From patchwork Fri Apr 28 09:50:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Wenlong X-Patchwork-Id: 13226176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 30896C77B7C for ; Fri, 28 Apr 2023 09:57:24 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.527220.819575 (Exim 4.92) (envelope-from ) id 1psKql-0001nM-61; Fri, 28 Apr 2023 09:57:11 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 527220.819575; Fri, 28 Apr 2023 09:57:11 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKql-0001nF-37; Fri, 28 Apr 2023 09:57:11 +0000 Received: by outflank-mailman (input) for mailman id 527220; Fri, 28 Apr 2023 09:52:42 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKmQ-0001fa-6g for xen-devel@lists.xenproject.org; Fri, 28 Apr 2023 09:52:42 +0000 Received: from out0-199.mail.aliyun.com (out0-199.mail.aliyun.com [140.205.0.199]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 66dc1d4b-e5aa-11ed-8611-37d641c3527e; Fri, 28 Apr 2023 11:52:37 +0200 (CEST) Received: from localhost(mailfrom:houwenlong.hwl@antgroup.com fp:SMTPD_---.STCEPV9_1682675548) by smtp.aliyun-inc.com; Fri, 28 Apr 2023 17:52:29 +0800 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 66dc1d4b-e5aa-11ed-8611-37d641c3527e X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R181e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018047198;MF=houwenlong.hwl@antgroup.com;NM=1;PH=DS;RN=17;SR=0;TI=SMTPD_---.STCEPV9_1682675548; From: "Hou Wenlong" To: linux-kernel@vger.kernel.org Cc: "Thomas Garnier" , "Lai Jiangshan" , "Kees Cook" , "Hou Wenlong" , "Juergen Gross" , "Boris Ostrovsky" , "Darren Hart" , "Andy Shevchenko" , "Thomas Gleixner" , "Ingo Molnar" , "Borislav Petkov" , "Dave Hansen" , , "H. Peter Anvin" , , Subject: [PATCH RFC 15/43] x86/PVH: Use fixed_percpu_data to set up GS base Date: Fri, 28 Apr 2023 17:50:55 +0800 Message-Id: <4fdb800ce6f1a2315918cb02eec3efbec1032cb8.1682673543.git.houwenlong.hwl@antgroup.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 startup_64() and startup_xen() both use fixed_percpu_data to set up GS base. So for consitency, use it too in PVH entry. Signed-off-by: Hou Wenlong Cc: Thomas Garnier Cc: Lai Jiangshan Cc: Kees Cook --- arch/x86/platform/pvh/head.S | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/arch/x86/platform/pvh/head.S b/arch/x86/platform/pvh/head.S index c4365a05ab83..b093996b7e19 100644 --- a/arch/x86/platform/pvh/head.S +++ b/arch/x86/platform/pvh/head.S @@ -96,7 +96,7 @@ SYM_CODE_START_LOCAL(pvh_start_xen) 1: /* Set base address in stack canary descriptor. */ mov $MSR_GS_BASE,%ecx - mov $_pa(canary), %eax + mov $_pa(INIT_PER_CPU_VAR(fixed_percpu_data)), %eax xor %edx, %edx wrmsr @@ -156,8 +156,6 @@ SYM_DATA_START_LOCAL(gdt_start) SYM_DATA_END_LABEL(gdt_start, SYM_L_LOCAL, gdt_end) .balign 16 -SYM_DATA_LOCAL(canary, .fill 48, 1, 0) - SYM_DATA_START_LOCAL(early_stack) .fill BOOT_STACK_SIZE, 1, 0 SYM_DATA_END_LABEL(early_stack, SYM_L_LOCAL, early_stack_end) From patchwork Fri Apr 28 09:50:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Wenlong X-Patchwork-Id: 13226179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 28FB2C77B7F for ; Fri, 28 Apr 2023 09:57:27 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.527222.819580 (Exim 4.92) (envelope-from ) id 1psKql-0001pY-DD; Fri, 28 Apr 2023 09:57:11 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 527222.819580; Fri, 28 Apr 2023 09:57:11 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKql-0001oz-9g; Fri, 28 Apr 2023 09:57:11 +0000 Received: by outflank-mailman (input) for mailman id 527222; Fri, 28 Apr 2023 09:52:52 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKma-0001fa-Sr for xen-devel@lists.xenproject.org; Fri, 28 Apr 2023 09:52:52 +0000 Received: from out0-214.mail.aliyun.com (out0-214.mail.aliyun.com [140.205.0.214]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 6ebca779-e5aa-11ed-8611-37d641c3527e; Fri, 28 Apr 2023 11:52:50 +0200 (CEST) Received: from localhost(mailfrom:houwenlong.hwl@antgroup.com fp:SMTPD_---.STFQGKD_1682675559) by smtp.aliyun-inc.com; Fri, 28 Apr 2023 17:52:41 +0800 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 6ebca779-e5aa-11ed-8611-37d641c3527e X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018047194;MF=houwenlong.hwl@antgroup.com;NM=1;PH=DS;RN=35;SR=0;TI=SMTPD_---.STFQGKD_1682675559; From: "Hou Wenlong" To: linux-kernel@vger.kernel.org Cc: "Thomas Garnier" , "Lai Jiangshan" , "Kees Cook" , "Hou Wenlong" , "Brian Gerst" , "Thomas Gleixner" , "Ingo Molnar" , "Borislav Petkov" , "Dave Hansen" , , "H. Peter Anvin" , "Andy Lutomirski" , "Juergen Gross" , "Boris Ostrovsky" , "Darren Hart" , "Andy Shevchenko" , "Nathan Chancellor" , "Nick Desaulniers" , "Tom Rix" , "Peter Zijlstra" , " =?utf-8?q?Mike_Rapoport_=28IBM?= =?utf-8?q?=29?= " , "Ashok Raj" , "Rick Edgecombe" , "Catalin Marinas" , "Guo Ren" , "Greg Kroah-Hartman" , "Jason A. Donenfeld" , "Pawan Gupta" , "Kim Phillips" , "David Woodhouse" , "Josh Poimboeuf" , , , Subject: [PATCH RFC 16/43] x86-64: Use per-cpu stack canary if supported by compiler Date: Fri, 28 Apr 2023 17:50:56 +0800 Message-Id: <7cee0c83225ffd8cf8fd0065bea9348f6db3b12a.1682673543.git.houwenlong.hwl@antgroup.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 From: Brian Gerst From: Brian Gerst If the compiler supports it, use a standard per-cpu variable for the stack protector instead of the old fixed location. Keep the fixed location code for compatibility with older compilers. [Hou Wenlong: Disable it on Clang, adapt new code change and adapt missing GS set up path in pvh_start_xen()] Signed-off-by: Brian Gerst Co-developed-by: Hou Wenlong Signed-off-by: Hou Wenlong Cc: Thomas Garnier Cc: Lai Jiangshan Cc: Kees Cook --- arch/x86/Kconfig | 12 ++++++++++++ arch/x86/Makefile | 21 ++++++++++++++------- arch/x86/entry/entry_64.S | 6 +++++- arch/x86/include/asm/processor.h | 17 ++++++++++++----- arch/x86/include/asm/stackprotector.h | 16 +++++++--------- arch/x86/kernel/asm-offsets_64.c | 2 +- arch/x86/kernel/cpu/common.c | 15 +++++++-------- arch/x86/kernel/head_64.S | 16 ++++++++++------ arch/x86/kernel/vmlinux.lds.S | 4 +++- arch/x86/platform/pvh/head.S | 8 ++++++++ arch/x86/xen/xen-head.S | 14 +++++++++----- 11 files changed, 88 insertions(+), 43 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 68e5da464b96..55cce8cdf9bd 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -410,6 +410,18 @@ config CC_HAS_SANE_STACKPROTECTOR the compiler produces broken code or if it does not let us control the segment on 32-bit kernels. +config CC_HAS_CUSTOMIZED_STACKPROTECTOR + bool + # Although clang supports -mstack-protector-guard-reg option, it + # would generate GOT reference for __stack_chk_guard even with + # -fno-PIE flag. + default y if (!CC_IS_CLANG && $(cc-option,-mstack-protector-guard-reg=gs)) + +config STACKPROTECTOR_FIXED + bool + depends on X86_64 && STACKPROTECTOR + default !CC_HAS_CUSTOMIZED_STACKPROTECTOR + menu "Processor type and features" config SMP diff --git a/arch/x86/Makefile b/arch/x86/Makefile index b39975977c03..57e4dbbf501d 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -111,13 +111,7 @@ ifeq ($(CONFIG_X86_32),y) # temporary until string.h is fixed KBUILD_CFLAGS += -ffreestanding - ifeq ($(CONFIG_STACKPROTECTOR),y) - ifeq ($(CONFIG_SMP),y) - KBUILD_CFLAGS += -mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard - else - KBUILD_CFLAGS += -mstack-protector-guard=global - endif - endif + percpu_seg := fs else BITS := 64 UTS_MACHINE := x86_64 @@ -167,6 +161,19 @@ else KBUILD_CFLAGS += -mcmodel=kernel KBUILD_RUSTFLAGS += -Cno-redzone=y KBUILD_RUSTFLAGS += -Ccode-model=kernel + + percpu_seg := gs +endif + +ifeq ($(CONFIG_STACKPROTECTOR),y) + ifneq ($(CONFIG_STACKPROTECTOR_FIXED),y) + ifeq ($(CONFIG_SMP),y) + KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg) \ + -mstack-protector-guard-symbol=__stack_chk_guard + else + KBUILD_CFLAGS += -mstack-protector-guard=global + endif + endif endif # diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 6f2297ebb15f..df79b7aa65bb 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -229,6 +229,10 @@ SYM_INNER_LABEL(entry_SYSRETQ_end, SYM_L_GLOBAL) int3 SYM_CODE_END(entry_SYSCALL_64) +#ifdef CONFIG_STACKPROTECTOR_FIXED +#define __stack_chk_guard fixed_percpu_data + FIXED_stack_canary +#endif + /* * %rdi: prev task * %rsi: next task @@ -252,7 +256,7 @@ SYM_FUNC_START(__switch_to_asm) #ifdef CONFIG_STACKPROTECTOR movq TASK_stack_canary(%rsi), %rbx - movq %rbx, PER_CPU_VAR(fixed_percpu_data) + FIXED_stack_canary + movq %rbx, PER_CPU_VAR(__stack_chk_guard) #endif /* diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 2a5ec5750ba7..3890f609569d 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -379,6 +379,8 @@ struct irq_stack { } __aligned(IRQ_STACK_SIZE); #ifdef CONFIG_X86_64 + +#ifdef CONFIG_STACKPROTECTOR_FIXED struct fixed_percpu_data { /* * GCC hardcodes the stack canary as %gs:40. Since the @@ -394,21 +396,26 @@ struct fixed_percpu_data { DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible; DECLARE_INIT_PER_CPU(fixed_percpu_data); +#endif /* CONFIG_STACKPROTECTOR_FIXED */ static inline unsigned long cpu_kernelmode_gs_base(int cpu) { +#ifdef CONFIG_STACKPROTECTOR_FIXED return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu); +#else +#ifdef CONFIG_SMP + return per_cpu_offset(cpu); +#else + return 0; +#endif +#endif } extern asmlinkage void ignore_sysret(void); /* Save actual FS/GS selectors and bases to current->thread */ void current_save_fsgs(void); -#else /* X86_64 */ -#ifdef CONFIG_STACKPROTECTOR -DECLARE_PER_CPU(unsigned long, __stack_chk_guard); -#endif -#endif /* !X86_64 */ +#endif /* X86_64 */ struct perf_event; diff --git a/arch/x86/include/asm/stackprotector.h b/arch/x86/include/asm/stackprotector.h index 00473a650f51..24aa0e2ad0dd 100644 --- a/arch/x86/include/asm/stackprotector.h +++ b/arch/x86/include/asm/stackprotector.h @@ -36,6 +36,12 @@ #include +#ifdef CONFIG_STACKPROTECTOR_FIXED +#define __stack_chk_guard fixed_percpu_data.stack_canary +#else +DECLARE_PER_CPU(unsigned long, __stack_chk_guard); +#endif + /* * Initialize the stackprotector canary value. * @@ -51,25 +57,17 @@ static __always_inline void boot_init_stack_canary(void) { unsigned long canary = get_random_canary(); -#ifdef CONFIG_X86_64 +#ifdef CONFIG_STACKPROTECTOR_FIXED BUILD_BUG_ON(offsetof(struct fixed_percpu_data, stack_canary) != 40); #endif current->stack_canary = canary; -#ifdef CONFIG_X86_64 - this_cpu_write(fixed_percpu_data.stack_canary, canary); -#else this_cpu_write(__stack_chk_guard, canary); -#endif } static inline void cpu_init_stack_canary(int cpu, struct task_struct *idle) { -#ifdef CONFIG_X86_64 - per_cpu(fixed_percpu_data.stack_canary, cpu) = idle->stack_canary; -#else per_cpu(__stack_chk_guard, cpu) = idle->stack_canary; -#endif } #else /* STACKPROTECTOR */ diff --git a/arch/x86/kernel/asm-offsets_64.c b/arch/x86/kernel/asm-offsets_64.c index bb65371ea9df..f39baf90126c 100644 --- a/arch/x86/kernel/asm-offsets_64.c +++ b/arch/x86/kernel/asm-offsets_64.c @@ -56,7 +56,7 @@ int main(void) BLANK(); -#ifdef CONFIG_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR_FIXED OFFSET(FIXED_stack_canary, fixed_percpu_data, stack_canary); BLANK(); #endif diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 3ea06b0b4570..972b1babf731 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2051,10 +2051,6 @@ DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) = { EXPORT_PER_CPU_SYMBOL(pcpu_hot); #ifdef CONFIG_X86_64 -DEFINE_PER_CPU_FIRST(struct fixed_percpu_data, - fixed_percpu_data) __aligned(PAGE_SIZE) __visible; -EXPORT_PER_CPU_SYMBOL_GPL(fixed_percpu_data); - static void wrmsrl_cstar(unsigned long val) { /* @@ -2102,15 +2098,18 @@ void syscall_init(void) X86_EFLAGS_IOPL|X86_EFLAGS_NT|X86_EFLAGS_RF| X86_EFLAGS_AC|X86_EFLAGS_ID); } - -#else /* CONFIG_X86_64 */ +#endif /* CONFIG_X86_64 */ #ifdef CONFIG_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR_FIXED +DEFINE_PER_CPU_FIRST(struct fixed_percpu_data, + fixed_percpu_data) __aligned(PAGE_SIZE) __visible; +EXPORT_PER_CPU_SYMBOL_GPL(fixed_percpu_data); +#else DEFINE_PER_CPU(unsigned long, __stack_chk_guard); EXPORT_PER_CPU_SYMBOL(__stack_chk_guard); #endif - -#endif /* CONFIG_X86_64 */ +#endif /* * Clear all 6 debug registers: diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 21f0556d3ac0..61f1873d0ff7 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -68,7 +68,13 @@ SYM_CODE_START_NOALIGN(startup_64) /* Setup GSBASE to allow stack canary access for C code */ movl $MSR_GS_BASE, %ecx +#if defined(CONFIG_STACKPROTECTOR_FIXED) leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#elif defined(CONFIG_SMP) + movabs $__per_cpu_load, %rdx +#else + xorl %edx, %edx +#endif movl %edx, %eax shrq $32, %rdx wrmsr @@ -283,16 +289,14 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) movl %eax,%fs movl %eax,%gs - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + /* + * Set up GS base. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx -#ifndef CONFIG_SMP - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#if !defined(CONFIG_SMP) && defined(CONFIG_STACKPROTECTOR_FIXED) + leaq __per_cpu_load(%rip), %rdx #endif movl %edx, %eax shrq $32, %rdx diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 25f155205770..f02dcde9f8a8 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -500,12 +500,14 @@ SECTIONS */ #define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load INIT_PER_CPU(gdt_page); -INIT_PER_CPU(fixed_percpu_data); INIT_PER_CPU(irq_stack_backing_store); +#ifdef CONFIG_STACKPROTECTOR_FIXED +INIT_PER_CPU(fixed_percpu_data); #ifdef CONFIG_SMP . = ASSERT((fixed_percpu_data == 0), "fixed_percpu_data is not at start of per-cpu area"); #endif +#endif #endif /* CONFIG_X86_64 */ diff --git a/arch/x86/platform/pvh/head.S b/arch/x86/platform/pvh/head.S index b093996b7e19..5842fe0e4f96 100644 --- a/arch/x86/platform/pvh/head.S +++ b/arch/x86/platform/pvh/head.S @@ -96,8 +96,16 @@ SYM_CODE_START_LOCAL(pvh_start_xen) 1: /* Set base address in stack canary descriptor. */ mov $MSR_GS_BASE,%ecx +#if defined(CONFIG_STACKPROTECTOR_FIXED) mov $_pa(INIT_PER_CPU_VAR(fixed_percpu_data)), %eax xor %edx, %edx +#elif defined(CONFIG_SMP) + mov $__per_cpu_load, %rax + cdq +#else + xor %eax, %eax + xor %edx, %edx +#endif wrmsr call xen_prepare_pvh diff --git a/arch/x86/xen/xen-head.S b/arch/x86/xen/xen-head.S index 643d02900fbb..09eaf59e8066 100644 --- a/arch/x86/xen/xen-head.S +++ b/arch/x86/xen/xen-head.S @@ -51,15 +51,19 @@ SYM_CODE_START(startup_xen) leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + /* + * Set up GS base. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx - movq $INIT_PER_CPU_VAR(fixed_percpu_data),%rax +#if defined(CONFIG_STACKPROTECTOR_FIXED) + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#elif defined(CONFIG_SMP) + movabs $__per_cpu_load, %rdx +#else + xorl %eax, %eax +#endif cdq wrmsr From patchwork Fri Apr 28 09:50:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Wenlong X-Patchwork-Id: 13226177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 59595C77B61 for ; Fri, 28 Apr 2023 09:57:24 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.527224.819586 (Exim 4.92) (envelope-from ) id 1psKql-0001vW-P6; Fri, 28 Apr 2023 09:57:11 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 527224.819586; Fri, 28 Apr 2023 09:57:11 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKql-0001uC-Gr; Fri, 28 Apr 2023 09:57:11 +0000 Received: by outflank-mailman (input) for mailman id 527224; Fri, 28 Apr 2023 09:52:59 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKmh-0001fa-9d for xen-devel@lists.xenproject.org; Fri, 28 Apr 2023 09:52:59 +0000 Received: from out0-197.mail.aliyun.com (out0-197.mail.aliyun.com [140.205.0.197]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 7280a1d6-e5aa-11ed-8611-37d641c3527e; Fri, 28 Apr 2023 11:52:57 +0200 (CEST) Received: from localhost(mailfrom:houwenlong.hwl@antgroup.com fp:SMTPD_---.STCEPaY_1682675567) by smtp.aliyun-inc.com; Fri, 28 Apr 2023 17:52:48 +0800 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7280a1d6-e5aa-11ed-8611-37d641c3527e X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018047192;MF=houwenlong.hwl@antgroup.com;NM=1;PH=DS;RN=32;SR=0;TI=SMTPD_---.STCEPaY_1682675567; From: "Hou Wenlong" To: linux-kernel@vger.kernel.org Cc: "Thomas Garnier" , "Lai Jiangshan" , "Kees Cook" , "Hou Wenlong" , "Andy Lutomirski" , "Thomas Gleixner" , "Ingo Molnar" , "Borislav Petkov" , "Dave Hansen" , , "H. Peter Anvin" , "Peter Zijlstra" , "Josh Poimboeuf" , "Pawan Gupta" , "Dennis Zhou" , "Tejun Heo" , "Christoph Lameter" , "Paolo Bonzini" , "Wanpeng Li" , "Vitaly Kuznetsov" , "Juergen Gross" , "Boris Ostrovsky" , "Nathan Chancellor" , "Nick Desaulniers" , "Tom Rix" , "David Woodhouse" , "Brian Gerst" , , , , Subject: [PATCH RFC 18/43] x86/percpu: Use PC-relative addressing for percpu variable references Date: Fri, 28 Apr 2023 17:50:58 +0800 Message-Id: <175116f75c38c15d8d73a03301eab805fea13a0a.1682673543.git.houwenlong.hwl@antgroup.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 For PIE binary, all symbol references are PC-relative addressing, even for percpu variable. So to keep compatible with PIE, add %rip suffix in percpu assembly macros if PIE is enabled. However, relocation of percpu variable references is broken now for PIE. It would be fixed later. Suggested-by: Lai Jiangshan Signed-off-by: Hou Wenlong Cc: Thomas Garnier Cc: Kees Cook --- arch/x86/entry/calling.h | 17 ++++++++++++---- arch/x86/include/asm/nospec-branch.h | 10 +++++----- arch/x86/include/asm/percpu.h | 29 +++++++++++++++++++++++++--- arch/x86/kernel/head_64.S | 2 +- arch/x86/kernel/kvm.c | 21 ++++++++++++++++---- arch/x86/lib/cmpxchg16b_emu.S | 8 ++++---- arch/x86/xen/xen-asm.S | 10 +++++----- 7 files changed, 71 insertions(+), 26 deletions(-) diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h index f6907627172b..11328578741d 100644 --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -173,7 +173,7 @@ For 32-bit we have the following conventions - kernel is built with .endm #define THIS_CPU_user_pcid_flush_mask \ - PER_CPU_VAR(cpu_tlbstate) + TLB_STATE_user_pcid_flush_mask + PER_CPU_VAR(cpu_tlbstate + TLB_STATE_user_pcid_flush_mask) .macro SWITCH_TO_USER_CR3_NOSTACK scratch_reg:req scratch_reg2:req ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI @@ -370,8 +370,8 @@ For 32-bit we have the following conventions - kernel is built with .endm .macro SAVE_AND_SET_GSBASE scratch_reg:req save_reg:req + GET_PERCPU_BASE \scratch_reg \save_reg rdgsbase \save_reg - GET_PERCPU_BASE \scratch_reg wrgsbase \scratch_reg .endm @@ -407,15 +407,24 @@ For 32-bit we have the following conventions - kernel is built with * Thus the kernel would consume a guest's TSC_AUX if an NMI arrives * while running KVM's run loop. */ -.macro GET_PERCPU_BASE reg:req +#ifdef CONFIG_X86_PIE +.macro GET_PERCPU_BASE reg:req scratch_reg:req + LOAD_CPU_AND_NODE_SEG_LIMIT \reg + andq $VDSO_CPUNODE_MASK, \reg + leaq __per_cpu_offset(%rip), \scratch_reg + movq (\scratch_reg, \reg, 8), \reg +.endm +#else +.macro GET_PERCPU_BASE reg:req scratch_reg:req LOAD_CPU_AND_NODE_SEG_LIMIT \reg andq $VDSO_CPUNODE_MASK, \reg movq __per_cpu_offset(, \reg, 8), \reg .endm +#endif /* CONFIG_X86_PIE */ #else -.macro GET_PERCPU_BASE reg:req +.macro GET_PERCPU_BASE reg:req scratch_reg:req movq pcpu_unit_offsets(%rip), \reg .endm diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index edb2b0cb8efe..d8fd935e0697 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -59,13 +59,13 @@ #ifdef CONFIG_CALL_THUNKS_DEBUG # define CALL_THUNKS_DEBUG_INC_CALLS \ - incq %gs:__x86_call_count; + incq %gs:(__x86_call_count)__percpu_rel; # define CALL_THUNKS_DEBUG_INC_RETS \ - incq %gs:__x86_ret_count; + incq %gs:(__x86_ret_count)__percpu_rel; # define CALL_THUNKS_DEBUG_INC_STUFFS \ - incq %gs:__x86_stuffs_count; + incq %gs:(__x86_stuffs_count)__percpu_rel; # define CALL_THUNKS_DEBUG_INC_CTXSW \ - incq %gs:__x86_ctxsw_count; + incq %gs:(__x86_ctxsw_count)__percpu_rel; #else # define CALL_THUNKS_DEBUG_INC_CALLS # define CALL_THUNKS_DEBUG_INC_RETS @@ -95,7 +95,7 @@ CALL_THUNKS_DEBUG_INC_CALLS #define INCREMENT_CALL_DEPTH \ - sarq $5, %gs:pcpu_hot + X86_call_depth; \ + sarq $5, %gs:(pcpu_hot + X86_call_depth)__percpu_rel;\ CALL_THUNKS_DEBUG_INC_CALLS #define ASM_INCREMENT_CALL_DEPTH \ diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 13c0d63ed55e..a627a073c6ea 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -4,16 +4,26 @@ #ifdef CONFIG_X86_64 #define __percpu_seg gs +#ifdef CONFIG_X86_PIE +#define __percpu_rel (%rip) +#else +#define __percpu_rel +#endif /* CONFIG_X86_PIE */ #else #define __percpu_seg fs +#define __percpu_rel #endif #ifdef __ASSEMBLY__ #ifdef CONFIG_SMP -#define PER_CPU_VAR(var) %__percpu_seg:var +/* Compatible with Position Independent Code */ +#define PER_CPU_VAR(var) %__percpu_seg:(var)##__percpu_rel +/* Rare absolute reference */ +#define PER_CPU_VAR_ABS(var) %__percpu_seg:var #else /* ! SMP */ -#define PER_CPU_VAR(var) var +#define PER_CPU_VAR(var) (var)##__percpu_rel +#define PER_CPU_VAR_ABS(var) var #endif /* SMP */ #ifdef CONFIG_X86_64_SMP @@ -148,10 +158,23 @@ do { \ (typeof(_var))(unsigned long) pfo_val__; \ }) +/* + * Position Independent code uses relative addresses only. + * The 'P' modifier prevents RIP-relative addressing in GCC, + * so use 'a' modifier instead. Howerver, 'P' modifier allows + * RIP-relative addressing in Clang but Clang doesn't support + * 'a' modifier. + */ +#if defined(CONFIG_X86_PIE) && defined(CONFIG_CC_IS_GCC) +#define __percpu_stable_arg __percpu_arg(a[var]) +#else +#define __percpu_stable_arg __percpu_arg(P[var]) +#endif + #define percpu_stable_op(size, op, _var) \ ({ \ __pcpu_type_##size pfo_val__; \ - asm(__pcpu_op2_##size(op, __percpu_arg(P[var]), "%[val]") \ + asm(__pcpu_op2_##size(op, __percpu_stable_arg, "%[val]") \ : [val] __pcpu_reg_##size("=", pfo_val__) \ : [var] "p" (&(_var))); \ (typeof(_var))(unsigned long) pfo_val__; \ diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 61f1873d0ff7..1eed50b7d1ac 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -396,7 +396,7 @@ SYM_CODE_START(start_cpu0) UNWIND_HINT_END_OF_STACK /* Find the idle task stack */ - movq PER_CPU_VAR(pcpu_hot) + X86_current_task, %rcx + movq PER_CPU_VAR(pcpu_hot + X86_current_task), %rcx movq TASK_threadsp(%rcx), %rsp jmp .Ljump_to_C_code diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 1cceac5984da..32d7b201f4f0 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -794,14 +794,27 @@ PV_CALLEE_SAVE_REGS_THUNK(__kvm_vcpu_is_preempted); extern bool __raw_callee_save___kvm_vcpu_is_preempted(long); +#ifndef CONFIG_X86_PIE +#define KVM_CHECK_VCPU_PREEMPTED \ + "movq __per_cpu_offset(,%rdi,8), %rax;" \ + "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax);" +#else +#define KVM_CHECK_VCPU_PREEMPTED \ + "pushq %rdi;" \ + "leaq __per_cpu_offset(%rip), %rax;" \ + "movq (%rax,%rdi,8), %rax;" \ + "leaq steal_time(%rip), %rdi;" \ + "cmpb $0, (%rax, %rdi, 1);" \ + "popq %rdi;" +#endif + /* * Hand-optimize version for x86-64 to avoid 8 64-bit register saving and * restoring to/from the stack. */ -#define PV_VCPU_PREEMPTED_ASM \ - "movq __per_cpu_offset(,%rdi,8), %rax\n\t" \ - "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \ - "setne %al\n\t" +#define PV_VCPU_PREEMPTED_ASM \ + KVM_CHECK_VCPU_PREEMPTED \ + "setne %al\n\t" DEFINE_PARAVIRT_ASM(__raw_callee_save___kvm_vcpu_is_preempted, PV_VCPU_PREEMPTED_ASM, .text); diff --git a/arch/x86/lib/cmpxchg16b_emu.S b/arch/x86/lib/cmpxchg16b_emu.S index 33c70c0160ea..891c5e9fd868 100644 --- a/arch/x86/lib/cmpxchg16b_emu.S +++ b/arch/x86/lib/cmpxchg16b_emu.S @@ -27,13 +27,13 @@ SYM_FUNC_START(this_cpu_cmpxchg16b_emu) pushfq cli - cmpq PER_CPU_VAR((%rsi)), %rax + cmpq PER_CPU_VAR_ABS((%rsi)), %rax jne .Lnot_same - cmpq PER_CPU_VAR(8(%rsi)), %rdx + cmpq PER_CPU_VAR_ABS(8(%rsi)), %rdx jne .Lnot_same - movq %rbx, PER_CPU_VAR((%rsi)) - movq %rcx, PER_CPU_VAR(8(%rsi)) + movq %rbx, PER_CPU_VAR_ABS((%rsi)) + movq %rcx, PER_CPU_VAR_ABS(8(%rsi)) popfq mov $1, %al diff --git a/arch/x86/xen/xen-asm.S b/arch/x86/xen/xen-asm.S index 9e5e68008785..448958ddbaf8 100644 --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -28,7 +28,7 @@ * non-zero. */ SYM_FUNC_START(xen_irq_disable_direct) - movb $1, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + movb $1, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) RET SYM_FUNC_END(xen_irq_disable_direct) @@ -69,7 +69,7 @@ SYM_FUNC_END(check_events) SYM_FUNC_START(xen_irq_enable_direct) FRAME_BEGIN /* Unmask events */ - movb $0, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + movb $0, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) /* * Preempt here doesn't matter because that will deal with any @@ -78,7 +78,7 @@ SYM_FUNC_START(xen_irq_enable_direct) */ /* Test for pending */ - testb $0xff, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_pending + testb $0xff, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_pending) jz 1f call check_events @@ -97,7 +97,7 @@ SYM_FUNC_END(xen_irq_enable_direct) * x86 use opposite senses (mask vs enable). */ SYM_FUNC_START(xen_save_fl_direct) - testb $0xff, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + testb $0xff, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) setz %ah addb %ah, %ah RET @@ -113,7 +113,7 @@ SYM_FUNC_END(xen_read_cr2); SYM_FUNC_START(xen_read_cr2_direct) FRAME_BEGIN - _ASM_MOV PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_arch_cr2, %_ASM_AX + _ASM_MOV PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_arch_cr2), %_ASM_AX FRAME_END RET SYM_FUNC_END(xen_read_cr2_direct); From patchwork Fri Apr 28 09:51:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Wenlong X-Patchwork-Id: 13226174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7F1EBC77B60 for ; Fri, 28 Apr 2023 09:57:22 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.527226.819591 (Exim 4.92) (envelope-from ) id 1psKql-00021c-WD; Fri, 28 Apr 2023 09:57:12 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 527226.819591; Fri, 28 Apr 2023 09:57:11 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKql-0001z7-QA; Fri, 28 Apr 2023 09:57:11 +0000 Received: by outflank-mailman (input) for mailman id 527226; Fri, 28 Apr 2023 09:53:35 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKnH-0001iA-Dx for xen-devel@lists.xenproject.org; Fri, 28 Apr 2023 09:53:35 +0000 Received: from out0-200.mail.aliyun.com (out0-200.mail.aliyun.com [140.205.0.200]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 88079f65-e5aa-11ed-8611-37d641c3527e; Fri, 28 Apr 2023 11:53:33 +0200 (CEST) Received: from localhost(mailfrom:houwenlong.hwl@antgroup.com fp:SMTPD_---.STFoGYl_1682675602) by smtp.aliyun-inc.com; Fri, 28 Apr 2023 17:53:23 +0800 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 88079f65-e5aa-11ed-8611-37d641c3527e X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R211e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018047212;MF=houwenlong.hwl@antgroup.com;NM=1;PH=DS;RN=17;SR=0;TI=SMTPD_---.STFoGYl_1682675602; From: "Hou Wenlong" To: linux-kernel@vger.kernel.org Cc: "Thomas Garnier" , "Lai Jiangshan" , "Kees Cook" , "Hou Wenlong" , "Juergen Gross" , "Boris Ostrovsky" , "Darren Hart" , "Andy Shevchenko" , "Thomas Gleixner" , "Ingo Molnar" , "Borislav Petkov" , "Dave Hansen" , , "H. Peter Anvin" , , Subject: [PATCH RFC 29/43] x86/PVH: Adapt PVH booting for PIE support Date: Fri, 28 Apr 2023 17:51:09 +0800 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 If PIE is enabled, all symbol references would be RIP-relative. However, PVH booting runs in low address space, which could cause wrong x86_init callbacks assignment. Since init_top_pgt has building high kernel address mapping, let PVH booting runs in high address space to make all things right. PVH booting assumes that no relocation happened. Since the kernel compile address is still in top 2G, so it is allowed to use R_X86_64_32S for symbol references in pvh_start_xen(). Signed-off-by: Hou Wenlong Cc: Thomas Garnier Cc: Lai Jiangshan Cc: Kees Cook --- arch/x86/platform/pvh/head.S | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/arch/x86/platform/pvh/head.S b/arch/x86/platform/pvh/head.S index 5842fe0e4f96..09518d4de042 100644 --- a/arch/x86/platform/pvh/head.S +++ b/arch/x86/platform/pvh/head.S @@ -94,6 +94,13 @@ SYM_CODE_START_LOCAL(pvh_start_xen) /* 64-bit entry point. */ .code64 1: +#ifdef CONFIG_X86_PIE + movabs $2f, %rax + ANNOTATE_RETPOLINE_SAFE + jmp *%rax +2: + ANNOTATE_NOENDBR // above +#endif /* Set base address in stack canary descriptor. */ mov $MSR_GS_BASE,%ecx #if defined(CONFIG_STACKPROTECTOR_FIXED) @@ -149,9 +156,15 @@ SYM_CODE_END(pvh_start_xen) .section ".init.data","aw" .balign 8 SYM_DATA_START_LOCAL(gdt) + /* + * Use an ASM_PTR (quad on x64) for _pa(gdt_start) because PIE requires + * a pointer size storage value before applying the relocation. On + * 32-bit _ASM_PTR will be a long which is aligned the space needed for + * relocation. + */ .word gdt_end - gdt_start - .long _pa(gdt_start) - .word 0 + _ASM_PTR _pa(gdt_start) + .balign 8 SYM_DATA_END(gdt) SYM_DATA_START_LOCAL(gdt_start) .quad 0x0000000000000000 /* NULL descriptor */ From patchwork Fri Apr 28 09:51:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Wenlong X-Patchwork-Id: 13226178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7C033C7EE22 for ; Fri, 28 Apr 2023 09:57:25 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.527229.819607 (Exim 4.92) (envelope-from ) id 1psKqm-0002Fa-OS; Fri, 28 Apr 2023 09:57:12 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 527229.819607; Fri, 28 Apr 2023 09:57:12 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKqm-0002DX-DO; Fri, 28 Apr 2023 09:57:12 +0000 Received: by outflank-mailman (input) for mailman id 527229; Fri, 28 Apr 2023 09:54:00 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKng-0001in-Rn for xen-devel@lists.xenproject.org; Fri, 28 Apr 2023 09:54:00 +0000 Received: from out0-193.mail.aliyun.com (out0-193.mail.aliyun.com [140.205.0.193]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 9675234d-e5aa-11ed-b224-6b7b168915f2; Fri, 28 Apr 2023 11:53:57 +0200 (CEST) Received: from localhost(mailfrom:houwenlong.hwl@antgroup.com fp:SMTPD_---.STFoGgp_1682675627) by smtp.aliyun-inc.com; Fri, 28 Apr 2023 17:53:48 +0800 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 9675234d-e5aa-11ed-b224-6b7b168915f2 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R721e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018047188;MF=houwenlong.hwl@antgroup.com;NM=1;PH=DS;RN=24;SR=0;TI=SMTPD_---.STFoGgp_1682675627; From: "Hou Wenlong" To: linux-kernel@vger.kernel.org Cc: "Thomas Garnier" , "Lai Jiangshan" , "Kees Cook" , "Hou Wenlong" , "Andy Lutomirski" , "Thomas Gleixner" , "Ingo Molnar" , "Borislav Petkov" , "Dave Hansen" , , "H. Peter Anvin" , "Juergen Gross" , " =?utf-8?q?Srivatsa_S=2E_Bhat_=28VMware=29?= " , "Alexey Makhalov" , "VMware PV-Drivers Reviewers" , "Boris Ostrovsky" , "Andrew Morton" , " =?utf-8?q?Mike_Rapoport_=28I?= =?utf-8?q?BM=29?= " , "Liam R. Howlett" , "Suren Baghdasaryan" , "Kirill A. Shutemov" , , Subject: [PATCH RFC 36/43] x86/vsyscall: Don't use set_fixmap() to map vsyscall page Date: Fri, 28 Apr 2023 17:51:16 +0800 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 In order to unify FIXADDR_TOP for x86 and allow fixmap area to be moveable, vsyscall page should be mapped individually. However, for XENPV guest, vsyscall page needs to be mapped into user pagetable too. So introduce a new PVMMU op to help to map vsyscall page. Suggested-by: Lai Jiangshan Signed-off-by: Hou Wenlong Cc: Thomas Garnier Cc: Kees Cook --- arch/x86/entry/vsyscall/vsyscall_64.c | 3 +-- arch/x86/include/asm/paravirt.h | 7 +++++++ arch/x86/include/asm/paravirt_types.h | 4 ++++ arch/x86/include/asm/vsyscall.h | 13 +++++++++++++ arch/x86/kernel/paravirt.c | 4 ++++ arch/x86/xen/mmu_pv.c | 20 ++++++++++++++------ 6 files changed, 43 insertions(+), 8 deletions(-) diff --git a/arch/x86/entry/vsyscall/vsyscall_64.c b/arch/x86/entry/vsyscall/vsyscall_64.c index e0ca8120aea8..4373460ebbde 100644 --- a/arch/x86/entry/vsyscall/vsyscall_64.c +++ b/arch/x86/entry/vsyscall/vsyscall_64.c @@ -385,8 +385,7 @@ void __init map_vsyscall(void) * page. */ if (vsyscall_mode == EMULATE) { - __set_fixmap(VSYSCALL_PAGE, physaddr_vsyscall, - PAGE_KERNEL_VVAR); + __set_vsyscall_page(physaddr_vsyscall, PAGE_KERNEL_VVAR); set_vsyscall_pgtable_user_bits(swapper_pg_dir); } diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index 2350ceb43db0..dcc0706287ee 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -576,6 +576,13 @@ static inline void __set_fixmap(unsigned /* enum fixed_addresses */ idx, { pv_ops.mmu.set_fixmap(idx, phys, flags); } + +#ifdef CONFIG_X86_VSYSCALL_EMULATION +static inline void __set_vsyscall_page(phys_addr_t phys, pgprot_t flags) +{ + pv_ops.mmu.set_vsyscall_page(phys, flags); +} +#endif #endif #if defined(CONFIG_SMP) && defined(CONFIG_PARAVIRT_SPINLOCKS) diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h index 982a234f5a06..e79f38232849 100644 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -224,6 +224,10 @@ struct pv_mmu_ops { an mfn. We can tell which is which from the index. */ void (*set_fixmap)(unsigned /* enum fixed_addresses */ idx, phys_addr_t phys, pgprot_t flags); + +#ifdef CONFIG_X86_VSYSCALL_EMULATION + void (*set_vsyscall_page)(phys_addr_t phys, pgprot_t flags); +#endif #endif } __no_randomize_layout; diff --git a/arch/x86/include/asm/vsyscall.h b/arch/x86/include/asm/vsyscall.h index ab60a71a8dcb..73691fc60924 100644 --- a/arch/x86/include/asm/vsyscall.h +++ b/arch/x86/include/asm/vsyscall.h @@ -2,6 +2,7 @@ #ifndef _ASM_X86_VSYSCALL_H #define _ASM_X86_VSYSCALL_H +#include #include #include @@ -15,6 +16,18 @@ extern void set_vsyscall_pgtable_user_bits(pgd_t *root); */ extern bool emulate_vsyscall(unsigned long error_code, struct pt_regs *regs, unsigned long address); +static inline void native_set_vsyscall_page(phys_addr_t phys, pgprot_t flags) +{ + pgprot_val(flags) &= __default_kernel_pte_mask; + set_pte_vaddr(VSYSCALL_ADDR, pfn_pte(phys >> PAGE_SHIFT, flags)); +} + +#ifndef CONFIG_PARAVIRT_XXL +#define __set_vsyscall_page native_set_vsyscall_page +#else +#include +#endif + #else static inline void map_vsyscall(void) {} static inline bool emulate_vsyscall(unsigned long error_code, diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index ac10b46c5832..13c81402f377 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -33,6 +33,7 @@ #include #include #include +#include /* * nop stub, which must not clobber anything *including the stack* to @@ -357,6 +358,9 @@ struct paravirt_patch_template pv_ops = { }, .mmu.set_fixmap = native_set_fixmap, +#ifdef CONFIG_X86_VSYSCALL_EMULATION + .mmu.set_vsyscall_page = native_set_vsyscall_page, +#endif #endif /* CONFIG_PARAVIRT_XXL */ #if defined(CONFIG_PARAVIRT_SPINLOCKS) diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index fdc91deece7e..a59bc013ee5b 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -59,6 +59,7 @@ #include #include +#include #include #include #include @@ -2020,9 +2021,6 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot) switch (idx) { case FIX_BTMAP_END ... FIX_BTMAP_BEGIN: -#ifdef CONFIG_X86_VSYSCALL_EMULATION - case VSYSCALL_PAGE: -#endif /* All local page mappings */ pte = pfn_pte(phys, prot); break; @@ -2058,14 +2056,21 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot) vaddr = __fix_to_virt(idx); if (HYPERVISOR_update_va_mapping(vaddr, pte, UVMF_INVLPG)) BUG(); +} #ifdef CONFIG_X86_VSYSCALL_EMULATION +static void xen_set_vsyscall_page(phys_addr_t phys, pgprot_t prot) +{ + pte_t pte = pfn_pte(phys >> PAGE_SHIFT, prot); + + if (HYPERVISOR_update_va_mapping(VSYSCALL_ADDR, pte, UVMF_INVLPG)) + BUG(); + /* Replicate changes to map the vsyscall page into the user pagetable vsyscall mapping. */ - if (idx == VSYSCALL_PAGE) - set_pte_vaddr_pud(level3_user_vsyscall, vaddr, pte); -#endif + set_pte_vaddr_pud(level3_user_vsyscall, VSYSCALL_ADDR, pte); } +#endif static void __init xen_post_allocator_init(void) { @@ -2156,6 +2161,9 @@ static const typeof(pv_ops) xen_mmu_ops __initconst = { }, .set_fixmap = xen_set_fixmap, +#ifdef CONFIG_X86_VSYSCALL_EMULATION + .set_vsyscall_page = xen_set_vsyscall_page, +#endif }, }; From patchwork Fri Apr 28 09:51:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Wenlong X-Patchwork-Id: 13226175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3B477C77B7F for ; Fri, 28 Apr 2023 09:57:24 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.527228.819598 (Exim 4.92) (envelope-from ) id 1psKqm-00028d-B6; Fri, 28 Apr 2023 09:57:12 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 527228.819598; Fri, 28 Apr 2023 09:57:12 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKqm-00025t-31; Fri, 28 Apr 2023 09:57:12 +0000 Received: by outflank-mailman (input) for mailman id 527228; Fri, 28 Apr 2023 09:54:00 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1psKng-0001in-KE for xen-devel@lists.xenproject.org; Fri, 28 Apr 2023 09:54:00 +0000 Received: from out0-208.mail.aliyun.com (out0-208.mail.aliyun.com [140.205.0.208]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 96c1f67c-e5aa-11ed-b224-6b7b168915f2; Fri, 28 Apr 2023 11:53:57 +0200 (CEST) Received: from localhost(mailfrom:houwenlong.hwl@antgroup.com fp:SMTPD_---.STFoGhB_1682675630) by smtp.aliyun-inc.com; Fri, 28 Apr 2023 17:53:50 +0800 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 96c1f67c-e5aa-11ed-b224-6b7b168915f2 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018047206;MF=houwenlong.hwl@antgroup.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---.STFoGhB_1682675630; From: "Hou Wenlong" To: linux-kernel@vger.kernel.org Cc: "Thomas Garnier" , "Lai Jiangshan" , "Kees Cook" , "Hou Wenlong" , "Juergen Gross" , "Boris Ostrovsky" , "Thomas Gleixner" , "Ingo Molnar" , "Borislav Petkov" , "Dave Hansen" , , "H. Peter Anvin" , Subject: [PATCH RFC 37/43] x86/xen: Pin up to VSYSCALL_ADDR when vsyscall page is out of fixmap area Date: Fri, 28 Apr 2023 17:51:17 +0800 Message-Id: <13975abd9b8b2e2e1e2efd3be6c341542b08af24.1682673543.git.houwenlong.hwl@antgroup.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 If vsyscall page is moved out of fixmap area, then FIXADDR_TOP would be below vsyscall page. So it should pin up to VSYSCALL_ADDR if vsyscall is enabled. Suggested-by: Lai Jiangshan Signed-off-by: Hou Wenlong Cc: Thomas Garnier Cc: Kees Cook --- arch/x86/xen/mmu_pv.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index a59bc013ee5b..28392f3478a0 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -587,6 +587,12 @@ static void xen_p4d_walk(struct mm_struct *mm, p4d_t *p4d, xen_pud_walk(mm, pud, func, last, limit); } +#ifdef CONFIG_X86_VSYSCALL_EMULATION +#define __KERNEL_MAP_TOP (VSYSCALL_ADDR + PAGE_SIZE) +#else +#define __KERNEL_MAP_TOP FIXADDR_TOP +#endif + /* * (Yet another) pagetable walker. This one is intended for pinning a * pagetable. This means that it walks a pagetable and calls the @@ -594,7 +600,7 @@ static void xen_p4d_walk(struct mm_struct *mm, p4d_t *p4d, * at every level. It walks the entire pagetable, but it only bothers * pinning pte pages which are below limit. In the normal case this * will be STACK_TOP_MAX, but at boot we need to pin up to - * FIXADDR_TOP. + * __KERNEL_MAP_TOP. * * We must skip the Xen hole in the middle of the address space, just after * the big x86-64 virtual hole. @@ -609,7 +615,7 @@ static void __xen_pgd_walk(struct mm_struct *mm, pgd_t *pgd, /* The limit is the last byte to be touched */ limit--; - BUG_ON(limit >= FIXADDR_TOP); + BUG_ON(limit >= __KERNEL_MAP_TOP); /* * 64-bit has a great big hole in the middle of the address @@ -797,7 +803,7 @@ static void __init xen_after_bootmem(void) #ifdef CONFIG_X86_VSYSCALL_EMULATION SetPagePinned(virt_to_page(level3_user_vsyscall)); #endif - xen_pgd_walk(&init_mm, xen_mark_pinned, FIXADDR_TOP); + xen_pgd_walk(&init_mm, xen_mark_pinned, __KERNEL_MAP_TOP); } static void xen_unpin_page(struct mm_struct *mm, struct page *page,