From patchwork Tue Jul 18 22:33:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 9850087 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D98F060392 for ; Tue, 18 Jul 2017 22:37:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C4A25285CB for ; Tue, 18 Jul 2017 22:37:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B8CE6285D8; Tue, 18 Jul 2017 22:37:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, RCVD_IN_DNSWL_MED, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F199D285CB for ; Tue, 18 Jul 2017 22:37:29 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXb4i-0000pI-0G; Tue, 18 Jul 2017 22:34:40 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXb4g-0000mk-PH for xen-devel@lists.xenproject.org; Tue, 18 Jul 2017 22:34:38 +0000 Received: from [85.158.139.211] by server-13.bemta-5.messagelabs.com id C4/12-01732-EFC8E695; Tue, 18 Jul 2017 22:34:38 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrMIsWRWlGSWpSXmKPExsXiVRusqfunJy/ SoH2apcX3LZOZHBg9Dn+4whLAGMWamZeUX5HAmjFp+SHGgq/+FQc6ZzI2MK5z7mLk4hASmM4o 0dXUxwTisAi8YpH4s/IxO4gjIdDPKnFr/ivGLkZOICdO4vDsi6wQdqXEzu5j7CC2kICSxNYNS 5khRv1jlOjZsoYNJMEmoCWxp2E+2FgRgdkcEou7zrCAOAICH9kkbmw5DNbCLNDCJPHk3zYmkB ZhAVuJhhuLWUBsFgFVifYN18FsXgEriV2HZgA1cADtNpb4+0UfxOQECi+/zwNxhaVEb28DywR GwQWMDKsY1YtTi8pSi3Qt9JKKMtMzSnITM3N0DQ1M9XJTi4sT01NzEpOK9ZLzczcxAoOOAQh2 MB5sdj7EKMnBpCTKu1U5L1KILyk/pTIjsTgjvqg0J7X4EKMMB4eSBO+hbqCcYFFqempFWmYOM Pxh0hIcPEoivD4gad7igsTc4sx0iNQpRmOOSQe2f2HieDXh/zcmIZa8/LxUKXHeBSClAiClGa V5cINgcXmJUVZKmJcR6DQhnoLUotzMElT5V4ziHIxKwrzswCgX4snMK4Hb9wroFCagU4R9c0B OKUlESEk1MC6ZevR1esvKBKvXsaG+uVN/rxDaYay0ut7zr/2GY5om37sa2rOSOeLtbq4T/v1/ U1zokjcHk2uD8rsea0fyxh26ZLDK9ecau8kVz6/f+r1u/pYLOh8uROoIn3qi/+WNF/sJGWGeH W7mHy8v+FOwdYeauWX262/rHnWkrN+crfaqYEXWE89Q1j4lluKMREMt5qLiRACABjPPxgIAAA == X-Env-Sender: thgarnie@google.com X-Msg-Ref: server-10.tower-206.messagelabs.com!1500417275!75037227!1 X-Originating-IP: [74.125.83.41] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.4.25; banners=-,-,- X-VirusChecked: Checked Received: (qmail 63381 invoked from network); 18 Jul 2017 22:34:36 -0000 Received: from mail-pg0-f41.google.com (HELO mail-pg0-f41.google.com) (74.125.83.41) by server-10.tower-206.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 18 Jul 2017 22:34:36 -0000 Received: by mail-pg0-f41.google.com with SMTP id u5so19921302pgq.3 for ; Tue, 18 Jul 2017 15:34:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=DJaGCihN+qFf9HWA+KLNP4wKBs3LnoTuhcW3zdyy9I4=; b=V03abFzckek5wxVoftskhYzBGOb0qHdiS24tN5/SLSWl2+ng/6WzDO1zRkDpIUhpKp WQEOWeCjj4kovHvw94WXpDJeqenQQnoCtUpmDrEHU1lkRaU2R2EO72VdA66eKLrOe47K ev0VTYrPahd1OJJRFheG05lyNLdw6kFhmClx+8kXn2yqqIfAcJhWcsR3GNLEToEe1+Y9 b4/NxHtyK9nD0pAnq5E2Z3blODXLBvxvqH6/ssJ14ewclrtU10Zj9/Xaxj4xWm83zmy1 A7E4ZQDpxcLb9Lttdi1bdCtgW/XM2G9LlJ9kdxLIMQbTSx2aPVrIIAACynYMP83mM+Z9 Fq5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=DJaGCihN+qFf9HWA+KLNP4wKBs3LnoTuhcW3zdyy9I4=; b=k5VZduuRatOrN6dg9beYSG9ysg59VQcKRq2oUBJudizEmLzojlTz/R1yqc8+xkkTFX XbBpFrLUfRAUeFTe+j7bZrVkHTSgKyWKZH9710FDVlnmXCJY1u0pGUHfvZN5Z8Fz9w4P 3DrS0oUHDIJfPbEcTeGX3KY6SDskc2LR6ip0jeXubfurTdOjSSZCGg+tFPLWwwcczUFj A1+ndJ7DIjcXj4kqzv6xQOsyICkQaq1hkwTfhGZJ0L8fydYYevVoVMvegw/Iaww0XMDK I/RoxA7Bq1mBYyT6u5XrnBgcqAFrZutzdXvNwfG3W2hwG4g9mjeGAoSLqKoHGgZSLxJq CvXw== X-Gm-Message-State: AIVw111zYGtFTf2Nv0Kv+4jFBsZakv/gTIsyjKRoKMzfeB6dHz6x5oZi oDZ2m4GXXtT/qomY X-Received: by 10.99.238.83 with SMTP id n19mr3975551pgk.268.1500417274486; Tue, 18 Jul 2017 15:34:34 -0700 (PDT) Received: from skynet.sea.corp.google.com ([100.100.206.164]) by smtp.gmail.com with ESMTPSA id b6sm6444242pgn.67.2017.07.18.15.34.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 18 Jul 2017 15:34:33 -0700 (PDT) From: Thomas Garnier To: Herbert Xu , "David S . Miller" , Thomas Gleixner , Ingo Molnar , "H . Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Thomas Garnier , Arnd Bergmann , Matthias Kaehlcke , Boris Ostrovsky , Juergen Gross , Paolo Bonzini , =?UTF-8?q?Radim=20Kr=C4=8Dm=C3=A1=C5=99?= , Joerg Roedel , Andy Lutomirski , Borislav Petkov , "Kirill A . Shutemov" , Brian Gerst , Borislav Petkov , Christian Borntraeger , "Rafael J . Wysocki" , Len Brown , Pavel Machek , Tejun Heo , Christoph Lameter , Kees Cook , Paul Gortmaker , Chris Metcalf , "Paul E . McKenney" , Andrew Morton , Christopher Li , Dou Liyang , Masahiro Yamada , Daniel Borkmann , Markus Trippelsdorf , Peter Foley , Steven Rostedt , Tim Chen , Ard Biesheuvel , Catalin Marinas , Matthew Wilcox , Michal Hocko , Rob Landley , Jiri Kosina , "H . J . Lu" , Paul Bolle , Baoquan He , Daniel Micay Date: Tue, 18 Jul 2017 15:33:27 -0700 Message-Id: <20170718223333.110371-17-thgarnie@google.com> X-Mailer: git-send-email 2.13.2.932.g7449e964c-goog In-Reply-To: <20170718223333.110371-1-thgarnie@google.com> References: <20170718223333.110371-1-thgarnie@google.com> Cc: linux-arch@vger.kernel.org, kvm@vger.kernel.org, linux-pm@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, linux-sparse@vger.kernel.org, linux-crypto@vger.kernel.org, kernel-hardening@lists.openwall.com, xen-devel@lists.xenproject.org Subject: [Xen-devel] [RFC 16/22] x86/percpu: Adapt percpu for PIE support X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Perpcu uses a clever design where the .percu ELF section has a virtual address of zero and the relocation code avoid relocating specific symbols. It makes the code simple and easily adaptable with or without SMP support. This design is incompatible with PIE because generated code always try to access the zero virtual address relative to the default mapping address. It becomes impossible when KASLR is configured to go below -2G. This patch solves this problem by removing the zero mapping and adapting the GS base to be relative to the expected address. These changes are done only when PIE is enabled. The original implementation is kept as-is by default. The assembly and PER_CPU macros are changed to use relative references when PIE is enabled. The KALLSYMS_ABSOLUTE_PERCPU configuration is disabled with PIE given percpu symbols are not absolute in this case. Position Independent Executable (PIE) support will allow to extended the KASLR randomization range below the -2G memory limit. Signed-off-by: Thomas Garnier --- arch/x86/entry/entry_64.S | 4 ++-- arch/x86/include/asm/percpu.h | 25 +++++++++++++++++++------ arch/x86/kernel/cpu/common.c | 4 +++- arch/x86/kernel/head_64.S | 4 ++++ arch/x86/kernel/setup_percpu.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 13 +++++++++++-- arch/x86/lib/cmpxchg16b_emu.S | 8 ++++---- arch/x86/xen/xen-asm.S | 12 ++++++------ init/Kconfig | 2 +- 9 files changed, 51 insertions(+), 23 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 691c4755269b..be198c0a2a8c 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -388,7 +388,7 @@ ENTRY(__switch_to_asm) #ifdef CONFIG_CC_STACKPROTECTOR movq TASK_stack_canary(%rsi), %rbx - movq %rbx, PER_CPU_VAR(irq_stack_union)+stack_canary_offset + movq %rbx, PER_CPU_VAR(irq_stack_union + stack_canary_offset) #endif /* restore callee-saved registers */ @@ -739,7 +739,7 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt /* * Exception entry points. */ -#define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss) + (TSS_ist + ((x) - 1) * 8) +#define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss + (TSS_ist + ((x) - 1) * 8)) .macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 ENTRY(\sym) diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 9fa03604b2b3..862eb771f0e5 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -4,9 +4,11 @@ #ifdef CONFIG_X86_64 #define __percpu_seg gs #define __percpu_mov_op movq +#define __percpu_rel (%rip) #else #define __percpu_seg fs #define __percpu_mov_op movl +#define __percpu_rel #endif #ifdef __ASSEMBLY__ @@ -27,10 +29,14 @@ #define PER_CPU(var, reg) \ __percpu_mov_op %__percpu_seg:this_cpu_off, reg; \ lea var(reg), reg -#define PER_CPU_VAR(var) %__percpu_seg:var +/* Compatible with Position Independent Code */ +#define PER_CPU_VAR(var) %__percpu_seg:(var)##__percpu_rel +/* Rare absolute reference */ +#define PER_CPU_VAR_ABS(var) %__percpu_seg:var #else /* ! SMP */ #define PER_CPU(var, reg) __percpu_mov_op $var, reg -#define PER_CPU_VAR(var) var +#define PER_CPU_VAR(var) (var)##__percpu_rel +#define PER_CPU_VAR_ABS(var) var #endif /* SMP */ #ifdef CONFIG_X86_64_SMP @@ -208,27 +214,34 @@ do { \ pfo_ret__; \ }) +/* Position Independent code uses relative addresses only */ +#ifdef CONFIG_X86_PIE +#define __percpu_stable_arg __percpu_arg(a1) +#else +#define __percpu_stable_arg __percpu_arg(P1) +#endif + #define percpu_stable_op(op, var) \ ({ \ typeof(var) pfo_ret__; \ switch (sizeof(var)) { \ case 1: \ - asm(op "b "__percpu_arg(P1)",%0" \ + asm(op "b "__percpu_stable_arg ",%0" \ : "=q" (pfo_ret__) \ : "p" (&(var))); \ break; \ case 2: \ - asm(op "w "__percpu_arg(P1)",%0" \ + asm(op "w "__percpu_stable_arg ",%0" \ : "=r" (pfo_ret__) \ : "p" (&(var))); \ break; \ case 4: \ - asm(op "l "__percpu_arg(P1)",%0" \ + asm(op "l "__percpu_stable_arg ",%0" \ : "=r" (pfo_ret__) \ : "p" (&(var))); \ break; \ case 8: \ - asm(op "q "__percpu_arg(P1)",%0" \ + asm(op "q "__percpu_stable_arg ",%0" \ : "=r" (pfo_ret__) \ : "p" (&(var))); \ break; \ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index b95cd94ca97b..31300767ec0f 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -480,7 +480,9 @@ void load_percpu_segment(int cpu) loadsegment(fs, __KERNEL_PERCPU); #else __loadsegment_simple(gs, 0); - wrmsrl(MSR_GS_BASE, (unsigned long)per_cpu(irq_stack_union.gs_base, cpu)); + wrmsrl(MSR_GS_BASE, + (unsigned long)per_cpu(irq_stack_union.gs_base, cpu) - + (unsigned long)__per_cpu_start); #endif load_stack_canary_segment(); } diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 7e4f7a83a15a..4d0a7e68bfe8 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -256,7 +256,11 @@ ENDPROC(start_cpu0) GLOBAL(initial_code) .quad x86_64_start_kernel GLOBAL(initial_gs) +#ifdef CONFIG_X86_PIE + .quad 0 +#else .quad INIT_PER_CPU_VAR(irq_stack_union) +#endif GLOBAL(initial_stack) /* * The SIZEOF_PTREGS gap is a convention which helps the in-kernel diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c index 10edd1e69a68..ce1c58a29def 100644 --- a/arch/x86/kernel/setup_percpu.c +++ b/arch/x86/kernel/setup_percpu.c @@ -25,7 +25,7 @@ DEFINE_PER_CPU_READ_MOSTLY(int, cpu_number); EXPORT_PER_CPU_SYMBOL(cpu_number); -#ifdef CONFIG_X86_64 +#if defined(CONFIG_X86_64) && !defined(CONFIG_X86_PIE) #define BOOT_PERCPU_OFFSET ((unsigned long)__per_cpu_load) #else #define BOOT_PERCPU_OFFSET 0 diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index c8a3b61be0aa..77f1b0622539 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -183,9 +183,14 @@ SECTIONS /* * percpu offsets are zero-based on SMP. PERCPU_VADDR() changes the * output PHDR, so the next output section - .init.text - should - * start another segment - init. + * start another segment - init. For Position Independent Code, the + * per-cpu section cannot be zero-based because everything is relative. */ +#ifdef CONFIG_X86_PIE + PERCPU_SECTION(INTERNODE_CACHE_BYTES) +#else PERCPU_VADDR(INTERNODE_CACHE_BYTES, 0, :percpu) +#endif ASSERT(SIZEOF(.data..percpu) < CONFIG_PHYSICAL_START, "per-CPU data too large - increase CONFIG_PHYSICAL_START") #endif @@ -361,7 +366,11 @@ SECTIONS * Per-cpu symbols which need to be offset from __per_cpu_load * for the boot processor. */ +#ifdef CONFIG_X86_PIE +#define INIT_PER_CPU(x) init_per_cpu__##x = x +#else #define INIT_PER_CPU(x) init_per_cpu__##x = x + __per_cpu_load +#endif INIT_PER_CPU(gdt_page); INIT_PER_CPU(irq_stack_union); @@ -371,7 +380,7 @@ INIT_PER_CPU(irq_stack_union); . = ASSERT((_end - _text <= KERNEL_IMAGE_SIZE), "kernel image bigger than KERNEL_IMAGE_SIZE"); -#ifdef CONFIG_SMP +#if defined(CONFIG_SMP) && !defined(CONFIG_X86_PIE) . = ASSERT((irq_stack_union == 0), "irq_stack_union is not at start of per-cpu area"); #endif diff --git a/arch/x86/lib/cmpxchg16b_emu.S b/arch/x86/lib/cmpxchg16b_emu.S index 9b330242e740..254950604ae4 100644 --- a/arch/x86/lib/cmpxchg16b_emu.S +++ b/arch/x86/lib/cmpxchg16b_emu.S @@ -33,13 +33,13 @@ ENTRY(this_cpu_cmpxchg16b_emu) pushfq cli - cmpq PER_CPU_VAR((%rsi)), %rax + cmpq PER_CPU_VAR_ABS((%rsi)), %rax jne .Lnot_same - cmpq PER_CPU_VAR(8(%rsi)), %rdx + cmpq PER_CPU_VAR_ABS(8(%rsi)), %rdx jne .Lnot_same - movq %rbx, PER_CPU_VAR((%rsi)) - movq %rcx, PER_CPU_VAR(8(%rsi)) + movq %rbx, PER_CPU_VAR_ABS((%rsi)) + movq %rcx, PER_CPU_VAR_ABS(8(%rsi)) popfq mov $1, %al diff --git a/arch/x86/xen/xen-asm.S b/arch/x86/xen/xen-asm.S index eff224df813f..40410969fd3c 100644 --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -26,7 +26,7 @@ ENTRY(xen_irq_enable_direct) FRAME_BEGIN /* Unmask events */ - movb $0, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + movb $0, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) /* * Preempt here doesn't matter because that will deal with any @@ -35,7 +35,7 @@ ENTRY(xen_irq_enable_direct) */ /* Test for pending */ - testb $0xff, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_pending + testb $0xff, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_pending) jz 1f 2: call check_events @@ -52,7 +52,7 @@ ENDPATCH(xen_irq_enable_direct) * non-zero. */ ENTRY(xen_irq_disable_direct) - movb $1, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + movb $1, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) ENDPATCH(xen_irq_disable_direct) ret ENDPROC(xen_irq_disable_direct) @@ -68,7 +68,7 @@ ENDPATCH(xen_irq_disable_direct) * x86 use opposite senses (mask vs enable). */ ENTRY(xen_save_fl_direct) - testb $0xff, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + testb $0xff, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) setz %ah addb %ah, %ah ENDPATCH(xen_save_fl_direct) @@ -91,7 +91,7 @@ ENTRY(xen_restore_fl_direct) #else testb $X86_EFLAGS_IF>>8, %ah #endif - setz PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + setz PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) /* * Preempt here doesn't matter because that will deal with any * pending interrupts. The pending check may end up being run @@ -99,7 +99,7 @@ ENTRY(xen_restore_fl_direct) */ /* check for unmasked and pending */ - cmpw $0x0001, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_pending + cmpw $0x0001, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_pending) jnz 1f 2: call check_events 1: diff --git a/init/Kconfig b/init/Kconfig index 8514b25db21c..4fb5d6fc2c4f 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1201,7 +1201,7 @@ config KALLSYMS_ALL config KALLSYMS_ABSOLUTE_PERCPU bool depends on KALLSYMS - default X86_64 && SMP + default X86_64 && SMP && !X86_PIE config KALLSYMS_BASE_RELATIVE bool