From patchwork Tue Apr 25 22:56:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 9700047 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id DA8AB60224 for ; Tue, 25 Apr 2017 22:57:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BC0D42839C for ; Tue, 25 Apr 2017 22:57:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B06502848E; Tue, 25 Apr 2017 22:57:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id DD2882839C for ; Tue, 25 Apr 2017 22:57:07 +0000 (UTC) Received: (qmail 9745 invoked by uid 550); 25 Apr 2017 22:56:57 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 9669 invoked from network); 25 Apr 2017 22:56:55 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hYujhtfHHdKB781FtqR6d9CpCYuobUhy4lT7v7ggp/8=; b=eSM1CQKJoxQUSMzlxt3/KmOHACxH6aRIP10kiPRMflOiMmg8MNQjG+7GFndqvTbK1n gITy6sZGD05+SQ/qPf23NB7fTLOqlmnSWV5Sjt2Cem3OjAnOYartISe0KwpN+QBV9ZSQ plVaaIf4dICI3nEV8OY+cWmj308yYeO0LD9HA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hYujhtfHHdKB781FtqR6d9CpCYuobUhy4lT7v7ggp/8=; b=qswx+1uvctyDj492M4KXP0uZzoIWGQ+CmSpWUT9uC/MbX+Cjz8HyBs3tSaafT+IRQL kQws77JwyxNbmwQq+KFcvGTCVr2iC0TmyrCxWVW6TYgZr3hJXL1AeJvqWAgb4xZ4cupG nnqdgCFZkLuSyYvsUvR/tLdmbsT4mEsAK7vFMKup7rrDa18wPO0X6u1P3wNJEPUmzKQR cKM0dgjrZVUC+TDfLPjeAtXhxCU1eMCVDl5/jPMsCcmSksXUM9W/WjUS26pOTmhcbxKa wszixOzBMtFmJx6NmeQY+HiqBAZWe8bKUpeP/coFT44QHKKOeSV7oa+C6x5Ir4hD5AMZ Hrzg== X-Gm-Message-State: AN3rC/7LHuQ9AUcKvUVV8V5RoSZxS+srk9lsaqLSBeh+NVDC33/uwA3d 5WkJsOBq9svvLUDp X-Received: by 10.36.93.4 with SMTP id w4mr8052368ita.32.1493161003736; Tue, 25 Apr 2017 15:56:43 -0700 (PDT) From: Kees Cook To: linux-kernel@vger.kernel.org Cc: Kees Cook , Peter Zijlstra , PaX Team , Jann Horn , Eric Biggers , Christoph Hellwig , "axboe@kernel.dk" , James Bottomley , Elena Reshetova , Hans Liljestrand , David Windsor , "x86@kernel.org" , Ingo Molnar , Arnd Bergmann , Greg Kroah-Hartman , "David S. Miller" , Rik van Riel , linux-arch , "kernel-hardening@lists.openwall.com" Date: Tue, 25 Apr 2017 15:56:37 -0700 Message-Id: <1493160997-126108-3-git-send-email-keescook@chromium.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1493160997-126108-1-git-send-email-keescook@chromium.org> References: <1493160997-126108-1-git-send-email-keescook@chromium.org> Subject: [kernel-hardening] [PATCH v2 2/2] x86, refcount: Implement fast refcount overflow protection X-Virus-Scanned: ClamAV using ClamSMTP This protection is a modified version of the x86 PAX_REFCOUNT implementation from PaX/grsecurity. This speeds up the refcount_t API by duplicating the existing atomic_t implementation with a single instruction added to detect if the refcount has wrapped past INT_MAX (or below 0) resulting in a signed value. Note that this protection is only meaningful for the overflow case, as that can be detected and stopped before the reference is freed and left to be abused by an attacker. Catching the "inc from 0" case is nice to have, but only indicates that a use-after-free has already happened. Such notifications are likely avoidable by an attacker that has already exploited a use-after-free vulnerability. With this overflow protection, the use-after-free cannot happen in the first place, avoiding the vulnerability entirely. On overflow detection (actually "signed value" detection), the offending process is killed, a report is generated, and the refcount value is reset to INT_MAX. This allows the system to attempt to keep operating. Another option, not done in this patch, would be to reset the counter to (INT_MIN / 2) to trap all future refcount inc or dec actions. Yet another option would be to choose (INT_MAX - N) with some small N to provide some headroom for legitimate users of the reference counter. On the matter of races, since the entire range beyond INT_MAX but before 0 is signed, every inc will trap, leaving no overflow-only race condition. As for performance, this implementation adds a single "js" instruction to a copy of the regular atomic_t operations, making this comparable to the existing atomic_t operations. The detection routine uses a combination of an alternative section exception handler and trap to return back to C for handling the error condition with minimal increase in text size. Various differences from PaX: - applied only to refcount_t, not atomic_t - rebased to -next - reorganized refcount error handler and section declaration locations - uses "js" instead of "jo" to trap all signed results instead of just under/overflow transitions Signed-off-by: Kees Cook --- arch/Kconfig | 19 ++++++++ arch/x86/Kconfig | 1 + arch/x86/entry/entry_32.S | 9 ++++ arch/x86/entry/entry_64.S | 3 ++ arch/x86/include/asm/irq_vectors.h | 3 ++ arch/x86/include/asm/refcount.h | 97 ++++++++++++++++++++++++++++++++++++++ arch/x86/include/asm/traps.h | 5 ++ arch/x86/kernel/traps.c | 44 ++++++++++++++++- include/asm-generic/sections.h | 4 ++ include/asm-generic/vmlinux.lds.h | 9 ++++ include/linux/kernel.h | 2 + include/linux/refcount.h | 4 ++ kernel/panic.c | 25 ++++++++++ lib/refcount.c | 5 +- 14 files changed, 228 insertions(+), 2 deletions(-) create mode 100644 arch/x86/include/asm/refcount.h diff --git a/arch/Kconfig b/arch/Kconfig index 6c00e5b00f8b..3983737fd919 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -867,4 +867,23 @@ config STRICT_MODULE_RWX config ARCH_WANT_RELAX_ORDER bool +config ARCH_HAS_FAST_REFCOUNT + bool + help + An architecture selects this when it has implemented refcount_t + using primitizes that provide a faster runtime at the expense + of some refcount state checks. The refcount overflow condition, + however, must be retained. Catching overflows is the primary + security concern for protecting against bugs in reference counts. + +config FAST_REFCOUNT + bool "Speed up reference counting at the expense of full validation" + depends on ARCH_HAS_FAST_REFCOUNT + help + The regular reference counting infrastructure in the kernel checks + many error conditions. If this option is selected, refcounting + is made faster using architecture-specific implementions that may + only check for reference count overflows (which is the most common + way reference counting bugs are turned into security exploits). + source "kernel/gcov/Kconfig" diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index a694d0002758..a1bbb09ae667 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -50,6 +50,7 @@ config X86 select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_ELF_RANDOMIZE select ARCH_HAS_FAST_MULTIPLIER + select ARCH_HAS_FAST_REFCOUNT select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_KCOV if X86_64 select ARCH_HAS_MMIO_FLUSH diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 50bc26949e9e..bba69761ec24 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -789,6 +789,15 @@ ENTRY(spurious_interrupt_bug) jmp common_exception END(spurious_interrupt_bug) +#ifdef CONFIG_FAST_REFCOUNT +ENTRY(refcount_error) + ASM_CLAC + pushl $0 + pushl $do_refcount_error + jmp common_exception +ENDPROC(refcount_error) +#endif + #ifdef CONFIG_XEN ENTRY(xen_hypervisor_callback) pushl $-1 /* orig_ax = -1 => not a system call */ diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 607d72c4a485..783045d3887c 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -855,6 +855,9 @@ idtentry coprocessor_error do_coprocessor_error has_error_code=0 idtentry alignment_check do_alignment_check has_error_code=1 idtentry simd_coprocessor_error do_simd_coprocessor_error has_error_code=0 +#ifdef CONFIG_FAST_REFCOUNT +idtentry refcount_error do_refcount_error has_error_code=0 +#endif /* * Reload gs selector with exception handling diff --git a/arch/x86/include/asm/irq_vectors.h b/arch/x86/include/asm/irq_vectors.h index 6ca9fd6234e1..64ca4dcc29ec 100644 --- a/arch/x86/include/asm/irq_vectors.h +++ b/arch/x86/include/asm/irq_vectors.h @@ -48,6 +48,9 @@ #define IA32_SYSCALL_VECTOR 0x80 +/* Refcount Overflow or Underflow Exception. */ +#define X86_REFCOUNT_VECTOR 0x81 + /* * Vectors 0x30-0x3f are used for ISA interrupts. * round up to the next 16-vector boundary diff --git a/arch/x86/include/asm/refcount.h b/arch/x86/include/asm/refcount.h new file mode 100644 index 000000000000..3d3125717154 --- /dev/null +++ b/arch/x86/include/asm/refcount.h @@ -0,0 +1,97 @@ +#ifndef __ASM_X86_REFCOUNT_H +#define __ASM_X86_REFCOUNT_H +/* + * x86-specific implementation of refcount_t. Ported from PAX_REFCOUNT in + * PaX/grsecurity and changed to use "js" instead of "jo" to trap on all + * signed results, not just when overflowing. + */ +#include +#include + +#define __REFCOUNT_EXCEPTION(size) \ + ".if "__stringify(size)" == 4\n\t" \ + ".pushsection .text.refcount_overflow\n" \ + ".elseif "__stringify(size)" == -4\n\t" \ + ".pushsection .text.refcount_underflow\n" \ + ".else\n" \ + ".error \"invalid size\"\n" \ + ".endif\n" \ + "111:\tlea %[counter],%%"_ASM_CX"\n\t" \ + "int $"__stringify(X86_REFCOUNT_VECTOR)"\n" \ + "222:\n\t" \ + ".popsection\n" \ + "333:\n" \ + _ASM_EXTABLE(222b, 333b) + +#define __REFCOUNT_CHECK(size) \ + "js 111f\n" \ + __REFCOUNT_EXCEPTION(size) + +#define __REFCOUNT_ERROR(size) \ + "jmp 111f\n" \ + __REFCOUNT_EXCEPTION(size) + +#define REFCOUNT_CHECK_OVERFLOW(size) __REFCOUNT_CHECK(size) +#define REFCOUNT_CHECK_UNDERFLOW(size) __REFCOUNT_CHECK(-(size)) + +static __always_inline void refcount_add(unsigned int i, refcount_t *r) +{ + asm volatile(LOCK_PREFIX "addl %1,%0\n\t" + REFCOUNT_CHECK_OVERFLOW(4) + : [counter] "+m" (r->refs.counter) + : "ir" (i) + : "cc", "cx"); +} + +static __always_inline void refcount_inc(refcount_t *r) +{ + asm volatile(LOCK_PREFIX "incl %0\n\t" + REFCOUNT_CHECK_OVERFLOW(4) + : [counter] "+m" (r->refs.counter) + : : "cc", "cx"); +} + +static __always_inline void refcount_dec(refcount_t *r) +{ + asm volatile(LOCK_PREFIX "decl %0\n\t" + REFCOUNT_CHECK_UNDERFLOW(4) + : [counter] "+m" (r->refs.counter) + : : "cc", "cx"); +} + +static __always_inline __must_check +bool refcount_sub_and_test(unsigned int i, refcount_t *r) +{ + GEN_BINARY_SUFFIXED_RMWcc(LOCK_PREFIX "subl", + REFCOUNT_CHECK_UNDERFLOW(4), r->refs.counter, + "er", i, "%0", e); +} + +static __always_inline __must_check bool refcount_dec_and_test(refcount_t *r) +{ + GEN_UNARY_SUFFIXED_RMWcc(LOCK_PREFIX "decl", + REFCOUNT_CHECK_UNDERFLOW(4), r->refs.counter, + "%0", e); +} + +static __always_inline __must_check bool refcount_inc_not_zero(refcount_t *r) +{ + int c; + + c = atomic_read(&(r->refs)); + do { + if (unlikely(c <= 0)) + break; + } while (!atomic_try_cmpxchg(&(r->refs), &c, c + 1)); + + /* Did we start or finish in an undesirable state? */ + if (unlikely(c <= 0 || c + 1 < 0)) { + asm volatile(__REFCOUNT_ERROR(4) + : : [counter] "m" (r->refs.counter) + : "cc", "cx"); + } + + return c != 0; +} + +#endif diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 01fd0a7f48cd..e4d8db75d85e 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -38,6 +38,10 @@ asmlinkage void machine_check(void); #endif /* CONFIG_X86_MCE */ asmlinkage void simd_coprocessor_error(void); +#ifdef CONFIG_FAST_REFCOUNT +asmlinkage void refcount_error(void); +#endif + #ifdef CONFIG_TRACING asmlinkage void trace_page_fault(void); #define trace_stack_segment stack_segment @@ -54,6 +58,7 @@ asmlinkage void trace_page_fault(void); #define trace_alignment_check alignment_check #define trace_simd_coprocessor_error simd_coprocessor_error #define trace_async_page_fault async_page_fault +#define trace_refcount_error refcount_error #endif dotraplinkage void do_divide_error(struct pt_regs *, long); diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 3995d3a777d4..4b1c318c96ff 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -218,8 +218,13 @@ do_trap_no_signal(struct task_struct *tsk, int trapnr, char *str, } if (!user_mode(regs)) { - if (fixup_exception(regs, trapnr)) + if (fixup_exception(regs, trapnr)) { + if (IS_ENABLED(CONFIG_FAST_REFCOUNT) && + trapnr == X86_REFCOUNT_VECTOR) + refcount_error_report(regs, str); + return 0; + } if (fixup_bug(regs, trapnr)) return 0; @@ -342,6 +347,38 @@ __visible void __noreturn handle_stack_overflow(const char *message, } #endif +#ifdef CONFIG_FAST_REFCOUNT + +dotraplinkage void do_refcount_error(struct pt_regs *regs, long error_code) +{ + const char *str = NULL; + + BUG_ON(!(regs->flags & X86_EFLAGS_SF)); + +#define range_check(size, dir, type, value) \ + do { \ + if ((unsigned long)__##size##_##dir##_start <= regs->ip && \ + regs->ip < (unsigned long)__##size##_##dir##_end) { \ + *(type *)regs->cx = (value); \ + str = #size " " #dir; \ + } \ + } while (0) + + /* + * Reset to INT_MAX in both cases to attempt to let system + * continue operating. + */ + range_check(refcount, overflow, int, INT_MAX); + range_check(refcount, underflow, int, INT_MAX); + +#undef range_check + + BUG_ON(!str); + do_error_trap(regs, error_code, (char *)str, X86_REFCOUNT_VECTOR, + SIGILL); +} +#endif + #ifdef CONFIG_X86_64 /* Runs on IST stack */ dotraplinkage void do_double_fault(struct pt_regs *regs, long error_code) @@ -1017,6 +1054,11 @@ void __init trap_init(void) set_bit(IA32_SYSCALL_VECTOR, used_vectors); #endif +#ifdef CONFIG_FAST_REFCOUNT + set_intr_gate(X86_REFCOUNT_VECTOR, refcount_error); + set_bit(X86_REFCOUNT_VECTOR, used_vectors); +#endif + /* * Set the IDT descriptor to a fixed read-only location, so that the * "sidt" instruction will not leak the location of the kernel, and diff --git a/include/asm-generic/sections.h b/include/asm-generic/sections.h index 532372c6cf15..0590f384f234 100644 --- a/include/asm-generic/sections.h +++ b/include/asm-generic/sections.h @@ -20,6 +20,8 @@ * may be out of this range on some architectures. * [_sinittext, _einittext]: contains .init.text.* sections * [__bss_start, __bss_stop]: contains BSS sections + * [__refcount_overflow/underflow_start, ..._end]: contains .text sections + * for refcount error handling. * * Following global variables are optional and may be unavailable on some * architectures and/or kernel configurations. @@ -39,6 +41,8 @@ extern char __per_cpu_load[], __per_cpu_start[], __per_cpu_end[]; extern char __kprobes_text_start[], __kprobes_text_end[]; extern char __entry_text_start[], __entry_text_end[]; extern char __start_rodata[], __end_rodata[]; +extern char __refcount_overflow_start[], __refcount_overflow_end[]; +extern char __refcount_underflow_start[], __refcount_underflow_end[]; /* Start and end of .ctors section - used for constructor calls. */ extern char __ctors_start[], __ctors_end[]; diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 314a0b9219c6..9b94c3f1e5ec 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -446,9 +446,18 @@ ALIGN_FUNCTION(); \ *(.text.hot .text .text.fixup .text.unlikely) \ *(.ref.text) \ + REFCOUNT_TEXT \ MEM_KEEP(init.text) \ MEM_KEEP(exit.text) \ +#define __REFCOUNT_TEXT(section) \ + VMLINUX_SYMBOL(__##section##_start) = .; \ + *(.text.##section) \ + VMLINUX_SYMBOL(__##section##_end) = .; + +#define REFCOUNT_TEXT \ + __REFCOUNT_TEXT(refcount_overflow) \ + __REFCOUNT_TEXT(refcount_underflow) /* sched.text is aling to function alignment to secure we have same * address even at second ld pass when generating System.map */ diff --git a/include/linux/kernel.h b/include/linux/kernel.h index 13bc08aba704..94f87d5642e4 100644 --- a/include/linux/kernel.h +++ b/include/linux/kernel.h @@ -276,6 +276,8 @@ extern int oops_may_print(void); void do_exit(long error_code) __noreturn; void complete_and_exit(struct completion *, long) __noreturn; +void refcount_error_report(struct pt_regs *regs, const char *kind); + /* Internal, do not use. */ int __must_check _kstrtoul(const char *s, unsigned int base, unsigned long *res); int __must_check _kstrtol(const char *s, unsigned int base, long *res); diff --git a/include/linux/refcount.h b/include/linux/refcount.h index b34aa649d204..d09ad4e91e55 100644 --- a/include/linux/refcount.h +++ b/include/linux/refcount.h @@ -41,6 +41,9 @@ static inline unsigned int refcount_read(const refcount_t *r) return atomic_read(&r->refs); } +#ifdef CONFIG_FAST_REFCOUNT +#include +#else extern __must_check bool refcount_add_not_zero(unsigned int i, refcount_t *r); extern void refcount_add(unsigned int i, refcount_t *r); @@ -52,6 +55,7 @@ extern void refcount_sub(unsigned int i, refcount_t *r); extern __must_check bool refcount_dec_and_test(refcount_t *r); extern void refcount_dec(refcount_t *r); +#endif extern __must_check bool refcount_dec_if_one(refcount_t *r); extern __must_check bool refcount_dec_not_one(refcount_t *r); diff --git a/kernel/panic.c b/kernel/panic.c index a58932b41700..7d5a3eedd1bb 100644 --- a/kernel/panic.c +++ b/kernel/panic.c @@ -26,6 +26,7 @@ #include #include #include +#include #define PANIC_TIMER_STEP 100 #define PANIC_BLINK_SPD 18 @@ -601,6 +602,30 @@ EXPORT_SYMBOL(__stack_chk_fail); #endif +#ifdef CONFIG_FAST_REFCOUNT +static DEFINE_RATELIMIT_STATE(refcount_ratelimit, 15 * HZ, 3); + +void refcount_error_report(struct pt_regs *regs, const char *kind) +{ + do_send_sig_info(SIGKILL, SEND_SIG_FORCED, current, true); + + if (!__ratelimit(&refcount_ratelimit)) + return; + + pr_emerg("%s detected in: %s:%d, uid/euid: %u/%u\n", + kind ? kind : "refcount error", + current->comm, task_pid_nr(current), + from_kuid_munged(&init_user_ns, current_uid()), + from_kuid_munged(&init_user_ns, current_euid())); + print_symbol(KERN_EMERG "refcount error occurred at: %s\n", + instruction_pointer(regs)); + preempt_disable(); + show_regs(regs); + preempt_enable(); +} +EXPORT_SYMBOL(refcount_error_report); +#endif + core_param(panic, panic_timeout, int, 0644); core_param(pause_on_oops, pause_on_oops, int, 0644); core_param(panic_on_warn, panic_on_warn, int, 0644); diff --git a/lib/refcount.c b/lib/refcount.c index f42124ccf295..6bfe1b7f3e30 100644 --- a/lib/refcount.c +++ b/lib/refcount.c @@ -37,6 +37,9 @@ #include #include +/* Leave out architecture-specific implementations. */ +#ifndef CONFIG_FAST_REFCOUNT + /** * refcount_add_not_zero - add a value to a refcount unless it is 0 * @i: the value to add to the refcount @@ -225,6 +228,7 @@ void refcount_dec(refcount_t *r) WARN_ONCE(refcount_dec_and_test(r), "refcount_t: decrement hit 0; leaking memory.\n"); } EXPORT_SYMBOL_GPL(refcount_dec); +#endif /* CONFIG_FAST_REFCOUNT */ /** * refcount_dec_if_one - decrement a refcount if it is 1 @@ -345,4 +349,3 @@ bool refcount_dec_and_lock(refcount_t *r, spinlock_t *lock) return true; } EXPORT_SYMBOL_GPL(refcount_dec_and_lock); -