From patchwork Mon Apr 27 16:00:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512925 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3702413B2 for ; Mon, 27 Apr 2020 16:00:47 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 3E486206BF for ; Mon, 27 Apr 2020 16:00:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="s7wahkpC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3E486206BF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18637-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 10118 invoked by uid 550); 27 Apr 2020 16:00:38 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 10017 invoked from network); 27 Apr 2020 16:00:37 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=QBR6Lbc359enZ9GCab/6TfVF8noXTmNT1IXJtHim5eo=; b=s7wahkpCHDvKVqyAdAzdYD9E2iIkX0QxZ2uWff9IdYz7cIH8w3dZYbRll1nlGNY8Ol /Iz+TGe6pY5G1fxDi/RDNT9D3k1VGAhDZnthmSTpwmoo6UmGy7Ntd6qxgQLnuwafxpfV /rWs1NzleEaUVzNyaxRA23BWkgXxxaVzS1TkiC6SzK3erPidXLqYnZteyZ+mP+yPjen2 se5wE5LfLkDibsKs+zL7MF6u2aYQ9SKASNsrE11tYtfliRRMHKF1a2mIegifWsb/iQof Wx58HkUfpJ+vmcluKuznlOVb7rSbtIA3Dq3tLXHjWJs1ZwRhPRUxTkshNKedBNYlcffu V7Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=QBR6Lbc359enZ9GCab/6TfVF8noXTmNT1IXJtHim5eo=; b=WZdXGVJqc1wyCT8FtEcNbhhE9X/nw/25iKMOjzq3jvtHUPjrqwrGo5LLDMqf0kjxeB bEpZ4R+Xq2H2nXsEKuqf+BkerC9/y+v4+gSuKogb+y6jEE/JN6PQ5pp5+TbQAm7vYDRh uFS2S0Dlpxl6FKl3ztsATlfCCXnF85lWi5n3FEA7Kn+MFlYZbCbPrTvhIVi2/YY3BEDR X0gKYabbLqIBqcZ/NFGbRV1Oy0aqKKtqityiWLOWz6+tZ/JvSYS+6UWZ0O6xP0841K4R vguarkrhrHgEGglvoxRig0rDcP2WIcJcYdQ2MxSM1Ihr5MII8NFydlMMGA0qMKeE9lS7 Emgg== X-Gm-Message-State: AGi0PuZvPCR2FzPvQidgmW+oonHnzHn8/oMj/jGNbTKjr1EZydIX6k25 pHL4cDNxfKF4kIkTjlK5r0rBpnKmvecM6PL4ApI= X-Google-Smtp-Source: APiQypIg1Fxqu2AHiC+o+ltA4wFCSgwiteBygrQAq5XLPj1AxM9UmaZg0zP0NYErifBqdO4h510rY8jJhoQl/C1Hg+g= X-Received: by 2002:a25:a3a1:: with SMTP id e30mr35450525ybi.10.1588003225936; Mon, 27 Apr 2020 09:00:25 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:07 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-2-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 01/12] add support for Clang's Shadow Call Stack (SCS) From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here: https://clang.llvm.org/docs/ShadowCallStack.html Note that security guarantees in the kernel differ from the ones documented for user space. The kernel must store addresses of shadow stacks in memory, which means an attacker capable reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying the stacks. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook Reviewed-by: Miguel Ojeda --- Makefile | 6 +++ arch/Kconfig | 24 ++++++++++++ include/linux/compiler-clang.h | 6 +++ include/linux/compiler_types.h | 4 ++ include/linux/scs.h | 72 ++++++++++++++++++++++++++++++++++ init/init_task.c | 8 ++++ kernel/Makefile | 1 + kernel/fork.c | 9 +++++ kernel/sched/core.c | 2 + kernel/scs.c | 68 ++++++++++++++++++++++++++++++++ 10 files changed, 200 insertions(+) create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.c diff --git a/Makefile b/Makefile index 679f302a8b8b..33dc0d0cdd08 100644 --- a/Makefile +++ b/Makefile @@ -866,6 +866,12 @@ ifdef CONFIG_LIVEPATCH KBUILD_CFLAGS += $(call cc-option, -flive-patching=inline-clone) endif +ifdef CONFIG_SHADOW_CALL_STACK +CC_FLAGS_SCS := -fsanitize=shadow-call-stack +KBUILD_CFLAGS += $(CC_FLAGS_SCS) +export CC_FLAGS_SCS +endif + # arch Makefile may override CC so keep this after arch Makefile is included NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include) diff --git a/arch/Kconfig b/arch/Kconfig index 786a85d4ad40..334a3d9b19df 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -533,6 +533,30 @@ config STACKPROTECTOR_STRONG about 20% of all kernel functions, which increases the kernel code size by about 2%. +config ARCH_SUPPORTS_SHADOW_CALL_STACK + bool + help + An architecture should select this if it supports Clang's Shadow + Call Stack, has asm/scs.h, and implements runtime support for shadow + stack switching. + +config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on CC_IS_CLANG && ARCH_SUPPORTS_SHADOW_CALL_STACK + help + This option enables Clang's Shadow Call Stack, which uses a + shadow stack to protect function return addresses from being + overwritten by an attacker. More information can be found in + Clang's documentation: + + https://clang.llvm.org/docs/ShadowCallStack.html + + Note that security guarantees in the kernel differ from the + ones documented for user space. The kernel must store addresses + of shadow stacks in memory, which means an attacker capable of + reading and writing arbitrary memory may be able to locate them + and hijack control flow by modifying the stacks. + config HAVE_ARCH_WITHIN_STACK_FRAMES bool help diff --git a/include/linux/compiler-clang.h b/include/linux/compiler-clang.h index 333a6695a918..18fc4d29ef27 100644 --- a/include/linux/compiler-clang.h +++ b/include/linux/compiler-clang.h @@ -42,3 +42,9 @@ * compilers, like ICC. */ #define barrier() __asm__ __volatile__("" : : : "memory") + +#if __has_feature(shadow_call_stack) +# define __noscs __attribute__((__no_sanitize__("shadow-call-stack"))) +#else +# define __noscs +#endif diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h index e970f97a7fcb..97b62f47a80d 100644 --- a/include/linux/compiler_types.h +++ b/include/linux/compiler_types.h @@ -193,6 +193,10 @@ struct ftrace_likely_data { # define randomized_struct_fields_end #endif +#ifndef __noscs +# define __noscs +#endif + #ifndef asm_volatile_goto #define asm_volatile_goto(x...) asm goto(x) #endif diff --git a/include/linux/scs.h b/include/linux/scs.h new file mode 100644 index 000000000000..060eeb3d1390 --- /dev/null +++ b/include/linux/scs.h @@ -0,0 +1,72 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Shadow Call Stack support. + * + * Copyright (C) 2019 Google LLC + */ + +#ifndef _LINUX_SCS_H +#define _LINUX_SCS_H + +#include +#include +#include +#include + +#ifdef CONFIG_SHADOW_CALL_STACK + +/* + * In testing, 1 KiB shadow stack size (i.e. 128 stack frames on a 64-bit + * architecture) provided ~40% safety margin on stack usage while keeping + * memory allocation overhead reasonable. + */ +#define SCS_SIZE 1024UL +#define GFP_SCS (GFP_KERNEL | __GFP_ZERO) + +/* An illegal pointer value to mark the end of the shadow stack. */ +#define SCS_END_MAGIC (0x5f6UL + POISON_POINTER_DELTA) + +#define task_scs(tsk) (task_thread_info(tsk)->scs_base) +#define task_scs_offset(tsk) (task_thread_info(tsk)->scs_offset) + +extern void scs_init(void); + +static inline void scs_task_reset(struct task_struct *tsk) +{ + /* + * Reset the shadow stack to the base address in case the task + * is reused. + */ + task_scs_offset(tsk) = 0; +} + +extern int scs_prepare(struct task_struct *tsk, int node); + +static inline unsigned long *__scs_magic(void *s) +{ + return (unsigned long *)(s + SCS_SIZE) - 1; +} + +static inline bool scs_corrupted(struct task_struct *tsk) +{ + unsigned long *magic = __scs_magic(task_scs(tsk)); + + return (task_scs_offset(tsk) >= SCS_SIZE - 1 || + READ_ONCE_NOCHECK(*magic) != SCS_END_MAGIC); +} + +extern void scs_release(struct task_struct *tsk); + +#else /* CONFIG_SHADOW_CALL_STACK */ + +#define task_scs(tsk) NULL + +static inline void scs_init(void) {} +static inline void scs_task_reset(struct task_struct *tsk) {} +static inline int scs_prepare(struct task_struct *tsk, int node) { return 0; } +static inline bool scs_corrupted(struct task_struct *tsk) { return false; } +static inline void scs_release(struct task_struct *tsk) {} + +#endif /* CONFIG_SHADOW_CALL_STACK */ + +#endif /* _LINUX_SCS_H */ diff --git a/init/init_task.c b/init/init_task.c index bd403ed3e418..169e34066d35 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -50,6 +51,13 @@ static struct sighand_struct init_sighand = { .signalfd_wqh = __WAIT_QUEUE_HEAD_INITIALIZER(init_sighand.signalfd_wqh), }; +#ifdef CONFIG_SHADOW_CALL_STACK +unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)] + __init_task_data = { + [(SCS_SIZE / sizeof(long)) - 1] = SCS_END_MAGIC +}; +#endif + /* * Set up the first task table, touch at your own risk!. Base=0, * limit=0x1fffff (=2MB) diff --git a/kernel/Makefile b/kernel/Makefile index 4cb4130ced32..c332eb9d4841 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -103,6 +103,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_PERF_EVENTS) += events/ diff --git a/kernel/fork.c b/kernel/fork.c index 8c700f881d92..f6339f9d232d 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -94,6 +94,7 @@ #include #include #include +#include #include #include @@ -456,6 +457,8 @@ void put_task_stack(struct task_struct *tsk) void free_task(struct task_struct *tsk) { + scs_release(tsk); + #ifndef CONFIG_THREAD_INFO_IN_TASK /* * The task is finally done with both the stack and thread_info, @@ -840,6 +843,8 @@ void __init fork_init(void) NULL, free_vm_stack_cache); #endif + scs_init(); + lockdep_init_task(&init_task); uprobes_init(); } @@ -899,6 +904,10 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) if (err) goto free_stack; + err = scs_prepare(tsk, node); + if (err) + goto free_stack; + #ifdef CONFIG_SECCOMP /* * We must handle setting up seccomp filters once we're under diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 9a2fbf98fd6f..934e03cfaec7 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -11,6 +11,7 @@ #include #include +#include #include #include @@ -6040,6 +6041,7 @@ void init_idle(struct task_struct *idle, int cpu) idle->se.exec_start = sched_clock(); idle->flags |= PF_IDLE; + scs_task_reset(idle); kasan_unpoison_task_stack(idle); #ifdef CONFIG_SMP diff --git a/kernel/scs.c b/kernel/scs.c new file mode 100644 index 000000000000..43624be9ad90 --- /dev/null +++ b/kernel/scs.c @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Shadow Call Stack support. + * + * Copyright (C) 2019 Google LLC + */ + +#include +#include +#include +#include + +static struct kmem_cache *scs_cache; + +static void *scs_alloc(int node) +{ + void *s; + + s = kmem_cache_alloc_node(scs_cache, GFP_SCS, node); + if (s) { + *__scs_magic(s) = SCS_END_MAGIC; + /* + * Poison the allocation to catch unintentional accesses to + * the shadow stack when KASAN is enabled. + */ + kasan_poison_object_data(scs_cache, s); + } + + return s; +} + +static void scs_free(void *s) +{ + kasan_unpoison_object_data(scs_cache, s); + kmem_cache_free(scs_cache, s); +} + +void __init scs_init(void) +{ + scs_cache = kmem_cache_create("scs_cache", SCS_SIZE, 0, 0, NULL); +} + +int scs_prepare(struct task_struct *tsk, int node) +{ + void *s; + + s = scs_alloc(node); + if (!s) + return -ENOMEM; + + task_scs(tsk) = s; + task_scs_offset(tsk) = 0; + + return 0; +} + +void scs_release(struct task_struct *tsk) +{ + void *s; + + s = task_scs(tsk); + if (!s) + return; + + WARN_ON(scs_corrupted(tsk)); + + scs_free(s); +} From patchwork Mon Apr 27 16:00:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512927 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 44B101392 for ; Mon, 27 Apr 2020 16:01:00 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 3E9F1205C9 for ; Mon, 27 Apr 2020 16:00:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pkChly/M" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3E9F1205C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18638-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 11405 invoked by uid 550); 27 Apr 2020 16:00:41 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 11338 invoked from network); 27 Apr 2020 16:00:41 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=VMU4Dxx5VtpJ9uKOr46aqgBjHsAjXld8C2DfnSdv8zo=; b=pkChly/MR4gg1F/vqNKdS/sRvRHC9h2GTUAegFBSMak98utNlZ3ShOdFIbT8e3bAOS Qw8Q9Bej6lzq2ZoUHa9FOdVh6GHVnGPiKRrlCRqY56CqAwxR1tVy2B5KvqLQKlfr44mJ XzWmpehjC2j7wr+O0l2nKdIuS+q5brNPlRPb9CkFAGjYfLQjvqvg3b+3drlHjancStvP peyjrnBzg7WUQqKDslZ8rxgMPATe+dCJ7ChBdJXxt4UTJg1AoT2va9kPF7K74NEZshI8 Tnhnhsr1OmuzskrQ5UhyDnyxXZggId4mQCkOufqIWd4kL7UJZzZOREaJQ/QmsfCBs6xr 5oQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=VMU4Dxx5VtpJ9uKOr46aqgBjHsAjXld8C2DfnSdv8zo=; b=a8EMFJD7Ru5ze0TZ8PW7Q0n8JU9CttgXFUgroWH60tqry3foEdgGKxlKbhXHK93Tcx xqs/iTZLfkybkzY7BWQD0ntsoHnXB47jRiFja5rB50B3WveLixrBbKp3IY+mJdMMz29R UqiVkf4rIEAbVBBS14NzRjntKD+oI4OUg27h/bQmNsAJ1vp9T8fgMbkQShPhzA0ShA8F yD/gcojtzuOw0DGit5tFSAhk3gYcuycpwFXtVgBhSOUWHnTYQ3zf8KBLxwE5ZYMq+2Ue rN7hKBbCt4mQ8PN15Avdf3j0VdLoPiONtdSbd7wOhlXx6dDvC2uABSQddxJ1ZMhmyMhK 9zKQ== X-Gm-Message-State: AGi0PubCtdXf4dpj9wj6Kvd36F4+3hR+xeCYTXLCxm62wsk/IfScOesh YpmYBtAYu4WnSnXIH1jXdKfOFIYmZ3xZ4tHj8Vo= X-Google-Smtp-Source: APiQypIyrY142CMDGq82JJi1Gkln1wo6uLkxCKbOaTA9gyTh2SBuyrZ7K/0KHyeX6IkZStbpWE1vne7HEdDXyYcdEfs= X-Received: by 2002:ad4:4b6b:: with SMTP id m11mr22911491qvx.130.1588003229016; Mon, 27 Apr 2020 09:00:29 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:08 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-3-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 02/12] scs: add accounting From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen This change adds accounting for the memory allocated for shadow stacks. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook Acked-by: Will Deacon --- drivers/base/node.c | 6 ++++++ fs/proc/meminfo.c | 4 ++++ include/linux/mmzone.h | 3 +++ kernel/scs.c | 15 +++++++++++++++ mm/page_alloc.c | 6 ++++++ mm/vmstat.c | 3 +++ 6 files changed, 37 insertions(+) diff --git a/drivers/base/node.c b/drivers/base/node.c index 10d7e818e118..50b8c0d43859 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -415,6 +415,9 @@ static ssize_t node_read_meminfo(struct device *dev, "Node %d AnonPages: %8lu kB\n" "Node %d Shmem: %8lu kB\n" "Node %d KernelStack: %8lu kB\n" +#ifdef CONFIG_SHADOW_CALL_STACK + "Node %d ShadowCallStack:%8lu kB\n" +#endif "Node %d PageTables: %8lu kB\n" "Node %d NFS_Unstable: %8lu kB\n" "Node %d Bounce: %8lu kB\n" @@ -438,6 +441,9 @@ static ssize_t node_read_meminfo(struct device *dev, nid, K(node_page_state(pgdat, NR_ANON_MAPPED)), nid, K(i.sharedram), nid, sum_zone_node_page_state(nid, NR_KERNEL_STACK_KB), +#ifdef CONFIG_SHADOW_CALL_STACK + nid, sum_zone_node_page_state(nid, NR_KERNEL_SCS_KB), +#endif nid, K(sum_zone_node_page_state(nid, NR_PAGETABLE)), nid, K(node_page_state(pgdat, NR_UNSTABLE_NFS)), nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)), diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index 8c1f1bb1a5ce..09cd51c8d23d 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -103,6 +103,10 @@ static int meminfo_proc_show(struct seq_file *m, void *v) show_val_kb(m, "SUnreclaim: ", sunreclaim); seq_printf(m, "KernelStack: %8lu kB\n", global_zone_page_state(NR_KERNEL_STACK_KB)); +#ifdef CONFIG_SHADOW_CALL_STACK + seq_printf(m, "ShadowCallStack:%8lu kB\n", + global_zone_page_state(NR_KERNEL_SCS_KB)); +#endif show_val_kb(m, "PageTables: ", global_zone_page_state(NR_PAGETABLE)); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 1b9de7d220fb..acffc3bc6178 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -156,6 +156,9 @@ enum zone_stat_item { NR_MLOCK, /* mlock()ed pages found and moved off LRU */ NR_PAGETABLE, /* used for pagetables */ NR_KERNEL_STACK_KB, /* measured in KiB */ +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) + NR_KERNEL_SCS_KB, /* measured in KiB */ +#endif /* Second 128 byte cacheline */ NR_BOUNCE, #if IS_ENABLED(CONFIG_ZSMALLOC) diff --git a/kernel/scs.c b/kernel/scs.c index 43624be9ad90..8769016c714c 100644 --- a/kernel/scs.c +++ b/kernel/scs.c @@ -6,8 +6,10 @@ */ #include +#include #include #include +#include #include static struct kmem_cache *scs_cache; @@ -40,6 +42,17 @@ void __init scs_init(void) scs_cache = kmem_cache_create("scs_cache", SCS_SIZE, 0, 0, NULL); } +static struct page *__scs_page(struct task_struct *tsk) +{ + return virt_to_page(task_scs(tsk)); +} + +static void scs_account(struct task_struct *tsk, int account) +{ + mod_zone_page_state(page_zone(__scs_page(tsk)), NR_KERNEL_SCS_KB, + account * (SCS_SIZE / 1024)); +} + int scs_prepare(struct task_struct *tsk, int node) { void *s; @@ -50,6 +63,7 @@ int scs_prepare(struct task_struct *tsk, int node) task_scs(tsk) = s; task_scs_offset(tsk) = 0; + scs_account(tsk, 1); return 0; } @@ -64,5 +78,6 @@ void scs_release(struct task_struct *tsk) WARN_ON(scs_corrupted(tsk)); + scs_account(tsk, -1); scs_free(s); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 69827d4fa052..83743d7a6177 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5411,6 +5411,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask) " managed:%lukB" " mlocked:%lukB" " kernel_stack:%lukB" +#ifdef CONFIG_SHADOW_CALL_STACK + " shadow_call_stack:%lukB" +#endif " pagetables:%lukB" " bounce:%lukB" " free_pcp:%lukB" @@ -5433,6 +5436,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask) K(zone_managed_pages(zone)), K(zone_page_state(zone, NR_MLOCK)), zone_page_state(zone, NR_KERNEL_STACK_KB), +#ifdef CONFIG_SHADOW_CALL_STACK + zone_page_state(zone, NR_KERNEL_SCS_KB), +#endif K(zone_page_state(zone, NR_PAGETABLE)), K(zone_page_state(zone, NR_BOUNCE)), K(free_pcp), diff --git a/mm/vmstat.c b/mm/vmstat.c index 96d21a792b57..2435d2c24657 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1119,6 +1119,9 @@ const char * const vmstat_text[] = { "nr_mlock", "nr_page_table_pages", "nr_kernel_stack", +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) + "nr_shadow_call_stack", +#endif "nr_bounce", #if IS_ENABLED(CONFIG_ZSMALLOC) "nr_zspages", From patchwork Mon Apr 27 16:00:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512931 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A881814B4 for ; Mon, 27 Apr 2020 16:01:15 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 13382205C9 for ; Mon, 27 Apr 2020 16:01:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LKPo6Bc/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 13382205C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18639-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 11715 invoked by uid 550); 27 Apr 2020 16:00:46 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 11643 invoked from network); 27 Apr 2020 16:00:45 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=kikYfFSTfca/HkjJiT4VwZFMKgohIqKorfyIql1wDNc=; b=LKPo6Bc/Q5MtQzUK3oQa6NKadNaC15z3lO/zjdaounxLr3Yr/P6IlUdUGblL+xU5k4 spAtU8NF5ubk32gUq0m5VXjp6kv3GAFhZ4b47WHO1jV9Re1P0wgxqNnIy78PYlqK6iHg Lz7JdX7MXdIx+DHcDceIZRSq68+3ppbTa32fDBBMVQUsXCdSeZ2Rk0+iayOXLcIpH0mh 1y8jfaqoFffOSgqi3glfbRygtn76YwRgqeTcaTsWT5Ma8vEPSbQu5Sf1lQXSpiWIQ5jH osRX2DeCrw1RtV2JiczAkw6ANMNeNCpzDPw4w/B5fxj55cgpesof+s+eghyxXIYIpJqz Cp7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=kikYfFSTfca/HkjJiT4VwZFMKgohIqKorfyIql1wDNc=; b=Y+UkuGAZLnOe8kFVGhmYyl+6FdRlLISv6+xxjWajW6T2E5oB9zCphloR53GJ25vlVA NW83fj8QJPye/egyWQEnEBHpav2tLF0SQ35f9mC8Z4C+cTZ+Wyc9Gwl9PxpEgLCsgXo7 HgRmgItBbaeF7ZU3aK2KVFvZ3o5Qn+cW4FoVbBVO98SLifZgX33GO5MkzC12bmlcWFv+ Fr4MC+7+8M10sYs7Ls7g6DyW5mtOjGEpc/mlT6sO5D39vc56lPFbdtcuT0d+eu9h5Fj1 R3H/L2uQ6LJMELVCvJdwOpe8ZK4IfoWd0fVevB/W0a8BR7euCVMj2uoDM6qqcHwvCoF8 zIlA== X-Gm-Message-State: AGi0PuaAgqaPGoI7msxJeRMh158ccDn4paAq9lWe7IvI5vvaYbbZ/26S ndGsN/GPyd9lILRwcoqZTv0E1mPrXKilu68+VdE= X-Google-Smtp-Source: APiQypIgO4clcO6Z7kvhUaBuvYACtL+F9hDoKmeBmaTp5Edf0BzJT+qKnh+MwC69KhaGLcQ8wc5C0JEBbMo44i2AOQM= X-Received: by 2002:a25:ddc3:: with SMTP id u186mr36888616ybg.383.1588003232559; Mon, 27 Apr 2020 09:00:32 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:09 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-4-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 03/12] scs: add support for stack usage debugging From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen Implements CONFIG_DEBUG_STACK_USAGE for shadow stacks. When enabled, also prints out the highest shadow stack usage per process. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook Acked-by: Will Deacon --- kernel/scs.c | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/kernel/scs.c b/kernel/scs.c index 8769016c714c..2a96573f2b1b 100644 --- a/kernel/scs.c +++ b/kernel/scs.c @@ -68,6 +68,43 @@ int scs_prepare(struct task_struct *tsk, int node) return 0; } +#ifdef CONFIG_DEBUG_STACK_USAGE +static unsigned long __scs_used(struct task_struct *tsk) +{ + unsigned long *p = task_scs(tsk); + unsigned long *end = __scs_magic(p); + unsigned long s = (unsigned long)p; + + while (p < end && READ_ONCE_NOCHECK(*p)) + p++; + + return (unsigned long)p - s; +} + +static void scs_check_usage(struct task_struct *tsk) +{ + static unsigned long highest; + unsigned long used = __scs_used(tsk); + unsigned long prev; + unsigned long curr = highest; + + while (used > curr) { + prev = cmpxchg_relaxed(&highest, curr, used); + + if (prev == curr) { + pr_info("%s (%d): highest shadow stack usage: " + "%lu bytes\n", + tsk->comm, task_pid_nr(tsk), used); + break; + } + + curr = prev; + } +} +#else +static inline void scs_check_usage(struct task_struct *tsk) {} +#endif + void scs_release(struct task_struct *tsk) { void *s; @@ -77,6 +114,7 @@ void scs_release(struct task_struct *tsk) return; WARN_ON(scs_corrupted(tsk)); + scs_check_usage(tsk); scs_account(tsk, -1); scs_free(s); From patchwork Mon Apr 27 16:00:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512937 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3BC311392 for ; Mon, 27 Apr 2020 16:01:52 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 98FE4205C9 for ; Mon, 27 Apr 2020 16:01:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="fqLi0Sky" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 98FE4205C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18640-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 12234 invoked by uid 550); 27 Apr 2020 16:00:54 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 12177 invoked from network); 27 Apr 2020 16:00:53 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=jTb4YPIUPKL1E1OLE2ifNkQg2x2HSYmo0hj4Ci3DZ3A=; b=fqLi0SkyPjbXxPX/hPsXZeMNhq35aYqBNJAvT/Xx+BzPh7YlBx6i8WTACzT4FBHIRv yB3PamKTCHUBg04d9KVAhujS595QnxeeGH1EBSaBLJkOM7C2p6uhVroooOZkVWSFANBo EQq7r79O7lYvn3jVBEw+TI04/17VIVUaDIXO9ddhSuge+F4kykHowiB4XA1PixFt7f06 knShlLB1Y6N5OLuGmCs4M7eKqc6BblFu2RW9cdCpUqsLmxzOIrCdokmS7rXLST3WR3EJ 4YBYToSahIZ4xPJWmk/lOEd0e03hxC16yaJ1xBfFhei9FtsLMetBOg+CemUkcnsn0gik hibA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=jTb4YPIUPKL1E1OLE2ifNkQg2x2HSYmo0hj4Ci3DZ3A=; b=mzjOOI4qNZ8KDFQLW5EJ3Jt0L9Gc+OxY916GiX9Ki+EyPraSg8+snbC5TgtTHkQpJM bnhguJmb15DIsB0sfYsYk/S29Jitw/9ltySu/FzKbYhAO7OvNFK18NfbouUayKIhTsHB VYTbUwwgGaeLwJFvOSRzBe2Pu3swvpQrg8Deq10W1vGY5RR7DiTeJl8v8EaHEskfI1K5 E+zsszobZtnSZU0b4uhB3K2r77zHDjH3ENCkMXC8WWe+o3qra05tAGOkscF7/bkI0Wb1 05pf9fExPg5SNgUr6qgrKgcHIwT8MXow+wobfvyQVllPTss469k5YMN6ajxl86OgQc5N gRbA== X-Gm-Message-State: AGi0PuZQY/awW5Jhw6e/O+unuMJ0gjuYrNXW7u4h0C1+/8VQM+QIk55/ IDdQS4pIXEjckzAtyrWG98hPhKoKdzRxC8XZw5U= X-Google-Smtp-Source: APiQypLDtauRhcKiTQyzgC2LQ3XamOTxT1OCw5Clsp9y71dsXWMyuVGmw4gnWIV3Yp64XvmFFiqiRAvcGkJl+hJ0vkc= X-Received: by 2002:a05:6214:1513:: with SMTP id e19mr22507959qvy.75.1588003237430; Mon, 27 Apr 2020 09:00:37 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:10 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-5-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 04/12] scs: disable when function graph tracing is enabled From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen The graph tracer hooks returns by modifying frame records on the (regular) stack, but with SCS the return address is taken from the shadow stack, and the value in the frame record has no effect. As we don't currently have a mechanism to determine the corresponding slot on the shadow stack (and to pass this through the ftrace infrastructure), for now let's disable SCS when the graph tracer is enabled. With SCS the return address is taken from the shadow stack and the value in the frame record has no effect. The mcount based graph tracer hooks returns by modifying frame records on the (regular) stack, and thus is not compatible. The patchable-function-entry graph tracer used for DYNAMIC_FTRACE_WITH_REGS modifies the LR before it is saved to the shadow stack, and is compatible. Modifying the mcount based graph tracer to work with SCS would require a mechanism to determine the corresponding slot on the shadow stack (and to pass this through the ftrace infrastructure), and we expect that everyone will eventually move to the patchable-function-entry based graph tracer anyway, so for now let's disable SCS when the mcount-based graph tracer is enabled. SCS and patchable-function-entry are both supported from LLVM 10.x. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook Reviewed-by: Mark Rutland --- arch/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/Kconfig b/arch/Kconfig index 334a3d9b19df..45dfca9a98d3 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -543,6 +543,7 @@ config ARCH_SUPPORTS_SHADOW_CALL_STACK config SHADOW_CALL_STACK bool "Clang Shadow Call Stack" depends on CC_IS_CLANG && ARCH_SUPPORTS_SHADOW_CALL_STACK + depends on DYNAMIC_FTRACE_WITH_REGS || !FUNCTION_GRAPH_TRACER help This option enables Clang's Shadow Call Stack, which uses a shadow stack to protect function return addresses from being From patchwork Mon Apr 27 16:00:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512935 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 33F8E14B4 for ; Mon, 27 Apr 2020 16:01:32 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 9279D205C9 for ; Mon, 27 Apr 2020 16:01:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="kiSv8CiU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9279D205C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18641-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 13319 invoked by uid 550); 27 Apr 2020 16:00:54 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 12195 invoked from network); 27 Apr 2020 16:00:53 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=vWKjMu4YpZkMxmoaABUBOKf1igZnxFZ9c3Ig1J3O4Og=; b=kiSv8CiUDdv0MhnhG7OLvK4HRjHRHD2p5Xk8z4MNjfWWTLx6u3zwpYo+VXYo00s9lQ 2SiB14ebJv+tpR/lJ6lBiS6NtwBJkpElgf6+/UxLuf4ODhwnkipaZqAOpGAlWYJpWfCS z/5ymvrE+OkDEIiguUTShhBQhJ6y715cKJGNQcwyaJsde7L0h/IXtozn5mtgfyeHg/Ym H43sTz24QuaAjK8JJGNtzpBldyBrDbq8uPivT4CI9K7/qC+hhiObIxt3J7MJfK5BmrKL +bLkpaJXkzjZcdx/s4M6wTt3HBXcwxs7ZUVUd46Y+nf5P0oIYdPegXdb9NdLI9tk1xzd GS1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vWKjMu4YpZkMxmoaABUBOKf1igZnxFZ9c3Ig1J3O4Og=; b=arPiOm5Be4VwK7GiOuLyFSIfTJfVxdUaihibJTipy9micuSdKqDhAiqn/n9ueK2s2F IplZ9gcEkiYtc0BSxgVlrd670f4+UxsAvRhcwyidsmwRVxZgT/p7DrAVpFk3sUzJDChI HXYdIBg6fKkXsFTz0ML6bvqukQ1z1r+rC8xG2jv1fFV4o1nmkk6CyfB7zfBpb9aLkG3a d14Ss5yS6N6HNLOIaYUaZ2Z868dpX3G6s0YIz4mBo53QQw+25hE1Tz7k5MlnC4RnQHDt O3qyvFqYJghdzZHfXpjK6FIpUU88PG2zbRSFEveI8ya9ay1bs4z7Kw0Cnw3OcgUEx0L8 VdXQ== X-Gm-Message-State: AGi0PuZP68PyUq7T/tmSvMaE1BcfxkXgqWuFq9mlC779VCx9eYZSYpJS Qo42z0SIpixosaUU5H0ZLPLoOUO3adYWmdcZY3w= X-Google-Smtp-Source: APiQypJpHxnUVa+HJ+OCdJz02DOHfWXmMr67lw1U6ExG9PpcAMVbKyAwbyBII6uQ9n51wuhyBZXu9qOSoWBoJcrJ8Lg= X-Received: by 2002:a05:6214:150e:: with SMTP id e14mr23211009qvy.65.1588003241754; Mon, 27 Apr 2020 09:00:41 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:11 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-6-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 05/12] arm64: reserve x18 from general allocation with SCS From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen Reserve the x18 register from general allocation when SCS is enabled, because the compiler uses the register to store the current task's shadow stack pointer. Note that all external kernel modules must also be compiled with -ffixed-x18 if the kernel has SCS enabled. Signed-off-by: Sami Tolvanen Reviewed-by: Nick Desaulniers Reviewed-by: Kees Cook Acked-by: Will Deacon --- arch/arm64/Makefile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 85e4149cc5d5..409a6c1be8cc 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -81,6 +81,10 @@ endif KBUILD_CFLAGS += $(branch-prot-flags-y) +ifeq ($(CONFIG_SHADOW_CALL_STACK), y) +KBUILD_CFLAGS += -ffixed-x18 +endif + ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__ From patchwork Mon Apr 27 16:00:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512947 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CBC7014B4 for ; Mon, 27 Apr 2020 16:02:28 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 325CB205C9 for ; Mon, 27 Apr 2020 16:02:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZfWmmWWe" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 325CB205C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18643-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 13880 invoked by uid 550); 27 Apr 2020 16:01:04 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 13805 invoked from network); 27 Apr 2020 16:01:02 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=aDem5II6XtVFnIoK0CnL/HzpdSNcG3sRMln2VgftrwU=; b=ZfWmmWWeg8Xv5CC2meB1osNUPG8RstNTWpr4cI8ESrfdKMVwQ7dxJXztTGQFimsM+Q CNvseJXVkYZoisMem4nRmxocSgBUZWRX2WM22kEjc8yGFWBw+H2CgMCAGJRvh5nA4tpL psKwxJzZBPKnH8RZhpmunbTEeHw7mqoGl1Chyd7YQFp1F8NuL79hZl5YpbN3INQoxV+B bs2zi3POqfqPXq6HRcKeb0fQ5M+8KRlgMPb3DVTPJden1T0c6jIaMjOnnX/iCef8UIHH 0wTi0R/2zRQc9P2Tgoryc+izyibMdoy6SFeJchSyHM1zbcKBR0WBctCIMUrRhaOOp1KA baSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=aDem5II6XtVFnIoK0CnL/HzpdSNcG3sRMln2VgftrwU=; b=LBuUiL10Wjs5O+hiZIAfePFSjwXhzkLH2IiuW7mdnNOpZCrHIl0doQTWWPCRZkNxod n06P/InUcjQhkgh4kTWilgrxtEQuoQrjS67pA667SU9NFRGjPiH93QqRj2MHb3n6plJ3 7bXUocn5vKuN92yqRrCpDucyJkGgREbrSy77UJTfyMQ/rcJs0eYePLaJGJtEi5qwn57X Xsf+mUIlbU2owhkogITMVuRVm5gYWs/7MhPN4aKTUz/3ApHNBsqRV7L3C4IsQ2HdUONk VZ1ut+fTf41w/YEnxPH+sCRsETQqwO6AjtSmJYwRdSRojbsoqiuPUfJA9LaxoCqAr/aA /uGQ== X-Gm-Message-State: AGi0PubaJMafeNRKdrYQNzhZ1fJceSjc9GinwajwP+fjThBAaL+O3hPt e5xSbCuQ5PducGA3FDt9hpdy2js0s/sA2cUJKWo= X-Google-Smtp-Source: APiQypJO4M2N3eKFmynDymgRKzs8tsPGJgj0wNazOuUIlfBTsE5WgPC/94hr99zCwp1irzlSc77dDJj+XxgS7YX4+qA= X-Received: by 2002:a0c:8262:: with SMTP id h89mr21907606qva.173.1588003245161; Mon, 27 Apr 2020 09:00:45 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:12 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-7-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 06/12] arm64: preserve x18 when CPU is suspended From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen Don't lose the current task's shadow stack when the CPU is suspended. Signed-off-by: Sami Tolvanen Reviewed-by: Nick Desaulniers Reviewed-by: Kees Cook Reviewed-by: Mark Rutland Acked-by: Will Deacon --- arch/arm64/include/asm/suspend.h | 2 +- arch/arm64/mm/proc.S | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/suspend.h b/arch/arm64/include/asm/suspend.h index 8939c87c4dce..0cde2f473971 100644 --- a/arch/arm64/include/asm/suspend.h +++ b/arch/arm64/include/asm/suspend.h @@ -2,7 +2,7 @@ #ifndef __ASM_SUSPEND_H #define __ASM_SUSPEND_H -#define NR_CTX_REGS 12 +#define NR_CTX_REGS 13 #define NR_CALLEE_SAVED_REGS 12 /* diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 197a9ba2d5ea..ed15be0f8103 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -58,6 +58,8 @@ * cpu_do_suspend - save CPU registers context * * x0: virtual address of context pointer + * + * This must be kept in sync with struct cpu_suspend_ctx in . */ SYM_FUNC_START(cpu_do_suspend) mrs x2, tpidr_el0 @@ -82,6 +84,11 @@ alternative_endif stp x8, x9, [x0, #48] stp x10, x11, [x0, #64] stp x12, x13, [x0, #80] + /* + * Save x18 as it may be used as a platform register, e.g. by shadow + * call stack. + */ + str x18, [x0, #96] ret SYM_FUNC_END(cpu_do_suspend) @@ -98,6 +105,13 @@ SYM_FUNC_START(cpu_do_resume) ldp x9, x10, [x0, #48] ldp x11, x12, [x0, #64] ldp x13, x14, [x0, #80] + /* + * Restore x18, as it may be used as a platform register, and clear + * the buffer to minimize the risk of exposure when used for shadow + * call stack. + */ + ldr x18, [x0, #96] + str xzr, [x0, #96] msr tpidr_el0, x2 msr tpidrro_el0, x3 msr contextidr_el1, x4 From patchwork Mon Apr 27 16:00:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512941 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57A7814B4 for ; Mon, 27 Apr 2020 16:02:09 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id B5E1B205C9 for ; Mon, 27 Apr 2020 16:02:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Z0D9RdpH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B5E1B205C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18642-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 13737 invoked by uid 550); 27 Apr 2020 16:01:01 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 13626 invoked from network); 27 Apr 2020 16:00:59 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=5K2KfWGbdPDvciUxJWKmXs8jymeyhVJTI52mBn5Jn2U=; b=Z0D9RdpHbbOHHtNvqHRo52aZbZAj07cdZRmpWEeU3iQTueml5HmYOfEBcmC2kKVIKm gi/I42ma62LvSvGNYMm2NpEhpa6fsNrdLgOepXOs2lXtqi4B13V7mob+8FE88DmkGTAj cIfoOAEzDwtio1vu3/bEbEwVj3iXpD61m74J35yn1WLDpR++aGJg9GG9gk87kfObQfaq 2Fr6/vhs+4bP5L3TBzKyaTEUUQB4mwZj8HZ3pjL3rr671/M4/cAQ1XkhNhG0tTSV5H0Y Lghd93Heuw59N+AcYzl5+VmFD+sORBxuAIoCgApPPsaEMCghNZl1+DyNOu8XOZdSuzS/ tqpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5K2KfWGbdPDvciUxJWKmXs8jymeyhVJTI52mBn5Jn2U=; b=U7BfBCV4wmkjOt0sfhGE1jeYbtvhIlI9QmUbVlS1gJO15ohS+brgiBvo2smbhsSqTf WhD0ringPT/E6W+Bj1Ba2Wv7MZPAm7CZeou89LhCpMcrR7MSKjwq1UbE+ugnSyXq+gdI SdORsNgssEwaAjZN4EZBuK825KT7UXt4UIvEGRIrL1B7ExgJKgyM+Lak9Z6VxpIvVUct /Hds8bEUxXsZDO7esv5WcFWkhzoqXkwsYNKlYpQ7RLp6mQDve5NsLTMQmWngO1/t2TEj q+n4tdgC21TdEBhZQEQts37Q7HPwZw17LuDajAlSrOcecR2m3qNBD1AUlb/65LwjA/K4 CAAg== X-Gm-Message-State: AGi0PuaUyll3ULfkxGUH5JD29+IsCYI0iJZBGIxhFKnbg8Ljkg6yosNr 4rhGC44Hs0MMVbwuSE/GoWz0QWZbmAobryeB1E4= X-Google-Smtp-Source: APiQypKZHPN6pHK1vtX6gJq9lobiR8zzmYnoXCSFDICaarTBdgedlZVYx+cSajiHci65J+75kJpbiA+ZLbes2n1w/OI= X-Received: by 2002:a0c:bec4:: with SMTP id f4mr23363061qvj.26.1588003248063; Mon, 27 Apr 2020 09:00:48 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:13 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 07/12] arm64: efi: restore x18 if it was corrupted From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen If we detect a corrupted x18, restore the register before jumping back to potentially SCS instrumented code. This is safe, because the wrapper is called with preemption disabled and a separate shadow stack is used for interrupt handling. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook Acked-by: Will Deacon --- arch/arm64/kernel/efi-rt-wrapper.S | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/efi-rt-wrapper.S b/arch/arm64/kernel/efi-rt-wrapper.S index 3fc71106cb2b..6ca6c0dc11a1 100644 --- a/arch/arm64/kernel/efi-rt-wrapper.S +++ b/arch/arm64/kernel/efi-rt-wrapper.S @@ -34,5 +34,14 @@ ENTRY(__efi_rt_asm_wrapper) ldp x29, x30, [sp], #32 b.ne 0f ret -0: b efi_handle_corrupted_x18 // tail call +0: + /* + * With CONFIG_SHADOW_CALL_STACK, the kernel uses x18 to store a + * shadow stack pointer, which we need to restore before returning to + * potentially instrumented code. This is safe because the wrapper is + * called with preemption disabled and a separate shadow stack is used + * for interrupts. + */ + mov x18, x2 + b efi_handle_corrupted_x18 // tail call ENDPROC(__efi_rt_asm_wrapper) From patchwork Mon Apr 27 16:00:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512951 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F4D01392 for ; Mon, 27 Apr 2020 16:02:47 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 8BD03206BF for ; Mon, 27 Apr 2020 16:02:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="aKKOaYv8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8BD03206BF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18644-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 13922 invoked by uid 550); 27 Apr 2020 16:01:04 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 13858 invoked from network); 27 Apr 2020 16:01:03 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=s5XAabWLI2BrMJh1ovnuwgRiP/5q7kOCrh3Tv97TlA0=; b=aKKOaYv8P84S78HqqnTCVvsVcMXU5SuB2uvNssb9dPS+wTPsWhzH4tykYH1PsvII8g Jd/hsa/Mgbp25Jeg8kQqz5FruH6lUYId1eE5kTscIN8QaYWmGyGrs5QWCnBs4rAdSgXQ zC0Nlt1KC7ARBPh/nOWpPnDWG0rRtqwjW8jYc28CX08RgkR6SRw7KhBHzjhgOoqClVSa 24U1Bn/ippHj5neJKtLWLh9eYKgPspBRD588z3b2ONTvZ+YfHBsPfig2qNeCQSypkVrT Yzp1OhGBKTNMPpcFLH0mbf95THoQkUSgEmKq+w0AYmnWxJt379tb0t0TUMJlMmHfTMKd cfmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=s5XAabWLI2BrMJh1ovnuwgRiP/5q7kOCrh3Tv97TlA0=; b=tl+xMCYRARoMoAXBArrw8KLGIlQwfXbLPm1aXmxMYt4Zcx7Wwb+eVIDUoe3J1YIA3e MFscsjUYbAm+6t1A/PE/rc9OBl5oIxxIts57niZfogfhNA4yBO0BIe/nbgOXkdenITTA boNgASHjE2MaKuUAgYjAYnCv/70z9cvwWP9Ochahed5YdKcd0/5JPyEEWMkNgvnckO8q TYHbuYlpuA26F1yvjv4R+fQIUU+GZi4xz2VHRjHQQmLWQEOzM7wBB1Sosj93SnM9RIWU w8OWpo3KZ6jgHFZHyWX9lNFYbuGksmCsqySwHhY0XtjE5Q2VZzkr6pq+Tlu+3KvdcGZ4 og+g== X-Gm-Message-State: AGi0PuasUgp1BqrZ0GDPOQ+npXZhfZC9YKP70VDX5se7Y8bHGDegMth6 XbN6SDLqNqc6i9kzSNRvLGBNgfe6i5YIxTyeWNk= X-Google-Smtp-Source: APiQypIC7qXsP5qbhdeCuYbEm/RDRXHOxeiSr4jdV4m9YjAk4hfpiSrYO6gGIVT/CP156YV9CFT4O6s/Zh/4ptwP/o4= X-Received: by 2002:a25:abed:: with SMTP id v100mr3797424ybi.96.1588003251771; Mon, 27 Apr 2020 09:00:51 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:14 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-9-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 08/12] arm64: vdso: disable Shadow Call Stack From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen Shadow stacks are only available in the kernel, so disable SCS instrumentation for the vDSO. Signed-off-by: Sami Tolvanen Reviewed-by: Nick Desaulniers Reviewed-by: Kees Cook Reviewed-by: Mark Rutland Acked-by: Will Deacon --- arch/arm64/kernel/vdso/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile index dd2514bb1511..a87a4f11724e 100644 --- a/arch/arm64/kernel/vdso/Makefile +++ b/arch/arm64/kernel/vdso/Makefile @@ -25,7 +25,7 @@ ccflags-y += -DDISABLE_BRANCH_PROFILING VDSO_LDFLAGS := -Bsymbolic -CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os +CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) KBUILD_CFLAGS += $(DISABLE_LTO) KASAN_SANITIZE := n UBSAN_SANITIZE := n From patchwork Mon Apr 27 16:00:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512955 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DBE914B4 for ; Mon, 27 Apr 2020 16:03:07 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 7A5EB206B8 for ; Mon, 27 Apr 2020 16:03:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="DHL29TiS" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7A5EB206B8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18645-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 14126 invoked by uid 550); 27 Apr 2020 16:01:07 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 14078 invoked from network); 27 Apr 2020 16:01:06 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=MhrpwHs0AsLMtVzb820LtwCNl0W7a7WFfLC0oPkvUmQ=; b=DHL29TiSaxcDKg26ae3snSvd9KBniehMx3v50e59HOzn23b9luEep1Uut6Rh1bgH// KO956X4m67E0WpTwHcyuZx/xco9K1MfUnOei1/NJ7DvhzpuBWAQQoieFIzyDvLo6Xj/U LSa76QE4JoiA3ockCGPZD1FfXXH04NOXGtZ1PxX9YzsvIand259bV5kmRZPWkfPadlSz XWC+qkJlG8khvQVMgO8moC3HdAi7FoVaXCNT/Dr0tnUqnPV0HoVJ/fMpxxfRfRmbMoG3 yBAUhuFYHDK2Ll4Rry67J9w3Df6Y/7mnJGUxW1bj4MawAaAPsThGV9/Mf4P8wQ1n4vgO xZBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=MhrpwHs0AsLMtVzb820LtwCNl0W7a7WFfLC0oPkvUmQ=; b=cCQ8blsWwWpsi8qRE2VFjSLCcB2eE7d8KP4t2l98DTtWD6kXLjoJ6TIzsDGh6nmh/L +3obcqEUNwk+hmbmC5wlA/mG3Y4LxQx6BhGc/mnKhJtSKt+94eHXsCwAj8x9HXn92UL3 GGarjxlaJ34PJhgwSAcMUesiayTy5QehEpEZUXnRcGoXxR24iiLyHedSFW//NCoKuPr7 +hZS+WSJqvSC0EzC8jj2ZcQ2xlRDUTGSvRD18ZdLf0q8SKpNWW2lebhXV2sUU3xsN9QF gzDS4LYgBQ73WA0w3Vh4ZL+deYJ7v2vMwYn4bNXdAr/Y/unTw7JhAIxnTwvf6M5vRg8P d9IQ== X-Gm-Message-State: AGi0PuZk0lyzYC+ftDl4baEYlocTuNiU6GXgcuc5LIYmYi1fWwlFBM0O xHNWhYl3SFG2e4F4TUzGTn6w/c10DoeiTesr5Wk= X-Google-Smtp-Source: APiQypIKYYCqrjrGDUY1B1bGA9o4qwmVhzAAlAiSD0oA2v0bEfRWWMuFhQtbWcozWJZsq/jS263B3BZnI77ktov0l6U= X-Received: by 2002:a5b:44b:: with SMTP id s11mr21987518ybp.399.1588003254867; Mon, 27 Apr 2020 09:00:54 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:15 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-10-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 09/12] arm64: disable SCS for hypervisor code From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen Disable SCS for code that runs at a different exception level by adding __noscs to __hyp_text. Suggested-by: James Morse Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook Acked-by: Marc Zyngier --- arch/arm64/include/asm/kvm_hyp.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index fe57f60f06a8..875b106c5d98 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -13,7 +13,7 @@ #include #include -#define __hyp_text __section(.hyp.text) notrace +#define __hyp_text __section(.hyp.text) notrace __noscs #define read_sysreg_elx(r,nvh,vh) \ ({ \ From patchwork Mon Apr 27 16:00:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512959 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 98A9E92C for ; Mon, 27 Apr 2020 16:03:26 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id A1429206BF for ; Mon, 27 Apr 2020 16:03:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="DIh3ZpqH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A1429206BF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18646-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 14288 invoked by uid 550); 27 Apr 2020 16:01:09 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 14220 invoked from network); 27 Apr 2020 16:01:08 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=mTuo+Q1m3yKttGwW9SEPV1YXcR0ZfdxZB+S+BHE7GOg=; b=DIh3ZpqH3JorDafuYNCRIaDH7QVxSM16VgoQMighevMCD4FrDjlhM3IVNYJOjOWaE9 u8C6UWBxnucwvlXWJNM0ZSWYzn9+SI1rMDMNpvXpnJ8GGC6yy7Bjs3/KirGL0Ecy84e/ dPcN+UWEJ4U79yFPR2tYBNwJGosY0VUt4CL7ZZZWwYCPWfUovsujNRT06S/K43Gbc/if K2tAGPhsmq0wlV3dzi+xhSG4RKZxDr+5DZmpLHIq+u9l9HqgyVzxeFdpl5LBJ6Ib3Leu YqLifV/Huv23y05O1T83NYLZWlVqYOXd4W2NH2m9Sc6NF88AUFdLjMW4cPboprmzjCGF 4Tzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=mTuo+Q1m3yKttGwW9SEPV1YXcR0ZfdxZB+S+BHE7GOg=; b=NDQLKA/77leGwyLVXCmgPpGxJzLLRA+xjPkRVc1yisYsdpTn89Xvp65Itm7Mv90Y3k WYDP6NBT6jilyg5biXc4cBV9+GrknQmlhXBVZZrDEt6LzkZMKSrYzLKSqsIZ/OZbKV3j fa/z2p6JGu/vAU/2d4Qydqa6PQ7mZbAS8ryiDOdhy8bCt7Vzp+MF0G4eZV1BfADsPu4q 5QLd9P6tsrGbDQjj0r4AJoS9CiTwV2+/IrrNqAqE5zCgtusjwwOlnb+9REPjlWXiRR4Y o0lFDsrMjjAG8eldNY9E08fvvN3d4eHHWPDLuMv6GRuiCWB1Z2FQrgc05FemUFG6isFt E0xA== X-Gm-Message-State: AGi0PubScVTzdo08SI5ICYIT/rIZ0jmRqh0dIT4qsOCkfYV/sLaD8kvy pvQDIGsDFa2rqangLYP2+fkc5ZfHAq34+/XkH9Q= X-Google-Smtp-Source: APiQypLOMueDmQsfet9TCWUzm3JBg3meRr/pTKYD6act4/mQMOJMVmQOPBTLPJeLGAgmLRgTR+erltvq+zOuSfT2ri8= X-Received: by 2002:a0c:8324:: with SMTP id j33mr1090078qva.23.1588003256950; Mon, 27 Apr 2020 09:00:56 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:16 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-11-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 10/12] arm64: implement Shadow Call Stack From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen This change implements shadow stack switching, initial SCS set-up, and interrupt shadow stacks for arm64. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook --- arch/arm64/Kconfig | 5 +++ arch/arm64/include/asm/scs.h | 46 ++++++++++++++++++++++++++++ arch/arm64/include/asm/thread_info.h | 13 ++++++++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/asm-offsets.c | 4 +++ arch/arm64/kernel/entry.S | 24 +++++++++++++-- arch/arm64/kernel/head.S | 6 ++++ arch/arm64/kernel/process.c | 2 ++ arch/arm64/kernel/scs.c | 15 +++++++++ 9 files changed, 114 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/include/asm/scs.h create mode 100644 arch/arm64/kernel/scs.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 40fb05d96c60..c380a16533f6 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -64,6 +64,7 @@ config ARM64 select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS select ARCH_SUPPORTS_MEMORY_FAILURE + select ARCH_SUPPORTS_SHADOW_CALL_STACK if CC_HAVE_SHADOW_CALL_STACK select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 && (GCC_VERSION >= 50000 || CC_IS_CLANG) select ARCH_SUPPORTS_NUMA_BALANCING @@ -1025,6 +1026,10 @@ config ARCH_HAS_CACHE_LINE_SIZE config ARCH_ENABLE_SPLIT_PMD_PTLOCK def_bool y if PGTABLE_LEVELS > 2 +# Supported by clang >= 7.0 +config CC_HAVE_SHADOW_CALL_STACK + def_bool $(cc-option, -fsanitize=shadow-call-stack -ffixed-x18) + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" ---help--- diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h new file mode 100644 index 000000000000..96549353b0cb --- /dev/null +++ b/arch/arm64/include/asm/scs.h @@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_SCS_H +#define _ASM_SCS_H + +#ifdef __ASSEMBLY__ + +#ifdef CONFIG_SHADOW_CALL_STACK + .macro scs_load tsk, tmp + ldp x18, \tmp, [\tsk, #TSK_TI_SCS_BASE] + add x18, x18, \tmp + .endm + + .macro scs_save tsk, tmp + ldr \tmp, [\tsk, #TSK_TI_SCS_BASE] + sub \tmp, x18, \tmp + str \tmp, [\tsk, #TSK_TI_SCS_OFFSET] + .endm +#else + .macro scs_load tsk, tmp + .endm + + .macro scs_save tsk, tmp + .endm +#endif /* CONFIG_SHADOW_CALL_STACK */ + +#else /* __ASSEMBLY__ */ + +#include + +#ifdef CONFIG_SHADOW_CALL_STACK + +static inline void scs_overflow_check(struct task_struct *tsk) +{ + if (unlikely(scs_corrupted(tsk))) + panic("corrupted shadow stack detected inside scheduler\n"); +} + +#else /* CONFIG_SHADOW_CALL_STACK */ + +static inline void scs_overflow_check(struct task_struct *tsk) {} + +#endif /* CONFIG_SHADOW_CALL_STACK */ + +#endif /* __ASSEMBLY __ */ + +#endif /* _ASM_SCS_H */ diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index 512174a8e789..9df79c0a4c43 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -41,6 +41,10 @@ struct thread_info { #endif } preempt; }; +#ifdef CONFIG_SHADOW_CALL_STACK + void *scs_base; + unsigned long scs_offset; +#endif }; #define thread_saved_pc(tsk) \ @@ -100,11 +104,20 @@ void arch_release_task_struct(struct task_struct *tsk); _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \ _TIF_SYSCALL_EMU) +#ifdef CONFIG_SHADOW_CALL_STACK +#define INIT_SCS \ + .scs_base = init_shadow_call_stack, \ + .scs_offset = 0, +#else +#define INIT_SCS +#endif + #define INIT_THREAD_INFO(tsk) \ { \ .flags = _TIF_FOREIGN_FPSTATE, \ .preempt_count = INIT_PREEMPT_COUNT, \ .addr_limit = KERNEL_DS, \ + INIT_SCS \ } #endif /* __ASM_THREAD_INFO_H */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 4e5b8ee31442..151f28521f1e 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -63,6 +63,7 @@ obj-$(CONFIG_CRASH_CORE) += crash_core.o obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_SSBD) += ssbd.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-y += vdso/ probes/ obj-$(CONFIG_COMPAT_VDSO) += vdso32/ diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 9981a0a5a87f..d7934250b68c 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -33,6 +33,10 @@ int main(void) DEFINE(TSK_TI_ADDR_LIMIT, offsetof(struct task_struct, thread_info.addr_limit)); #ifdef CONFIG_ARM64_SW_TTBR0_PAN DEFINE(TSK_TI_TTBR0, offsetof(struct task_struct, thread_info.ttbr0)); +#endif +#ifdef CONFIG_SHADOW_CALL_STACK + DEFINE(TSK_TI_SCS_BASE, offsetof(struct task_struct, thread_info.scs_base)); + DEFINE(TSK_TI_SCS_OFFSET, offsetof(struct task_struct, thread_info.scs_offset)); #endif DEFINE(TSK_STACK, offsetof(struct task_struct, stack)); #ifdef CONFIG_STACKPROTECTOR diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index ddcde093c433..244268d5ae47 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -179,6 +180,8 @@ alternative_cb_end apply_ssbd 1, x22, x23 ptrauth_keys_install_kernel tsk, 1, x20, x22, x23 + + scs_load tsk, x20 .else add x21, sp, #S_FRAME_SIZE get_current_task tsk @@ -343,6 +346,8 @@ alternative_else_nop_endif msr cntkctl_el1, x1 4: #endif + scs_save tsk, x0 + /* No kernel C function calls after this as user keys are set. */ ptrauth_keys_install_user tsk, x0, x1, x2 @@ -388,6 +393,9 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 .macro irq_stack_entry mov x19, sp // preserve the original sp +#ifdef CONFIG_SHADOW_CALL_STACK + mov x24, x18 // preserve the original shadow stack +#endif /* * Compare sp with the base of the task stack. @@ -405,15 +413,25 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 /* switch to the irq stack */ mov sp, x26 + +#ifdef CONFIG_SHADOW_CALL_STACK + /* also switch to the irq shadow stack */ + adr_this_cpu x18, irq_shadow_call_stack, x26 +#endif + 9998: .endm /* - * x19 should be preserved between irq_stack_entry and - * irq_stack_exit. + * The callee-saved regs (x19-x29) should be preserved between + * irq_stack_entry and irq_stack_exit, but note that kernel_entry + * uses x20-x23 to store data for later use. */ .macro irq_stack_exit mov sp, x19 +#ifdef CONFIG_SHADOW_CALL_STACK + mov x18, x24 +#endif .endm /* GPRs used by entry code */ @@ -901,6 +919,8 @@ SYM_FUNC_START(cpu_switch_to) mov sp, x9 msr sp_el0, x1 ptrauth_keys_install_kernel x1, 1, x8, x9, x10 + scs_save x0, x8 + scs_load x1, x8 ret SYM_FUNC_END(cpu_switch_to) NOKPROBE(cpu_switch_to) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 57a91032b4c2..2b01c19c5483 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -424,6 +425,10 @@ SYM_FUNC_START_LOCAL(__primary_switched) stp xzr, x30, [sp, #-16]! mov x29, sp +#ifdef CONFIG_SHADOW_CALL_STACK + adr_l x18, init_shadow_call_stack // Set shadow call stack +#endif + str_l x21, __fdt_pointer, x5 // Save FDT pointer ldr_l x4, kimage_vaddr // Save the offset between @@ -737,6 +742,7 @@ SYM_FUNC_START_LOCAL(__secondary_switched) ldr x2, [x0, #CPU_BOOT_TASK] cbz x2, __secondary_too_slow msr sp_el0, x2 + scs_load x2, x3 mov x29, #0 mov x30, #0 b secondary_start_kernel diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 56be4cbf771f..a35d3318492c 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_STACKPROTECTOR_PER_TASK) @@ -515,6 +516,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev, entry_task_switch(next); uao_thread_switch(next); ssbs_thread_switch(next); + scs_overflow_check(next); /* * Complete any pending TLB or cache maintenance on this CPU in case diff --git a/arch/arm64/kernel/scs.c b/arch/arm64/kernel/scs.c new file mode 100644 index 000000000000..acc6741d1a40 --- /dev/null +++ b/arch/arm64/kernel/scs.c @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Shadow Call Stack support. + * + * Copyright (C) 2019 Google LLC + */ + +#include +#include + +/* Allocate a static per-CPU shadow stack */ +#define DEFINE_SCS(name) \ + DEFINE_PER_CPU(unsigned long [SCS_SIZE/sizeof(long)], name) \ + +DEFINE_SCS(irq_shadow_call_stack); From patchwork Mon Apr 27 16:00:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512965 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 89D0192C for ; Mon, 27 Apr 2020 16:03:52 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id E9492205C9 for ; Mon, 27 Apr 2020 16:03:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CRokN2oO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E9492205C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18647-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 15436 invoked by uid 550); 27 Apr 2020 16:01:11 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 15380 invoked from network); 27 Apr 2020 16:01:10 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qkETA9PcFfLLgtHJjON9o2tnpUEMYOCdGfYROMcqbgY=; b=CRokN2oO5qHaIwDnfhPQm4WUGN3dL/7L085WmXGBSTSWdMgIglwt0wRRJelCXVTkUE evOGTcMnXuR7Gjk4FVLYN7tW5CCc6BokYtvDyoKf0BZkGqxaFodx3U9MT4v6NLYHqmZ3 iKYOMczKSYycS/Y78HO0iUKm+Cybr6RYQTwonAcmpQMj4NBDWRV7rBluW3RDsp+lkYsB XjiNReA/J9iMZUvAfy9f6Kp0qq+eqtAmtSd9c8FAhrtbFh6vgYjxE0lF9hnWHmvKb9K6 9WkijNUXXW4KO1tiyvrcZh1oyagjEj80iU2m3X3BlmfDkqRZqoPzeopP0mihx1UMFM7T 1fAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qkETA9PcFfLLgtHJjON9o2tnpUEMYOCdGfYROMcqbgY=; b=ACjlJocT2zbCKw7DCB9TBOAeKzrLF6iDMXHreWeP2daiixoCv2M4NYK7YhjguZjkpg k7ScczUsFbsqaEb/3D48N0NKIIE5rolsMUvjT2JPxlf6FTNM0gZroEmDxDtY0SDW3bc3 XuWWdonbfZiT6xDW71YCsKuHPWzKmY+RBrPEr9o/DSvn/7YGvSgs2tZB5/xzXrrdOmfi nPwqrwHZ5i1hV9UcOFZqstYVOFh+htfSwCeBZXDzhppKFjh0wvd8kBPYQzW49X0SW5J4 Ci0HKctUVAEMGRp3D9MTvl6pIaO2GTLyTnH/0H5PgHylBK1xZxI9MPr60B3IKDtpodOf An9w== X-Gm-Message-State: AGi0PuYngm497xpm8SLhSv4JBieYeAK0pDeyQDlF2W2NnU0iexpkowj5 NVy1KGoMuy1wzrr1usuWYCkZfjzUnk18DOIG5mQ= X-Google-Smtp-Source: APiQypLYy8Cf6kn4jY7fWjXaYMyZ9zM7eZaWrEh0s8K72Jfk+q/yv/f8RANanjFwMmhFbjtpLFic2YH4QuKkPtwOEog= X-Received: by 2002:a25:d14c:: with SMTP id i73mr9231289ybg.116.1588003258779; Mon, 27 Apr 2020 09:00:58 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:17 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-12-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 11/12] arm64: scs: add shadow stacks for SDEI From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen This change adds per-CPU shadow call stacks for the SDEI handler. Similarly to how the kernel stacks are handled, we add separate shadow stacks for normal and critical events. Signed-off-by: Sami Tolvanen Reviewed-by: James Morse Tested-by: James Morse --- arch/arm64/kernel/entry.S | 14 +++++++++++++- arch/arm64/kernel/scs.c | 5 +++++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 244268d5ae47..cb0516e6f963 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -1049,13 +1049,16 @@ SYM_CODE_START(__sdei_asm_handler) mov x19, x1 +#if defined(CONFIG_VMAP_STACK) || defined(CONFIG_SHADOW_CALL_STACK) + ldrb w4, [x19, #SDEI_EVENT_PRIORITY] +#endif + #ifdef CONFIG_VMAP_STACK /* * entry.S may have been using sp as a scratch register, find whether * this is a normal or critical event and switch to the appropriate * stack for this CPU. */ - ldrb w4, [x19, #SDEI_EVENT_PRIORITY] cbnz w4, 1f ldr_this_cpu dst=x5, sym=sdei_stack_normal_ptr, tmp=x6 b 2f @@ -1065,6 +1068,15 @@ SYM_CODE_START(__sdei_asm_handler) mov sp, x5 #endif +#ifdef CONFIG_SHADOW_CALL_STACK + /* Use a separate shadow call stack for normal and critical events */ + cbnz w4, 3f + adr_this_cpu dst=x18, sym=sdei_shadow_call_stack_normal, tmp=x6 + b 4f +3: adr_this_cpu dst=x18, sym=sdei_shadow_call_stack_critical, tmp=x6 +4: +#endif + /* * We may have interrupted userspace, or a guest, or exit-from or * return-to either of these. We can't trust sp_el0, restore it. diff --git a/arch/arm64/kernel/scs.c b/arch/arm64/kernel/scs.c index acc6741d1a40..adc97f826fab 100644 --- a/arch/arm64/kernel/scs.c +++ b/arch/arm64/kernel/scs.c @@ -13,3 +13,8 @@ DEFINE_PER_CPU(unsigned long [SCS_SIZE/sizeof(long)], name) \ DEFINE_SCS(irq_shadow_call_stack); + +#ifdef CONFIG_ARM_SDE_INTERFACE +DEFINE_SCS(sdei_shadow_call_stack_normal); +DEFINE_SCS(sdei_shadow_call_stack_critical); +#endif From patchwork Mon Apr 27 16:00:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11512969 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1D21B92C for ; Mon, 27 Apr 2020 16:04:15 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 7A84B20661 for ; Mon, 27 Apr 2020 16:04:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ijnQ0fnO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7A84B20661 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18648-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 15558 invoked by uid 550); 27 Apr 2020 16:01:13 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 15501 invoked from network); 27 Apr 2020 16:01:12 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=aF2klQa9rRamHf/M4CeCxFGtvwc1mZVJXnokoniINw4=; b=ijnQ0fnOBOZz8ohYGbprOcWD6bXoahdyQ/EdPHNcahHH1UNi6yI4DR5R2vlPmVt+MA 4ppIwntZ2J5xzVAeqXwe8sImyb647bo4hHg8kf1yP8EUjew4IgKXlLT+cyN+LZsy9Bq6 UC/DmCNG2YyUikRIYaXEhti+n0bf2Uk+eSvlUyy0VkRfNHrV9zVYezDs+dHfFwKCE9hw sMB0tZWkO4naa4enGnEraDahmtXjGEJ9z7j4fYeSXhaXwN3tRENHyK+EQjCXHOgGEjXH LRUwN4NXMSq2HIBjq6AGd84jx2T6eVKALaDwXc+kFG7UvOg+1/Elzk8FlgUN6WR5CVNf +rUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=aF2klQa9rRamHf/M4CeCxFGtvwc1mZVJXnokoniINw4=; b=UESnawemxkgBvtxe0GPMNho0AiDbd3dhfP1oNDPcoBhPQyeKeQgt1BLDL1pbFmnvHx 7b/wN68j0sXt/P33nLd4R8XPe8qiErIo7bxa4W/sneReKkdiNXdJlLrIg/YfxubZWbTx yQJgxx0DXEShKb5wNDKpBtk9HqD914cA5l+yWF+PNi9PM7JsAVsJBVfgn/3tQf7QNa9y VImtXRG0Kda3TFB0Ikgc+GFxuJ/3dAM6cyjj3naAbISR+u/oFXDyITbovliwDgJSIT0d HzstktbL9cDfyEV+rpUVsMmOVjPg4kWg0o+xGc2QS9NMWxZOGB7DW/bAlPBRIqkbqaw/ zivw== X-Gm-Message-State: AGi0PuYzy/HowT5PIfwgfKzBGDiLE/NQ5UU4aP3Myw6ys6Gpn8GyYaRN +7DkM458MzFWt6Sc06V8Kz2u2EPsuNKF36P/tmo= X-Google-Smtp-Source: APiQypKOsjWQ+jO+l4BxG62MWOPRhB/PQQHwz7wCFC1Vfqvf4Ou89RrbjPCh/lsRauohpj0Cl1hSrISC+Brb9ZNFufc= X-Received: by 2002:a25:23d4:: with SMTP id j203mr35345989ybj.97.1588003260531; Mon, 27 Apr 2020 09:01:00 -0700 (PDT) Date: Mon, 27 Apr 2020 09:00:18 -0700 In-Reply-To: <20200427160018.243569-1-samitolvanen@google.com> Message-Id: <20200427160018.243569-13-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20200427160018.243569-1-samitolvanen@google.com> X-Mailer: git-send-email 2.26.2.303.gf8c07b1a785-goog Subject: [PATCH v13 12/12] efi/libstub: disable SCS From: Sami Tolvanen To: Will Deacon , Catalin Marinas , James Morse , Steven Rostedt , Ard Biesheuvel , Mark Rutland , Masahiro Yamada , Michal Marek , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dave Martin , Kees Cook , Laura Abbott , Marc Zyngier , Masami Hiramatsu , Nick Desaulniers , Jann Horn , Miguel Ojeda , clang-built-linux@googlegroups.com, kernel-hardening@lists.openwall.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Sami Tolvanen , Ard Biesheuvel Shadow stacks are not available in the EFI stub, filter out SCS flags. Suggested-by: James Morse Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook Acked-by: Ard Biesheuvel --- drivers/firmware/efi/libstub/Makefile | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile index 094eabdecfe6..b52ae8c29560 100644 --- a/drivers/firmware/efi/libstub/Makefile +++ b/drivers/firmware/efi/libstub/Makefile @@ -32,6 +32,9 @@ KBUILD_CFLAGS := $(cflags-y) -DDISABLE_BRANCH_PROFILING \ $(call cc-option,-fno-stack-protector) \ -D__DISABLE_EXPORTS +# remove SCS flags from all objects in this directory +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) + GCOV_PROFILE := n KASAN_SANITIZE := n UBSAN_SANITIZE := n