From patchwork Mon Apr 25 11:55:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12825650 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 49D2AC433EF for ; Mon, 25 Apr 2022 11:57:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ihCC+6PU9263rtuiEtGLVhjmiHbczk3W7XfTWjTKCYo=; b=f0MRWf3J018IPn /Rk0sO+ItgRj/nsYCBPhKmklHy6DtfV35Vu5EwjH6fDfberCuYCDtMi2CHqL9v6he+zuu8EuCMkat f98jroAT3z036YGSetHLHQe1fZLcyDkrwOZAi3JAeEy4k5fDx0vurqvopYljOA2F/uA9WTB1nTl0U 8RxX7f1TfXZ1gx5V5I4mfWqg9oWdERHI5kc78pSzOqv9MzUcZgjBGeb2KASVrVw6MiCOh6MwzM8dk Iw1JMjMYGh4pLWn4EmNSDCB1Zj00qvGizmdEi+t5lvcyIg80TTTWu9BnHFIM7cxsXMEE2nCbBAa/c upWCt9Q6xXNx+UaCGUeA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKN-009NsG-Ao; Mon, 25 Apr 2022 11:56:28 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKA-009NoI-5Z for linux-arm-kernel@lists.infradead.org; Mon, 25 Apr 2022 11:56:16 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EDF77ED1; Mon, 25 Apr 2022 04:56:12 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B14D83F73B; Mon, 25 Apr 2022 04:56:11 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, alex.popov@linux.com, catalin.marinas@arm.com, keescook@chromium.org, linux-arm-kernel@lists.infradead.org, luto@kernel.org, mark.rutland@arm.com, will@kernel.org Subject: [PATCH 1/8] arm64: stackleak: fix current_top_of_stack() Date: Mon, 25 Apr 2022 12:55:56 +0100 Message-Id: <20220425115603.781311-2-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220425115603.781311-1-mark.rutland@arm.com> References: <20220425115603.781311-1-mark.rutland@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220425_045614_337574_708F07D9 X-CRM114-Status: GOOD ( 18.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Due to some historical confusion, arm64's current_top_of_stack() isn't what the stackleak code expects. This could in theory result in a number of problems, and practically results in an unnecessary performance hit. We can avoid this by aligning the arm64 implementation with the x86 implementation. The arm64 implementation of current_top_of_stack() was added specifically for stackleak in commit: 0b3e336601b82c6a ("arm64: Add support for STACKLEAK gcc plugin") This was intended to be equivalent to the x86 implementation, but the implementation, semantics, and performance characteristics differ wildly: * On x86, current_top_of_stack() returns the top of the current task's task stack, regardless of which stack is in active use. The implementation accesses a percpu variable which the x86 entry code maintains, and returns the location immediately above the pt_regs on the task stack (above which x86 has some padding). * On arm64 current_top_of_stack() returns the top of the stack in active use (i.e. the one which is currently being used). The implementation checks the SP against a number of potentially-accessible stacks, and will BUG() if no stack is found. The core stackleak_erase() code determines the upper bound of stack to erase with: | if (on_thread_stack()) | boundary = current_stack_pointer; | else | boundary = current_top_of_stack(); On arm64 stackleak_erase() is always called on a task stack, and on_thread_stack() should always be true. On x86, stackleak_erase() is mostly called on a trampoline stack, and is sometimes called on a task stack. Currently, this results in a lot of unnecessary code being generated for arm64 for the impossible !on_thread_stack() case. Some of this is inlined, bloating stackleak_erase(), while portions of this are left out-of-line and permitted to be instrumented (which would be a functional problem if that code were reachable). As a first step towards improving this, this patch aligns arm64's implementation of current_top_of_stack() with x86's, always returning the top of the current task's stack. With GCC 11.1.0 this results in the bulk of the unnecessary code being removed, including all of the out-of-line instrumentable code. While I don't believe there's a functional problem in practice I've marked this as a fix since the semantic was clearly wrong, the fix itself is simple, and other code might rely upon this in future. Fixes: 0b3e336601b82c6a ("arm64: Add support for STACKLEAK gcc plugin") Signed-off-by: Mark Rutland Cc: Alexander Popov Cc: Andrew Morton Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Kees Cook Cc: Will Deacon --- arch/arm64/include/asm/processor.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 73e38d9a540ce..6b1a12c23fe77 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -381,12 +381,10 @@ long get_tagged_addr_ctrl(struct task_struct *task); * of header definitions for the use of task_stack_page. */ -#define current_top_of_stack() \ -({ \ - struct stack_info _info; \ - BUG_ON(!on_accessible_stack(current, current_stack_pointer, 1, &_info)); \ - _info.high; \ -}) +/* + * The top of the current task's task stack + */ +#define current_top_of_stack() ((unsigned long)current->stack + THREAD_SIZE) #define on_thread_stack() (on_task_stack(current, current_stack_pointer, 1, NULL)) #endif /* __ASSEMBLY__ */ From patchwork Mon Apr 25 11:55:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12825651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E5B71C433EF for ; Mon, 25 Apr 2022 11:57:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Xxk5sUU4xfLKD4j1wFOyLl5yvHaA+16f4Kd96FOT7bI=; b=qkrL4LBKGZqc6C 1WyWHvIGFGK1dnm6iZtFQnVYHUkn8YHb9w3qb9aZZmxYpZB2DvggtAS+mHvZjdEdDUgjR+CnCorO0 bvpR65dT5a9D21mLEoDg2bsPzFl//6VV9m8xmR4TCorHkEuf77Zg859dgbSGJc2z6UrCH0IMiouC2 zBrJZjeqs+96o67PKRYkhiqI383wfwt++Ci0UvBvotNiWXY04oEyuHO8AvCcrkt72JyPZ2O6OU8ea +/gmN4DHbsN2PBjXwzK08yjJrPCgPRdnrVJ0dYks2eMkLaywAHJZLObjNfPxEsZEl9vgRNqYZsmQG 6ywdJst9O1duGIdu5pQA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKX-009Nvl-Hu; Mon, 25 Apr 2022 11:56:37 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKC-009Nov-2r for linux-arm-kernel@lists.infradead.org; Mon, 25 Apr 2022 11:56:19 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EEEA7113E; Mon, 25 Apr 2022 04:56:14 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B62003F73B; Mon, 25 Apr 2022 04:56:13 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, alex.popov@linux.com, catalin.marinas@arm.com, keescook@chromium.org, linux-arm-kernel@lists.infradead.org, luto@kernel.org, mark.rutland@arm.com, will@kernel.org Subject: [PATCH 2/8] stackleak: move skip_erasing() check earlier Date: Mon, 25 Apr 2022 12:55:57 +0100 Message-Id: <20220425115603.781311-3-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220425115603.781311-1-mark.rutland@arm.com> References: <20220425115603.781311-1-mark.rutland@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220425_045616_980091_8E0BCF3A X-CRM114-Status: GOOD ( 16.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In stackleak_erase() we check skip_erasing() after accessing some fields from current. As generating the address of current uses asm which hazards with the static branch asm, this work is always performed, even when the static branch is patched to jump to the return a the end of the function. This patch avoids this redundant work by moving the skip_erasing() check earlier. To avoid complicating initialization within stackleak_erase(), the body of the function is split out into a __stackleak_erase() helper, with the check left in a wrapper function. The __stackleak_erase() helper is marked __always_inline to ensure that this is inlined into stackleak_erase() and not instrumented. Before this patch, on x86-64 w/ GCC 11.1.0 the start of the function is: : 65 48 8b 04 25 00 00 mov %gs:0x0,%rax 00 00 48 8b 48 20 mov 0x20(%rax),%rcx 48 8b 80 98 0a 00 00 mov 0xa98(%rax),%rax 66 90 xchg %ax,%ax <------------ static branch 48 89 c2 mov %rax,%rdx 48 29 ca sub %rcx,%rdx 48 81 fa ff 3f 00 00 cmp $0x3fff,%rdx After this patch, on x86-64 w/ GCC 11.1.0 the start of the function is: : 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) <--- static branch 65 48 8b 04 25 00 00 mov %gs:0x0,%rax 00 00 48 8b 48 20 mov 0x20(%rax),%rcx 48 8b 80 98 0a 00 00 mov 0xa98(%rax),%rax 48 89 c2 mov %rax,%rdx 48 29 ca sub %rcx,%rdx 48 81 fa ff 3f 00 00 cmp $0x3fff,%rdx Before this patch, on arm64 w/ GCC 11.1.0 the start of the function is: : d503245f bti c d5384100 mrs x0, sp_el0 f9401003 ldr x3, [x0, #32] f9451000 ldr x0, [x0, #2592] d503201f nop <------------------------------- static branch d503233f paciasp cb030002 sub x2, x0, x3 d287ffe1 mov x1, #0x3fff eb01005f cmp x2, x1 After this patch, on arm64 w/ GCC 11.1.0 the start of the function is: : d503245f bti c d503201f nop <------------------------------- static branch d503233f paciasp d5384100 mrs x0, sp_el0 f9401003 ldr x3, [x0, #32] d287ffe1 mov x1, #0x3fff f9451000 ldr x0, [x0, #2592] cb030002 sub x2, x0, x3 eb01005f cmp x2, x1 While this may not be a huge win on its own, moving the static branch will permit further optimization of the body of the function in subsequent patches. Signed-off-by: Mark Rutland Cc: Alexander Popov Cc: Andrew Morton Cc: Andy Lutomirski Cc: Kees Cook --- kernel/stackleak.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/kernel/stackleak.c b/kernel/stackleak.c index ddb5a7f48d69e..753eab797a04d 100644 --- a/kernel/stackleak.c +++ b/kernel/stackleak.c @@ -70,7 +70,7 @@ late_initcall(stackleak_sysctls_init); #define skip_erasing() false #endif /* CONFIG_STACKLEAK_RUNTIME_DISABLE */ -asmlinkage void noinstr stackleak_erase(void) +static __always_inline void __stackleak_erase(void) { /* It would be nice not to have 'kstack_ptr' and 'boundary' on stack */ unsigned long kstack_ptr = current->lowest_stack; @@ -78,9 +78,6 @@ asmlinkage void noinstr stackleak_erase(void) unsigned int poison_count = 0; const unsigned int depth = STACKLEAK_SEARCH_DEPTH / sizeof(unsigned long); - if (skip_erasing()) - return; - /* Check that 'lowest_stack' value is sane */ if (unlikely(kstack_ptr - boundary >= THREAD_SIZE)) kstack_ptr = boundary; @@ -125,6 +122,14 @@ asmlinkage void noinstr stackleak_erase(void) current->lowest_stack = current_top_of_stack() - THREAD_SIZE/64; } +asmlinkage void noinstr stackleak_erase(void) +{ + if (skip_erasing()) + return; + + __stackleak_erase(); +} + void __used __no_caller_saved_registers noinstr stackleak_track_stack(void) { unsigned long sp = current_stack_pointer; From patchwork Mon Apr 25 11:55:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12825652 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 05996C433EF for ; Mon, 25 Apr 2022 11:57:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=U2kA03aJZkfjoiux3jDmyn2ycuzX2VE3S0S8Bc7BRJY=; b=nCf8y4WBHWn+XQ fx1dMkdMZJzNHkCLLVHjT8F5H1C0DjdPV0lKlgW7vwLgyaZUXEJW1XGDIUFu/IOosXyESQFChXdRX ZH0nQeqk3TmN99iRN9rvDLMjSxSDAHm8oRwo3VSIYsLFFDDHruj9dG38h+3cCuWwK7tWk8qQ8Zz6t ymz6qe54pYnpLvksbjz6HwjryfJFVKtYl+JvTy9HHIZXGFUveNMh5/GsZooSKXxprEMELyzhkdWCH zeYnGex1g7U1n00xeu38sed0En8EcnQWdYtpY+1lGJdlaD6ovPM7qGzRabN6uPGH7YvbO/HwvSDIu eo8B1nUPPg3k7lIyCgtA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKg-009NyR-R9; Mon, 25 Apr 2022 11:56:46 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKE-009Nq1-Gt for linux-arm-kernel@lists.infradead.org; Mon, 25 Apr 2022 11:56:21 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E69AC11FB; Mon, 25 Apr 2022 04:56:16 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AA4FC3F73B; Mon, 25 Apr 2022 04:56:15 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, alex.popov@linux.com, catalin.marinas@arm.com, keescook@chromium.org, linux-arm-kernel@lists.infradead.org, luto@kernel.org, mark.rutland@arm.com, will@kernel.org Subject: [PATCH 3/8] stackleak: rework stack low bound handling Date: Mon, 25 Apr 2022 12:55:58 +0100 Message-Id: <20220425115603.781311-4-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220425115603.781311-1-mark.rutland@arm.com> References: <20220425115603.781311-1-mark.rutland@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220425_045618_744882_B234C378 X-CRM114-Status: GOOD ( 19.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In stackleak_task_init(), stackleak_track_stack(), and __stackleak_erase(), we open-code skipping the STACK_END_MAGIC at the bottom of the stack. Each case is implemented slightly differently, and only the __stackleak_erase() case is commented. In stackleak_task_init() and stackleak_track_stack() we unconditionally add sizeof(unsigned long) to the lowest stack address. In stackleak_task_init() we use end_of_stack() for this, and in stackleak_track_stack() we use task_stack_page(). In __stackleak_erase() we handle this by detecting if `kstack_ptr` has hit the stack end boundary, and if so, conditionally moving it above the magic. This patch adds a new stackleak_task_low_bound() helper which is used in all three cases, which unconditionally adds sizeof(unsigned long) to the lowest address on the task stack, with commentary as to why. This uses end_of_stack() as stackleak_task_init() did prior to this patch, as this is consistent with the code in kernel/fork.c which initializes the STACK_END_MAGIC value. In __stackleak_erase() we no longer need to check whether we've spilled into the STACK_END_MAGIC value, as stackleak_track_stack() ensures that `current->lowest_stack` stops immediately above this, and similarly the poison scan will stop immediately above this. For stackleak_task_init() and stackleak_track_stack() this results in no change to code generation. For __stackleak_erase() the generated assembly is slightly simpler and shorter. Signed-off-by: Mark Rutland Cc: Alexander Popov Cc: Andrew Morton Cc: Andy Lutomirski Cc: Kees Cook --- include/linux/stackleak.h | 15 ++++++++++++++- kernel/stackleak.c | 14 ++++---------- 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/include/linux/stackleak.h b/include/linux/stackleak.h index ccaab2043fcd5..67430faa5c518 100644 --- a/include/linux/stackleak.h +++ b/include/linux/stackleak.h @@ -15,9 +15,22 @@ #ifdef CONFIG_GCC_PLUGIN_STACKLEAK #include +/* + * The lowest address on tsk's stack which we can plausibly erase. + */ +static __always_inline unsigned long +stackleak_task_low_bound(const struct task_struct *tsk) +{ + /* + * The lowest unsigned long on the task stack contains STACK_END_MAGIC, + * which we must not corrupt. + */ + return (unsigned long)end_of_stack(tsk) + sizeof(unsigned long); +} + static inline void stackleak_task_init(struct task_struct *t) { - t->lowest_stack = (unsigned long)end_of_stack(t) + sizeof(unsigned long); + t->lowest_stack = stackleak_task_low_bound(t); # ifdef CONFIG_STACKLEAK_METRICS t->prev_lowest_stack = t->lowest_stack; # endif diff --git a/kernel/stackleak.c b/kernel/stackleak.c index 753eab797a04d..0472956d9a2ce 100644 --- a/kernel/stackleak.c +++ b/kernel/stackleak.c @@ -72,9 +72,11 @@ late_initcall(stackleak_sysctls_init); static __always_inline void __stackleak_erase(void) { + const unsigned long task_stack_low = stackleak_task_low_bound(current); + /* It would be nice not to have 'kstack_ptr' and 'boundary' on stack */ unsigned long kstack_ptr = current->lowest_stack; - unsigned long boundary = (unsigned long)end_of_stack(current); + unsigned long boundary = task_stack_low; unsigned int poison_count = 0; const unsigned int depth = STACKLEAK_SEARCH_DEPTH / sizeof(unsigned long); @@ -92,13 +94,6 @@ static __always_inline void __stackleak_erase(void) kstack_ptr -= sizeof(unsigned long); } - /* - * One 'long int' at the bottom of the thread stack is reserved and - * should not be poisoned (see CONFIG_SCHED_STACK_END_CHECK=y). - */ - if (kstack_ptr == boundary) - kstack_ptr += sizeof(unsigned long); - #ifdef CONFIG_STACKLEAK_METRICS current->prev_lowest_stack = kstack_ptr; #endif @@ -144,8 +139,7 @@ void __used __no_caller_saved_registers noinstr stackleak_track_stack(void) /* 'lowest_stack' should be aligned on the register width boundary */ sp = ALIGN(sp, sizeof(unsigned long)); if (sp < current->lowest_stack && - sp >= (unsigned long)task_stack_page(current) + - sizeof(unsigned long)) { + sp >= stackleak_task_low_bound(current)) { current->lowest_stack = sp; } } From patchwork Mon Apr 25 11:55:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12825653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 492E6C433EF for ; Mon, 25 Apr 2022 11:57:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bRuLu+l74v5J0EuqytzyOj3KK/zOrcyFdpxz1U50uAc=; b=XWkyhDXCJ74Lg7 mo7EkiHfCrVunD3eRr8O/H0SRoZXvTHcnFC+SEa8L+2sezrIWEphrvcPqCjJ9RKQ3mRgkjZOrFJvn 6+hdprs8JAnBUczeJyd+xkuZLNeEqyQjPQyDqV7JQFD12AXo6LMw8P8DYD4Ei0LEvu4rbev2pU69K sCOEVJWBTsMTQ045F7Qb1lGv+6zp+T9C4r/zD+RHHj8MMaNCRo+KkBNC3I/Nu1bW+lKXyojVr9SNx XMMYgD82W9PUbo2feqkcazxFMinLu3D9AeLc6ehYMvgH7QCNQ7IFlIMQ3/Cv52hMJSh0p7eRKl4KY 34RDqFXei3AoDKHZ96HQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKr-009O1d-37; Mon, 25 Apr 2022 11:56:57 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKF-009Nov-Pd for linux-arm-kernel@lists.infradead.org; Mon, 25 Apr 2022 11:56:22 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EAA8D1FB; Mon, 25 Apr 2022 04:56:18 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 92BDF3F73B; Mon, 25 Apr 2022 04:56:17 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, alex.popov@linux.com, catalin.marinas@arm.com, keescook@chromium.org, linux-arm-kernel@lists.infradead.org, luto@kernel.org, mark.rutland@arm.com, will@kernel.org Subject: [PATCH 4/8] stackleak: clarify variable names Date: Mon, 25 Apr 2022 12:55:59 +0100 Message-Id: <20220425115603.781311-5-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220425115603.781311-1-mark.rutland@arm.com> References: <20220425115603.781311-1-mark.rutland@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220425_045620_476235_28C66778 X-CRM114-Status: GOOD ( 17.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The logic within __stackleak_erase() can be a little hard to follow, as `boundary` switches from being the low bound to the high bound mid way through the function, and `kstack_ptr` is used to represent the start of the region to erase while `boundary` represents the end of the region to erase. Make this a little clearer by consistently using clearer variable names. The `boundary` variable is removed, the bounds of the region to erase are described by `erase_low` and `erase_high`, and bounds of the task stack are described by `task_stack_low` and `task_stck_high`. As the same time, remove the comment above the variables, since it is unclear whether it's intended as rationale, a complaint, or a TODO, and is more confusing than helpful. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Alexander Popov Cc: Andrew Morton Cc: Andy Lutomirski Cc: Kees Cook --- kernel/stackleak.c | 34 ++++++++++++++++------------------ 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/kernel/stackleak.c b/kernel/stackleak.c index 0472956d9a2ce..8fbc1e4c21435 100644 --- a/kernel/stackleak.c +++ b/kernel/stackleak.c @@ -73,44 +73,42 @@ late_initcall(stackleak_sysctls_init); static __always_inline void __stackleak_erase(void) { const unsigned long task_stack_low = stackleak_task_low_bound(current); - - /* It would be nice not to have 'kstack_ptr' and 'boundary' on stack */ - unsigned long kstack_ptr = current->lowest_stack; - unsigned long boundary = task_stack_low; + unsigned long erase_low = current->lowest_stack; + unsigned long erase_high; unsigned int poison_count = 0; const unsigned int depth = STACKLEAK_SEARCH_DEPTH / sizeof(unsigned long); /* Check that 'lowest_stack' value is sane */ - if (unlikely(kstack_ptr - boundary >= THREAD_SIZE)) - kstack_ptr = boundary; + if (unlikely(erase_low - task_stack_low >= THREAD_SIZE)) + erase_low = task_stack_low; /* Search for the poison value in the kernel stack */ - while (kstack_ptr > boundary && poison_count <= depth) { - if (*(unsigned long *)kstack_ptr == STACKLEAK_POISON) + while (erase_low > task_stack_low && poison_count <= depth) { + if (*(unsigned long *)erase_low == STACKLEAK_POISON) poison_count++; else poison_count = 0; - kstack_ptr -= sizeof(unsigned long); + erase_low -= sizeof(unsigned long); } #ifdef CONFIG_STACKLEAK_METRICS - current->prev_lowest_stack = kstack_ptr; + current->prev_lowest_stack = erase_low; #endif /* - * Now write the poison value to the kernel stack. Start from - * 'kstack_ptr' and move up till the new 'boundary'. We assume that - * the stack pointer doesn't change when we write poison. + * Now write the poison value to the kernel stack between 'erase_low' + * and 'erase_high'. We assume that the stack pointer doesn't change + * when we write poison. */ if (on_thread_stack()) - boundary = current_stack_pointer; + erase_high = current_stack_pointer; else - boundary = current_top_of_stack(); + erase_high = current_top_of_stack(); - while (kstack_ptr < boundary) { - *(unsigned long *)kstack_ptr = STACKLEAK_POISON; - kstack_ptr += sizeof(unsigned long); + while (erase_low < erase_high) { + *(unsigned long *)erase_low = STACKLEAK_POISON; + erase_low += sizeof(unsigned long); } /* Reset the 'lowest_stack' value for the next syscall */ From patchwork Mon Apr 25 11:56:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12825654 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2E197C433EF for ; Mon, 25 Apr 2022 11:58:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7h9E9F6w2aBnN6PKHY665ixULA+F8PsB5xlZk0VJEOM=; b=lXLkOapUqd4szW udMY55vSxQo+gl+geb7RVmsx9UxCqt71cNNBQC1GAvPdvArHQtizi6KR6mGlViVwTQkHQ2NoIXsSX Vuubc/yHCeAKGlz0HmjCDMLKXK7+ROjwB99+KkOnxSng8qnxi97OzL0OrQdY3PEGncPXha9m+YHpQ MfPuKU5or58YwSaMnS+y7ML3W1HbuodI2HIFZEyarK9r7qkkQnSx+A/mXvpXvnINWr0l/ISvU56y/ yDQGSbDou+2Jyt4RKAcfMbGxF0YqHGWZqGfqs1oJcpuFVpJlC5wpX6dauH8UA0x/6dEKQ/s52LN3z 9yUxrFm7vXkY2bs/8ikw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixL5-009O8O-N3; Mon, 25 Apr 2022 11:57:11 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKH-009Nq1-Nx for linux-arm-kernel@lists.infradead.org; Mon, 25 Apr 2022 11:56:24 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B6DEAED1; Mon, 25 Apr 2022 04:56:20 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7826C3F73B; Mon, 25 Apr 2022 04:56:19 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, alex.popov@linux.com, catalin.marinas@arm.com, keescook@chromium.org, linux-arm-kernel@lists.infradead.org, luto@kernel.org, mark.rutland@arm.com, will@kernel.org Subject: [PATCH 5/8] stackleak: rework stack high bound handling Date: Mon, 25 Apr 2022 12:56:00 +0100 Message-Id: <20220425115603.781311-6-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220425115603.781311-1-mark.rutland@arm.com> References: <20220425115603.781311-1-mark.rutland@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220425_045622_005361_CB0243FC X-CRM114-Status: GOOD ( 23.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Prior to returning to userpace, we reset current->lowest_stack to a reasonable high bound. Currently we do this by subtracting the arbitrary value `THREAD_SIZE/64` from the top of the stack, for reasons lost to history. Looking at configurations today: * On i386 where THREAD_SIZE is 8K, the bound will be 128 bytes. The pt_regs at the top of the stack is 68 bytes (with 0 to 16 bytes of padding above), and so this covers an additional portion of 44 to 60 bytes. * On x86_64 where THREAD_SIZE is at least 16K (up to 32K with KASAN) the bound will be at least 256 bytes (up to 512 with KASAN). The pt_regs at the top of the stack is 168 bytes, and so this cover an additional 88 bytes of stack (up to 344 with KASAN). * On arm64 where THREAD_SIZE is at least 16K (up to 64K with 64K pages and VMAP_STACK), the bound will be at least 256 bytes (up to 1024 with KASAN). The pt_regs at the top of the stack is 336 bytes, so this can fall within the pt_regs, or can cover an additional 688 bytes of stack. Clearly the `THREAD_SIZE/64` value doesn't make much sense -- in the worst case, this will cause more than 600 bytes of stack to be erased for every syscall, even if actual stack usage were substantially smaller. This patches makes this slightly less nonsensical by consistently resetting current->lowest_stack to the base of the task pt_regs. For clarity and for consistency with the handling of the low bound, the generation of the high bound is split into a helper with commentary explaining why. Since the pt_regs at the top of the stack will be clobbered upon the next exception entry, we don't need to poison these at exception exit. By using task_pt_regs() as the high stack boundary instead of current_top_of_stack() we avoid some redundant poisoning, and the compiler can share the address generation between the poisoning and restting of `current->lowest_stack`, making the generated code more optimal. It's not clear to me whether the existing `THREAD_SIZE/64` offset was a dodgy heuristic to skip the pt_regs, or whether it was attempting to minimize the number of times stackleak_check_stack() would have to update `current->lowest_stack` when stack usage was shallow at the cost of unconditionally poisoning a small portion of the stack for every exit to userspace. For now I've simply removed the offset, and if we need/want to minimize updates for shallow stack usage it should be easy to add a better heuristic atop, with appropriate commentary so we know what's going on. Signed-off-by: Mark Rutland Cc: Alexander Popov Cc: Andrew Morton Cc: Andy Lutomirski Cc: Kees Cook --- include/linux/stackleak.h | 14 ++++++++++++++ kernel/stackleak.c | 19 ++++++++++++++----- 2 files changed, 28 insertions(+), 5 deletions(-) diff --git a/include/linux/stackleak.h b/include/linux/stackleak.h index 67430faa5c518..467661aeb4136 100644 --- a/include/linux/stackleak.h +++ b/include/linux/stackleak.h @@ -28,6 +28,20 @@ stackleak_task_low_bound(const struct task_struct *tsk) return (unsigned long)end_of_stack(tsk) + sizeof(unsigned long); } +/* + * The address immediately after the highest address on tsk's stack which we + * can plausibly erase. + */ +static __always_inline unsigned long +stackleak_task_high_bound(const struct task_struct *tsk) +{ + /* + * The task's pt_regs lives at the top of the task stack and will be + * overwritten by exception entry, so there's no need to erase them. + */ + return (unsigned long)task_pt_regs(tsk); +} + static inline void stackleak_task_init(struct task_struct *t) { t->lowest_stack = stackleak_task_low_bound(t); diff --git a/kernel/stackleak.c b/kernel/stackleak.c index 8fbc1e4c21435..f597d3323729d 100644 --- a/kernel/stackleak.c +++ b/kernel/stackleak.c @@ -73,6 +73,7 @@ late_initcall(stackleak_sysctls_init); static __always_inline void __stackleak_erase(void) { const unsigned long task_stack_low = stackleak_task_low_bound(current); + const unsigned long task_stack_high = stackleak_task_high_bound(current); unsigned long erase_low = current->lowest_stack; unsigned long erase_high; unsigned int poison_count = 0; @@ -97,14 +98,22 @@ static __always_inline void __stackleak_erase(void) #endif /* - * Now write the poison value to the kernel stack between 'erase_low' - * and 'erase_high'. We assume that the stack pointer doesn't change - * when we write poison. + * Write poison to the task's stack between 'erase_low' and + * 'erase_high'. + * + * If we're running on a different stack (e.g. an entry trampoline + * stack) we can erase everything below the pt_regs at the top of the + * task stack. + * + * If we're running on the task stack itself, we must not clobber any + * stack used by this function and its caller. We assume that this + * function has a fixed-size stack frame, and the current stack pointer + * doesn't change while we write poison. */ if (on_thread_stack()) erase_high = current_stack_pointer; else - erase_high = current_top_of_stack(); + erase_high = task_stack_high; while (erase_low < erase_high) { *(unsigned long *)erase_low = STACKLEAK_POISON; @@ -112,7 +121,7 @@ static __always_inline void __stackleak_erase(void) } /* Reset the 'lowest_stack' value for the next syscall */ - current->lowest_stack = current_top_of_stack() - THREAD_SIZE/64; + current->lowest_stack = task_stack_high; } asmlinkage void noinstr stackleak_erase(void) From patchwork Mon Apr 25 11:56:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12825667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 11C67C433EF for ; Mon, 25 Apr 2022 11:59:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ifb3Kx9X3qM5FpTFoXvzFvsJPHaWw/mcv3Kn879aUU0=; b=ZZMmh6i98CC2DK D1cj3XEZkP0luNjfsrCNixUjy2XRdXeQoM6+R3PUhGLSfdEKcxmRpTdRWiDw61YjX89ufVB8A+KUf Z8djx1H7UpYY6ue/mXt5+4YMgn55sgN+Lsck66IgXO63VXYMn9ZR0pjVkl+da9THVdbWM7dsQZ7dX 0l0XqV2EpsUm+YmrmHn8czibkjbRcqk3MOmXdjp9pzeFUkmo4GaiEjFbew3dBkTAjXS6dByKHJg1A glG9AiTare2/3XljlZoa6gSOfqUT6PUzLGMlzPuU3Lx21stT+DMbQiSx9ERjoGD+LdH1Lf+iydTwV Ucmh12vqp+sPZsEqEa3w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixMA-009OnD-D6; Mon, 25 Apr 2022 11:58:18 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKO-009NsX-OT for linux-arm-kernel@lists.infradead.org; Mon, 25 Apr 2022 11:56:30 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C8A04113E; Mon, 25 Apr 2022 04:56:22 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7120A3F73B; Mon, 25 Apr 2022 04:56:21 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, alex.popov@linux.com, catalin.marinas@arm.com, keescook@chromium.org, linux-arm-kernel@lists.infradead.org, luto@kernel.org, mark.rutland@arm.com, will@kernel.org Subject: [PATCH 6/8] stackleak: remove redundant check Date: Mon, 25 Apr 2022 12:56:01 +0100 Message-Id: <20220425115603.781311-7-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220425115603.781311-1-mark.rutland@arm.com> References: <20220425115603.781311-1-mark.rutland@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220425_045628_894959_DCD31F73 X-CRM114-Status: GOOD ( 11.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In __stackleak_erase() we check that the `erase_low` value derived from `current->lowest_stack` is above the lowest legitimate stack pointer value, but this is already enforced by stackleak_track_stack() when recording the lowest stack value. Remove the redundant check. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Alexander Popov Cc: Andrew Morton Cc: Andy Lutomirski Cc: Kees Cook --- kernel/stackleak.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/kernel/stackleak.c b/kernel/stackleak.c index f597d3323729d..ba346d46218f5 100644 --- a/kernel/stackleak.c +++ b/kernel/stackleak.c @@ -79,10 +79,6 @@ static __always_inline void __stackleak_erase(void) unsigned int poison_count = 0; const unsigned int depth = STACKLEAK_SEARCH_DEPTH / sizeof(unsigned long); - /* Check that 'lowest_stack' value is sane */ - if (unlikely(erase_low - task_stack_low >= THREAD_SIZE)) - erase_low = task_stack_low; - /* Search for the poison value in the kernel stack */ while (erase_low > task_stack_low && poison_count <= depth) { if (*(unsigned long *)erase_low == STACKLEAK_POISON) From patchwork Mon Apr 25 11:56:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12825665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 68E50C433F5 for ; Mon, 25 Apr 2022 11:58:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yAPWfc6av/FyWIfVbhSkN/+rqfzLa7hrixWUYSAqLo0=; b=kEDmGrwGpisTJp FERQdPwNzSXH+RndGDQWNLp7E17T0Ptlr53LtLxZJQdeXVT1ypUGIERAfWEVZxzYdecR2+7Rni3vd 6tWxYkrWqL9pojgT3b6/mBE+ikyBl+IT9dSnUoCA7IPr8OTesVQ0D6FE0i65KJzKFZqDT3J5kuvQ0 7F6jZg11FVCx6y24O3hY2bPPcmblLMesNGCj6wQJ74GfOCGt1uPfQTi3G1Nuv2PTti8NXtub1g6gE Nl6Y44A2WqhYjRyyRv6l86wsyN9r1ZeHaMIavDxd9jFDYL7DC3Sjv0K57jxrmSu/QnMh/narm5KbM hX3VHcS1mK3p/t8QlTkA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixLR-009OKA-O5; Mon, 25 Apr 2022 11:57:33 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKL-009Nq1-AH for linux-arm-kernel@lists.infradead.org; Mon, 25 Apr 2022 11:56:27 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AF4121FB; Mon, 25 Apr 2022 04:56:24 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6A8283F73B; Mon, 25 Apr 2022 04:56:23 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, alex.popov@linux.com, catalin.marinas@arm.com, keescook@chromium.org, linux-arm-kernel@lists.infradead.org, luto@kernel.org, mark.rutland@arm.com, will@kernel.org Subject: [PATCH 7/8] stackleak: add on/off stack variants Date: Mon, 25 Apr 2022 12:56:02 +0100 Message-Id: <20220425115603.781311-8-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220425115603.781311-1-mark.rutland@arm.com> References: <20220425115603.781311-1-mark.rutland@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220425_045625_504255_AB201746 X-CRM114-Status: GOOD ( 15.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The stackleak_erase() code dynamically handles being on a task stack or another stack. In most cases, this is a fixed property of the caller, which the caller is aware of, as an architecture might always return using the task stack, or might always return using a trampoline stack. This patch adds stackleak_erase_on_task_stack() and stackleak_erase_off_task_stack() functions which callers can use to avoid on_thread_stack() check and associated redundant work when the calling stack is known. The existing stackleak_erase() is retained as a safe default. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Alexander Popov Cc: Andrew Morton Cc: Andy Lutomirski Cc: Kees Cook --- kernel/stackleak.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/kernel/stackleak.c b/kernel/stackleak.c index ba346d46218f5..4c3f82066d7fb 100644 --- a/kernel/stackleak.c +++ b/kernel/stackleak.c @@ -70,7 +70,7 @@ late_initcall(stackleak_sysctls_init); #define skip_erasing() false #endif /* CONFIG_STACKLEAK_RUNTIME_DISABLE */ -static __always_inline void __stackleak_erase(void) +static __always_inline void __stackleak_erase(bool on_task_stack) { const unsigned long task_stack_low = stackleak_task_low_bound(current); const unsigned long task_stack_high = stackleak_task_high_bound(current); @@ -106,7 +106,7 @@ static __always_inline void __stackleak_erase(void) * function has a fixed-size stack frame, and the current stack pointer * doesn't change while we write poison. */ - if (on_thread_stack()) + if (on_task_stack) erase_high = current_stack_pointer; else erase_high = task_stack_high; @@ -120,12 +120,41 @@ static __always_inline void __stackleak_erase(void) current->lowest_stack = task_stack_high; } +/* + * Erase and poison the portion of the task stack used since the last erase. + * Can be called from the task stack or an entry stack when the task stack is + * no longer in use. + */ asmlinkage void noinstr stackleak_erase(void) { if (skip_erasing()) return; - __stackleak_erase(); + __stackleak_erase(on_thread_stack()); +} + +/* + * Erase and poison the portion of the task stack used since the last erase. + * Can only be called from the task stack. + */ +asmlinkage void noinstr stackleak_erase_on_task_stack(void) +{ + if (skip_erasing()) + return; + + __stackleak_erase(true); +} + +/* + * Erase and poison the portion of the task stack used since the last erase. + * Can only be called from a stack other than the task stack. + */ +asmlinkage void noinstr stackleak_erase_off_task_stack(void) +{ + if (skip_erasing()) + return; + + __stackleak_erase(false); } void __used __no_caller_saved_registers noinstr stackleak_track_stack(void) From patchwork Mon Apr 25 11:56:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12825666 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 74EC0C433F5 for ; Mon, 25 Apr 2022 11:59:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=aPBPHS4ZmYucz50JgF4Bln82sfk/5GbAHkMBI9YbdqU=; b=YOr4G26MOi/Wyr rqCd6RMb5SzqdJ5Jh7TTIPnYUiJZ7HtYSJBWwFjwJgLOYS0eyqwGo1xPS/PBkZPuKu4Uz2hScczNp gmphaAio9nXR+5di/v9VN2WFCoyCXrAnShsAMGJI+qqJeoHjLoyI8rUal5sDs3pc5wIfBKmrAyU9m w9bQHXnomgGsMiT+kMJt+aWDCxrtLvXC2FACwTKLHcvQtGUOCQM38tqjIfjflPdo/DyQv2kAzrUoK YAi0bBrms1FvH2xYC0lasGgmFf6neiZcaQMOTNAkhEV87VwcU4LFhOeLcrmU5qpLKYy2PNKRc30NJ cOhgYlW0q9xa3EYJbC5Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixLt-009OcO-Da; Mon, 25 Apr 2022 11:58:01 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nixKN-009NsI-Jc for linux-arm-kernel@lists.infradead.org; Mon, 25 Apr 2022 11:56:30 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9A462ED1; Mon, 25 Apr 2022 04:56:26 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5C54E3F73B; Mon, 25 Apr 2022 04:56:25 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, alex.popov@linux.com, catalin.marinas@arm.com, keescook@chromium.org, linux-arm-kernel@lists.infradead.org, luto@kernel.org, mark.rutland@arm.com, will@kernel.org Subject: [PATCH 8/8] arm64: entry: use stackleak_erase_on_task_stack() Date: Mon, 25 Apr 2022 12:56:03 +0100 Message-Id: <20220425115603.781311-9-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220425115603.781311-1-mark.rutland@arm.com> References: <20220425115603.781311-1-mark.rutland@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220425_045628_421875_3F8AF6A4 X-CRM114-Status: GOOD ( 11.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On arm64 we always call stackleak_erase() on a task stack, and never call it on another stack. We can avoid some redundant work by using stackleak_erase_on_task_stack(), telling the stackleak code that it's being called on a task stack. Signed-off-by: Mark Rutland Cc: Alexander Popov Cc: Andrew Morton Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Kees Cook Cc: Will Deacon --- arch/arm64/kernel/entry.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index ede028dee81b0..5b82b92924005 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -596,7 +596,7 @@ SYM_CODE_START_LOCAL(ret_to_user) ldr x19, [tsk, #TSK_TI_FLAGS] // re-check for single-step enable_step_tsk x19, x2 #ifdef CONFIG_GCC_PLUGIN_STACKLEAK - bl stackleak_erase + bl stackleak_erase_on_task_stack #endif kernel_exit 0 SYM_CODE_END(ret_to_user)