From patchwork Wed Apr 13 19:26:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 12812550 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3D0E5C433F5 for ; Wed, 13 Apr 2022 19:46:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mbzVo0Y/BdoolubeVMZMSOzOcgRP/NBxrpUCE+xzSyo=; b=Azpie1u3wj+Fza 5P+nQc7rwgfU+j6u+c830QUmsw6w++YShUAOP5L+ZZnkSGe/Ms4ErEeGsD8MnicLP/BGz0ffQnCxo SDNRTPNuQ0LtgO4o9zqsiZLii6D4g5atRw1vHdOm1X/m7E/y/fQAoiO6iDOyPmB8TwVdovdqnuZ6R b49Ejq3BLE5EfxAhHReinTYrfPLoOrPBDbALeVMx2BO6sSGW/zhmQSMXhdWGIdGqv/EO4K1zIws4N LVJSkWjCoGmD4dVPWtcNNBxsZQyM6tgeTFyYFYdFYT0DHKERtse5r4QgKDiWfqnOJmgB/XsCV5Zr4 SAto/2d5Fcr7Jj2i4Msw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1neive-002YjF-LN; Wed, 13 Apr 2022 19:45:27 +0000 Received: from out2.migadu.com ([188.165.223.204]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1neif8-002ROA-EH for linux-arm-kernel@lists.infradead.org; Wed, 13 Apr 2022 19:28:24 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1649878095; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k1Ti9WI7kQjXH/LtbtJpqTp8bomGV2aI21uzEA+/hgk=; b=nykjmmr0sgX45Kb38uAO4roxOKHuFvGqKVhaS7htTLEqnKboJlyPfvkJ7xqMlLAFN7fzbn 3YYgvkl0RZ4Ec+gP9PtkVsgLLXTl5jPx7gAw4RlzgAdUTBNII6VaYVnmBXQoYKlKY9XQnv 1JRuxhqmlIdwpkIELUxCGbVISHZ0MR8= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko , Mark Rutland Cc: Andrey Konovalov , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, Catalin Marinas , Will Deacon , Vincenzo Frascino , Sami Tolvanen , linux-arm-kernel@lists.infradead.org, Peter Collingbourne , Evgenii Stepanov , Florian Mayer , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v3 1/3] arm64, scs: expose irq_shadow_call_stack_ptr Date: Wed, 13 Apr 2022 21:26:44 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220413_122822_789271_D0DF04F4 X-CRM114-Status: UNSURE ( 9.44 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Andrey Konovalov When a hardirq happens, the Shadow Call Stack (SCS) pointer is switched to a per-hardirq one, which is stored in irq_shadow_call_stack_ptr per-CPU. Collecting stack traces based on SCS in hardirqs requires access to irq_shadow_call_stack_ptr. Expose irq_shadow_call_stack_ptr in arch/arm64/include/asm/scs.h. For symmetry, move sdei_shadow_call_stack_normal_ptr and sdei_shadow_call_stack_critical_ptr declarations there too. Signed-off-by: Andrey Konovalov --- arch/arm64/include/asm/scs.h | 10 +++++++++- arch/arm64/kernel/irq.c | 4 +--- arch/arm64/kernel/sdei.c | 3 --- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 8297bccf0784..9d9ac04014f1 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -24,6 +24,14 @@ .endm #endif /* CONFIG_SHADOW_CALL_STACK */ -#endif /* __ASSEMBLY __ */ +#else /* __ASSEMBLY__ */ + +#include + +DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); +DECLARE_PER_CPU(unsigned long *, sdei_shadow_call_stack_normal_ptr); +DECLARE_PER_CPU(unsigned long *, sdei_shadow_call_stack_critical_ptr); + +#endif /* __ASSEMBLY__ */ #endif /* _ASM_SCS_H */ diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index bda49430c9ea..9d85c82f0b1c 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -21,6 +21,7 @@ #include #include #include +#include #include /* Only access this in an NMI enter/exit */ @@ -28,9 +29,6 @@ DEFINE_PER_CPU(struct nmi_ctx, nmi_contexts); DEFINE_PER_CPU(unsigned long *, irq_stack_ptr); - -DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); - #ifdef CONFIG_SHADOW_CALL_STACK DEFINE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); #endif diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c index d20620a1c51a..882f88ce8d4d 100644 --- a/arch/arm64/kernel/sdei.c +++ b/arch/arm64/kernel/sdei.c @@ -39,9 +39,6 @@ DEFINE_PER_CPU(unsigned long *, sdei_stack_normal_ptr); DEFINE_PER_CPU(unsigned long *, sdei_stack_critical_ptr); #endif -DECLARE_PER_CPU(unsigned long *, sdei_shadow_call_stack_normal_ptr); -DECLARE_PER_CPU(unsigned long *, sdei_shadow_call_stack_critical_ptr); - #ifdef CONFIG_SHADOW_CALL_STACK DEFINE_PER_CPU(unsigned long *, sdei_shadow_call_stack_normal_ptr); DEFINE_PER_CPU(unsigned long *, sdei_shadow_call_stack_critical_ptr); From patchwork Wed Apr 13 19:26:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 12812548 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7DA9BC433EF for ; Wed, 13 Apr 2022 19:45:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=POC/SDjLdNT+NossS15irMFPLFE0DObUepXV4jA9Ro4=; b=W3YtUppCFme6ME hrgR+O7F1D0S6C7ZSu7LXzNwjwJkPS4pq7aDUv3Bsz62/jZ1d5avLPlcEgsEoAcReIkhFcOmrioYf cN4tVNOeq65Qf9qSln6O0wI58KCTmj0JTS/DB0SGZ4E9QXbgWFp/cql/u3e7lvXE2Jn6YkJY2Bw3O mEMSIKYkdCa0rRkdyRyuZy69gc/IPnTGRjXYGjTX6OK/oQze/jlesTEm0lLgWF7lCj8Y5ueirGpAL BMXa+wv1FUYk2JEXM6j4r/8rX38NWNTv1cWFPEbzu2+7PcPk1JtJd8Ksfo/9qhS5KKT0SQddvG0Pk YEZAtRTVaHQMeQhIzkNg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1neiu3-002Y1M-8d; Wed, 13 Apr 2022 19:43:47 +0000 Received: from out2.migadu.com ([2001:41d0:2:aacc::]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1neif7-002ROj-70 for linux-arm-kernel@lists.infradead.org; Wed, 13 Apr 2022 19:28:23 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1649878096; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1W/iPGxZZe0dTxtRIT/L4inWkaCNOxDFyHeQ0vhNfa0=; b=gnwkJMKPPygIVkZ/9Wh1mh3OT1cMRRjuS5qS8IK6a6j48F8Mu3vVEr6wfetKEVn+G0kwVH VpnSFOGe6lHZ16nJIfVYj83+P9+IFGSaJ9oP5g4LZ79gI0ZQ4hAcFfPGXw47pP2GkGazjs X7uQvydkAJIrVzsEfjwJGtTH9md0qaI= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko , Mark Rutland Cc: Andrey Konovalov , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, Catalin Marinas , Will Deacon , Vincenzo Frascino , Sami Tolvanen , linux-arm-kernel@lists.infradead.org, Peter Collingbourne , Evgenii Stepanov , Florian Mayer , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v3 2/3] kasan, arm64: implement stack_trace_save_shadow Date: Wed, 13 Apr 2022 21:26:45 +0200 Message-Id: <78cd352296ceb14da1d0136ff7d0a6818e594ab7.1649877511.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220413_122821_606697_5B6C955D X-CRM114-Status: GOOD ( 17.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Andrey Konovalov Implement stack_trace_save_shadow() that collects stack traces based on the Shadow Call Stack (SCS) for arm64 by copiing the frames from SCS. The implementation is best-effort and thus has limitations. stack_trace_save_shadow() fully handles task and softirq contexts, which are both processed on the per-task SCS. For hardirqs, the support is limited: stack_trace_save_shadow() does not collect the task part of the stack trace. For KASAN, this is not a problem, as stack depot only saves the interrupt part of the stack anyway. Otherwise, stack_trace_save_shadow() also takes a best-effort approach with a focus on performance. Thus, it: - Does not try to collect stack traces from other exceptions like SDEI. - Does not try to recover frames modified by KRETPROBES or by FTRACE. However, stack_trace_save_shadow() does strip PTR_AUTH tags to avoid leaking them in stack traces. The -ENOSYS return value is deliberatly used to match stack_trace_save_tsk_reliable(). Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index d9079ec11f31..23b30fa6e270 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -30,6 +30,68 @@ #include "kasan.h" #include "../slab.h" +#ifdef CONFIG_SHADOW_CALL_STACK +#include +#include + +/* + * Collect the stack trace from the Shadow Call Stack in a best-effort manner: + * + * - Do not collect the task part of the stack trace when in a hardirq. + * - Do not collect stack traces from other exception levels like SDEI. + * - Do not recover frames modified by KRETPROBES or by FTRACE. + * + * Note that marking the function with __noscs leads to unnacceptable + * performance impact, as helper functions stop being inlined. + */ +static inline int stack_trace_save_shadow(unsigned long *store, + unsigned int size) +{ + unsigned long *scs_top, *scs_base, *frame; + unsigned int len = 0; + + /* Get the SCS base. */ + if (in_task() || in_serving_softirq()) { + /* Softirqs reuse the task SCS area. */ + scs_base = task_scs(current); + } else if (in_hardirq()) { + /* Hardirqs use a per-CPU SCS area. */ + scs_base = *this_cpu_ptr(&irq_shadow_call_stack_ptr); + } else { + /* Ignore other exception levels. */ + return 0; + } + + /* + * Get the SCS pointer. + * + * Note that this assembly might be placed before the function's + * prologue. In this case, the last stack frame will be lost. This is + * acceptable: the lost frame will correspond to an internal KASAN + * function, which is not relevant to identify the external call site. + */ + asm volatile("mov %0, x18" : "=&r" (scs_top)); + + /* The top SCS slot is empty. */ + scs_top -= 1; + + for (frame = scs_top; frame >= scs_base; frame--) { + if (len >= size) + break; + /* Do not leak PTR_AUTH tags in stack traces. */ + store[len++] = ptrauth_strip_insn_pac(*frame); + } + + return len; +} +#else /* CONFIG_SHADOW_CALL_STACK */ +static inline int stack_trace_save_shadow(unsigned long *store, + unsigned int size) +{ + return -ENOSYS; +} +#endif /* CONFIG_SHADOW_CALL_STACK */ + depot_stack_handle_t kasan_save_stack(gfp_t flags, bool can_alloc) { unsigned long entries[KASAN_STACK_DEPTH]; From patchwork Wed Apr 13 19:26:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 12812549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 65BB2C433FE for ; Wed, 13 Apr 2022 19:45:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=j4ZeduU5LNu3UiWCkJ34aEjlp8Es/3YbG6+SdaVZues=; b=AqSSHwFVyNlawu bU2FUNcLQJ+8JNBSsAAv/4zEVw+nSR9mGvYhB3TbhyOSRve2FrdTCOMllNr/QayklHr3Uk6vy9DO9 wmBLvCNx4LQSsazD2pUqRqkXsmY/owH5bu2JCg9j9W0PmW2oY3Z0W73PgKMtT/8AtKlYlPE3CqG2X ZhNoxeWi6eB8FnpDlaYNAqf5LzPWCXzLK1gQvO6zUcsQ7bOyJu27wH4b8G9maLLYlDM3vBGaUbW3+ 6tdwkh+805dzDDF/sN9r1jB7J4r1wa2mZFvaJGjKalnuIcwxx0iYWr1tmA8hHxziIGxzp1bZMyV+p 4M8cNJRnS3s4dBFrNU0A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1neiuw-002YPl-GG; Wed, 13 Apr 2022 19:44:43 +0000 Received: from out2.migadu.com ([2001:41d0:2:aacc::]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1neif7-002RPP-8M for linux-arm-kernel@lists.infradead.org; Wed, 13 Apr 2022 19:28:22 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1649878097; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JDSn48VO/uynw53/sb4SAMVT5nBlYcr9rW/XWz8GxpM=; b=sgF7TidZ/P5xqTORl4THKhdzQ00G5F6Fn39wehah+VVZmo+WJOxBeHdzsNbNNIShn9WE6w Qt9Bazc4jEh/KaM/hD9VnKJ780fSIt7YjOm05xnrVlhblVT2a3/vrvFbj/2vEF002aW9G4 2nQnx8WcHwg902UfuFYSedzVMroR/2M= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko , Mark Rutland Cc: Andrey Konovalov , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, Catalin Marinas , Will Deacon , Vincenzo Frascino , Sami Tolvanen , linux-arm-kernel@lists.infradead.org, Peter Collingbourne , Evgenii Stepanov , Florian Mayer , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v3 3/3] kasan: use stack_trace_save_shadow Date: Wed, 13 Apr 2022 21:26:46 +0200 Message-Id: <05fb7a41510f471f82aa1f3930ed3aac8abe2410.1649877511.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220413_122821_476986_B010F972 X-CRM114-Status: UNSURE ( 9.96 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Andrey Konovalov Use stack_trace_save_shadow() to collect stack traces whenever CONFIG_HAVE_SHADOW_STACKTRACE is enabled. This improves the boot time of a defconfig build by ~30% for all KASAN modes. Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 23b30fa6e270..00fef2e5fe90 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -97,7 +97,9 @@ depot_stack_handle_t kasan_save_stack(gfp_t flags, bool can_alloc) unsigned long entries[KASAN_STACK_DEPTH]; unsigned int nr_entries; - nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0); + nr_entries = stack_trace_save_shadow(entries, ARRAY_SIZE(entries)); + if (nr_entries < 0) + nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0); return __stack_depot_save(entries, nr_entries, flags, can_alloc); }