From patchwork Wed Jan 8 18:56:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324361 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 01F5D1871 for ; Wed, 8 Jan 2020 18:57:16 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C854220705 for ; Wed, 8 Jan 2020 18:57:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Y2EmzNiR" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C854220705 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Kql5LVKSXq4fdpEQycE88gi4In78BACbe7oRB4FiIjo=; b=Y2EmzNiRnh/PEUkFeUNRXtlqQd 9aShC6z9dHGQIk10GretDGpvAeLqWlaS5h/0fifv4HQW/sKz1QZXRM4e8UHc3l+djxxc2S6FZVzhx ILIDpZ5pI0d1capaAHEy0ynveO6v3JXZPFnu/WDn8KeN0VG6YqF9imct/HTwXk/OlB/5W1Q7movwI uYSTnVxlR+kJ3se0SVriYYxAhVseeXnSrQsF9E9TdAa8rqxSEJ/PVSZ471tOlRI4/HTGtunoRMQfI SgbO0HDbBI69iHC1MNEfwFMpw3Hsa7+vJAXq6i4vRySc5jkmro5c5+/AkwBmgpy2rMxAtyDdeWMyf vBRN9VRA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGW0-0007cS-Vd; Wed, 08 Jan 2020 18:57:13 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGVh-0007IY-4V for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:56:54 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7DA98328; Wed, 8 Jan 2020 10:56:52 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EE0303F534; Wed, 8 Jan 2020 10:56:50 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 01/17] arm64: entry: mark all entry code as notrace Date: Wed, 8 Jan 2020 18:56:18 +0000 Message-Id: <20200108185634.1163-2-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105653_216753_D54D42B8 X-CRM114-Status: GOOD ( 11.07 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Almost all functions in entry-common.c are marked notrace, with el1_undef and el1_inv being the only exceptions. We appear to have done this on the assumption that there were no exception registers that we needed to snapshot, and thus it was safe to run trace code that might result in further exceptions and clobber those registers. However, until we inherit the DAIF flags, our irq flag tracing is stale, and this discrepancy could set off warnings in some configurations. Given we don't expect to trigger el1_undef or el1_inv unless something is already wrong, any irqflag warnigns are liable to mask the information we'd actually care about. Let's keep things simple and mark el1_undef and el1_inv as notrace. Developers can trace do_undefinstr and bad_mode if they really want to monitor these cases. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Will Deacon --- arch/arm64/kernel/entry-common.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 5dce5e56995a..67198142a0fc 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -36,14 +36,14 @@ static void notrace el1_pc(struct pt_regs *regs, unsigned long esr) } NOKPROBE_SYMBOL(el1_pc); -static void el1_undef(struct pt_regs *regs) +static void notrace el1_undef(struct pt_regs *regs) { local_daif_inherit(regs); do_undefinstr(regs); } NOKPROBE_SYMBOL(el1_undef); -static void el1_inv(struct pt_regs *regs, unsigned long esr) +static void notrace el1_inv(struct pt_regs *regs, unsigned long esr) { local_daif_inherit(regs); bad_mode(regs, 0, esr); From patchwork Wed Jan 8 18:56:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324363 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 75DF0930 for ; Wed, 8 Jan 2020 18:57:30 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 53EB620678 for ; Wed, 8 Jan 2020 18:57:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="L8daHkb7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 53EB620678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=QHQ+ZgrPJ5u6FaJ00jUh15UoI68FrC23SqJ84WRn2kQ=; b=L8daHkb7MBXhvzJBOV6zAREbXz E6ECAoto0STIiUusOauhjtoNwQxxT57VaBD9tuDSzLRMpb+zx9o8n95qWYMYiuCkS5cc4AYJzhX4l 1PJDYIQ/2T+gh/EIo5AgfuPCoFMmMToo1gxPwSWNWD7nTAX0QClzEHLblxuPtSXd+H+y2W+QjAsDc 5BEhgDY4bIEW5eSVf4xCoCUigY1yjowdSM49KsB/0NlVLKNMU5MftdjMrgda4XgjYw8Cv2EixO2En HGwJDQzBP2a6Fx4ToeFm5GUqer/fcmRci5Kmlpsazs1y3nwk225oG+aZmbu6XWfF6V7qHPDh8j+oj CjLac1PQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGWH-0007s2-Dh; Wed, 08 Jan 2020 18:57:29 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGVj-0007Jw-FI for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:56:57 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B84D01FB; Wed, 8 Jan 2020 10:56:54 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 347753F534; Wed, 8 Jan 2020 10:56:53 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 02/17] arm64: entry: cleanup el0 svc handler naming Date: Wed, 8 Jan 2020 18:56:19 +0000 Message-Id: <20200108185634.1163-3-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105655_574682_245D6BD3 X-CRM114-Status: UNSURE ( 9.26 ) X-CRM114-Notice: Please train this message. X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org For most of the exception entry code, _handler() is the first C function called from the entry assembly in entry-common.c, and external functions handling the bulk of the logic are called do_(). For consistency, apply this scheme to el0_svc_handler and el0_svc_compat_hander, renaming them to do_el0_svc and do_el0_svc_compat respectively. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Will Deacon Reviewed-by: Anshuman Khandual --- arch/arm64/include/asm/exception.h | 4 ++-- arch/arm64/kernel/entry-common.c | 4 ++-- arch/arm64/kernel/syscall.c | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/exception.h b/arch/arm64/include/asm/exception.h index 4d5f3b5f50cd..b87c6e276ab1 100644 --- a/arch/arm64/include/asm/exception.h +++ b/arch/arm64/include/asm/exception.h @@ -45,8 +45,8 @@ void do_sysinstr(unsigned int esr, struct pt_regs *regs); void do_sp_pc_abort(unsigned long addr, unsigned int esr, struct pt_regs *regs); void bad_el0_sync(struct pt_regs *regs, int reason, unsigned int esr); void do_cp15instr(unsigned int esr, struct pt_regs *regs); -void el0_svc_handler(struct pt_regs *regs); -void el0_svc_compat_handler(struct pt_regs *regs); +void do_el0_svc(struct pt_regs *regs); +void do_el0_svc_compat(struct pt_regs *regs); void do_el0_ia_bp_hardening(unsigned long addr, unsigned int esr, struct pt_regs *regs); diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 67198142a0fc..fde59981445c 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -215,7 +215,7 @@ static void notrace el0_svc(struct pt_regs *regs) if (system_uses_irq_prio_masking()) gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); - el0_svc_handler(regs); + do_el0_svc(regs); } NOKPROBE_SYMBOL(el0_svc); @@ -281,7 +281,7 @@ static void notrace el0_svc_compat(struct pt_regs *regs) if (system_uses_irq_prio_masking()) gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); - el0_svc_compat_handler(regs); + do_el0_svc_compat(regs); } NOKPROBE_SYMBOL(el0_svc_compat); diff --git a/arch/arm64/kernel/syscall.c b/arch/arm64/kernel/syscall.c index 9a9d98a443fc..a12c0c88d345 100644 --- a/arch/arm64/kernel/syscall.c +++ b/arch/arm64/kernel/syscall.c @@ -154,14 +154,14 @@ static inline void sve_user_discard(void) sve_user_disable(); } -void el0_svc_handler(struct pt_regs *regs) +void do_el0_svc(struct pt_regs *regs) { sve_user_discard(); el0_svc_common(regs, regs->regs[8], __NR_syscalls, sys_call_table); } #ifdef CONFIG_COMPAT -void el0_svc_compat_handler(struct pt_regs *regs) +void do_el0_svc_compat(struct pt_regs *regs) { el0_svc_common(regs, regs->regs[7], __NR_compat_syscalls, compat_sys_call_table); From patchwork Wed Jan 8 18:56:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324365 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 61300930 for ; Wed, 8 Jan 2020 18:57:45 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3B25520678 for ; Wed, 8 Jan 2020 18:57:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="b/HnqCFb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3B25520678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=M0Zz5l/4prVsPnviSH9ZqHmDAj1nI4k0VeGd7fmQNZo=; b=b/HnqCFbnk2VHYD2ASw+MODpDZ SXLTEF6JPdmrJTEuZ6U397kUiu6NE9xwQm5hNtQSaU+TpTHm8MS3NnBnMN++y7icNfU69Pwzzc3kx JDUW4ZNbT+TcrRRPGO3Jlb+UupsBXkJJqR3eraH6LTrd8vLFZ2muWjWIbT2oQh0AF5rUaGAccJ5QK txtD3sunC2TJXLgMBmf2R4Z8IL90qrbZLk2CeOq7U79BgZhdZXsxJVhYnp5ZkrRT3Vc+GPwi8gWMg U1y7lB+Z76TCNPr7HqzO7+W5fYZENf4phNq+UXCDLvzm5uNhtcdaTHxmy9VKNmg+WohfiJhhV6EDO MWNQqzEQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGWW-00085l-0O; Wed, 08 Jan 2020 18:57:44 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGVm-0007Of-QK for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:00 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EAA2B328; Wed, 8 Jan 2020 10:56:56 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 66AFB3F534; Wed, 8 Jan 2020 10:56:55 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 03/17] arm64: entry: move arm64_preempt_schedule_irq to entry-common.c Date: Wed, 8 Jan 2020 18:56:20 +0000 Message-Id: <20200108185634.1163-4-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105659_038022_7A39E0D2 X-CRM114-Status: GOOD ( 12.57 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Subsequent patches will pull more of the IRQ entry handling into C. To keep this in one place, let's move arm64_preempt_schedule_irq() into entry-common.c along with the other entry management functions. We no longer need to include in process.c, so the include directive is removed. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Julien Thierry Cc: Will Deacon Reviewed-by: Anshuman Khandual --- --- arch/arm64/kernel/entry-common.c | 19 +++++++++++++++++++ arch/arm64/kernel/process.c | 17 ----------------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index fde59981445c..4fa058453468 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -6,7 +6,10 @@ */ #include +#include +#include #include +#include #include #include @@ -330,3 +333,19 @@ asmlinkage void notrace el0_sync_compat_handler(struct pt_regs *regs) } NOKPROBE_SYMBOL(el0_sync_compat_handler); #endif /* CONFIG_COMPAT */ + +asmlinkage void __sched arm64_preempt_schedule_irq(void) +{ + lockdep_assert_irqs_disabled(); + + /* + * Preempting a task from an IRQ means we leave copies of PSTATE + * on the stack. cpufeature's enable calls may modify PSTATE, but + * resuming one of these preempted tasks would undo those changes. + * + * Only allow a task to be preempted once cpufeatures have been + * enabled. + */ + if (static_branch_likely(&arm64_const_caps_ready)) + preempt_schedule_irq(); +} diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 71f788cd2b18..94b3ae351af9 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -17,7 +17,6 @@ #include #include #include -#include #include #include #include @@ -633,19 +632,3 @@ static int __init tagged_addr_init(void) core_initcall(tagged_addr_init); #endif /* CONFIG_ARM64_TAGGED_ADDR_ABI */ - -asmlinkage void __sched arm64_preempt_schedule_irq(void) -{ - lockdep_assert_irqs_disabled(); - - /* - * Preempting a task from an IRQ means we leave copies of PSTATE - * on the stack. cpufeature's enable calls may modify PSTATE, but - * resuming one of these preempted tasks would undo those changes. - * - * Only allow a task to be preempted once cpufeatures have been - * enabled. - */ - if (static_branch_likely(&arm64_const_caps_ready)) - preempt_schedule_irq(); -} From patchwork Wed Jan 8 18:56:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324367 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 99197930 for ; Wed, 8 Jan 2020 18:57:58 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 77DB220705 for ; Wed, 8 Jan 2020 18:57:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="OGm9tv8e" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 77DB220705 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Tp+R7q4qDqR6QmmxVhzdR/ymgXX4hQxrOFivPuCDT+E=; b=OGm9tv8eaIDKhRrDh5bVEGVAuI +8WPE75rmdPRNXe0WAocpBJAfwRCVXLnfpyv3yd8/tvEAJ8RB2amyaij8BjnAsVLcNWIhq6S5ILvg 75nvCn0boqvjPf2keNpFiCjUm4cWFawIGYg1JUeGK4XWpY84/aewvsCxB7r2vVlzsdW3EHDoaOI/L gkxr/cWIWKEdG0PQrELslWcbmckMsIR9MlTHR/oKpCbap3pt9WfrDbwUxDkZTLW3XRqBqi2FgwWhW VnRaS+Dd2KUDCVL2G+WIzLKW1FzUZhFwCZsWELtK0uLWqgpbL84NxgsXopONU3Bns+u2hq99HZKtR TORhzHJQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGWi-0008KE-IA; Wed, 08 Jan 2020 18:57:56 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGVn-0007Pn-OU for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:01 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1C8EEDA7; Wed, 8 Jan 2020 10:56:59 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8A5EC3F534; Wed, 8 Jan 2020 10:56:57 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 04/17] arm64: entry: move preempt logic to C Date: Wed, 8 Jan 2020 18:56:21 +0000 Message-Id: <20200108185634.1163-5-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105659_912048_86657B51 X-CRM114-Status: GOOD ( 12.38 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Some of our preeemption logic is in C, while a portion of it is in assembly. Let's pull the remainder of it into C so that it lives in one place. At the same time, let's improve the comments regarding NMI preemption to make it clearer why we cannot preempt from NMIs. Subsequent patches will covert the caller of el1_preempt() to C. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Julien Thierry Cc: Will Deacon --- arch/arm64/kernel/entry-common.c | 18 +++++++++++++++++- arch/arm64/kernel/entry.S | 13 +------------ 2 files changed, 18 insertions(+), 13 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 4fa058453468..b93ca2148a20 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -334,8 +335,23 @@ asmlinkage void notrace el0_sync_compat_handler(struct pt_regs *regs) NOKPROBE_SYMBOL(el0_sync_compat_handler); #endif /* CONFIG_COMPAT */ -asmlinkage void __sched arm64_preempt_schedule_irq(void) +asmlinkage void __sched el1_preempt(void) { + if (!IS_ENABLED(CONFIG_PREEMPT) || preempt_count()) + return; + + /* + * To avoid nesting NMIs and overflowing the stack, we must leave NMIs + * masked until the exception return. We want to context-switch with + * IRQs masked but NMIs enabled, so cannot preempt an NMI. + * + * PSTATE.{D,A,F} are cleared for IRQ and NMI by el1_irq(). + * When gic_handle_irq() handles an NMI, it leaves PSTATE.I set. + * If anything is set in DAIF, this is an NMI. + */ + if (system_uses_irq_prio_masking() && read_sysreg(daif) != 0) + return; + lockdep_assert_irqs_disabled(); /* diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 7c6a0a41676f..53ce1877a4aa 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -606,18 +606,7 @@ el1_irq: irq_handler #ifdef CONFIG_PREEMPT - ldr x24, [tsk, #TSK_TI_PREEMPT] // get preempt count -alternative_if ARM64_HAS_IRQ_PRIO_MASKING - /* - * DA_F were cleared at start of handling. If anything is set in DAIF, - * we come back from an NMI, so skip preemption - */ - mrs x0, daif - orr x24, x24, x0 -alternative_else_nop_endif - cbnz x24, 1f // preempt count != 0 || NMI return path - bl arm64_preempt_schedule_irq // irq en/disable is done inside -1: + bl el1_preempt #endif #ifdef CONFIG_ARM64_PSEUDO_NMI From patchwork Wed Jan 8 18:56:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324369 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4BB45139A for ; Wed, 8 Jan 2020 18:58:15 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2A8F620720 for ; Wed, 8 Jan 2020 18:58:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="YIi119Ts" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2A8F620720 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=kRhPP+WJxgYLZ6O08slBChTQT2pd9DWGC5cSujMfO3I=; b=YIi119TsozpuWq0jRXHp8IknBH 0Nfe3EKUwLcp2f758e7M6X8UB7zaC43olAoKb/3oXLyNxAm+DH35D3dZzXDqt9PuFfvb8ZPJo7jEU zTiXwGAMQvBTVKrIacz2If5mhgBpGVsq47mHspHYmnMDgFz9eaqKLWH8efVE9sqLqC8vUIUFm4WGR WdfClfv3Q7XXE7YWebxyGXRUUhd6vxURrwkZshNm/lR+MHTlQMHJkMltp05Qs0yA0n2LEObN5Vame +ps6jpNSkCgV22RhZjnjaEcevk6pPwj0GiB+5iyIA++BhlG4HfKLJ55QzogkyT512BTzHOaW6vmsg zSn0MBhg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGWz-00008m-4T; Wed, 08 Jan 2020 18:58:13 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGVq-0007Td-Lx for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:08 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 450861FB; Wed, 8 Jan 2020 10:57:01 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B55873F534; Wed, 8 Jan 2020 10:56:59 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 05/17] arm64: entry: add a call_on_stack helper Date: Wed, 8 Jan 2020 18:56:22 +0000 Message-Id: <20200108185634.1163-6-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105703_005708_4AD25F71 X-CRM114-Status: GOOD ( 10.66 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org In some cases, we want to call a function from C code, using an alternative stack. Add a helper that we can use in such cases. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Will Deacon Reviewed-by: Anshuman Khandual --- arch/arm64/include/asm/exception.h | 2 ++ arch/arm64/kernel/entry.S | 21 +++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/arch/arm64/include/asm/exception.h b/arch/arm64/include/asm/exception.h index b87c6e276ab1..a49038fa4faf 100644 --- a/arch/arm64/include/asm/exception.h +++ b/arch/arm64/include/asm/exception.h @@ -31,6 +31,8 @@ static inline u32 disr_to_esr(u64 disr) return esr; } +asmlinkage void call_on_stack(struct pt_regs *, void (*)(struct pt_regs *), + unsigned long); asmlinkage void enter_from_user_mode(void); void do_mem_abort(unsigned long addr, unsigned int esr, struct pt_regs *regs); void do_sp_pc_abort(unsigned long addr, unsigned int esr, struct pt_regs *regs); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 53ce1877a4aa..184313c773ea 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -901,6 +901,27 @@ ENTRY(ret_from_fork) ENDPROC(ret_from_fork) NOKPROBE(ret_from_fork) +/* + * x0 = argument to function + * x1 = function to call + * x2 = new stack pointer + */ +ENTRY(call_on_stack) + /* Create a frame record to save our LR and SP (implicit in FP) */ + stp x29, x30, [sp, #-16]! + mov x29, sp + + /* Move to the new stack and call the function there */ + mov sp, x2 + blr x1 + + /* Restore SP from the FP, FP and LR from the record, and return */ + mov sp, x29 + ldp x29, x30, [sp], #16 + ret +ENDPROC(call_on_stack) +NOKPROBE(call_on_stack) + #ifdef CONFIG_ARM_SDE_INTERFACE #include From patchwork Wed Jan 8 18:56:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324371 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7D0F2139A for ; Wed, 8 Jan 2020 18:58:33 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5B08D20705 for ; Wed, 8 Jan 2020 18:58:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="VpCCpVM1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5B08D20705 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=iD0UMz5ZbTokGTBLIlwgX/Da9ResmpwZXrWl1lP2zYE=; b=VpCCpVM1JeVZXxfmVnNLAEexiI t5QybFtP7njcpPAOKU137lxnBpZsgrMPbjB6ncWHvs7ASpO1IEwzlGhrecJIfk2/PPjCL8FohD/TP Wg24BHdd2XRo8hhyoKfr10vGb68ZvSRjfnwBZQFcTUrpSoTi4BR7ejdVSxMvHath5YpO8+ajteaLn apBMsh8nRK9CoRfvLUySTSJND4+OqxHxLEMnjX+sKS3QGKKJYlxFfhCEAwvyURtPLnMtz0ruAXNT+ sOaxq9gOK8Hqs/qESkGsIomWpkU4aw/Zu46iCmS7Br/Cfj7Tu+rp4imvQqCByxkY2tJemN81JJnOT Nmidmf7w==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGXD-0000MV-VX; Wed, 08 Jan 2020 18:58:27 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGVs-0007V5-4Q for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:09 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6CAB2328; Wed, 8 Jan 2020 10:57:03 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DC6793F534; Wed, 8 Jan 2020 10:57:01 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 06/17] arm64: entry: convert irq entry to C Date: Wed, 8 Jan 2020 18:56:23 +0000 Message-Id: <20200108185634.1163-7-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105704_280142_2507E12C X-CRM114-Status: GOOD ( 17.01 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Portions of our IRQ entry logic are handled in C while other parts are handled in assembly. Let's migrate the bulk of it to C so that it's easier to follow and maintain. The original assembly called trace_hardirqs_off() for IRQs and NMIs, but did not call trace_hardirqs_on() for NMIs where IRQs were priority masked. The C code always balances these calls. The now unused asm_nmi_{enter,exit}() wrappers are removed. The EL0 BP hardening is moved from fault.c so that it's in the same compilation unit as its only caller, where the compiler can inline it. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Julien Thierry Cc: Will Deacon --- arch/arm64/kernel/entry-common.c | 81 +++++++++++++++++++++++- arch/arm64/kernel/entry.S | 133 +++------------------------------------ arch/arm64/kernel/irq.c | 15 ----- arch/arm64/mm/fault.c | 7 --- 4 files changed, 87 insertions(+), 149 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index b93ca2148a20..45155d9f72da 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -6,8 +6,12 @@ */ #include +#include +#include +#include #include #include +#include #include #include #include @@ -19,6 +23,7 @@ #include #include #include +#include #include static void notrace el1_abort(struct pt_regs *regs, unsigned long esr) @@ -335,7 +340,7 @@ asmlinkage void notrace el0_sync_compat_handler(struct pt_regs *regs) NOKPROBE_SYMBOL(el0_sync_compat_handler); #endif /* CONFIG_COMPAT */ -asmlinkage void __sched el1_preempt(void) +static void __sched el1_preempt(void) { if (!IS_ENABLED(CONFIG_PREEMPT) || preempt_count()) return; @@ -345,7 +350,7 @@ asmlinkage void __sched el1_preempt(void) * masked until the exception return. We want to context-switch with * IRQs masked but NMIs enabled, so cannot preempt an NMI. * - * PSTATE.{D,A,F} are cleared for IRQ and NMI by el1_irq(). + * PSTATE.{D,A,F} are cleared for IRQ and NMI by el1_irq_handler(). * When gic_handle_irq() handles an NMI, it leaves PSTATE.I set. * If anything is set in DAIF, this is an NMI. */ @@ -365,3 +370,75 @@ asmlinkage void __sched el1_preempt(void) if (static_branch_likely(&arm64_const_caps_ready)) preempt_schedule_irq(); } + +static void notrace invoke_irq_handler(struct pt_regs *regs) +{ + unsigned long irq_stack = (unsigned long)raw_cpu_read(irq_stack_ptr); + + irq_stack += IRQ_STACK_SIZE; + + if (on_thread_stack()) + call_on_stack(regs, handle_arch_irq, irq_stack); + else + handle_arch_irq(regs); +} +NOKPROBE_SYMBOL(invoke_irq_handler); + +asmlinkage void notrace el1_irq_handler(struct pt_regs *regs) +{ + bool masked; + + if (system_uses_irq_prio_masking()) + gic_write_pmr(regs->pmr_save | GIC_PRIO_PSR_I_SET); + + /* + * We can't use local_daif_restore(DAIF_PROCCTX_NOIRQ) here as it will + * see the A flag is clear and try to unmask NMIs. + */ + write_sysreg(DAIF_PROCCTX_NOIRQ, daif); + + /* + * If IRQs were masked, this is definitely an NMI. If IRQs were + * unmasked, this may be an IRQ or an NMI, and gic_handle_nmi() will + * handle nmi_{enter,exit} as necessary. + */ + masked = !irqs_priority_unmasked(regs); + + if (masked) + nmi_enter(); + else + trace_hardirqs_off(); + + invoke_irq_handler(regs); + + if (masked) { + nmi_exit(); + } else { + el1_preempt(); + trace_hardirqs_on(); + } +} +NOKPROBE_SYMBOL(el1_irq_handler); + +static inline void notrace do_el0_irq_bp_hardening(struct pt_regs *regs) +{ + if (!IS_ENABLED(CONFIG_HARDEN_BRANCH_PREDICTOR)) + return; + if (regs->pc & BIT(55)) + arm64_apply_bp_hardening(); +} +NOKPROBE_SYMBOL(do_el0_irq_bp_hardening); + +asmlinkage void notrace el0_irq_handler(struct pt_regs *regs) +{ + if (system_uses_irq_prio_masking()) + gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); + + user_exit_irqoff(); + local_daif_restore(DAIF_PROCCTX_NOIRQ); + trace_hardirqs_off(); + do_el0_irq_bp_hardening(regs); + invoke_irq_handler(regs); + trace_hardirqs_on(); +} +NOKPROBE_SYMBOL(el0_irq_handler); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 184313c773ea..55c4be1e996a 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -384,64 +384,9 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 sb .endm - .macro irq_stack_entry - mov x19, sp // preserve the original sp - - /* - * Compare sp with the base of the task stack. - * If the top ~(THREAD_SIZE - 1) bits match, we are on a task stack, - * and should switch to the irq stack. - */ - ldr x25, [tsk, TSK_STACK] - eor x25, x25, x19 - and x25, x25, #~(THREAD_SIZE - 1) - cbnz x25, 9998f - - ldr_this_cpu x25, irq_stack_ptr, x26 - mov x26, #IRQ_STACK_SIZE - add x26, x25, x26 - - /* switch to the irq stack */ - mov sp, x26 -9998: - .endm - - /* - * x19 should be preserved between irq_stack_entry and - * irq_stack_exit. - */ - .macro irq_stack_exit - mov sp, x19 - .endm - /* GPRs used by entry code */ tsk .req x28 // current thread_info -/* - * Interrupt handling. - */ - .macro irq_handler - ldr_l x1, handle_arch_irq - mov x0, sp - irq_stack_entry - blr x1 - irq_stack_exit - .endm - -#ifdef CONFIG_ARM64_PSEUDO_NMI - /* - * Set res to 0 if irqs were unmasked in interrupted context. - * Otherwise set res to non-0 value. - */ - .macro test_irqs_unmasked res:req, pmr:req -alternative_if ARM64_HAS_IRQ_PRIO_MASKING - sub \res, \pmr, #GIC_PRIO_IRQON -alternative_else - mov \res, xzr -alternative_endif - .endm -#endif - .macro gic_prio_kentry_setup, tmp:req #ifdef CONFIG_ARM64_PSEUDO_NMI alternative_if ARM64_HAS_IRQ_PRIO_MASKING @@ -451,15 +396,6 @@ alternative_endif #endif .endm - .macro gic_prio_irq_setup, pmr:req, tmp:req -#ifdef CONFIG_ARM64_PSEUDO_NMI - alternative_if ARM64_HAS_IRQ_PRIO_MASKING - orr \tmp, \pmr, #GIC_PRIO_PSR_I_SET - msr_s SYS_ICC_PMR_EL1, \tmp - alternative_else_nop_endif -#endif - .endm - .text /* @@ -589,47 +525,8 @@ ENDPROC(el1_sync) .align 6 el1_irq: kernel_entry 1 - gic_prio_irq_setup pmr=x20, tmp=x1 - enable_da_f - -#ifdef CONFIG_ARM64_PSEUDO_NMI - test_irqs_unmasked res=x0, pmr=x20 - cbz x0, 1f - bl asm_nmi_enter -1: -#endif - -#ifdef CONFIG_TRACE_IRQFLAGS - bl trace_hardirqs_off -#endif - - irq_handler - -#ifdef CONFIG_PREEMPT - bl el1_preempt -#endif - -#ifdef CONFIG_ARM64_PSEUDO_NMI - /* - * When using IRQ priority masking, we can get spurious interrupts while - * PMR is set to GIC_PRIO_IRQOFF. An NMI might also have occurred in a - * section with interrupts disabled. Skip tracing in those cases. - */ - test_irqs_unmasked res=x0, pmr=x20 - cbz x0, 1f - bl asm_nmi_exit -1: -#endif - -#ifdef CONFIG_TRACE_IRQFLAGS -#ifdef CONFIG_ARM64_PSEUDO_NMI - test_irqs_unmasked res=x0, pmr=x20 - cbnz x0, 1f -#endif - bl trace_hardirqs_on -1: -#endif - + mov x0, sp + bl el1_irq_handler kernel_exit 1 ENDPROC(el1_irq) @@ -655,7 +552,10 @@ ENDPROC(el0_sync) .align 6 el0_irq_compat: kernel_entry 0, 32 - b el0_irq_naked + mov x0, sp + bl el0_irq_handler + b ret_to_user +ENDPROC(el0_irq_compat) el0_error_compat: kernel_entry 0, 32 @@ -665,25 +565,8 @@ el0_error_compat: .align 6 el0_irq: kernel_entry 0 -el0_irq_naked: - gic_prio_irq_setup pmr=x20, tmp=x0 - ct_user_exit_irqoff - enable_da_f - -#ifdef CONFIG_TRACE_IRQFLAGS - bl trace_hardirqs_off -#endif - -#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR - tbz x22, #55, 1f - bl do_el0_irq_bp_hardening -1: -#endif - irq_handler - -#ifdef CONFIG_TRACE_IRQFLAGS - bl trace_hardirqs_on -#endif + mov x0, sp + bl el0_irq_handler b ret_to_user ENDPROC(el0_irq) diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index 04a327ccf84d..49a2d7b68531 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -76,18 +76,3 @@ void __init init_IRQ(void) local_daif_restore(DAIF_PROCCTX_NOIRQ); } } - -/* - * Stubs to make nmi_enter/exit() code callable from ASM - */ -asmlinkage void notrace asm_nmi_enter(void) -{ - nmi_enter(); -} -NOKPROBE_SYMBOL(asm_nmi_enter); - -asmlinkage void notrace asm_nmi_exit(void) -{ - nmi_exit(); -} -NOKPROBE_SYMBOL(asm_nmi_exit); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 077b02a2d4d3..43aea8012f62 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -745,13 +745,6 @@ void do_mem_abort(unsigned long addr, unsigned int esr, struct pt_regs *regs) } NOKPROBE_SYMBOL(do_mem_abort); -void do_el0_irq_bp_hardening(void) -{ - /* PC has already been checked in entry.S */ - arm64_apply_bp_hardening(); -} -NOKPROBE_SYMBOL(do_el0_irq_bp_hardening); - void do_sp_pc_abort(unsigned long addr, unsigned int esr, struct pt_regs *regs) { arm64_notify_die("SP/PC alignment exception", regs, From patchwork Wed Jan 8 18:56:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324373 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 55D4413A0 for ; Wed, 8 Jan 2020 18:58:42 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2B50A20678 for ; Wed, 8 Jan 2020 18:58:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="toRKhaMQ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2B50A20678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=8kqRzMChhcJxCEQc9sR1VsYDgM7Whkw4GMj+RO4EI+I=; b=toRKhaMQlbwSd4qwDt1ZbZzLmR 6tN8j/jJ5vvz18dRWU/ImPzfMDG9SdQvDlt6JMvs+AyIgNOcu8YR5TqxGy9q53Wr2WL+3Lq5no0Wq A3U6+sKNqO+sxGxX561ru5wr3zK31E+jUBqNbF8pk+95H/Y1nz+iXSdusEzLuQmJ+zWNzuRp+bl5V HECZWO63CEroUOMFuVdo7CveoUe4ro6RU+p4IqxQHCZ+E/OPscRe8k31Dv2DTCKU/QtPZGUeNM9zb jZdb5+1O3sfJ2PBLhWKI9iKngTYNeRfZZu4vchAac+dr4P9PGu56fGUWwVY1PxnWJ0fi4N2DyGz25 Xkwmjd/Q==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGXQ-0000a2-3p; Wed, 08 Jan 2020 18:58:40 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGVu-0007XU-To for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:10 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8F931DA7; Wed, 8 Jan 2020 10:57:05 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0B7293F534; Wed, 8 Jan 2020 10:57:03 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 07/17] arm64: entry: convert error entry to C Date: Wed, 8 Jan 2020 18:56:24 +0000 Message-Id: <20200108185634.1163-8-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105707_298106_F1C25C57 X-CRM114-Status: GOOD ( 11.27 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Portions of our error entry logic are handled in C while other parts are handled in assembly. Let's migrate the bulk of it to C so that it's easier to follow and maintain. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Will Deacon --- arch/arm64/include/asm/exception.h | 1 + arch/arm64/kernel/entry-common.c | 26 ++++++++++++++++++++++++++ arch/arm64/kernel/entry.S | 19 ++++++------------- arch/arm64/kernel/traps.c | 2 +- 4 files changed, 34 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/exception.h b/arch/arm64/include/asm/exception.h index a49038fa4faf..220a7c3971d8 100644 --- a/arch/arm64/include/asm/exception.h +++ b/arch/arm64/include/asm/exception.h @@ -51,5 +51,6 @@ void do_el0_svc(struct pt_regs *regs); void do_el0_svc_compat(struct pt_regs *regs); void do_el0_ia_bp_hardening(unsigned long addr, unsigned int esr, struct pt_regs *regs); +void do_serror(struct pt_regs *regs, unsigned int esr); #endif /* __ASM_EXCEPTION_H */ diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 45155d9f72da..bf9d497e620c 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -442,3 +442,29 @@ asmlinkage void notrace el0_irq_handler(struct pt_regs *regs) trace_hardirqs_on(); } NOKPROBE_SYMBOL(el0_irq_handler); + +asmlinkage void el1_error_handler(struct pt_regs *regs) +{ + unsigned long esr = read_sysreg(esr_el1); + + if (system_uses_irq_prio_masking()) + gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); + + local_daif_restore(DAIF_ERRCTX); + do_serror(regs, esr); +} +NOKPROBE_SYMBOL(el1_error_handler); + +asmlinkage void el0_error_handler(struct pt_regs *regs) +{ + unsigned long esr = read_sysreg(esr_el1); + + if (system_uses_irq_prio_masking()) + gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); + + user_exit_irqoff(); + local_daif_restore(DAIF_ERRCTX); + do_serror(regs, esr); + local_daif_restore(DAIF_PROCCTX_NOIRQ); +} +NOKPROBE_SYMBOL(el0_error_handler); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 55c4be1e996a..0c5117ef7c3c 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -559,7 +559,10 @@ ENDPROC(el0_irq_compat) el0_error_compat: kernel_entry 0, 32 - b el0_error_naked + mov x0, sp + bl el0_error_handler + b ret_to_user +ENDPROC(el0_error_compat) #endif .align 6 @@ -572,25 +575,15 @@ ENDPROC(el0_irq) el1_error: kernel_entry 1 - mrs x1, esr_el1 - gic_prio_kentry_setup tmp=x2 - enable_dbg mov x0, sp - bl do_serror + bl el1_error_handler kernel_exit 1 ENDPROC(el1_error) el0_error: kernel_entry 0 -el0_error_naked: - mrs x25, esr_el1 - gic_prio_kentry_setup tmp=x2 - ct_user_exit_irqoff - enable_dbg mov x0, sp - mov x1, x25 - bl do_serror - enable_da_f + bl el0_error_handler b ret_to_user ENDPROC(el0_error) diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 73caf35c2262..170e637bb87b 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -901,7 +901,7 @@ bool arm64_is_fatal_ras_serror(struct pt_regs *regs, unsigned int esr) } } -asmlinkage void do_serror(struct pt_regs *regs, unsigned int esr) +void do_serror(struct pt_regs *regs, unsigned int esr) { const bool was_in_nmi = in_nmi(); From patchwork Wed Jan 8 18:56:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324375 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ADB65139A for ; Wed, 8 Jan 2020 18:58:59 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8C77120705 for ; Wed, 8 Jan 2020 18:58:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="dFH5ng0B" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C77120705 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bgA/LWdcYEF7bJ9kFBUOZX2x/f11m4mWOL7RAX3xZII=; b=dFH5ng0Bw48a8afYan24UXinuD rDaSmvlL5CsVAzeuW4PKY2VwYdlafS0mI7+yBwbcy9y62DVGaYM+UHyagnuo9Fsy3DfRBm+7HliBk HKVEcbySuIF03tzmLgUbAkWQ58CIspS3whA0t9wj5mKSWkFytgxpu+IFZO7lpfGnSjByRPskShRfm 2HaaD3UqpyxL/LJ+FI86BLtnV/HWfuKluH4lzeMIjZE479EvFoPfBo8J1Kzbo6NaRyKMSOWTkf/BZ HIMuga4ye9q1DtG750geLih19nmq9z0YlziDm2J/PxAadUOaUjVKjnoKnJawYcnswN+S9e5X0ro59 LXyfPqOw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGXe-0000nr-OY; Wed, 08 Jan 2020 18:58:54 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGVw-0007Td-De for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:13 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B141C106F; Wed, 8 Jan 2020 10:57:07 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 24D693F534; Wed, 8 Jan 2020 10:57:06 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 08/17] arm64: entry: Split el0_sync_compat from el0_sync Date: Wed, 8 Jan 2020 18:56:25 +0000 Message-Id: <20200108185634.1163-9-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105708_525486_10678DCF X-CRM114-Status: UNSURE ( 8.68 ) X-CRM114-Notice: Please train this message. X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Currently we treat el0_sync_compat as-if it's a portion of el0_sync, which is unlike all the other exception entry stubs. Let's split it out and give it it's own ENDPROC(), so that we can treat it as a separate path entirely. Reported-by: Mark Brown Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Will Deacon Reviewed-by: Anshuman Khandual --- arch/arm64/kernel/entry.S | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 0c5117ef7c3c..2c3de577f720 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -539,6 +539,7 @@ el0_sync: mov x0, sp bl el0_sync_handler b ret_to_user +ENDPROC(el0_sync) #ifdef CONFIG_COMPAT .align 6 @@ -547,7 +548,7 @@ el0_sync_compat: mov x0, sp bl el0_sync_compat_handler b ret_to_user -ENDPROC(el0_sync) +ENDPROC(el0_sync_compat) .align 6 el0_irq_compat: From patchwork Wed Jan 8 18:56:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324377 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4B1CC13A0 for ; Wed, 8 Jan 2020 18:59:19 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0F1F220678 for ; Wed, 8 Jan 2020 18:59:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="gF+fOr8/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0F1F220678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=oS9/XNrMv/B4Fo82mBfqPuemjdTTDQ0nixTc9MBD8GI=; b=gF+fOr8/zDvubxC5OsTTX9DbC8 SPO78V6tnlS+fgfrP2VANWBJ1Gq/lOrFiYu5pJeQcoRv92d5YAN4a5VGhxKJYghME7UfWYwfVUcxw 2OaG4vrtT/6rvvQuF+YxlKNU9wBn/5MjLpk+rR/rYbwl5sriXRq1guwcV+0wl51TyHgEkENxFPIjD lRFshQJy5r+R1iNuiXbXTAyVXDs/y+1QAcow9YcqhPd+9bYTmW2mjCL4Ra3dYtjowrep/OULU8AdE 96Zh/+3N7p4YSHQB+dri6OYmjpuPSh3K0dHby5ylBA+sZk76umxz4aMe1YBmkL3jhFw7qXupS6fFz h+HKZU0A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGY2-00019q-4I; Wed, 08 Jan 2020 18:59:18 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGVy-0007V5-7a for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:13 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D44E91FB; Wed, 8 Jan 2020 10:57:09 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4ECC73F534; Wed, 8 Jan 2020 10:57:08 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 09/17] arm64: entry: organise handler stubs consistently Date: Wed, 8 Jan 2020 18:56:26 +0000 Message-Id: <20200108185634.1163-10-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105710_367489_C355F63B X-CRM114-Status: UNSURE ( 9.03 ) X-CRM114-Notice: Please train this message. X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The exception handler stubs are laid out in a random order, violating the EL1/EL0 split described by the comments. In an attempt to get more optimal cache usage for commonly-invoked handlers, some handlers are given special alignment while others are not. To make things less surprising, This patch reorganises the hander stubs so they're in a consistent order, with the EL1 sync/irq/error stubs first, followed by the EL0 sync/irq/error stubs, then the EL0 compat sync/irq/error stubs. All the stubs are given the same alignment. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Will Deacon Reviewed-by: Anshuman Khandual --- arch/arm64/kernel/entry.S | 48 +++++++++++++++++++++++++---------------------- 1 file changed, 26 insertions(+), 22 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 2c3de577f720..35a8c56b0582 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -530,6 +530,14 @@ el1_irq: kernel_exit 1 ENDPROC(el1_irq) + .align 6 +el1_error: + kernel_entry 1 + mov x0, sp + bl el1_error_handler + kernel_exit 1 +ENDPROC(el1_error) + /* * EL0 mode handlers. */ @@ -541,6 +549,23 @@ el0_sync: b ret_to_user ENDPROC(el0_sync) + .align 6 +el0_irq: + kernel_entry 0 + mov x0, sp + bl el0_irq_handler + b ret_to_user +ENDPROC(el0_irq) + + .align 6 +el0_error: + kernel_entry 0 + mov x0, sp + bl el0_error_handler + b ret_to_user +ENDPROC(el0_error) + + #ifdef CONFIG_COMPAT .align 6 el0_sync_compat: @@ -558,6 +583,7 @@ el0_irq_compat: b ret_to_user ENDPROC(el0_irq_compat) + .align 6 el0_error_compat: kernel_entry 0, 32 mov x0, sp @@ -566,28 +592,6 @@ el0_error_compat: ENDPROC(el0_error_compat) #endif - .align 6 -el0_irq: - kernel_entry 0 - mov x0, sp - bl el0_irq_handler - b ret_to_user -ENDPROC(el0_irq) - -el1_error: - kernel_entry 1 - mov x0, sp - bl el1_error_handler - kernel_exit 1 -ENDPROC(el1_error) - -el0_error: - kernel_entry 0 - mov x0, sp - bl el0_error_handler - b ret_to_user -ENDPROC(el0_error) - /* * Ok, we need to do extra processing, enter the slow path. */ From patchwork Wed Jan 8 18:56:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324379 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2083E139A for ; Wed, 8 Jan 2020 18:59:32 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DC82C20678 for ; Wed, 8 Jan 2020 18:59:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="RQcYkCaH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DC82C20678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=9tWu59R8YQk4ZNpFRnr+nyIr7YdwpUctZ+Gl/ZuKugA=; b=RQcYkCaH0/VygR5sR8PGvNE/tS /rFoZifGK9Grf/He2wcQZXn+QhOllOKObXm9m5FN7CogFfs8MEG64agDpkuLiGS5954Opd8yGn++Q zavgHUzCrh7TWB7LQPi+NC5jAXRxlCrkRw9ue/7DVCK3o1c0Rn3Hu/eom10yW5sWURfgZQOT4mzC6 ad5alHm976D7+hHziRYqWXQBwU04dN/sflE1X+8BQvtWi6l+/BNo3fwku55m3/ITeBD2KuE02XSXc MDX27AMKLCqO8l84uzGN7j4NffOzFfCPBZpjva7Pb+lVkhLcS8s042OP+Mj4d5HcDHjmeZZ164Xzi S7hszghQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGYF-0001MO-4o; Wed, 08 Jan 2020 18:59:31 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGW0-0007cT-Gz for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:15 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 09F2C328; Wed, 8 Jan 2020 10:57:12 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 79ECB3F534; Wed, 8 Jan 2020 10:57:10 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 10/17] arm64: entry: consolidate EL1 return paths Date: Wed, 8 Jan 2020 18:56:27 +0000 Message-Id: <20200108185634.1163-11-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105712_635776_59AC6983 X-CRM114-Status: UNSURE ( 9.97 ) X-CRM114-Notice: Please train this message. X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Each of the EL1 exception handler stubs has an identical copy of the kernel_exit code. While each handler needs its own kernel_entry sequence, there's no need to duplicate this for each handler, and we can consolidate them for better I-cache usage. This patch makes the EL1 handlers all use a common kernel_exit stub called ret_to_kernel, matching the ret_to_user stub used by EL0 handlers. As with the handlers, ret_to_kenerl is aligned for better I-cache and brapnch predictor utilization, and for consistency the same alignment is applied to ret_to_user. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Will Deacon Reviewed-by: Anshuman Khandual --- arch/arm64/kernel/entry.S | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 35a8c56b0582..e76326feb1da 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -519,7 +519,7 @@ el1_sync: kernel_entry 1 mov x0, sp bl el1_sync_handler - kernel_exit 1 + b ret_to_kernel ENDPROC(el1_sync) .align 6 @@ -527,7 +527,7 @@ el1_irq: kernel_entry 1 mov x0, sp bl el1_irq_handler - kernel_exit 1 + b ret_to_kernel ENDPROC(el1_irq) .align 6 @@ -535,10 +535,18 @@ el1_error: kernel_entry 1 mov x0, sp bl el1_error_handler - kernel_exit 1 + b ret_to_kernel ENDPROC(el1_error) /* + * Common EL1 exception return path + */ + .align 6 +ret_to_kernel: + kernel_exit 1 +ENDPROC(ret_to_kernel) + +/* * EL0 mode handlers. */ .align 6 @@ -606,6 +614,7 @@ work_pending: /* * "slow" syscall return path. */ + .align 6 ret_to_user: disable_daif gic_prio_kentry_setup tmp=x3 From patchwork Wed Jan 8 18:56:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324381 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5BB8313A0 for ; Wed, 8 Jan 2020 18:59:48 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1D04020678 for ; Wed, 8 Jan 2020 18:59:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="HY+YSV0O" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1D04020678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=kmoS75KGcr2/xThMP7eQ6E+vHmaOPuDxcJP3mpPr2wc=; b=HY+YSV0OWeCJcll2I4nGzQdVq8 EotxwbeKDHwuAVxatnpkRBOn720K7jY9lpcMzBLuAnwKCcLdw8nFyXb3kbD0t+1QnU55GrinNghmd KKj4i/FHENIIb09Xiti3OrvFXuzj3r8CFcwpm4cA29nlC2zCGi8kLzU4+O86fsumnpg44ASAhtFWO RjkowCUq1shxoPfFl686PDR+84Pp2VmEx9CSLt9vDmEAOvaxgE4Is+rnM1Qx2ppe4XxOQqDeXG0ju YdlZGOZBbvCEJF/w9GUDdoucSP2fB4j1MyQprM5qpATSgc4j3+3cTyg1OpBroNIjo7frjn5n4mmB3 sQvOkJNw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGYS-0001bG-7k; Wed, 08 Jan 2020 18:59:44 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGW2-0007Td-ED for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:16 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3506F1FB; Wed, 8 Jan 2020 10:57:14 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A539C3F534; Wed, 8 Jan 2020 10:57:12 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 11/17] stackleak: allow C to call stackleak_erase() Date: Wed, 8 Jan 2020 18:56:28 +0000 Message-Id: <20200108185634.1163-12-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105714_537802_F9FC412B X-CRM114-Status: UNSURE ( 9.14 ) X-CRM114-Notice: Please train this message. X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Currently, stackleak_erase() has no prototype in a header file, and has to be called directly from low-level architecture entry assembly code. This necessitates ifdeffery and complicates the entry assembly. To ameliorate matters, let's provide a prototype so that architecture can call stackleak_erase() from slightly higher level C code used as part of the entry flow. This makes things easier to read and maintain. Signed-off-by: Mark Rutland Cc: Alexander Popov Cc: Catalin Marinas Cc: Kees Cook Cc: Laura Abbott Cc: Will Deacon Acked-by: Kees Cook --- include/linux/stackleak.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/include/linux/stackleak.h b/include/linux/stackleak.h index 3d5c3271a9a8..2b09d3759c76 100644 --- a/include/linux/stackleak.h +++ b/include/linux/stackleak.h @@ -15,6 +15,8 @@ #ifdef CONFIG_GCC_PLUGIN_STACKLEAK #include +asmlinkage void notrace stackleak_erase(void); + static inline void stackleak_task_init(struct task_struct *t) { t->lowest_stack = (unsigned long)end_of_stack(t) + sizeof(unsigned long); @@ -30,6 +32,7 @@ int stack_erasing_sysctl(struct ctl_table *table, int write, #else /* !CONFIG_GCC_PLUGIN_STACKLEAK */ static inline void stackleak_task_init(struct task_struct *t) { } +static inline void stackleak_erase(void) { } #endif #endif From patchwork Wed Jan 8 18:56:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324383 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 739101398 for ; Wed, 8 Jan 2020 19:00:10 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 398E820678 for ; Wed, 8 Jan 2020 19:00:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="bRNno6nq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 398E820678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=BW7E/02KJijlxLPjDvDXi1ggc0j1yv6wPFBJyYtsA0c=; b=bRNno6nqN+UWJjpfZu5VmAYeLK Y2k3VehX8+tYVqFKeh5MgOvbkTqFKdY7nxsey7Zby/4fqTpLZPfnn70AcAeOXN6wwLDkICkKNIRrI ImXNa8Of6rdXzEl0p12gYeUHC6D/IeTXyOYRYnUuAtzLA8n12XUg2K4X2acxME6Q6yozuKJOVCflI 7Ad0hoN+qI5AnEj8ILx1Lt11OtgvRwjCjyzYVQsxpiH2IlOaoCjuNoYOEmJQpDoXKms0tIM89rnjt Sm1Cw/GKdQ8rnIJq+W2tRmyi1UooW3UXazob4nbL3xsP1K2BCLQ5AQd+eJKTK7Qh2Ls8kdJEhgOEp PiS6wmZg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGYm-0001tF-3c; Wed, 08 Jan 2020 19:00:04 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGW5-0007hV-5O for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:19 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 65F771FB; Wed, 8 Jan 2020 10:57:16 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D68743F534; Wed, 8 Jan 2020 10:57:14 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 12/17] arm64: debug-monitors: refactor MDSCR manipulation Date: Wed, 8 Jan 2020 18:56:29 +0000 Message-Id: <20200108185634.1163-13-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105717_319615_52464AE5 X-CRM114-Status: GOOD ( 13.01 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org When we convert the ret_to_user/work_pending code to C, we're going to want to poke the MDSCR to enable/disable single-step. Let's factor out the existing code for this from debug-monitors.c. At the same time, we can make use of {read,write}_sysreg() directly, and get rid of the mdscr_{read,write} wrappers. The existing code masked DAIF when manipulating MDSCR_EL1, but this should not be necessary. Exceptions can be taken immediately before DAIF is masked, and given the lack of an ISB can also be taken after DAIF is unmasked as writes to DAIF are only self-synchronizing and not context-synchronizing in general. We may want to add an ISB to ensure that updates to MDSCR have taken effect, however. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Will Deacon --- arch/arm64/include/asm/debug-monitors.h | 10 ++++++++++ arch/arm64/kernel/debug-monitors.c | 32 +++++++------------------------- 2 files changed, 17 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/debug-monitors.h b/arch/arm64/include/asm/debug-monitors.h index 7619f473155f..342867e50c54 100644 --- a/arch/arm64/include/asm/debug-monitors.h +++ b/arch/arm64/include/asm/debug-monitors.h @@ -107,6 +107,16 @@ enum dbg_active_el { void enable_debug_monitors(enum dbg_active_el el); void disable_debug_monitors(enum dbg_active_el el); +static __always_inline void __enable_single_step_nosync(void) +{ + sysreg_clear_set(mdscr_el1, 0, DBG_MDSCR_SS); +} + +static __always_inline void __disable_single_step_nosync(void) +{ + sysreg_clear_set(mdscr_el1, DBG_MDSCR_SS, 0); +} + void user_rewind_single_step(struct task_struct *task); void user_fastforward_single_step(struct task_struct *task); diff --git a/arch/arm64/kernel/debug-monitors.c b/arch/arm64/kernel/debug-monitors.c index 48222a4760c2..fa2d4145bd07 100644 --- a/arch/arm64/kernel/debug-monitors.c +++ b/arch/arm64/kernel/debug-monitors.c @@ -32,24 +32,6 @@ u8 debug_monitors_arch(void) } /* - * MDSCR access routines. - */ -static void mdscr_write(u32 mdscr) -{ - unsigned long flags; - flags = local_daif_save(); - write_sysreg(mdscr, mdscr_el1); - local_daif_restore(flags); -} -NOKPROBE_SYMBOL(mdscr_write); - -static u32 mdscr_read(void) -{ - return read_sysreg(mdscr_el1); -} -NOKPROBE_SYMBOL(mdscr_read); - -/* * Allow root to disable self-hosted debug from userspace. * This is useful if you want to connect an external JTAG debugger. */ @@ -91,9 +73,9 @@ void enable_debug_monitors(enum dbg_active_el el) enable |= DBG_MDSCR_KDE; if (enable && debug_enabled) { - mdscr = mdscr_read(); + mdscr = read_sysreg(mdscr_el1); mdscr |= enable; - mdscr_write(mdscr); + write_sysreg(mdscr, mdscr_el1); } } NOKPROBE_SYMBOL(enable_debug_monitors); @@ -112,9 +94,9 @@ void disable_debug_monitors(enum dbg_active_el el) disable &= ~DBG_MDSCR_KDE; if (disable) { - mdscr = mdscr_read(); + mdscr = read_sysreg(mdscr_el1); mdscr &= disable; - mdscr_write(mdscr); + write_sysreg(mdscr, mdscr_el1); } } NOKPROBE_SYMBOL(disable_debug_monitors); @@ -409,7 +391,7 @@ void kernel_enable_single_step(struct pt_regs *regs) { WARN_ON(!irqs_disabled()); set_regs_spsr_ss(regs); - mdscr_write(mdscr_read() | DBG_MDSCR_SS); + __enable_single_step_nosync(); enable_debug_monitors(DBG_ACTIVE_EL1); } NOKPROBE_SYMBOL(kernel_enable_single_step); @@ -417,7 +399,7 @@ NOKPROBE_SYMBOL(kernel_enable_single_step); void kernel_disable_single_step(void) { WARN_ON(!irqs_disabled()); - mdscr_write(mdscr_read() & ~DBG_MDSCR_SS); + __disable_single_step_nosync(); disable_debug_monitors(DBG_ACTIVE_EL1); } NOKPROBE_SYMBOL(kernel_disable_single_step); @@ -425,7 +407,7 @@ NOKPROBE_SYMBOL(kernel_disable_single_step); int kernel_active_single_step(void) { WARN_ON(!irqs_disabled()); - return mdscr_read() & DBG_MDSCR_SS; + return read_sysreg(mdscr_el1) & DBG_MDSCR_SS; } NOKPROBE_SYMBOL(kernel_active_single_step); From patchwork Wed Jan 8 18:56:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324385 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9215C1398 for ; Wed, 8 Jan 2020 19:00:21 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3F77F20678 for ; Wed, 8 Jan 2020 19:00:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="GBHoHpS+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3F77F20678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=I9Nrj9vEMJ4M3QE28TKpXd2d1lEZ+rDopM3Rhmjzuzg=; b=GBHoHpS+JH/XThdbLAxFXtuo0+ ZU+sOOliIp1DrN9wD3SRoBVB5Z+SvwTpB237w6om34jeytrwdDbOb1xYaNbrGoCjdDr4GgGO/jpya TXzkyZbKH4iYqHxVricvGL6WYUVZY060WNsXepnwOexRCKYtU+J+6+GXrBLnVUESEcWu/Ti49Ja4i UdvDFWox2nNoWLKEilRyU0F8yfD92kdpYM0cKFesPz+fSqUsaDU1oso4TAhEdwoMmEuXw6afzMVE4 ZLtf11fRu+zW5e3QHcx7ErubGHVPaTth6196rjHgYtFhoqXfD6bs7QBK8vfvXSqKKQKGmFwTDRA4P YYYcTFnw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGZ2-0003Ju-L4; Wed, 08 Jan 2020 19:00:20 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGW7-0007j8-QH for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:24 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 95221328; Wed, 8 Jan 2020 10:57:18 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 114313F534; Wed, 8 Jan 2020 10:57:16 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 13/17] arm64: entry: move common el0 entry/return work to C Date: Wed, 8 Jan 2020 18:56:30 +0000 Message-Id: <20200108185634.1163-14-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105719_980751_0C6FE89A X-CRM114-Status: GOOD ( 17.26 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Portions of common EL0 exception entry/return logic are handled in C while other parts are handled in assembly. Let's migrate the bulk of it to C so that it's easier to follow and maintain. This patch moves the ret_to_user/work_pending logic to C. As that handled enabling singlestep for userspace, the matching disable is similarly factored out of the entry code. Additionally user_enter() is factored out of kernel_exit as all the corresponding user_enter() calls have already been converted to C. Rather than add tedious boilerplate to each top-level exception handler to perform this entry/return work, a new EL0_HANDLER() handles this automatically. This takes the full name of each handler to keep them easy to find with grep, and also takes a name for the pt_regs argument so that we don't have a confusing implicit variable. Since local_daif_mask() handles the GIC priority, we don't need to mess with this explicitly as we did in the old assembly, and the now unused gic_prio_kentry_setup macro can be removed. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Will Deacon --- arch/arm64/include/asm/assembler.h | 18 ---------- arch/arm64/include/asm/exception.h | 1 + arch/arm64/kernel/entry-common.c | 67 +++++++++++++++++++++++++++++++++----- arch/arm64/kernel/entry.S | 56 ++----------------------------- arch/arm64/kernel/signal.c | 3 +- 5 files changed, 64 insertions(+), 81 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index b8cf7c85ffa2..505beb369f1e 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -67,24 +67,6 @@ msr daifclr, #8 .endm - .macro disable_step_tsk, flgs, tmp - tbz \flgs, #TIF_SINGLESTEP, 9990f - mrs \tmp, mdscr_el1 - bic \tmp, \tmp, #DBG_MDSCR_SS - msr mdscr_el1, \tmp - isb // Synchronise with enable_dbg -9990: - .endm - - /* call with daif masked */ - .macro enable_step_tsk, flgs, tmp - tbz \flgs, #TIF_SINGLESTEP, 9990f - mrs \tmp, mdscr_el1 - orr \tmp, \tmp, #DBG_MDSCR_SS - msr mdscr_el1, \tmp -9990: - .endm - /* * SMP data memory barrier */ diff --git a/arch/arm64/include/asm/exception.h b/arch/arm64/include/asm/exception.h index 220a7c3971d8..7f367d720ca4 100644 --- a/arch/arm64/include/asm/exception.h +++ b/arch/arm64/include/asm/exception.h @@ -52,5 +52,6 @@ void do_el0_svc_compat(struct pt_regs *regs); void do_el0_ia_bp_hardening(unsigned long addr, unsigned int esr, struct pt_regs *regs); void do_serror(struct pt_regs *regs, unsigned int esr); +void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags); #endif /* __ASM_EXCEPTION_H */ diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index bf9d497e620c..17dbfadb2fb8 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -15,10 +15,12 @@ #include #include #include +#include #include #include #include +#include #include #include #include @@ -107,6 +109,59 @@ asmlinkage void notrace el1_sync_handler(struct pt_regs *regs) } NOKPROBE_SYMBOL(el1_sync_handler); +static void notrace el0_prepare_entry(struct pt_regs *regs) +{ + if (test_thread_flag(TIF_SINGLESTEP)) { + __disable_single_step_nosync(); + isb(); + } +} +NOKPROBE_SYMBOL(el0_prepare_entry); + +static void notrace el0_prepare_return(struct pt_regs *regs) +{ + unsigned long flags; + + local_daif_mask(); + + flags = current_thread_info()->flags; + if (unlikely(flags & _TIF_WORK_MASK)) { + do_notify_resume(regs, flags); + trace_hardirqs_on(); + } + + if (test_thread_flag(TIF_SINGLESTEP)) + __enable_single_step_nosync(); + + user_enter(); + + stackleak_erase(); +} +NOKPROBE_SYMBOL(el0_prepare_return); + +asmlinkage void notrace el0_prepare_return_from_fork(void) +{ + el0_prepare_return(current_pt_regs()); +} +NOKPROBE_SYMBOL(el0_prepare_return_from_fork); + +/* + * Top-level exception handlers need to perform common entry/exit work. Use + * this macro when defining a handler for exceptions from EL0, so that work is + * handled automatically. + */ +#define EL0_HANDLER(handlername, regsname) \ +static __always_inline void notrace __raw_##handlername(struct pt_regs *regsname); \ +NOKPROBE_SYMBOL(__raw_##handlername); \ +asmlinkage void notrace handlername(struct pt_regs *regs) \ +{ \ + el0_prepare_entry(regs); \ + __raw_##handlername(regs); \ + el0_prepare_return(regs); \ +} \ +NOKPROBE_SYMBOL(handlername); \ +static __always_inline void notrace __raw_##handlername(struct pt_regs *regsname) + static void notrace el0_da(struct pt_regs *regs, unsigned long esr) { unsigned long far = read_sysreg(far_el1); @@ -228,7 +283,7 @@ static void notrace el0_svc(struct pt_regs *regs) } NOKPROBE_SYMBOL(el0_svc); -asmlinkage void notrace el0_sync_handler(struct pt_regs *regs) +EL0_HANDLER(el0_sync_handler, regs) { unsigned long esr = read_sysreg(esr_el1); @@ -274,7 +329,6 @@ asmlinkage void notrace el0_sync_handler(struct pt_regs *regs) el0_inv(regs, esr); } } -NOKPROBE_SYMBOL(el0_sync_handler); #ifdef CONFIG_COMPAT static void notrace el0_cp15(struct pt_regs *regs, unsigned long esr) @@ -294,7 +348,7 @@ static void notrace el0_svc_compat(struct pt_regs *regs) } NOKPROBE_SYMBOL(el0_svc_compat); -asmlinkage void notrace el0_sync_compat_handler(struct pt_regs *regs) +EL0_HANDLER(el0_sync_compat_handler, regs) { unsigned long esr = read_sysreg(esr_el1); @@ -337,7 +391,6 @@ asmlinkage void notrace el0_sync_compat_handler(struct pt_regs *regs) el0_inv(regs, esr); } } -NOKPROBE_SYMBOL(el0_sync_compat_handler); #endif /* CONFIG_COMPAT */ static void __sched el1_preempt(void) @@ -429,7 +482,7 @@ static inline void notrace do_el0_irq_bp_hardening(struct pt_regs *regs) } NOKPROBE_SYMBOL(do_el0_irq_bp_hardening); -asmlinkage void notrace el0_irq_handler(struct pt_regs *regs) +EL0_HANDLER(el0_irq_handler, regs) { if (system_uses_irq_prio_masking()) gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); @@ -441,7 +494,6 @@ asmlinkage void notrace el0_irq_handler(struct pt_regs *regs) invoke_irq_handler(regs); trace_hardirqs_on(); } -NOKPROBE_SYMBOL(el0_irq_handler); asmlinkage void el1_error_handler(struct pt_regs *regs) { @@ -455,7 +507,7 @@ asmlinkage void el1_error_handler(struct pt_regs *regs) } NOKPROBE_SYMBOL(el1_error_handler); -asmlinkage void el0_error_handler(struct pt_regs *regs) +EL0_HANDLER(el0_error_handler, regs) { unsigned long esr = read_sysreg(esr_el1); @@ -467,4 +519,3 @@ asmlinkage void el0_error_handler(struct pt_regs *regs) do_serror(regs, esr); local_daif_restore(DAIF_PROCCTX_NOIRQ); } -NOKPROBE_SYMBOL(el0_error_handler); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index e76326feb1da..e67c8ad94cce 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -26,22 +26,6 @@ #include #include -/* - * Context tracking subsystem. Used to instrument transitions - * between user and kernel mode. - */ - .macro ct_user_exit_irqoff -#ifdef CONFIG_CONTEXT_TRACKING - bl enter_from_user_mode -#endif - .endm - - .macro ct_user_enter -#ifdef CONFIG_CONTEXT_TRACKING - bl context_tracking_user_enter -#endif - .endm - .macro clear_gp_regs .irp n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29 mov x\n, xzr @@ -167,9 +151,7 @@ alternative_cb_end .if \el == 0 clear_gp_regs mrs x21, sp_el0 - ldr_this_cpu tsk, __entry_task, x20 // Ensure MDSCR_EL1.SS is clear, - ldr x19, [tsk, #TSK_TI_FLAGS] // since we can unmask debug - disable_step_tsk x19, x20 // exceptions when scheduling. + ldr_this_cpu tsk, __entry_task, x20 apply_ssbd 1, x22, x23 @@ -277,9 +259,6 @@ alternative_if ARM64_HAS_IRQ_PRIO_MASKING alternative_else_nop_endif ldp x21, x22, [sp, #S_PC] // load ELR, SPSR - .if \el == 0 - ct_user_enter - .endif #ifdef CONFIG_ARM64_SW_TTBR0_PAN /* @@ -387,15 +366,6 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 /* GPRs used by entry code */ tsk .req x28 // current thread_info - .macro gic_prio_kentry_setup, tmp:req -#ifdef CONFIG_ARM64_PSEUDO_NMI - alternative_if ARM64_HAS_IRQ_PRIO_MASKING - mov \tmp, #(GIC_PRIO_PSR_I_SET | GIC_PRIO_IRQON) - msr_s SYS_ICC_PMR_EL1, \tmp - alternative_else_nop_endif -#endif - .endm - .text /* @@ -601,31 +571,10 @@ ENDPROC(el0_error_compat) #endif /* - * Ok, we need to do extra processing, enter the slow path. - */ -work_pending: - mov x0, sp // 'regs' - bl do_notify_resume -#ifdef CONFIG_TRACE_IRQFLAGS - bl trace_hardirqs_on // enabled while in userspace -#endif - ldr x1, [tsk, #TSK_TI_FLAGS] // re-check for single-step - b finish_ret_to_user -/* - * "slow" syscall return path. + * Common EL0 exception return path */ .align 6 ret_to_user: - disable_daif - gic_prio_kentry_setup tmp=x3 - ldr x1, [tsk, #TSK_TI_FLAGS] - and x2, x1, #_TIF_WORK_MASK - cbnz x2, work_pending -finish_ret_to_user: - enable_step_tsk x1, x2 -#ifdef CONFIG_GCC_PLUGIN_STACKLEAK - bl stackleak_erase -#endif kernel_exit 0 ENDPROC(ret_to_user) @@ -787,6 +736,7 @@ ENTRY(ret_from_fork) mov x0, x20 blr x19 1: get_current_task tsk + bl el0_prepare_return_from_fork b ret_to_user ENDPROC(ret_from_fork) NOKPROBE(ret_from_fork) diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c index dd2cdc0d5be2..22e726d57846 100644 --- a/arch/arm64/kernel/signal.c +++ b/arch/arm64/kernel/signal.c @@ -896,8 +896,7 @@ static void do_signal(struct pt_regs *regs) restore_saved_sigmask(); } -asmlinkage void do_notify_resume(struct pt_regs *regs, - unsigned long thread_flags) +void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags) { /* * The assembly code enters us with IRQs off, but it hasn't From patchwork Wed Jan 8 18:56:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324389 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E6DDE13A0 for ; Wed, 8 Jan 2020 19:00:40 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C470620678 for ; Wed, 8 Jan 2020 19:00:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="TVzzMwIi" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C470620678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=4vnoWqJkpmOdc1e7kyfb8UhSncEb78Q5AGGZGthgJzA=; b=TVzzMwIihM4RoT7yWpe63qubut u3jt75wy7snbMEdmnTBA4iarwTkCbcM2iB4/PJCDNYTQv1R12kdk534VW+Q6mso7GyX7yA6HTC6VU oPBBzHLidZPVt7X7evkLq4LdSPN1ygWT5+pGLpE0t5CS0BDl7SNi5m2FV+F/WE/YvnSYCaTCQwzyw MheOgsOy5XO8o0yZOxdDpeYbs16oLDh1uoKCUEkvhKtGOlTHEM9tC+gZDN1B1alq/CkfA8Ay5Xa0F SfM06hfPqEVgW6Y/a7vVRleU5K0C56mLGSdoAUKuxVYv5h9RVBUVE2K/lm1KfanviD4aB5/4d5xJU ju/MUiBQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGZK-0003bT-T0; Wed, 08 Jan 2020 19:00:38 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGW9-0007lB-K1 for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:23 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C7F651FB; Wed, 8 Jan 2020 10:57:20 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 441153F534; Wed, 8 Jan 2020 10:57:19 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 14/17] arm64: entry: move NO_SYSCALL setup to C Date: Wed, 8 Jan 2020 18:56:31 +0000 Message-Id: <20200108185634.1163-15-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105721_725794_28330F8D X-CRM114-Status: GOOD ( 11.88 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org In the entry assembly we set up pt_regs::syscallno to NO_SYSCALL so that any ptrace calls will see a sensible value. For real syscalls, the actual syscall number is setup in C code, in do_el0_svc or do_el0_svc_compat. Given that tracing isn't performed until the usual EL0 entry work is performed, we can move the default syscallno setup to C code, making things simpler and more legible. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Will Deacon Reviewed-by: Anshuman Khandual --- arch/arm64/kernel/asm-offsets.c | 1 - arch/arm64/kernel/entry-common.c | 3 +++ arch/arm64/kernel/entry.S | 6 ------ 3 files changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index a5bdce8af65b..0b6ffa9ecc08 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -61,7 +61,6 @@ int main(void) DEFINE(S_SP, offsetof(struct pt_regs, sp)); DEFINE(S_PSTATE, offsetof(struct pt_regs, pstate)); DEFINE(S_PC, offsetof(struct pt_regs, pc)); - DEFINE(S_SYSCALLNO, offsetof(struct pt_regs, syscallno)); DEFINE(S_ORIG_ADDR_LIMIT, offsetof(struct pt_regs, orig_addr_limit)); DEFINE(S_PMR_SAVE, offsetof(struct pt_regs, pmr_save)); DEFINE(S_STACKFRAME, offsetof(struct pt_regs, stackframe)); diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 17dbfadb2fb8..fa568284e73f 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -111,6 +111,9 @@ NOKPROBE_SYMBOL(el1_sync_handler); static void notrace el0_prepare_entry(struct pt_regs *regs) { + /* Not in a syscall by default; do_el0_svc{,_compat} overwrite this */ + regs->syscallno = NO_SYSCALL; + if (test_thread_flag(TIF_SINGLESTEP)) { __disable_single_step_nosync(); isb(); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index e67c8ad94cce..d84718d272e9 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -208,12 +208,6 @@ alternative_else_nop_endif stp x22, x23, [sp, #S_PC] - /* Not in a syscall by default (el0_svc overwrites for real syscall) */ - .if \el == 0 - mov w21, #NO_SYSCALL - str w21, [sp, #S_SYSCALLNO] - .endif - /* * Set sp_el0 to current thread_info. */ From patchwork Wed Jan 8 18:56:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324391 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7433C1398 for ; Wed, 8 Jan 2020 19:00:55 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 35DCD20678 for ; Wed, 8 Jan 2020 19:00:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ZwF58Nnx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 35DCD20678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=32TDCvFvLCLBPrUjOI05i7Y2nM3qqJWN6hhmKIMv6Gc=; b=ZwF58NnxmuyHkSkCLBE8YnzMpl oGMrTN7zoDM+Q1FKa3JEdbCcnzRhPhe+Mt5NV3AU7k+yHI2/dqlqvUYx6yKVtOBf3+A7bkvs+tTj6 tOrW8YTOatm26MJ1Talif9Wfx4pGAT8Hp+8iA9ljUGQ24E265ri6uKMiMmKrw+QCgH10D+Lif38uT 8zdOVdXuA47AcZaUP4/zVkRhHRobOCxcDx2CHdepicsgWgZUdIXbp4TnU37bcvT4GF2Z2wcSh++er V2wNqsJwk+Ov2iKNXT3CN/cBdIONB291y1nDZO/7se4AGTwwKUvjUEenchQBPitTfGwqv40e98rce 3DqDMRBA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGZZ-0003oq-Ro; Wed, 08 Jan 2020 19:00:53 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGWB-0007mN-GZ for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:25 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 09FC6DA7; Wed, 8 Jan 2020 10:57:23 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7A1D03F534; Wed, 8 Jan 2020 10:57:21 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 15/17] arm64: entry: move ARM64_ERRATUM_845719 workaround to C Date: Wed, 8 Jan 2020 18:56:32 +0000 Message-Id: <20200108185634.1163-16-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105723_634049_A0884FB7 X-CRM114-Status: GOOD ( 11.19 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org To make the entry code less of a rats nest of overlapping labels and shared state, and to make the code easier to debug and maintain, let's move the workaround for ARM64_ERRATUM_845719 to C. The workaround requires us to perform a write to CONTEXTIDR_EL1 at AArch64 EL1 before retuning to an AArch32 EL0 task. There are no additional requirements on the state of the CPU, or on subsequent instructions prior to the ERET, so this can safely be performed in C code. As with the assembly version, we preserve the value of CONTEXTIDR if CONFIG_PID_IN_CONTEXTIDR is selected. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/kernel/entry-common.c | 17 +++++++++++++++++ arch/arm64/kernel/entry.S | 14 -------------- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index fa568284e73f..28b241cfd8f0 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -121,6 +121,21 @@ static void notrace el0_prepare_entry(struct pt_regs *regs) } NOKPROBE_SYMBOL(el0_prepare_entry); +static void notrace workaround_arm64_erratum_845719(void) +{ + unsigned long val = 0; + + if (!IS_ENABLED(CONFIG_ARM64_ERRATUM_845719) || + !cpus_have_const_cap(ARM64_WORKAROUND_845719) || + !is_compat_task()) + return; + + if (IS_ENABLED(CONFIG_PID_IN_CONTEXTIDR)) + val = read_sysreg(contextidr_el1); + write_sysreg(val, contextidr_el1); +} +NOKPROBE_SYMBOL(workaround_arm64_erratum_845719); + static void notrace el0_prepare_return(struct pt_regs *regs) { unsigned long flags; @@ -138,6 +153,8 @@ static void notrace el0_prepare_return(struct pt_regs *regs) user_enter(); + workaround_arm64_erratum_845719(); + stackleak_erase(); } NOKPROBE_SYMBOL(el0_prepare_return); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index d84718d272e9..a7340e551589 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -288,20 +288,6 @@ alternative_else_nop_endif .if \el == 0 ldr x23, [sp, #S_SP] // load return stack pointer msr sp_el0, x23 - tst x22, #PSR_MODE32_BIT // native task? - b.eq 3f - -#ifdef CONFIG_ARM64_ERRATUM_845719 -alternative_if ARM64_WORKAROUND_845719 -#ifdef CONFIG_PID_IN_CONTEXTIDR - mrs x29, contextidr_el1 - msr contextidr_el1, x29 -#else - msr contextidr_el1, xzr -#endif -alternative_else_nop_endif -#endif -3: #ifdef CONFIG_ARM64_ERRATUM_1418040 alternative_if_not ARM64_WORKAROUND_1418040 b 4f From patchwork Wed Jan 8 18:56:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324393 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5A7C41398 for ; Wed, 8 Jan 2020 19:01:14 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3395420678 for ; Wed, 8 Jan 2020 19:01:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="t/X8hWd9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3395420678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+ArpmF9w5EefC9+648ggV9/4RQDsQrvPhm99bLPcz38=; b=t/X8hWd9/qPGQ6d/3ETzsXcm6l YWTAUrsikpntjbjCCqmVzaHH+0zdmt2oCVZ95AQQzcQFZF5NShE6Q8G5tBEIkv+kSvgjpCd2hqPGo pqfMoJLFG10WyA2O8DKpA5/BXAo+MOmGaU5Uaj0rYS1DMRZX15YZFhdlzy3Q+CKtdo6BjeIkwhM5a e19OEcQLmMLW49JnjdrXNBmlJG2bRCaB7PC5uwOUyUTjMAp6H9F1XL2qUQqdExId0xKWTfJnqjcyU MZCWrk4YsiQjKxxhIcA/uU8d/WL0DyuPwZUG/3ezRRXCrEXlrRWirlki9NpcB3Mc5phlinuWCs35+ t/dKcrjg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGZt-00048y-B4; Wed, 08 Jan 2020 19:01:13 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGWF-0007px-QC for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:29 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 16FF81FB; Wed, 8 Jan 2020 10:57:27 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 876243F534; Wed, 8 Jan 2020 10:57:25 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 16/17] arm64: entry: move ARM64_ERRATUM_1418040 workaround to C Date: Wed, 8 Jan 2020 18:56:33 +0000 Message-Id: <20200108185634.1163-17-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105727_977390_4AE59363 X-CRM114-Status: GOOD ( 11.86 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org To make the entry code less of a rats nest of overlapping labels and shared state, and to make the code easier to debug and maintain, lets move the workaround for ARM64_ERRATUM_1418040 to C. The workaround requires us to disable EL0 access to the virtual counter, and emulate these accesses when they are trapped. The assembly code is only responsible for manipulating the trap control, which we can safely do in C code. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Robin Murphy Cc: Will Deacon --- arch/arm64/kernel/entry-common.c | 20 ++++++++++++++++++++ arch/arm64/kernel/entry.S | 15 --------------- 2 files changed, 20 insertions(+), 15 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 28b241cfd8f0..a7bebc3ce2a4 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -28,6 +28,8 @@ #include #include +#include + static void notrace el1_abort(struct pt_regs *regs, unsigned long esr) { unsigned long far = read_sysreg(far_el1); @@ -136,6 +138,23 @@ static void notrace workaround_arm64_erratum_845719(void) } NOKPROBE_SYMBOL(workaround_arm64_erratum_845719); +static void notrace workaround_arm64_erratum_1418040(void) +{ + unsigned long clear = 0, set = 0; + + if (!IS_ENABLED(CONFIG_ARM64_ERRATUM_1418040) || + !cpus_have_const_cap(ARM64_WORKAROUND_1418040)) + return; + + if (is_compat_task()) + clear = ARCH_TIMER_USR_VCT_ACCESS_EN; + else + set = ARCH_TIMER_USR_VCT_ACCESS_EN; + + sysreg_clear_set(cntkctl_el1, clear, set); +} +NOKPROBE_SYMBOL(workaround_arm64_erratum_1418040); + static void notrace el0_prepare_return(struct pt_regs *regs) { unsigned long flags; @@ -154,6 +173,7 @@ static void notrace el0_prepare_return(struct pt_regs *regs) user_enter(); workaround_arm64_erratum_845719(); + workaround_arm64_erratum_1418040(); stackleak_erase(); } diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index a7340e551589..537b44c413df 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -288,21 +288,6 @@ alternative_else_nop_endif .if \el == 0 ldr x23, [sp, #S_SP] // load return stack pointer msr sp_el0, x23 -#ifdef CONFIG_ARM64_ERRATUM_1418040 -alternative_if_not ARM64_WORKAROUND_1418040 - b 4f -alternative_else_nop_endif - /* - * if (x22.mode32 == cntkctl_el1.el0vcten) - * cntkctl_el1.el0vcten = ~cntkctl_el1.el0vcten - */ - mrs x1, cntkctl_el1 - eon x0, x1, x22, lsr #3 - tbz x0, #1, 4f - eor x1, x1, #2 // ARCH_TIMER_USR_VCT_ACCESS_EN - msr cntkctl_el1, x1 -4: -#endif apply_ssbd 0, x0, x1 .endif From patchwork Wed Jan 8 18:56:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 11324395 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 257101398 for ; Wed, 8 Jan 2020 19:01:33 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 03D4F20678 for ; Wed, 8 Jan 2020 19:01:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="cakDYEls" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 03D4F20678 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=xjuynAlolSc1R5QeBVd2VhEahygS6ol9TnJlp/RACH4=; b=cakDYElsSnnL3htH6uE42yiGib xiwsBuYvC6Lf7lBf+6kUVPHvzWKs1ODgCF2hgEKf0sNWAnw3H5wPgjYobdQDKFSPXBtDh4tDx771H WDq3Y74EmLmdvlAl1bCEYQDRfgznsVFywSs8JHA1aEOmjyNb73p/es4AtTe6oH2YmHiXbmv1lOppa eVpjAY0JLvM0ytChSghaISfsdyumzpdzg5quAKSPPGj5aImR9OcsFvwuJ7e2mI4LmhX3GHdLCkyu2 XrH97GpOTH6HprnCg8Xqfz+TOmOep2U9b6JyvXZrVsvUIZMAJI4tHWbangDUtUCNvha1GweDMo06Q vRvLTbAg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGaA-0004OX-CF; Wed, 08 Jan 2020 19:01:30 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ipGWJ-0007td-4b for linux-arm-kernel@lists.infradead.org; Wed, 08 Jan 2020 18:57:33 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A8A871FB; Wed, 8 Jan 2020 10:57:30 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 24C563F534; Wed, 8 Jan 2020 10:57:29 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com Subject: [PATCH 17/17] arm64: entry: cleanup sp_el0 manipulation Date: Wed, 8 Jan 2020 18:56:34 +0000 Message-Id: <20200108185634.1163-18-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200108185634.1163-1-mark.rutland@arm.com> References: <20200108185634.1163-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200108_105731_742233_E92E4AA9 X-CRM114-Status: GOOD ( 10.19 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, keescook@chromium.org, maz@kernel.org, broonie@kernel.org, labbott@redhat.com, robin.murphy@arm.com, julien.thierry.kdev@gmail.com, alex.popov@linux.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The kernel stashes the current task struct in sp_el0 so that this can be acquired consistently/cheaply when required. When we take an exception from EL0 we have to: 1) stash the original sp_el0 value 2) find the current task 3) update sp_el0 with the current task pointer Currently steps #1 and #2 occur in one place, and step #3 a while later. As the value of sp_el0 is immaterial between these points, let's move them together to make the code clearer and minimize ifdeffery. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Will Deacon --- arch/arm64/kernel/entry.S | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 537b44c413df..40d1cd8fb4c6 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -152,6 +152,7 @@ alternative_cb_end clear_gp_regs mrs x21, sp_el0 ldr_this_cpu tsk, __entry_task, x20 + msr sp_el0, tsk apply_ssbd 1, x22, x23 @@ -208,13 +209,6 @@ alternative_else_nop_endif stp x22, x23, [sp, #S_PC] - /* - * Set sp_el0 to current thread_info. - */ - .if \el == 0 - msr sp_el0, tsk - .endif - /* Save pmr */ alternative_if ARM64_HAS_IRQ_PRIO_MASKING mrs_s x20, SYS_ICC_PMR_EL1