From patchwork Mon May 10 15:56:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 874C3C433B4 for ; Mon, 10 May 2021 15:58:24 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4D520615FF for ; Mon, 10 May 2021 15:58:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4D520615FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=HSLho3fdL0lSkKVxkLvC08lO5qqum+twi4GjMcKcNLg=; b=BrvlRFylcL3HxZBLeYfcjvpL+l nn84uXGpc9zJl88KXR3+4HE1TZBPd7MnKhthkNbLOmqZQohV+X/WeeJo2kOoiKAHg1NLnLBgxaBOc H9FwR17a4qY2o9QoimQWP52xHfleWQZDJSFGy/Sfep5rcrr844iMSQdvonmS4bkk2viZ/U553VhBE a+DCExwvv3piNQ4Rz8eEJaNzz64AXKgq3rvhvXA5ZE4Cw9UFKJWy+HHINapM9Jnj3JOgbS/C2e3y6 hD1sX0L1S5cjka9AOWcoGux3pTPFkEhNsCr+GEfGuxB5D5gQOSZDQFGfASl98NiMR5tFLuNs2ooY8 Mve5/EqQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HB-00EqQK-IS; Mon, 10 May 2021 15:56:57 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8H7-00EqPV-8F for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:56:53 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=6HvimQ9YZ9Qtnw8hYEeAnHRkuSpRXYT13Jvnf/fmJCc=; b=uNQRhWMSjBkgd5kPdXcboTilCc CenGvylAhHcMmkBFCs0k5uGS0ZlAYGK3AqzsdbEQ+uSHBqcpl2MMTsGZTgFs7plqJw4Twc/7tARl3 H399kaQ46vnMqgt0IuL6EGdZKC/2ChllHNAFBncN5UUtI4ncr4nmcFTjKi2ePXPFXfs6WKIt7DNDP IqaxUDQUZIQFUVp7934dIsp183H17JopPQZbCej05qCnopIEeSkzRBnMBNofGPX9d0CdUO/K0AhuU 3xNRaq0UiNbgdDUtNRSe1rLaddgPhfWXb3UrY97RgmbSv1bpSMiXZQVbSZ7mkCFPIEaFXZ9eTt2aD 46nxWr+g==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8H4-008tKy-Mm for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:56:52 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9B9251691; Mon, 10 May 2021 08:56:49 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4CF0D3F73B; Mon, 10 May 2021 08:56:48 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 01/14] arm64: remove redundant local_daif_mask() in bad_mode() Date: Mon, 10 May 2021 16:56:08 +0100 Message-Id: <20210510155621.52811-2-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085650_810039_2A366E1D X-CRM114-Status: GOOD ( 10.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Upon taking an exception, the CPU sets all the DAIF bits. We never clear any of these bits prior to calling bad_mode(), and bad_mode() itself never clears any of these bits, so there's no need to call local_daif_mask(). This patch removes the redundant call. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/kernel/traps.c | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index a05d34f0e82a..41f0aa92022a 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -765,7 +765,6 @@ asmlinkage void notrace bad_mode(struct pt_regs *regs, int reason, unsigned int esr_get_class_string(esr)); __show_regs(regs); - local_daif_mask(); panic("bad mode"); } From patchwork Mon May 10 15:56:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A3F9C433ED for ; Mon, 10 May 2021 15:58:42 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 926F16161E for ; Mon, 10 May 2021 15:58:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 926F16161E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=yOpu9CpNstZScTwklbEZSk5POcHfPW068gdeuKJWZfs=; b=P5uLVdAm3hHoQONQVSQUbPlDuR FM0gSw5vAL/Tl9UhUWGCMHx31TaitFljk3s/3k97rXmWzAS2gZ0lSGcjHFo/Tew2V1DB+YZlNJQCc vTsa4MS4DmgJfwWZWwqe7udTzCGhTfh3cOPVnzPP8SlSWXM0pjRfPf7cpOxvsWlqgfAuRVvxxf6iE NhBLkqZEtxmXoZKd1J3WKYB/NBxomRwByWHyIViHL8/T83uVbAf/mW45gQMpq3VbdUYSp+utOTvyN C7VtpTMg/nVKbGdQo4rQbpAOKkP68TuN6pUfwpTxPAVNxcuZHlF87yzc7Z9LJqmHFHWdYHcUpxIiG WKGIUc0w==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HX-00EqVc-1q; Mon, 10 May 2021 15:57:19 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8H9-00EqPq-Ml for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:56:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=t9Voq7+s67K4JPJuXRWDo+IPXsDBov1yUL51f/GJyz4=; b=V1/E5aBmsWnJUHT0taPDV4Jhj0 rh4LZZNImyntLUpLd4zB4biyyYBAGbumxdi7j03WFM8SFNTmX4PPM1mLKZG6thjvOhxkQn7gLKfqr OG14emXcqNnEzMHvIa3ov4ZhJ9+tXLaxBfJKfUmt6ljlajVRKX9S/Y65rngbFwB8cASKqp7CbtFjf MWkmRKxIrDFL4KJvcEGZPI7eJrn3XAf8RqM8fCWexJWzEZLkVk1gU2WVXvX1u6ygBLeevNgWzEMMF CLce0++2jKC8W1Z6oRk9b7nvZ/dzNGYYR7V+LJNMmA0TQEybrE13jPljTYfegQwSDLYRPk/RyvSmi WEYZUhmQ==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8H6-008tLu-NB for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:56:54 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CEFC11692; Mon, 10 May 2021 08:56:51 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E2BC33F73B; Mon, 10 May 2021 08:56:50 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 02/14] arm64: entry: unmask IRQ+FIQ after EL0 handling Date: Mon, 10 May 2021 16:56:09 +0100 Message-Id: <20210510155621.52811-3-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085652_827245_3357CCE5 X-CRM114-Status: GOOD ( 14.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For non-fatal exceptions taken from EL0, we expect that at some point during exception handling it is possible to return to a regular process context with all exceptions unmasked (e.g. as we do in do_notify_resume()), and we generally aim to unmask exceptions wherever possible. While handling SError and debug exceptions from EL0, we need to leave some exceptions masked during handling. Handling SError requires us to mask SError (which also requires masking IRQ+FIQ), and handing debug exceptions requires us to mask debug (which also requires masking SError+IRQ+FIQ). Once do_serror() or do_debug_exception() has returned, we no longer need to mask exceptions, and can unmask them all, which is what we did prior to commit: 9034f6251572a474 ("arm64: Do not enable IRQs for ct_user_exit") ... where we had to mask IRQs as for context_tracking_user_exit() expected IRQs to be masked. Since then, we realised that our context tracking wasn't entirely correct, and reworked the entry code to fix this. As of commit: 23529049c6842382 ("arm64: entry: fix non-NMI user<->kernel transitions") ... we consistently call context_tracking_user_exit() later as part of ret_to_user. Prior to this we can transiently unmask exceptions (e.g. as part of do_notify_resume), and we always mask all exceptions prior to calling context_tracking_user_exit(). Thus, there's no longer a reason to leave IRQs or FIQs masked at the end of el0_dbg() or el0_error(), so let's bring these into line with other EL0 exceptions handlers and unmask all exceptions after the handler is finished. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/kernel/entry-common.c | 2 +- arch/arm64/kernel/entry.S | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 340d04e13617..02be1517e08f 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -398,7 +398,7 @@ static void noinstr el0_dbg(struct pt_regs *regs, unsigned long esr) enter_from_user_mode(); do_debug_exception(far, esr, regs); - local_daif_restore(DAIF_PROCCTX_NOIRQ); + local_daif_restore(DAIF_PROCCTX); } static void noinstr el0_svc(struct pt_regs *regs) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 3513984a88bd..6b2f6f5c5bb8 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -794,7 +794,7 @@ el0_error_naked: mov x0, sp mov x1, x25 bl do_serror - enable_da + enable_daif b ret_to_user SYM_CODE_END(el0_error) From patchwork Mon May 10 15:56:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72DBDC433B4 for ; Mon, 10 May 2021 15:58:55 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DB10061622 for ; Mon, 10 May 2021 15:58:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DB10061622 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Z1bjhwTocJgLS1kWjxkmlYGjcKJfUPZWaK9j/K231Ys=; b=psS5ofGK3PaZR30GjUp+YGvMyV VStI7B2LYwpDInwfnCU9FaFxua3F8r0jn3lOa0ec/imtUp6/HXEyg6ekVm15KYQkkLmG2zF63osWG 1Oba3SdZnKcGISaJq1yG9xkVNx2aik5v7/9wd8eegNeSpDW/WEuDBgBqS+yZ4stXmnCUdBAnni72G ccqIiiz7pZAyEF24UGFj7rl6jmgYdTcIBlbPgkOUepDYjKDFkj8uXr8M6BDOEd8lVhuidNJNULNWb 0QBbYp+ZmQxNa3s7IGOka7OqCHYImTiGLFANooxH0T6GCo8Ds4eFFw0Poz4ZDv9elY0J0R3aY3yn2 Nk9hFrKQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hl-00EqYd-7n; Mon, 10 May 2021 15:57:33 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HF-00EqQn-0B for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:01 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=tdCO8a+tngZQJ/TCCQ/83e7pZ5laL8J5CG9EFlHPCRQ=; b=4jQa0xj8SVwid/N6rIZpolbIdg wKzD5ayo2RPQH1k9uIEFUlMfCWCH9UOyWCkTEWXLZbdKooX7EMqycKpnjtjJK6xwmkxTNK8aK5/79 T6YYb9F6KgHtFoiz82vp0lxreUWYDO9ROCmdDM77A7gw0j+igMtCsrMRbIiba5ZST1Cr7EfnsHeCJ P4dZooijofL2B0gxMUYSBn1P7iAaZ2IodMM9SqhnS39Li+mhZugB16ARtdBCIsxgef7quag2alrR4 nF1nipTTbGMUCtm8a9xvCWlC7JxfNO5AdvH6OQ+rO5VkHNK8XhQoQNoTrHsQlTNLNz/rvGwRyfux0 1EKnVunQ==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HC-008tMW-5w for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:56:59 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CCC33168F; Mon, 10 May 2021 08:56:55 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E1F273F73B; Mon, 10 May 2021 08:56:54 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 03/14] arm64: entry: convert SError handlers to C Date: Mon, 10 May 2021 16:56:10 +0100 Message-Id: <20210510155621.52811-4-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085658_336437_5C4B200F X-CRM114-Status: GOOD ( 12.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For various reasons we'd like to convert the bulk of arm64's exception triage logic to C. As a step towards that, this patch converts the EL1 and EL0 SError triage logic to C. Separate C functions are added for the native and compat cases so that in subsequent patches we can handle native/compat differences in C. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/include/asm/exception.h | 4 ++++ arch/arm64/kernel/entry-common.c | 28 ++++++++++++++++++++++++++++ arch/arm64/kernel/entry.S | 16 +++++----------- 3 files changed, 37 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/exception.h b/arch/arm64/include/asm/exception.h index 6546158d2f2d..3a859d4e8b59 100644 --- a/arch/arm64/include/asm/exception.h +++ b/arch/arm64/include/asm/exception.h @@ -32,8 +32,11 @@ static inline u32 disr_to_esr(u64 disr) } asmlinkage void el1_sync_handler(struct pt_regs *regs); +asmlinkage void el1_error_handler(struct pt_regs *regs); asmlinkage void el0_sync_handler(struct pt_regs *regs); +asmlinkage void el0_error_handler(struct pt_regs *regs); asmlinkage void el0_sync_compat_handler(struct pt_regs *regs); +asmlinkage void el0_error_compat_handler(struct pt_regs *regs); asmlinkage void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs); asmlinkage void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs); @@ -57,4 +60,5 @@ void do_cp15instr(unsigned int esr, struct pt_regs *regs); void do_el0_svc(struct pt_regs *regs); void do_el0_svc_compat(struct pt_regs *regs); void do_ptrauth_fault(struct pt_regs *regs, unsigned int esr); +void do_serror(struct pt_regs *regs, unsigned int esr); #endif /* __ASM_EXCEPTION_H */ diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 02be1517e08f..a890ba550af3 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -279,6 +279,14 @@ asmlinkage void noinstr el1_sync_handler(struct pt_regs *regs) } } +asmlinkage void noinstr el1_error_handler(struct pt_regs *regs) +{ + unsigned long esr = read_sysreg(esr_el1); + + local_daif_restore(DAIF_ERRCTX); + do_serror(regs, esr); +} + asmlinkage void noinstr enter_from_user_mode(void) { lockdep_hardirqs_off(CALLER_ADDR0); @@ -468,6 +476,21 @@ asmlinkage void noinstr el0_sync_handler(struct pt_regs *regs) } } +static void __el0_error_handler_common(struct pt_regs *regs) +{ + unsigned long esr = read_sysreg(esr_el1); + + enter_from_user_mode(); + local_daif_restore(DAIF_ERRCTX); + do_serror(regs, esr); + local_daif_restore(DAIF_PROCCTX); +} + +asmlinkage void noinstr el0_error_handler(struct pt_regs *regs) +{ + __el0_error_handler_common(regs); +} + #ifdef CONFIG_COMPAT static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr) { @@ -526,4 +549,9 @@ asmlinkage void noinstr el0_sync_compat_handler(struct pt_regs *regs) el0_inv(regs, esr); } } + +asmlinkage void noinstr el0_error_compat_handler(struct pt_regs *regs) +{ + __el0_error_handler_common(regs); +} #endif /* CONFIG_COMPAT */ diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 6b2f6f5c5bb8..656f3129bfef 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -757,7 +757,9 @@ SYM_CODE_END(el0_fiq_compat) SYM_CODE_START_LOCAL_NOALIGN(el0_error_compat) kernel_entry 0, 32 - b el0_error_naked + mov x0, sp + bl el0_error_compat_handler + b ret_to_user SYM_CODE_END(el0_error_compat) #endif @@ -778,23 +780,15 @@ SYM_CODE_END(el0_fiq) SYM_CODE_START_LOCAL(el1_error) kernel_entry 1 - mrs x1, esr_el1 - enable_dbg mov x0, sp - bl do_serror + bl el1_error_handler kernel_exit 1 SYM_CODE_END(el1_error) SYM_CODE_START_LOCAL(el0_error) kernel_entry 0 -el0_error_naked: - mrs x25, esr_el1 - user_exit_irqoff - enable_dbg mov x0, sp - mov x1, x25 - bl do_serror - enable_daif + bl el0_error_handler b ret_to_user SYM_CODE_END(el0_error) From patchwork Mon May 10 15:56:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CA30C433ED for ; Mon, 10 May 2021 15:59:20 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 870A06161C for ; Mon, 10 May 2021 15:59:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 870A06161C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FnLuamhhoxCgW2T+VvTS/rLRMD/MbzR0KuvniwvZ5x4=; b=SAE2jFzhAAzUTcvb1IRRIedGOn WdXD/mppQPsSvWG3yD6pflRGINwNcD6krRT2SZRdbxfzYjbFzCtmUREgpRmTGrPBMf/hyt0oPRsfm 7VRN6JlCZSSn7qVomOY+FRw6383NhMhZdsyTEQbqG5LF0eYsQxeo9+KrP6r8nwR0R2qtvYC/ldOzb qSDPmUiTqXt5H6cXoWiMdTLKuXJQB5utE+5H4D7i4qDkIhXqeFwJyDhOD5zebL5nM08iAj9mMPFmA 6JkPJAjQTKdlN8mblLaH89pDqsjuGndgoYaYEACT9cTsaeJHZDP3h1MjABd20BRpGOrSsEpvOPSH9 h6qsk/Cg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hx-00EqeC-IP; Mon, 10 May 2021 15:57:45 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HG-00EqRP-CC for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=egG6gGelnl1bkST+xUhJK7xBFzOLSL6Yf8EavrP7Gms=; b=tQLY/9TMMynpoHGJ/vNrHjRYYE 7sfc6yTcUNEE8h3jcvivzdesatFd65gw04Pjrd7Wd+yC9n4ljf0OGWWSA/TbqhLzVnzeTWkIg8NYH GJLZ0Xmk5BN/YOzSkKlJT7X4p7zs4La0ZMDJ5h4764HNnnRrbp1OsZCwDaEPL+y0OWFuc+rmNxuzT gWYJwrtOPfkRVow0kUc+fUmEqjFCFgLOMb3SQSPLcsA8GtXwuvZjplwTdmcEtxJf3GjCV5OG0B888 Jx2WMpFUCjM9ysydqwVow8Kk2eviTGTLRBtUE3KL+Xq0gGNXZBt6CHToshg0qeGzp1TT/X7XIVouO v0LoljgA==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HD-008tMk-N6 for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:57:01 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 008691691; Mon, 10 May 2021 08:56:58 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 125103F73B; Mon, 10 May 2021 08:56:56 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 04/14] arm64: entry: move arm64_preempt_schedule_irq to entry-common.c Date: Mon, 10 May 2021 16:56:11 +0100 Message-Id: <20210510155621.52811-5-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085659_824299_DE37CAD8 X-CRM114-Status: GOOD ( 13.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Subsequent patches will pull more of the IRQ entry handling into C. To keep this in one place, let's move arm64_preempt_schedule_irq() into entry-common.c along with the other entry management functions. We no longer need to include in process.c, so the include directive is removed. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/kernel/entry-common.c | 20 ++++++++++++++++++++ arch/arm64/kernel/process.c | 17 ----------------- 2 files changed, 20 insertions(+), 17 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index a890ba550af3..33b3660beea3 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -6,7 +6,11 @@ */ #include +#include +#include #include +#include +#include #include #include @@ -113,6 +117,22 @@ asmlinkage void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs) exit_to_kernel_mode(regs); } +asmlinkage void __sched arm64_preempt_schedule_irq(void) +{ + lockdep_assert_irqs_disabled(); + + /* + * Preempting a task from an IRQ means we leave copies of PSTATE + * on the stack. cpufeature's enable calls may modify PSTATE, but + * resuming one of these preempted tasks would undo those changes. + * + * Only allow a task to be preempted once cpufeatures have been + * enabled. + */ + if (system_capabilities_finalized()) + preempt_schedule_irq(); +} + #ifdef CONFIG_ARM64_ERRATUM_1463225 static DEFINE_PER_CPU(int, __in_cortex_a76_erratum_1463225_wa); diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index b4bb67f17a2c..2e7337709155 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -18,7 +18,6 @@ #include #include #include -#include #include #include #include @@ -724,22 +723,6 @@ static int __init tagged_addr_init(void) core_initcall(tagged_addr_init); #endif /* CONFIG_ARM64_TAGGED_ADDR_ABI */ -asmlinkage void __sched arm64_preempt_schedule_irq(void) -{ - lockdep_assert_irqs_disabled(); - - /* - * Preempting a task from an IRQ means we leave copies of PSTATE - * on the stack. cpufeature's enable calls may modify PSTATE, but - * resuming one of these preempted tasks would undo those changes. - * - * Only allow a task to be preempted once cpufeatures have been - * enabled. - */ - if (system_capabilities_finalized()) - preempt_schedule_irq(); -} - #ifdef CONFIG_BINFMT_ELF int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state, bool has_interp, bool is_interp) From patchwork Mon May 10 15:56:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81927C433ED for ; Mon, 10 May 2021 15:59:55 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 33A596161C for ; Mon, 10 May 2021 15:59:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 33A596161C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=UqEaVDIzje3InoWLYFBseCIaaa6HiS/q+4qJcfi9BXc=; b=km+YjXsTvypgNB5NZVzUE3IHzw /wTZNd0xNno7mbXsfvkNaDQx0U0Ly2/VANbKkCYWcCNXn9AnNkDeSeiIAG0JrYWTLsO4z1Cu2W3rp e5iNPESYoKQSHx/1AW6ayd0KUuTBp47w1zJDycQ+OltwUk6Nrn12UYEKT9rydli1S5jRzpeHF1Fu9 im6+m3WQPS6LKXKzPqJCbWHUGemwsgHe3BBsWFwpkxy0Yn48mBhuLM/PXCeFjFKRD3o+SqCaI/Iv4 ICUob+/2jmHqUK8NRuxtmI35M5cdaP06wDKbfLkMwSHLA1K37yGzMCQ+svkQI9GXoetaky8zq19Qx T4kenG/g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8I9-00Eqj6-O0; Mon, 10 May 2021 15:57:57 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HH-00EqRU-Pg for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=+psv8Ofgdst1QY1yXYTRfTx96O07fCxWWDty7kcHZX8=; b=DJ7avxry4m1LMzIHvtBk3Fd1Nt FpWGTMCeJ/F0pOvn4xfJ3DkwHyhZE29MYcCXCIocgu9cxAXvzWXsgVltWU3/Zmv0TW14I52dVx9JR 6kOojVi82sLiGNSEnEHfmmWPWy+HHv7T4w5QrAwiGpxv2uhD9hTVrk7NqXIKyv9fyHb9JiAcnt/pD pgd9865ug53XLimukq4UV19R3l4r9svXJILrXnUGctbq7UB9HJY5beyAsZm2cDydid0+MqJN1iLD2 0SrRWRQsQaeyEDHVGAOqBNtInU4T3tOpVcESUTRsGoZow7WyOWezbhppjxF9R1+y8JdUyDFpvO17J aIA/vhqA==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HF-008tN1-1l for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:57:02 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3C3B7168F; Mon, 10 May 2021 08:57:00 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 566053F73B; Mon, 10 May 2021 08:56:59 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 05/14] arm64: entry: move preempt logic to C Date: Mon, 10 May 2021 16:56:12 +0100 Message-Id: <20210510155621.52811-6-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085701_156734_64E3BEC2 X-CRM114-Status: GOOD ( 15.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently portions of our preempt logic are written in C while other parts are written in assembly. There's no reason any of this needs to live in assembly, so let's move the rest of the lgoic to C. At the same time, let's make the comment a bit clearer. Other than the increased lockdep coverage there should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/kernel/entry-common.c | 12 ++++++++++++ arch/arm64/kernel/entry.S | 13 ------------- 2 files changed, 12 insertions(+), 13 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 33b3660beea3..87997a4a0936 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -121,6 +121,18 @@ asmlinkage void __sched arm64_preempt_schedule_irq(void) { lockdep_assert_irqs_disabled(); + if (preempt_count() != 0) + return; + + /* + * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC + * priority masking is used the GIC irqchip driver will clear DAIF.IF + * using gic_arch_enable_irqs() for normal IRQs. If anything is set in + * DAIF we must have handled an NMI, so skip preemption. + */ + if (system_uses_irq_prio_masking() && read_sysreg(daif)) + return; + /* * Preempting a task from an IRQ means we leave copies of PSTATE * on the stack. cpufeature's enable calls may modify PSTATE, but diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 656f3129bfef..8c7ddd651756 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -561,20 +561,7 @@ tsk .req x28 // current thread_info irq_handler \handler #ifdef CONFIG_PREEMPTION - ldr x24, [tsk, #TSK_TI_PREEMPT] // get preempt count -alternative_if ARM64_HAS_IRQ_PRIO_MASKING - /* - * DA were cleared at start of handling, and IF are cleared by - * the GIC irqchip driver using gic_arch_enable_irqs() for - * normal IRQs. If anything is set, it means we come back from - * an NMI instead of a normal IRQ, so skip preemption - */ - mrs x0, daif - orr x24, x24, x0 -alternative_else_nop_endif - cbnz x24, 1f // preempt count != 0 || NMI return path bl arm64_preempt_schedule_irq // irq en/disable is done inside -1: #endif mov x0, sp From patchwork Mon May 10 15:56:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1141C43462 for ; Mon, 10 May 2021 16:00:06 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 127276161E for ; Mon, 10 May 2021 16:00:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 127276161E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ezrVmjqflrOeVgmgEs43SMnNugisnS+xYFCL+DAK0JM=; b=MkuPY924KjstudE2WPLM64OGwo BhvX7YkIkl8c1kgn+5tB1sXW5dhwLkB5CTReO8vymDEx0Txh2Ud/q9Yf81AghaQHC2zGQRNn45Sx8 kXq8ZqAesiV879K/q+Oj/T6CU5c2cBuJL+vLZw2BrjFEJ3dMnYYzroSAV6o6Awe44ejgAiroedkti zBsi3NCdQrScrx37q+2hRm9gTrvXlZFtUs5iL0TZCYoNs1ZW0YIOlnf8k5y9jMTUxwFkyN4+kQOEw gGJfSHOms7YEZm62CBSa5X26dgNjDBBUWq4dfmlQR90gYNeadE4JUHtCjnikDn7e0nbxMPDYHbbwN lF3AeqTg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8IN-00Eqnu-Ln; Mon, 10 May 2021 15:58:11 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HO-00EqSR-NN for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:10 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=/yeenI8VhMLzj3WTjtBFRYYxiEZ5UUwb/UERhfVI3gs=; b=RXGb6b/dDaCeSQKnols0Q+Y927 4MvJTM3V3ahhqoZzq1WQKbUFHQh7H4Il6VNgFp42eDrvCwXoETM23gi08A1bzjPftuL/EhNB652Se IEzmK2iOmeZM/dOOPlWsi23zwr28RSl8INUlRJo60XFZeHubmr+JsoSY0O78Jf2A6yiMuGic+rAWg 9XRsY+zxsz4Sn+VXZaRBPrk+S1b0N2Ywn7x09fFbhaZCj/oRcNCEmVkcVRQkgDYTbQXHhqimj5Dmg 3ImyGMzQXIPCtJt8hAaNtkM4IYwUIJ0COzLgYpDsS2SWg6oD43HrPUeoJNhHZtzz6FCO0XA3iKIWn 0FJWnfag==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HM-008tNL-1H for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:57:09 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CA724168F; Mon, 10 May 2021 08:57:06 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E33BD3F73B; Mon, 10 May 2021 08:57:05 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 06/14] arm64: entry: add a call_on_irq_stack helper Date: Mon, 10 May 2021 16:56:13 +0100 Message-Id: <20210510155621.52811-7-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085708_142529_5FD2D9C6 X-CRM114-Status: GOOD ( 13.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When handling IRQ/FIQ exceptions the entry assembly may transition from a task's stack to that CPU's IRQ stack (and IRQ shadow call stack). In subsequent patches we want to migrate the IRQ/FIQ triage logic to C, and as we want to perform some actions on the task stack (e.g. EL1 preemption), we need to switch stacks within the C handler. So that we can do so, this patch adds a helper to call a function on a CPU's IRQ stack (and shadow stack as appropriate). Subsequent patches will make use of the new helper function. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/include/asm/exception.h | 2 ++ arch/arm64/kernel/entry.S | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/arch/arm64/include/asm/exception.h b/arch/arm64/include/asm/exception.h index 3a859d4e8b59..c24b69c0c589 100644 --- a/arch/arm64/include/asm/exception.h +++ b/arch/arm64/include/asm/exception.h @@ -40,6 +40,8 @@ asmlinkage void el0_error_compat_handler(struct pt_regs *regs); asmlinkage void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs); asmlinkage void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs); +asmlinkage void call_on_irq_stack(struct pt_regs *regs, + void (*func)(struct pt_regs *)); asmlinkage void enter_from_user_mode(void); asmlinkage void exit_to_user_mode(void); void arm64_enter_nmi(struct pt_regs *regs); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 8c7ddd651756..327a559679f7 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -979,6 +979,42 @@ SYM_CODE_START(ret_from_fork) SYM_CODE_END(ret_from_fork) NOKPROBE(ret_from_fork) +/* + * void call_on_irq_stack(struct pt_regs *regs, + * void (*func)(struct pt_regs *)); + * + * Calls func(regs) using this CPU's irq stack and shadow irq stack. + */ +SYM_FUNC_START(call_on_irq_stack) +#ifdef CONFIG_SHADOW_CALL_STACK + stp scs_sp, xzr, [sp, #-16]! + adr_this_cpu scs_sp, irq_shadow_call_stack, x17 +#endif + /* Create a frame record to save our LR and SP (implicit in FP) */ + stp x29, x30, [sp, #-16]! + mov x29, sp + + ldr_this_cpu x16, irq_stack_ptr, x17 + mov x15, #IRQ_STACK_SIZE + add x16, x16, x15 + + /* Move to the new stack and call the function there */ + mov sp, x16 + blr x1 + + /* + * Restore the SP from the FP, and restore the FP and LR from the frame + * record. + */ + mov sp, x29 + ldp x29, x30, [sp], #16 +#ifdef CONFIG_SHADOW_CALL_STACK + ldp scs_sp, xzr, [sp], #16 +#endif + ret +SYM_FUNC_END(call_on_irq_stack) +NOKPROBE(call_on_irq_stack) + #ifdef CONFIG_ARM_SDE_INTERFACE #include From patchwork Mon May 10 15:56:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C462C433B4 for ; Mon, 10 May 2021 16:00:22 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 956916161E for ; Mon, 10 May 2021 16:00:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 956916161E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=337Jc/f7NOPFm2jY1zIu308xUXZ+y1jqa1Uv331q/eQ=; b=UF3MkEJ4waGT41i0GXG+7TX45j 2MEgLiyuLEcVLOlf+zrvWc+3jDdWfq1MToGpkFX3sT+INrajCqpkAMxoQ6ajLDLUMKnY3/8F0XqF5 FLDVmUDAAo53SkEETmKnnn67jOS/CCpz6FJ6fbT3NMS+8vEXeBKCatp+NKwjzGGQ+/2eSNLVFgGH3 cFLTZ4bzkBcyxLmsQNhaOjhPfXbfOFchz8ooadDDeG1UJUjVusjqz0vhhBSHz9giZ8buiHAJavh+w gRJmgNN5A4tIBZo3bhPPpfB05fOql92fcu63lRIWFCynlbDY9g76ehnGe+4khYuT85Lxn11EPuI4G i4ConRjQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Id-00EquR-W9; Mon, 10 May 2021 15:58:28 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HT-00EqUy-An for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=nhiB0bfBqxGrcTqvPDD1CueO2hdiDyQh8YHbNZt33kY=; b=D2j/Md9Sp715hIzXUo/sxpHL5k IXsXhCqiEmNcQIhAct6CVzYSQCG+9izX+7fSKHs/PsFVEogeJL90HC0AgZ1Yl4lDTc4Qyt80r6/bP 7b+l/83wCAwx+ssBwTyV9O76FesPb5aSgHPKNitko+acOJyhPE7FBK1Pf22S6IhTKgDyvevU8tTUU v0axfHBEc/vYh7l1B6F9XAGX7hBexnWmAyQ5oVB9ZOt7CG+vxJdlsMoQH75RrVcAogEKkJzWDvqsm A1mxmCRJ2ZTD83u3JaTIesRUWP64aYmP/kLcJRZ6WpDd3kalN4DkkSIIyGqBCrBFOghCXpjgthvLx UdXhsz/g==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HQ-008tNd-1v for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:57:14 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 98856168F; Mon, 10 May 2021 08:57:11 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AE3103F73B; Mon, 10 May 2021 08:57:10 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 07/14] arm64: entry: convert IRQ+FIQ handlers to C Date: Mon, 10 May 2021 16:56:14 +0100 Message-Id: <20210510155621.52811-8-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085712_219420_39533462 X-CRM114-Status: GOOD ( 17.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For various reasons we'd like to convert the bulk of arm64's exception triage logic to C. As a step towards that, this patch converts the EL1 and EL0 IRQ+FIQ triage logic to C. Separate C functions are added for the native and compat cases so that in subsequent patches we can handle native/compat differences in C. Since the triage functions can now call arm64_apply_bp_hardening() directly, the do_el0_irq_bp_hardening() wrapper function is removed. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/include/asm/exception.h | 8 ++- arch/arm64/include/asm/processor.h | 2 - arch/arm64/kernel/entry-common.c | 86 +++++++++++++++++++++++++++++++-- arch/arm64/kernel/entry.S | 99 ++++++-------------------------------- arch/arm64/mm/fault.c | 7 --- 5 files changed, 102 insertions(+), 100 deletions(-) diff --git a/arch/arm64/include/asm/exception.h b/arch/arm64/include/asm/exception.h index c24b69c0c589..4284ee57a9a5 100644 --- a/arch/arm64/include/asm/exception.h +++ b/arch/arm64/include/asm/exception.h @@ -32,14 +32,18 @@ static inline u32 disr_to_esr(u64 disr) } asmlinkage void el1_sync_handler(struct pt_regs *regs); +asmlinkage void el1_irq_handler(struct pt_regs *regs); +asmlinkage void el1_fiq_handler(struct pt_regs *regs); asmlinkage void el1_error_handler(struct pt_regs *regs); asmlinkage void el0_sync_handler(struct pt_regs *regs); +asmlinkage void el0_irq_handler(struct pt_regs *regs); +asmlinkage void el0_fiq_handler(struct pt_regs *regs); asmlinkage void el0_error_handler(struct pt_regs *regs); asmlinkage void el0_sync_compat_handler(struct pt_regs *regs); +asmlinkage void el0_irq_compat_handler(struct pt_regs *regs); +asmlinkage void el0_fiq_compat_handler(struct pt_regs *regs); asmlinkage void el0_error_compat_handler(struct pt_regs *regs); -asmlinkage void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs); -asmlinkage void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs); asmlinkage void call_on_irq_stack(struct pt_regs *regs, void (*func)(struct pt_regs *)); asmlinkage void enter_from_user_mode(void); diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 9df3feeee890..2f21c76324bb 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -257,8 +257,6 @@ void set_task_sctlr_el1(u64 sctlr); extern struct task_struct *cpu_switch_to(struct task_struct *prev, struct task_struct *next); -asmlinkage void arm64_preempt_schedule_irq(void); - #define task_pt_regs(p) \ ((struct pt_regs *)(THREAD_SIZE + task_stack_page(p)) - 1) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 87997a4a0936..0e876062481a 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -19,6 +19,8 @@ #include #include #include +#include +#include #include /* @@ -101,7 +103,7 @@ void noinstr arm64_exit_nmi(struct pt_regs *regs) __nmi_exit(); } -asmlinkage void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs) +static void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs) { if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs)) arm64_enter_nmi(regs); @@ -109,7 +111,7 @@ asmlinkage void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs) enter_from_kernel_mode(regs); } -asmlinkage void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs) +static void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs) { if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs)) arm64_exit_nmi(regs); @@ -117,11 +119,11 @@ asmlinkage void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs) exit_to_kernel_mode(regs); } -asmlinkage void __sched arm64_preempt_schedule_irq(void) +static void __sched arm64_preempt_schedule_irq(void) { lockdep_assert_irqs_disabled(); - if (preempt_count() != 0) + if (!IS_ENABLED(CONFIG_PREEMPTION) || preempt_count() != 0) return; /* @@ -145,6 +147,18 @@ asmlinkage void __sched arm64_preempt_schedule_irq(void) preempt_schedule_irq(); } +static void do_interrupt_handler(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) +{ + if (on_thread_stack()) + call_on_irq_stack(regs, handler); + else + handler(regs); +} + +extern void (*handle_arch_irq)(struct pt_regs *); +extern void (*handle_arch_fiq)(struct pt_regs *); + #ifdef CONFIG_ARM64_ERRATUM_1463225 static DEFINE_PER_CPU(int, __in_cortex_a76_erratum_1463225_wa); @@ -311,6 +325,27 @@ asmlinkage void noinstr el1_sync_handler(struct pt_regs *regs) } } +static void noinstr el1_interrupt(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) +{ + write_sysreg(DAIF_PROCCTX_NOIRQ, daif); + + enter_el1_irq_or_nmi(regs); + do_interrupt_handler(regs, handler); + arm64_preempt_schedule_irq(); + exit_el1_irq_or_nmi(regs); +} + +asmlinkage void noinstr el1_irq_handler(struct pt_regs *regs) +{ + el1_interrupt(regs, handle_arch_irq); +} + +asmlinkage void noinstr el1_fiq_handler(struct pt_regs *regs) +{ + el1_interrupt(regs, handle_arch_fiq); +} + asmlinkage void noinstr el1_error_handler(struct pt_regs *regs) { unsigned long esr = read_sysreg(esr_el1); @@ -508,6 +543,39 @@ asmlinkage void noinstr el0_sync_handler(struct pt_regs *regs) } } +static void noinstr el0_interrupt(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) +{ + enter_from_user_mode(); + + write_sysreg(DAIF_PROCCTX_NOIRQ, daif); + + if (regs->pc & BIT(55)) + arm64_apply_bp_hardening(); + + do_interrupt_handler(regs, handler); +} + +static void noinstr __el0_irq_handler_common(struct pt_regs *regs) +{ + el0_interrupt(regs, handle_arch_irq); +} + +asmlinkage void noinstr el0_irq_handler(struct pt_regs *regs) +{ + __el0_irq_handler_common(regs); +} + +static void noinstr __el0_fiq_handler_common(struct pt_regs *regs) +{ + el0_interrupt(regs, handle_arch_fiq); +} + +asmlinkage void noinstr el0_fiq_handler(struct pt_regs *regs) +{ + __el0_fiq_handler_common(regs); +} + static void __el0_error_handler_common(struct pt_regs *regs) { unsigned long esr = read_sysreg(esr_el1); @@ -582,6 +650,16 @@ asmlinkage void noinstr el0_sync_compat_handler(struct pt_regs *regs) } } +asmlinkage void noinstr el0_irq_compat_handler(struct pt_regs *regs) +{ + __el0_irq_handler_common(regs); +} + +asmlinkage void noinstr el0_fiq_compat_handler(struct pt_regs *regs) +{ + __el0_fiq_handler_common(regs); +} + asmlinkage void noinstr el0_error_compat_handler(struct pt_regs *regs) { __el0_error_handler_common(regs); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 327a559679f7..eebc6e72125c 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -486,63 +486,12 @@ SYM_CODE_START_LOCAL(__swpan_exit_el0) SYM_CODE_END(__swpan_exit_el0) #endif - .macro irq_stack_entry - mov x19, sp // preserve the original sp -#ifdef CONFIG_SHADOW_CALL_STACK - mov x24, scs_sp // preserve the original shadow stack -#endif - - /* - * Compare sp with the base of the task stack. - * If the top ~(THREAD_SIZE - 1) bits match, we are on a task stack, - * and should switch to the irq stack. - */ - ldr x25, [tsk, TSK_STACK] - eor x25, x25, x19 - and x25, x25, #~(THREAD_SIZE - 1) - cbnz x25, 9998f - - ldr_this_cpu x25, irq_stack_ptr, x26 - mov x26, #IRQ_STACK_SIZE - add x26, x25, x26 - - /* switch to the irq stack */ - mov sp, x26 - -#ifdef CONFIG_SHADOW_CALL_STACK - /* also switch to the irq shadow stack */ - ldr_this_cpu scs_sp, irq_shadow_call_stack_ptr, x26 -#endif - -9998: - .endm - - /* - * The callee-saved regs (x19-x29) should be preserved between - * irq_stack_entry and irq_stack_exit, but note that kernel_entry - * uses x20-x23 to store data for later use. - */ - .macro irq_stack_exit - mov sp, x19 -#ifdef CONFIG_SHADOW_CALL_STACK - mov scs_sp, x24 -#endif - .endm - /* GPRs used by entry code */ tsk .req x28 // current thread_info /* * Interrupt handling. */ - .macro irq_handler, handler:req - ldr_l x1, \handler - mov x0, sp - irq_stack_entry - blr x1 - irq_stack_exit - .endm - .macro gic_prio_kentry_setup, tmp:req #ifdef CONFIG_ARM64_PSEUDO_NMI alternative_if ARM64_HAS_IRQ_PRIO_MASKING @@ -552,32 +501,6 @@ tsk .req x28 // current thread_info #endif .endm - .macro el1_interrupt_handler, handler:req - enable_da - - mov x0, sp - bl enter_el1_irq_or_nmi - - irq_handler \handler - -#ifdef CONFIG_PREEMPTION - bl arm64_preempt_schedule_irq // irq en/disable is done inside -#endif - - mov x0, sp - bl exit_el1_irq_or_nmi - .endm - - .macro el0_interrupt_handler, handler:req - user_exit_irqoff - enable_da - - tbz x22, #55, 1f - bl do_el0_irq_bp_hardening -1: - irq_handler \handler - .endm - .text /* @@ -701,13 +624,15 @@ SYM_CODE_END(el1_sync) .align 6 SYM_CODE_START_LOCAL_NOALIGN(el1_irq) kernel_entry 1 - el1_interrupt_handler handle_arch_irq + mov x0, sp + bl el1_irq_handler kernel_exit 1 SYM_CODE_END(el1_irq) SYM_CODE_START_LOCAL_NOALIGN(el1_fiq) kernel_entry 1 - el1_interrupt_handler handle_arch_fiq + mov x0, sp + bl el1_fiq_handler kernel_exit 1 SYM_CODE_END(el1_fiq) @@ -734,12 +659,16 @@ SYM_CODE_END(el0_sync_compat) .align 6 SYM_CODE_START_LOCAL_NOALIGN(el0_irq_compat) kernel_entry 0, 32 - b el0_irq_naked + mov x0, sp + bl el0_irq_compat_handler + b ret_to_user SYM_CODE_END(el0_irq_compat) SYM_CODE_START_LOCAL_NOALIGN(el0_fiq_compat) kernel_entry 0, 32 - b el0_fiq_naked + mov x0, sp + bl el0_fiq_compat_handler + b ret_to_user SYM_CODE_END(el0_fiq_compat) SYM_CODE_START_LOCAL_NOALIGN(el0_error_compat) @@ -753,15 +682,15 @@ SYM_CODE_END(el0_error_compat) .align 6 SYM_CODE_START_LOCAL_NOALIGN(el0_irq) kernel_entry 0 -el0_irq_naked: - el0_interrupt_handler handle_arch_irq + mov x0, sp + bl el0_irq_handler b ret_to_user SYM_CODE_END(el0_irq) SYM_CODE_START_LOCAL_NOALIGN(el0_fiq) kernel_entry 0 -el0_fiq_naked: - el0_interrupt_handler handle_arch_fiq + mov x0, sp + bl el0_fiq_handler b ret_to_user SYM_CODE_END(el0_fiq) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 871c82ab0a30..3b4a4adfddfd 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -836,13 +836,6 @@ void do_mem_abort(unsigned long far, unsigned int esr, struct pt_regs *regs) } NOKPROBE_SYMBOL(do_mem_abort); -void do_el0_irq_bp_hardening(void) -{ - /* PC has already been checked in entry.S */ - arm64_apply_bp_hardening(); -} -NOKPROBE_SYMBOL(do_el0_irq_bp_hardening); - void do_sp_pc_abort(unsigned long addr, unsigned int esr, struct pt_regs *regs) { arm64_notify_die("SP/PC alignment exception", regs, SIGBUS, BUS_ADRALN, From patchwork Mon May 10 15:56:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B877BC433B4 for ; Mon, 10 May 2021 16:00:36 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2EE56615FF for ; Mon, 10 May 2021 16:00:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2EE56615FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=2ajoO1ig74Da+yXkpGUevqyRnWwRWei1FG6xUFCLKX4=; b=CrD9Gb6FKKZ5qJX9Rff9ZE9AdT iCRD98T7A2mxmk3rJA3isFVMNoO1lQQOyPxYh0EKBmX6JWdls5tYrMvFmaLkBCBsy8RYZDvZql4i0 4TIl316tdAXcD9O4uVckpjELFoE9eWiSAscn0Rwcd3V6vtjS4lx5zEBBC+xtufif2FQUWx6uc8+fL g3h6MvcXWaUCSc//qkT79WqXQnSr8DaL5981vxpH7BsTJ3/buPMSXi37iFS5zF+67TiZA+7J3M7Ok SLLvoGrWNUKRZqBfplv5vMXO+aQt+Fd1JnvxUKHEG80wJvzdhu8fzr4e+OTGCQ2QvXVCLUEnoGfN3 +VXswPgA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Is-00Er13-Ih; Mon, 10 May 2021 15:58:42 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HW-00EqVZ-Hh for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=iIByC3Boxr/MJb22T5OC1nWtLAGtE70kcXzm8CYYjgs=; b=ZIWQUmKBFU/m8ChSmrlaz3+LAt eUJAg5yGVc54yc2Qh6YuUt0fB4Y117bf6aH3Ooye1eecpRg6mKdfo0qq26M1fQnQtoxpulGxaoJWu uyNY5+fDWZNd0lyy7ISjfh16E59p7M32QtrsMvLp2LEQ8Uc1cnwfUH/W6xJBhmUR5rVPPoZlhRxyL fRPVFKNSs7zPC8xRWK5hLshd/T0RXD5UCEoXBjOcSo1dhGGVkm0WYoCwsBiTuklBx8TlTv0cgIBal LmhSGoScu4bD2zAFcxHmvI8EtEIH3kIatCIhimj99OJ46S0uGz1UI25kiIto0z+4qMQOIusC+QhPC tPNHOHjQ==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HT-008tNw-Q4 for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:57:17 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 20878168F; Mon, 10 May 2021 08:57:14 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3096A3F73B; Mon, 10 May 2021 08:57:13 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 08/14] arm64: entry: organise entry handlers consistently Date: Mon, 10 May 2021 16:56:15 +0100 Message-Id: <20210510155621.52811-9-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085715_958045_F3F4AF74 X-CRM114-Status: GOOD ( 11.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In entry.S we have two comments which distinguish EL0 and EL1 exception handlers, but the code isn't actually laid out this way, and there are a few other inconsitencies that would be good to clear up. This patch organizes the entry handers consistently: * The handlers are laid out in order of the vectors, to make them easier to navigate. * All handlers are given the same alignment, which was previously applied inconsitently. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/kernel/entry.S | 64 ++++++++++++++++++++++++++--------------------- 1 file changed, 35 insertions(+), 29 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index eebc6e72125c..a1dd730a474d 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -629,6 +629,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_irq) kernel_exit 1 SYM_CODE_END(el1_irq) + .align 6 SYM_CODE_START_LOCAL_NOALIGN(el1_fiq) kernel_entry 1 mov x0, sp @@ -636,6 +637,14 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_fiq) kernel_exit 1 SYM_CODE_END(el1_fiq) + .align 6 +SYM_CODE_START_LOCAL_NOALIGN(el1_error) + kernel_entry 1 + mov x0, sp + bl el1_error_handler + kernel_exit 1 +SYM_CODE_END(el1_error) + /* * EL0 mode handlers. */ @@ -647,6 +656,30 @@ SYM_CODE_START_LOCAL_NOALIGN(el0_sync) b ret_to_user SYM_CODE_END(el0_sync) + .align 6 +SYM_CODE_START_LOCAL_NOALIGN(el0_irq) + kernel_entry 0 + mov x0, sp + bl el0_irq_handler + b ret_to_user +SYM_CODE_END(el0_irq) + + .align 6 +SYM_CODE_START_LOCAL_NOALIGN(el0_fiq) + kernel_entry 0 + mov x0, sp + bl el0_fiq_handler + b ret_to_user +SYM_CODE_END(el0_fiq) + + .align 6 +SYM_CODE_START_LOCAL_NOALIGN(el0_error) + kernel_entry 0 + mov x0, sp + bl el0_error_handler + b ret_to_user +SYM_CODE_END(el0_error) + #ifdef CONFIG_COMPAT .align 6 SYM_CODE_START_LOCAL_NOALIGN(el0_sync_compat) @@ -664,6 +697,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el0_irq_compat) b ret_to_user SYM_CODE_END(el0_irq_compat) + .align 6 SYM_CODE_START_LOCAL_NOALIGN(el0_fiq_compat) kernel_entry 0, 32 mov x0, sp @@ -671,6 +705,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el0_fiq_compat) b ret_to_user SYM_CODE_END(el0_fiq_compat) + .align 6 SYM_CODE_START_LOCAL_NOALIGN(el0_error_compat) kernel_entry 0, 32 mov x0, sp @@ -679,35 +714,6 @@ SYM_CODE_START_LOCAL_NOALIGN(el0_error_compat) SYM_CODE_END(el0_error_compat) #endif - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el0_irq) - kernel_entry 0 - mov x0, sp - bl el0_irq_handler - b ret_to_user -SYM_CODE_END(el0_irq) - -SYM_CODE_START_LOCAL_NOALIGN(el0_fiq) - kernel_entry 0 - mov x0, sp - bl el0_fiq_handler - b ret_to_user -SYM_CODE_END(el0_fiq) - -SYM_CODE_START_LOCAL(el1_error) - kernel_entry 1 - mov x0, sp - bl el1_error_handler - kernel_exit 1 -SYM_CODE_END(el1_error) - -SYM_CODE_START_LOCAL(el0_error) - kernel_entry 0 - mov x0, sp - bl el0_error_handler - b ret_to_user -SYM_CODE_END(el0_error) - /* * "slow" syscall return path. */ From patchwork Mon May 10 15:56:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E4F3C433ED for ; Mon, 10 May 2021 16:00:53 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1D22D6161C for ; Mon, 10 May 2021 16:00:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1D22D6161C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Cir6GfWW82LOa3wx02LFuvumHaUmLVMTRc2SaJMG1ts=; b=qd/YsHJlz8EiImhZHqCHILF0he YllyYs481CvUjp3DjvtDRShLxTm2ernDB8Fh48I8wAb24IzQsm/0/2JWFdRYT3QDNlWSuwm5RNgqc gb+xhStLttomvm3Xo0myq5/zKsmFCkVIzN5C6BxUfi5MJE6TghcOQVjtL+vZf+XlKAf4ZJdDz1Ke9 fS9F0s5RAde1qaIXSoAjTjuPiTLsrOpkHawLWhuMdwCiRhDWmsvZOej0Sl5wM7CUCStLNOg5xQMtq WdVENEi7oWQ2PzAOcAT3bcBS7R+TOkHp4oA6dXZT8p0dQ82YNGTA+oOM3myEuHzxZ6eZHWbCNqn5P RY6BXZvw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8JC-00Er9i-80; Mon, 10 May 2021 15:59:02 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HY-00EqVr-5w for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:20 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=ny0jS7CATh9skTziCAftAG/GVTs9hZq1ognXQDrQF50=; b=KQd9tktDr0Q/u2Hj4IYmZYIW4T YUQakN3vDncD5vTsSk+7f+txyn0KarNNwx37McxYZOfhVqc3PTb4XapHAKJTZ2SuwSaIU2BiUm4n9 n6bJglg7pP+h5pHuE6Yfsge/pL98GjuvEJKM0JZogNgBH01QxiM6fon2lSwF64cJHZafrBCQfMu1s 30OnjNwqUyedCrwfg4OssPNcsVWLOoWxMtD++YI0+F1U5lKIfBlNu3CWJwQEzqvI7MVjgBuHyRV2Z mx/0ashA6tqEErQJRzKGXghdgXLc+0Lat2xHadGi+Hmbv6xy6yopcLnws3gkeCT2dRsS7olTIO22M LK/KjNjg==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8HV-008tOA-D8 for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:57:18 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 05AC01691; Mon, 10 May 2021 08:57:17 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1AEE33F73B; Mon, 10 May 2021 08:57:16 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 09/14] arm64: entry: organise entry vectors consistently Date: Mon, 10 May 2021 16:56:16 +0100 Message-Id: <20210510155621.52811-10-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085717_555366_5A578A4E X-CRM114-Status: GOOD ( 10.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In subsequent patches we'll rename the entry handlers based on their original EL, register width, and exception class. To do so, we need to make all 3 mandatory arguments to the `kernel_ventry` macro, and distinguish EL1h from EL1t. In preparation for this, let's make the current set of arguments mandatory, and move the `regsize` column before the branch label suffix, making the vectors easier to read column-wise. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/kernel/entry.S | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index a1dd730a474d..be8e596af75a 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -60,7 +60,7 @@ #define BAD_FIQ 2 #define BAD_ERROR 3 - .macro kernel_ventry, el, label, regsize = 64 + .macro kernel_ventry, el:req, regsize:req, label:req .align 7 #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 .if \el == 0 @@ -510,31 +510,31 @@ tsk .req x28 // current thread_info .align 11 SYM_CODE_START(vectors) - kernel_ventry 1, sync_invalid // Synchronous EL1t - kernel_ventry 1, irq_invalid // IRQ EL1t - kernel_ventry 1, fiq_invalid // FIQ EL1t - kernel_ventry 1, error_invalid // Error EL1t + kernel_ventry 1, 64, sync_invalid // Synchronous EL1t + kernel_ventry 1, 64, irq_invalid // IRQ EL1t + kernel_ventry 1, 64, fiq_invalid // FIQ EL1t + kernel_ventry 1, 64, error_invalid // Error EL1t - kernel_ventry 1, sync // Synchronous EL1h - kernel_ventry 1, irq // IRQ EL1h - kernel_ventry 1, fiq // FIQ EL1h - kernel_ventry 1, error // Error EL1h + kernel_ventry 1, 64, sync // Synchronous EL1h + kernel_ventry 1, 64, irq // IRQ EL1h + kernel_ventry 1, 64, fiq // FIQ EL1h + kernel_ventry 1, 64, error // Error EL1h - kernel_ventry 0, sync // Synchronous 64-bit EL0 - kernel_ventry 0, irq // IRQ 64-bit EL0 - kernel_ventry 0, fiq // FIQ 64-bit EL0 - kernel_ventry 0, error // Error 64-bit EL0 + kernel_ventry 0, 64, sync // Synchronous 64-bit EL0 + kernel_ventry 0, 64, irq // IRQ 64-bit EL0 + kernel_ventry 0, 64, fiq // FIQ 64-bit EL0 + kernel_ventry 0, 64, error // Error 64-bit EL0 #ifdef CONFIG_COMPAT - kernel_ventry 0, sync_compat, 32 // Synchronous 32-bit EL0 - kernel_ventry 0, irq_compat, 32 // IRQ 32-bit EL0 - kernel_ventry 0, fiq_compat, 32 // FIQ 32-bit EL0 - kernel_ventry 0, error_compat, 32 // Error 32-bit EL0 + kernel_ventry 0, 32, sync_compat // Synchronous 32-bit EL0 + kernel_ventry 0, 32, irq_compat // IRQ 32-bit EL0 + kernel_ventry 0, 32, fiq_compat // FIQ 32-bit EL0 + kernel_ventry 0, 32, error_compat // Error 32-bit EL0 #else - kernel_ventry 0, sync_invalid, 32 // Synchronous 32-bit EL0 - kernel_ventry 0, irq_invalid, 32 // IRQ 32-bit EL0 - kernel_ventry 0, fiq_invalid, 32 // FIQ 32-bit EL0 - kernel_ventry 0, error_invalid, 32 // Error 32-bit EL0 + kernel_ventry 0, 32, sync_invalid // Synchronous 32-bit EL0 + kernel_ventry 0, 32, irq_invalid // IRQ 32-bit EL0 + kernel_ventry 0, 32, fiq_invalid // FIQ 32-bit EL0 + kernel_ventry 0, 32, error_invalid // Error 32-bit EL0 #endif SYM_CODE_END(vectors) From patchwork Mon May 10 15:56:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76930C433ED for ; Mon, 10 May 2021 16:00:57 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2C1756161C for ; Mon, 10 May 2021 16:00:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2C1756161C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=YIYQNEOg7Y3b9feTL4mf77mvO8I6mKy+pgqVTCgOoNY=; b=WIZBPS47RekhaHWxzNJvGF/9su MUUu1sE2nB3nCpLTafyElEAzBxykGOXE3abtbZqtjn8DzP4TQGlY5DdVJJvUeXB1xlDLM09O7CaRS Khdrd4IZyM5cyki/86cIaLYyuZhQMkEwtUYD570p1ZPOq35D1zxLBVEJPNw+qRmQDRGSrIo4KMBYp V8o+ARWcwZFoQE9gpt6ybMBe+SniMBaU+gk+/65d9MFf5dGSXI+a3v673NcoG9azwH60nwnpM3a5H 1/Dybrz8Z1e9aTVI7sKFZBmQQhLbGMsNBKdMzat/uFVkBBM/w/wyzu/cbLbpJPsYBxB1WLEXXLs52 iLeh2JYg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8JS-00ErI9-Ga; Mon, 10 May 2021 15:59:18 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hg-00EqXT-6A for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=2mTtX6iY2hzmdquNPod1WlCD6UWozzacPvy7Ql/qZxA=; b=XCd4WcjGBfZUcPKkZMtFDTOi0m ceXAzMpmzcWgIkEuNyi/mRwPPfNNZpsBnlFFoxuMOHgRK9F0ncJ734Vb/2DRqPwn8eUsTtq0NlXE9 oKmQ9tZYedsXdd8K6Mv3yo85txhj6Ti8c7XQctZKF9e4PxCS7ewD4OM/MH0GtfMm7aVMsSSLw1SFJ VNcfwjHi9CZVuU0S4Grrj3JIDDGnduqfZKXVhnZx/y2k/KwlkXSfqnxuK/tDp1hNEoLyYPB3ASHlY zTeymsrKd6Ijewl/UV1mawzhSQ3nwGuWZRMK+mFfD+QjPjM3X2stdpzWUHKR3aur6XLPHWjJvZoj9 +Iqp+2cg==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hd-008tOX-JQ for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:57:26 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 37217168F; Mon, 10 May 2021 08:57:23 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 407993F73B; Mon, 10 May 2021 08:57:22 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 10/14] arm64: entry: consolidate EL1 exception returns Date: Mon, 10 May 2021 16:56:17 +0100 Message-Id: <20210510155621.52811-11-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085725_702468_127ACD89 X-CRM114-Status: UNSURE ( 8.63 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Following the example of ret_to_user, let's consolidate all the EL1 return paths with a ret_to_kernel helper, rather than each entry point having its own copy of the return code. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/kernel/entry.S | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index be8e596af75a..9b662d4c09d8 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -618,7 +618,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_sync) kernel_entry 1 mov x0, sp bl el1_sync_handler - kernel_exit 1 + b ret_to_kernel SYM_CODE_END(el1_sync) .align 6 @@ -626,7 +626,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_irq) kernel_entry 1 mov x0, sp bl el1_irq_handler - kernel_exit 1 + b ret_to_kernel SYM_CODE_END(el1_irq) .align 6 @@ -634,7 +634,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_fiq) kernel_entry 1 mov x0, sp bl el1_fiq_handler - kernel_exit 1 + b ret_to_kernel SYM_CODE_END(el1_fiq) .align 6 @@ -642,9 +642,13 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_error) kernel_entry 1 mov x0, sp bl el1_error_handler - kernel_exit 1 + b ret_to_kernel SYM_CODE_END(el1_error) +SYM_CODE_START_LOCAL(ret_to_kernel) + kernel_exit 1 +SYM_CODE_END(ret_to_kernel) + /* * EL0 mode handlers. */ From patchwork Mon May 10 15:56:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D15AC433B4 for ; Mon, 10 May 2021 16:01:53 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B7C89611CA for ; Mon, 10 May 2021 16:01:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B7C89611CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=NtUudBY5R6GB08jv6Vu6ULKDnvaAVVQGD7TAVD/POMc=; b=JnrlvVWtMUplOH7mx8E6KCsKPJ NN7niyEipKMjlfHXZASoE2QzIiQ1ip4eHSxNhVHKnHMi5wtBmIU/ipUbbdRu++XJvmWTM3gQH3KGz QOhnZ723qY6/gD9p3rpC+ka1TaP9AHyvegploI3bHotAsqcJEUKEWrzr5SmQV8PXqwChRyH8hWmZ0 /SYGTIh27mF+Rze55gaK0WCK6BggQS9fvMuaFmw6C1YBdfDcfPQuLGoibcNeLYBXlOItFhSfb9LCr KDu4DYsi2rEh6HEveVSkmMvU/c9V6DI1zWGYtBRr4UskDhkbPSnXUBmbYGyEtBs7Y86rnPfW4Z12W 6rcKjPog==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Jr-00ErS3-Rs; Mon, 10 May 2021 15:59:44 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hl-00EqYv-Ld for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=nb6SBfPXJfMmQjOlOlXL29+aGvDa3UnUlZd6RHXnSGQ=; b=Li7GuQy7sz5oVNW9dNOascDIFK P+pzsjDQd2XwepAVNUwKDzv2ESCfgPFnu+VxtC/xO1Dxa9IL9L1DNS7ZH2C69o1BgdGdQv8Lfetps nH9j1z20s3pdqg5JUZXHSxJlCLNnhkh6Fhel1uRCFuaJEdZjOxiKPZ/GI/zNNJEs6hsVxV+Y1VRln n67OR/4gk7LhBmAOHM7vOPBG9UBN4W3PTG2FrNxppKRKD34ChTue4foytF3mRDLGUEJDS9FEz3scf g3qW/pLSOjQ3uOWOdIY0njsezfEiRMLIP8FsEJ6ud37Ej2XnIbfd4jEP4JINPP8R5O5AWL/uYa/lY Aiqm/l2A==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hi-008tOm-Ue for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:57:32 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 39AA81691; Mon, 10 May 2021 08:57:25 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4A8AB3F73B; Mon, 10 May 2021 08:57:24 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 11/14] arm64: entry: move bad_mode() to entry-common.c Date: Mon, 10 May 2021 16:56:18 +0100 Message-Id: <20210510155621.52811-12-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085731_099585_3637E178 X-CRM114-Status: GOOD ( 14.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In subsequent patches we'll rework the way bad_mode is called by exception entry code. In preparation for this, let's move bad_mode() itself into entry-common.c. Let's also mark it as noinstr (e.g. to prevent it being kprobed), and let's also make the `handler` array a local variable, as this is only use by bad_mode(), and will be removed entirely in a subsequent patch. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/kernel/entry-common.c | 27 +++++++++++++++++++++++++++ arch/arm64/kernel/traps.c | 25 ------------------------- 2 files changed, 27 insertions(+), 25 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 0e876062481a..1b63687c38cd 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -22,6 +22,7 @@ #include #include #include +#include /* * This is intended to match the logic in irqentry_enter(), handling the kernel @@ -159,6 +160,32 @@ static void do_interrupt_handler(struct pt_regs *regs, extern void (*handle_arch_irq)(struct pt_regs *); extern void (*handle_arch_fiq)(struct pt_regs *); +/* + * bad_mode handles the impossible case in the exception vector. This is always + * fatal. + */ +asmlinkage void noinstr bad_mode(struct pt_regs *regs, int reason, unsigned int esr) +{ + const char *handler[] = { + "Synchronous Abort", + "IRQ", + "FIQ", + "Error" + }; + + arm64_enter_nmi(regs); + + console_verbose(); + + pr_crit("Bad mode in %s handler detected on CPU%d, code 0x%08x -- %s\n", + handler[reason], smp_processor_id(), esr, + esr_get_class_string(esr)); + + __show_regs(regs); + panic("bad mode"); +} + + #ifdef CONFIG_ARM64_ERRATUM_1463225 static DEFINE_PER_CPU(int, __in_cortex_a76_erratum_1463225_wa); diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 41f0aa92022a..e62b92b32156 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -45,13 +45,6 @@ #include #include -static const char *handler[] = { - "Synchronous Abort", - "IRQ", - "FIQ", - "Error" -}; - int show_unhandled_signals = 0; static void dump_kernel_instr(const char *lvl, struct pt_regs *regs) @@ -751,24 +744,6 @@ const char *esr_get_class_string(u32 esr) } /* - * bad_mode handles the impossible case in the exception vector. This is always - * fatal. - */ -asmlinkage void notrace bad_mode(struct pt_regs *regs, int reason, unsigned int esr) -{ - arm64_enter_nmi(regs); - - console_verbose(); - - pr_crit("Bad mode in %s handler detected on CPU%d, code 0x%08x -- %s\n", - handler[reason], smp_processor_id(), esr, - esr_get_class_string(esr)); - - __show_regs(regs); - panic("bad mode"); -} - -/* * bad_el0_sync handles unexpected, but potentially recoverable synchronous * exceptions taken from EL0. Unlike bad_mode, this returns. */ From patchwork Mon May 10 15:56:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BBF5C433B4 for ; Mon, 10 May 2021 16:02:46 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 07DDB6121E for ; Mon, 10 May 2021 16:02:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 07DDB6121E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Z4sO1YBvwGYZDMPUhEp+dzKvP3DOQ5FSqyR9lwXenhc=; b=WChK7uoTOlBhuMPLOdG/zHo87e X4isKfAo9OHIDrpDvQQhudsX3iG4YggxKu0PDOBdKnknSuue56QSgNBNcC8vTYRqQ6hOEVDbyzb4f X6fKu6Uc/r6d4saITsefXyG/hSNXi/80ok2goza+u3RA/4aiTyK+ITVas1YkxkczCEN4ybuVwpuMg /68K2SCUcjrIE9xKlVppNPbjx1Cr4tlXR0Jx5wsTKTbDQ11INZ/2dDf3RgjUlhYUstKiXfKhqggCA iD3RLgd69ByYaLmUw279YidsnhR0SBUnaYMrdMfX9RLqrQq2BtqhUjTsB7OxfHLo0RGajeJ2FkIxQ lZFGauBQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Kf-00ErpA-H5; Mon, 10 May 2021 16:00:33 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hn-00EqZZ-Lh for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=VImY9bU9dISHsggWJFI7Hu5M55woXd3+jwEXDJSLsus=; b=j7u5swPDY09BNY6i+p9JdKU1Zu HkvPrz/t7xU74cGKi2jSyieo0SuhYx0T0nTUDM1jPit1ZFE3SyeyvOr6DQvxbOdi6MaM5GZeY+Dlg yfnJGJDDjUgTSjItIMYzA5/x9fQ6S4Pydb+pchJKKnd4pX3LG2YrUpbGAk3av4XT9YS8141uWQzG6 YrEpBdqTf7hE4570RLw0R0u7PXi3uvxNpbn9WGYFXHG/7BQSz62RfUIN0zvSYXwoqvfKbcl4PkrwB t+fwz/tl1uZ8JseFfdtYdMJAD7RC7wLNzw3PxZpi3xQsboxHUquhlxOBq29TIGX3HEXKKG3vnS4wC NGYgl8eg==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hk-008tPG-Vg for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:57:34 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4F5D7169C; Mon, 10 May 2021 08:57:32 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 67EEC3F73B; Mon, 10 May 2021 08:57:26 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 12/14] arm64: entry: improve bad_mode() Date: Mon, 10 May 2021 16:56:19 +0100 Message-Id: <20210510155621.52811-13-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085733_084384_0DE9B35C X-CRM114-Status: GOOD ( 14.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Our use of bad_mode() has a few rough edges: * AArch64 doesn't use the term "mode", and refers to "Execution states", "Exception levels", and "Selected stack pointer". * We log the exception type (SYNC/IRQ/FIQ/SError), but not the actual "mode" (though this can be deocded from the SPSR value). * We use bad_mode() as a second-level handler for unexpected synchronous exceptions, where the "mode" is legitimate, but the specific exception is not. * We dump the ESR value, but call this "code", and so it's not clear to all readers that this is the ESR. ... and all of this can be someqhat opaque to those who aren't extremely familiar with the code. Let's make this a bit clearer by having bad_mode() log "Unhandled ${TYPE} exception" rather than "Bad mode in ${TYPE} handler", using "ESR" rather than "code", and having the final panic() log "Unhandled exception" rather than "Bad mode". In future we'd like to log the specific architectural vector rather than just the type of exception, so we also split the core of basd_mode() out into a helper called __panic_unhandled(), which takes the vector as a string argument. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/kernel/entry-common.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 1b63687c38cd..d7087e22ffc1 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -160,31 +160,32 @@ static void do_interrupt_handler(struct pt_regs *regs, extern void (*handle_arch_irq)(struct pt_regs *); extern void (*handle_arch_fiq)(struct pt_regs *); -/* - * bad_mode handles the impossible case in the exception vector. This is always - * fatal. - */ -asmlinkage void noinstr bad_mode(struct pt_regs *regs, int reason, unsigned int esr) +static void noinstr __panic_unhandled(struct pt_regs *regs, const char *vector, + unsigned int esr) { - const char *handler[] = { - "Synchronous Abort", - "IRQ", - "FIQ", - "Error" - }; - arm64_enter_nmi(regs); console_verbose(); - pr_crit("Bad mode in %s handler detected on CPU%d, code 0x%08x -- %s\n", - handler[reason], smp_processor_id(), esr, + pr_crit("Unhandled %s exception on CPU%d, ESR 0x%08x -- %s\n", + vector, smp_processor_id(), esr, esr_get_class_string(esr)); __show_regs(regs); - panic("bad mode"); + panic("Unhandled exception"); } +asmlinkage void noinstr bad_mode(struct pt_regs *regs, int reason, unsigned int esr) +{ + const char *handler[] = { + "Synchronous Abort", + "IRQ", + "FIQ", + "Error" + }; + + __panic_unhandled(regs, handler[reason], esr); +} #ifdef CONFIG_ARM64_ERRATUM_1463225 static DEFINE_PER_CPU(int, __in_cortex_a76_erratum_1463225_wa); From patchwork Mon May 10 15:56:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BF15C433ED for ; Mon, 10 May 2021 16:03:08 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8861C611CA for ; Mon, 10 May 2021 16:03:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8861C611CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=tRzz9tunZvKt1L4EEUarn1Gvn1h+c/28Qw5DRxGQEZc=; b=UC9gefgUL7LPsrzoshYwi+77y1 1Jzl9RPO3XSOQ9f6l7WuoyiaTcozx2kZuTJKjXFJdkTJ7/Otv6fIU0l/SypOJGD6fw20Pn92mmmZo L+KhEYkNK6ZzBXl2PEshNuBwWs3t017DxU5DvePtulAUSGEJexTqQ901uhPFnlwU0D23k5sPxqsH1 CZzqW/L0O3dHvdwem1vIXvn4V7WU3hp+COTbhl+lyAoOq4qfwSfHyKR0X3HGwrN04fXBsoSslGXBz Lg/C/pXnHZ2ZIRdR1b2qe/T/TO7zlEEOVYUlf/zZFc4DQ55gik5GcGLZANZZ0ns5FCFH35xWJh2hs OtjICLzA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8LA-00Es1u-2M; Mon, 10 May 2021 16:01:04 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hq-00EqcP-0b for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=TcRyWRhkX3o976xkKzyB5hIgbTW4WaBADqaw8SAQna4=; b=w8ohLficAE0Iv0Fbau7UE8HGTs GPb0xvcnwDWf/KnjUNkidJ113UsHr3qn0Pz515BdicGiZpSoqCL/SeaJY3VE8DGzZoqGOIA6wWlKL Y/hdGyxG3dSrFPuyVldvCHU1vltZ4iM3NRm3X8fmkxSti508W7zh45ELBu4jjJsVPg3RoIKW3y9FO FTVQdGmkAfFD5WP2F4sR+4R6+JDiJ+Mj3gyuon0suU3Dkx5mc0vgaLAuhJIe44l79IetWG4c+gvQx iBROUbElFOQl40NHba0IxJQRLumd5ZyMMU+Hgnp+9ZxkqqjcFN1orswIAqdMzJZ3Kbgf3cSI8uWXZ Dp27RkfQ==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hn-008tPZ-8j for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:57:36 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0A578168F; Mon, 10 May 2021 08:57:29 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 20C043F7F5; Mon, 10 May 2021 08:57:28 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 13/14] arm64: entry: template the entry asm functions Date: Mon, 10 May 2021 16:56:20 +0100 Message-Id: <20210510155621.52811-14-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085735_418690_5EF0EB0E X-CRM114-Status: GOOD ( 10.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that the majority of the exception triage logic has been converted to C, the entry assembly functions all have a uniform structure. Let's generate them all with an assembly macro to reduce the amount of code and to ensure they all remain in sync if we make changes in future. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/kernel/entry.S | 124 ++++++++++------------------------------------ 1 file changed, 27 insertions(+), 97 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 9b662d4c09d8..d4f80b9df621 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -610,114 +610,44 @@ SYM_CODE_START_LOCAL(el1_error_invalid) inv_entry 1, BAD_ERROR SYM_CODE_END(el1_error_invalid) -/* - * EL1 mode handlers. - */ - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el1_sync) - kernel_entry 1 - mov x0, sp - bl el1_sync_handler - b ret_to_kernel -SYM_CODE_END(el1_sync) - - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el1_irq) - kernel_entry 1 - mov x0, sp - bl el1_irq_handler - b ret_to_kernel -SYM_CODE_END(el1_irq) - - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el1_fiq) - kernel_entry 1 - mov x0, sp - bl el1_fiq_handler - b ret_to_kernel -SYM_CODE_END(el1_fiq) - + .macro entry_handler el:req, regsize:req, label:req .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el1_error) - kernel_entry 1 +SYM_CODE_START_LOCAL_NOALIGN(el\el\()_\label) + kernel_entry \el, \regsize mov x0, sp - bl el1_error_handler + bl el\el\()_\label\()_handler + .if \el == 0 + b ret_to_user + .else b ret_to_kernel -SYM_CODE_END(el1_error) - -SYM_CODE_START_LOCAL(ret_to_kernel) - kernel_exit 1 -SYM_CODE_END(ret_to_kernel) + .endif +SYM_CODE_END(el\el\()_\label) + .endm /* - * EL0 mode handlers. + * Early exception handlers */ - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el0_sync) - kernel_entry 0 - mov x0, sp - bl el0_sync_handler - b ret_to_user -SYM_CODE_END(el0_sync) + entry_handler 1, 64, sync + entry_handler 1, 64, irq + entry_handler 1, 64, fiq + entry_handler 1, 64, error - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el0_irq) - kernel_entry 0 - mov x0, sp - bl el0_irq_handler - b ret_to_user -SYM_CODE_END(el0_irq) - - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el0_fiq) - kernel_entry 0 - mov x0, sp - bl el0_fiq_handler - b ret_to_user -SYM_CODE_END(el0_fiq) - - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el0_error) - kernel_entry 0 - mov x0, sp - bl el0_error_handler - b ret_to_user -SYM_CODE_END(el0_error) + entry_handler 0, 64, sync + entry_handler 0, 64, irq + entry_handler 0, 64, fiq + entry_handler 0, 64, error #ifdef CONFIG_COMPAT - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el0_sync_compat) - kernel_entry 0, 32 - mov x0, sp - bl el0_sync_compat_handler - b ret_to_user -SYM_CODE_END(el0_sync_compat) - - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el0_irq_compat) - kernel_entry 0, 32 - mov x0, sp - bl el0_irq_compat_handler - b ret_to_user -SYM_CODE_END(el0_irq_compat) - - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el0_fiq_compat) - kernel_entry 0, 32 - mov x0, sp - bl el0_fiq_compat_handler - b ret_to_user -SYM_CODE_END(el0_fiq_compat) - - .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el0_error_compat) - kernel_entry 0, 32 - mov x0, sp - bl el0_error_compat_handler - b ret_to_user -SYM_CODE_END(el0_error_compat) + entry_handler 0, 32, sync_compat + entry_handler 0, 32, irq_compat + entry_handler 0, 32, fiq_compat + entry_handler 0, 32, error_compat #endif +SYM_CODE_START_LOCAL(ret_to_kernel) + kernel_exit 1 +SYM_CODE_END(ret_to_kernel) + /* * "slow" syscall return path. */ From patchwork Mon May 10 15:56:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12248645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEE27C433ED for ; Mon, 10 May 2021 16:02:43 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 39CE461260 for ; Mon, 10 May 2021 16:02:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 39CE461260 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=L/7QQp97pD+z1P/nI6oUdulrPoMCo13mCxfd/SLHAN4=; b=XfxtaXsU6YmI28TdbfYelOi7jB heS4II5OGWKi0r1XFh36BwnhBJdxT3wW6tGoocpyYSKAQkTv7JeLwfNJlaVbJkT6Ernk3LDWEPCo8 XxUKGduUVAiPJ1SWHPUL1bUw5GCE+PrBikB7pT10Td6chrcq6PboIFQYJs4tMRNILqsECgsSjMlH3 A9axs8uL8WXON8BSfbY/+Uo9/I7eMLFILF/H9v/hi6nKxvnIKcakkLP6FsQ3bdcG8FAlqPiUqmjzg Pj8Y0yys5aw9KVU8srBlH2iyeC7o6HnnK9YmRhxO8CET1tblNJKesXjWW28qTCZ+28pvqo2++qR9i 5iNpY3Cw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8KH-00ErfG-Rp; Mon, 10 May 2021 16:00:10 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hm-00EqZN-UU for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 15:57:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=5S1AM1wjV2sO+TWHT2ywtG1SfQBpJaSKIW9Kvi05swU=; b=ZkrrDPe8qCnE/mSt0SpCXzOMSf VJ2O5Sm/jiHBjRCX04owBjW1JSrwDFJSTYQFDNtG2UvhpGLoqfpbeOF1q4wUYFYRKC6sQezYinRzq eFvK9l6ljcTzvamKd05U54SKrM2CSm6urZoNhcaQGILLl/a75IfWMCkky65caIAhFh/RGFchYfdji oXW6U9uYTFhuMz/ATl6NP6mNcKqydXhP3JoHYMd37UQCBzySEOct/8H+S0e35GwB7hLpbgYPlAGln GyCC8+p+2qpeVpd3hXQ5+YRqbMgDgEswRss46aV3x/2wSayLAKIHEE2MP61JaRgfi2qMmxwMJ1Tak J4Zn11+g==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg8Hj-008tP0-CK for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 15:57:33 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C5E1A1692; Mon, 10 May 2021 08:57:30 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DEADD3F7F5; Mon, 10 May 2021 08:57:29 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Cc: james.morse@arm.com, mark.rutland@arm.com, maz@kernel.org Subject: [PATCH 14/14] arm64: entry: handle all vectors with C Date: Mon, 10 May 2021 16:56:21 +0100 Message-Id: <20210510155621.52811-15-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210510155621.52811-1-mark.rutland@arm.com> References: <20210510155621.52811-1-mark.rutland@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_085731_540156_45A4B94A X-CRM114-Status: GOOD ( 18.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We have 16 architectural exception vectors, and depending on kernel configuration we handle 8 or 12 of these with C code, and we handle 8 or 4 of these as sepcial cases in the entry assembly. It would be nicer if the entry assembly were uniform for all exceptions, and we deferred any specific handling of the exceptions to C code. This way the entry assembly can be more easily templated without ifdeffery or special cases, and it's easier to modify the handling of these cases in future (e.g. to dump additional registers other context). This patch reworks the entry code so that we always have a C handle for every architectural exception vector, with the entry assembly being completely uniform. We now have to handle exceptions from EL1t and EL1h, and also have to handle exceptions from AArch32 even when the kernel is built without CONFIG_COMPAT. To make this clear and to simplify templating, we rename the top-level exception handlers with a consistent naming scheme: asm: __ c: ___handler .. where: is `el1t`, `el1h`, or `el0` is `64` or `32` is `sync`, `irq`, `fiq`, or `error` ... e.g. asm: el1h_64_sync c: el1h_64_sync_handler ... with lower-level handlers simply using "el1" and "compat" as today. For unexpected exceptions, this information is passed to panic_unandled(), so it can report the specific vector an unexpected exception was taken from, e.g. | Unexpected 64-bit el1t sync exception For vectors we never expect to enter legitimately, the C code is gnerated using a macro to avoid code duplication. The `kernel_ventry` and `entry_handler` assembly macros are update to handle the new naming scheme. In theory it should be possible to generate the entry functions at the same time as the vectors using a single table, but this will require reworking the linker script to split the two into separate sections, so for now we duplicate the two. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Will Deacon --- arch/arm64/include/asm/exception.h | 31 +++++--- arch/arm64/kernel/entry-common.c | 51 +++++++------ arch/arm64/kernel/entry.S | 146 ++++++++++++------------------------- 3 files changed, 92 insertions(+), 136 deletions(-) diff --git a/arch/arm64/include/asm/exception.h b/arch/arm64/include/asm/exception.h index 4284ee57a9a5..40a3a20dca1c 100644 --- a/arch/arm64/include/asm/exception.h +++ b/arch/arm64/include/asm/exception.h @@ -31,18 +31,25 @@ static inline u32 disr_to_esr(u64 disr) return esr; } -asmlinkage void el1_sync_handler(struct pt_regs *regs); -asmlinkage void el1_irq_handler(struct pt_regs *regs); -asmlinkage void el1_fiq_handler(struct pt_regs *regs); -asmlinkage void el1_error_handler(struct pt_regs *regs); -asmlinkage void el0_sync_handler(struct pt_regs *regs); -asmlinkage void el0_irq_handler(struct pt_regs *regs); -asmlinkage void el0_fiq_handler(struct pt_regs *regs); -asmlinkage void el0_error_handler(struct pt_regs *regs); -asmlinkage void el0_sync_compat_handler(struct pt_regs *regs); -asmlinkage void el0_irq_compat_handler(struct pt_regs *regs); -asmlinkage void el0_fiq_compat_handler(struct pt_regs *regs); -asmlinkage void el0_error_compat_handler(struct pt_regs *regs); +asmlinkage void el1t_64_sync_handler(struct pt_regs *regs); +asmlinkage void el1t_64_irq_handler(struct pt_regs *regs); +asmlinkage void el1t_64_fiq_handler(struct pt_regs *regs); +asmlinkage void el1t_64_error_handler(struct pt_regs *regs); + +asmlinkage void el1h_64_sync_handler(struct pt_regs *regs); +asmlinkage void el1h_64_irq_handler(struct pt_regs *regs); +asmlinkage void el1h_64_fiq_handler(struct pt_regs *regs); +asmlinkage void el1h_64_error_handler(struct pt_regs *regs); + +asmlinkage void el0_64_sync_handler(struct pt_regs *regs); +asmlinkage void el0_64_irq_handler(struct pt_regs *regs); +asmlinkage void el0_64_fiq_handler(struct pt_regs *regs); +asmlinkage void el0_64_error_handler(struct pt_regs *regs); + +asmlinkage void el0_32_sync_handler(struct pt_regs *regs); +asmlinkage void el0_32_irq_handler(struct pt_regs *regs); +asmlinkage void el0_32_fiq_handler(struct pt_regs *regs); +asmlinkage void el0_32_error_handler(struct pt_regs *regs); asmlinkage void call_on_irq_stack(struct pt_regs *regs, void (*func)(struct pt_regs *)); diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index d7087e22ffc1..9b6451e836d8 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -175,16 +175,11 @@ static void noinstr __panic_unhandled(struct pt_regs *regs, const char *vector, panic("Unhandled exception"); } -asmlinkage void noinstr bad_mode(struct pt_regs *regs, int reason, unsigned int esr) -{ - const char *handler[] = { - "Synchronous Abort", - "IRQ", - "FIQ", - "Error" - }; - - __panic_unhandled(regs, handler[reason], esr); +#define UNHANDLED(el, regsize, vector) \ +asmlinkage void noinstr el##_##regsize##_##vector##_handler(struct pt_regs *regs) \ +{ \ + const char *desc = #regsize "-bit " #el " " #vector; \ + __panic_unhandled(regs, desc, read_sysreg(esr_el1)); \ } #ifdef CONFIG_ARM64_ERRATUM_1463225 @@ -236,6 +231,11 @@ static bool cortex_a76_erratum_1463225_debug_handler(struct pt_regs *regs) } #endif /* CONFIG_ARM64_ERRATUM_1463225 */ +UNHANDLED(el1t, 64, sync) +UNHANDLED(el1t, 64, irq) +UNHANDLED(el1t, 64, fiq) +UNHANDLED(el1t, 64, error) + static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr) { unsigned long far = read_sysreg(far_el1); @@ -271,7 +271,7 @@ static void noinstr el1_inv(struct pt_regs *regs, unsigned long esr) { enter_from_kernel_mode(regs); local_daif_inherit(regs); - bad_mode(regs, 0, esr); + __panic_unhandled(regs, "el1h sync", esr); local_daif_mask(); exit_to_kernel_mode(regs); } @@ -319,7 +319,7 @@ static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr) exit_to_kernel_mode(regs); } -asmlinkage void noinstr el1_sync_handler(struct pt_regs *regs) +asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs) { unsigned long esr = read_sysreg(esr_el1); @@ -364,17 +364,17 @@ static void noinstr el1_interrupt(struct pt_regs *regs, exit_el1_irq_or_nmi(regs); } -asmlinkage void noinstr el1_irq_handler(struct pt_regs *regs) +asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs) { el1_interrupt(regs, handle_arch_irq); } -asmlinkage void noinstr el1_fiq_handler(struct pt_regs *regs) +asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs) { el1_interrupt(regs, handle_arch_fiq); } -asmlinkage void noinstr el1_error_handler(struct pt_regs *regs) +asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs) { unsigned long esr = read_sysreg(esr_el1); @@ -518,7 +518,7 @@ static void noinstr el0_fpac(struct pt_regs *regs, unsigned long esr) do_ptrauth_fault(regs, esr); } -asmlinkage void noinstr el0_sync_handler(struct pt_regs *regs) +asmlinkage void noinstr el0_64_sync_handler(struct pt_regs *regs) { unsigned long esr = read_sysreg(esr_el1); @@ -589,7 +589,7 @@ static void noinstr __el0_irq_handler_common(struct pt_regs *regs) el0_interrupt(regs, handle_arch_irq); } -asmlinkage void noinstr el0_irq_handler(struct pt_regs *regs) +asmlinkage void noinstr el0_64_irq_handler(struct pt_regs *regs) { __el0_irq_handler_common(regs); } @@ -599,7 +599,7 @@ static void noinstr __el0_fiq_handler_common(struct pt_regs *regs) el0_interrupt(regs, handle_arch_fiq); } -asmlinkage void noinstr el0_fiq_handler(struct pt_regs *regs) +asmlinkage void noinstr el0_64_fiq_handler(struct pt_regs *regs) { __el0_fiq_handler_common(regs); } @@ -614,7 +614,7 @@ static void __el0_error_handler_common(struct pt_regs *regs) local_daif_restore(DAIF_PROCCTX); } -asmlinkage void noinstr el0_error_handler(struct pt_regs *regs) +asmlinkage void noinstr el0_64_error_handler(struct pt_regs *regs) { __el0_error_handler_common(regs); } @@ -634,7 +634,7 @@ static void noinstr el0_svc_compat(struct pt_regs *regs) do_el0_svc_compat(regs); } -asmlinkage void noinstr el0_sync_compat_handler(struct pt_regs *regs) +asmlinkage void noinstr el0_32_sync_handler(struct pt_regs *regs) { unsigned long esr = read_sysreg(esr_el1); @@ -678,18 +678,23 @@ asmlinkage void noinstr el0_sync_compat_handler(struct pt_regs *regs) } } -asmlinkage void noinstr el0_irq_compat_handler(struct pt_regs *regs) +asmlinkage void noinstr el0_32_irq_handler(struct pt_regs *regs) { __el0_irq_handler_common(regs); } -asmlinkage void noinstr el0_fiq_compat_handler(struct pt_regs *regs) +asmlinkage void noinstr el0_32_fiq_handler(struct pt_regs *regs) { __el0_fiq_handler_common(regs); } -asmlinkage void noinstr el0_error_compat_handler(struct pt_regs *regs) +asmlinkage void noinstr el0_32_error_handler(struct pt_regs *regs) { __el0_error_handler_common(regs); } +#else /* CONFIG_COMPAT */ +UNHANDLED(el0, 32, sync) +UNHANDLED(el0, 32, irq) +UNHANDLED(el0, 32, fiq) +UNHANDLED(el0, 32, error) #endif /* CONFIG_COMPAT */ diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index d4f80b9df621..257e8192e8d8 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -51,16 +51,7 @@ .endr .endm -/* - * Bad Abort numbers - *----------------- - */ -#define BAD_SYNC 0 -#define BAD_IRQ 1 -#define BAD_FIQ 2 -#define BAD_ERROR 3 - - .macro kernel_ventry, el:req, regsize:req, label:req + .macro kernel_ventry, el:req, ht, regsize:req, label:req .align 7 #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 .if \el == 0 @@ -87,7 +78,7 @@ alternative_else_nop_endif tbnz x0, #THREAD_SHIFT, 0f sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp - b el\()\el\()_\label + b el\el\ht\()_\regsize\()_\label 0: /* @@ -119,7 +110,7 @@ alternative_else_nop_endif sub sp, sp, x0 mrs x0, tpidrro_el0 #endif - b el\()\el\()_\label + b el\el\ht\()_\regsize\()_\label .endm .macro tramp_alias, dst, sym @@ -510,32 +501,25 @@ tsk .req x28 // current thread_info .align 11 SYM_CODE_START(vectors) - kernel_ventry 1, 64, sync_invalid // Synchronous EL1t - kernel_ventry 1, 64, irq_invalid // IRQ EL1t - kernel_ventry 1, 64, fiq_invalid // FIQ EL1t - kernel_ventry 1, 64, error_invalid // Error EL1t - - kernel_ventry 1, 64, sync // Synchronous EL1h - kernel_ventry 1, 64, irq // IRQ EL1h - kernel_ventry 1, 64, fiq // FIQ EL1h - kernel_ventry 1, 64, error // Error EL1h - - kernel_ventry 0, 64, sync // Synchronous 64-bit EL0 - kernel_ventry 0, 64, irq // IRQ 64-bit EL0 - kernel_ventry 0, 64, fiq // FIQ 64-bit EL0 - kernel_ventry 0, 64, error // Error 64-bit EL0 - -#ifdef CONFIG_COMPAT - kernel_ventry 0, 32, sync_compat // Synchronous 32-bit EL0 - kernel_ventry 0, 32, irq_compat // IRQ 32-bit EL0 - kernel_ventry 0, 32, fiq_compat // FIQ 32-bit EL0 - kernel_ventry 0, 32, error_compat // Error 32-bit EL0 -#else - kernel_ventry 0, 32, sync_invalid // Synchronous 32-bit EL0 - kernel_ventry 0, 32, irq_invalid // IRQ 32-bit EL0 - kernel_ventry 0, 32, fiq_invalid // FIQ 32-bit EL0 - kernel_ventry 0, 32, error_invalid // Error 32-bit EL0 -#endif + kernel_ventry 1, t, 64, sync // Synchronous EL1t + kernel_ventry 1, t, 64, irq // IRQ EL1t + kernel_ventry 1, t, 64, fiq // FIQ EL1h + kernel_ventry 1, t, 64, error // Error EL1t + + kernel_ventry 1, h, 64, sync // Synchronous EL1h + kernel_ventry 1, h, 64, irq // IRQ EL1h + kernel_ventry 1, h, 64, fiq // FIQ EL1h + kernel_ventry 1, h, 64, error // Error EL1h + + kernel_ventry 0, , 64, sync // Synchronous 64-bit EL0 + kernel_ventry 0, , 64, irq // IRQ 64-bit EL0 + kernel_ventry 0, , 64, fiq // FIQ 64-bit EL0 + kernel_ventry 0, , 64, error // Error 64-bit EL0 + + kernel_ventry 0, , 32, sync // Synchronous 32-bit EL0 + kernel_ventry 0, , 32, irq // IRQ 32-bit EL0 + kernel_ventry 0, , 32, fiq // FIQ 32-bit EL0 + kernel_ventry 0, , 32, error // Error 32-bit EL0 SYM_CODE_END(vectors) #ifdef CONFIG_VMAP_STACK @@ -566,83 +550,43 @@ __bad_stack: ASM_BUG() #endif /* CONFIG_VMAP_STACK */ -/* - * Invalid mode handlers - */ - .macro inv_entry, el, reason, regsize = 64 - kernel_entry \el, \regsize - mov x0, sp - mov x1, #\reason - mrs x2, esr_el1 - bl bad_mode - ASM_BUG() - .endm - -SYM_CODE_START_LOCAL(el0_sync_invalid) - inv_entry 0, BAD_SYNC -SYM_CODE_END(el0_sync_invalid) - -SYM_CODE_START_LOCAL(el0_irq_invalid) - inv_entry 0, BAD_IRQ -SYM_CODE_END(el0_irq_invalid) - -SYM_CODE_START_LOCAL(el0_fiq_invalid) - inv_entry 0, BAD_FIQ -SYM_CODE_END(el0_fiq_invalid) - -SYM_CODE_START_LOCAL(el0_error_invalid) - inv_entry 0, BAD_ERROR -SYM_CODE_END(el0_error_invalid) -SYM_CODE_START_LOCAL(el1_sync_invalid) - inv_entry 1, BAD_SYNC -SYM_CODE_END(el1_sync_invalid) - -SYM_CODE_START_LOCAL(el1_irq_invalid) - inv_entry 1, BAD_IRQ -SYM_CODE_END(el1_irq_invalid) - -SYM_CODE_START_LOCAL(el1_fiq_invalid) - inv_entry 1, BAD_FIQ -SYM_CODE_END(el1_fiq_invalid) - -SYM_CODE_START_LOCAL(el1_error_invalid) - inv_entry 1, BAD_ERROR -SYM_CODE_END(el1_error_invalid) - - .macro entry_handler el:req, regsize:req, label:req + .macro entry_handler el:req, ht, regsize:req, label:req .align 6 -SYM_CODE_START_LOCAL_NOALIGN(el\el\()_\label) +SYM_CODE_START_LOCAL_NOALIGN(el\el\ht\()_\regsize\()_\label) kernel_entry \el, \regsize mov x0, sp - bl el\el\()_\label\()_handler + bl el\el\ht\()_\regsize\()_\label\()_handler .if \el == 0 b ret_to_user .else b ret_to_kernel .endif -SYM_CODE_END(el\el\()_\label) +SYM_CODE_END(el\el\ht\()_\regsize\()_\label) .endm /* * Early exception handlers */ - entry_handler 1, 64, sync - entry_handler 1, 64, irq - entry_handler 1, 64, fiq - entry_handler 1, 64, error - - entry_handler 0, 64, sync - entry_handler 0, 64, irq - entry_handler 0, 64, fiq - entry_handler 0, 64, error - -#ifdef CONFIG_COMPAT - entry_handler 0, 32, sync_compat - entry_handler 0, 32, irq_compat - entry_handler 0, 32, fiq_compat - entry_handler 0, 32, error_compat -#endif + entry_handler 1, t, 64, sync + entry_handler 1, t, 64, irq + entry_handler 1, t, 64, fiq + entry_handler 1, t, 64, error + + entry_handler 1, h, 64, sync + entry_handler 1, h, 64, irq + entry_handler 1, h, 64, fiq + entry_handler 1, h, 64, error + + entry_handler 0, , 64, sync + entry_handler 0, , 64, irq + entry_handler 0, , 64, fiq + entry_handler 0, , 64, error + + entry_handler 0, , 32, sync + entry_handler 0, , 32, irq + entry_handler 0, , 32, fiq + entry_handler 0, , 32, error SYM_CODE_START_LOCAL(ret_to_kernel) kernel_exit 1