From patchwork Fri Feb 12 06:15:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 12084763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89B7FC433E0 for ; Fri, 12 Feb 2021 06:17:06 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 35B4064E6B for ; Fri, 12 Feb 2021 06:17:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 35B4064E6B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=/Qg/nRMtX4WtkPYr6uXh5aCEzJrBWbXsdlFm0S+rfrM=; b=SYzZEj0fkcpJyQdVx8o8tIEO0 pkHADBUnkH6VHdvq1V/TqVLTFTtLjW+efgN61aJVeh/iUy1hGCObPe5kPU9R41e7754ybLVyh4bOE Trtg13eH6644nGq6CulW+lJF+l1Aptw1GRH7P7C0Om2/JiJHvrWkZYp8QdMYChFBVr5UZMexJtSMY I5sspkBcPvtS5DaQ4MRpkpTUiebcZv/V4TT3Rn1pmhNVt4ppuVAtE29qi5+92/McUhFJGhDAbjnIW kQ33ihHwzab643qQQkYWxwbVYrJDDe7gsdwJawvAzranLFOA8ijroBKgPlbRM0c/E/MVinyoYz5kC Dqzhc1Dkg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1lARkH-0001Bc-DS; Fri, 12 Feb 2021 06:16:01 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1lARk9-000191-SF for linux-arm-kernel@lists.infradead.org; Fri, 12 Feb 2021 06:15:56 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id g17so6114684ybh.4 for ; Thu, 11 Feb 2021 22:15:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=CjUD5J0k0hQzfTB0DmQ6t7Ua1l58gIuKMJB1voH1UbU=; b=p3AjqchzOkyaWN1IfZZMVGXeLJtc53RxH9SAeHabjsFjPKk7UtxvhfNZRC40DVfhrM 65iAYti+Bcb8pdeKB9wemICJraNV1AcEdT86HoNWEdfbtk6PoCDiLo5LtUShSncSRaHB +8AiJREqe265h2E7/5hjnjgtQWetwdj/w+UkwCRZvBKb9j2ckKuCjfgvNObn+TC3vEeA N0q9/agJFmytD6sJ80/w83IIBjBwFISctL3J3jWrJ0u68NTPOT8mY0odyB/hSr4+nWFJ M0tHdsO7DkGeOGk/0mE5U1BR88fRt4PqO6bKx4JPjWyqfO9CwWhjhQnPbJno9neSiUMI PmIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=CjUD5J0k0hQzfTB0DmQ6t7Ua1l58gIuKMJB1voH1UbU=; b=Xd4Eym7oBUoHotZjMR3Cxt6AXhCVTbfGjZ1JvIh2rCHn6vZ6j17XyEd5L0lh28Ght9 KLrmTxF3H/7WKzEMfkE+AXrY86sZdRX5WMhl3LUYzswBPoEiepMNGJU3CjPEbVlDx8pb fdOA5AqSskqMhz6y6FaOhVt6JgEd0UpfCPHzzJo0kHdcftNYOyIJmGvNl2DIOMqmTLSp YTuCgTb0eISUzRxZVvdCjZfA0VUC/uqNyh9mL4vqHg7bFJbO9NdoWp5rAa4p4OhP8prD +KCNqp/lsI93RQwVik6pq8ewtIfUSZkm/Sb08btmPcW9Np6/Di2YdBJ2JftYLWqclOkF oQ2Q== X-Gm-Message-State: AOAM530/dGfIACcZHWAMrZM/jnt+mt3CqfWsiMO7Igmg7MPGSmgPOPld 6oBSj0kPCcFXkWwGcCCmUvu/Mq8= X-Google-Smtp-Source: ABdhPJzNC02oeU6zG8n4aIB0JC7fHASd2qxupXmFwcZlJPtDl7he3J3jTRY7bPlMtIOXHSAA6YUjrYE= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:0:300c:869a:ec89:df82]) (user=pcc job=sendgmr) by 2002:a25:b992:: with SMTP id r18mr2181703ybg.86.1613110550217; Thu, 11 Feb 2021 22:15:50 -0800 (PST) Date: Thu, 11 Feb 2021 22:15:39 -0800 In-Reply-To: <51dbab807a063b05bd9fbf37b9333a1a20cc15fc.1613110413.git.pcc@google.com> Message-Id: Mime-Version: 1.0 References: <51dbab807a063b05bd9fbf37b9333a1a20cc15fc.1613110413.git.pcc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH v7 3/3] arm64: pac: Optimize kernel entry/exit key installation code paths From: Peter Collingbourne To: Catalin Marinas , Evgenii Stepanov , Kostya Serebryany , Vincenzo Frascino , Dave Martin , Szabolcs Nagy , Florian Weimer X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210212_011554_004123_46D254AA X-CRM114-Status: GOOD ( 24.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: libc-alpha@sourceware.org, Peter Collingbourne , Andrey Konovalov , Kevin Brodsky , linux-api@vger.kernel.org, Will Deacon , Linux ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The kernel does not use any keys besides IA so we don't need to install IB/DA/DB/GA on kernel exit if we arrange to install them on task switch instead, which we can expect to happen an order of magnitude less often. Furthermore we can avoid installing the user IA in the case where the user task has IA disabled and just leave the kernel IA installed. This also lets us avoid needing to install IA on kernel entry. On an Apple M1 under a hypervisor, the overhead of kernel entry/exit has been measured to be reduced by 15.6ns in the case where IA is enabled, and 31.9ns in the case where IA is disabled. Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/Ieddf6b580d23c9e0bed45a822dabe72d2ffc9a8e --- arch/arm64/include/asm/asm_pointer_auth.h | 20 +------------ arch/arm64/include/asm/pointer_auth.h | 36 ++++++++++++++++++----- arch/arm64/kernel/asm-offsets.c | 4 --- arch/arm64/kernel/entry.S | 33 ++++++++++++--------- arch/arm64/kernel/pointer_auth.c | 1 + arch/arm64/kernel/process.c | 1 + arch/arm64/kernel/suspend.c | 3 +- 7 files changed, 53 insertions(+), 45 deletions(-) diff --git a/arch/arm64/include/asm/asm_pointer_auth.h b/arch/arm64/include/asm/asm_pointer_auth.h index 52dead2a8640..8ca2dc0661ee 100644 --- a/arch/arm64/include/asm/asm_pointer_auth.h +++ b/arch/arm64/include/asm/asm_pointer_auth.h @@ -13,30 +13,12 @@ * so use the base value of ldp as thread.keys_user and offset as * thread.keys_user.ap*. */ - .macro ptrauth_keys_install_user tsk, tmp1, tmp2, tmp3 + .macro __ptrauth_keys_install_user tsk, tmp1, tmp2, tmp3 mov \tmp1, #THREAD_KEYS_USER add \tmp1, \tsk, \tmp1 -alternative_if_not ARM64_HAS_ADDRESS_AUTH - b .Laddr_auth_skip_\@ -alternative_else_nop_endif ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APIA] msr_s SYS_APIAKEYLO_EL1, \tmp2 msr_s SYS_APIAKEYHI_EL1, \tmp3 - ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APIB] - msr_s SYS_APIBKEYLO_EL1, \tmp2 - msr_s SYS_APIBKEYHI_EL1, \tmp3 - ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APDA] - msr_s SYS_APDAKEYLO_EL1, \tmp2 - msr_s SYS_APDAKEYHI_EL1, \tmp3 - ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APDB] - msr_s SYS_APDBKEYLO_EL1, \tmp2 - msr_s SYS_APDBKEYHI_EL1, \tmp3 -.Laddr_auth_skip_\@: -alternative_if ARM64_HAS_GENERIC_AUTH - ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APGA] - msr_s SYS_APGAKEYLO_EL1, \tmp2 - msr_s SYS_APGAKEYHI_EL1, \tmp3 -alternative_else_nop_endif .endm .macro __ptrauth_keys_install_kernel_nosync tsk, tmp1, tmp2, tmp3 diff --git a/arch/arm64/include/asm/pointer_auth.h b/arch/arm64/include/asm/pointer_auth.h index 1a85e25d98ba..3e40fc776ea3 100644 --- a/arch/arm64/include/asm/pointer_auth.h +++ b/arch/arm64/include/asm/pointer_auth.h @@ -35,6 +35,25 @@ struct ptrauth_keys_kernel { struct ptrauth_key apia; }; +#define __ptrauth_key_install_nosync(k, v) \ +do { \ + struct ptrauth_key __pki_v = (v); \ + write_sysreg_s(__pki_v.lo, SYS_ ## k ## KEYLO_EL1); \ + write_sysreg_s(__pki_v.hi, SYS_ ## k ## KEYHI_EL1); \ +} while (0) + +static inline void ptrauth_keys_install_user(struct ptrauth_keys_user *keys) +{ + if (system_supports_address_auth()) { + __ptrauth_key_install_nosync(APIB, keys->apib); + __ptrauth_key_install_nosync(APDA, keys->apda); + __ptrauth_key_install_nosync(APDB, keys->apdb); + } + + if (system_supports_generic_auth()) + __ptrauth_key_install_nosync(APGA, keys->apga); +} + static inline void ptrauth_keys_init_user(struct ptrauth_keys_user *keys) { if (system_supports_address_auth()) { @@ -46,14 +65,9 @@ static inline void ptrauth_keys_init_user(struct ptrauth_keys_user *keys) if (system_supports_generic_auth()) get_random_bytes(&keys->apga, sizeof(keys->apga)); -} -#define __ptrauth_key_install_nosync(k, v) \ -do { \ - struct ptrauth_key __pki_v = (v); \ - write_sysreg_s(__pki_v.lo, SYS_ ## k ## KEYLO_EL1); \ - write_sysreg_s(__pki_v.hi, SYS_ ## k ## KEYHI_EL1); \ -} while (0) + ptrauth_keys_install_user(keys); +} static __always_inline void ptrauth_keys_init_kernel(struct ptrauth_keys_kernel *keys) { @@ -81,6 +95,9 @@ static inline unsigned long ptrauth_strip_insn_pac(unsigned long ptr) return ptrauth_clear_pac(ptr); } +#define ptrauth_suspend_exit() \ + ptrauth_keys_install_user(¤t->thread.keys_user) + #define ptrauth_thread_init_user() \ do { \ ptrauth_keys_init_user(¤t->thread.keys_user); \ @@ -92,6 +109,9 @@ static inline unsigned long ptrauth_strip_insn_pac(unsigned long ptr) SCTLR_ELx_ENDA | SCTLR_ELx_ENDB); \ } while (0) +#define ptrauth_thread_switch_user(tsk) \ + ptrauth_keys_install_user(&(tsk)->thread.keys_user) + #define ptrauth_thread_init_kernel(tsk) \ ptrauth_keys_init_kernel(&(tsk)->thread.keys_kernel) #define ptrauth_thread_switch_kernel(tsk) \ @@ -102,8 +122,10 @@ static inline unsigned long ptrauth_strip_insn_pac(unsigned long ptr) #define ptrauth_set_enabled_keys(tsk, keys, enabled) (-EINVAL) #define ptrauth_get_enabled_keys(tsk) (-EINVAL) #define ptrauth_strip_insn_pac(lr) (lr) +#define ptrauth_suspend_exit() #define ptrauth_thread_init_user() #define ptrauth_thread_init_kernel(tsk) +#define ptrauth_thread_switch_user(tsk) #define ptrauth_thread_switch_kernel(tsk) #endif /* CONFIG_ARM64_PTR_AUTH */ diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 30c7694f3ec2..2c660e677728 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -145,10 +145,6 @@ int main(void) #endif #ifdef CONFIG_ARM64_PTR_AUTH DEFINE(PTRAUTH_USER_KEY_APIA, offsetof(struct ptrauth_keys_user, apia)); - DEFINE(PTRAUTH_USER_KEY_APIB, offsetof(struct ptrauth_keys_user, apib)); - DEFINE(PTRAUTH_USER_KEY_APDA, offsetof(struct ptrauth_keys_user, apda)); - DEFINE(PTRAUTH_USER_KEY_APDB, offsetof(struct ptrauth_keys_user, apdb)); - DEFINE(PTRAUTH_USER_KEY_APGA, offsetof(struct ptrauth_keys_user, apga)); DEFINE(PTRAUTH_KERNEL_KEY_APIA, offsetof(struct ptrauth_keys_kernel, apia)); BLANK(); #endif diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 02625e75d35c..22711d60a323 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -247,21 +247,26 @@ alternative_else_nop_endif check_mte_async_tcf x19, x22 apply_ssbd 1, x22, x23 - ptrauth_keys_install_kernel_nosync tsk, x20, x22, x23 - #ifdef CONFIG_ARM64_PTR_AUTH alternative_if ARM64_HAS_ADDRESS_AUTH /* * Enable IA for in-kernel PAC if the task had it disabled. Although * this could be implemented with an unconditional MRS which would avoid * a load, this was measured to be slower on Cortex-A75 and Cortex-A76. + * + * Install the kernel IA key only if IA was enabled in the task. If IA + * was disabled on kernel exit then we would have left the kernel IA + * installed so there is no need to install it again. */ ldr x0, [tsk, THREAD_SCTLR_USER] - tbnz x0, SCTLR_ELx_ENIA_SHIFT, 1f + tbz x0, SCTLR_ELx_ENIA_SHIFT, 1f + __ptrauth_keys_install_kernel_nosync tsk, x20, x22, x23 + b 2f +1: mrs x0, sctlr_el1 orr x0, x0, SCTLR_ELx_ENIA msr sctlr_el1, x0 -1: +2: isb alternative_else_nop_endif #endif @@ -368,24 +373,24 @@ alternative_else_nop_endif 3: scs_save tsk, x0 - /* - * No kernel C function calls after this as user keys are set and IA may - * be disabled. - */ - ptrauth_keys_install_user tsk, x0, x1, x2 - #ifdef CONFIG_ARM64_PTR_AUTH alternative_if ARM64_HAS_ADDRESS_AUTH /* - * IA was enabled for in-kernel PAC. Disable it now if needed. - * All other per-task SCTLR bits were updated on task switch. + * IA was enabled for in-kernel PAC. Disable it now if needed, or + * alternatively install the user's IA. All other per-task keys and + * SCTLR bits were updated on task switch. + * + * No kernel C function calls after this. */ ldr x0, [tsk, THREAD_SCTLR_USER] - tbnz x0, SCTLR_ELx_ENIA_SHIFT, 1f + tbz x0, SCTLR_ELx_ENIA_SHIFT, 1f + __ptrauth_keys_install_user tsk, x0, x1, x2 + b 2f +1: mrs x0, sctlr_el1 bic x0, x0, SCTLR_ELx_ENIA msr sctlr_el1, x0 -1: +2: alternative_else_nop_endif #endif diff --git a/arch/arm64/kernel/pointer_auth.c b/arch/arm64/kernel/pointer_auth.c index f03e5bfe4490..60901ab0a7fe 100644 --- a/arch/arm64/kernel/pointer_auth.c +++ b/arch/arm64/kernel/pointer_auth.c @@ -43,6 +43,7 @@ int ptrauth_prctl_reset_keys(struct task_struct *tsk, unsigned long arg) get_random_bytes(&keys->apdb, sizeof(keys->apdb)); if (arg & PR_PAC_APGAKEY) get_random_bytes(&keys->apga, sizeof(keys->apga)); + ptrauth_keys_install_user(keys); return 0; } diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 34cb5cc6ca6d..532487656f8e 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -568,6 +568,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev, entry_task_switch(next); ssbs_thread_switch(next); erratum_1418040_thread_switch(prev, next); + ptrauth_thread_switch_user(next); /* * Complete any pending TLB or cache maintenance on this CPU in case diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c index a67b37a7a47e..c5a6614a41f9 100644 --- a/arch/arm64/kernel/suspend.c +++ b/arch/arm64/kernel/suspend.c @@ -74,8 +74,9 @@ void notrace __cpu_suspend_exit(void) */ spectre_v4_enable_mitigation(NULL); - /* Restore additional MTE-specific configuration */ + /* Restore additional feature-specific configuration */ mte_suspend_exit(); + ptrauth_suspend_exit(); } /*