From patchwork Fri Jul 9 02:35:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 12366557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A808CC07E96 for ; Fri, 9 Jul 2021 02:37:19 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6BA206145E for ; Fri, 9 Jul 2021 02:37:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6BA206145E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version: Message-Id:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=OpZ1me1PbVXUf8AgO+q5YXSQq62e9bzugs0Hbc/ZPoI=; b=ZkT lORAoBkEFwn+TW0ptKoCa+Y4pFH0UOG9OB1LzOVKOYoqdmCah56EFlgxSxJ7fuIkxRb6PTINev8Os y1Wjke7gI4qtKVdPTWaTdwCEQQWLjlGIAUnKL7I03PCenihua9esKhl5iPGBlaU+GJZqrYV41pURY d0mgjyH93y24Jlnp1dAVT3JUd9eT57woWPu+r3i6Ar/I5A9JIeBhiPhMUfO+zaBhUAKWNLCl4n6N4 PXSg+O9Kz2PBUF1Up2Pe3JEve5cS3d4X7EZxZ8bZWTdoiUkOKrWYcfLshfXL8wGBj89t7gHIV9SNc AFLDCPRp4Nqt2Gw0IVW+rBLIs8Jtb2A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m1gMj-000dRX-Qv; Fri, 09 Jul 2021 02:35:45 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m1gMg-000dR1-AP for linux-arm-kernel@lists.infradead.org; Fri, 09 Jul 2021 02:35:44 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id q10-20020a056902150ab02905592911c932so9683402ybu.15 for ; Thu, 08 Jul 2021 19:35:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=7spP/IGcUC97w7lxujnj4yiCIAPcnzqhrq7xgk+8thg=; b=Zv/pKacEBu3HNB9urLznWVYm+zg30SBLoG4TVr1r7oqEKR674ml1golhbwz71ucLE6 yXlaD0sAN7NNaMQIspSsZyGutF0m/AvTOYoFYHafr7yFeP3XNiBsXEjkaH/XgXQdRhS5 vttFRIMfGT6AGGQwS3U0DUYANfRxNgBAQZE4ih+a8ejZM91R6NI3yWRSkBikCNgTRQb2 5wlkSaBCjI2UAHk3b7T8UpGFvXWoNXYcj0Nl96SqhZechp1K51DrSKhHBXUIVgrxZwmR RfrN3sfbrF5JXtOWUfEWeO1p6Er7TpnmPCwjuckfr63uqzRGeYyI08O9VR4ChRNp63x1 cNMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=7spP/IGcUC97w7lxujnj4yiCIAPcnzqhrq7xgk+8thg=; b=IlIjW+SPQKUtTmh5NcDfG1WZHF7B26rmMkPZmnWC43H8lqjL85A0B2RPHIxrtKzy5l L0flbl/ZIS8zzuKMab2h2XULL/vWFLGgxwSc/WwZe2XINpAJl9dws9M3FPZ2AeNjr7qV VaiS+Lw/m7E6PO8FVkNYFoIdZueh3FMxGxSntbBOm2bwXnJhacnCfyq2vQYsiXfCsJ3G Unxwfz/1Pyanwq5LeHWtrCFDDsnWV8cm+GKDtPG4w6Xq7sCfdZiWprSwdCyG03e9S8Rl iqPF6FDu7FOwdcILiEpIhZvSQGWYRvqqTFhJfWM23cV/ShgFOzN00dEYVlQTEyH6RQ/L 5EkA== X-Gm-Message-State: AOAM5324dLS2U8RJ6rkQgA3SXZiA0D+ENpnevwZ1zHyjMJXCErqptRCc nL9gV8l3JyHbARAAPK9BKGH9wSo= X-Google-Smtp-Source: ABdhPJypMyB9D1NvcxXJi8vuoWiyHpHxg+bTlAid66/deCqW1WNo2gWlEtr7qQf2/G/kvD+Qo0E6mV4= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:200:4cff:5ae6:4a68:ed58]) (user=pcc job=sendgmr) by 2002:a25:9343:: with SMTP id g3mr43938311ybo.154.1625798140392; Thu, 08 Jul 2021 19:35:40 -0700 (PDT) Date: Thu, 8 Jul 2021 19:35:32 -0700 Message-Id: <20210709023532.2133673-1-pcc@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.32.0.93.g670b81a890-goog Subject: [PATCH v2] arm64: mte: avoid TFSR related operations unless in async mode From: Peter Collingbourne To: Catalin Marinas , Vincenzo Frascino , Will Deacon Cc: Peter Collingbourne , Evgenii Stepanov , linux-arm-kernel@lists.infradead.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210708_193542_422367_959B0646 X-CRM114-Status: GOOD ( 15.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org There is no reason to touch TFSR nor issue a DSB unless our task is in asynchronous mode. Since these operations (especially the DSB) may be expensive on certain microarchitectures, only perform them if necessary. Furthermore, stop clearing TFSR on entry because it will be cleared on exit and it is not necessary to have any particular value in TFSR between entry and exit. Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/Ib353a63e3d0abc2b0b008e96aa2d9692cfc1b815 --- v2: - avoid clearing TFSR on entry arch/arm64/kernel/entry.S | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 863d44f73028..5cf160135411 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -133,29 +133,37 @@ alternative_cb_end .endm /* Check for MTE asynchronous tag check faults */ - .macro check_mte_async_tcf, tmp, ti_flags + .macro check_mte_async_tcf, tmp, ti_flags, thread_sctlr #ifdef CONFIG_ARM64_MTE .arch_extension lse alternative_if_not ARM64_MTE b 1f alternative_else_nop_endif + /* + * Asynchronous tag check faults are only possible in ASYNC (2) or + * ASYM (3) modes. In each of these modes bit 1 of SCTLR_EL1.TCF0 is + * set, so skip the check if it is unset. + */ + tbz \thread_sctlr, #(SCTLR_EL1_TCF0_SHIFT + 1), 1f mrs_s \tmp, SYS_TFSRE0_EL1 tbz \tmp, #SYS_TFSR_EL1_TF0_SHIFT, 1f /* Asynchronous TCF occurred for TTBR0 access, set the TI flag */ mov \tmp, #_TIF_MTE_ASYNC_FAULT add \ti_flags, tsk, #TSK_TI_FLAGS stset \tmp, [\ti_flags] - msr_s SYS_TFSRE0_EL1, xzr 1: #endif .endm /* Clear the MTE asynchronous tag check faults */ - .macro clear_mte_async_tcf + .macro clear_mte_async_tcf thread_sctlr #ifdef CONFIG_ARM64_MTE alternative_if ARM64_MTE + /* See comment in check_mte_async_tcf above. */ + tbz \thread_sctlr, #(SCTLR_EL1_TCF0_SHIFT + 1), 1f dsb ish msr_s SYS_TFSRE0_EL1, xzr +1: alternative_else_nop_endif #endif .endm @@ -231,8 +239,8 @@ alternative_else_nop_endif disable_step_tsk x19, x20 /* Check for asynchronous tag check faults in user space */ - check_mte_async_tcf x22, x23 - apply_ssbd 1, x22, x23 + ldr x0, [tsk, THREAD_SCTLR_USER] + check_mte_async_tcf x22, x23, x0 #ifdef CONFIG_ARM64_PTR_AUTH alternative_if ARM64_HAS_ADDRESS_AUTH @@ -245,7 +253,6 @@ alternative_if ARM64_HAS_ADDRESS_AUTH * was disabled on kernel exit then we would have left the kernel IA * installed so there is no need to install it again. */ - ldr x0, [tsk, THREAD_SCTLR_USER] tbz x0, SCTLR_ELx_ENIA_SHIFT, 1f __ptrauth_keys_install_kernel_nosync tsk, x20, x22, x23 b 2f @@ -258,6 +265,8 @@ alternative_if ARM64_HAS_ADDRESS_AUTH alternative_else_nop_endif #endif + apply_ssbd 1, x22, x23 + mte_set_kernel_gcr x22, x23 scs_load tsk @@ -362,6 +371,10 @@ alternative_else_nop_endif 3: scs_save tsk + /* Ignore asynchronous tag check faults in the uaccess routines */ + ldr x0, [tsk, THREAD_SCTLR_USER] + clear_mte_async_tcf x0 + #ifdef CONFIG_ARM64_PTR_AUTH alternative_if ARM64_HAS_ADDRESS_AUTH /* @@ -371,7 +384,6 @@ alternative_if ARM64_HAS_ADDRESS_AUTH * * No kernel C function calls after this. */ - ldr x0, [tsk, THREAD_SCTLR_USER] tbz x0, SCTLR_ELx_ENIA_SHIFT, 1f __ptrauth_keys_install_user tsk, x0, x1, x2 b 2f @@ -599,8 +611,6 @@ SYM_CODE_START_LOCAL(ret_to_user) cbnz x2, work_pending finish_ret_to_user: user_enter_irqoff - /* Ignore asynchronous tag check faults in the uaccess routines */ - clear_mte_async_tcf enable_step_tsk x19, x2 #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase