From patchwork Sat Feb 19 01:29:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 12752127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 268E5C433F5 for ; Sat, 19 Feb 2022 01:30:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A7BBC6B0075; Fri, 18 Feb 2022 20:30:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DA5B6B0078; Fri, 18 Feb 2022 20:30:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A5086B007B; Fri, 18 Feb 2022 20:30:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0103.hostedemail.com [216.40.44.103]) by kanga.kvack.org (Postfix) with ESMTP id 78B126B0075 for ; Fri, 18 Feb 2022 20:30:06 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3DFF69F84A for ; Sat, 19 Feb 2022 01:30:06 +0000 (UTC) X-FDA: 79157798412.20.98C0B32 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf02.hostedemail.com (Postfix) with ESMTP id AD9EB80003 for ; Sat, 19 Feb 2022 01:30:05 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-2d6d36ec646so26418997b3.23 for ; Fri, 18 Feb 2022 17:30:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=HAlYyn9KnmbmoD8rPWV1wNqJvLzz5Tpsb0HkUX4HIHw=; b=tb5C0jiJAUc+t0dRPvAiENwb0uKF7ukIlY7RFbbn/6odLTMsWDFKSufoHsC999Rade W78aZyoguLRulqKo74pOISX9hB3N19AHOi2nRoVkqaY/CCuSkal/cr1q981EJna+PUjL Da+lJWVkUQS/H0rYRq4CSE5PoEoxADt1LDLVqG3ik9DGc3ydUFLGGn/KLB7R6rIrd22c +Xiw403EeX2qPxcUJ2xm7XcKGybdMsZJD85wV2hST0NT4Cjhiu67ZNwDC9SgmvdBKTY7 gcGM2y76UC3F3HwWOdkyafrW0vyh8fNtG2icBf7cQMequz8kXq2gCumBbQkqF3QiX5cy W1tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=HAlYyn9KnmbmoD8rPWV1wNqJvLzz5Tpsb0HkUX4HIHw=; b=0lVx7P7oS1G4NVt0w5o1W+Xci6ZwPMrLv/6OtQ7Nkz8VxYGqONBfB1qQ2i5PBHdCEK 50frRSAkkEXKQiR6RW7hxWdjbplTelzDW+/Y0DTgkirfoqYUw4vbFwSoaEjSUW6Tgqf5 1g2Bc9A8ApTuV0yQev91uI2j4YprKwxjsM5Vtpn0LBtT7lOn38BVHG21KLOYCddLDSHv DnrbrkcQZoak4RPS40TKY6yD5WBvIpy7JYETzRbWLE9kotiO7W2n+ghg4Cs9L8kxbWms fi+kaPGa4uL9DIbJ+0faOs2TMszYgqGp26mZ5F0enrWa73ZfRA49Uy9yaXbPR9s5wtvN m1rg== X-Gm-Message-State: AOAM532Dohkf7Efx9B1apLgT2EZUNOZSSnI57LPuwBhOW9w5SaWWTsBi 0vl8UpbrVFEDaiwwi+6AzNXwL0k= X-Google-Smtp-Source: ABdhPJyXqqPv1gHzxgxRtD2X8PQLKtXIr+c5HMVFUnLMzRhNSPgnVpsajrw568Zv0/+/eyjZOTU4ppI= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:200:4926:4660:7cdf:2d]) (user=pcc job=sendgmr) by 2002:a25:e90b:0:b0:61d:9552:6f1a with SMTP id n11-20020a25e90b000000b0061d95526f1amr9592310ybd.400.1645234204963; Fri, 18 Feb 2022 17:30:04 -0800 (PST) Date: Fri, 18 Feb 2022 17:29:45 -0800 In-Reply-To: <20220219012945.894950-1-pcc@google.com> Message-Id: <20220219012945.894950-2-pcc@google.com> Mime-Version: 1.0 References: <20220219012945.894950-1-pcc@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v5 2/2] arm64: mte: avoid clearing PSTATE.TCO on entry unless necessary From: Peter Collingbourne To: Catalin Marinas , Vincenzo Frascino , Will Deacon , Andrey Konovalov , Mark Rutland Cc: Peter Collingbourne , Evgenii Stepanov , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AD9EB80003 X-Stat-Signature: tz3rbpqwzmntt3b6ck8phyjtm5tf6yay Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=tb5C0jiJ; spf=pass (imf02.hostedemail.com: domain of 3HEgQYgMKCAUuhhlttlqj.htrqnsz2-rrp0fhp.twl@flex--pcc.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3HEgQYgMKCAUuhhlttlqj.htrqnsz2-rrp0fhp.twl@flex--pcc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-HE-Tag: 1645234205-697238 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On some microarchitectures, clearing PSTATE.TCO is expensive. Clearing TCO is only necessary if in-kernel MTE is enabled, or if MTE is enabled in the userspace process in synchronous (or, soon, asymmetric) mode, because we do not report uaccess faults to userspace in none or asynchronous modes. Therefore, adjust the kernel entry code to clear TCO only if necessary. Because it is now possible to switch to a task in which TCO needs to be clear from a task in which TCO is set, we also need to do the same thing on task switch. Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/I52d82a580bd0500d420be501af2c35fa8c90729e Reviewed-by: Catalin Marinas --- v5: - fix header circular dependency with KASAN_GENERIC/KASAN_SW_TAGS enabled with a dependent patch v4: - some changes suggested by Catalin v3: - switch to a C implementation v2: - do the same thing in cpu_switch_to() arch/arm64/include/asm/mte.h | 22 ++++++++++++++++++++++ arch/arm64/kernel/entry-common.c | 3 +++ arch/arm64/kernel/entry.S | 7 ------- arch/arm64/kernel/mte.c | 3 +++ 4 files changed, 28 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 075539f5f1c8..adcb937342f1 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -11,7 +11,9 @@ #ifndef __ASSEMBLY__ #include +#include #include +#include #include #include @@ -86,6 +88,26 @@ static inline int mte_ptrace_copy_tags(struct task_struct *child, #endif /* CONFIG_ARM64_MTE */ +static inline void mte_disable_tco_entry(struct task_struct *task) +{ + if (!system_supports_mte()) + return; + + /* + * Re-enable tag checking (TCO set on exception entry). This is only + * necessary if MTE is enabled in either the kernel or the userspace + * task in synchronous or asymmetric mode (SCTLR_EL1.TCF0 bit 0 is set + * for both). With MTE disabled in the kernel and disabled or + * asynchronous in userspace, tag check faults (including in uaccesses) + * are not reported, therefore there is no need to re-enable checking. + * This is beneficial on microarchitectures where re-enabling TCO is + * expensive. + */ + if (kasan_hw_tags_enabled() || + (task->thread.sctlr_user & (1UL << SCTLR_EL1_TCF0_SHIFT))) + asm volatile(SET_PSTATE_TCO(0)); +} + #ifdef CONFIG_KASAN_HW_TAGS /* Whether the MTE asynchronous mode is enabled. */ DECLARE_STATIC_KEY_FALSE(mte_async_or_asymm_mode); diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index ef7fcefb96bd..7093b578e325 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -56,6 +57,7 @@ static void noinstr enter_from_kernel_mode(struct pt_regs *regs) { __enter_from_kernel_mode(regs); mte_check_tfsr_entry(); + mte_disable_tco_entry(current); } /* @@ -103,6 +105,7 @@ static __always_inline void __enter_from_user_mode(void) CT_WARN_ON(ct_state() != CONTEXT_USER); user_exit_irqoff(); trace_hardirqs_off_finish(); + mte_disable_tco_entry(current); } static __always_inline void enter_from_user_mode(struct pt_regs *regs) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 772ec2ecf488..e1013a83d4f0 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -308,13 +308,6 @@ alternative_if ARM64_HAS_IRQ_PRIO_MASKING msr_s SYS_ICC_PMR_EL1, x20 alternative_else_nop_endif - /* Re-enable tag checking (TCO set on exception entry) */ -#ifdef CONFIG_ARM64_MTE -alternative_if ARM64_MTE - SET_PSTATE_TCO(0) -alternative_else_nop_endif -#endif - /* * Registers that may be useful after this macro is invoked: * diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index f418ebc65f95..f983795b5eda 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -253,6 +253,9 @@ void mte_thread_switch(struct task_struct *next) mte_update_sctlr_user(next); mte_update_gcr_excl(next); + /* TCO may not have been disabled on exception entry for the current task. */ + mte_disable_tco_entry(next); + /* * Check if an async tag exception occurred at EL1. *