From patchwork Thu Mar 18 17:07:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 12148723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4E75C433E0 for ; Thu, 18 Mar 2021 17:09:46 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3E2A464F1F for ; Thu, 18 Mar 2021 17:09:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3E2A464F1F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=VTjW18HGk5id0pQOZ6m2YLg72Clg5PXxPCIKIu0bRbY=; b=b7J9j66jZRa5e4WfPXP2CgaWl vIxdZXXBLEZn1foS0EiQdn/BPWbDbZR1pXSEnwZL1FxkriMTfXtokw+QkkIazG0BxUE+SCnJ3U25J KzAlnS8QDWeU2QX99j40YLTAGKdErDiUqdrGSICamrSQpFQQcAc3OjmzDvOYYs4+IIRmA3sXKS5r2 SiEH9NKGyKl5hTNYnyf+0VjXMdYFBK0RpVE6UfagAVR4uFvAz/9MCNGBGt1EEdKb2fNXSRJBIH7r1 NwMPEVC7YIas2tdolnjsJDh7QNrGUGvoKDzBtSl64+UovwNfor8aJdIqFGxKZGnta6ifJfu/PDrhf +x+2QL0XQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lMw8D-005eop-3C; Thu, 18 Mar 2021 17:08:21 +0000 Received: from mail.kernel.org ([198.145.29.99]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lMw7n-005em5-8N for linux-arm-kernel@lists.infradead.org; Thu, 18 Mar 2021 17:07:57 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id A1DE464F3B; Thu, 18 Mar 2021 17:07:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616087274; bh=JmyKBgbj5+tDKwZoejlEI4HSvnyEkw5LKk+u2Tj61fI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jmsnc2dtZXHjNn5R0O1G1AjTJrfpsJPkrb5dusjSozFLiXMNe7TsYXxLst1MQoHT6 QC5bQ515jAwJT2GwUnUuxP0dZ8BjMoIhbeVdQB0uVFnlgQ5JMHtGkziyktTQjdsyFC qKVTnz3FCrYk3gaMq7MpZtVzPcrM5rfXEalFWxzxPFezuXNNL6yZoyLlMUh/b85Ohr DJpGmg5CQNyaMHjwyJefnELFW93lhbs3S0xm0uDoWyaBR18lg+3519z19LLfIlPFV2 MZZP61G2V4vpjNWyQ8GZMnaIYPUaMmANylYIOhO5kTCOPWJshatJZcy032DMJ+oTZ+ FJZ0WEpJe/Abw== From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: kernel-team@android.com, Will Deacon , Catalin Marinas , Mark Rutland , Marc Zyngier , Vincenzo Frascino , Russell King Subject: [PATCH 4/5] arm64: vdso: Avoid ISB after reading from cntvct_el0 Date: Thu, 18 Mar 2021 17:07:37 +0000 Message-Id: <20210318170738.7756-5-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210318170738.7756-1-will@kernel.org> References: <20210318170738.7756-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210318_170755_664887_7A0A19C9 X-CRM114-Status: GOOD ( 12.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We can avoid the expensive ISB instruction after reading the counter in the vDSO gettime functions by creating a fake address hazard against a dummy stack read, just like we do inside the kernel. Signed-off-by: Will Deacon Reviewed-by: Vincenzo Frascino --- arch/arm64/include/asm/arch_timer.h | 21 --------------------- arch/arm64/include/asm/barrier.h | 19 +++++++++++++++++++ arch/arm64/include/asm/vdso/gettimeofday.h | 6 +----- 3 files changed, 20 insertions(+), 26 deletions(-) diff --git a/arch/arm64/include/asm/arch_timer.h b/arch/arm64/include/asm/arch_timer.h index 9f0ec21d6327..88d20f04c64a 100644 --- a/arch/arm64/include/asm/arch_timer.h +++ b/arch/arm64/include/asm/arch_timer.h @@ -165,25 +165,6 @@ static inline void arch_timer_set_cntkctl(u32 cntkctl) isb(); } -/* - * Ensure that reads of the counter are treated the same as memory reads - * for the purposes of ordering by subsequent memory barriers. - * - * This insanity brought to you by speculative system register reads, - * out-of-order memory accesses, sequence locks and Thomas Gleixner. - * - * http://lists.infradead.org/pipermail/linux-arm-kernel/2019-February/631195.html - */ -#define arch_counter_enforce_ordering(val) do { \ - u64 tmp, _val = (val); \ - \ - asm volatile( \ - " eor %0, %1, %1\n" \ - " add %0, sp, %0\n" \ - " ldr xzr, [%0]" \ - : "=r" (tmp) : "r" (_val)); \ -} while (0) - static __always_inline u64 __arch_counter_get_cntpct_stable(void) { u64 cnt; @@ -224,8 +205,6 @@ static __always_inline u64 __arch_counter_get_cntvct(void) return cnt; } -#undef arch_counter_enforce_ordering - static inline int arch_timer_arch_init(void) { return 0; diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h index c3009b0e5239..37d891af8ea5 100644 --- a/arch/arm64/include/asm/barrier.h +++ b/arch/arm64/include/asm/barrier.h @@ -70,6 +70,25 @@ static inline unsigned long array_index_mask_nospec(unsigned long idx, return mask; } +/* + * Ensure that reads of the counter are treated the same as memory reads + * for the purposes of ordering by subsequent memory barriers. + * + * This insanity brought to you by speculative system register reads, + * out-of-order memory accesses, sequence locks and Thomas Gleixner. + * + * http://lists.infradead.org/pipermail/linux-arm-kernel/2019-February/631195.html + */ +#define arch_counter_enforce_ordering(val) do { \ + u64 tmp, _val = (val); \ + \ + asm volatile( \ + " eor %0, %1, %1\n" \ + " add %0, sp, %0\n" \ + " ldr xzr, [%0]" \ + : "=r" (tmp) : "r" (_val)); \ +} while (0) + #define __smp_mb() dmb(ish) #define __smp_rmb() dmb(ishld) #define __smp_wmb() dmb(ishst) diff --git a/arch/arm64/include/asm/vdso/gettimeofday.h b/arch/arm64/include/asm/vdso/gettimeofday.h index 631ab1281633..4b4c0dac0e14 100644 --- a/arch/arm64/include/asm/vdso/gettimeofday.h +++ b/arch/arm64/include/asm/vdso/gettimeofday.h @@ -83,11 +83,7 @@ static __always_inline u64 __arch_get_hw_counter(s32 clock_mode, */ isb(); asm volatile("mrs %0, cntvct_el0" : "=r" (res) :: "memory"); - /* - * This isb() is required to prevent that the seq lock is - * speculated.# - */ - isb(); + arch_counter_enforce_ordering(res); return res; }