From patchwork Wed Dec 18 18:26:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ionela Voinescu X-Patchwork-Id: 11301605 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A9D7B109A for ; Wed, 18 Dec 2019 18:27:06 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 80EFA20716 for ; Wed, 18 Dec 2019 18:27:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="rHDjHso1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 80EFA20716 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=eq/ZWJrKHNrs+gMtXpNdD7FIuAWypJhniqmvsIAcsYM=; b=rHDjHso1b2PU3ddssIUip5VLRd c+6MC2mwxCSlyy89Xw1zn/q6xm+JbXUwQ4jvYRmFi2KsokoxLknxomb/wJVb2lN4+/u8aZT1TE2Lu fzXz1vmibDsCYndlYPi2fS5UGGnKLc81tQxX5Ci6KD4M6FYpBM1PQupAs/KaV5ABE8hjqI3KHO/yp 520+fH+/LdH7UEY7V8FSYakW7izSN8WvDISpt1i08moCKVp4zMeqFm9iQKmPmMRORifzTdQaZFDYG w1xtQ/99rrHtc/vU0kuEbLL8YqzgXCpeSh49fV/HuLAtJqxp7Sbj3OJ9wTLjxzsIRbNn0DjNnk71x ZRHOyctA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe2L-0007eo-AV; Wed, 18 Dec 2019 18:27:05 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe2H-0007dz-On for linux-arm-kernel@lists.infradead.org; Wed, 18 Dec 2019 18:27:03 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C64801FB; Wed, 18 Dec 2019 10:27:00 -0800 (PST) Received: from e108754-lin.cambridge.arm.com (unknown [10.1.198.81]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E695A3F67D; Wed, 18 Dec 2019 10:26:58 -0800 (PST) From: Ionela Voinescu To: catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, maz@kernel.org, suzuki.poulose@arm.com, sudeep.holla@arm.com, dietmar.eggemann@arm.com, ionela.voinescu@arm.com Subject: [PATCH v2 1/6] arm64: add support for the AMU extension v1 Date: Wed, 18 Dec 2019 18:26:02 +0000 Message-Id: <20191218182607.21607-2-ionela.voinescu@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191218182607.21607-1-ionela.voinescu@arm.com> References: <20191218182607.21607-1-ionela.voinescu@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191218_102701_892377_8E25F044 X-CRM114-Status: GOOD ( 23.10 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-doc@vger.kernel.org, peterz@infradead.org, linux-kernel@vger.kernel.org, mingo@redhat.com, ggherdovich@suse.cz, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The activity monitors extension is an optional extension introduced by the ARMv8.4 CPU architecture. This implements basic support for version 1 of the activity monitors architecture, AMUv1. This support includes: - Extension detection on each CPU (boot, secondary, hotplugged) - Register interface for AMU aarch64 registers - (while here) create defines for ID_PFR0_EL1 fields when adding the AMU field information. Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc: Will Deacon Cc: Suzuki K Poulose Cc: Marc Zyngier Cc: Mark Rutland --- arch/arm64/Kconfig | 27 ++++++++++ arch/arm64/include/asm/cpucaps.h | 3 +- arch/arm64/include/asm/cpufeature.h | 4 ++ arch/arm64/include/asm/sysreg.h | 44 ++++++++++++++++ arch/arm64/kernel/cpufeature.c | 81 +++++++++++++++++++++++++++-- 5 files changed, 154 insertions(+), 5 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index ac31ed6184d0..6ae7bfa5812e 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1485,6 +1485,33 @@ config ARM64_PTR_AUTH endmenu +menu "ARMv8.4 architectural features" + +config ARM64_AMU_EXTN + bool "Enable support for the Activity Monitors Unit CPU extension" + default y + help + The activity monitors extension is an optional extension introduced + by the ARMv8.4 CPU architecture. This enables support for version 1 + of the activity monitors architecture, AMUv1. + + To enable the use of this extension on CPUs that implement it, say Y. + + Note that for architectural reasons, firmware _must_ implement AMU + support when running on CPUs that present the activity monitors + extension. The required support is present in: + * Version 1.5 and later of the ARM Trusted Firmware + + For kernels that have this configuration enabled but boot with broken + firmware, you may need to say N here until the firmware is fixed. + Otherwise you may experience firmware panics or lockups when + accessing the counter registers. Even if you are not observing these + symptoms, the values returned by the register reads might not + correctly reflect reality. Most commonly, the value read will be 0, + indicating that the counter is not enabled. + +endmenu + config ARM64_SVE bool "ARM Scalable Vector Extension support" default y diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index b92683871119..7dde890bde50 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -56,7 +56,8 @@ #define ARM64_WORKAROUND_CAVIUM_TX2_219_PRFM 46 #define ARM64_WORKAROUND_1542419 47 #define ARM64_WORKAROUND_1319367 48 +#define ARM64_HAS_AMU_EXTN 49 -#define ARM64_NCAPS 49 +#define ARM64_NCAPS 50 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 4261d55e8506..b89e799d6972 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -673,6 +673,10 @@ static inline bool cpu_has_hw_af(void) ID_AA64MMFR1_HADBS_SHIFT); } +#ifdef CONFIG_ARM64_AMU_EXTN +extern inline bool cpu_has_amu_feat(void); +#endif + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 6e919fafb43d..bfcc87953a68 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -382,6 +382,42 @@ #define SYS_TPIDR_EL0 sys_reg(3, 3, 13, 0, 2) #define SYS_TPIDRRO_EL0 sys_reg(3, 3, 13, 0, 3) +/* Definitions for system register interface to AMU for ARMv8.4 onwards */ +#define SYS_AM_EL0(crm, op2) sys_reg(3, 3, 13, crm, op2) +#define SYS_AMCR_EL0 SYS_AM_EL0(2, 0) +#define SYS_AMCFGR_EL0 SYS_AM_EL0(2, 1) +#define SYS_AMCGCR_EL0 SYS_AM_EL0(2, 2) +#define SYS_AMUSERENR_EL0 SYS_AM_EL0(2, 3) +#define SYS_AMCNTENCLR0_EL0 SYS_AM_EL0(2, 4) +#define SYS_AMCNTENSET0_EL0 SYS_AM_EL0(2, 5) +#define SYS_AMCNTENCLR1_EL0 SYS_AM_EL0(3, 0) +#define SYS_AMCNTENSET1_EL0 SYS_AM_EL0(3, 1) + +/* + * Group 0 of activity monitors (architected): + * op0 CRn op1 op2 CRm + * Counter: 11 1101 011 n<2:0> 010:n<3> + * Type: 11 1101 011 n<2:0> 011:n<3> + * n: 0-3 + * + * Group 1 of activity monitors (auxiliary): + * op0 CRn op1 op2 CRm + * Counter: 11 1101 011 n<2:0> 110:n<3> + * Type: 11 1101 011 n<2:0> 111:n<3> + * n: 0-15 + */ + +#define SYS_AMEVCNTR0_EL0(n) SYS_AM_EL0(4 + ((n) >> 3), (n) & 0x7) +#define SYS_AMEVTYPE0_EL0(n) SYS_AM_EL0(6 + ((n) >> 3), (n) & 0x7) +#define SYS_AMEVCNTR1_EL0(n) SYS_AM_EL0(12 + ((n) >> 3), (n) & 0x7) +#define SYS_AMEVTYPE1_EL0(n) SYS_AM_EL0(14 + ((n) >> 3), (n) & 0x7) + +/* V1: Fixed (architecturally defined) activity monitors */ +#define SYS_AMEVCNTR0_CORE_EL0 SYS_AMEVCNTR0_EL0(0) +#define SYS_AMEVCNTR0_CONST_EL0 SYS_AMEVCNTR0_EL0(1) +#define SYS_AMEVCNTR0_INST_RET_EL0 SYS_AMEVCNTR0_EL0(2) +#define SYS_AMEVCNTR0_MEM_STALL SYS_AMEVCNTR0_EL0(3) + #define SYS_CNTFRQ_EL0 sys_reg(3, 3, 14, 0, 0) #define SYS_CNTP_TVAL_EL0 sys_reg(3, 3, 14, 2, 0) @@ -577,6 +613,7 @@ #define ID_AA64PFR0_CSV3_SHIFT 60 #define ID_AA64PFR0_CSV2_SHIFT 56 #define ID_AA64PFR0_DIT_SHIFT 48 +#define ID_AA64PFR0_AMU_SHIFT 44 #define ID_AA64PFR0_SVE_SHIFT 32 #define ID_AA64PFR0_RAS_SHIFT 28 #define ID_AA64PFR0_GIC_SHIFT 24 @@ -587,6 +624,7 @@ #define ID_AA64PFR0_EL1_SHIFT 4 #define ID_AA64PFR0_EL0_SHIFT 0 +#define ID_AA64PFR0_AMU 0x1 #define ID_AA64PFR0_SVE 0x1 #define ID_AA64PFR0_RAS_V1 0x1 #define ID_AA64PFR0_FP_NI 0xf @@ -709,6 +747,12 @@ #define ID_AA64MMFR0_TGRAN16_NI 0x0 #define ID_AA64MMFR0_TGRAN16_SUPPORTED 0x1 +#define ID_PFR0_AMU_SHIFT 20 +#define ID_PFR0_STATE3_SHIFT 12 +#define ID_PFR0_STATE2_SHIFT 8 +#define ID_PFR0_STATE1_SHIFT 4 +#define ID_PFR0_STATE0_SHIFT 0 + #if defined(CONFIG_ARM64_4K_PAGES) #define ID_AA64MMFR0_TGRAN_SHIFT ID_AA64MMFR0_TGRAN4_SHIFT #define ID_AA64MMFR0_TGRAN_SUPPORTED ID_AA64MMFR0_TGRAN4_SUPPORTED diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 04cf64e9f0c9..c639b3e052d7 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -156,6 +156,7 @@ static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = { ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV3_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV2_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_DIT_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_AMU_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SVE), FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_SVE_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_RAS_SHIFT, 4, 0), @@ -314,10 +315,11 @@ static const struct arm64_ftr_bits ftr_id_mmfr4[] = { }; static const struct arm64_ftr_bits ftr_id_pfr0[] = { - ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, 12, 4, 0), /* State3 */ - ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, 8, 4, 0), /* State2 */ - ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, 4, 4, 0), /* State1 */ - ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, 0, 4, 0), /* State0 */ + ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_PFR0_AMU_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_PFR0_STATE3_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_PFR0_STATE2_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_PFR0_STATE1_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_PFR0_STATE0_SHIFT, 4, 0), ARM64_FTR_END, }; @@ -1150,6 +1152,59 @@ static bool has_hw_dbm(const struct arm64_cpu_capabilities *cap, #endif +#ifdef CONFIG_ARM64_AMU_EXTN + +/* + * This per cpu variable only signals that the CPU implementation supports + * the Activity Monitors Unit (AMU) but does not provide information + * regarding all the events that it supports. + * When this amu_feat per CPU variable is true, the user of this feature + * can only rely on the presence of the 4 fixed counters. But this does + * not guarantee that the counters are enabled or access to these counters + * is provided by code executed at higher exception levels. + * + * Also, to ensure the safe use of this per_cpu variable, the following + * accessor is defined to allow a read of amu_feat for the current cpu only + * from the current cpu. + * - cpu_has_amu_feat() + */ +static DEFINE_PER_CPU_READ_MOSTLY(u8, amu_feat); + +inline bool cpu_has_amu_feat(void) +{ + return !!this_cpu_read(amu_feat); +} + +static void cpu_amu_enable(struct arm64_cpu_capabilities const *cap) +{ + if (has_cpuid_feature(cap, SCOPE_LOCAL_CPU)) { + pr_info("detected CPU%d: Activity Monitors Unit (AMU)\n", + smp_processor_id()); + this_cpu_write(amu_feat, 1); + } +} + +static bool has_amu(const struct arm64_cpu_capabilities *cap, + int __unused) +{ + /* + * The AMU extension is a non-conflicting feature: the kernel can + * safely run a mix of CPUs with and without support for the + * activity monitors extension. + * Therefore, unconditionally enable the capability to allow + * any late CPU to use the feature. + * + * With this feature unconditionally enabled, the cpu_enable + * function will be called for all CPUs that match the criteria, + * including secondary and hotplugged, marking this feature as + * present on that respective CPU. The enable function will also + * print a detection message. + */ + + return true; +} +#endif + #ifdef CONFIG_ARM64_VHE static bool runs_at_el2(const struct arm64_cpu_capabilities *entry, int __unused) { @@ -1419,6 +1474,24 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .cpu_enable = cpu_clear_disr, }, #endif /* CONFIG_ARM64_RAS_EXTN */ +#ifdef CONFIG_ARM64_AMU_EXTN + { + /* + * The feature is enabled by default if CONFIG_ARM64_AMU_EXTN=y. + * Therefore, don't provide .desc as we don't want the detection + * message to be shown until at least one CPU is detected to + * support the feature. + */ + .capability = ARM64_HAS_AMU_EXTN, + .type = ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE, + .matches = has_amu, + .sys_reg = SYS_ID_AA64PFR0_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64PFR0_AMU_SHIFT, + .min_field_value = ID_AA64PFR0_AMU, + .cpu_enable = cpu_amu_enable, + }, +#endif /* CONFIG_ARM64_AMU_EXTN */ { .desc = "Data cache clean to the PoU not required for I/D coherence", .capability = ARM64_HAS_CACHE_IDC, From patchwork Wed Dec 18 18:26:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ionela Voinescu X-Patchwork-Id: 11301607 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A00ED17EF for ; Wed, 18 Dec 2019 18:27:15 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 793E020716 for ; Wed, 18 Dec 2019 18:27:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="SnCaTCrW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 793E020716 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ewue3clJkDsEnppWaFCW6Z/rp+pL4tyy3iVvV62mqmY=; b=SnCaTCrW9GI2h8PFsQehb9qE8S tjD0b8ZuQxy6onGu/EBadOBZymZcYUBzZnW81hTf3vLyXUTX3SPmrshrv+Qt+BYnTbEvkdCW/UcxX c9irs5GcK5GQcH2TqI/s3wNgE+cudtDTOtIMzb50c4Dq3fGkxNv7uZwXqy7iovvDDGJLWsK7CrIoN An5jRXCfI0sROLueT77sgv2ssywsiKlqndEJ6rRXzkAJAmaGM5yh/2dbWuzCjKsjdaf8bX6p6CZJw 4i0Z3YrJfx0RLYhGt5aY+Khb6PepntGT+h1gqhW0gzsEwnocxgricSQ7AcyWRjm1LUYEWQxBPujxT mWwyEG2A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe2T-0007p0-Sk; Wed, 18 Dec 2019 18:27:13 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe2P-0007k4-Us for linux-arm-kernel@lists.infradead.org; Wed, 18 Dec 2019 18:27:11 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 684051FB; Wed, 18 Dec 2019 10:27:09 -0800 (PST) Received: from e108754-lin.cambridge.arm.com (unknown [10.1.198.81]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6A1383F67D; Wed, 18 Dec 2019 10:27:07 -0800 (PST) From: Ionela Voinescu To: catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, maz@kernel.org, suzuki.poulose@arm.com, sudeep.holla@arm.com, dietmar.eggemann@arm.com, ionela.voinescu@arm.com Subject: [PATCH v2 2/6] arm64: trap to EL1 accesses to AMU counters from EL0 Date: Wed, 18 Dec 2019 18:26:03 +0000 Message-Id: <20191218182607.21607-3-ionela.voinescu@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191218182607.21607-1-ionela.voinescu@arm.com> References: <20191218182607.21607-1-ionela.voinescu@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191218_102710_058504_392165F3 X-CRM114-Status: UNSURE ( 8.48 ) X-CRM114-Notice: Please train this message. X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-doc@vger.kernel.org, peterz@infradead.org, Steve Capper , linux-kernel@vger.kernel.org, mingo@redhat.com, ggherdovich@suse.cz, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The activity monitors extension is an optional extension introduced by the ARMv8.4 CPU architecture. In order to access the activity monitors counters safely, if desired, the kernel should detect the presence of the extension through the feature register, and mediate the access. Therefore, disable direct accesses to activity monitors counters from EL0 (userspace) and trap them to EL1 (kernel). Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: Steve Capper --- arch/arm64/include/asm/assembler.h | 10 ++++++++++ arch/arm64/mm/proc.S | 3 +++ 2 files changed, 13 insertions(+) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 2cc0dd8bd9f7..83bb499e8916 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -443,6 +443,16 @@ USER(\label, ic ivau, \tmp2) // invalidate I line PoU 9000: .endm +/* + * reset_amuserenr_el0 - reset AMUSERENR_EL0 if AMUv1 present + */ + .macro reset_amuserenr_el0, tmpreg + mrs \tmpreg, id_aa64pfr0_el1 // Check ID_AA64PFR0_EL1 + ubfx \tmpreg, \tmpreg, #ID_AA64PFR0_AMU_SHIFT, #4 + cbz \tmpreg, 9000f // Skip if no AMU present + msr_s SYS_AMUSERENR_EL0, xzr // Disable AMU access from EL0 +9000: + .endm /* * copy_page - copy src to dest using temp registers t1-t8 */ diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index a1e0592d1fbc..d8aae1152c08 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -124,6 +124,7 @@ alternative_endif ubfx x11, x11, #1, #1 msr oslar_el1, x11 reset_pmuserenr_el0 x0 // Disable PMU access from EL0 + reset_amuserenr_el0 x0 // Disable AMU access from EL0 alternative_if ARM64_HAS_RAS_EXTN msr_s SYS_DISR_EL1, xzr @@ -415,6 +416,8 @@ ENTRY(__cpu_setup) isb // Unmask debug exceptions now, enable_dbg // since this is per-cpu reset_pmuserenr_el0 x0 // Disable PMU access from EL0 + reset_amuserenr_el0 x0 // Disable AMU access from EL0 + /* * Memory region attributes for LPAE: * From patchwork Wed Dec 18 18:26:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ionela Voinescu X-Patchwork-Id: 11301609 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E3F0A109A for ; Wed, 18 Dec 2019 18:27:42 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A8BB12465E for ; Wed, 18 Dec 2019 18:27:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="kV8DL+9T" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A8BB12465E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=tmYCjfx28VBPRa1B6NR7F5nWoBXPitIM7hHmH6p0PQU=; b=kV8DL+9TaA5PKvHSDKsxXjvkQx vT9P1Bb2ZPmhW92MwxUEP2jfEEjGna/R6loheuyMhkICxsEHGzPWH+dOVWUs95VGiCt0B/r6IEU01 OaUzt3QSsFBoxvDL2DUef9fd3RjkbbWlXPW1mfs4yki4fG+qZdmXA3OGRzldICj9hIjTZxwcziN4s +mAmWOuFl/plIb4YIM6n+jDvRbmNUvaAt5/MJ4aZXxAlxB+n+0PHvBNM0wiBhIxWV13SqM2aunq+W GQsp+2Dc3IoA4K7C7tkgG4kU3s75+fcOTw1UAj9KKnpgvnpP08wDP82jLf7sxIRZGimwQceH5UA45 C1JueqSw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe2t-00088d-LG; Wed, 18 Dec 2019 18:27:39 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe2W-0007sN-H1 for linux-arm-kernel@lists.infradead.org; Wed, 18 Dec 2019 18:27:18 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0B3AC1FB; Wed, 18 Dec 2019 10:27:16 -0800 (PST) Received: from e108754-lin.cambridge.arm.com (unknown [10.1.198.81]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EA33A3F67D; Wed, 18 Dec 2019 10:27:13 -0800 (PST) From: Ionela Voinescu To: catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, maz@kernel.org, suzuki.poulose@arm.com, sudeep.holla@arm.com, dietmar.eggemann@arm.com, ionela.voinescu@arm.com Subject: [PATCH v2 3/6] arm64/kvm: disable access to AMU registers from kvm guests Date: Wed, 18 Dec 2019 18:26:04 +0000 Message-Id: <20191218182607.21607-4-ionela.voinescu@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191218182607.21607-1-ionela.voinescu@arm.com> References: <20191218182607.21607-1-ionela.voinescu@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191218_102716_690587_CA81F9F1 X-CRM114-Status: GOOD ( 13.46 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-doc@vger.kernel.org, peterz@infradead.org, linux-kernel@vger.kernel.org, mingo@redhat.com, ggherdovich@suse.cz, Julien Thierry , James Morse , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Access to the AMU counters should be disabled by default in kvm guests, as information from the counters might reveal activity in other guests or activity on the host. Therefore, disable access to AMU registers from EL0 and EL1 in kvm guests by: - Hiding the presence of the extension in the feature register (SYS_ID_AA64PFR0_EL1 and SYS_ID_PFR0_EL1) on the VCPU. - Disabling access to the AMU registers before switching to the guest. - Trapping accesses and injecting an undefined instruction into the guest. Signed-off-by: Ionela Voinescu Cc: Marc Zyngier Cc: James Morse Cc: Julien Thierry Cc: Suzuki K Poulose Cc: Catalin Marinas Cc: Will Deacon --- arch/arm64/include/asm/kvm_arm.h | 7 ++- arch/arm64/kvm/hyp/switch.c | 13 ++++- arch/arm64/kvm/sys_regs.c | 95 +++++++++++++++++++++++++++++++- 3 files changed, 109 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 6e5d839f42b5..dd20fb185d56 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -266,10 +266,11 @@ #define CPTR_EL2_TFP_SHIFT 10 /* Hyp Coprocessor Trap Register */ -#define CPTR_EL2_TCPAC (1 << 31) -#define CPTR_EL2_TTA (1 << 20) -#define CPTR_EL2_TFP (1 << CPTR_EL2_TFP_SHIFT) #define CPTR_EL2_TZ (1 << 8) +#define CPTR_EL2_TFP (1 << CPTR_EL2_TFP_SHIFT) +#define CPTR_EL2_TTA (1 << 20) +#define CPTR_EL2_TAM (1 << 30) +#define CPTR_EL2_TCPAC (1 << 31) #define CPTR_EL2_RES1 0x000032ff /* known RES1 bits in CPTR_EL2 */ #define CPTR_EL2_DEFAULT CPTR_EL2_RES1 diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 72fbbd86eb5e..0bca87a2621f 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -90,6 +90,17 @@ static void activate_traps_vhe(struct kvm_vcpu *vcpu) val = read_sysreg(cpacr_el1); val |= CPACR_EL1_TTA; val &= ~CPACR_EL1_ZEN; + + /* + * With VHE enabled, we have HCR_EL2.{E2H,TGE} = {1,1}. Note that in + * this case CPACR_EL1 has the same bit layout as CPTR_EL2, and + * CPACR_EL1 accessing instructions are redefined to access CPTR_EL2. + * Therefore use CPTR_EL2.TAM bit reference to activate AMU register + * traps. + */ + + val |= CPTR_EL2_TAM; + if (update_fp_enabled(vcpu)) { if (vcpu_has_sve(vcpu)) val |= CPACR_EL1_ZEN; @@ -111,7 +122,7 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu) __activate_traps_common(vcpu); val = CPTR_EL2_DEFAULT; - val |= CPTR_EL2_TTA | CPTR_EL2_TZ; + val |= CPTR_EL2_TTA | CPTR_EL2_TZ | CPTR_EL2_TAM; if (!update_fp_enabled(vcpu)) { val |= CPTR_EL2_TFP; __activate_traps_fpsimd32(vcpu); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 9f2165937f7d..940ab9b4c98b 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1003,6 +1003,20 @@ static bool access_pmuserenr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, { SYS_DESC(SYS_PMEVTYPERn_EL0(n)), \ access_pmu_evtyper, reset_unknown, (PMEVTYPER0_EL0 + n), } +static bool access_amu(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + kvm_inject_undefined(vcpu); + + return false; +} + +/* Macro to expand the AMU counter and type registers*/ +#define AMU_AMEVCNTR0_EL0(n) { SYS_DESC(SYS_AMEVCNTR0_EL0(n)), access_amu } +#define AMU_AMEVTYPE0_EL0(n) { SYS_DESC(SYS_AMEVTYPE0_EL0(n)), access_amu } +#define AMU_AMEVCNTR1_EL0(n) { SYS_DESC(SYS_AMEVCNTR1_EL0(n)), access_amu } +#define AMU_AMEVTYPE1_EL0(n) { SYS_DESC(SYS_AMEVTYPE1_EL0(n)), access_amu } + static bool trap_ptrauth(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *rd) @@ -1078,8 +1092,12 @@ static u64 read_id_reg(const struct kvm_vcpu *vcpu, (u32)r->CRn, (u32)r->CRm, (u32)r->Op2); u64 val = raz ? 0 : read_sanitised_ftr_reg(id); - if (id == SYS_ID_AA64PFR0_EL1 && !vcpu_has_sve(vcpu)) { - val &= ~(0xfUL << ID_AA64PFR0_SVE_SHIFT); + if (id == SYS_ID_AA64PFR0_EL1) { + if (!vcpu_has_sve(vcpu)) + val &= ~(0xfUL << ID_AA64PFR0_SVE_SHIFT); + val &= ~(0xfUL << ID_AA64PFR0_AMU_SHIFT); + } else if (id == SYS_ID_PFR0_EL1) { + val &= ~(0xfUL << ID_PFR0_AMU_SHIFT); } else if (id == SYS_ID_AA64ISAR1_EL1 && !vcpu_has_ptrauth(vcpu)) { val &= ~((0xfUL << ID_AA64ISAR1_APA_SHIFT) | (0xfUL << ID_AA64ISAR1_API_SHIFT) | @@ -1565,6 +1583,79 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_TPIDR_EL0), NULL, reset_unknown, TPIDR_EL0 }, { SYS_DESC(SYS_TPIDRRO_EL0), NULL, reset_unknown, TPIDRRO_EL0 }, + { SYS_DESC(SYS_AMCR_EL0), access_amu }, + { SYS_DESC(SYS_AMCFGR_EL0), access_amu }, + { SYS_DESC(SYS_AMCGCR_EL0), access_amu }, + { SYS_DESC(SYS_AMUSERENR_EL0), access_amu }, + { SYS_DESC(SYS_AMCNTENCLR0_EL0), access_amu }, + { SYS_DESC(SYS_AMCNTENSET0_EL0), access_amu }, + { SYS_DESC(SYS_AMCNTENCLR1_EL0), access_amu }, + { SYS_DESC(SYS_AMCNTENSET1_EL0), access_amu }, + AMU_AMEVCNTR0_EL0(0), + AMU_AMEVCNTR0_EL0(1), + AMU_AMEVCNTR0_EL0(2), + AMU_AMEVCNTR0_EL0(3), + AMU_AMEVCNTR0_EL0(4), + AMU_AMEVCNTR0_EL0(5), + AMU_AMEVCNTR0_EL0(6), + AMU_AMEVCNTR0_EL0(7), + AMU_AMEVCNTR0_EL0(8), + AMU_AMEVCNTR0_EL0(9), + AMU_AMEVCNTR0_EL0(10), + AMU_AMEVCNTR0_EL0(11), + AMU_AMEVCNTR0_EL0(12), + AMU_AMEVCNTR0_EL0(13), + AMU_AMEVCNTR0_EL0(14), + AMU_AMEVCNTR0_EL0(15), + AMU_AMEVTYPE0_EL0(0), + AMU_AMEVTYPE0_EL0(1), + AMU_AMEVTYPE0_EL0(2), + AMU_AMEVTYPE0_EL0(3), + AMU_AMEVTYPE0_EL0(4), + AMU_AMEVTYPE0_EL0(5), + AMU_AMEVTYPE0_EL0(6), + AMU_AMEVTYPE0_EL0(7), + AMU_AMEVTYPE0_EL0(8), + AMU_AMEVTYPE0_EL0(9), + AMU_AMEVTYPE0_EL0(10), + AMU_AMEVTYPE0_EL0(11), + AMU_AMEVTYPE0_EL0(12), + AMU_AMEVTYPE0_EL0(13), + AMU_AMEVTYPE0_EL0(14), + AMU_AMEVTYPE0_EL0(15), + AMU_AMEVCNTR1_EL0(0), + AMU_AMEVCNTR1_EL0(1), + AMU_AMEVCNTR1_EL0(2), + AMU_AMEVCNTR1_EL0(3), + AMU_AMEVCNTR1_EL0(4), + AMU_AMEVCNTR1_EL0(5), + AMU_AMEVCNTR1_EL0(6), + AMU_AMEVCNTR1_EL0(7), + AMU_AMEVCNTR1_EL0(8), + AMU_AMEVCNTR1_EL0(9), + AMU_AMEVCNTR1_EL0(10), + AMU_AMEVCNTR1_EL0(11), + AMU_AMEVCNTR1_EL0(12), + AMU_AMEVCNTR1_EL0(13), + AMU_AMEVCNTR1_EL0(14), + AMU_AMEVCNTR1_EL0(15), + AMU_AMEVTYPE1_EL0(0), + AMU_AMEVTYPE1_EL0(1), + AMU_AMEVTYPE1_EL0(2), + AMU_AMEVTYPE1_EL0(3), + AMU_AMEVTYPE1_EL0(4), + AMU_AMEVTYPE1_EL0(5), + AMU_AMEVTYPE1_EL0(6), + AMU_AMEVTYPE1_EL0(7), + AMU_AMEVTYPE1_EL0(8), + AMU_AMEVTYPE1_EL0(9), + AMU_AMEVTYPE1_EL0(10), + AMU_AMEVTYPE1_EL0(11), + AMU_AMEVTYPE1_EL0(12), + AMU_AMEVTYPE1_EL0(13), + AMU_AMEVTYPE1_EL0(14), + AMU_AMEVTYPE1_EL0(15), + { SYS_DESC(SYS_CNTP_TVAL_EL0), access_arch_timer }, { SYS_DESC(SYS_CNTP_CTL_EL0), access_arch_timer }, { SYS_DESC(SYS_CNTP_CVAL_EL0), access_arch_timer }, From patchwork Wed Dec 18 18:26:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ionela Voinescu X-Patchwork-Id: 11301613 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4BE11109A for ; Wed, 18 Dec 2019 18:28:17 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 114ED20716 for ; Wed, 18 Dec 2019 18:28:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="msouINOA" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 114ED20716 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=SRiQqDmfyhOtLwyJXNJZNDCd7y9fyhcwDcBZcXXcLlQ=; b=msouINOAVoqM+m7cJ2DRFw7yCT IHFIetaR1X2FKtLXqyEm9ZoZGq8qqUc7q+B6s7ot2hEZFqAGPpf/q/7015JUlIVKo42QfIFOZf7Yy Z8Y6ea8LWcRIppm0WTCOtLfrrJUg8tboj+NXc8EAxBIoOFpBuok0OYz8GCAzruaT2vbtXjOBJNFFR 19jpAlFokwBCxXoHTcDjb5Ln11vCKgM6NTXIHxMHdQbaqcHM5/QegbYM9boPCv6pCs25OIPP7Awh9 HmCmtGerNB7a+EHJxnYsszRgQkmEnfNlC0quze6+fPjTJcLGX0CnbKFu0xhNSQZsvvCkc4e4AKGtq MWL4pObQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe3S-0000B4-Ie; Wed, 18 Dec 2019 18:28:14 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe2Z-0007vP-Rr for linux-arm-kernel@lists.infradead.org; Wed, 18 Dec 2019 18:27:27 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4693B1FB; Wed, 18 Dec 2019 10:27:19 -0800 (PST) Received: from e108754-lin.cambridge.arm.com (unknown [10.1.198.81]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4B8033F67D; Wed, 18 Dec 2019 10:27:17 -0800 (PST) From: Ionela Voinescu To: catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, maz@kernel.org, suzuki.poulose@arm.com, sudeep.holla@arm.com, dietmar.eggemann@arm.com, ionela.voinescu@arm.com Subject: [PATCH v2 4/6] Documentation: arm64: document support for the AMU extension Date: Wed, 18 Dec 2019 18:26:05 +0000 Message-Id: <20191218182607.21607-5-ionela.voinescu@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191218182607.21607-1-ionela.voinescu@arm.com> References: <20191218182607.21607-1-ionela.voinescu@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191218_102720_143278_3BB9638D X-CRM114-Status: GOOD ( 19.13 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-doc@vger.kernel.org, peterz@infradead.org, Jonathan Corbet , linux-kernel@vger.kernel.org, mingo@redhat.com, ggherdovich@suse.cz, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The activity monitors extension is an optional extension introduced by the ARMv8.4 CPU architecture. Add initial documentation for the AMUv1 extension: - arm64/amu.txt: AMUv1 documentation - arm64/booting.txt: system registers initialisation - arm64/cpu-feature-registers.txt: visibility to userspace Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc: Will Deacon Cc: Jonathan Corbet --- Documentation/arm64/amu.rst | 107 ++++++++++++++++++ Documentation/arm64/booting.rst | 14 +++ Documentation/arm64/cpu-feature-registers.rst | 2 + Documentation/arm64/index.rst | 1 + 4 files changed, 124 insertions(+) create mode 100644 Documentation/arm64/amu.rst diff --git a/Documentation/arm64/amu.rst b/Documentation/arm64/amu.rst new file mode 100644 index 000000000000..62a6635522e1 --- /dev/null +++ b/Documentation/arm64/amu.rst @@ -0,0 +1,107 @@ +======================================================= +Activity Monitors Unit (AMU) extension in AArch64 Linux +======================================================= + +Author: Ionela Voinescu + +Date: 2019-09-10 + +This document briefly describes the provision of Activity Monitors Unit +support in AArch64 Linux. + + +Architecture overview +--------------------- + +The activity monitors extension is an optional extension introduced by the +ARMv8.4 CPU architecture. + +The activity monitors unit, implemented in each CPU, provides performance +counters intended for system management use. The AMU extension provides a +system register interface to the counter registers and also supports an +optional external memory-mapped interface. + +Version 1 of the Activity Monitors architecture implements a counter group +of four fixed and architecturally defined 64-bit event counters. + - CPU cycle counter: increments at the frequency of the CPU. + - Constant counter: increments at the fixed frequency of the system + clock. + - Instructions retired: increments with every architecturally executed + instruction. + - Memory stall cycles: counts instruction dispatch stall cycles caused by + misses in the last level cache within the clock domain. + +When in WFI or WFE these counters do not increment. + +The Activity Monitors architecture provides space for up to 16 architected +event counters. Future versions of the architecture may use this space to +implement additional architected event counters. + +Additionally, version 1 implements a counter group of up to 16 auxiliary +64-bit event counters. + +On cold reset all counters reset to 0. + + +Basic support +------------- + +The kernel can safely run a mix of CPUs with and without support for the +activity monitors extension. Therefore, when CONFIG_ARM64_AMU_EXTN is +selected we unconditionally enable the capability to allow any late CPU +(secondary or hotplugged) to detect and use the feature. + +When the feature is detected on a CPU, a per-CPU variable (amu_feat) is +set, but this does not guarantee the correct functionality of the +counters, only the presence of the extension. + +Firmware (code running at higher exception levels, e.g. arm-tf) support is +needed to: + - Enable access for lower exception levels (EL2 and EL1) to the AMU + registers. + - Enable the counters. If not enabled these will read as 0. + - Save/restore the counters before/after the CPU is being put/brought up + from the 'off' power state. + +When using kernels that have this configuration enabled but boot with +broken firmware the user may experience panics or lockups when accessing +the counter registers. Even if these symptoms are not observed, the +values returned by the register reads might not correctly reflect reality. +Most commonly, the counters will read as 0, indicating that they are not +enabled. If proper support is not provided in firmware it's best to disable +CONFIG_ARM64_AMU_EXTN. + +The fixed counters of AMUv1 are accessible though the following system +register definitions: + - SYS_AMEVCNTR0_CORE_EL0 + - SYS_AMEVCNTR0_CONST_EL0 + - SYS_AMEVCNTR0_INST_RET_EL0 + - SYS_AMEVCNTR0_MEM_STALL_EL0 + +Auxiliary platform specific counters can be accessed using +SYS_AMEVCNTR1_EL0(n), where n is a value between 0 and 15. + +Details can be found in: arch/arm64/include/asm/sysreg.h. + + +Userspace access +---------------- + +Currently, access from userspace to the AMU registers is disabled due to: + - Security reasons: they might expose information about code executed in + secure mode. + - Purpose: AMU counters are intended for system management use. + +Also, the presence of the feature is not visible to userspace. + + +Virtualization +-------------- + +Currently, access from userspace (EL0) and kernelspace (EL1) on the KVM +guest side is disabled due to: + - Security reasons: they might expose information about code executed + by other guests or the host. + +Any attempt to access the AMU registers will result in an UNDEFINED +exception being injected into the guest. diff --git a/Documentation/arm64/booting.rst b/Documentation/arm64/booting.rst index 5d78a6f5b0ae..a3f1a47b6f1c 100644 --- a/Documentation/arm64/booting.rst +++ b/Documentation/arm64/booting.rst @@ -248,6 +248,20 @@ Before jumping into the kernel, the following conditions must be met: - HCR_EL2.APK (bit 40) must be initialised to 0b1 - HCR_EL2.API (bit 41) must be initialised to 0b1 + For CPUs with Activity Monitors Unit v1 (AMUv1) extension present: + - If EL3 is present: + CPTR_EL3.TAM (bit 30) must be initialised to 0b0 + CPTR_EL2.TAM (bit 30) must be initialised to 0b0 + AMCNTENSET0_EL0 must be initialised to 0b1111 + AMCNTENSET1_EL0 must be initialised to a platform specific value + having 0b1 set for the corresponding bit for each of the auxiliary + counters present. + - If the kernel is entered at EL1: + AMCNTENSET0_EL0 must be initialised to 0b1111 + AMCNTENSET1_EL0 must be initialised to a platform specific value + having 0b1 set for the corresponding bit for each of the auxiliary + counters present. + The requirements described above for CPU mode, caches, MMUs, architected timers, coherency and system registers apply to all CPUs. All CPUs must enter the kernel in the same exception level. diff --git a/Documentation/arm64/cpu-feature-registers.rst b/Documentation/arm64/cpu-feature-registers.rst index b6e44884e3ad..4770ae54032b 100644 --- a/Documentation/arm64/cpu-feature-registers.rst +++ b/Documentation/arm64/cpu-feature-registers.rst @@ -150,6 +150,8 @@ infrastructure: +------------------------------+---------+---------+ | DIT | [51-48] | y | +------------------------------+---------+---------+ + | AMU | [47-44] | n | + +------------------------------+---------+---------+ | SVE | [35-32] | y | +------------------------------+---------+---------+ | GIC | [27-24] | n | diff --git a/Documentation/arm64/index.rst b/Documentation/arm64/index.rst index 5c0c69dc58aa..09cbb4ed2237 100644 --- a/Documentation/arm64/index.rst +++ b/Documentation/arm64/index.rst @@ -6,6 +6,7 @@ ARM64 Architecture :maxdepth: 1 acpi_object_usage + amu arm-acpi booting cpu-feature-registers From patchwork Wed Dec 18 18:26:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ionela Voinescu X-Patchwork-Id: 11301611 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4E94C17EF for ; Wed, 18 Dec 2019 18:28:02 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2867720716 for ; Wed, 18 Dec 2019 18:28:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="i8CNEzxA" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2867720716 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bPhR+A1QWbXz22wu+88UCBty+QeclJpP/s1exBuJIUk=; b=i8CNEzxAla6BVaEXqZ8KmvinQI zdAp6o7L+rlRHSwGQL7HpmS5U7qjJS1oa4Tjlmp6MMHBY6WdKztsJHxOWV58ntd+YaHquAgheWBMl 5F0T+ffvueT0aXJCDpmbtoyv34INbgiinsc1T51CWJwD3QF2KLE6LPVl+BvzhMu8/bKt8XBjB0lJM EplidWsmRf0PwsTvSwv1yVJjYLtdgC0r2/oLajkh4bEmxcnzG5qZBUsDx+aUL9EfR0MPJ7WGhVBfM 9A6F0sxWvSwjuinXEAShhQjG2u2FnInBaxVfgDZf+7pLismkR3oQ2pRcAZpD4H23Urp+rLvXGMgqp Vl50ryvw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe3E-0008Qq-0l; Wed, 18 Dec 2019 18:28:00 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe2d-0007yY-HP for linux-arm-kernel@lists.infradead.org; Wed, 18 Dec 2019 18:27:27 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9863931B; Wed, 18 Dec 2019 10:27:22 -0800 (PST) Received: from e108754-lin.cambridge.arm.com (unknown [10.1.198.81]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9DE453F67D; Wed, 18 Dec 2019 10:27:20 -0800 (PST) From: Ionela Voinescu To: catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, maz@kernel.org, suzuki.poulose@arm.com, sudeep.holla@arm.com, dietmar.eggemann@arm.com, ionela.voinescu@arm.com Subject: [PATCH v2 5/6] TEMP: sched: add interface for counter-based frequency invariance Date: Wed, 18 Dec 2019 18:26:06 +0000 Message-Id: <20191218182607.21607-6-ionela.voinescu@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191218182607.21607-1-ionela.voinescu@arm.com> References: <20191218182607.21607-1-ionela.voinescu@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191218_102723_692977_DDA25EB3 X-CRM114-Status: GOOD ( 10.50 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Juri Lelli , linux-doc@vger.kernel.org, peterz@infradead.org, linux-kernel@vger.kernel.org, mingo@redhat.com, ggherdovich@suse.cz, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org To be noted that this patch is a temporary one. It introduces the interface added by the patches at [1] to allow update of the frequency invariance scale factor based on counters. If [1] is merged there is not need for this patch. For platforms that support counters (x86 - APERF/MPERF, arm64 - AMU counters) the frequency invariance correction factor can be obtained using a core counter and a fixed counter to get information on the performance (frequency based only) obtained in a period of time. This will more accurately reflect the actual current frequency of the CPU, compared with the alternative implementation that reflects the request of a performance level from the OS through the cpufreq framework (arch_set_freq_scale). Therefore, introduce an interface - arch_scale_freq_tick, to be implemented by each architecture and called for each CPU on the tick to update the scale factor based on the delta in the counter values, if counter support is present on the CPU. Either because reading counters is expensive or because reading counters from remote CPUs is not possible or is expensive, only update the counter based frequency scale factor on the tick for now. A tick based update will definitely be necessary either due to it being the only point of update for certain architectures or in order to cache the counter values for a particular CPU, if a further update from that CPU is not possible. [1] https://lore.kernel.org/lkml/20191113124654.18122-1-ggherdovich@suse.cz/ Suggested-by: Peter Zijlstra Suggested-by: Giovanni Gherdovich Signed-off-by: Ionela Voinescu Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann --- kernel/sched/core.c | 1 + kernel/sched/sched.h | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 90e4b00ace89..e0b70b9fb5cc 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3594,6 +3594,7 @@ void scheduler_tick(void) struct task_struct *curr = rq->curr; struct rq_flags rf; + arch_scale_freq_tick(); sched_clock_tick(); rq_lock(rq, &rf); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 280a3c735935..afdafcf7f9da 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1771,6 +1771,13 @@ static inline void set_next_task(struct rq *rq, struct task_struct *next) next->sched_class->set_next_task(rq, next, false); } +#ifndef arch_scale_freq_tick +static __always_inline +void arch_scale_freq_tick(void) +{ +} +#endif + #ifdef CONFIG_SMP #define sched_class_highest (&stop_sched_class) #else From patchwork Wed Dec 18 18:26:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ionela Voinescu X-Patchwork-Id: 11301615 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8CAAD109A for ; Wed, 18 Dec 2019 18:28:34 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 55B8B2465E for ; Wed, 18 Dec 2019 18:28:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="PqSMTJpJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 55B8B2465E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=gFMapZobBCZ20JKxw4kk609Tn7JFkQgrtVO6BzCHG8c=; b=PqSMTJpJsk0JuZFCEvG3gFonOW l3BOedx4Vx2cgWYvdD7jXP8QItwxif3qFImUcrZt1sRTqahrPu4wL58VVyMtHjHbN8OErIKHA5B07 6E1UTNwrW22JxzQSGst4frbMFUPRqkx2wsfVaNWkjII/gNW9gk3GgauWbWt93sYNfpeGAwlTPcg1G /KTRdMaqihs0HRDIa9Eumnitlwh3MO4PHwtTC7+p7usXXH2iCEupN5V2QDw6dQuW9HaAfUPHdS2XC JV2iYlUSyZB/0ozThiepBvcV6b1lzNPwNjoSB+y8SXNGgnGiqmp//fOmfXsF6DKtp5L2Ru3WsZY6l 6djxON1A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe3k-0000RN-G7; Wed, 18 Dec 2019 18:28:32 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihe2g-00080L-2S for linux-arm-kernel@lists.infradead.org; Wed, 18 Dec 2019 18:27:30 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 85AB2113E; Wed, 18 Dec 2019 10:27:25 -0800 (PST) Received: from e108754-lin.cambridge.arm.com (unknown [10.1.198.81]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A09FB3F67D; Wed, 18 Dec 2019 10:27:23 -0800 (PST) From: Ionela Voinescu To: catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, maz@kernel.org, suzuki.poulose@arm.com, sudeep.holla@arm.com, dietmar.eggemann@arm.com, ionela.voinescu@arm.com Subject: [PATCH v2 6/6] arm64: use activity monitors for frequency invariance Date: Wed, 18 Dec 2019 18:26:07 +0000 Message-Id: <20191218182607.21607-7-ionela.voinescu@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191218182607.21607-1-ionela.voinescu@arm.com> References: <20191218182607.21607-1-ionela.voinescu@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191218_102726_251761_A3B95287 X-CRM114-Status: GOOD ( 21.86 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [217.140.110.172 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-doc@vger.kernel.org, peterz@infradead.org, linux-kernel@vger.kernel.org, mingo@redhat.com, ggherdovich@suse.cz, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The Frequency Invariance Engine (FIE) is providing a frequency scaling correction factor that helps achieve more accurate load-tracking. So far, for arm and arm64 platforms, this scale factor has been obtained based on the ratio between the current frequency and the maximum supported frequency recorded by the cpufreq policy. The setting of this scale factor is triggered from cpufreq drivers by calling arch_set_freq_scale. The current frequency used in computation is the frequency requested by a governor, but it may not be the frequency that was implemented by the platform. This correction factor can also be obtained using a core counter and a constant counter to get information on the performance (frequency based only) obtained in a period of time. This will more accurately reflect the actual current frequency of the CPU, compared with the alternative implementation that reflects the request of a performance level from the OS. Therefore, implement arch_scale_freq_tick to use activity monitors, if present, for the computation of the frequency scale factor. The use of AMU counters depends on: - CONFIG_ARM64_AMU_EXTN - depents on the AMU extension being present - CONFIG_CPU_FREQ - the current frequency obtained using counter information is divided by the maximum frequency obtained from the cpufreq policy. While it is possible to have a combination of CPUs in the system with and without support for activity monitors, the use of counters for frequency invariance is only enabled for a CPU, if all related CPUs (CPUs in the same frequency domain) support and have enabled the core and constant activity monitor counters. In this way, there is a clear separation between the policies for which arch_set_freq_scale (cpufreq based FIE) is used, and the policies for which arch_scale_freq_tick (counter based FIE) is used to set the frequency scale factor. For this purpose, a cpufreq notifier is registered to trigger validation work for CPUs and policies at policy creation that will enable or disable the use of AMU counters for frequency invariance. Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc: Will Deacon Cc: Sudeep Holla --- arch/arm64/include/asm/topology.h | 9 ++ arch/arm64/kernel/topology.c | 233 ++++++++++++++++++++++++++++++ drivers/base/arch_topology.c | 16 ++ 3 files changed, 258 insertions(+) diff --git a/arch/arm64/include/asm/topology.h b/arch/arm64/include/asm/topology.h index a4d945db95a2..98412dd27565 100644 --- a/arch/arm64/include/asm/topology.h +++ b/arch/arm64/include/asm/topology.h @@ -19,6 +19,15 @@ int pcibus_to_node(struct pci_bus *bus); /* Replace task scheduler's default frequency-invariant accounting */ #define arch_scale_freq_capacity topology_get_freq_scale +#if defined(CONFIG_ARM64_AMU_EXTN) && defined(CONFIG_CPU_FREQ) +void topology_scale_freq_tick(void); +/* + * Replace task scheduler's default counter-based frequency-invariance + * scale factor setting. + */ +#define arch_scale_freq_tick topology_scale_freq_tick +#endif + /* Replace task scheduler's default cpu-invariant accounting */ #define arch_scale_cpu_capacity topology_get_cpu_scale diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index fa9528dfd0ce..61f8264afec9 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -120,4 +121,236 @@ int __init parse_acpi_topology(void) } #endif +#if defined(CONFIG_ARM64_AMU_EXTN) && defined(CONFIG_CPU_FREQ) +#undef pr_fmt +#define pr_fmt(fmt) "AMU: " fmt + +static void init_fie_counters_done_workfn(struct work_struct *work); +static DECLARE_WORK(init_fie_counters_done_work, + init_fie_counters_done_workfn); + +static struct workqueue_struct *policy_amu_fie_init_wq; +static struct workqueue_struct *cpu_amu_fie_init_wq; + +struct cpu_amu_work { + struct work_struct cpu_work; + struct work_struct policy_work; + unsigned int cpuinfo_max_freq; + struct cpumask policy_cpus; + bool cpu_amu_fie; +}; +static struct cpu_amu_work __percpu *works; +static cpumask_var_t cpus_to_visit; + +static DEFINE_PER_CPU_READ_MOSTLY(unsigned long, arch_max_freq_scale); +static DEFINE_PER_CPU(u64, arch_const_cycles_prev); +static DEFINE_PER_CPU(u64, arch_core_cycles_prev); +DECLARE_PER_CPU(u8, amu_scale_freq); + +static void cpu_amu_fie_init_workfn(struct work_struct *work) +{ + u64 core_cnt, const_cnt, ratio; + struct cpu_amu_work *amu_work; + int cpu = smp_processor_id(); + + if (!cpu_has_amu_feat()) { + pr_debug("CPU%d: counters are not supported.\n", cpu); + return; + } + + core_cnt = read_sysreg_s(SYS_AMEVCNTR0_CORE_EL0); + const_cnt = read_sysreg_s(SYS_AMEVCNTR0_CONST_EL0); + + if (unlikely(!core_cnt || !const_cnt)) { + pr_err("CPU%d: cycle counters are not enabled.\n", cpu); + return; + } + + amu_work = container_of(work, struct cpu_amu_work, cpu_work); + if (unlikely(!(amu_work->cpuinfo_max_freq))) { + pr_err("CPU%d: invalid maximum frequency.\n", cpu); + return; + } + + /* + * Pre-compute the fixed ratio between the frequency of the + * constant counter and the maximum frequency of the CPU (hz). + */ + ratio = (u64)arch_timer_get_rate() << (2 * SCHED_CAPACITY_SHIFT); + ratio = div64_u64(ratio, amu_work->cpuinfo_max_freq * 1000); + this_cpu_write(arch_max_freq_scale, (unsigned long)ratio); + + this_cpu_write(arch_core_cycles_prev, core_cnt); + this_cpu_write(arch_const_cycles_prev, const_cnt); + amu_work->cpu_amu_fie = true; +} + +static void policy_amu_fie_init_workfn(struct work_struct *work) +{ + struct cpu_amu_work *amu_work; + u8 enable; + int cpu; + + amu_work = container_of(work, struct cpu_amu_work, policy_work); + + flush_workqueue(cpu_amu_fie_init_wq); + + for_each_cpu(cpu, &amu_work->policy_cpus) + if (!(per_cpu_ptr(works, cpu)->cpu_amu_fie)) + break; + + enable = (cpu >= nr_cpu_ids) ? 1 : 0; + + for_each_cpu(cpu, &amu_work->policy_cpus) + per_cpu(amu_scale_freq, cpu) = enable; + + pr_info("CPUs[%*pbl]: counters %s be used for FIE.", + cpumask_pr_args(&amu_work->policy_cpus), + enable ? "will" : "WON'T"); +} + +static int init_fie_counters_callback(struct notifier_block *nb, + unsigned long val, + void *data) +{ + struct cpufreq_policy *policy = data; + struct cpu_amu_work *work; + int cpu; + + if (val != CPUFREQ_CREATE_POLICY) + return 0; + + /* Return if not all related CPUs are online */ + if (!cpumask_equal(policy->cpus, policy->related_cpus)) { + pr_info("CPUs[%*pbl]: counters WON'T be used for FIE.", + cpumask_pr_args(policy->related_cpus)); + return 0; + } + + /* + * Queue functions on all online CPUs from policy to: + * - Check support and enablement for AMU counters + * - Store system freq to max freq ratio per cpu + * - Flag CPU as valid for use of counters for FIE + */ + for_each_cpu(cpu, policy->cpus) { + work = per_cpu_ptr(works, cpu); + work->cpuinfo_max_freq = policy->cpuinfo.max_freq; + work->cpu_amu_fie = false; + INIT_WORK(&work->cpu_work, cpu_amu_fie_init_workfn); + queue_work_on(cpu, cpu_amu_fie_init_wq, &work->cpu_work); + } + + /* + * Queue function to validate support at policy level: + * - Flush all work on online policy CPUs + * - Verify that all online policy CPUs are flagged as + * valid for use of counters for FIE + * - Enable or disable use of counters for FIE on CPUs + */ + work = per_cpu_ptr(works, cpumask_first(policy->cpus)); + cpumask_copy(&work->policy_cpus, policy->cpus); + INIT_WORK(&work->policy_work, policy_amu_fie_init_workfn); + queue_work(policy_amu_fie_init_wq, &work->policy_work); + + cpumask_andnot(cpus_to_visit, cpus_to_visit, policy->cpus); + if (cpumask_empty(cpus_to_visit)) + schedule_work(&init_fie_counters_done_work); + + return 0; +} + +static struct notifier_block init_fie_counters_notifier = { + .notifier_call = init_fie_counters_callback, +}; + +static void init_fie_counters_done_workfn(struct work_struct *work) +{ + cpufreq_unregister_notifier(&init_fie_counters_notifier, + CPUFREQ_POLICY_NOTIFIER); + + /* + * Destroy policy_amu_fie_init_wq first to ensure all policy + * work is finished, which includes flushing of the per-CPU + * work, before cpu_amu_fie_init_wq is destroyed. + */ + destroy_workqueue(policy_amu_fie_init_wq); + destroy_workqueue(cpu_amu_fie_init_wq); + + free_percpu(works); + free_cpumask_var(cpus_to_visit); +} + +static int __init register_fie_counters_cpufreq_notifier(void) +{ + int ret = -ENOMEM; + + if (!alloc_cpumask_var(&cpus_to_visit, GFP_KERNEL)) + goto out; + + cpumask_copy(cpus_to_visit, cpu_possible_mask); + + cpu_amu_fie_init_wq = create_workqueue("cpu_amu_fie_init_wq"); + if (!cpu_amu_fie_init_wq) + goto free_cpumask; + + policy_amu_fie_init_wq = create_workqueue("policy_amu_fie_init_wq"); + if (!cpu_amu_fie_init_wq) + goto free_cpu_wq; + + works = alloc_percpu(struct cpu_amu_work); + if (!works) + goto free_policy_wq; + + ret = cpufreq_register_notifier(&init_fie_counters_notifier, + CPUFREQ_POLICY_NOTIFIER); + if (ret) + goto free_works; + + return 0; + +free_works: + free_percpu(works); +free_policy_wq: + destroy_workqueue(policy_amu_fie_init_wq); +free_cpu_wq: + destroy_workqueue(cpu_amu_fie_init_wq); +free_cpumask: + free_cpumask_var(cpus_to_visit); +out: + return ret; +} +core_initcall(register_fie_counters_cpufreq_notifier); + +void topology_scale_freq_tick(void) +{ + u64 prev_core_cnt, prev_const_cnt; + u64 core_cnt, const_cnt, scale; + + if (!this_cpu_read(amu_scale_freq)) + return; + + const_cnt = read_sysreg_s(SYS_AMEVCNTR0_CONST_EL0); + core_cnt = read_sysreg_s(SYS_AMEVCNTR0_CORE_EL0); + prev_const_cnt = this_cpu_read(arch_const_cycles_prev); + prev_core_cnt = this_cpu_read(arch_core_cycles_prev); + + if (unlikely(core_cnt <= prev_core_cnt || + const_cnt <= prev_const_cnt)) + goto store_and_exit; + + scale = core_cnt - prev_core_cnt; + scale *= this_cpu_read(arch_max_freq_scale); + scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT, + const_cnt - prev_const_cnt); + + scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE); + this_cpu_write(freq_scale, (unsigned long)scale); + +store_and_exit: + this_cpu_write(arch_core_cycles_prev, core_cnt); + this_cpu_write(arch_const_cycles_prev, const_cnt); +} + +#endif diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index 1eb81f113786..3ae6091d845e 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -23,12 +23,28 @@ DEFINE_PER_CPU(unsigned long, freq_scale) = SCHED_CAPACITY_SCALE; +#if defined(CONFIG_ARM64_AMU_EXTN) && defined(CONFIG_CPU_FREQ) +DEFINE_PER_CPU_READ_MOSTLY(u8, amu_scale_freq); +#endif + void arch_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq, unsigned long max_freq) { unsigned long scale; int i; +#if defined(CONFIG_ARM64_AMU_EXTN) && defined(CONFIG_CPU_FREQ) + /* + * This function will only be called from CPUFREQ drivers. + * If the use of counters for FIE is enabled, establish if a CPU, + * the first one, supports counters and if they are valid. If they + * are, just return as we don't want to update with information + * from CPUFREQ. In this case the scale factor will be updated + * from arch_scale_freq_tick. + */ + if (per_cpu(amu_scale_freq, cpumask_first(cpus))) + return; +#endif scale = (cur_freq << SCHED_CAPACITY_SHIFT) / max_freq; for_each_cpu(i, cpus)