From patchwork Sun May 8 16:07:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 12842482 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9E274C433F5 for ; Sun, 8 May 2022 16:16:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CWHOB/nzS4nqMsvw7H05boi7TvtgsdAWigYJJyTEwVI=; b=4C+CjrvxritVet MjyTU6pzfmnxYpVJ4DyUhD8OnCX/u+GrpTRbjX0XDQtCUskbwQ5al6QrQwlRemFb/zlggShhno4zu kSCaSyQmLM/CcM96e4d+kPhF376/FWClqQp7HrnidHUusk5bo9e9ilDSeV1iVbsgoyrKoF0GbblXq 4XYk2+i2E8GTsxVbCCSB38eVNJ+FycGb9jd0DEI/c7ps7yISc06SzxSzah3TN56aceS8iHjnca+CA awMo8ThIL/A0onSHTBMEi6KPPgVqd95gaCDV4G56mPvf4I658eyc6UPsdqEYOtR5yQWP4y7I8rKew 5mlY+FtLNVpPK0TLu4EA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nnjaF-00Ab0D-C9; Sun, 08 May 2022 16:16:35 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nnjaC-00AazC-JH for linux-riscv@lists.infradead.org; Sun, 08 May 2022 16:16:33 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 507696121C; Sun, 8 May 2022 16:16:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BCB75C385A4; Sun, 8 May 2022 16:16:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1652026589; bh=cai4u5JKDr1cLjhdt9yaycCa4Y3iBnxfkpZjmnNWcVw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mdPAo0t5mrvr4GmAHEi7fUpzOr52OdPOdTfvMG2GDQeOpWO4nWnz+BT8ei8S8bldv KGF56HqI6nCm50Jia/awUHAqUlXLc5KirY9OTAv+cDqV+3VHJOJNG66cS9GlDqP5wP l4+sOr1tt2o3b7/L6ifGSp7h0MxMkLSy9wfI8RXXzx5TJtIaSd8MK8o9VM+DXOt7xy Po0SUd/H6NhU4PshiM8nAIBCnU85lUhGkdQXtmz5iY5gMnEkGC3VQTcQBPLoRKq/D9 bESc0bsVD7erCPt7hJGlgaoO0YbJUdtISbpRRey/rCYsW+7jIV+hrrQTGz/IPqvvYI biSZhsinVL0Rg== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Alexandre Ghiti Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: [PATCH v2 1/4] riscv: mm: init: make pt_ops_set_[early|late|fixmap] static Date: Mon, 9 May 2022 00:07:46 +0800 Message-Id: <20220508160749.984-2-jszhang@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220508160749.984-1-jszhang@kernel.org> References: <20220508160749.984-1-jszhang@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220508_091632_708144_A23BC0F9 X-CRM114-Status: GOOD ( 12.85 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org These three functions are only used in init.c, so make them static. Fix W=1 warnings like below: arch/riscv/mm/init.c:721:13: warning: no previous prototype for function 'pt_ops_set_early' [-Wmissing-prototypes] void __init pt_ops_set_early(void) ^ Signed-off-by: Jisheng Zhang Reviewed-by: Anup Patel --- arch/riscv/mm/init.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 05ed641a1134..5f3f26dd9f21 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -849,7 +849,7 @@ static void __init create_fdt_early_page_table(pgd_t *pgdir, uintptr_t dtb_pa) * MMU is not enabled, the page tables are allocated directly using * early_pmd/pud/p4d and the address returned is the physical one. */ -void __init pt_ops_set_early(void) +static void __init pt_ops_set_early(void) { pt_ops.alloc_pte = alloc_pte_early; pt_ops.get_pte_virt = get_pte_virt_early; @@ -871,7 +871,7 @@ void __init pt_ops_set_early(void) * Note that this is called with MMU disabled, hence kernel_mapping_pa_to_va, * but it will be used as described above. */ -void __init pt_ops_set_fixmap(void) +static void __init pt_ops_set_fixmap(void) { pt_ops.alloc_pte = kernel_mapping_pa_to_va((uintptr_t)alloc_pte_fixmap); pt_ops.get_pte_virt = kernel_mapping_pa_to_va((uintptr_t)get_pte_virt_fixmap); @@ -889,7 +889,7 @@ void __init pt_ops_set_fixmap(void) * MMU is enabled and page table setup is complete, so from now, we can use * generic page allocation functions to setup page table. */ -void __init pt_ops_set_late(void) +static void __init pt_ops_set_late(void) { pt_ops.alloc_pte = alloc_pte_late; pt_ops.get_pte_virt = get_pte_virt_late; From patchwork Sun May 8 16:07:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 12842481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 48DC7C433FE for ; Sun, 8 May 2022 16:16:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bOEdLEhtnHuUlugt4YBqVIfOKU6E+UCEyetiU7Yh1E4=; b=BfFWWT8WGQvlI2 7hPBAWHVXIsMlZyw7T6hfUokxuyDLLgptOb7FK/D4Rou212i52hB7FnbGI+xjV3C6u2Dr4zLiBhD5 3aRljQJVl/HD+Fi3TItGZpFH+88H7+QDTsrgBp01Sj40rVOSj9wFbNQZf/xWBQnbs20anXfPzNpTy OAtGgkD6hdCgkVuMQBTaRAEg8MpGvz+t9v9uo3XbcngMg/nEuQ0GPQ/AbgpGBntCU/iSXTBW6xguO jprN6FgFpJGYBVMTb5QP1aT3kFOObXIYHjf66xHr9g9ghKhclsJ93wRhrGPzSEBhm7vbtv1kDtd1I XnCP73Ffwj9IlymDHUwQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nnjaJ-00Ab1K-NL; Sun, 08 May 2022 16:16:39 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nnjaG-00Ab0J-Bc for linux-riscv@lists.infradead.org; Sun, 08 May 2022 16:16:38 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id EFA5C6121C; Sun, 8 May 2022 16:16:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1CDC1C385C0; Sun, 8 May 2022 16:16:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1652026595; bh=1D217r1rSI/uUeSIUljN5J553Ajo8zPgW0SHZ2rk+io=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Bkxkj8L1JiFS7MiCImm50YEZA/y0LoDC6dK+ZhYwc59l2AdRol9CFhGrkpuAUDYjy Wdv7f9x4kCNhOQdPCB4cKzmvR8VRc1tJchGSEuv5jGbQ2/8WhXcS3CBxGK07H1P2S0 feZH21gUL3s8D9XG0WkxWHeDVH2G5SlXZkg8nE1O+Wl5lAFCBfEhtsmKSCyMtarMww Y6MF9P2PeoJi937nqmfY8bazzEtLdNZ8/3bLh/+Wd2I2D0zhnaFhx6DQG/fF7KUW0y hK2FqBj+6hTibsMrhxPSE8UthdSkITPTWpFTtX8MJBE1il89sfJrAeHsnWFfBmWWLE zCQK2m8bJ9Xkw== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Alexandre Ghiti Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: [PATCH v2 2/4] riscv: introduce unified static key mechanism for CPU features Date: Mon, 9 May 2022 00:07:47 +0800 Message-Id: <20220508160749.984-3-jszhang@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220508160749.984-1-jszhang@kernel.org> References: <20220508160749.984-1-jszhang@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220508_091636_519809_C3C1FFF6 X-CRM114-Status: GOOD ( 25.87 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Currently, riscv has several features why may not be supported on all riscv platforms, for example, FPU, SV48 and so on. To support unified kernel Image style, we need to check whether the feature is suportted or not. If the check sits at hot code path, then performance will be impacted a lot. static key can be used to solve the issue. In the past FPU support has been converted to use static key mechanism. I believe we will have similar cases in the future. Similar as arm64 does(in fact, some code is borrowed from arm64), this patch tries to add an unified mechanism to use static keys for all the cpu features by implementing an array of default-false static keys and enabling them when detected. The cpus_have_*_cap() check uses the static keys if riscv_const_caps_ready is finalized, otherwise the compiler generates the bitmap test. Signed-off-by: Jisheng Zhang --- arch/riscv/Makefile | 3 + arch/riscv/include/asm/cpufeature.h | 94 +++++++++++++++++++++++++++++ arch/riscv/kernel/cpufeature.c | 23 +++++++ arch/riscv/tools/Makefile | 22 +++++++ arch/riscv/tools/cpucaps | 5 ++ arch/riscv/tools/gen-cpucaps.awk | 40 ++++++++++++ 6 files changed, 187 insertions(+) create mode 100644 arch/riscv/include/asm/cpufeature.h create mode 100644 arch/riscv/tools/Makefile create mode 100644 arch/riscv/tools/cpucaps create mode 100755 arch/riscv/tools/gen-cpucaps.awk diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 7d81102cffd4..f4df67369d84 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -154,3 +154,6 @@ PHONY += rv64_randconfig rv64_randconfig: $(Q)$(MAKE) KCONFIG_ALLCONFIG=$(srctree)/arch/riscv/configs/64-bit.config \ -f $(srctree)/Makefile randconfig + +archprepare: + $(Q)$(MAKE) $(build)=arch/riscv/tools kapi diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h new file mode 100644 index 000000000000..d80ddd2f3b49 --- /dev/null +++ b/arch/riscv/include/asm/cpufeature.h @@ -0,0 +1,94 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2014 Linaro Ltd. + * Copyright (C) 2022 Jisheng Zhang + */ + +#ifndef __ASM_CPUFEATURE_H +#define __ASM_CPUFEATURE_H + +#include + +#include +#include +#include + +extern DECLARE_BITMAP(cpu_hwcaps, RISCV_NCAPS); +extern struct static_key_false cpu_hwcap_keys[RISCV_NCAPS]; +extern struct static_key_false riscv_const_caps_ready; + +static __always_inline bool system_capabilities_finalized(void) +{ + return static_branch_likely(&riscv_const_caps_ready); +} + +/* + * Test for a capability with a runtime check. + * + * Before the capability is detected, this returns false. + */ +static inline bool cpus_have_cap(unsigned int num) +{ + if (num >= RISCV_NCAPS) + return false; + return test_bit(num, cpu_hwcaps); +} + +/* + * Test for a capability without a runtime check. + * + * Before capabilities are finalized, this returns false. + * After capabilities are finalized, this is patched to avoid a runtime check. + * + * @num must be a compile-time constant. + */ +static __always_inline bool __cpus_have_const_cap(int num) +{ + if (num >= RISCV_NCAPS) + return false; + return static_branch_unlikely(&cpu_hwcap_keys[num]); +} + +/* + * Test for a capability without a runtime check. + * + * Before capabilities are finalized, this will BUG(). + * After capabilities are finalized, this is patched to avoid a runtime check. + * + * @num must be a compile-time constant. + */ +static __always_inline bool cpus_have_final_cap(int num) +{ + if (system_capabilities_finalized()) + return __cpus_have_const_cap(num); + else + BUG(); +} + +/* + * Test for a capability, possibly with a runtime check. + * + * Before capabilities are finalized, this behaves as cpus_have_cap(). + * After capabilities are finalized, this is patched to avoid a runtime check. + * + * @num must be a compile-time constant. + */ +static __always_inline bool cpus_have_const_cap(int num) +{ + if (system_capabilities_finalized()) + return __cpus_have_const_cap(num); + else + return cpus_have_cap(num); +} + +static inline void cpus_set_cap(unsigned int num) +{ + if (num >= RISCV_NCAPS) { + pr_warn("Attempt to set an illegal CPU capability (%d >= %d)\n", + num, RISCV_NCAPS); + } else { + __set_bit(num, cpu_hwcaps); + } +} + +#endif diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 1b2d42d7f589..e6c72cad0c1c 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -25,6 +26,15 @@ static DECLARE_BITMAP(riscv_isa, RISCV_ISA_EXT_MAX) __read_mostly; __ro_after_init DEFINE_STATIC_KEY_FALSE(cpu_hwcap_fpu); #endif +DECLARE_BITMAP(cpu_hwcaps, RISCV_NCAPS); +EXPORT_SYMBOL(cpu_hwcaps); + +DEFINE_STATIC_KEY_ARRAY_FALSE(cpu_hwcap_keys, RISCV_NCAPS); +EXPORT_SYMBOL(cpu_hwcap_keys); + +DEFINE_STATIC_KEY_FALSE(riscv_const_caps_ready); +EXPORT_SYMBOL(riscv_const_caps_ready); + /** * riscv_isa_extension_base() - Get base extension word * @@ -62,6 +72,17 @@ bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, int bit) } EXPORT_SYMBOL_GPL(__riscv_isa_extension_available); +static void __init enable_cpu_capabilities(void) +{ + int i; + + for (i = 0; i < RISCV_NCAPS; i++) { + if (!cpus_have_cap(i)) + continue; + static_branch_enable(&cpu_hwcap_keys[i]); + } +} + void __init riscv_fill_hwcap(void) { struct device_node *node; @@ -236,4 +257,6 @@ void __init riscv_fill_hwcap(void) if (elf_hwcap & (COMPAT_HWCAP_ISA_F | COMPAT_HWCAP_ISA_D)) static_branch_enable(&cpu_hwcap_fpu); #endif + enable_cpu_capabilities(); + static_branch_enable(&riscv_const_caps_ready); } diff --git a/arch/riscv/tools/Makefile b/arch/riscv/tools/Makefile new file mode 100644 index 000000000000..932b4fe5c768 --- /dev/null +++ b/arch/riscv/tools/Makefile @@ -0,0 +1,22 @@ +# SPDX-License-Identifier: GPL-2.0 + +gen := arch/$(ARCH)/include/generated +kapi := $(gen)/asm + +kapi-hdrs-y := $(kapi)/cpucaps.h + +targets += $(addprefix ../../../,$(gen-y) $(kapi-hdrs-y)) + +PHONY += kapi + +kapi: $(kapi-hdrs-y) $(gen-y) + +# Create output directory if not already present +_dummy := $(shell [ -d '$(kapi)' ] || mkdir -p '$(kapi)') + +quiet_cmd_gen_cpucaps = GEN $@ + cmd_gen_cpucaps = mkdir -p $(dir $@) && \ + $(AWK) -f $(filter-out $(PHONY),$^) > $@ + +$(kapi)/cpucaps.h: $(src)/gen-cpucaps.awk $(src)/cpucaps FORCE + $(call if_changed,gen_cpucaps) diff --git a/arch/riscv/tools/cpucaps b/arch/riscv/tools/cpucaps new file mode 100644 index 000000000000..cb1ff2747859 --- /dev/null +++ b/arch/riscv/tools/cpucaps @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Internal CPU capabilities constants, keep this list sorted + +HAS_NO_FPU diff --git a/arch/riscv/tools/gen-cpucaps.awk b/arch/riscv/tools/gen-cpucaps.awk new file mode 100755 index 000000000000..52a1e1b064ad --- /dev/null +++ b/arch/riscv/tools/gen-cpucaps.awk @@ -0,0 +1,40 @@ +#!/bin/awk -f +# SPDX-License-Identifier: GPL-2.0 +# gen-cpucaps.awk: riscv cpucaps header generator +# +# Usage: awk -f gen-cpucaps.awk cpucaps.txt + +# Log an error and terminate +function fatal(msg) { + print "Error at line " NR ": " msg > "/dev/stderr" + exit 1 +} + +# skip blank lines and comment lines +/^$/ { next } +/^#/ { next } + +BEGIN { + print "#ifndef __ASM_CPUCAPS_H" + print "#define __ASM_CPUCAPS_H" + print "" + print "/* Generated file - do not edit */" + cap_num = 0 + print "" +} + +/^[vA-Z0-9_]+$/ { + printf("#define RISCV_%-30s\t%d\n", $0, cap_num++) + next +} + +END { + printf("#define RISCV_NCAPS\t\t\t\t%d\n", cap_num) + print "" + print "#endif /* __ASM_CPUCAPS_H */" +} + +# Any lines not handled by previous rules are unexpected +{ + fatal("unhandled statement") +} From patchwork Sun May 8 16:07:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 12842483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C99A8C433F5 for ; Sun, 8 May 2022 16:16:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OJL8E4bZ+oQdEibxVyxKwFkWU1DqxGZE4cd6VYIreHQ=; b=z12fo0tibUlw0d 9DGuiC7tZ8U87tC//l4xlPfb3apfkZyh7ZAhvFzitA2cP5BBgL4D4ZPTAEFIXec1nSk8Vw4ry/wmu HL/4FPxqajzj+v6R3ZgqI/gAdWnQO6ep1KbimINk7YDYiZkpbv/8fJYqAT1mc/DZPP+CHbSvxChHS uthpdj5r7TBWy9zANhmsvRCmVd0cAHXnmJpdI3kXrVTafn6m2CtGLTb9y4+UK7YCklSLFuxq1DTal 0GN+K8CaPCBVh/6UL79zrh5z/0Umnfg6Wm2Yl9aPbOieXn74CGkDIVVfjhQXKWKxnLSK+YaukMZ9t D6ZHtvg2t5eyxylff21g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nnjaQ-00Ab3V-Al; Sun, 08 May 2022 16:16:46 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nnjaN-00Ab2L-NF for linux-riscv@lists.infradead.org; Sun, 08 May 2022 16:16:45 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4AC4B60F60; Sun, 8 May 2022 16:16:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E2514C385AC; Sun, 8 May 2022 16:16:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1652026602; bh=aSM+qwsoybBfG7hu+B5KpBzvce1SYI7kPkbgAdNlN7o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Wp0TGBxIHnaLHDpUy5porx22k0sHC+tq4kGeDQZDt0zliePYfQGhXAr9y58Wt2f6D 1g6ToqH1B2Lck1CtURjw6B8XR08DGyAT22UdbU9NMb1Dh5tRoU4Sx64wjpPeJV9N/q OA+4hUM9DVB7D9TmpcjtmZJx92JFEHWHvTs9lYlN0mu5oGGkYtckfc+yDaesLfCV8y csR1+KlsdBZAm0vsEGMYBPFtF0K3hABUwFIw+yuaUNe0DLrAsDo7lu4wODBF/ZQlXi E9hGVIpyVGzb59SKNzWhk43ymCQcrgcDaael0Ya3+vKm23ph83QoJLpnFV6eBVyo/R v8GDvMN8x9VLQ== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Alexandre Ghiti Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: [PATCH v2 3/4] riscv: replace has_fpu() with system_supports_fpu() Date: Mon, 9 May 2022 00:07:48 +0800 Message-Id: <20220508160749.984-4-jszhang@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220508160749.984-1-jszhang@kernel.org> References: <20220508160749.984-1-jszhang@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220508_091643_874844_AAB4B8A6 X-CRM114-Status: GOOD ( 17.01 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This is to use the unified cpus_have_{final|const}_cap() instead of putting static key related here and there. Signed-off-by: Jisheng Zhang --- arch/riscv/include/asm/cpufeature.h | 5 +++++ arch/riscv/include/asm/switch_to.h | 9 ++------- arch/riscv/kernel/cpufeature.c | 8 ++------ arch/riscv/kernel/process.c | 2 +- arch/riscv/kernel/signal.c | 4 ++-- 5 files changed, 12 insertions(+), 16 deletions(-) diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index d80ddd2f3b49..634a653c7fa2 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -91,4 +91,9 @@ static inline void cpus_set_cap(unsigned int num) } } +static inline bool system_supports_fpu(void) +{ + return IS_ENABLED(CONFIG_FPU) && !cpus_have_final_cap(RISCV_HAS_NO_FPU); +} + #endif diff --git a/arch/riscv/include/asm/switch_to.h b/arch/riscv/include/asm/switch_to.h index 0a3f4f95c555..362cb18d12d5 100644 --- a/arch/riscv/include/asm/switch_to.h +++ b/arch/riscv/include/asm/switch_to.h @@ -8,6 +8,7 @@ #include #include +#include #include #include #include @@ -56,13 +57,7 @@ static inline void __switch_to_aux(struct task_struct *prev, fstate_restore(next, task_pt_regs(next)); } -extern struct static_key_false cpu_hwcap_fpu; -static __always_inline bool has_fpu(void) -{ - return static_branch_likely(&cpu_hwcap_fpu); -} #else -static __always_inline bool has_fpu(void) { return false; } #define fstate_save(task, regs) do { } while (0) #define fstate_restore(task, regs) do { } while (0) #define __switch_to_aux(__prev, __next) do { } while (0) @@ -75,7 +70,7 @@ extern struct task_struct *__switch_to(struct task_struct *, do { \ struct task_struct *__prev = (prev); \ struct task_struct *__next = (next); \ - if (has_fpu()) \ + if (system_supports_fpu()) \ __switch_to_aux(__prev, __next); \ ((last) = __switch_to(__prev, __next)); \ } while (0) diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index e6c72cad0c1c..1edf3c3f8f62 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -22,10 +22,6 @@ unsigned long elf_hwcap __read_mostly; /* Host ISA bitmap */ static DECLARE_BITMAP(riscv_isa, RISCV_ISA_EXT_MAX) __read_mostly; -#ifdef CONFIG_FPU -__ro_after_init DEFINE_STATIC_KEY_FALSE(cpu_hwcap_fpu); -#endif - DECLARE_BITMAP(cpu_hwcaps, RISCV_NCAPS); EXPORT_SYMBOL(cpu_hwcaps); @@ -254,8 +250,8 @@ void __init riscv_fill_hwcap(void) pr_info("riscv: ELF capabilities %s\n", print_str); #ifdef CONFIG_FPU - if (elf_hwcap & (COMPAT_HWCAP_ISA_F | COMPAT_HWCAP_ISA_D)) - static_branch_enable(&cpu_hwcap_fpu); + if (!(elf_hwcap & (COMPAT_HWCAP_ISA_F | COMPAT_HWCAP_ISA_D))) + cpus_set_cap(RISCV_HAS_NO_FPU); #endif enable_cpu_capabilities(); static_branch_enable(&riscv_const_caps_ready); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 504b496787aa..c9cd0b42299e 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -88,7 +88,7 @@ void start_thread(struct pt_regs *regs, unsigned long pc, unsigned long sp) { regs->status = SR_PIE; - if (has_fpu()) { + if (system_supports_fpu()) { regs->status |= SR_FS_INITIAL; /* * Restore the initial value to the FP register diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 9f4e59f80551..96aa593a989e 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -90,7 +90,7 @@ static long restore_sigcontext(struct pt_regs *regs, /* sc_regs is structured the same as the start of pt_regs */ err = __copy_from_user(regs, &sc->sc_regs, sizeof(sc->sc_regs)); /* Restore the floating-point state. */ - if (has_fpu()) + if (system_supports_fpu()) err |= restore_fp_state(regs, &sc->sc_fpregs); return err; } @@ -143,7 +143,7 @@ static long setup_sigcontext(struct rt_sigframe __user *frame, /* sc_regs is structured the same as the start of pt_regs */ err = __copy_to_user(&sc->sc_regs, regs, sizeof(sc->sc_regs)); /* Save the floating-point state. */ - if (has_fpu()) + if (system_supports_fpu()) err |= save_fp_state(regs, &sc->sc_fpregs); return err; } From patchwork Sun May 8 16:07:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 12842484 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B5C61C433F5 for ; Sun, 8 May 2022 16:17:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=H60QhSrcJaK7IM+r1zy8IQbbl43v0M5lz/R1sHm1vsw=; b=kAM0pdMbylN/Xw MnQAqf1kPetkc1sV9dNsUGzn+kGNVemMv6vbyJzaylN+OeWru6WqBzA83iSyEAR7Y4YvUnv7sgAxr l4/mtBGFKbe/0ane93V7+uM5H7ZtIWFQ9NV+g/yZohZn66u/BsuWo16TYxmMuWT3bEq0m+7s4PCGU IaGzcsojTFUzTYwhkh3EHoeRtxI9Ml1OWDfuQrEnn22emxiIhIVCZEWjJezrRSLWJOGH5pVh9ZMYu n2Cxeef7sPGwLCy8oIll3WuFLtbP7KIomDd/NiXKfw6cquMqTFTtw3crXWeSf4E/hMnIsnI6tuQsi 1PcyZElGvEP2Zw6t8XBw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nnjaa-00Ab8C-Ot; Sun, 08 May 2022 16:16:56 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nnjaW-00Ab6n-ID for linux-riscv@lists.infradead.org; Sun, 08 May 2022 16:16:55 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1E2EF60F60; Sun, 8 May 2022 16:16:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3EA36C385AF; Sun, 8 May 2022 16:16:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1652026611; bh=s+Wtb80a0asBOvxsopCXgpsx46DIS6cRWDYhtKPxr+o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WkoeFI5VxGNdiR6dEMJprMVwqcPIjixFK0LGBNa2FnQ3UrE07s0u9P3DP1rvrdZeP MjVyztSXMpr6XlyKbKIKAZk9EAF6Q3Z58kbIRrsEQUfXTxzIsZd133tcgFLyPIamdB lCkHlPr/0Ya6r/JV2T7J0td6fuD5Ksl6lRHCvrghv2nkn4DknPDk0S9NTKfIFvFO8t HnxZy4EQRE2TVtaARRaJsT60uO+AyOS2frIwmINVmwesgrQJ2DGDLBiJE23xYJeNe6 ha9WEF64F7Ayrkszj/m3S+GY9JmDLBiJJlBl9EUogF3g8Xtujc2g+cTGrxXBbRgsux B0XAhQ86+iykg== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Alexandre Ghiti Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: [PATCH v2 4/4] riscv: convert pgtable_l4|[l5]_enabled to static key Date: Mon, 9 May 2022 00:07:49 +0800 Message-Id: <20220508160749.984-5-jszhang@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220508160749.984-1-jszhang@kernel.org> References: <20220508160749.984-1-jszhang@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220508_091652_727930_A7BE8C58 X-CRM114-Status: GOOD ( 17.53 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On a specific HW platform, pgtable_l4|[l5]_enabled won't change after boot, and the check sits at hot code path, this characteristic makes it suitable for optimization with static key. Signed-off-by: Jisheng Zhang --- arch/riscv/include/asm/cpufeature.h | 11 +++++++ arch/riscv/include/asm/pgalloc.h | 16 +++++----- arch/riscv/include/asm/pgtable-64.h | 40 ++++++++++++------------- arch/riscv/include/asm/pgtable.h | 5 ++-- arch/riscv/kernel/cpu.c | 4 +-- arch/riscv/mm/init.c | 46 +++++++++++++---------------- arch/riscv/mm/kasan_init.c | 16 +++++----- arch/riscv/tools/cpucaps | 2 ++ 8 files changed, 73 insertions(+), 67 deletions(-) diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 634a653c7fa2..a51f2602a0e3 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -96,4 +96,15 @@ static inline bool system_supports_fpu(void) return IS_ENABLED(CONFIG_FPU) && !cpus_have_final_cap(RISCV_HAS_NO_FPU); } +static inline bool system_supports_sv48(void) +{ + return IS_ENABLED(CONFIG_64BIT) && !IS_ENABLED(CONFIG_XIP_KERNEL) && + !cpus_have_const_cap(RISCV_HAS_NO_SV48); +} + +static inline bool system_supports_sv57(void) +{ + return IS_ENABLED(CONFIG_64BIT) && !IS_ENABLED(CONFIG_XIP_KERNEL) && + !cpus_have_const_cap(RISCV_HAS_NO_SV57); +} #endif diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 947f23d7b6af..f49233ca696a 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -41,7 +41,7 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4d, pud_t *pud) { - if (pgtable_l4_enabled) { + if (system_supports_sv48()) { unsigned long pfn = virt_to_pfn(pud); set_p4d(p4d, __p4d((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); @@ -51,7 +51,7 @@ static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4d, pud_t *pud) static inline void p4d_populate_safe(struct mm_struct *mm, p4d_t *p4d, pud_t *pud) { - if (pgtable_l4_enabled) { + if (system_supports_sv48()) { unsigned long pfn = virt_to_pfn(pud); set_p4d_safe(p4d, @@ -61,7 +61,7 @@ static inline void p4d_populate_safe(struct mm_struct *mm, p4d_t *p4d, static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4d) { - if (pgtable_l5_enabled) { + if (system_supports_sv57()) { unsigned long pfn = virt_to_pfn(p4d); set_pgd(pgd, __pgd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); @@ -71,7 +71,7 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4d) static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4d) { - if (pgtable_l5_enabled) { + if (system_supports_sv57()) { unsigned long pfn = virt_to_pfn(p4d); set_pgd_safe(pgd, @@ -82,7 +82,7 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, #define pud_alloc_one pud_alloc_one static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) { - if (pgtable_l4_enabled) + if (system_supports_sv48()) return __pud_alloc_one(mm, addr); return NULL; @@ -91,7 +91,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) #define pud_free pud_free static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (pgtable_l4_enabled) + if (system_supports_sv48()) __pud_free(mm, pud); } @@ -100,7 +100,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - if (pgtable_l5_enabled) { + if (system_supports_sv57()) { gfp_t gfp = GFP_PGTABLE_USER; if (mm == &init_mm) @@ -120,7 +120,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) #define p4d_free p4d_free static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (pgtable_l5_enabled) + if (system_supports_sv57()) __p4d_free(mm, p4d); } diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 7e246e9f8d70..9ee4abf0f528 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -7,17 +7,15 @@ #define _ASM_RISCV_PGTABLE_64_H #include - -extern bool pgtable_l4_enabled; -extern bool pgtable_l5_enabled; +#include #define PGDIR_SHIFT_L3 30 #define PGDIR_SHIFT_L4 39 #define PGDIR_SHIFT_L5 48 #define PGDIR_SIZE_L3 (_AC(1, UL) << PGDIR_SHIFT_L3) -#define PGDIR_SHIFT (pgtable_l5_enabled ? PGDIR_SHIFT_L5 : \ - (pgtable_l4_enabled ? PGDIR_SHIFT_L4 : PGDIR_SHIFT_L3)) +#define PGDIR_SHIFT (system_supports_sv57() ? PGDIR_SHIFT_L5 : \ + (system_supports_sv48() ? PGDIR_SHIFT_L4 : PGDIR_SHIFT_L3)) /* Size of region mapped by a page global directory */ #define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT) #define PGDIR_MASK (~(PGDIR_SIZE - 1)) @@ -119,7 +117,7 @@ static inline struct page *pud_page(pud_t pud) #define mm_p4d_folded mm_p4d_folded static inline bool mm_p4d_folded(struct mm_struct *mm) { - if (pgtable_l5_enabled) + if (system_supports_sv57()) return false; return true; @@ -128,7 +126,7 @@ static inline bool mm_p4d_folded(struct mm_struct *mm) #define mm_pud_folded mm_pud_folded static inline bool mm_pud_folded(struct mm_struct *mm) { - if (pgtable_l4_enabled) + if (system_supports_sv48()) return false; return true; @@ -159,7 +157,7 @@ static inline unsigned long _pmd_pfn(pmd_t pmd) static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) { - if (pgtable_l4_enabled) + if (system_supports_sv48()) *p4dp = p4d; else set_pud((pud_t *)p4dp, (pud_t){ p4d_val(p4d) }); @@ -167,7 +165,7 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) static inline int p4d_none(p4d_t p4d) { - if (pgtable_l4_enabled) + if (system_supports_sv48()) return (p4d_val(p4d) == 0); return 0; @@ -175,7 +173,7 @@ static inline int p4d_none(p4d_t p4d) static inline int p4d_present(p4d_t p4d) { - if (pgtable_l4_enabled) + if (system_supports_sv48()) return (p4d_val(p4d) & _PAGE_PRESENT); return 1; @@ -183,7 +181,7 @@ static inline int p4d_present(p4d_t p4d) static inline int p4d_bad(p4d_t p4d) { - if (pgtable_l4_enabled) + if (system_supports_sv48()) return !p4d_present(p4d); return 0; @@ -191,7 +189,7 @@ static inline int p4d_bad(p4d_t p4d) static inline void p4d_clear(p4d_t *p4d) { - if (pgtable_l4_enabled) + if (system_supports_sv48()) set_p4d(p4d, __p4d(0)); } @@ -207,7 +205,7 @@ static inline unsigned long _p4d_pfn(p4d_t p4d) static inline pud_t *p4d_pgtable(p4d_t p4d) { - if (pgtable_l4_enabled) + if (system_supports_sv48()) return (pud_t *)pfn_to_virt(p4d_val(p4d) >> _PAGE_PFN_SHIFT); return (pud_t *)pud_pgtable((pud_t) { p4d_val(p4d) }); @@ -224,7 +222,7 @@ static inline struct page *p4d_page(p4d_t p4d) #define pud_offset pud_offset static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) { - if (pgtable_l4_enabled) + if (system_supports_sv48()) return p4d_pgtable(*p4d) + pud_index(address); return (pud_t *)p4d; @@ -232,7 +230,7 @@ static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) { - if (pgtable_l5_enabled) + if (system_supports_sv57()) *pgdp = pgd; else set_p4d((p4d_t *)pgdp, (p4d_t){ pgd_val(pgd) }); @@ -240,7 +238,7 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) static inline int pgd_none(pgd_t pgd) { - if (pgtable_l5_enabled) + if (system_supports_sv57()) return (pgd_val(pgd) == 0); return 0; @@ -248,7 +246,7 @@ static inline int pgd_none(pgd_t pgd) static inline int pgd_present(pgd_t pgd) { - if (pgtable_l5_enabled) + if (system_supports_sv57()) return (pgd_val(pgd) & _PAGE_PRESENT); return 1; @@ -256,7 +254,7 @@ static inline int pgd_present(pgd_t pgd) static inline int pgd_bad(pgd_t pgd) { - if (pgtable_l5_enabled) + if (system_supports_sv57()) return !pgd_present(pgd); return 0; @@ -264,13 +262,13 @@ static inline int pgd_bad(pgd_t pgd) static inline void pgd_clear(pgd_t *pgd) { - if (pgtable_l5_enabled) + if (system_supports_sv57()) set_pgd(pgd, __pgd(0)); } static inline p4d_t *pgd_pgtable(pgd_t pgd) { - if (pgtable_l5_enabled) + if (system_supports_sv57()) return (p4d_t *)pfn_to_virt(pgd_val(pgd) >> _PAGE_PFN_SHIFT); return (p4d_t *)p4d_pgtable((p4d_t) { pgd_val(pgd) }); @@ -288,7 +286,7 @@ static inline struct page *pgd_page(pgd_t pgd) #define p4d_offset p4d_offset static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) { - if (pgtable_l5_enabled) + if (system_supports_sv57()) return pgd_pgtable(*pgd) + p4d_index(address); return (p4d_t *)pgd; diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 046b44225623..ef2a1654100a 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -63,8 +63,8 @@ * position vmemmap directly below the VMALLOC region. */ #ifdef CONFIG_64BIT -#define VA_BITS (pgtable_l5_enabled ? \ - 57 : (pgtable_l4_enabled ? 48 : 39)) +#define VA_BITS (system_supports_sv57() ? \ + 57 : (system_supports_sv48() ? 48 : 39)) #else #define VA_BITS 32 #endif @@ -738,7 +738,6 @@ extern uintptr_t _dtb_early_pa; #define dtb_early_pa _dtb_early_pa #endif /* CONFIG_XIP_KERNEL */ extern u64 satp_mode; -extern bool pgtable_l4_enabled; void paging_init(void); void misc_mem_init(void); diff --git a/arch/riscv/kernel/cpu.c b/arch/riscv/kernel/cpu.c index ccb617791e56..c8f3989b08f3 100644 --- a/arch/riscv/kernel/cpu.c +++ b/arch/riscv/kernel/cpu.c @@ -141,9 +141,9 @@ static void print_mmu(struct seq_file *f) #if defined(CONFIG_32BIT) strncpy(sv_type, "sv32", 5); #elif defined(CONFIG_64BIT) - if (pgtable_l5_enabled) + if (system_supports_sv57()) strncpy(sv_type, "sv57", 5); - else if (pgtable_l4_enabled) + else if (system_supports_sv48()) strncpy(sv_type, "sv48", 5); else strncpy(sv_type, "sv39", 5); diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 5f3f26dd9f21..b6a59a5d1a7f 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -21,6 +21,7 @@ #include #include +#include #include #include #include @@ -44,11 +45,6 @@ u64 satp_mode __ro_after_init = SATP_MODE_32; #endif EXPORT_SYMBOL(satp_mode); -bool pgtable_l4_enabled = IS_ENABLED(CONFIG_64BIT) && !IS_ENABLED(CONFIG_XIP_KERNEL); -bool pgtable_l5_enabled = IS_ENABLED(CONFIG_64BIT) && !IS_ENABLED(CONFIG_XIP_KERNEL); -EXPORT_SYMBOL(pgtable_l4_enabled); -EXPORT_SYMBOL(pgtable_l5_enabled); - phys_addr_t phys_ram_base __ro_after_init; EXPORT_SYMBOL(phys_ram_base); @@ -555,26 +551,26 @@ static void __init create_p4d_mapping(p4d_t *p4dp, } #define pgd_next_t p4d_t -#define alloc_pgd_next(__va) (pgtable_l5_enabled ? \ - pt_ops.alloc_p4d(__va) : (pgtable_l4_enabled ? \ +#define alloc_pgd_next(__va) (system_supports_sv57() ? \ + pt_ops.alloc_p4d(__va) : (system_supports_sv48() ? \ pt_ops.alloc_pud(__va) : pt_ops.alloc_pmd(__va))) -#define get_pgd_next_virt(__pa) (pgtable_l5_enabled ? \ - pt_ops.get_p4d_virt(__pa) : (pgd_next_t *)(pgtable_l4_enabled ? \ +#define get_pgd_next_virt(__pa) (system_supports_sv57() ? \ + pt_ops.get_p4d_virt(__pa) : (pgd_next_t *)(system_supports_sv48() ? \ pt_ops.get_pud_virt(__pa) : (pud_t *)pt_ops.get_pmd_virt(__pa))) #define create_pgd_next_mapping(__nextp, __va, __pa, __sz, __prot) \ - (pgtable_l5_enabled ? \ + (system_supports_sv57() ? \ create_p4d_mapping(__nextp, __va, __pa, __sz, __prot) : \ - (pgtable_l4_enabled ? \ + (system_supports_sv48() ? \ create_pud_mapping((pud_t *)__nextp, __va, __pa, __sz, __prot) : \ create_pmd_mapping((pmd_t *)__nextp, __va, __pa, __sz, __prot))) -#define fixmap_pgd_next (pgtable_l5_enabled ? \ - (uintptr_t)fixmap_p4d : (pgtable_l4_enabled ? \ +#define fixmap_pgd_next (system_supports_sv57() ? \ + (uintptr_t)fixmap_p4d : (system_supports_sv48() ? \ (uintptr_t)fixmap_pud : (uintptr_t)fixmap_pmd)) -#define trampoline_pgd_next (pgtable_l5_enabled ? \ - (uintptr_t)trampoline_p4d : (pgtable_l4_enabled ? \ +#define trampoline_pgd_next (system_supports_sv57() ? \ + (uintptr_t)trampoline_p4d : (system_supports_sv48() ? \ (uintptr_t)trampoline_pud : (uintptr_t)trampoline_pmd)) -#define early_dtb_pgd_next (pgtable_l5_enabled ? \ - (uintptr_t)early_dtb_p4d : (pgtable_l4_enabled ? \ +#define early_dtb_pgd_next (system_supports_sv57() ? \ + (uintptr_t)early_dtb_p4d : (system_supports_sv48() ? \ (uintptr_t)early_dtb_pud : (uintptr_t)early_dtb_pmd)) #else #define pgd_next_t pte_t @@ -680,14 +676,14 @@ static __init pgprot_t pgprot_from_va(uintptr_t va) #ifdef CONFIG_64BIT static void __init disable_pgtable_l5(void) { - pgtable_l5_enabled = false; + cpus_set_cap(RISCV_HAS_NO_SV57); kernel_map.page_offset = PAGE_OFFSET_L4; satp_mode = SATP_MODE_48; } static void __init disable_pgtable_l4(void) { - pgtable_l4_enabled = false; + cpus_set_cap(RISCV_HAS_NO_SV48); kernel_map.page_offset = PAGE_OFFSET_L3; satp_mode = SATP_MODE_39; } @@ -816,11 +812,11 @@ static void __init create_fdt_early_page_table(pgd_t *pgdir, uintptr_t dtb_pa) PGDIR_SIZE, IS_ENABLED(CONFIG_64BIT) ? PAGE_TABLE : PAGE_KERNEL); - if (pgtable_l5_enabled) + if (system_supports_sv57()) create_p4d_mapping(early_dtb_p4d, DTB_EARLY_BASE_VA, (uintptr_t)early_dtb_pud, P4D_SIZE, PAGE_TABLE); - if (pgtable_l4_enabled) + if (system_supports_sv48()) create_pud_mapping(early_dtb_pud, DTB_EARLY_BASE_VA, (uintptr_t)early_dtb_pmd, PUD_SIZE, PAGE_TABLE); @@ -961,11 +957,11 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) #ifndef __PAGETABLE_PMD_FOLDED /* Setup fixmap P4D and PUD */ - if (pgtable_l5_enabled) + if (system_supports_sv57()) create_p4d_mapping(fixmap_p4d, FIXADDR_START, (uintptr_t)fixmap_pud, P4D_SIZE, PAGE_TABLE); /* Setup fixmap PUD and PMD */ - if (pgtable_l4_enabled) + if (system_supports_sv48()) create_pud_mapping(fixmap_pud, FIXADDR_START, (uintptr_t)fixmap_pmd, PUD_SIZE, PAGE_TABLE); create_pmd_mapping(fixmap_pmd, FIXADDR_START, @@ -973,10 +969,10 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) /* Setup trampoline PGD and PMD */ create_pgd_mapping(trampoline_pg_dir, kernel_map.virt_addr, trampoline_pgd_next, PGDIR_SIZE, PAGE_TABLE); - if (pgtable_l5_enabled) + if (system_supports_sv57()) create_p4d_mapping(trampoline_p4d, kernel_map.virt_addr, (uintptr_t)trampoline_pud, P4D_SIZE, PAGE_TABLE); - if (pgtable_l4_enabled) + if (system_supports_sv48()) create_pud_mapping(trampoline_pud, kernel_map.virt_addr, (uintptr_t)trampoline_pmd, PUD_SIZE, PAGE_TABLE); #ifdef CONFIG_XIP_KERNEL diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index a22e418dbd82..7b662661f7a9 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -209,15 +209,15 @@ static void __init kasan_populate_p4d(pgd_t *pgd, set_pgd(pgd, pfn_pgd(PFN_DOWN(__pa(base_p4d)), PAGE_TABLE)); } -#define kasan_early_shadow_pgd_next (pgtable_l5_enabled ? \ +#define kasan_early_shadow_pgd_next (system_supports_sv57() ? \ (uintptr_t)kasan_early_shadow_p4d : \ - (pgtable_l4_enabled ? \ + (system_supports_sv48() ? \ (uintptr_t)kasan_early_shadow_pud : \ (uintptr_t)kasan_early_shadow_pmd)) #define kasan_populate_pgd_next(pgdp, vaddr, next, early) \ - (pgtable_l5_enabled ? \ + (system_supports_sv57() ? \ kasan_populate_p4d(pgdp, vaddr, next, early) : \ - (pgtable_l4_enabled ? \ + (system_supports_sv48() ? \ kasan_populate_pud(pgdp, vaddr, next, early) : \ kasan_populate_pmd((pud_t *)pgdp, vaddr, next))) @@ -274,7 +274,7 @@ asmlinkage void __init kasan_early_init(void) (__pa((uintptr_t)kasan_early_shadow_pte)), PAGE_TABLE)); - if (pgtable_l4_enabled) { + if (system_supports_sv48()) { for (i = 0; i < PTRS_PER_PUD; ++i) set_pud(kasan_early_shadow_pud + i, pfn_pud(PFN_DOWN @@ -282,7 +282,7 @@ asmlinkage void __init kasan_early_init(void) PAGE_TABLE)); } - if (pgtable_l5_enabled) { + if (system_supports_sv57()) { for (i = 0; i < PTRS_PER_P4D; ++i) set_p4d(kasan_early_shadow_p4d + i, pfn_p4d(PFN_DOWN @@ -393,9 +393,9 @@ static void __init kasan_shallow_populate_p4d(pgd_t *pgdp, } #define kasan_shallow_populate_pgd_next(pgdp, vaddr, next) \ - (pgtable_l5_enabled ? \ + (system_supports_sv57() ? \ kasan_shallow_populate_p4d(pgdp, vaddr, next) : \ - (pgtable_l4_enabled ? \ + (system_supports_sv48() ? \ kasan_shallow_populate_pud(pgdp, vaddr, next) : \ kasan_shallow_populate_pmd(pgdp, vaddr, next))) diff --git a/arch/riscv/tools/cpucaps b/arch/riscv/tools/cpucaps index cb1ff2747859..0b9e19ec8371 100644 --- a/arch/riscv/tools/cpucaps +++ b/arch/riscv/tools/cpucaps @@ -3,3 +3,5 @@ # Internal CPU capabilities constants, keep this list sorted HAS_NO_FPU +HAS_NO_SV48 +HAS_NO_SV57