From patchwork Tue Jan 31 09:23:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122592 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C6FBC636CC for ; Tue, 31 Jan 2023 09:27:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=h8LzIi0GFLWLH1huwWH97VcvhEPce9MYNrR/665lX9s=; b=wzpTCd3z7iULUA YxVk0PdkedoaaEInT0eDplJVc2zZ7/aSVrg2gTsEAStV8npI3vwhgGU6m+FvjbzUPdjwk60Ugj14f SaTPiM+xqtFXU7bLWPPdi8T0fdjfJUKwlzKVL2b/P1J4tON8wOKKpm9/ZnoRZpS0i7I48xp+sS1/W JiZnEZc5pFZA8SWV/SA8IJb4qCYePg2+Tq1Ngfb2Rv9XgQaLQftsfPSKnuKdiFtk5ARvsyNakAgbd OFywx1HzqUIOfT0Qxkt7JfMEBMNR03FswmppRBC++JB9b9jm3PcXpn/51nxUoOb9hJPaG5R+Vfjs3 R1sR6XHdPSY9HdvT+uOg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmuT-006ysA-IY; Tue, 31 Jan 2023 09:26:37 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtT-006yPL-5A for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:25:37 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 82511B81A87; Tue, 31 Jan 2023 09:25:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 34552C4339B; Tue, 31 Jan 2023 09:25:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675157132; bh=t4xEJAO5BfLuaX8mjpeucO4oU7Eyo3etANQIAyyRnTk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LTyqvKlSOe2+bczruOsTReb9ysevGCLqjz0Y3HaTYUzbN94xU4XsSvSSyLxRakQDi f+NFJFahx0Wgr8i+XhEjP+41SUFBH/+koahZ+/ObnlDxfb9tdxZkY2KP74jFT5VZbF iRD2rbxXDUwSLp1UBDvImhxeAftXKhI7X7cGICAGbAk1YFB7sJvR/2oYjE29v/pe9c 835I/BYID6gNRwJ7ZJWBy0jh1OTFWIfP7+C48QXejXYFWMXLVSsYnmlKq4t43YmfIX UY7dCIFU6NJJ1o/3QsDZab7t2kEjbs+1EsZHWezw83GgBSAQiRDdO6XxPjSP9XjrgY b3+TyJHZ/NA5g== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtN-0067U2-T4; Tue, 31 Jan 2023 09:25:29 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 01/69] arm64: Add ARM64_HAS_NESTED_VIRT cpufeature Date: Tue, 31 Jan 2023 09:23:56 +0000 Message-Id: <20230131092504.2880505-2-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_012535_538617_F5BFE20C X-CRM114-Status: GOOD ( 21.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim Add a new ARM64_HAS_NESTED_VIRT feature to indicate that the CPU has the ARMv8.3 nested virtualization capability, together with the 'kvm-arm.mode=nested' command line option. This will be used to support nested virtualization in KVM. Reviewed-by: Russell King (Oracle) Signed-off-by: Jintack Lim Signed-off-by: Andre Przywara Signed-off-by: Christoffer Dall [maz: moved the command-line option to kvm-arm.mode] Signed-off-by: Marc Zyngier Acked-by: Catalin Marinas --- .../admin-guide/kernel-parameters.txt | 7 +++++- arch/arm64/include/asm/kvm_host.h | 5 ++++ arch/arm64/kernel/cpufeature.c | 25 +++++++++++++++++++ arch/arm64/kvm/arm.c | 5 ++++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 42 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 6cfa6e3996cf..b7b0704e360e 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2553,9 +2553,14 @@ protected: nVHE-based mode with support for guests whose state is kept private from the host. + nested: VHE-based mode with support for nested + virtualization. Requires at least ARMv8.3 + hardware. + Defaults to VHE/nVHE based on hardware support. Setting mode to "protected" will disable kexec and hibernation - for the host. + for the host. "nested" is experimental and should be + used with extreme caution. kvm-arm.vgic_v3_group0_trap= [KVM,ARM] Trap guest accesses to GICv3 group-0 diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index b14a0199eba4..186f1b759763 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -60,9 +60,14 @@ enum kvm_mode { KVM_MODE_DEFAULT, KVM_MODE_PROTECTED, + KVM_MODE_NV, KVM_MODE_NONE, }; +#ifdef CONFIG_KVM enum kvm_mode kvm_get_mode(void); +#else +static inline enum kvm_mode kvm_get_mode(void) { return KVM_MODE_NONE; }; +#endif DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index a77315b338e6..3fc14ee86239 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1956,6 +1956,20 @@ static void cpu_copy_el2regs(const struct arm64_cpu_capabilities *__unused) write_sysreg(read_sysreg(tpidr_el1), tpidr_el2); } +static bool has_nested_virt_support(const struct arm64_cpu_capabilities *cap, + int scope) +{ + if (kvm_get_mode() != KVM_MODE_NV) + return false; + + if (!has_cpuid_feature(cap, scope)) { + pr_warn("unavailable: %s\n", cap->desc); + return false; + } + + return true; +} + #ifdef CONFIG_ARM64_PAN static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused) { @@ -2215,6 +2229,17 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = runs_at_el2, .cpu_enable = cpu_copy_el2regs, }, + { + .desc = "Nested Virtualization Support", + .capability = ARM64_HAS_NESTED_VIRT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_nested_virt_support, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_EL1_NV_SHIFT, + .field_width = 4, + .min_field_value = ID_AA64MMFR2_EL1_NV_IMP, + }, { .capability = ARM64_HAS_32BIT_EL0_DO_NOT_USE, .type = ARM64_CPUCAP_SYSTEM_FEATURE, diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 698787ed87e9..4e1943e3aa42 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -2325,6 +2325,11 @@ static int __init early_kvm_mode_cfg(char *arg) return 0; } + if (strcmp(arg, "nested") == 0 && !WARN_ON(!is_kernel_in_hyp_mode())) { + kvm_mode = KVM_MODE_NV; + return 0; + } + return -EINVAL; } early_param("kvm-arm.mode", early_kvm_mode_cfg); diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index dfeb2c51e257..1af77a3657f7 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -31,6 +31,7 @@ HAS_GENERIC_AUTH_IMP_DEF HAS_IRQ_PRIO_MASKING HAS_LDAPR HAS_LSE_ATOMICS +HAS_NESTED_VIRT HAS_NO_FPSIMD HAS_NO_HW_PREFETCH HAS_PAN From patchwork Tue Jan 31 09:23:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122588 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 701E8C636CC for ; Tue, 31 Jan 2023 09:26:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9xgVKA7hn247+ZnRt/ZJhCDKI8kYPXo9umfSi+h6ZzU=; b=1xArbofwcVQzrO Va6kRJCVbOnhG6wbzxwiUWO7IWkxQwn53vVq1NZQxxEgjpCY0KOMiPEsyE3PFKCxKMdauURhYc04S 9BOeKcnpWVl+WuQ8/Of+hHnDw240/5HS+05d+w/L3wOsk1MLcpJQAWqCrwk0gLlnXrPOkw07c6RHV /AMlx118OMSomIN21bzC112fjsvpUhd/VQad6Uro+0sPkjmOKA7bxbWrKX1aWN2lipC17sGO8HQSy LweZ2DD5+0I37MC8T7LQMQ7GWGOsXpt7lQnaDSzxEEcGthIiAilCV+Jzr7reHEv0By3kyxLgTSKDY PHo4N6ammPvkaYwXFOQg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtW-006ySK-7F; Tue, 31 Jan 2023 09:25:38 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtR-006yP1-T6 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:25:35 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B284461471; Tue, 31 Jan 2023 09:25:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 23055C433D2; Tue, 31 Jan 2023 09:25:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675157132; bh=bs/YrObwitf4kEvM0zuHy+3mBo5vDAOzoljTOjtCkfU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fQ1QOsZf6KwuzEQEaeJ9P3zOpcfONingCoxqPHCDwc8Dv8QgAQPEaW6YLJEGgZovl eoRYZVxJlnY2IU9yjTJIlDtFDQ1tT8kyPdXAouzpo/dyw7Iewv9l6TcnDomk4TBE8M mFM065ybHZjxld4voOhyIkONOjlg7qgfziKd7Zf5X7y0KmjDTur+c8RbAom/3OpVXT cTJlGd67QG2i6BBZFhX1NWS6ux/4p0/NUveGirjn8TwgOhBK/SBrIltXusFgyqetVS fMavNVZCrrPIXpsiSrGZWFZWXsW1c+OiFopIbL3UrJrEZdxQ2KUVXESahEMUCkkDdO qYFHE248snePw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtO-0067U2-5T; Tue, 31 Jan 2023 09:25:30 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 02/69] KVM: arm64: Use the S2 MMU context to iterate over S2 table Date: Tue, 31 Jan 2023 09:23:57 +0000 Message-Id: <20230131092504.2880505-3-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_012534_057674_0240AEA8 X-CRM114-Status: GOOD ( 15.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Most of our S2 helpers take a kvm_s2_mmu pointer, but quickly revert back to using the kvm structure. By doing so, we lose track of which S2 MMU context we were initially using, and fallback to the "canonical" context. If we were trying to unmap a S2 context managed by a guest hypervisor, we end-up parsing the wrong set of page tables, and bad stuff happens (as this is often happening on the back of a trapped TLBI from the guest hypervisor). Instead, make sure we always use the provided MMU context all the way. This has no impact on non-NV, as we always pass the canonical MMU context. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/mmu.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index cfc7777fa490..217a511eb271 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -46,16 +46,17 @@ static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end) * long will also starve other vCPUs. We have to also make sure that the page * tables are not freed while we released the lock. */ -static int stage2_apply_range(struct kvm *kvm, phys_addr_t addr, +static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end, int (*fn)(struct kvm_pgtable *, u64, u64), bool resched) { + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); int ret; u64 next; do { - struct kvm_pgtable *pgt = kvm->arch.mmu.pgt; + struct kvm_pgtable *pgt = mmu->pgt; if (!pgt) return -EINVAL; @@ -71,8 +72,8 @@ static int stage2_apply_range(struct kvm *kvm, phys_addr_t addr, return ret; } -#define stage2_apply_range_resched(kvm, addr, end, fn) \ - stage2_apply_range(kvm, addr, end, fn, true) +#define stage2_apply_range_resched(mmu, addr, end, fn) \ + stage2_apply_range(mmu, addr, end, fn, true) static bool memslot_is_logging(struct kvm_memory_slot *memslot) { @@ -235,7 +236,7 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 lockdep_assert_held_write(&kvm->mmu_lock); WARN_ON(size & ~PAGE_MASK); - WARN_ON(stage2_apply_range(kvm, start, end, kvm_pgtable_stage2_unmap, + WARN_ON(stage2_apply_range(mmu, start, end, kvm_pgtable_stage2_unmap, may_block)); } @@ -934,8 +935,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, */ static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) { - struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); - stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_wrprotect); + stage2_apply_range_resched(mmu, addr, end, kvm_pgtable_stage2_wrprotect); } /** From patchwork Tue Jan 31 09:23:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122590 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9ABE7C636CC for ; Tue, 31 Jan 2023 09:27:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TlEgTAxXhjr1lGXFw3abz5T77Gp4p9467Z7cBgNd3n0=; b=QZKlpgD1poWwmi gXBarbXXTVCgOcmaG/p+zo1/Zo/xUsGHcRzEM4WkPCu6yIESA7Y2lTR0wRVn0i5dzUdtdTEmGWf5i lLlZ5y2R8AHqb3aeNu/XYTm2SOJlDqORUHW7qBcAtIkYK/iMtdcg9e+6WR0ekKlOl6ZupzmVcKb3A xFirBCKjek/h992RBrOB0ke1vQQv2Coc76jZUblWI/4bSNth6x893Rvz4mFkfIiZ2Uje4xXo5ANe5 vdczFfs8Z+ncAKVDVW1oukcVPO+VpjhmpLeagWzNxiPdEHHpdHIaK2bBtMcyQ6zFiXL7Zu0lA8Kcy GIHJ5ZfB2BXjNOclhvhA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtw-006ydm-Ea; Tue, 31 Jan 2023 09:26:04 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtS-006yP7-4b for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:25:36 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1582D6147A; Tue, 31 Jan 2023 09:25:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7A9D5C4339E; Tue, 31 Jan 2023 09:25:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675157132; bh=caJCdIDxEcs2dm46yIwmQS+/hs5HuoKGvfg5KefJxGc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MXtonjIjLDsKqbhIJW/ynZWojNqOAs0Mcw4MauTh6Xg4UqtaR+FUR7DvWa1ZaSD5j ldBODNWUqLlEsF+n/dwH7wxmgZcqqnz5jR4zsLmbICBJ4EWT3HeQ3pyOmWjzmjo+PW m0n2z5IPCw9iy2Cza/tZxjj7MTpb8+AZmiaOm1BMKqoTjp6ptrIdC2w3KfML8ZQTX0 xpHfXyra6T1eiPrba899w37GixJ2lHRFGNmf4Bva+J3yFhimG5loyv5+R3vA+v9xRf qC5euF93xu24YRBCN3/YCiIG3nLctdRa5xF0RyTacYks8K2JG7whrDuPqmWj/qJViE 139bAk+jjca8Q== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtO-0067U2-Dq; Tue, 31 Jan 2023 09:25:30 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 03/69] KVM: arm64: nv: Introduce nested virtualization VCPU feature Date: Tue, 31 Jan 2023 09:23:58 +0000 Message-Id: <20230131092504.2880505-4-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_012534_295991_D268EDC4 X-CRM114-Status: GOOD ( 17.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall Introduce the feature bit and a primitive that checks if the feature is set behind a static key check based on the cpus_have_const_cap check. Checking vcpu_has_nv() on systems without nested virt enabled should have negligible overhead. We don't yet allow userspace to actually set this feature. Reviewed-by: Ganapatrao Kulkarni Reviewed-by: Russell King (Oracle) Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 14 ++++++++++++++ arch/arm64/include/uapi/asm/kvm.h | 1 + 2 files changed, 15 insertions(+) create mode 100644 arch/arm64/include/asm/kvm_nested.h diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h new file mode 100644 index 000000000000..fd601ea68d13 --- /dev/null +++ b/arch/arm64/include/asm/kvm_nested.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ARM64_KVM_NESTED_H +#define __ARM64_KVM_NESTED_H + +#include + +static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu) +{ + return (!__is_defined(__KVM_NVHE_HYPERVISOR__) && + cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) && + test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)); +} + +#endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index a7a857f1784d..f8129c624b07 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -109,6 +109,7 @@ struct kvm_regs { #define KVM_ARM_VCPU_SVE 4 /* enable SVE for this CPU */ #define KVM_ARM_VCPU_PTRAUTH_ADDRESS 5 /* VCPU uses address authentication */ #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6 /* VCPU uses generic authentication */ +#define KVM_ARM_VCPU_HAS_EL2 7 /* Support nested virtualization */ struct kvm_vcpu_init { __u32 target; From patchwork Tue Jan 31 09:23:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 76800C636CC for ; Tue, 31 Jan 2023 09:27:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GTCc1ewyf0Av93G68g8F+uhqjAUY4lyR7a9sl883Bso=; b=k8K9/IZVwkVKQM 1EOO5KlzsGfAb5kYUDsDV6AAlI0msaxbsx6rMa90S5QsHA2DW9CHhDl7PEatK+Mk+3RXW2HAqWzux hEIthZ0cnRpRncz3mYHPx99ZrW2W+H81VlkolWTHoIAL9wad7gomD/Evw3HhpWAHHxUAt6Cs3qi9q 6aihddk1neLKXvSyfQQlS0DFEMTnRgovA4GZLYCPyd7JUNqnOqWSo+MqdVFpC+/KZ0bhpUx4ejchH 6dpUeSSYhxNjDXoPwU44srASkE4+0jLsD2tbETFBnu0e4XSm3z9HEw/bt3Nne0OdWU/NK6BmzswDT hiVBHiR6VkHIRNsMrkxw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmuC-006ykh-Ii; Tue, 31 Jan 2023 09:26:21 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtS-006yP6-Bs for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:25:36 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1420261478; Tue, 31 Jan 2023 09:25:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7348FC4339C; Tue, 31 Jan 2023 09:25:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675157132; bh=IN38zzqmT51zPoMDhWxwQcjhiI4SC5y83JbwCGL5vQY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IBTIrytTLVVZhPJULkNoMwTIUOXNaFJRsDu+00XQSQIfniD+sduAb3eprXo3+Py8K mFnuAT+2wPTu7B+owO1swQfcCtuaAlWkhvIRjZX0KyAtQDd875eha21vK59k1wZVGe zcNct7YvGcY5fYtdb4HpenaWwiLVTA9Z5KJ7tyoesl/bBgBabJuheo8srTnayFi2HX agPcDdReJEWvEez7QZElM3gvcplUlTVmuCbOWoZB0k6XpYVVhjkckTrcAxNFr2r5LP pHtpTZ0BUcDC2HbAO8m7JgmLpL4wTbLqhUimuW9zurG2QxvF5R9+F96LaAK4TzFtcz ON641OcH+HWbg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtO-0067U2-MB; Tue, 31 Jan 2023 09:25:30 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 04/69] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set Date: Tue, 31 Jan 2023 09:23:59 +0000 Message-Id: <20230131092504.2880505-5-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_012534_505481_785E00EB X-CRM114-Status: GOOD ( 17.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall Reset the VCPU with PSTATE.M = EL2h when the nested virtualization feature is enabled on the VCPU. Reviewed-by: Russell King (Oracle) Reviewed-by: Alexandru Elisei Signed-off-by: Christoffer Dall [maz: rework register reset not to use empty data structures] Signed-off-by: Marc Zyngier --- arch/arm64/kvm/reset.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 4a39da302b88..49a3257dec46 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -27,6 +27,7 @@ #include #include #include +#include #include /* Maximum phys_shift supported for any VM on this host */ @@ -38,6 +39,9 @@ static u32 __ro_after_init kvm_ipa_limit; #define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \ PSR_F_BIT | PSR_D_BIT) +#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \ + PSR_F_BIT | PSR_D_BIT) + #define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \ PSR_AA32_I_BIT | PSR_AA32_F_BIT) @@ -221,6 +225,10 @@ static int kvm_set_vm_width(struct kvm_vcpu *vcpu) if (kvm_has_mte(kvm) && is32bit) return -EINVAL; + /* NV is incompatible with AArch32 */ + if (vcpu_has_nv(vcpu) && is32bit) + return -EINVAL; + if (is32bit) set_bit(KVM_ARCH_FLAG_EL1_32BIT, &kvm->arch.flags); @@ -273,6 +281,12 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) if (loaded) kvm_arch_vcpu_put(vcpu); + /* Disallow NV+SVE for the time being */ + if (vcpu_has_nv(vcpu) && vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) { + ret = -EINVAL; + goto out; + } + if (!kvm_arm_vcpu_sve_finalized(vcpu)) { if (test_bit(KVM_ARM_VCPU_SVE, vcpu->arch.features)) { ret = kvm_vcpu_enable_sve(vcpu); @@ -295,6 +309,8 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) default: if (vcpu_el1_is_32bit(vcpu)) { pstate = VCPU_RESET_PSTATE_SVC; + } else if (vcpu_has_nv(vcpu)) { + pstate = VCPU_RESET_PSTATE_EL2; } else { pstate = VCPU_RESET_PSTATE_EL1; } From patchwork Tue Jan 31 09:24:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 41D8AC38142 for ; Tue, 31 Jan 2023 09:26:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sAqaL/HGYBAU7uF9xNZEgg9rAbtmuyigxmG9Sp01Vn4=; b=JA8gqJqmVZC0+K tBETeCVLYJHuqx95c3PNTVlR4RhvqcvfjBXscWKrqhCQ4vdUOKcrrdthdsUWdZ0wjyDKWIfK7y12r 2WbGLKJ1+NgwvMj0rYKUCEZ4TceCmr0eVYt5JSvUKzG5/S4gLjNJczWXecy8m+FaijqwDv0mkGuGV GneN4ndNC7Zp5yEZz6NzaWhO/SQm/LzJ2Ema0w/ccjlguaqp2b1qsu0VsaC8nrpJ+koPGqygBYRm0 DYBxxug7VYFsoLwMgHRJAufUH+3Ex2yCGIPBO3KGIstjcX0i+rlruHqVqLZ4TwL9h3wp7DPzIgMhe 8fVK7p9ARG8BgnM40X2w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtm-006yZg-L7; Tue, 31 Jan 2023 09:25:54 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtS-006yPK-6M for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:25:36 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B1F1F6142F; Tue, 31 Jan 2023 09:25:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1E591C433A1; Tue, 31 Jan 2023 09:25:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675157133; bh=I1ke+JEZimM+0OzjneFTWAmwFB/8ZVWgBFGIEfgwjLg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fNmbiZC9zM1JFPex2Ek1pu4ZTHCi+XbmEAYAl1zT0dJ3Iw8O5KfxV7J/oJ8zUxDnD +Ve5Ru8AvJc1uAcj/xD1lK5rSeokGrfnO4M0fCE0FGiOYRy7J8NU9eAhDFBpsJj4D+ ojm/X08sNVLwk1oY/lLD8+TRuE0fZVTaUinWqGp6gtqDucoFTmjN/ZhiUcGYikAGHZ C4kWm/KUbfF7a4OyX+KdDuiBBIKXDZ5Ez/cZidsQMxc0/EBkBYscqWoGxwvaEiIJd7 oWDjJPWGyRLD7Urw1pv5tuGMmDxvwdDjnnaTbcOun8h70iQRh3q/McvKqrczlToj8+ VhElU4hatV6Pw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtO-0067U2-UR; Tue, 31 Jan 2023 09:25:31 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 05/69] KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x Date: Tue, 31 Jan 2023 09:24:00 +0000 Message-Id: <20230131092504.2880505-6-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_012534_320935_97E0FFD1 X-CRM114-Status: GOOD ( 13.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall We were not allowing userspace to set a more privileged mode for the VCPU than EL1, but we should allow this when nested virtualization is enabled for the VCPU. Reviewed-by: Russell King (Oracle) Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/guest.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index cf4c495a4321..07444fa22888 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include "trace.h" @@ -253,6 +254,11 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (!vcpu_el1_is_32bit(vcpu)) return -EINVAL; break; + case PSR_MODE_EL2h: + case PSR_MODE_EL2t: + if (!vcpu_has_nv(vcpu)) + return -EINVAL; + fallthrough; case PSR_MODE_EL0t: case PSR_MODE_EL1t: case PSR_MODE_EL1h: From patchwork Tue Jan 31 09:24:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0E28FC38142 for ; Tue, 31 Jan 2023 09:29:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RVRMUK/Sgt6oxvxwW+a2dNQHdzEHKlkm2FDMmgafCLI=; b=2A2RBTEiuVeGxZ fEmfyyO/4cai09EbN3gp/kUU8rDLihXX9YRNvUs17VdTT3MbnM4Vmimh3P498YaIKiLHBaMAQSwxE LLZ3FIT8rMQbsv/o4bRzB/jAfkSYgTlvcdQNNoQkW00lDmMA5G1t8J/c6a+Aypc8Jtx21PtWDy4Xm bx3+ryDiRyAIsUzxclUINaLZ1bld7qWAa3v3wrDWbg4ZrDiov5HJWV4mE0SkhlWff948BsPXo7EdL yUbbIlV9BzWF2Buo5HyNaC/ijlbaM8+5topNwUO6t7WQ08zEBoH5WZpeyanR0VhBII1gmR4EL5R1H dkxI7spQC6eHf3O0A0IQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmvp-006zWT-Im; Tue, 31 Jan 2023 09:28:01 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtU-006yQl-21 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:25:38 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A4187B81A8C; Tue, 31 Jan 2023 09:25:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4FA56C433A4; Tue, 31 Jan 2023 09:25:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675157133; bh=zxjsgrcxZ/lfwZIgtetN/YI3tL15PTV/X2bbrNxchKs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=n6RThDFdmRUNV67FxsutcJYx8LF9GV/7pfVmQaQ71HL225QmCkOMF7Hc3h3BXnlOT 7bIb5+Mo+hXHDOUskC9pg9YT3ZEzwaRwUonRauHKSN3hXc8uKK+x1mZCX2Vs7Kolxk M8FDBtxa2FqP8hKycNWkyFjC0iQu5PMjsOZW2L3BcqXgutIwJCPgNcWKASNC9nbiVf HS6VsCyfyb4ANB9wKBr0l3cpZ2+M3XtA3qoI+z/GXX5EalPe+puBAT9s6MjJlubi91 NwiGKkBAeE0OlQo9w8ljbopD6HbgQpokbeNntzLrv0ufmBnNlhT2hoDIFoMsRxroRJ n8vPJoX/DN+xg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtP-0067U2-70; Tue, 31 Jan 2023 09:25:31 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 06/69] KVM: arm64: nv: Add EL2 system registers to vcpu context Date: Tue, 31 Jan 2023 09:24:01 +0000 Message-Id: <20230131092504.2880505-7-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_012536_434073_E8131A66 X-CRM114-Status: GOOD ( 13.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add the minimal set of EL2 system registers to the vcpu context. Nothing uses them just yet. Reviewed-by: Andre Przywara Reviewed-by: Russell King (Oracle) Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 33 ++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 186f1b759763..8ae20f57cf9a 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -326,12 +326,43 @@ enum vcpu_sysreg { TFSR_EL1, /* Tag Fault Status Register (EL1) */ TFSRE0_EL1, /* Tag Fault Status Register (EL0) */ - /* 32bit specific registers. Keep them at the end of the range */ + /* 32bit specific registers. */ DACR32_EL2, /* Domain Access Control Register */ IFSR32_EL2, /* Instruction Fault Status Register */ FPEXC32_EL2, /* Floating-Point Exception Control Register */ DBGVCR32_EL2, /* Debug Vector Catch Register */ + /* EL2 registers */ + VPIDR_EL2, /* Virtualization Processor ID Register */ + VMPIDR_EL2, /* Virtualization Multiprocessor ID Register */ + SCTLR_EL2, /* System Control Register (EL2) */ + ACTLR_EL2, /* Auxiliary Control Register (EL2) */ + HCR_EL2, /* Hypervisor Configuration Register */ + MDCR_EL2, /* Monitor Debug Configuration Register (EL2) */ + CPTR_EL2, /* Architectural Feature Trap Register (EL2) */ + HSTR_EL2, /* Hypervisor System Trap Register */ + HACR_EL2, /* Hypervisor Auxiliary Control Register */ + TTBR0_EL2, /* Translation Table Base Register 0 (EL2) */ + TTBR1_EL2, /* Translation Table Base Register 1 (EL2) */ + TCR_EL2, /* Translation Control Register (EL2) */ + VTTBR_EL2, /* Virtualization Translation Table Base Register */ + VTCR_EL2, /* Virtualization Translation Control Register */ + SPSR_EL2, /* EL2 saved program status register */ + ELR_EL2, /* EL2 exception link register */ + AFSR0_EL2, /* Auxiliary Fault Status Register 0 (EL2) */ + AFSR1_EL2, /* Auxiliary Fault Status Register 1 (EL2) */ + ESR_EL2, /* Exception Syndrome Register (EL2) */ + FAR_EL2, /* Fault Address Register (EL2) */ + HPFAR_EL2, /* Hypervisor IPA Fault Address Register */ + MAIR_EL2, /* Memory Attribute Indirection Register (EL2) */ + AMAIR_EL2, /* Auxiliary Memory Attribute Indirection Register (EL2) */ + VBAR_EL2, /* Vector Base Address Register (EL2) */ + RVBAR_EL2, /* Reset Vector Base Address Register */ + CONTEXTIDR_EL2, /* Context ID Register (EL2) */ + TPIDR_EL2, /* EL2 Software Thread ID Register */ + CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */ + SP_EL2, /* EL2 Stack Pointer */ + NR_SYS_REGS /* Nothing after this line! */ }; From patchwork Tue Jan 31 09:24:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122594 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 06195C38142 for ; Tue, 31 Jan 2023 09:28:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vJ3J89rtX33BmDIUjOD87JbVNtGE+2+8CWsuB14WwQI=; b=Dj+AQGkVtjY1YB 8ftFgbL1E0ktNXbhTkR2pHuKg7pUX75Fo+a/sOS3vlzyU5J8tNKFD3Z6cuA4r5DSt01YVzHmfY+Cr jkg9HfNzHDXQzNyJaAqhmaKQYzHjThkyfV/CI6eUkt2QaL3VJqlhcTHhQvFxgbEuEK0IUrIVTrTbJ Rf96Ru81OvVp1bfhnVO9DASF4NqSGs3T60iXrB/1qwIMGGFigSQ0QLpqJ39hfYwIlMGhds7Dm1zBN +3rNCCIMIEHxnlKvTjKejYehgb2UpixCQGtPW9G5iAvxB+QH2WqE+m2mk84551qxEzS8ZP8Xj+BAo Dz+mfJ+nK9GpG+ZFDSvA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmvF-006zDk-7V; Tue, 31 Jan 2023 09:27:25 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtT-006yR6-Ra for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:25:38 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CDDF061480; Tue, 31 Jan 2023 09:25:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3DFE5C433A0; Tue, 31 Jan 2023 09:25:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675157133; bh=nwsdSOlnJTwNwaWG4XAdthRFZHWhNVZUaQlLe4SqDzc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hOswpLLY/1n5cs6HOqLqAxwa51n0hv9VvJjLu6gqrxk3ovn8oFOZogDkUK4JcCr3a d/9T83b/g2vkn5hplOr3lmiIfRHIKkLyY+pmwOcorxNzsA2VFCY7CgARtJlyBx20FK Uiv6Cdtc1i7LFhsi6LcCabw0ilarfBcUAopsqqGRQA0ZdWhR3mea1BOi/YRKD8Upsr vHp8xbFtG1S5o2PdIhihCuDhxTMaiOyqN6ALV3P2YQnCYBdT6Cxn76G06spgV4A+DP G1tFnj2IvL2PWrOJaE/7V2GeHq13kmy39NNSo1fz65X3ZVO6I0UL6r/w9CJYGXNZEX DzAyidrY5HEQw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtP-0067U2-FL; Tue, 31 Jan 2023 09:25:31 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 07/69] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state Date: Tue, 31 Jan 2023 09:24:02 +0000 Message-Id: <20230131092504.2880505-8-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_012536_057417_E215D492 X-CRM114-Status: GOOD ( 15.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall When running a nested hypervisor we commonly have to figure out if the VCPU mode is running in the context of a guest hypervisor or guest guest, or just a normal guest. Add convenient primitives for this. Reviewed-by: Russell King (Oracle) Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 56 ++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index b1134af843bd..c12287345ede 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -179,6 +179,62 @@ static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, vcpu_gp_regs(vcpu)->regs[reg_num] = val; } +static inline bool vcpu_is_el2_ctxt(const struct kvm_cpu_context *ctxt) +{ + switch (ctxt->regs.pstate & (PSR_MODE32_BIT | PSR_MODE_MASK)) { + case PSR_MODE_EL2h: + case PSR_MODE_EL2t: + return true; + default: + return false; + } +} + +static inline bool vcpu_is_el2(const struct kvm_vcpu *vcpu) +{ + return vcpu_is_el2_ctxt(&vcpu->arch.ctxt); +} + +static inline bool __vcpu_el2_e2h_is_set(const struct kvm_cpu_context *ctxt) +{ + return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_E2H; +} + +static inline bool vcpu_el2_e2h_is_set(const struct kvm_vcpu *vcpu) +{ + return __vcpu_el2_e2h_is_set(&vcpu->arch.ctxt); +} + +static inline bool __vcpu_el2_tge_is_set(const struct kvm_cpu_context *ctxt) +{ + return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_TGE; +} + +static inline bool vcpu_el2_tge_is_set(const struct kvm_vcpu *vcpu) +{ + return __vcpu_el2_tge_is_set(&vcpu->arch.ctxt); +} + +static inline bool __is_hyp_ctxt(const struct kvm_cpu_context *ctxt) +{ + /* + * We are in a hypervisor context if the vcpu mode is EL2 or + * E2H and TGE bits are set. The latter means we are in the user space + * of the VHE kernel. ARMv8.1 ARM describes this as 'InHost' + * + * Note that the HCR_EL2.{E2H,TGE}={0,1} isn't really handled in the + * rest of the KVM code, and will result in a misbehaving guest. + */ + return vcpu_is_el2_ctxt(ctxt) || + (__vcpu_el2_e2h_is_set(ctxt) && __vcpu_el2_tge_is_set(ctxt)) || + __vcpu_el2_tge_is_set(ctxt); +} + +static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu) +{ + return __is_hyp_ctxt(&vcpu->arch.ctxt); +} + /* * The layout of SPSR for an AArch32 state is different when observed from an * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32 From patchwork Tue Jan 31 09:24:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122612 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA695C38142 for ; Tue, 31 Jan 2023 09:29:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qJYd0IftmuFQgaOeHOi/irOcQK8xqTXAXlYN2Y1TTOU=; b=i95v8DPvtnjpST 9+RHwITn4UmIEQ3kL+zR1QYl2k+lGCcw/uG/zuOfbwYhvJhQmmLqaB4euXIDObcAu9Jy8a7SpVvhb 1oMeRoz3LVKHLVmGuy7gFJxVzmnMhZZuGDPdZI70VSq7Ywvyst/W+47fu7m/VBw2J7YRQ3duZWooy uMQ9h6CsVvvYu2mfNupQTO4KJUFDWj9wiPt3Vn9Kw95Fo71VnBzd+WixAoVnf6lYCPv7EIrD3+GHf 6WKCnbvf1YKLigsbZ2lcQMKEDn90QrusewcMlHfztcQkCHiSX3vvS2YiU0OuWDODAoxHejh6gHnOy sk4V4gtahxqEov9HxVlQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmw5-006zfb-IB; Tue, 31 Jan 2023 09:28:17 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtV-006ySE-5n for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:25:40 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 44F4961484; Tue, 31 Jan 2023 09:25:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 89E46C433A8; Tue, 31 Jan 2023 09:25:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675157133; bh=evBxBwvQi55IdXQWYpG5lKq19hyUi69QGBj/PXaQ/zU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZNbyHkCCaNsFAD18VGVEkIi9IAxaBKdv3yBe0R5oCmxcFm0sYlf9jHRDpz1+JWri0 CQSsm8gIL68NDT04P/Z8VoMXBJZM7P2bNi778z4+suscC8Ck/MG1vjE8mJG31dtBm9 EXW/aU3N74KtVrj6g+5vgD562h93NfFH3WvHQRbGjzC8rbX1HmipFuF1fuKcxo+zTN ZjsgKFTW+UIiuPRf5ClLAWhfeR6PAWef6TX94kAjt6FZn7q2Sn5VvxJjT0W+OA7Q+r nR03H3fHR8/bUMEBjYSV0FiS+otZQXq2PrF2khXKaioziJ5tqanZ1wrVq0ImLjL/O5 X/M63U3rJeyKg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtP-0067U2-Ol; Tue, 31 Jan 2023 09:25:31 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 08/69] KVM: arm64: nv: Handle HCR_EL2.NV system register traps Date: Tue, 31 Jan 2023 09:24:03 +0000 Message-Id: <20230131092504.2880505-9-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_012537_407592_93022827 X-CRM114-Status: GOOD ( 21.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim ARM v8.3 introduces a new bit in the HCR_EL2, which is the NV bit. When this bit is set, accessing EL2 registers in EL1 traps to EL2. In addition, executing the following instructions in EL1 will trap to EL2: tlbi, at, eret, and msr/mrs instructions to access SP_EL1. Most of the instructions that trap to EL2 with the NV bit were undef at EL1 prior to ARM v8.3. The only instruction that was not undef is eret. This patch sets up a handler for EL2 registers and SP_EL1 register accesses at EL1. The host hypervisor keeps those register values in memory, and will emulate their behavior. This patch doesn't set the NV bit yet. It will be set in a later patch once nested virtualization support is completed. Reviewed-by: Russell King (Oracle) Signed-off-by: Jintack Lim [maz: EL2_REG() macros] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/sysreg.h | 40 ++++++++++++- arch/arm64/kvm/sys_regs.c | 99 +++++++++++++++++++++++++++++++-- 2 files changed, 133 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index a2a93b3fc557..4b9024216eba 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -489,23 +489,51 @@ #define SYS_PMCCFILTR_EL0 sys_reg(3, 3, 14, 15, 7) +#define SYS_VPIDR_EL2 sys_reg(3, 4, 0, 0, 0) +#define SYS_VMPIDR_EL2 sys_reg(3, 4, 0, 0, 5) + #define SYS_SCTLR_EL2 sys_reg(3, 4, 1, 0, 0) +#define SYS_ACTLR_EL2 sys_reg(3, 4, 1, 0, 1) +#define SYS_HCR_EL2 sys_reg(3, 4, 1, 1, 0) +#define SYS_MDCR_EL2 sys_reg(3, 4, 1, 1, 1) +#define SYS_CPTR_EL2 sys_reg(3, 4, 1, 1, 2) +#define SYS_HSTR_EL2 sys_reg(3, 4, 1, 1, 3) #define SYS_HFGRTR_EL2 sys_reg(3, 4, 1, 1, 4) #define SYS_HFGWTR_EL2 sys_reg(3, 4, 1, 1, 5) #define SYS_HFGITR_EL2 sys_reg(3, 4, 1, 1, 6) +#define SYS_HACR_EL2 sys_reg(3, 4, 1, 1, 7) + +#define SYS_TTBR0_EL2 sys_reg(3, 4, 2, 0, 0) +#define SYS_TTBR1_EL2 sys_reg(3, 4, 2, 0, 1) +#define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2) +#define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0) +#define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2) + #define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1) #define SYS_HDFGRTR_EL2 sys_reg(3, 4, 3, 1, 4) #define SYS_HDFGWTR_EL2 sys_reg(3, 4, 3, 1, 5) #define SYS_HAFGRTR_EL2 sys_reg(3, 4, 3, 1, 6) #define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0) #define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1) +#define SYS_SP_EL1 sys_reg(3, 4, 4, 1, 0) #define SYS_IFSR32_EL2 sys_reg(3, 4, 5, 0, 1) +#define SYS_AFSR0_EL2 sys_reg(3, 4, 5, 1, 0) +#define SYS_AFSR1_EL2 sys_reg(3, 4, 5, 1, 1) #define SYS_ESR_EL2 sys_reg(3, 4, 5, 2, 0) #define SYS_VSESR_EL2 sys_reg(3, 4, 5, 2, 3) #define SYS_FPEXC32_EL2 sys_reg(3, 4, 5, 3, 0) #define SYS_TFSR_EL2 sys_reg(3, 4, 5, 6, 0) -#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1) +#define SYS_FAR_EL2 sys_reg(3, 4, 6, 0, 0) +#define SYS_HPFAR_EL2 sys_reg(3, 4, 6, 0, 4) + +#define SYS_MAIR_EL2 sys_reg(3, 4, 10, 2, 0) +#define SYS_AMAIR_EL2 sys_reg(3, 4, 10, 3, 0) + +#define SYS_VBAR_EL2 sys_reg(3, 4, 12, 0, 0) +#define SYS_RVBAR_EL2 sys_reg(3, 4, 12, 0, 1) +#define SYS_RMR_EL2 sys_reg(3, 4, 12, 0, 2) +#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1) #define __SYS__AP0Rx_EL2(x) sys_reg(3, 4, 12, 8, x) #define SYS_ICH_AP0R0_EL2 __SYS__AP0Rx_EL2(0) #define SYS_ICH_AP0R1_EL2 __SYS__AP0Rx_EL2(1) @@ -547,13 +575,21 @@ #define SYS_ICH_LR14_EL2 __SYS__LR8_EL2(6) #define SYS_ICH_LR15_EL2 __SYS__LR8_EL2(7) +#define SYS_CONTEXTIDR_EL2 sys_reg(3, 4, 13, 0, 1) +#define SYS_TPIDR_EL2 sys_reg(3, 4, 13, 0, 2) + +#define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3) +#define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0) + /* VHE encodings for architectural EL0/1 system registers */ #define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0) #define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0) #define SYS_TTBR1_EL12 sys_reg(3, 5, 2, 0, 1) #define SYS_TCR_EL12 sys_reg(3, 5, 2, 0, 2) + #define SYS_SPSR_EL12 sys_reg(3, 5, 4, 0, 0) #define SYS_ELR_EL12 sys_reg(3, 5, 4, 0, 1) + #define SYS_AFSR0_EL12 sys_reg(3, 5, 5, 1, 0) #define SYS_AFSR1_EL12 sys_reg(3, 5, 5, 1, 1) #define SYS_ESR_EL12 sys_reg(3, 5, 5, 2, 0) @@ -569,6 +605,8 @@ #define SYS_CNTV_CTL_EL02 sys_reg(3, 5, 14, 3, 1) #define SYS_CNTV_CVAL_EL02 sys_reg(3, 5, 14, 3, 2) +#define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0) + /* Common SCTLR_ELx flags. */ #define SCTLR_ELx_ENTP2 (BIT(60)) #define SCTLR_ELx_DSSBS (BIT(44)) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 8d2747ad50ba..d11c9c4c177a 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -25,6 +25,7 @@ #include #include #include +#include #include #include @@ -175,6 +176,18 @@ static int set_ccsidr(struct kvm_vcpu *vcpu, u32 csselr, u32 val) return 0; } +static bool access_rw(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + vcpu_write_sys_reg(vcpu, p->regval, r->reg); + else + p->regval = vcpu_read_sys_reg(vcpu, r->reg); + + return true; +} + /* * See note at ARMv7 ARM B1.14.4 (TL;DR: S/W ops are not easily virtualized). */ @@ -333,6 +346,14 @@ static bool trap_raz_wi(struct kvm_vcpu *vcpu, return read_zero(vcpu, p); } +static bool trap_undef(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + kvm_inject_undefined(vcpu); + return false; +} + /* * ARMv8.1 mandates at least a trivial LORegion implementation, where all the * RW registers are RES0 (which we can implement as RAZ/WI). On an ARMv8.0 @@ -443,12 +464,9 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - if (p->is_write) { - vcpu_write_sys_reg(vcpu, p->regval, r->reg); + access_rw(vcpu, p, r); + if (p->is_write) vcpu_set_flag(vcpu, DEBUG_DIRTY); - } else { - p->regval = vcpu_read_sys_reg(vcpu, r->reg); - } trace_trap_reg(__func__, r->reg, p->is_write, p->regval); @@ -1583,6 +1601,24 @@ static unsigned int mte_visibility(const struct kvm_vcpu *vcpu, .visibility = mte_visibility, \ } +static unsigned int el2_visibility(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *rd) +{ + if (vcpu_has_nv(vcpu)) + return 0; + + return REG_HIDDEN; +} + +#define EL2_REG(name, acc, rst, v) { \ + SYS_DESC(SYS_##name), \ + .access = acc, \ + .reset = rst, \ + .reg = name, \ + .visibility = el2_visibility, \ + .val = v, \ +} + /* sys_reg_desc initialiser for known cpufeature ID registers */ #define ID_SANITISED(name) { \ SYS_DESC(SYS_##name), \ @@ -1627,6 +1663,18 @@ static unsigned int mte_visibility(const struct kvm_vcpu *vcpu, .visibility = raz_visibility, \ } +static bool access_sp_el1(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + __vcpu_sys_reg(vcpu, SP_EL1) = p->regval; + else + p->regval = __vcpu_sys_reg(vcpu, SP_EL1); + + return true; +} + /* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2 @@ -2052,9 +2100,50 @@ static const struct sys_reg_desc sys_reg_descs[] = { { PMU_SYS_REG(SYS_PMCCFILTR_EL0), .access = access_pmu_evtyper, .reset = reset_val, .reg = PMCCFILTR_EL0, .val = 0 }, + EL2_REG(VPIDR_EL2, access_rw, reset_val, 0), + EL2_REG(VMPIDR_EL2, access_rw, reset_val, 0), + EL2_REG(SCTLR_EL2, access_rw, reset_val, SCTLR_EL2_RES1), + EL2_REG(ACTLR_EL2, access_rw, reset_val, 0), + EL2_REG(HCR_EL2, access_rw, reset_val, 0), + EL2_REG(MDCR_EL2, access_rw, reset_val, 0), + EL2_REG(CPTR_EL2, access_rw, reset_val, CPTR_EL2_DEFAULT ), + EL2_REG(HSTR_EL2, access_rw, reset_val, 0), + EL2_REG(HACR_EL2, access_rw, reset_val, 0), + + EL2_REG(TTBR0_EL2, access_rw, reset_val, 0), + EL2_REG(TTBR1_EL2, access_rw, reset_val, 0), + EL2_REG(TCR_EL2, access_rw, reset_val, TCR_EL2_RES1), + EL2_REG(VTTBR_EL2, access_rw, reset_val, 0), + EL2_REG(VTCR_EL2, access_rw, reset_val, 0), + { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 }, + EL2_REG(SPSR_EL2, access_rw, reset_val, 0), + EL2_REG(ELR_EL2, access_rw, reset_val, 0), + { SYS_DESC(SYS_SP_EL1), access_sp_el1}, + { SYS_DESC(SYS_IFSR32_EL2), NULL, reset_unknown, IFSR32_EL2 }, + EL2_REG(AFSR0_EL2, access_rw, reset_val, 0), + EL2_REG(AFSR1_EL2, access_rw, reset_val, 0), + EL2_REG(ESR_EL2, access_rw, reset_val, 0), { SYS_DESC(SYS_FPEXC32_EL2), NULL, reset_val, FPEXC32_EL2, 0x700 }, + + EL2_REG(FAR_EL2, access_rw, reset_val, 0), + EL2_REG(HPFAR_EL2, access_rw, reset_val, 0), + + EL2_REG(MAIR_EL2, access_rw, reset_val, 0), + EL2_REG(AMAIR_EL2, access_rw, reset_val, 0), + + EL2_REG(VBAR_EL2, access_rw, reset_val, 0), + EL2_REG(RVBAR_EL2, access_rw, reset_val, 0), + { SYS_DESC(SYS_RMR_EL2), trap_undef }, + + EL2_REG(CONTEXTIDR_EL2, access_rw, reset_val, 0), + EL2_REG(TPIDR_EL2, access_rw, reset_val, 0), + + EL2_REG(CNTVOFF_EL2, access_rw, reset_val, 0), + EL2_REG(CNTHCTL_EL2, access_rw, reset_val, 0), + + EL2_REG(SP_EL2, NULL, reset_unknown, 0), }; static bool trap_dbgdidr(struct kvm_vcpu *vcpu, From patchwork Tue Jan 31 09:24:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 03FBDC636CC for ; Tue, 31 Jan 2023 09:28:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fA+9ut/gZKjNo36wa/BOB0SQx8mwMk32rwzm90PcdJU=; b=qLFeaFqvJf9oTQ 0/2bwMrPDIvtWrkthn+6m4UO6XOrhLb2AFpCWrvuh4Qpu77PVuZm5X7VjUvkLkCIjmWxH8+VkBCGY T7OCNRNarfsJNCjZgY4qvWuDGrxZaM7HiFOMbTdYh0Rc7fSBKDTsTXnLvu/IHEKAl9VN2fT/3tUuT LetZd3heNXYhjK7vPKF4By+R6LKqMNKUPRZV3vj4xRVMBOrvR/uAokgAGpqIGuTLvbnfbdOdYJOjV mU5KHjLHndMNEm5OhrIua6moCEO1J8hmn0Kdj9C4c2kNco9pbobYMOIUHWhM/aur97RST6MkGGNB9 9NrGPpi4FN+2nm0QiliQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmvY-006zNi-9m; Tue, 31 Jan 2023 09:27:44 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMmtT-006yR9-RX for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:25:38 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0C33A61482; Tue, 31 Jan 2023 09:25:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D0AD4C433A7; Tue, 31 Jan 2023 09:25:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675157133; bh=mQifo5ColVaTyS46g1UPvijlpz/VgvuSAnasRmnAFuU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qSMVG0zoP42WIXCNPniPSqt5ky5LwAafUok5NOp4lu/Ce9YDWDUGp7kgsvVsD2zXt AZYuyqkEZbdeqGx4gZt12rwm7cUrt2HrWsFOmnS7ww5xm94OZjSLKw5PpwOnMNTs8Q CmhXfubmcNfFz79t/tZVFNBF55BTjMaZreR8+cZTF7i/KW2oO40RXlTT5de44qUa6B 77hVkk2QhpnE41m7evSVHvNiJe0LJkWiyz8PFUEg401vS9Gl0iHZLKwHjO6qIPJpyK zrVO8eiRwUHezdv52S4pPqDFAHZpLC41wPsW66Fp8tyhZLlOE07D4jQ5IAOWmnnsBm tVqu+317QEeFA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtQ-0067U2-1C; Tue, 31 Jan 2023 09:25:32 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 09/69] KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values Date: Tue, 31 Jan 2023 09:24:04 +0000 Message-Id: <20230131092504.2880505-10-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_012536_053947_30F8E3CC X-CRM114-Status: GOOD ( 13.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall The VMPIDR_EL2 and VPIDR_EL2 are architecturally UNKNOWN at reset, but let's be nice to a guest hypervisor behaving foolishly and reset these to something reasonable anyway. Reviewed-by: Russell King (Oracle) Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index d11c9c4c177a..2a1e8f77b56c 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -667,7 +667,7 @@ static void reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) vcpu_write_sys_reg(vcpu, actlr, ACTLR_EL1); } -static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +static u64 compute_reset_mpidr(struct kvm_vcpu *vcpu) { u64 mpidr; @@ -681,7 +681,24 @@ static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0); mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1); mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2); - vcpu_write_sys_reg(vcpu, (1ULL << 31) | mpidr, MPIDR_EL1); + mpidr |= (1ULL << 31); + + return mpidr; +} + +static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +{ + vcpu_write_sys_reg(vcpu, compute_reset_mpidr(vcpu), MPIDR_EL1); +} + +static void reset_vmpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +{ + vcpu_write_sys_reg(vcpu, compute_reset_mpidr(vcpu), VMPIDR_EL2); +} + +static void reset_vpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +{ + vcpu_write_sys_reg(vcpu, read_cpuid_id(), VPIDR_EL2); } static unsigned int pmu_visibility(const struct kvm_vcpu *vcpu, @@ -2100,8 +2117,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { { PMU_SYS_REG(SYS_PMCCFILTR_EL0), .access = access_pmu_evtyper, .reset = reset_val, .reg = PMCCFILTR_EL0, .val = 0 }, - EL2_REG(VPIDR_EL2, access_rw, reset_val, 0), - EL2_REG(VMPIDR_EL2, access_rw, reset_val, 0), + EL2_REG(VPIDR_EL2, access_rw, reset_vpidr, 0), + EL2_REG(VMPIDR_EL2, access_rw, reset_vmpidr, 0), EL2_REG(SCTLR_EL2, access_rw, reset_val, SCTLR_EL2_RES1), EL2_REG(ACTLR_EL2, access_rw, reset_val, 0), EL2_REG(HCR_EL2, access_rw, reset_val, 0), From patchwork Tue Jan 31 09:24:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 468DBC636CC for ; Tue, 31 Jan 2023 10:13:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WJst9KQjaK/09Pax0STqqux8HIH4FwkXVf0Vyyyfqeo=; b=N1KNz8NminG3s4 vJ8+1zKl45B8AVQhQZGuuNlUZSvy8sdTFGYUwxO4sgYXwy8B6vResnPmEUjHQZXxn0tiYwmPBuLoq NTrI0+hMZlHTV1rPGfGFUcDn1T/2N0VK8ajZzhPEjaZQzlojY8ARUdIWF+MUmA2gxqDoibilPqvLF 0JI1Xt8+0jVKSJRhdtkQsjjjIOwIuFjxvyO8Xr79BO3WH7Vm4KelSrXmkSr+bFS0gX6fc8XkMs6FH ECEUgOvi2CUrOjVyUMg30zHwcvEgkoKVP7FC2+Bp73jwn+w/Rngs7/nME8acUZyIXDceXVmKFV7RO yhMRRUyXmhuV7f94Tr9Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnc2-007IEj-4d; Tue, 31 Jan 2023 10:11:40 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9i-0075cv-EJ for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:25 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D98EBB81AFA; Tue, 31 Jan 2023 09:42:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 42FBDC433D2; Tue, 31 Jan 2023 09:42:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158139; bh=YACg1Bg1F6tlU5rlGxllCmzvxEEHh3zo3innGE1lwMY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PC8LmBEk47rDqgtw/At7PeT0sY4eGAzUEsovmVEEv7EreoV6XUaeUzJqy6u9kl/gj xe0nzX5Zwl18PZGu4Kp0fEjQKIcI4fDEhq2ySd5U411b2x+ApQ62kyvYFPxHMPgCiG jJsuXcOklKJbm9ba0WWdTWbjhZLQ8pSWvDpMQrtTJKczV/60tz3iO3Rew0psM3lgLS uariw+MjP43X2U/d1VvaQUFzW2HKvdOufVrfSGpLNPFQxlWhykpJm3bJ9sKQUUNUIR 23NjYNucxsqkhZqda68BnR05DleZsvwQbaadyQ32cIimCmxsa5xmY5lw6HGFjGMcrk rO6QtaxUHa/fg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtQ-0067U2-9m; Tue, 31 Jan 2023 09:25:32 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 10/69] KVM: arm64: nv: Support virtual EL2 exceptions Date: Tue, 31 Jan 2023 09:24:05 +0000 Message-Id: <20230131092504.2880505-11-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014222_889191_DCA16E41 X-CRM114-Status: GOOD ( 34.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor). Reviewed-by: Ganapatrao Kulkarni Signed-off-by: Jintack Lim Signed-off-by: Christoffer Dall [maz: switch to common exception injection framework, illegal exeption return handling] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 17 +++ arch/arm64/include/asm/kvm_emulate.h | 10 ++ arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/emulate-nested.c | 203 +++++++++++++++++++++++++++ arch/arm64/kvm/hyp/exception.c | 48 +++++-- arch/arm64/kvm/inject_fault.c | 61 +++++++- arch/arm64/kvm/trace_arm.h | 59 ++++++++ 8 files changed, 382 insertions(+), 20 deletions(-) create mode 100644 arch/arm64/kvm/emulate-nested.c diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 5a4b2342d571..c101a611af9c 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -350,4 +350,21 @@ #define CPACR_EL1_DEFAULT (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN |\ CPACR_EL1_ZEN_EL1EN) +#define kvm_mode_names \ + { PSR_MODE_EL0t, "EL0t" }, \ + { PSR_MODE_EL1t, "EL1t" }, \ + { PSR_MODE_EL1h, "EL1h" }, \ + { PSR_MODE_EL2t, "EL2t" }, \ + { PSR_MODE_EL2h, "EL2h" }, \ + { PSR_MODE_EL3t, "EL3t" }, \ + { PSR_MODE_EL3h, "EL3h" }, \ + { PSR_AA32_MODE_USR, "32-bit USR" }, \ + { PSR_AA32_MODE_FIQ, "32-bit FIQ" }, \ + { PSR_AA32_MODE_IRQ, "32-bit IRQ" }, \ + { PSR_AA32_MODE_SVC, "32-bit SVC" }, \ + { PSR_AA32_MODE_ABT, "32-bit ABT" }, \ + { PSR_AA32_MODE_HYP, "32-bit HYP" }, \ + { PSR_AA32_MODE_UND, "32-bit UND" }, \ + { PSR_AA32_MODE_SYS, "32-bit SYS" } + #endif /* __ARM64_KVM_ARM_H__ */ diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index c12287345ede..b31b32ecbe2d 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -33,6 +33,12 @@ enum exception_type { except_type_serror = 0x180, }; +#define kvm_exception_type_names \ + { except_type_sync, "SYNC" }, \ + { except_type_irq, "IRQ" }, \ + { except_type_fiq, "FIQ" }, \ + { except_type_serror, "SERROR" } + bool kvm_condition_valid32(const struct kvm_vcpu *vcpu); void kvm_skip_instr32(struct kvm_vcpu *vcpu); @@ -44,6 +50,10 @@ void kvm_inject_size_fault(struct kvm_vcpu *vcpu); void kvm_vcpu_wfi(struct kvm_vcpu *vcpu); +void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu); +int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2); +int kvm_inject_nested_irq(struct kvm_vcpu *vcpu); + #if defined(__KVM_VHE_HYPERVISOR__) || defined(__KVM_NVHE_HYPERVISOR__) static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu) { diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 8ae20f57cf9a..640997251724 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -638,7 +638,7 @@ struct kvm_vcpu_arch { #define EXCEPT_AA64_EL1_IRQ __vcpu_except_flags(1) #define EXCEPT_AA64_EL1_FIQ __vcpu_except_flags(2) #define EXCEPT_AA64_EL1_SERR __vcpu_except_flags(3) -/* For AArch64 with NV (one day): */ +/* For AArch64 with NV: */ #define EXCEPT_AA64_EL2_SYNC __vcpu_except_flags(4) #define EXCEPT_AA64_EL2_IRQ __vcpu_except_flags(5) #define EXCEPT_AA64_EL2_FIQ __vcpu_except_flags(6) diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 5e33c2d4645a..31b07f2b2186 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -14,7 +14,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ inject_fault.o va_layout.o handle_exit.o \ guest.o debug.o reset.o sys_regs.o stacktrace.o \ vgic-sys-reg-v3.o fpsimd.o pkvm.o \ - arch_timer.o trng.o vmid.o \ + arch_timer.o trng.o vmid.o emulate-nested.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c new file mode 100644 index 000000000000..b96662029fb1 --- /dev/null +++ b/arch/arm64/kvm/emulate-nested.c @@ -0,0 +1,203 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2016 - Linaro and Columbia University + * Author: Jintack Lim + */ + +#include +#include + +#include +#include + +#include "hyp/include/hyp/adjust_pc.h" + +#include "trace.h" + +static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr) +{ + u64 mode = spsr & PSR_MODE_MASK; + + /* + * Possible causes for an Illegal Exception Return from EL2: + * - trying to return to EL3 + * - trying to return to an illegal M value + * - trying to return to a 32bit EL + * - trying to return to EL1 with HCR_EL2.TGE set + */ + if (mode == PSR_MODE_EL3t || mode == PSR_MODE_EL3h || + mode == 0b00001 || (mode & BIT(1)) || + (spsr & PSR_MODE32_BIT) || + (vcpu_el2_tge_is_set(vcpu) && (mode == PSR_MODE_EL1t || + mode == PSR_MODE_EL1h))) { + /* + * The guest is playing with our nerves. Preserve EL, SP, + * masks, flags from the existing PSTATE, and set IL. + * The HW will then generate an Illegal State Exception + * immediately after ERET. + */ + spsr = *vcpu_cpsr(vcpu); + + spsr &= (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | + PSR_N_BIT | PSR_Z_BIT | PSR_C_BIT | PSR_V_BIT | + PSR_MODE_MASK | PSR_MODE32_BIT); + spsr |= PSR_IL_BIT; + } + + return spsr; +} + +void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) +{ + u64 spsr, elr, mode; + bool direct_eret; + + /* + * Going through the whole put/load motions is a waste of time + * if this is a VHE guest hypervisor returning to its own + * userspace, or the hypervisor performing a local exception + * return. No need to save/restore registers, no need to + * switch S2 MMU. Just do the canonical ERET. + */ + spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2); + spsr = kvm_check_illegal_exception_return(vcpu, spsr); + + mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); + + direct_eret = (mode == PSR_MODE_EL0t && + vcpu_el2_e2h_is_set(vcpu) && + vcpu_el2_tge_is_set(vcpu)); + direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); + + if (direct_eret) { + *vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2); + *vcpu_cpsr(vcpu) = spsr; + trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr); + return; + } + + preempt_disable(); + kvm_arch_vcpu_put(vcpu); + + elr = __vcpu_sys_reg(vcpu, ELR_EL2); + + trace_kvm_nested_eret(vcpu, elr, spsr); + + /* + * Note that the current exception level is always the virtual EL2, + * since we set HCR_EL2.NV bit only when entering the virtual EL2. + */ + *vcpu_pc(vcpu) = elr; + *vcpu_cpsr(vcpu) = spsr; + + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + preempt_enable(); +} + +static void kvm_inject_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2, + enum exception_type type) +{ + trace_kvm_inject_nested_exception(vcpu, esr_el2, type); + + switch (type) { + case except_type_sync: + kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SYNC); + vcpu_write_sys_reg(vcpu, esr_el2, ESR_EL2); + break; + case except_type_irq: + kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_IRQ); + break; + default: + WARN_ONCE(1, "Unsupported EL2 exception injection %d\n", type); + } +} + +/* + * Emulate taking an exception to EL2. + * See ARM ARM J8.1.2 AArch64.TakeException() + */ +static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2, + enum exception_type type) +{ + u64 pstate, mode; + bool direct_inject; + + if (!vcpu_has_nv(vcpu)) { + kvm_err("Unexpected call to %s for the non-nesting configuration\n", + __func__); + return -EINVAL; + } + + /* + * As for ERET, we can avoid doing too much on the injection path by + * checking that we either took the exception from a VHE host + * userspace or from vEL2. In these cases, there is no change in + * translation regime (or anything else), so let's do as little as + * possible. + */ + pstate = *vcpu_cpsr(vcpu); + mode = pstate & (PSR_MODE_MASK | PSR_MODE32_BIT); + + direct_inject = (mode == PSR_MODE_EL0t && + vcpu_el2_e2h_is_set(vcpu) && + vcpu_el2_tge_is_set(vcpu)); + direct_inject |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); + + if (direct_inject) { + kvm_inject_el2_exception(vcpu, esr_el2, type); + return 1; + } + + preempt_disable(); + + /* + * We may have an exception or PC update in the EL0/EL1 context. + * Commit it before entering EL2. + */ + __kvm_adjust_pc(vcpu); + + kvm_arch_vcpu_put(vcpu); + + kvm_inject_el2_exception(vcpu, esr_el2, type); + + /* + * A hard requirement is that a switch between EL1 and EL2 + * contexts has to happen between a put/load, so that we can + * pick the correct timer and interrupt configuration, among + * other things. + * + * Make sure the exception actually took place before we load + * the new context. + */ + __kvm_adjust_pc(vcpu); + + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + preempt_enable(); + + return 1; +} + +int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2) +{ + return kvm_inject_nested(vcpu, esr_el2, except_type_sync); +} + +int kvm_inject_nested_irq(struct kvm_vcpu *vcpu) +{ + /* + * Do not inject an irq if the: + * - Current exception level is EL2, and + * - virtual HCR_EL2.TGE == 0 + * - virtual HCR_EL2.IMO == 0 + * + * See Table D1-17 "Physical interrupt target and masking when EL3 is + * not implemented and EL2 is implemented" in ARM DDI 0487C.a. + */ + + if (vcpu_is_el2(vcpu) && !vcpu_el2_tge_is_set(vcpu) && + !(__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_IMO)) + return 1; + + /* esr_el2 value doesn't matter for exits due to irqs. */ + return kvm_inject_nested(vcpu, 0, except_type_irq); +} diff --git a/arch/arm64/kvm/hyp/exception.c b/arch/arm64/kvm/hyp/exception.c index 791d3de76771..424a5107cddb 100644 --- a/arch/arm64/kvm/hyp/exception.c +++ b/arch/arm64/kvm/hyp/exception.c @@ -14,6 +14,7 @@ #include #include #include +#include #if !defined (__KVM_NVHE_HYPERVISOR__) && !defined (__KVM_VHE_HYPERVISOR__) #error Hypervisor code only! @@ -23,7 +24,9 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) { u64 val; - if (__vcpu_read_sys_reg_from_cpu(reg, &val)) + if (unlikely(vcpu_has_nv(vcpu))) + return vcpu_read_sys_reg(vcpu, reg); + else if (__vcpu_read_sys_reg_from_cpu(reg, &val)) return val; return __vcpu_sys_reg(vcpu, reg); @@ -31,18 +34,25 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) { - if (__vcpu_write_sys_reg_to_cpu(val, reg)) - return; - - __vcpu_sys_reg(vcpu, reg) = val; + if (unlikely(vcpu_has_nv(vcpu))) + vcpu_write_sys_reg(vcpu, val, reg); + else if (!__vcpu_write_sys_reg_to_cpu(val, reg)) + __vcpu_sys_reg(vcpu, reg) = val; } -static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, u64 val) +static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode, + u64 val) { - if (has_vhe()) + if (unlikely(vcpu_has_nv(vcpu))) { + if (target_mode == PSR_MODE_EL1h) + vcpu_write_sys_reg(vcpu, val, SPSR_EL1); + else + vcpu_write_sys_reg(vcpu, val, SPSR_EL2); + } else if (has_vhe()) { write_sysreg_el1(val, SYS_SPSR); - else + } else { __vcpu_sys_reg(vcpu, SPSR_EL1) = val; + } } static void __vcpu_write_spsr_abt(struct kvm_vcpu *vcpu, u64 val) @@ -101,6 +111,11 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1); __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1); break; + case PSR_MODE_EL2h: + vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2); + sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2); + __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2); + break; default: /* Don't do that */ BUG(); @@ -153,7 +168,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, new |= target_mode; *vcpu_cpsr(vcpu) = new; - __vcpu_write_spsr(vcpu, old); + __vcpu_write_spsr(vcpu, target_mode, old); } /* @@ -323,11 +338,20 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu) case unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC): enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync); break; + + case unpack_vcpu_flag(EXCEPT_AA64_EL2_SYNC): + enter_exception64(vcpu, PSR_MODE_EL2h, except_type_sync); + break; + + case unpack_vcpu_flag(EXCEPT_AA64_EL2_IRQ): + enter_exception64(vcpu, PSR_MODE_EL2h, except_type_irq); + break; + default: /* - * Only EL1_SYNC makes sense so far, EL2_{SYNC,IRQ} - * will be implemented at some point. Everything - * else gets silently ignored. + * Only EL1_SYNC and EL2_{SYNC,IRQ} makes + * sense so far. Everything else gets silently + * ignored. */ break; } diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c index f32f4a2a347f..64c3aec0d937 100644 --- a/arch/arm64/kvm/inject_fault.c +++ b/arch/arm64/kvm/inject_fault.c @@ -12,17 +12,55 @@ #include #include +#include #include +static void pend_sync_exception(struct kvm_vcpu *vcpu) +{ + /* If not nesting, EL1 is the only possible exception target */ + if (likely(!vcpu_has_nv(vcpu))) { + kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC); + return; + } + + /* + * With NV, we need to pick between EL1 and EL2. Note that we + * never deal with a nesting exception here, hence never + * changing context, and the exception itself can be delayed + * until the next entry. + */ + switch(*vcpu_cpsr(vcpu) & PSR_MODE_MASK) { + case PSR_MODE_EL2h: + case PSR_MODE_EL2t: + kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SYNC); + break; + case PSR_MODE_EL1h: + case PSR_MODE_EL1t: + kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC); + break; + case PSR_MODE_EL0t: + if (vcpu_el2_tge_is_set(vcpu)) + kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SYNC); + else + kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC); + break; + default: + BUG(); + } +} + +static bool match_target_el(struct kvm_vcpu *vcpu, unsigned long target) +{ + return (vcpu_get_flag(vcpu, EXCEPT_MASK) == target); +} + static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr) { unsigned long cpsr = *vcpu_cpsr(vcpu); bool is_aarch32 = vcpu_mode_is_32bit(vcpu); u64 esr = 0; - kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC); - - vcpu_write_sys_reg(vcpu, addr, FAR_EL1); + pend_sync_exception(vcpu); /* * Build an {i,d}abort, depending on the level and the @@ -43,14 +81,22 @@ static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr if (!is_iabt) esr |= ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT; - vcpu_write_sys_reg(vcpu, esr | ESR_ELx_FSC_EXTABT, ESR_EL1); + esr |= ESR_ELx_FSC_EXTABT; + + if (match_target_el(vcpu, unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC))) { + vcpu_write_sys_reg(vcpu, addr, FAR_EL1); + vcpu_write_sys_reg(vcpu, esr, ESR_EL1); + } else { + vcpu_write_sys_reg(vcpu, addr, FAR_EL2); + vcpu_write_sys_reg(vcpu, esr, ESR_EL2); + } } static void inject_undef64(struct kvm_vcpu *vcpu) { u64 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT); - kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC); + pend_sync_exception(vcpu); /* * Build an unknown exception, depending on the instruction @@ -59,7 +105,10 @@ static void inject_undef64(struct kvm_vcpu *vcpu) if (kvm_vcpu_trap_il_is32bit(vcpu)) esr |= ESR_ELx_IL; - vcpu_write_sys_reg(vcpu, esr, ESR_EL1); + if (match_target_el(vcpu, unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC))) + vcpu_write_sys_reg(vcpu, esr, ESR_EL1); + else + vcpu_write_sys_reg(vcpu, esr, ESR_EL2); } #define DFSR_FSC_EXTABT_LPAE 0x10 diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h index 33e4e7dd2719..f3e46a976125 100644 --- a/arch/arm64/kvm/trace_arm.h +++ b/arch/arm64/kvm/trace_arm.h @@ -2,6 +2,7 @@ #if !defined(_TRACE_ARM_ARM64_KVM_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_ARM_ARM64_KVM_H +#include #include #include @@ -301,6 +302,64 @@ TRACE_EVENT(kvm_timer_emulate, __entry->timer_idx, __entry->should_fire) ); +TRACE_EVENT(kvm_nested_eret, + TP_PROTO(struct kvm_vcpu *vcpu, unsigned long elr_el2, + unsigned long spsr_el2), + TP_ARGS(vcpu, elr_el2, spsr_el2), + + TP_STRUCT__entry( + __field(struct kvm_vcpu *, vcpu) + __field(unsigned long, elr_el2) + __field(unsigned long, spsr_el2) + __field(unsigned long, target_mode) + __field(unsigned long, hcr_el2) + ), + + TP_fast_assign( + __entry->vcpu = vcpu; + __entry->elr_el2 = elr_el2; + __entry->spsr_el2 = spsr_el2; + __entry->target_mode = spsr_el2 & (PSR_MODE_MASK | PSR_MODE32_BIT); + __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + ), + + TP_printk("elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx", + __entry->elr_el2, __entry->spsr_el2, + __print_symbolic(__entry->target_mode, kvm_mode_names), + __entry->hcr_el2) +); + +TRACE_EVENT(kvm_inject_nested_exception, + TP_PROTO(struct kvm_vcpu *vcpu, u64 esr_el2, int type), + TP_ARGS(vcpu, esr_el2, type), + + TP_STRUCT__entry( + __field(struct kvm_vcpu *, vcpu) + __field(unsigned long, esr_el2) + __field(int, type) + __field(unsigned long, spsr_el2) + __field(unsigned long, pc) + __field(unsigned long, source_mode) + __field(unsigned long, hcr_el2) + ), + + TP_fast_assign( + __entry->vcpu = vcpu; + __entry->esr_el2 = esr_el2; + __entry->type = type; + __entry->spsr_el2 = *vcpu_cpsr(vcpu); + __entry->pc = *vcpu_pc(vcpu); + __entry->source_mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT); + __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + ), + + TP_printk("%s: esr_el2 0x%lx elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx", + __print_symbolic(__entry->type, kvm_exception_type_names), + __entry->esr_el2, __entry->pc, __entry->spsr_el2, + __print_symbolic(__entry->source_mode, kvm_mode_names), + __entry->hcr_el2) +); + #endif /* _TRACE_ARM_ARM64_KVM_H */ #undef TRACE_INCLUDE_PATH From patchwork Tue Jan 31 09:24:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 46980C636D3 for ; Tue, 31 Jan 2023 09:53:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7dJlWMG+0iZNGVxa0MAqJkgzvkdUcqk+p1LfNWPB6AA=; b=1t1HyrgkhPwnFt GiGxGyNTCfkg7x+P4qyjAB84GkAalhwYkMev1dJY/3BcXcYbnHUebNf3V0BLoCIgcAHQ0mL3jt+xW 0DmJU51vPrMpGmr1liuVceRBgX3NCfDnoXsjBrSOqRBM6UHV3gK4dxNd1T81pn4iWz5gDXHD7b7vZ eV8pAf0KVtMkX9ZGxJe4FtJbYPFRAzI0eCa37hkBX1wysIKvpVeJfiieSNiV6QqxllmS/ufieoE3w WNHAx2pS+VPZi8rDSxN+0E6ltqfw6lrSYmZwZShDjJN1HrIgwvIcVjODMW15h5pBxT0ZRCA+dx99n 94J1rTrcZU6lkLsRwjdA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnJ1-0079Uz-R7; Tue, 31 Jan 2023 09:51:59 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8o-0075C0-Jr for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:28 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 292EEB81AE0; Tue, 31 Jan 2023 09:41:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CCBDDC433D2; Tue, 31 Jan 2023 09:41:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158084; bh=ycJQTa90vNc+zMPsmHdkSUMnSh/e/B8/iWuzQP1sK/U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eW1KAYTImXH4uKcG50l+3lzAf+qP+IcoWswxpH+ZhKTG5Bfi3Gd31I/pVgMc0Jatd s7SyMVT1BvQMLjDJ6te0nZcf/h30TutFLflJZJT/ny/v0fPZGzIyuNB6+nHUNjRy4N BUEL12QfPQzU4R3bpOXvj+ZGEuebunB4F6/m5z9g7KDZ6MiKTCCOT5yvViovu4g0sr VHg/i+Ru1LzlxO4w/3Yviogbg1LKraWDy4NKn79Rlv0XaEU+k9YFr2ihbuS8Xue5pX 26dl3mT+Cqgu2SWkfiZGjXBqfLp9rixYbWuc0fMzMblLOcroLtxfQqxnV4wV3x1dwZ 9SLCLshDkVLyQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtQ-0067U2-Jt; Tue, 31 Jan 2023 09:25:32 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 11/69] KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 Date: Tue, 31 Jan 2023 09:24:06 +0000 Message-Id: <20230131092504.2880505-12-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014126_992804_D80B894C X-CRM114-Status: GOOD ( 15.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim As we expect all PSCI calls from the L1 hypervisor to be performed using SMC when nested virtualization is enabled, it is clear that all HVC instruction from the VM (including from the virtual EL2) are supposed to handled in the virtual EL2. Forward these to EL2 as required. Reviewed-by: Russell King (Oracle) Reviewed-by: Alexandru Elisei Signed-off-by: Jintack Lim [maz: add handling of HCR_EL2.HCD] Signed-off-by: Marc Zyngier --- arch/arm64/kvm/handle_exit.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index e778eefcf214..2d8c09cf3e49 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -41,6 +42,16 @@ static int handle_hvc(struct kvm_vcpu *vcpu) kvm_vcpu_hvc_get_imm(vcpu)); vcpu->stat.hvc_exit_stat++; + /* Forward hvc instructions to the virtual EL2 if the guest has EL2. */ + if (vcpu_has_nv(vcpu)) { + if (vcpu_read_sys_reg(vcpu, HCR_EL2) & HCR_HCD) + kvm_inject_undefined(vcpu); + else + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + + return 1; + } + ret = kvm_hvc_call_handler(vcpu); if (ret < 0) { vcpu_set_reg(vcpu, 0, ~0UL); From patchwork Tue Jan 31 09:24:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9000CC636CC for ; Tue, 31 Jan 2023 09:43:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=BaGyWaOlJ0w8jPHrlpx/TkgfSmzSFkS8A7k0cLiXwnA=; b=KPvqkd0siI60SB J3QBteCwbdDEXH79AGYQpKfYpkqD8n4bZ6M1M5BuDLlCjyhKtLwW7lfWvKIxclo5oFodfSXuEoUpZ wLA0V6eXqai6/D3oEpMd7q0F87S9EcGHnDbLDe23OD+MUC1kMAEq+fxQULF8gjs7HwMn72cKgiIp0 pVOylmKkeevR4UZFQTmGj/65F/j2FPBmMQE+M1fauVDerPhp5Qi/Z9Wr1YD5wSoAk6SYk58DazovA E9EgCeN2DOzJavoqtn0NM1R5Hzkca86ul2oMeZDGaXy7zML0DvbPNsnWQvjfaFQ7P87y0OO+lST5J H624T50bBIzPJwBNRT8g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9w-0075jz-0T; Tue, 31 Jan 2023 09:42:36 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8B-0074qy-HQ for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:40:49 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BDFA861484; Tue, 31 Jan 2023 09:40:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 90B30C433D2; Tue, 31 Jan 2023 09:40:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158046; bh=AEdo1QjTraKeiiK+6yLem0poWl5PLEbqaOt2rrGehMk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F2D6Fb/wH+F2p9hPsgJI1fXs/Pi27D5UFC6v+TcQkVqxMgLVVbck6cTxV4pXHnOtI X371jqiOUbHQv00VyRB02WI9WkSIYNbkxVDAQxn2j8g4QfcxmhQc+Y32kLwZQgJGsu G8SfCrpCIA7UBAr7+N42x+6dVdZJzi7vg36PwlWyjtZJIMIIiyl7qUKzQFtL5PjtW4 lMohoNnBnaFk81rwyRrYNFXkHReh7QGMZewXNc/toe/rbvsuOhHjaypOOTGkQxsJUh 7UyXBov93lH0b+3wrrLQmJY4IJO3N2ggEtXpXIWRj8HCRDUPpuyms0Bl0hJRsC4rfl Fw1HIHwy0CBqw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtQ-0067U2-Tw; Tue, 31 Jan 2023 09:25:32 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 12/69] KVM: arm64: nv: Handle trapped ERET from virtual EL2 Date: Tue, 31 Jan 2023 09:24:07 +0000 Message-Id: <20230131092504.2880505-13-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014047_681853_355FF00A X-CRM114-Status: GOOD ( 15.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall When a guest hypervisor running virtual EL2 in EL1 executes an ERET instruction, we will have set HCR_EL2.NV which traps ERET to EL2, so that we can emulate the exception return in software. Reviewed-by: Russell King (Oracle) Reviewed-by: Alexandru Elisei Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/esr.h | 4 ++++ arch/arm64/include/asm/kvm_arm.h | 2 +- arch/arm64/kvm/handle_exit.c | 10 ++++++++++ 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index 206de10524e3..d4dd949b921e 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -272,6 +272,10 @@ (((e) & ESR_ELx_SYS64_ISS_OP2_MASK) >> \ ESR_ELx_SYS64_ISS_OP2_SHIFT)) +/* ISS field definitions for ERET/ERETAA/ERETAB trapping */ +#define ESR_ELx_ERET_ISS_ERET 0x2 +#define ESR_ELx_ERET_ISS_ERETA 0x1 + /* * ISS field definitions for floating-point exception traps * (FP_EXC_32/FP_EXC_64). diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index c101a611af9c..baef29fcbeee 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -345,7 +345,7 @@ ECN(SP_ALIGN), ECN(FP_EXC32), ECN(FP_EXC64), ECN(SERROR), \ ECN(BREAKPT_LOW), ECN(BREAKPT_CUR), ECN(SOFTSTP_LOW), \ ECN(SOFTSTP_CUR), ECN(WATCHPT_LOW), ECN(WATCHPT_CUR), \ - ECN(BKPT32), ECN(VECTOR32), ECN(BRK64) + ECN(BKPT32), ECN(VECTOR32), ECN(BRK64), ECN(ERET) #define CPACR_EL1_DEFAULT (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN |\ CPACR_EL1_ZEN_EL1EN) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 2d8c09cf3e49..e75101f2aa6c 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -207,6 +207,15 @@ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu) return 1; } +static int kvm_handle_eret(struct kvm_vcpu *vcpu) +{ + if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERET) + return kvm_handle_ptrauth(vcpu); + + kvm_emulate_nested_eret(vcpu); + return 1; +} + static exit_handle_fn arm_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = kvm_handle_unknown_ec, [ESR_ELx_EC_WFx] = kvm_handle_wfx, @@ -222,6 +231,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_SMC64] = handle_smc, [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg, [ESR_ELx_EC_SVE] = handle_sve, + [ESR_ELx_EC_ERET] = kvm_handle_eret, [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort, [ESR_ELx_EC_DABT_LOW] = kvm_handle_guest_abort, [ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug, From patchwork Tue Jan 31 09:24:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8B3F3C636CC for ; Tue, 31 Jan 2023 09:55:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=uIXe5LNsdxAAgkvxGk9O6UYx2TimXM9RH517oJsIMbA=; b=BBY3BzcxzhiNW2 rq+WN/7DKVTpxaw+hIQr0uHlmyQjiPOtJBb/U1PO5hVgJ8//d0iRnW2Nafny2/KefanZb/xfDOSue gpM54ncTf2moLKTLsm5046shIJyVGHL6OYk/3uZ+SfrjpmQzsHZDP5M9Rn199utgqp8SjyEhu2Zwh pS2VsdbO9kUPo9p6WBS7SU/YaWxVLWHuG10xUWi0Rts46grFckOZkaJo6rysPsKdrFAsxMZRynBNN gFkAgyn7NWIdzHJtHw5ks+jE+nMpkco+C79ezCjB/XrmnpPMR4AWk85tA00ZiT9Js+O6oVsjLj28G 5POcE63GP5H3P10lOXCw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnLl-007AxU-MU; Tue, 31 Jan 2023 09:54:50 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn90-0075Ih-TO for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:40 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6B81D6148B; Tue, 31 Jan 2023 09:41:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3EC0AC433D2; Tue, 31 Jan 2023 09:41:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158098; bh=bqx6jidbc2f+LQLxjJDORJmIUTqcTIYjVPiIBgFGjyA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gmKFHnXXVz0blDbRJ32NQO9lsKJP4itsbuR+hZPuvqfkiOXw7k0505YSATJOSSgPm 6j5QVj90MsSIeUFtipigqMiPZGNTuYkjAw77jt7tltcuW1x9hTiVlnqjCXoDGfduKB yNmIHyf3HTzXwd4DmAOzeQrMksXtdM/am7NrpJw2SMUTz2cpCfMQuRkx4ExYPo+NE2 ojN2awb3DL8OUO6WeLyNiC3qmmfRLSsKUbNmEuGpoAQc85jJtApwB8F7ivl4w6m6Xe LcdSpuABt+4pDqY8+WaJrXX+1oMVAA2tPcAapzuej2riJlSFrqhN8pbQh8t0SEl3ln FeVRW2oX64uGQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtR-0067U2-5i; Tue, 31 Jan 2023 09:25:33 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 13/69] KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers Date: Tue, 31 Jan 2023 09:24:08 +0000 Message-Id: <20230131092504.2880505-14-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014139_059142_CEC3FBF6 X-CRM114-Status: GOOD ( 15.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Some EL2 system registers immediately affect the current execution of the system, so we need to use their respective EL1 counterparts. For this we need to define a mapping between the two. In general, this only affects non-VHE guest hypervisors, as VHE system registers are compatible with the EL1 counterparts. These helpers will get used in subsequent patches. Co-developed-by: Andre Przywara Signed-off-by: Andre Przywara Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 48 +++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index fd601ea68d13..f7bb1f0d0954 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -2,6 +2,7 @@ #ifndef __ARM64_KVM_NESTED_H #define __ARM64_KVM_NESTED_H +#include #include static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu) @@ -11,4 +12,51 @@ static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu) test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)); } +/* Translation helpers from non-VHE EL2 to EL1 */ +static inline u64 tcr_el2_ps_to_tcr_el1_ips(u64 tcr_el2) +{ + return (u64)FIELD_GET(TCR_EL2_PS_MASK, tcr_el2) << TCR_IPS_SHIFT; +} + +static inline u64 translate_tcr_el2_to_tcr_el1(u64 tcr) +{ + return TCR_EPD1_MASK | /* disable TTBR1_EL1 */ + ((tcr & TCR_EL2_TBI) ? TCR_TBI0 : 0) | + tcr_el2_ps_to_tcr_el1_ips(tcr) | + (tcr & TCR_EL2_TG0_MASK) | + (tcr & TCR_EL2_ORGN0_MASK) | + (tcr & TCR_EL2_IRGN0_MASK) | + (tcr & TCR_EL2_T0SZ_MASK); +} + +static inline u64 translate_cptr_el2_to_cpacr_el1(u64 cptr_el2) +{ + u64 cpacr_el1 = 0; + + if (cptr_el2 & CPTR_EL2_TTA) + cpacr_el1 |= CPACR_ELx_TTA; + if (!(cptr_el2 & CPTR_EL2_TFP)) + cpacr_el1 |= CPACR_ELx_FPEN; + if (!(cptr_el2 & CPTR_EL2_TZ)) + cpacr_el1 |= CPACR_ELx_ZEN; + + return cpacr_el1; +} + +static inline u64 translate_sctlr_el2_to_sctlr_el1(u64 val) +{ + /* Only preserve the minimal set of bits we support */ + val &= (SCTLR_ELx_M | SCTLR_ELx_A | SCTLR_ELx_C | SCTLR_ELx_SA | + SCTLR_ELx_I | SCTLR_ELx_IESB | SCTLR_ELx_WXN | SCTLR_ELx_EE); + val |= SCTLR_EL1_RES1; + + return val; +} + +static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0) +{ + /* Clear the ASID field */ + return ttbr0 & ~GENMASK_ULL(63, 48); +} + #endif /* __ARM64_KVM_NESTED_H */ From patchwork Tue Jan 31 09:24:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122830 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5A7B4C636CC for ; Tue, 31 Jan 2023 10:26:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=iSUp71aSaof53xbyQYlcEceqzEbrh4uC/g5xy56PdCA=; b=brRhhX8sb8fFtX KYiYeWA2D8A2tqXGBcpsWtJcIydiPzOWQziF1lTnbKfi5HorRt2AtOLxebEYbkJVI4YPTRo3q+FPM 0jo5KirpPdVMcl+tDLVqeObc5ruMTuR7id5snpgvROAweQpr4he5gF9qOzzedciugbH9PzuJ4oevV H6n3AK2+5dAOThleeuYUJKV+ixYcDpXq7ZCzjW1KUKdgn/swkJO+sSNAMWI9DJuq51ChpQDyfX+u6 r4bNhxwWon6Mfo2DtcIr0TUFHAg7I1WgaYEaontpYH1iwIsT8wPG0kGGutfaDZKxDFe2+ndxCWZvt vxpdSDlmGVMxkPwBqk8A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnpG-007Nz5-N7; Tue, 31 Jan 2023 10:25:18 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnAN-00760y-MS for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:43:07 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 109FA61490; Tue, 31 Jan 2023 09:43:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 585BAC433D2; Tue, 31 Jan 2023 09:43:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158182; bh=IUtvb7PsSsSK+uHDS4ibr2J82DAxYycRjntArVr2heY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pKwmkyatO7alq8CrJ3jxxn3OETU6xf6bVvh0PWfJUajZHxoERY3HznMvtyC/rurdC 2wtJUx4TqcyDgEUYkmL9m2e/XpMhdtJbxrDlpxlKAIvGIiwMN9SuR+ULCernnLIodK KbvuQnkemM4Cv25hYsF7l7ywkfy9UCSQprFVyNyUwIUyB/C7geHcB+iY2wcYh5V3M7 kL4tqovMhWjk/SyjYIpPyyM2JNgkxXakgFa2Z9wNU6nDxF0Npacg9VsN2ZhYBYhxEw tCLNZt/xRcnC6fg6YeXxj2dmKVwFOxmZRV9NqNB7r5zROvPwlbiT21wefxDhAiPNBh FGj9/gSJ5atlQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtR-0067U2-Fv; Tue, 31 Jan 2023 09:25:33 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 14/69] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg() Date: Tue, 31 Jan 2023 09:24:09 +0000 Message-Id: <20230131092504.2880505-15-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014303_890549_5DFD3675 X-CRM114-Status: GOOD ( 27.68 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org KVM internally uses accessor functions when reading or writing the guest's system registers. This takes care of accessing either the stored copy or using the "live" EL1 system registers when the host uses VHE. With the introduction of virtual EL2 we add a bunch of EL2 system registers, which now must also be taken care of: - If the guest is running in vEL2, and we access an EL1 sysreg, we must revert to the stored version of that, and not use the CPU's copy. - If the guest is running in vEL1, and we access an EL2 sysreg, we must also use the stored version, since the CPU carries the EL1 copy. - Some EL2 system registers are supposed to affect the current execution of the system, so we need to put them into their respective EL1 counterparts. For this we need to define a mapping between the two. This is done using the newly introduced struct el2_sysreg_map. - Some EL2 system registers have a different format than their EL1 counterpart, so we need to translate them before writing them to the CPU. This is done using an (optional) translate function in the map. - There are the three special registers SP_EL2, SPSR_EL2 and ELR_EL2, which need some separate handling (SPSR_EL2 is being handled in a separate patch). All of these cases are now wrapped into the existing accessor functions, so KVM users wouldn't need to care whether they access EL2 or EL1 registers and also which state the guest is in. This handles what was formerly known as the "shadow state" dynamically, without requiring a separate copy for each vCPU EL. Reviewed-by: Ganapatrao Kulkarni Reviewed-by: Alexandru Elisei Reviewed-by: Russell King (Oracle) Co-developed-by: Andre Przywara Signed-off-by: Andre Przywara Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 141 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 137 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 2a1e8f77b56c..6704b64e3982 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -63,23 +63,156 @@ static bool write_to_read_only(struct kvm_vcpu *vcpu, return false; } +#define PURE_EL2_SYSREG(el2) \ + case el2: { \ + *el1r = el2; \ + return true; \ + } + +#define MAPPED_EL2_SYSREG(el2, el1, fn) \ + case el2: { \ + *xlate = fn; \ + *el1r = el1; \ + return true; \ + } + +static bool get_el2_to_el1_mapping(unsigned int reg, + unsigned int *el1r, u64 (**xlate)(u64)) +{ + switch (reg) { + PURE_EL2_SYSREG( VPIDR_EL2 ); + PURE_EL2_SYSREG( VMPIDR_EL2 ); + PURE_EL2_SYSREG( ACTLR_EL2 ); + PURE_EL2_SYSREG( HCR_EL2 ); + PURE_EL2_SYSREG( MDCR_EL2 ); + PURE_EL2_SYSREG( HSTR_EL2 ); + PURE_EL2_SYSREG( HACR_EL2 ); + PURE_EL2_SYSREG( VTTBR_EL2 ); + PURE_EL2_SYSREG( VTCR_EL2 ); + PURE_EL2_SYSREG( RVBAR_EL2 ); + PURE_EL2_SYSREG( TPIDR_EL2 ); + PURE_EL2_SYSREG( HPFAR_EL2 ); + PURE_EL2_SYSREG( ELR_EL2 ); + PURE_EL2_SYSREG( SPSR_EL2 ); + PURE_EL2_SYSREG( CNTHCTL_EL2 ); + MAPPED_EL2_SYSREG(SCTLR_EL2, SCTLR_EL1, + translate_sctlr_el2_to_sctlr_el1 ); + MAPPED_EL2_SYSREG(CPTR_EL2, CPACR_EL1, + translate_cptr_el2_to_cpacr_el1 ); + MAPPED_EL2_SYSREG(TTBR0_EL2, TTBR0_EL1, + translate_ttbr0_el2_to_ttbr0_el1 ); + MAPPED_EL2_SYSREG(TTBR1_EL2, TTBR1_EL1, NULL ); + MAPPED_EL2_SYSREG(TCR_EL2, TCR_EL1, + translate_tcr_el2_to_tcr_el1 ); + MAPPED_EL2_SYSREG(VBAR_EL2, VBAR_EL1, NULL ); + MAPPED_EL2_SYSREG(AFSR0_EL2, AFSR0_EL1, NULL ); + MAPPED_EL2_SYSREG(AFSR1_EL2, AFSR1_EL1, NULL ); + MAPPED_EL2_SYSREG(ESR_EL2, ESR_EL1, NULL ); + MAPPED_EL2_SYSREG(FAR_EL2, FAR_EL1, NULL ); + MAPPED_EL2_SYSREG(MAIR_EL2, MAIR_EL1, NULL ); + MAPPED_EL2_SYSREG(AMAIR_EL2, AMAIR_EL1, NULL ); + default: + return false; + } +} + u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) { u64 val = 0x8badf00d8badf00d; + u64 (*xlate)(u64) = NULL; + unsigned int el1r; - if (vcpu_get_flag(vcpu, SYSREGS_ON_CPU) && - __vcpu_read_sys_reg_from_cpu(reg, &val)) + if (!vcpu_get_flag(vcpu, SYSREGS_ON_CPU)) + goto memory_read; + + if (unlikely(get_el2_to_el1_mapping(reg, &el1r, &xlate))) { + if (!is_hyp_ctxt(vcpu)) + goto memory_read; + + /* + * ELR_EL2 is special cased for now. + */ + switch (reg) { + case ELR_EL2: + return read_sysreg_el1(SYS_ELR); + } + + /* + * If this register does not have an EL1 counterpart, + * then read the stored EL2 version. + */ + if (reg == el1r) + goto memory_read; + + /* + * If we have a non-VHE guest and that the sysreg + * requires translation to be used at EL1, use the + * in-memory copy instead. + */ + if (!vcpu_el2_e2h_is_set(vcpu) && xlate) + goto memory_read; + + /* Get the current version of the EL1 counterpart. */ + WARN_ON(!__vcpu_read_sys_reg_from_cpu(el1r, &val)); + return val; + } + + /* EL1 register can't be on the CPU if the guest is in vEL2. */ + if (unlikely(is_hyp_ctxt(vcpu))) + goto memory_read; + + if (__vcpu_read_sys_reg_from_cpu(reg, &val)) return val; +memory_read: return __vcpu_sys_reg(vcpu, reg); } void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) { - if (vcpu_get_flag(vcpu, SYSREGS_ON_CPU) && - __vcpu_write_sys_reg_to_cpu(val, reg)) + u64 (*xlate)(u64) = NULL; + unsigned int el1r; + + if (!vcpu_get_flag(vcpu, SYSREGS_ON_CPU)) + goto memory_write; + + if (unlikely(get_el2_to_el1_mapping(reg, &el1r, &xlate))) { + if (!is_hyp_ctxt(vcpu)) + goto memory_write; + + /* + * Always store a copy of the write to memory to avoid having + * to reverse-translate virtual EL2 system registers for a + * non-VHE guest hypervisor. + */ + __vcpu_sys_reg(vcpu, reg) = val; + + switch (reg) { + case ELR_EL2: + write_sysreg_el1(val, SYS_ELR); + return; + } + + /* No EL1 counterpart? We're done here.? */ + if (reg == el1r) + return; + + if (!vcpu_el2_e2h_is_set(vcpu) && xlate) + val = xlate(val); + + /* Redirect this to the EL1 version of the register. */ + WARN_ON(!__vcpu_write_sys_reg_to_cpu(val, el1r)); + return; + } + + /* EL1 register can't be on the CPU if the guest is in vEL2. */ + if (unlikely(is_hyp_ctxt(vcpu))) + goto memory_write; + + if (__vcpu_write_sys_reg_to_cpu(val, reg)) return; +memory_write: __vcpu_sys_reg(vcpu, reg) = val; } From patchwork Tue Jan 31 09:24:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E599CC636CC for ; Tue, 31 Jan 2023 09:42:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=v0mpS8Oq0mOY3YexBxE1KBymY7CaNk2ItDb9klEGg7g=; b=Zx+UyTSfRcX7FV SwKwC0OsFR8POAuL/O6zzHkROVyqI8gHF4y6gJJ1dg5/XDp0d3v9WLPUi5j3NhA6ZLZP0vTgdSpAp O3YXvXPuRVXC4EnT+4Eo3H8nW1iZ6Nkw/wE8DegO6jSdd/9RthwlIakudOdF3oJQjFxAcQe2q38sB T+CeGfsjukbL1rn4YrxTmkPq5XgqNJVVgHUXvSgOuzO/qM4D6bd7sYyXFs/9mXYTy4hq268f25C4H PqPJd7Wqm8HjhFlThn0MkFBuLSRAWjTQLYEYUndrYbFQLmyK8X8F4mH4zoDUurvw0vu0rOebddrB0 H8Rc5JOdB3YZ+m5tFNqA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn98-0075LN-R8; Tue, 31 Jan 2023 09:41:47 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn89-0074q0-9H for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:40:47 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A7F01B818EB; Tue, 31 Jan 2023 09:40:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 600BDC433D2; Tue, 31 Jan 2023 09:40:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158042; bh=2casMehS8IylmJ+68hnzrywmz9DomVYzfqnqCRV3bno=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KTzM6Wa3dR3BAgoSq8imiW+RR6fop7ej/SR/bGgFH0cfp/snKNd3XNwJWYeiVq72i Dg3DrggrQ+KlsnDZuhLeBQM5sdP4AZMdMWNM3DG8il488Eg8smJg+mN+/LOA8dvGsk 3XWAqGHITyGuO/IlnkX+GCg908iJpNQ1aTTgqAz4oP4JpZ12KsCo2vmS3JhK9Mo67l V4a3uPpbonXGq8JyTOLBI8WcElhyPhoEpNhzGj+cy0Dr7ZTAra4zsuaGaWz6C/74l6 YW8C79JzR+1u4m941/2+CtZOYU++yc+kjbjEMasrTUI0nTQ/WETc2yurOHX3HfYZUL bZO3ROnctAtiA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtR-0067U2-Ot; Tue, 31 Jan 2023 09:25:33 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 15/69] KVM: arm64: nv: Handle SPSR_EL2 specially Date: Tue, 31 Jan 2023 09:24:10 +0000 Message-Id: <20230131092504.2880505-16-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014045_687957_15A6EA10 X-CRM114-Status: GOOD ( 24.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org SPSR_EL2 needs special attention when running nested on ARMv8.3: If taking an exception while running at vEL2 (actually EL1), the HW will update the SPSR_EL1 register with the EL1 mode. We need to track this in order to make sure that accesses to the virtual view of SPSR_EL2 is correct. To do so, we place an illegal value in SPSR_EL1.M, and patch it accordingly if required when accessing it. Reviewed-by: Alexandru Elisei Reviewed-by: Russell King (Oracle) Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 37 ++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 23 +++++++++++++++-- 2 files changed, 58 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index b31b32ecbe2d..2f5d3e27c017 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -245,6 +245,43 @@ static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu) return __is_hyp_ctxt(&vcpu->arch.ctxt); } +static inline u64 __fixup_spsr_el2_write(struct kvm_cpu_context *ctxt, u64 val) +{ + if (!__vcpu_el2_e2h_is_set(ctxt)) { + /* + * Clear the .M field when writing SPSR to the CPU, so that we + * can detect when the CPU clobbered our SPSR copy during a + * local exception. + */ + val &= ~0xc; + } + + return val; +} + +static inline u64 __fixup_spsr_el2_read(const struct kvm_cpu_context *ctxt, u64 val) +{ + if (__vcpu_el2_e2h_is_set(ctxt)) + return val; + + /* + * SPSR.M == 0 means the CPU has not touched the SPSR, so the + * register has still the value we saved on the last write. + */ + if ((val & 0xc) == 0) + return ctxt_sys_reg(ctxt, SPSR_EL2); + + /* + * Otherwise there was a "local" exception on the CPU, + * which from the guest's point of view was being taken from + * EL2 to EL2, although it actually happened to be from + * EL1 to EL1. + * So we need to fix the .M field in SPSR, to make it look + * like EL2, which is what the guest would expect. + */ + return (val & ~0x0c) | CurrentEL_EL2; +} + /* * The layout of SPSR for an AArch32 state is different when observed from an * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32 diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 6704b64e3982..4975c5222f73 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -130,11 +130,14 @@ u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) goto memory_read; /* - * ELR_EL2 is special cased for now. + * ELR_EL2 and SPSR_EL2 are special cased for now. */ switch (reg) { case ELR_EL2: return read_sysreg_el1(SYS_ELR); + case SPSR_EL2: + val = read_sysreg_el1(SYS_SPSR); + return __fixup_spsr_el2_read(&vcpu->arch.ctxt, val); } /* @@ -191,6 +194,10 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) case ELR_EL2: write_sysreg_el1(val, SYS_ELR); return; + case SPSR_EL2: + val = __fixup_spsr_el2_write(&vcpu->arch.ctxt, val); + write_sysreg_el1(val, SYS_SPSR); + return; } /* No EL1 counterpart? We're done here.? */ @@ -1825,6 +1832,18 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, return true; } +static bool access_spsr_el2(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2); + else + p->regval = vcpu_read_sys_reg(vcpu, SPSR_EL2); + + return true; +} + /* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2 @@ -2267,7 +2286,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL2_REG(VTCR_EL2, access_rw, reset_val, 0), { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 }, - EL2_REG(SPSR_EL2, access_rw, reset_val, 0), + EL2_REG(SPSR_EL2, access_spsr_el2, reset_val, 0), EL2_REG(ELR_EL2, access_rw, reset_val, 0), { SYS_DESC(SYS_SP_EL1), access_sp_el1}, From patchwork Tue Jan 31 09:24:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 867E5C38142 for ; Tue, 31 Jan 2023 09:45:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PeL9JMz/t14R3NmQSmxvBHqXjIOH/uaRPQzojaL9L1Q=; b=nYPYx1zm2rh/pt VKHfvHACHL8CxQEnH7jK5aY9/vKiLkZTgMwonQ9zDNTzVjDe2zGMxTFsMYbxjmafLxJaKZiMqGSsi 0eRXhXYq/xkmCxoJsK8b8w7IOPOcJ3QPbQCoQcpaUTCt9u0ssue1Os30j8O8HbJmldQ+xjAaxTOPs hAYXHa6hfEfSjF7zORSI/WUT9V2Uk/TP691m2X6kiaLEjuWm5sA4/7fbNyh4eCO8oJAwN0CAJJ7KH stAUMlEv9PrbMlsuwT99hr6fS3tXrA0z9b8RFoYj10MTEocJbqIsVf2uIlsMlwZNWuDAHschrMzUW WPeK4PMuTuNDhL7/ZOAA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnBS-0076Wp-Ve; Tue, 31 Jan 2023 09:44:11 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8J-0074vC-L4 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:40:57 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id EDEE5B81AC8; Tue, 31 Jan 2023 09:40:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 996CDC433D2; Tue, 31 Jan 2023 09:40:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158052; bh=CZqRuMv0iI2pw1EuDKhGV5K14+ejIqjRP+HkaP/T+Ps=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qaIWrPtAs5oaSoGqCoVEQhpz22vcA9d8GJZPu2lqevEMC5JsFSUcj9xIwSVzAKOZ9 lxsgOtktYdzsfwPd5Bz0gzWICu8XoGn1wUCL4Oa7AjAzHw2qCP2L4elUz75LZMEPXT czSiOmvoo2qz8QuYhlNGBN85Bdg204e5Sd0y2EM762f8ejJp/ueaVhjNwZNK2zsR8w msG1fzXyvcPM0xOOcTYlsflbBGnnN7QTW1U9dCWBBJxr/YVY7Pl0M5Dh2HWzl4lLVI 0gOCGXVdE1UZMfv51wwq3vy4lg/nJpDHS2UuJ5wz2CGu2PPQaNh4LIz1iQ2pZhAFkz utAXklKtMS0Ag== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtS-0067U2-1E; Tue, 31 Jan 2023 09:25:34 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 16/69] KVM: arm64: nv: Handle HCR_EL2.E2H specially Date: Tue, 31 Jan 2023 09:24:11 +0000 Message-Id: <20230131092504.2880505-17-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014056_054884_4270A558 X-CRM114-Status: GOOD ( 15.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org HCR_EL2.E2H is nasty, as a flip of this bit completely changes the way we deal with a lot of the state. So when the guest flips this bit (sysregs are live), do the put/load dance so that we have a consistent state. Yes, this is slow. Don't do it. Suggested-by: Alexandru Elisei Reviewed-by: Russell King (Oracle) Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 4975c5222f73..26d189f908a7 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -180,9 +180,24 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) goto memory_write; if (unlikely(get_el2_to_el1_mapping(reg, &el1r, &xlate))) { + bool need_put_load; + if (!is_hyp_ctxt(vcpu)) goto memory_write; + /* + * HCR_EL2.E2H is nasty: it changes the way we interpret a + * lot of the EL2 state, so treat is as a full state + * transition. + */ + need_put_load = ((reg == HCR_EL2) && + vcpu_el2_e2h_is_set(vcpu) != !!(val & HCR_E2H)); + + if (need_put_load) { + preempt_disable(); + kvm_arch_vcpu_put(vcpu); + } + /* * Always store a copy of the write to memory to avoid having * to reverse-translate virtual EL2 system registers for a @@ -190,6 +205,11 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) */ __vcpu_sys_reg(vcpu, reg) = val; + if (need_put_load) { + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + preempt_enable(); + } + switch (reg) { case ELR_EL2: write_sysreg_el1(val, SYS_ELR); From patchwork Tue Jan 31 09:24:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1924BC38142 for ; Tue, 31 Jan 2023 09:55:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=zX5mBpYjgVlHWmJQwu57YnW51W6taLOzyUoTd6Qe2iQ=; b=duxYf/wZR2rsWV Ow5/vvcfvRFgaY537KfNHDjdKYsXw9guDRZjG57pIpLFOhcNdTCvGbiaQfj8TiU42nX+1g/RjIKmO Hvns2/2ofBxc7WJZ+BnwRDRUir5xzYhqBHWYlAVvYzmDy/V/ylREUHcKe6GBshO/MWCxCeK7l01N2 HrWa4puC06j20lXgSBu93UosTVZWgD+WkVl/KWCRu2wIjX5DLEMhsa4r1pHcwZUrIdwKSoecESgMy 5M2zdXi0wDJwCAbjThw/hw2gEYWLUJgfvg7YEhxghiRpj/2qiQzwifanux5KcR45QmgktdL7SXBOw c2uJbBFA8Bo1xRlB4IOA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnLC-007Afx-Ub; Tue, 31 Jan 2023 09:54:15 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8y-0075HN-VS for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:39 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 682D761485; Tue, 31 Jan 2023 09:41:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B0A11C433EF; Tue, 31 Jan 2023 09:41:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158095; bh=N5tRQKnXlunBVyOAUzNmdn8T0F7OZgAp4E5DfQ8qmhs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EJGoNGmV6zpuLpuwkWs/9D2wHaCbU3oXqH+FsoBL6akS1MZJpLjQOCwsSWCGUNdiy l7s247N15//4MEvqBIhfaqL7pjKbN8TJWwx/aaRI0XwZ9V+2l94hScPfBPDUkioMRp vUq/swS0WuHw/bQ9vNHnm7pgmfk7+trkg4z6XZ+iOVLweWYLYpZENGV0SFGc9qjvM2 8xZ3a334s60v3QhDwvVqUo1DpEmsVoXdw5GWJyQS8rp53lGsiECyatasBBwR0m2UdR E1vTxOSalV4e92uFrYrOLk0t4HxtQzVoIyYy2yt1TciIN7VMnJE3tbrfQbjwzW/UFI Kr6Io2E+sO06A== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtS-0067U2-9C; Tue, 31 Jan 2023 09:25:34 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 17/69] KVM: arm64: nv: Save/Restore vEL2 sysregs Date: Tue, 31 Jan 2023 09:24:12 +0000 Message-Id: <20230131092504.2880505-18-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014137_160938_65F25918 X-CRM114-Status: GOOD ( 23.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Whenever we need to restore the guest's system registers to the CPU, we now need to take care of the EL2 system registers as well. Most of them are accessed via traps only, but some have an immediate effect and also a guest running in VHE mode would expect them to be accessible via their EL1 encoding, which we do not trap. For vEL2 we write the virtual EL2 registers with an identical format directly into their EL1 counterpart, and translate the few registers that have a different format for the same effect on the execution when running a non-VHE guest guest hypervisor. Based on an initial patch from Andre Przywara, rewritten many times since. Reviewed-by: Alexandru Elisei Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 5 +- arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 2 +- arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 125 ++++++++++++++++++++- 3 files changed, 127 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index 147cb4c846c6..c506eabc4e21 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -91,9 +91,10 @@ static inline void __sysreg_restore_user_state(struct kvm_cpu_context *ctxt) write_sysreg(ctxt_sys_reg(ctxt, TPIDRRO_EL0), tpidrro_el0); } -static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) +static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt, + u64 mpidr) { - write_sysreg(ctxt_sys_reg(ctxt, MPIDR_EL1), vmpidr_el2); + write_sysreg(mpidr, vmpidr_el2); if (has_vhe() || !cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) { diff --git a/arch/arm64/kvm/hyp/nvhe/sysreg-sr.c b/arch/arm64/kvm/hyp/nvhe/sysreg-sr.c index 29305022bc04..dba101565de3 100644 --- a/arch/arm64/kvm/hyp/nvhe/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/nvhe/sysreg-sr.c @@ -28,7 +28,7 @@ void __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt) void __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt) { - __sysreg_restore_el1_state(ctxt); + __sysreg_restore_el1_state(ctxt, ctxt_sys_reg(ctxt, MPIDR_EL1)); __sysreg_restore_common_state(ctxt); __sysreg_restore_user_state(ctxt); __sysreg_restore_el2_return_state(ctxt); diff --git a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c index 7b44f6b3b547..3f9d6952fbd7 100644 --- a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c @@ -13,6 +13,96 @@ #include #include #include +#include + +static void __sysreg_save_vel2_state(struct kvm_cpu_context *ctxt) +{ + /* These registers are common with EL1 */ + ctxt_sys_reg(ctxt, PAR_EL1) = read_sysreg(par_el1); + ctxt_sys_reg(ctxt, TPIDR_EL1) = read_sysreg(tpidr_el1); + + ctxt_sys_reg(ctxt, ESR_EL2) = read_sysreg_el1(SYS_ESR); + ctxt_sys_reg(ctxt, AFSR0_EL2) = read_sysreg_el1(SYS_AFSR0); + ctxt_sys_reg(ctxt, AFSR1_EL2) = read_sysreg_el1(SYS_AFSR1); + ctxt_sys_reg(ctxt, FAR_EL2) = read_sysreg_el1(SYS_FAR); + ctxt_sys_reg(ctxt, MAIR_EL2) = read_sysreg_el1(SYS_MAIR); + ctxt_sys_reg(ctxt, VBAR_EL2) = read_sysreg_el1(SYS_VBAR); + ctxt_sys_reg(ctxt, CONTEXTIDR_EL2) = read_sysreg_el1(SYS_CONTEXTIDR); + ctxt_sys_reg(ctxt, AMAIR_EL2) = read_sysreg_el1(SYS_AMAIR); + + /* + * In VHE mode those registers are compatible between EL1 and EL2, + * and the guest uses the _EL1 versions on the CPU naturally. + * So we save them into their _EL2 versions here. + * For nVHE mode we trap accesses to those registers, so our + * _EL2 copy in sys_regs[] is always up-to-date and we don't need + * to save anything here. + */ + if (__vcpu_el2_e2h_is_set(ctxt)) { + ctxt_sys_reg(ctxt, SCTLR_EL2) = read_sysreg_el1(SYS_SCTLR); + ctxt_sys_reg(ctxt, CPTR_EL2) = read_sysreg_el1(SYS_CPACR); + ctxt_sys_reg(ctxt, TTBR0_EL2) = read_sysreg_el1(SYS_TTBR0); + ctxt_sys_reg(ctxt, TTBR1_EL2) = read_sysreg_el1(SYS_TTBR1); + ctxt_sys_reg(ctxt, TCR_EL2) = read_sysreg_el1(SYS_TCR); + ctxt_sys_reg(ctxt, CNTHCTL_EL2) = read_sysreg_el1(SYS_CNTKCTL); + } + + ctxt_sys_reg(ctxt, SP_EL2) = read_sysreg(sp_el1); + ctxt_sys_reg(ctxt, ELR_EL2) = read_sysreg_el1(SYS_ELR); + ctxt_sys_reg(ctxt, SPSR_EL2) = __fixup_spsr_el2_read(ctxt, read_sysreg_el1(SYS_SPSR)); +} + +static void __sysreg_restore_vel2_state(struct kvm_cpu_context *ctxt) +{ + u64 val; + + /* These registers are common with EL1 */ + write_sysreg(ctxt_sys_reg(ctxt, PAR_EL1), par_el1); + write_sysreg(ctxt_sys_reg(ctxt, TPIDR_EL1), tpidr_el1); + + write_sysreg(read_cpuid_id(), vpidr_el2); + write_sysreg(ctxt_sys_reg(ctxt, MPIDR_EL1), vmpidr_el2); + write_sysreg_el1(ctxt_sys_reg(ctxt, MAIR_EL2), SYS_MAIR); + write_sysreg_el1(ctxt_sys_reg(ctxt, VBAR_EL2), SYS_VBAR); + write_sysreg_el1(ctxt_sys_reg(ctxt, CONTEXTIDR_EL2),SYS_CONTEXTIDR); + write_sysreg_el1(ctxt_sys_reg(ctxt, AMAIR_EL2), SYS_AMAIR); + + if (__vcpu_el2_e2h_is_set(ctxt)) { + /* + * In VHE mode those registers are compatible between + * EL1 and EL2. + */ + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL2), SYS_SCTLR); + write_sysreg_el1(ctxt_sys_reg(ctxt, CPTR_EL2), SYS_CPACR); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL2), SYS_TTBR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL2), SYS_TTBR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL2), SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, CNTHCTL_EL2), SYS_CNTKCTL); + } else { + /* + * CNTHCTL_EL2 only affects EL1 when running nVHE, so + * no need to restore it. + */ + val = translate_sctlr_el2_to_sctlr_el1(ctxt_sys_reg(ctxt, SCTLR_EL2)); + write_sysreg_el1(val, SYS_SCTLR); + val = translate_cptr_el2_to_cpacr_el1(ctxt_sys_reg(ctxt, CPTR_EL2)); + write_sysreg_el1(val, SYS_CPACR); + val = translate_ttbr0_el2_to_ttbr0_el1(ctxt_sys_reg(ctxt, TTBR0_EL2)); + write_sysreg_el1(val, SYS_TTBR0); + val = translate_tcr_el2_to_tcr_el1(ctxt_sys_reg(ctxt, TCR_EL2)); + write_sysreg_el1(val, SYS_TCR); + } + + write_sysreg_el1(ctxt_sys_reg(ctxt, ESR_EL2), SYS_ESR); + write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR0_EL2), SYS_AFSR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR1_EL2), SYS_AFSR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, FAR_EL2), SYS_FAR); + write_sysreg(ctxt_sys_reg(ctxt, SP_EL2), sp_el1); + write_sysreg_el1(ctxt_sys_reg(ctxt, ELR_EL2), SYS_ELR); + + val = __fixup_spsr_el2_write(ctxt, ctxt_sys_reg(ctxt, SPSR_EL2)); + write_sysreg_el1(val, SYS_SPSR); +} /* * VHE: Host and guest must save mdscr_el1 and sp_el0 (and the PC and @@ -65,6 +155,7 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu) { struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt; struct kvm_cpu_context *host_ctxt; + u64 mpidr; host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; __sysreg_save_user_state(host_ctxt); @@ -77,7 +168,29 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu) */ __sysreg32_restore_state(vcpu); __sysreg_restore_user_state(guest_ctxt); - __sysreg_restore_el1_state(guest_ctxt); + + if (unlikely(__is_hyp_ctxt(guest_ctxt))) { + __sysreg_restore_vel2_state(guest_ctxt); + } else { + if (vcpu_has_nv(vcpu)) { + /* + * Only set VPIDR_EL2 for nested VMs, as this is the + * only time it changes. We'll restore the MIDR_EL1 + * view on put. + */ + write_sysreg(ctxt_sys_reg(guest_ctxt, VPIDR_EL2), vpidr_el2); + + /* + * As we're restoring a nested guest, set the value + * provided by the guest hypervisor. + */ + mpidr = ctxt_sys_reg(guest_ctxt, VMPIDR_EL2); + } else { + mpidr = ctxt_sys_reg(guest_ctxt, MPIDR_EL1); + } + + __sysreg_restore_el1_state(guest_ctxt, mpidr); + } vcpu_set_flag(vcpu, SYSREGS_ON_CPU); @@ -103,12 +216,20 @@ void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu) host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; deactivate_traps_vhe_put(vcpu); - __sysreg_save_el1_state(guest_ctxt); + if (unlikely(__is_hyp_ctxt(guest_ctxt))) + __sysreg_save_vel2_state(guest_ctxt); + else + __sysreg_save_el1_state(guest_ctxt); + __sysreg_save_user_state(guest_ctxt); __sysreg32_save_state(vcpu); /* Restore host user state */ __sysreg_restore_user_state(host_ctxt); + /* If leaving a nesting guest, restore MPIDR_EL1 default view */ + if (vcpu_has_nv(vcpu)) + write_sysreg(read_cpuid_id(), vpidr_el2); + vcpu_clear_flag(vcpu, SYSREGS_ON_CPU); } From patchwork Tue Jan 31 09:24:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D02F4C636CC for ; Tue, 31 Jan 2023 09:49:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=drWJwiqcGkeAmXitvUt14uZ/ttzHwPtdG3oJjGqtVgY=; b=JlLSp7nuqPj6Ut raDq08KSMH3aAS20LGMUpY8n1TM97+1FhlkG3GX0Ss7n2Rf1/2f/7Lm0r9gKW5Rqy3aeiyxWMMMiM yt2MswB0K2HBaO3nUCikDWZ7WbuYCgBoeNRlTrQ7ue8lhFnR7GkL2Nwi9eTDQL67BgIPcA3bx8CDs 6bU5Reg0G/N20w3qUSPw4I4OfPm5xITCDVgo+YR8JEHuJFhQZlzBRnLSJYq7QfEad+sbU/M6T8p+6 SLm9A8srCxGWeM1UkKV4TpVFa4BUDhtLgBILZwiKYDI15aZy17Yz1nbn3J28VntXCKPqky3/qxpDL N5bFd8/JYMZY8fLosiKg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnFk-0078Gt-F9; Tue, 31 Jan 2023 09:48:37 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8W-00753P-8g for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:12 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id AAD5061445; Tue, 31 Jan 2023 09:41:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1FB0BC433D2; Tue, 31 Jan 2023 09:41:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158067; bh=xBhjj6OSLXZCK722O+yY3j4d+austxISjMyQn27kvjI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=a8LARrqCKSD/43xVEGLlmbmIpsKSaY3PaoONEIy1xkF8LYw+B1cftPEp2/bridyw5 2XbahgkGl1pZJ8/YgufHVAoqO4Yq/w8XowPIlyM+h1x09zbIECiaSWrA96dhVYbx+R JKZZlyj17tSpvr5/7CWZSSXgVGdyu8TYC6+B0bgADCOWRpoS/awGnhkQMbx1Y/K8lF 3QT5IFHJyT75jfRkAIUvRDo12GOf6wKijs16UuU2DxEYu4o8vtrkalpfkxJpPwvVLh EyV8MSWscrxzHulAJpn3dEbabGWIe/CnxY6BEsgTneCAGuEIDdhvPMF6UNzn/V5Ypl Y2k1DXIF0BKTA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtS-0067U2-IH; Tue, 31 Jan 2023 09:25:34 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 18/69] KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor Date: Tue, 31 Jan 2023 09:24:13 +0000 Message-Id: <20230131092504.2880505-19-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014108_449550_DC42632E X-CRM114-Status: GOOD ( 19.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We can no longer blindly copy the VCPU's PSTATE into SPSR_EL2 and return to the guest and vice versa when taking an exception to the hypervisor, because we emulate virtual EL2 in EL1 and therefore have to translate the mode field from EL2 to EL1 and vice versa. This requires keeping track of the state we enter the guest, for which we transiently use a dedicated flag. Reviewed-by: Russell King (Oracle) Reviewed-by: Alexandru Elisei Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 19 ++++++++++++++++- arch/arm64/kvm/hyp/vhe/switch.c | 24 ++++++++++++++++++++++ 3 files changed, 44 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 640997251724..a1892a8f6032 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -649,6 +649,8 @@ struct kvm_vcpu_arch { #define DEBUG_STATE_SAVE_SPE __vcpu_single_flag(iflags, BIT(5)) /* Save TRBE context if active */ #define DEBUG_STATE_SAVE_TRBE __vcpu_single_flag(iflags, BIT(6)) +/* vcpu running in HYP context */ +#define VCPU_HYP_CONTEXT __vcpu_single_flag(iflags, BIT(7)) /* SVE enabled for host EL0 */ #define HOST_SVE_ENABLED __vcpu_single_flag(sflags, BIT(0)) diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index c506eabc4e21..d67c620f2af7 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -155,9 +155,26 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt, write_sysreg_el1(ctxt_sys_reg(ctxt, SPSR_EL1), SYS_SPSR); } +/* Read the VCPU state's PSTATE, but translate (v)EL2 to EL1. */ +static inline u64 to_hw_pstate(const struct kvm_cpu_context *ctxt) +{ + u64 mode = ctxt->regs.pstate & (PSR_MODE_MASK | PSR_MODE32_BIT); + + switch (mode) { + case PSR_MODE_EL2t: + mode = PSR_MODE_EL1t; + break; + case PSR_MODE_EL2h: + mode = PSR_MODE_EL1h; + break; + } + + return (ctxt->regs.pstate & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; +} + static inline void __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt) { - u64 pstate = ctxt->regs.pstate; + u64 pstate = to_hw_pstate(ctxt); u64 mode = pstate & PSR_AA32_MODE_MASK; /* diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 81bf236d9c27..cd3f3117bf16 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -120,6 +120,25 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu) static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) { + /* + * If we were in HYP context on entry, adjust the PSTATE view + * so that the usual helpers work correctly. + */ + if (unlikely(vcpu_get_flag(vcpu, VCPU_HYP_CONTEXT))) { + u64 mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT); + + switch (mode) { + case PSR_MODE_EL1t: + mode = PSR_MODE_EL2t; + break; + case PSR_MODE_EL1h: + mode = PSR_MODE_EL2h; + break; + } + + *vcpu_cpsr(vcpu) &= ~(PSR_MODE_MASK | PSR_MODE32_BIT); + *vcpu_cpsr(vcpu) |= mode; + } } /* Switch to the guest for VHE systems running in EL2 */ @@ -154,6 +173,11 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) sysreg_restore_guest_state_vhe(guest_ctxt); __debug_switch_to_guest(vcpu); + if (is_hyp_ctxt(vcpu)) + vcpu_set_flag(vcpu, VCPU_HYP_CONTEXT); + else + vcpu_clear_flag(vcpu, VCPU_HYP_CONTEXT); + do { /* Jump in the fire! */ exit_code = __guest_enter(vcpu); From patchwork Tue Jan 31 09:24:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 89B98C38142 for ; Tue, 31 Jan 2023 09:48:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7B7x+c9RiNeqh2sy/OfRk5EKz0P3FxeTSFA1JZ3nMXw=; b=W/j9iyifcvxDMJ 9w6WP6pjDwNFPYhARr6/s4UUMAZuAHhXbz6O785HoskQYLN7TJaoWquv+q03qeECzSIDmv5QeG8MF ZxTx3WjCtDExvUc3xGWH9igGFHjEn6H8Pr/Dnp9C0PB3ez2rD4o9rjm1sG//70z/Ahd8kw4/7hvOc K9Yi9AVZZ02gSfpEk3FOftL6OfdgzQ6WCdJqwrpUrTd2cLfBu4qj54CWR3jSEY3V4+hhd6P/7gVcF 6IUqMkVHcoj9uNv8tYbogHACNIkkyXwtp5sXU+/szo1lLKXYZbnGIw99LJfPhL+3Am/keoGnube/3 2T5urUElj3nqa7dx6QcQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnDz-0077aa-BR; Tue, 31 Jan 2023 09:46:48 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8S-00751D-Ou for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:09 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 56AF3B81ACC; Tue, 31 Jan 2023 09:41:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 05C36C433D2; Tue, 31 Jan 2023 09:41:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158062; bh=uuVf166OaKEfm0pMgDt6BSPXE2oAhYfqL5lWMvrQvcA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rQ+BiRr3M+1hoFoov0Qh9f0RlLRKLGXicsyYuRNkIP0Ew2GCEUk488MZyqk2S3jOZ wK/+IwAH6NRUuDkfSNjnCY1fGRyBzTVBFvZ9bD0UISjnRPcX+1YxLkg74LKfhN6YTo v0sgdwl3RLfUHyHfTBvd6LVNmK/Cta2fh6sCEAlIJuKuzntdbyG3EcNP8HBlzBou6i 3Pqs+lNcpT+9GpuSWpOhd3HKT6sln4B2kL+qYIGQBOT4dp/OPmIVibk+niNHJg5fKD MLj0JjH5PQYUnC6Xs3YmGnVzin8Jl/tThyms20EgxwC41PHveqHeGkIvtDyaqCoz3h ifkC04lE1IZHA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtS-0067U2-SG; Tue, 31 Jan 2023 09:25:34 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 19/69] KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2 Date: Tue, 31 Jan 2023 09:24:14 +0000 Message-Id: <20230131092504.2880505-20-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014105_205745_9DB39EF5 X-CRM114-Status: GOOD ( 20.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall When running in virtual EL2 mode, we actually run the hardware in EL1 and therefore have to use the EL1 registers to ensure correct operation. By setting the HCR.TVM and HCR.TVRM we ensure that the virtual EL2 mode doesn't shoot itself in the foot when setting up what it believes to be a different mode's system register state (for example when preparing to switch to a VM). We can leverage the existing sysregs infrastructure to support trapped accesses to these registers. Reviewed-by: Russell King (Oracle) Reviewed-by: Alexandru Elisei Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/include/hyp/switch.h | 4 +--- arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 7 ++++++- arch/arm64/kvm/sys_regs.c | 19 ++++++++++++++++--- 4 files changed, 24 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 07d37ff88a3f..e0bcaf000251 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -118,10 +118,8 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu) } } -static inline void ___activate_traps(struct kvm_vcpu *vcpu) +static inline void ___activate_traps(struct kvm_vcpu *vcpu, u64 hcr) { - u64 hcr = vcpu->arch.hcr_el2; - if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM)) hcr |= HCR_TVM; diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index c2cb46ca4fb6..efac8fbe0b20 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -40,7 +40,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) { u64 val; - ___activate_traps(vcpu); + ___activate_traps(vcpu, vcpu->arch.hcr_el2); __activate_traps_common(vcpu); val = vcpu->arch.cptr_el2; diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index cd3f3117bf16..c8da8d350453 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -35,9 +35,14 @@ DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); static void __activate_traps(struct kvm_vcpu *vcpu) { + u64 hcr = vcpu->arch.hcr_el2; u64 val; - ___activate_traps(vcpu); + /* Trap VM sysreg accesses if an EL2 guest is not using VHE. */ + if (vcpu_is_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu)) + hcr |= HCR_TVM | HCR_TRVM; + + ___activate_traps(vcpu, hcr); val = read_sysreg(cpacr_el1); val |= CPACR_ELx_TTA; diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 26d189f908a7..f83d8fe47573 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -391,8 +391,15 @@ static void get_access_mask(const struct sys_reg_desc *r, u64 *mask, u64 *shift) /* * Generic accessor for VM registers. Only called as long as HCR_TVM - * is set. If the guest enables the MMU, we stop trapping the VM - * sys_regs and leave it in complete control of the caches. + * is set. + * + * This is set in two cases: either (1) we're running at vEL2, or (2) + * we're running at EL1 and the guest has its MMU off. + * + * (1) TVM/TRVM is set, as we need to virtualise some of the VM + * registers for the guest hypervisor + * (2) Once the guest enables the MMU, we stop trapping the VM sys_regs + * and leave it in complete control of the caches. */ static bool access_vm_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p, @@ -401,7 +408,13 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, bool was_enabled = vcpu_has_cache_enabled(vcpu); u64 val, mask, shift; - BUG_ON(!p->is_write); + /* We don't expect TRVM on the host */ + BUG_ON(!vcpu_is_el2(vcpu) && !p->is_write); + + if (!p->is_write) { + p->regval = vcpu_read_sys_reg(vcpu, r->reg); + return true; + } get_access_mask(r, &mask, &shift); From patchwork Tue Jan 31 09:24:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122734 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 25547C38142 for ; Tue, 31 Jan 2023 09:53:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AIdklbkGTn1Lowe+3OvGbgy3xuIyY8FtnXbOsLZC4Ec=; b=tuV8rVw5LkWRPT 4VhUX/WYYsu9+5Tgo8ZnAajcF0agDR2MbvzF7A1qN6+n/UzgpKTeQYEeeBgUYnBXfnVrGNdW7kGZ/ y6nUJHt61Do6cPFueZ2Wdh/11L418bTbjZ96j1BNEsHEb/sqwVEviJl6+P5qx1lWZcdcEHt4hCtkt V3WBcyddasbpsF1Z6rk5MBG2IfvYaQ+T4ZvaXIO/ppWmr9bcnOoTxRX0/FieQiksBd2BXr/MzfTgr X6s+grQFLvLtFz9ySYarfbHkT3IdURaXK7VfI+8FtZRsWsPie1PgJQWv8IwQty0ewzRYouRER3geD JWv5lAZEbG4aZtBGHhSQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnJe-0079pN-DX; Tue, 31 Jan 2023 09:52:38 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8t-0075EW-FW for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:33 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 135AEB81AF2; Tue, 31 Jan 2023 09:41:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B4516C433EF; Tue, 31 Jan 2023 09:41:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158088; bh=3fiUQI+IVEytzUGgbIMfdBGA45t4h/pe2cWbkWSI4CU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iLCspZsbW9LrANCbkF2DLTdZF28unLizpPQNBixippPj+ZEzOeTpNrYHWtP0G3i6J FsaEF96dNbXjogLLxBsV40feyyQLWlVP/LvQ5yP+IkwY8aR3iFxMxA6ew9uLsFcSzy ifSibLnlLZnlJ0d/E1EHwycE4Wn3FR5IwtzsVTxVUeREuDTJ08syK/SHjlkwq6+wlF nsGV0gnW76PyP226Ji8sMsf/LjRsFyCoAKn/e0H0F9YWZ5od68fJ4cQew8CIhJa+oP SSMRiQdIQwFoyffPTAXeVSwwZWyJVWyxBEIfw5kOlnMp6EXlOHrPJapyNCk2jdGmqM bv2v167lmmSRA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtT-0067U2-58; Tue, 31 Jan 2023 09:25:35 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 20/69] KVM: arm64: nv: Add accessors for SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2 Date: Tue, 31 Jan 2023 09:24:15 +0000 Message-Id: <20230131092504.2880505-21-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014131_850422_313D0D6A X-CRM114-Status: GOOD ( 15.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim For the same reason we trap virtual memory register accesses at virtual EL2, we need to trap SPSR_EL1, ELR_EL1 and VBAR_EL1 accesses. ARM v8.3 introduces the HCR_EL2.NV1 bit to be able to trap on those register accesses in EL1. Do not set this bit until the whole nesting support is completed, which happens further down the line... Reviewed-by: Russell King (Oracle) Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index f83d8fe47573..b559a289d996 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1865,6 +1865,30 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, return true; } +static bool access_elr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1); + else + p->regval = vcpu_read_sys_reg(vcpu, ELR_EL1); + + return true; +} + +static bool access_spsr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + __vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval; + else + p->regval = __vcpu_sys_reg(vcpu, SPSR_EL1); + + return true; +} + static bool access_spsr_el2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -2033,6 +2057,9 @@ static const struct sys_reg_desc sys_reg_descs[] = { PTRAUTH_KEY(APDB), PTRAUTH_KEY(APGA), + { SYS_DESC(SYS_SPSR_EL1), access_spsr}, + { SYS_DESC(SYS_ELR_EL1), access_elr}, + { SYS_DESC(SYS_AFSR0_EL1), access_vm_reg, reset_unknown, AFSR0_EL1 }, { SYS_DESC(SYS_AFSR1_EL1), access_vm_reg, reset_unknown, AFSR1_EL1 }, { SYS_DESC(SYS_ESR_EL1), access_vm_reg, reset_unknown, ESR_EL1 }, @@ -2080,7 +2107,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_LORC_EL1), trap_loregion }, { SYS_DESC(SYS_LORID_EL1), trap_loregion }, - { SYS_DESC(SYS_VBAR_EL1), NULL, reset_val, VBAR_EL1, 0 }, + { SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 }, { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 }, { SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only }, From patchwork Tue Jan 31 09:24:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DF8EEC38142 for ; Tue, 31 Jan 2023 09:51:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=MwdHwh4ZLX7iQQKXGXkEsYNTFQFmw4sqw+CoWzSHKvU=; b=JKDvgtaj2KbFaK 2uYxVkkfXB2fSOMgr0ViCgURbaH02yM1S03xOmFX7txz3ss+Ou3KGIZ0XTQFsFA/uT9LAxDyq2PIM 2I/PcWC97v1OAJ9w+kXaWzcETqzaEOQSQrCjMKTiBbw6ua7ICbkUcvyjte9BLWq7Mrg4So5wDKGDQ kU5S91t+iTxXMxSwQm/DGdLLcf629X0QEthZ/ebcgsuh2avXgPURmsjE+9k3IBXwiSLA6xe1YQ1ST fnAEAE84/Y+B5k4Yj35ZogUi4eljL2KiPnGDLk+hOpFwy7tUEazMOGQ2ZREVBgotd6tJN+t1fyTf/ jX/2DgnM7W9q+Ht3AWQA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnHQ-0078r4-KL; Tue, 31 Jan 2023 09:50:20 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8b-00756e-Ty for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:16 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 50683B81AC8; Tue, 31 Jan 2023 09:41:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ED1A4C433D2; Tue, 31 Jan 2023 09:41:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158071; bh=+GltZpNflPk7u9VdDh7mZpVKmw9p7r9TMmLIybZwI6k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rQRVCFoPS0E+D8hra1XrkuIZ6flmXIycElJa/94fr33d18W0aRjsc1ti+b/5VA5Zp NZlwOnRss11X0bvqC60AZ99YRdKtDlxHOJUkdfxF+dz1RgjoL438r4bjfjPtUrAfeL aVm0ra+U/1TnnOp7TBhyj9MgFxaXdOxfdrCxUAX+jPWhYZLUydasMQk6Jqn9UmXQ+h lehOD/yZeXDTJlpxGhDetODQlkwFzE8NE8t9AjobK+SmSZfEQMqVNKPXUNvpfpVMn7 LPIR3OMYpAvQ/rZifZe77G5cvuLEwi7j75G5PbLcfdTBl6+1OAUuj24+g8bK7jK21w +9X0f0FYl/Yqw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtT-0067U2-D2; Tue, 31 Jan 2023 09:25:35 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 21/69] KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2 Date: Tue, 31 Jan 2023 09:24:16 +0000 Message-Id: <20230131092504.2880505-22-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014114_336793_66707D95 X-CRM114-Status: GOOD ( 15.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim For the same reason we trap virtual memory register accesses in virtual EL2, we trap CPACR_EL1 access too; We allow the virtual EL2 mode to access EL1 system register state instead of the virtual EL2 one. Reviewed-by: Alexandru Elisei Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/vhe/switch.c | 3 +++ arch/arm64/kvm/sys_regs.c | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index c8da8d350453..41385bb02d6b 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -68,6 +68,9 @@ static void __activate_traps(struct kvm_vcpu *vcpu) __activate_traps_fpsimd32(vcpu); } + if (vcpu_is_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu)) + val |= CPTR_EL2_TCPAC; + write_sysreg(val, cpacr_el1); write_sysreg(__this_cpu_read(kvm_hyp_vector), vbar_el1); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index b559a289d996..1cc68ca9eb20 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2038,7 +2038,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_SCTLR_EL1), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 }, { SYS_DESC(SYS_ACTLR_EL1), access_actlr, reset_actlr, ACTLR_EL1 }, - { SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 }, + { SYS_DESC(SYS_CPACR_EL1), access_rw, reset_val, CPACR_EL1, 0 }, MTE_REG(RGSR_EL1), MTE_REG(GCR_EL1), From patchwork Tue Jan 31 09:24:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF226C636CC for ; Tue, 31 Jan 2023 09:56:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OuNnpIk36qT9N4mncvAXB2i1qCTyCxoZD7iLsFMCVEg=; b=wqRsKUqKFpOwHn 2caeQsB/LGWEnsjNfxPwvC9F4AJ1EmWRekkb7iFCdXbtFi8Q9y3kDyq1Dqh2tcDUZzqHWHyAGTTBi Y9Z9mCnOjjrlBicINFGmo1D8K0zqeUVXriG8SaWpIuc7oqQNmVcS0UzZUuk+QBUBGmuEFmmSXwNlq 3r3enW4b9wwzCSvnfHjL2qEnPrjchlkmyYQLuKHxx6zP4641YCQi4gGuLrdPv57JNkizw2XPPIt16 MGCkQKc4KAIy4YHT7oJ57r24UB86GUB7kzwX7KGFNeBQs8keSKDoX8TaL0DPjEKEc6lwiF/Pr6WZj 5pVwDhaSCCEWjyRH2iEA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnMW-007BJy-Pq; Tue, 31 Jan 2023 09:55:37 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn97-0075LF-BP for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:47 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id F1E22B81AC8; Tue, 31 Jan 2023 09:41:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A65A1C433D2; Tue, 31 Jan 2023 09:41:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158102; bh=dhvTz42JuyazXpHz1ww4ODtTIeXpTYgqJ/HxgzVe05I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=n3bxXxfv1nzA7Q3ssHo+NQw0xe1S0gZuLWQAF9Y1zz7Yl1CCVPhrpV8CJy2C0Fl6M Kcz3CS/KQ8i8w3UlZn3FSeXML/7tVyU7Brq+f+FLNk8hE1Y/uxTIMQMgtNh/Pqktij 8fi76IpMBi3ftiJTD2euN11U496HzLmw+Y+au9KHlPFpcmPBxEbgWE8S8PSAGfv0wo PQtx1wdjJ7bwOAo1nHUX8FNVfQm6C2yEST6qJhlYABiTS2kzY9O4thNzwVm/i0tdjF emeUGGFyhV3SKQkLGjsBXl7rYzihXol6st7JCH4Lh0sWx+sbG8rjXddc6DgcmPxwIW 9GCWxZREMwnfQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtT-0067U2-LQ; Tue, 31 Jan 2023 09:25:35 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 22/69] KVM: arm64: nv: Handle PSCI call via smc from the guest Date: Tue, 31 Jan 2023 09:24:17 +0000 Message-Id: <20230131092504.2880505-23-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014145_727266_00EC942A X-CRM114-Status: GOOD ( 19.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim VMs used to execute hvc #0 for the psci call if EL3 is not implemented. However, when we come to provide the virtual EL2 mode to the VM, the host OS inside the VM calls kvm_call_hyp() which is also hvc #0. So, it's hard to differentiate between them from the host hypervisor's point of view. So, let the VM execute smc instruction for the psci call. On ARMv8.3, even if EL3 is not implemented, a smc instruction executed at non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than being treated as UNDEFINED. So, the host hypervisor can handle this psci call without any confusion. Reviewed-by: Alexandru Elisei Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/handle_exit.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index e75101f2aa6c..b0c343c4e062 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -63,6 +63,8 @@ static int handle_hvc(struct kvm_vcpu *vcpu) static int handle_smc(struct kvm_vcpu *vcpu) { + int ret; + /* * "If an SMC instruction executed at Non-secure EL1 is * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a @@ -70,10 +72,28 @@ static int handle_smc(struct kvm_vcpu *vcpu) * * We need to advance the PC after the trap, as it would * otherwise return to the same address... + * + * If imm is non-zero, it's not defined, so just skip it. + */ + if (kvm_vcpu_hvc_get_imm(vcpu)) { + vcpu_set_reg(vcpu, 0, ~0UL); + kvm_incr_pc(vcpu); + return 1; + } + + /* + * If imm is zero, it's a psci call. + * Note that on ARMv8.3, even if EL3 is not implemented, SMC executed + * at Non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than + * being treated as UNDEFINED. */ - vcpu_set_reg(vcpu, 0, ~0UL); + ret = kvm_hvc_call_handler(vcpu); + if (ret < 0) + vcpu_set_reg(vcpu, 0, ~0UL); + kvm_incr_pc(vcpu); - return 1; + + return ret; } /* From patchwork Tue Jan 31 09:24:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C2943C38142 for ; Tue, 31 Jan 2023 09:56:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mV9Vf/seRxVWpBbL51okLvDkkPNCsXfdvOx/ObbeGK4=; b=YfBeUt+jFxgQqi LjVI6UapBm3Ek3NDZ7Ut6bPu5vElSH5f7r6R85CVxVqdd8bsu2nppsg8xSPtTZ5R7q4tGFmVht06b PME9WAVj/4gzW9ZNsgWblWJ29afpUIPhKnfv7dNlTd02w/EQ3qbKFEQecJQh2cHruXWv3X1ZvRbQ8 uMtRC6I6MO7mIv/NB4Ak9x6SljE60aAl3xKSk9Qe36XjE3h1UyI5MHMDMMHWvflfd9NuTJpoIDITW Zit+Y52AK6/f43Sn+UMq+/VhnkGC7MJv0rD8RlXLZlcDWrovRxQwcF2gy+8zJTWf0OgDb5I5+a2YL IgKEqsS+m2lSCi4txHMg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnM0-007B4P-AB; Tue, 31 Jan 2023 09:55:04 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn93-0075KM-8d for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:43 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A6CDD61481; Tue, 31 Jan 2023 09:41:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 17DABC433EF; Tue, 31 Jan 2023 09:41:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158100; bh=/QY1FPyGRGaFdavJIbYCnyzWzBu7xwfwmRblKc+JKKs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cNOvakJ6KCxiNC2UdKi5Sf0dZeX0nDqa5Zkp8T82mZ7zRqnkXXnXjX6kQ/PoM2MRE LedmDUu2cbkFXnQiGD0QiYuALCXDRHccVe3HI8qa2nQe8zbmQdEwn1+iotwBPGXbiF Y1ODgnW/jwjAttTAOXinLsT8r/qP0cGrMGGfNcecaa2kd3g5MLN6jrYhJwwZlMiOQm 1ogTEhwGDUTKBgUcID4yHc28sv7YHL1JRbD60+861rdBLP2JwAwnYVCgJXBw5OUHcq esS3MMlS4cVaU0eaxcI6xIYSnexvktKrTTKHjXLYinKZHNI1TL/VxcEK5Lxe9Fr4ob RAh5hqY5kt1FA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtT-0067U2-Te; Tue, 31 Jan 2023 09:25:35 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 23/69] KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting Date: Tue, 31 Jan 2023 09:24:18 +0000 Message-Id: <20230131092504.2880505-24-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014141_456970_090B04DB X-CRM114-Status: GOOD ( 15.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim Forward exceptions due to WFI or WFE instructions to the virtual EL2 if they are not coming from the virtual EL2 and virtual HCR_EL2.TWX is set. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 14 ++++++++++++++ arch/arm64/kvm/handle_exit.c | 6 +++++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 2f5d3e27c017..6e9c0d76a818 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -333,6 +334,19 @@ static __always_inline u64 kvm_vcpu_get_esr(const struct kvm_vcpu *vcpu) return vcpu->arch.fault.esr_el2; } +static inline bool guest_hyp_wfx_traps_enabled(const struct kvm_vcpu *vcpu) +{ + u64 esr = kvm_vcpu_get_esr(vcpu); + bool is_wfe = !!(esr & ESR_ELx_WFx_ISS_WFE); + u64 hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + + if (!vcpu_has_nv(vcpu) || vcpu_is_el2(vcpu)) + return false; + + return ((is_wfe && (hcr_el2 & HCR_TWE)) || + (!is_wfe && (hcr_el2 & HCR_TWI))); +} + static __always_inline int kvm_vcpu_get_condition(const struct kvm_vcpu *vcpu) { u64 esr = kvm_vcpu_get_esr(vcpu); diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index b0c343c4e062..ea73b47b99ac 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -124,8 +124,12 @@ static int handle_no_fpsimd(struct kvm_vcpu *vcpu) static int kvm_handle_wfx(struct kvm_vcpu *vcpu) { u64 esr = kvm_vcpu_get_esr(vcpu); + bool is_wfe = !!(esr & ESR_ELx_WFx_ISS_WFE); - if (esr & ESR_ELx_WFx_ISS_WFE) { + if (guest_hyp_wfx_traps_enabled(vcpu)) + return kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + + if (is_wfe) { trace_kvm_wfx_arm64(*vcpu_pc(vcpu), true); vcpu->stat.wfe_exit_stat++; } else { From patchwork Tue Jan 31 09:24:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122774 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0BBC8C38142 for ; Tue, 31 Jan 2023 10:04:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sDVKPcduZt4+LqVtoM1sHyz9hnJfgE7Le8OBEK6Om0E=; b=aB+Umb9LmS1dT4 LkxvVqng5l1GFKGxSqKEejxhmEHLYhkm9QPPhvyJMiwlfk4lqGJ1qe1qIdEH6KY+7OhwiFUWaBQ79 K5hnR/rmuVXkj566Fvm5QEJH7b/pgfwriNCnVkxmxw4/0YCvYNY7hcLxnn38OQA8JfDV9JCx3OQ++ 3klQew/UBl68utV/4m69e6vBqRZ2HwkHDMzsIF09W4CD7rSWdrJcKiUuw/XEXwEENJiI+St1dAXLH OJK3an9EkhFDbRFdozYTd3I+TWNif7X0Z7CCsR/RTPl3ObnZC6OI/Tyw5gAxf/X7IgjDYFJrO9y/J NuTSN31tCJg5NOiUIFJw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnU0-007EcV-Mx; Tue, 31 Jan 2023 10:03:21 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9S-0075VQ-Vk for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:09 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7C310B81AB2; Tue, 31 Jan 2023 09:42:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2F2C1C433EF; Tue, 31 Jan 2023 09:42:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158124; bh=g54Lb44ynwGN1CfXwVNqV8s+ncgc24jCGSUiaf50FVc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=e5ObThjNsSk4YzsHDNwSBqVG8eElmddG7rXBr9fsCEzeo88uvJCH1qTPNHyms7Sx0 sSMuO7Vasc+O2dZUUa6o93NYM3LKV1kvLCe/8hfCCdGybgqEUJn4ca5jEdym6ZKMeS YOZkfLx/4ythPm7SBz+3WRTeDbycoykD9gTK+41vuHktI48R1DnLsSe5D3iGNi7hlt 3dbV6nunELHsMP6lUqhxuaz+YPeXEcxKqh7tdPeZfsIlb926/ZqzjkPrMTIQtlP8+2 9ObkGrSJvftp74OPW8adcspD0Gt6eIECMxE1OvTqyyzaQQ3fJjU/RodO03RWkAap+c a0FhJ09V+I0lQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtU-0067U2-6I; Tue, 31 Jan 2023 09:25:36 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 24/69] KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings Date: Tue, 31 Jan 2023 09:24:19 +0000 Message-Id: <20230131092504.2880505-25-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014207_404222_E40EA9AF X-CRM114-Status: GOOD ( 21.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim Forward traps due to FP/ASIMD register accesses to the virtual EL2 if virtual CPTR_EL2.TFP is set (with HCR_EL2.E2H == 0) or CPTR_EL2.FPEN is configure to do so (with HCR_EL2.E2h == 1). Signed-off-by: Jintack Lim Signed-off-by: Christoffer Dall [maz: account for HCR_EL2.E2H when testing for TFP/FPEN, with all the hard work actually being done by Chase Conklin] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 25 +++++++++++++++++++++++++ arch/arm64/kvm/handle_exit.c | 16 ++++++++++++---- arch/arm64/kvm/hyp/include/hyp/switch.h | 8 ++++++-- 3 files changed, 43 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 6e9c0d76a818..fe4b4b893fb8 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -11,6 +11,7 @@ #ifndef __ARM64_KVM_EMULATE_H__ #define __ARM64_KVM_EMULATE_H__ +#include #include #include @@ -329,6 +330,30 @@ static inline bool vcpu_mode_priv(const struct kvm_vcpu *vcpu) return mode != PSR_MODE_EL0t; } +static inline bool guest_hyp_fpsimd_traps_enabled(const struct kvm_vcpu *vcpu) +{ + u64 val; + + if (!vcpu_has_nv(vcpu)) + return false; + + val = vcpu_read_sys_reg(vcpu, CPTR_EL2); + + if (!vcpu_el2_e2h_is_set(vcpu)) + return (val & CPTR_EL2_TFP); + + switch (FIELD_GET(CPACR_ELx_FPEN, val)) { + case 0b00: + case 0b10: + return true; + case 0b01: + return vcpu_el2_tge_is_set(vcpu) && !vcpu_is_el2(vcpu); + case 0b11: + default: /* GCC is dumb */ + return false; + } +} + static __always_inline u64 kvm_vcpu_get_esr(const struct kvm_vcpu *vcpu) { return vcpu->arch.fault.esr_el2; diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index ea73b47b99ac..2aefe35409c9 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -97,11 +97,19 @@ static int handle_smc(struct kvm_vcpu *vcpu) } /* - * Guest access to FP/ASIMD registers are routed to this handler only - * when the system doesn't support FP/ASIMD. + * This handles the cases where the system does not support FP/ASIMD or when + * we are running nested virtualization and the guest hypervisor is trapping + * FP/ASIMD accesses by its guest guest. + * + * All other handling of guest vs. host FP/ASIMD register state is handled in + * fixup_guest_exit(). */ -static int handle_no_fpsimd(struct kvm_vcpu *vcpu) +static int kvm_handle_fpasimd(struct kvm_vcpu *vcpu) { + if (guest_hyp_fpsimd_traps_enabled(vcpu)) + return kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + + /* This is the case when the system doesn't support FP/ASIMD. */ kvm_inject_undefined(vcpu); return 1; } @@ -263,7 +271,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_BREAKPT_LOW]= kvm_handle_guest_debug, [ESR_ELx_EC_BKPT32] = kvm_handle_guest_debug, [ESR_ELx_EC_BRK64] = kvm_handle_guest_debug, - [ESR_ELx_EC_FP_ASIMD] = handle_no_fpsimd, + [ESR_ELx_EC_FP_ASIMD] = kvm_handle_fpasimd, [ESR_ELx_EC_PAC] = kvm_handle_ptrauth, }; diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index e0bcaf000251..f458c2439271 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -174,8 +174,12 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) sve_guest = vcpu_has_sve(vcpu); esr_ec = kvm_vcpu_trap_get_class(vcpu); - /* Don't handle SVE traps for non-SVE vcpus here: */ - if (!sve_guest && esr_ec != ESR_ELx_EC_FP_ASIMD) + /* + * Don't handle SVE traps for non-SVE vcpus here. This + * includes NV guests for the time being. + */ + if (!sve_guest && (esr_ec != ESR_ELx_EC_FP_ASIMD || + guest_hyp_fpsimd_traps_enabled(vcpu))) return false; /* Valid trap. Switch the context: */ From patchwork Tue Jan 31 09:24:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4BF82C636CC for ; Tue, 31 Jan 2023 09:53:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=oAzf911samv79J43wdN6jCaZAYzfglgd4WS4TUupdb4=; b=OlO0dne6TkZTDg kNuz2Z1TfSW977lG2jcM5oBvp65CXIx3yf9Tz3DcUwtmmjBSdguu2GyeUn1VusHSLsOVppFrxYmSa 2WZ/InST2kmMiaXs0lvvUAfxmRco1DKAKsuVid3zY4aYuKspYzE+9azEz1VbMxGU/DYzagHuhe0+E 8g10ezn9riGCSWITs1ZE3tXqTLtVSwp59kGdRHiYB4SWCvS20VWB9vGLb70Y5RFsb/pE6qvTZ4jCf 8rTE72KfEIvHj7uGW2jHCFD0oTegrAVJdaLtFBKU7vHeh6Z2PNHxQ8nCk7dzmqXPYjb5sBRJInQcN x85rIol5bIoLnikqM14Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnJP-0079gd-PF; Tue, 31 Jan 2023 09:52:23 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8q-0075DA-0Y for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:29 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7F7976147A; Tue, 31 Jan 2023 09:41:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E5387C433EF; Tue, 31 Jan 2023 09:41:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158086; bh=ESIDo1L7SOPJ/b6U02mH7Qt5+IYpfRiML1qv/jzpvY0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aTqaVmHnovuLu8wF2D4pilipbNzLwFTQyQmhu0G5xU6MfMJrRaFFd5nb+Rhry0max skZpeKqbudYYYSpBnKBaEKET8ocWIZ9MuhN1l5N5DBfBdF7fINKa/zxMhJEOqB8WaT 0gXeZOGm9Jx2FE+i0dDKWB+xNlmG9pOY9EX0o57Y2WJCwPGFZllonsNkqaR+pwjZf1 9r/bi9wGWpxC/vmUKa8blvFiycMDLYeQr7O26kNtSy0GRCJ5lbiOpZQRHnGRZLjk3p qARdyXwQafB+qW7qctYuPXsqlsUcs1VHquyatSvEsfqio9wHR/YXD94fe5UPfHkKRC CuD110vOzTmtg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtU-0067U2-ET; Tue, 31 Jan 2023 09:25:36 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 25/69] KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting Date: Tue, 31 Jan 2023 09:24:20 +0000 Message-Id: <20230131092504.2880505-26-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014128_185626_AAFE8F5D X-CRM114-Status: GOOD ( 21.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim Forward traps due to HCR_EL2.NV bit to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.NV bit is set. In addition to EL2 register accesses, setting NV bit will also make EL12 register accesses trap to EL2. To emulate this for the virtual EL2, forword traps due to EL12 register accessses to the virtual EL2 if the virtual HCR_EL2.NV bit is set. Same thing is done for SMCs issued by a guest and trapped by the virtual EL2. This is for recursive nested virtualization. Signed-off-by: Jintack Lim [Moved code to emulate-nested.c] Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/kvm_nested.h | 3 +++ arch/arm64/kvm/emulate-nested.c | 27 +++++++++++++++++++++++++++ arch/arm64/kvm/handle_exit.c | 7 +++++++ arch/arm64/kvm/sys_regs.c | 21 +++++++++++++++++++++ 5 files changed, 59 insertions(+) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index baef29fcbeee..e47faadbfde1 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -20,6 +20,7 @@ #define HCR_AMVOFFEN (UL(1) << 51) #define HCR_FIEN (UL(1) << 47) #define HCR_FWB (UL(1) << 46) +#define HCR_NV (UL(1) << 42) #define HCR_API (UL(1) << 41) #define HCR_APK (UL(1) << 40) #define HCR_TEA (UL(1) << 37) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index f7bb1f0d0954..befd2501d5ba 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -59,4 +59,7 @@ static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0) return ttbr0 & ~GENMASK_ULL(63, 48); } +extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); +extern bool forward_nv_traps(struct kvm_vcpu *vcpu); + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index b96662029fb1..75cf6f15accc 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -14,6 +14,26 @@ #include "trace.h" +bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) +{ + bool control_bit_set; + + if (!vcpu_has_nv(vcpu)) + return false; + + control_bit_set = __vcpu_sys_reg(vcpu, HCR_EL2) & control_bit; + if (!vcpu_is_el2(vcpu) && control_bit_set) { + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + return true; + } + return false; +} + +bool forward_nv_traps(struct kvm_vcpu *vcpu) +{ + return forward_traps(vcpu, HCR_NV); +} + static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr) { u64 mode = spsr & PSR_MODE_MASK; @@ -52,6 +72,13 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) u64 spsr, elr, mode; bool direct_eret; + /* + * Forward this trap to the virtual EL2 if the virtual + * HCR_EL2.NV bit is set and this is coming from !EL2. + */ + if (forward_nv_traps(vcpu)) + return; + /* * Going through the whole put/load motions is a waste of time * if this is a VHE guest hypervisor returning to its own diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 2aefe35409c9..cfc50d67a6bc 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -65,6 +65,13 @@ static int handle_smc(struct kvm_vcpu *vcpu) { int ret; + /* + * Forward this trapped smc instruction to the virtual EL2 if + * the guest has asked for it. + */ + if (forward_traps(vcpu, HCR_TSC)) + return 1; + /* * "If an SMC instruction executed at Non-secure EL1 is * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 1cc68ca9eb20..58c0c0eed28b 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -336,10 +336,19 @@ static int set_ccsidr(struct kvm_vcpu *vcpu, u32 csselr, u32 val) return 0; } +static bool el12_reg(struct sys_reg_params *p) +{ + /* All *_EL12 registers have Op1=5. */ + return (p->Op1 == 5); +} + static bool access_rw(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, r->reg); else @@ -408,6 +417,9 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, bool was_enabled = vcpu_has_cache_enabled(vcpu); u64 val, mask, shift; + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + /* We don't expect TRVM on the host */ BUG_ON(!vcpu_is_el2(vcpu) && !p->is_write); @@ -1869,6 +1881,9 @@ static bool access_elr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1); else @@ -1881,6 +1896,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) __vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval; else @@ -1893,6 +1911,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2); else From patchwork Tue Jan 31 09:24:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122822 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C4F88C636CC for ; Tue, 31 Jan 2023 10:21:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hO7HGrm5ZuV2TwI+JJlo9Q50zI/dOVz2Edgp1EHg7Ok=; b=vDOx0o4IG7L+bu JeRXY1CKhXtXJTXd4Xl6cKjRhKfsCjibYhqJ1oj9ZRnnnhB6/uxQn0e9cpWY8XrNsBIkim07cAUBI HF+dc2E76fySNw/BzQ4VO2ygQVks9AvzaC34AzoH+0l9RXoMu9l2vfi12d7RpXNAubeIsbm4LpPBg fCJxDd+phG5z7AvOqSyicEbIORS2goNkdD3Tms8Hb9WHhCPH20qlht8dpC5DriKzQTlB1qutRLe5K mQiMwsxZLk8WSLHLbGnhT7kqNwwLTo91XSWXSQc26TmuQb8bZWYqclwyu37TnVgDD9nAoiDvQXcEX Z0SNqp5hr5ByDLR8+WvA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnkm-007M51-1s; Tue, 31 Jan 2023 10:20:40 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnA5-0075oU-BM for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:46 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DB463B81AB2; Tue, 31 Jan 2023 09:42:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8F488C433A1; Tue, 31 Jan 2023 09:42:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158162; bh=h1J6OM3LavEJHyI9EDArAeIgKAt25X6tuhGyuMdntGg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lMXT0eNAcMcouv79m9l0sidh0hY/YEhh6SR+OViOmkA+w2oDhDYdlCXg/cSb0xJiI iQPQvxspCZSxjoim3dyVisBznrcjXqn7fMsbvK++mQnrq56KurpeMY83fLT0w+Akqd DSpccikrNmm9u/3wvFqIjE2ZTCMJCZDq9yNAX0AHl+SYfn8AHLMF7j6Qx7HVz2ybcz 9VoZ/wz/wbhgIOQkak4XKdBaM4NYIrkf0XWUf9GTzNYWbPLFiV8arEtQ9vsmXsciEc 0kkXXMDVqcLVKYtVvgsqoFT7OVIuMV4LQ4cXSCW4QBWLsi6j0X66mghHUIjdV/IyYK Gd4GHIZkD5ASQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtU-0067U2-Nq; Tue, 31 Jan 2023 09:25:36 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 26/69] KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings Date: Tue, 31 Jan 2023 09:24:21 +0000 Message-Id: <20230131092504.2880505-27-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014245_576944_076579DA X-CRM114-Status: GOOD ( 12.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim Forward the EL1 virtual memory register traps to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.TVM or TRVM bit is set. This is for recursive nested virtualization. Reviewed-by: Alexandru Elisei Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 58c0c0eed28b..35ed217e39d8 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -420,6 +420,13 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p)) { + u64 bit = p->is_write ? HCR_TVM : HCR_TRVM; + + if (forward_traps(vcpu, bit)) + return false; + } + /* We don't expect TRVM on the host */ BUG_ON(!vcpu_is_el2(vcpu) && !p->is_write); From patchwork Tue Jan 31 09:24:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ABD1DC38142 for ; Tue, 31 Jan 2023 10:19:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=US4G6N1HfPQXMZ8ZHgMCnfK0aFYkqemLwg+P0BqCpso=; b=s+jvsVFAMmULg+ s2KGg/V5isQERIS7kJq+IUxIAwU2l5CyemmKtF68oH1nBAkGA4kcAF8Vsp4pprashUm/n7X8j3Vgw qVx6+8+PJGJS0XDvP8kqurLuJPfeD295KuziFqFVn6Ux462lBEHPG/BzzE0hslUhL+mnuNY+RWLhl xEx4qhSCeZpxU4oLjhrrn+4ScavumFY0cZGUA7u8XX9wvD7B1de4uoADzFrR/wWRBtdHrmiZBef0T Cq7jL469lRxPbQWn/pSnHp0ubFYwGPNS72syH56tmbcVbbDwtnf6Fp34nOeZVaLuZQg4aUXQF4711 HchWt5QoEaWehS3WZsbA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMniw-007LHx-NU; Tue, 31 Jan 2023 10:18:46 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9y-0075ku-01 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:39 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A6B9AB81AFD; Tue, 31 Jan 2023 09:42:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4A852C4339C; Tue, 31 Jan 2023 09:42:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158155; bh=Rkcxe5kU0tMZeWRSHUAEUvKkAP9dJ5NOUWFsfN+/6s4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zc2wpfB00uaPS36ztlgCBIuBAI6FGOdG1Gb1Ck0ZVVYdtlrNM/Jbo3l/LluRXtcO1 sjq7V6AjvI+CnYss2hqX4eH8dUrHGVAeBWOpjlFBWpI9tiZDgaEtACnjt5/+wP20Jd ml2OQ+8ESukG68IHYmoTOzenJauSs9z5Znh7mLdbByIEsRdlipx3LMLjAbjuxGaev/ tVbgtjd8G+WeXfsFoBtUUnS0tZfJHG4IYdTXDhrp6cctRKZySpTx2xtIspGxy6zwdU ClZz04UHM53jgjUQHDekgq2DCby9Qnv5ykkZpxhlXwPlNwsUMDXUos3ujEOmu9VC+d KGPzGSwcZepTw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtV-0067U2-00; Tue, 31 Jan 2023 09:25:37 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 27/69] KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting Date: Tue, 31 Jan 2023 09:24:22 +0000 Message-Id: <20230131092504.2880505-28-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014238_364586_8DD43375 X-CRM114-Status: GOOD ( 16.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim Forward ELR_EL1, SPSR_EL1 and VBAR_EL1 traps to the virtual EL2 if the virtual HCR_EL2.{NV,NV1}={1,1}. This is for recursive nested virtualization. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/kvm_nested.h | 1 + arch/arm64/kvm/emulate-nested.c | 5 +++++ arch/arm64/kvm/sys_regs.c | 21 ++++++++++++++++++++- 4 files changed, 27 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index e47faadbfde1..8e2b0bf1f484 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -20,6 +20,7 @@ #define HCR_AMVOFFEN (UL(1) << 51) #define HCR_FIEN (UL(1) << 47) #define HCR_FWB (UL(1) << 46) +#define HCR_NV1 (UL(1) << 43) #define HCR_NV (UL(1) << 42) #define HCR_API (UL(1) << 41) #define HCR_APK (UL(1) << 40) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index befd2501d5ba..7633a8c845c4 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -61,5 +61,6 @@ static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0) extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); +extern bool forward_nv1_traps(struct kvm_vcpu *vcpu); #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 75cf6f15accc..7c06844a5113 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -34,6 +34,11 @@ bool forward_nv_traps(struct kvm_vcpu *vcpu) return forward_traps(vcpu, HCR_NV); } +bool forward_nv1_traps(struct kvm_vcpu *vcpu) +{ + return forward_traps(vcpu, HCR_NV1); +} + static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr) { u64 mode = spsr & PSR_MODE_MASK; diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 35ed217e39d8..732d622f5fab 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -357,6 +357,16 @@ static bool access_rw(struct kvm_vcpu *vcpu, return true; } +static bool access_vbar_el1(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (forward_nv1_traps(vcpu)) + return false; + + return access_rw(vcpu, p, r); +} + /* * See note at ARMv7 ARM B1.14.4 (TL;DR: S/W ops are not easily virtualized). */ @@ -1891,6 +1901,9 @@ static bool access_elr(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p) && forward_nv1_traps(vcpu)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1); else @@ -1906,6 +1919,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p) && forward_nv1_traps(vcpu)) + return false; + if (p->is_write) __vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval; else @@ -1921,6 +1937,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p) && forward_nv1_traps(vcpu)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2); else @@ -2135,7 +2154,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_LORC_EL1), trap_loregion }, { SYS_DESC(SYS_LORID_EL1), trap_loregion }, - { SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 }, + { SYS_DESC(SYS_VBAR_EL1), access_vbar_el1, reset_val, VBAR_EL1, 0 }, { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 }, { SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only }, From patchwork Tue Jan 31 09:24:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 85EB5C636CC for ; Tue, 31 Jan 2023 09:50:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=In6tpuhTfZB9lB/UCokKG/m4b6v+XoFP75WBT0L/eq0=; b=p+IVpzUCICh340 LOXXBz3Ro8aHYh9J4sby/JcN8QU5NmxVvPs554D8+mjFu1Zer3vQ4s0xdvNksSiUDsOl5fMoIq79a z1+toFP8ctQZVwBzqlcZYxarSfU/qvTo0XopFfTuSHw0Gjfup8QI8oOZiVxD3f5H1spgOrGDSOdWf FBemaOEiJ7SwsN4iww6tIDxtRzOXMtQAKIOqUAljlaqh0IigS5Hwwr1p1Qi18TSSKKFrlzJOtWWMW sZTfa0Ah0s8J+9TS14KfCudW1WDfewO1EdzLPxitvPlBl8ww6XffARKpzRlpoqVKavQ46gOgWrPLR +ZogDdjlJw0P/YW4wfEw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnGe-0078bb-V5; Tue, 31 Jan 2023 09:49:33 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8X-00754N-J5 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:12 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1338561484; Tue, 31 Jan 2023 09:41:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D8D0CC433EF; Tue, 31 Jan 2023 09:41:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158068; bh=zIBjd6J+UmxYLNdtehRAaCh+NDaHVqX+rFHMmdOT+Uk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BL3U7P3ltb9X9443gNLgyIa9OQlwTXrL9C6tolP9i3ocp+H5HadE0meo9ZFnnuh1s qIR5oBZhducmAtgc2eatJ7tElul1FIGCcXpVLXJnYqQVM1xnZuB5FTWkcptpNNg5aU MKOmLeUo/J+lAZMW+WCIh709Gna2Tou/RiRyPjfEb9LsWj/SdiI2Wt1a1j71fHP4p+ Bi2w4bjTok+zk2KLNh6oiyKzmKVWhzctF0tJruX+oQe0glZL3NEaq5vKbxVqgL9vJi X3geC2u2KTbX+CJuFN6euamVnKlWFRjR3l8ev8gTwfqX+OgWWzRt2+VQzXfOXtsFbi VyBMnxFb2rfjA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtV-0067U2-9A; Tue, 31 Jan 2023 09:25:37 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 28/69] KVM: arm64: nv: Allow a sysreg to be hidden from userspace only Date: Tue, 31 Jan 2023 09:24:23 +0000 Message-Id: <20230131092504.2880505-29-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014109_799898_7FEB72A7 X-CRM114-Status: GOOD ( 18.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org So far, we never needed to distinguish between registers hidden from userspace and being hidden from a guest (they are always either visible to both, or hidden from both). With NV, we have the ugly case of the EL02 and EL12 registers, which are only a view on the EL0 and EL1 registers. It makes absolutely no sense to expose them to userspace, since it already has the canonical view. Add a new visibility flag (REG_HIDDEN_USER) and a new helper that checks for it and REG_HIDDEN when checking whether to expose a sysreg to userspace. Subsequent patches will make use of it. Reviewed-by: Alexandru Elisei Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 6 +++--- arch/arm64/kvm/sys_regs.h | 14 ++++++++++++-- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 732d622f5fab..eff4d41bf1bb 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -3341,7 +3341,7 @@ int kvm_sys_reg_get_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg, int ret; r = id_to_sys_reg_desc(vcpu, reg->id, table, num); - if (!r) + if (!r || sysreg_hidden_user(vcpu, r)) return -ENOENT; if (r->get_user) { @@ -3385,7 +3385,7 @@ int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg, return -EFAULT; r = id_to_sys_reg_desc(vcpu, reg->id, table, num); - if (!r) + if (!r || sysreg_hidden_user(vcpu, r)) return -ENOENT; if (sysreg_user_write_ignore(vcpu, r)) @@ -3471,7 +3471,7 @@ static int walk_one_sys_reg(const struct kvm_vcpu *vcpu, if (!(rd->reg || rd->get_user)) return 0; - if (sysreg_hidden(vcpu, rd)) + if (sysreg_hidden_user(vcpu, rd)) return 0; if (!copy_reg_to_user(rd, uind)) diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h index e4ebb3a379fd..6b11f2cc7146 100644 --- a/arch/arm64/kvm/sys_regs.h +++ b/arch/arm64/kvm/sys_regs.h @@ -85,8 +85,9 @@ struct sys_reg_desc { }; #define REG_HIDDEN (1 << 0) /* hidden from userspace and guest */ -#define REG_RAZ (1 << 1) /* RAZ from userspace and guest */ -#define REG_USER_WI (1 << 2) /* WI from userspace only */ +#define REG_HIDDEN_USER (1 << 1) /* hidden from userspace only */ +#define REG_RAZ (1 << 2) /* RAZ from userspace and guest */ +#define REG_USER_WI (1 << 3) /* WI from userspace only */ static __printf(2, 3) inline void print_sys_reg_msg(const struct sys_reg_params *p, @@ -152,6 +153,15 @@ static inline bool sysreg_hidden(const struct kvm_vcpu *vcpu, return sysreg_visibility(vcpu, r) & REG_HIDDEN; } +static inline bool sysreg_hidden_user(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *r) +{ + if (likely(!r->visibility)) + return false; + + return r->visibility(vcpu, r) & (REG_HIDDEN | REG_HIDDEN_USER); +} + static inline bool sysreg_visible_as_raz(const struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) { From patchwork Tue Jan 31 09:24:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7127CC38142 for ; Tue, 31 Jan 2023 09:46:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=FAVYyBlcdg0fMEgB6peK7OfJCw+pP/Lx7Ikmvta+UUM=; b=fAhBIzlCui23eU hPY7bSsS+2TFrgZVRIJ7Yn+esC9OeESI4iRPMOiSu6SnizqNpWZUzzZz9Um+kq7uXZ7W29OFxcdoE ikPsFHU/3lN+fogdTKCCn6gxDP9NS1ikTSgSXqkRTAYTUgDdpJ1pipfufmgRWtED1JehzFyQwx84q PPQBLetosa3N+s99eRR+u7uk7BBkqLGfqN/Nz3UC+L0iBL6+VE7sVL1X/rjvmXLzRhh/0lDSLrCo7 kJ9ovPp+VHvX57Cv2pmkIp6YzPcanNH7eUIvMK1zwl3TzqvSV9JzVosX3OorYFlucmvoGOHDxYCwV xeAsZdLpaJRpZrFQYTnQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnCi-00774z-Nc; Tue, 31 Jan 2023 09:45:29 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8M-0074wo-H2 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:00 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 28AC1B81AB2; Tue, 31 Jan 2023 09:40:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CA9E5C4339B; Tue, 31 Jan 2023 09:40:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158056; bh=+kPGdWhIzeMaeETryS9YnLRT6NMj2RoKNhMIXWZZC/s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eRI2oXUmTfvH0+P6+a0Hvh3eHSm7uH25WPGwGGFjeHTE0nGsRrIn6tLul8PX1Ea2C ppeZEvpvc2ewTjoQQy0gXEXLN7SRfYMrwUpHebboMtcI71y++6BU/pV/z4Fo/Ootez eMkMiN5/z8E7e5SNjIYrVS3doXbURMm7aW7SBqgBWoIjr2gW0CQH46yMuqZyUlBnXx fPDEVFZ/M+x7wYVDi7bxNjaxWnZ24Pnffk3jAJQKaLlPCXlgTXCPN5d/0Qajwvhe0P jlpGvWHvKAGyhOEp+VXWMIZDGjztX7Q2G8WFpTG5sE6/MtZZ84dFsmQ3NQ6IdMUdhQ 8Bhl+zzu2P30w== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtV-0067U2-H2; Tue, 31 Jan 2023 09:25:37 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 29/69] KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 Date: Tue, 31 Jan 2023 09:24:24 +0000 Message-Id: <20230131092504.2880505-30-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014058_941258_56821888 X-CRM114-Status: GOOD ( 14.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim With HCR_EL2.NV bit set, accesses to EL12 registers in the virtual EL2 trap to EL2. Handle those traps just like we do for EL1 registers. One exception is CNTKCTL_EL12. We don't trap on CNTKCTL_EL1 for non-VHE virtual EL2 because we don't have to. However, accessing CNTKCTL_EL12 will trap since it's one of the EL12 registers controlled by HCR_EL2.NV bit. Therefore, add a handler for it and don't treat it as a non-trap-registers when preparing a shadow context. These registers, being only a view on their EL1 counterpart, are permanently hidden from userspace. Reviewed-by: Alexandru Elisei Signed-off-by: Jintack Lim [maz: EL12_REG(), register visibility] Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index eff4d41bf1bb..2206fedccfc6 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1838,6 +1838,26 @@ static unsigned int el2_visibility(const struct kvm_vcpu *vcpu, .val = v, \ } +/* + * EL{0,1}2 registers are the EL2 view on an EL0 or EL1 register when + * HCR_EL2.E2H==1, and only in the sysreg table for convenience of + * handling traps. Given that, they are always hidden from userspace. + */ +static unsigned int elx2_visibility(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *rd) +{ + return REG_HIDDEN_USER; +} + +#define EL12_REG(name, acc, rst, v) { \ + SYS_DESC(SYS_##name##_EL12), \ + .access = acc, \ + .reset = rst, \ + .reg = name##_EL1, \ + .val = v, \ + .visibility = elx2_visibility, \ +} + /* sys_reg_desc initialiser for known cpufeature ID registers */ #define ID_SANITISED(name) { \ SYS_DESC(SYS_##name), \ @@ -2419,6 +2439,23 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL2_REG(CNTVOFF_EL2, access_rw, reset_val, 0), EL2_REG(CNTHCTL_EL2, access_rw, reset_val, 0), + EL12_REG(SCTLR, access_vm_reg, reset_val, 0x00C50078), + EL12_REG(CPACR, access_rw, reset_val, 0), + EL12_REG(TTBR0, access_vm_reg, reset_unknown, 0), + EL12_REG(TTBR1, access_vm_reg, reset_unknown, 0), + EL12_REG(TCR, access_vm_reg, reset_val, 0), + { SYS_DESC(SYS_SPSR_EL12), access_spsr}, + { SYS_DESC(SYS_ELR_EL12), access_elr}, + EL12_REG(AFSR0, access_vm_reg, reset_unknown, 0), + EL12_REG(AFSR1, access_vm_reg, reset_unknown, 0), + EL12_REG(ESR, access_vm_reg, reset_unknown, 0), + EL12_REG(FAR, access_vm_reg, reset_unknown, 0), + EL12_REG(MAIR, access_vm_reg, reset_unknown, 0), + EL12_REG(AMAIR, access_vm_reg, reset_amair_el1, 0), + EL12_REG(VBAR, access_rw, reset_val, 0), + EL12_REG(CONTEXTIDR, access_vm_reg, reset_val, 0), + EL12_REG(CNTKCTL, access_rw, reset_val, 0), + EL2_REG(SP_EL2, NULL, reset_unknown, 0), }; From patchwork Tue Jan 31 09:24:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 40071C636CC for ; Tue, 31 Jan 2023 09:52:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NP7IJbUl49mi7AzyJ9BjB+WS2wxuYYOVoaCXGVzFMjE=; b=J7c6IL8b5cjYVK lln40b7KQR/ZmXREbtCvqKfIZBrwkRIViNCznhnK1T9GWKzi18j6TlPdWk0JZlIXoKyqIfnqCxGwf /ei1wSZAf3mjjzeWouv+kSTut3uQWvAG4V6/GxvRi7A4IvvKfTy+6e3jl686lkxzAib8/pClWPpWp VZW+jb5tC4IDEpHq5xYQ3DZ4tXu16z2x6vCvBQ7mETT0W5RSGXjkeN6JhpjZwJLcbFyL0ydbhUq8o ONUZ1SJA6432Rnjh2G1suea8OKun3G0xr376jW+hg2yy3KqZQ/PwLcnpvbqfngqMy6rkbnEc5k6Mb lIFL7Y6GQpXz9sTY+LGw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnIo-0079O5-Du; Tue, 31 Jan 2023 09:51:46 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8j-0075AL-Uf for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:23 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 9A162B81AB2; Tue, 31 Jan 2023 09:41:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5712DC433EF; Tue, 31 Jan 2023 09:41:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158079; bh=UkFdG6zfNlpkdCXoHcPWCNATAtk2HHLyoOVHUn2Pg4s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f8BMziAhFCLDqNgFuRYNIEOBQweA+zqn893JxYb72bh9b0IEf8ejdoPvdz8C9DVcz nve1+rxmIdNpwwFpW/RQInwQKkUbDtroGV66JlP6GBXbwd84cpkaPcq0q9fGa2tY3V A2MFOEaeEkcvf7J/b1qscUVVtKhog8yv7nnAPk0x5IAc2mVk0Xqpcui+8i4Fxn2DjZ QYTE1YwGMxKXtFyTuZH1cpnfr2TvmEZEfMFPyBtksUUzjc0hm6BigLxWcNgPiHd/wx ViFLI9lNRTyfQS3whdMqvicpf2Yza+cNepzez/UO6R9YSKmdOCiR0LWNodDfGeP6gc Qxf27nFyKnpog== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtV-0067U2-Q0; Tue, 31 Jan 2023 09:25:37 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 30/69] KVM: arm64: nv: Forward debug traps to the nested guest Date: Tue, 31 Jan 2023 09:24:25 +0000 Message-Id: <20230131092504.2880505-31-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014122_337822_7D546D5D X-CRM114-Status: GOOD ( 14.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On handling a debug trap, check whether we need to forward it to the guest before handling it. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 2 ++ arch/arm64/kvm/emulate-nested.c | 9 +++++++-- arch/arm64/kvm/sys_regs.c | 5 +++++ 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 7633a8c845c4..2fe5847b401f 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -59,6 +59,8 @@ static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0) return ttbr0 & ~GENMASK_ULL(63, 48); } +extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, + u64 control_bit); extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); extern bool forward_nv1_traps(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 7c06844a5113..555771e1260d 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -14,14 +14,14 @@ #include "trace.h" -bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) +bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit) { bool control_bit_set; if (!vcpu_has_nv(vcpu)) return false; - control_bit_set = __vcpu_sys_reg(vcpu, HCR_EL2) & control_bit; + control_bit_set = __vcpu_sys_reg(vcpu, reg) & control_bit; if (!vcpu_is_el2(vcpu) && control_bit_set) { kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); return true; @@ -29,6 +29,11 @@ bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) return false; } +bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) +{ + return __forward_traps(vcpu, HCR_EL2, control_bit); +} + bool forward_nv_traps(struct kvm_vcpu *vcpu) { return forward_traps(vcpu, HCR_NV); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 2206fedccfc6..ab4383a5cfb1 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -666,6 +666,11 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (forward_traps(vcpu, HCR_TGE) || + __forward_traps(vcpu, MDCR_EL2, MDCR_EL2_TDE) || + __forward_traps(vcpu, MDCR_EL2, MDCR_EL2_TDA)) + return false; + access_rw(vcpu, p, r); if (p->is_write) vcpu_set_flag(vcpu, DEBUG_DIRTY); From patchwork Tue Jan 31 09:24:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 432B1C636CC for ; Tue, 31 Jan 2023 09:59:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SO+GxO2lKwEVZmquGNE9XhiL4OUm1DW+n/dI3XsoTEQ=; b=SrK92bGdHRX/M0 7PcgPAIN0XoQQKKXHvH+pY6z38hcXOiLjKUDhoZ0otgt9g/Jvi9ms+RQafyRjBAPz4qFjssCUs+4M p32WznW9z+0UZHcZ/CSuc2Q/644pvXjDkDC5GxJOAbw7Lwv0UNmT5At4W2tWCOWGYyRYcj4UpRwda ETfPWBv8LjSBhZcCBc51JuAlTE/J8CBtdBe/fqYWsxLajC1nG6sSymmwH8vv4WjbS3Se+CNF9pZFe JTWp+dgOqrnQjDRC7sQlHhEHj775JZ8Y7JGmya3QQkpOw3FQ++9J8oo8jeDPMH0k5Jm3tN5DPv5Bo 9QPonid7hi9T1GqBbkYw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnOo-007COV-HE; Tue, 31 Jan 2023 09:57:58 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9G-0075Pd-By for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:56 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C7D4EB81AC8; Tue, 31 Jan 2023 09:41:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 81361C433EF; Tue, 31 Jan 2023 09:41:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158111; bh=YcShdA+tKDmSUCRb0tpq6I2td/HqIONC//0886kHbTk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=morjQ+tG9sde25Llo9Lz9zE6bCYM0OyRnuS5c2K7rtRsVRY3B5iYhn8fDesHV2ppG Uwgfg+PYxu1ulx7wniIq/F7opK2bYxYExcOssjrc/SXvEB32uAqjWwnziBpTeJ4EeE jAxxm+BTVj7DnD6UU+5E4DPjZriMUArSlqtqSZPxTiXfCn8QZ6hyx+rYDronSiieBE bfvL0N7kqDkTGwmLJX0OTxaMsuldda2vPsJFuhOVMZ2ZQSoKDUmmoEYHT5hceFEs+t CIS9/kF+GnGFKQ13nu2JzFIsUCmnzRjzbN8CDtXlG/XcrOAnQh/kDKhoh8g4TeEdYT w4jhfwFF+dg6Q== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtW-0067U2-1g; Tue, 31 Jan 2023 09:25:38 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 31/69] KVM: arm64: nv: Configure HCR_EL2 for nested virtualization Date: Tue, 31 Jan 2023 09:24:26 +0000 Message-Id: <20230131092504.2880505-32-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014154_748938_F04DBDE1 X-CRM114-Status: GOOD ( 18.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim We enable nested virtualization by setting the HCR NV and NV1 bit. When the virtual E2H bit is set, we can support EL2 register accesses via EL1 registers from the virtual EL2 by doing trap-and-emulate. A better alternative, however, is to allow the virtual EL2 to access EL2 register states without trap. This can be easily achieved by not traping EL1 registers since those registers already have EL2 register states. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/kvm/hyp/vhe/switch.c | 38 +++++++++++++++++++++++++++++--- 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 8e2b0bf1f484..1ea71d26823c 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -89,6 +89,7 @@ HCR_BSU_IS | HCR_FB | HCR_TACR | \ HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \ HCR_FMO | HCR_IMO | HCR_PTW | HCR_TID3 | HCR_TID2) +#define HCR_GUEST_NV_FILTER_FLAGS (HCR_ATA | HCR_API | HCR_APK | HCR_FIEN) #define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF) #define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA) #define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 41385bb02d6b..36b50e315504 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -38,9 +38,41 @@ static void __activate_traps(struct kvm_vcpu *vcpu) u64 hcr = vcpu->arch.hcr_el2; u64 val; - /* Trap VM sysreg accesses if an EL2 guest is not using VHE. */ - if (vcpu_is_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu)) - hcr |= HCR_TVM | HCR_TRVM; + if (is_hyp_ctxt(vcpu)) { + hcr |= HCR_NV; + + if (!vcpu_el2_e2h_is_set(vcpu)) { + /* + * For a guest hypervisor on v8.0, trap and emulate + * the EL1 virtual memory control register accesses. + */ + hcr |= HCR_TVM | HCR_TRVM | HCR_NV1; + } else { + /* + * For a guest hypervisor on v8.1 (VHE), allow to + * access the EL1 virtual memory control registers + * natively. These accesses are to access EL2 register + * states. + * Note that we still need to respect the virtual + * HCR_EL2 state. + */ + u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + + vhcr_el2 &= ~HCR_GUEST_NV_FILTER_FLAGS; + + /* + * We already set TVM to handle set/way cache maint + * ops traps, this somewhat collides with the nested + * virt trapping for nVHE. So turn this off for now + * here, in the hope that VHE guests won't ever do this. + * TODO: find out whether it's worth to support both + * cases at the same time. + */ + hcr &= ~HCR_TVM; + + hcr |= vhcr_el2 & (HCR_TVM | HCR_TRVM); + } + } ___activate_traps(vcpu, hcr); From patchwork Tue Jan 31 09:24:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 583E2C636CC for ; Tue, 31 Jan 2023 10:27:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8Ni2sNLUFzTkJVBuU7kSrCb/eKKH8RC3xIjTHtLQwck=; b=xH6fN01p5cyPhm frQZGdoJhVW2l1P5gUs1ohkmnTZ2v1BjZioNtBXGl0a7jhpOPT21mCMXmOxqWo3/saCljsX2fvQa6 pgQJ/sFF4m3MZqropxieYjLD9jCrnsBd+5hPhV/o/uSmHQQIn1SH0inEy2oLXtxTS1mlw+LVfaEvp +tETIAlAsmfBph7oKBs3PeMGCzT73PkZIa0yOFlnHP94cKDBcu0RHn/yQOaxlCDnCl9Z4HmFald58 oG+9RfB5Xyvyq/4EAE6OED2GtTySb0zdH7UMT6AvLc4uhBUVgR4vvdtQsI0onnpwkIet5AbAb44K5 9eY3/+cWmVpPaFlV0Lpg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnqA-007OK3-3S; Tue, 31 Jan 2023 10:26:14 +0000 Received: from sin.source.kernel.org ([2604:1380:40e1:4800::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnAP-00760s-FQ for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:43:08 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 42C12CE1CCA; Tue, 31 Jan 2023 09:43:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 92B9BC433D2; Tue, 31 Jan 2023 09:42:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158178; bh=Efazmd/qzlkJ1RsoTc6vX3o6kREDZ7Tx29SGwB1GbTc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LVasHHns2Ruu9zSYNLZURb2jr5/vcLW/lXSdosPWB7jOZ+5vtarB+y/N/Sox1Hzvb 7aAJnBMCVvB83ZuI+wICgMfBpRpPBCkXh0XauMjZ81chG23V3nrM7lPkZHJzHAg0yg 3lPbA6Im9UfuLgvFEGdxbnAqq90kP9/FvYgsMXsZ+Lb/Sl32Q9FqYyfBGgVyhNgakS 5kgUstWMAFQJEpjIMHg8yI2MlajKGWbzlUPSnENsLiMng1b84ivlifdEdqS40Vhat2 5bdpb9b3gurX6ODf/Agq6/whY/ojifwlTjCehUd5Fhnh5e60vXOZGjl8lpyyEs1Hlu /cRSN0an9idrA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtW-0067U2-Au; Tue, 31 Jan 2023 09:25:38 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 32/69] KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes Date: Tue, 31 Jan 2023 09:24:27 +0000 Message-Id: <20230131092504.2880505-33-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014306_606501_662B9FFE X-CRM114-Status: GOOD ( 17.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall So far we were flushing almost the entire universe whenever a VM would load/unload the SCTLR_EL1 and the two versions of that register had different MMU enabled settings. This turned out to be so slow that it prevented forward progress for a nested VM, because a scheduler timer tick interrupt would always be pending when we reached the nested VM. To avoid this problem, we consider the SCTLR_EL2 when evaluating if caches are on or off when entering virtual EL2 (because this is the value that we end up shadowing onto the hardware EL1 register). Reviewed-by: Alexandru Elisei Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_mmu.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 7f7c1231679e..083cc47dca08 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -115,6 +115,7 @@ alternative_cb_end #include #include #include +#include #include void kvm_update_va_mask(struct alt_instr *alt, @@ -192,7 +193,15 @@ struct kvm; static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu) { - return (vcpu_read_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101; + u64 cache_bits = SCTLR_ELx_M | SCTLR_ELx_C; + int reg; + + if (vcpu_is_el2(vcpu)) + reg = SCTLR_EL2; + else + reg = SCTLR_EL1; + + return (vcpu_read_sys_reg(vcpu, reg) & cache_bits) == cache_bits; } static inline void __clean_dcache_guest_page(void *va, size_t size) From patchwork Tue Jan 31 09:24:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122790 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3F1E2C636CC for ; Tue, 31 Jan 2023 10:11:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/WDUL1PcMaYvOIs78ZVwT0DIO9EX3tEX3UMpPxdvOnw=; b=zgRBGBQsJg5OZa kFDvbfO3giMFJVim/Lvj77tYDYcbObyOvAGsvTUEZSC1KK5ntM6ifIZMB7ikcD6N0VXicUf+dkJQ9 hEtYpwTAFTGeW4+EyVIRw4PVZzyoABEvGhOPfoKDrJbCGUgIK/SxeRJHbeb+HQvOuJdTu1vHqJSpP YuztfDOb/lkVGjF9ELZzZ0Msn53oyoaOqD+dfo24A20AVEzGaeUjZoC7Fwplban9IuJy/33YZw5Em 0JMGe2lcz4UtMpD9sv0EZ7XRUUAFFVeM/IaNLRANSGCpD4MnZY7On8Exr0UW3R7DTjYwlzOKCDPSZ QNT+xxrZ3R7QoVVvbl0g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMna7-007HKx-O6; Tue, 31 Jan 2023 10:09:42 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9e-0075aj-2i for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:20 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8BEB961481; Tue, 31 Jan 2023 09:42:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D02E8C433EF; Tue, 31 Jan 2023 09:42:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158137; bh=MtqBLnspx9ej9GF9hhLCxMcAsDjdcqbIfWAbIRlaTQk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cMikvgOBcr+aa8bARqpEKwTYS33VqrgnmUauL2K2H/5TVzfJnGXp46kzBY8erSdWw YfvyACVAGL/KFFswU5LYOWxm7ub/Tyinm/mXPYHDhZXPxOdm68wvuQlgEenhISZfM1 oSM9Ge0mn2PaGS4niqGk+m/Ci7w5rcmbLG0dYNihiwTXYIJsDVvTVVnozNX9GtonDX 9Mnfxy0/3TzAjBmCxVxWeULKC6qYTHe2M+WPQkOhaMkWIYQUTXRR8fFFcVhM1vZWV9 kqLHfvr8g7yzMPZwuXDbl0vIP4BxOVQ916Af38sks6AtibWJsTpHaO6ARj5VQDndVF bnisNBbjfRYCw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtW-0067U2-Jg; Tue, 31 Jan 2023 09:25:38 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 33/69] KVM: arm64: nv: Filter out unsupported features from ID regs Date: Tue, 31 Jan 2023 09:24:28 +0000 Message-Id: <20230131092504.2880505-34-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014218_255588_A9CA4338 X-CRM114-Status: GOOD ( 23.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest. Whilst we are at it, advertize ARMv8.4-TTL as well as ARMv8.5-GTG. Reviewed-by: Ganapatrao Kulkarni Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 6 ++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/nested.c | 162 ++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 3 + 4 files changed, 172 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/nested.c diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 2fe5847b401f..f2820c82e956 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -65,4 +65,10 @@ extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); extern bool forward_nv1_traps(struct kvm_vcpu *vcpu); +struct sys_reg_params; +struct sys_reg_desc; + +void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, + const struct sys_reg_desc *r); + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 31b07f2b2186..c0c050e53157 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -14,7 +14,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ inject_fault.o va_layout.o handle_exit.o \ guest.o debug.o reset.o sys_regs.o stacktrace.o \ vgic-sys-reg-v3.o fpsimd.o pkvm.o \ - arch_timer.o trng.o vmid.o emulate-nested.o \ + arch_timer.o trng.o vmid.o emulate-nested.o nested.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c new file mode 100644 index 000000000000..f7ec27c27a4f --- /dev/null +++ b/arch/arm64/kvm/nested.c @@ -0,0 +1,162 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2017 - Columbia University and Linaro Ltd. + * Author: Jintack Lim + */ + +#include +#include + +#include +#include +#include + +#include "sys_regs.h" + +/* Protection against the sysreg repainting madness... */ +#define NV_FTR(r, f) ID_AA64##r##_EL1_##f + +/* + * Our emulated CPU doesn't support all the possible features. For the + * sake of simplicity (and probably mental sanity), wipe out a number + * of feature bits we don't intend to support for the time being. + * This list should get updated as new features get added to the NV + * support, and new extension to the architecture. + */ +void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + u32 id = sys_reg((u32)r->Op0, (u32)r->Op1, + (u32)r->CRn, (u32)r->CRm, (u32)r->Op2); + u64 val, tmp; + + val = p->regval; + + switch (id) { + case SYS_ID_AA64ISAR0_EL1: + /* Support everything but TME, O.S. and Range TLBIs */ + val &= ~(NV_FTR(ISAR0, TLB) | + NV_FTR(ISAR0, TME)); + break; + + case SYS_ID_AA64ISAR1_EL1: + /* Support everything but PtrAuth and Spec Invalidation */ + val &= ~(GENMASK_ULL(63, 56) | + NV_FTR(ISAR1, SPECRES) | + NV_FTR(ISAR1, GPI) | + NV_FTR(ISAR1, GPA) | + NV_FTR(ISAR1, API) | + NV_FTR(ISAR1, APA)); + break; + + case SYS_ID_AA64PFR0_EL1: + /* No AMU, MPAM, S-EL2, RAS or SVE */ + val &= ~(GENMASK_ULL(55, 52) | + NV_FTR(PFR0, AMU) | + NV_FTR(PFR0, MPAM) | + NV_FTR(PFR0, SEL2) | + NV_FTR(PFR0, RAS) | + NV_FTR(PFR0, SVE) | + NV_FTR(PFR0, EL3) | + NV_FTR(PFR0, EL2) | + NV_FTR(PFR0, EL1)); + /* 64bit EL1/EL2/EL3 only */ + val |= FIELD_PREP(NV_FTR(PFR0, EL1), 0b0001); + val |= FIELD_PREP(NV_FTR(PFR0, EL2), 0b0001); + val |= FIELD_PREP(NV_FTR(PFR0, EL3), 0b0001); + break; + + case SYS_ID_AA64PFR1_EL1: + /* Only support SSBS */ + val &= NV_FTR(PFR1, SSBS); + break; + + case SYS_ID_AA64MMFR0_EL1: + /* Hide ECV, FGT, ExS, Secure Memory */ + val &= ~(GENMASK_ULL(63, 43) | + NV_FTR(MMFR0, TGRAN4_2) | + NV_FTR(MMFR0, TGRAN16_2) | + NV_FTR(MMFR0, TGRAN64_2) | + NV_FTR(MMFR0, SNSMEM)); + + /* Disallow unsupported S2 page sizes */ + switch (PAGE_SIZE) { + case SZ_64K: + val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN16_2), 0b0001); + fallthrough; + case SZ_16K: + val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN4_2), 0b0001); + fallthrough; + case SZ_4K: + /* Support everything */ + break; + } + /* + * Since we can't support a guest S2 page size smaller than + * the host's own page size (due to KVM only populating its + * own S2 using the kernel's page size), advertise the + * limitation using FEAT_GTG. + */ + switch (PAGE_SIZE) { + case SZ_4K: + val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN4_2), 0b0010); + fallthrough; + case SZ_16K: + val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN16_2), 0b0010); + fallthrough; + case SZ_64K: + val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN64_2), 0b0010); + break; + } + /* Cap PARange to 48bits */ + tmp = FIELD_GET(NV_FTR(MMFR0, PARANGE), val); + if (tmp > 0b0101) { + val &= ~NV_FTR(MMFR0, PARANGE); + val |= FIELD_PREP(NV_FTR(MMFR0, PARANGE), 0b0101); + } + break; + + case SYS_ID_AA64MMFR1_EL1: + val &= (NV_FTR(MMFR1, PAN) | + NV_FTR(MMFR1, LO) | + NV_FTR(MMFR1, HPDS) | + NV_FTR(MMFR1, VH) | + NV_FTR(MMFR1, VMIDBits)); + break; + + case SYS_ID_AA64MMFR2_EL1: + val &= ~(NV_FTR(MMFR2, EVT) | + NV_FTR(MMFR2, BBM) | + NV_FTR(MMFR2, TTL) | + GENMASK_ULL(47, 44) | + NV_FTR(MMFR2, ST) | + NV_FTR(MMFR2, CCIDX) | + NV_FTR(MMFR2, VARange)); + + /* Force TTL support */ + val |= FIELD_PREP(NV_FTR(MMFR2, TTL), 0b0001); + break; + + case SYS_ID_AA64DFR0_EL1: + /* Only limited support for PMU, Debug, BPs and WPs */ + val &= (NV_FTR(DFR0, PMUVer) | + NV_FTR(DFR0, WRPs) | + NV_FTR(DFR0, BRPs) | + NV_FTR(DFR0, DebugVer)); + + /* Cap Debug to ARMv8.1 */ + tmp = FIELD_GET(NV_FTR(DFR0, DebugVer), val); + if (tmp > 0b0111) { + val &= ~NV_FTR(DFR0, DebugVer); + val |= FIELD_PREP(NV_FTR(DFR0, DebugVer), 0b0111); + } + break; + + default: + /* Unknown register, just wipe it clean */ + val = 0; + break; + } + + p->regval = val; +} diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index ab4383a5cfb1..b117c43ba15e 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1528,6 +1528,9 @@ static bool access_id_reg(struct kvm_vcpu *vcpu, return write_to_read_only(vcpu, p, r); p->regval = read_id_reg(vcpu, r); + if (vcpu_has_nv(vcpu)) + access_nested_id_reg(vcpu, p, r); + return true; } From patchwork Tue Jan 31 09:24:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 96A29C38142 for ; Tue, 31 Jan 2023 10:15:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0kBusvgcUZaEBLYoVdoXXM/d61I2kABmRuIxbMxm12E=; b=PyZAImjmqjb8o8 z69lWMBEgAFAg14L08C6ovKC9agVaRy5BU+nqmnM8UyMGkhp7QNFIK5MxE+EFUeFRqk97O/WrhoYS szuWicsizef1kvaz/pnnDS5d5TzLyaU/E4nr0bcQuAgUqZkJ5igeZfePgqAkOLc5Bo1hKHZag4XlE ni+5vps+rgEzfVtimDvPxLHuU3lK7Lb7R+2IyqGj3ADHPtzgwsO7L/2PVCZYwjPVMlKLFkfxK/zrS Fot+y4i3IK1yD85YvWoaBYdUJ4/wdUtFRTWQ1wlBHBaH126HujkrVLW3uV4YxI3eIeDFd9762ezG5 8O7XslRtKWKCEzN79uIQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMneM-007JE8-ME; Tue, 31 Jan 2023 10:14:03 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9o-0075ft-Au for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:29 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E504EB81AF7; Tue, 31 Jan 2023 09:42:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8FEEAC433EF; Tue, 31 Jan 2023 09:42:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158145; bh=xfH4X07JD136hpkodYH2sYgjnhcmppQC6xkM+e9e7P8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RQaaelSw7vp7vuKhDcG5pm9k7r+er+v7I7BUocxRbaTusUSSCfkggi6TCNKmaYgXZ nvXKdn8QUV+9cOpCXUe9yrJj5oWSvGNIpEmvKfa1ZIpna3KheVRF6bTL7U1ZGvaYmp /01jtlIB35KaQeKzUakUoRLl5OotK5mU3GjbWPmHTkKd4PZGmDSzoyL49EKIPfHoi5 UMaG52kRMCrCNOD+3uxN9NmekSrM/xt/df7ocWKdAIYzhBH4IikPnM+gYWJM40AN3X wBdbEeBOXSB3DWDVopJjsCpLW83nfjv8BiqsJZbHDRro3Qu2/zRL3WhWQNa9gujZzJ 3/YyTBI7MVAsg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtW-0067U2-Sw; Tue, 31 Jan 2023 09:25:38 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 34/69] KVM: arm64: nv: Hide RAS from nested guests Date: Tue, 31 Jan 2023 09:24:29 +0000 Message-Id: <20230131092504.2880505-35-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014228_567777_0DD073A9 X-CRM114-Status: GOOD ( 11.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We don't want to expose complicated features to guests until we have a good grasp on the basic CPU emulation. So let's pretend that RAS, doesn't exist in a nested guest. We already hide the feature bits, let's now make sure VDISR_EL1 will UNDEF. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index b117c43ba15e..93a0c10d8e0c 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2440,6 +2440,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL2_REG(VBAR_EL2, access_rw, reset_val, 0), EL2_REG(RVBAR_EL2, access_rw, reset_val, 0), { SYS_DESC(SYS_RMR_EL2), trap_undef }, + { SYS_DESC(SYS_VDISR_EL2), trap_undef }, EL2_REG(CONTEXTIDR_EL2, access_rw, reset_val, 0), EL2_REG(TPIDR_EL2, access_rw, reset_val, 0), From patchwork Tue Jan 31 09:24:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9CCC2C38142 for ; Tue, 31 Jan 2023 09:52:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0i8BNEWDht4o9l4VSi0R5oPkyrXSMKPqxkEAC4dEELQ=; b=zGYslmEgppICW8 yTINvSerlzlHPenXe5/h3NE+RlBXBsabFLwBVZt2SfQC+jgCAvyHCMT6FX3z4J7ikWKO+12t2wFLO e/OXqvrU3DKrDjJ9Mfk2GSa19kkH2OgwN6Rqeha/JQWXQgZ8IL4/Yj9E6mKixpKbAK2j+SJaA4Gh9 sKKPfrsFejsri/9XW9V2fMlzjjnauqq1z3lnpEG7W4LJYJ7W13JStRn50pA/1gc1Z9BgbDWAfRxHz BOdMFSdjN+jrtyv+JjMAMVVVaSCvFeEuHD4EW/dR18nnMH6MCce0j/dOL3C/NHJFCBWmGmDMojHtK bgo+fV77wLgz7lAtEulQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnIZ-0079Ie-AC; Tue, 31 Jan 2023 09:51:31 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8c-007571-5p for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:19 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8DE6661478; Tue, 31 Jan 2023 09:41:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1C74EC433D2; Tue, 31 Jan 2023 09:41:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158073; bh=1vhB8vbEWarbqcojPXxNg8st3WTjgl0QHzUUPttq1bE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eRF+GGXneZbca7CV2ulWQVmr9U+WkV20a5/kREqY/KH0fKgnjjzPRRTzVetBWcIgW UUVLC+tHFevs2tucWm1AJ1zKyh46jknEUqAb0yz5xtxcRVI+QitWdcV2Sk1FslCdoC JkObJYeau7UuVTdWi+vFX8T2R2x8dEoQPrQnzh2MpzJkTLJyCjocwW+hS8lr6xB1Ta 4aUgHVLehQP5dq5gQMAOcPqv66XpCwcci0I+cSNC22JRgijm9c85yfO6+waR2PvCTS x3T+7aXeKBcWC/nHRNG3GKTqmUtqeG6/krcAhHlSuffqkvRh5NMSO0zlIKHv6+CjYF rmsfS8B0/b9cA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtY-0067U2-7i; Tue, 31 Jan 2023 09:25:42 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 35/69] KVM: arm64: nv: Support multiple nested Stage-2 mmu structures Date: Tue, 31 Jan 2023 09:24:30 +0000 Message-Id: <20230131092504.2880505-36-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014114_385337_620AA284 X-CRM114-Status: GOOD ( 37.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add Stage-2 mmu data structures for virtual EL2 and for nested guests. We don't yet populate shadow Stage-2 page tables, but we now have a framework for getting to a shadow Stage-2 pgd. We allocate twice the number of vcpus as Stage-2 mmu structures because that's sufficient for each vcpu running two translation regimes without having to flush the Stage-2 page tables. Co-developed-by: Christoffer Dall Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 35 +++++ arch/arm64/include/asm/kvm_mmu.h | 9 ++ arch/arm64/include/asm/kvm_nested.h | 7 + arch/arm64/kvm/arm.c | 18 ++- arch/arm64/kvm/mmu.c | 77 +++++++---- arch/arm64/kvm/nested.c | 196 ++++++++++++++++++++++++++++ 6 files changed, 318 insertions(+), 24 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index a1892a8f6032..a348c93b1094 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -159,8 +159,35 @@ struct kvm_s2_mmu { int __percpu *last_vcpu_ran; struct kvm_arch *arch; + + /* + * For a shadow stage-2 MMU, the virtual vttbr used by the + * host to parse the guest S2. + * This either contains: + * - the virtual VTTBR programmed by the guest hypervisor with + * CnP cleared + * - The value 1 (VMID=0, BADDR=0, CnP=1) if invalid + */ + u64 vttbr; + + /* + * true when this represents a nested context where virtual + * HCR_EL2.VM == 1 + */ + bool nested_stage2_enabled; + + /* + * 0: Nobody is currently using this, check vttbr for validity + * >0: Somebody is actively using this. + */ + atomic_t refcnt; }; +static inline bool kvm_s2_mmu_valid(struct kvm_s2_mmu *mmu) +{ + return !(mmu->vttbr & 1); +} + struct kvm_arch_memory_slot { }; @@ -187,6 +214,14 @@ struct kvm_protected_vm { struct kvm_arch { struct kvm_s2_mmu mmu; + /* + * Stage 2 paging state for VMs with nested S2 using a virtual + * VMID. + */ + struct kvm_s2_mmu *nested_mmus; + size_t nested_mmus_size; + int nested_mmus_next; + /* VTCR_EL2 value for this VM */ u64 vtcr; diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 083cc47dca08..0cb5ad7b0852 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -117,6 +117,7 @@ alternative_cb_end #include #include #include +#include void kvm_update_va_mask(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst); @@ -166,6 +167,7 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void **haddr); void __init free_hyp_pgds(void); +void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size); void stage2_unmap_vm(struct kvm *kvm); int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long type); void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu); @@ -307,5 +309,12 @@ static inline struct kvm *kvm_s2_mmu_to_kvm(struct kvm_s2_mmu *mmu) { return container_of(mmu->arch, struct kvm, arch); } + +static inline u64 get_vmid(u64 vttbr) +{ + return (vttbr & VTTBR_VMID_MASK(kvm_get_vmid_bits())) >> + VTTBR_VMID_SHIFT; +} + #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index f2820c82e956..e3bcb351aae1 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -59,6 +59,13 @@ static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0) return ttbr0 & ~GENMASK_ULL(63, 48); } +extern void kvm_init_nested(struct kvm *kvm); +extern int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu); +extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu); +extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr); +extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); +extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); + extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 4e1943e3aa42..92d56959cb27 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -36,9 +36,10 @@ #include #include #include +#include #include +#include #include -#include #include #include @@ -128,6 +129,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) { int ret; + kvm_init_nested(kvm); + ret = kvm_share_hyp(kvm, kvm + 1); if (ret) return ret; @@ -387,6 +390,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) struct kvm_s2_mmu *mmu; int *last_ran; + if (vcpu_has_nv(vcpu)) + kvm_vcpu_load_hw_mmu(vcpu); + mmu = vcpu->arch.hw_mmu; last_ran = this_cpu_ptr(mmu->last_vcpu_ran); @@ -437,9 +443,12 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) kvm_timer_vcpu_put(vcpu); kvm_vgic_put(vcpu); kvm_vcpu_pmu_restore_host(vcpu); + if (vcpu_has_nv(vcpu)) + kvm_vcpu_put_hw_mmu(vcpu); kvm_arm_vmid_clear_active(); vcpu_clear_on_unsupported_cpu(vcpu); + vcpu->cpu = -1; } @@ -1169,8 +1178,13 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu, vcpu->arch.target = phys_target; + /* Prepare for nested if required */ + ret = kvm_vcpu_init_nested(vcpu); + /* Now we know what it is, we can reset it. */ - ret = kvm_reset_vcpu(vcpu); + if (!ret) + ret = kvm_reset_vcpu(vcpu); + if (ret) { vcpu->arch.target = -1; bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 217a511eb271..9637049f74ae 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -217,7 +217,7 @@ static void invalidate_icache_guest_page(void *va, size_t size) * does. */ /** - * unmap_stage2_range -- Clear stage2 page table entries to unmap a range + * __unmap_stage2_range -- Clear stage2 page table entries to unmap a range * @mmu: The KVM stage-2 MMU pointer * @start: The intermediate physical base address of the range to unmap * @size: The size of the area to unmap @@ -240,7 +240,7 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 may_block)); } -static void unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size) +void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size) { __unmap_stage2_range(mmu, start, size, true); } @@ -692,21 +692,9 @@ static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = { .icache_inval_pou = invalidate_icache_guest_page, }; -/** - * kvm_init_stage2_mmu - Initialise a S2 MMU structure - * @kvm: The pointer to the KVM structure - * @mmu: The pointer to the s2 MMU structure - * @type: The machine type of the virtual machine - * - * Allocates only the stage-2 HW PGD level table(s). - * Note we don't need locking here as this is only called when the VM is - * created, which can only be done once. - */ -int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long type) +static int kvm_init_ipa_range(struct kvm *kvm, unsigned long type) { u32 kvm_ipa_limit = get_kvm_ipa_limit(); - int cpu, err; - struct kvm_pgtable *pgt; u64 mmfr0, mmfr1; u32 phys_shift; @@ -733,7 +721,53 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1); kvm->arch.vtcr = kvm_get_vtcr(mmfr0, mmfr1, phys_shift); + return 0; +} + +/** + * kvm_init_stage2_mmu - Initialise a S2 MMU structure + * @kvm: The pointer to the KVM structure + * @mmu: The pointer to the s2 MMU structure + * @type: The machine type of the virtual machine + * + * Allocates only the stage-2 HW PGD level table(s). + * Note we don't need locking here as this is only called in two cases: + * + * - when the VM is created, which can't race against anything + * + * - when secondary kvm_s2_mmu structures are initialised for NV + * guests, and the caller must hold kvm->lock as this is called on a + * per-vcpu basis. + */ +int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long type) +{ + int cpu, err; + struct kvm_pgtable *pgt; + + /* + * We only initialise the IPA range on the canonical MMU, so + * the type is meaningless in all other situations. + */ + if (&kvm->arch.mmu == mmu) { + err = kvm_init_ipa_range(kvm, type); + if (err) + return err; + } + + /* + * If we already have our page tables in place, and that the + * MMU context is the canonical one, we have a bug somewhere, + * as this is only supposed to ever happen once per VM. + * + * Otherwise, we're building nested page tables, and that's + * probably because userspace called KVM_ARM_VCPU_INIT more + * than once on the same vcpu. Since that's actually legal, + * don't kick a fuss and leave gracefully. + */ if (mmu->pgt != NULL) { + if (&kvm->arch.mmu != mmu) + return 0; + kvm_err("kvm_arch already initialized?\n"); return -EINVAL; } @@ -758,6 +792,10 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t mmu->pgt = pgt; mmu->pgd_phys = __pa(pgt->pgd); + + if (&kvm->arch.mmu != mmu) + kvm_init_nested_s2_mmu(mmu); + return 0; out_destroy_pgtable: @@ -803,7 +841,7 @@ static void stage2_unmap_memslot(struct kvm *kvm, if (!(vma->vm_flags & VM_PFNMAP)) { gpa_t gpa = addr + (vm_start - memslot->userspace_addr); - unmap_stage2_range(&kvm->arch.mmu, gpa, vm_end - vm_start); + kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, vm_end - vm_start); } hva = vm_end; } while (hva < reg_end); @@ -1846,11 +1884,6 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) { } -void kvm_arch_flush_shadow_all(struct kvm *kvm) -{ - kvm_free_stage2_pgd(&kvm->arch.mmu); -} - void kvm_arch_flush_shadow_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) { @@ -1858,7 +1891,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, phys_addr_t size = slot->npages << PAGE_SHIFT; write_lock(&kvm->mmu_lock); - unmap_stage2_range(&kvm->arch.mmu, gpa, size); + kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, size); write_unlock(&kvm->mmu_lock); } diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index f7ec27c27a4f..5514116429af 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -7,7 +7,9 @@ #include #include +#include #include +#include #include #include @@ -16,6 +18,200 @@ /* Protection against the sysreg repainting madness... */ #define NV_FTR(r, f) ID_AA64##r##_EL1_##f +void kvm_init_nested(struct kvm *kvm) +{ + kvm->arch.nested_mmus = NULL; + kvm->arch.nested_mmus_size = 0; +} + +int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + struct kvm_s2_mmu *tmp; + int num_mmus; + int ret = -ENOMEM; + + if (!test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)) + return 0; + + if (!cpus_have_final_cap(ARM64_HAS_NESTED_VIRT)) + return -EINVAL; + + mutex_lock(&kvm->lock); + + /* + * Let's treat memory allocation failures as benign: If we fail to + * allocate anything, return an error and keep the allocated array + * alive. Userspace may try to recover by intializing the vcpu + * again, and there is no reason to affect the whole VM for this. + */ + num_mmus = atomic_read(&kvm->online_vcpus) * 2; + tmp = krealloc(kvm->arch.nested_mmus, + num_mmus * sizeof(*kvm->arch.nested_mmus), + GFP_KERNEL_ACCOUNT | __GFP_ZERO); + if (tmp) { + /* + * If we went through a realocation, adjust the MMU + * back-pointers in the previously initialised + * pg_table structures. + */ + if (kvm->arch.nested_mmus != tmp) { + int i; + + for (i = 0; i < num_mmus - 2; i++) + tmp[i].pgt->mmu = &tmp[i]; + } + + if (kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 1], 0) || + kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 2], 0)) { + kvm_free_stage2_pgd(&tmp[num_mmus - 1]); + kvm_free_stage2_pgd(&tmp[num_mmus - 2]); + } else { + kvm->arch.nested_mmus_size = num_mmus; + ret = 0; + } + + kvm->arch.nested_mmus = tmp; + } + + mutex_unlock(&kvm->lock); + return ret; +} + +/* Must be called with kvm->mmu_lock held */ +struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) +{ + bool nested_stage2_enabled = hcr & HCR_VM; + int i; + + /* Don't consider the CnP bit for the vttbr match */ + vttbr = vttbr & ~VTTBR_CNP_BIT; + + /* + * Two possibilities when looking up a S2 MMU context: + * + * - either S2 is enabled in the guest, and we need a context that + * is S2-enabled and matches the full VTTBR (VMID+BADDR), which + * makes it safe from a TLB conflict perspective (a broken guest + * won't be able to generate them), + * + * - or S2 is disabled, and we need a context that is S2-disabled + * and matches the VMID only, as all TLBs are tagged by VMID even + * if S2 translation is disabled. + */ + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (!kvm_s2_mmu_valid(mmu)) + continue; + + if (nested_stage2_enabled && + mmu->nested_stage2_enabled && + vttbr == mmu->vttbr) + return mmu; + + if (!nested_stage2_enabled && + !mmu->nested_stage2_enabled && + get_vmid(vttbr) == get_vmid(mmu->vttbr)) + return mmu; + } + return NULL; +} + +/* Must be called with kvm->mmu_lock held */ +static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + u64 hcr= vcpu_read_sys_reg(vcpu, HCR_EL2); + struct kvm_s2_mmu *s2_mmu; + int i; + + s2_mmu = lookup_s2_mmu(kvm, vttbr, hcr); + if (s2_mmu) + goto out; + + /* + * Make sure we don't always search from the same point, or we + * will always reuse a potentially active context, leaving + * free contexts unused. + */ + for (i = kvm->arch.nested_mmus_next; + i < (kvm->arch.nested_mmus_size + kvm->arch.nested_mmus_next); + i++) { + s2_mmu = &kvm->arch.nested_mmus[i % kvm->arch.nested_mmus_size]; + + if (atomic_read(&s2_mmu->refcnt) == 0) + break; + } + BUG_ON(atomic_read(&s2_mmu->refcnt)); /* We have struct MMUs to spare */ + + /* Set the scene for the next search */ + kvm->arch.nested_mmus_next = (i + 1) % kvm->arch.nested_mmus_size; + + if (kvm_s2_mmu_valid(s2_mmu)) { + /* Clear the old state */ + kvm_unmap_stage2_range(s2_mmu, 0, kvm_phys_size(kvm)); + if (atomic64_read(&s2_mmu->vmid.id)) + kvm_call_hyp(__kvm_tlb_flush_vmid, s2_mmu); + } + + /* + * The virtual VMID (modulo CnP) will be used as a key when matching + * an existing kvm_s2_mmu. + */ + s2_mmu->vttbr = vttbr & ~VTTBR_CNP_BIT; + s2_mmu->nested_stage2_enabled = hcr & HCR_VM; + +out: + atomic_inc(&s2_mmu->refcnt); + return s2_mmu; +} + +void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu) +{ + mmu->vttbr = 1; + mmu->nested_stage2_enabled = false; + atomic_set(&mmu->refcnt, 0); +} + +void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu) +{ + if (is_hyp_ctxt(vcpu)) { + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; + } else { + write_lock(&vcpu->kvm->mmu_lock); + vcpu->arch.hw_mmu = get_s2_mmu_nested(vcpu); + write_unlock(&vcpu->kvm->mmu_lock); + } +} + +void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu) +{ + if (vcpu->arch.hw_mmu != &vcpu->kvm->arch.mmu) { + atomic_dec(&vcpu->arch.hw_mmu->refcnt); + vcpu->arch.hw_mmu = NULL; + } +} + +void kvm_arch_flush_shadow_all(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + WARN_ON(atomic_read(&mmu->refcnt)); + + if (!atomic_read(&mmu->refcnt)) + kvm_free_stage2_pgd(mmu); + } + kfree(kvm->arch.nested_mmus); + kvm->arch.nested_mmus = NULL; + kvm->arch.nested_mmus_size = 0; + kvm_free_stage2_pgd(&kvm->arch.mmu); +} + /* * Our emulated CPU doesn't support all the possible features. For the * sake of simplicity (and probably mental sanity), wipe out a number From patchwork Tue Jan 31 09:24:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C02E8C38142 for ; Tue, 31 Jan 2023 09:58:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/Ju5Ucyqkgr7LYYLGWxvKp86yu8RFxcKFF8HkpQ6T4k=; b=w2mRvaOB1vMYNw W1U5T8edswlr9khSUMNh5yc/SnrThZr2fyZN7NXJOjdo0DpInqB9s64wjPmgZJy9hJccXH/tPRaX1 VJxViT3AhmoCYxgGxfd4PgR69PQd7AArxWZFEphv7XAo1gDtvQdGFzyQtpSRdPBbu/G/1xXftUoEc j8uATN3L218V9XIRiJx9t8pkIJpvlKs6jrxOn92fEwvKHoQib8O4XNVxo93HT9xPBM6+UpC9UuZWR 6X3AEq5LBVBWWbykGEmCjszjM5zJE8h546LvCGJGXOfmFG+J8aENF6eL8QBy+xyCTfduDcbdjAu+g vOX/8gMyWL5nWX4xRIcA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnNf-007Bt6-Ph; Tue, 31 Jan 2023 09:56:48 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9E-0075OY-4s for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:54 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A90D7B81ACC; Tue, 31 Jan 2023 09:41:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3AB3CC433EF; Tue, 31 Jan 2023 09:41:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158109; bh=B6OGYq5srkPnhGDNIl+tTbIP52HB70C3rO+EGN6g+aU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hplXYla8+EJW+RtqLjcVQ1vK9sWpEJ04tXCToA+6J+7mzFoXguWRc1zvkEZ02to7f T9LmCfbM8HzAOuZgi+ZaiDaWHq6RPIUlg+4JhZv8tZLKbIC99Ucr6qDr0CCSYUMauQ EQtCBLchpMOX5o50kRuvONlJ2VIj0LAgSJS1FfRHaq8JlHnjPjEIX+lQ0DPq1/uXB6 1DSj6kN2rqv3pv6FHy+RMdqoLoAqjbuk1tFjgvvrZ/ECED5taiasGYlLF2CzQ3RytL dE+i/Kv/c7RT4+NyT+6GrkP9hg51jel5npi9fer9OY7EpBcvvhm954yQxNvD0r1Qow ncRF5ekz6jTtg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtc-0067U2-K4; Tue, 31 Jan 2023 09:25:46 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 36/69] KVM: arm64: nv: Implement nested Stage-2 page table walk logic Date: Tue, 31 Jan 2023 09:24:31 +0000 Message-Id: <20230131092504.2880505-37-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014152_550915_AF4D18AE X-CRM114-Status: GOOD ( 26.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall Based on the pseudo-code in the ARM ARM, implement a stage 2 software page table walker. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim [maz: heavily reworked for future ARMv8.4-TTL support] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/esr.h | 1 + arch/arm64/include/asm/kvm_arm.h | 2 + arch/arm64/include/asm/kvm_nested.h | 13 ++ arch/arm64/kvm/nested.c | 270 ++++++++++++++++++++++++++++ 4 files changed, 286 insertions(+) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index d4dd949b921e..f1ad97375961 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -141,6 +141,7 @@ #define ESR_ELx_CM (UL(1) << ESR_ELx_CM_SHIFT) /* ISS field definitions for exceptions taken in to Hyp */ +#define ESR_ELx_FSC_ADDRSZ (0x00) #define ESR_ELx_CV (UL(1) << 24) #define ESR_ELx_COND_SHIFT (20) #define ESR_ELx_COND_MASK (UL(0xF) << ESR_ELx_COND_SHIFT) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 1ea71d26823c..680c02d8f38f 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -274,6 +274,8 @@ #define VTTBR_VMID_SHIFT (UL(48)) #define VTTBR_VMID_MASK(size) (_AT(u64, (1 << size) - 1) << VTTBR_VMID_SHIFT) +#define SCTLR_EE (UL(1) << 25) + /* Hyp System Trap Register */ #define HSTR_EL2_T(x) (1 << x) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index e3bcb351aae1..77badf494a9d 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -66,6 +66,19 @@ extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr); extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); +struct kvm_s2_trans { + phys_addr_t output; + unsigned long block_size; + bool writable; + bool readable; + int level; + u32 esr; + u64 upper_attr; +}; + +extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, + struct kvm_s2_trans *result); + extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 5514116429af..3ac3a0f334b0 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -78,6 +78,276 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) return ret; } +struct s2_walk_info { + int (*read_desc)(phys_addr_t pa, u64 *desc, void *data); + void *data; + u64 baddr; + unsigned int max_pa_bits; + unsigned int max_ipa_bits; + unsigned int pgshift; + unsigned int pgsize; + unsigned int ps; + unsigned int sl; + unsigned int t0sz; + bool be; + bool el1_aarch32; +}; + +static unsigned int ps_to_output_size(unsigned int ps) +{ + switch (ps) { + case 0: return 32; + case 1: return 36; + case 2: return 40; + case 3: return 42; + case 4: return 44; + case 5: + default: + return 48; + } +} + +static u32 compute_fsc(int level, u32 fsc) +{ + return fsc | (level & 0x3); +} + +static int check_base_s2_limits(struct s2_walk_info *wi, + int level, int input_size, int stride) +{ + int start_size; + + /* Check translation limits */ + switch (wi->pgsize) { + case SZ_64K: + if (level == 0 || (level == 1 && wi->max_ipa_bits <= 42)) + return -EFAULT; + break; + case SZ_16K: + if (level == 0 || (level == 1 && wi->max_ipa_bits <= 40)) + return -EFAULT; + break; + case SZ_4K: + if (level < 0 || (level == 0 && wi->max_ipa_bits <= 42)) + return -EFAULT; + break; + } + + /* Check input size limits */ + if (input_size > wi->max_ipa_bits && + (!wi->el1_aarch32 || input_size > 40)) + return -EFAULT; + + /* Check number of entries in starting level table */ + start_size = input_size - ((3 - level) * stride + wi->pgshift); + if (start_size < 1 || start_size > stride + 4) + return -EFAULT; + + return 0; +} + +/* Check if output is within boundaries */ +static int check_output_size(struct s2_walk_info *wi, phys_addr_t output) +{ + unsigned int output_size = ps_to_output_size(wi->ps); + + if (output_size > wi->max_pa_bits) + output_size = wi->max_pa_bits; + + if (output_size != 48 && (output & GENMASK_ULL(47, output_size))) + return -1; + + return 0; +} + +/* + * This is essentially a C-version of the pseudo code from the ARM ARM + * AArch64.TranslationTableWalk function. I strongly recommend looking at + * that pseudocode in trying to understand this. + * + * Must be called with the kvm->srcu read lock held + */ +static int walk_nested_s2_pgd(phys_addr_t ipa, + struct s2_walk_info *wi, struct kvm_s2_trans *out) +{ + int first_block_level, level, stride, input_size, base_lower_bound; + phys_addr_t base_addr; + unsigned int addr_top, addr_bottom; + u64 desc; /* page table entry */ + int ret; + phys_addr_t paddr; + + switch (wi->pgsize) { + case SZ_64K: + case SZ_16K: + level = 3 - wi->sl; + first_block_level = 2; + break; + case SZ_4K: + level = 2 - wi->sl; + first_block_level = 1; + break; + default: + /* GCC is braindead */ + unreachable(); + } + + stride = wi->pgshift - 3; + input_size = 64 - wi->t0sz; + if (input_size > 48 || input_size < 25) + return -EFAULT; + + ret = check_base_s2_limits(wi, level, input_size, stride); + if (WARN_ON(ret)) + return ret; + + base_lower_bound = 3 + input_size - ((3 - level) * stride + + wi->pgshift); + base_addr = wi->baddr & GENMASK_ULL(47, base_lower_bound); + + if (check_output_size(wi, base_addr)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ADDRSZ); + return 1; + } + + addr_top = input_size - 1; + + while (1) { + phys_addr_t index; + + addr_bottom = (3 - level) * stride + wi->pgshift; + index = (ipa & GENMASK_ULL(addr_top, addr_bottom)) + >> (addr_bottom - 3); + + paddr = base_addr | index; + ret = wi->read_desc(paddr, &desc, wi->data); + if (ret < 0) + return ret; + + /* + * Handle reversedescriptors if endianness differs between the + * host and the guest hypervisor. + */ + if (wi->be) + desc = be64_to_cpu(desc); + else + desc = le64_to_cpu(desc); + + /* Check for valid descriptor at this point */ + if (!(desc & 1) || ((desc & 3) == 1 && level == 3)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_FAULT); + out->upper_attr = desc; + return 1; + } + + /* We're at the final level or block translation level */ + if ((desc & 3) == 1 || level == 3) + break; + + if (check_output_size(wi, desc)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ADDRSZ); + out->upper_attr = desc; + return 1; + } + + base_addr = desc & GENMASK_ULL(47, wi->pgshift); + + level += 1; + addr_top = addr_bottom - 1; + } + + if (level < first_block_level) { + out->esr = compute_fsc(level, ESR_ELx_FSC_FAULT); + out->upper_attr = desc; + return 1; + } + + /* + * We don't use the contiguous bit in the stage-2 ptes, so skip check + * for misprogramming of the contiguous bit. + */ + + if (check_output_size(wi, desc)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ADDRSZ); + out->upper_attr = desc; + return 1; + } + + if (!(desc & BIT(10))) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ACCESS); + out->upper_attr = desc; + return 1; + } + + /* Calculate and return the result */ + paddr = (desc & GENMASK_ULL(47, addr_bottom)) | + (ipa & GENMASK_ULL(addr_bottom - 1, 0)); + out->output = paddr; + out->block_size = 1UL << ((3 - level) * stride + wi->pgshift); + out->readable = desc & (0b01 << 6); + out->writable = desc & (0b10 << 6); + out->level = level; + out->upper_attr = desc & GENMASK_ULL(63, 52); + return 0; +} + +static int read_guest_s2_desc(phys_addr_t pa, u64 *desc, void *data) +{ + struct kvm_vcpu *vcpu = data; + + return kvm_read_guest(vcpu->kvm, pa, desc, sizeof(*desc)); +} + +static void vtcr_to_walk_info(u64 vtcr, struct s2_walk_info *wi) +{ + wi->t0sz = vtcr & TCR_EL2_T0SZ_MASK; + + switch (vtcr & VTCR_EL2_TG0_MASK) { + case VTCR_EL2_TG0_4K: + wi->pgshift = 12; break; + case VTCR_EL2_TG0_16K: + wi->pgshift = 14; break; + case VTCR_EL2_TG0_64K: + default: + wi->pgshift = 16; break; + } + + wi->pgsize = BIT(wi->pgshift); + wi->ps = FIELD_GET(VTCR_EL2_PS_MASK, vtcr); + wi->sl = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr); + wi->max_ipa_bits = VTCR_EL2_IPA(vtcr); + /* Global limit for now, should eventually be per-VM */ + wi->max_pa_bits = get_kvm_ipa_limit(); +} + +int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, + struct kvm_s2_trans *result) +{ + u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + struct s2_walk_info wi; + int ret; + + result->esr = 0; + + if (!vcpu_has_nv(vcpu)) + return 0; + + wi.read_desc = read_guest_s2_desc; + wi.data = vcpu; + wi.baddr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + + vtcr_to_walk_info(vtcr, &wi); + + wi.be = vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_EE; + wi.el1_aarch32 = vcpu_mode_is_32bit(vcpu); + + ret = walk_nested_s2_pgd(gipa, &wi, result); + if (ret) + result->esr |= (kvm_vcpu_get_esr(vcpu) & ~ESR_ELx_FSC); + + return ret; +} + /* Must be called with kvm->mmu_lock held */ struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) { From patchwork Tue Jan 31 09:24:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7E45BC636CC for ; Tue, 31 Jan 2023 10:18:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PjXvKF3gv0OUn6TDPrWItwrSrbtiR0lEsaZqI8UcsD8=; b=NgcztYg75YPAzq B78zfOlOv6FFOQ9xscbcgpHNKk9qXxohJbENQMEDAbHCQqpxgPHK3cQkUsySLh/woB/cdcuBM8Ygw 4DJfvhuvbkI374AXAU6pjUbZsflDoP34jFCoi8BaDlmvpvt3+znYr6C3kSURO3fvLPBcPIPewEGmU Axbo6Go6LFqcJgzs8+Zl6GtDApOSY8+UtUTytq3P11FFzYV5S9J6PtkqMIbY1dpajJBi2vGQ7Agtc Aj4Mvg4HkbsB4PthN4m3OA31DC4Qbt2oo7Euqwb8sXNTO9dttZUFRvI7Yw694lfW5VerpiDb8n3gT +HQOyk5vU9AD1D4gSqRQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnhF-007KXp-1I; Tue, 31 Jan 2023 10:17:03 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9v-0075jT-H7 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:37 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 0664EB81AFB; Tue, 31 Jan 2023 09:42:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9BFC6C4339B; Tue, 31 Jan 2023 09:42:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158152; bh=XSH31bwQC/eV4T+wcxcgHsBjPTHFZSaxim9E8WZPu3c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MiFW2QrGQzYz3W2/PVXlRhpx9dsc83iXC1C7TA61PK3/vj35xBfuMFBoR/kj9pzEh Mpot9Ho0yFVw+rZBJuRZvr3EW6Q9xBQUtJN9nYF4Oue5TEBLuUYs7XLDMAVyg/k6k/ +RvbgcDzsEz+y3Hok2yMse3JyRxt7VLT8OPpv0096eYdf42qDT8v+4GWySANkB1PEJ NGu8Ht1PspoubpHH4g22gBJ5A6BBBJXMeQmoviYtczDJ0whUDkygE1eoLjN8ckdod0 xA5Oe/uy2Tzs8VO8bSrRpqo6yY5GZzQUk4GmTtRP4Badc5EGLqQIGqkfaKyrAO6FZa 8LrjOBId0foTg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtf-0067U2-Hh; Tue, 31 Jan 2023 09:25:48 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 37/69] KVM: arm64: nv: Handle shadow stage 2 page faults Date: Tue, 31 Jan 2023 09:24:32 +0000 Message-Id: <20230131092504.2880505-38-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014235_903822_55433CA9 X-CRM114-Status: GOOD ( 28.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If we are faulting on a shadow stage 2 translation, we first walk the guest hypervisor's stage 2 page table to see if it has a mapping. If not, we inject a stage 2 page fault to the virtual EL2. Otherwise, we create a mapping in the shadow stage 2 page table. Note that we have to deal with two IPAs when we got a shadow stage 2 page fault. One is the address we faulted on, and is in the L2 guest phys space. The other is from the guest stage-2 page table walk, and is in the L1 guest phys space. To differentiate them, we rename variables so that fault_ipa is used for the former and ipa is used for the latter. Co-developed-by: Christoffer Dall Co-developed-by: Jintack Lim Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim [maz: rewrote this multiple times...] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 6 ++ arch/arm64/include/asm/kvm_nested.h | 19 ++++++ arch/arm64/kvm/mmu.c | 90 ++++++++++++++++++++++++---- arch/arm64/kvm/nested.c | 48 +++++++++++++++ 4 files changed, 153 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index fe4b4b893fb8..1aa059ebb569 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -646,4 +646,10 @@ static inline bool vcpu_has_feature(struct kvm_vcpu *vcpu, int feature) return test_bit(feature, vcpu->arch.features); } +static inline bool kvm_is_shadow_s2_fault(struct kvm_vcpu *vcpu) +{ + return (vcpu->arch.hw_mmu != &vcpu->kvm->arch.mmu && + vcpu->arch.hw_mmu->nested_stage2_enabled); +} + #endif /* __ARM64_KVM_EMULATE_H__ */ diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 77badf494a9d..80371dea3652 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -76,9 +76,28 @@ struct kvm_s2_trans { u64 upper_attr; }; +static inline phys_addr_t kvm_s2_trans_output(struct kvm_s2_trans *trans) +{ + return trans->output; +} + +static inline unsigned long kvm_s2_trans_size(struct kvm_s2_trans *trans) +{ + return trans->block_size; +} + +static inline u32 kvm_s2_trans_esr(struct kvm_s2_trans *trans) +{ + return trans->esr; +} + extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, struct kvm_s2_trans *result); +extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, + struct kvm_s2_trans *trans); +extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2); +int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 9637049f74ae..7865acd4b924 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1225,14 +1225,16 @@ static bool kvm_vma_mte_allowed(struct vm_area_struct *vma) } static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, - struct kvm_memory_slot *memslot, unsigned long hva, - unsigned long fault_status) + struct kvm_s2_trans *nested, + struct kvm_memory_slot *memslot, + unsigned long hva, unsigned long fault_status) { int ret = 0; bool write_fault, writable, force_pte = false; bool exec_fault; bool device = false; unsigned long mmu_seq; + phys_addr_t ipa = fault_ipa; struct kvm *kvm = vcpu->kvm; struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache; struct vm_area_struct *vma; @@ -1303,10 +1305,39 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, } vma_pagesize = 1UL << vma_shift; + + if (nested) { + unsigned long max_map_size; + + max_map_size = force_pte ? PUD_SIZE : PAGE_SIZE; + + ipa = kvm_s2_trans_output(nested); + + /* + * If we're about to create a shadow stage 2 entry, then we + * can only create a block mapping if the guest stage 2 page + * table uses at least as big a mapping. + */ + max_map_size = min(kvm_s2_trans_size(nested), max_map_size); + + /* + * Be careful that if the mapping size falls between + * two host sizes, take the smallest of the two. + */ + if (max_map_size >= PMD_SIZE && max_map_size < PUD_SIZE) + max_map_size = PMD_SIZE; + else if (max_map_size >= PAGE_SIZE && max_map_size < PMD_SIZE) + max_map_size = PAGE_SIZE; + + force_pte = (max_map_size == PAGE_SIZE); + vma_pagesize = min(vma_pagesize, max_map_size); + } + if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE) fault_ipa &= ~(vma_pagesize - 1); - gfn = fault_ipa >> PAGE_SHIFT; + gfn = ipa >> PAGE_SHIFT; + mmap_read_unlock(current->mm); /* @@ -1468,8 +1499,10 @@ static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) */ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) { + struct kvm_s2_trans nested_trans, *nested = NULL; unsigned long fault_status; - phys_addr_t fault_ipa; + phys_addr_t fault_ipa; /* The address we faulted on */ + phys_addr_t ipa; /* Always the IPA in the L1 guest phys space */ struct kvm_memory_slot *memslot; unsigned long hva; bool is_iabt, write_fault, writable; @@ -1478,7 +1511,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) fault_status = kvm_vcpu_trap_get_fault_type(vcpu); - fault_ipa = kvm_vcpu_get_fault_ipa(vcpu); + ipa = fault_ipa = kvm_vcpu_get_fault_ipa(vcpu); is_iabt = kvm_vcpu_trap_is_iabt(vcpu); if (fault_status == ESR_ELx_FSC_FAULT) { @@ -1519,6 +1552,12 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) if (fault_status != ESR_ELx_FSC_FAULT && fault_status != ESR_ELx_FSC_PERM && fault_status != ESR_ELx_FSC_ACCESS) { + /* + * We must never see an address size fault on shadow stage 2 + * page table walk, because we would have injected an addr + * size fault when we walked the nested s2 page and not + * create the shadow entry. + */ kvm_err("Unsupported FSC: EC=%#x xFSC=%#lx ESR_EL2=%#lx\n", kvm_vcpu_trap_get_class(vcpu), (unsigned long)kvm_vcpu_trap_get_fault(vcpu), @@ -1528,7 +1567,37 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) idx = srcu_read_lock(&vcpu->kvm->srcu); - gfn = fault_ipa >> PAGE_SHIFT; + /* + * We may have faulted on a shadow stage 2 page table if we are + * running a nested guest. In this case, we have to resolve the L2 + * IPA to the L1 IPA first, before knowing what kind of memory should + * back the L1 IPA. + * + * If the shadow stage 2 page table walk faults, then we simply inject + * this to the guest and carry on. + */ + if (kvm_is_shadow_s2_fault(vcpu)) { + u32 esr; + + ret = kvm_walk_nested_s2(vcpu, fault_ipa, &nested_trans); + if (ret) { + esr = kvm_s2_trans_esr(&nested_trans); + kvm_inject_s2_fault(vcpu, esr); + goto out_unlock; + } + + ret = kvm_s2_handle_perm_fault(vcpu, &nested_trans); + if (ret) { + esr = kvm_s2_trans_esr(&nested_trans); + kvm_inject_s2_fault(vcpu, esr); + goto out_unlock; + } + + ipa = kvm_s2_trans_output(&nested_trans); + nested = &nested_trans; + } + + gfn = ipa >> PAGE_SHIFT; memslot = gfn_to_memslot(vcpu->kvm, gfn); hva = gfn_to_hva_memslot_prot(memslot, gfn, &writable); write_fault = kvm_is_write_fault(vcpu); @@ -1572,13 +1641,13 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) * faulting VA. This is always 12 bits, irrespective * of the page size. */ - fault_ipa |= kvm_vcpu_get_hfar(vcpu) & ((1 << 12) - 1); - ret = io_mem_abort(vcpu, fault_ipa); + ipa |= kvm_vcpu_get_hfar(vcpu) & ((1 << 12) - 1); + ret = io_mem_abort(vcpu, ipa); goto out_unlock; } /* Userspace should not be able to register out-of-bounds IPAs */ - VM_BUG_ON(fault_ipa >= kvm_phys_size(vcpu->kvm)); + VM_BUG_ON(ipa >= kvm_phys_size(vcpu->kvm)); if (fault_status == ESR_ELx_FSC_ACCESS) { handle_access_fault(vcpu, fault_ipa); @@ -1586,7 +1655,8 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) goto out_unlock; } - ret = user_mem_abort(vcpu, fault_ipa, memslot, hva, fault_status); + ret = user_mem_abort(vcpu, fault_ipa, nested, + memslot, hva, fault_status); if (ret == 0) ret = 1; out: diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 3ac3a0f334b0..5fb5e93248c1 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -112,6 +112,15 @@ static u32 compute_fsc(int level, u32 fsc) return fsc | (level & 0x3); } +static int esr_s2_fault(struct kvm_vcpu *vcpu, int level, u32 fsc) +{ + u32 esr; + + esr = kvm_vcpu_get_esr(vcpu) & ~ESR_ELx_FSC; + esr |= compute_fsc(level, fsc); + return esr; +} + static int check_base_s2_limits(struct s2_walk_info *wi, int level, int input_size, int stride) { @@ -464,6 +473,45 @@ void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu) } } +/* + * Returns non-zero if permission fault is handled by injecting it to the next + * level hypervisor. + */ +int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans) +{ + unsigned long fault_status = kvm_vcpu_trap_get_fault_type(vcpu); + bool forward_fault = false; + + trans->esr = 0; + + if (fault_status != ESR_ELx_FSC_PERM) + return 0; + + if (kvm_vcpu_trap_is_iabt(vcpu)) { + forward_fault = (trans->upper_attr & BIT(54)); + } else { + bool write_fault = kvm_is_write_fault(vcpu); + + forward_fault = ((write_fault && !trans->writable) || + (!write_fault && !trans->readable)); + } + + if (forward_fault) { + trans->esr = esr_s2_fault(vcpu, trans->level, ESR_ELx_FSC_PERM); + return 1; + } + + return 0; +} + +int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2) +{ + vcpu_write_sys_reg(vcpu, vcpu->arch.fault.far_el2, FAR_EL2); + vcpu_write_sys_reg(vcpu, vcpu->arch.fault.hpfar_el2, HPFAR_EL2); + + return kvm_inject_nested_sync(vcpu, esr_el2); +} + void kvm_arch_flush_shadow_all(struct kvm *kvm) { int i; From patchwork Tue Jan 31 09:24:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 085FDC38142 for ; Tue, 31 Jan 2023 10:23:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JyS6Em9xbERfJi/yAZNLJGLPbGXz5TRyd7Fppa2+RvI=; b=d70oHgqoyby8O7 eyKJ7hbb/Tng/MeWcxDyiXhj0Df69DY5nZKYBztFtuv7W7MULHXv/az6WJN7xKmkx4ymGje8r6j97 GtpvAV5q+Ea9hfGXGHhAfWVCiP1/wJ994Gs/t3Bi1TtrDzh8rWvv628r4LmcCiCQyeZwizPZrT5bt sFyoAZ4pQmbE0RjhIc1jR6A1wQd0GMRHQyRW0YjaoriDSa35LsXIc+TCrrU8UbADEsc60e2hskg8i 9ChPS5+kTgePp+m8QHNVqZVau7CYiQVpR3mJw6Ov7+EJvpWlkargLbUvDlUmcqgRj715XIRwpu2pJ 4HL6L7S7bPLnpZIVreLw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnmR-007MoV-BJ; Tue, 31 Jan 2023 10:22:23 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnAE-0075ux-Ck for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:56 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E89286148E; Tue, 31 Jan 2023 09:42:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 58548C433EF; Tue, 31 Jan 2023 09:42:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158173; bh=VxmRcf1P6efLkPjQVx9LunzmqtSb81ioZU+Jf+6ji0k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rTVE5nDEKUP0Q1BcjJj+CsVYJLt+OuemeegeEkGeNrwA0IuELFjwBcP32a16t7X5H vKhdVE+tum2D2XwxYhxOukDk7+IZmu7HEpJMeXpD/qacEph5ugzy9JAxsh9lkvJGfl 4+EM9B4oi7prIk/vwMMYA/2DOh+EEHOTwYTlT1xhHbipHAfRooZsJvwqT66Eexj032 i7MPDHZjowp13Ap0Md78Jcaq/Z4eN1ttirz/s9lUNalJv9hFGDSgEF0AQbMsEhPxGp II1G9OYW6CJNa1mDIMsYPCRWqbRYkJItOAgZb8JVaiDM+UPacNSJyWLDGNpjUhGQ+Z XpN3gEMFlLERg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtg-0067U2-RA; Tue, 31 Jan 2023 09:25:52 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 38/69] KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's Date: Tue, 31 Jan 2023 09:24:33 +0000 Message-Id: <20230131092504.2880505-39-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014254_542175_C6F871EB X-CRM114-Status: GOOD ( 16.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When mapping a page in a shadow stage-2, special care must be taken not to be more permissive than the guest is (writable or readable page when the guest hasn't set that permission). Reviewed-by: Alexandru Elisei Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 15 +++++++++++++++ arch/arm64/kvm/mmu.c | 14 +++++++++++++- arch/arm64/kvm/nested.c | 2 +- 3 files changed, 29 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 80371dea3652..fc96873b6dfe 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -91,6 +91,21 @@ static inline u32 kvm_s2_trans_esr(struct kvm_s2_trans *trans) return trans->esr; } +static inline bool kvm_s2_trans_readable(struct kvm_s2_trans *trans) +{ + return trans->readable; +} + +static inline bool kvm_s2_trans_writable(struct kvm_s2_trans *trans) +{ + return trans->writable; +} + +static inline bool kvm_s2_trans_executable(struct kvm_s2_trans *trans) +{ + return !(trans->upper_attr & BIT(54)); +} + extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, struct kvm_s2_trans *result); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 7865acd4b924..a42e7dc2be51 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1402,6 +1402,17 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (exec_fault && device) return -ENOEXEC; + /* + * Potentially reduce shadow S2 permissions to match the guest's own + * S2. For exec faults, we'd only reach this point if the guest + * actually allowed it (see kvm_s2_handle_perm_fault). + */ + if (nested) { + writable &= kvm_s2_trans_writable(nested); + if (!kvm_s2_trans_readable(nested)) + prot &= ~KVM_PGTABLE_PROT_R; + } + read_lock(&kvm->mmu_lock); pgt = vcpu->arch.hw_mmu->pgt; if (mmu_invalidate_retry(kvm, mmu_seq)) @@ -1439,7 +1450,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (device) prot |= KVM_PGTABLE_PROT_DEVICE; - else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC)) + else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC) && + (!nested || kvm_s2_trans_executable(nested))) prot |= KVM_PGTABLE_PROT_X; /* diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 5fb5e93248c1..dc74c9b8db89 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -488,7 +488,7 @@ int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans) return 0; if (kvm_vcpu_trap_is_iabt(vcpu)) { - forward_fault = (trans->upper_attr & BIT(54)); + forward_fault = !kvm_s2_trans_executable(trans); } else { bool write_fault = kvm_is_write_fault(vcpu); From patchwork Tue Jan 31 09:24:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4E465C636CC for ; Tue, 31 Jan 2023 10:21:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2BXX9aUfbx/dKre/t614Srtj3HAmdfrGJN9sZ+5iujI=; b=EmC1PorfC3DbBG zj3TNhpfIjPw6/6vIjuD2sOrK6bVdwD0lo/hqE0Kyr/i9XJ5JOK4Mm9PAyxvAFh76q2OukHZm5eyw j3ahhhB4u6/R5vHig6TiiqztKdlLSyt/bWiIvDMGwUmbC0knD8KrFoLZDv+8faKRtOf8wXWfiRaom wGZtF2empeVUh29ecnLRJmGPzuuzaTBBON7qrJ4jKI955+/W3pOUMCq7myLwOirIbqXl9iwZ7mWyi jSr5o7j39S7Vtqj1Rge0ZkvcYT9besj6G3Cnsa+4T2Wy5IGeXZXZ+7sw6cKZwfACwYSa81YG/d4oh slB1rnCdRibtl0siFUfA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnk9-007Lpb-UZ; Tue, 31 Jan 2023 10:20:02 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnA3-0075nb-Bd for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:45 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id EE934B81AF2; Tue, 31 Jan 2023 09:42:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 83199C433EF; Tue, 31 Jan 2023 09:42:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158160; bh=tzeat7kH9efZCoFbmQliqntFTi+c1k8RuHaFmpvRQiU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ojvOh5q1Gro88RS5EnQztKZIWgwAyDR53oycuqaeMfYJRE6s2hzekvZpsXStVgj8R T8oYW2tT1+1PndR0rF82vlR5/bqtZg9BtgJUB9iX98E0DQJ8cNPKVeU0fTaMw2tTPB xiQYiRpOlPy1wQEtkWh85j+p5mi3msBwPcevvpiXpRstaQKUnaJKrN3m6SAWFZ+GTj 34DMjOEcahfMebpQ1h7+xkZkxRROpHDYBDuvQTRamGrGP0aX8mrVVkgbHIe2E6y6jO rKn6LwnJA/bURUXiA0wuO0uk9kgNLCWpXWdAhaus133jKFOsLJD3QVMvDDSfIwIWF0 LcVBXdgeQsjFA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtl-0067U2-5R; Tue, 31 Jan 2023 09:25:53 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 39/69] KVM: arm64: nv: Unmap/flush shadow stage 2 page tables Date: Tue, 31 Jan 2023 09:24:34 +0000 Message-Id: <20230131092504.2880505-40-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014243_758469_CDD9C2D0 X-CRM114-Status: GOOD ( 19.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall Unmap/flush shadow stage 2 page tables for the nested VMs as well as the stage 2 page table for the guest hypervisor. Note: A bunch of the code in mmu.c relating to MMU notifiers is currently dealt with in an extremely abrupt way, for example by clearing out an entire shadow stage-2 table. This will be handled in a more efficient way using the reverse mapping feature in a later version of the patch series. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_mmu.h | 3 +++ arch/arm64/include/asm/kvm_nested.h | 3 +++ arch/arm64/kvm/mmu.c | 31 +++++++++++++++++++---- arch/arm64/kvm/nested.c | 39 +++++++++++++++++++++++++++++ 4 files changed, 71 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 0cb5ad7b0852..679d87b72fec 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -165,6 +165,8 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size, void __iomem **haddr); int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void **haddr); +void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu, + phys_addr_t addr, phys_addr_t end); void __init free_hyp_pgds(void); void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size); @@ -173,6 +175,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu); int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, phys_addr_t pa, unsigned long size, bool writable); +void kvm_stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end); int kvm_handle_guest_abort(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index fc96873b6dfe..282fe63ed7b2 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -112,6 +112,9 @@ extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans); extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2); +extern void kvm_nested_s2_wp(struct kvm *kvm); +extern void kvm_nested_s2_unmap(struct kvm *kvm); +extern void kvm_nested_s2_flush(struct kvm *kvm); int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index a42e7dc2be51..6dc82b0384ce 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -245,13 +245,20 @@ void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size) __unmap_stage2_range(mmu, start, size, true); } +void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu, + phys_addr_t addr, phys_addr_t end) +{ + stage2_apply_range_resched(mmu, addr, end, kvm_pgtable_stage2_flush); +} + static void stage2_flush_memslot(struct kvm *kvm, struct kvm_memory_slot *memslot) { phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT; phys_addr_t end = addr + PAGE_SIZE * memslot->npages; + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; - stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_flush); + kvm_stage2_flush_range(mmu, addr, end); } /** @@ -274,6 +281,8 @@ static void stage2_flush_vm(struct kvm *kvm) kvm_for_each_memslot(memslot, bkt, slots) stage2_flush_memslot(kvm, memslot); + kvm_nested_s2_flush(kvm); + write_unlock(&kvm->mmu_lock); srcu_read_unlock(&kvm->srcu, idx); } @@ -868,6 +877,8 @@ void stage2_unmap_vm(struct kvm *kvm) kvm_for_each_memslot(memslot, bkt, slots) stage2_unmap_memslot(kvm, memslot); + kvm_nested_s2_unmap(kvm); + write_unlock(&kvm->mmu_lock); mmap_read_unlock(current->mm); srcu_read_unlock(&kvm->srcu, idx); @@ -966,12 +977,12 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, } /** - * stage2_wp_range() - write protect stage2 memory region range + * kvm_stage2_wp_range() - write protect stage2 memory region range * @mmu: The KVM stage-2 MMU pointer * @addr: Start address of range * @end: End address of range */ -static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) +void kvm_stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) { stage2_apply_range_resched(mmu, addr, end, kvm_pgtable_stage2_wrprotect); } @@ -1002,7 +1013,8 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; write_lock(&kvm->mmu_lock); - stage2_wp_range(&kvm->arch.mmu, start, end); + kvm_stage2_wp_range(&kvm->arch.mmu, start, end); + kvm_nested_s2_wp(kvm); write_unlock(&kvm->mmu_lock); kvm_flush_remote_tlbs(kvm); } @@ -1026,7 +1038,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; - stage2_wp_range(&kvm->arch.mmu, start, end); + kvm_stage2_wp_range(&kvm->arch.mmu, start, end); } /* @@ -1041,6 +1053,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, gfn_t gfn_offset, unsigned long mask) { kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask); + kvm_nested_s2_wp(kvm); } static void kvm_send_hwpoison_signal(unsigned long address, short lsb) @@ -1690,6 +1703,7 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) (range->end - range->start) << PAGE_SHIFT, range->may_block); + kvm_nested_s2_unmap(kvm); return false; } @@ -1724,6 +1738,7 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) PAGE_SIZE, __pfn_to_phys(pfn), KVM_PGTABLE_PROT_R, NULL, 0); + kvm_nested_s2_unmap(kvm); return false; } @@ -1742,6 +1757,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) range->start << PAGE_SHIFT); pte = __pte(kpte); return pte_valid(pte) && pte_young(pte); + + /* + * TODO: Handle nested_mmu structures here using the reverse mapping in + * a later version of patch series. + */ } bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) @@ -1974,6 +1994,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, write_lock(&kvm->mmu_lock); kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, size); + kvm_nested_s2_unmap(kvm); write_unlock(&kvm->mmu_lock); } diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index dc74c9b8db89..4356af6cbed0 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -512,6 +512,45 @@ int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2) return kvm_inject_nested_sync(vcpu, esr_el2); } +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_wp(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (kvm_s2_mmu_valid(mmu)) + kvm_stage2_wp_range(mmu, 0, kvm_phys_size(kvm)); + } +} + +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_unmap(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (kvm_s2_mmu_valid(mmu)) + kvm_unmap_stage2_range(mmu, 0, kvm_phys_size(kvm)); + } +} + +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_flush(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (kvm_s2_mmu_valid(mmu)) + kvm_stage2_flush_range(mmu, 0, kvm_phys_size(kvm)); + } +} + void kvm_arch_flush_shadow_all(struct kvm *kvm) { int i; From patchwork Tue Jan 31 09:24:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 57F3FC38142 for ; Tue, 31 Jan 2023 09:54:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2RTx2L4TmtB7fbGkB6492qWPf5FAOS2niRjXv6rK3jY=; b=YzvVeW9pKezS4F C691mGwfOty/bTMM0NjyCEVLKFAEQoRX9KL3LdkYW84Blh7zwMwQFUkvPGkItLTF6eKzXNpRPKPRg EqKGBOms+ct0iFdDT/H4R2KWuf4l65jKPaQsnrSvUVrBZ3DoiSmkZvKpwvhPkVZen+JCXzpou0qrs mmR44WIzTHkRvhm7OZ1ypipRz43Lo97ds3KS0prpt5V/nxFPpIOO13+PCVRXLpdq/c4O2amKWtKuU DXD+7Ar41C7OlRSbPZG3CEP1yNeAP2z/JASNVXAE8dxTPdBYRlTpGfLs5gQxaShqEEv9pIxqClVLj HC3vTZIPqnmVxE3HTSig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnKY-007AKk-4t; Tue, 31 Jan 2023 09:53:34 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8y-0075Gm-EU for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:38 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 120D2B81ACC; Tue, 31 Jan 2023 09:41:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BCBFAC433D2; Tue, 31 Jan 2023 09:41:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158093; bh=/5D0aubgwrzAq9dww2bq0tpKJiXgd+SVS/PFO/Z3p2g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JqXS59ZfgrAfAKi6djSX4nUB4s0Wuu70/Z+ZWkUrltbD06v5hAOpTImYp2MpdGipt pdtFiDvUnH3xq/hXWORQ4UcpTFx1FUamAebOYkm7F/POVTZQ6V42i51LfVLACpjwWG 84P4gFdi93ta1XUBI6G0eBAAk3Uep7qwC+Gomc+/6d25q4EaJOEu9fWqD4SWDqChUJ DC09NHL00yShwyf5bJfPV27q7Yq+OqeYwrmKKYf1BLcLiB5uESscLASIX5U9+QKTz0 oOhwT5S7yJDsC8lk6z+TcjjxREusI3ae5FiM5yDhURgx6TyJvL4nxO3kLgX6OApB9R 4dwiCjB6g5dkw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtn-0067U2-3N; Tue, 31 Jan 2023 09:25:57 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 40/69] KVM: arm64: nv: Set a handler for the system instruction traps Date: Tue, 31 Jan 2023 09:24:35 +0000 Message-Id: <20230131092504.2880505-41-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014136_842770_4927BE61 X-CRM114-Status: GOOD ( 18.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When HCR.NV bit is set, execution of the EL2 translation regime address aranslation instructions and TLB maintenance instructions are trapped to EL2. In addition, execution of the EL1 translation regime address aranslation instructions and TLB maintenance instructions that are only accessible from EL2 and above are trapped to EL2. In these cases, ESR_EL2.EC will be set to 0x18. Rework the system instruction emulation framework to handle potentially all system instruction traps other than MSR/MRS instructions. Those system instructions would be AT and TLBI instructions controlled by HCR_EL2.NV, AT, and TTLB bits. Signed-off-by: Jintack Lim [maz: squashed two patches together, redispatched various bits around] Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 47 +++++++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 93a0c10d8e0c..f049f6f052bd 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1988,10 +1988,6 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, * guest... */ static const struct sys_reg_desc sys_reg_descs[] = { - { SYS_DESC(SYS_DC_ISW), access_dcsw }, - { SYS_DESC(SYS_DC_CSW), access_dcsw }, - { SYS_DESC(SYS_DC_CISW), access_dcsw }, - DBG_BCR_BVR_WCR_WVR_EL1(0), DBG_BCR_BVR_WCR_WVR_EL1(1), { SYS_DESC(SYS_MDCCINT_EL1), trap_debug_regs, reset_val, MDCCINT_EL1, 0 }, @@ -2468,6 +2464,12 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL2_REG(SP_EL2, NULL, reset_unknown, 0), }; +static struct sys_reg_desc sys_insn_descs[] = { + { SYS_DESC(SYS_DC_ISW), access_dcsw }, + { SYS_DESC(SYS_DC_CSW), access_dcsw }, + { SYS_DESC(SYS_DC_CISW), access_dcsw }, +}; + static bool trap_dbgdidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -3156,6 +3158,24 @@ static bool emulate_sys_reg(struct kvm_vcpu *vcpu, return false; } +static int emulate_sys_instr(struct kvm_vcpu *vcpu, struct sys_reg_params *p) +{ + const struct sys_reg_desc *r; + + /* Search from the system instruction table. */ + r = find_reg(p, sys_insn_descs, ARRAY_SIZE(sys_insn_descs)); + + if (likely(r)) { + perform_access(vcpu, p, r); + } else { + kvm_err("Unsupported guest sys instruction at: %lx\n", + *vcpu_pc(vcpu)); + print_sys_reg_instr(p); + kvm_inject_undefined(vcpu); + } + return 1; +} + /** * kvm_reset_sys_regs - sets system registers to reset value * @vcpu: The VCPU pointer @@ -3173,7 +3193,8 @@ void kvm_reset_sys_regs(struct kvm_vcpu *vcpu) } /** - * kvm_handle_sys_reg -- handles a mrs/msr trap on a guest sys_reg access + * kvm_handle_sys_reg -- handles a system instruction or mrs/msr instruction + * trap on a guest execution * @vcpu: The VCPU pointer */ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu) @@ -3187,12 +3208,19 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu) params = esr_sys64_to_params(esr); params.regval = vcpu_get_reg(vcpu, Rt); - if (!emulate_sys_reg(vcpu, ¶ms)) + /* System register? */ + if (params.Op0 == 2 || params.Op0 == 3) { + if (!emulate_sys_reg(vcpu, ¶ms)) + return 1; + + if (!params.is_write) + vcpu_set_reg(vcpu, Rt, params.regval); + return 1; + } - if (!params.is_write) - vcpu_set_reg(vcpu, Rt, params.regval); - return 1; + /* Hints, PSTATE (Op0 == 0) and System instructions (Op0 == 1) */ + return emulate_sys_instr(vcpu, ¶ms); } /****************************************************************************** @@ -3584,6 +3612,7 @@ int __init kvm_sys_reg_table_init(void) valid &= check_sysreg_table(cp15_regs, ARRAY_SIZE(cp15_regs), true); valid &= check_sysreg_table(cp15_64_regs, ARRAY_SIZE(cp15_64_regs), true); valid &= check_sysreg_table(invariant_sys_regs, ARRAY_SIZE(invariant_sys_regs), false); + valid &= check_sysreg_table(sys_insn_descs, ARRAY_SIZE(sys_insn_descs), false); if (!valid) return -EINVAL; From patchwork Tue Jan 31 09:24:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122820 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 52705C38142 for ; Tue, 31 Jan 2023 10:20:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QRAjh4Nd540pSX+2U9fCNmulnTqbGmfuWFBS6wKPz1g=; b=lhn+40iFt3ihBG Z20bqYoHW0v3SRTFFZ7gwvsMiiBfL/9A++BwNIguhOfG/cYSRcc5D3Ebe01SwvtaTQJEzdmYvLWyL 1M0iBaPgqeLIolQPyx3ySlu+CUUjezOY44NTRp89ZMqXJBnwvSX5BYGR3pQxaroJie73y9gq4ZqTG IqKP4lWRqiK+N4L6CMZ774mCh3biMTJaoC6chR8gmFIY5iHNi2F6bhYWpThdpy3s8r4Vdf7QUJiUd Cx9k5sz+X9B4hKcSUVq0CvDb5IAFRa6bBW/tYQdIu6YcXChcFld0HZ5nzYDfgumESEQ95MXb01rMj ENYXsFln4nl4LiQdsqyg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnjK-007LS9-4w; Tue, 31 Jan 2023 10:19:11 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9y-0075lT-L2 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:41 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1443461489; Tue, 31 Jan 2023 09:42:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 32C8AC4339B; Tue, 31 Jan 2023 09:42:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158157; bh=HcxymYHr0cCn+xy800QZobpP+MV4lGj5zZMBwtm6OXo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GaHG7NfwHlIOawjN6Xrr0V63Y59rnM33ei4Q0nyrRicaxEun5vcGRYHwVIU+bEFIh wQ2O75zrRbHxrgEo0lAePWth9C9uJmbMF4G9h1h7mLnqWlzX7xk9hrwt1uyuohdJtH 01LO2TIklAUMXUR1LORd6eujaT/tzg1rGJIARSzR9kKAdzG+kZH/JKy/FPNoWgbVGU ZETtMHQFPeHTy6MPHP9U4nnqTrbRBgeN+D1mn+lb5ykokp2gb1+nO7os+ehLDSASEh 78KfAn3Ay/xVLsmMbhxlJe+7UKLmWMbSNfez12ee2W+QnCG/lEmB9ONH3fFNSfILD3 ulpgJlidPaWYw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtp-0067U2-MN; Tue, 31 Jan 2023 09:25:57 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 41/69] KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2 Date: Tue, 31 Jan 2023 09:24:36 +0000 Message-Id: <20230131092504.2880505-42-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014238_845125_9746D351 X-CRM114-Status: GOOD ( 33.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When supporting nested virtualization a guest hypervisor executing AT instructions must be trapped and emulated by the host hypervisor, because untrapped AT instructions operating on S1E1 will use the wrong translation regieme (the one used to emulate virtual EL2 in EL1 instead of virtual EL1) and AT instructions operating on S12 will not work from EL1. This patch does several things. 1. List and define all AT system instructions to emulate and document the emulation design. 2. Implement AT instruction handling logic in EL2. This will be used to emulate AT instructions executed in the virtual EL2. AT instruction emulation works by loading the proper processor context, which depends on the trapped instruction and the virtual HCR_EL2, to the EL1 virtual memory control registers and executing AT instructions. Note that ctxt->hw_sys_regs is expected to have the proper processor context before calling the handling function(__kvm_at_insn) implemented in this patch. 4. Emulate AT S1E[01] instructions by issuing the same instructions in EL2. We set the physical EL1 registers, NV and NV1 bits as described in the AT instruction emulation overview. 5. Emulate AT A12E[01] instructions in two steps: First, do the stage-1 translation by reusing the existing AT emulation functions. Second, do the stage-2 translation by walking the guest hypervisor's stage-2 page table in software. Record the translation result to PAR_EL1. 6. Emulate AT S1E2 instructions by issuing the corresponding S1E1 instructions in EL2. We set the physical EL1 registers and the HCR_EL2 register as described in the AT instruction emulation overview. 7. Forward system instruction traps to the virtual EL2 if the corresponding virtual AT bit is set in the virtual HCR_EL2. [ Much logic above has been reworked by Marc Zyngier ] Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier Signed-off-by: Christoffer Dall --- arch/arm64/include/asm/kvm_arm.h | 2 + arch/arm64/include/asm/kvm_asm.h | 2 + arch/arm64/include/asm/sysreg.h | 17 +++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/at.c | 221 ++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/switch.c | 13 +- arch/arm64/kvm/sys_regs.c | 226 +++++++++++++++++++++++++++++++ 7 files changed, 480 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/kvm/at.c diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 680c02d8f38f..d446fccdca5a 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -20,6 +20,7 @@ #define HCR_AMVOFFEN (UL(1) << 51) #define HCR_FIEN (UL(1) << 47) #define HCR_FWB (UL(1) << 46) +#define HCR_AT (UL(1) << 44) #define HCR_NV1 (UL(1) << 43) #define HCR_NV (UL(1) << 42) #define HCR_API (UL(1) << 41) @@ -120,6 +121,7 @@ #define VTCR_EL2_TG0_16K TCR_TG0_16K #define VTCR_EL2_TG0_64K TCR_TG0_64K #define VTCR_EL2_SH0_MASK TCR_SH0_MASK +#define VTCR_EL2_SH0_SHIFT TCR_SH0_SHIFT #define VTCR_EL2_SH0_INNER TCR_SH0_INNER #define VTCR_EL2_ORGN0_MASK TCR_ORGN0_MASK #define VTCR_EL2_ORGN0_WBWA TCR_ORGN0_WBWA diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 43c3bc0f9544..373558ea24d8 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -228,6 +228,8 @@ extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_timer_set_cntvoff(u64 cntvoff); +extern void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr); +extern void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr); extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 4b9024216eba..9c718f4af944 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -607,6 +607,23 @@ #define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0) +/* AT instructions */ +#define AT_Op0 1 +#define AT_CRn 7 + +#define OP_AT_S1E1R sys_insn(AT_Op0, 0, AT_CRn, 8, 0) +#define OP_AT_S1E1W sys_insn(AT_Op0, 0, AT_CRn, 8, 1) +#define OP_AT_S1E0R sys_insn(AT_Op0, 0, AT_CRn, 8, 2) +#define OP_AT_S1E0W sys_insn(AT_Op0, 0, AT_CRn, 8, 3) +#define OP_AT_S1E1RP sys_insn(AT_Op0, 0, AT_CRn, 9, 0) +#define OP_AT_S1E1WP sys_insn(AT_Op0, 0, AT_CRn, 9, 1) +#define OP_AT_S1E2R sys_insn(AT_Op0, 4, AT_CRn, 8, 0) +#define OP_AT_S1E2W sys_insn(AT_Op0, 4, AT_CRn, 8, 1) +#define OP_AT_S12E1R sys_insn(AT_Op0, 4, AT_CRn, 8, 4) +#define OP_AT_S12E1W sys_insn(AT_Op0, 4, AT_CRn, 8, 5) +#define OP_AT_S12E0R sys_insn(AT_Op0, 4, AT_CRn, 8, 6) +#define OP_AT_S12E0W sys_insn(AT_Op0, 4, AT_CRn, 8, 7) + /* Common SCTLR_ELx flags. */ #define SCTLR_ELx_ENTP2 (BIT(60)) #define SCTLR_ELx_DSSBS (BIT(44)) diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index c0c050e53157..c2717a8f12f5 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -14,7 +14,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ inject_fault.o va_layout.o handle_exit.o \ guest.o debug.o reset.o sys_regs.o stacktrace.o \ vgic-sys-reg-v3.o fpsimd.o pkvm.o \ - arch_timer.o trng.o vmid.o emulate-nested.o nested.o \ + arch_timer.o trng.o vmid.o emulate-nested.o nested.o at.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ diff --git a/arch/arm64/kvm/at.c b/arch/arm64/kvm/at.c new file mode 100644 index 000000000000..d11059f9f077 --- /dev/null +++ b/arch/arm64/kvm/at.c @@ -0,0 +1,221 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2017 - Linaro Ltd + * Author: Jintack Lim + */ + +#include +#include + +struct mmu_config { + u64 ttbr0; + u64 ttbr1; + u64 tcr; + u64 sctlr; + u64 vttbr; + u64 vtcr; + u64 hcr; +}; + +static void __mmu_config_save(struct mmu_config *config) +{ + config->ttbr0 = read_sysreg_el1(SYS_TTBR0); + config->ttbr1 = read_sysreg_el1(SYS_TTBR1); + config->tcr = read_sysreg_el1(SYS_TCR); + config->sctlr = read_sysreg_el1(SYS_SCTLR); + config->vttbr = read_sysreg(vttbr_el2); + config->vtcr = read_sysreg(vtcr_el2); + config->hcr = read_sysreg(hcr_el2); +} + +static void __mmu_config_restore(struct mmu_config *config) +{ + write_sysreg_el1(config->ttbr0, SYS_TTBR0); + write_sysreg_el1(config->ttbr1, SYS_TTBR1); + write_sysreg_el1(config->tcr, SYS_TCR); + write_sysreg_el1(config->sctlr, SYS_SCTLR); + write_sysreg(config->vttbr, vttbr_el2); + write_sysreg(config->vtcr, vtcr_el2); + write_sysreg(config->hcr, hcr_el2); + + isb(); +} + +void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) +{ + struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; + struct mmu_config config; + struct kvm_s2_mmu *mmu; + bool fail; + + write_lock(&vcpu->kvm->mmu_lock); + + /* + * If HCR_EL2.{E2H,TGE} == {1,1}, the MMU context is already + * the right one (as we trapped from vEL2). + */ + if (vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu)) + goto skip_mmu_switch; + + /* + * FIXME: Obtaining the S2 MMU for a guest guest is horribly + * racy, and we may not find it (evicted by another vcpu, for + * example). + */ + mmu = lookup_s2_mmu(vcpu->kvm, + vcpu_read_sys_reg(vcpu, VTTBR_EL2), + vcpu_read_sys_reg(vcpu, HCR_EL2)); + + if (WARN_ON(!mmu)) + goto out; + + /* We've trapped, so everything is live on the CPU. */ + __mmu_config_save(&config); + + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL1), SYS_TTBR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL1), SYS_TTBR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL1), SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL1), SYS_SCTLR); + write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); + /* + * REVISIT: do we need anything from the guest's VTCR_EL2? If + * looks like keeping the hosts configuration is the right + * thing to do at this stage (and we could avoid save/restore + * it. Keep the host's version for now. + */ + write_sysreg((config.hcr & ~HCR_TGE) | HCR_VM, hcr_el2); + + isb(); + +skip_mmu_switch: + + switch (op) { + case OP_AT_S1E1R: + case OP_AT_S1E1RP: + fail = __kvm_at("s1e1r", vaddr); + break; + case OP_AT_S1E1W: + case OP_AT_S1E1WP: + fail = __kvm_at("s1e1w", vaddr); + break; + case OP_AT_S1E0R: + fail = __kvm_at("s1e0r", vaddr); + break; + case OP_AT_S1E0W: + fail = __kvm_at("s1e0w", vaddr); + break; + default: + WARN_ON_ONCE(1); + break; + } + + if (!fail) + ctxt_sys_reg(ctxt, PAR_EL1) = read_sysreg(par_el1); + else + ctxt_sys_reg(ctxt, PAR_EL1) = SYS_PAR_EL1_F; + + /* + * Failed? let's leave the building now. + * + * FIXME: how about a failed translation because the shadow S2 + * wasn't populated? We may need to perform a SW PTW, + * populating our shadow S2 and retry the instruction. + */ + if (ctxt_sys_reg(ctxt, PAR_EL1) & SYS_PAR_EL1_F) + goto nopan; + + /* No PAN? No problem. */ + if (!vcpu_el2_e2h_is_set(vcpu) || !(*vcpu_cpsr(vcpu) & PSR_PAN_BIT)) + goto nopan; + + /* + * For PAN-involved AT operations, perform the same + * translation, using EL0 this time. + */ + switch (op) { + case OP_AT_S1E1RP: + fail = __kvm_at("s1e0r", vaddr); + break; + case OP_AT_S1E1WP: + fail = __kvm_at("s1e0w", vaddr); + break; + default: + goto nopan; + } + + /* + * If the EL0 translation has succeeded, we need to pretend + * the AT operation has failed, as the PAN setting forbids + * such a translation. + * + * FIXME: we hardcode a Level-3 permission fault. We really + * should return the real fault level. + */ + if (fail || !(read_sysreg(par_el1) & SYS_PAR_EL1_F)) + ctxt_sys_reg(ctxt, PAR_EL1) = (0xf << 1) | SYS_PAR_EL1_F; + +nopan: + if (!(vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu))) + __mmu_config_restore(&config); + +out: + write_unlock(&vcpu->kvm->mmu_lock); +} + +void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) +{ + struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; + struct mmu_config config; + struct kvm_s2_mmu *mmu; + u64 val; + + write_lock(&vcpu->kvm->mmu_lock); + + mmu = &vcpu->kvm->arch.mmu; + + /* We've trapped, so everything is live on the CPU. */ + __mmu_config_save(&config); + + if (vcpu_el2_e2h_is_set(vcpu)) { + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL2), SYS_TTBR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL2), SYS_TTBR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL2), SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL2), SYS_SCTLR); + + val = config.hcr; + } else { + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL2), SYS_TTBR0); + val = translate_tcr_el2_to_tcr_el1(ctxt_sys_reg(ctxt, TCR_EL2)); + write_sysreg_el1(val, SYS_TCR); + val = translate_sctlr_el2_to_sctlr_el1(ctxt_sys_reg(ctxt, SCTLR_EL2)); + write_sysreg_el1(val, SYS_SCTLR); + + val = config.hcr | HCR_NV | HCR_NV1; + } + + write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); + /* FIXME: write S2 MMU VTCR_EL2? */ + write_sysreg((val & ~HCR_TGE) | HCR_VM, hcr_el2); + + isb(); + + switch (op) { + case OP_AT_S1E2R: + asm volatile("at s1e1r, %0" : : "r" (vaddr)); + break; + case OP_AT_S1E2W: + asm volatile("at s1e1w, %0" : : "r" (vaddr)); + break; + default: + WARN_ON_ONCE(1); + break; + } + + isb(); + + /* FIXME: handle failed translation due to shadow S2 */ + ctxt_sys_reg(ctxt, PAR_EL1) = read_sysreg(par_el1); + + __mmu_config_restore(&config); + write_unlock(&vcpu->kvm->mmu_lock); +} diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 36b50e315504..6d186a3e5e5f 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -44,9 +44,10 @@ static void __activate_traps(struct kvm_vcpu *vcpu) if (!vcpu_el2_e2h_is_set(vcpu)) { /* * For a guest hypervisor on v8.0, trap and emulate - * the EL1 virtual memory control register accesses. + * the EL1 virtual memory control register accesses + * as well as the AT S1 operations. */ - hcr |= HCR_TVM | HCR_TRVM | HCR_NV1; + hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_NV1; } else { /* * For a guest hypervisor on v8.1 (VHE), allow to @@ -71,6 +72,14 @@ static void __activate_traps(struct kvm_vcpu *vcpu) hcr &= ~HCR_TVM; hcr |= vhcr_el2 & (HCR_TVM | HCR_TRVM); + + /* + * If we're using the EL1 translation regime + * (TGE clear), then ensure that AT S1 ops are + * trapped too. + */ + if (!vcpu_el2_tge_is_set(vcpu)) + hcr |= HCR_AT; } } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index f049f6f052bd..66a379baf8c7 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2464,10 +2464,236 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL2_REG(SP_EL2, NULL, reset_unknown, 0), }; +static bool handle_s1e01(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + int sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + if (vcpu_has_nv(vcpu) && forward_traps(vcpu, HCR_AT)) + return false; + + __kvm_at_s1e01(vcpu, sys_encoding, p->regval); + + return true; +} + +static bool handle_s1e2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + int sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + if (vcpu_has_nv(vcpu) && forward_nv_traps(vcpu)) + return false; + + __kvm_at_s1e2(vcpu, sys_encoding, p->regval); + + return true; +} + +static u64 setup_par_aborted(u32 esr) +{ + u64 par = 0; + + /* S [9]: fault in the stage 2 translation */ + par |= (1 << 9); + /* FST [6:1]: Fault status code */ + par |= (esr << 1); + /* F [0]: translation is aborted */ + par |= 1; + + return par; +} + +static u64 setup_par_completed(struct kvm_vcpu *vcpu, struct kvm_s2_trans *out) +{ + u64 par, vtcr_sh0; + + /* F [0]: Translation is completed successfully */ + par = 0; + /* ATTR [63:56] */ + par |= out->upper_attr; + /* PA [47:12] */ + par |= out->output & GENMASK_ULL(11, 0); + /* RES1 [11] */ + par |= (1UL << 11); + /* SH [8:7]: Shareability attribute */ + vtcr_sh0 = vcpu_read_sys_reg(vcpu, VTCR_EL2) & VTCR_EL2_SH0_MASK; + par |= (vtcr_sh0 >> VTCR_EL2_SH0_SHIFT) << 7; + + return par; +} + +static bool handle_s12(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r, bool write) +{ + u64 par, va; + u32 esr, op; + phys_addr_t ipa; + struct kvm_s2_trans out; + int ret; + + if (vcpu_has_nv(vcpu) && forward_nv_traps(vcpu)) + return false; + + /* Do the stage-1 translation */ + va = p->regval; + op = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + switch (op) { + case OP_AT_S12E1R: + op = OP_AT_S1E1R; + break; + case OP_AT_S12E1W: + op = OP_AT_S1E1W; + break; + case OP_AT_S12E0R: + op = OP_AT_S1E0R; + break; + case OP_AT_S12E0W: + op = OP_AT_S1E0W; + break; + default: + WARN_ON_ONCE(1); + return true; + } + + __kvm_at_s1e01(vcpu, op, va); + par = vcpu_read_sys_reg(vcpu, PAR_EL1); + if (par & 1) { + /* The stage-1 translation aborted */ + return true; + } + + /* Do the stage-2 translation */ + ipa = (par & GENMASK_ULL(47, 12)) | (va & GENMASK_ULL(11, 0)); + out.esr = 0; + ret = kvm_walk_nested_s2(vcpu, ipa, &out); + if (ret < 0) + return false; + + /* Check if the stage-2 PTW is aborted */ + if (out.esr) { + esr = out.esr; + goto s2_trans_abort; + } + + /* Check the access permission */ + if ((!write && !out.readable) || (write && !out.writable)) { + esr = ESR_ELx_FSC_PERM; + esr |= out.level & 0x3; + goto s2_trans_abort; + } + + vcpu_write_sys_reg(vcpu, setup_par_completed(vcpu, &out), PAR_EL1); + return true; + +s2_trans_abort: + vcpu_write_sys_reg(vcpu, setup_par_aborted(esr), PAR_EL1); + return true; +} + +static bool handle_s12r(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + return handle_s12(vcpu, p, r, false); +} + +static bool handle_s12w(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + return handle_s12(vcpu, p, r, true); +} + +/* + * AT instruction emulation + * + * We emulate AT instructions executed in the virtual EL2. + * Basic strategy for the stage-1 translation emulation is to load proper + * context, which depends on the trapped instruction and the virtual HCR_EL2, + * to the EL1 virtual memory control registers and execute S1E[01] instructions + * in EL2. See below for more detail. + * + * For the stage-2 translation, which is necessary for S12E[01] emulation, + * we walk the guest hypervisor's stage-2 page table in software. + * + * The stage-1 translation emulations can be divided into two groups depending + * on the translation regime. + * + * 1. EL2 AT instructions: S1E2x + * +-----------------------------------------------------------------------+ + * | | Setting for the emulation | + * | Virtual HCR_EL2.E2H on trap |-----------------------------------------+ + * | | Phys EL1 regs | Phys NV, NV1 | Phys TGE | + * |-----------------------------------------------------------------------| + * | 0 | vEL2 | (1, 1) | 0 | + * | 1 | vEL2 | (0, 0) | 0 | + * +-----------------------------------------------------------------------+ + * + * We emulate the EL2 AT instructions by loading virtual EL2 context + * to the EL1 virtual memory control registers and executing corresponding + * EL1 AT instructions. + * + * We set physical NV and NV1 bits to use EL2 page table format for non-VHE + * guest hypervisor (i.e. HCR_EL2.E2H == 0). As a VHE guest hypervisor uses the + * EL1 page table format, we don't set those bits. + * + * We should clear physical TGE bit not to use the EL2 translation regime when + * the host uses the VHE feature. + * + * + * 2. EL0/EL1 AT instructions: S1E[01]x, S12E1x + * +----------------------------------------------------------------------+ + * | Virtual HCR_EL2 on trap | Setting for the emulation | + * |----------------------------------------------------------------------+ + * | (vE2H, vTGE) | (vNV, vNV1) | Phys EL1 regs | Phys NV, NV1 | Phys TGE | + * |----------------------------------------------------------------------| + * | (0, 0)* | (0, 0) | vEL1 | (0, 0) | 0 | + * | (0, 0) | (1, 1) | vEL1 | (1, 1) | 0 | + * | (1, 1) | (0, 0) | vEL2 | (0, 0) | 0 | + * | (1, 1) | (1, 1) | vEL2 | (1, 1) | 0 | + * +----------------------------------------------------------------------+ + * + * *For (0, 0) in the 'Virtual HCR_EL2 on trap' column, it actually means + * (1, 1). Keep them (0, 0) just for the readability. + * + * We set physical EL1 virtual memory control registers depending on + * (vE2H, vTGE) pair. When the pair is (0, 0) where AT instructions are + * supposed to use EL0/EL1 translation regime, we load the EL1 registers with + * the virtual EL1 registers (i.e. EL1 registers from the guest hypervisor's + * point of view). When the pair is (1, 1), however, AT instructions are defined + * to apply EL2 translation regime. To emulate this behavior, we load the EL1 + * registers with the virtual EL2 context. (i.e the shadow registers) + * + * We respect the virtual NV and NV1 bit for the emulation. When those bits are + * set, it means that a guest hypervisor would like to use EL2 page table format + * for the EL1 translation regime. We emulate this by setting the physical + * NV and NV1 bits. + */ + +#define SYS_INSN(insn, access_fn) \ + { \ + SYS_DESC(OP_##insn), \ + .access = (access_fn), \ + } + static struct sys_reg_desc sys_insn_descs[] = { { SYS_DESC(SYS_DC_ISW), access_dcsw }, + + SYS_INSN(AT_S1E1R, handle_s1e01), + SYS_INSN(AT_S1E1W, handle_s1e01), + SYS_INSN(AT_S1E0R, handle_s1e01), + SYS_INSN(AT_S1E0W, handle_s1e01), + SYS_INSN(AT_S1E1RP, handle_s1e01), + SYS_INSN(AT_S1E1WP, handle_s1e01), + { SYS_DESC(SYS_DC_CSW), access_dcsw }, { SYS_DESC(SYS_DC_CISW), access_dcsw }, + + SYS_INSN(AT_S1E2R, handle_s1e2), + SYS_INSN(AT_S1E2W, handle_s1e2), + SYS_INSN(AT_S12E1R, handle_s12r), + SYS_INSN(AT_S12E1W, handle_s12w), + SYS_INSN(AT_S12E0R, handle_s12r), + SYS_INSN(AT_S12E0W, handle_s12w), }; static bool trap_dbgdidr(struct kvm_vcpu *vcpu, From patchwork Tue Jan 31 09:24:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0B70FC636CC for ; Tue, 31 Jan 2023 10:00:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Na1d5PAnAVo/f6AgYE/hFmeCOzObkCWMOolYuVINycw=; b=wozY2H0B+qL4bl 8Kz1BmdoMWaVyZniw9suzqJXViLqkP9smSW9+RR3OoZzdJBhKGa2oCOueNwbfmKsS2qFC0BeaPWa7 BX5Ryq1MskaLhdN7GAKDDskQMnBxEi6lR0BtMOcVfprV5hYye80e9pym3oY40CO1rztBB5TgXOqhr ZRvPDVFC6pGBdITFxRuRzc5xwSRPY99R1BNAFnNAhPtnXAkpkfkerKrVCyPc/tn5TmEZ3BwhLSOPb i+NLUlRa4JHcnG+lJlVTKQtpfP3RzQ4QUVfqrxwCDwlSg+XoaApktc1h9pvHmuEXx2l8e0yl7J13S qcD/MUlJ3okNEqvxSz1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnQ2-007Cuz-HN; Tue, 31 Jan 2023 09:59:15 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9G-0075Q5-W5 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:57 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 719E36148D; Tue, 31 Jan 2023 09:41:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 92DE7C433D2; Tue, 31 Jan 2023 09:41:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158113; bh=QZ9sfbCZ9IRXc9MzUR105iVsSLjIPandL0TtmB8/Q1E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F/Bb9mKojS1C11v8ENr/BTqHdPOQgKrjbtLftfMH2p5BAu1Owq9TzqAFvjcDWpwvv wNldrxVmjbAiJ46k9AaSt9n7B5BxCYMX+6hsMkF+/+zygwGef04xyzEPp2VDrQn8/Y +4prVwIdgRwV3i9H+QAD8PKff/S+KPCvoDyCYP4E76if/dGs4Zw2P8lYvfqqEPwWNy GtZ1QUAidWjEc1cC5Cf93TsmjPBDfPIbeH8/rw70wdzuHWf0kUEWm9y9behwY9pZyK fvU+c7fNK8gavXwhu7707VbMAtn9liVPv9QpdvLR3daCh2LlaVq08vjWjLXiMqXLlz DPhS7woqjlAng== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtp-0067U2-VB; Tue, 31 Jan 2023 09:25:58 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 42/69] KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2 Date: Tue, 31 Jan 2023 09:24:37 +0000 Message-Id: <20230131092504.2880505-43-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014155_210358_73796B80 X-CRM114-Status: GOOD ( 36.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim When supporting nested virtualization a guest hypervisor executing TLBI instructions must be trapped and emulated by the host hypervisor, because the guest hypervisor can only affect physical TLB entries relating to its own execution environment (virtual EL2 in EL1) but not to the nested guests as required by the semantics of the instructions and TLBI instructions might also result in updates (invalidations) to shadow page tables. This patch does several things. 1. List and define all TLBI system instructions to emulate. 2. Emulate TLBI ALLE2(IS) instruction executed in the virtual EL2. Since we emulate the virtual EL2 in the EL1, we invalidate EL1&0 regime stage 1 TLB entries with setting vttbr_el2 having the VMID of the virtual EL2. 3. Emulate TLBI VAE2* instruction executed in the virtual EL2. Based on the same principle as TLBI ALLE2 instruction, we can simply emulate those instructions by executing corresponding VAE1* instructions with the virtual EL2's VMID assigned by the host hypervisor. Note that we are able to emulate TLBI ALLE2IS precisely by only invalidating stage 1 TLB entries via TLBI VMALL1IS instruction, but to make it simeple, we reuse the existing function, __kvm_tlb_flush_vmid(), which invalidates both of stage 1 and 2 TLB entries. 4. TLBI ALLE1(IS) instruction invalidates all EL1&0 regime stage 1 and 2 TLB entries (on all PEs in the same Inner Shareable domain). To emulate these instructions, we first need to clear all the mappings in the shadow page tables since executing those instructions implies the change of mappings in the stage 2 page tables maintained by the guest hypervisor. We then need to invalidate all EL1&0 regime stage 1 and 2 TLB entries of all VMIDs, which are assigned by the host hypervisor, for this VM. 5. Based on the same principle as TLBI ALLE1(IS) emulation, we clear the mappings in the shadow stage-2 page tables and invalidate TLB entries. But this time we do it only for the current VMID from the guest hypervisor's perspective, not for all VMIDs. 6. Based on the same principle as TLBI ALLE1(IS) and TLBI VMALLS12E1(IS) emulation, we clear the mappings in the shadow stage-2 page tables and invalidate TLB entries. We do it only for one mapping for the current VMID from the guest hypervisor's view. 7. Forward system instruction traps to the virtual EL2 if a corresponding bit in the virtual HCR_EL2 is set. 8. Even though a guest hypervisor can execute TLBI instructions that are accesible at EL1 without trap, it's wrong; All those TLBI instructions work based on current VMID, and when running a guest hypervisor current VMID is the one for itself, not the one from the virtual vttbr_el2. So letting a guest hypervisor execute those TLBI instructions results in invalidating its own TLB entries and leaving invalid TLB entries unhandled. Therefore we trap and emulate those TLBI instructions. The emulation is simple; we find a shadow VMID mapped to the virtual vttbr_el2, set it in the physical vttbr_el2, then execute the same instruction in EL2. We don't set HCR_EL2.TTLB bit yet. [ Changes performed by Marc Zynger: The TLBI handling code more or less directly execute the same instruction that has been trapped (with an EL2->EL1 conversion in the case of an EL2 TLBI), but that's unfortunately not enough: - TLBIs must be upgraded to the Inner Shareable domain to account for vcpu migration, just like we already have with HCR_EL2.FB. - The DSB instruction that synchronises these must thus be on the Inner Shareable domain as well. - Prior to executing the TLBI, we need another DSB ISHST to make sure that the update to the page tables is now visible. Ordering of system instructions fixed - The current TLB invalidation code is pretty buggy, as it assume a page mapping. On the contrary, it is likely that TLB invalidation will cover more than a single page, and the size should be decided by the guests configuration (and not the host's). Since we don't cache the guest mapping sizes in the shadow PT yet, let's assume the worse case (a block mapping) and invalidate that. Take this opportunity to fix the decoding of the parameter (it isn't a straight IPA). - In general, we always emulate local TBL invalidations as being as upgraded to the Inner Shareable domain so that we can easily deal with vcpu migration. This is consistent with the fact that we set HCR_EL2.FB when running non-nested VMs. So let's emulate TLBI ALLE2 as ALLE2IS. ] [ Changes performed by Christoffer Dall: Sometimes when we are invalidating the TLB for a certain S2 MMU context, this context can also have EL2 context associated with it and we have to invalidate this too. ] Signed-off-by: Jintack Lim Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_asm.h | 2 + arch/arm64/include/asm/sysreg.h | 36 +++++ arch/arm64/kvm/hyp/vhe/switch.c | 8 +- arch/arm64/kvm/hyp/vhe/tlb.c | 81 +++++++++++ arch/arm64/kvm/mmu.c | 17 ++- arch/arm64/kvm/sys_regs.c | 227 +++++++++++++++++++++++++++++++ 6 files changed, 366 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 373558ea24d8..d73f8d5a8a25 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -226,6 +226,8 @@ extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, int level); extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); +extern void __kvm_tlb_vae2is(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding); +extern void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding); extern void __kvm_timer_set_cntvoff(u64 cntvoff); extern void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr); diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 9c718f4af944..35147f369336 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -624,6 +624,42 @@ #define OP_AT_S12E0R sys_insn(AT_Op0, 4, AT_CRn, 8, 6) #define OP_AT_S12E0W sys_insn(AT_Op0, 4, AT_CRn, 8, 7) +/* TLBI instructions */ +#define TLBI_Op0 1 +#define TLBI_Op1_EL1 0 /* Accessible from EL1 or higher */ +#define TLBI_Op1_EL2 4 /* Accessible from EL2 or higher */ +#define TLBI_CRn 8 +#define tlbi_insn_el1(CRm, Op2) sys_insn(TLBI_Op0, TLBI_Op1_EL1, TLBI_CRn, (CRm), (Op2)) +#define tlbi_insn_el2(CRm, Op2) sys_insn(TLBI_Op0, TLBI_Op1_EL2, TLBI_CRn, (CRm), (Op2)) + +#define OP_TLBI_VMALLE1IS tlbi_insn_el1(3, 0) +#define OP_TLBI_VAE1IS tlbi_insn_el1(3, 1) +#define OP_TLBI_ASIDE1IS tlbi_insn_el1(3, 2) +#define OP_TLBI_VAAE1IS tlbi_insn_el1(3, 3) +#define OP_TLBI_VALE1IS tlbi_insn_el1(3, 5) +#define OP_TLBI_VAALE1IS tlbi_insn_el1(3, 7) +#define OP_TLBI_VMALLE1 tlbi_insn_el1(7, 0) +#define OP_TLBI_VAE1 tlbi_insn_el1(7, 1) +#define OP_TLBI_ASIDE1 tlbi_insn_el1(7, 2) +#define OP_TLBI_VAAE1 tlbi_insn_el1(7, 3) +#define OP_TLBI_VALE1 tlbi_insn_el1(7, 5) +#define OP_TLBI_VAALE1 tlbi_insn_el1(7, 7) + +#define OP_TLBI_IPAS2E1IS tlbi_insn_el2(0, 1) +#define OP_TLBI_IPAS2LE1IS tlbi_insn_el2(0, 5) +#define OP_TLBI_ALLE2IS tlbi_insn_el2(3, 0) +#define OP_TLBI_VAE2IS tlbi_insn_el2(3, 1) +#define OP_TLBI_ALLE1IS tlbi_insn_el2(3, 4) +#define OP_TLBI_VALE2IS tlbi_insn_el2(3, 5) +#define OP_TLBI_VMALLS12E1IS tlbi_insn_el2(3, 6) +#define OP_TLBI_IPAS2E1 tlbi_insn_el2(4, 1) +#define OP_TLBI_IPAS2LE1 tlbi_insn_el2(4, 5) +#define OP_TLBI_ALLE2 tlbi_insn_el2(7, 0) +#define OP_TLBI_VAE2 tlbi_insn_el2(7, 1) +#define OP_TLBI_ALLE1 tlbi_insn_el2(7, 4) +#define OP_TLBI_VALE2 tlbi_insn_el2(7, 5) +#define OP_TLBI_VMALLS12E1 tlbi_insn_el2(7, 6) + /* Common SCTLR_ELx flags. */ #define SCTLR_ELx_ENTP2 (BIT(60)) #define SCTLR_ELx_DSSBS (BIT(44)) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 6d186a3e5e5f..a17bf261d501 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -47,7 +47,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) * the EL1 virtual memory control register accesses * as well as the AT S1 operations. */ - hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_NV1; + hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_TTLB | HCR_NV1; } else { /* * For a guest hypervisor on v8.1 (VHE), allow to @@ -75,11 +75,11 @@ static void __activate_traps(struct kvm_vcpu *vcpu) /* * If we're using the EL1 translation regime - * (TGE clear), then ensure that AT S1 ops are - * trapped too. + * (TGE clear), then ensure that AT S1 and + * TLBI E1 ops are trapped too. */ if (!vcpu_el2_tge_is_set(vcpu)) - hcr |= HCR_AT; + hcr |= HCR_AT | HCR_TTLB; } } diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index 24cef9b87f9e..c4389db4cc22 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -161,3 +161,84 @@ void __kvm_flush_vm_context(void) dsb(ish); } + +void __kvm_tlb_vae2is(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding) +{ + struct tlb_inv_context cxt; + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + /* + * Execute the EL1 version of TLBI VAE2* instruction, forcing + * an upgrade to the Inner Shareable domain in order to + * perform the invalidation on all CPUs. + */ + switch (sys_encoding) { + case OP_TLBI_VAE2: + case OP_TLBI_VAE2IS: + __tlbi(vae1is, va); + break; + case OP_TLBI_VALE2: + case OP_TLBI_VALE2IS: + __tlbi(vale1is, va); + break; + default: + break; + } + dsb(ish); + isb(); + + __tlb_switch_to_host(&cxt); +} + +void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding) +{ + struct tlb_inv_context cxt; + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + /* + * Execute the same instruction as the guest hypervisor did, + * expanding the scope of local TLB invalidations to the Inner + * Shareable domain so that it takes place on all CPUs. This + * is equivalent to having HCR_EL2.FB set. + */ + switch (sys_encoding) { + case OP_TLBI_VMALLE1: + case OP_TLBI_VMALLE1IS: + __tlbi(vmalle1is); + break; + case OP_TLBI_VAE1: + case OP_TLBI_VAE1IS: + __tlbi(vae1is, val); + break; + case OP_TLBI_ASIDE1: + case OP_TLBI_ASIDE1IS: + __tlbi(aside1is, val); + break; + case OP_TLBI_VAAE1: + case OP_TLBI_VAAE1IS: + __tlbi(vaae1is, val); + break; + case OP_TLBI_VALE1: + case OP_TLBI_VALE1IS: + __tlbi(vale1is, val); + break; + case OP_TLBI_VAALE1: + case OP_TLBI_VAALE1IS: + __tlbi(vaale1is, val); + break; + default: + break; + } + dsb(ish); + isb(); + + __tlb_switch_to_host(&cxt); +} diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 6dc82b0384ce..4680a703847b 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -89,7 +89,22 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot) void kvm_flush_remote_tlbs(struct kvm *kvm) { ++kvm->stat.generic.remote_tlb_flush_requests; - kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); + + if (!kvm->arch.nested_mmus) { + /* + * For a normal (i.e. non-nested) guest, flush entries for the + * given VMID. + */ + kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); + } else { + /* + * When supporting nested virtualization, we can have multiple + * VMIDs in play for each VCPU in the VM, so it's really not + * worth it to try to quiesce the system and flush all the + * VMIDs that may be in use, instead just nuke the whole thing. + */ + kvm_call_hyp(__kvm_flush_vm_context); + } } static bool kvm_is_device_pfn(unsigned long pfn) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 66a379baf8c7..40215710ede2 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2603,6 +2603,205 @@ static bool handle_s12w(struct kvm_vcpu *vcpu, struct sys_reg_params *p, return handle_s12(vcpu, p, r, true); } +static bool handle_alle2is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (vcpu_has_nv(vcpu) && forward_nv_traps(vcpu)) + return false; + + /* + * To emulate invalidating all EL2 regime stage 1 TLB entries for all + * PEs, executing TLBI VMALLE1IS is enough. But reuse the existing + * interface for the simplicity; invalidating stage 2 entries doesn't + * affect the correctness. + */ + __kvm_tlb_flush_vmid(&vcpu->kvm->arch.mmu); + return true; +} + +static bool handle_vae2is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + int sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + if (vcpu_has_nv(vcpu) && forward_nv_traps(vcpu)) + return false; + + /* + * Based on the same principle as TLBI ALLE2 instruction + * emulation, we emulate TLBI VAE2* instructions by executing + * corresponding TLBI VAE1* instructions with the virtual + * EL2's VMID assigned by the host hypervisor. + */ + __kvm_tlb_vae2is(&vcpu->kvm->arch.mmu, p->regval, sys_encoding); + return true; +} + +static bool handle_alle1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct kvm_s2_mmu *mmu = &vcpu->kvm->arch.mmu; + + if (vcpu_has_nv(vcpu) && forward_nv_traps(vcpu)) + return false; + + write_lock(&vcpu->kvm->mmu_lock); + + /* + * Clear all mappings in the shadow page tables and invalidate the stage + * 1 and 2 TLB entries via kvm_tlb_flush_vmid_ipa(). + */ + kvm_nested_s2_unmap(vcpu->kvm); + + if (atomic64_read(&mmu->vmid.id)) { + /* + * Invalidate the stage 1 and 2 TLB entries for the host OS + * in a VM only if there is one. + */ + __kvm_tlb_flush_vmid(mmu); + } + + write_unlock(&vcpu->kvm->mmu_lock); + + return true; +} + +static bool handle_vmalls12e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct kvm_s2_mmu *mmu; + u64 vttbr; + + if (vcpu_has_nv(vcpu) && forward_nv_traps(vcpu)) + return false; + + vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + write_lock(&vcpu->kvm->mmu_lock); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, HCR_VM); + if (mmu) + kvm_unmap_stage2_range(mmu, 0, kvm_phys_size(vcpu->kvm)); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, 0); + if (mmu) + kvm_unmap_stage2_range(mmu, 0, kvm_phys_size(vcpu->kvm)); + + write_unlock(&vcpu->kvm->mmu_lock); + + return true; +} + +static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + struct kvm_s2_mmu *mmu; + u64 base_addr; + int max_size; + + if (vcpu_has_nv(vcpu) && forward_nv_traps(vcpu)) + return false; + + /* + * We drop a number of things from the supplied value: + * + * - NS bit: we're non-secure only. + * + * - TTL field: We already have the granule size from the + * VTCR_EL2.TG0 field, and the level is only relevant to the + * guest's S2PT. + * + * - IPA[51:48]: We don't support 52bit IPA just yet... + * + * And of course, adjust the IPA to be on an actual address. + */ + base_addr = (p->regval & GENMASK_ULL(35, 0)) << 12; + + /* Compute the maximum extent of the invalidation */ + switch ((vtcr & VTCR_EL2_TG0_MASK)) { + case VTCR_EL2_TG0_4K: + max_size = SZ_1G; + break; + case VTCR_EL2_TG0_16K: + max_size = SZ_32M; + break; + case VTCR_EL2_TG0_64K: + /* + * No, we do not support 52bit IPA in nested yet. Once + * we do, this should be 4TB. + */ + /* FIXME: remove the 52bit PA support from the IDregs */ + max_size = SZ_512M; + break; + default: + BUG(); + } + + write_lock(&vcpu->kvm->mmu_lock); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, HCR_VM); + if (mmu) + kvm_unmap_stage2_range(mmu, base_addr, max_size); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, 0); + if (mmu) + kvm_unmap_stage2_range(mmu, base_addr, max_size); + + write_unlock(&vcpu->kvm->mmu_lock); + + return true; +} + +static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + u32 sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + if (vcpu_has_nv(vcpu) && forward_traps(vcpu, HCR_TTLB)) + return false; + + /* + * If we're here, this is because we've trapped on a EL1 TLBI + * instruction that affects the EL1 translation regime while + * we're running in a context that doesn't allow us to let the + * HW do its thing (aka vEL2): + * + * - HCR_EL2.E2H == 0 : a non-VHE guest + * - HCR_EL2.{E2H,TGE} == { 1, 0 } : a VHE guest in guest mode + * + * We don't expect these helpers to ever be called when running + * in a vEL1 context. + */ + + WARN_ON(!vcpu_is_el2(vcpu)); + + mutex_lock(&vcpu->kvm->lock); + + if ((__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_E2H | HCR_TGE)) != (HCR_E2H | HCR_TGE)) { + u64 virtual_vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + struct kvm_s2_mmu *mmu; + + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, HCR_VM); + if (mmu) + __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); + + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, 0); + if (mmu) + __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); + } else { + /* + * ARMv8.4-NV allows the guest to change TGE behind + * our back, so we always trap EL1 TLBIs from vEL2... + */ + __kvm_tlb_el1_instr(&vcpu->kvm->arch.mmu, p->regval, sys_encoding); + } + + mutex_unlock(&vcpu->kvm->lock); + + return true; +} + /* * AT instruction emulation * @@ -2688,12 +2887,40 @@ static struct sys_reg_desc sys_insn_descs[] = { { SYS_DESC(SYS_DC_CSW), access_dcsw }, { SYS_DESC(SYS_DC_CISW), access_dcsw }, + SYS_INSN(TLBI_VMALLE1IS, handle_tlbi_el1), + SYS_INSN(TLBI_VAE1IS, handle_tlbi_el1), + SYS_INSN(TLBI_ASIDE1IS, handle_tlbi_el1), + SYS_INSN(TLBI_VAAE1IS, handle_tlbi_el1), + SYS_INSN(TLBI_VALE1IS, handle_tlbi_el1), + SYS_INSN(TLBI_VAALE1IS, handle_tlbi_el1), + SYS_INSN(TLBI_VMALLE1, handle_tlbi_el1), + SYS_INSN(TLBI_VAE1, handle_tlbi_el1), + SYS_INSN(TLBI_ASIDE1, handle_tlbi_el1), + SYS_INSN(TLBI_VAAE1, handle_tlbi_el1), + SYS_INSN(TLBI_VALE1, handle_tlbi_el1), + SYS_INSN(TLBI_VAALE1, handle_tlbi_el1), + SYS_INSN(AT_S1E2R, handle_s1e2), SYS_INSN(AT_S1E2W, handle_s1e2), SYS_INSN(AT_S12E1R, handle_s12r), SYS_INSN(AT_S12E1W, handle_s12w), SYS_INSN(AT_S12E0R, handle_s12r), SYS_INSN(AT_S12E0W, handle_s12w), + + SYS_INSN(TLBI_IPAS2E1IS, handle_ipas2e1is), + SYS_INSN(TLBI_IPAS2LE1IS, handle_ipas2e1is), + SYS_INSN(TLBI_ALLE2IS, handle_alle2is), + SYS_INSN(TLBI_VAE2IS, handle_vae2is), + SYS_INSN(TLBI_ALLE1IS, handle_alle1is), + SYS_INSN(TLBI_VALE2IS, handle_vae2is), + SYS_INSN(TLBI_VMALLS12E1IS, handle_vmalls12e1is), + SYS_INSN(TLBI_IPAS2E1, handle_ipas2e1is), + SYS_INSN(TLBI_IPAS2LE1, handle_ipas2e1is), + SYS_INSN(TLBI_ALLE2, handle_alle2is), + SYS_INSN(TLBI_VAE2, handle_vae2is), + SYS_INSN(TLBI_ALLE1, handle_alle1is), + SYS_INSN(TLBI_VALE2, handle_vae2is), + SYS_INSN(TLBI_VMALLS12E1, handle_vmalls12e1is), }; static bool trap_dbgdidr(struct kvm_vcpu *vcpu, From patchwork Tue Jan 31 09:24:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 86DB8C636D3 for ; Tue, 31 Jan 2023 09:44:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2WK+3Bu1AhCF7MyAVBQx8w0XNRE4lKgsT9osYd3zVyE=; b=MzdvOkfvkBgiia 2oOR1wbEUWPSNsVH1QOazkFJ9EdZAq0JkPvuYHEnxc5jleAioYwJJgmJlVniZcAmwA0ZTXv87Jyuf RrqfaFrvlaQF2tyHA2idagp0Ee5K5NCWmO42Rd5CnSUGxgAfKwIG2LtJSVmKmsTEPfv4+DZRQc6JD S5SCpGS87rrXgWsezu9l6xoT50am5pxVqrHmYpAtmhBNaYDflvUWyxsA5Q1ZqaKeN6cX91oD2sTjc 7QwHHjwawML1Qm31krIRJVvJiscLnPpub3l/FedUDBgpomPbaXCVs55dYSFJ/y7fzQGfnV6jjLMoh VGx9MhI4XCqV3UFXMDCg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnAj-0076BT-PF; Tue, 31 Jan 2023 09:43:26 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8F-0074tR-FO for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:40:53 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D8DBD61445; Tue, 31 Jan 2023 09:40:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4991DC433D2; Tue, 31 Jan 2023 09:40:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158050; bh=BqWGGG6P0Q1EZNUaobu4OneFEfRFSI4gFWADamGwyEI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vLz3gBvtu5uXbisjJp6/wBgoiYRrDUuso9bZ0UtYCXKd9o2MOzCjVUuz9PTqgq59O VTiPCw2Ga4Wh3WIQc8qcbq0EIBxs3ZmbsxpgBO4Mh4hi/t2JlF92Em0R8ViUZefhXk BYQiqZU5pwKVnaNPg9XbXMsar1M4idPjDAZIFFF9m741tHb7bUFKGYGKhQD2fYYFdN Uh9XnynOTP5tMmqME+dBVjJkB3OrRZ6NVYmiaXESLeX8y08g5cQiRn9PUwFan8Swxd 296sGxOtA+QoUPcIG3mWqSI4bu6qaejbtsF+BNm32TKzFfk/Xc67H6g7v7h3LfMUQA svOBA56JK7X+Q== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtq-0067U2-7d; Tue, 31 Jan 2023 09:25:58 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 43/69] KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's Date: Tue, 31 Jan 2023 09:24:38 +0000 Message-Id: <20230131092504.2880505-44-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014051_612082_40BD2B2A X-CRM114-Status: GOOD ( 13.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When entering a L2 guest (nested virt enabled, but not in hypervisor context), we need to honor the traps the L1 guest has asked enabled. For now, just OR the guest's HCR_EL2 into the host's. We may have to do some filtering in the future though. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/vhe/switch.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index a17bf261d501..ff77b37b6f45 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -81,6 +81,11 @@ static void __activate_traps(struct kvm_vcpu *vcpu) if (!vcpu_el2_tge_is_set(vcpu)) hcr |= HCR_AT | HCR_TTLB; } + } else if (vcpu_has_nv(vcpu)) { + u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + + vhcr_el2 &= ~HCR_GUEST_NV_FILTER_FLAGS; + hcr |= vhcr_el2; } ___activate_traps(vcpu, hcr); From patchwork Tue Jan 31 09:24:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122776 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2EA32C38142 for ; Tue, 31 Jan 2023 10:07:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=K6dFQImP+mw+8dv4WQEX9tN0izKuYMZYfizZnPAld+k=; b=Pcf66uYaYENlJP PaKKFChuBmZ9a4K25FRuDlNKR9em+t+dRyHwX9q90TTA3iT0BkKny+xc6U5n08AuvXcrmyIlOhztz 6fx3ev6jTHsdcob0xIPYW7aNkFKInkx03UKFKt+LqOdilAzClXKWSy1mWz9CK66EfVOs9W5I0QQgX iNNdmifOlCgDqnmgv+eXXHa3wIfITZCzqtdMb1bznekfbzgL6sTwCirEvhdaIy5i+ukferFFl13dM FOQcpUccH82HkZp+7koyY85LEc4S3BT/0cacVhMVnszTO9j9nDpAke+Gg9mwcft6sxdGx/JM5KUts qIGxqkcy+TI8KEQsEm1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnWG-007FaP-Mc; Tue, 31 Jan 2023 10:05:41 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9X-0075XY-6E for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:14 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B570E61485; Tue, 31 Jan 2023 09:42:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D3836C4339B; Tue, 31 Jan 2023 09:42:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158130; bh=bZsmY6UAcvugWrDjX7Ym/lqZ6zS77NYQrzzwfsI+gtU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kxUs8Uq2Wf/yUaRFFSUV8B04pGHaJCOuk9lvoLiG6XiruUMLDQp74JosqWv2iJlhQ YSUuoO6Cav60qLxlY3WEz3YcXBlIKpm3Nhz/3LZh7JXQKZTvoiqgzl78FJsIAE/ZqC GCVmMmkgn4dH4iMmF3zizXCapdtURLB3ACMK/JOtC7Hf84vXTvhKDVg7Bt9ZOpESPO m09uw6jy4Ktvvr/rfwqkKXw1CHA1sq7auEufi935RBNO62sdZOqLSlY3CfNfoGUUFI vtwenc76DuOon0Jxzn8Ii0KL4xVqrYTplNX6RZxw0AFczySH8EZmgyrup0gND9TxJs /MEBT+Czk013g== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtq-0067U2-GD; Tue, 31 Jan 2023 09:25:58 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 44/69] KVM: arm64: nv: arch_timer: Support hyp timer emulation Date: Tue, 31 Jan 2023 09:24:39 +0000 Message-Id: <20230131092504.2880505-45-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014211_349647_23032605 X-CRM114-Status: GOOD ( 28.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Emulating EL2 also means emulating the EL2 timers. To do so, we expand our timer framework to deal with at most 4 timers. At any given time, two timers are using the HW timers, and the two others are purely emulated. The role of deciding which is which at any given time is left to a mapping function which is called every time we need to make such a decision. Co-developed-by: Christoffer Dall Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 4 + arch/arm64/include/asm/sysreg.h | 2 + arch/arm64/kvm/arch_timer.c | 186 ++++++++++++++++++++++++++++-- arch/arm64/kvm/sys_regs.c | 49 +++++++- arch/arm64/kvm/trace_arm.h | 6 +- arch/arm64/kvm/vgic/vgic.c | 15 +++ include/kvm/arm_arch_timer.h | 8 +- include/kvm/arm_vgic.h | 1 + 8 files changed, 257 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index a348c93b1094..a486aef61b3f 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -397,6 +397,10 @@ enum vcpu_sysreg { TPIDR_EL2, /* EL2 Software Thread ID Register */ CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */ SP_EL2, /* EL2 Stack Pointer */ + CNTHP_CTL_EL2, + CNTHP_CVAL_EL2, + CNTHV_CTL_EL2, + CNTHV_CVAL_EL2, NR_SYS_REGS /* Nothing after this line! */ }; diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 35147f369336..0457170760d8 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -467,6 +467,8 @@ #define SYS_CNTFRQ_EL0 sys_reg(3, 3, 14, 0, 0) +#define SYS_CNTPCT_EL0 sys_reg(3, 3, 14, 0, 1) +#define SYS_CNTVCT_EL0 sys_reg(3, 3, 14, 0, 2) #define SYS_CNTPCTSS_EL0 sys_reg(3, 3, 14, 0, 5) #define SYS_CNTVCTSS_EL0 sys_reg(3, 3, 14, 0, 6) diff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.c index 00610477ec7b..a8425a88c750 100644 --- a/arch/arm64/kvm/arch_timer.c +++ b/arch/arm64/kvm/arch_timer.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -40,6 +41,16 @@ static const struct kvm_irq_level default_vtimer_irq = { .level = 1, }; +static const struct kvm_irq_level default_hptimer_irq = { + .irq = 26, + .level = 1, +}; + +static const struct kvm_irq_level default_hvtimer_irq = { + .irq = 28, + .level = 1, +}; + static bool kvm_timer_irq_can_fire(struct arch_timer_context *timer_ctx); static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, struct arch_timer_context *timer_ctx); @@ -51,6 +62,11 @@ static void kvm_arm_timer_write(struct kvm_vcpu *vcpu, static u64 kvm_arm_timer_read(struct kvm_vcpu *vcpu, struct arch_timer_context *timer, enum kvm_arch_timer_regs treg); +static bool kvm_arch_timer_get_input_level(int vintid); + +static struct irq_ops arch_timer_irq_ops = { + .get_input_level = kvm_arch_timer_get_input_level, +}; u32 timer_get_ctl(struct arch_timer_context *ctxt) { @@ -61,6 +77,10 @@ u32 timer_get_ctl(struct arch_timer_context *ctxt) return __vcpu_sys_reg(vcpu, CNTV_CTL_EL0); case TIMER_PTIMER: return __vcpu_sys_reg(vcpu, CNTP_CTL_EL0); + case TIMER_HVTIMER: + return __vcpu_sys_reg(vcpu, CNTHV_CTL_EL2); + case TIMER_HPTIMER: + return __vcpu_sys_reg(vcpu, CNTHP_CTL_EL2); default: WARN_ON(1); return 0; @@ -76,6 +96,10 @@ u64 timer_get_cval(struct arch_timer_context *ctxt) return __vcpu_sys_reg(vcpu, CNTV_CVAL_EL0); case TIMER_PTIMER: return __vcpu_sys_reg(vcpu, CNTP_CVAL_EL0); + case TIMER_HVTIMER: + return __vcpu_sys_reg(vcpu, CNTHV_CVAL_EL2); + case TIMER_HPTIMER: + return __vcpu_sys_reg(vcpu, CNTHP_CVAL_EL2); default: WARN_ON(1); return 0; @@ -105,6 +129,12 @@ static void timer_set_ctl(struct arch_timer_context *ctxt, u32 ctl) case TIMER_PTIMER: __vcpu_sys_reg(vcpu, CNTP_CTL_EL0) = ctl; break; + case TIMER_HVTIMER: + __vcpu_sys_reg(vcpu, CNTHV_CTL_EL2) = ctl; + break; + case TIMER_HPTIMER: + __vcpu_sys_reg(vcpu, CNTHP_CTL_EL2) = ctl; + break; default: WARN_ON(1); } @@ -121,6 +151,12 @@ static void timer_set_cval(struct arch_timer_context *ctxt, u64 cval) case TIMER_PTIMER: __vcpu_sys_reg(vcpu, CNTP_CVAL_EL0) = cval; break; + case TIMER_HVTIMER: + __vcpu_sys_reg(vcpu, CNTHV_CVAL_EL2) = cval; + break; + case TIMER_HPTIMER: + __vcpu_sys_reg(vcpu, CNTHP_CVAL_EL2) = cval; + break; default: WARN_ON(1); } @@ -146,13 +182,27 @@ u64 kvm_phys_timer_read(void) static void get_timer_map(struct kvm_vcpu *vcpu, struct timer_map *map) { - if (has_vhe()) { + if (vcpu_has_nv(vcpu)) { + if (is_hyp_ctxt(vcpu)) { + map->direct_vtimer = vcpu_hvtimer(vcpu); + map->direct_ptimer = vcpu_hptimer(vcpu); + map->emul_vtimer = vcpu_vtimer(vcpu); + map->emul_ptimer = vcpu_ptimer(vcpu); + } else { + map->direct_vtimer = vcpu_vtimer(vcpu); + map->direct_ptimer = vcpu_ptimer(vcpu); + map->emul_vtimer = vcpu_hvtimer(vcpu); + map->emul_ptimer = vcpu_hptimer(vcpu); + } + } else if (has_vhe()) { map->direct_vtimer = vcpu_vtimer(vcpu); map->direct_ptimer = vcpu_ptimer(vcpu); + map->emul_vtimer = NULL; map->emul_ptimer = NULL; } else { map->direct_vtimer = vcpu_vtimer(vcpu); map->direct_ptimer = NULL; + map->emul_vtimer = NULL; map->emul_ptimer = vcpu_ptimer(vcpu); } @@ -345,9 +395,11 @@ static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx) switch (index) { case TIMER_VTIMER: + case TIMER_HVTIMER: cnt_ctl = read_sysreg_el0(SYS_CNTV_CTL); break; case TIMER_PTIMER: + case TIMER_HPTIMER: cnt_ctl = read_sysreg_el0(SYS_CNTP_CTL); break; case NR_KVM_TIMERS: @@ -455,6 +507,7 @@ static void timer_save_state(struct arch_timer_context *ctx) switch (index) { case TIMER_VTIMER: + case TIMER_HVTIMER: timer_set_ctl(ctx, read_sysreg_el0(SYS_CNTV_CTL)); timer_set_cval(ctx, read_sysreg_el0(SYS_CNTV_CVAL)); @@ -480,6 +533,7 @@ static void timer_save_state(struct arch_timer_context *ctx) set_cntvoff(0); break; case TIMER_PTIMER: + case TIMER_HPTIMER: timer_set_ctl(ctx, read_sysreg_el0(SYS_CNTP_CTL)); timer_set_cval(ctx, read_sysreg_el0(SYS_CNTP_CVAL)); @@ -517,6 +571,7 @@ static void kvm_timer_blocking(struct kvm_vcpu *vcpu) */ if (!kvm_timer_irq_can_fire(map.direct_vtimer) && !kvm_timer_irq_can_fire(map.direct_ptimer) && + !kvm_timer_irq_can_fire(map.emul_vtimer) && !kvm_timer_irq_can_fire(map.emul_ptimer) && !vcpu_has_wfit_active(vcpu)) return; @@ -552,11 +607,14 @@ static void timer_restore_state(struct arch_timer_context *ctx) switch (index) { case TIMER_VTIMER: set_cntvoff(timer_get_offset(ctx)); + fallthrough; + case TIMER_HVTIMER: write_sysreg_el0(timer_get_cval(ctx), SYS_CNTV_CVAL); isb(); write_sysreg_el0(timer_get_ctl(ctx), SYS_CNTV_CTL); break; case TIMER_PTIMER: + case TIMER_HPTIMER: write_sysreg_el0(timer_get_cval(ctx), SYS_CNTP_CVAL); isb(); write_sysreg_el0(timer_get_ctl(ctx), SYS_CNTP_CTL); @@ -628,6 +686,59 @@ static void kvm_timer_vcpu_load_nogic(struct kvm_vcpu *vcpu) enable_percpu_irq(host_vtimer_irq, host_vtimer_irq_flags); } +static void kvm_timer_vcpu_load_nested_switch(struct kvm_vcpu *vcpu, + struct timer_map *map) +{ + int hw, ret; + + if (!irqchip_in_kernel(vcpu->kvm)) + return; + + /* + * We only ever unmap the vtimer irq on a VHE system that runs nested + * virtualization, in which case we have both a valid emul_vtimer, + * emul_ptimer, direct_vtimer, and direct_ptimer. + * + * Since this is called from kvm_timer_vcpu_load(), a change between + * vEL2 and vEL1/0 will have just happened, and the timer_map will + * represent this, and therefore we switch the emul/direct mappings + * below. + */ + hw = kvm_vgic_get_map(vcpu, map->direct_vtimer->irq.irq); + if (hw < 0) { + kvm_vgic_unmap_phys_irq(vcpu, map->emul_vtimer->irq.irq); + kvm_vgic_unmap_phys_irq(vcpu, map->emul_ptimer->irq.irq); + + ret = kvm_vgic_map_phys_irq(vcpu, + map->direct_vtimer->host_timer_irq, + map->direct_vtimer->irq.irq, + &arch_timer_irq_ops); + WARN_ON_ONCE(ret); + ret = kvm_vgic_map_phys_irq(vcpu, + map->direct_ptimer->host_timer_irq, + map->direct_ptimer->irq.irq, + &arch_timer_irq_ops); + WARN_ON_ONCE(ret); + } + + /* + * Apply the trapping bits that the guest hypervisor has + * requested for its own guest. + */ + if (!is_hyp_ctxt(vcpu)) { + u64 val = __vcpu_sys_reg(vcpu, CNTHCTL_EL2); + + if (vcpu_el2_e2h_is_set(vcpu)) + val &= (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN) << 10; + else + val = (val & (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN)) << 10; + + sysreg_clear_set(cntkctl_el1, + (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN) << 10, + val); + } +} + void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) { struct arch_timer_cpu *timer = vcpu_timer(vcpu); @@ -639,6 +750,9 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) get_timer_map(vcpu, &map); if (static_branch_likely(&has_gic_active_state)) { + if (vcpu_has_nv(vcpu)) + kvm_timer_vcpu_load_nested_switch(vcpu, &map); + kvm_timer_vcpu_load_gic(map.direct_vtimer); if (map.direct_ptimer) kvm_timer_vcpu_load_gic(map.direct_ptimer); @@ -652,6 +766,8 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) if (map.direct_ptimer) timer_restore_state(map.direct_ptimer); + if (map.emul_vtimer) + timer_emulate(map.emul_vtimer); if (map.emul_ptimer) timer_emulate(map.emul_ptimer); } @@ -696,6 +812,8 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu) * In any case, we re-schedule the hrtimer for the physical timer when * coming back to the VCPU thread in kvm_timer_vcpu_load(). */ + if (map.emul_vtimer) + soft_timer_cancel(&map.emul_vtimer->hrtimer); if (map.emul_ptimer) soft_timer_cancel(&map.emul_ptimer->hrtimer); @@ -747,10 +865,14 @@ int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu) */ timer_set_ctl(vcpu_vtimer(vcpu), 0); timer_set_ctl(vcpu_ptimer(vcpu), 0); + timer_set_ctl(vcpu_hvtimer(vcpu), 0); + timer_set_ctl(vcpu_hptimer(vcpu), 0); if (timer->enabled) { kvm_timer_update_irq(vcpu, false, vcpu_vtimer(vcpu)); kvm_timer_update_irq(vcpu, false, vcpu_ptimer(vcpu)); + kvm_timer_update_irq(vcpu, false, vcpu_hvtimer(vcpu)); + kvm_timer_update_irq(vcpu, false, vcpu_hptimer(vcpu)); if (irqchip_in_kernel(vcpu->kvm)) { kvm_vgic_reset_mapped_irq(vcpu, map.direct_vtimer->irq.irq); @@ -759,6 +881,8 @@ int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu) } } + if (map.emul_vtimer) + soft_timer_cancel(&map.emul_vtimer->hrtimer); if (map.emul_ptimer) soft_timer_cancel(&map.emul_ptimer->hrtimer); @@ -789,30 +913,47 @@ void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu) struct arch_timer_cpu *timer = vcpu_timer(vcpu); struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); struct arch_timer_context *ptimer = vcpu_ptimer(vcpu); + struct arch_timer_context *hvtimer = vcpu_hvtimer(vcpu); + struct arch_timer_context *hptimer = vcpu_hptimer(vcpu); vtimer->vcpu = vcpu; ptimer->vcpu = vcpu; + hvtimer->vcpu = vcpu; + hptimer->vcpu = vcpu; /* Synchronize cntvoff across all vtimers of a VM. */ update_vtimer_cntvoff(vcpu, kvm_phys_timer_read()); timer_set_offset(ptimer, 0); + timer_set_offset(hvtimer, 0); + timer_set_offset(hptimer, 0); hrtimer_init(&timer->bg_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); timer->bg_timer.function = kvm_bg_timer_expire; hrtimer_init(&vtimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); hrtimer_init(&ptimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); + hrtimer_init(&hvtimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); + hrtimer_init(&hptimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); + vtimer->hrtimer.function = kvm_hrtimer_expire; ptimer->hrtimer.function = kvm_hrtimer_expire; + hvtimer->hrtimer.function = kvm_hrtimer_expire; + hptimer->hrtimer.function = kvm_hrtimer_expire; vtimer->irq.irq = default_vtimer_irq.irq; ptimer->irq.irq = default_ptimer_irq.irq; + hvtimer->irq.irq = default_hvtimer_irq.irq; + hptimer->irq.irq = default_hptimer_irq.irq; vtimer->host_timer_irq = host_vtimer_irq; ptimer->host_timer_irq = host_ptimer_irq; + hvtimer->host_timer_irq = host_vtimer_irq; + hptimer->host_timer_irq = host_ptimer_irq; vtimer->host_timer_irq_flags = host_vtimer_irq_flags; ptimer->host_timer_irq_flags = host_ptimer_irq_flags; + hvtimer->host_timer_irq_flags = host_vtimer_irq_flags; + hptimer->host_timer_irq_flags = host_ptimer_irq_flags; } void kvm_timer_cpu_up(void) @@ -927,6 +1068,10 @@ static u64 kvm_arm_timer_read(struct kvm_vcpu *vcpu, val = kvm_phys_timer_read() - timer_get_offset(timer); break; + case TIMER_REG_VOFF: + val = timer_get_offset(timer); + break; + default: BUG(); } @@ -945,7 +1090,7 @@ u64 kvm_arm_timer_read_sysreg(struct kvm_vcpu *vcpu, get_timer_map(vcpu, &map); timer = vcpu_get_timer(vcpu, tmr); - if (timer == map.emul_ptimer) + if (timer == map.emul_vtimer || timer == map.emul_ptimer) return kvm_arm_timer_read(vcpu, timer, treg); preempt_disable(); @@ -977,6 +1122,10 @@ static void kvm_arm_timer_write(struct kvm_vcpu *vcpu, timer_set_cval(timer, val); break; + case TIMER_REG_VOFF: + timer_set_offset(timer, val); + break; + default: BUG(); } @@ -992,7 +1141,7 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, get_timer_map(vcpu, &map); timer = vcpu_get_timer(vcpu, tmr); - if (timer == map.emul_ptimer) { + if (timer == map.emul_vtimer || timer == map.emul_ptimer) { soft_timer_cancel(&timer->hrtimer); kvm_arm_timer_write(vcpu, timer, treg, val); timer_emulate(timer); @@ -1072,10 +1221,6 @@ static const struct irq_domain_ops timer_domain_ops = { .free = timer_irq_domain_free, }; -static struct irq_ops arch_timer_irq_ops = { - .get_input_level = kvm_arch_timer_get_input_level, -}; - static void kvm_irq_fixup_flags(unsigned int virq, u32 *flags) { *flags = irq_get_trigger_type(virq); @@ -1217,7 +1362,7 @@ void kvm_timer_vcpu_terminate(struct kvm_vcpu *vcpu) static bool timer_irqs_are_valid(struct kvm_vcpu *vcpu) { - int vtimer_irq, ptimer_irq, ret; + int vtimer_irq, ptimer_irq, hvtimer_irq, hptimer_irq, ret; unsigned long i; vtimer_irq = vcpu_vtimer(vcpu)->irq.irq; @@ -1230,16 +1375,28 @@ static bool timer_irqs_are_valid(struct kvm_vcpu *vcpu) if (ret) return false; + hvtimer_irq = vcpu_hvtimer(vcpu)->irq.irq; + ret = kvm_vgic_set_owner(vcpu, hvtimer_irq, vcpu_hvtimer(vcpu)); + if (ret) + return false; + + hptimer_irq = vcpu_hptimer(vcpu)->irq.irq; + ret = kvm_vgic_set_owner(vcpu, hptimer_irq, vcpu_hptimer(vcpu)); + if (ret) + return false; + kvm_for_each_vcpu(i, vcpu, vcpu->kvm) { if (vcpu_vtimer(vcpu)->irq.irq != vtimer_irq || - vcpu_ptimer(vcpu)->irq.irq != ptimer_irq) + vcpu_ptimer(vcpu)->irq.irq != ptimer_irq || + vcpu_hvtimer(vcpu)->irq.irq != hvtimer_irq || + vcpu_hptimer(vcpu)->irq.irq != hptimer_irq) return false; } return true; } -bool kvm_arch_timer_get_input_level(int vintid) +static bool kvm_arch_timer_get_input_level(int vintid) { struct kvm_vcpu *vcpu = kvm_get_running_vcpu(); struct arch_timer_context *timer; @@ -1251,6 +1408,10 @@ bool kvm_arch_timer_get_input_level(int vintid) timer = vcpu_vtimer(vcpu); else if (vintid == vcpu_ptimer(vcpu)->irq.irq) timer = vcpu_ptimer(vcpu); + else if (vintid == vcpu_hvtimer(vcpu)->irq.irq) + timer = vcpu_hvtimer(vcpu); + else if (vintid == vcpu_hptimer(vcpu)->irq.irq) + timer = vcpu_hptimer(vcpu); else BUG(); @@ -1333,6 +1494,7 @@ static void set_timer_irqs(struct kvm *kvm, int vtimer_irq, int ptimer_irq) kvm_for_each_vcpu(i, vcpu, kvm) { vcpu_vtimer(vcpu)->irq.irq = vtimer_irq; vcpu_ptimer(vcpu)->irq.irq = ptimer_irq; + /* TODO: Add support for hv/hp timers */ } } @@ -1343,6 +1505,8 @@ int kvm_arm_timer_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr) struct arch_timer_context *ptimer = vcpu_ptimer(vcpu); int irq; + /* TODO: Add support for hv/hp timers */ + if (!irqchip_in_kernel(vcpu->kvm)) return -EINVAL; @@ -1375,6 +1539,8 @@ int kvm_arm_timer_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr) struct arch_timer_context *timer; int irq; + /* TODO: Add support for hv/hp timers */ + switch (attr->attr) { case KVM_ARM_VCPU_TIMER_IRQ_VTIMER: timer = vcpu_vtimer(vcpu); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 40215710ede2..a58235dec296 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1363,12 +1363,57 @@ static bool access_arch_timer(struct kvm_vcpu *vcpu, tmr = TIMER_PTIMER; treg = TIMER_REG_CVAL; break; + case SYS_CNTVOFF_EL2: + tmr = TIMER_VTIMER; + treg = TIMER_REG_VOFF; + break; + + case SYS_CNTPCT_EL0: + case SYS_CNTPCTSS_EL0: + if (is_hyp_ctxt(vcpu)) + tmr = TIMER_HPTIMER; + else + tmr = TIMER_PTIMER; + treg = TIMER_REG_CNT; + break; + default: print_sys_reg_msg(p, "%s", "Unhandled trapped timer register"); kvm_inject_undefined(vcpu); return false; } + if (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu) && tmr == TIMER_PTIMER) { + u64 val = __vcpu_sys_reg(vcpu, CNTHCTL_EL2); + bool trap; + + if (vcpu_el2_e2h_is_set(vcpu)) + val &= (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN) << 10; + else + val = (val & (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN)) << 10; + + switch (treg) { + case TIMER_REG_CVAL: + case TIMER_REG_CTL: + case TIMER_REG_TVAL: + trap = val & (CNTHCTL_EL1PCEN << 10); + break; + + case TIMER_REG_CNT: + trap = val & (CNTHCTL_EL1PCTEN << 10); + break; + + default: + trap = false; + break; + } + + if (trap) { + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + return false; + } + } + if (p->is_write) kvm_arm_timer_write_sysreg(vcpu, tmr, treg, p->regval); else @@ -2325,6 +2370,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { AMU_AMEVTYPER1_EL0(14), AMU_AMEVTYPER1_EL0(15), + { SYS_DESC(SYS_CNTPCT_EL0), access_arch_timer }, + { SYS_DESC(SYS_CNTPCTSS_EL0), access_arch_timer }, { SYS_DESC(SYS_CNTP_TVAL_EL0), access_arch_timer }, { SYS_DESC(SYS_CNTP_CTL_EL0), access_arch_timer }, { SYS_DESC(SYS_CNTP_CVAL_EL0), access_arch_timer }, @@ -2441,7 +2488,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL2_REG(CONTEXTIDR_EL2, access_rw, reset_val, 0), EL2_REG(TPIDR_EL2, access_rw, reset_val, 0), - EL2_REG(CNTVOFF_EL2, access_rw, reset_val, 0), + { SYS_DESC(SYS_CNTVOFF_EL2), access_arch_timer }, EL2_REG(CNTHCTL_EL2, access_rw, reset_val, 0), EL12_REG(SCTLR, access_vm_reg, reset_val, 0x00C50078), diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h index f3e46a976125..6ce5c025218d 100644 --- a/arch/arm64/kvm/trace_arm.h +++ b/arch/arm64/kvm/trace_arm.h @@ -206,6 +206,7 @@ TRACE_EVENT(kvm_get_timer_map, __field( unsigned long, vcpu_id ) __field( int, direct_vtimer ) __field( int, direct_ptimer ) + __field( int, emul_vtimer ) __field( int, emul_ptimer ) ), @@ -214,14 +215,17 @@ TRACE_EVENT(kvm_get_timer_map, __entry->direct_vtimer = arch_timer_ctx_index(map->direct_vtimer); __entry->direct_ptimer = (map->direct_ptimer) ? arch_timer_ctx_index(map->direct_ptimer) : -1; + __entry->emul_vtimer = + (map->emul_vtimer) ? arch_timer_ctx_index(map->emul_vtimer) : -1; __entry->emul_ptimer = (map->emul_ptimer) ? arch_timer_ctx_index(map->emul_ptimer) : -1; ), - TP_printk("VCPU: %ld, dv: %d, dp: %d, ep: %d", + TP_printk("VCPU: %ld, dv: %d, dp: %d, ev: %d, ep: %d", __entry->vcpu_id, __entry->direct_vtimer, __entry->direct_ptimer, + __entry->emul_vtimer, __entry->emul_ptimer) ); diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index d97e6080b421..ae491ef97188 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -573,6 +573,21 @@ int kvm_vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, unsigned int vintid) return 0; } +int kvm_vgic_get_map(struct kvm_vcpu *vcpu, unsigned int vintid) +{ + struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, vintid); + unsigned long flags; + int ret = -1; + + raw_spin_lock_irqsave(&irq->irq_lock, flags); + if (irq->hw) + ret = irq->hwintid; + raw_spin_unlock_irqrestore(&irq->irq_lock, flags); + + vgic_put_irq(vcpu->kvm, irq); + return ret; +} + /** * kvm_vgic_set_owner - Set the owner of an interrupt for a VM * diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h index 71916de7c6c4..24dad4aaa6a3 100644 --- a/include/kvm/arm_arch_timer.h +++ b/include/kvm/arm_arch_timer.h @@ -13,6 +13,8 @@ enum kvm_arch_timers { TIMER_PTIMER, TIMER_VTIMER, + TIMER_HVTIMER, + TIMER_HPTIMER, NR_KVM_TIMERS }; @@ -21,6 +23,7 @@ enum kvm_arch_timer_regs { TIMER_REG_CVAL, TIMER_REG_TVAL, TIMER_REG_CTL, + TIMER_REG_VOFF, }; struct arch_timer_context { @@ -47,6 +50,7 @@ struct arch_timer_context { struct timer_map { struct arch_timer_context *direct_vtimer; struct arch_timer_context *direct_ptimer; + struct arch_timer_context *emul_vtimer; struct arch_timer_context *emul_ptimer; }; @@ -83,12 +87,12 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu); void kvm_timer_init_vhe(void); -bool kvm_arch_timer_get_input_level(int vintid); - #define vcpu_timer(v) (&(v)->arch.timer_cpu) #define vcpu_get_timer(v,t) (&vcpu_timer(v)->timers[(t)]) #define vcpu_vtimer(v) (&(v)->arch.timer_cpu.timers[TIMER_VTIMER]) #define vcpu_ptimer(v) (&(v)->arch.timer_cpu.timers[TIMER_PTIMER]) +#define vcpu_hvtimer(v) (&(v)->arch.timer_cpu.timers[TIMER_HVTIMER]) +#define vcpu_hptimer(v) (&(v)->arch.timer_cpu.timers[TIMER_HPTIMER]) #define arch_timer_ctx_index(ctx) ((ctx) - vcpu_timer((ctx)->vcpu)->timers) diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 0629e3532ad0..67c50f42df9c 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -380,6 +380,7 @@ int kvm_vgic_inject_irq(struct kvm *kvm, int cpuid, unsigned int intid, int kvm_vgic_map_phys_irq(struct kvm_vcpu *vcpu, unsigned int host_irq, u32 vintid, struct irq_ops *ops); int kvm_vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, unsigned int vintid); +int kvm_vgic_get_map(struct kvm_vcpu *vcpu, unsigned int vintid); bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int vintid); int kvm_vgic_vcpu_pending_irq(struct kvm_vcpu *vcpu); From patchwork Tue Jan 31 09:24:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9C201C636CC for ; Tue, 31 Jan 2023 09:52:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=z7Dy1Ti+j3f3alrohv2mzbqAK4sFjlQ+HHMfysJLW2A=; b=qGyJT++CWIwwFJ ZPwvtk2y6mWkcJ040ZkrDNtu7siwczSYsRJRwaMIHeUNCL5vcovMQLG8fxktQ0L0IMWbe/aoWpyLW WbPkGFEH+FzN+bhqpqIda+koBPqcGVMTQ6fhA9BLNkfc3FAASInjmk9jjkyr2WNEHqBabODZGdnpO dTWGU4n7kHA9N3+CTfsvwZBLmw32umnT2bSx8+2SjC1WycRvX4KmNJwL5fjyUytkPPwjuuwTkUqwC dbjKPClruMdW42fv2p/qmtNzUzY3Fv+Au2rR19oYkZZs3nTGAjnbox7wV4FA5eeVIWjfZZZ+C6RKz DN5A7fu5DENHfIRAeR9w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnI5-00797n-Ri; Tue, 31 Jan 2023 09:51:02 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8f-00758M-Ck for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:19 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id ECCA1B818EB; Tue, 31 Jan 2023 09:41:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9D78AC433D2; Tue, 31 Jan 2023 09:41:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158075; bh=i18k0iKMiryF2NTRy0/SZgBklcgli3gpOzjeka6Mt20=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FSrBoFBX8jOHEr5u7923KtGrnCRiBXn+grvucdO51fW5gkLT3wxM6nOZfwud+rJZj qYcRO+Qbz3syv/n2Zoz725+gpPe7txdRD4hafqQ+0CO3Skue+SkHMe6Ttppxn9Rbmd tk7Jd7Evrn8N0EobqLDemAUSulB+FbQXjK62LfnwS3NheuvabyI8PheuI8QFcmnlDz f7/vohDxwRpGmQOWEZzhexlbik4Qgz/TGEKV1KjeUft+Yq6YUSWLbi3UXnBw078mwS iVBCsaKYykWra9R6H9BLPxJs6HZiihoZ992Fbb4Gvv9o5FCIFC65EMenX5quqSodmH YkdwTLqRD7ZEA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtq-0067U2-Pv; Tue, 31 Jan 2023 09:25:58 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 45/69] KVM: arm64: nv: Add handling of EL2-specific timer registers Date: Tue, 31 Jan 2023 09:24:40 +0000 Message-Id: <20230131092504.2880505-46-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014117_782711_79E3A380 X-CRM114-Status: GOOD ( 13.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add the required handling for EL2 and EL02 registers, as well as EL1 registers used in the E2H context. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/sysreg.h | 6 +++ arch/arm64/kvm/sys_regs.c | 87 +++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+) diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 0457170760d8..06200ea47ad4 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -582,6 +582,12 @@ #define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3) #define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0) +#define SYS_CNTHP_TVAL_EL2 sys_reg(3, 4, 14, 2, 0) +#define SYS_CNTHP_CTL_EL2 sys_reg(3, 4, 14, 2, 1) +#define SYS_CNTHP_CVAL_EL2 sys_reg(3, 4, 14, 2, 2) +#define SYS_CNTHV_TVAL_EL2 sys_reg(3, 4, 14, 3, 0) +#define SYS_CNTHV_CTL_EL2 sys_reg(3, 4, 14, 3, 1) +#define SYS_CNTHV_CVAL_EL2 sys_reg(3, 4, 14, 3, 2) /* VHE encodings for architectural EL0/1 system registers */ #define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index a58235dec296..9ab97b5d31dc 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1349,20 +1349,92 @@ static bool access_arch_timer(struct kvm_vcpu *vcpu, switch (reg) { case SYS_CNTP_TVAL_EL0: + if (vcpu_is_el2(vcpu) && vcpu_el2_e2h_is_set(vcpu)) + tmr = TIMER_HPTIMER; + else + tmr = TIMER_PTIMER; + treg = TIMER_REG_TVAL; + break; + case SYS_AARCH32_CNTP_TVAL: + case SYS_CNTP_TVAL_EL02: tmr = TIMER_PTIMER; treg = TIMER_REG_TVAL; break; + + case SYS_CNTV_TVAL_EL02: + tmr = TIMER_VTIMER; + treg = TIMER_REG_TVAL; + break; + + case SYS_CNTHP_TVAL_EL2: + tmr = TIMER_HPTIMER; + treg = TIMER_REG_TVAL; + break; + + case SYS_CNTHV_TVAL_EL2: + tmr = TIMER_HVTIMER; + treg = TIMER_REG_TVAL; + break; + case SYS_CNTP_CTL_EL0: + if (vcpu_is_el2(vcpu) && vcpu_el2_e2h_is_set(vcpu)) + tmr = TIMER_HPTIMER; + else + tmr = TIMER_PTIMER; + treg = TIMER_REG_CTL; + break; + case SYS_AARCH32_CNTP_CTL: + case SYS_CNTP_CTL_EL02: tmr = TIMER_PTIMER; treg = TIMER_REG_CTL; break; + + case SYS_CNTV_CTL_EL02: + tmr = TIMER_VTIMER; + treg = TIMER_REG_CTL; + break; + + case SYS_CNTHP_CTL_EL2: + tmr = TIMER_HPTIMER; + treg = TIMER_REG_CTL; + break; + + case SYS_CNTHV_CTL_EL2: + tmr = TIMER_HVTIMER; + treg = TIMER_REG_CTL; + break; + case SYS_CNTP_CVAL_EL0: + if (vcpu_is_el2(vcpu) && vcpu_el2_e2h_is_set(vcpu)) + tmr = TIMER_HPTIMER; + else + tmr = TIMER_PTIMER; + treg = TIMER_REG_CVAL; + break; + case SYS_AARCH32_CNTP_CVAL: + case SYS_CNTP_CVAL_EL02: tmr = TIMER_PTIMER; treg = TIMER_REG_CVAL; break; + + case SYS_CNTV_CVAL_EL02: + tmr = TIMER_VTIMER; + treg = TIMER_REG_CVAL; + break; + + case SYS_CNTHP_CVAL_EL2: + tmr = TIMER_HPTIMER; + treg = TIMER_REG_CVAL; + break; + + case SYS_CNTHV_CVAL_EL2: + tmr = TIMER_HVTIMER; + treg = TIMER_REG_CVAL; + break; + case SYS_CNTVOFF_EL2: tmr = TIMER_VTIMER; treg = TIMER_REG_VOFF; @@ -2491,6 +2563,13 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CNTVOFF_EL2), access_arch_timer }, EL2_REG(CNTHCTL_EL2, access_rw, reset_val, 0), + { SYS_DESC(SYS_CNTHP_TVAL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHP_CTL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHP_CVAL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHV_TVAL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHV_CTL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHV_CVAL_EL2), access_arch_timer }, + EL12_REG(SCTLR, access_vm_reg, reset_val, 0x00C50078), EL12_REG(CPACR, access_rw, reset_val, 0), EL12_REG(TTBR0, access_vm_reg, reset_unknown, 0), @@ -2508,6 +2587,14 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL12_REG(CONTEXTIDR, access_vm_reg, reset_val, 0), EL12_REG(CNTKCTL, access_rw, reset_val, 0), + { SYS_DESC(SYS_CNTP_TVAL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTP_CTL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTP_CVAL_EL02), access_arch_timer }, + + { SYS_DESC(SYS_CNTV_TVAL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTV_CTL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTV_CVAL_EL02), access_arch_timer }, + EL2_REG(SP_EL2, NULL, reset_unknown, 0), }; From patchwork Tue Jan 31 09:24:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EE80EC38142 for ; Tue, 31 Jan 2023 09:45:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rpDMzwoLDdPGCDJ02hBz25ncxYt/LRWlpwQ4qaHU8SY=; b=A2hpthmimMycfR yGGfjRZZsjMdq+UEoHoCKq5QmWvxXSaJD3ciMYBFse0DlVpwndGlly1L3QqaIEhbpfjWjpGGkDN+9 lBYJbkonzGOmWyhCc6TOVfDsfEJkB4Fdnpzg2prIcqJIJ6C0Hbcdr6yMdijTAJeX4jMlTC02bqd5Z pQFmKZ88w2DLIznjb6jhCsvWbW7z5g8B/gbUpzYO2Zu55qvubBH9UmPGG5fFIryg0YdNU04cXfZc8 hXipMdQaqVe4HogR0aYIqeCkHc/GcfQNtmSdKccXUuUcOiuvq2TAvC92cfc/Ih/FuPU5eEyx1dA2V y0yKlQaIdfkLQIcPsf5g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnC8-0076ng-7r; Tue, 31 Jan 2023 09:44:53 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8M-0074wn-Ex for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:40:59 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C8125B81AB1; Tue, 31 Jan 2023 09:40:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7D3F6C433D2; Tue, 31 Jan 2023 09:40:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158054; bh=62VNiXgtEcJnQhdzskHxVPAOADe2yEDFvlnzy+HXPMc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oa6e04L/D3HuuMc27vDX9Oly0DzfTktapDjHxbhP/nWByPoSU8HAdWY5/AMuwul8R hV1sHB7H8UcXt1/vetJiWda+AaK8S8+Z7QkVMoz5p0vmAwMQRr1X0WGyx3pMiIavS7 9jZBeEzqW/wFiGyzshP5fPZq51tePhOmt0tfGjgG7WnVypFWlMLbDRe+jEZwiShoJg Ugfz2j4INxOf8K/XXXlp8F/3nsGRMyLQjg/HlPdvzZ92Qsr8ngwqCdPVi/jg3rL+qm 7d6bTw2B99xhLjYJX6yjA3YP0mMslT+n7eIN3iyWJXWBEDyMGmjyl4LJfNxisvyvj4 X2v7PotbKQSuw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtr-0067U2-73; Tue, 31 Jan 2023 09:25:59 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 46/69] KVM: arm64: nv: Load timer before the GIC Date: Tue, 31 Jan 2023 09:24:41 +0000 Message-Id: <20230131092504.2880505-47-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014058_671652_5C0742CE X-CRM114-Status: GOOD ( 15.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order for vgic_v3_load_nested to be able to observe which timer interrupts have the HW bit set for the current context, the timers must have been loaded in the new mode and the right timer mapped to their corresponding HW IRQs. At the moment, we load the GIC first, meaning that timer interrupts injected to an L2 guest will never have the HW bit set (we see the old configuration). Swapping the two loads solves this particular problem. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/arm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 92d56959cb27..cdeb4cbd4aa7 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -412,8 +412,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) vcpu->cpu = cpu; - kvm_vgic_load(vcpu); kvm_timer_vcpu_load(vcpu); + kvm_vgic_load(vcpu); if (has_vhe()) kvm_vcpu_load_sysregs_vhe(vcpu); kvm_arch_vcpu_load_fp(vcpu); From patchwork Tue Jan 31 09:24:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4ADBFC636CC for ; Tue, 31 Jan 2023 10:02:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ZFwplL1ePLm2NgWQQuUvgbJZF7fHMdvZlewd8DrJHSI=; b=ALRSh3S+oqPhfg /FkCiynJ4DVc1SgKcxxESCynYXhyPfeK3HS7nz+cTzUGqyofD21M/SIE+0hf81hmNtAt4tp9He2hP vwV4bhLXZ0+8WwCpJbZwGcfKxMLzLi41C+CkDB8mx+T79Qjkg8EBUUMb7OI/mBMce0R7bdFG0cMk1 2jQPBnBQaf0EQZVscCyehE1crhUMmKvppkdDnRRtJ58xwMnMi0hodo4ddXTX9gs9lb9XYashqTXJz jgFb/64sXmfDWyurlVMytgb2W1or/sf1ibmB5S53nAgldnak3jZ/BNmMJreBIBpNro7Uvc4SmNyqu 5lZ7LlTUqDagF1r+m2kA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnS6-007Dly-2x; Tue, 31 Jan 2023 10:01:22 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9P-0075UX-AS for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:07 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id AA7B56148E; Tue, 31 Jan 2023 09:42:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CD2ADC433D2; Tue, 31 Jan 2023 09:42:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158122; bh=NJ6HkCb9nPDizLxl69fkXTghWg2SM5hrR7t5VUJanNc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=m5aHY2GiS33+80f6L89S7NFdg4rWKbLCJ82kuxnlmRHORVVFWVq+niwjO8ZJS4jkf Jz5onA94U5CqNE55Sy6t4s9IQOCpbHhz1CdK4HX6n1YP9if2DqWjDsLEKB4b0VDsnj iw601Yjo9+7PU+lFAHhFtfDxjxdj4/mIDvwdIF+Nkbi8wD7A2kPAl/gGsDV7yq+RKk jH5e+MBRJ2Eh3uuwv2rHrx+1tZVLY3W+PbJUQs7LEL30enEMEn1cAxKBZ7R+Y2AlDS qzgzBxnUcNrTZ4Oi9lAWcJLe4M3DR7BywsC5EpqVX1+KzM3LlTB2reKZ1W0LoM/IVI m0MzTnAt4OElg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtr-0067U2-FD; Tue, 31 Jan 2023 09:25:59 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 47/69] KVM: arm64: nv: Nested GICv3 Support Date: Tue, 31 Jan 2023 09:24:42 +0000 Message-Id: <20230131092504.2880505-48-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014203_556095_E3C3F28D X-CRM114-Status: GOOD ( 32.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jintack Lim When entering a nested VM, we set up the hypervisor control interface based on what the guest hypervisor has set. Especially, we investigate each list register written by the guest hypervisor whether HW bit is set. If so, we translate hw irq number from the guest's point of view to the real hardware irq number if there is a mapping. Signed-off-by: Jintack Lim [Redesigned execution flow around vcpu load/put] Signed-off-by: Christoffer Dall [Rewritten to support GICv3 instead of GICv2] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 8 +- arch/arm64/include/asm/kvm_host.h | 13 +- arch/arm64/include/asm/kvm_nested.h | 1 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 7 ++ arch/arm64/kvm/nested.c | 16 +++ arch/arm64/kvm/sys_regs.c | 179 +++++++++++++++++++++++++- arch/arm64/kvm/vgic/vgic-v3-nested.c | 180 +++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-v3.c | 26 ++++ arch/arm64/kvm/vgic/vgic.c | 27 ++++ include/kvm/arm_vgic.h | 18 +++ 11 files changed, 468 insertions(+), 9 deletions(-) create mode 100644 arch/arm64/kvm/vgic/vgic-v3-nested.c diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 1aa059ebb569..d4721555efc9 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -540,7 +540,13 @@ static inline bool kvm_is_write_fault(struct kvm_vcpu *vcpu) static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu) { - return vcpu_read_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK; + /* + * Use the in-memory view for MPIDR_EL1. It can't be changed by the + * guest, and is also accessed from the context of *another* vcpu, + * so anything using some other state (such as the NV state that is + * used by vcpu_read_sys_reg) will eventually go wrong. + */ + return __vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK; } static inline void kvm_vcpu_set_be(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index a486aef61b3f..5db9e74b86f6 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -41,12 +41,13 @@ #define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) -#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) -#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) -#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3) -#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4) -#define KVM_REQ_RELOAD_PMU KVM_ARCH_REQ(5) -#define KVM_REQ_SUSPEND KVM_ARCH_REQ(6) +#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) +#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) +#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3) +#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4) +#define KVM_REQ_RELOAD_PMU KVM_ARCH_REQ(5) +#define KVM_REQ_SUSPEND KVM_ARCH_REQ(6) +#define KVM_REQ_GUEST_HYP_IRQ_PENDING KVM_ARCH_REQ(7) #define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 282fe63ed7b2..6e44a01403dc 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -65,6 +65,7 @@ extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu); extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr); extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); +extern void check_nested_vcpu_requests(struct kvm_vcpu *vcpu); struct kvm_s2_trans { phys_addr_t output; diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index c2717a8f12f5..4462bede5c60 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -20,7 +20,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ - vgic/vgic-its.o vgic/vgic-debug.o + vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o kvm-$(CONFIG_HW_PERF_EVENTS) += pmu-emul.o pmu.o diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index cdeb4cbd4aa7..505cb72d8535 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -766,6 +766,8 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu) if (kvm_dirty_ring_check_request(vcpu)) return 0; + + check_nested_vcpu_requests(vcpu); } return 1; @@ -900,6 +902,11 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) * preserved on VMID roll-over if the task was preempted, * making a thread's VMID inactive. So we need to call * kvm_arm_vmid_update() in non-premptible context. + * + * Note that this must happen after the check_vcpu_request() + * call to pick the correct s2_mmu structure, as a pending + * nested exception (IRQ, for example) can trigger a change + * in translation regime. */ kvm_arm_vmid_update(&vcpu->arch.hw_mmu->vmid); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 4356af6cbed0..981a4ff75da6 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -569,6 +569,22 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm) kvm_free_stage2_pgd(&kvm->arch.mmu); } +bool vgic_state_is_nested(struct kvm_vcpu *vcpu) +{ + bool imo = __vcpu_sys_reg(vcpu, HCR_EL2) & HCR_IMO; + bool fmo = __vcpu_sys_reg(vcpu, HCR_EL2) & HCR_FMO; + + WARN_ONCE(imo != fmo, "Separate virtual IRQ/FIQ settings not supported\n"); + + return vcpu_has_nv(vcpu) && imo && fmo && !is_hyp_ctxt(vcpu); +} + +void check_nested_vcpu_requests(struct kvm_vcpu *vcpu) +{ + if (kvm_check_request(KVM_REQ_GUEST_HYP_IRQ_PENDING, vcpu)) + kvm_inject_nested_irq(vcpu); +} + /* * Our emulated CPU doesn't support all the possible features. For the * sake of simplicity (and probably mental sanity), wipe out a number diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 9ab97b5d31dc..f02cc109a767 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -17,6 +17,8 @@ #include #include +#include + #include #include #include @@ -476,6 +478,19 @@ static bool access_actlr(struct kvm_vcpu *vcpu, return true; } +/* + * The architecture says that non-secure write accesses to this register from + * EL1 are trapped to EL2, if either: + * - HCR_EL2.FMO==1, or + * - HCR_EL2.IMO==1 + */ +static bool sgi_traps_to_vel2(struct kvm_vcpu *vcpu) +{ + return (vcpu_has_nv(vcpu) && + !vcpu_is_el2(vcpu) && + !!(vcpu_read_sys_reg(vcpu, HCR_EL2) & (HCR_IMO | HCR_FMO))); +} + /* * Trap handler for the GICv3 SGI generation system register. * Forward the request to the VGIC emulation. @@ -491,6 +506,11 @@ static bool access_gic_sgi(struct kvm_vcpu *vcpu, if (!p->is_write) return read_from_write_only(vcpu, p, r); + if (sgi_traps_to_vel2(vcpu)) { + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + return false; + } + /* * In a system where GICD_CTLR.DS=1, a ICC_SGI0R_EL1 access generates * Group0 SGIs only, while ICC_SGI1R_EL1 can generate either group, @@ -534,7 +554,13 @@ static bool access_gic_sre(struct kvm_vcpu *vcpu, if (p->is_write) return ignore_write(vcpu, p); - p->regval = vcpu->arch.vgic_cpu.vgic_v3.vgic_sre; + if (p->Op1 == 4) { /* ICC_SRE_EL2 */ + p->regval = (ICC_SRE_EL2_ENABLE | ICC_SRE_EL2_SRE | + ICC_SRE_EL1_DIB | ICC_SRE_EL1_DFB); + } else { /* ICC_SRE_EL1 */ + p->regval = vcpu->arch.vgic_cpu.vgic_v3.vgic_sre; + } + return true; } @@ -2093,6 +2119,122 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, return true; } +static bool access_gic_apr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + u32 index, *base; + + index = r->Op2; + if (r->CRm == 8) + base = cpu_if->vgic_ap0r; + else + base = cpu_if->vgic_ap1r; + + if (p->is_write) + base[index] = p->regval; + else + p->regval = base[index]; + + return true; +} + +static bool access_gic_hcr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + + if (p->is_write) + cpu_if->vgic_hcr = p->regval; + else + p->regval = cpu_if->vgic_hcr; + + return true; +} + +static bool access_gic_vtr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = kvm_vgic_global_state.ich_vtr_el2; + + return true; +} + +static bool access_gic_misr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = vgic_v3_get_misr(vcpu); + + return true; +} + +static bool access_gic_eisr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = vgic_v3_get_eisr(vcpu); + + return true; +} + +static bool access_gic_elrsr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = vgic_v3_get_elrsr(vcpu); + + return true; +} + +static bool access_gic_vmcr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + + if (p->is_write) + cpu_if->vgic_vmcr = p->regval; + else + p->regval = cpu_if->vgic_vmcr; + + return true; +} + +static bool access_gic_lr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + u32 index; + + index = p->Op2; + if (p->CRm == 13) + index += 8; + + if (p->is_write) + cpu_if->vgic_lr[index] = p->regval; + else + p->regval = cpu_if->vgic_lr[index]; + + return true; +} + /* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2 @@ -2557,6 +2699,41 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_RMR_EL2), trap_undef }, { SYS_DESC(SYS_VDISR_EL2), trap_undef }, + { SYS_DESC(SYS_ICH_AP0R0_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP0R1_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP0R2_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP0R3_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R0_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R1_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R2_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R3_EL2), access_gic_apr }, + + { SYS_DESC(SYS_ICC_SRE_EL2), access_gic_sre }, + + { SYS_DESC(SYS_ICH_HCR_EL2), access_gic_hcr }, + { SYS_DESC(SYS_ICH_VTR_EL2), access_gic_vtr }, + { SYS_DESC(SYS_ICH_MISR_EL2), access_gic_misr }, + { SYS_DESC(SYS_ICH_EISR_EL2), access_gic_eisr }, + { SYS_DESC(SYS_ICH_ELRSR_EL2), access_gic_elrsr }, + { SYS_DESC(SYS_ICH_VMCR_EL2), access_gic_vmcr }, + + { SYS_DESC(SYS_ICH_LR0_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR1_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR2_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR3_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR4_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR5_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR6_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR7_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR8_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR9_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR10_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR11_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR12_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR13_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR14_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR15_EL2), access_gic_lr }, + EL2_REG(CONTEXTIDR_EL2, access_rw, reset_val, 0), EL2_REG(TPIDR_EL2, access_rw, reset_val, 0), diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c new file mode 100644 index 000000000000..ab8ddf490b31 --- /dev/null +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -0,0 +1,180 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include + +#include "vgic.h" + +static inline struct vgic_v3_cpu_if *vcpu_nested_if(struct kvm_vcpu *vcpu) +{ + return &vcpu->arch.vgic_cpu.nested_vgic_v3; +} + +static inline struct vgic_v3_cpu_if *vcpu_shadow_if(struct kvm_vcpu *vcpu) +{ + return &vcpu->arch.vgic_cpu.shadow_vgic_v3; +} + +static inline bool lr_triggers_eoi(u64 lr) +{ + return !(lr & (ICH_LR_STATE | ICH_LR_HW)) && (lr & ICH_LR_EOI); +} + +u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + u16 reg = 0; + int i; + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + if (lr_triggers_eoi(cpu_if->vgic_lr[i])) + reg |= BIT(i); + } + + return reg; +} + +u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + u16 reg = 0; + int i; + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + if (!(cpu_if->vgic_lr[i] & ICH_LR_STATE)) + reg |= BIT(i); + } + + return reg; +} + +u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + int nr_lr = kvm_vgic_global_state.nr_lr; + u64 reg = 0; + + if (vgic_v3_get_eisr(vcpu)) + reg |= ICH_MISR_EOI; + + if (cpu_if->vgic_hcr & ICH_HCR_UIE) { + int used_lrs; + + used_lrs = nr_lr - hweight16(vgic_v3_get_elrsr(vcpu)); + if (used_lrs <= 1) + reg |= ICH_MISR_U; + } + + /* TODO: Support remaining bits in this register */ + return reg; +} + +/* + * For LRs which have HW bit set such as timer interrupts, we modify them to + * have the host hardware interrupt number instead of the virtual one programmed + * by the guest hypervisor. + */ +static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + struct vgic_irq *irq; + int i, used_lrs = 0; + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + u64 lr = cpu_if->vgic_lr[i]; + int l1_irq; + + if (!(lr & ICH_LR_HW)) + goto next; + + /* We have the HW bit set */ + l1_irq = (lr & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT; + irq = vgic_get_irq(vcpu->kvm, vcpu, l1_irq); + + if (!irq || !irq->hw) { + /* There was no real mapping, so nuke the HW bit */ + lr &= ~ICH_LR_HW; + if (irq) + vgic_put_irq(vcpu->kvm, irq); + goto next; + } + + /* Translate the virtual mapping to the real one */ + lr &= ~ICH_LR_EOI; /* Why? */ + lr &= ~ICH_LR_PHYS_ID_MASK; + lr |= (u64)irq->hwintid << ICH_LR_PHYS_ID_SHIFT; + vgic_put_irq(vcpu->kvm, irq); + +next: + s_cpu_if->vgic_lr[i] = lr; + used_lrs = i + 1; + } + + s_cpu_if->used_lrs = used_lrs; +} + +/* + * Change the shadow HWIRQ field back to the virtual value before copying over + * the entire shadow struct to the nested state. + */ +static void vgic_v3_fixup_shadow_lr_state(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + int lr; + + for (lr = 0; lr < kvm_vgic_global_state.nr_lr; lr++) { + s_cpu_if->vgic_lr[lr] &= ~ICH_LR_PHYS_ID_MASK; + s_cpu_if->vgic_lr[lr] |= cpu_if->vgic_lr[lr] & ICH_LR_PHYS_ID_MASK; + } +} + +void vgic_v3_load_nested(struct kvm_vcpu *vcpu) +{ + struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + + vgic_cpu->shadow_vgic_v3 = vgic_cpu->nested_vgic_v3; + vgic_v3_create_shadow_lr(vcpu); + __vgic_v3_restore_state(vcpu_shadow_if(vcpu)); +} + +void vgic_v3_put_nested(struct kvm_vcpu *vcpu) +{ + struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + + __vgic_v3_save_state(vcpu_shadow_if(vcpu)); + + /* + * Translate the shadow state HW fields back to the virtual ones + * before copying the shadow struct back to the nested one. + */ + vgic_v3_fixup_shadow_lr_state(vcpu); + vgic_cpu->nested_vgic_v3 = vgic_cpu->shadow_vgic_v3; +} + +void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + + /* + * If we exit a nested VM with a pending maintenance interrupt from the + * GIC, then we need to forward this to the guest hypervisor so that it + * can re-sync the appropriate LRs and sample level triggered interrupts + * again. + */ + if (vgic_state_is_nested(vcpu) && + (cpu_if->vgic_hcr & ICH_HCR_EN) && + vgic_v3_get_misr(vcpu)) + kvm_inject_nested_irq(vcpu); +} diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c index 60b3eab760ca..911bfd80f72c 100644 --- a/arch/arm64/kvm/vgic/vgic-v3.c +++ b/arch/arm64/kvm/vgic/vgic-v3.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include "vgic.h" @@ -278,6 +279,12 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu) vgic_v3->vgic_sre = (ICC_SRE_EL1_DIB | ICC_SRE_EL1_DFB | ICC_SRE_EL1_SRE); + /* + * If nesting is allowed, force GICv3 onto the nested + * guests as well. + */ + if (vcpu_has_nv(vcpu)) + vcpu->arch.vgic_cpu.nested_vgic_v3.vgic_sre = vgic_v3->vgic_sre; vcpu->arch.vgic_cpu.pendbaser = INITIAL_PENDBASER_VALUE; } else { vgic_v3->vgic_sre = 0; @@ -724,6 +731,13 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) { struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; + /* + * vgic_v3_load_nested only affects the LRs in the shadow + * state, so it is fine to pass the nested state around. + */ + if (vgic_state_is_nested(vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + /* * If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen * is dependent on ICC_SRE_EL1.SRE, and we have to perform the @@ -737,6 +751,9 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) if (has_vhe()) __vgic_v3_activate_traps(cpu_if); + if (vgic_state_is_nested(vcpu)) + vgic_v3_load_nested(vcpu); + WARN_ON(vgic_v4_load(vcpu)); } @@ -744,6 +761,9 @@ void vgic_v3_vmcr_sync(struct kvm_vcpu *vcpu) { struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; + if (vgic_state_is_nested(vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + if (likely(cpu_if->vgic_sre)) cpu_if->vgic_vmcr = kvm_call_hyp_ret(__vgic_v3_read_vmcr); } @@ -756,8 +776,14 @@ void vgic_v3_put(struct kvm_vcpu *vcpu) vgic_v3_vmcr_sync(vcpu); + if (vgic_state_is_nested(vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + kvm_call_hyp(__vgic_v3_save_aprs, cpu_if); if (has_vhe()) __vgic_v3_deactivate_traps(cpu_if); + + if (vgic_state_is_nested(vcpu)) + vgic_v3_put_nested(vcpu); } diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index ae491ef97188..2c5a4bb6b2f9 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -876,6 +876,10 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu) { int used_lrs; + /* If nesting, this is a load/put affair, not flush/sync. */ + if (vgic_state_is_nested(vcpu)) + return; + /* An empty ap_list_head implies used_lrs == 0 */ if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head)) return; @@ -920,6 +924,29 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu) !vgic_supports_direct_msis(vcpu->kvm)) return; + /* + * If in a nested state, we must return early. Two possibilities: + * + * - If we have any pending IRQ for the guest and the guest + * expects IRQs to be handled in its virtual EL2 mode (the + * virtual IMO bit is set) and it is not already running in + * virtual EL2 mode, then we have to emulate an IRQ + * exception to virtual EL2. + * + * We do that by placing a request to ourselves which will + * abort the entry procedure and inject the exception at the + * beginning of the run loop. + * + * - Otherwise, do exactly *NOTHING*. The guest state is + * already loaded, and we can carry on with running it. + */ + if (vgic_state_is_nested(vcpu)) { + if (kvm_vgic_vcpu_pending_irq(vcpu)) + kvm_make_request(KVM_REQ_GUEST_HYP_IRQ_PENDING, vcpu); + + return; + } + DEBUG_SPINLOCK_BUG_ON(!irqs_disabled()); if (!list_empty(&vcpu->arch.vgic_cpu.ap_list_head)) { diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 67c50f42df9c..c8af6642450d 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -331,6 +331,15 @@ struct vgic_cpu { struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS]; + /* CPU vif control registers for the virtual GICH interface */ + struct vgic_v3_cpu_if nested_vgic_v3; + + /* + * The shadow vif control register loaded to the hardware when + * running a nested L2 guest with the virtual IMO/FMO bit set. + */ + struct vgic_v3_cpu_if shadow_vgic_v3; + raw_spinlock_t ap_list_lock; /* Protects the ap_list */ /* @@ -389,6 +398,13 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu); void kvm_vgic_put(struct kvm_vcpu *vcpu); void kvm_vgic_vmcr_sync(struct kvm_vcpu *vcpu); +void vgic_v3_load_nested(struct kvm_vcpu *vcpu); +void vgic_v3_put_nested(struct kvm_vcpu *vcpu); +void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); +u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu); +u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu); +u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu); + #define irqchip_in_kernel(k) (!!((k)->arch.vgic.in_kernel)) #define vgic_initialized(k) ((k)->arch.vgic.initialized) #define vgic_ready(k) ((k)->arch.vgic.ready) @@ -433,6 +449,8 @@ int vgic_v4_load(struct kvm_vcpu *vcpu); void vgic_v4_commit(struct kvm_vcpu *vcpu); int vgic_v4_put(struct kvm_vcpu *vcpu, bool need_db); +bool vgic_state_is_nested(struct kvm_vcpu *vcpu); + /* CPU HP callbacks */ void kvm_vgic_cpu_up(void); void kvm_vgic_cpu_down(void); From patchwork Tue Jan 31 09:24:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6F7D3C636CC for ; Tue, 31 Jan 2023 10:09:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=avaY3d57gLYE5BirBJPpMQftY3rgwi7pFTbyzR9ivXk=; b=xinlpdM0KbQG0V 65SQ9+3oBx104tS8siIpcoHX83VTPxTywmo16TGT1LB0PWhupkuopcngEsG0CqwJeGje5KXJxTzea tybH5eZxLyHHVWXO1CMXr3/eGUTED6D/eaIgtJ34v3vP78nOkwW1rTJIMWAdohaD2Rv2MutoIn3It NCE6Vh6JYKwQLbv3NFR4ObZopl/2/LUoywvk6V035McagRlXcT+4Ii04SHJYZJIWqruEq8u7g5k8a xhUhQ1KtT089rKWbJ0JrWJ63pnITGkeSvGmmqui3GsAfsuQ1ky6XqVZdtGkI7TVYC2btNU//lEbGX jTQQiA83ZVNSj8xG7Tpw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnYi-007Gcz-F1; Tue, 31 Jan 2023 10:08:14 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9b-0075ZO-8r for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:16 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CBFEE61484; Tue, 31 Jan 2023 09:42:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3D56FC433D2; Tue, 31 Jan 2023 09:42:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158134; bh=dcYEQCrt9haLJXBiQNsjtFOMISpjMi07IsrJjrtHAqs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sRgR/570L1+GLMCCW5xYqIiPLMhgaxUBzMs61uW2Q9VEANE42oSUC56MwWMNen13/ yPmNRGNQsLtEDTbDyJs1l71s1QAURMZpRwn/x45HSwqhCOldxStQRMTBRDdpddmi9w SDBRrRwjnhEffYMqYUGo4PajZQnZ4oy340d/AHA9Ewq5ze1TozW4+vjwGzVHpugR4P ct+irbaheFNL0rRgAsMqWng/qsaraezLGWYHMFLGyNxiFsW8hXuyjCJh0/LKeDTP1o e+hoyD9nbbJbh5PkE7idla7HYUAzlvwsEVWCiOs3S4vRLXv+RvV3jcT0zM08emZGYi Ki9sYVFJtE/EA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtr-0067U2-SC; Tue, 31 Jan 2023 09:25:59 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 48/69] KVM: arm64: nv: Don't load the GICv4 context on entering a nested guest Date: Tue, 31 Jan 2023 09:24:43 +0000 Message-Id: <20230131092504.2880505-49-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014215_426804_BE9A63E2 X-CRM114-Status: GOOD ( 15.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When entering a nested guest (vgic_state_is_nested() == true), special care must be taken *not* to make the vPE resident, as these are interrupts targetting the L1 guest, and not any nested guest. By not making the vPE resident, we guarantee that the delivery of an vLPI will result in a doorbell, forcing an exit from the nested guest and a switch to the L1 guest to handle the interrupt. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/vgic/vgic-v3.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c index 911bfd80f72c..3a1ade1b592e 100644 --- a/arch/arm64/kvm/vgic/vgic-v3.c +++ b/arch/arm64/kvm/vgic/vgic-v3.c @@ -753,8 +753,8 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) if (vgic_state_is_nested(vcpu)) vgic_v3_load_nested(vcpu); - - WARN_ON(vgic_v4_load(vcpu)); + else + WARN_ON(vgic_v4_load(vcpu)); } void vgic_v3_vmcr_sync(struct kvm_vcpu *vcpu) @@ -772,6 +772,12 @@ void vgic_v3_put(struct kvm_vcpu *vcpu) { struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; + /* + * vgic_v4_put will do nothing if we were not resident. This + * covers both the cases where we've blocked (we already have + * done a vgic_v4_put) and when running a nested guest (the + * vPE was never resident in order to generate a doorbell). + */ WARN_ON(vgic_v4_put(vcpu, false)); vgic_v3_vmcr_sync(vcpu); From patchwork Tue Jan 31 09:24:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 45DCCC38142 for ; Tue, 31 Jan 2023 10:22:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=f4nUmU4f5tPOyUXcFFv/CaNF1n5aZ51sqwYs58xgYjE=; b=k9wbYelLvlEzqp /Q/rPpakZOLoWI+T1Rjqx1SbMIS2+lxtGwDCIQZZ1bVFdN5JVzCe2DHP5B+ss2BHUFVoXiwKaKtZ6 i83R0sFVbn8EmiBBQJqlpqwcTPD3qRTSCtBmuZVTKabyb5MX2f7shAomokv9jHMx4p+J9xgaTSTHW c5sgpAv0dJX2PR29n0AN9mC05p35NAS000bhhfFNncty6viCCszi9iTf2hb3lo0Wd0H27fl7l9fXF fOp7QBsFeoUc44LG4t8ib6LvQsxPJjD8qowTAdSCr6w8DBvUSpPNp3V+d9xZppz6NflySRFnlS2nQ EtjowJ275INhU/Kt6iJw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnlH-007MI8-CD; Tue, 31 Jan 2023 10:21:11 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnA8-0075qE-6n for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:50 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D7D90B81AE0; Tue, 31 Jan 2023 09:42:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8B095C433EF; Tue, 31 Jan 2023 09:42:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158165; bh=dHvQMGh1EyBj7A6q5y/oVjPmQ6OuDbClXpT6HV6dj3U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IIOz5kMC1gXJpQhnx9uX2uxR5gxFZ+8I1wTIiqmjzPkKKC/cQMZHTmEsS11lwZENn I15lelHxpibhXUpHktzKc1BAaqIgCjHPXTnxkS3HVoeNrra6uJF4T6XCP59RLETEL+ CZMEQ8bAWTMfdblDIg1sM9jwAcuq8GB9dXp1EiGkGvMqiwfZkV7AA0OmcZP6fxpBtL 6ZjUn2jaArWzFHoCi2NtQSXBqEyPgonq17s7vQYXrAhQang4Zthfhf++KkL7iD3eCC tjtalFu/MBfPIQ720XA4N0EmxDVBm7D8ekne0uZG4O3k176t2+CrCg7Xz3zC+QLJWO cWhiEnmQHLiew== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmts-0067U2-4X; Tue, 31 Jan 2023 09:26:00 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 49/69] KVM: arm64: nv: vgic: Emulate the HW bit in software Date: Tue, 31 Jan 2023 09:24:44 +0000 Message-Id: <20230131092504.2880505-50-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014248_574835_244BB9AC X-CRM114-Status: GOOD ( 21.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall Should the guest hypervisor use the HW bit in the LRs, we need to emulate the deactivation from the L2 guest into the L1 distributor emulation, which is handled by L0. It's all good fun. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_hyp.h | 2 ++ arch/arm64/kvm/hyp/vgic-v3-sr.c | 2 +- arch/arm64/kvm/vgic/vgic-v3-nested.c | 32 ++++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic.c | 6 ++++-- include/kvm/arm_vgic.h | 1 + 5 files changed, 40 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 6797eafe7890..bd4ef970ecfa 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -57,6 +57,8 @@ DECLARE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu); +u64 __gic_v3_get_lr(unsigned int lr); + void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if); void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if); void __vgic_v3_activate_traps(struct vgic_v3_cpu_if *cpu_if); diff --git a/arch/arm64/kvm/hyp/vgic-v3-sr.c b/arch/arm64/kvm/hyp/vgic-v3-sr.c index 6cb638b184b1..75152c1ce646 100644 --- a/arch/arm64/kvm/hyp/vgic-v3-sr.c +++ b/arch/arm64/kvm/hyp/vgic-v3-sr.c @@ -18,7 +18,7 @@ #define vtr_to_nr_pre_bits(v) ((((u32)(v) >> 26) & 7) + 1) #define vtr_to_nr_apr_regs(v) (1 << (vtr_to_nr_pre_bits(v) - 5)) -static u64 __gic_v3_get_lr(unsigned int lr) +u64 __gic_v3_get_lr(unsigned int lr) { switch (lr & 0xf) { case 0: diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c index ab8ddf490b31..e88c75e79010 100644 --- a/arch/arm64/kvm/vgic/vgic-v3-nested.c +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -140,6 +140,38 @@ static void vgic_v3_fixup_shadow_lr_state(struct kvm_vcpu *vcpu) } } +void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + struct vgic_irq *irq; + int i; + + for (i = 0; i < s_cpu_if->used_lrs; i++) { + u64 lr = cpu_if->vgic_lr[i]; + int l1_irq; + + if (!(lr & ICH_LR_HW) || !(lr & ICH_LR_STATE)) + continue; + + /* + * If we had a HW lr programmed by the guest hypervisor, we + * need to emulate the HW effect between the guest hypervisor + * and the nested guest. + */ + l1_irq = (lr & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT; + irq = vgic_get_irq(vcpu->kvm, vcpu, l1_irq); + if (!irq) + continue; /* oh well, the guest hyp is broken */ + + lr = __gic_v3_get_lr(i); + if (!(lr & ICH_LR_STATE)) + irq->active = false; + + vgic_put_irq(vcpu->kvm, irq); + } +} + void vgic_v3_load_nested(struct kvm_vcpu *vcpu) { struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index 2c5a4bb6b2f9..05809462e199 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -876,9 +876,11 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu) { int used_lrs; - /* If nesting, this is a load/put affair, not flush/sync. */ - if (vgic_state_is_nested(vcpu)) + /* If nesting, emulate the HW effect from L0 to L1 */ + if (vgic_state_is_nested(vcpu)) { + vgic_v3_sync_nested(vcpu); return; + } /* An empty ap_list_head implies used_lrs == 0 */ if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head)) diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index c8af6642450d..a466cda3a739 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -398,6 +398,7 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu); void kvm_vgic_put(struct kvm_vcpu *vcpu); void kvm_vgic_vmcr_sync(struct kvm_vcpu *vcpu); +void vgic_v3_sync_nested(struct kvm_vcpu *vcpu); void vgic_v3_load_nested(struct kvm_vcpu *vcpu); void vgic_v3_put_nested(struct kvm_vcpu *vcpu); void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); From patchwork Tue Jan 31 09:24:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1E467C636CC for ; Tue, 31 Jan 2023 09:57:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Xhxg8Q68aCBz28EpiECcwRb+PJcWX3jJW8cWzJv8k1M=; b=Cx9PTGvddPodbG H8LMMi2lJPi2Nj790USE5av2FvNDNtwdlrH93H2123OqAP4cYcH+HMiFGu9ZkGtdkeaUVwrDUketc AIkasYrmUUeo2rQD0AMTc0oyS2cPnvhyInDT3+YsgGzUnUw/Z2BVr+oHth41VuxVVRP7uozvTkQh/ BLCpJfgR0hEeXxhYqbNjDm7/GNizvZQTCb5BMGwQEcZy3ps7mzJtDpZJ88j5EgtEvi3Qlkc9VW07G EkpLD5Vtz8l7STIoQ8+38d3phgLl994wKwHLnXL9P8krJx8YzZ6DmEUuflLNOyPi7VjKePIg+B49s hEdazm+LU69aitcGKAzw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnNA-007Be8-5V; Tue, 31 Jan 2023 09:56:16 +0000 Received: from sin.source.kernel.org ([2604:1380:40e1:4800::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9C-0075Md-18 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:52 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 57B5FCE1CCB; Tue, 31 Jan 2023 09:41:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8A5D0C433D2; Tue, 31 Jan 2023 09:41:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158104; bh=NQLd+kx2LypSnl9dYhw+RFVUBmaP9stvi5I5iXIIKUQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UUjSto7JkCOi6T85SzTiaOjWe2d/JMgCVe647t0KCMo6VEIz6NUt4kbsvFRQXm1UI su7aU1rkqm2UnMGFY8BdByZElYjC+qfgTbtLCNrx+FMmspjP6e20ygOBooAEosmGW3 KDgThZlDH/sz4EWcVHuJatEmCLLH+CNnJKl+8DrUezAPCnKTljGT6YUkwWdggfVDt4 6SDp6JYsfaekJhQCokt2zEBJsd0AkKEanP5yffBhnGFp/xd16snbp2+PPsdUEmoilE UGF7SA9f4WKQz6W9fjrDdZBRysK3EZ4sWZfmjgWVOqP0hm0WzOiojSUIHYRedR4+JQ UnrRRj5vwjtWg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmts-0067U2-Cp; Tue, 31 Jan 2023 09:26:00 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 50/69] KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ Date: Tue, 31 Jan 2023 09:24:45 +0000 Message-Id: <20230131092504.2880505-51-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014150_951222_563667D4 X-CRM114-Status: GOOD ( 23.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Andre Przywara The VGIC maintenance IRQ signals various conditions about the LRs, when the GIC's virtualization extension is used. So far we didn't need it, but nested virtualization needs to know about this interrupt, so add a userland interface to setup the IRQ number. The architecture mandates that it must be a PPI, on top of that this code only exports a per-device option, so the PPI is the same on all VCPUs. Signed-off-by: Andre Przywara [added some bits of documentation] Signed-off-by: Marc Zyngier --- .../virt/kvm/devices/arm-vgic-v3.rst | 12 +++++++- arch/arm64/include/uapi/asm/kvm.h | 1 + arch/arm64/kvm/vgic/vgic-kvm-device.c | 29 +++++++++++++++++-- include/kvm/arm_vgic.h | 3 ++ tools/arch/arm/include/uapi/asm/kvm.h | 1 + 5 files changed, 43 insertions(+), 3 deletions(-) diff --git a/Documentation/virt/kvm/devices/arm-vgic-v3.rst b/Documentation/virt/kvm/devices/arm-vgic-v3.rst index 51e5e5762571..1901e651cc00 100644 --- a/Documentation/virt/kvm/devices/arm-vgic-v3.rst +++ b/Documentation/virt/kvm/devices/arm-vgic-v3.rst @@ -284,8 +284,18 @@ Groups: | Aff3 | Aff2 | Aff1 | Aff0 | Errors: - ======= ============================================= -EINVAL vINTID is not multiple of 32 or info field is not VGIC_LEVEL_INFO_LINE_LEVEL ======= ============================================= + + KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ + Attributes: + + The attr field of kvm_device_attr encodes the following values: + + bits: | 31 .... 5 | 4 .... 0 | + values: | RES0 | vINTID | + + The vINTID specifies which interrupt is generated when the vGIC + must generate a maintenance interrupt. This must be a PPI. diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index f8129c624b07..8bf7b1a62e23 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -390,6 +390,7 @@ enum { #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7 #define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 +#define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ 9 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT) diff --git a/arch/arm64/kvm/vgic/vgic-kvm-device.c b/arch/arm64/kvm/vgic/vgic-kvm-device.c index edeac2380591..0cc40b492ee1 100644 --- a/arch/arm64/kvm/vgic/vgic-kvm-device.c +++ b/arch/arm64/kvm/vgic/vgic-kvm-device.c @@ -288,6 +288,12 @@ static int vgic_get_common_attr(struct kvm_device *dev, VGIC_NR_PRIVATE_IRQS, uaddr); break; } + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: { + u32 __user *uaddr = (u32 __user *)(long)attr->addr; + + r = put_user(dev->kvm->arch.vgic.maint_irq, uaddr); + break; + } } return r; @@ -541,7 +547,7 @@ static int vgic_v3_attr_regs_access(struct kvm_device *dev, struct vgic_reg_attr reg_attr; gpa_t addr; struct kvm_vcpu *vcpu; - bool uaccess; + bool uaccess, post_init = true; u32 val; int ret; @@ -557,6 +563,9 @@ static int vgic_v3_attr_regs_access(struct kvm_device *dev, /* Sysregs uaccess is performed by the sysreg handling code */ uaccess = false; break; + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: + post_init = false; + fallthrough; default: uaccess = true; } @@ -569,7 +578,7 @@ static int vgic_v3_attr_regs_access(struct kvm_device *dev, mutex_lock(&dev->kvm->lock); - if (unlikely(!vgic_initialized(dev->kvm))) { + if (post_init != vgic_initialized(dev->kvm)) { ret = -EBUSY; goto out; } @@ -604,6 +613,19 @@ static int vgic_v3_attr_regs_access(struct kvm_device *dev, } break; } + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: + if (!is_write) { + val = dev->kvm->arch.vgic.maint_irq; + ret = 0; + break; + } + + ret = -EINVAL; + if ((val < VGIC_NR_PRIVATE_IRQS) && (val >= VGIC_NR_SGIS)) { + dev->kvm->arch.vgic.maint_irq = val; + ret = 0; + } + break; default: ret = -EINVAL; break; @@ -629,6 +651,7 @@ static int vgic_v3_set_attr(struct kvm_device *dev, case KVM_DEV_ARM_VGIC_GRP_REDIST_REGS: case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS: case KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO: + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: return vgic_v3_attr_regs_access(dev, attr, true); default: return vgic_set_common_attr(dev, attr); @@ -643,6 +666,7 @@ static int vgic_v3_get_attr(struct kvm_device *dev, case KVM_DEV_ARM_VGIC_GRP_REDIST_REGS: case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS: case KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO: + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: return vgic_v3_attr_regs_access(dev, attr, false); default: return vgic_get_common_attr(dev, attr); @@ -666,6 +690,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev, case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS: return vgic_v3_has_attr_regs(dev, attr); case KVM_DEV_ARM_VGIC_GRP_NR_IRQS: + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: return 0; case KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO: { if (((attr->attr & KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK) >> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index a466cda3a739..38643d9b3813 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -243,6 +243,9 @@ struct vgic_dist { int nr_spis; + /* The GIC maintenance IRQ for nested hypervisors. */ + u32 maint_irq; + /* base addresses in guest physical address space: */ gpa_t vgic_dist_base; /* distributor */ union { diff --git a/tools/arch/arm/include/uapi/asm/kvm.h b/tools/arch/arm/include/uapi/asm/kvm.h index 03cd7c19a683..d5dd96902817 100644 --- a/tools/arch/arm/include/uapi/asm/kvm.h +++ b/tools/arch/arm/include/uapi/asm/kvm.h @@ -246,6 +246,7 @@ struct kvm_vcpu_events { #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7 #define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 +#define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ 9 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT) From patchwork Tue Jan 31 09:24:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122829 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5E864C38142 for ; Tue, 31 Jan 2023 10:26:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3CB3JKIOb2rn97QtF1H7qRYwrxaVgRUl7BH0o+aTeAA=; b=TEZVrP4FLTtktU lZAQFP5WCvtpQ3jVr8TjH+/kQsUOSUkV3K8sIGpBxhA5C3+pqW7dcLyIckyT5QMjKrogiWEhETKXF A0dh13xq6vepl/ydRCagw+jXkhkOJbQJhhDa+Z6UAOmGakYCN0veRErNXD16YD965ujVba0k+VRlk DsJ2KjMAflcmbCuyr3pnX/zhI3Aszj+i3lRn+DFmDaGM4wVpP0nriAivNFBTS0vShrxm9cEJcivBp 6dUjkN+ZTqIkWam0b8IO8YaDi4YQ7prwd+2YfHzdyZuIHwXYjygwKI2Ow0NzaLlkyX1OgZOaRlvSI 5ul9lzPWqmeksDFH4QjQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnoe-007NjH-Nh; Tue, 31 Jan 2023 10:24:41 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnAL-0075zb-4B for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:43:03 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 94DC361494; Tue, 31 Jan 2023 09:43:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 65C4CC4339B; Tue, 31 Jan 2023 09:43:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158180; bh=A8APb/ShbfEk/49VtpSdFlMSQpzv8xpXTIJI/KZaCbQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=t4eliMzL9aXBNN+dY8khuA2nxK5Bsxq7JJEq51BbGVlU68h4Hbu8KpXdfyXLcMiWo Avu3VlKFwQFnF9rch+przGe2q10cHfcTwNXNWYctFXgAn2euIU92ABFLSiPG9HAGPS ULNNENv+Qk1RPgo75JdfFko+TG6QWs+sdmR+FoOTINN0HQWx9oMQunAzzRIBwTL8Fz Q7BlGz4IGOEMBwjy9ju5Zi1oTyqr44tduoEpp3EnSid10brVpEXkQWTFxCQQBomoWE eXjl0mC27EN+I5JRBWihZYzFY//2gtQvoC7YVdC799sekT0EBOeURA/i6tkzZUvGOR rI25RZc37uqRg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmts-0067U2-LD; Tue, 31 Jan 2023 09:26:00 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 51/69] KVM: arm64: nv: Implement maintenance interrupt forwarding Date: Tue, 31 Jan 2023 09:24:46 +0000 Message-Id: <20230131092504.2880505-52-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014301_299733_2B56CB9D X-CRM114-Status: GOOD ( 23.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When we take a maintenance interrupt, we need to decide whether it is generated on an action from the guest, or if it is something that needs to be forwarded to the guest hypervisor. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/vgic/vgic-init.c | 33 ++++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-v3-nested.c | 28 ++++++++++++++++++----- 2 files changed, 55 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-init.c b/arch/arm64/kvm/vgic/vgic-init.c index cd134db41a57..a97279f82bbd 100644 --- a/arch/arm64/kvm/vgic/vgic-init.c +++ b/arch/arm64/kvm/vgic/vgic-init.c @@ -6,10 +6,12 @@ #include #include #include +#include #include #include #include #include +#include #include "vgic.h" /* @@ -222,6 +224,16 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu) if (!irqchip_in_kernel(vcpu->kvm)) return 0; + if (vcpu_has_nv(vcpu)) { + /* Cope with vintage userspace. Maybe we should fail instead */ + if (vcpu->kvm->arch.vgic.maint_irq == 0) + vcpu->kvm->arch.vgic.maint_irq = kvm_vgic_global_state.maint_irq; + ret = kvm_vgic_set_owner(vcpu, vcpu->kvm->arch.vgic.maint_irq, + vcpu); + if (ret) + return ret; + } + /* * If we are creating a VCPU with a GICv3 we must also register the * KVM io device for the redistributor that belongs to this VCPU. @@ -478,12 +490,23 @@ void kvm_vgic_cpu_down(void) static irqreturn_t vgic_maintenance_handler(int irq, void *data) { + struct kvm_vcpu *vcpu = *(struct kvm_vcpu **)data; + /* * We cannot rely on the vgic maintenance interrupt to be * delivered synchronously. This means we can only use it to * exit the VM, and we perform the handling of EOIed * interrupts on the exit path (see vgic_fold_lr_state). */ + + /* If not nested, deactivate */ + if (!vcpu || !vgic_state_is_nested(vcpu)) { + irq_set_irqchip_state(irq, IRQCHIP_STATE_ACTIVE, false); + return IRQ_HANDLED; + } + + /* Assume nested from now */ + vgic_v3_handle_nested_maint_irq(vcpu); return IRQ_HANDLED; } @@ -582,6 +605,16 @@ int kvm_vgic_hyp_init(void) return ret; } + if (has_mask) { + ret = irq_set_vcpu_affinity(kvm_vgic_global_state.maint_irq, + kvm_get_running_vcpus()); + if (ret) { + kvm_err("Error setting vcpu affinity\n"); + free_percpu_irq(kvm_vgic_global_state.maint_irq, kvm_get_running_vcpus()); + return ret; + } + } + kvm_info("vgic interrupt IRQ%d\n", kvm_vgic_global_state.maint_irq); return 0; } diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c index e88c75e79010..755e4819603a 100644 --- a/arch/arm64/kvm/vgic/vgic-v3-nested.c +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -175,10 +175,20 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) void vgic_v3_load_nested(struct kvm_vcpu *vcpu) { struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + struct vgic_irq *irq; + unsigned long flags; vgic_cpu->shadow_vgic_v3 = vgic_cpu->nested_vgic_v3; vgic_v3_create_shadow_lr(vcpu); __vgic_v3_restore_state(vcpu_shadow_if(vcpu)); + + irq = vgic_get_irq(vcpu->kvm, vcpu, vcpu->kvm->arch.vgic.maint_irq); + raw_spin_lock_irqsave(&irq->irq_lock, flags); + if (irq->line_level || irq->active) + irq_set_irqchip_state(kvm_vgic_global_state.maint_irq, + IRQCHIP_STATE_ACTIVE, true); + raw_spin_unlock_irqrestore(&irq->irq_lock, flags); + vgic_put_irq(vcpu->kvm, irq); } void vgic_v3_put_nested(struct kvm_vcpu *vcpu) @@ -193,20 +203,26 @@ void vgic_v3_put_nested(struct kvm_vcpu *vcpu) */ vgic_v3_fixup_shadow_lr_state(vcpu); vgic_cpu->nested_vgic_v3 = vgic_cpu->shadow_vgic_v3; + irq_set_irqchip_state(kvm_vgic_global_state.maint_irq, + IRQCHIP_STATE_ACTIVE, false); } void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); - /* * If we exit a nested VM with a pending maintenance interrupt from the * GIC, then we need to forward this to the guest hypervisor so that it * can re-sync the appropriate LRs and sample level triggered interrupts * again. */ - if (vgic_state_is_nested(vcpu) && - (cpu_if->vgic_hcr & ICH_HCR_EN) && - vgic_v3_get_misr(vcpu)) - kvm_inject_nested_irq(vcpu); + if (vgic_state_is_nested(vcpu)) { + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + bool state; + + state = cpu_if->vgic_hcr & ICH_HCR_EN; + state &= vgic_v3_get_misr(vcpu); + + kvm_vgic_inject_irq(vcpu->kvm, vcpu->vcpu_id, + vcpu->kvm->arch.vgic.maint_irq, state, vcpu); + } } From patchwork Tue Jan 31 09:24:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8147AC38142 for ; Tue, 31 Jan 2023 10:15:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=H8Y/Rx8mSM0jgMea5v3cFdRBZQaVmFHPqkVWVf8W0bQ=; b=gmkHm9DZGYxl1o DCtCc/fUniTOAWBHibK4LmFlF0Wvu5WNE0e3pfE/4TFsAJQKogAykeO7JI68xcy6Q7mU4bmWQqu4k aqXJTW62BWhAJQDwK+yYRPJ2v4aD3TVD/CPQRjES1S8ndSg7GOgO+BCrRtbT6/Nn74hQJXr6MPwdG rQOD+36PsMOFAoGMuUiWbZikPfvsTzm2Rr1bCa1ufoLrtn6n96X2+CyEZGxYj5ZZebu+bt1FsgyXs w3pycTqa6po0odAwCFnZWlJmwQwewI8sat5B3HuFCKfS/o/AY34SAsUCf0X9RF7xLObkCCC9bXW6U CN84c9N9W38A2aBQ6wlg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnf6-007JZO-Ik; Tue, 31 Jan 2023 10:14:49 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9o-0075gH-Hx for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:30 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 107E261480; Tue, 31 Jan 2023 09:42:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 74DF6C433EF; Tue, 31 Jan 2023 09:42:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158147; bh=HzXSsUwByW2jAR2hO6nb9l7RS/h2EPWpRifFw83LriQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S9anC6S2sQ2cUxkZ13UPJKavWae38cmwSk6DnFkKl2WdZMkof8olpQrJp5gSjkjNg NhOnrHNCt6bjbcbJrcl6cXlnodBxQXS3mz9753XJTQKc3r0OofkVaf8AEok7/g0gpc 3fgAQV/TwQ95R4YgXYQEE+TtCfSFtyV+wwl5u2Uwhy/QpRNcjEyMoPtfwFp6pRPhaH u+yM6Nlv1auXb21Nk85J27STv/JQs4Uhz3fCrQuUYe1zM3i4u2e67G4LJJJNDZIm8A M/EGLAdle1OExgQ7W2QXEs8T3Q8QkbKMms/d/5pIap84WmADCDAlOZGz1yOTJ4zOiA 1oQMiuj6ckoog== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmts-0067U2-Tz; Tue, 31 Jan 2023 09:26:00 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 52/69] KVM: arm64: nv: Deal with broken VGIC on maintenance interrupt delivery Date: Tue, 31 Jan 2023 09:24:47 +0000 Message-Id: <20230131092504.2880505-53-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014228_737598_95853AB1 X-CRM114-Status: GOOD ( 14.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Normal, non-nesting KVM deals with maintenance interrupt in a very simple way: we don't even try to handle it and just turn it off as soon as we exit, long before the kernel can handle it. However, with NV, we rely on the actual handling of the interrupt to leave it active and pass it down to the L1 guest hypervisor (we effectively treat it as an assigned interrupt, just like the timer). This doesn't work with something like the Apple M2, which doesn't have an active state that allows the interrupt to be masked. Instead, just disable the vgic after having taken the interrupt and injected a virtual interrupt. This is enough for the guest to make forward progress, but will limit its ability to handle further interrupts until it next exits (IAR will always report "spurious"). Oh well. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/vgic/vgic-v3-nested.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c index 755e4819603a..919275b94625 100644 --- a/arch/arm64/kvm/vgic/vgic-v3-nested.c +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -225,4 +225,7 @@ void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) kvm_vgic_inject_irq(vcpu->kvm, vcpu->vcpu_id, vcpu->kvm->arch.vgic.maint_irq, state, vcpu); } + + if (unlikely(kvm_vgic_global_state.no_hw_deactivation)) + sysreg_clear_set_s(SYS_ICH_HCR_EL2, ICH_HCR_EN, 0); } From patchwork Tue Jan 31 09:24:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ABBD1C636CD for ; Tue, 31 Jan 2023 10:01:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ytwi46OkkVaGTRYgy+5hzqzFJ76imZL00Vdy8Hp9aOQ=; b=imTOlRnZktM0E+ +umidYKzaKGHWY1aBaKdLi8bhzVx43LOh9Yy09CPB7E9BRQZyXKPXaIPSErA7gKPScdW8nJvfRMzN DdYdAiFM97d2kG4x5af0DmIZ/SZfMdbO5aZ2BchgcizG4oL4Ttc1gBu5OXL0B/c1S84K1DmpcyeKE NU1IYTzvDMamHF/eRRT/lWZeSP7VuiEsxHjOXxD4j+c/SJ4m25zeGZ1Anwv0q/wtcWJUXa2Aq34EK y1BsXZgSttO9m5efwZBQix9EhVcO+vJ0dNu07yVyt6m8cWhxLVcbF8e7ttMv1VCCJoVyoqWfVU235 0Vx9MpqDc70g4DqC9oAw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnR8-007DJT-MA; Tue, 31 Jan 2023 10:00:23 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9L-0075S0-87 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:01 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D0B87B81AE0; Tue, 31 Jan 2023 09:41:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 60ADFC433D2; Tue, 31 Jan 2023 09:41:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158116; bh=VuNXutmixGrigENDAWh4xcOZ1P+J8v17NsHrHKS1Th0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QJUai05lCUjV+hmeSuYdTedSMezavCTDYC6JxCeDZXsNfBvg/dLjZUYSfpITGmHPH ocEQ0iCH0yK615bA4//RpM2hnRa660YCFPDMSuHgyCrsz9WdPE5WB8lGDO4dywfNP7 oEWnES2Pb1XEGLctYRdwPjgcnkgnshSli4cXHbqc+D4pO1UPWOkT4BBzcAuZIhS77y 1q+BLTuAKVKeyrFAvKoFeP+L6MYZr4vr8yrWvJNcOWGXSMxF6ArKK72EXSd1kvYc7Q OGORK/sPVwpz7qwn3JhD32b7JQqp3JoJ7zqwKDv9M4mbrVM9piyc0YYBz05afpFnMJ ibO83lc2BCkdQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtt-0067U2-5z; Tue, 31 Jan 2023 09:26:01 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 53/69] KVM: arm64: nv: Add nested GICv3 tracepoints Date: Tue, 31 Jan 2023 09:24:48 +0000 Message-Id: <20230131092504.2880505-54-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014159_636714_1338216C X-CRM114-Status: GOOD ( 19.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall Adding tracepoints to be able to peek into the shadow LRs used when running a guest guest. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/vgic/vgic-nested-trace.h | 137 ++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-v3-nested.c | 13 ++- 2 files changed, 149 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/vgic/vgic-nested-trace.h diff --git a/arch/arm64/kvm/vgic/vgic-nested-trace.h b/arch/arm64/kvm/vgic/vgic-nested-trace.h new file mode 100644 index 000000000000..f1a074c791a6 --- /dev/null +++ b/arch/arm64/kvm/vgic/vgic-nested-trace.h @@ -0,0 +1,137 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(_TRACE_VGIC_NESTED_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_VGIC_NESTED_H + +#include + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM kvm + +#define SLR_ENTRY_VALS(x) \ + " ", \ + !!(__entry->lrs[x] & ICH_LR_HW), \ + !!(__entry->lrs[x] & ICH_LR_PENDING_BIT), \ + !!(__entry->lrs[x] & ICH_LR_ACTIVE_BIT), \ + __entry->lrs[x] & ICH_LR_VIRTUAL_ID_MASK, \ + (__entry->lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT, \ + (__entry->orig_lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT + +TRACE_EVENT(vgic_create_shadow_lrs, + TP_PROTO(struct kvm_vcpu *vcpu, int nr_lr, u64 *lrs, u64 *orig_lrs), + TP_ARGS(vcpu, nr_lr, lrs, orig_lrs), + + TP_STRUCT__entry( + __field( int, nr_lr ) + __array( u64, lrs, 16 ) + __array( u64, orig_lrs, 16 ) + ), + + TP_fast_assign( + __entry->nr_lr = nr_lr; + memcpy(__entry->lrs, lrs, 16 * sizeof(u64)); + memcpy(__entry->orig_lrs, orig_lrs, 16 * sizeof(u64)); + ), + + TP_printk("nr_lr: %d\n" + "%50sLR[ 0]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 1]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 2]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 3]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 4]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 5]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 6]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 7]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 8]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 9]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[10]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[11]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[12]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[13]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[14]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[15]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)", + __entry->nr_lr, + SLR_ENTRY_VALS(0), SLR_ENTRY_VALS(1), SLR_ENTRY_VALS(2), + SLR_ENTRY_VALS(3), SLR_ENTRY_VALS(4), SLR_ENTRY_VALS(5), + SLR_ENTRY_VALS(6), SLR_ENTRY_VALS(7), SLR_ENTRY_VALS(8), + SLR_ENTRY_VALS(9), SLR_ENTRY_VALS(10), SLR_ENTRY_VALS(11), + SLR_ENTRY_VALS(12), SLR_ENTRY_VALS(13), SLR_ENTRY_VALS(14), + SLR_ENTRY_VALS(15)) +); + +#define LR_ENTRY_VALS(x) \ + " ", \ + !!(__entry->lrs[x] & ICH_LR_HW), \ + !!(__entry->lrs[x] & ICH_LR_PENDING_BIT), \ + !!(__entry->lrs[x] & ICH_LR_ACTIVE_BIT), \ + __entry->lrs[x] & ICH_LR_VIRTUAL_ID_MASK, \ + (__entry->lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT + +TRACE_EVENT(vgic_put_nested, + TP_PROTO(struct kvm_vcpu *vcpu, int nr_lr, u64 *lrs), + TP_ARGS(vcpu, nr_lr, lrs), + + TP_STRUCT__entry( + __field( int, nr_lr ) + __array( u64, lrs, 16 ) + ), + + TP_fast_assign( + __entry->nr_lr = nr_lr; + memcpy(__entry->lrs, lrs, 16 * sizeof(u64)); + ), + + TP_printk("nr_lr: %d\n" + "%50sLR[ 0]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 1]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 2]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 3]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 4]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 5]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 6]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 7]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 8]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 9]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[10]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[11]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[12]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[13]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[14]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[15]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu", + __entry->nr_lr, + LR_ENTRY_VALS(0), LR_ENTRY_VALS(1), LR_ENTRY_VALS(2), + LR_ENTRY_VALS(3), LR_ENTRY_VALS(4), LR_ENTRY_VALS(5), + LR_ENTRY_VALS(6), LR_ENTRY_VALS(7), LR_ENTRY_VALS(8), + LR_ENTRY_VALS(9), LR_ENTRY_VALS(10), LR_ENTRY_VALS(11), + LR_ENTRY_VALS(12), LR_ENTRY_VALS(13), LR_ENTRY_VALS(14), + LR_ENTRY_VALS(15)) +); + +TRACE_EVENT(vgic_nested_hw_emulate, + TP_PROTO(int lr, u64 lr_val, u32 l1_intid), + TP_ARGS(lr, lr_val, l1_intid), + + TP_STRUCT__entry( + __field( int, lr ) + __field( u64, lr_val ) + __field( u32, l1_intid ) + ), + + TP_fast_assign( + __entry->lr = lr; + __entry->lr_val = lr_val; + __entry->l1_intid = l1_intid; + ), + + TP_printk("lr: %d LR %llx L1 INTID: %u\n", + __entry->lr, __entry->lr_val, __entry->l1_intid) +); + +#endif /* _TRACE_VGIC_NESTED_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH vgic/ +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_FILE vgic-nested-trace + +/* This part must be outside protection */ +#include diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c index 919275b94625..98b21adb6e9b 100644 --- a/arch/arm64/kvm/vgic/vgic-v3-nested.c +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -15,6 +15,9 @@ #include "vgic.h" +#define CREATE_TRACE_POINTS +#include "vgic-nested-trace.h" + static inline struct vgic_v3_cpu_if *vcpu_nested_if(struct kvm_vcpu *vcpu) { return &vcpu->arch.vgic_cpu.nested_vgic_v3; @@ -121,6 +124,9 @@ static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) used_lrs = i + 1; } + trace_vgic_create_shadow_lrs(vcpu, kvm_vgic_global_state.nr_lr, + s_cpu_if->vgic_lr, cpu_if->vgic_lr); + s_cpu_if->used_lrs = used_lrs; } @@ -165,8 +171,10 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) continue; /* oh well, the guest hyp is broken */ lr = __gic_v3_get_lr(i); - if (!(lr & ICH_LR_STATE)) + if (!(lr & ICH_LR_STATE)) { + trace_vgic_nested_hw_emulate(i, lr, l1_irq); irq->active = false; + } vgic_put_irq(vcpu->kvm, irq); } @@ -197,6 +205,9 @@ void vgic_v3_put_nested(struct kvm_vcpu *vcpu) __vgic_v3_save_state(vcpu_shadow_if(vcpu)); + trace_vgic_put_nested(vcpu, kvm_vgic_global_state.nr_lr, + vcpu_shadow_if(vcpu)->vgic_lr); + /* * Translate the shadow state HW fields back to the virtual ones * before copying the shadow struct back to the nested one. From patchwork Tue Jan 31 09:24:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 56624C636CD for ; Tue, 31 Jan 2023 09:53:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GWj8JRjJu6GeUj5V2nYHWAUMYm73ARtuMtjZWclUbX0=; b=tyEcpMo+TyGWcU XS9qCrBN5Mwmv01bw9VG9BDDs2ZJka2LxMQAQidFn+IWR4tmup7o9mc2xhxjBFzr5BviY3gQXaBwe +yp7THNW8hXoondEOJ9lXCyfZVBcScgGu0b4X7xo16Tbc7ewur9IiZwKYHeE18GmzcgdWj3abGwql BURHhSdU42vHZbgjYVENDlN9c2LrHVeag+IM6SsgI0RYg0W8pc2JKfusgv5EStB9sUa2JklauMV0x /5wRzXrHg9xYq4XhD4b5lWm5FpHYb+MVNsD3AAcabYQQLk4akhSa5fn5kCP+jDqL7MkrNlKD3n9W1 4ut6ADatuA77hvL3FWKQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnJB-0079ZH-T5; Tue, 31 Jan 2023 09:52:10 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8o-0075Ba-LM for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:28 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 7BC11CE1CCA; Tue, 31 Jan 2023 09:41:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AE505C433D2; Tue, 31 Jan 2023 09:41:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158081; bh=91p16N15uIviVqlWpa+2xR/bZv/r5Wt0gGrieUD+SGs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZTbDI2RxuZnQ1Gvup8ko3Ra5tWSWJGFDd52nFr5CI9Mq0HibGwiCjgpNRBOhVxLzI Hpt8GJ1stT6ChdRCZGH3Ryl98sB+oR7wGDGi53yQOz5dg9kMVEXdQGhUsyUAi/IAsc kV1y+7Kmm03pJhYLaQEh0Dyxiu56tlJ4BTngqS0Sx7xGvn+CDEoV1txt/Bd7QCOB/q wMaW58bP8ctlBRq8DkJN7X5Vlc3MkFj4TjbLdk74yoiSIH647J+qMi13tc3NaCqUo3 dlnppf9v5yUzK/PgFizVEJlP/+25vGhpI2hEBjgSEbyYQQxiMsCLL9MziaKQi+nZGk N9l+JHK/Mc0XA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtt-0067U2-FQ; Tue, 31 Jan 2023 09:26:01 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 54/69] KVM: arm64: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT Date: Tue, 31 Jan 2023 09:24:49 +0000 Message-Id: <20230131092504.2880505-55-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014127_230585_1B56F4D9 X-CRM114-Status: GOOD ( 15.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Since we're (almost) feature complete, let's allow userspace to request KVM_ARM_VCPU_NESTED_VIRT by bumping the KVM_VCPU_MAX_FEATURES up. We also now advertise the feature to userspace with a new capability. It's going to be great... Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kvm/arm.c | 3 +++ include/uapi/linux/kvm.h | 1 + 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 5db9e74b86f6..a1fc3b893af8 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -37,7 +37,7 @@ #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS -#define KVM_VCPU_MAX_FEATURES 7 +#define KVM_VCPU_MAX_FEATURES 8 #define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 505cb72d8535..bfcf6db27448 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -269,6 +269,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_ARM_EL1_32BIT: r = cpus_have_const_cap(ARM64_HAS_32BIT_EL1); break; + case KVM_CAP_ARM_EL2: + r = cpus_have_const_cap(ARM64_HAS_NESTED_VIRT); + break; case KVM_CAP_GUEST_DEBUG_HW_BPS: r = get_num_brps(); break; diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 55155e262646..573fa03bb002 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1175,6 +1175,7 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224 #define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225 +#define KVM_CAP_ARM_EL2 226 #ifdef KVM_CAP_IRQ_ROUTING From patchwork Tue Jan 31 09:24:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122764 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 01038C636CC for ; Tue, 31 Jan 2023 10:03:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xPcP9SdvFmQLFdtXXkfJLwQv11gxemtUS8/XzhEkL18=; b=wKdfs9acrujESG DHHvtFjwKrR/ZdOwX4qwSGlZGEyT6loRZgPHoU+37/vZgMFaQaGM4A6oInGrI1KWNWSSONw0CWOrE 7GDc8dRobnFWw+cS3Oho7jMKkMJ6nfz/T3ffG5sWQuaKa6lyq56bRKkbigbbjr7ZjygV/A23Ko9Il a+Efe0ECcWHnVlfYQGQt9pU0kfQe/7L7tABA0oZPNhnVcFPEK2TTI63p3eFB62lDCofWLBCczhKEF JHVmkRHuUu32Odpx0ZcjS7qdF/GPJViFU49sgOM6xP/KyJ70JCnsk7O2Sb+7Y7UlU/B8EJIGlowsB TbXSheeo579Ka2+X0qoA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnT2-007EBk-JC; Tue, 31 Jan 2023 10:02:21 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9R-0075Uy-7A for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:08 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id BB94CCE1BD9; Tue, 31 Jan 2023 09:42:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EE4BFC433EF; Tue, 31 Jan 2023 09:41:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158120; bh=3YAcWVZAjhAN6J7Oiw9EGD8rLRdIRPR56p+Gaauwfeo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RUAkMiMZawI5Q9oBIP/XlYnIto2EIwe/8hWMUWUW7nUR2Q63PV1mqSyQoISxvSxQ2 6iSaQ73xykoEeSXtJSpECKaW7xzs6V9f9Y3WJEE6J5aObUyT2n9PBuI7i3vkv6sFRi cc6pjNnDgbs+psexMQwxqok86U2ajJQTp6Qlt5mUL5vqJmNEd6UizdAs3HGZI3Vgid TAKHcyElpUbm8xCJKZB1RLRokouB4WPwtdr9WAlImSR0e+KZG5dew7MUlywMorUm+u t+Sho5fJEG/pl0K4TLYvJzy6GNeX6SBDaQjI5Kb408IpFp9/ogsWyzvKdxEN+aqjSi Xf9wV0B8Kcqow== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtt-0067U2-RF; Tue, 31 Jan 2023 09:26:01 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 55/69] KVM: arm64: nv: Add handling of FEAT_TTL TLB invalidation Date: Tue, 31 Jan 2023 09:24:50 +0000 Message-Id: <20230131092504.2880505-56-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014206_004887_655BEDF1 X-CRM114-Status: GOOD ( 19.68 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Support guest-provided information information to find out about the range of required invalidation. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 1 + arch/arm64/kvm/nested.c | 56 +++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 78 ++++++++++++++++++----------- 3 files changed, 107 insertions(+), 28 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 6e44a01403dc..c5b661517cb6 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -122,6 +122,7 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); extern bool forward_nv1_traps(struct kvm_vcpu *vcpu); +unsigned int ttl_to_size(u8 ttl); struct sys_reg_params; struct sys_reg_desc; diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 981a4ff75da6..8b22984a3112 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -357,6 +357,62 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, return ret; } +unsigned int ttl_to_size(u8 ttl) +{ + int level = ttl & 3; + int gran = (ttl >> 2) & 3; + unsigned int max_size = 0; + + switch (gran) { + case TLBI_TTL_TG_4K: + switch (level) { + case 0: + break; + case 1: + max_size = SZ_1G; + break; + case 2: + max_size = SZ_2M; + break; + case 3: + max_size = SZ_4K; + break; + } + break; + case TLBI_TTL_TG_16K: + switch (level) { + case 0: + case 1: + break; + case 2: + max_size = SZ_32M; + break; + case 3: + max_size = SZ_16K; + break; + } + break; + case TLBI_TTL_TG_64K: + switch (level) { + case 0: + case 1: + /* No 52bit IPA support */ + break; + case 2: + max_size = SZ_512M; + break; + case 3: + max_size = SZ_64K; + break; + } + break; + default: /* No size information */ + break; + } + + return max_size; +} + /* Must be called with kvm->mmu_lock held */ struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) { diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index f02cc109a767..08acb1f8d551 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -3002,14 +3002,54 @@ static bool handle_vmalls12e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, return true; } +static unsigned long compute_tlb_inval_range(struct kvm_vcpu *vcpu, + struct kvm_s2_mmu *mmu, + u64 val) +{ + unsigned long max_size; + u8 ttl = 0; + + if (cpus_have_const_cap(ARM64_HAS_ARMv8_4_TTL)) { + ttl = FIELD_GET(GENMASK_ULL(47, 44), val); + } + + max_size = ttl_to_size(ttl); + + if (!max_size) { + u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + + /* Compute the maximum extent of the invalidation */ + switch ((vtcr & VTCR_EL2_TG0_MASK)) { + case VTCR_EL2_TG0_4K: + max_size = SZ_1G; + break; + case VTCR_EL2_TG0_16K: + max_size = SZ_32M; + break; + case VTCR_EL2_TG0_64K: + /* + * No, we do not support 52bit IPA in nested yet. Once + * we do, this should be 4TB. + */ + /* FIXME: remove the 52bit PA support from the IDregs */ + max_size = SZ_512M; + break; + default: + BUG(); + } + } + + WARN_ON(!max_size); + return max_size; +} + static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); - u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); struct kvm_s2_mmu *mmu; u64 base_addr; - int max_size; + unsigned long max_size; if (vcpu_has_nv(vcpu) && forward_nv_traps(vcpu)) return false; @@ -3019,45 +3059,27 @@ static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, * * - NS bit: we're non-secure only. * - * - TTL field: We already have the granule size from the - * VTCR_EL2.TG0 field, and the level is only relevant to the - * guest's S2PT. - * * - IPA[51:48]: We don't support 52bit IPA just yet... * * And of course, adjust the IPA to be on an actual address. */ base_addr = (p->regval & GENMASK_ULL(35, 0)) << 12; - /* Compute the maximum extent of the invalidation */ - switch ((vtcr & VTCR_EL2_TG0_MASK)) { - case VTCR_EL2_TG0_4K: - max_size = SZ_1G; - break; - case VTCR_EL2_TG0_16K: - max_size = SZ_32M; - break; - case VTCR_EL2_TG0_64K: - /* - * No, we do not support 52bit IPA in nested yet. Once - * we do, this should be 4TB. - */ - /* FIXME: remove the 52bit PA support from the IDregs */ - max_size = SZ_512M; - break; - default: - BUG(); - } - write_lock(&vcpu->kvm->mmu_lock); mmu = lookup_s2_mmu(vcpu->kvm, vttbr, HCR_VM); - if (mmu) + if (mmu) { + max_size = compute_tlb_inval_range(vcpu, mmu, p->regval); + base_addr &= ~(max_size - 1); kvm_unmap_stage2_range(mmu, base_addr, max_size); + } mmu = lookup_s2_mmu(vcpu->kvm, vttbr, 0); - if (mmu) + if (mmu) { + max_size = compute_tlb_inval_range(vcpu, mmu, p->regval); + base_addr &= ~(max_size - 1); kvm_unmap_stage2_range(mmu, base_addr, max_size); + } write_unlock(&vcpu->kvm->mmu_lock); From patchwork Tue Jan 31 09:24:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C699BC636D7 for ; Tue, 31 Jan 2023 10:14:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yW8DMM7lQDsvDDxfZ59ltabI1sWt5L7nk9EavOFPlxU=; b=B8Dyhqo8dZ3kj9 WyMEd/W37HFXR+nkiQPQkPDr09ozaK44p1jJn15B6dqeqLzfaPTra/Ih6Zf/HMN3Yjkx9d5xo6xFi Vnzxn/zI6UIXwBCbFfEpRCJeHReYTp+/0q7OMN7fK2Er7a6iA2db0Odft+Hr+Eswz/mi3UEx2ixAu moSc32cE3QzQ5UoHubZ2QOS6GvN0mqpxqY8NZsFnbBpc2trPiX8gyyfyl/5ZhvaruzkAj6jw5CIud 3snTgHnn75sGjP+vqr/5H9dzQ6TVbtIL4mA9uaSk7fzsFA0ySggw33EiFChooZAKSZPNRs55n1LHN 3m/bCoqFX2Ie5n1ej02Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMndH-007Iju-0u; Tue, 31 Jan 2023 10:12:55 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9k-0075dn-80 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:27 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 78E5F61481; Tue, 31 Jan 2023 09:42:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E218DC433D2; Tue, 31 Jan 2023 09:42:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158142; bh=afC5jtCFVZQy5hocE46xdocxPGsbQ2dJvCDg6ApT0NM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JmPgFlZ/4ZiEvEsCC4rHS/gzmK83mZVEdODQSkKypWdOs/RNdIHTFVWyWLCLD0A5d JAS7lcfhhtc0xlyut7/JFvI4Ze+9AQDNXY8WXBWlGhd+3rWyhOMeftk3vh8G0MffhX CVfGqvaC1AwPr1Wga0csaLyEMGltyDPHWABTbRAQYN6UfuzJLWiKgZ0sxH4Lm25c3f hcupcDUg6sK6JEQguuq7KhXgBAEFwizH3Ue3G+JAszjkSPtDJaWVy3Abf2ASvN6lbe h6fwbE1kyL8OmNMi9LqcQNoHQN6Ox7wq0d2l9vpIv8b8Uz/8m0wo3L65S86LyiUhzQ NnfLK41CIgwPg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtu-0067U2-4v; Tue, 31 Jan 2023 09:26:02 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 56/69] KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like information Date: Tue, 31 Jan 2023 09:24:51 +0000 Message-Id: <20230131092504.2880505-57-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014224_503279_C46FD77F X-CRM114-Status: GOOD ( 23.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to be able to make S2 TLB invalidations more performant on NV, let's use a scheme derived from the ARMv8.4 TTL extension. If bits [56:55] in the descriptor are non-zero, they indicate a level which can be used as an invalidation range. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 4 ++ arch/arm64/kvm/nested.c | 99 +++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 9 ++- 3 files changed, 109 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index c5b661517cb6..63bb4898e2d9 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -122,6 +122,8 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); extern bool forward_nv1_traps(struct kvm_vcpu *vcpu); +u8 get_guest_mapping_ttl(struct kvm_vcpu *vcpu, struct kvm_s2_mmu *mmu, + u64 addr); unsigned int ttl_to_size(u8 ttl); struct sys_reg_params; @@ -130,4 +132,6 @@ struct sys_reg_desc; void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, const struct sys_reg_desc *r); +#define KVM_NV_GUEST_MAP_SZ (KVM_PGTABLE_PROT_SW1 | KVM_PGTABLE_PROT_SW0) + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 8b22984a3112..f93f4248c6f3 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -4,6 +4,7 @@ * Author: Jintack Lim */ +#include #include #include @@ -357,6 +358,30 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, return ret; } +static int read_host_s2_desc(phys_addr_t pa, u64 *desc, void *data) +{ + u64 *va = phys_to_virt(pa); + + *desc = *va; + + return 0; +} + +static int kvm_walk_shadow_s2(struct kvm_s2_mmu *mmu, phys_addr_t gipa, + struct kvm_s2_trans *result) +{ + struct s2_walk_info wi = { }; + + wi.read_desc = read_host_s2_desc; + wi.baddr = mmu->pgd_phys; + + vtcr_to_walk_info(mmu->arch->vtcr, &wi); + + wi.be = IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + + return walk_nested_s2_pgd(gipa, &wi, result); +} + unsigned int ttl_to_size(u8 ttl) { int level = ttl & 3; @@ -413,6 +438,80 @@ unsigned int ttl_to_size(u8 ttl) return max_size; } +/* + * Compute the equivalent of the TTL field by parsing the shadow PT. + * The granule size is extracted from VTCR_EL2.TG0 while the level is + * retrieved from first entry carrying the level as a tag. + */ +u8 get_guest_mapping_ttl(struct kvm_vcpu *vcpu, struct kvm_s2_mmu *mmu, + u64 addr) +{ + u64 tmp, sz = 0, vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + struct kvm_s2_trans out; + u8 ttl, level; + + switch (vtcr & VTCR_EL2_TG0_MASK) { + case VTCR_EL2_TG0_4K: + ttl = (1 << 2); + break; + case VTCR_EL2_TG0_16K: + ttl = (2 << 2); + break; + case VTCR_EL2_TG0_64K: + ttl = (3 << 2); + break; + default: + BUG(); + } + + tmp = addr; + +again: + /* Iteratively compute the block sizes for a particular granule size */ + switch (vtcr & VTCR_EL2_TG0_MASK) { + case VTCR_EL2_TG0_4K: + if (sz < SZ_4K) sz = SZ_4K; + else if (sz < SZ_2M) sz = SZ_2M; + else if (sz < SZ_1G) sz = SZ_1G; + else sz = 0; + break; + case VTCR_EL2_TG0_16K: + if (sz < SZ_16K) sz = SZ_16K; + else if (sz < SZ_32M) sz = SZ_32M; + else sz = 0; + break; + case VTCR_EL2_TG0_64K: + if (sz < SZ_64K) sz = SZ_64K; + else if (sz < SZ_512M) sz = SZ_512M; + else sz = 0; + break; + default: + BUG(); + } + + if (sz == 0) + return 0; + + tmp &= ~(sz - 1); + out = (struct kvm_s2_trans) { }; + kvm_walk_shadow_s2(mmu, tmp, &out); + level = FIELD_GET(KVM_NV_GUEST_MAP_SZ, out.upper_attr); + if (!level) + goto again; + + ttl |= level; + + /* + * We now have found some level information in the shadow S2. Check + * that the resulting range is actually including the original IPA. + */ + sz = ttl_to_size(ttl); + if (addr < (tmp + sz)) + return ttl; + + return 0; +} + /* Must be called with kvm->mmu_lock held */ struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) { diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 08acb1f8d551..8f8e1afeae8c 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -3007,10 +3007,13 @@ static unsigned long compute_tlb_inval_range(struct kvm_vcpu *vcpu, u64 val) { unsigned long max_size; - u8 ttl = 0; + u8 ttl; - if (cpus_have_const_cap(ARM64_HAS_ARMv8_4_TTL)) { - ttl = FIELD_GET(GENMASK_ULL(47, 44), val); + ttl = FIELD_GET(GENMASK_ULL(47, 44), val); + + if (!(cpus_have_const_cap(ARM64_HAS_ARMv8_4_TTL) && ttl)) { + u64 addr = (val & GENMASK_ULL(35, 0)) << 12; + ttl = get_guest_mapping_ttl(vcpu, mmu, addr); } max_size = ttl_to_size(ttl); From patchwork Tue Jan 31 09:24:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 39C14C636CD for ; Tue, 31 Jan 2023 09:44:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2aS7dad75+xJqRUvKbNLIPc6kkdsx95wxVs1ot6F7A4=; b=UvyrvWCqp+7/To ioh0H7sD3wpIyAltWfaAJ5oa7kkILHJsQhCYMf5Wydg1LFWOADt/cK/RMV2DObIQ2EimIVfaMFbRX wxghK1p9kLZZ874g4IJzGgszTvaAseuk1Tl4XJKBBxW78KBhMGSGEnbGhxE6MMwmW0M3uuEKCJIRZ 6ZWttZQnlLr5vsfkmJFLTiSr0J7OUpL8yZA8w/LfxYJSgXH9UwwAQAgs0gqNBXV+OcwvUmoVsFPs0 CHUv4UnL4u9pVm/fFpFvpGxsuPP2orzysz3eMeMFvA0UQnLJAHIJ+o4yNurJx+t59ktK+CtDa8KMk 2J2EywtEWdq8H0zP37BA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnBB-0076Np-Dl; Tue, 31 Jan 2023 09:43:53 +0000 Received: from sin.source.kernel.org ([2604:1380:40e1:4800::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8F-0074sR-Vy for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:40:54 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 559ACCE1C83; Tue, 31 Jan 2023 09:40:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 86FC6C4339B; Tue, 31 Jan 2023 09:40:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158044; bh=oiUBW/YD2grtuGltNhErJ48Wq1qmFw6GmVI9itZ6Bk0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VDQc49UCIdgkS+BM1njCDtQa8hfIk4rwIawdibXG5fgvd4H2TzoYlyHuR77kbf2Wp w/M1plcy19lsa81U8L7SAhpJuKKs4jGrk4c76jrUdyBEp9zG+6UC8X+8B4ghGDXJWo 2BTxyDZCnr27v8JU9L26pQvMyPvIUkfQY7inftXMlsL4idlsasmSx3/oJd+2FgNzKE xzDDJitdgXZZX81oruCsbwT0RtfGUqOr1VJ5mKkfk315nDgYgjWmTXuyP3crlweJ0A 43Y8mPJnKpvv3xIO6rib+gD7jU49YYjgcV+KW/0FSsmYerwI5v7PkJOx5MoNFM2TsU Rwn+EEqm2VMIQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtu-0067U2-ED; Tue, 31 Jan 2023 09:26:02 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 57/69] KVM: arm64: nv: Tag shadow S2 entries with nested level Date: Tue, 31 Jan 2023 09:24:52 +0000 Message-Id: <20230131092504.2880505-58-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014052_955681_3F886624 X-CRM114-Status: GOOD ( 18.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Populate bits [56:55] of the leaf entry with the level provided by the guest's S2 translation. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 7 +++++++ arch/arm64/kvm/mmu.c | 16 ++++++++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 63bb4898e2d9..5dd6b12b6ec0 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -5,6 +5,8 @@ #include #include +#include + static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu) { return (!__is_defined(__KVM_NVHE_HYPERVISOR__) && @@ -134,4 +136,9 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, #define KVM_NV_GUEST_MAP_SZ (KVM_PGTABLE_PROT_SW1 | KVM_PGTABLE_PROT_SW0) +static inline u64 kvm_encode_nested_level(struct kvm_s2_trans *trans) +{ + return FIELD_PREP(KVM_NV_GUEST_MAP_SZ, trans->level); +} + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 4680a703847b..b473a135732a 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1434,11 +1434,17 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * Potentially reduce shadow S2 permissions to match the guest's own * S2. For exec faults, we'd only reach this point if the guest * actually allowed it (see kvm_s2_handle_perm_fault). + * + * Also encode the level of the nested translation in the SW bits of + * the PTE/PMD/PUD. This will be retrived on TLB invalidation from + * the guest. */ if (nested) { writable &= kvm_s2_trans_writable(nested); if (!kvm_s2_trans_readable(nested)) prot &= ~KVM_PGTABLE_PROT_R; + + prot |= kvm_encode_nested_level(nested); } read_lock(&kvm->mmu_lock); @@ -1487,14 +1493,20 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * permissions only if vma_pagesize equals fault_granule. Otherwise, * kvm_pgtable_stage2_map() should be called to change block size. */ - if (fault_status == ESR_ELx_FSC_PERM && vma_pagesize == fault_granule) + if (fault_status == ESR_ELx_FSC_PERM && vma_pagesize == fault_granule) { + /* + * Drop the SW bits in favour of those stored in the + * PTE, which will be preserved. + */ + prot &= ~KVM_NV_GUEST_MAP_SZ; ret = kvm_pgtable_stage2_relax_perms(pgt, fault_ipa, prot); - else + } else { ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize, __pfn_to_phys(pfn), prot, memcache, KVM_PGTABLE_WALK_HANDLE_FAULT | KVM_PGTABLE_WALK_SHARED); + } /* Mark the page dirty only if the fault is handled successfully */ if (writable && !ret) { From patchwork Tue Jan 31 09:24:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122724 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF87DC636CC for ; Tue, 31 Jan 2023 09:49:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=s4CrbQo5NQ5SWlhpvl+SsNmUPh1EOpQJFowm0l9+HWU=; b=akvfgxLCtQSpNe U5uFP/IphnLfyiJPkezcdJHIRTrf9kB5hTl2ftou0vV3LA7ebTKBo8LYrwWaoPu1XPj4p61Vsn5l9 qn8LzH5Aeb0rN2Mehiu9GSuTI5hNDSdpuQzI9vyRJHJgJpO7HRpeKw3jW8x5qkiskBmJjkjc7ibke 3AYCPDehCxD8c3OXeTqjIVYwEXQHqRsVJcNRl21LAmaZgFlxMUEtinhpD8nf8EajIC/K1F/ZX/PGc p0t75CPGxxlRq4ANZ1Bwc8NOWTR2v6Fr4STUMBrnV8n6pbMx2JMjQJxD6rb+xoI7Yl/ylNzECrtMn sneDXR3PNDfWhqgBQ3Sw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnFF-00784E-Rt; Tue, 31 Jan 2023 09:48:06 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8V-00752d-B6 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:09 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E5797B81AC8; Tue, 31 Jan 2023 09:41:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9D6DBC4339C; Tue, 31 Jan 2023 09:41:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158064; bh=PHs4gaxAaWQmUmtjbLODLMqbuuO3t58QTS/ww8BSn6A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PqEnly6VtSah8bKiZGvHJpsabc1azUQaEDAW0DpKRc/XOrQ8498ux+4f+ql6qYNqA f1yx69dJv9gymHGf9pvVWFSZsmvHOVX0GMhpBpgHfy0+6LUqj7xcTKylt9j+ki8ujI V9NDH+FTauWviNn5jKOghjDWlvfIozA2ZifFgemXrZXCN1sO5EkjgqlsBRCZBbQAv9 DPPLlroUv38ckNwlkJYFniYImVDoTOlpgX0lFjYSSBa4Ns+k9tMy50g+SP8e5TR0or HPofpeNoVkYTXUu0/dmBeuzCk2OZie3FM3vh1JEd0NrZ6hOhSutWNZb7lC7N/VbidD TyUaShca6FlCQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtu-0067U2-N0; Tue, 31 Jan 2023 09:26:02 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 58/69] KVM: arm64: nv: Add include containing the VNCR_EL2 offsets Date: Tue, 31 Jan 2023 09:24:53 +0000 Message-Id: <20230131092504.2880505-59-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014107_734966_BA300E50 X-CRM114-Status: GOOD ( 14.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org VNCR_EL2 points to a page containing a number of system registers accessed by a guest hypervisor when ARMv8.4-NV is enabled. Let's document the offsets in that page, as we are going to use this layout. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/vncr_mapping.h | 74 +++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 arch/arm64/include/asm/vncr_mapping.h diff --git a/arch/arm64/include/asm/vncr_mapping.h b/arch/arm64/include/asm/vncr_mapping.h new file mode 100644 index 000000000000..c0a2acd5045c --- /dev/null +++ b/arch/arm64/include/asm/vncr_mapping.h @@ -0,0 +1,74 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * System register offsets in the VNCR page + * All offsets are *byte* displacements! + */ + +#ifndef __ARM64_VNCR_MAPPING_H__ +#define __ARM64_VNCR_MAPPING_H__ + +#define VNCR_VTTBR_EL2 0x020 +#define VNCR_VTCR_EL2 0x040 +#define VNCR_VMPIDR_EL2 0x050 +#define VNCR_CNTVOFF_EL2 0x060 +#define VNCR_HCR_EL2 0x078 +#define VNCR_HSTR_EL2 0x080 +#define VNCR_VPIDR_EL2 0x088 +#define VNCR_TPIDR_EL2 0x090 +#define VNCR_VNCR_EL2 0x0B0 +#define VNCR_CPACR_EL1 0x100 +#define VNCR_CONTEXTIDR_EL1 0x108 +#define VNCR_SCTLR_EL1 0x110 +#define VNCR_ACTLR_EL1 0x118 +#define VNCR_TCR_EL1 0x120 +#define VNCR_AFSR0_EL1 0x128 +#define VNCR_AFSR1_EL1 0x130 +#define VNCR_ESR_EL1 0x138 +#define VNCR_MAIR_EL1 0x140 +#define VNCR_AMAIR_EL1 0x148 +#define VNCR_MDSCR_EL1 0x158 +#define VNCR_SPSR_EL1 0x160 +#define VNCR_CNTV_CVAL_EL0 0x168 +#define VNCR_CNTV_CTL_EL0 0x170 +#define VNCR_CNTP_CVAL_EL0 0x178 +#define VNCR_CNTP_CTL_EL0 0x180 +#define VNCR_SCXTNUM_EL1 0x188 +#define VNCR_TFSR_EL1 0x190 +#define VNCR_ZCR_EL1 0x1E0 +#define VNCR_TTBR0_EL1 0x200 +#define VNCR_TTBR1_EL1 0x210 +#define VNCR_FAR_EL1 0x220 +#define VNCR_ELR_EL1 0x230 +#define VNCR_SP_EL1 0x240 +#define VNCR_VBAR_EL1 0x250 +#define VNCR_ICH_LR0_EL2 0x400 +// VNCR_ICH_LRN_EL2(n) VNCR_ICH_LR0_EL2+8*((n) & 7) +#define VNCR_ICH_AP0R0_EL2 0x480 +// VNCR_ICH_AP0RN_EL2(n) VNCR_ICH_AP0R0_EL2+8*((n) & 3) +#define VNCR_ICH_AP1R0_EL2 0x4A0 +// VNCR_ICH_AP1RN_EL2(n) VNCR_ICH_AP1R0_EL2+8*((n) & 3) +#define VNCR_ICH_HCR_EL2 0x4C0 +#define VNCR_ICH_VMCR_EL2 0x4C8 +#define VNCR_VDISR_EL2 0x500 +#define VNCR_PMBLIMITR_EL1 0x800 +#define VNCR_PMBPTR_EL1 0x810 +#define VNCR_PMBSR_EL1 0x820 +#define VNCR_PMSCR_EL1 0x828 +#define VNCR_PMSEVFR_EL1 0x830 +#define VNCR_PMSICR_EL1 0x838 +#define VNCR_PMSIRR_EL1 0x840 +#define VNCR_PMSLATFR_EL1 0x848 +#define VNCR_TRFCR_EL1 0x880 +#define VNCR_MPAM1_EL1 0x900 +#define VNCR_MPAMHCR_EL2 0x930 +#define VNCR_MPAMVPMV_EL2 0x938 +#define VNCR_MPAMVPM0_EL2 0x940 +#define VNCR_MPAMVPM1_EL2 0x948 +#define VNCR_MPAMVPM2_EL2 0x950 +#define VNCR_MPAMVPM3_EL2 0x958 +#define VNCR_MPAMVPM4_EL2 0x960 +#define VNCR_MPAMVPM5_EL2 0x968 +#define VNCR_MPAMVPM6_EL2 0x970 +#define VNCR_MPAMVPM7_EL2 0x978 + +#endif /* __ARM64_VNCR_MAPPING_H__ */ From patchwork Tue Jan 31 09:24:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122775 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 76E9AC38142 for ; Tue, 31 Jan 2023 10:06:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=KglCmqVtGyHApL8DEZmGAdsNMMtwFUjo9EzyO5+V64Y=; b=bn3xpmi1+kdxEL zhQN7ikPgIjZH4GXHBa7naGow8z5vd4Z32kQQndzrUe8RFf+4LNmNufqv+m+hkKrzedxhHfX1i+wr MRoftiuXfktGOxlw4ljlhlHG+MLhE3nCvZFXjmOBLB9UgLoM2oTg0VTMij1s0nK8qYqNP3kMz23H2 lpS54DxROgOIVEN9B/rWPjkYwuU/XBIBg40CclbRBAP+Q+ay6DJskDw8lrC/cqbpRY9dKFgVXYdmo JtfXktzaHlfJ2af78aFRFIx8nPg3Pb13dlaZknFAZmhj4m3uID9wMpQT2rsEJ1FBMuqBthm6/hhe7 sU4LtrY+7bE584hW8g0Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnVG-007FBx-6I; Tue, 31 Jan 2023 10:04:38 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9V-0075WS-J0 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:12 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 30A70B81AF5; Tue, 31 Jan 2023 09:42:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B3581C433EF; Tue, 31 Jan 2023 09:42:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158126; bh=vfQcisVUfTNZ+XKnX7cBnUV8pycWE/NPonanGcXhsp8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tcESIQVhPFNu2DoOzPrEl4R/63x8a52JeZOMd1IUDBUx6UEpnkVtcMl9NV4ayyZHS GHclYWSMbzejrtp1ljlVzHjMopbYuLeXTQ0dWe4DPYK2ARBw0frlmOX1jNO3CYoBpp LbNMZtx40DkJUrKMgoklo0EZbj8K5o3uZzG2bwR6XBeAtHsq/bgRs4+vVH2lkKjC1G n6VkyMXUTrzDv1y4jHcqI+jTU9/iDFs+5IgKZbxWt4RsTWXL9UfETT/mBW6feRospA VqUS0XvDbb4hk4fcxw6jHCB5UPPnWgvmof0wueDbD+Hxm5pfTEzO9yU2NdJQfZtcVB ez/nLfBOyF4FA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtu-0067U2-Vs; Tue, 31 Jan 2023 09:26:03 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 59/69] KVM: arm64: nv: Map VNCR-capable registers to a separate page Date: Tue, 31 Jan 2023 09:24:54 +0000 Message-Id: <20230131092504.2880505-60-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014209_972179_0802841D X-CRM114-Status: GOOD ( 22.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org With ARMv8.4-NV, registers that can be directly accessed in memory by the guest have to live at architected offsets in a special page. Let's annotate the sysreg enum to reflect the offset at which they are in this page, whith a little twist: If running on HW that doesn't have the ARMv8.4-NV feature, or even a VM that doesn't use NV, we store all the system registers in the usual sys_regs array. The only difference with the pre-8.4 situation is that VNCR-capable registers are at a "similar" offset as in the VNCR page (we can compute the actual offset at compile time), and that the sys_regs array is both bigger and sparse. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 104 ++++++++++++++++++++---------- arch/arm64/tools/cpucaps | 1 + 2 files changed, 70 insertions(+), 35 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index a1fc3b893af8..f58e953d6845 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -26,6 +26,7 @@ #include #include #include +#include #define __KVM_HAVE_ARCH_INTC_INITIALIZED @@ -290,32 +291,33 @@ struct kvm_vcpu_fault_info { u64 disr_el1; /* Deferred [SError] Status Register */ }; +/* + * VNCR() just places the VNCR_capable registers in the enum after + * __VNCR_START__, and the value (after correction) to be an 8-byte offset + * from the VNCR base. As we don't require the enum to be otherwise ordered, + * we need the terrible hack below to ensure that we correctly size the + * sys_regs array, no matter what. + * + * The __MAX__ macro has been lifted from Sean Eron Anderson's wonderful + * treasure trove of bit hacks: + * https://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax + */ +#define __MAX__(x,y) ((x) ^ (((x) ^ (y)) & -((x) < (y)))) +#define VNCR(r) \ + __before_##r, \ + r = __VNCR_START__ + ((VNCR_ ## r) / 8), \ + __after_##r = __MAX__(__before_##r - 1, r) + enum vcpu_sysreg { __INVALID_SYSREG__, /* 0 is reserved as an invalid value */ MPIDR_EL1, /* MultiProcessor Affinity Register */ CLIDR_EL1, /* Cache Level ID Register */ CSSELR_EL1, /* Cache Size Selection Register */ - SCTLR_EL1, /* System Control Register */ - ACTLR_EL1, /* Auxiliary Control Register */ - CPACR_EL1, /* Coprocessor Access Control */ - ZCR_EL1, /* SVE Control */ - TTBR0_EL1, /* Translation Table Base Register 0 */ - TTBR1_EL1, /* Translation Table Base Register 1 */ - TCR_EL1, /* Translation Control Register */ - ESR_EL1, /* Exception Syndrome Register */ - AFSR0_EL1, /* Auxiliary Fault Status Register 0 */ - AFSR1_EL1, /* Auxiliary Fault Status Register 1 */ - FAR_EL1, /* Fault Address Register */ - MAIR_EL1, /* Memory Attribute Indirection Register */ - VBAR_EL1, /* Vector Base Address Register */ - CONTEXTIDR_EL1, /* Context ID Register */ TPIDR_EL0, /* Thread ID, User R/W */ TPIDRRO_EL0, /* Thread ID, User R/O */ TPIDR_EL1, /* Thread ID, Privileged */ - AMAIR_EL1, /* Aux Memory Attribute Indirection Register */ CNTKCTL_EL1, /* Timer Control Register (EL1) */ PAR_EL1, /* Physical Address Register */ - MDSCR_EL1, /* Monitor Debug System Control Register */ MDCCINT_EL1, /* Monitor Debug Comms Channel Interrupt Enable Reg */ OSLSR_EL1, /* OS Lock Status Register */ DISR_EL1, /* Deferred Interrupt Status Register */ @@ -346,20 +348,9 @@ enum vcpu_sysreg { APGAKEYLO_EL1, APGAKEYHI_EL1, - ELR_EL1, - SP_EL1, - SPSR_EL1, - - CNTVOFF_EL2, - CNTV_CVAL_EL0, - CNTV_CTL_EL0, - CNTP_CVAL_EL0, - CNTP_CTL_EL0, - /* Memory Tagging Extension registers */ RGSR_EL1, /* Random Allocation Tag Seed Register */ GCR_EL1, /* Tag Control Register */ - TFSR_EL1, /* Tag Fault Status Register (EL1) */ TFSRE0_EL1, /* Tag Fault Status Register (EL0) */ /* 32bit specific registers. */ @@ -369,20 +360,14 @@ enum vcpu_sysreg { DBGVCR32_EL2, /* Debug Vector Catch Register */ /* EL2 registers */ - VPIDR_EL2, /* Virtualization Processor ID Register */ - VMPIDR_EL2, /* Virtualization Multiprocessor ID Register */ SCTLR_EL2, /* System Control Register (EL2) */ ACTLR_EL2, /* Auxiliary Control Register (EL2) */ - HCR_EL2, /* Hypervisor Configuration Register */ MDCR_EL2, /* Monitor Debug Configuration Register (EL2) */ CPTR_EL2, /* Architectural Feature Trap Register (EL2) */ - HSTR_EL2, /* Hypervisor System Trap Register */ HACR_EL2, /* Hypervisor Auxiliary Control Register */ TTBR0_EL2, /* Translation Table Base Register 0 (EL2) */ TTBR1_EL2, /* Translation Table Base Register 1 (EL2) */ TCR_EL2, /* Translation Control Register (EL2) */ - VTTBR_EL2, /* Virtualization Translation Table Base Register */ - VTCR_EL2, /* Virtualization Translation Control Register */ SPSR_EL2, /* EL2 saved program status register */ ELR_EL2, /* EL2 exception link register */ AFSR0_EL2, /* Auxiliary Fault Status Register 0 (EL2) */ @@ -395,7 +380,6 @@ enum vcpu_sysreg { VBAR_EL2, /* Vector Base Address Register (EL2) */ RVBAR_EL2, /* Reset Vector Base Address Register */ CONTEXTIDR_EL2, /* Context ID Register (EL2) */ - TPIDR_EL2, /* EL2 Software Thread ID Register */ CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */ SP_EL2, /* EL2 Stack Pointer */ CNTHP_CTL_EL2, @@ -403,6 +387,42 @@ enum vcpu_sysreg { CNTHV_CTL_EL2, CNTHV_CVAL_EL2, + __VNCR_START__, /* Any VNCR-capable reg goes after this point */ + + VNCR(SCTLR_EL1),/* System Control Register */ + VNCR(ACTLR_EL1),/* Auxiliary Control Register */ + VNCR(CPACR_EL1),/* Coprocessor Access Control */ + VNCR(ZCR_EL1), /* SVE Control */ + VNCR(TTBR0_EL1),/* Translation Table Base Register 0 */ + VNCR(TTBR1_EL1),/* Translation Table Base Register 1 */ + VNCR(TCR_EL1), /* Translation Control Register */ + VNCR(ESR_EL1), /* Exception Syndrome Register */ + VNCR(AFSR0_EL1),/* Auxiliary Fault Status Register 0 */ + VNCR(AFSR1_EL1),/* Auxiliary Fault Status Register 1 */ + VNCR(FAR_EL1), /* Fault Address Register */ + VNCR(MAIR_EL1), /* Memory Attribute Indirection Register */ + VNCR(VBAR_EL1), /* Vector Base Address Register */ + VNCR(CONTEXTIDR_EL1), /* Context ID Register */ + VNCR(AMAIR_EL1),/* Aux Memory Attribute Indirection Register */ + VNCR(MDSCR_EL1),/* Monitor Debug System Control Register */ + VNCR(ELR_EL1), + VNCR(SP_EL1), + VNCR(SPSR_EL1), + VNCR(TFSR_EL1), /* Tag Fault Status Register (EL1) */ + VNCR(VPIDR_EL2),/* Virtualization Processor ID Register */ + VNCR(VMPIDR_EL2),/* Virtualization Multiprocessor ID Register */ + VNCR(HCR_EL2), /* Hypervisor Configuration Register */ + VNCR(HSTR_EL2), /* Hypervisor System Trap Register */ + VNCR(VTTBR_EL2),/* Virtualization Translation Table Base Register */ + VNCR(VTCR_EL2), /* Virtualization Translation Control Register */ + VNCR(TPIDR_EL2),/* EL2 Software Thread ID Register */ + + VNCR(CNTVOFF_EL2), + VNCR(CNTV_CVAL_EL0), + VNCR(CNTV_CTL_EL0), + VNCR(CNTP_CVAL_EL0), + VNCR(CNTP_CTL_EL0), + NR_SYS_REGS /* Nothing after this line! */ }; @@ -419,6 +439,9 @@ struct kvm_cpu_context { u64 sys_regs[NR_SYS_REGS]; struct kvm_vcpu *__hyp_running_vcpu; + + /* This pointer has to be 4kB aligned. */ + u64 *vncr_array; }; struct kvm_host_data { @@ -760,8 +783,19 @@ struct kvm_vcpu_arch { * accessed by a running VCPU. For example, for userspace access or * for system registers that are never context switched, but only * emulated. + * + * Don't bother with VNCR-based accesses in the nVHE code, it has no + * business dealing with NV. */ -#define __ctxt_sys_reg(c,r) (&(c)->sys_regs[(r)]) +static inline u64 *__ctxt_sys_reg(const struct kvm_cpu_context *ctxt, int r) +{ +#if !defined (__KVM_NVHE_HYPERVISOR__) + if (unlikely(cpus_have_final_cap(ARM64_HAS_ENHANCED_NESTED_VIRT) && + r >= __VNCR_START__ && ctxt->vncr_array)) + return &ctxt->vncr_array[r - __VNCR_START__]; +#endif + return (u64 *)&ctxt->sys_regs[r]; +} #define ctxt_sys_reg(c,r) (*__ctxt_sys_reg(c,r)) diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 1af77a3657f7..e17d0f8f4182 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -23,6 +23,7 @@ HAS_DCPOP HAS_DIT HAS_E0PD HAS_ECV +HAS_ENHANCED_NESTED_VIRT HAS_EPAN HAS_GENERIC_AUTH HAS_GENERIC_AUTH_ARCH_QARMA3 From patchwork Tue Jan 31 09:24:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122824 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 823F6C636CC for ; Tue, 31 Jan 2023 10:23:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=pZB2mN1NxtcXueUmXKa0zOORaw4VMnvSxd4s85AA0I4=; b=1INBIt7SjZOQ6j ncUmuUYMD+Qjc5jl44Gio1yHTXZdXrh7H8b73kLfvRBvINKJ6dSActpfJ/nnqnY6nL3ZX2XKldQ+n 7zGVzw/KKxdrTDO9naUOeXii1dOe89Mb3ZRZKWzP5mDj+xTdLS7gSGSNaa1fKjXRp+n2Lqat8Gqjj ZHahih7ksNjA1AZPWKht5BKBAPTcZraAgSb0EMMfeDCtDHZVlCgxlLQuQzhX7/Di+fZI7bGWeidf6 SLv52lp1VVvMTNKSeyz+yDUSoOO1mK4eo8a0AKrE7BaA2WZ5yJr5rj+QmrRW83RJ2Z9Bp9CEXJnOL TYK7MhFtAD/9kzK0WZlg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnm0-007Mbe-V5; Tue, 31 Jan 2023 10:21:57 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnAA-0075rl-6u for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:52 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B7A276148B; Tue, 31 Jan 2023 09:42:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0767CC433D2; Tue, 31 Jan 2023 09:42:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158169; bh=Y+tV/BlnxtDG516mV+rfhjJ7z/a0Mppn1SSIXwFyQ0c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hiEi+9R5CQNRPlKHYSyyTipp9cGFsxsEJ56WNhAtcBusBlcagv3ZSC5rOmkTa0kZl MrMj1vN12l6eBk+tEzH231/cW7l23Vu+cUZsVYMzo4cWjhi/hWa48T/RYJtxXtQ24H Y4sqHN/kSDz3C/XAUbrU+s85APunptcq8N6vTTwggFYZmPZtaV8gPF3rUCcXSUJYk8 PPT7Fu7Ba+JLe4jTMViZJe64dKqJU3zalv7Y/QCJaIFgnXTFF4olyagu0IK2H5ytwh GTS0Vq2ScdfXOmiWjRdT09ITo9MkOemWp+p3gjKLJBDtdD8CKwUCcycmyLwAGxQkU7 Dz9z7BTZNY6Pg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtv-0067U2-C7; Tue, 31 Jan 2023 09:26:03 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 60/69] KVM: arm64: nv: Move nested vgic state into the sysreg file Date: Tue, 31 Jan 2023 09:24:55 +0000 Message-Id: <20230131092504.2880505-61-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014250_404645_E79CFF3D X-CRM114-Status: GOOD ( 25.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The vgic nested state needs to be accessible from the VNCR page, and thus needs to be part of the normal sysreg file. Let's move it there. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 9 +++ arch/arm64/kvm/sys_regs.c | 53 +++++++++++------ arch/arm64/kvm/vgic/vgic-v3-nested.c | 88 ++++++++++++++-------------- arch/arm64/kvm/vgic/vgic-v3.c | 17 ++++-- arch/arm64/kvm/vgic/vgic.h | 10 ++++ include/kvm/arm_vgic.h | 7 --- 6 files changed, 110 insertions(+), 74 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index f58e953d6845..62a0c79d0918 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -423,6 +423,15 @@ enum vcpu_sysreg { VNCR(CNTP_CVAL_EL0), VNCR(CNTP_CTL_EL0), + VNCR(ICH_LR0_EL2), + ICH_LR15_EL2 = ICH_LR0_EL2 + 15, + VNCR(ICH_AP0R0_EL2), + ICH_AP0R3_EL2 = ICH_AP0R0_EL2 + 3, + VNCR(ICH_AP1R0_EL2), + ICH_AP1R3_EL2 = ICH_AP1R0_EL2 + 3, + VNCR(ICH_HCR_EL2), + VNCR(ICH_VMCR_EL2), + NR_SYS_REGS /* Nothing after this line! */ }; diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 8f8e1afeae8c..1a1ae7ff218e 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2123,17 +2123,17 @@ static bool access_gic_apr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; - u32 index, *base; + u64 *base; + u8 index; index = r->Op2; if (r->CRm == 8) - base = cpu_if->vgic_ap0r; + base = __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_AP0R0_EL2); else - base = cpu_if->vgic_ap1r; + base = __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_AP1R0_EL2); if (p->is_write) - base[index] = p->regval; + base[index] = lower_32_bits(p->regval); else p->regval = base[index]; @@ -2144,12 +2144,10 @@ static bool access_gic_hcr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; - if (p->is_write) - cpu_if->vgic_hcr = p->regval; + __vcpu_sys_reg(vcpu, ICH_HCR_EL2) = lower_32_bits(p->regval); else - p->regval = cpu_if->vgic_hcr; + p->regval = __vcpu_sys_reg(vcpu, ICH_HCR_EL2); return true; } @@ -2206,12 +2204,19 @@ static bool access_gic_vmcr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; - if (p->is_write) - cpu_if->vgic_vmcr = p->regval; + __vcpu_sys_reg(vcpu, ICH_VMCR_EL2) = (p->regval & + (ICH_VMCR_ENG0_MASK | + ICH_VMCR_ENG1_MASK | + ICH_VMCR_PMR_MASK | + ICH_VMCR_BPR0_MASK | + ICH_VMCR_BPR1_MASK | + ICH_VMCR_EOIM_MASK | + ICH_VMCR_CBPR_MASK | + ICH_VMCR_FIQ_EN_MASK | + ICH_VMCR_ACK_CTL_MASK)); else - p->regval = cpu_if->vgic_vmcr; + p->regval = __vcpu_sys_reg(vcpu, ICH_VMCR_EL2); return true; } @@ -2220,17 +2225,29 @@ static bool access_gic_lr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; u32 index; + u64 *base; + base = __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_LR0_EL2); index = p->Op2; if (p->CRm == 13) index += 8; - if (p->is_write) - cpu_if->vgic_lr[index] = p->regval; - else - p->regval = cpu_if->vgic_lr[index]; + if (p->is_write) { + u64 mask = (ICH_LR_VIRTUAL_ID_MASK | + ICH_LR_GROUP | + ICH_LR_HW | + ICH_LR_STATE); + + if (p->regval & ICH_LR_HW) + mask |= ICH_LR_PHYS_ID_MASK; + else + mask |= ICH_LR_EOI; + + base[index] = p->regval & mask; + } else { + p->regval = base[index]; + } return true; } diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c index 98b21adb6e9b..bb6c74909a78 100644 --- a/arch/arm64/kvm/vgic/vgic-v3-nested.c +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -18,11 +18,6 @@ #define CREATE_TRACE_POINTS #include "vgic-nested-trace.h" -static inline struct vgic_v3_cpu_if *vcpu_nested_if(struct kvm_vcpu *vcpu) -{ - return &vcpu->arch.vgic_cpu.nested_vgic_v3; -} - static inline struct vgic_v3_cpu_if *vcpu_shadow_if(struct kvm_vcpu *vcpu) { return &vcpu->arch.vgic_cpu.shadow_vgic_v3; @@ -35,12 +30,11 @@ static inline bool lr_triggers_eoi(u64 lr) u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); u16 reg = 0; int i; for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { - if (lr_triggers_eoi(cpu_if->vgic_lr[i])) + if (lr_triggers_eoi(__vcpu_sys_reg(vcpu, ICH_LRN(i)))) reg |= BIT(i); } @@ -49,12 +43,11 @@ u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu) u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); u16 reg = 0; int i; for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { - if (!(cpu_if->vgic_lr[i] & ICH_LR_STATE)) + if (!(__vcpu_sys_reg(vcpu, ICH_LRN(i)) & ICH_LR_STATE)) reg |= BIT(i); } @@ -63,14 +56,13 @@ u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu) u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); int nr_lr = kvm_vgic_global_state.nr_lr; u64 reg = 0; if (vgic_v3_get_eisr(vcpu)) reg |= ICH_MISR_EOI; - if (cpu_if->vgic_hcr & ICH_HCR_UIE) { + if (__vcpu_sys_reg(vcpu, ICH_HCR_EL2) & ICH_HCR_UIE) { int used_lrs; used_lrs = nr_lr - hweight16(vgic_v3_get_elrsr(vcpu)); @@ -89,13 +81,12 @@ u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu) */ static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); struct vgic_irq *irq; int i, used_lrs = 0; for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { - u64 lr = cpu_if->vgic_lr[i]; + u64 lr = __vcpu_sys_reg(vcpu, ICH_LRN(i)); int l1_irq; if (!(lr & ICH_LR_HW)) @@ -125,36 +116,20 @@ static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) } trace_vgic_create_shadow_lrs(vcpu, kvm_vgic_global_state.nr_lr, - s_cpu_if->vgic_lr, cpu_if->vgic_lr); + s_cpu_if->vgic_lr, + __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_LR0_EL2)); s_cpu_if->used_lrs = used_lrs; } -/* - * Change the shadow HWIRQ field back to the virtual value before copying over - * the entire shadow struct to the nested state. - */ -static void vgic_v3_fixup_shadow_lr_state(struct kvm_vcpu *vcpu) -{ - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); - struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); - int lr; - - for (lr = 0; lr < kvm_vgic_global_state.nr_lr; lr++) { - s_cpu_if->vgic_lr[lr] &= ~ICH_LR_PHYS_ID_MASK; - s_cpu_if->vgic_lr[lr] |= cpu_if->vgic_lr[lr] & ICH_LR_PHYS_ID_MASK; - } -} - void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); struct vgic_irq *irq; int i; for (i = 0; i < s_cpu_if->used_lrs; i++) { - u64 lr = cpu_if->vgic_lr[i]; + u64 lr = __vcpu_sys_reg(vcpu, ICH_LRN(i)); int l1_irq; if (!(lr & ICH_LR_HW) || !(lr & ICH_LR_STATE)) @@ -180,14 +155,27 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) } } +void vgic_v3_create_shadow_state(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + int i; + + cpu_if->vgic_hcr = __vcpu_sys_reg(vcpu, ICH_HCR_EL2); + cpu_if->vgic_vmcr = __vcpu_sys_reg(vcpu, ICH_VMCR_EL2); + + for (i = 0; i < 4; i++) { + cpu_if->vgic_ap0r[i] = __vcpu_sys_reg(vcpu, ICH_AP0RN(i)); + cpu_if->vgic_ap1r[i] = __vcpu_sys_reg(vcpu, ICH_AP1RN(i)); + } + + vgic_v3_create_shadow_lr(vcpu); +} + void vgic_v3_load_nested(struct kvm_vcpu *vcpu) { - struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; struct vgic_irq *irq; unsigned long flags; - vgic_cpu->shadow_vgic_v3 = vgic_cpu->nested_vgic_v3; - vgic_v3_create_shadow_lr(vcpu); __vgic_v3_restore_state(vcpu_shadow_if(vcpu)); irq = vgic_get_irq(vcpu->kvm, vcpu, vcpu->kvm->arch.vgic.maint_irq); @@ -201,19 +189,34 @@ void vgic_v3_load_nested(struct kvm_vcpu *vcpu) void vgic_v3_put_nested(struct kvm_vcpu *vcpu) { - struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + int i; - __vgic_v3_save_state(vcpu_shadow_if(vcpu)); + __vgic_v3_save_state(s_cpu_if); - trace_vgic_put_nested(vcpu, kvm_vgic_global_state.nr_lr, - vcpu_shadow_if(vcpu)->vgic_lr); + trace_vgic_put_nested(vcpu, kvm_vgic_global_state.nr_lr, s_cpu_if->vgic_lr); /* * Translate the shadow state HW fields back to the virtual ones * before copying the shadow struct back to the nested one. */ - vgic_v3_fixup_shadow_lr_state(vcpu); - vgic_cpu->nested_vgic_v3 = vgic_cpu->shadow_vgic_v3; + __vcpu_sys_reg(vcpu, ICH_HCR_EL2) = s_cpu_if->vgic_hcr; + __vcpu_sys_reg(vcpu, ICH_VMCR_EL2) = s_cpu_if->vgic_vmcr; + + for (i = 0; i < 4; i++) { + __vcpu_sys_reg(vcpu, ICH_AP0RN(i)) = s_cpu_if->vgic_ap0r[i]; + __vcpu_sys_reg(vcpu, ICH_AP1RN(i)) = s_cpu_if->vgic_ap1r[i]; + } + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + u64 val = __vcpu_sys_reg(vcpu, ICH_LRN(i)); + + val &= ~ICH_LR_STATE; + val |= s_cpu_if->vgic_lr[i] & ICH_LR_STATE; + + __vcpu_sys_reg(vcpu, ICH_LRN(i)) = val; + } + irq_set_irqchip_state(kvm_vgic_global_state.maint_irq, IRQCHIP_STATE_ACTIVE, false); } @@ -227,10 +230,9 @@ void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) * again. */ if (vgic_state_is_nested(vcpu)) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); bool state; - state = cpu_if->vgic_hcr & ICH_HCR_EN; + state = __vcpu_sys_reg(vcpu, ICH_HCR_EL2) & ICH_HCR_EN; state &= vgic_v3_get_misr(vcpu); kvm_vgic_inject_irq(vcpu->kvm, vcpu->vcpu_id, diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c index 3a1ade1b592e..ed6be738528a 100644 --- a/arch/arm64/kvm/vgic/vgic-v3.c +++ b/arch/arm64/kvm/vgic/vgic-v3.c @@ -281,10 +281,11 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu) ICC_SRE_EL1_SRE); /* * If nesting is allowed, force GICv3 onto the nested - * guests as well. + * guests as well by setting the shadow state to the + * same value. */ if (vcpu_has_nv(vcpu)) - vcpu->arch.vgic_cpu.nested_vgic_v3.vgic_sre = vgic_v3->vgic_sre; + vcpu->arch.vgic_cpu.shadow_vgic_v3.vgic_sre = vgic_v3->vgic_sre; vcpu->arch.vgic_cpu.pendbaser = INITIAL_PENDBASER_VALUE; } else { vgic_v3->vgic_sre = 0; @@ -732,11 +733,15 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; /* - * vgic_v3_load_nested only affects the LRs in the shadow - * state, so it is fine to pass the nested state around. + * If the vgic is in nested state, populate the shadow state + * from the guest's nested state. As vgic_v3_load_nested() + * will only load LRs, let's deal with the rest of the state + * here as if it was a non-nested state. Cunning. */ - if (vgic_state_is_nested(vcpu)) - cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + if (vgic_state_is_nested(vcpu)) { + vgic_v3_create_shadow_state(vcpu); + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + } /* * If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h index 23e280fa0a16..41fc4bc0ce56 100644 --- a/arch/arm64/kvm/vgic/vgic.h +++ b/arch/arm64/kvm/vgic/vgic.h @@ -333,4 +333,14 @@ void vgic_v4_configure_vsgis(struct kvm *kvm); void vgic_v4_get_vlpi_state(struct vgic_irq *irq, bool *val); int vgic_v4_request_vpe_irq(struct kvm_vcpu *vcpu, int irq); +void vgic_v3_sync_nested(struct kvm_vcpu *vcpu); +void vgic_v3_create_shadow_state(struct kvm_vcpu *vcpu); +void vgic_v3_load_nested(struct kvm_vcpu *vcpu); +void vgic_v3_put_nested(struct kvm_vcpu *vcpu); +void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); + +#define ICH_LRN(n) (ICH_LR0_EL2 + (n)) +#define ICH_AP0RN(n) (ICH_AP0R0_EL2 + (n)) +#define ICH_AP1RN(n) (ICH_AP1R0_EL2 + (n)) + #endif diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 38643d9b3813..593b27a818de 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -334,9 +334,6 @@ struct vgic_cpu { struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS]; - /* CPU vif control registers for the virtual GICH interface */ - struct vgic_v3_cpu_if nested_vgic_v3; - /* * The shadow vif control register loaded to the hardware when * running a nested L2 guest with the virtual IMO/FMO bit set. @@ -401,10 +398,6 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu); void kvm_vgic_put(struct kvm_vcpu *vcpu); void kvm_vgic_vmcr_sync(struct kvm_vcpu *vcpu); -void vgic_v3_sync_nested(struct kvm_vcpu *vcpu); -void vgic_v3_load_nested(struct kvm_vcpu *vcpu); -void vgic_v3_put_nested(struct kvm_vcpu *vcpu); -void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu); u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu); u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu); From patchwork Tue Jan 31 09:24:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D064FC636CD for ; Tue, 31 Jan 2023 10:28:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DMPVgcqnuCYxeH0PjCXbmBT1tErMDbTbUEsIusxpF9E=; b=Szihw5musfpqne 5pcroqan2ct2T1Z5DUKVYu7ydNVUZ9oMG8hZ+rD+uAhxLITdA+/S3Vn0NHTi+KoasPO55/hScHOJQ 65rDzPb5Pk55Uo0a4N5WU6CaAa3eYGLAxgGtd6qM3JkcfZSXtT0lSYLaOmuqAIehnxvNwUIDI7jY6 1oV9+wpZLtbmylBStVs9OwimitJYhGaNC0B14QNrTa/nrV52Y+kmI8kVHwS7RvTjWoEH7oJYWYxPE NpBcec6/hr424zvCZRPLYpfEsYr+kmKHgYIGHTEJM77ssLGBDRPmmmg9pqVMZnacDomVPeIji4Gdb fO3L2WAqW5P7yeaqLQ0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnrP-007OoR-H0; Tue, 31 Jan 2023 10:27:32 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnAU-00764C-H6 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:43:12 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 42931CE1CCC; Tue, 31 Jan 2023 09:43:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ABA5FC4339C; Tue, 31 Jan 2023 09:43:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158186; bh=ia+7lU7uvByBUeiEEpZKtYPX0L15wbvLID+PpsSZeJk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eGw8YvKfADeIFZt5hZ5RUYvftQu2xkhQiJgxCguzfo3/631Md02ftCvLxfKmXZdyf 8V6CXNiZzN7LVDgOjmb09RGFb9awX9dkIH6gbTDark+HeVM1vHAva7Cz8FEPp/tQW7 T12HBD2fcbvNfTMYabZQyoy5D/Eqcb52x8T9TOJRDHqpB+TjKwX1N1/AWn4vNvmE8P HekJkeTUnmdAWVJUxBge/bSnkEHJEtljbd9LPBuiMpAiC4pPsOncrbOymafRhQlCU3 dOiDXwuZS3Ujs8ovQ1N2x3/yPTlG7l10qAChSGevrnp5jTp1t9a8OsYlfxtiQFz4Jb FlxvqQcYDT6zA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtv-0067U2-LQ; Tue, 31 Jan 2023 09:26:03 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 61/69] KVM: arm64: Add FEAT_NV2 cpu feature Date: Tue, 31 Jan 2023 09:24:56 +0000 Message-Id: <20230131092504.2880505-62-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014311_007627_77BCDEC8 X-CRM114-Status: GOOD ( 13.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add the detection code for the FEAT_NV2 feature. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 6 ++++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 2 files changed, 17 insertions(+) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 5dd6b12b6ec0..86e89ace1746 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -14,6 +14,12 @@ static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu) test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)); } +static inline bool vcpu_has_nv2(const struct kvm_vcpu *vcpu) +{ + return cpus_have_final_cap(ARM64_HAS_ENHANCED_NESTED_VIRT) && + vcpu_has_nv(vcpu); +} + /* Translation helpers from non-VHE EL2 to EL1 */ static inline u64 tcr_el2_ps_to_tcr_el1_ips(u64 tcr_el2) { diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 3fc14ee86239..49e632aa1756 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2240,6 +2240,17 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .field_width = 4, .min_field_value = ID_AA64MMFR2_EL1_NV_IMP, }, + { + .desc = "Enhanced Nested Virtualization Support", + .capability = ARM64_HAS_ENHANCED_NESTED_VIRT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_nested_virt_support, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_EL1_NV_SHIFT, + .field_width = 4, + .min_field_value = ID_AA64MMFR2_EL1_NV_NV2, + }, { .capability = ARM64_HAS_32BIT_EL0_DO_NOT_USE, .type = ARM64_CPUCAP_SYSTEM_FEATURE, From patchwork Tue Jan 31 09:24:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122832 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E6903C38142 for ; Tue, 31 Jan 2023 10:27:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QSEpS9VIz86uAYaF0306CoBFjMU8Ih2TmqImT7DFnVU=; b=j3lciJ7CAWYGUC 7kSXPXBx3FLPNDAurKllvExeUXKqPhwIQePf9T1XGww9tU2oPyXZuXL35AGei80Qx9nquSWQ1Ab1S 0CLlPVA26/0N6CNVvJ9LYKg+Mpz7Og2V8enoFV316Ig9Ev51C8SW2YVCyR4owek3PNlQszNGtX+fe DZsDqoIFM6vmqu2Eo9oajBY29y8Bs8P3rn+mqgD9VDu444OKk8bUqFq7FTIUl1Bp6kcn+3LX0XsSt aZou90CB1JIaZ4fpwE4YwbGn5ViNKFHTyyAgeFUwF9OWIfjWjQUHGmsSVedDq9R9KMXgCQDN+Rmol qEzUnzL8nDxoANIgqdvA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnqe-007OX1-Ke; Tue, 31 Jan 2023 10:26:45 +0000 Received: from sin.source.kernel.org ([2604:1380:40e1:4800::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnAT-00762y-CV for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:43:11 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id F33FDCE1BD9; Tue, 31 Jan 2023 09:43:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4FBBCC433EF; Tue, 31 Jan 2023 09:43:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158184; bh=214nS00mDoR6gQmTbYQrUICmLS4jFOzpb9raruKfJk4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PsBFdT+ekKSYArMF18IjZW+dsoEYiuHDuD2xezvL+eLFwDa+647ZC50bxEzxnWcYg 2StUg+vyy4Z9gPJSAnu5AkNU5PahOHoWwsH45hqUmbTB0OrPfw2wovSbPL8a0f0E4f v/IZSpFxvd0wZQ/iNVEZVlNjXemj8iGbWb+WY+NrQgvBwhmofE3n4A7As/kbPE+Gpc 5hr6y9G/oD/2VrVI3YBC1920HnhWs1PItNBfzI3dvW65g/lkroe1IhUsKwT3olxkCs a5LsP/GFTmLnUitjEh/1U6kKgSMm9gjk39Eso7xKq6mlBqAAuWDB1vUH2sYo2RBv2g hppsr4rC1N0Hg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtv-0067U2-Ta; Tue, 31 Jan 2023 09:26:03 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 62/69] KVM: arm64: nv: Sync nested timer state with FEAT_NV2 Date: Tue, 31 Jan 2023 09:24:57 +0000 Message-Id: <20230131092504.2880505-63-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014310_354683_46A13FAA X-CRM114-Status: GOOD ( 20.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Christoffer Dall Emulating the timers with FEAT_NV2 is a bit odd, as the timers can be reconfigured behind our back without the hypervisor even noticing. In the VHE case, that's an actual regression in the architecture... Co-developed-by: Christoffer Dall Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/arch_timer.c | 44 ++++++++++++++++++++++++++++++++++++ arch/arm64/kvm/arm.c | 3 +++ include/kvm/arm_arch_timer.h | 1 + 3 files changed, 48 insertions(+) diff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.c index a8425a88c750..cc308310c0a0 100644 --- a/arch/arm64/kvm/arch_timer.c +++ b/arch/arm64/kvm/arch_timer.c @@ -821,6 +821,50 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu) kvm_timer_blocking(vcpu); } +void kvm_timer_sync_nested(struct kvm_vcpu *vcpu) +{ + /* + * When NV2 is on, guest hypervisors have their EL0 timer register + * accesses redirected to the VNCR page. Any guest action taken on + * the timer is postponed until the next exit, leading to a very + * poor quality of emulation. + */ + if (!is_hyp_ctxt(vcpu)) + return; + + if (!vcpu_el2_e2h_is_set(vcpu)) { + /* + * A non-VHE guest hypervisor doesn't have any direct access + * to its timers: the EL2 registers trap (and the HW is + * fully emulated), while the EL0 registers access memory + * despite the access being notionally direct. Boo. + * + * We update the hardware timer registers with the + * latest value written by the guest to the VNCR page + * and let the hardware take care of the rest. + */ + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTV_CTL_EL0), SYS_CNTV_CTL); + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTV_CVAL_EL0), SYS_CNTV_CVAL); + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTP_CTL_EL0), SYS_CNTP_CTL); + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTP_CVAL_EL0), SYS_CNTP_CVAL); + } else { + /* + * For a VHE guest hypervisor, the EL2 state is directly + * stored in the host EL0 timers, while the emulated EL0 + * state is stored in the VNCR page. The latter could have + * been updated behind our back, and we must reset the + * emulation of the timers. + */ + struct timer_map map; + get_timer_map(vcpu, &map); + + soft_timer_cancel(&map.emul_vtimer->hrtimer); + soft_timer_cancel(&map.emul_ptimer->hrtimer); + timer_emulate(map.emul_vtimer); + timer_emulate(map.emul_ptimer); + } +} + /* * With a userspace irqchip we have to check if the guest de-asserted the * timer and if so, unmask the timer irq signal on the host interrupt diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index bfcf6db27448..b895ede3970b 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -982,6 +982,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) if (static_branch_unlikely(&userspace_irqchip_in_use)) kvm_timer_sync_user(vcpu); + if (vcpu_has_nv2(vcpu)) + kvm_timer_sync_nested(vcpu); + kvm_arch_vcpu_ctxsync_fp(vcpu); /* diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h index 24dad4aaa6a3..1a05fabc7b43 100644 --- a/include/kvm/arm_arch_timer.h +++ b/include/kvm/arm_arch_timer.h @@ -68,6 +68,7 @@ int __init kvm_timer_hyp_init(bool has_gic); int kvm_timer_enable(struct kvm_vcpu *vcpu); int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu); void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu); +void kvm_timer_sync_nested(struct kvm_vcpu *vcpu); void kvm_timer_sync_user(struct kvm_vcpu *vcpu); bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu); void kvm_timer_update_run(struct kvm_vcpu *vcpu); From patchwork Tue Jan 31 09:24:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 01611C636CC for ; Tue, 31 Jan 2023 10:24:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=LIzf0d+JorAaREnHdCoHsZGA4PU4+Sa15GhOMoZ8f0Y=; b=kKxVEVtvizxd4V x5RbWKnoWxeODc6gJqEoOn+UAbYSVuN85JvdTjVUTbw9WNM6conG435+GMRlc8qCYoZrXjEuUvIEa zg8Up9zKXrOVCjk388XdsQmHWERjkm2SxnHNi1jk5CYe0yR3lfXK1B3aEcMoDBjc/iXoWNX5jaPCH QHSa/XjJx4L1o4E1RKMaxBpC7FIG4BvEfxZrmqzttBloBOi5nOwAdqeYyhx/WoojjvQDIPNTlaFWg uJrWKMf4ycWUuTI2v6UmEels4KNR2EsOTNaWwx/ywYnFmGNe+g7TB3dJHtc5902oVPUgfzb/8CZvQ 6Jx07rOl2UmIUKg3YnAQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnn5-007N5r-Sh; Tue, 31 Jan 2023 10:23:04 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnAE-0075u6-9R for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:56 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C266EB81AE0; Tue, 31 Jan 2023 09:42:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 77078C433EF; Tue, 31 Jan 2023 09:42:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158171; bh=R9R8lSJiZlGHXpk1NuhDArGYtHjEsAnmxDGWI+EDPbg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DhJ6P6JM67Dve40ZbjsDCX3n4VeWxLIhE1Op5dPKxfCLP6Vy73CmFdtQIH6lj6FRM 7Vkh7l5j+OpRymdDTLQSwglMA4JMX6TEbhKMFeo5lSiYNR1zgkA/yYKTH5IkzjjtiK KL2lN6OBwaj2LmRzP52hbNZQjvle3H1z5gva7gmmbPaWvuhCP7FyPB3ASxWWrzkliJ m+JJljE/UD2EISlC/ShLxo4AVxPRJT3lv5CwDQiepUFSrQoXp+GvKy/2BXatZHk86k 1cojyEL2rSlIXpW/iA3a3A9m/0s3DwQTvZ76mINbqag/++5fo3NLXEpBwmcAyIGCzV LZX2kYn8dPHsQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtw-0067U2-5k; Tue, 31 Jan 2023 09:26:04 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 63/69] KVM: arm64: nv: Publish emulated timer interrupt state in the in-memory state Date: Tue, 31 Jan 2023 09:24:58 +0000 Message-Id: <20230131092504.2880505-64-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014254_679547_305E803D X-CRM114-Status: GOOD ( 14.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org With FEAT_NV2, the EL0 timer state is entirely stored in memory, meaning that the hypervisor can only provide a very poor emulation. The only thing we can really do is to publish the interrupt state in the guest view of CNT{P,V}_CTL_EL0, and defer everything else to the next exit. Only FEAT_ECV will allow us to fix it, at the cost of extra trapping. Suggested-by: Chase Conklin Suggested-by: Ganapatrao Kulkarni Signed-off-by: Marc Zyngier --- arch/arm64/kvm/arch_timer.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.c index cc308310c0a0..f9e98dd5c100 100644 --- a/arch/arm64/kvm/arch_timer.c +++ b/arch/arm64/kvm/arch_timer.c @@ -450,6 +450,25 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, { int ret; + /* + * Paper over NV2 brokenness by publishing the interrupt status + * bit. This still results in a poor quality of emulation (guest + * writes will have no effect until the next exit). + * + * But hey, it's fast, right? + */ + if (vcpu_has_nv2(vcpu) && is_hyp_ctxt(vcpu) && + (timer_ctx == vcpu_vtimer(vcpu) || timer_ctx == vcpu_ptimer(vcpu))) { + u32 ctl = timer_get_ctl(timer_ctx); + + if (new_level) + ctl |= ARCH_TIMER_CTRL_IT_STAT; + else + ctl &= ~ARCH_TIMER_CTRL_IT_STAT; + + timer_set_ctl(timer_ctx, ctl); + } + timer_ctx->irq.level = new_level; trace_kvm_timer_update_irq(vcpu->vcpu_id, timer_ctx->irq.irq, timer_ctx->irq.level); From patchwork Tue Jan 31 09:24:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DE901C636CC for ; Tue, 31 Jan 2023 09:57:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GcgfWx2Bn6N4lQH5hvgSSJ67ikecosSkkF/Ehdn35AI=; b=uEQ9hBen/IA/pR vRHPtrufK0t6tlCt+DC9VQmoC88vXDncpDQWNXLZ3VtQFzUAh0nMyzcvzwFglt0EbuVgKravSXM4d JD+wFbNHPXau4ceP1GhlQfMZG93vftfRzLPOrneL+HKPuWOtAf5QVMDYLRrMFre/RJdgyGJIyuF1+ bPd2IcD8M3+K9XXIFcATHbuopAc2S9wMle8UNj7fYONt9ziwd4vGk2QIxktS/GlZlfb6wB978O3ii KQmj5ox52gB7C00v7Sx1WgfOrvvHBRu43D533MLokHJeHgYPgK9SCHdE+uLCL8iShgFiECRjskpdI tnxA2cOFQTtFfvwRsBvA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnMp-007BTg-NQ; Tue, 31 Jan 2023 09:55:56 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn99-0075Mn-TQ for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:49 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7B6156148B; Tue, 31 Jan 2023 09:41:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4F925C433D2; Tue, 31 Jan 2023 09:41:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158107; bh=xEoffwONEvVbElUwXlLMsQt0ti0TTqB++Y6iLHQ/o9I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BnS8gk5sGqcWRQnUSdZKMMEqdsszvAu9BiYvsv6QbkEYwjuly3JC9jyockNqo7Oxm AQMiyLEn2Jls+QiCszzMFVd3pIar8PcDD1cwqhsJ/nx0qo3qRHhp7P200/oEgM80xo LZ79E+VbJufhYiefiWiybJ371G9mKv+1IpG/PsIBCCIvulwnOOwMJbQ/RLtno0orlY K5dh1urP88k7WyZAi2aHE/ucqIkgYUEz1SM4Hz9JJSR3OwAvYaeoCuG3pStYfaN99P TmpmS9NFfPle6+1Z+UtTCtYRx4wH4q6EqPyrPQ3xcYFQ7sFgwW2b/c9M63E6dgKj7N bBAFSpzs5PneQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtw-0067U2-Dl; Tue, 31 Jan 2023 09:26:04 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 64/69] KVM: arm64: nv: Allocate VNCR page when required Date: Tue, 31 Jan 2023 09:24:59 +0000 Message-Id: <20230131092504.2880505-65-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014148_058048_5265F8AB X-CRM114-Status: GOOD ( 14.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If running a NV guest on an ARMv8.4-NV capable system, let's allocate an additional page that will be used by the hypervisor to fulfill system register accesses. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/nested.c | 10 ++++++++++ arch/arm64/kvm/reset.c | 1 + 2 files changed, 11 insertions(+) diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index f93f4248c6f3..19f07aa8fa68 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -38,6 +38,14 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) if (!cpus_have_final_cap(ARM64_HAS_NESTED_VIRT)) return -EINVAL; + if (cpus_have_final_cap(ARM64_HAS_ENHANCED_NESTED_VIRT)) { + if (!vcpu->arch.ctxt.vncr_array) + vcpu->arch.ctxt.vncr_array = (u64 *)__get_free_page(GFP_KERNEL | __GFP_ZERO); + + if (!vcpu->arch.ctxt.vncr_array) + return -ENOMEM; + } + mutex_lock(&kvm->lock); /* @@ -67,6 +75,8 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 2], 0)) { kvm_free_stage2_pgd(&tmp[num_mmus - 1]); kvm_free_stage2_pgd(&tmp[num_mmus - 2]); + free_page((unsigned long)vcpu->arch.ctxt.vncr_array); + vcpu->arch.ctxt.vncr_array = NULL; } else { kvm->arch.nested_mmus_size = num_mmus; ret = 0; diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 49a3257dec46..8417d2fa0ebd 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -161,6 +161,7 @@ void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu) if (sve_state) kvm_unshare_hyp(sve_state, sve_state + vcpu_sve_state_size(vcpu)); kfree(sve_state); + free_page((unsigned long)vcpu->arch.ctxt.vncr_array); kfree(vcpu->arch.ccsidr); } From patchwork Tue Jan 31 09:25:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 84E7EC636CC for ; Tue, 31 Jan 2023 09:43:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IA8ro0xEFm8a4fl5hvOCBEmYr5C5QGZ0EFZgL8FQ1M0=; b=zMH2wvITS6cnEX NVv8vtsnsfPveIsBKYB+Zue3rgv4jEvG3cC40gNiwVM74a9tM9kNnDjGq5yQDNgXDI8wqUwx/t91P 61yGmbEq9zzbI/ba9ykTXNd0A3LZ99s4WZR5nS79Ov2WAfmkjUGvDkdo+0B6YSjtI9kt19CMl+Okc 7Ttxf0kfuXe6ebIEmfVOhej6/jks3oaOQOvhoPkBknkQXwpti0DsIhO9nTqNzXjf6kEp7pEWP8eRR IRSBT0wnA8DesN+SP4J7h1sGle8SeDzXo/EL4aMi00DW84wXU7doLQx/LvvCVQDDtqEJSZbA1jhNS p1qmSP18Gn4HoOGlLTJw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnAE-0075uO-BQ; Tue, 31 Jan 2023 09:42:54 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8D-0074sQ-Co for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:40:51 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CF4746148C; Tue, 31 Jan 2023 09:40:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6E1F1C433D2; Tue, 31 Jan 2023 09:40:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158048; bh=QmcIvSqOHyUQqggyWtwOqI7ZEscQ7oLFdlrm1jPrqms=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XEDQTx1WP3distnUXlcatCaT99f8YjYn3PUwskaPGoxx2MnW6dEA+IimWIKGFiZ3C 0MmViZZqhDR7QWDUAbDW+45F8ne0NbeOaxGtHUMla0UGlV0UA5HKaMS/FwQGhzGVwb xkd9yE2anX0zO8k0R7XfSQB25otF3Cfh7Z1tNfn/xOaZ8jtXmswln7SvkFHNF2ZHHA cbUk7jDIzfQMO5BbKwo7/PoCbYxqrLNL8KXOYsmV66bJwQIEjIbh/GbR/8zKN5aQKg Iv8+1G33Q51Axl0Pb/aQjvwfc54OAlg03EmLI3ROBKlUy8nRQ6TEBBdWKhlbJpqMXA xirVv/MT92aSQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtw-0067U2-Lx; Tue, 31 Jan 2023 09:26:04 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 65/69] KVM: arm64: nv: Enable ARMv8.4-NV support Date: Tue, 31 Jan 2023 09:25:00 +0000 Message-Id: <20230131092504.2880505-66-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014049_553470_438C1A5B X-CRM114-Status: GOOD ( 18.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As all the VNCR-capable system registers are nicely separated from the rest of the crowd, let's set HCR_EL2.NV2 on and let the ball rolling. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/kvm_emulate.h | 23 +++++++++++++---------- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kvm/hyp/vhe/switch.c | 14 +++++++++++++- 4 files changed, 28 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index d446fccdca5a..6e20fcc142bb 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -20,6 +20,7 @@ #define HCR_AMVOFFEN (UL(1) << 51) #define HCR_FIEN (UL(1) << 47) #define HCR_FWB (UL(1) << 46) +#define HCR_NV2 (UL(1) << 45) #define HCR_AT (UL(1) << 44) #define HCR_NV1 (UL(1) << 43) #define HCR_NV (UL(1) << 42) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index d4721555efc9..98600f2bbcc9 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -249,21 +249,24 @@ static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu) static inline u64 __fixup_spsr_el2_write(struct kvm_cpu_context *ctxt, u64 val) { - if (!__vcpu_el2_e2h_is_set(ctxt)) { - /* - * Clear the .M field when writing SPSR to the CPU, so that we - * can detect when the CPU clobbered our SPSR copy during a - * local exception. - */ - val &= ~0xc; - } + struct kvm_vcpu *vcpu = container_of(ctxt, struct kvm_vcpu, arch.ctxt); + + if (vcpu_has_nv2(vcpu) || __vcpu_el2_e2h_is_set(ctxt)) + return val; - return val; + /* + * Clear the .M field when writing SPSR to the CPU, so that we + * can detect when the CPU clobbered our SPSR copy during a + * local exception. + */ + return val &= ~0xc; } static inline u64 __fixup_spsr_el2_read(const struct kvm_cpu_context *ctxt, u64 val) { - if (__vcpu_el2_e2h_is_set(ctxt)) + struct kvm_vcpu *vcpu = container_of(ctxt, struct kvm_vcpu, arch.ctxt); + + if (vcpu_has_nv2(vcpu) || __vcpu_el2_e2h_is_set(ctxt)) return val; /* diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 06200ea47ad4..7b197d6449de 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -510,6 +510,7 @@ #define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2) #define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0) #define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2) +#define SYS_VNCR_EL2 sys_reg(3, 4, 2, 2, 0) #define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1) #define SYS_HDFGRTR_EL2 sys_reg(3, 4, 3, 1, 4) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index ff77b37b6f45..7682d2f13eaa 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -47,7 +47,13 @@ static void __activate_traps(struct kvm_vcpu *vcpu) * the EL1 virtual memory control register accesses * as well as the AT S1 operations. */ - hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_TTLB | HCR_NV1; + if (vcpu_has_nv2(vcpu)) { + hcr &= ~HCR_TVM; + } else { + hcr |= HCR_TVM | HCR_TRVM | HCR_TTLB; + } + + hcr |= HCR_AT | HCR_NV1; } else { /* * For a guest hypervisor on v8.1 (VHE), allow to @@ -81,6 +87,12 @@ static void __activate_traps(struct kvm_vcpu *vcpu) if (!vcpu_el2_tge_is_set(vcpu)) hcr |= HCR_AT | HCR_TTLB; } + + if (vcpu_has_nv2(vcpu)) { + hcr |= HCR_AT | HCR_TTLB | HCR_NV2; + write_sysreg_s(vcpu->arch.ctxt.vncr_array, + SYS_VNCR_EL2); + } } else if (vcpu_has_nv(vcpu)) { u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); From patchwork Tue Jan 31 09:25:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62C23C636CC for ; Tue, 31 Jan 2023 09:54:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fMcclAvqJTp43dQ+yc53LJd/lWoQs7i1Pk+NJMwQc54=; b=qqfLny+meKhAYN fYG7+HVD8DPN87YA/35viDQEegKMvoH6YOPbUSSbd0x1ICey8qNQtFmE1jMpXPlLbTkNY1EA7fRxU ji5bjcJVjLozZur9obFygGkT2iFwQvw1gZEV0N1iPIj5t/f/1fNvG5x668PT16O0xbyREnS3a5ZEU Jr8YQn4tBOTGbbNUDV615NU/98io4AeLD828kqz2wbWWesXBOVkbJr7qCjGv/MAB1Dn9cDn/c2VBj Rkaymw8IGtHXQ7uAzc77tbHKueQp3V4eAEf5xOEMGphel+qdKjfWAEZ3TtUfYhY/KG+eBhT1NTYE9 Ybni3JVQ/7uGSYSEkrVw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnJz-007A1O-I7; Tue, 31 Jan 2023 09:52:59 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8v-0075FP-Ew for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:35 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 081ACB81AF1; Tue, 31 Jan 2023 09:41:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B349DC433D2; Tue, 31 Jan 2023 09:41:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158090; bh=C193qCDjqiF5HkfsJlWly9Jsvkj9TvtpD7IYpBzHGpI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uNYz1SAYLmOJCe+uWcBDN+NDEqD/VQmLamiThqyte4jsBwE6JCNo8x/VBrn2dnqAb IZsgQ+L3E9T2jpkXkLhU0Faja2IOxeYREguEcohp0ccr79RLc00IFkZLr3VOyJzq+F iL1+x0xVGdH7ky8Yfj2f1W+JReZ7XKdMqIf1j0elslU16lJdVRHs5t89jEYmlUxPys w3BbJ9/46vSL0pNGEuAw8ryX3GiBdspCryUtx7S8Xf9MO+xKvLx8ogtT/xBLgo9KNn drUWEr9aaDEx0bOX9hgK8u5irfRlLWH1Jmrhrstapb+TWwKq6VaBXpSwpF15oD15NC lLiGFyOY61NAQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtw-0067U2-U8; Tue, 31 Jan 2023 09:26:04 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 66/69] KVM: arm64: nv: Fast-track 'InHost' exception returns Date: Tue, 31 Jan 2023 09:25:01 +0000 Message-Id: <20230131092504.2880505-67-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014133_832453_63CC64D8 X-CRM114-Status: GOOD ( 22.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A significant part of the ARMv8.3-NV extension is to trap ERET instructions so that the hypervisor gets a chance to switch from a vEL2 L1 guest to an EL1 L2 guest. But this also has the unfortunate consequence of trapping ERET in unsuspecting circumstances, such as staying at vEL2 (interrupt handling while being in the guest hypervisor), or returning to host userspace in the case of a VHE guest. Although we already make some effort to handle these ERET quicker by not doing the put/load dance, it is still way too far down the line for it to be efficient enough. For these cases, it would ideal to ERET directly, no question asked. Of course, we can't do that. But the next best thing is to do it as early as possible, in fixup_guest_exit(), much as we would handle FPSIMD exceptions. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/emulate-nested.c | 29 +++------------------ arch/arm64/kvm/hyp/vhe/switch.c | 46 +++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 26 deletions(-) diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 555771e1260d..e2d23f624115 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -79,8 +79,7 @@ static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr) void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) { - u64 spsr, elr, mode; - bool direct_eret; + u64 spsr, elr; /* * Forward this trap to the virtual EL2 if the virtual @@ -89,33 +88,11 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) if (forward_nv_traps(vcpu)) return; - /* - * Going through the whole put/load motions is a waste of time - * if this is a VHE guest hypervisor returning to its own - * userspace, or the hypervisor performing a local exception - * return. No need to save/restore registers, no need to - * switch S2 MMU. Just do the canonical ERET. - */ - spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2); - spsr = kvm_check_illegal_exception_return(vcpu, spsr); - - mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); - - direct_eret = (mode == PSR_MODE_EL0t && - vcpu_el2_e2h_is_set(vcpu) && - vcpu_el2_tge_is_set(vcpu)); - direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); - - if (direct_eret) { - *vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2); - *vcpu_cpsr(vcpu) = spsr; - trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr); - return; - } - preempt_disable(); kvm_arch_vcpu_put(vcpu); + spsr = __vcpu_sys_reg(vcpu, SPSR_EL2); + spsr = kvm_check_illegal_exception_return(vcpu, spsr); elr = __vcpu_sys_reg(vcpu, ELR_EL2); trace_kvm_nested_eret(vcpu, elr, spsr); diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 7682d2f13eaa..238c6613cf47 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -168,6 +168,51 @@ void deactivate_traps_vhe_put(struct kvm_vcpu *vcpu) __deactivate_traps_common(vcpu); } +static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; + u64 spsr, mode; + + /* + * Going through the whole put/load motions is a waste of time + * if this is a VHE guest hypervisor returning to its own + * userspace, or the hypervisor performing a local exception + * return. No need to save/restore registers, no need to + * switch S2 MMU. Just do the canonical ERET. + * + * Unless the trap has to be forwarded further down the line, + * of course... + */ + if (__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_NV) + return false; + + spsr = read_sysreg_el1(SYS_SPSR); + spsr = __fixup_spsr_el2_read(ctxt, spsr); + mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); + + switch (mode) { + case PSR_MODE_EL0t: + if (!(vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu))) + return false; + break; + case PSR_MODE_EL2t: + mode = PSR_MODE_EL1t; + break; + case PSR_MODE_EL2h: + mode = PSR_MODE_EL1h; + break; + default: + return false; + } + + spsr = (spsr & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; + + write_sysreg_el2(spsr, SYS_SPSR); + write_sysreg_el2(read_sysreg_el1(SYS_ELR), SYS_ELR); + + return true; +} + static const exit_handler_fn hyp_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = NULL, [ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32, @@ -177,6 +222,7 @@ static const exit_handler_fn hyp_exit_handlers[] = { [ESR_ELx_EC_IABT_LOW] = kvm_hyp_handle_iabt_low, [ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low, [ESR_ELx_EC_PAC] = kvm_hyp_handle_ptrauth, + [ESR_ELx_EC_ERET] = kvm_hyp_handle_eret, }; static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu) From patchwork Tue Jan 31 09:25:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122828 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 12115C636CC for ; Tue, 31 Jan 2023 10:25:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=U363uGRLaFT+u/HTfjS00j0Fuyffs2IGb2/RAbrmM5M=; b=FkOEdD/gXe48RH euaM1YJVOjfPHpAEMVQ+4kWlc0op2tsti1Plf80sCvqsVazLuwUFnmK6Eg2V4X1pJU+r0dXJaU9b0 wK36yB/O9xBcmUh/E1v3hrhpe5QMnsK6zIhQCDJGqdjVY4ohy68hjl+EvbK0CEdeDSBZwAe8rMxf8 sjlrZBqehhEzBG7H0FmYbBQkvoD5XXCr57/IZ6ouDgAa8H+BJhtAH0SgL1hvNN4PxQwj0l04+oifk KYnV/6ZXfrhrkMzyiUskmsRrDZbxXdfXkikRZc2cPp8ul6Me7KjTGTVoHctCOsn6IX26oW58b2E5B V89w8w19OURPxvsr9bUA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnny-007NQh-Rg; Tue, 31 Jan 2023 10:23:59 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnAG-0075xI-Bd for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:59 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CF39161478; Tue, 31 Jan 2023 09:42:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 417F2C433D2; Tue, 31 Jan 2023 09:42:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158175; bh=PPX1R4dKA2pD7SNKEyWdQQpgFAbK3cl/QLyVGGXHwPw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ud+xfjbctmnU/YTbbr8m3Xn72Eyv7VLLjxSafSZUVACfVb97qc+tD2C4ZBS1co0q0 N4mbNS22DTgfBZz3Kq6ndwsXkglHMmfznKGOGLjps0P1BjaYpPNVqcW1YCm5BT/46R mkJBUgj85pZF5Q40b37li3QQYr9Yl5J6KnbZkpS7B5yekvYFYEwLtaSgMhzv8CKbwy dO4+J2WCLUzmL88XRDErmKjnaOO0xugrMRmuohPBaWHw0msrVawkcrfw92Tf7AU6Vg dFRXFo3gFF2rjIamN1bliKJRlAB1A0A+eBOeBS1nB+vx1ShKlLCgAHQtMeIkGXy158 kUQPlAaPNooHg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtx-0067U2-70; Tue, 31 Jan 2023 09:26:05 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 67/69] KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests Date: Tue, 31 Jan 2023 09:25:02 +0000 Message-Id: <20230131092504.2880505-68-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014256_563968_5FEEAEED X-CRM114-Status: GOOD ( 21.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Due to the way ARMv8.4-NV suppresses traps when accessing EL2 system registers, we can't track when the guest changes its HCR_EL2.TGE setting. This means we always trap EL1 TLBIs, even if they don't affect any guest. This obviously has a huge impact on performance, as we handle TLBI traps as a normal exit, and a normal VHE host issues thousands of TLBIs when booting (and quite a few when running userspace). A cheap way to reduce the overhead is to handle the limited case of {E2H,TGE}=={1,1} as a guest fixup, as we already have the right mmu configuration in place. Just execute the decoded instruction right away and return to the guest. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/vhe/switch.c | 43 ++++++++++++++++++++++++++++++++- arch/arm64/kvm/hyp/vhe/tlb.c | 6 +++-- arch/arm64/kvm/sys_regs.c | 25 ++++++------------- 3 files changed, 54 insertions(+), 20 deletions(-) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 238c6613cf47..a3555b90d9e1 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -168,6 +168,47 @@ void deactivate_traps_vhe_put(struct kvm_vcpu *vcpu) __deactivate_traps_common(vcpu); } +static bool kvm_hyp_handle_tlbi_el1(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + u32 instr; + u64 val; + + /* + * Ideally, we would never trap on EL1 TLB invalidations when the + * guest's HCR_EL2.{E2H,TGE} == {1,1}. But "thanks" to ARMv8.4, we + * don't trap writes to HCR_EL2, meaning that we can't track + * changes to the virtual TGE bit. So we leave HCR_EL2.TTLB set on + * the host. Oopsie... + * + * In order to speed-up EL1 TLBIs from the vEL2 guest when TGE is + * set, try and handle these invalidation as quickly as possible, + * without fully exiting (unless this needs forwarding). + */ + if (!vcpu_has_nv2(vcpu) || + !vcpu_is_el2(vcpu) || + (__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_E2H | HCR_TGE)) != (HCR_E2H | HCR_TGE)) + return false; + + instr = esr_sys64_to_sysreg(kvm_vcpu_get_esr(vcpu)); + if (sys_reg_Op0(instr) != TLBI_Op0 || + sys_reg_Op1(instr) != TLBI_Op1_EL1) + return false; + + val = vcpu_get_reg(vcpu, kvm_vcpu_sys_get_rt(vcpu)); + __kvm_tlb_el1_instr(NULL, val, instr); + __kvm_skip_instr(vcpu); + + return true; +} + +static bool kvm_hyp_handle_sysreg_vhe(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + if (kvm_hyp_handle_tlbi_el1(vcpu, exit_code)) + return true; + + return kvm_hyp_handle_sysreg(vcpu, exit_code); +} + static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) { struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; @@ -216,7 +257,7 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) static const exit_handler_fn hyp_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = NULL, [ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32, - [ESR_ELx_EC_SYS64] = kvm_hyp_handle_sysreg, + [ESR_ELx_EC_SYS64] = kvm_hyp_handle_sysreg_vhe, [ESR_ELx_EC_SVE] = kvm_hyp_handle_fpsimd, [ESR_ELx_EC_FP_ASIMD] = kvm_hyp_handle_fpsimd, [ESR_ELx_EC_IABT_LOW] = kvm_hyp_handle_iabt_low, diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index c4389db4cc22..beb162468c0b 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -201,7 +201,8 @@ void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding) dsb(ishst); /* Switch to requested VMID */ - __tlb_switch_to_guest(mmu, &cxt); + if (mmu) + __tlb_switch_to_guest(mmu, &cxt); /* * Execute the same instruction as the guest hypervisor did, @@ -240,5 +241,6 @@ void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding) dsb(ish); isb(); - __tlb_switch_to_host(&cxt); + if (mmu) + __tlb_switch_to_host(&cxt); } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 1a1ae7ff218e..924afc40ab8b 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -3110,6 +3110,8 @@ static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { u32 sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + u64 virtual_vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + struct kvm_s2_mmu *mmu; if (vcpu_has_nv(vcpu) && forward_traps(vcpu, HCR_TTLB)) return false; @@ -3131,24 +3133,13 @@ static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, mutex_lock(&vcpu->kvm->lock); - if ((__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_E2H | HCR_TGE)) != (HCR_E2H | HCR_TGE)) { - u64 virtual_vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); - struct kvm_s2_mmu *mmu; - - mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, HCR_VM); - if (mmu) - __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, HCR_VM); + if (mmu) + __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); - mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, 0); - if (mmu) - __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); - } else { - /* - * ARMv8.4-NV allows the guest to change TGE behind - * our back, so we always trap EL1 TLBIs from vEL2... - */ - __kvm_tlb_el1_instr(&vcpu->kvm->arch.mmu, p->regval, sys_encoding); - } + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, 0); + if (mmu) + __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); mutex_unlock(&vcpu->kvm->lock); From patchwork Tue Jan 31 09:25:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122700 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5DEA5C38142 for ; Tue, 31 Jan 2023 09:47:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=FbYJOsoUIzpu2kKx7ov8sL/NZ4wK3ADpsmnK7+GtBpk=; b=Q0YblYFbC8wrtF Oqn2GZ1mPYz+w57bJD/e9z6lEmKhdIK4KHnJKEIBDMpJK1l2F13+Zv1DQC+Jn26uf1WvA7LSw1aJ+ p0IyktcbUv0btHfjzmM4+fn/sDkVvN1l2oNJCY1eNLpSIMq5uv4xM2QHO7Mi9Ws4znHhwwzOQ7eTV 0DaIVcGNeWkxMGq7R6UheO/lL9OHHu/4AbH0WUVhwOH22wo/nKSR9mAHlmgIgZj1nQXCeFGMIEog+ GgksO8j9znR3r+tl7CryXXOqaeyrgfoY6zjCle+PzPlNEq9CftF5Y12vhGk2GXAkOKC73gc7vd7EP d93LywTRYDPfBYm8N2Nw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMnDR-0077MF-5A; Tue, 31 Jan 2023 09:46:13 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn8O-0074yT-BT for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:41:02 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D163561478; Tue, 31 Jan 2023 09:40:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 45F6EC4339B; Tue, 31 Jan 2023 09:40:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158059; bh=FiC2D5JqzQBNXMgtEyqErD1oya6FjA8qSK9Th4x5NI8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BkHIvCKx/vMYSKYXXle/jX3dw2/g8Q3KdTyX808oelvE1FNz8wn1+eo4LIFZtUs8M Va306V3l+kbbwOVMhjhGr4OmqsDHKLa4k+y+clpb9QEF1JOXqwYeZakegZlaweUsrz jlM19gV1agI6RkDD5ePV4EM3+EzEQ9sbER8KqeZDxF7xDc9UfMVQzwYFaxLEqllT5u lby9uQdMKzE200ShwJA58vCM0VBSFAMOiJRCAl9ipExcBGa8yGY3955soNDULfMtDU 0YUwvs5Y1QE8oygslxVLocglHRw0EP2qA5m5Mo5m7jEcaRpC49jLpkQFOEVXoDLMfp FNToYQj/CX/0g== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtx-0067U2-FC; Tue, 31 Jan 2023 09:26:05 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 68/69] KVM: arm64: nv: Use FEAT_ECV to trap access to EL0 timers Date: Tue, 31 Jan 2023 09:25:03 +0000 Message-Id: <20230131092504.2880505-69-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014100_518063_D07F0C77 X-CRM114-Status: GOOD ( 22.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Although FEAT_NV2 makes most things fast, it also makes it impossible to correctly emulate the timers, as the sysreg accesses are redirected to memory. FEAT_ECV addresses this by giving a hypervisor the ability to trap the EL02 sysregs as well as the virtual timer. Add the required trap setting to make use of the feature, allowing us to elide the ugly resync in the middle of the run loop. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/arch_timer.c | 54 +++++++++++++++++++++++++++- include/clocksource/arm_arch_timer.h | 4 +++ 2 files changed, 57 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.c index f9e98dd5c100..213b15a16ef2 100644 --- a/arch/arm64/kvm/arch_timer.c +++ b/arch/arm64/kvm/arch_timer.c @@ -740,6 +740,44 @@ static void kvm_timer_vcpu_load_nested_switch(struct kvm_vcpu *vcpu, WARN_ON_ONCE(ret); } + /* + * NV2 badly breaks the timer semantics by redirecting accesses to + * the EL0 timer state to memory, so let's call ECV to the rescue if + * available: we trap all CNT{P,V}_{CTL,CVAL,TVAL}_EL0 accesses. + * + * The treatment slightly varies depending whether we run a nVHE or + * VHE guest: nVHE will use the _EL0 registers directly, while VHE + * will use the _EL02 accessors. This translates in different trap + * bits. + * + * None of the trapping is engaged when running in non-HYP context, + * unless required by the L1 hypervisor settings once we advertise + * ECV+NV in the guest. + */ + if (vcpu_has_nv2(vcpu) && cpus_have_const_cap(ARM64_HAS_ECV)) { + u64 clr = 0, set = 0; + + if (vcpu_el2_e2h_is_set(vcpu)) { + set |= CNTHCTL_EL1NVVCT | CNTHCTL_EL1NVPCT; + } else { + /* + * Only trap the timers, not the counters, as they + * are in the "direct" state for nVHE. + */ + clr |= CNTHCTL_EL1PCEN << 10; + set |= CNTHCTL_EL1TVT; + } + + /* + * Apply these values for HYP context, and reverse them + * otherwise + */ + if (is_hyp_ctxt(vcpu)) + sysreg_clear_set(cntkctl_el1, clr, set); + else + sysreg_clear_set(cntkctl_el1, set, clr); + } + /* * Apply the trapping bits that the guest hypervisor has * requested for its own guest. @@ -838,6 +876,16 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu) if (kvm_vcpu_is_blocking(vcpu)) kvm_timer_blocking(vcpu); + + /* + * Restore the standard, non-nested configuration so that we're + * ready to run a non-NV vcpu. + */ + if (vcpu_has_nv2(vcpu)) + sysreg_clear_set(cntkctl_el1, + (CNTHCTL_EL1NVPCT | CNTHCTL_EL1NVVCT | + CNTHCTL_EL1TVT | CNTHCTL_EL1TVCT), + (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN) << 10); } void kvm_timer_sync_nested(struct kvm_vcpu *vcpu) @@ -847,8 +895,12 @@ void kvm_timer_sync_nested(struct kvm_vcpu *vcpu) * accesses redirected to the VNCR page. Any guest action taken on * the timer is postponed until the next exit, leading to a very * poor quality of emulation. + * + * This is an unmitigated disaster, only papered over by FEAT_ECV, + * which allows trapping of the timer registers even with NV2. + * Still, this is still worse than FEAT_NV on its own. Meh. */ - if (!is_hyp_ctxt(vcpu)) + if (cpus_have_const_cap(ARM64_HAS_ECV) || !is_hyp_ctxt(vcpu)) return; if (!vcpu_el2_e2h_is_set(vcpu)) { diff --git a/include/clocksource/arm_arch_timer.h b/include/clocksource/arm_arch_timer.h index 057c8964aefb..e3c3816d19ba 100644 --- a/include/clocksource/arm_arch_timer.h +++ b/include/clocksource/arm_arch_timer.h @@ -21,6 +21,10 @@ #define CNTHCTL_EVNTEN (1 << 2) #define CNTHCTL_EVNTDIR (1 << 3) #define CNTHCTL_EVNTI (0xF << 4) +#define CNTHCTL_EL1TVT (1 << 13) +#define CNTHCTL_EL1TVCT (1 << 14) +#define CNTHCTL_EL1NVPCT (1 << 15) +#define CNTHCTL_EL1NVVCT (1 << 16) enum arch_timer_reg { ARCH_TIMER_REG_CTRL, From patchwork Tue Jan 31 09:25:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13122816 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6CF8CC38142 for ; Tue, 31 Jan 2023 10:17:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=v9YVfXxe1nwoQQoWB0FbfJM+W7g+W1bVjM/nk/WUnEU=; b=RXjy8wtBj6cVMY UNCU8CKI0qFWk4Pp/lsFPO3i9WSMsmKsxxpXbG4y+z4eaomLkFbiGQFiQEXEggbofADeWkcLDWb6Y b332CWRDKCS8TViBtBwS0eaEyNKHMQZZOZgdVsaVIZoAwJgayBQ39aiHzXiOdDyfRUKMy0MWN3W+f oFGCJNODRTQ3xmPRQo714ohcFUBK68jDCnLt3O8EWPKpicUB0LnfNainDqrURq9/XINI3Z2fA9Aob Onvpqu6LEsZXmeIbZkuHIKqUb2q4JwqJ0kiByGwoJOr8+pr9NwOncJtfMSTFs17ZRZsZN4in/Di4t 4MY8JAqdUu3QiTaxFIrQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMng0-007Jzb-5x; Tue, 31 Jan 2023 10:15:45 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pMn9q-0075hj-U9 for linux-arm-kernel@lists.infradead.org; Tue, 31 Jan 2023 09:42:32 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6D30B6147F; Tue, 31 Jan 2023 09:42:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D2BCDC433EF; Tue, 31 Jan 2023 09:42:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675158149; bh=R76HaO8qoXwTusW877BsCKlHkjCFGvuoQfdtYgi9v/k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E1SJhhLuWllct1hyqZ0/VugyusW1hDA+OdrBr2YwCiNp9G6lNodZodWtGWkgkffsW kZ3tgdjL4OxLoxxcbq1LgAk2tob58cPo9YqXmNHjG/NaUf8UbrJbVLZ6X6xwAf3SOx JVfEeMBendDCAcJNY4aD1U/VVCrUAT198jYbXbwuiHy0e0Z3XOhy3rmwZr/xJmofUG 41secMmr75MPiNms2BmzufWBUBjDi/zB9rMF69JdWjOsGEtgP4GFtMwhwb/XLvb36i rwiUiP0ZusNBqRVeypKcwJIGI2PLonzkr7f8HyWaH0raFoGp05SHmKafdG5NiBMlHz 9c0Gx222Z2XJw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pMmtx-0067U2-NT; Tue, 31 Jan 2023 09:26:05 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: [PATCH v8 69/69] KVM: arm64: nv: Accelerate EL0 timer read accesses when FEAT_ECV is on Date: Tue, 31 Jan 2023 09:25:04 +0000 Message-Id: <20230131092504.2880505-70-maz@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230131092504.2880505-1-maz@kernel.org> References: <20230131092504.2880505-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230131_014231_088736_160E8475 X-CRM114-Status: GOOD ( 14.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Although FEAT_ECV allows us to correctly enable the timers, it also reduces performances pretty badly (a L2 guest doing a lot of virtio emulated in L1 userspace results in a 30% degradation). Mitigate this by emulating the CTL/CVAL register reads in the inner run loop, without returning to the general kernel. This halves the overhead described above. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/vhe/switch.c | 49 +++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index a3555b90d9e1..a9ac61505a86 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -201,11 +201,60 @@ static bool kvm_hyp_handle_tlbi_el1(struct kvm_vcpu *vcpu, u64 *exit_code) return true; } +static bool kvm_hyp_handle_ecv(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + u64 esr, val; + + /* + * Having FEAT_ECV allows for a better quality of timer emulation. + * However, this comes at a huge cost in terms of traps. Try and + * satisfy the reads without returning to the kernel if we can. + */ + if (!cpus_have_final_cap(ARM64_HAS_ECV)) + return false; + + if (!vcpu_has_nv2(vcpu)) + return false; + + esr = kvm_vcpu_get_esr(vcpu); + if ((esr & ESR_ELx_SYS64_ISS_DIR_MASK) != ESR_ELx_SYS64_ISS_DIR_READ) + return false; + + switch (esr_sys64_to_sysreg(esr)) { + case SYS_CNTP_CTL_EL02: + case SYS_CNTP_CTL_EL0: + val = __vcpu_sys_reg(vcpu, CNTP_CTL_EL0); + break; + case SYS_CNTP_CVAL_EL02: + case SYS_CNTP_CVAL_EL0: + val = __vcpu_sys_reg(vcpu, CNTP_CVAL_EL0); + break; + case SYS_CNTV_CTL_EL02: + case SYS_CNTV_CTL_EL0: + val = __vcpu_sys_reg(vcpu, CNTV_CTL_EL0); + break; + case SYS_CNTV_CVAL_EL02: + case SYS_CNTV_CVAL_EL0: + val = __vcpu_sys_reg(vcpu, CNTV_CVAL_EL0); + break; + default: + return false; + } + + vcpu_set_reg(vcpu, kvm_vcpu_sys_get_rt(vcpu), val); + __kvm_skip_instr(vcpu); + + return true; +} + static bool kvm_hyp_handle_sysreg_vhe(struct kvm_vcpu *vcpu, u64 *exit_code) { if (kvm_hyp_handle_tlbi_el1(vcpu, exit_code)) return true; + if (kvm_hyp_handle_ecv(vcpu, exit_code)) + return true; + return kvm_hyp_handle_sysreg(vcpu, exit_code); }