From patchwork Wed Sep 11 14:34:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fares Mehanna X-Patchwork-Id: 13800696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B375FEE49BC for ; Wed, 11 Sep 2024 14:38:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4542794004E; Wed, 11 Sep 2024 10:38:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DD56940021; Wed, 11 Sep 2024 10:38:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22F4D94004E; Wed, 11 Sep 2024 10:38:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id ED5BE940021 for ; Wed, 11 Sep 2024 10:38:08 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9F9EB41542 for ; Wed, 11 Sep 2024 14:38:08 +0000 (UTC) X-FDA: 82552712256.24.8534837 Received: from smtp-fw-80007.amazon.com (smtp-fw-80007.amazon.com [99.78.197.218]) by imf06.hostedemail.com (Postfix) with ESMTP id 997F3180004 for ; Wed, 11 Sep 2024 14:38:06 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=VNeBsVfc; dmarc=pass (policy=quarantine) header.from=amazon.de; spf=pass (imf06.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 99.78.197.218 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726065482; a=rsa-sha256; cv=none; b=oenNOOB/A4Hi0TNStSPPMMqEx7vc+/tcY/SYKz7qR+OwmU4NIyTSOdJcFlvvc2NpTSjZbY U0RDFXksicZwCXUTBBcfW374yHlhgW8RhQ3AQpFnzufRyrzEhJdVhwaWHbEhjkZXYLAHpg HpDZ+GVeU3LDmERuFhRP9vUPdO+J5oQ= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=VNeBsVfc; dmarc=pass (policy=quarantine) header.from=amazon.de; spf=pass (imf06.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 99.78.197.218 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726065482; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Hqkci3dj0GUqOn7J4NBabiUfpbZVZgBkN05sOca7hnY=; b=SZUJdBBK/rPbPkqt86g/+dz4+1zOnbs+w7dHdvcb8jT69kIBT3umnNH5NP8WMo/aabi77G rmHAl/kNNgGo8V8hT+jLwZTSi+eqC83UCWiCSJgcj9z8ILzlGgFFk1zPne/NoV6Uhy9sUr whU8FZYSJrNlxBg1JzPmnm47c+oZzJM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1726065487; x=1757601487; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Hqkci3dj0GUqOn7J4NBabiUfpbZVZgBkN05sOca7hnY=; b=VNeBsVfcLxdsajkuQf/MCCjTETZVKDFNUmJcN3Gm3d4jfBxA/iD7hgD3 hoiKr3QcHAw4Hr5+38wNsc7ALcYTJZP2UXFOvLv1AOQsDCCb/BR1gnpPR HnjZU9QjTlD/JEIL6y20sWL+3V6+J1Pg5gygt7/2IRCpttDSYLhYg77h1 g=; X-IronPort-AV: E=Sophos;i="6.10,220,1719878400"; d="scan'208";a="329956973" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.25.36.210]) by smtp-border-fw-80007.pdx80.corp.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2024 14:38:05 +0000 Received: from EX19MTAEUC001.ant.amazon.com [10.0.17.79:41644] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.25.181:2525] with esmtp (Farcaster) id 4f621809-6571-476e-b300-f14101ba8a55; Wed, 11 Sep 2024 14:38:03 +0000 (UTC) X-Farcaster-Flow-ID: 4f621809-6571-476e-b300-f14101ba8a55 Received: from EX19D007EUA001.ant.amazon.com (10.252.50.133) by EX19MTAEUC001.ant.amazon.com (10.252.51.193) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Wed, 11 Sep 2024 14:38:00 +0000 Received: from EX19MTAUEC001.ant.amazon.com (10.252.135.222) by EX19D007EUA001.ant.amazon.com (10.252.50.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Wed, 11 Sep 2024 14:37:59 +0000 Received: from dev-dsk-faresx-1b-27755bf1.eu-west-1.amazon.com (10.253.79.181) by mail-relay.amazon.com (10.252.135.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Wed, 11 Sep 2024 14:37:56 +0000 From: Fares Mehanna To: CC: , Fares Mehanna , "Marc Zyngier" , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , "Will Deacon" , Andrew Morton , "Kemeng Shi" , =?utf-8?q?Pierre-Cl=C3=A9ment_Tos?= =?utf-8?q?i?= , Ard Biesheuvel , Mark Rutland , Javier Martinez Canillas , "Arnd Bergmann" , Fuad Tabba , Mark Brown , Joey Gouly , Kristina Martsenko , Randy Dunlap , "Bjorn Helgaas" , Jean-Philippe Brucker , "Mike Rapoport (IBM)" , "David Hildenbrand" , Roman Kagan , "moderated list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , "open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , open list , "open list:MEMORY MANAGEMENT" Subject: [RFC PATCH 7/7] arm64: KVM: Allocate vCPU fp-regs dynamically on VHE and KERNEL_SECRETMEM enabled systems Date: Wed, 11 Sep 2024 14:34:06 +0000 Message-ID: <20240911143421.85612-8-faresx@amazon.de> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240911143421.85612-1-faresx@amazon.de> References: <20240911143421.85612-1-faresx@amazon.de> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 7gjtrmsopjrqxffxx1f5yhbu53bojtcq X-Rspamd-Queue-Id: 997F3180004 X-Rspamd-Server: rspam02 X-HE-Tag: 1726065486-285593 X-HE-Meta: U2FsdGVkX1/olbb64ts/HkD2u+hT8O2BL4AvJEq9I4qOS/NlUHKC/MUcIrvJQzPUpwBFXJjKEKeZ/qleDtkwcKIQBXfhogJ+V8S08/68XaAGR9nv4AnsJHHshI6369ItIklisrFMFsQ1d33P1PnrsbhLxCZMRaUK0mMaZXki5PcQLSq619pn5Jf7kxezWQHzLt8cN6k0XRXLe9pjiO6SZv2cgWR/SuxQoiHzM5xaOK7JW3NoK2moxxzO10NKi2hlZTygufBawynuhiGekUU1FqvM/pTFYqjnTtBASZKhP7WycDhDVOIQsTmZSfTkLvpu0tME6+qbYVWH5V/1iZ3Er15VVn/YlVfEx07jpYN/Y+y2MM1sWW/pa34jQoNNFX/brRO7nH39z6rr3Duvqs0WH0CU3mw17DSpeiIpBBF1AvrvgwZ4DM6+d0Mha8u3I44lR5u/A3V2TEpnXx1MNOmtQKI826bNyH2WtvvbaGxbLYu+vsq5FIVN8jWZSpX2AO3Jz9rLSD6QSoi6C6SrZdUdxpyd447/w7NLK9dHOYQxf82Xy/tjc2R49h7GKx7ik4Coxfv1Qo6rC35n3ZQYCxuLcR0906Af5Kd+2evKlE+yFTH2GO8/fAQ0owt7Us1yddoNUiQHf5X5cgvN5l/XFEimVXXm3vQf4lnd0RDfSzQkMosxut1OG2iu4PUCNnUElVXX3aOaWJlLE0x8o1VBhXfGz3Mz2ijknh+o/N9/30MW/zEFylB+PYLCUloFYPe0kdPby16vBPG4VGe1duP+uvkhLpGwrlRhyB2ZiT0C0mu8cbKcdXRywuX+efhmUqMASiK6fMN18I2vazL0yxSS6om51yFdOs3s1SOUOrENFA+Iz8x3eMzaQ/x8I9vX6yV/+AGMtfh0qsejZZhvJNkWo8UiT4YGeB8mqOP3a1EpP1aChv/e3+Nuajdxzpw4qaPaiwkm4axaTe2+KUHjsximoHm YNKZAA5S E7xKw5FiYrjqGElkrxfhdRq6zDQdErBU44no7RI2vRdhT2Cx5SKAQND37mzdFaEgpDeU8Ox9+7MTJ/nlAVjBBx/rVha/lEzjK6ahw4tycLvmPT9f9KnEynrTHpTHtLRX7jlpORRGn0FWCyWvmFoYobIOevJoc1IvZz6shQqV1ViH+hCtJPAxzMYAQrc0j2AcM4yJVFEpUTMwOj0dccyxhHcIBaRMcRt4TN83Or4iLVjwO8yjEnApdisa4x5yutUqSzXgDqHZFuIoe2nvSgdYZkTfnRsfA4w+c/JpRJB8ua6eCooN9SjMVK6Pafr3waJ1CflDRQQ/JSgnpQBQorW64OPYk6WHscXM16Ug/zXQz6iqj+7IYLxoBGYajEzXBo+1HTAWCG+mQ/M6QUtIlAfosX4phUYUbsGoGZgh0LfbMR8Pw7UvxpW0Zu8flsqT6z11aI/mC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similar to what was done in this commit: "arm64: KVM: Allocate vCPU gp-regs dynamically on VHE and KERNEL_SECRETMEM enabled systems" We're moving fp-regs to dynamic memory for systems supporting VHE and compiled with KERNEL_SECRETMEM support. Otherwise, we will use the "fp_regs_storage" struct embedded in the vCPU context. Accessing fp-regs embedded in the vCPU context without de-reference is done as: add \regs, \ctxt, #offsetof(struct kvm_cpu_context, fp_regs_storage) Accessing the dynamically allocated fp-regs with de-reference is done as: ldr \regs, [\ctxt, #offsetof(struct kvm_cpu_context, fp_regs)] Signed-off-by: Fares Mehanna --- arch/arm64/include/asm/kvm_host.h | 16 ++++++++++++++-- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kvm/arm.c | 29 +++++++++++++++++++++++++++-- arch/arm64/kvm/va_layout.c | 23 +++++++++++++++++++---- 4 files changed, 61 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index e8ed2c12479f..4132c57d7e69 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -550,7 +550,9 @@ struct kvm_cpu_context { u64 spsr_irq; u64 spsr_fiq; - struct user_fpsimd_state fp_regs; + struct user_fpsimd_state *fp_regs; + struct user_fpsimd_state fp_regs_storage; + struct secretmem_area *fp_regs_area; u64 sys_regs[NR_SYS_REGS]; @@ -968,7 +970,17 @@ static __always_inline struct user_pt_regs *ctxt_gp_regs(const struct kvm_cpu_co return regs; } #define vcpu_gp_regs(v) (ctxt_gp_regs(&(v)->arch.ctxt)) -#define ctxt_fp_regs(ctxt) (&(ctxt).fp_regs) + +static __always_inline struct user_fpsimd_state *ctxt_fp_regs(const struct kvm_cpu_context *ctxt) +{ + struct user_fpsimd_state *fp_regs = (void *) ctxt; + asm volatile(ALTERNATIVE_CB("add %0, %0, %1\n", + ARM64_HAS_VIRT_HOST_EXTN, + kvm_update_ctxt_fp_regs) + : "+r" (fp_regs) + : "I" (offsetof(struct kvm_cpu_context, fp_regs_storage))); + return fp_regs; +} #define vcpu_fp_regs(v) (ctxt_fp_regs(&(v)->arch.ctxt)) /* diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index e3bb626e299c..904573598e0f 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -87,6 +87,7 @@ KVM_NVHE_ALIAS(kvm_update_va_mask); KVM_NVHE_ALIAS(kvm_get_kimage_voffset); KVM_NVHE_ALIAS(kvm_compute_final_ctr_el0); KVM_NVHE_ALIAS(kvm_update_ctxt_gp_regs); +KVM_NVHE_ALIAS(kvm_update_ctxt_fp_regs); KVM_NVHE_ALIAS(spectre_bhb_patch_loop_iter); KVM_NVHE_ALIAS(spectre_bhb_patch_loop_mitigation_enable); KVM_NVHE_ALIAS(spectre_bhb_patch_wa3); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 7542af3f766a..17b42e9099c3 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -477,6 +477,14 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) if (!vcpu->arch.ctxt.regs_area) return -ENOMEM; vcpu->arch.ctxt.regs = vcpu->arch.ctxt.regs_area->ptr; + + pages_needed = (sizeof(*vcpu_fp_regs(vcpu)) + PAGE_SIZE - 1) / PAGE_SIZE; + vcpu->arch.ctxt.fp_regs_area = secretmem_allocate_pages(fls(pages_needed - 1)); + if (!vcpu->arch.ctxt.fp_regs_area) { + err = -ENOMEM; + goto free_vcpu_ctxt; + } + vcpu->arch.ctxt.fp_regs = vcpu->arch.ctxt.fp_regs_area->ptr; } /* Set up the timer */ @@ -504,8 +512,10 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) return kvm_share_hyp(vcpu, vcpu + 1); free_vcpu_ctxt: - if (kvm_use_dynamic_regs()) + if (kvm_use_dynamic_regs()) { secretmem_release_pages(vcpu->arch.ctxt.regs_area); + secretmem_release_pages(vcpu->arch.ctxt.fp_regs_area); + } return err; } @@ -524,8 +534,10 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) kvm_vgic_vcpu_destroy(vcpu); kvm_arm_vcpu_destroy(vcpu); - if (kvm_use_dynamic_regs()) + if (kvm_use_dynamic_regs()) { secretmem_release_pages(vcpu->arch.ctxt.regs_area); + secretmem_release_pages(vcpu->arch.ctxt.fp_regs_area); + } } void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) @@ -2729,12 +2741,25 @@ static int init_hyp_hve_mode(void) per_cpu(kvm_host_data, cpu).host_ctxt.regs = kvm_host_data_regs; } + /* Allocate fp-regs */ + for_each_possible_cpu(cpu) { + void *kvm_host_data_regs; + + kvm_host_data_regs = kzalloc(sizeof(struct user_fpsimd_state), GFP_KERNEL); + if (!kvm_host_data_regs) { + err = -ENOMEM; + goto free_regs; + } + per_cpu(kvm_host_data, cpu).host_ctxt.fp_regs = kvm_host_data_regs; + } + return 0; free_regs: for_each_possible_cpu(cpu) { kfree(per_cpu(kvm_hyp_ctxt, cpu).regs); kfree(per_cpu(kvm_host_data, cpu).host_ctxt.regs); + kfree(per_cpu(kvm_host_data, cpu).host_ctxt.fp_regs); } return err; diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c index fcef7e89d042..ba1030fa5b08 100644 --- a/arch/arm64/kvm/va_layout.c +++ b/arch/arm64/kvm/va_layout.c @@ -185,10 +185,12 @@ void __init kvm_update_va_mask(struct alt_instr *alt, } } -void __init kvm_update_ctxt_gp_regs(struct alt_instr *alt, - __le32 *origptr, __le32 *updptr, int nr_inst) +static __always_inline void __init kvm_update_ctxt_regs(struct alt_instr *alt, + __le32 *origptr, + __le32 *updptr, + int nr_inst, u32 imm) { - u32 rd, rn, imm, insn, oinsn; + u32 rd, rn, insn, oinsn; BUG_ON(nr_inst != 1); @@ -198,7 +200,6 @@ void __init kvm_update_ctxt_gp_regs(struct alt_instr *alt, oinsn = le32_to_cpu(origptr[0]); rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn); rn = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RN, oinsn); - imm = offsetof(struct kvm_cpu_context, regs); insn = aarch64_insn_gen_load_store_imm(rd, rn, imm, AARCH64_INSN_SIZE_64, @@ -208,6 +209,20 @@ void __init kvm_update_ctxt_gp_regs(struct alt_instr *alt, updptr[0] = cpu_to_le32(insn); } +void __init kvm_update_ctxt_gp_regs(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst) +{ + u32 offset = offsetof(struct kvm_cpu_context, regs); + kvm_update_ctxt_regs(alt, origptr, updptr, nr_inst, offset); +} + +void __init kvm_update_ctxt_fp_regs(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst) +{ + u32 offset = offsetof(struct kvm_cpu_context, fp_regs); + kvm_update_ctxt_regs(alt, origptr, updptr, nr_inst, offset); +} + void kvm_patch_vector_branch(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst) {