From patchwork Wed Sep 11 14:34:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fares Mehanna X-Patchwork-Id: 13800693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF3CBEE49BA for ; Wed, 11 Sep 2024 14:36:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42D1F94004A; Wed, 11 Sep 2024 10:36:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DB31940021; Wed, 11 Sep 2024 10:36:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27C6294004A; Wed, 11 Sep 2024 10:36:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0A13E940021 for ; Wed, 11 Sep 2024 10:36:58 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B751C81370 for ; Wed, 11 Sep 2024 14:36:57 +0000 (UTC) X-FDA: 82552709274.21.C2AD2A8 Received: from smtp-fw-52003.amazon.com (smtp-fw-52003.amazon.com [52.119.213.152]) by imf13.hostedemail.com (Postfix) with ESMTP id CE05520005 for ; Wed, 11 Sep 2024 14:36:55 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=l1LNfeOx; spf=pass (imf13.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 52.119.213.152 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de"; dmarc=pass (policy=quarantine) header.from=amazon.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726065311; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gUT0NHH1rkNY0jIUGubcfNpyrpnEMMZzd9a2sKPpzdA=; b=E2aICx0IhZq5b2reyDy4RNREUMTGf5oHVvLS2dY86supMl8U22+eC9QOBZfSMSveTWLEdY H57lJDzyo2WTvnQQTzcXisQCfy3zg3ckjKDNL+ASYg1YLsE9pTJYdlSwlLmZlKpdhP0isF CFCpM9uwg5B7stcUO0U094s2mjgXEpI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726065311; a=rsa-sha256; cv=none; b=xFDDu7VIEXTn/cvH7qgdDY6m+3XYBHdt3yxCY8qduS0rVxw5+BPs8NKHzCZXchN56s+0JM QkA933ZRp+w9TsVZ9QQq4Eyb6yz7MtqyFGRt3tY+tmvkQ4VJlogHGYzFbLiq+nJk+34ZhM EeVtn4DEEIF8zhbE1iO2YU4S39kOE6g= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=l1LNfeOx; spf=pass (imf13.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 52.119.213.152 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de"; dmarc=pass (policy=quarantine) header.from=amazon.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1726065416; x=1757601416; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gUT0NHH1rkNY0jIUGubcfNpyrpnEMMZzd9a2sKPpzdA=; b=l1LNfeOxKv1zSTZ050NCcTkkz0oFWBQhnQHI8qs0OF5Gr47WiUb0n95l 0gYALIm7v2NL36MK+ZEIHT05vpZe7T51/xndOEcKJr0BWBTX3A3bk7xMT ENXjS+T7zUi83kuRfQ4hkonBaiQP8xp9T2+SBTXKukMuIr+6CgpNLqnCT E=; X-IronPort-AV: E=Sophos;i="6.10,220,1719878400"; d="scan'208";a="24916368" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO smtpout.prod.us-east-1.prod.farcaster.email.amazon.dev) ([10.43.8.6]) by smtp-border-fw-52003.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2024 14:36:54 +0000 Received: from EX19MTAEUA001.ant.amazon.com [10.0.10.100:10017] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.27.59:2525] with esmtp (Farcaster) id f9f29f1f-caf9-4ac7-bc5d-27f9e417256f; Wed, 11 Sep 2024 14:36:52 +0000 (UTC) X-Farcaster-Flow-ID: f9f29f1f-caf9-4ac7-bc5d-27f9e417256f Received: from EX19D007EUA001.ant.amazon.com (10.252.50.133) by EX19MTAEUA001.ant.amazon.com (10.252.50.223) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Wed, 11 Sep 2024 14:36:52 +0000 Received: from EX19MTAUEC001.ant.amazon.com (10.252.135.222) by EX19D007EUA001.ant.amazon.com (10.252.50.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Wed, 11 Sep 2024 14:36:51 +0000 Received: from dev-dsk-faresx-1b-27755bf1.eu-west-1.amazon.com (10.253.79.181) by mail-relay.amazon.com (10.252.135.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Wed, 11 Sep 2024 14:36:49 +0000 From: Fares Mehanna To: CC: , Fares Mehanna , "Marc Zyngier" , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , "Will Deacon" , Andrew Morton , "Kemeng Shi" , =?utf-8?q?Pierre-Cl=C3=A9ment_Tos?= =?utf-8?q?i?= , Ard Biesheuvel , Mark Rutland , Javier Martinez Canillas , "Arnd Bergmann" , Fuad Tabba , Mark Brown , Joey Gouly , Kristina Martsenko , Randy Dunlap , "Bjorn Helgaas" , Jean-Philippe Brucker , "Mike Rapoport (IBM)" , "David Hildenbrand" , Roman Kagan , "moderated list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , "open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , open list , "open list:MEMORY MANAGEMENT" Subject: [RFC PATCH 4/7] KVM: Refactor Assembly-code to access vCPU gp-registers through a macro Date: Wed, 11 Sep 2024 14:34:03 +0000 Message-ID: <20240911143421.85612-5-faresx@amazon.de> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240911143421.85612-1-faresx@amazon.de> References: <20240911143421.85612-1-faresx@amazon.de> MIME-Version: 1.0 X-Stat-Signature: c6kwh661x53umfyys1znqiyjzk156ycu X-Rspamd-Queue-Id: CE05520005 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1726065415-760725 X-HE-Meta: U2FsdGVkX1+E5nwvVmRX9wRDjQLBUN1CHzVRjhCuZ5cGwmJkvqYJ25fXbz+igMH0KqvE7RxNMYuwXqOkESf/4FPB2w5elv4aicEfzOWow5rCFd0HcZn/p2U9hlh2hizR8b/GUg1rRCsVHegymMGiHBcObLxr2riAeC6uQY5o1zfwMZ3wLwmC+oKHPiW4PVjQJnmwc4TqS9TOLhVoGu6M7Qnxh1Co8bn/5oV0DRJPZ2sMfRjVmvi+2FG7t+Vpu1Xbw+WOWUy5F1NCu9aN7FmM12vB8I3+Hby+NE1+aXQk1qGMgAz1ulg59K0TH4rEXYSLhrNhF7soqoGHKGx5vB/zMh96iqj53nKakxOyN5ZX+w66r4dLLcxkc3oORk+NvK8OfQLPMQrfs/elCJS3nCD9Z3A17velkcjt/xzmcC8ABKCRiIfgWGyecJ+0B8Bgqb+IOJMIFjE5hleEEGhxGohdwUIGTSyzaHuW5YqhEPwXON1OoSCotsybpcbk4A6m1fUqgRmfX0iv48oqid4Psd+0pY5DlNDhlGtBXoHYPYEo2IlC8fi8yqmAbMZSCcGOJkhflhgWinUNtVN07GfZCegJqHwWlFJmnWNO6TaxeDjMBJ6WcwEPk8M3lnaZ3ar8bd1AeUiOehxKLDVGQVJs8SGWfYTYql6tqZ2OlUetWgKwwTHLanhVbRmklR5VHlmi1uXTuLJsnxJjH4iLrgEk9lJL5LBVvQrjGNyeUGS9M5v+KX02cJNcmpBewjJBtsYY7WQWm54+CgQdBKCEoGkk3Diy5eYz89ob+8VevkwWF0Da2BDlkDBKLW0klXugl8szrQpccQ6cAo99fSRPoKQdeG41EVMnkiXVPfWLcw+CMe+Z0iMi0cDFY97lnyv0eSvIm+V6kciznY+OiNkGSWdHCOQcZKFIgAOQhA5hW6tgUlCP8cT3J+/8dkLGHHcwDg7bN9tQmdFdh0dxh82v5HBl1ZL CxrBZR7O Si97pMTGRUOib6nAtic1CMWI87okCGYb/tzYHeFxXLDemGWGsQMLzeCiZg4PKZsFBBSxkNzaEPINGYuYwwR4u6BjnRyCwBIjvOhVNSqYLZnJ0tsOfmTJKSXKwh1v9En0uLi8VzveJBDV/Nm6XlPpA4Rtoa+yXVu8nikEG0I3ZblJaY+uuOotyIB7ZjabAzpJu4deD111K1Bm1PQLfFte/hjMhiOmekO9VXis1WcEdQQNQGAG/NLZ5MFOpuJEsYCwA6b9Os0xNGdZqIbIRHHIzc24HLj9dncjNhPdrCORJoZvz/fO2RTzuAgnjjgFzycZGM7GEb8sZuuJYMuLQSkMWne8beXDD3ms8Z7BlAyWCynnrJQSZ06xmbrC531YbR2jR8ZNbhBJ0jjE+z8Eek8l0WuEKuw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Right now assembly code accesses vCPU gp-regs directly from the context struct "struct kvm_cpu_context" using "CPU_XREG_OFFSET()". Since we want to move gp-regs to dynamic memory, we can no longer assume that gp-regs will be embedded in the context struct, thus split the access to two steps. The first is to get the gp-regs from the context using the assembly macro "get_ctxt_gp_regs". And the second is to access the gp-registers directly from within the "struct user_pt_regs" by removing the offset "CPU_USER_PT_REGS" from the access macro "CPU_XREG_OFFSET()". I also changed variable naming and comments where appropriate. Signed-off-by: Fares Mehanna --- arch/arm64/include/asm/kvm_asm.h | 48 +++++++++++++++++--------------- arch/arm64/kvm/hyp/entry.S | 15 ++++++++++ arch/arm64/kvm/hyp/nvhe/host.S | 20 ++++++++++--- 3 files changed, 57 insertions(+), 26 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 2181a11b9d92..fa4fb642a5f5 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -313,6 +313,10 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt, str \vcpu, [\ctxt, #HOST_CONTEXT_VCPU] .endm +.macro get_ctxt_gp_regs ctxt, regs + add \regs, \ctxt, #CPU_USER_PT_REGS +.endm + /* * KVM extable for unexpected exceptions. * Create a struct kvm_exception_table_entry output to a section that can be @@ -329,7 +333,7 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt, .popsection .endm -#define CPU_XREG_OFFSET(x) (CPU_USER_PT_REGS + 8*x) +#define CPU_XREG_OFFSET(x) (8 * (x)) #define CPU_LR_OFFSET CPU_XREG_OFFSET(30) #define CPU_SP_EL0_OFFSET (CPU_LR_OFFSET + 8) @@ -337,34 +341,34 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt, * We treat x18 as callee-saved as the host may use it as a platform * register (e.g. for shadow call stack). */ -.macro save_callee_saved_regs ctxt - str x18, [\ctxt, #CPU_XREG_OFFSET(18)] - stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] - stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] - stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)] - stp x25, x26, [\ctxt, #CPU_XREG_OFFSET(25)] - stp x27, x28, [\ctxt, #CPU_XREG_OFFSET(27)] - stp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)] +.macro save_callee_saved_regs regs + str x18, [\regs, #CPU_XREG_OFFSET(18)] + stp x19, x20, [\regs, #CPU_XREG_OFFSET(19)] + stp x21, x22, [\regs, #CPU_XREG_OFFSET(21)] + stp x23, x24, [\regs, #CPU_XREG_OFFSET(23)] + stp x25, x26, [\regs, #CPU_XREG_OFFSET(25)] + stp x27, x28, [\regs, #CPU_XREG_OFFSET(27)] + stp x29, lr, [\regs, #CPU_XREG_OFFSET(29)] .endm -.macro restore_callee_saved_regs ctxt - // We require \ctxt is not x18-x28 - ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)] - ldp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] - ldp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] - ldp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)] - ldp x25, x26, [\ctxt, #CPU_XREG_OFFSET(25)] - ldp x27, x28, [\ctxt, #CPU_XREG_OFFSET(27)] - ldp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)] +.macro restore_callee_saved_regs regs + // We require \regs is not x18-x28 + ldr x18, [\regs, #CPU_XREG_OFFSET(18)] + ldp x19, x20, [\regs, #CPU_XREG_OFFSET(19)] + ldp x21, x22, [\regs, #CPU_XREG_OFFSET(21)] + ldp x23, x24, [\regs, #CPU_XREG_OFFSET(23)] + ldp x25, x26, [\regs, #CPU_XREG_OFFSET(25)] + ldp x27, x28, [\regs, #CPU_XREG_OFFSET(27)] + ldp x29, lr, [\regs, #CPU_XREG_OFFSET(29)] .endm -.macro save_sp_el0 ctxt, tmp +.macro save_sp_el0 regs, tmp mrs \tmp, sp_el0 - str \tmp, [\ctxt, #CPU_SP_EL0_OFFSET] + str \tmp, [\regs, #CPU_SP_EL0_OFFSET] .endm -.macro restore_sp_el0 ctxt, tmp - ldr \tmp, [\ctxt, #CPU_SP_EL0_OFFSET] +.macro restore_sp_el0 regs, tmp + ldr \tmp, [\regs, #CPU_SP_EL0_OFFSET] msr sp_el0, \tmp .endm diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index 4433a234aa9b..628a123bcdc1 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -28,6 +28,9 @@ SYM_FUNC_START(__guest_enter) adr_this_cpu x1, kvm_hyp_ctxt, x2 + // Get gp-regs pointer from the context + get_ctxt_gp_regs x1, x1 + // Store the hyp regs save_callee_saved_regs x1 @@ -62,6 +65,9 @@ alternative_else_nop_endif // when this feature is enabled for kernel code. ptrauth_switch_to_guest x29, x0, x1, x2 + // Get gp-regs pointer from the context + get_ctxt_gp_regs x29, x29 + // Restore the guest's sp_el0 restore_sp_el0 x29, x0 @@ -108,6 +114,7 @@ SYM_INNER_LABEL(__guest_exit_panic, SYM_L_GLOBAL) // current state is saved to the guest context but it will only be // accurate if the guest had been completely restored. adr_this_cpu x0, kvm_hyp_ctxt, x1 + get_ctxt_gp_regs x0, x0 adr_l x1, hyp_panic str x1, [x0, #CPU_XREG_OFFSET(30)] @@ -120,6 +127,7 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL) // vcpu x0-x1 on the stack add x1, x1, #VCPU_CONTEXT + get_ctxt_gp_regs x1, x1 ALTERNATIVE(nop, SET_PSTATE_PAN(1), ARM64_HAS_PAN, CONFIG_ARM64_PAN) @@ -145,6 +153,10 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL) // Store the guest's sp_el0 save_sp_el0 x1, x2 + // Recover vCPU context to x1 + get_vcpu_ptr x1, x2 + add x1, x1, #VCPU_CONTEXT + adr_this_cpu x2, kvm_hyp_ctxt, x3 // Macro ptrauth_switch_to_hyp format: @@ -157,6 +169,9 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL) // mte_switch_to_hyp(g_ctxt, h_ctxt, reg1) mte_switch_to_hyp x1, x2, x3 + // Get gp-regs pointer from the context + get_ctxt_gp_regs x2, x2 + // Restore hyp's sp_el0 restore_sp_el0 x2, x3 diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S index 3d610fc51f4d..31afa7396294 100644 --- a/arch/arm64/kvm/hyp/nvhe/host.S +++ b/arch/arm64/kvm/hyp/nvhe/host.S @@ -17,6 +17,12 @@ SYM_FUNC_START(__host_exit) get_host_ctxt x0, x1 + /* Keep host context in x1 */ + mov x1, x0 + + /* Get gp-regs pointer from the context */ + get_ctxt_gp_regs x0, x0 + /* Store the host regs x2 and x3 */ stp x2, x3, [x0, #CPU_XREG_OFFSET(2)] @@ -36,7 +42,10 @@ SYM_FUNC_START(__host_exit) /* Store the host regs x18-x29, lr */ save_callee_saved_regs x0 - /* Save the host context pointer in x29 across the function call */ + /* Save the host context pointer in x28 across the function call */ + mov x28, x1 + + /* Save the host gp-regs pointer in x29 across the function call */ mov x29, x0 #ifdef CONFIG_ARM64_PTR_AUTH_KERNEL @@ -46,7 +55,7 @@ alternative_else_nop_endif alternative_if ARM64_KVM_PROTECTED_MODE /* Save kernel ptrauth keys. */ - add x18, x29, #CPU_APIAKEYLO_EL1 + add x18, x28, #CPU_APIAKEYLO_EL1 ptrauth_save_state x18, x19, x20 /* Use hyp keys. */ @@ -58,6 +67,7 @@ alternative_else_nop_endif __skip_pauth_save: #endif /* CONFIG_ARM64_PTR_AUTH_KERNEL */ + mov x0, x28 bl handle_trap __host_enter_restore_full: @@ -68,7 +78,7 @@ b __skip_pauth_restore alternative_else_nop_endif alternative_if ARM64_KVM_PROTECTED_MODE - add x18, x29, #CPU_APIAKEYLO_EL1 + add x18, x28, #CPU_APIAKEYLO_EL1 ptrauth_restore_state x18, x19, x20 alternative_else_nop_endif __skip_pauth_restore: @@ -101,7 +111,8 @@ SYM_FUNC_END(__host_exit) * void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt); */ SYM_FUNC_START(__host_enter) - mov x29, x0 + mov x28, x0 + get_ctxt_gp_regs x0, x29 b __host_enter_restore_full SYM_FUNC_END(__host_enter) @@ -141,6 +152,7 @@ SYM_FUNC_START(__hyp_do_panic) /* Enter the host, conditionally restoring the host context. */ cbz x29, __host_enter_without_restoring + get_ctxt_gp_regs x29, x29 b __host_enter_for_panic SYM_FUNC_END(__hyp_do_panic)