From patchwork Wed Sep 11 14:34:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fares Mehanna X-Patchwork-Id: 13800674 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DBA96EE49B7 for ; Wed, 11 Sep 2024 14:35:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 786AA940047; Wed, 11 Sep 2024 10:35:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 70FA7940021; Wed, 11 Sep 2024 10:35:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B00C940047; Wed, 11 Sep 2024 10:35:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3B88D940021 for ; Wed, 11 Sep 2024 10:35:52 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E5123161905 for ; Wed, 11 Sep 2024 14:35:51 +0000 (UTC) X-FDA: 82552706502.29.C84E356 Received: from smtp-fw-52005.amazon.com (smtp-fw-52005.amazon.com [52.119.213.156]) by imf18.hostedemail.com (Postfix) with ESMTP id 0181C1C0019 for ; Wed, 11 Sep 2024 14:35:49 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=Ql7vwXSp; dmarc=pass (policy=quarantine) header.from=amazon.de; spf=pass (imf18.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 52.119.213.156 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726065273; a=rsa-sha256; cv=none; b=fWs26cJLUvnPIenj0edXMatd3+6D4WRTaqFeaCoGXkVvOco9hU6tirTxEJ/Gg1hPipsyD9 GfkCe9pPTrwqQeQFbBP4cHmOzKULECJN9Os+b/LwglmyDRFk7zQ4cbjgPHdkE6gjO9dFPf uDAbEqa2Bo6kjBkzw8iH5tPKapyruYY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=Ql7vwXSp; dmarc=pass (policy=quarantine) header.from=amazon.de; spf=pass (imf18.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 52.119.213.156 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726065273; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M6VWy3IvZCNc4zvvTM+7ldu6BbKiq6bFPxrlGyI8xuQ=; b=cMi3tEosJx1unmn73gdE5jW9T06IX1qWJ90lFtYKvIF4/zpugDD1+ktRFr3OMm0ywm8L58 QUk2J3npL7ERiHamRLdZXCVBoFVlALhIp7zaNEOwdBxUQEhIr2bs2qmuO1ytDKCTTeWltB 0at7kOvQRYx8dybRDtOHsg13QDCHGls= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1726065350; x=1757601350; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=M6VWy3IvZCNc4zvvTM+7ldu6BbKiq6bFPxrlGyI8xuQ=; b=Ql7vwXSpGRkoQ47dOei6pI1jd8YmZY/5RvPOfZC4Fzr2X7cmX1NS8Wf7 PVPaSsWqQT4TGb25QKAuSrV4J7OSK2brJ9nqjiO0CZ0sIs1MRngECbQpg eca71bzQ0t9DJki1x/4Bx4kQk3kzm5CqbDt/Lpwn94g+ECkWhvszv1Bpx Q=; X-IronPort-AV: E=Sophos;i="6.10,220,1719878400"; d="scan'208";a="679649476" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO smtpout.prod.us-east-1.prod.farcaster.email.amazon.dev) ([10.43.8.6]) by smtp-border-fw-52005.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2024 14:35:48 +0000 Received: from EX19MTAEUC002.ant.amazon.com [10.0.43.254:50131] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.39.168:2525] with esmtp (Farcaster) id ce0e4cf6-8c87-42ac-ae89-33a20b34e603; Wed, 11 Sep 2024 14:35:46 +0000 (UTC) X-Farcaster-Flow-ID: ce0e4cf6-8c87-42ac-ae89-33a20b34e603 Received: from EX19D007EUA004.ant.amazon.com (10.252.50.76) by EX19MTAEUC002.ant.amazon.com (10.252.51.245) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Wed, 11 Sep 2024 14:35:42 +0000 Received: from EX19MTAUEC001.ant.amazon.com (10.252.135.222) by EX19D007EUA004.ant.amazon.com (10.252.50.76) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Wed, 11 Sep 2024 14:35:41 +0000 Received: from dev-dsk-faresx-1b-27755bf1.eu-west-1.amazon.com (10.253.79.181) by mail-relay.amazon.com (10.252.135.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Wed, 11 Sep 2024 14:35:39 +0000 From: Fares Mehanna To: CC: , Fares Mehanna , Roman Kagan , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Andrew Morton , Kemeng Shi , =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= , Ard Biesheuvel , Mark Rutland , "Javier Martinez Canillas" , Arnd Bergmann , Fuad Tabba , Mark Brown , Joey Gouly , Kristina Martsenko , "Randy Dunlap" , Bjorn Helgaas , Jean-Philippe Brucker , "Mike Rapoport (IBM)" , David Hildenbrand , "moderated list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , "open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , open list , "open list:MEMORY MANAGEMENT" Subject: [RFC PATCH 1/7] mseal: expose interface to seal / unseal user memory ranges Date: Wed, 11 Sep 2024 14:34:00 +0000 Message-ID: <20240911143421.85612-2-faresx@amazon.de> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240911143421.85612-1-faresx@amazon.de> References: <20240911143421.85612-1-faresx@amazon.de> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0181C1C0019 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: ocxu9yj6o6k3h7wzydiy64w8tezdzkah X-HE-Tag: 1726065349-300569 X-HE-Meta: U2FsdGVkX18AaL63ErfDEsvKfIs5YHc1Etsmj7kMPMMfxMPsNUhrla9Xn4dC8bv+mEgHeze5gP+daF3L6ty8i+lRJWgZKixZNN3GVVmzSm1LzZ0za5zn3mKVk5tsHlsK85g3Fu4MOinTaHZcWep8Fha0P5H4ZeDi2Qo/E94ZYslRRDCHS5IxkXJtzOk03rAspV0o405cXoObreiRscs9qKOdDuzENSqQOE8YkQgKNXBo+PH6VCp+IrPl+Qv+q2y4W95obimq81JoqwqXhZPmFrQF7up5kE6TEjl9azPMLFyyKwID6bjIPT8q+NOX9ZQElrbm8GvdN2ky8mnDtXrgM5MToVzngxJfE5pOTvhc/GUI5aRJRISYzQiR/2b7GnSQCkaGueBWX/YSlCSCKsQUuTRCGH12OberG95e79ZAb320V3c/pdU17+Ke5/Su37ppnN3VW9gh4ftmF/deZR46LDO+ahKpl1zGBo+SZso2qXXSDV/Hd8HHxgOZazQAUurywamizvqOm5SM9NNdXMHWTxykRKoT93NRNQxNwY0Dmn+sBLblKie3jfwdTqdLxz0f8uZ2pHA1bB+VnLnEuEl/Gs++8NdoMBp/UkZ5398rQQiAZBSE/sct+M1gtBhoZxBvYleoLEs0GlrVIu7noGTH1xuPDlqSn8hCUGwJG1THL7R8nchajVhBoWu+J5gilU/j+fZwoTOAOHvhb97xBtIWFE5vEqMVFfd8k82H3LO5UA1xzj0NID987OkPAvX7dMSnPI35e5dEPebxqs8cRKqNh5ydNYyBDhosNrT0+s7aeGBEqBV/fW23WSXdweEWcBYI0YjAdxgWOvjou/eK+NT/2YfWUznyda3SKibTASiyQy2IYXXlK1ea3GKtlHdoTzntied3YV0/1eaifJJ4GMXWYRH7ps7W8odtH3iJXyNR5cPA0+XL8pcsTkKJMfuQr9qo/y0iCfjYoI/qBDHe56D WzyYq1yh X2Ijl6thxtmYzaGk/a9ZKyzkIdmFg0L3+KuOQOcCE5t8R6mFgJO2t4zgs3K6AKN1xL30LPOELQx/rbGYyyhzs6jPXHdwxNOF9H/wybEBT5S+ce7VuC64pXToffYOPnbD4J/RpLtMnrnPmFg654jKoqLn/TBIdMoCqxLPsXfYzcg2FM/dLJFGw7qxWOJxRm3yN1/0fZFjCnWAeW2hwkNy6OKppSnhEs3MGFLv6ezEUvoFeFUwNiTsnkOz4HbSVnjwUNEVqhF4GyePRQtVbEqCXdX68AtyHo8/mr5Lhi2R1mjDwXsPZY3dxtDLLwh/f8bMFV/FaCD+l7gdbh2HzOua7KYPKDqP4PgXhK9VSWo8sCin3WNPAeY25D/ubkcyOFPFStHJk28mcF2k+ZG+bK06C3FrNGA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To make sure the kernel mm-local mapping is untouched by the user, we will seal the VMA before changing the protection to be used by the kernel. This will guarantee that userspace can't unmap or alter this VMA while it is being used by the kernel. After the kernel is done with the secret memory, it will unseal the VMA to be able to unmap and free it. Unseal operation is not exposed to userspace. Signed-off-by: Fares Mehanna Signed-off-by: Roman Kagan --- mm/internal.h | 7 +++++ mm/mseal.c | 81 ++++++++++++++++++++++++++++++++------------------- 2 files changed, 58 insertions(+), 30 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index b4d86436565b..cf7280d101e9 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1501,6 +1501,8 @@ bool can_modify_mm(struct mm_struct *mm, unsigned long start, unsigned long end); bool can_modify_mm_madv(struct mm_struct *mm, unsigned long start, unsigned long end, int behavior); +/* mm's mmap write lock must be taken before seal/unseal operation */ +int do_mseal(unsigned long start, unsigned long end, bool seal); #else static inline int can_do_mseal(unsigned long flags) { @@ -1518,6 +1520,11 @@ static inline bool can_modify_mm_madv(struct mm_struct *mm, unsigned long start, { return true; } + +static inline int do_mseal(unsigned long start, unsigned long end, bool seal) +{ + return -EINVAL; +} #endif #ifdef CONFIG_SHRINKER_DEBUG diff --git a/mm/mseal.c b/mm/mseal.c index 15bba28acc00..aac9399ffd5d 100644 --- a/mm/mseal.c +++ b/mm/mseal.c @@ -26,6 +26,11 @@ static inline void set_vma_sealed(struct vm_area_struct *vma) vm_flags_set(vma, VM_SEALED); } +static inline void clear_vma_sealed(struct vm_area_struct *vma) +{ + vm_flags_clear(vma, VM_SEALED); +} + /* * check if a vma is sealed for modification. * return true, if modification is allowed. @@ -117,7 +122,7 @@ bool can_modify_mm_madv(struct mm_struct *mm, unsigned long start, unsigned long static int mseal_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, struct vm_area_struct **prev, unsigned long start, - unsigned long end, vm_flags_t newflags) + unsigned long end, vm_flags_t newflags, bool seal) { int ret = 0; vm_flags_t oldflags = vma->vm_flags; @@ -131,7 +136,10 @@ static int mseal_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, goto out; } - set_vma_sealed(vma); + if (seal) + set_vma_sealed(vma); + else + clear_vma_sealed(vma); out: *prev = vma; return ret; @@ -167,9 +175,9 @@ static int check_mm_seal(unsigned long start, unsigned long end) } /* - * Apply sealing. + * Apply sealing / unsealing. */ -static int apply_mm_seal(unsigned long start, unsigned long end) +static int apply_mm_seal(unsigned long start, unsigned long end, bool seal) { unsigned long nstart; struct vm_area_struct *vma, *prev; @@ -191,11 +199,14 @@ static int apply_mm_seal(unsigned long start, unsigned long end) unsigned long tmp; vm_flags_t newflags; - newflags = vma->vm_flags | VM_SEALED; + if (seal) + newflags = vma->vm_flags | VM_SEALED; + else + newflags = vma->vm_flags & ~(VM_SEALED); tmp = vma->vm_end; if (tmp > end) tmp = end; - error = mseal_fixup(&vmi, vma, &prev, nstart, tmp, newflags); + error = mseal_fixup(&vmi, vma, &prev, nstart, tmp, newflags, seal); if (error) return error; nstart = vma_iter_end(&vmi); @@ -204,6 +215,37 @@ static int apply_mm_seal(unsigned long start, unsigned long end) return 0; } +int do_mseal(unsigned long start, unsigned long end, bool seal) +{ + int ret; + + if (end < start) + return -EINVAL; + + if (end == start) + return 0; + + /* + * First pass, this helps to avoid + * partial sealing in case of error in input address range, + * e.g. ENOMEM error. + */ + ret = check_mm_seal(start, end); + if (ret) + goto out; + + /* + * Second pass, this should success, unless there are errors + * from vma_modify_flags, e.g. merge/split error, or process + * reaching the max supported VMAs, however, those cases shall + * be rare. + */ + ret = apply_mm_seal(start, end, seal); + +out: + return ret; +} + /* * mseal(2) seals the VM's meta data from * selected syscalls. @@ -256,7 +298,7 @@ static int apply_mm_seal(unsigned long start, unsigned long end) * * unseal() is not supported. */ -static int do_mseal(unsigned long start, size_t len_in, unsigned long flags) +static int __do_mseal(unsigned long start, size_t len_in, unsigned long flags) { size_t len; int ret = 0; @@ -277,33 +319,12 @@ static int do_mseal(unsigned long start, size_t len_in, unsigned long flags) return -EINVAL; end = start + len; - if (end < start) - return -EINVAL; - - if (end == start) - return 0; if (mmap_write_lock_killable(mm)) return -EINTR; - /* - * First pass, this helps to avoid - * partial sealing in case of error in input address range, - * e.g. ENOMEM error. - */ - ret = check_mm_seal(start, end); - if (ret) - goto out; - - /* - * Second pass, this should success, unless there are errors - * from vma_modify_flags, e.g. merge/split error, or process - * reaching the max supported VMAs, however, those cases shall - * be rare. - */ - ret = apply_mm_seal(start, end); + ret = do_mseal(start, end, true); -out: mmap_write_unlock(current->mm); return ret; } @@ -311,5 +332,5 @@ static int do_mseal(unsigned long start, size_t len_in, unsigned long flags) SYSCALL_DEFINE3(mseal, unsigned long, start, size_t, len, unsigned long, flags) { - return do_mseal(start, len, flags); + return __do_mseal(start, len, flags); } From patchwork Wed Sep 11 14:34:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fares Mehanna X-Patchwork-Id: 13800675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFBFAEE49BA for ; Wed, 11 Sep 2024 14:36:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A052940048; Wed, 11 Sep 2024 10:36:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 42B53940021; Wed, 11 Sep 2024 10:36:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CA14940048; Wed, 11 Sep 2024 10:36:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0BD50940021 for ; Wed, 11 Sep 2024 10:36:16 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8EA671404D8 for ; Wed, 11 Sep 2024 14:36:15 +0000 (UTC) X-FDA: 82552707510.26.F8D7394 Received: from smtp-fw-52004.amazon.com (smtp-fw-52004.amazon.com [52.119.213.154]) by imf01.hostedemail.com (Postfix) with ESMTP id 8E90940010 for ; Wed, 11 Sep 2024 14:36:13 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=jxtw08f9; spf=pass (imf01.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 52.119.213.154 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de"; dmarc=pass (policy=quarantine) header.from=amazon.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726065270; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Y9TjOwsiSXZl9P0/RyFMFJbYNQPYXtbeuapzd5F/ewk=; b=lJ5vDy1H8AVD7WbOS9inAf3q25UpjTWgc0H0j1Evi8kMhQ4UCCd9NPocJ+QwLu5dBCpoDZ 7N2Ov0Txqe0605wLc4BXPdAjBk22aDvJxPsFyTjEc/2hI7q5jDuMFpNndB6roGV3YW+3xl /eTii/EpA5kIjVPCVRyaUttsTiaSDug= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726065270; a=rsa-sha256; cv=none; b=fBHJMPo8E0nFzXo5GMMnSNiQ1zG5watVq+KEMeR0FqEdLWXBj1GdDQRd0rKND+n+P6QvpI t5c0Rb+H4Vha2LF5wIIggkYKlHGDsc6qyIU4U91JjpzfNvo7ZPhqyUI110EEP9UGeCMMOP xQN9R4frppOi4uz3Df+XcVNjMcTIzmo= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=jxtw08f9; spf=pass (imf01.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 52.119.213.154 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de"; dmarc=pass (policy=quarantine) header.from=amazon.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1726065374; x=1757601374; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Y9TjOwsiSXZl9P0/RyFMFJbYNQPYXtbeuapzd5F/ewk=; b=jxtw08f9uvtGnFscZb2dGxwajbFL+rpYEHVl+DAxyCAf0OKY1IpSEZYh tlfsPUJaY41QRPyUK84vLYkdD5tC/t1Sx6gNNhBM9esU2SeLh4nNzWuaf 9giL9wLHLxq468JQoFvH1yJX7mE31GxdMYUQ/By1ert5j3OoaXOoMi4WD c=; X-IronPort-AV: E=Sophos;i="6.10,220,1719878400"; d="scan'208";a="231193640" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.43.8.2]) by smtp-border-fw-52004.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2024 14:36:07 +0000 Received: from EX19MTAEUC001.ant.amazon.com [10.0.17.79:61643] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.38.136:2525] with esmtp (Farcaster) id 3550e7db-38ee-4821-b3d5-b8907b20a81c; Wed, 11 Sep 2024 14:36:06 +0000 (UTC) X-Farcaster-Flow-ID: 3550e7db-38ee-4821-b3d5-b8907b20a81c Received: from EX19D033EUB004.ant.amazon.com (10.252.61.103) by EX19MTAEUC001.ant.amazon.com (10.252.51.193) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Wed, 11 Sep 2024 14:36:01 +0000 Received: from EX19MTAUEC001.ant.amazon.com (10.252.135.222) by EX19D033EUB004.ant.amazon.com (10.252.61.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Wed, 11 Sep 2024 14:36:01 +0000 Received: from dev-dsk-faresx-1b-27755bf1.eu-west-1.amazon.com (10.253.79.181) by mail-relay.amazon.com (10.252.135.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Wed, 11 Sep 2024 14:35:59 +0000 From: Fares Mehanna To: CC: , Fares Mehanna , Roman Kagan , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Andrew Morton , Kemeng Shi , =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= , Ard Biesheuvel , Mark Rutland , "Javier Martinez Canillas" , Arnd Bergmann , Fuad Tabba , Mark Brown , Joey Gouly , Kristina Martsenko , "Randy Dunlap" , Bjorn Helgaas , Jean-Philippe Brucker , "Mike Rapoport (IBM)" , David Hildenbrand , "moderated list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , "open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , open list , "open list:MEMORY MANAGEMENT" Subject: [RFC PATCH 2/7] mm/secretmem: implement mm-local kernel allocations Date: Wed, 11 Sep 2024 14:34:01 +0000 Message-ID: <20240911143421.85612-3-faresx@amazon.de> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240911143421.85612-1-faresx@amazon.de> References: <20240911143421.85612-1-faresx@amazon.de> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8E90940010 X-Stat-Signature: kgw4jgxffeezwhsww7dggfd4eehipdzb X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1726065373-816204 X-HE-Meta: U2FsdGVkX1/xFl/IXPQFrYNJd6ZXhd506HvKZwQvrgjbNSQ1xvvypbqKXCSh6B3q1uIt53OTaYqyh9F8LRsh3gBRoPEGK/Mu3Tka9ozdY0yNX6XcNGuKaNuCFxKtaMdb8Q1B2HpoXR66XpNIfTmFPazJ1xAlQX3ZngNLx/5SKUyFjPSmwpVrI6K2z+7U/GTS04n6fm0L/G6/P8V33Bl+qU6kXBbqIIAjwq0qnSjnwXRxADBg8XtEi6dpJmzkV485T44hkViCvH4fWxcxtCj68HhB2RQvkpQd+IVwLHb9+7sbC/J4W+JT9DBHOamb+hXCz1QK2CSqR5k0O4VY09MKa8/Gt3jzBIGr1jdAC5IhJ2TFsD1Mfwi+yVvZRdRhq8b9xYB8ZRUuMwwjU8gpYUh7yx/87gMjfKa4zClu5t7T3uHCh865iFIFKbrOb0DAaqTp9LLAvMyUeeSFaDgZ0nenNbPgAUvne5YMxsMTBqk30Bk1pgvz7bLtczcTCj5wAonbawjKF1aIRP3NQcVQLeFdhxaJ9q0Bncpj+wUen4vXgW1rFEBIqs9cwphypQXD3RuQuzcU/rBc75nEZo7aDlSS3jlGqpE+NaGfhx6pFa9fsWS9JaaAEuAt9X3Q9HB9P1ENh8/fOQ3My8i4FLIFuBq3AcCfFZoPY6OX12m+s/puE6SJLxsskgwQwgGV5La29b1sbuyQ1EFbZ6auGMnbZz2IAjDsRnHzdcXLUcqcAubDiiA6UvZMSecPnI3wKkRonhHJVlPLarLorevYiOn8QRoyMc/7BCBch+fRblXTmGteU6g4NpLjum8X9/ChbgyA7bhpZEJmDvAO8SOhaGNSnB6fqmekyX4IXEWmlGQbuidxQ8vjNsZJhlLX6MJhGj4MrdoFl9BP4XEx3o9oDumOdp+tB1639S9hnGc6jV3PGH1u+aqjCLakPsUPK+t/1NOIVKFOawy1ddll/Y1267O3Zlh Tr1GVAuG +7SsWmwcZ+W3o/2XVAaENkQBvnT7dqM4X+FUwXRJ1BLurHf/KgrJr6Bw8pcQS3UhYEqFDicK3Swk63MGZNWI1/UjzfrAqmk1NRUxExX5gNM5ifuk4uLgumzr2DlBb8q1a9sgD6FrOvYURUVWGWOtRamFkJ5pgw7xhIUcgqeYT6fsPv5H2Xii51q0CZd58LEwFRicA3vo4C7N8rSVyErg5kQFORa0q5+JTEf3iaKwgkI37pP+aQ2wIEOnb6TT3YLQ6Wv0ZV2D0D3DptNJNbKep24lP/ad5GkpEfdJH7LNpnmwMG/DKH38AQ9RcHnTRADSEBy3KFisOmDapdBQEdPbXhSsa0zHds+4dhOw1F76UQtkGPOZERDUhoLb284t5tMSfIIKVyH6WD9+S+vx4zpojnB06eA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to be resilient against cross-process speculation-based attacks, it makes sense to store certain (secret) items in kernel memory local to mm. Implement such allocations on top of secretmem infrastructure. Specifically, on allocate 1. Create secretmem file. 2. To distinguish it from the conventional memfd_secret()-created one and to maintain associated mm-local allocation context, put the latter on ->private_data of the file. 3. Create virtual mapping in user virtual address space using mmap(). 4. Seal the virtual mapping to disallow the user from affecting it in any way. 5. Fault the pages in, effectively calling secretmem fault handler to remove the pages from kernel linear address and make them local to process mm. 6. Change the PTE from user mode to kernel mode, any access from userspace will result in segmentation fault. Kernel can access this virtual address now. 7. Return the secure area as a struct containing the pointer to the actual memory and providing the context for the release function later. On release, - if called while mm is still in use, remove the mapping - otherwise, if performed at mm teardown, no unmapping is necessary The rest is taken care of by secretmem file cleanup, including returning the pages to the kernel direct map. Signed-off-by: Fares Mehanna Signed-off-by: Roman Kagan --- include/linux/secretmem.h | 29 ++++++ mm/Kconfig | 10 ++ mm/gup.c | 4 +- mm/secretmem.c | 213 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 254 insertions(+), 2 deletions(-) diff --git a/include/linux/secretmem.h b/include/linux/secretmem.h index e918f96881f5..39cc73a0e4bd 100644 --- a/include/linux/secretmem.h +++ b/include/linux/secretmem.h @@ -2,6 +2,10 @@ #ifndef _LINUX_SECRETMEM_H #define _LINUX_SECRETMEM_H +struct secretmem_area { + void *ptr; +}; + #ifdef CONFIG_SECRETMEM extern const struct address_space_operations secretmem_aops; @@ -33,4 +37,29 @@ static inline bool secretmem_active(void) #endif /* CONFIG_SECRETMEM */ +#ifdef CONFIG_KERNEL_SECRETMEM + +bool can_access_secretmem_vma(struct vm_area_struct *vma); +struct secretmem_area *secretmem_allocate_pages(unsigned int order); +void secretmem_release_pages(struct secretmem_area *data); + +#else + +static inline bool can_access_secretmem_vma(struct vm_area_struct *vma) +{ + return true; +} + +static inline struct secretmem_area *secretmem_allocate_pages(unsigned int order) +{ + return NULL; +} + +static inline void secretmem_release_pages(struct secretmem_area *data) +{ + WARN_ONCE(1, "Called secret memory release page without support\n"); +} + +#endif /* CONFIG_KERNEL_SECRETMEM */ + #endif /* _LINUX_SECRETMEM_H */ diff --git a/mm/Kconfig b/mm/Kconfig index b72e7d040f78..a327d8def179 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1168,6 +1168,16 @@ config SECRETMEM memory areas visible only in the context of the owning process and not mapped to other processes and other kernel page tables. +config KERNEL_SECRETMEM + default y + bool "Enable kernel usage of memfd_secret()" if EXPERT + depends on SECRETMEM + depends on MMU + help + Enable the kernel usage of memfd_secret() for kernel memory allocations, + The allocated memory is visible only to the kernel in the context of + the owning process. + config ANON_VMA_NAME bool "Anonymous VMA name support" depends on PROC_FS && ADVISE_SYSCALLS && MMU diff --git a/mm/gup.c b/mm/gup.c index 54d0dc3831fb..6c2c6a0cbe2a 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1076,7 +1076,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, struct follow_page_context ctx = { NULL }; struct page *page; - if (vma_is_secretmem(vma)) + if (!can_access_secretmem_vma(vma)) return NULL; if (WARN_ON_ONCE(foll_flags & FOLL_PIN)) @@ -1281,7 +1281,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) if ((gup_flags & FOLL_LONGTERM) && vma_is_fsdax(vma)) return -EOPNOTSUPP; - if (vma_is_secretmem(vma)) + if (!can_access_secretmem_vma(vma)) return -EFAULT; if (write) { diff --git a/mm/secretmem.c b/mm/secretmem.c index 3afb5ad701e1..86afedc65889 100644 --- a/mm/secretmem.c +++ b/mm/secretmem.c @@ -13,13 +13,17 @@ #include #include #include +#include #include #include #include #include #include +#include +#include #include +#include #include @@ -42,6 +46,16 @@ MODULE_PARM_DESC(secretmem_enable, static atomic_t secretmem_users; +/* secretmem file private context */ +struct secretmem_ctx { + struct secretmem_area _area; + struct page **_pages; + unsigned long _nr_pages; + struct file *_file; + struct mm_struct *_mm; +}; + + bool secretmem_active(void) { return !!atomic_read(&secretmem_users); @@ -116,6 +130,7 @@ static const struct vm_operations_struct secretmem_vm_ops = { static int secretmem_release(struct inode *inode, struct file *file) { + kfree(file->private_data); atomic_dec(&secretmem_users); return 0; } @@ -123,13 +138,23 @@ static int secretmem_release(struct inode *inode, struct file *file) static int secretmem_mmap(struct file *file, struct vm_area_struct *vma) { unsigned long len = vma->vm_end - vma->vm_start; + struct secretmem_ctx *ctx = file->private_data; + unsigned long kernel_no_permissions; + + kernel_no_permissions = (VM_READ | VM_WRITE | VM_EXEC | VM_MAYEXEC); if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0) return -EINVAL; + if (ctx && (vma->vm_flags & kernel_no_permissions)) + return -EINVAL; + if (!mlock_future_ok(vma->vm_mm, vma->vm_flags | VM_LOCKED, len)) return -EAGAIN; + if (ctx) + vm_flags_set(vma, VM_MIXEDMAP); + vm_flags_set(vma, VM_LOCKED | VM_DONTDUMP); vma->vm_ops = &secretmem_vm_ops; @@ -230,6 +255,194 @@ static struct file *secretmem_file_create(unsigned long flags) return file; } +#ifdef CONFIG_KERNEL_SECRETMEM + +struct secretmem_area *secretmem_allocate_pages(unsigned int order) +{ + unsigned long uvaddr, uvaddr_inc, unused, nr_pages, bytes_length; + struct file *kernel_secfile; + struct vm_area_struct *vma; + struct secretmem_ctx *ctx; + struct page **sec_pages; + struct mm_struct *mm; + long nr_pinned_pages; + pte_t pte, old_pte; + spinlock_t *ptl; + pte_t *upte; + int rc; + + nr_pages = (1 << order); + bytes_length = nr_pages * PAGE_SIZE; + mm = current->mm; + + if (!mm || !mmget_not_zero(mm)) + return NULL; + + /* Create secret memory file / truncate it */ + kernel_secfile = secretmem_file_create(0); + if (IS_ERR(kernel_secfile)) + goto put_mm; + + ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); + if (IS_ERR(ctx)) + goto close_secfile; + kernel_secfile->private_data = ctx; + + rc = do_truncate(file_mnt_idmap(kernel_secfile), + file_dentry(kernel_secfile), bytes_length, 0, NULL); + if (rc) + goto close_secfile; + + if (mmap_write_lock_killable(mm)) + goto close_secfile; + + /* Map pages to the secretmem file */ + uvaddr = do_mmap(kernel_secfile, 0, bytes_length, PROT_NONE, + MAP_SHARED, 0, 0, &unused, NULL); + if (IS_ERR_VALUE(uvaddr)) + goto unlock_mmap; + + /* mseal() the VMA to make sure it won't change */ + rc = do_mseal(uvaddr, uvaddr + bytes_length, true); + if (rc) + goto unmap_pages; + + /* Make sure VMA is there, and is kernel-secure */ + vma = find_vma(current->mm, uvaddr); + if (!vma) + goto unseal_vma; + + if (!vma_is_secretmem(vma) || + !can_access_secretmem_vma(vma)) + goto unseal_vma; + + /* Pin user pages; fault them in */ + sec_pages = kzalloc(sizeof(struct page *) * nr_pages, GFP_KERNEL); + if (!sec_pages) + goto unseal_vma; + + nr_pinned_pages = pin_user_pages(uvaddr, nr_pages, FOLL_FORCE | FOLL_LONGTERM, sec_pages); + if (nr_pinned_pages < 0) + goto free_sec_pages; + if (nr_pinned_pages != nr_pages) + goto unpin_pages; + + /* Modify the existing mapping to be kernel accessible, local to this process mm */ + uvaddr_inc = uvaddr; + while (uvaddr_inc < uvaddr + bytes_length) { + upte = get_locked_pte(mm, uvaddr_inc, &ptl); + if (!upte) + goto unpin_pages; + old_pte = ptep_modify_prot_start(vma, uvaddr_inc, upte); + pte = pte_modify(old_pte, PAGE_KERNEL); + ptep_modify_prot_commit(vma, uvaddr_inc, upte, old_pte, pte); + pte_unmap_unlock(upte, ptl); + uvaddr_inc += PAGE_SIZE; + } + flush_tlb_range(vma, uvaddr, uvaddr + bytes_length); + + /* Return data */ + mmgrab(mm); + ctx->_area.ptr = (void *) uvaddr; + ctx->_pages = sec_pages; + ctx->_nr_pages = nr_pages; + ctx->_mm = mm; + ctx->_file = kernel_secfile; + + mmap_write_unlock(mm); + mmput(mm); + + return &ctx->_area; + +unpin_pages: + unpin_user_pages(sec_pages, nr_pinned_pages); +free_sec_pages: + kfree(sec_pages); +unseal_vma: + rc = do_mseal(uvaddr, uvaddr + bytes_length, false); + if (rc) + BUG(); +unmap_pages: + rc = do_munmap(mm, uvaddr, bytes_length, NULL); + if (rc) + BUG(); +unlock_mmap: + mmap_write_unlock(mm); +close_secfile: + fput(kernel_secfile); +put_mm: + mmput(mm); + return NULL; +} + +void secretmem_release_pages(struct secretmem_area *data) +{ + unsigned long uvaddr, bytes_length; + struct secretmem_ctx *ctx; + int rc; + + if (!data || !data->ptr) + BUG(); + + ctx = container_of(data, struct secretmem_ctx, _area); + if (!ctx || !ctx->_file || !ctx->_pages || !ctx->_mm) + BUG(); + + bytes_length = ctx->_nr_pages * PAGE_SIZE; + uvaddr = (unsigned long) data->ptr; + + /* + * Remove the mapping if mm is still in use. + * Not secure to continue if unmapping failed. + */ + if (mmget_not_zero(ctx->_mm)) { + mmap_write_lock(ctx->_mm); + rc = do_mseal(uvaddr, uvaddr + bytes_length, false); + if (rc) { + mmap_write_unlock(ctx->_mm); + BUG(); + } + rc = do_munmap(ctx->_mm, uvaddr, bytes_length, NULL); + if (rc) { + mmap_write_unlock(ctx->_mm); + BUG(); + } + mmap_write_unlock(ctx->_mm); + mmput(ctx->_mm); + } + + mmdrop(ctx->_mm); + unpin_user_pages(ctx->_pages, ctx->_nr_pages); + fput(ctx->_file); + kfree(ctx->_pages); + + ctx->_nr_pages = 0; + ctx->_pages = NULL; + ctx->_file = NULL; + ctx->_mm = NULL; + ctx->_area.ptr = NULL; +} + +bool can_access_secretmem_vma(struct vm_area_struct *vma) +{ + struct secretmem_ctx *ctx; + + if (!vma_is_secretmem(vma)) + return true; + + /* + * If VMA is owned by running process, and marked for kernel + * usage, then allow access. + */ + ctx = vma->vm_file->private_data; + if (ctx && current->mm == vma->vm_mm) + return true; + + return false; +} + +#endif /* CONFIG_KERNEL_SECRETMEM */ + SYSCALL_DEFINE1(memfd_secret, unsigned int, flags) { struct file *file; From patchwork Wed Sep 11 14:34:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fares Mehanna X-Patchwork-Id: 13800692 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76709EE49BC for ; Wed, 11 Sep 2024 14:36:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D090940049; Wed, 11 Sep 2024 10:36:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 080E4940021; Wed, 11 Sep 2024 10:36:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6273940049; Wed, 11 Sep 2024 10:36:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C7B6E940021 for ; Wed, 11 Sep 2024 10:36:41 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 84A14A9F30 for ; Wed, 11 Sep 2024 14:36:41 +0000 (UTC) X-FDA: 82552708602.12.0DB7FBF Received: from smtp-fw-9102.amazon.com (smtp-fw-9102.amazon.com [207.171.184.29]) by imf14.hostedemail.com (Postfix) with ESMTP id 6675A100004 for ; Wed, 11 Sep 2024 14:36:39 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=MVcorqqc; spf=pass (imf14.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 207.171.184.29 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de"; dmarc=pass (policy=quarantine) header.from=amazon.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726065296; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rCn+xTWUjnpsWnBBnpI3iR0NNWrWMHIE+UutE4x6c8o=; b=n/sw+IS+F1AokT2lWc1TvKAJGTwJ9+Ph0Q3IGEsP5iYnCCC90UQotK+VpeV0XO2eeKqG00 AK25NBktHYI7TxhHJylB9YJWXwDWWkFREHJ/6weegOXEgiOqrIBzFlHEDYpVXutwM/G+3B qoclE0iCkCWjhwm2BA3nnDcYMdE80l8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726065296; a=rsa-sha256; cv=none; b=P8Ne8FlFSe6sVwtZgd0lOERiGwyjcE8m5xRW8vWLfVub43SUahHftQih2lCRwP/aLDrfoP vQi98K7eh0SUYQDSuqkoqPE2APFn+KJdfuE69XqbWBYNJ4WfX+sucryOAofBWDyk6xpX+n pdfw9K8PHkhax5KLd07CLjKv2e3l1Ag= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=MVcorqqc; spf=pass (imf14.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 207.171.184.29 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de"; dmarc=pass (policy=quarantine) header.from=amazon.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1726065400; x=1757601400; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rCn+xTWUjnpsWnBBnpI3iR0NNWrWMHIE+UutE4x6c8o=; b=MVcorqqchM3HnXSUsp2s1aN7/33Eox2UNj53Bas+gQV9CXeXgdeJPHqK 5+0mRRkD2ypxv5uzhkEugjm7JaQ7XhPtr5AGcly1lszkXrOxYy7JP8tKG coiiwN4OHVDlfNYMEdOpxYLwqnX6iN2qhHqw+bywMHggtgF92ClsTJrZP c=; X-IronPort-AV: E=Sophos;i="6.10,220,1719878400"; d="scan'208";a="452953260" Received: from pdx4-co-svc-p1-lb2-vlan3.amazon.com (HELO smtpout.prod.us-east-1.prod.farcaster.email.amazon.dev) ([10.25.36.214]) by smtp-border-fw-9102.sea19.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2024 14:36:30 +0000 Received: from EX19MTAEUC001.ant.amazon.com [10.0.17.79:20048] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.13.80:2525] with esmtp (Farcaster) id 337e7469-1cf4-42a1-a92c-28b8d968fd53; Wed, 11 Sep 2024 14:36:28 +0000 (UTC) X-Farcaster-Flow-ID: 337e7469-1cf4-42a1-a92c-28b8d968fd53 Received: from EX19D007EUB001.ant.amazon.com (10.252.51.82) by EX19MTAEUC001.ant.amazon.com (10.252.51.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Wed, 11 Sep 2024 14:36:22 +0000 Received: from EX19MTAUEC001.ant.amazon.com (10.252.135.222) by EX19D007EUB001.ant.amazon.com (10.252.51.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Wed, 11 Sep 2024 14:36:21 +0000 Received: from dev-dsk-faresx-1b-27755bf1.eu-west-1.amazon.com (10.253.79.181) by mail-relay.amazon.com (10.252.135.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Wed, 11 Sep 2024 14:36:19 +0000 From: Fares Mehanna To: CC: , Fares Mehanna , "Marc Zyngier" , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , "Will Deacon" , Andrew Morton , "Kemeng Shi" , =?utf-8?q?Pierre-Cl=C3=A9ment_Tos?= =?utf-8?q?i?= , Ard Biesheuvel , Mark Rutland , Javier Martinez Canillas , "Arnd Bergmann" , Fuad Tabba , Mark Brown , Joey Gouly , Kristina Martsenko , Randy Dunlap , "Bjorn Helgaas" , Jean-Philippe Brucker , "Mike Rapoport (IBM)" , "David Hildenbrand" , Roman Kagan , "moderated list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , "open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , open list , "open list:MEMORY MANAGEMENT" Subject: [RFC PATCH 3/7] arm64: KVM: Refactor C-code to access vCPU gp-registers through macros Date: Wed, 11 Sep 2024 14:34:02 +0000 Message-ID: <20240911143421.85612-4-faresx@amazon.de> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240911143421.85612-1-faresx@amazon.de> References: <20240911143421.85612-1-faresx@amazon.de> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6675A100004 X-Stat-Signature: rr3qeo3nmnw9gtbgkr5sqpgihgumy7sz X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1726065399-34643 X-HE-Meta: U2FsdGVkX1/0056xJA7wA8/nx4kUnpGsBIKwyhgZlTQYWllqg/dFc5ShoSXtzgQ5r2d9Mf8Un4diIk/tN+T2WrIUz/qk2ZlmbqTD8CONP9UYIcs111Z3n2g4AoMuRfVVJncQv/wPNgywnNK1E3tgzv7LyqlC3scgfDbNF50+O5eLObpwdP5Vwo4Jn6/4SFN6LY1JDjrBmPfk/QLrGftCg1alXZwiuDQFwUfzzY/cPiZZuo3S1Q2/6N3Kk+alTi+lXVjGk70JLR3fW1vSETYp3WLuROi1lvUjRYiw465+tsbVm63fIHCZxVA8M9Gf1I1LUhTzNv8sAvj2rgAKGG5/njBQ2shDTUyPcBFO+/sbnNlo9ydaTQDGXnKFicIo9R71gb3Rhj5g6V38HFLcH9pL6UOaB9cWx32nWAdzCm9M6wvrQzMgCP46BpaSPFDjbOHQwWobkyk4UwE+A8TBYEDKsgV/axg4a98XjcDN+0UfycTObJxjNh6Y456OHKT+1ZhJ2JWxkdRUbEXT47i7XObGzd7/VLbRZC8FMGElh3kWfMpS3EMprdrIWjDkdzx3uAMBeU1Llq1xfxWVDS7rVXst7Vpp42BPAB7VPvhPVUDP7gP9eE65wnlM/rm8Dcq6a75uqkzcs5Y+gJvrnymU2pd7HH9XaP8wM/ZlfuN0zctA6bOI31PRKuNMecce6NJ3YZ9BvULWVnKG6PEO/FhVAjPrhuhDJl7/JBheZa29saC94uM9vk3Zo9/QylImgY7w8rGUfSVq1IAE1oV2I4WP7MndYobpDUnjMUYVaD90SQd738zSvBcFwWsg9uiAcYn57EwxfPVn7mxhJNVa6FARGS2VMJb5UYx8sArYQIqbLCWmcy5YyKuOPsQagzqx/aZh3ICOo0kzSiztlnzzp8IUooAe9qOe4PI7sw+76ib1wYbNxrbqzCl3XPXAsb68dYOhaFRcQC8PFVX3pd6gX/WG0pB wBL+cjoz 328A/kJ2e3GPi1vGMiCRBOIcGCTlcGcewuH3ysQmzf93VzDISI+O97yfMG0uPazUDIzr0dyhGBMTR4Z7ByBv5ldHTUysq6iBmf+ziodIkmJ2RK5MSmg61qeftXqPN1H5oQifrQwo0k5PFkCs3AhRm+ce8hmCXLeqTpXDnEPBhVqf9Mb35jq17+ig710w7E4omNLhFtF5uMZV7oB5fnuuVfWG0ZOUeoBaCzO8pVMnKqkJUJYWpYMnUG8u5hkdI5szJEzF06AEkT8ItmUZ47E4Su0LyHWuwlXRuR5YzlE6zf5yaQYFxZOjQoPGYkyOKjtKw4+hzpQhhosyTj+NzKtdQnFYRpuKeFVG7gVPEXD8YrN7K1SL5WS4gwT5B9h0GBNF3+/CL0++m7e0VNjoQyaU5NcjxBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Unify how KVM accesses vCPU gp-regs by using two macros vcpu_gp_regs() and ctxt_gp_regs(). This is prerequisite to move the gp-regs later to be dynamically allocated for vCPUs. Signed-off-by: Fares Mehanna --- arch/arm64/include/asm/kvm_emulate.h | 2 +- arch/arm64/include/asm/kvm_host.h | 3 ++- arch/arm64/kvm/guest.c | 8 ++++---- arch/arm64/kvm/hyp/include/hyp/switch.h | 2 +- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 10 +++++----- arch/arm64/kvm/hyp/include/nvhe/trap_handler.h | 2 +- 6 files changed, 14 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index a601a9305b10..cabfb76ca514 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -170,7 +170,7 @@ static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, static inline bool vcpu_is_el2_ctxt(const struct kvm_cpu_context *ctxt) { - switch (ctxt->regs.pstate & (PSR_MODE32_BIT | PSR_MODE_MASK)) { + switch (ctxt_gp_regs(ctxt)->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK)) { case PSR_MODE_EL2h: case PSR_MODE_EL2t: return true; diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index a33f5996ca9f..31cbd62a5d06 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -946,7 +946,8 @@ struct kvm_vcpu_arch { #define vcpu_clear_on_unsupported_cpu(vcpu) \ vcpu_clear_flag(vcpu, ON_UNSUPPORTED_CPU) -#define vcpu_gp_regs(v) (&(v)->arch.ctxt.regs) +#define ctxt_gp_regs(ctxt) (&(ctxt)->regs) +#define vcpu_gp_regs(v) (ctxt_gp_regs(&(v)->arch.ctxt)) /* * Only use __vcpu_sys_reg/ctxt_sys_reg if you know you want the diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 11098eb7eb44..821a2b7de388 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -134,16 +134,16 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) KVM_REG_ARM_CORE_REG(regs.regs[30]): off -= KVM_REG_ARM_CORE_REG(regs.regs[0]); off /= 2; - return &vcpu->arch.ctxt.regs.regs[off]; + return &vcpu_gp_regs(vcpu)->regs[off]; case KVM_REG_ARM_CORE_REG(regs.sp): - return &vcpu->arch.ctxt.regs.sp; + return &vcpu_gp_regs(vcpu)->sp; case KVM_REG_ARM_CORE_REG(regs.pc): - return &vcpu->arch.ctxt.regs.pc; + return &vcpu_gp_regs(vcpu)->pc; case KVM_REG_ARM_CORE_REG(regs.pstate): - return &vcpu->arch.ctxt.regs.pstate; + return &vcpu_gp_regs(vcpu)->pstate; case KVM_REG_ARM_CORE_REG(sp_el1): return __ctxt_sys_reg(&vcpu->arch.ctxt, SP_EL1); diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 37ff87d782b6..d2ed0938fc90 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -649,7 +649,7 @@ static inline void synchronize_vcpu_pstate(struct kvm_vcpu *vcpu, u64 *exit_code ESR_ELx_EC(read_sysreg_el2(SYS_ESR)) == ESR_ELx_EC_PAC) write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR); - vcpu->arch.ctxt.regs.pstate = read_sysreg_el2(SYS_SPSR); + vcpu_gp_regs(vcpu)->pstate = read_sysreg_el2(SYS_SPSR); } /* diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index 4c0fdabaf8ae..d17033766010 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -105,13 +105,13 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) static inline void __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt) { - ctxt->regs.pc = read_sysreg_el2(SYS_ELR); + ctxt_gp_regs(ctxt)->pc = read_sysreg_el2(SYS_ELR); /* * Guest PSTATE gets saved at guest fixup time in all * cases. We still need to handle the nVHE host side here. */ if (!has_vhe() && ctxt->__hyp_running_vcpu) - ctxt->regs.pstate = read_sysreg_el2(SYS_SPSR); + ctxt_gp_regs(ctxt)->pstate = read_sysreg_el2(SYS_SPSR); if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN)) ctxt_sys_reg(ctxt, DISR_EL1) = read_sysreg_s(SYS_VDISR_EL2); @@ -202,7 +202,7 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) /* Read the VCPU state's PSTATE, but translate (v)EL2 to EL1. */ static inline u64 to_hw_pstate(const struct kvm_cpu_context *ctxt) { - u64 mode = ctxt->regs.pstate & (PSR_MODE_MASK | PSR_MODE32_BIT); + u64 mode = ctxt_gp_regs(ctxt)->pstate & (PSR_MODE_MASK | PSR_MODE32_BIT); switch (mode) { case PSR_MODE_EL2t: @@ -213,7 +213,7 @@ static inline u64 to_hw_pstate(const struct kvm_cpu_context *ctxt) break; } - return (ctxt->regs.pstate & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; + return (ctxt_gp_regs(ctxt)->pstate & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; } static inline void __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt) @@ -235,7 +235,7 @@ static inline void __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctx if (!(mode & PSR_MODE32_BIT) && mode >= PSR_MODE_EL2t) pstate = PSR_MODE_EL2h | PSR_IL_BIT; - write_sysreg_el2(ctxt->regs.pc, SYS_ELR); + write_sysreg_el2(ctxt_gp_regs(ctxt)->pc, SYS_ELR); write_sysreg_el2(pstate, SYS_SPSR); if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN)) diff --git a/arch/arm64/kvm/hyp/include/nvhe/trap_handler.h b/arch/arm64/kvm/hyp/include/nvhe/trap_handler.h index 45a84f0ade04..dfe5be0d70ef 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/trap_handler.h +++ b/arch/arm64/kvm/hyp/include/nvhe/trap_handler.h @@ -11,7 +11,7 @@ #include -#define cpu_reg(ctxt, r) (ctxt)->regs.regs[r] +#define cpu_reg(ctxt, r) (ctxt_gp_regs((ctxt))->regs[r]) #define DECLARE_REG(type, name, ctxt, reg) \ type name = (type)cpu_reg(ctxt, (reg)) From patchwork Wed Sep 11 14:34:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fares Mehanna X-Patchwork-Id: 13800693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF3CBEE49BA for ; Wed, 11 Sep 2024 14:36:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42D1F94004A; Wed, 11 Sep 2024 10:36:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DB31940021; Wed, 11 Sep 2024 10:36:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27C6294004A; Wed, 11 Sep 2024 10:36:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0A13E940021 for ; Wed, 11 Sep 2024 10:36:58 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B751C81370 for ; Wed, 11 Sep 2024 14:36:57 +0000 (UTC) X-FDA: 82552709274.21.C2AD2A8 Received: from smtp-fw-52003.amazon.com (smtp-fw-52003.amazon.com [52.119.213.152]) by imf13.hostedemail.com (Postfix) with ESMTP id CE05520005 for ; Wed, 11 Sep 2024 14:36:55 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=l1LNfeOx; spf=pass (imf13.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 52.119.213.152 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de"; dmarc=pass (policy=quarantine) header.from=amazon.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726065311; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gUT0NHH1rkNY0jIUGubcfNpyrpnEMMZzd9a2sKPpzdA=; b=E2aICx0IhZq5b2reyDy4RNREUMTGf5oHVvLS2dY86supMl8U22+eC9QOBZfSMSveTWLEdY H57lJDzyo2WTvnQQTzcXisQCfy3zg3ckjKDNL+ASYg1YLsE9pTJYdlSwlLmZlKpdhP0isF CFCpM9uwg5B7stcUO0U094s2mjgXEpI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726065311; a=rsa-sha256; cv=none; b=xFDDu7VIEXTn/cvH7qgdDY6m+3XYBHdt3yxCY8qduS0rVxw5+BPs8NKHzCZXchN56s+0JM QkA933ZRp+w9TsVZ9QQq4Eyb6yz7MtqyFGRt3tY+tmvkQ4VJlogHGYzFbLiq+nJk+34ZhM EeVtn4DEEIF8zhbE1iO2YU4S39kOE6g= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=l1LNfeOx; spf=pass (imf13.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 52.119.213.152 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de"; dmarc=pass (policy=quarantine) header.from=amazon.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1726065416; x=1757601416; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gUT0NHH1rkNY0jIUGubcfNpyrpnEMMZzd9a2sKPpzdA=; b=l1LNfeOxKv1zSTZ050NCcTkkz0oFWBQhnQHI8qs0OF5Gr47WiUb0n95l 0gYALIm7v2NL36MK+ZEIHT05vpZe7T51/xndOEcKJr0BWBTX3A3bk7xMT ENXjS+T7zUi83kuRfQ4hkonBaiQP8xp9T2+SBTXKukMuIr+6CgpNLqnCT E=; X-IronPort-AV: E=Sophos;i="6.10,220,1719878400"; d="scan'208";a="24916368" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO smtpout.prod.us-east-1.prod.farcaster.email.amazon.dev) ([10.43.8.6]) by smtp-border-fw-52003.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2024 14:36:54 +0000 Received: from EX19MTAEUA001.ant.amazon.com [10.0.10.100:10017] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.27.59:2525] with esmtp (Farcaster) id f9f29f1f-caf9-4ac7-bc5d-27f9e417256f; Wed, 11 Sep 2024 14:36:52 +0000 (UTC) X-Farcaster-Flow-ID: f9f29f1f-caf9-4ac7-bc5d-27f9e417256f Received: from EX19D007EUA001.ant.amazon.com (10.252.50.133) by EX19MTAEUA001.ant.amazon.com (10.252.50.223) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Wed, 11 Sep 2024 14:36:52 +0000 Received: from EX19MTAUEC001.ant.amazon.com (10.252.135.222) by EX19D007EUA001.ant.amazon.com (10.252.50.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Wed, 11 Sep 2024 14:36:51 +0000 Received: from dev-dsk-faresx-1b-27755bf1.eu-west-1.amazon.com (10.253.79.181) by mail-relay.amazon.com (10.252.135.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Wed, 11 Sep 2024 14:36:49 +0000 From: Fares Mehanna To: CC: , Fares Mehanna , "Marc Zyngier" , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , "Will Deacon" , Andrew Morton , "Kemeng Shi" , =?utf-8?q?Pierre-Cl=C3=A9ment_Tos?= =?utf-8?q?i?= , Ard Biesheuvel , Mark Rutland , Javier Martinez Canillas , "Arnd Bergmann" , Fuad Tabba , Mark Brown , Joey Gouly , Kristina Martsenko , Randy Dunlap , "Bjorn Helgaas" , Jean-Philippe Brucker , "Mike Rapoport (IBM)" , "David Hildenbrand" , Roman Kagan , "moderated list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , "open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , open list , "open list:MEMORY MANAGEMENT" Subject: [RFC PATCH 4/7] KVM: Refactor Assembly-code to access vCPU gp-registers through a macro Date: Wed, 11 Sep 2024 14:34:03 +0000 Message-ID: <20240911143421.85612-5-faresx@amazon.de> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240911143421.85612-1-faresx@amazon.de> References: <20240911143421.85612-1-faresx@amazon.de> MIME-Version: 1.0 X-Stat-Signature: c6kwh661x53umfyys1znqiyjzk156ycu X-Rspamd-Queue-Id: CE05520005 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1726065415-760725 X-HE-Meta: U2FsdGVkX1+E5nwvVmRX9wRDjQLBUN1CHzVRjhCuZ5cGwmJkvqYJ25fXbz+igMH0KqvE7RxNMYuwXqOkESf/4FPB2w5elv4aicEfzOWow5rCFd0HcZn/p2U9hlh2hizR8b/GUg1rRCsVHegymMGiHBcObLxr2riAeC6uQY5o1zfwMZ3wLwmC+oKHPiW4PVjQJnmwc4TqS9TOLhVoGu6M7Qnxh1Co8bn/5oV0DRJPZ2sMfRjVmvi+2FG7t+Vpu1Xbw+WOWUy5F1NCu9aN7FmM12vB8I3+Hby+NE1+aXQk1qGMgAz1ulg59K0TH4rEXYSLhrNhF7soqoGHKGx5vB/zMh96iqj53nKakxOyN5ZX+w66r4dLLcxkc3oORk+NvK8OfQLPMQrfs/elCJS3nCD9Z3A17velkcjt/xzmcC8ABKCRiIfgWGyecJ+0B8Bgqb+IOJMIFjE5hleEEGhxGohdwUIGTSyzaHuW5YqhEPwXON1OoSCotsybpcbk4A6m1fUqgRmfX0iv48oqid4Psd+0pY5DlNDhlGtBXoHYPYEo2IlC8fi8yqmAbMZSCcGOJkhflhgWinUNtVN07GfZCegJqHwWlFJmnWNO6TaxeDjMBJ6WcwEPk8M3lnaZ3ar8bd1AeUiOehxKLDVGQVJs8SGWfYTYql6tqZ2OlUetWgKwwTHLanhVbRmklR5VHlmi1uXTuLJsnxJjH4iLrgEk9lJL5LBVvQrjGNyeUGS9M5v+KX02cJNcmpBewjJBtsYY7WQWm54+CgQdBKCEoGkk3Diy5eYz89ob+8VevkwWF0Da2BDlkDBKLW0klXugl8szrQpccQ6cAo99fSRPoKQdeG41EVMnkiXVPfWLcw+CMe+Z0iMi0cDFY97lnyv0eSvIm+V6kciznY+OiNkGSWdHCOQcZKFIgAOQhA5hW6tgUlCP8cT3J+/8dkLGHHcwDg7bN9tQmdFdh0dxh82v5HBl1ZL CxrBZR7O Si97pMTGRUOib6nAtic1CMWI87okCGYb/tzYHeFxXLDemGWGsQMLzeCiZg4PKZsFBBSxkNzaEPINGYuYwwR4u6BjnRyCwBIjvOhVNSqYLZnJ0tsOfmTJKSXKwh1v9En0uLi8VzveJBDV/Nm6XlPpA4Rtoa+yXVu8nikEG0I3ZblJaY+uuOotyIB7ZjabAzpJu4deD111K1Bm1PQLfFte/hjMhiOmekO9VXis1WcEdQQNQGAG/NLZ5MFOpuJEsYCwA6b9Os0xNGdZqIbIRHHIzc24HLj9dncjNhPdrCORJoZvz/fO2RTzuAgnjjgFzycZGM7GEb8sZuuJYMuLQSkMWne8beXDD3ms8Z7BlAyWCynnrJQSZ06xmbrC531YbR2jR8ZNbhBJ0jjE+z8Eek8l0WuEKuw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Right now assembly code accesses vCPU gp-regs directly from the context struct "struct kvm_cpu_context" using "CPU_XREG_OFFSET()". Since we want to move gp-regs to dynamic memory, we can no longer assume that gp-regs will be embedded in the context struct, thus split the access to two steps. The first is to get the gp-regs from the context using the assembly macro "get_ctxt_gp_regs". And the second is to access the gp-registers directly from within the "struct user_pt_regs" by removing the offset "CPU_USER_PT_REGS" from the access macro "CPU_XREG_OFFSET()". I also changed variable naming and comments where appropriate. Signed-off-by: Fares Mehanna --- arch/arm64/include/asm/kvm_asm.h | 48 +++++++++++++++++--------------- arch/arm64/kvm/hyp/entry.S | 15 ++++++++++ arch/arm64/kvm/hyp/nvhe/host.S | 20 ++++++++++--- 3 files changed, 57 insertions(+), 26 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 2181a11b9d92..fa4fb642a5f5 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -313,6 +313,10 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt, str \vcpu, [\ctxt, #HOST_CONTEXT_VCPU] .endm +.macro get_ctxt_gp_regs ctxt, regs + add \regs, \ctxt, #CPU_USER_PT_REGS +.endm + /* * KVM extable for unexpected exceptions. * Create a struct kvm_exception_table_entry output to a section that can be @@ -329,7 +333,7 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt, .popsection .endm -#define CPU_XREG_OFFSET(x) (CPU_USER_PT_REGS + 8*x) +#define CPU_XREG_OFFSET(x) (8 * (x)) #define CPU_LR_OFFSET CPU_XREG_OFFSET(30) #define CPU_SP_EL0_OFFSET (CPU_LR_OFFSET + 8) @@ -337,34 +341,34 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt, * We treat x18 as callee-saved as the host may use it as a platform * register (e.g. for shadow call stack). */ -.macro save_callee_saved_regs ctxt - str x18, [\ctxt, #CPU_XREG_OFFSET(18)] - stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] - stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] - stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)] - stp x25, x26, [\ctxt, #CPU_XREG_OFFSET(25)] - stp x27, x28, [\ctxt, #CPU_XREG_OFFSET(27)] - stp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)] +.macro save_callee_saved_regs regs + str x18, [\regs, #CPU_XREG_OFFSET(18)] + stp x19, x20, [\regs, #CPU_XREG_OFFSET(19)] + stp x21, x22, [\regs, #CPU_XREG_OFFSET(21)] + stp x23, x24, [\regs, #CPU_XREG_OFFSET(23)] + stp x25, x26, [\regs, #CPU_XREG_OFFSET(25)] + stp x27, x28, [\regs, #CPU_XREG_OFFSET(27)] + stp x29, lr, [\regs, #CPU_XREG_OFFSET(29)] .endm -.macro restore_callee_saved_regs ctxt - // We require \ctxt is not x18-x28 - ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)] - ldp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] - ldp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] - ldp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)] - ldp x25, x26, [\ctxt, #CPU_XREG_OFFSET(25)] - ldp x27, x28, [\ctxt, #CPU_XREG_OFFSET(27)] - ldp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)] +.macro restore_callee_saved_regs regs + // We require \regs is not x18-x28 + ldr x18, [\regs, #CPU_XREG_OFFSET(18)] + ldp x19, x20, [\regs, #CPU_XREG_OFFSET(19)] + ldp x21, x22, [\regs, #CPU_XREG_OFFSET(21)] + ldp x23, x24, [\regs, #CPU_XREG_OFFSET(23)] + ldp x25, x26, [\regs, #CPU_XREG_OFFSET(25)] + ldp x27, x28, [\regs, #CPU_XREG_OFFSET(27)] + ldp x29, lr, [\regs, #CPU_XREG_OFFSET(29)] .endm -.macro save_sp_el0 ctxt, tmp +.macro save_sp_el0 regs, tmp mrs \tmp, sp_el0 - str \tmp, [\ctxt, #CPU_SP_EL0_OFFSET] + str \tmp, [\regs, #CPU_SP_EL0_OFFSET] .endm -.macro restore_sp_el0 ctxt, tmp - ldr \tmp, [\ctxt, #CPU_SP_EL0_OFFSET] +.macro restore_sp_el0 regs, tmp + ldr \tmp, [\regs, #CPU_SP_EL0_OFFSET] msr sp_el0, \tmp .endm diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index 4433a234aa9b..628a123bcdc1 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -28,6 +28,9 @@ SYM_FUNC_START(__guest_enter) adr_this_cpu x1, kvm_hyp_ctxt, x2 + // Get gp-regs pointer from the context + get_ctxt_gp_regs x1, x1 + // Store the hyp regs save_callee_saved_regs x1 @@ -62,6 +65,9 @@ alternative_else_nop_endif // when this feature is enabled for kernel code. ptrauth_switch_to_guest x29, x0, x1, x2 + // Get gp-regs pointer from the context + get_ctxt_gp_regs x29, x29 + // Restore the guest's sp_el0 restore_sp_el0 x29, x0 @@ -108,6 +114,7 @@ SYM_INNER_LABEL(__guest_exit_panic, SYM_L_GLOBAL) // current state is saved to the guest context but it will only be // accurate if the guest had been completely restored. adr_this_cpu x0, kvm_hyp_ctxt, x1 + get_ctxt_gp_regs x0, x0 adr_l x1, hyp_panic str x1, [x0, #CPU_XREG_OFFSET(30)] @@ -120,6 +127,7 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL) // vcpu x0-x1 on the stack add x1, x1, #VCPU_CONTEXT + get_ctxt_gp_regs x1, x1 ALTERNATIVE(nop, SET_PSTATE_PAN(1), ARM64_HAS_PAN, CONFIG_ARM64_PAN) @@ -145,6 +153,10 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL) // Store the guest's sp_el0 save_sp_el0 x1, x2 + // Recover vCPU context to x1 + get_vcpu_ptr x1, x2 + add x1, x1, #VCPU_CONTEXT + adr_this_cpu x2, kvm_hyp_ctxt, x3 // Macro ptrauth_switch_to_hyp format: @@ -157,6 +169,9 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL) // mte_switch_to_hyp(g_ctxt, h_ctxt, reg1) mte_switch_to_hyp x1, x2, x3 + // Get gp-regs pointer from the context + get_ctxt_gp_regs x2, x2 + // Restore hyp's sp_el0 restore_sp_el0 x2, x3 diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S index 3d610fc51f4d..31afa7396294 100644 --- a/arch/arm64/kvm/hyp/nvhe/host.S +++ b/arch/arm64/kvm/hyp/nvhe/host.S @@ -17,6 +17,12 @@ SYM_FUNC_START(__host_exit) get_host_ctxt x0, x1 + /* Keep host context in x1 */ + mov x1, x0 + + /* Get gp-regs pointer from the context */ + get_ctxt_gp_regs x0, x0 + /* Store the host regs x2 and x3 */ stp x2, x3, [x0, #CPU_XREG_OFFSET(2)] @@ -36,7 +42,10 @@ SYM_FUNC_START(__host_exit) /* Store the host regs x18-x29, lr */ save_callee_saved_regs x0 - /* Save the host context pointer in x29 across the function call */ + /* Save the host context pointer in x28 across the function call */ + mov x28, x1 + + /* Save the host gp-regs pointer in x29 across the function call */ mov x29, x0 #ifdef CONFIG_ARM64_PTR_AUTH_KERNEL @@ -46,7 +55,7 @@ alternative_else_nop_endif alternative_if ARM64_KVM_PROTECTED_MODE /* Save kernel ptrauth keys. */ - add x18, x29, #CPU_APIAKEYLO_EL1 + add x18, x28, #CPU_APIAKEYLO_EL1 ptrauth_save_state x18, x19, x20 /* Use hyp keys. */ @@ -58,6 +67,7 @@ alternative_else_nop_endif __skip_pauth_save: #endif /* CONFIG_ARM64_PTR_AUTH_KERNEL */ + mov x0, x28 bl handle_trap __host_enter_restore_full: @@ -68,7 +78,7 @@ b __skip_pauth_restore alternative_else_nop_endif alternative_if ARM64_KVM_PROTECTED_MODE - add x18, x29, #CPU_APIAKEYLO_EL1 + add x18, x28, #CPU_APIAKEYLO_EL1 ptrauth_restore_state x18, x19, x20 alternative_else_nop_endif __skip_pauth_restore: @@ -101,7 +111,8 @@ SYM_FUNC_END(__host_exit) * void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt); */ SYM_FUNC_START(__host_enter) - mov x29, x0 + mov x28, x0 + get_ctxt_gp_regs x0, x29 b __host_enter_restore_full SYM_FUNC_END(__host_enter) @@ -141,6 +152,7 @@ SYM_FUNC_START(__hyp_do_panic) /* Enter the host, conditionally restoring the host context. */ cbz x29, __host_enter_without_restoring + get_ctxt_gp_regs x29, x29 b __host_enter_for_panic SYM_FUNC_END(__hyp_do_panic) From patchwork Wed Sep 11 14:34:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fares Mehanna X-Patchwork-Id: 13800694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 981E1EE49BC for ; Wed, 11 Sep 2024 14:37:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CBEB94004B; Wed, 11 Sep 2024 10:37:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 27CD2940021; Wed, 11 Sep 2024 10:37:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11CE994004B; Wed, 11 Sep 2024 10:37:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E31F6940021 for ; Wed, 11 Sep 2024 10:37:21 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 92B291C4EEB for ; Wed, 11 Sep 2024 14:37:21 +0000 (UTC) X-FDA: 82552710282.12.AEF657A Received: from smtp-fw-52002.amazon.com (smtp-fw-52002.amazon.com [52.119.213.150]) by imf04.hostedemail.com (Postfix) with ESMTP id 4A94C40022 for ; Wed, 11 Sep 2024 14:37:19 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=JiBBg45Y; spf=pass (imf04.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 52.119.213.150 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de"; dmarc=pass (policy=quarantine) header.from=amazon.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726065411; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y7aHv3GtJyRu8tFW/46gCzZ/hJvvqzg3ZHRgqRHdqdU=; b=sdYoZjqnB+f9ayfDWdQ+0L6Q9WpghwoKRTXRvdBmzfgqi8GDa53682/HFT7Su9ik8aViCb bUjnIDoiNs+RYeU8kstgFlaO4HIxTSu7IdfRkgFJ89AQS3sbEAfFS7vY+VwcTexVTgXrUk 1bb5ZZ6gFEguQyoGDVjl2R7sqRDLRuc= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=JiBBg45Y; spf=pass (imf04.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 52.119.213.150 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de"; dmarc=pass (policy=quarantine) header.from=amazon.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726065411; a=rsa-sha256; cv=none; b=0UHC8KcS9Su36G03wnLfXtpaag+/z1n5EV9oLgOE7ZsFIAvo4JN1YcfQ41frKLTBxH0Z/G pLd7amWutiYHx9M+eqMq/rJVY2M1bljiXTSotJ+svOYmHqXBPeLGvV1o5llskzXRGLGfY0 LtkMTZJSdfZ3+X0ByrWm0IQd5V95XMo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1726065439; x=1757601439; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=y7aHv3GtJyRu8tFW/46gCzZ/hJvvqzg3ZHRgqRHdqdU=; b=JiBBg45YxJWZTg0mQ6nJxo/d1DmwULd03zKblwytHsO+r4qYTEJdZ5MW Bc7lcjrBmnB1Gf7nUA1UVlFmaZiM0/gp4w+kWgG3+IVcfF0EQuHV0TitD DLJ1+519O3ZQQPwyAGgPd8TDr7Aoy2LESg6vJF5dmYwXN2qlg2SnVlN9C o=; X-IronPort-AV: E=Sophos;i="6.10,220,1719878400"; d="scan'208";a="658274111" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.43.8.6]) by smtp-border-fw-52002.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2024 14:37:14 +0000 Received: from EX19MTAEUC002.ant.amazon.com [10.0.43.254:16521] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.27.59:2525] with esmtp (Farcaster) id c7fcd23e-81e6-4810-ad32-43bfceb7c5f9; Wed, 11 Sep 2024 14:37:13 +0000 (UTC) X-Farcaster-Flow-ID: c7fcd23e-81e6-4810-ad32-43bfceb7c5f9 Received: from EX19D007EUA004.ant.amazon.com (10.252.50.76) by EX19MTAEUC002.ant.amazon.com (10.252.51.245) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Wed, 11 Sep 2024 14:37:12 +0000 Received: from EX19MTAUEC001.ant.amazon.com (10.252.135.222) by EX19D007EUA004.ant.amazon.com (10.252.50.76) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Wed, 11 Sep 2024 14:37:12 +0000 Received: from dev-dsk-faresx-1b-27755bf1.eu-west-1.amazon.com (10.253.79.181) by mail-relay.amazon.com (10.252.135.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Wed, 11 Sep 2024 14:37:09 +0000 From: Fares Mehanna To: CC: , Fares Mehanna , "Marc Zyngier" , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , "Will Deacon" , Andrew Morton , "Kemeng Shi" , =?utf-8?q?Pierre-Cl=C3=A9ment_Tos?= =?utf-8?q?i?= , Ard Biesheuvel , Mark Rutland , Javier Martinez Canillas , "Arnd Bergmann" , Fuad Tabba , Mark Brown , Joey Gouly , Kristina Martsenko , Randy Dunlap , "Bjorn Helgaas" , Jean-Philippe Brucker , "Mike Rapoport (IBM)" , "David Hildenbrand" , Roman Kagan , "moderated list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , "open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , open list , "open list:MEMORY MANAGEMENT" Subject: [RFC PATCH 5/7] arm64: KVM: Allocate vCPU gp-regs dynamically on VHE and KERNEL_SECRETMEM enabled systems Date: Wed, 11 Sep 2024 14:34:04 +0000 Message-ID: <20240911143421.85612-6-faresx@amazon.de> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240911143421.85612-1-faresx@amazon.de> References: <20240911143421.85612-1-faresx@amazon.de> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: tfgje9pguarff7zf3zqmdwpgmw49jyyg X-Rspamd-Queue-Id: 4A94C40022 X-Rspamd-Server: rspam11 X-HE-Tag: 1726065439-136944 X-HE-Meta: U2FsdGVkX18EKUh6Mpysm7As2S4YspnQkypj0A9J0NskGwR70nG5La296EzsL0V/0DvaWNwPNMw6lRRnkHX2EccIeYpmsgRORP4DcyMT7ZnswZ479RTSqylm0OQBE/xLPLgoWovn6UjZt+ZEGDVz5wS4jZD7o9p3YwWwieqcWrVzoI/pMeSDnYlGtf4yXOnGnd09N5B6FUHairp6GMln+GLIuuSu8uMjTFkkuZafZtEpFkXE4Ms+UwcuA0dcHksln8jshSAUDZg6cx/Zz15sgNLTQu0OKj0deYhMYGiTwIuelpQ4X1dY7TYI1wiTFfpe4GZI5FQqHozVpvW3ISimOIbkR40Y4AvX5zcrBGBJvkMLYGJTTQShc9PfC8cKzBHvtbVcoP++UYcQhOwQmTv0WVeHflMaFd3fheT2eEnQgE4DkNjE/mYDaPuKCWQxcEkZe5a7KzQ8Vn463S3/HKNJHJdU1Tn5h/R+MKTE2swrgJGBXBZ/BXjnXOS6bpX/c+NMOS8sK6I8DoV7E5oEN+vTV4yqo5aCQa49aTfUwY2zBhjDnGuL9dVI80kAdouMj3L4DEvsXpn0vZf+y3phn+Lxjc5ikc6JlY2Jx8jC5fbRJ7nX2qPZlRLBy/YWH2C7CSBvWKPFhnPaRkh0tzjm080yFVEULUrSgvG3XSQmXQG7tS/vxv7c3lwdC7YEJ4qTbZ2zVhOQ7Nt2x1V7wEtsX+xqGFBmnBeEg7wvvpE/3RPp1mhC0TpSVvlkBWSKQ9TMbCuLiv1xjrlSC2KA6xCHRyxeXhcvCUwvD+oPZh4rOo4fNVz0eGT8Bcn0nb0dZElGo0Pe2ZicxvthNScY/OEsg31lW9HoFC0dMfcYmPJJVb3+oX1oWZ/Z/63e7X/1Py9sbxhul8e7DqX72Baek4ftzKCCE0USrTburQ7jUeM1xiwNe3x7kiEWohHdLipey7HBXsuiMw12LTRm6YF4dZ1Jx0i TEdLZVGL cJxu02S9t4ho75ANFsiLi9PrwxWH8IaFU4+djdF/aPeiB+HVdpiEBfMC9uJPIjPvk4dEyvP88a0dmOfB7bWl8PlVHwvDbXfz2+WbpY8QdxS++VupMbQOkXzZNHy1QYQL468oNUceuwBHwIjkwwn9yfShXpL7r1WGfqUrqVbmpTH/VT1uR0YzTJhJiCXTVh13HYD6Lfr5Gb7jJlDm3sUG8mLr0gta6pfKP0C+suZS+2MQeLFIgTPw7JvvhiBl5AMOS9AVbnSQFKenCg7a/Za69vl1J9lczo0CNy84BuscE1xoCgqCJAO3hL7B5PwWqiAyCDjjiKlGdhVEspSOl8heLomHg4LExT1JbrnQEoxubstbbqwToJUj6ObstSD637NkD5cFHQ9uFnIxoLDVADplYFdCQXuEO2wyCaN3B4+og+bJRXj703nBtR/E/We5VV9+dBKzB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To allocate the vCPU gp-regs using secret memory, we need to dynamically allocate the vCPU gp-regs first. This is tricky with NVHE (Non-Virtualization Host Extensions) since it will require adjusting the virtual address on every access. With a large shared codebase between the OS and the hypervisor, it would be cumbersome to duplicate the code with one version using `kern_hyp_va()`. To avoid this issue, and since the secret memory feature will not be enabled on NVHE systems, we're introducing the following changes: 1. Maintain a `struct user_pt_regs regs_storage` in the vCPU context struct as a fallback storage for the vCPU gp-regs. 2. Introduce a pointer `struct user_pt_regs *regs` in the vCPU context struct to hold the dynamically allocated vCPU gp-regs. If we are on an NVHE system or a VHE (Virtualization Host Extensions) system that doesn't support `KERNEL_SECRETMEM`, we will use `ctxt_storage`. Accessing the context in this case will not require a de-reference operation. If we are on a VHE system with support for `KERNEL_SECRETMEM`, we will use the `regs` pointer. In this case, we will add one de-reference operation every time the vCPU gp-reg is accessed. Accessing the gp-regs embedded in the vCPU context without de-reference is done as: add \regs, \ctxt, #CPU_USER_PT_REGS_STRG Accessing the dynamically allocated gp-regs with de-reference is done as: ldr \regs, [\ctxt, #CPU_USER_PT_REGS] By default, we are using the first version. If we are booting on a system that supports VHE and `KERNEL_SECRETMEM`, we switch to the second version. We are also allocating the needed gp-regs allocations for vCPU, kvm_hyp_ctxt and kvm_host_data structs when needed. Signed-off-by: Fares Mehanna --- arch/arm64/include/asm/kvm_asm.h | 4 +- arch/arm64/include/asm/kvm_host.h | 24 +++++++++++- arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kvm/arm.c | 63 ++++++++++++++++++++++++++++++- arch/arm64/kvm/va_layout.c | 23 +++++++++++ 6 files changed, 112 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index fa4fb642a5f5..1d6de0806dbd 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -314,7 +314,9 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt, .endm .macro get_ctxt_gp_regs ctxt, regs - add \regs, \ctxt, #CPU_USER_PT_REGS +alternative_cb ARM64_HAS_VIRT_HOST_EXTN, kvm_update_ctxt_gp_regs + add \regs, \ctxt, #CPU_USER_PT_REGS_STRG +alternative_cb_end .endm /* diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 31cbd62a5d06..23a10178d1b0 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -541,7 +541,9 @@ struct kvm_sysreg_masks { }; struct kvm_cpu_context { - struct user_pt_regs regs; /* sp = sp_el0 */ + struct user_pt_regs *regs; /* sp = sp_el0 */ + struct user_pt_regs regs_storage; + struct secretmem_area *regs_area; u64 spsr_abt; u64 spsr_und; @@ -946,7 +948,25 @@ struct kvm_vcpu_arch { #define vcpu_clear_on_unsupported_cpu(vcpu) \ vcpu_clear_flag(vcpu, ON_UNSUPPORTED_CPU) -#define ctxt_gp_regs(ctxt) (&(ctxt)->regs) +/* Static allocation is used if NVHE-host or if KERNEL_SECRETMEM is not enabled */ +static __inline bool kvm_use_dynamic_regs(void) +{ +#ifndef CONFIG_KERNEL_SECRETMEM + return false; +#endif + return cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN); +} + +static __always_inline struct user_pt_regs *ctxt_gp_regs(const struct kvm_cpu_context *ctxt) +{ + struct user_pt_regs *regs = (void *) ctxt; + asm volatile(ALTERNATIVE_CB("add %0, %0, %1\n", + ARM64_HAS_VIRT_HOST_EXTN, + kvm_update_ctxt_gp_regs) + : "+r" (regs) + : "I" (offsetof(struct kvm_cpu_context, regs_storage))); + return regs; +} #define vcpu_gp_regs(v) (ctxt_gp_regs(&(v)->arch.ctxt)) /* diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 27de1dddb0ab..275d480f5e65 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -128,6 +128,7 @@ int main(void) DEFINE(VCPU_FAULT_DISR, offsetof(struct kvm_vcpu, arch.fault.disr_el1)); DEFINE(VCPU_HCR_EL2, offsetof(struct kvm_vcpu, arch.hcr_el2)); DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_cpu_context, regs)); + DEFINE(CPU_USER_PT_REGS_STRG, offsetof(struct kvm_cpu_context, regs_storage)); DEFINE(CPU_ELR_EL2, offsetof(struct kvm_cpu_context, sys_regs[ELR_EL2])); DEFINE(CPU_RGSR_EL1, offsetof(struct kvm_cpu_context, sys_regs[RGSR_EL1])); DEFINE(CPU_GCR_EL1, offsetof(struct kvm_cpu_context, sys_regs[GCR_EL1])); diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 8f5422ed1b75..e3bb626e299c 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -86,6 +86,7 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch); KVM_NVHE_ALIAS(kvm_update_va_mask); KVM_NVHE_ALIAS(kvm_get_kimage_voffset); KVM_NVHE_ALIAS(kvm_compute_final_ctr_el0); +KVM_NVHE_ALIAS(kvm_update_ctxt_gp_regs); KVM_NVHE_ALIAS(spectre_bhb_patch_loop_iter); KVM_NVHE_ALIAS(spectre_bhb_patch_loop_mitigation_enable); KVM_NVHE_ALIAS(spectre_bhb_patch_wa3); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 9bef7638342e..78c562a060de 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -452,6 +453,7 @@ int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id) int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) { + unsigned long pages_needed; int err; spin_lock_init(&vcpu->arch.mp_state_lock); @@ -469,6 +471,14 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO; + if (kvm_use_dynamic_regs()) { + pages_needed = (sizeof(*vcpu_gp_regs(vcpu)) + PAGE_SIZE - 1) / PAGE_SIZE; + vcpu->arch.ctxt.regs_area = secretmem_allocate_pages(fls(pages_needed - 1)); + if (!vcpu->arch.ctxt.regs_area) + return -ENOMEM; + vcpu->arch.ctxt.regs = vcpu->arch.ctxt.regs_area->ptr; + } + /* Set up the timer */ kvm_timer_vcpu_init(vcpu); @@ -489,9 +499,14 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) err = kvm_vgic_vcpu_init(vcpu); if (err) - return err; + goto free_vcpu_ctxt; return kvm_share_hyp(vcpu, vcpu + 1); + +free_vcpu_ctxt: + if (kvm_use_dynamic_regs()) + secretmem_release_pages(vcpu->arch.ctxt.regs_area); + return err; } void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu) @@ -508,6 +523,9 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) kvm_pmu_vcpu_destroy(vcpu); kvm_vgic_vcpu_destroy(vcpu); kvm_arm_vcpu_destroy(vcpu); + + if (kvm_use_dynamic_regs()) + secretmem_release_pages(vcpu->arch.ctxt.regs_area); } void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) @@ -2683,6 +2701,45 @@ static int __init init_hyp_mode(void) return err; } +static int init_hyp_hve_mode(void) +{ + int cpu; + int err = 0; + + if (!kvm_use_dynamic_regs()) + return 0; + + /* Allocate gp-regs */ + for_each_possible_cpu(cpu) { + void *hyp_ctxt_regs; + void *kvm_host_data_regs; + + hyp_ctxt_regs = kzalloc(sizeof(struct user_pt_regs), GFP_KERNEL); + if (!hyp_ctxt_regs) { + err = -ENOMEM; + goto free_regs; + } + per_cpu(kvm_hyp_ctxt, cpu).regs = hyp_ctxt_regs; + + kvm_host_data_regs = kzalloc(sizeof(struct user_pt_regs), GFP_KERNEL); + if (!kvm_host_data_regs) { + err = -ENOMEM; + goto free_regs; + } + per_cpu(kvm_host_data, cpu).host_ctxt.regs = kvm_host_data_regs; + } + + return 0; + +free_regs: + for_each_possible_cpu(cpu) { + kfree(per_cpu(kvm_hyp_ctxt, cpu).regs); + kfree(per_cpu(kvm_host_data, cpu).host_ctxt.regs); + } + + return err; +} + struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr) { struct kvm_vcpu *vcpu = NULL; @@ -2806,6 +2863,10 @@ static __init int kvm_arm_init(void) err = init_hyp_mode(); if (err) goto out_err; + } else { + err = init_hyp_hve_mode(); + if (err) + goto out_err; } err = kvm_init_vector_slots(); diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c index 91b22a014610..fcef7e89d042 100644 --- a/arch/arm64/kvm/va_layout.c +++ b/arch/arm64/kvm/va_layout.c @@ -185,6 +185,29 @@ void __init kvm_update_va_mask(struct alt_instr *alt, } } +void __init kvm_update_ctxt_gp_regs(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst) +{ + u32 rd, rn, imm, insn, oinsn; + + BUG_ON(nr_inst != 1); + + if (!kvm_use_dynamic_regs()) + return; + + oinsn = le32_to_cpu(origptr[0]); + rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn); + rn = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RN, oinsn); + imm = offsetof(struct kvm_cpu_context, regs); + + insn = aarch64_insn_gen_load_store_imm(rd, rn, imm, + AARCH64_INSN_SIZE_64, + AARCH64_INSN_LDST_LOAD_IMM_OFFSET); + BUG_ON(insn == AARCH64_BREAK_FAULT); + + updptr[0] = cpu_to_le32(insn); +} + void kvm_patch_vector_branch(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst) { From patchwork Wed Sep 11 14:34:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fares Mehanna X-Patchwork-Id: 13800695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5068DEE49BC for ; Wed, 11 Sep 2024 14:37:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D7A8594004C; Wed, 11 Sep 2024 10:37:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D2AB3940021; Wed, 11 Sep 2024 10:37:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCBA094004C; Wed, 11 Sep 2024 10:37:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9F0ED940021 for ; Wed, 11 Sep 2024 10:37:42 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5DE541202C3 for ; Wed, 11 Sep 2024 14:37:42 +0000 (UTC) X-FDA: 82552711164.10.CB68F34 Received: from smtp-fw-80007.amazon.com (smtp-fw-80007.amazon.com [99.78.197.218]) by imf24.hostedemail.com (Postfix) with ESMTP id 59CC3180004 for ; Wed, 11 Sep 2024 14:37:39 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=uaQOaXOE; dmarc=pass (policy=quarantine) header.from=amazon.de; spf=pass (imf24.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 99.78.197.218 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726065375; a=rsa-sha256; cv=none; b=r+1fltdW9G2+5mZJUzPVKSIU3QjEJjaZk584Cb9twEURs48Fktdh+gidhTYIZKRIlrhjcy 2oyGuPNsqEal7Y8W0o1BYKJcXmMqUPFLWO64JFWsAYV+U8F9YLfnhP8Hd7xs9X5am1zuQh nSG6Qom5LrQnjJHI7JPM029zGEL8iuU= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=uaQOaXOE; dmarc=pass (policy=quarantine) header.from=amazon.de; spf=pass (imf24.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 99.78.197.218 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726065375; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ttqjEP9NZOH/UBgVCbYLfsAknImtx66onLISyRd5NFM=; b=Zg9cDrtlqjLyObYwYYY/OExvG1emIjkSe0RY8kA1WBI812OPNOC9Iy+KwgkdEUURc/g4cC sAWY2O5OSfssmTo5My1/m55tJoam/mGAumreG620WHev/7RdIG8NZUeRe/rCd+OVr0koZd UuLvccXtroTfzxTDWtN5F4bPRqii4tk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1726065459; x=1757601459; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ttqjEP9NZOH/UBgVCbYLfsAknImtx66onLISyRd5NFM=; b=uaQOaXOENcpgQc/hhB8UQAG+FvHrUq4XkeDhTLi0Jut+XNyJt8x8dgYi rqdxQYQviTOzHvg0boMaQG/+NlOdYT1s60WO53oIvQIYCgyM7SP3X9W5m ZlGRCcKUyAlyaGQXwzsU++fwYrP26PjvRoT9vJszgljzfcuIyi2FRRdaf k=; X-IronPort-AV: E=Sophos;i="6.10,220,1719878400"; d="scan'208";a="329956727" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO smtpout.prod.us-east-1.prod.farcaster.email.amazon.dev) ([10.25.36.210]) by smtp-border-fw-80007.pdx80.corp.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2024 14:37:36 +0000 Received: from EX19MTAEUA001.ant.amazon.com [10.0.10.100:3852] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.38.136:2525] with esmtp (Farcaster) id ac0f2d97-7ed9-430d-a2d7-58bab541e8ef; Wed, 11 Sep 2024 14:37:35 +0000 (UTC) X-Farcaster-Flow-ID: ac0f2d97-7ed9-430d-a2d7-58bab541e8ef Received: from EX19D007EUB001.ant.amazon.com (10.252.51.82) by EX19MTAEUA001.ant.amazon.com (10.252.50.50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Wed, 11 Sep 2024 14:37:33 +0000 Received: from EX19MTAUEC001.ant.amazon.com (10.252.135.222) by EX19D007EUB001.ant.amazon.com (10.252.51.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Wed, 11 Sep 2024 14:37:33 +0000 Received: from dev-dsk-faresx-1b-27755bf1.eu-west-1.amazon.com (10.253.79.181) by mail-relay.amazon.com (10.252.135.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Wed, 11 Sep 2024 14:37:30 +0000 From: Fares Mehanna To: CC: , Fares Mehanna , "Marc Zyngier" , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , "Will Deacon" , Andrew Morton , "Kemeng Shi" , =?utf-8?q?Pierre-Cl=C3=A9ment_Tos?= =?utf-8?q?i?= , Ard Biesheuvel , Mark Rutland , Javier Martinez Canillas , "Arnd Bergmann" , Fuad Tabba , Mark Brown , Joey Gouly , Kristina Martsenko , Randy Dunlap , "Bjorn Helgaas" , Jean-Philippe Brucker , "Mike Rapoport (IBM)" , "David Hildenbrand" , Roman Kagan , "moderated list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , "open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , open list , "open list:MEMORY MANAGEMENT" Subject: [RFC PATCH 6/7] arm64: KVM: Refactor C-code to access vCPU fp-registers through macros Date: Wed, 11 Sep 2024 14:34:05 +0000 Message-ID: <20240911143421.85612-7-faresx@amazon.de> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240911143421.85612-1-faresx@amazon.de> References: <20240911143421.85612-1-faresx@amazon.de> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 59CC3180004 X-Stat-Signature: qnt7hpygk4ap8ugaae6jjobies1bikci X-Rspam-User: X-HE-Tag: 1726065459-817732 X-HE-Meta: U2FsdGVkX1+K3425TXkYN2zrWWlD5gDYgZrbupiCEwj2ZDrzo0BLY+0OCeIOprf52U2WAl3MzOCbyUf3nBkI7TdpAAVeuWAS+JCLL0ZHLK4739jH75Bdsaln7/f1Z4fIKaOR6EAm8XrTK2N15E7knLzkSuQRJSL/Xu2fIFTEwkaNZpc1SAyVZuWkfmICi7hsbvTN3Mxf1PMW/KNxQGAYDs/gkKi+B7pHsm00mdTnKzZBa/vmrKqqB5zV+vPGzDO7NSBYDjkvXXcLOVDDT9JPrdjyjTvdngRV1LwuYouJjPb1jaDkVZt604BDbtv2XEY7h9nm0MuxbPITpFZOukhr0kEY/DB5jKrKiHfb2IryJmS4lu9eHHHmxJDOVAPyJQWo3buPCgftFOLvYajieElfQzuTadjyHPGqysXkmaRajy7c26bt4HDJOSrPFTqZ3gOKyr2lPEtWrE3b8dLFQoYwLiT7G4asrtTYvvSYiwIU33LfZei9uWAgsHhsIyZING/MZCZHkgMOqvWBaws+6mYGQt0cn+VBjID2lspn7COxiwskGatjrsDuYl/IL7BrI8K/PQBh/yf6Ui9Jl2bd+ZUsi9IB6wFpVy4GpAiJ1iGUpNkQSfJc/DxQxqf2WsxY//thNEpi61YGb/js6QuaLfYn6TR3HJvO22A2W96O4zDYVlceeaOp12cg6TjfFyZkSZkuPRBIQ9JiUeij1G6akz0y4+h2hV8fdNfCzDT1QrPgW1Y5nPAbEipPU9jtrXpu8bx/uXkPv8vzMHISfAPxmQI1J/wsQkkylwJDLIeL1XvvZsJyAZQdk/X4DzDJ1ZnO3IBmdruE7I3HQaoyOq0A9gc4J+eG0eevK4ArWFjHmLpm0XObjKotPHm3WYH2yVOVqzGDfVlR2VU5AMH7Gt+RpxvlJZWGRx79Q7ASXOPNZOcngw2RIvH4ihPwC9E7SPf3T/mWn9UmTcr3uJzEuGnUn9S wuKo5SfE wVu4KcUlaefPbMukKup+mBDzrM4cIlkoFJqVD1LlYxzValI31mySz+xh9vXzSCJwaBCWxW0Ck2V+XznWHViUJFGYKo6vO+d0wiYDUgg9kwc0M78sTR3c5nkwIunPEqe7Uyuzjpr5MULSTp4kVFQFQdwn4sI7RqZWNCo3m6j4Lpk+eK0i/Zbn9t+c5KxcyiafGkmDGe8yOF9pJiOzzSS3JwMpj3/WU6I9cpVJkN/H9TurXX9qpcFMdS2cshMRr+b0awRk5QcGrhR2a4h+uJMKn07wcYLdKhGt3gMv9Jw0wM+M0ptNajKyLClQmyFNreWNYQ2IO8SpOiHqwLm3+TIxZtMnD81QYxPXlaGHJiKEIgFNTLamYiwjoxNFfqKdW5cKO7Hu1c47Li430yfM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Unify how KVM accesses vCPU fp-regs by using vcpu_fp_regs(). This is a prerequisite to move the fp-regs later to be dynamically allocated for vCPUs. Signed-off-by: Fares Mehanna --- arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/kvm/arm.c | 2 +- arch/arm64/kvm/fpsimd.c | 2 +- arch/arm64/kvm/guest.c | 6 +++--- arch/arm64/kvm/hyp/include/hyp/switch.h | 4 ++-- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 4 ++-- arch/arm64/kvm/reset.c | 2 +- 7 files changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 23a10178d1b0..e8ed2c12479f 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -968,6 +968,8 @@ static __always_inline struct user_pt_regs *ctxt_gp_regs(const struct kvm_cpu_co return regs; } #define vcpu_gp_regs(v) (ctxt_gp_regs(&(v)->arch.ctxt)) +#define ctxt_fp_regs(ctxt) (&(ctxt).fp_regs) +#define vcpu_fp_regs(v) (ctxt_fp_regs(&(v)->arch.ctxt)) /* * Only use __vcpu_sys_reg/ctxt_sys_reg if you know you want the diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 78c562a060de..7542af3f766a 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -2507,7 +2507,7 @@ static void finalize_init_hyp_mode(void) for_each_possible_cpu(cpu) { struct user_fpsimd_state *fpsimd_state; - fpsimd_state = &per_cpu_ptr_nvhe_sym(kvm_host_data, cpu)->host_ctxt.fp_regs; + fpsimd_state = ctxt_fp_regs(&per_cpu_ptr_nvhe_sym(kvm_host_data, cpu)->host_ctxt); per_cpu_ptr_nvhe_sym(kvm_host_data, cpu)->fpsimd_state = kern_hyp_va(fpsimd_state); } diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index c53e5b14038d..c27c96ae22e1 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -130,7 +130,7 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) * Currently we do not support SME guests so SVCR is * always 0 and we just need a variable to point to. */ - fp_state.st = &vcpu->arch.ctxt.fp_regs; + fp_state.st = vcpu_fp_regs(vcpu); fp_state.sve_state = vcpu->arch.sve_state; fp_state.sve_vl = vcpu->arch.sve_max_vl; fp_state.sme_state = NULL; diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 821a2b7de388..3474874a00a7 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -170,13 +170,13 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) KVM_REG_ARM_CORE_REG(fp_regs.vregs[31]): off -= KVM_REG_ARM_CORE_REG(fp_regs.vregs[0]); off /= 4; - return &vcpu->arch.ctxt.fp_regs.vregs[off]; + return &vcpu_fp_regs(vcpu)->vregs[off]; case KVM_REG_ARM_CORE_REG(fp_regs.fpsr): - return &vcpu->arch.ctxt.fp_regs.fpsr; + return &vcpu_fp_regs(vcpu)->fpsr; case KVM_REG_ARM_CORE_REG(fp_regs.fpcr): - return &vcpu->arch.ctxt.fp_regs.fpcr; + return &vcpu_fp_regs(vcpu)->fpcr; default: return NULL; diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index d2ed0938fc90..1444bad519db 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -319,7 +319,7 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu) */ sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, SYS_ZCR_EL2); __sve_restore_state(vcpu_sve_pffr(vcpu), - &vcpu->arch.ctxt.fp_regs.fpsr, + &vcpu_fp_regs(vcpu)->fpsr, true); /* @@ -401,7 +401,7 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) if (sve_guest) __hyp_sve_restore_guest(vcpu); else - __fpsimd_restore_state(&vcpu->arch.ctxt.fp_regs); + __fpsimd_restore_state(vcpu_fp_regs(vcpu)); /* Skip restoring fpexc32 for AArch64 guests */ if (!(read_sysreg(hcr_el2) & HCR_RW)) diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index f43d845f3c4e..feb1dd37f2a5 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -32,7 +32,7 @@ static void __hyp_sve_save_guest(struct kvm_vcpu *vcpu) * on the VL, so use a consistent (i.e., the maximum) guest VL. */ sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, SYS_ZCR_EL2); - __sve_save_state(vcpu_sve_pffr(vcpu), &vcpu->arch.ctxt.fp_regs.fpsr, true); + __sve_save_state(vcpu_sve_pffr(vcpu), &vcpu_fp_regs(vcpu)->fpsr, true); write_sysreg_s(ZCR_ELx_LEN_MASK, SYS_ZCR_EL2); } @@ -71,7 +71,7 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu) if (vcpu_has_sve(vcpu)) __hyp_sve_save_guest(vcpu); else - __fpsimd_save_state(&vcpu->arch.ctxt.fp_regs); + __fpsimd_save_state(vcpu_fp_regs(vcpu)); if (system_supports_sve()) __hyp_sve_restore_host(); diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 0b0ae5ae7bc2..5f38acf5d156 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -229,7 +229,7 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu) /* Reset core registers */ memset(vcpu_gp_regs(vcpu), 0, sizeof(*vcpu_gp_regs(vcpu))); - memset(&vcpu->arch.ctxt.fp_regs, 0, sizeof(vcpu->arch.ctxt.fp_regs)); + memset(vcpu_fp_regs(vcpu), 0, sizeof(*vcpu_fp_regs(vcpu))); vcpu->arch.ctxt.spsr_abt = 0; vcpu->arch.ctxt.spsr_und = 0; vcpu->arch.ctxt.spsr_irq = 0; From patchwork Wed Sep 11 14:34:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fares Mehanna X-Patchwork-Id: 13800696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B375FEE49BC for ; Wed, 11 Sep 2024 14:38:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4542794004E; Wed, 11 Sep 2024 10:38:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DD56940021; Wed, 11 Sep 2024 10:38:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22F4D94004E; Wed, 11 Sep 2024 10:38:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id ED5BE940021 for ; Wed, 11 Sep 2024 10:38:08 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9F9EB41542 for ; Wed, 11 Sep 2024 14:38:08 +0000 (UTC) X-FDA: 82552712256.24.8534837 Received: from smtp-fw-80007.amazon.com (smtp-fw-80007.amazon.com [99.78.197.218]) by imf06.hostedemail.com (Postfix) with ESMTP id 997F3180004 for ; Wed, 11 Sep 2024 14:38:06 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=VNeBsVfc; dmarc=pass (policy=quarantine) header.from=amazon.de; spf=pass (imf06.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 99.78.197.218 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726065482; a=rsa-sha256; cv=none; b=oenNOOB/A4Hi0TNStSPPMMqEx7vc+/tcY/SYKz7qR+OwmU4NIyTSOdJcFlvvc2NpTSjZbY U0RDFXksicZwCXUTBBcfW374yHlhgW8RhQ3AQpFnzufRyrzEhJdVhwaWHbEhjkZXYLAHpg HpDZ+GVeU3LDmERuFhRP9vUPdO+J5oQ= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=amazon.de header.s=amazon201209 header.b=VNeBsVfc; dmarc=pass (policy=quarantine) header.from=amazon.de; spf=pass (imf06.hostedemail.com: domain of "prvs=97728e23b=faresx@amazon.de" designates 99.78.197.218 as permitted sender) smtp.mailfrom="prvs=97728e23b=faresx@amazon.de" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726065482; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Hqkci3dj0GUqOn7J4NBabiUfpbZVZgBkN05sOca7hnY=; b=SZUJdBBK/rPbPkqt86g/+dz4+1zOnbs+w7dHdvcb8jT69kIBT3umnNH5NP8WMo/aabi77G rmHAl/kNNgGo8V8hT+jLwZTSi+eqC83UCWiCSJgcj9z8ILzlGgFFk1zPne/NoV6Uhy9sUr whU8FZYSJrNlxBg1JzPmnm47c+oZzJM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1726065487; x=1757601487; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Hqkci3dj0GUqOn7J4NBabiUfpbZVZgBkN05sOca7hnY=; b=VNeBsVfcLxdsajkuQf/MCCjTETZVKDFNUmJcN3Gm3d4jfBxA/iD7hgD3 hoiKr3QcHAw4Hr5+38wNsc7ALcYTJZP2UXFOvLv1AOQsDCCb/BR1gnpPR HnjZU9QjTlD/JEIL6y20sWL+3V6+J1Pg5gygt7/2IRCpttDSYLhYg77h1 g=; X-IronPort-AV: E=Sophos;i="6.10,220,1719878400"; d="scan'208";a="329956973" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.25.36.210]) by smtp-border-fw-80007.pdx80.corp.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2024 14:38:05 +0000 Received: from EX19MTAEUC001.ant.amazon.com [10.0.17.79:41644] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.25.181:2525] with esmtp (Farcaster) id 4f621809-6571-476e-b300-f14101ba8a55; Wed, 11 Sep 2024 14:38:03 +0000 (UTC) X-Farcaster-Flow-ID: 4f621809-6571-476e-b300-f14101ba8a55 Received: from EX19D007EUA001.ant.amazon.com (10.252.50.133) by EX19MTAEUC001.ant.amazon.com (10.252.51.193) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Wed, 11 Sep 2024 14:38:00 +0000 Received: from EX19MTAUEC001.ant.amazon.com (10.252.135.222) by EX19D007EUA001.ant.amazon.com (10.252.50.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Wed, 11 Sep 2024 14:37:59 +0000 Received: from dev-dsk-faresx-1b-27755bf1.eu-west-1.amazon.com (10.253.79.181) by mail-relay.amazon.com (10.252.135.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Wed, 11 Sep 2024 14:37:56 +0000 From: Fares Mehanna To: CC: , Fares Mehanna , "Marc Zyngier" , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , "Will Deacon" , Andrew Morton , "Kemeng Shi" , =?utf-8?q?Pierre-Cl=C3=A9ment_Tos?= =?utf-8?q?i?= , Ard Biesheuvel , Mark Rutland , Javier Martinez Canillas , "Arnd Bergmann" , Fuad Tabba , Mark Brown , Joey Gouly , Kristina Martsenko , Randy Dunlap , "Bjorn Helgaas" , Jean-Philippe Brucker , "Mike Rapoport (IBM)" , "David Hildenbrand" , Roman Kagan , "moderated list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , "open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , open list , "open list:MEMORY MANAGEMENT" Subject: [RFC PATCH 7/7] arm64: KVM: Allocate vCPU fp-regs dynamically on VHE and KERNEL_SECRETMEM enabled systems Date: Wed, 11 Sep 2024 14:34:06 +0000 Message-ID: <20240911143421.85612-8-faresx@amazon.de> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240911143421.85612-1-faresx@amazon.de> References: <20240911143421.85612-1-faresx@amazon.de> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 7gjtrmsopjrqxffxx1f5yhbu53bojtcq X-Rspamd-Queue-Id: 997F3180004 X-Rspamd-Server: rspam02 X-HE-Tag: 1726065486-285593 X-HE-Meta: U2FsdGVkX1/olbb64ts/HkD2u+hT8O2BL4AvJEq9I4qOS/NlUHKC/MUcIrvJQzPUpwBFXJjKEKeZ/qleDtkwcKIQBXfhogJ+V8S08/68XaAGR9nv4AnsJHHshI6369ItIklisrFMFsQ1d33P1PnrsbhLxCZMRaUK0mMaZXki5PcQLSq619pn5Jf7kxezWQHzLt8cN6k0XRXLe9pjiO6SZv2cgWR/SuxQoiHzM5xaOK7JW3NoK2moxxzO10NKi2hlZTygufBawynuhiGekUU1FqvM/pTFYqjnTtBASZKhP7WycDhDVOIQsTmZSfTkLvpu0tME6+qbYVWH5V/1iZ3Er15VVn/YlVfEx07jpYN/Y+y2MM1sWW/pa34jQoNNFX/brRO7nH39z6rr3Duvqs0WH0CU3mw17DSpeiIpBBF1AvrvgwZ4DM6+d0Mha8u3I44lR5u/A3V2TEpnXx1MNOmtQKI826bNyH2WtvvbaGxbLYu+vsq5FIVN8jWZSpX2AO3Jz9rLSD6QSoi6C6SrZdUdxpyd447/w7NLK9dHOYQxf82Xy/tjc2R49h7GKx7ik4Coxfv1Qo6rC35n3ZQYCxuLcR0906Af5Kd+2evKlE+yFTH2GO8/fAQ0owt7Us1yddoNUiQHf5X5cgvN5l/XFEimVXXm3vQf4lnd0RDfSzQkMosxut1OG2iu4PUCNnUElVXX3aOaWJlLE0x8o1VBhXfGz3Mz2ijknh+o/N9/30MW/zEFylB+PYLCUloFYPe0kdPby16vBPG4VGe1duP+uvkhLpGwrlRhyB2ZiT0C0mu8cbKcdXRywuX+efhmUqMASiK6fMN18I2vazL0yxSS6om51yFdOs3s1SOUOrENFA+Iz8x3eMzaQ/x8I9vX6yV/+AGMtfh0qsejZZhvJNkWo8UiT4YGeB8mqOP3a1EpP1aChv/e3+Nuajdxzpw4qaPaiwkm4axaTe2+KUHjsximoHm YNKZAA5S E7xKw5FiYrjqGElkrxfhdRq6zDQdErBU44no7RI2vRdhT2Cx5SKAQND37mzdFaEgpDeU8Ox9+7MTJ/nlAVjBBx/rVha/lEzjK6ahw4tycLvmPT9f9KnEynrTHpTHtLRX7jlpORRGn0FWCyWvmFoYobIOevJoc1IvZz6shQqV1ViH+hCtJPAxzMYAQrc0j2AcM4yJVFEpUTMwOj0dccyxhHcIBaRMcRt4TN83Or4iLVjwO8yjEnApdisa4x5yutUqSzXgDqHZFuIoe2nvSgdYZkTfnRsfA4w+c/JpRJB8ua6eCooN9SjMVK6Pafr3waJ1CflDRQQ/JSgnpQBQorW64OPYk6WHscXM16Ug/zXQz6iqj+7IYLxoBGYajEzXBo+1HTAWCG+mQ/M6QUtIlAfosX4phUYUbsGoGZgh0LfbMR8Pw7UvxpW0Zu8flsqT6z11aI/mC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similar to what was done in this commit: "arm64: KVM: Allocate vCPU gp-regs dynamically on VHE and KERNEL_SECRETMEM enabled systems" We're moving fp-regs to dynamic memory for systems supporting VHE and compiled with KERNEL_SECRETMEM support. Otherwise, we will use the "fp_regs_storage" struct embedded in the vCPU context. Accessing fp-regs embedded in the vCPU context without de-reference is done as: add \regs, \ctxt, #offsetof(struct kvm_cpu_context, fp_regs_storage) Accessing the dynamically allocated fp-regs with de-reference is done as: ldr \regs, [\ctxt, #offsetof(struct kvm_cpu_context, fp_regs)] Signed-off-by: Fares Mehanna --- arch/arm64/include/asm/kvm_host.h | 16 ++++++++++++++-- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kvm/arm.c | 29 +++++++++++++++++++++++++++-- arch/arm64/kvm/va_layout.c | 23 +++++++++++++++++++---- 4 files changed, 61 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index e8ed2c12479f..4132c57d7e69 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -550,7 +550,9 @@ struct kvm_cpu_context { u64 spsr_irq; u64 spsr_fiq; - struct user_fpsimd_state fp_regs; + struct user_fpsimd_state *fp_regs; + struct user_fpsimd_state fp_regs_storage; + struct secretmem_area *fp_regs_area; u64 sys_regs[NR_SYS_REGS]; @@ -968,7 +970,17 @@ static __always_inline struct user_pt_regs *ctxt_gp_regs(const struct kvm_cpu_co return regs; } #define vcpu_gp_regs(v) (ctxt_gp_regs(&(v)->arch.ctxt)) -#define ctxt_fp_regs(ctxt) (&(ctxt).fp_regs) + +static __always_inline struct user_fpsimd_state *ctxt_fp_regs(const struct kvm_cpu_context *ctxt) +{ + struct user_fpsimd_state *fp_regs = (void *) ctxt; + asm volatile(ALTERNATIVE_CB("add %0, %0, %1\n", + ARM64_HAS_VIRT_HOST_EXTN, + kvm_update_ctxt_fp_regs) + : "+r" (fp_regs) + : "I" (offsetof(struct kvm_cpu_context, fp_regs_storage))); + return fp_regs; +} #define vcpu_fp_regs(v) (ctxt_fp_regs(&(v)->arch.ctxt)) /* diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index e3bb626e299c..904573598e0f 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -87,6 +87,7 @@ KVM_NVHE_ALIAS(kvm_update_va_mask); KVM_NVHE_ALIAS(kvm_get_kimage_voffset); KVM_NVHE_ALIAS(kvm_compute_final_ctr_el0); KVM_NVHE_ALIAS(kvm_update_ctxt_gp_regs); +KVM_NVHE_ALIAS(kvm_update_ctxt_fp_regs); KVM_NVHE_ALIAS(spectre_bhb_patch_loop_iter); KVM_NVHE_ALIAS(spectre_bhb_patch_loop_mitigation_enable); KVM_NVHE_ALIAS(spectre_bhb_patch_wa3); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 7542af3f766a..17b42e9099c3 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -477,6 +477,14 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) if (!vcpu->arch.ctxt.regs_area) return -ENOMEM; vcpu->arch.ctxt.regs = vcpu->arch.ctxt.regs_area->ptr; + + pages_needed = (sizeof(*vcpu_fp_regs(vcpu)) + PAGE_SIZE - 1) / PAGE_SIZE; + vcpu->arch.ctxt.fp_regs_area = secretmem_allocate_pages(fls(pages_needed - 1)); + if (!vcpu->arch.ctxt.fp_regs_area) { + err = -ENOMEM; + goto free_vcpu_ctxt; + } + vcpu->arch.ctxt.fp_regs = vcpu->arch.ctxt.fp_regs_area->ptr; } /* Set up the timer */ @@ -504,8 +512,10 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) return kvm_share_hyp(vcpu, vcpu + 1); free_vcpu_ctxt: - if (kvm_use_dynamic_regs()) + if (kvm_use_dynamic_regs()) { secretmem_release_pages(vcpu->arch.ctxt.regs_area); + secretmem_release_pages(vcpu->arch.ctxt.fp_regs_area); + } return err; } @@ -524,8 +534,10 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) kvm_vgic_vcpu_destroy(vcpu); kvm_arm_vcpu_destroy(vcpu); - if (kvm_use_dynamic_regs()) + if (kvm_use_dynamic_regs()) { secretmem_release_pages(vcpu->arch.ctxt.regs_area); + secretmem_release_pages(vcpu->arch.ctxt.fp_regs_area); + } } void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) @@ -2729,12 +2741,25 @@ static int init_hyp_hve_mode(void) per_cpu(kvm_host_data, cpu).host_ctxt.regs = kvm_host_data_regs; } + /* Allocate fp-regs */ + for_each_possible_cpu(cpu) { + void *kvm_host_data_regs; + + kvm_host_data_regs = kzalloc(sizeof(struct user_fpsimd_state), GFP_KERNEL); + if (!kvm_host_data_regs) { + err = -ENOMEM; + goto free_regs; + } + per_cpu(kvm_host_data, cpu).host_ctxt.fp_regs = kvm_host_data_regs; + } + return 0; free_regs: for_each_possible_cpu(cpu) { kfree(per_cpu(kvm_hyp_ctxt, cpu).regs); kfree(per_cpu(kvm_host_data, cpu).host_ctxt.regs); + kfree(per_cpu(kvm_host_data, cpu).host_ctxt.fp_regs); } return err; diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c index fcef7e89d042..ba1030fa5b08 100644 --- a/arch/arm64/kvm/va_layout.c +++ b/arch/arm64/kvm/va_layout.c @@ -185,10 +185,12 @@ void __init kvm_update_va_mask(struct alt_instr *alt, } } -void __init kvm_update_ctxt_gp_regs(struct alt_instr *alt, - __le32 *origptr, __le32 *updptr, int nr_inst) +static __always_inline void __init kvm_update_ctxt_regs(struct alt_instr *alt, + __le32 *origptr, + __le32 *updptr, + int nr_inst, u32 imm) { - u32 rd, rn, imm, insn, oinsn; + u32 rd, rn, insn, oinsn; BUG_ON(nr_inst != 1); @@ -198,7 +200,6 @@ void __init kvm_update_ctxt_gp_regs(struct alt_instr *alt, oinsn = le32_to_cpu(origptr[0]); rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn); rn = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RN, oinsn); - imm = offsetof(struct kvm_cpu_context, regs); insn = aarch64_insn_gen_load_store_imm(rd, rn, imm, AARCH64_INSN_SIZE_64, @@ -208,6 +209,20 @@ void __init kvm_update_ctxt_gp_regs(struct alt_instr *alt, updptr[0] = cpu_to_le32(insn); } +void __init kvm_update_ctxt_gp_regs(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst) +{ + u32 offset = offsetof(struct kvm_cpu_context, regs); + kvm_update_ctxt_regs(alt, origptr, updptr, nr_inst, offset); +} + +void __init kvm_update_ctxt_fp_regs(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst) +{ + u32 offset = offsetof(struct kvm_cpu_context, fp_regs); + kvm_update_ctxt_regs(alt, origptr, updptr, nr_inst, offset); +} + void kvm_patch_vector_branch(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst) {