From patchwork Mon Mar 14 20:01:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12780744 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A380EC433EF for ; Mon, 14 Mar 2022 20:04:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=98R9rrRfZcZDPT0MhXQG+cEMh/X4JH04y+reUwDz6f8=; b=IeqkP1XjNu4gfV hizqQY0eh8lKJd8MLCuW0sH96kRdmoKutltyRCkkt07muWpCsgrmAnUkAfvNxqCIeisl5YaW5lkgV Lj4Qw5qC4Si6ePCW6+c5LIMqSKPnxfXy6MV6s312K6GUvUuT8Nt4PZkInYONCsGx/X5VqecaDOEaJ Cpl9ZnKP6R52lEr0mkB+NOy9f3ep14l9nwXmN8sYK1bpITN+Zvu17oSSK1pPlzMNjcMRGTwJ1Ar03 n97QHZfZDOa9VjPX6WD73HuYsXGtWiF6bTEtURR4XowHgYdRG5FitGQ7gcKwoTbNdcCPM9FPuTul6 Tj+vg2AbJWptrsUuU6Og==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqu2-006h94-Ek; Mon, 14 Mar 2022 20:02:50 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqtx-006h76-3M for linux-arm-kernel@lists.infradead.org; Mon, 14 Mar 2022 20:02:46 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-2e584d5159eso1680017b3.23 for ; Mon, 14 Mar 2022 13:02:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=fM7xdaekwmagyuKy0kdx3p5v2FfzYu7Y7vGXarjRfKM=; b=Jju9oT+vQW6Extb2//d+2hPcrzdqRTRz8LLvYdwkCzvVoHbTYTsLeq6DDN9pwSg4ex dJuDeCwtJBErKNwSBqyIMYZ6AFZs+yiF2LMOhtVE+Cb37YqZS3LzXs37tEsjiA8u3Vsn RMHL0VeQfR4FN1Xux9ZdmyC65BaUdLmGfAI+3qxiGPwG/xUCjE9QBN701VL5QjJIAUK2 sZcfFSMlrS9+17Y5ylq6szGPggUXnm/2kkUQ32XTCsVEKMh+gx80kgkhvJ6ikwn8kqNU GAoNNxigrbH/N2vf6cWg6NpJhPXDn4FL/pVYIKx5ca8jpSu3DjX0MUTI1xeaBoH/PlkP kahg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=fM7xdaekwmagyuKy0kdx3p5v2FfzYu7Y7vGXarjRfKM=; b=YhEBrnvGJ1UvIeh1FjLcxTqY8iu+BJw4dNw8PjFKBO1Vrex9dCu5sABFYuO9kwTDYb CqbmB3FJf1VjbFQ/gMI+TAxhgqN4iGhW+h8y+uo6QVM/CcHLQ8Tb3vsQ2gCUAS6ATtZL XAM2EY2LOAlaQjuYVlNxhZq+YGKmUMirPfZYkYY2D/e95brn5aO4tp/n4jJHR7FHqdti QKwdQKHvaZh2NkSPAf6AC6TSWBEfO4bz+TlD0U0dP213y57dNDs1th3jzluz/fQc1gPX JVLtSH2cIShRTJ6DxM4Pvw9As34zq5hbo1vMtfKbi92JdtQlIsD+ZnZh8nAWS47kqARH Fu6g== X-Gm-Message-State: AOAM530cExuWVsjS1nqbBAgnUO7plEyGpzl8VlUPJrhX1dyPdldnonCN BwJQzgqb9PDW7VIewe5UKPnSw6v8w+Sw/g2MMQ== X-Google-Smtp-Source: ABdhPJx9FWdZStCzmeuHXnB38FrYj2l6xi5QsttqJvp/Oho17IbEa6feRFDkIv3tMXYmLFL0VOkW/PwFarS6dq/e3Q== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:500a:9ac2:e73e:69e3]) (user=kaleshsingh job=sendgmr) by 2002:a0d:d546:0:b0:2dc:5667:8642 with SMTP id x67-20020a0dd546000000b002dc56678642mr20779116ywd.111.1647288163318; Mon, 14 Mar 2022 13:02:43 -0700 (PDT) Date: Mon, 14 Mar 2022 13:01:10 -0700 In-Reply-To: <20220314200148.2695206-1-kaleshsingh@google.com> Message-Id: <20220314200148.2695206-2-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220314200148.2695206-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.723.g4982287a31-goog Subject: [PATCH v6 1/8] KVM: arm64: Introduce hyp_alloc_private_va_range() From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Mark Rutland , Mark Brown , Masami Hiramatsu , Peter Collingbourne , "Madhavan T. Venkataraman" , Stephen Boyd , Andrew Scull , Andrew Jones , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220314_130245_171025_44407112 X-CRM114-Status: GOOD ( 19.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org hyp_alloc_private_va_range() can be used to reserve private VA ranges in the nVHE hypervisor. Allocations are aligned based on the order of the requested size. This will be used to implement stack guard pages for KVM nVHE hypervisor (nVHE Hyp mode / not pKVM), in a subsequent patch in the series. Signed-off-by: Kalesh Singh Tested-by: Fuad Tabba Reviewed-by: Fuad Tabba --- Changes in v6: - Update kernel-doc for hyp_alloc_private_va_range() and add return description, per Stephen - Update hyp_alloc_private_va_range() to return an int error code, per Stephen - Replace IS_ERR() checks with IS_ERR_VALUE() check, per Stephen - Clean up goto, per Stephen Changes in v5: - Align private allocations based on the order of their size, per Marc Changes in v4: - Handle null ptr in hyp_alloc_private_va_range() and replace IS_ERR_OR_NULL checks in callers with IS_ERR checks, per Fuad - Fix kernel-doc comments format, per Fuad Changes in v3: - Handle null ptr in IS_ERR_OR_NULL checks, per Mark arch/arm64/include/asm/kvm_mmu.h | 1 + arch/arm64/kvm/mmu.c | 66 +++++++++++++++++++++----------- 2 files changed, 45 insertions(+), 22 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 81839e9a8a24..3cc9aa25f510 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -153,6 +153,7 @@ static __always_inline unsigned long __kern_hyp_va(unsigned long v) int kvm_share_hyp(void *from, void *to); void kvm_unshare_hyp(void *from, void *to); int create_hyp_mappings(void *from, void *to, enum kvm_pgtable_prot prot); +int hyp_alloc_private_va_range(size_t size, unsigned long *haddr); int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size, void __iomem **kaddr, void __iomem **haddr); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index bc2aba953299..7326d683c500 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -457,23 +457,22 @@ int create_hyp_mappings(void *from, void *to, enum kvm_pgtable_prot prot) return 0; } -static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, - unsigned long *haddr, - enum kvm_pgtable_prot prot) + +/** + * hyp_alloc_private_va_range - Allocates a private VA range. + * @size: The size of the VA range to reserve. + * @haddr: The hypervisor virtual start address of the allocation. + * + * The private virtual address (VA) range is allocated below io_map_base + * and aligned based on the order of @size. + * + * Return: 0 on success or negative error code on failure. + */ +int hyp_alloc_private_va_range(size_t size, unsigned long *haddr) { unsigned long base; int ret = 0; - if (!kvm_host_owns_hyp_mappings()) { - base = kvm_call_hyp_nvhe(__pkvm_create_private_mapping, - phys_addr, size, prot); - if (IS_ERR_OR_NULL((void *)base)) - return PTR_ERR((void *)base); - *haddr = base; - - return 0; - } - mutex_lock(&kvm_hyp_pgd_mutex); /* @@ -484,30 +483,53 @@ static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, * * The allocated size is always a multiple of PAGE_SIZE. */ - size = PAGE_ALIGN(size + offset_in_page(phys_addr)); - base = io_map_base - size; + base = io_map_base - PAGE_ALIGN(size); + + /* Align the allocation based on the order of its size */ + base = ALIGN_DOWN(base, PAGE_SIZE << get_order(size)); /* * Verify that BIT(VA_BITS - 1) hasn't been flipped by * allocating the new area, as it would indicate we've * overflowed the idmap/IO address range. */ - if ((base ^ io_map_base) & BIT(VA_BITS - 1)) + if (!base || (base ^ io_map_base) & BIT(VA_BITS - 1)) ret = -ENOMEM; else - io_map_base = base; + *haddr = io_map_base = base; mutex_unlock(&kvm_hyp_pgd_mutex); + return ret; +} + +static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, + unsigned long *haddr, + enum kvm_pgtable_prot prot) +{ + unsigned long addr; + int ret = 0; + + if (!kvm_host_owns_hyp_mappings()) { + addr = kvm_call_hyp_nvhe(__pkvm_create_private_mapping, + phys_addr, size, prot); + if (IS_ERR_VALUE(addr)) + return addr; + *haddr = addr; + + return 0; + } + + size += offset_in_page(phys_addr); + ret = hyp_alloc_private_va_range(size, &addr); if (ret) - goto out; + return ret; - ret = __create_hyp_mappings(base, size, phys_addr, prot); + ret = __create_hyp_mappings(addr, size, phys_addr, prot); if (ret) - goto out; + return ret; - *haddr = base + offset_in_page(phys_addr); -out: + *haddr = addr + offset_in_page(phys_addr); return ret; } From patchwork Mon Mar 14 20:01:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12780745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 45FCBC433EF for ; Mon, 14 Mar 2022 20:04:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rSWw7M2VweKlt1WcEeVieRIIOdPPfwsHJwzZVEisb7U=; b=b5b5meEKjsOf++ rqZVikvEXv0VikUirh5fYyU4QHjsf+Lp0ehxWkY2XmNcwjLnmX90liCBnIYbpN8lYzHjp3sCFkdpj OwXy/FOsp8cBKP0Vu0w+x0xChMlD1kfSPtg9dGA2ckW9vjDwtMF9iuxqVDoXoHX8rOr0LmETSnKy7 hrKtU/BsGWg/oZwPQ48fr3VaPPUXn7wvUdcz5rpbX+IGQgnH/AsqVtS508ZeaQm0zO1WBd9qqtdxP +7kGIEy7itAcG/aEsdMP9tmQEuvIhUmsV+e85eNKMA1dg1zBUK5CQ3xIof0ObrK91CZjDK5EBJhZv ETk55eCvipCBDrPhNBqw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqui-006hQy-DN; Mon, 14 Mar 2022 20:03:32 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTquX-006hML-Ev for linux-arm-kernel@lists.infradead.org; Mon, 14 Mar 2022 20:03:23 +0000 Received: by mail-yb1-xb49.google.com with SMTP id n66-20020a254045000000b0062883b59ddbso14807520yba.12 for ; Mon, 14 Mar 2022 13:03:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=InfIvBONUVycW2B9vnVERVjcwgpgQZgScpxjxPux7vM=; b=m6KnHhjBv1ZGUvKA7QmFBvxppRt+2fWSM144rzVlJJBNef5bT8lnGxFLP0cELierCY Di8x42Gx1gxIVanbebVBH/wjuCmaePoSo3pbH6E6iKD0zt3s64lzngeZNqp/+iNcg1kn D829qhNDfaiXArwz5n4u8KIGtz6zFrXdOOm7iMwSemR6bVr1C1qOGMhGIzVb3Tz5V8Al 3BySosfiDojf++JKrXIImdL5ZzJV9XnOVCtRkVhFVIci915KPsXX6i4STHifuotpJr8c +6l0kkpRVwZ1vFs2DgcrM2OmLqgk+Y3KVzR5fghIve8YQJWrRO+9EGKfUUrdPjKbWbb2 qeDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=InfIvBONUVycW2B9vnVERVjcwgpgQZgScpxjxPux7vM=; b=lt6DfiCgeApXWXX2XMbYDOyzOD4P0Trkfrc346o6hCPxOQlCWe/egOXx67qGn+EAaR oZOBmADsN6WlcBJGrW+6kmOOrn0nMQGKTmWjCv9jQtCls8JU1+asWLeDoYFY2kUFwGqx giwtPVHdGDbenWax5Wddi+zEZ3xTuYVohS5VHPIFcWNCA5ztV3vRLr7YcpBO7HGJodlN eUuvLfOHen8EkFd9am7e0nXgY1iG9bm/pQAMCevhNn+dD1FkTgXMAeE67EP4/1l2LbYz VGfDNrUb5nY/NSC9/R+Etz6HJNivxLIh7RkMPsZniUeu4tkvG8Ravrfs2FhHil4ZEiMC V1nA== X-Gm-Message-State: AOAM532FACwl2ihILylN/4YETfwGQFI5CA06V3WaTw9pLW7Wdd/v7BYd U4BGF8lhYrpQl49HQO32kakdInnzjrUyB0AShw== X-Google-Smtp-Source: ABdhPJzhmr48eR68AWLXHoJ7fNXQmlSPhOD1eD7H6QfiAv+Of+aksPwIKH+V8byt7+ewjZYJEnlZl6LUjNKc4yySdw== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:500a:9ac2:e73e:69e3]) (user=kaleshsingh job=sendgmr) by 2002:a25:8109:0:b0:633:73fc:208d with SMTP id o9-20020a258109000000b0063373fc208dmr113300ybk.462.1647288199690; Mon, 14 Mar 2022 13:03:19 -0700 (PDT) Date: Mon, 14 Mar 2022 13:01:11 -0700 In-Reply-To: <20220314200148.2695206-1-kaleshsingh@google.com> Message-Id: <20220314200148.2695206-3-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220314200148.2695206-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.723.g4982287a31-goog Subject: [PATCH v6 2/8] KVM: arm64: Introduce pkvm_alloc_private_va_range() From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Mark Rutland , Mark Brown , Masami Hiramatsu , Peter Collingbourne , "Madhavan T. Venkataraman" , Stephen Boyd , Andrew Scull , Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220314_130321_556060_7263BE4B X-CRM114-Status: GOOD ( 21.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org pkvm_hyp_alloc_private_va_range() can be used to reserve private VA ranges in the pKVM nVHE hypervisor. Allocations are aligned based on the order of the requested size. This will be used to implement stack guard pages for pKVM nVHE hypervisor (in a subsequent patch in the series). Credits to Quentin Perret for the idea of moving private VA allocation out of __pkvm_create_private_mapping() Signed-off-by: Kalesh Singh Tested-by: Fuad Tabba Reviewed-by: Fuad Tabba --- Changes in v6: - Update kernel-doc for pkvm_alloc_private_va_range() and add return description, per Stephen - Update pkvm_alloc_private_va_range() to return an int error code, per Stephen - Update __pkvm_create_private_mapping to return an in error code, per Quentin - Update callers of __pkvm_create_private_mapping() to handle new return value and params. Changes in v5: - Align private allocations based on the order of their size, per Marc Changes in v4: - Handle null ptr in pkvm_alloc_private_va_range() and replace IS_ERR_OR_NULL checks in callers with IS_ERR checks, per Fuad - Fix kernel-doc comments format, per Fuad - Format __pkvm_create_private_mapping() prototype args (< 80 col), per Fuad Changes in v3: - Handle null ptr in IS_ERR_OR_NULL checks, per Mark Changes in v2: - Allow specifying an alignment for the private VA allocations, per Marc arch/arm64/kvm/hyp/include/nvhe/mm.h | 6 ++- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 18 ++++++- arch/arm64/kvm/hyp/nvhe/mm.c | 78 ++++++++++++++++++---------- 3 files changed, 72 insertions(+), 30 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h index 2d08510c6cc1..42d8eb9bfe72 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -19,8 +19,10 @@ int hyp_back_vmemmap(phys_addr_t phys, unsigned long size, phys_addr_t back); int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot); int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot); int pkvm_create_mappings_locked(void *from, void *to, enum kvm_pgtable_prot prot); -unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, - enum kvm_pgtable_prot prot); +int __pkvm_create_private_mapping(phys_addr_t phys, size_t size, + enum kvm_pgtable_prot prot, + unsigned long *haddr); +int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr); static inline void hyp_vmemmap_range(phys_addr_t phys, unsigned long size, unsigned long *start, unsigned long *end) diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 5e2197db0d32..3cea4b6ac23e 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -160,7 +160,23 @@ static void handle___pkvm_create_private_mapping(struct kvm_cpu_context *host_ct DECLARE_REG(size_t, size, host_ctxt, 2); DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 3); - cpu_reg(host_ctxt, 1) = __pkvm_create_private_mapping(phys, size, prot); + /* + * __pkvm_create_private_mapping() populates a pointer with the + * hypervisor start address of the allocation. + * + * However, handle___pkvm_create_private_mapping() hypercall crosses the + * EL1/EL2 boundary so the pointer would not be valid in this context. + * + * Instead pass the allocation address as the return value (or return + * ERR_PTR() on failure). + */ + unsigned long haddr; + int err = __pkvm_create_private_mapping(phys, size, prot, &haddr); + + if (err) + haddr = (unsigned long)ERR_PTR(err); + + cpu_reg(host_ctxt, 1) = haddr; } static void handle___pkvm_prot_finalize(struct kvm_cpu_context *host_ctxt) diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c index cdbe8e246418..670f11349070 100644 --- a/arch/arm64/kvm/hyp/nvhe/mm.c +++ b/arch/arm64/kvm/hyp/nvhe/mm.c @@ -37,36 +37,60 @@ static int __pkvm_create_mappings(unsigned long start, unsigned long size, return err; } -unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, - enum kvm_pgtable_prot prot) +/** + * pkvm_alloc_private_va_range - Allocates a private VA range. + * @size: The size of the VA range to reserve. + * @haddr: The hypervisor virtual start address of the allocation. + * + * The private virtual address (VA) range is allocated above __io_map_base + * and aligned based on the order of @size. + * + * Return: 0 on success or negative error code on failure. + */ +int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr) { - unsigned long addr; - int err; + unsigned long base, addr; + int ret = 0; hyp_spin_lock(&pkvm_pgd_lock); - size = PAGE_ALIGN(size + offset_in_page(phys)); - addr = __io_map_base; - __io_map_base += size; + /* Align the allocation based on the order of its size */ + addr = ALIGN(__io_map_base, PAGE_SIZE << get_order(size)); - /* Are we overflowing on the vmemmap ? */ - if (__io_map_base > __hyp_vmemmap) { - __io_map_base -= size; - addr = (unsigned long)ERR_PTR(-ENOMEM); - goto out; - } + /* The allocated size is always a multiple of PAGE_SIZE */ + base = addr + PAGE_ALIGN(size); - err = kvm_pgtable_hyp_map(&pkvm_pgtable, addr, size, phys, prot); - if (err) { - addr = (unsigned long)ERR_PTR(err); - goto out; + /* Are we overflowing on the vmemmap ? */ + if (!addr || base > __hyp_vmemmap) + ret = -ENOMEM; + else { + __io_map_base = base; + *haddr = addr; } - addr = addr + offset_in_page(phys); -out: hyp_spin_unlock(&pkvm_pgd_lock); - return addr; + return ret; +} + +int __pkvm_create_private_mapping(phys_addr_t phys, size_t size, + enum kvm_pgtable_prot prot, + unsigned long *haddr) +{ + unsigned long addr; + int err; + + size += offset_in_page(phys); + err = pkvm_alloc_private_va_range(size, &addr); + if (err) + return err; + + err = __pkvm_create_mappings(addr, size, phys, prot); + if (err) + return err; + + *haddr = addr + offset_in_page(phys); + return err; } int pkvm_create_mappings_locked(void *from, void *to, enum kvm_pgtable_prot prot) @@ -146,7 +170,8 @@ int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot) int hyp_map_vectors(void) { phys_addr_t phys; - void *bp_base; + unsigned long bp_base; + int ret; if (!kvm_system_needs_idmapped_vectors()) { __hyp_bp_vect_base = __bp_harden_hyp_vecs; @@ -154,13 +179,12 @@ int hyp_map_vectors(void) } phys = __hyp_pa(__bp_harden_hyp_vecs); - bp_base = (void *)__pkvm_create_private_mapping(phys, - __BP_HARDEN_HYP_VECS_SZ, - PAGE_HYP_EXEC); - if (IS_ERR_OR_NULL(bp_base)) - return PTR_ERR(bp_base); + ret = __pkvm_create_private_mapping(phys, __BP_HARDEN_HYP_VECS_SZ, + PAGE_HYP_EXEC, &bp_base); + if (ret) + return ret; - __hyp_bp_vect_base = bp_base; + __hyp_bp_vect_base = (void *)bp_base; return 0; } From patchwork Mon Mar 14 20:01:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12780746 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A86CEC4332F for ; Mon, 14 Mar 2022 20:05:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ARJifowxiM/I/Oj5WW1SlJOw4CG1RTXHhaGKS1bR6ZM=; b=qeagyptLbEEbtk g2VnhmOR7sDpQz6w65Z1GhnUCXuFcT0zNxvegqt6YhW6F8TR+muGZAvoCM6ffBcW+Lu5GJ7sRrvQs aAVUzccDVrqLIo7Ql0eaq7XnKDnF8gvJ8uq+IEdC2p+1sSwD+Bu1HwZytSPtvB65PrYYPXERMhpoa NZAF6Ywyz5nEfk8aUBmbUjz7nSGONHXzWIZgMFUE0qJfKgB1lHD2IQ251ci8vAPg/k5fX9a/d+edY NIJY3Svo7c+ZljPCZmegj4taSKfwZfqfeIs4gfZ5j1WIdDIlDkRN1wOTK6bP3hsMpvxL73p8/IFMd J80SZy0egcAtN2EQUy4A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqvG-006hek-48; Mon, 14 Mar 2022 20:04:06 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqv7-006hbS-9W for linux-arm-kernel@lists.infradead.org; Mon, 14 Mar 2022 20:04:00 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-2d7eaa730d9so141851887b3.13 for ; Mon, 14 Mar 2022 13:03:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=KBKuM8nQbHEsk5KoB9KDoIyUXnj0iRI96x05Zc5GVwU=; b=pSlak9X+zf0pVcMyDGHXsdj/0ZvCoyfAdmZkLL2IoXQ/eoplgnCHVzUHV6625bNCMO 5W/FfGYImrd/5arptPi/Rh1DZ3oNGT2WHcLtPHwLYae1RG+2+WzzpSLpx2IBnWYENwKv olSKeenA96Zm6AESUkoS6AaJro4cv3oA8G58W7nRuRrabP+0XkTX3+FgmM2KLrYrKfxI fh3CE8qwaMgsfxjjWdwl2sVShwIJXNUkC9SCrJe9gGVTROtWlcut+zWSPLRbuIgzhnlp LaMbeaJkLAlGGeT8FDnOUeP6d721ULqiYxQkoP6I2jelyXGKa7OrDJwlWFpyCLR4N3P3 sZSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=KBKuM8nQbHEsk5KoB9KDoIyUXnj0iRI96x05Zc5GVwU=; b=kGkIbRMIBo2lf6pZQJ0RCZkc6RWdyly3kj07e7Nf/dFNH96ByxYGatik7m6F/oC9rL RKwBHARbuhQN5r24gLf6MgqZnTC6eXKn3HQEImRFp0tARxI58mMlfVNYYr6USeRTiRxl Ci6h/hQXSwBd9Lg+3TtCyDXc9kSCW2ZDWpZqyRzxv6I7ze4sCtCCO8u2rmpvL28UHqsU QwjlM4ZvyrCt+ZVtdGN6nWDcV5eOP8doNTjDasmoqxW9MdOycrMKFxKktedfAt9qDmwE nqprvRD0JeDa3alc6BWwxb5Z15En1JDpMqGASWKn4EFSQKx+TRI77TxDi3JGQTL2rLDE 6/QQ== X-Gm-Message-State: AOAM531GrJb1hWTbbJ9vTfmZcfrqgF4ntu4LMHmUU/egBpa71shZz4qf c810i/KvipBEM/dqen4Cje8EcfiqzqBTsJL7gw== X-Google-Smtp-Source: ABdhPJy9MIn39nszadJy4TGmciC+Phtwg3irGAJ8Z0MmmzIQBmJ0g8Fd8p3i/9X7ZdC1H/f6r8RSPw1B8z+qXReY+g== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:500a:9ac2:e73e:69e3]) (user=kaleshsingh job=sendgmr) by 2002:a0d:e244:0:b0:2db:d6a8:14c9 with SMTP id l65-20020a0de244000000b002dbd6a814c9mr21314233ywe.402.1647288235164; Mon, 14 Mar 2022 13:03:55 -0700 (PDT) Date: Mon, 14 Mar 2022 13:01:12 -0700 In-Reply-To: <20220314200148.2695206-1-kaleshsingh@google.com> Message-Id: <20220314200148.2695206-4-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220314200148.2695206-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.723.g4982287a31-goog Subject: [PATCH v6 3/8] KVM: arm64: Add guard pages for KVM nVHE hypervisor stack From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Mark Rutland , Mark Brown , Masami Hiramatsu , Peter Collingbourne , "Madhavan T. Venkataraman" , Stephen Boyd , Andrew Walbran , Andrew Scull , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220314_130357_387187_72F2E37B X-CRM114-Status: GOOD ( 20.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Map the stack pages in the flexible private VA range and allocate guard pages below the stack as unbacked VA space. The stack is aligned so that any valid stack address has PAGE_SHIFT bit as 1 - this is used for overflow detection (implemented in a subsequent patch in the series). Signed-off-by: Kalesh Singh Tested-by: Fuad Tabba Reviewed-by: Fuad Tabba --- Changes in v6: - Update call to hyp_alloc_private_va_range() (return val and params) Changes in v5: - Use a single allocation for stack and guard pages to ensure they are contiguous, per Marc Changes in v4: - Replace IS_ERR_OR_NULL check with IS_ERR check now that hyp_alloc_private_va_range() returns an error for null pointer, per Fuad - Format comments to < 80 cols, per Fuad Changes in v3: - Handle null ptr in IS_ERR_OR_NULL checks, per Mark arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/include/asm/kvm_mmu.h | 3 +++ arch/arm64/kvm/arm.c | 39 +++++++++++++++++++++++++++++--- arch/arm64/kvm/mmu.c | 4 ++-- 4 files changed, 42 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index d5b0386ef765..2e277f2ed671 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -169,6 +169,7 @@ struct kvm_nvhe_init_params { unsigned long tcr_el2; unsigned long tpidr_el2; unsigned long stack_hyp_va; + unsigned long stack_pa; phys_addr_t pgd_pa; unsigned long hcr_el2; unsigned long vttbr; diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 3cc9aa25f510..967a9ea9a47b 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -116,6 +116,9 @@ alternative_cb_end #include #include +extern struct kvm_pgtable *hyp_pgtable; +extern struct mutex kvm_hyp_pgd_mutex; + void kvm_update_va_mask(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst); void kvm_compute_layout(void); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 4dca6ffd03d4..72be7e695d8d 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1538,7 +1538,6 @@ static void cpu_prepare_hyp_mode(int cpu) tcr |= (idmap_t0sz & GENMASK(TCR_TxSZ_WIDTH - 1, 0)) << TCR_T0SZ_OFFSET; params->tcr_el2 = tcr; - params->stack_hyp_va = kern_hyp_va(per_cpu(kvm_arm_hyp_stack_page, cpu) + PAGE_SIZE); params->pgd_pa = kvm_mmu_get_httbr(); if (is_protected_kvm_enabled()) params->hcr_el2 = HCR_HOST_NVHE_PROTECTED_FLAGS; @@ -1987,14 +1986,48 @@ static int init_hyp_mode(void) * Map the Hyp stack pages */ for_each_possible_cpu(cpu) { + struct kvm_nvhe_init_params *params = per_cpu_ptr_nvhe_sym(kvm_init_params, cpu); char *stack_page = (char *)per_cpu(kvm_arm_hyp_stack_page, cpu); - err = create_hyp_mappings(stack_page, stack_page + PAGE_SIZE, - PAGE_HYP); + unsigned long hyp_addr; + /* + * Allocate a contiguous HYP private VA range for the stack + * and guard page. The allocation is also aligned based on + * the order of its size. + */ + err = hyp_alloc_private_va_range(PAGE_SIZE * 2, &hyp_addr); + if (err) { + kvm_err("Cannot allocate hyp stack guard page\n"); + goto out_err; + } + + /* + * Since the stack grows downwards, map the stack to the page + * at the higher address and leave the lower guard page + * unbacked. + * + * Any valid stack address now has the PAGE_SHIFT bit as 1 + * and addresses corresponding to the guard page have the + * PAGE_SHIFT bit as 0 - this is used for overflow detection. + */ + mutex_lock(&kvm_hyp_pgd_mutex); + err = kvm_pgtable_hyp_map(hyp_pgtable, hyp_addr + PAGE_SIZE, + PAGE_SIZE, __pa(stack_page), PAGE_HYP); + mutex_unlock(&kvm_hyp_pgd_mutex); if (err) { kvm_err("Cannot map hyp stack\n"); goto out_err; } + + /* + * Save the stack PA in nvhe_init_params. This will be needed + * to recreate the stack mapping in protected nVHE mode. + * __hyp_pa() won't do the right thing there, since the stack + * has been mapped in the flexible private VA space. + */ + params->stack_pa = __pa(stack_page); + + params->stack_hyp_va = hyp_addr + (2 * PAGE_SIZE); } for_each_possible_cpu(cpu) { diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 7326d683c500..9bfc6d8f3c49 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -22,8 +22,8 @@ #include "trace.h" -static struct kvm_pgtable *hyp_pgtable; -static DEFINE_MUTEX(kvm_hyp_pgd_mutex); +struct kvm_pgtable *hyp_pgtable; +DEFINE_MUTEX(kvm_hyp_pgd_mutex); static unsigned long hyp_idmap_start; static unsigned long hyp_idmap_end; From patchwork Mon Mar 14 20:01:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12780747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D6543C433F5 for ; Mon, 14 Mar 2022 20:06:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=VSmINSsClS5o9b2V9wDMzwzVSTHj/ENXyveyAAvaPtQ=; b=vW+vCPhWJkqD81 tzTiFjwzQN+DBa63mQG1cu/5tCgZVZ8/5PUvQLYzuMRj/e4ojdzribsBkgQYdL4xuOvOzeTlH22TQ v6PrQxB4zflYaE7rVUI4IOfpmSAlumTRCEz6qmDaaKvcB3EeMqaUC4/DkZt6VooJZc0Q1s2mRqqny TSEfGrimX48pidxtU4fc3Fq6W7o1xD8R1Mf1VUZxvVbnXah0W/Pi8tcPuW0nXs6gFA4kefgM6nT6P Fk0z+AQCDQNQDm8MPDZLlo8e7zhWx2vL9OxQqHcXYwSWt0KUxfIoMReJYDB2shEZ5Bu9b56ComyQf Or4yLgnrVf9rN7Q6vdcw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqvp-006hvT-V6; Mon, 14 Mar 2022 20:04:42 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqvj-006ht1-Eg for linux-arm-kernel@lists.infradead.org; Mon, 14 Mar 2022 20:04:37 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id e4-20020a056902034400b00633691534d5so1203743ybs.7 for ; Mon, 14 Mar 2022 13:04:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=wcrKUjLTdB/HMJdpeOMPENhm20Kul5UdkQa5+z1O4C4=; b=ly66flbISMQbhQBRlyU0Ey6SFlvMFzMx+MCgdnpmKL7gbVZVnWJLeZ+eBkxab3u2uQ /BAPbfsTDdal7NC9tiorOpcTnS1SPCwqZUI5C63IE+zdaccPQGfmYSxoacTwGxFep/2p sYVB6HiHODgLOoNDiEG83gzblrNaMgs4O6zNyjkLeAiPKK3DV/bg6YbvMdHsVKlKv7KY 550C2M5Wqi0qqi2+Rf5QnAcPMwneFdc65JXbl6PhmeS78GfP9FQWcDKMbAEnn8H1+9ZJ SCEW87y7rwuaOAGH1aDA7RKYVL+X6hcCfbwXAskLXQyF15WTxFS95qoyIpo7yUBKpcK4 iU7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=wcrKUjLTdB/HMJdpeOMPENhm20Kul5UdkQa5+z1O4C4=; b=4p4zMM7r4l72UIwdNmyo7qF4AZ6XkXB6dOvI0xktJw/8OixmlZKAv8CboRdXp5Uy// o94Hbs/+JG+IpZZqTbF/O2fiMr/AK5CTPfwfuaAnlV9S432gD36E6Dn42u5KZ6TFCkSJ LB9f6YXx9OcNkZR9bJqpWDY0QjY/YMb2SokrkwYduSfCo47Fu6bvSMqaFvB9SPFAuIXb H0AtszHRLuCd65ROz1y6itSdiG4p8wJITQowXUA8DmUrcZNp2DwPf4Vbqsz0RNasgp/W 0WuzT42oB+0oXD0jbOYJRBmggDEoH7XazevDcocZ1dqiaE0tZnOVxrHIdICqvqb1U8y3 xaeQ== X-Gm-Message-State: AOAM531vwlnXTk1buJVtO5cGN8y0E0yUIgvFwaT+uqT5lURgYPmKWmWC jOAkDC2ByipathBMEKmGZdonJiMIDzI/SZxcKw== X-Google-Smtp-Source: ABdhPJzCQjLXyp0zrSq0fPr0qCvgIfWkpLUI1kq/oOWycXP2KR703/KzwXb0OI2paMiDufMo2S2nH7jF9T3m4zbQpw== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:500a:9ac2:e73e:69e3]) (user=kaleshsingh job=sendgmr) by 2002:a81:9c4c:0:b0:2dc:ba4:4ee with SMTP id n12-20020a819c4c000000b002dc0ba404eemr20662044ywa.248.1647288273266; Mon, 14 Mar 2022 13:04:33 -0700 (PDT) Date: Mon, 14 Mar 2022 13:01:13 -0700 In-Reply-To: <20220314200148.2695206-1-kaleshsingh@google.com> Message-Id: <20220314200148.2695206-5-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220314200148.2695206-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.723.g4982287a31-goog Subject: [PATCH v6 4/8] KVM: arm64: Add guard pages for pKVM (protected nVHE) hypervisor stack From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Mark Rutland , Mark Brown , Masami Hiramatsu , Peter Collingbourne , "Madhavan T. Venkataraman" , Andrew Scull , Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220314_130435_537512_06A68F79 X-CRM114-Status: GOOD ( 17.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Map the stack pages in the flexible private VA range and allocate guard pages below the stack as unbacked VA space. The stack is aligned so that any valid stack address has PAGE_SHIFT bit as 1 - this is used for overflow detection (implemented in a subsequent patch in the series) Signed-off-by: Kalesh Singh Tested-by: Fuad Tabba Reviewed-by: Fuad Tabba --- Changes in v6: - Update call to pkvm_alloc_private_va_range() (return val and params) Changes in v5: - Use a single allocation for stack and guard pages to ensure they are contiguous, per Marc Changes in v4: - Replace IS_ERR_OR_NULL check with IS_ERR check now that pkvm_alloc_private_va_range() returns an error for null pointer, per Fuad Changes in v3: - Handle null ptr in IS_ERR_OR_NULL checks, per Mark arch/arm64/kvm/hyp/nvhe/setup.c | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index 27af337f9fea..e8d4ea2fcfa0 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -99,17 +99,42 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size, return ret; for (i = 0; i < hyp_nr_cpus; i++) { + struct kvm_nvhe_init_params *params = per_cpu_ptr(&kvm_init_params, i); + unsigned long hyp_addr; + start = (void *)kern_hyp_va(per_cpu_base[i]); end = start + PAGE_ALIGN(hyp_percpu_size); ret = pkvm_create_mappings(start, end, PAGE_HYP); if (ret) return ret; - end = (void *)per_cpu_ptr(&kvm_init_params, i)->stack_hyp_va; - start = end - PAGE_SIZE; - ret = pkvm_create_mappings(start, end, PAGE_HYP); + /* + * Allocate a contiguous HYP private VA range for the stack + * and guard page. The allocation is also aligned based on + * the order of its size. + */ + ret = pkvm_alloc_private_va_range(PAGE_SIZE * 2, &hyp_addr); + if (ret) + return ret; + + /* + * Since the stack grows downwards, map the stack to the page + * at the higher address and leave the lower guard page + * unbacked. + * + * Any valid stack address now has the PAGE_SHIFT bit as 1 + * and addresses corresponding to the guard page have the + * PAGE_SHIFT bit as 0 - this is used for overflow detection. + */ + hyp_spin_lock(&pkvm_pgd_lock); + ret = kvm_pgtable_hyp_map(&pkvm_pgtable, hyp_addr + PAGE_SIZE, + PAGE_SIZE, params->stack_pa, PAGE_HYP); + hyp_spin_unlock(&pkvm_pgd_lock); if (ret) return ret; + + /* Update stack_hyp_va to end of the stack's private VA range */ + params->stack_hyp_va = hyp_addr + (2 * PAGE_SIZE); } /* From patchwork Mon Mar 14 20:01:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12780748 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B14B4C433F5 for ; Mon, 14 Mar 2022 20:06:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=laBJ7nvgQl+0fhMbOZut8HHFjET84I92/F/VcPLAFvE=; b=urZXzNVJckuD5j IR2sGBrUCHRI/l2OjxZK6nm/oHmtSdFkeShz33Yx/RBJOjJLQbhIotugrxOm2GK9ybMjStJnsSLOo YyrCI6Ow5NVZbbavWxpR+0ahZtuph1jMmFD/L7bInINEC/W+ugQ+487squrMVgDJPIzbWgAOnL1D0 oeNYPNzjBPg8bkzQ9DGNhGDrOT0tGUek54Q7+j6WqhD8lpPFPFrEHRvFp2ZQNO+B3EYqOWGDHtBc6 z8wx5GKy531mHPPFneKmS/WZDjwnxFMuF+A9xEXdxP7A7BpW2i7DOr83Wn343qJFzAWT0+4o7+0Nf on61DaXSQgA9Kx5BQUcw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqwU-006iE9-7S; Mon, 14 Mar 2022 20:05:24 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqwM-006iBc-6q for linux-arm-kernel@lists.infradead.org; Mon, 14 Mar 2022 20:05:15 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-2e584d5159eso1757027b3.23 for ; Mon, 14 Mar 2022 13:05:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=6AJr5ee6eL/k85TbDtFKJ/Olir+hxbCOMzelCH/e6LI=; b=DbOLudXnBJV5WO3zhARrfN9GjpNNvscKjdv6Rrb8oyyeDVG6ym1PavfRRn0cQJZ30D +dFqE5+GylXb9B4WnXVx3fW2AATv8Ys88byJ+NHvH9967pm3jvonAL9PQ+5KIMYGIyWp iahue+tdX54Tb5cmZFWMlyT82Y2P+DzxdagH7gCkcNuWgReaLtuoow7tNlycZEfv41j0 Z3MuulV5hZJ6TR8aWwWD95mixAsJjU2HvKK721BkNiM0xuUVzaKJz5nHxIYba3/1tyZF 1ffE2x9gcqNJWFC4HieFmTCSuPIQPSl70MU+91mCcGGQ94la0DsGNiOYSyr1nMfBIRI3 pk2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=6AJr5ee6eL/k85TbDtFKJ/Olir+hxbCOMzelCH/e6LI=; b=X9ABrFfj39qrMjg4zWNWwWS4idYBvjL+vfugltlcIYgz5YDjegUVygdoOu67td9//t dI6LxGaPbNVwys0vmMWHhxxePdO1Y4eaN5iO0lj1RfJY8TP8+oQuh9r9G8FaxWS2QuVx bEQ17EGYSZM4iwkha2pTWCQGCRgJSKir4srp5gwevhOELtfPPMjSQOlb1aIbMHDY+myk k1eS7hc0lTtL10sR0EtHjzPR2xxjsk5TYYf5KrhZGP8QYrO91dxuKcFHPAY9pDGF3pon myp/SSeW79473Q9fibVBe5MJaBBOl2A1S9TWwrdWXN7X29kEKF0m+rA5Y9UfVQ3pT9L6 CRyg== X-Gm-Message-State: AOAM532iedxnn3P5Gb+c3RqyQ6gFk3TNJSdsSPloNKGxZaSvizolLQbu ePyFG83YPpcYzg7JAmIfLOJLqcheSiHBq/G6tQ== X-Google-Smtp-Source: ABdhPJw1XRCjbhTUYB79h5/vjUMks+3FK7y+nHEPTRAsppIX6ownwysU3XOcH6qXpY0ENzmCluQyvA8Bf+JBhORZGw== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:500a:9ac2:e73e:69e3]) (user=kaleshsingh job=sendgmr) by 2002:a25:764b:0:b0:628:6e65:1880 with SMTP id r72-20020a25764b000000b006286e651880mr19903957ybc.264.1647288312731; Mon, 14 Mar 2022 13:05:12 -0700 (PDT) Date: Mon, 14 Mar 2022 13:01:14 -0700 In-Reply-To: <20220314200148.2695206-1-kaleshsingh@google.com> Message-Id: <20220314200148.2695206-6-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220314200148.2695206-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.723.g4982287a31-goog Subject: [PATCH v6 5/8] KVM: arm64: Detect and handle hypervisor stack overflows From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Mark Rutland , Mark Brown , Masami Hiramatsu , Peter Collingbourne , "Madhavan T. Venkataraman" , Andrew Walbran , Andrew Scull , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220314_130514_303153_0261690D X-CRM114-Status: GOOD ( 15.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The hypervisor stacks (for both nVHE Hyp mode and nVHE protected mode) are aligned such that any valid stack address has PAGE_SHIFT bit as 1. This allows us to conveniently check for overflow in the exception entry without corrupting any GPRs. We won't recover from a stack overflow so panic the hypervisor. Signed-off-by: Kalesh Singh Tested-by: Fuad Tabba Reviewed-by: Fuad Tabba --- Changes in v5: - Valid stack addresses now have PAGE_SHIFT bit as 1 instead of 0 Changes in v3: - Remove test_sp_overflow macro, per Mark - Add asmlinkage attribute for hyp_panic, hyp_panic_bad_stack, per Ard arch/arm64/kvm/hyp/nvhe/host.S | 24 ++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/switch.c | 7 ++++++- 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S index 3d613e721a75..be6d844279b1 100644 --- a/arch/arm64/kvm/hyp/nvhe/host.S +++ b/arch/arm64/kvm/hyp/nvhe/host.S @@ -153,6 +153,18 @@ SYM_FUNC_END(__host_hvc) .macro invalid_host_el2_vect .align 7 + + /* + * Test whether the SP has overflowed, without corrupting a GPR. + * nVHE hypervisor stacks are aligned so that the PAGE_SHIFT bit + * of SP should always be 1. + */ + add sp, sp, x0 // sp' = sp + x0 + sub x0, sp, x0 // x0' = sp' - x0 = (sp + x0) - x0 = sp + tbz x0, #PAGE_SHIFT, .L__hyp_sp_overflow\@ + sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 + sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp + /* If a guest is loaded, panic out of it. */ stp x0, x1, [sp, #-16]! get_loaded_vcpu x0, x1 @@ -165,6 +177,18 @@ SYM_FUNC_END(__host_hvc) * been partially clobbered by __host_enter. */ b hyp_panic + +.L__hyp_sp_overflow\@: + /* + * Reset SP to the top of the stack, to allow handling the hyp_panic. + * This corrupts the stack but is ok, since we won't be attempting + * any unwinding here. + */ + ldr_this_cpu x0, kvm_init_params + NVHE_INIT_STACK_HYP_VA, x1 + mov sp, x0 + + bl hyp_panic_bad_stack + ASM_BUG() .endm .macro invalid_host_el1_vect diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 6410d21d8695..703a5d3f611b 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -347,7 +347,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) return exit_code; } -void __noreturn hyp_panic(void) +asmlinkage void __noreturn hyp_panic(void) { u64 spsr = read_sysreg_el2(SYS_SPSR); u64 elr = read_sysreg_el2(SYS_ELR); @@ -369,6 +369,11 @@ void __noreturn hyp_panic(void) unreachable(); } +asmlinkage void __noreturn hyp_panic_bad_stack(void) +{ + hyp_panic(); +} + asmlinkage void kvm_unexpected_el2_exception(void) { return __kvm_unexpected_el2_exception(); From patchwork Mon Mar 14 20:01:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12780749 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C4DF0C433EF for ; Mon, 14 Mar 2022 20:07:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SODJqK22wp0MfE6ytQ8AmKLB9IxVj+tlfJVuYE2Hh3w=; b=d3QR4HF0M+0YdO LfwWmuJ508HEekK/A2qu1+3WLrZVIZREibfYTnmITr/YjD4JVDb1dtbH1grW6hxqN+UtTupJh+Uka LUhD2x/LTH7++n5Mq9IsraQAs3+Uy+k35fBu5eT6rjwXS9itc/Y5udOasftgc+BVIC857J5j8JB8j 6+5x8RsH//YFyq+veFSpR+UXJ7kKCZ7/E/+yQpNIgl3ZxmG/OIh767Jd0URmPPsT/O/9w1iPhIo0f xzCsi6n5VWG1HAK9wnVjZCNYs1Ciujhkwc5fVcbJ3RO5UHW1W2AB8pK2cH+UFcp73GpvQpJy7/VEr kYSAjfMPiUIYhDnhJ6EQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqxC-006iX8-6x; Mon, 14 Mar 2022 20:06:06 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqwy-006iRJ-Jv for linux-arm-kernel@lists.infradead.org; Mon, 14 Mar 2022 20:05:54 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-2d6994a6942so140677677b3.17 for ; Mon, 14 Mar 2022 13:05:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=8f0s79ka6K+xZLjhEa6v9OQwM+zc/G84NFUgH0amXtY=; b=oWMxFcmEuQa7Oz8Msf5z2d2F/+cxSvJeY2AkshA00cCwmDongR92lT++R6u+2wswp/ KeHuRfDuIINh7QuVpmgim0OtP9TVidSBRd8VA4FhF9ErV1otSCPmz+AkktixGdSZwzT8 LCfWw99uiLBPTLjaM7e2GesV6Vr95aJhEjo56x1iSRDgcy1Pi4qC604b7DxcZjF0+c7w /2K0o8E7BBCgilr6vRGvf156M3tc+fnB8epOT0a5sLqt3zWx7hzNiJnGvxaWsM5waXja J6OMDvqIkNbTk3t3YyDVaC5Q3RlFOFXcGH/fE9ejjisH84XYUlCL+8IflYOkwpId7Erp aQEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=8f0s79ka6K+xZLjhEa6v9OQwM+zc/G84NFUgH0amXtY=; b=dT6tslna2Z+Y30bfle2Bbv8jiowGb+trRwateIr0YprmfcQm5VTlADGVr9/WjblqNQ plA7JsDDDar8dHoI/aFZZ0bYVn0DmRoFsE221kk0hQFLh6rjKlEoVI2I4CIjPGMhMSDf dFtUwvNob0Ski3bed04oFKkp8QYmgge9WDR4B44Wtgd8lX87EXtZLpNXLw4fuOfHJ//h t2Q4UuIv4GQ5H7RA9uQ9WLEcihnSvWzM+HDFwFlOZjlzqugWywK5t07u4PFNYm/h5uld KdRE8YNrHsW4AxHE8LJKAccu5Xv4Sns6LX6FqTgZhsh9HocUTXNNUzSU9XRzv05Q6r+U vLEw== X-Gm-Message-State: AOAM531BtY7i+EIfCoB6PYd3P3gA+Pjl3CZKm2Ugg7Cs8Vg76+Q+aNAz gcrYRdnDYbdycBelQMgoLi5+SblDMC6+TEYQRQ== X-Google-Smtp-Source: ABdhPJxeJ4ERpyXJrRQxcj/bYJNsQODEu20+d8yn9qPL1N8ZlV6KyQDHbDeVHPvOyZo+/DauOEQQySrLcRIvNw+kpA== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:500a:9ac2:e73e:69e3]) (user=kaleshsingh job=sendgmr) by 2002:a5b:712:0:b0:629:5d03:4b1b with SMTP id g18-20020a5b0712000000b006295d034b1bmr19139174ybq.380.1647288351094; Mon, 14 Mar 2022 13:05:51 -0700 (PDT) Date: Mon, 14 Mar 2022 13:01:15 -0700 In-Reply-To: <20220314200148.2695206-1-kaleshsingh@google.com> Message-Id: <20220314200148.2695206-7-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220314200148.2695206-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.723.g4982287a31-goog Subject: [PATCH v6 6/8] KVM: arm64: Add hypervisor overflow stack From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Mark Rutland , Mark Brown , Masami Hiramatsu , Peter Collingbourne , "Madhavan T. Venkataraman" , Andrew Walbran , Andrew Scull , Andrew Jones , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220314_130552_699692_963C58E3 X-CRM114-Status: GOOD ( 14.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allocate and switch to 16-byte aligned secondary stack on overflow. This provides us stack space to better handle overflows; and is used in a subsequent patch to dump the hypervisor stacktrace. The overflow stack is only allocated if CONFIG_NVHE_EL2_DEBUG is enabled, as hypervisor stacktraces is a debug feature dependent on CONFIG_NVHE_EL2_DEBUG. Signed-off-by: Kalesh Singh Tested-by: Fuad Tabba Reviewed-by: Fuad Tabba --- Changes in v4: - Update comment to clarify resetting the SP to the top of the stack only happens if CONFIG_NVHE_EL2_DEBUG is disabled, per Fuad arch/arm64/kvm/hyp/nvhe/host.S | 11 ++++++++--- arch/arm64/kvm/hyp/nvhe/switch.c | 5 +++++ 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S index be6d844279b1..a0c4b4f1549f 100644 --- a/arch/arm64/kvm/hyp/nvhe/host.S +++ b/arch/arm64/kvm/hyp/nvhe/host.S @@ -179,13 +179,18 @@ SYM_FUNC_END(__host_hvc) b hyp_panic .L__hyp_sp_overflow\@: +#ifdef CONFIG_NVHE_EL2_DEBUG + /* Switch to the overflow stack */ + adr_this_cpu sp, hyp_overflow_stack + PAGE_SIZE, x0 +#else /* - * Reset SP to the top of the stack, to allow handling the hyp_panic. - * This corrupts the stack but is ok, since we won't be attempting - * any unwinding here. + * If !CONFIG_NVHE_EL2_DEBUG, reset SP to the top of the stack, to + * allow handling the hyp_panic. This corrupts the stack but is ok, + * since we won't be attempting any unwinding here. */ ldr_this_cpu x0, kvm_init_params + NVHE_INIT_STACK_HYP_VA, x1 mov sp, x0 +#endif bl hyp_panic_bad_stack ASM_BUG() diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 703a5d3f611b..efc20273a352 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -34,6 +34,11 @@ DEFINE_PER_CPU(struct kvm_host_data, kvm_host_data); DEFINE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); +#ifdef CONFIG_NVHE_EL2_DEBUG +DEFINE_PER_CPU(unsigned long [PAGE_SIZE/sizeof(long)], hyp_overflow_stack) + __aligned(16); +#endif + static void __activate_traps(struct kvm_vcpu *vcpu) { u64 val; From patchwork Mon Mar 14 20:01:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12780750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 87742C433EF for ; Mon, 14 Mar 2022 20:08:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=olUi4mcB+5J0RgDMlopxSd/dSNN9qYnNG/MOOe/dxaQ=; b=BhZX9KbC4NpC+o KRLFb27f19z1jqXYdG4W3iNSCJZd4cj/ShjZbQ2xEUiCOuERJj1xk09uEfzplttw04KjuqAGnFzy8 ZOdYxVpXSh2CYh3FXaYWoSxLjGfUSOawwq4mByNeH+4vXtQERKSCIckViysih385o3umcWsGqHjTN EFgmNzemz16w4hqWnIDvJvgyt5uvBIRYWsiS3RYIHrERVuK5eMCooPPzCV3UGalI5NWsAHpYQyM3q iDFBEb7gW8HZPuvO0YjRakYTkbLaTTnKzdfOwAiZqsJbns+wkRHLG8r64jPLziWCKMU3Acuo6q9Ls GR2q7zvr3VKUKxO1RvFQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqxt-006iob-IK; Mon, 14 Mar 2022 20:06:50 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqxa-006igv-67 for linux-arm-kernel@lists.infradead.org; Mon, 14 Mar 2022 20:06:32 +0000 Received: by mail-yb1-xb49.google.com with SMTP id j5-20020a056902020500b00628ab64be30so14698132ybs.16 for ; Mon, 14 Mar 2022 13:06:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=mXdQx6y1X7XoYA3xZnPfmktj3a5xOv4cTUjZQXQ7QEY=; b=s7J80w9hld1XIBJ/UTgaMW2q9VzHN0dtHZZoUV5BQg3xdDL41swhyeznJf61mvD3SF ZrK0xsEVoJgUGtVkhfji8EEUdZ/oNvwrR8pyxz3elqKDPzdcuL80AX32ddZypz/aDNNB 5R5Qn8T4cQHXWE0CNYq/qXlfBGgOP0a94ijkLqw8yEOZaJCczfmtg3RWs8pyH9L8x2iY St27wJNyS2L6XjJ3RplTUSKUpndN7NJcXRv9tMlcVpX9+QeNdFvb/gSXei8TGBmbBfp8 DQWufgNgBlkUaevjzl3oaKRCCw55yO8oQ4D+RrXwTztu0d19K1hwQymb1eRE2VQiOPYd lHQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=mXdQx6y1X7XoYA3xZnPfmktj3a5xOv4cTUjZQXQ7QEY=; b=UlJ8/61sh9Rj8nfLsH3+ydoS+0vbq6JvE+esZ9x/QnPqDtjojodaLioWBHtC6FD/tU YCOHU/DRf1rCvJtxfapcuYkHlfkxxHndDZ44/olkzr/sHn8bx1tdGqWwCRwoYyfxrbQK XU2ncVIxzm4iD7G6Jbm+Km7qeW9uZT+F1bxTXGyQJWlFlZD7YmZOVrhq1Ps4xN5qeW5M JuNVi0ixPfoLb8/zidUNzuU5MpBfQubqw0FraBnXlaZw9xm2SRzXMCt8Eh5qE8AYF74H lNzLw85klYCwHF9ARQ5XPzRv4yRP4BXapzI+a5rezuznwjEbCNMPKfbe9zwBpgDUjB57 EMnA== X-Gm-Message-State: AOAM531MCEtyKN0z3L6nIcWdlnpqSAj7Rj+QLx4bIehk1p+njcV6LKWe +xZ8IuF0mTtCBkQ601zk4+YKhQqU/5dI6DdapQ== X-Google-Smtp-Source: ABdhPJyaQtecJX0eaDnEgcsZnTJi0QMftBFt1Ol5UUnDoGK1zS1zHPVIt42N4yE42gDDvltZoIa1oYpDV5lslmYG9w== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:500a:9ac2:e73e:69e3]) (user=kaleshsingh job=sendgmr) by 2002:a81:6606:0:b0:2e5:7ede:abad with SMTP id a6-20020a816606000000b002e57edeabadmr1871170ywc.405.1647288388280; Mon, 14 Mar 2022 13:06:28 -0700 (PDT) Date: Mon, 14 Mar 2022 13:01:16 -0700 In-Reply-To: <20220314200148.2695206-1-kaleshsingh@google.com> Message-Id: <20220314200148.2695206-8-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220314200148.2695206-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.723.g4982287a31-goog Subject: [PATCH v6 7/8] KVM: arm64: Unwind and dump nVHE HYP stacktrace From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Mark Rutland , Mark Brown , Masami Hiramatsu , Peter Collingbourne , "Madhavan T. Venkataraman" , Stephen Boyd , Andrew Walbran , Andrew Scull , Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220314_130630_285985_610FB7F1 X-CRM114-Status: GOOD ( 29.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Unwind the stack in EL1, when CONFIG_NVHE_EL2_DEBUG is enabled. This is possible because CONFIG_NVHE_EL2_DEBUG disables the host stage-2 protection on hyp_panic(), allowing the host to access the hypervisor stack pages in EL1. A simple stack overflow test produces the following output: [ 580.376051][ T412] kvm: nVHE hyp panic at: ffffffc0116145c4! [ 580.378034][ T412] kvm [412]: nVHE HYP call trace: [ 580.378591][ T412] kvm [412]: [] [ 580.378993][ T412] kvm [412]: [] [ 580.379386][ T412] kvm [412]: [] // Non-terminating recursive call [ 580.379772][ T412] kvm [412]: [] [ 580.380158][ T412] kvm [412]: [] [ 580.380544][ T412] kvm [412]: [] [ 580.380928][ T412] kvm [412]: [] . . . Since nVHE hyp symbols are not included by kallsyms to avoid issues with aliasing, we fallback to the vmlinux addresses. Symbolizing the addresses is handled in the next patch in this series. Signed-off-by: Kalesh Singh Tested-by: Fuad Tabba Reviewed-by: Fuad Tabba --- Changes in v4: - Update commit text and struct kvm_nvhe_panic_info kernel-doc comment to clarify that CONFIG_NVHE_EL2_DEBUG only disables the host stage-2 protection on hyp_panic(), per Fuad - Update NVHE_EL2_DEBUG Kconfig description to clarify that the hypervisor stack trace is printed when hyp_panic() is called, per Fuad Changes in v3: - The nvhe hyp stack unwinder now makes use of the core logic from the regular kernel unwinder to avoid duplication, per Mark Changes in v2: - Add cpu_prepare_nvhe_panic_info() - Move updating the panic info to hyp_panic(), so that unwinding also works for conventional nVHE Hyp-mode. arch/arm64/include/asm/kvm_asm.h | 20 +++ arch/arm64/include/asm/stacktrace.h | 12 ++ arch/arm64/kernel/stacktrace.c | 210 +++++++++++++++++++++++++--- arch/arm64/kvm/Kconfig | 5 +- arch/arm64/kvm/arm.c | 2 +- arch/arm64/kvm/handle_exit.c | 3 + arch/arm64/kvm/hyp/nvhe/switch.c | 18 +++ 7 files changed, 244 insertions(+), 26 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 2e277f2ed671..4abcf93c6662 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -176,6 +176,26 @@ struct kvm_nvhe_init_params { unsigned long vtcr; }; +#ifdef CONFIG_NVHE_EL2_DEBUG +/** + * struct kvm_nvhe_panic_info - nVHE hypervisor panic info. + * @hyp_stack_base: hyp VA of the hyp_stack base. + * @hyp_overflow_stack_base: hyp VA of the hyp_overflow_stack base. + * @fp: hyp FP where the backtrace begins. + * @pc: hyp PC where the backtrace begins. + * + * Used by the host in EL1 to dump the nVHE hypervisor backtrace on + * hyp_panic. This is possible because CONFIG_NVHE_EL2_DEBUG disables + * the host stage 2 protection on hyp_panic(). See: __hyp_do_panic() + */ +struct kvm_nvhe_panic_info { + unsigned long hyp_stack_base; + unsigned long hyp_overflow_stack_base; + unsigned long fp; + unsigned long pc; +}; +#endif /* CONFIG_NVHE_EL2_DEBUG */ + /* Translate a kernel address @ptr into its equivalent linear mapping */ #define kvm_ksym_ref(ptr) \ ({ \ diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h index e77cdef9ca29..18611a51cf14 100644 --- a/arch/arm64/include/asm/stacktrace.h +++ b/arch/arm64/include/asm/stacktrace.h @@ -22,6 +22,10 @@ enum stack_type { STACK_TYPE_OVERFLOW, STACK_TYPE_SDEI_NORMAL, STACK_TYPE_SDEI_CRITICAL, +#ifdef CONFIG_NVHE_EL2_DEBUG + STACK_TYPE_KVM_NVHE_HYP, + STACK_TYPE_KVM_NVHE_OVERFLOW, +#endif /* CONFIG_NVHE_EL2_DEBUG */ __NR_STACK_TYPES }; @@ -147,4 +151,12 @@ static inline bool on_accessible_stack(const struct task_struct *tsk, return false; } +#ifdef CONFIG_NVHE_EL2_DEBUG +void kvm_nvhe_dump_backtrace(unsigned long hyp_offset); +#else +static inline void kvm_nvhe_dump_backtrace(unsigned long hyp_offset) +{ +} +#endif /* CONFIG_NVHE_EL2_DEBUG */ + #endif /* __ASM_STACKTRACE_H */ diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c index e4103e085681..6ec85cb69b1f 100644 --- a/arch/arm64/kernel/stacktrace.c +++ b/arch/arm64/kernel/stacktrace.c @@ -15,6 +15,8 @@ #include #include +#include +#include #include #include @@ -64,26 +66,15 @@ NOKPROBE_SYMBOL(start_backtrace); * records (e.g. a cycle), determined based on the location and fp value of A * and the location (but not the fp value) of B. */ -static int notrace unwind_frame(struct task_struct *tsk, - struct stackframe *frame) +static int notrace __unwind_frame(struct stackframe *frame, struct stack_info *info, + unsigned long (*translate_fp)(unsigned long, enum stack_type)) { unsigned long fp = frame->fp; - struct stack_info info; - - if (!tsk) - tsk = current; - - /* Final frame; nothing to unwind */ - if (fp == (unsigned long)task_pt_regs(tsk)->stackframe) - return -ENOENT; if (fp & 0x7) return -EINVAL; - if (!on_accessible_stack(tsk, fp, 16, &info)) - return -EINVAL; - - if (test_bit(info.type, frame->stacks_done)) + if (test_bit(info->type, frame->stacks_done)) return -EINVAL; /* @@ -94,28 +85,62 @@ static int notrace unwind_frame(struct task_struct *tsk, * * TASK -> IRQ -> OVERFLOW -> SDEI_NORMAL * TASK -> SDEI_NORMAL -> SDEI_CRITICAL -> OVERFLOW + * KVM_NVHE_HYP -> KVM_NVHE_OVERFLOW * * ... but the nesting itself is strict. Once we transition from one * stack to another, it's never valid to unwind back to that first * stack. */ - if (info.type == frame->prev_type) { + if (info->type == frame->prev_type) { if (fp <= frame->prev_fp) return -EINVAL; } else { set_bit(frame->prev_type, frame->stacks_done); } + /* Record fp as prev_fp before attempting to get the next fp */ + frame->prev_fp = fp; + + /* + * If fp is not from the current address space perform the + * necessary translation before dereferencing it to get next fp. + */ + if (translate_fp) + fp = translate_fp(fp, info->type); + if (!fp) + return -EINVAL; + /* * Record this frame record's values and location. The prev_fp and - * prev_type are only meaningful to the next unwind_frame() invocation. + * prev_type are only meaningful to the next __unwind_frame() invocation. */ frame->fp = READ_ONCE_NOCHECK(*(unsigned long *)(fp)); frame->pc = READ_ONCE_NOCHECK(*(unsigned long *)(fp + 8)); - frame->prev_fp = fp; - frame->prev_type = info.type; - frame->pc = ptrauth_strip_insn_pac(frame->pc); + frame->prev_type = info->type; + + return 0; +} + +static int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) +{ + unsigned long fp = frame->fp; + struct stack_info info; + int err; + + if (!tsk) + tsk = current; + + /* Final frame; nothing to unwind */ + if (fp == (unsigned long)task_pt_regs(tsk)->stackframe) + return -ENOENT; + + if (!on_accessible_stack(tsk, fp, 16, &info)) + return -EINVAL; + + err = __unwind_frame(frame, &info, NULL); + if (err) + return err; #ifdef CONFIG_FUNCTION_GRAPH_TRACER if (tsk->ret_stack && @@ -143,20 +168,27 @@ static int notrace unwind_frame(struct task_struct *tsk, } NOKPROBE_SYMBOL(unwind_frame); -static void notrace walk_stackframe(struct task_struct *tsk, - struct stackframe *frame, - bool (*fn)(void *, unsigned long), void *data) +static void notrace __walk_stackframe(struct task_struct *tsk, struct stackframe *frame, + bool (*fn)(void *, unsigned long), void *data, + int (*unwind_frame_fn)(struct task_struct *tsk, struct stackframe *frame)) { while (1) { int ret; if (!fn(data, frame->pc)) break; - ret = unwind_frame(tsk, frame); + ret = unwind_frame_fn(tsk, frame); if (ret < 0) break; } } + +static void notrace walk_stackframe(struct task_struct *tsk, + struct stackframe *frame, + bool (*fn)(void *, unsigned long), void *data) +{ + __walk_stackframe(tsk, frame, fn, data, unwind_frame); +} NOKPROBE_SYMBOL(walk_stackframe); static bool dump_backtrace_entry(void *arg, unsigned long where) @@ -210,3 +242,135 @@ noinline notrace void arch_stack_walk(stack_trace_consume_fn consume_entry, walk_stackframe(task, &frame, consume_entry, cookie); } + +#ifdef CONFIG_NVHE_EL2_DEBUG +DECLARE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); +DECLARE_KVM_NVHE_PER_CPU(unsigned long [PAGE_SIZE/sizeof(long)], hyp_overflow_stack); +DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_panic_info, kvm_panic_info); + +static inline bool kvm_nvhe_on_overflow_stack(unsigned long sp, unsigned long size, + struct stack_info *info) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr_nvhe_sym(kvm_panic_info); + unsigned long low = (unsigned long)panic_info->hyp_overflow_stack_base; + unsigned long high = low + PAGE_SIZE; + + return on_stack(sp, size, low, high, STACK_TYPE_KVM_NVHE_OVERFLOW, info); +} + +static inline bool kvm_nvhe_on_hyp_stack(unsigned long sp, unsigned long size, + struct stack_info *info) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr_nvhe_sym(kvm_panic_info); + unsigned long low = (unsigned long)panic_info->hyp_stack_base; + unsigned long high = low + PAGE_SIZE; + + return on_stack(sp, size, low, high, STACK_TYPE_KVM_NVHE_HYP, info); +} + +static inline bool kvm_nvhe_on_accessible_stack(unsigned long sp, unsigned long size, + struct stack_info *info) +{ + if (info) + info->type = STACK_TYPE_UNKNOWN; + + if (kvm_nvhe_on_hyp_stack(sp, size, info)) + return true; + if (kvm_nvhe_on_overflow_stack(sp, size, info)) + return true; + + return false; +} + +static unsigned long kvm_nvhe_hyp_stack_kern_va(unsigned long addr) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr_nvhe_sym(kvm_panic_info); + unsigned long hyp_base, kern_base, hyp_offset; + + hyp_base = (unsigned long)panic_info->hyp_stack_base; + hyp_offset = addr - hyp_base; + + kern_base = (unsigned long)*this_cpu_ptr(&kvm_arm_hyp_stack_page); + + return kern_base + hyp_offset; +} + +static unsigned long kvm_nvhe_overflow_stack_kern_va(unsigned long addr) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr_nvhe_sym(kvm_panic_info); + unsigned long hyp_base, kern_base, hyp_offset; + + hyp_base = (unsigned long)panic_info->hyp_overflow_stack_base; + hyp_offset = addr - hyp_base; + + kern_base = (unsigned long)this_cpu_ptr_nvhe_sym(hyp_overflow_stack); + + return kern_base + hyp_offset; +} + +/* + * Convert KVM nVHE hypervisor stack VA to a kernel VA. + * + * The nVHE hypervisor stack is mapped in the flexible 'private' VA range, to allow + * for guard pages below the stack. Consequently, the fixed offset address + * translation macros won't work here. + * + * The kernel VA is calculated as an offset from the kernel VA of the hypervisor + * stack base. See: kvm_nvhe_hyp_stack_kern_va(), kvm_nvhe_overflow_stack_kern_va() + */ +static unsigned long kvm_nvhe_stack_kern_va(unsigned long addr, + enum stack_type type) +{ + switch (type) { + case STACK_TYPE_KVM_NVHE_HYP: + return kvm_nvhe_hyp_stack_kern_va(addr); + case STACK_TYPE_KVM_NVHE_OVERFLOW: + return kvm_nvhe_overflow_stack_kern_va(addr); + default: + return 0UL; + } +} + +static int notrace kvm_nvhe_unwind_frame(struct task_struct *tsk, + struct stackframe *frame) +{ + struct stack_info info; + + if (!kvm_nvhe_on_accessible_stack(frame->fp, 16, &info)) + return -EINVAL; + + return __unwind_frame(frame, &info, kvm_nvhe_stack_kern_va); +} + +static bool kvm_nvhe_dump_backtrace_entry(void *arg, unsigned long where) +{ + unsigned long va_mask = GENMASK_ULL(vabits_actual - 1, 0); + unsigned long hyp_offset = (unsigned long)arg; + + where &= va_mask; /* Mask tags */ + where += hyp_offset; /* Convert to kern addr */ + + kvm_err("[<%016lx>] %pB\n", where, (void *)where); + + return true; +} + +static void notrace kvm_nvhe_walk_stackframe(struct task_struct *tsk, + struct stackframe *frame, + bool (*fn)(void *, unsigned long), void *data) +{ + __walk_stackframe(tsk, frame, fn, data, kvm_nvhe_unwind_frame); +} + +void kvm_nvhe_dump_backtrace(unsigned long hyp_offset) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr_nvhe_sym(kvm_panic_info); + struct stackframe frame; + + start_backtrace(&frame, panic_info->fp, panic_info->pc); + pr_err("nVHE HYP call trace:\n"); + kvm_nvhe_walk_stackframe(NULL, &frame, kvm_nvhe_dump_backtrace_entry, + (void *)hyp_offset); + pr_err("---- end of nVHE HYP call trace ----\n"); +} +#endif /* CONFIG_NVHE_EL2_DEBUG */ diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 8a5fbbf084df..a7be4ef35fbf 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -51,8 +51,9 @@ config NVHE_EL2_DEBUG depends on KVM help Say Y here to enable the debug mode for the non-VHE KVM EL2 object. - Failure reports will BUG() in the hypervisor. This is intended for - local EL2 hypervisor development. + Failure reports will BUG() in the hypervisor; and calls to hyp_panic() + will result in printing the hypervisor call stack. + This is intended for local EL2 hypervisor development. If unsure, say N. diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 72be7e695d8d..c7216ce1d55c 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -49,7 +49,7 @@ DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized); DECLARE_KVM_HYP_PER_CPU(unsigned long, kvm_hyp_vector); -static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); +DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); unsigned long kvm_arm_hyp_percpu_base[NR_CPUS]; DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index e3140abd2e2e..ff69dff33700 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -326,6 +327,8 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, kvm_err("nVHE hyp panic at: %016llx!\n", elr_virt + hyp_offset); } + kvm_nvhe_dump_backtrace(hyp_offset); + /* * Hyp has panicked and we're going to handle that by panicking the * kernel. The kernel offset will be revealed in the panic so we're diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index efc20273a352..b8ecffc47424 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -37,6 +37,22 @@ DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); #ifdef CONFIG_NVHE_EL2_DEBUG DEFINE_PER_CPU(unsigned long [PAGE_SIZE/sizeof(long)], hyp_overflow_stack) __aligned(16); +DEFINE_PER_CPU(struct kvm_nvhe_panic_info, kvm_panic_info); + +static inline void cpu_prepare_nvhe_panic_info(void) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr(&kvm_panic_info); + struct kvm_nvhe_init_params *params = this_cpu_ptr(&kvm_init_params); + + panic_info->hyp_stack_base = (unsigned long)(params->stack_hyp_va - PAGE_SIZE); + panic_info->hyp_overflow_stack_base = (unsigned long)this_cpu_ptr(hyp_overflow_stack); + panic_info->fp = (unsigned long)__builtin_frame_address(0); + panic_info->pc = _THIS_IP_; +} + #else +static inline void cpu_prepare_nvhe_panic_info(void) +{ +} #endif static void __activate_traps(struct kvm_vcpu *vcpu) @@ -360,6 +376,8 @@ asmlinkage void __noreturn hyp_panic(void) struct kvm_cpu_context *host_ctxt; struct kvm_vcpu *vcpu; + cpu_prepare_nvhe_panic_info(); + host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; vcpu = host_ctxt->__hyp_running_vcpu; From patchwork Mon Mar 14 20:01:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12780751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F405AC433F5 for ; Mon, 14 Mar 2022 20:09:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QWeE7ealr7tsIkBE3buStFMYzgLHtUCW1cbSubpqixM=; b=XJ9CBYtPnM74LF srZ7Nv7VPPMEYtguhjyJslhMXfKIAWmdg+5xomOc5Rbjia5PQ9h3F2D7U9gxyRarWULOrODTefIqL HC9yGtRY1os14n2+BWZ0KIxr13et9YbXVCkHKZt55meymMZw6CSIfCM2UldNw2dnBIROp+8C2kZjh Luk8sdSR9OaXz7VMARrMShcE7B/6RSIn2YAoGts9K1So9PE0f+cnyyUi/xEt9MM0zAiE55tuwaQo+ eTvK6nTIyoMzhw+A0neo+z4xRpKm9uE1sb8yZn4kFP/N03PQkEP2Eh1QnlhYDZN2PBm0oba7VYL9o omf8/LdBYXB0aQnS9M7Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqyc-006j8I-Jn; Mon, 14 Mar 2022 20:07:35 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTqy9-006iwq-Op for linux-arm-kernel@lists.infradead.org; Mon, 14 Mar 2022 20:07:11 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-2d6914a097cso140729457b3.10 for ; Mon, 14 Mar 2022 13:07:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=7WPeIutFQ8MpWvUoMJwXeeMUz9b7tskLEPBl/3D4C6U=; b=cxwJyz+g5A+OyqgvqDWvQe+JtpGKB6vHjvy5/DobHD8VmnHbQT7A8rTMSYElYviMzV W2Ps0EwVwxAvO4EXrcUP+dDEnYoUg3I3HrOiQ1Fegi6FffpY/SExycCPO8B9as25wbEs mwIMQpE2KLgaM4IFNmfC7KuV140gff8y5mz7qMpS/pLSxfK5wmgHsyGsDCF+j20zyA/M YFwqfQz2bm5J7nmGQAcJNNF9upClL6AnaCzH+kLXvABXiE8HErJNMACfTubNk6o3WD6n kf0Pwd1Kc522idPnHKae37N1MgRr0LwMpNAJ0ASvNDXcruiU4hGYsN5Aj6i1mx1VtsKw yZ7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=7WPeIutFQ8MpWvUoMJwXeeMUz9b7tskLEPBl/3D4C6U=; b=K1Y0bYsKBY+iNOP+CSUffLb4C3mP4sN/12gRIX2ZlFw8wjk6jFrM/gsJHMD9DyFfKE 85coR2Zt2afiCZH6PUSjbEBgC4nqVLeShl9txlABm0zqobkLgRDvu02CTwg2KzC0n5mj hq0SSEBTR4Q8m7lVrm0zcIOQJ9EVYeKH0NK0qpniup3saw9AnPBf6SfZJN4en4z6Hgny 8hzFZkyrGf+mCJKNxNI2fpnFUx9PzDQCWdD4YfwftdCAA/qmIDD2n/ZrEvSTQpC/ZYnk E8iY9DBhUVCfOG6QMrs30rkoWdf0QvfBC5nnc8Qxsj0kXumoLDNy5wwKorIjmi6hDsij R39A== X-Gm-Message-State: AOAM532di3Jngtg4WDfCnxOwF0riegDImA+1O/TxKaonLYJNIg4CJ4gJ GJhLLsnOsSDkfsokHpW5R+6n88l9f9K7/kC/Hg== X-Google-Smtp-Source: ABdhPJwqmuZWietxle5e91HveZ+teLadkWuBWcTKe8lzVoiXl7o144fZFzG4YZ53zF6KF//5yrRIAJR7dEQ1e0ntag== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:500a:9ac2:e73e:69e3]) (user=kaleshsingh job=sendgmr) by 2002:a25:2514:0:b0:62e:a211:a940 with SMTP id l20-20020a252514000000b0062ea211a940mr17232935ybl.122.1647288424162; Mon, 14 Mar 2022 13:07:04 -0700 (PDT) Date: Mon, 14 Mar 2022 13:01:17 -0700 In-Reply-To: <20220314200148.2695206-1-kaleshsingh@google.com> Message-Id: <20220314200148.2695206-9-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220314200148.2695206-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.723.g4982287a31-goog Subject: [PATCH v6 8/8] KVM: arm64: Symbolize the nVHE HYP backtrace From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Mark Rutland , Mark Brown , Masami Hiramatsu , Peter Collingbourne , "Madhavan T. Venkataraman" , Peter Zijlstra , Andrew Scull , Andrew Jones , Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220314_130705_864820_0CF3B101 X-CRM114-Status: GOOD ( 15.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Reintroduce the __kvm_nvhe_ symbols in kallsyms, ignoring the local symbols in this namespace. The local symbols are not informative and can cause aliasing issues when symbolizing the addresses. With the necessary symbols now in kallsyms we can symbolize nVHE stacktrace addresses using the %pB print format specifier. Example call trace: [ 98.916444][ T426] kvm [426]: nVHE hyp panic at: [] __kvm_nvhe_overflow_stack+0x8/0x34! [ 98.918360][ T426] nVHE HYP call trace: [ 98.918692][ T426] kvm [426]: [] __kvm_nvhe_cpu_prepare_nvhe_panic_info+0x4c/0x68 [ 98.919545][ T426] kvm [426]: [] __kvm_nvhe_hyp_panic+0x2c/0xe8 [ 98.920107][ T426] kvm [426]: [] __kvm_nvhe_hyp_panic_bad_stack+0x10/0x10 [ 98.920665][ T426] kvm [426]: [] __kvm_nvhe___kvm_hyp_host_vector+0x24c/0x794 [ 98.921292][ T426] kvm [426]: [] __kvm_nvhe_overflow_stack+0x24/0x34 . . . [ 98.973382][ T426] kvm [426]: [] __kvm_nvhe_overflow_stack+0x24/0x34 [ 98.973816][ T426] kvm [426]: [] __kvm_nvhe___kvm_vcpu_run+0x38/0x438 [ 98.974255][ T426] kvm [426]: [] __kvm_nvhe_handle___kvm_vcpu_run+0x1c4/0x364 [ 98.974719][ T426] kvm [426]: [] __kvm_nvhe_handle_trap+0xa8/0x130 [ 98.975152][ T426] kvm [426]: [] __kvm_nvhe___host_exit+0x64/0x64 [ 98.975588][ T426] ---- end of nVHE HYP call trace ---- Signed-off-by: Kalesh Singh Tested-by: Fuad Tabba Reviewed-by: Fuad Tabba --- Changes in v2: - Fix printk warnings - %p expects (void *) arch/arm64/kvm/handle_exit.c | 13 +++++-------- scripts/kallsyms.c | 2 +- 2 files changed, 6 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index ff69dff33700..3a5c32017c6b 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -296,13 +296,8 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_in_kimg = __phys_to_kimg(elr_phys); u64 hyp_offset = elr_in_kimg - kaslr_offset() - elr_virt; u64 mode = spsr & PSR_MODE_MASK; + u64 panic_addr = elr_virt + hyp_offset; - /* - * The nVHE hyp symbols are not included by kallsyms to avoid issues - * with aliasing. That means that the symbols cannot be printed with the - * "%pS" format specifier, so fall back to the vmlinux address if - * there's no better option. - */ if (mode != PSR_MODE_EL2t && mode != PSR_MODE_EL2h) { kvm_err("Invalid host exception to nVHE hyp!\n"); } else if (ESR_ELx_EC(esr) == ESR_ELx_EC_BRK64 && @@ -322,9 +317,11 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, if (file) kvm_err("nVHE hyp BUG at: %s:%u!\n", file, line); else - kvm_err("nVHE hyp BUG at: %016llx!\n", elr_virt + hyp_offset); + kvm_err("nVHE hyp BUG at: [<%016llx>] %pB!\n", panic_addr, + (void *)panic_addr); } else { - kvm_err("nVHE hyp panic at: %016llx!\n", elr_virt + hyp_offset); + kvm_err("nVHE hyp panic at: [<%016llx>] %pB!\n", panic_addr, + (void *)panic_addr); } kvm_nvhe_dump_backtrace(hyp_offset); diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c index 54ad86d13784..19aba43d9da4 100644 --- a/scripts/kallsyms.c +++ b/scripts/kallsyms.c @@ -111,7 +111,7 @@ static bool is_ignored_symbol(const char *name, char type) ".LASANPC", /* s390 kasan local symbols */ "__crc_", /* modversions */ "__efistub_", /* arm64 EFI stub namespace */ - "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */ + "__kvm_nvhe_$", /* arm64 local symbols in non-VHE KVM namespace */ "__AArch64ADRPThunk_", /* arm64 lld */ "__ARMV5PILongThunk_", /* arm lld */ "__ARMV7PILongThunk_",