From patchwork Tue Feb 22 16:51:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12755684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 52E32C433EF for ; Tue, 22 Feb 2022 17:02:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ytKtwKYZ3qoKCMxMTVvZqgE5BU+y0XJ5UszCtlTopcg=; b=bFl6MaPcG7WZ1u g0/6zufD4ADhD9tFxPDlxUfJOMVXA3J5Zrjp9H7Mpz4dAm1AnLNi3ZJPAYeZhL6ISkwm+bRmvOi3i S3r7G89cEjiZbSdzjAdi4d1M5Z8wTHa3WvOTwf1828ryCpEWstLOh4nO7i696qjU5A4lCnwhhd+eC OMriERyCEpJ1jYr4/oT/wcNXH4f5ztKFeeYzaRBbS3rT3IA2LO0FrE9OoWjyLUjW6RwMXn+x8Sj8J OpMqKT7jzMGl+DFbcxSfYieV6GYiTFHDZA9WJWwNHnwIDzf+a7j1OB+qhgiUe+v3OvtcyCHuETMcD IYzRr7e7sp1F9FFj+Utw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYX1-00Ap5F-Mq; Tue, 22 Feb 2022 17:00:56 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYQC-00AmSL-Kr for linux-arm-kernel@lists.infradead.org; Tue, 22 Feb 2022 16:53:59 +0000 Received: by mail-yb1-xb49.google.com with SMTP id k7-20020a255607000000b00621afc793b8so24920841ybb.1 for ; Tue, 22 Feb 2022 08:53:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=HjTQZhkAd40lRuvS6cRV70Tpnwq9dG8Hrk7s02bufrw=; b=l9gvPlFOgpm93Vwqfc4x5Vuvn4H98TzVtnPztFKq1RVJHuCOwjRWnhPGwbxmv2gBju qni22+lZAoWeuHyAnrHLkUxmGVg1MGvOPtvysJUTpVnxGE9XNCgdGG3ihOcgDh4Ud0VO t81B3SWw8tC8AP32YUnfVl7SNtvpxpe1ER/dZExOrnQ3bwmGSIvoILXJpsKsbYOeEn5X eNRw3J9xbd+PwHflrNv9W7603a/Ig+5VqL31ZqaFq/h7ygZQBquGNSxpt5WNmcJEKeuh 799vMF4hFNUjwmMyaOlRkDXrCuhduGcEePOXQj+fAOq4j8VbFHa6wTIBzVfUzsaX0ujb tebQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=HjTQZhkAd40lRuvS6cRV70Tpnwq9dG8Hrk7s02bufrw=; b=oQXe9oF80bGDmqLkabhVz5Uiwon7RtasnDPazZK8NI2jZ5HSqX5shLRC7lcWRn3lOg ZRR5y7o5nSsHHhTcsuCWnpWnVknT+3fm9YMbSPF5261nVSBqvpG0NbVEUaujv//ihFIQ 0de+E/E+Tyd+14viOiq0bSPYjuOg1LaAUAb66uYaItQ90RID7dlr7oY0XwxPh3ujwqwT Bv8qEmnu2romgSG2f2mV91T1B/3zOZJp80h+5WgOE4c9kwYbGyEuuxW7Mv/rteRBFsYV EeUauGX1PyVeyJ8IAzT/oFI6s5Hp642CHvljHrZx93040hCjoppT1C1N6bFQbZ4wZzuY uaAQ== X-Gm-Message-State: AOAM533ul/MrwS8SfH8vYmzZ8kOc47psLJdR1nBWWkHST8nHmC1R5C6j TqWBJfaqbNNRHv9hYp1n8HdNWsbj+G4ZzTMjJw== X-Google-Smtp-Source: ABdhPJxqQdinLFWWaI+jsvW0KvgGVTkFzIldtere0/r0cpO5BgzuX/8Yyc3hdk7P6zZ0tb1mTeOyerwEEGUUJI16nw== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:5db7:1235:b3dd:cfcb]) (user=kaleshsingh job=sendgmr) by 2002:a25:4dd7:0:b0:621:a740:3988 with SMTP id a206-20020a254dd7000000b00621a7403988mr24600128ybb.58.1645548831008; Tue, 22 Feb 2022 08:53:51 -0800 (PST) Date: Tue, 22 Feb 2022 08:51:02 -0800 In-Reply-To: <20220222165212.2005066-1-kaleshsingh@google.com> Message-Id: <20220222165212.2005066-2-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220222165212.2005066-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v2 1/9] KVM: arm64: Introduce hyp_alloc_private_va_range() From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , Catalin Marinas , James Morse , Alexandru Elisei , Suzuki K Poulose , Ard Biesheuvel , Mark Rutland , Pasha Tatashin , Joey Gouly , Peter Collingbourne , Andrew Walbran , Andrew Scull , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.cs.columbia.edu X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220222_085352_740903_D1AE2381 X-CRM114-Status: GOOD ( 17.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org hyp_alloc_private_va_range() can be used to reserve private VA ranges in the nVHE hypervisor. Also update __create_hyp_private_mapping() to allow specifying an alignment for the private VA mapping. These will be used to implement stack guard pages for KVM nVHE hypervisor (nVHE Hyp mode / not pKVM), in a subsequent patch in the series. Signed-off-by: Kalesh Singh --- arch/arm64/include/asm/kvm_mmu.h | 4 +++ arch/arm64/kvm/mmu.c | 61 +++++++++++++++++++++----------- 2 files changed, 44 insertions(+), 21 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 81839e9a8a24..0b0c71302b92 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -153,6 +153,10 @@ static __always_inline unsigned long __kern_hyp_va(unsigned long v) int kvm_share_hyp(void *from, void *to); void kvm_unshare_hyp(void *from, void *to); int create_hyp_mappings(void *from, void *to, enum kvm_pgtable_prot prot); +unsigned long hyp_alloc_private_va_range(size_t size, size_t align); +int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, + size_t align, unsigned long *haddr, + enum kvm_pgtable_prot prot); int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size, void __iomem **kaddr, void __iomem **haddr); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index bc2aba953299..e5abcce44ad0 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -457,22 +457,16 @@ int create_hyp_mappings(void *from, void *to, enum kvm_pgtable_prot prot) return 0; } -static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, - unsigned long *haddr, - enum kvm_pgtable_prot prot) + +/* + * Allocates a private VA range below io_map_base. + * + * @size: The size of the VA range to reserve. + * @align: The required alignment for the allocation. + */ +unsigned long hyp_alloc_private_va_range(size_t size, size_t align) { unsigned long base; - int ret = 0; - - if (!kvm_host_owns_hyp_mappings()) { - base = kvm_call_hyp_nvhe(__pkvm_create_private_mapping, - phys_addr, size, prot); - if (IS_ERR_OR_NULL((void *)base)) - return PTR_ERR((void *)base); - *haddr = base; - - return 0; - } mutex_lock(&kvm_hyp_pgd_mutex); @@ -484,8 +478,8 @@ static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, * * The allocated size is always a multiple of PAGE_SIZE. */ - size = PAGE_ALIGN(size + offset_in_page(phys_addr)); - base = io_map_base - size; + base = io_map_base - PAGE_ALIGN(size); + base = ALIGN_DOWN(base, align); /* * Verify that BIT(VA_BITS - 1) hasn't been flipped by @@ -493,20 +487,45 @@ static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, * overflowed the idmap/IO address range. */ if ((base ^ io_map_base) & BIT(VA_BITS - 1)) - ret = -ENOMEM; + base = (unsigned long)ERR_PTR(-ENOMEM); else io_map_base = base; mutex_unlock(&kvm_hyp_pgd_mutex); + return base; +} + +int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, + size_t align, unsigned long *haddr, + enum kvm_pgtable_prot prot) +{ + unsigned long addr; + int ret = 0; + + if (!kvm_host_owns_hyp_mappings()) { + addr = kvm_call_hyp_nvhe(__pkvm_create_private_mapping, + phys_addr, size, prot); + if (IS_ERR_OR_NULL((void *)addr)) + return PTR_ERR((void *)addr); + *haddr = addr; + + return 0; + } + + size += offset_in_page(phys_addr); + addr = hyp_alloc_private_va_range(size, align); + if (IS_ERR_OR_NULL((void *)addr)) + return PTR_ERR((void *)addr); + if (ret) goto out; - ret = __create_hyp_mappings(base, size, phys_addr, prot); + ret = __create_hyp_mappings(addr, size, phys_addr, prot); if (ret) goto out; - *haddr = base + offset_in_page(phys_addr); + *haddr = addr + offset_in_page(phys_addr); out: return ret; } @@ -537,7 +556,7 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size, return 0; } - ret = __create_hyp_private_mapping(phys_addr, size, + ret = __create_hyp_private_mapping(phys_addr, size, PAGE_SIZE, &addr, PAGE_HYP_DEVICE); if (ret) { iounmap(*kaddr); @@ -564,7 +583,7 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, BUG_ON(is_kernel_in_hyp_mode()); - ret = __create_hyp_private_mapping(phys_addr, size, + ret = __create_hyp_private_mapping(phys_addr, size, PAGE_SIZE, &addr, PAGE_HYP_EXEC); if (ret) { *haddr = NULL; From patchwork Tue Feb 22 16:51:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12755702 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7EB02C433EF for ; Tue, 22 Feb 2022 17:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=82fZC9WKBFNpA59Jh/wzUUYdIDaq+I/VljkBDQwBcJM=; b=J9zIC70+Q4QOIY +EC/MuA4KTcQC8Am9ziGQdjnlSU0s3M0hIgbNtj+nbsiVg1RY04A9llQVKqzEqUsOa8GNd5hpLYNa a2QAREnz3Ja/wfGBDhWPeG3LNn1WwHlD8OTQkxKVUdZNkpdqHaOGgdt38dUSoOsnyO87FzKG//2ml 5jLvm89UFH7ba9OZM2a+IExiB69eetWMtvt6FAwa1wucOkJmUx+LcK4WU3D4GjCMs9OrTku44KLq4 5vH6vJSnEy6ERjhNrOmt5Bh6t3T/FWJpf7GuLqsk3tI9fBch66t44C2+7QpqHFoVWH8t+hc8ha31C 5ibvMZ7goHFAXnH6yHtA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYYN-00Apgn-BG; Tue, 22 Feb 2022 17:02:22 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYRG-00AmsM-Ar for linux-arm-kernel@lists.infradead.org; Tue, 22 Feb 2022 16:55:00 +0000 Received: by mail-yb1-xb49.google.com with SMTP id b11-20020a5b008b000000b00624ea481d55so687402ybp.19 for ; Tue, 22 Feb 2022 08:54:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=ZlyAAqqDhAE4WxbyZ31+jkqH/YpnLCKmJnh7POCxqTo=; b=TIooTTyH5x1F5nebrNIhkxrh2HuWrtPHN3fTIyBzs1a6KH6OGiKUp1PWOlO6NXsbQT k7h9zWn15cvo5sZhfXstOZ7RpAD+Gat9qLxoStKOCv0K160fImoXF6iu00kBuJnVyT6n YeThCRS0IPdnxIFck9JWsb0bj7sYMYA9LYn4MWVPyD2giNP+0cLZ8uSvz0IwXqXOXaOu G/dlwHj4gGGF4Y6nSgBvhRU/4Yq7YkrEaClamVuzK+tUFIXdOsSYoS/NWO0g8LCGrMsX It+EBn3ntQqg8tQNu7EC0xgAKjXEbbsSJs8P+w445Q4D6gLEB+GMhwcdWzAmnJsKbA4J Pgrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=ZlyAAqqDhAE4WxbyZ31+jkqH/YpnLCKmJnh7POCxqTo=; b=L33b6iEO8YBZ+Bz0sMapWy3ltl17ZPCQpCnKYc/Np9IlXamos/0u1p2eyvctp5F5Us MxhnrcyUnT/irHW4F7wmqplvYCYrcf08tIoYRaZNhjAvcPf6O5XWYweQV6adswMrcHHe 7McrElJSemYhM7/jlcnVKnSd0aZJrxPiiwvJPHTsPnObq8wFQ7YWxyIJoFKExm+yO2cu z3NlUwoB23VbMOXpPHzdIxk2ggfGTbgGUsAP1T1FTMaH56boHa+6at/qGH99aZKj3ip7 pXMqOAf5z00fP9NxOF8G+o6jIPF4TW1dAOuk5M6u9GvAzXXnXmNFbjeE6mpGFtbTVgUk XORQ== X-Gm-Message-State: AOAM531AqYmsfAcolcjrnPrkdhWfzLJ0uRKx3588RTG8OxZkhuTm9wcr t0pHN/CkQnD/R0TBIRrE+2JU4gONDhjZ7L9GBg== X-Google-Smtp-Source: ABdhPJwTHC6zcsc1CP1g9QcEYL9chbpjl/nOUt74qFxAcpxZyUBNLrnrRhwJaWgirDTS7AoSkxlDAybxGjmkVJWUJw== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:5db7:1235:b3dd:cfcb]) (user=kaleshsingh job=sendgmr) by 2002:a81:911:0:b0:2d2:c5c6:b4a1 with SMTP id 17-20020a810911000000b002d2c5c6b4a1mr24349571ywj.48.1645548896812; Tue, 22 Feb 2022 08:54:56 -0800 (PST) Date: Tue, 22 Feb 2022 08:51:03 -0800 In-Reply-To: <20220222165212.2005066-1-kaleshsingh@google.com> Message-Id: <20220222165212.2005066-3-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220222165212.2005066-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v2 2/9] KVM: arm64: Introduce pkvm_alloc_private_va_range() From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , Catalin Marinas , James Morse , Alexandru Elisei , Suzuki K Poulose , Ard Biesheuvel , Mark Rutland , Pasha Tatashin , Joey Gouly , Peter Collingbourne , Andrew Scull , Zenghui Yu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.cs.columbia.edu X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220222_085458_468091_93869347 X-CRM114-Status: GOOD ( 17.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org pkvm_hyp_alloc_private_va_range() can be used to reserve private VA ranges in the pKVM nVHE hypervisor (). Also update __pkvm_create_private_mapping() to allow specifying an alignment for the private VA mapping. These will be used to implement stack guard pages for pKVM nVHE hypervisor (in a subsequent patch in the series). Credits to Quentin Perret for the idea of moving private VA allocation out of __pkvm_create_private_mapping() Signed-off-by: Kalesh Singh --- Changes in v2: - Allow specifying an alignment for the private VA allocations, per Marc arch/arm64/kvm/hyp/include/nvhe/mm.h | 3 +- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 5 +-- arch/arm64/kvm/hyp/nvhe/mm.c | 49 +++++++++++++++++++--------- arch/arm64/kvm/mmu.c | 2 +- 4 files changed, 39 insertions(+), 20 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h index 2d08510c6cc1..05d06ad00347 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -20,7 +20,8 @@ int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot); int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot); int pkvm_create_mappings_locked(void *from, void *to, enum kvm_pgtable_prot prot); unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, - enum kvm_pgtable_prot prot); + size_t align, enum kvm_pgtable_prot prot); +unsigned long pkvm_alloc_private_va_range(size_t size, size_t align); static inline void hyp_vmemmap_range(phys_addr_t phys, unsigned long size, unsigned long *start, unsigned long *end) diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 5e2197db0d32..96b2312a0f1d 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -158,9 +158,10 @@ static void handle___pkvm_create_private_mapping(struct kvm_cpu_context *host_ct { DECLARE_REG(phys_addr_t, phys, host_ctxt, 1); DECLARE_REG(size_t, size, host_ctxt, 2); - DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 3); + DECLARE_REG(size_t, align, host_ctxt, 3); + DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 4); - cpu_reg(host_ctxt, 1) = __pkvm_create_private_mapping(phys, size, prot); + cpu_reg(host_ctxt, 1) = __pkvm_create_private_mapping(phys, size, align, prot); } static void handle___pkvm_prot_finalize(struct kvm_cpu_context *host_ctxt) diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c index 526a7d6fa86f..298fbbe4651d 100644 --- a/arch/arm64/kvm/hyp/nvhe/mm.c +++ b/arch/arm64/kvm/hyp/nvhe/mm.c @@ -37,26 +37,46 @@ static int __pkvm_create_mappings(unsigned long start, unsigned long size, return err; } -unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, - enum kvm_pgtable_prot prot) +/* + * Allocates a private VA range above __io_map_base. + * + * @size: The size of the VA range to reserve. + * @align: The required alignment for the allocation. + */ +unsigned long pkvm_alloc_private_va_range(size_t size, size_t align) { - unsigned long addr; - int err; + unsigned long base, addr; hyp_spin_lock(&pkvm_pgd_lock); - size = PAGE_ALIGN(size + offset_in_page(phys)); - addr = __io_map_base; - __io_map_base += size; + addr = ALIGN(__io_map_base, align); + + /* The allocated size is always a multiple of PAGE_SIZE */ + base = addr + PAGE_ALIGN(size); /* Are we overflowing on the vmemmap ? */ - if (__io_map_base > __hyp_vmemmap) { - __io_map_base -= size; + if (base > __hyp_vmemmap) addr = (unsigned long)ERR_PTR(-ENOMEM); + else + __io_map_base = base; + + hyp_spin_unlock(&pkvm_pgd_lock); + + return addr; +} + +unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, + size_t align, enum kvm_pgtable_prot prot) +{ + unsigned long addr; + int err; + + size += offset_in_page(phys); + addr = pkvm_alloc_private_va_range(size, align); + if (IS_ERR((void *)addr)) goto out; - } - err = kvm_pgtable_hyp_map(&pkvm_pgtable, addr, size, phys, prot); + err = __pkvm_create_mappings(addr, size, phys, prot); if (err) { addr = (unsigned long)ERR_PTR(err); goto out; @@ -64,8 +84,6 @@ unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, addr = addr + offset_in_page(phys); out: - hyp_spin_unlock(&pkvm_pgd_lock); - return addr; } @@ -152,9 +170,8 @@ int hyp_map_vectors(void) return 0; phys = __hyp_pa(__bp_harden_hyp_vecs); - bp_base = (void *)__pkvm_create_private_mapping(phys, - __BP_HARDEN_HYP_VECS_SZ, - PAGE_HYP_EXEC); + bp_base = (void *)__pkvm_create_private_mapping(phys, __BP_HARDEN_HYP_VECS_SZ, + PAGE_SIZE, PAGE_HYP_EXEC); if (IS_ERR_OR_NULL(bp_base)) return PTR_ERR(bp_base); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index e5abcce44ad0..18a711d6a52f 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -505,7 +505,7 @@ int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, if (!kvm_host_owns_hyp_mappings()) { addr = kvm_call_hyp_nvhe(__pkvm_create_private_mapping, - phys_addr, size, prot); + phys_addr, size, align, prot); if (IS_ERR_OR_NULL((void *)addr)) return PTR_ERR((void *)addr); *haddr = addr; From patchwork Tue Feb 22 16:51:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12755703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3B1A2C433F5 for ; Tue, 22 Feb 2022 17:05:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8wMfjECq6K1uWicH4xZ4zrsIzyg8uFgaoSWXMw7vR60=; b=nXcLowr/zIsvt6 KXpvGUVK9rYlAGWDzFMiZV0/Gbfv9QO0lffQCKu3cwJYccmlQPgeBXWXRAGzhzeZb9pTCBmlqwS6d 57tfDXGha+qjKLvMqgpeNqA3JsiRTKJmTB9hzEiP4/GoN3/Be/XZKKde4Dc4NMnhNPAnvIYqj52Qg x8NmeCLxTSsYIAjHPtBh29xcWllawFNiukELgWkCxyxjyAEY07q46NJcAV0lMkW68zHeMnZBr0t/I KLeAC3c7/JBLQROt/XnztovrAl5qzwEGEpsMOx7X107fJYb34sbjvrpRyrykuQCCNYSgYSRBncBcB ND4rTKKOwkNKnTQxmXjQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYa9-00AqJK-7a; Tue, 22 Feb 2022 17:04:10 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYSM-00AnI2-9R for linux-arm-kernel@lists.infradead.org; Tue, 22 Feb 2022 16:56:07 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-2d6914a097cso145615577b3.10 for ; Tue, 22 Feb 2022 08:56:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=R7NUgyhmz2Vzm1IbfrrDYD1qNpQGYjcjDbFVXUaMLH8=; b=KI3O3p6WIXcE6n5c1xlRqNCzstNa7C+hluEFjVsr0MOzveDFLD2VUOGSWGeKG0RHJy T/yqV6+ri+p/3rX+b1pYMjSNo3Ovj9yyZbKA1607HqCA3kbya8vyKQ2GDgWlYG9l1znh rWdySSjzOH0K6sfRsXIWIRMQFAHMD5jl44ausSef3FnfQgFOmyhWcQwfQoMj6etEc/LI hJHRQ66bszx5zigZGXaE9ntUDiXKmNtjFdm5S5h0xTnO2PAF2yNKcyiTDlX9Dq5Ziu1q t3RdfNDzIKa7Bq1TCGBRbeYsC4mW70m5JWLL7NBzuv6XS1rF0YDiOJcHIe7I88ZhGfi5 7YDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=R7NUgyhmz2Vzm1IbfrrDYD1qNpQGYjcjDbFVXUaMLH8=; b=p82gbJM2HWttUNCrNs0nTRjhJeYGGWhig7vqcyHA4ftx8adbPHYlGZGJDfWoxN6Whx zza24pB5cYTzlnRmQ1odlSOHNwuJCi7CEBAIxJKPN3/tfdhoWPirmcV67An0p21UedhT kdPQhk7l9LmEweDG68qL0sv1smoMr9pN1J/2CUGINK3+fqmnTXwpjkRXfR4xmo1CyJN9 m3Q0hv5VgZQuJD3H2UNVh59EoEaz8Cczi9o5mph6HoJ23kGDVDpQ/1CVMp+WyOjmZg8D AujWMwA+W7slkpm+t2B8OCBRWvVxtO/sMInIChLObRjrLDmuY1uOSvp/ra+Ofr2+bPLP SYcQ== X-Gm-Message-State: AOAM5301jo/3Wyam8Lt4cMwCtz7D9TiY9GtUXk7Erls2ZeUMcCoSi7PE Dva1FHzIJ74XsF1LEReQLOAxvWc3rQjcOIx3Tg== X-Google-Smtp-Source: ABdhPJzj4uTOWudxzzJokeTZXJ0Xc+OcxmAl9Nm7IKWdiyC4Dj/qqu6z+Dkz3NZLlc5P6ZEzL8Pz6NCeaRU6Ltd2cA== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:5db7:1235:b3dd:cfcb]) (user=kaleshsingh job=sendgmr) by 2002:a05:6902:ca:b0:5ff:5f2d:b533 with SMTP id i10-20020a05690200ca00b005ff5f2db533mr23967292ybs.606.1645548964571; Tue, 22 Feb 2022 08:56:04 -0800 (PST) Date: Tue, 22 Feb 2022 08:51:04 -0800 In-Reply-To: <20220222165212.2005066-1-kaleshsingh@google.com> Message-Id: <20220222165212.2005066-4-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220222165212.2005066-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v2 3/9] KVM: arm64: Add guard pages for KVM nVHE hypervisor stack From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , Catalin Marinas , James Morse , Alexandru Elisei , Suzuki K Poulose , Ard Biesheuvel , Mark Rutland , Pasha Tatashin , Joey Gouly , Peter Collingbourne , Andrew Scull , Paolo Bonzini , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.cs.columbia.edu X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220222_085606_377396_CAF9F6B1 X-CRM114-Status: GOOD ( 16.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Maps the stack pages in the flexible private VA range and allocates guard pages below the stack as unbacked VA space. The stack is aligned to twice its size to aid overflow detection (implemented in a subsequent patch in the series). Signed-off-by: Kalesh Singh --- arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/kvm/arm.c | 32 +++++++++++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index d5b0386ef765..2e277f2ed671 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -169,6 +169,7 @@ struct kvm_nvhe_init_params { unsigned long tcr_el2; unsigned long tpidr_el2; unsigned long stack_hyp_va; + unsigned long stack_pa; phys_addr_t pgd_pa; unsigned long hcr_el2; unsigned long vttbr; diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index ecc5958e27fe..7e2e680c3ffb 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1541,7 +1541,6 @@ static void cpu_prepare_hyp_mode(int cpu) tcr |= (idmap_t0sz & GENMASK(TCR_TxSZ_WIDTH - 1, 0)) << TCR_T0SZ_OFFSET; params->tcr_el2 = tcr; - params->stack_hyp_va = kern_hyp_va(per_cpu(kvm_arm_hyp_stack_page, cpu) + PAGE_SIZE); params->pgd_pa = kvm_mmu_get_httbr(); if (is_protected_kvm_enabled()) params->hcr_el2 = HCR_HOST_NVHE_PROTECTED_FLAGS; @@ -1990,14 +1989,41 @@ static int init_hyp_mode(void) * Map the Hyp stack pages */ for_each_possible_cpu(cpu) { + struct kvm_nvhe_init_params *params = per_cpu_ptr_nvhe_sym(kvm_init_params, cpu); char *stack_page = (char *)per_cpu(kvm_arm_hyp_stack_page, cpu); - err = create_hyp_mappings(stack_page, stack_page + PAGE_SIZE, - PAGE_HYP); + unsigned long stack_hyp_va, guard_hyp_va; + /* + * Private mappings are allocated downwards from io_map_base + * so allocate the stack first then the guard page. + * + * The stack is aligned to twice its size to facilitate overflow + * detection. + */ + err = __create_hyp_private_mapping(__pa(stack_page), PAGE_SIZE, + PAGE_SIZE * 2, &stack_hyp_va, PAGE_HYP); if (err) { kvm_err("Cannot map hyp stack\n"); goto out_err; } + + /* Allocate unbacked private VA range for stack guard page */ + guard_hyp_va = hyp_alloc_private_va_range(PAGE_SIZE, PAGE_SIZE); + if (IS_ERR((void *)guard_hyp_va)) { + err = PTR_ERR((void *)guard_hyp_va); + kvm_err("Cannot allocate hyp stack guard page\n"); + goto out_err; + } + + /* + * Save the stack PA in nvhe_init_params. This will be needed to recreate + * the stack mapping in protected nVHE mode. __hyp_pa() won't do the right + * thing there, since the stack has been mapped in the flexible private + * VA space. + */ + params->stack_pa = __pa(stack_page) + PAGE_SIZE; + + params->stack_hyp_va = stack_hyp_va + PAGE_SIZE; } for_each_possible_cpu(cpu) { From patchwork Tue Feb 22 16:51:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12755709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A9314C433EF for ; Tue, 22 Feb 2022 17:09:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HnrSJZE2+OVz+ltSOcf5wnwu3giSXoUmLzgx1/XPYD8=; b=ju3QPVEH7wud/r W2bgeLqG/UdXj394o92jLvCIkTJS/FyDbArbF0JJWByeUp4+XJC4IcQ5XEHmIvOB6ldDH+t7pTsow TMlcJDpiJjx3X+0vfhuE4qaBmEHiJwO8AV+JMg8V6vewDa+4peLvMPc2SBzZEYvwBode9Jk6AkyTd WX67j1wTTY7iBhyxzmgnA5rL3yQST+TxaaRJoT3Zr7pnVA7Ffv98LCWUmict194dpNRPD4hw9IvdZ zzkKbm0N/p0hIps+cF1FjSWFuSIpKof9cdpKKyVtfliCwZKYnBDmJjufVhyA+3rwu5vfm5E5KjK8f 6U6jtl/t8RccZq4YM8eA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYdb-00ArYX-AE; Tue, 22 Feb 2022 17:07:44 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYTQ-00AnkC-Mm for linux-arm-kernel@lists.infradead.org; Tue, 22 Feb 2022 16:57:14 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-2d6b40fc9bdso139014237b3.13 for ; Tue, 22 Feb 2022 08:57:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=knnzmuohJ2ih2y2NTiUEzUxj92qYLV4FmoYUekHSvUw=; b=tazhhxtdUk3rQttzgDm+V0n+dY4v2ZnB4i8xG8ax2HZdSoHAffJo42UTzU3IHvzAqj CzDDQtHudJDmOpjvt+NTOxsHiMMcd5AR1KeV4KZ04Anj9Hvg2wo5WdkKx0uhJOAuX1UG DMG5h+1UR544FLvyjxNA1/VvkZavi0pxiwKfoHP3qdoBAJ8849pS8kTNoGD5QX5f9/Px 48yknJpXBRw8xvBSwtZigYe1MGyv8eQBBWwjuMT8BSujAmFufVqRsnYE2LCmZqywZs8i wJddWj0UZSrpFMMQpEijc1j/Xch9CqJ1J2YUynBS8lvvuY7OGu4etP9u4qE8wqmUHhIw yLwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=knnzmuohJ2ih2y2NTiUEzUxj92qYLV4FmoYUekHSvUw=; b=N9fDa0VsRqcAEzq2xHF2LSw4jiOlKG4kqfWZJmqacSHVsATxviUCrPw3KjOMwpbgQH jAD8XeZTcyUR8z1iKZ5be/SEj/eJWrq3aOOBTGSGHDGpuwI0hy/WXHvwslbCrrNid+MM 0+/q3Cd+9f+NIDlXG7T06Gb9NIuQOPEPIXdrK+9uvDoNoeku5Cbf5hJGqYWdtyzd6tto 3fFKpbMzUmZB93aJnXo1XnmVRST6EJeLoj2rrroR3HttA/t2Ax7AS+qapWwRjCOZoZLT lPiw2FaFUxrO0VID5EDj4B9dsFDIoFgSEXIppFq0Mj9lnv30NUC1LGFk5eSJNQzQnxm2 s7lw== X-Gm-Message-State: AOAM531N5gKYTfvX+QLQfTPQ+dY+BS1hMjkenxvxaXrNONL4OBtfxIjl 5X5GqQIZcum9qteaaP/nvA/jNuUd5/uZU2U1TQ== X-Google-Smtp-Source: ABdhPJxsM9h50TZI3B0JLyBOhiXv0umcrv6dsBugtZNBeZtlKN8O9HTo3L2ie+qpRSFxA92HD2CPBV8kKYkIX9PJAQ== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:5db7:1235:b3dd:cfcb]) (user=kaleshsingh job=sendgmr) by 2002:a25:4cc1:0:b0:623:ca02:c1e5 with SMTP id z184-20020a254cc1000000b00623ca02c1e5mr22952503yba.95.1645549030879; Tue, 22 Feb 2022 08:57:10 -0800 (PST) Date: Tue, 22 Feb 2022 08:51:05 -0800 In-Reply-To: <20220222165212.2005066-1-kaleshsingh@google.com> Message-Id: <20220222165212.2005066-5-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220222165212.2005066-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v2 4/9] KVM: arm64: Add guard pages for pKVM (protected nVHE) hypervisor stack From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , Catalin Marinas , James Morse , Alexandru Elisei , Suzuki K Poulose , Ard Biesheuvel , Mark Rutland , Pasha Tatashin , Joey Gouly , Peter Collingbourne , Andrew Scull , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.cs.columbia.edu X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220222_085712_793716_4DBA7B80 X-CRM114-Status: GOOD ( 13.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Maps the stack pages in the flexible private VA range and allocates guard pages below the stack as unbacked VA space. The stack is aligned to twice its size to aid overflow detection (implemented in a subsequent patch in the series). Signed-off-by: Kalesh Singh --- arch/arm64/kvm/hyp/nvhe/setup.c | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index 27af337f9fea..69df21320b09 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -105,11 +105,28 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size, if (ret) return ret; - end = (void *)per_cpu_ptr(&kvm_init_params, i)->stack_hyp_va; + /* + * Private mappings are allocated upwards from __io_map_base + * so allocate the guard page first then the stack. + */ + start = (void *)pkvm_alloc_private_va_range(PAGE_SIZE, PAGE_SIZE); + if (IS_ERR_OR_NULL(start)) + return PTR_ERR(start); + + /* + * The stack is aligned to twice its size to facilitate overflow + * detection. + */ + end = (void *)per_cpu_ptr(&kvm_init_params, i)->stack_pa; start = end - PAGE_SIZE; - ret = pkvm_create_mappings(start, end, PAGE_HYP); - if (ret) - return ret; + start = (void *)__pkvm_create_private_mapping((phys_addr_t)start, + PAGE_SIZE, PAGE_SIZE * 2, PAGE_HYP); + if (IS_ERR_OR_NULL(start)) + return PTR_ERR(start); + end = start + PAGE_SIZE; + + /* Update stack_hyp_va to end of the stack's private VA range */ + per_cpu_ptr(&kvm_init_params, i)->stack_hyp_va = (unsigned long) end; } /* From patchwork Tue Feb 22 16:51:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12755710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B3C10C433F5 for ; Tue, 22 Feb 2022 17:12:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ynaqbzQNmyoLBOB59cmUChj7saAQfwAgizuMsCgblYk=; b=VDMPTCwmtegsns 0A9AInpDNoqQHbtirOv22//i52AJC4k8GCLkGr88oRj1vDxxOhla3hwd0UyMotfWft5buAJnw0vl0 85IZOyN6su0/x9+xFMvF/o/eex1xpbM9VyRp8nZyQuCJV2AuevyDyVSQa0LQIJHM+GnlSf1VezJNB CGX5lBekdMfJHe34OF7nLvvtUtTPyO+TZMIyytL/pMNzUILN44VqiE9tlCCdqgkpLHqyXIHEGCHJp mXjmtiQ8VRYY1YMBflbMlf6WLihrSjg6xmrmlicL4IuHByTb4uF5wpQ2eyufKU9JNFrfI07Bl2K63 v8nSY/wSP0ZvtA9IHVog==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYgW-00AseG-3o; Tue, 22 Feb 2022 17:10:45 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYUe-00AoAu-AU for linux-arm-kernel@lists.infradead.org; Tue, 22 Feb 2022 16:58:30 +0000 Received: by mail-yb1-xb49.google.com with SMTP id d17-20020a253611000000b006244e94b7b4so12351524yba.4 for ; Tue, 22 Feb 2022 08:58:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=LdOkseL/SVBmCmTgjcnNq9QzVzqqoPlVFVsUav/JFK0=; b=FNoLI1ytq66gPDfA9ohgBGXwe3ZXAMfQPU+vmETuazTxn1eRt4sfndjKQl5vNg1s6N aKsNQ+NyKz9hsu+jiKv0znYXLeJd3n/+di/bVx097otwKJ+Pm8qhNs67V2lcMqwTVhI1 wW9Xhb5tkIIU2yAp/NzXxghYQ+kMzgy+o2wG8PnfZAZIib9of+6+4qFBo+n7HfiKcBUT zgEVEI8Bx5lOplWa+VY0r+2kHSy9Bk1YC2Ipw+jivWuz7/saxNwcRLetDoqydMzCZ/+o DlsgOQQ986VSJnpFZSHCEQu861DTK2XF2rz2ttpajHfiMXQ90DWFxCmT05fG0qHlS8Nr E9dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=LdOkseL/SVBmCmTgjcnNq9QzVzqqoPlVFVsUav/JFK0=; b=H8/9FfYrMa6xDxO+1OEd5aeCHaq4GFIqTPeiL8oD671axKQDtk/GrUFtZLc1l+KQCt pCp4USgRo7BbXSrWV+o+gxXbcB7EwroFqNXkvA2GsFse73rzHnofRR7mo5H81ezIQ0SJ nyec3Ydw+ds++xe5FZCSXQ+itaTVzBYQf+GVV4qTcOMKRw2L7bapKu2L5flllgkmTGDq Ol7S6bi8g2PxhKiT1hEETbbv+6aKs0d6IbfkoqPwt8qBdKXUciXqfscaaWNUz5xUUhjp V2fwIsws/AhRWLdxvA8iiKMl/cIWY9RJqZl9chOw9NDRVR+APVsugvJdThjQVr2kRLAU ZHNA== X-Gm-Message-State: AOAM531RE45UTe4miiIg1/BnLcSSaKyPwJmFrXZz911KNw85vPaqCpl7 cCkw9u0y84xvTj3xzgdjPZQU6YOGqNIrMuoEVA== X-Google-Smtp-Source: ABdhPJxTMt4EP42XEzXDEACbHfSQnsjw9YnBj/jsWJgMJWkojIR6N0I/bKfbgjEHFp9OFaaN7ZlREpz8dxBvYs+xsg== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:5db7:1235:b3dd:cfcb]) (user=kaleshsingh job=sendgmr) by 2002:a81:b11:0:b0:2d7:3775:1eb9 with SMTP id 17-20020a810b11000000b002d737751eb9mr10900452ywl.99.1645549106468; Tue, 22 Feb 2022 08:58:26 -0800 (PST) Date: Tue, 22 Feb 2022 08:51:06 -0800 In-Reply-To: <20220222165212.2005066-1-kaleshsingh@google.com> Message-Id: <20220222165212.2005066-6-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220222165212.2005066-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v2 5/9] arm64: asm: Introduce test_sp_overflow macro From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , Catalin Marinas , James Morse , Alexandru Elisei , Suzuki K Poulose , Ard Biesheuvel , Mark Rutland , Pasha Tatashin , Joey Gouly , Peter Collingbourne , Andrew Scull , Paolo Bonzini , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.cs.columbia.edu X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220222_085828_420788_E4E42DFD X-CRM114-Status: GOOD ( 13.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Quentin Perret The asm entry code in the kernel uses a trick to check if VMAP'd stacks have overflowed by aligning them at THREAD_SHIFT * 2 granularity and checking the SP's THREAD_SHIFT bit. Protected KVM will soon make use of a similar trick to detect stack overflows, so factor out the asm code in a re-usable macro. Signed-off-by: Quentin Perret [Kalesh - Resolve minor conflicts] Signed-off-by: Kalesh Singh --- arch/arm64/include/asm/assembler.h | 11 +++++++++++ arch/arm64/kernel/entry.S | 7 +------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index e8bd0af0141c..ad40eb0eee83 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -850,4 +850,15 @@ alternative_endif #endif /* GNU_PROPERTY_AARCH64_FEATURE_1_DEFAULT */ +/* + * Test whether the SP has overflowed, without corrupting a GPR. + */ +.macro test_sp_overflow shift, label + add sp, sp, x0 // sp' = sp + x0 + sub x0, sp, x0 // x0' = sp' - x0 = (sp + x0) - x0 = sp + tbnz x0, #\shift, \label + sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 + sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp +.endm + #endif /* __ASM_ASSEMBLER_H */ diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 772ec2ecf488..ce99ee30c77e 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -53,15 +53,10 @@ alternative_else_nop_endif sub sp, sp, #PT_REGS_SIZE #ifdef CONFIG_VMAP_STACK /* - * Test whether the SP has overflowed, without corrupting a GPR. * Task and IRQ stacks are aligned so that SP & (1 << THREAD_SHIFT) * should always be zero. */ - add sp, sp, x0 // sp' = sp + x0 - sub x0, sp, x0 // x0' = sp' - x0 = (sp + x0) - x0 = sp - tbnz x0, #THREAD_SHIFT, 0f - sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 - sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp + test_sp_overflow THREAD_SHIFT, 0f b el\el\ht\()_\regsize\()_\label 0: From patchwork Tue Feb 22 16:51:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12755727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DF54AC433F5 for ; Tue, 22 Feb 2022 17:14:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xB6h7oY2EjMVKY3m8SKr/aM+ipmPicLZJ1qy0I4CghY=; b=cRp4cMvWVS3HhL pvvmbFi9z21Qylz3fMM9s7O12PE6FH6aq12fzFeQfVKgrXQ3eJjKGYJ4Zhr5CYKLYxivbDLvjfM2/ xyNXsM1QakJumUciVhrxPEPpJw2VnVMtmggj0svy6bUFLgo3RDJndZaF6Z9tH5FA+fQZfiOCsRU0n NtObBEw6R/2vCVpwVrf52DxvZCE9LIfBbKXh3XseofZBdNa37EuHoFtoQZpdpnLx5bdOF3itPQ8Dq g78c/OcF/g7tAhVEkvJqmtlHI3J/NqSYLAcKJj/TO+MxlDrw1h4MVLSKUi9e0h4M/fKJiIlyoCnij Zw6ahuDvbsmlZeYLXO9g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYif-00AteK-PI; Tue, 22 Feb 2022 17:12:58 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYVh-00Aoad-8n for linux-arm-kernel@lists.infradead.org; Tue, 22 Feb 2022 16:59:34 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-2d61b4ef6cdso146069417b3.11 for ; Tue, 22 Feb 2022 08:59:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=kA1uHTXTZgvCDEwUv0Venj02VbjFVl5MyBDxltrhOyc=; b=NP6fnietNid07WODom2crvgVApuYNwyBc+vaQeT9ELb0Wby0y027V1WoutortBDdER 6VKH2m5g/hhqE8jP+GV8Z52SXKndZFpGpqKGnvqTJ6saOgO9yiXq/Sl8KOE48ShXSFrf Yd/GQs7iTARZWcbcAbs2DZcziHvlIf4ws8th2T86PRoxGLRCAo79SHT5a0UzBseydVww c/kdTBUTDmKW4WhBmbRdCfs4Sm1ZyK65IyLMvKPqQFMBa2uiSRCXXe8ja7vMgkYdHVOd FDQ7AFIgQCuxZCe7VWKGtZS0vLPpQ4mLhJFEH4IE0gr/JO65ZVTSGIb2gzhuMTjBEe1w 9t+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=kA1uHTXTZgvCDEwUv0Venj02VbjFVl5MyBDxltrhOyc=; b=r+zmFUJ4w2JAf/P26F8cMb7YnkHQhIdmGHi9LgRPnoJF6x7tVFPeItYKDAmOBdFJn4 Di8X0V3PfyxtzdNDbSJs7DnPacTccYDyut6lbQ91Z0nxMJaEOyfjgdIhj/HKii0i8/Lu Pe19Zcxjqulj59e/TYNySphWWDtOMVnVqz7VMkzB2w8p066xUukB4kiuEu9e0d2vL187 7SiFopenyVcadycUmkeEVy+T/HZxGoneFtEEyh7OG6rdGO1i5dP0mbTHEb5CpD0Mymjg ffpUSvj80mG5Th1QLQp72XD6p3Uua/R/IQY1fbwUQMCtymLBsfvYbRS/x6JRCd9Gac+L j8rA== X-Gm-Message-State: AOAM530hl/gI8abbMBRWNOGek6GTMXUNppZKZ11AA/7sOGuNGzchYALa Q/lzzNYSlL3y5UEuaDbx4fdpsh/205AP7vQjyA== X-Google-Smtp-Source: ABdhPJw2THd1z2iAbfih7pFjkHmBG5AlQKbmEqgBiWlGmwrM/dGUOKWBoivdDJ4o6t47SOdmT8K0alB2LRMYcbGgbA== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:5db7:1235:b3dd:cfcb]) (user=kaleshsingh job=sendgmr) by 2002:a81:138a:0:b0:2d6:b391:90cd with SMTP id 132-20020a81138a000000b002d6b39190cdmr23404107ywt.480.1645549171601; Tue, 22 Feb 2022 08:59:31 -0800 (PST) Date: Tue, 22 Feb 2022 08:51:07 -0800 In-Reply-To: <20220222165212.2005066-1-kaleshsingh@google.com> Message-Id: <20220222165212.2005066-7-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220222165212.2005066-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v2 6/9] KVM: arm64: Detect and handle hypervisor stack overflows From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , Catalin Marinas , James Morse , Alexandru Elisei , Suzuki K Poulose , Ard Biesheuvel , Mark Rutland , Pasha Tatashin , Joey Gouly , Peter Collingbourne , Andrew Scull , Paolo Bonzini , Zenghui Yu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.cs.columbia.edu X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220222_085933_356564_71ABC080 X-CRM114-Status: GOOD ( 11.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The hypervisor stacks (for both nVHE Hyp mode and nVHE protected mode) are aligned to twice their size (PAGE_SIZE), meaning that any valid stack address has PAGE_SHIFT bit as 0. This allows us to conveniently check for overflow in the exception entry without corrupting any GPRs. We won't recover from a stack overflow so panic the hypervisor. Signed-off-by: Kalesh Singh Reported-by: kernel test robot Reported-by: kernel test robot --- arch/arm64/kvm/hyp/nvhe/host.S | 16 ++++++++++++++++ arch/arm64/kvm/hyp/nvhe/switch.c | 5 +++++ 2 files changed, 21 insertions(+) diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S index 3d613e721a75..78e4b612ac06 100644 --- a/arch/arm64/kvm/hyp/nvhe/host.S +++ b/arch/arm64/kvm/hyp/nvhe/host.S @@ -153,6 +153,10 @@ SYM_FUNC_END(__host_hvc) .macro invalid_host_el2_vect .align 7 + + /* Test stack overflow without corrupting GPRs */ + test_sp_overflow PAGE_SHIFT, .L__hyp_sp_overflow\@ + /* If a guest is loaded, panic out of it. */ stp x0, x1, [sp, #-16]! get_loaded_vcpu x0, x1 @@ -165,6 +169,18 @@ SYM_FUNC_END(__host_hvc) * been partially clobbered by __host_enter. */ b hyp_panic + +.L__hyp_sp_overflow\@: + /* + * Reset SP to the top of the stack, to allow handling the hyp_panic. + * This corrupts the stack but is ok, since we won't be attempting + * any unwinding here. + */ + ldr_this_cpu x0, kvm_init_params + NVHE_INIT_STACK_HYP_VA, x1 + mov sp, x0 + + bl hyp_panic_bad_stack + ASM_BUG() .endm .macro invalid_host_el1_vect diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 6410d21d8695..5a2e1ab79913 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -369,6 +369,11 @@ void __noreturn hyp_panic(void) unreachable(); } +void __noreturn hyp_panic_bad_stack(void) +{ + hyp_panic(); +} + asmlinkage void kvm_unexpected_el2_exception(void) { return __kvm_unexpected_el2_exception(); From patchwork Tue Feb 22 16:51:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12755728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 10E6CC433F5 for ; Tue, 22 Feb 2022 17:16:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=joYOrxFYIj7Z1oZAsElrT6kUaGzNsgmR2P/gT6AkWbE=; b=bZ8x4GxJUTv9ow jl+T0I6VFZv1U/FbLYA5J+s+1M+tjqWjRgkTllhPVykYEJnobl3nmMthtShRZvy4wXvrRIzSSOeE9 ibHvnC2YBVrN94NqGOIiXG6iNetYGMm5nTJAZTUJ4iDUf9nPYhNlTMKym3Tnwg56rGTLAHoFPHreX uCZtG+p+3Qt4oK/4ukr9mrqsY19dIpIBd+4bu7zXIVEyQ+7SDODnYWhJPj0aeV6C1rFBdRtyIZEd5 zIUDnQUElPqxEI0m5a6f6U5xtmAxKtVSYY161nv+p4IPEZ+O/Iju6HfZufUfySUirPOurvl4CfFPc AUgBtd8mcqUpAmgZdHkg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYkm-00AuQr-Ia; Tue, 22 Feb 2022 17:15:09 +0000 Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYWm-00Ap06-TB for linux-arm-kernel@lists.infradead.org; Tue, 22 Feb 2022 17:00:42 +0000 Received: by mail-pf1-x44a.google.com with SMTP id g19-20020aa796b3000000b004e136dcec34so7311792pfk.23 for ; Tue, 22 Feb 2022 09:00:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=5qb5Zh5e8x0hA/HCW8SK7letHwHax9Wce8Hz1k3JTdg=; b=ITgvGI+DDxrws5QD+aqNmanS1XXPkR5OB1mc/GjeEFKe1PUFInLivQOj2yo7QUg+Ms jiQqEBIgx8+pm/kTGUv4kJbSou6k/zNoYhVYDkZAoPQcycS+1hRaIFS0LySP1N6gBZzN UaUG12NsYuyKvxTJaL6deitrMeqYZr6WLyZD3VhFhAjEtFzDl9XHbYq3JslYm4d8OYjz X/npdOM7xhWI6vlrhC6uIMRYM7xjLfG1ykah3QVJqU0WAp7G1+U7twKq7GAqvOV9yNkk pKJSH3usycLKHhOjCm8vwteuS3A7gapZDcF2n0sPj1Iupc/Hz3asBMWMKtVvwjT0Yhf5 /1Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=5qb5Zh5e8x0hA/HCW8SK7letHwHax9Wce8Hz1k3JTdg=; b=wrUP19ERgbrbONSi1JyK8Yo49P1/HgI8F8+bWEx9rpZK/rWUYGftLVkei+6RdfmRAd n/h4rIA59JKlerUkHmFPYeJ4C8bBkiKP42K3nMd3KS9y38teBRKuc/GkAMI2GCVMuroD bkkibzKszHzN5NFW2I3+YIa9cz27PmQcQI9QueaA9fb1oQ1z2hzDt9YeV0yTjZh847xg l+mqEKS7i6jVGFRqp2fiVPXgn9nwTE8kljoLhvjEz5/+/n5iC0zPb33vzjGN72GYjr+v gmwG4M4s9IhkEOIPV7Zg+xamc7WNL99MVxaYbDLYSJh2erxvXls9gO4kOs+tlBMKrSXX RDgg== X-Gm-Message-State: AOAM532kaZZq2DCK0dWOn0m2EJZRlqHxUwVw+X0vhDptQ6ow+wlVGKDL 7WcLLw7lDQD+d1INjwP8VOW74oVykUpznh6iSA== X-Google-Smtp-Source: ABdhPJzH5Lfs/f/UpeVcOR5FEQsrODh9JTFqCt6nAPQwaG7BkrPBoWZU4Etxumn3mAF9Ju4JlTfm2YcSMf2h76tr4w== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:5db7:1235:b3dd:cfcb]) (user=kaleshsingh job=sendgmr) by 2002:a05:6a00:2296:b0:4e1:3029:ee2 with SMTP id f22-20020a056a00229600b004e130290ee2mr25396129pfe.22.1645549239077; Tue, 22 Feb 2022 09:00:39 -0800 (PST) Date: Tue, 22 Feb 2022 08:51:08 -0800 In-Reply-To: <20220222165212.2005066-1-kaleshsingh@google.com> Message-Id: <20220222165212.2005066-8-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220222165212.2005066-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v2 7/9] KVM: arm64: Add hypervisor overflow stack From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , Catalin Marinas , James Morse , Alexandru Elisei , Suzuki K Poulose , Ard Biesheuvel , Mark Rutland , Pasha Tatashin , Joey Gouly , Peter Collingbourne , Andrew Scull , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.cs.columbia.edu X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220222_090040_999407_0898C0E1 X-CRM114-Status: GOOD ( 13.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allocate and switch to 16-byte aligned secondary stack on overflow. This provides us stack space to better handle overflows; and is used in a subsequent patch to dump the hypervisor stacktrace. The overflow stack is only allocated if CONFIG_NVHE_EL2_DEBUG is enabled, as hypervisor stacktraces is a debug feature dependent on CONFIG_NVHE_EL2_DEBUG. Signed-off-by: Kalesh Singh --- arch/arm64/kvm/hyp/nvhe/host.S | 5 +++++ arch/arm64/kvm/hyp/nvhe/switch.c | 5 +++++ 2 files changed, 10 insertions(+) diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S index 78e4b612ac06..751a4b9e429f 100644 --- a/arch/arm64/kvm/hyp/nvhe/host.S +++ b/arch/arm64/kvm/hyp/nvhe/host.S @@ -171,6 +171,10 @@ SYM_FUNC_END(__host_hvc) b hyp_panic .L__hyp_sp_overflow\@: +#ifdef CONFIG_NVHE_EL2_DEBUG + /* Switch to the overflow stack */ + adr_this_cpu sp, hyp_overflow_stack + PAGE_SIZE, x0 +#else /* * Reset SP to the top of the stack, to allow handling the hyp_panic. * This corrupts the stack but is ok, since we won't be attempting @@ -178,6 +182,7 @@ SYM_FUNC_END(__host_hvc) */ ldr_this_cpu x0, kvm_init_params + NVHE_INIT_STACK_HYP_VA, x1 mov sp, x0 +#endif bl hyp_panic_bad_stack ASM_BUG() diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 5a2e1ab79913..2accc158210f 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -34,6 +34,11 @@ DEFINE_PER_CPU(struct kvm_host_data, kvm_host_data); DEFINE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); +#ifdef CONFIG_NVHE_EL2_DEBUG +DEFINE_PER_CPU(unsigned long [PAGE_SIZE/sizeof(long)], hyp_overflow_stack) + __aligned(16); +#endif + static void __activate_traps(struct kvm_vcpu *vcpu) { u64 val; From patchwork Tue Feb 22 16:51:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12755729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3EFC7C433F5 for ; Tue, 22 Feb 2022 17:17:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=X6gZzQ2AyHf8s0VKzQT1Iq016vZpIK56vPtZx4cjgmU=; b=Djhyl4Gg9ffEKk 30iKd2TEwp3RecAssMeQ4OgOOBq6UqsrGWATdyKf2fYIGP5sEJnTDLnYeFO45ZUulIoPPxq5oOqbf PPi8ggsEQdj4qzuGEXImiMbNB0c8pSQbOvZdkCI8h1A8azKi3oiQ2ShkyAvLXB6Y+zpqb/PuxmP1c 7af+KW0lzCT+HMIhF9tgz/90mSNn/K4QG3IkMUuZB07IJmm4BCzj19xewvwNQwouXDZKa4Z5dfkUx UHO2eUO0FbcrBG2YQckYx0rE8qSoGgSnAQs2Y3an83t2kCHmaL5mlotN7YFKgqNWATvfTfRgD9nXU jfeOEtXkANBVG+wPO4Xg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYld-00Auns-84; Tue, 22 Feb 2022 17:16:01 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYXo-00ApVN-P9 for linux-arm-kernel@lists.infradead.org; Tue, 22 Feb 2022 17:01:47 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id e129-20020a25d387000000b006245d830ca6so10497363ybf.13 for ; Tue, 22 Feb 2022 09:01:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=EEB0cF7W8Z4tzEJGAGYjBKXuJWIAUS9G1/uPvryZBvQ=; b=p7Id9dQujHyG29UdzWat8WdvrrPOw9krxSXiqk00tl1BZL9gYpFz1Lm+YwRDklhGUH 150VLbRfa0vcn/ERLt+En0egKXNgOC8lOqXzts0iqORAExDDaPBq3HdTShbvH1CSuDZK lCiPBIiXIti2UHtIQayQ0NSEjtzeT40JmK27ityUxx5sAtcfCy46cvf6FedW4XfLEfmZ fhR9famMNOpZEMx80sWjahdWq+EzlpJ4922/FHyVPoD0Ln4NAfbQO+vCazvCjeJOFkzM uavkGk1fmEM0XM2PtoEUoda5VCyP1ut3qzRDpZH2wkQuOscNUT3ISzbq5dQ2OWa6ykos IBYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=EEB0cF7W8Z4tzEJGAGYjBKXuJWIAUS9G1/uPvryZBvQ=; b=HhAAU2AyY5wVER5v+ix+4kkHTVC6zifu8PcwJbVdxwImElRo4wqr7EUbP5V8nMV7ta D49ftDi64OFo18LODxsajWAebwKUcKPVTM5y5+PzZreeFZV/8OafoA305PRB7RukckQw 8jcFkdA8EkvRsy+H7OoUpA0GeOF9w+iSuVFe4QiIxMPQ8wNs8t+DOZc8Ynestqv3pI/k TRfm6YXGtx7E+ALDNp17fpps0PbJ4g61WcauQuf8NFSzTe6pc8yX+CBV0JIOginCEns+ B71hfqvS/S7cxD9fOTY/loQgpm3a2UE5KY8GPmgXQps6Ir1BweT96VqGJhpph3d4Y5Nf 6cjQ== X-Gm-Message-State: AOAM5306Ghd1oG9vatTa6B7ZvOa0xPxcqkj9jiypyNPlhsgjwzIjakd8 feitlrpbFOtEa0Dla+sRfsaX8gW0DDGcQ0RUjg== X-Google-Smtp-Source: ABdhPJzEcdFuLjZ+IU/YVKrt6YVo9vU91uda9Y6U5r89FElLqqMG4NnbgdLFIeyeY+V8GzaBnO8HGFz4Jr+YvAi4wg== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:5db7:1235:b3dd:cfcb]) (user=kaleshsingh job=sendgmr) by 2002:a25:84cf:0:b0:621:a740:3249 with SMTP id x15-20020a2584cf000000b00621a7403249mr23161492ybm.275.1645549303006; Tue, 22 Feb 2022 09:01:43 -0800 (PST) Date: Tue, 22 Feb 2022 08:51:09 -0800 In-Reply-To: <20220222165212.2005066-1-kaleshsingh@google.com> Message-Id: <20220222165212.2005066-9-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220222165212.2005066-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v2 8/9] KVM: arm64: Unwind and dump nVHE HYP stacktrace From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , Catalin Marinas , James Morse , Alexandru Elisei , Suzuki K Poulose , Ard Biesheuvel , Mark Rutland , Pasha Tatashin , Joey Gouly , Peter Collingbourne , Andrew Scull , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.cs.columbia.edu X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220222_090144_882618_B762A5FF X-CRM114-Status: GOOD ( 31.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Unwind the stack in EL1, when CONFIG_NVHE_EL2_DEBUG is enabled. This is possible because CONFIG_NVHE_EL2_DEBUG disables the host stage 2 protection which allows host to access the hypervisor stack pages in EL1. Unwinding and dumping hyp call traces is gated on CONFIG_NVHE_EL2_DEBUG to avoid the potential leaking of information to the host. A simple stack overflow test produces the following output: [ 580.376051][ T412] kvm: nVHE hyp panic at: ffffffc0116145c4! [ 580.378034][ T412] kvm [412]: nVHE HYP call trace: [ 580.378591][ T412] kvm [412]: [] [ 580.378993][ T412] kvm [412]: [] [ 580.379386][ T412] kvm [412]: [] // Non-terminating recursive call [ 580.379772][ T412] kvm [412]: [] [ 580.380158][ T412] kvm [412]: [] [ 580.380544][ T412] kvm [412]: [] [ 580.380928][ T412] kvm [412]: [] . . . Since nVHE hyp symbols are not included by kallsyms to avoid issues with aliasing, we fallback to the vmlinux addresses. Symbolizing the addresses is handled in the next patch in this series. Signed-off-by: Kalesh Singh --- Changes in v2: - Add cpu_prepare_nvhe_panic_info() - Move updating the panic info to hyp_panic(), so that unwinding also works for conventional nVHE Hyp-mode. arch/arm64/include/asm/kvm_asm.h | 17 ++ arch/arm64/kvm/Kconfig | 5 +- arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/arm.c | 2 +- arch/arm64/kvm/handle_exit.c | 3 + arch/arm64/kvm/hyp/nvhe/switch.c | 19 ++ arch/arm64/kvm/stacktrace.c | 290 +++++++++++++++++++++++++++++++ arch/arm64/kvm/stacktrace.h | 17 ++ 8 files changed, 351 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/kvm/stacktrace.c create mode 100644 arch/arm64/kvm/stacktrace.h diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 2e277f2ed671..af44b3a0596b 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -176,6 +176,23 @@ struct kvm_nvhe_init_params { unsigned long vtcr; }; +#ifdef CONFIG_NVHE_EL2_DEBUG +/* + * Used by the host in EL1 to dump the nVHE hypervisor backtrace on + * hyp_panic. This is possible because CONFIG_NVHE_EL2_DEBUG disables + * the host stage 2 protection. See: __hyp_do_panic() + * + * @hyp_stack_base: hyp VA of the hyp_stack base. + * @hyp_overflow_stack_base: hyp VA of the hyp_overflow_stack base. + * @start_fp: hyp FP where the hyp backtrace should begin. + */ +struct kvm_nvhe_panic_info { + unsigned long hyp_stack_base; + unsigned long hyp_overflow_stack_base; + unsigned long start_fp; +}; +#endif + /* Translate a kernel address @ptr into its equivalent linear mapping */ #define kvm_ksym_ref(ptr) \ ({ \ diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 8a5fbbf084df..75f2c8255ff0 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -51,8 +51,9 @@ config NVHE_EL2_DEBUG depends on KVM help Say Y here to enable the debug mode for the non-VHE KVM EL2 object. - Failure reports will BUG() in the hypervisor. This is intended for - local EL2 hypervisor development. + Failure reports will BUG() in the hypervisor; and panics will print + the hypervisor call stack. This is intended for local EL2 hypervisor + development. If unsure, say N. diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 91861fd8b897..262b5c58cc62 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -23,6 +23,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ vgic/vgic-its.o vgic/vgic-debug.o kvm-$(CONFIG_HW_PERF_EVENTS) += pmu-emul.o +kvm-$(CONFIG_NVHE_EL2_DEBUG) += stacktrace.o always-y := hyp_constants.h hyp-constants.s diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 7e2e680c3ffb..491cf1eb28f6 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -49,7 +49,7 @@ DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized); DECLARE_KVM_HYP_PER_CPU(unsigned long, kvm_hyp_vector); -static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); +DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); unsigned long kvm_arm_hyp_percpu_base[NR_CPUS]; DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index e3140abd2e2e..b038c32a3236 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -23,6 +23,7 @@ #define CREATE_TRACE_POINTS #include "trace_handle_exit.h" +#include "stacktrace.h" typedef int (*exit_handle_fn)(struct kvm_vcpu *); @@ -326,6 +327,8 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, kvm_err("nVHE hyp panic at: %016llx!\n", elr_virt + hyp_offset); } + hyp_dump_backtrace(hyp_offset); + /* * Hyp has panicked and we're going to handle that by panicking the * kernel. The kernel offset will be revealed in the panic so we're diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 2accc158210f..57ab23f03b1e 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -37,6 +37,23 @@ DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); #ifdef CONFIG_NVHE_EL2_DEBUG DEFINE_PER_CPU(unsigned long [PAGE_SIZE/sizeof(long)], hyp_overflow_stack) __aligned(16); +DEFINE_PER_CPU(struct kvm_nvhe_panic_info, kvm_panic_info); + +DECLARE_PER_CPU(struct kvm_nvhe_panic_info, kvm_panic_info); + +static void cpu_prepare_nvhe_panic_info(void) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr(&kvm_panic_info); + struct kvm_nvhe_init_params *params = this_cpu_ptr(&kvm_init_params); + + panic_info->hyp_stack_base = (unsigned long)(params->stack_hyp_va - PAGE_SIZE); + panic_info->hyp_overflow_stack_base = (unsigned long)this_cpu_ptr(hyp_overflow_stack); + panic_info->start_fp = (unsigned long)__builtin_frame_address(0); +} +#else +static void cpu_prepare_nvhe_panic_info(void) +{ +} #endif static void __activate_traps(struct kvm_vcpu *vcpu) @@ -360,6 +377,8 @@ void __noreturn hyp_panic(void) struct kvm_cpu_context *host_ctxt; struct kvm_vcpu *vcpu; + cpu_prepare_nvhe_panic_info(); + host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; vcpu = host_ctxt->__hyp_running_vcpu; diff --git a/arch/arm64/kvm/stacktrace.c b/arch/arm64/kvm/stacktrace.c new file mode 100644 index 000000000000..cdd672bf0ea8 --- /dev/null +++ b/arch/arm64/kvm/stacktrace.c @@ -0,0 +1,290 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Stack unwinder for EL2 nVHE hypervisor. + * + * Code mostly copied from the arm64 kernel stack unwinder + * and adapted to the nVHE hypervisor. + * + * See: arch/arm64/kernel/stacktrace.c + * + * CONFIG_NVHE_EL2_DEBUG disables the host stage-2 protection + * allowing us to access the hypervisor stack pages and + * consequently unwind its stack from the host in EL1. + * + * See: __hyp_do_panic() + */ + +#include +#include +#include +#include "stacktrace.h" + +DECLARE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); +DECLARE_KVM_NVHE_PER_CPU(unsigned long [PAGE_SIZE/sizeof(long)], hyp_overflow_stack); +DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_panic_info, kvm_panic_info); + +enum hyp_stack_type { + HYP_STACK_TYPE_UNKNOWN, + HYP_STACK_TYPE_HYP, + HYP_STACK_TYPE_OVERFLOW, + __NR_HYP_STACK_TYPES +}; + +struct hyp_stack_info { + unsigned long low; + unsigned long high; + enum hyp_stack_type type; +}; + +/* + * A snapshot of a frame record or fp/lr register values, along with some + * accounting information necessary for robust unwinding. + * + * @fp: The fp value in the frame record (or the real fp) + * @pc: The pc value calculated from lr in the frame record. + * + * @stacks_done: Stacks which have been entirely unwound, for which it is no + * longer valid to unwind to. + * + * @prev_fp: The fp that pointed to this frame record, or a synthetic value + * of 0. This is used to ensure that within a stack, each + * subsequent frame record is at an increasing address. + * @prev_type: The type of stack this frame record was on, or a synthetic + * value of HYP_STACK_TYPE_UNKNOWN. This is used to detect a + * transition from one stack to another. + */ +struct hyp_stackframe { + unsigned long fp; + unsigned long pc; + DECLARE_BITMAP(stacks_done, __NR_HYP_STACK_TYPES); + unsigned long prev_fp; + enum hyp_stack_type prev_type; +}; + +static inline bool __on_hyp_stack(unsigned long hyp_sp, unsigned long size, + unsigned long low, unsigned long high, + enum hyp_stack_type type, + struct hyp_stack_info *info) +{ + if (!low) + return false; + + if (hyp_sp < low || hyp_sp + size < hyp_sp || hyp_sp + size > high) + return false; + + if (info) { + info->low = low; + info->high = high; + info->type = type; + } + return true; +} + +static inline bool on_hyp_overflow_stack(unsigned long hyp_sp, unsigned long size, + struct hyp_stack_info *info) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr_nvhe_sym(kvm_panic_info); + unsigned long low = (unsigned long)panic_info->hyp_overflow_stack_base; + unsigned long high = low + PAGE_SIZE; + + return __on_hyp_stack(hyp_sp, size, low, high, HYP_STACK_TYPE_OVERFLOW, info); +} + +static inline bool on_hyp_stack(unsigned long hyp_sp, unsigned long size, + struct hyp_stack_info *info) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr_nvhe_sym(kvm_panic_info); + unsigned long low = (unsigned long)panic_info->hyp_stack_base; + unsigned long high = low + PAGE_SIZE; + + return __on_hyp_stack(hyp_sp, size, low, high, HYP_STACK_TYPE_HYP, info); +} + +static inline bool on_hyp_accessible_stack(unsigned long hyp_sp, unsigned long size, + struct hyp_stack_info *info) +{ + if (info) + info->type = HYP_STACK_TYPE_UNKNOWN; + + if (on_hyp_stack(hyp_sp, size, info)) + return true; + if (on_hyp_overflow_stack(hyp_sp, size, info)) + return true; + + return false; +} + +static unsigned long __hyp_stack_kern_va(unsigned long hyp_va) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr_nvhe_sym(kvm_panic_info); + unsigned long hyp_base, kern_base, hyp_offset; + + hyp_base = (unsigned long)panic_info->hyp_stack_base; + hyp_offset = hyp_va - hyp_base; + + kern_base = (unsigned long)*this_cpu_ptr(&kvm_arm_hyp_stack_page); + + return kern_base + hyp_offset; +} + +static unsigned long __hyp_overflow_stack_kern_va(unsigned long hyp_va) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr_nvhe_sym(kvm_panic_info); + unsigned long hyp_base, kern_base, hyp_offset; + + hyp_base = (unsigned long)panic_info->hyp_overflow_stack_base; + hyp_offset = hyp_va - hyp_base; + + kern_base = (unsigned long)this_cpu_ptr_nvhe_sym(hyp_overflow_stack); + + return kern_base + hyp_offset; +} + +/* + * Convert hypervisor stack VA to a kernel VA. + * + * The hypervisor stack is mapped in the flexible 'private' VA range, to allow + * for guard pages below the stack. Consequently, the fixed offset address + * translation macros won't work here. + * + * The kernel VA is calculated as an offset from the kernel VA of the hypervisor + * stack base. See: __hyp_stack_kern_va(), __hyp_overflow_stack_kern_va() + */ +static unsigned long hyp_stack_kern_va(unsigned long hyp_va, + enum hyp_stack_type stack_type) +{ + switch (stack_type) { + case HYP_STACK_TYPE_HYP: + return __hyp_stack_kern_va(hyp_va); + case HYP_STACK_TYPE_OVERFLOW: + return __hyp_overflow_stack_kern_va(hyp_va); + default: + return 0UL; + } +} + +/* + * Unwind from one frame record (A) to the next frame record (B). + * + * We terminate early if the location of B indicates a malformed chain of frame + * records (e.g. a cycle), determined based on the location and fp value of A + * and the location (but not the fp value) of B. + */ +static int notrace hyp_unwind_frame(struct hyp_stackframe *frame) +{ + unsigned long fp = frame->fp, fp_kern_va; + struct hyp_stack_info info; + + if (fp & 0x7) + return -EINVAL; + + if (!on_hyp_accessible_stack(fp, 16, &info)) + return -EINVAL; + + if (test_bit(info.type, frame->stacks_done)) + return -EINVAL; + + /* + * As stacks grow downward, any valid record on the same stack must be + * at a strictly higher address than the prior record. + * + * Stacks can nest in the following order: + * + * HYP -> OVERFLOW + * + * ... but the nesting itself is strict. Once we transition from one + * stack to another, it's never valid to unwind back to that first + * stack. + */ + if (info.type == frame->prev_type) { + if (fp <= frame->prev_fp) + return -EINVAL; + } else { + set_bit(frame->prev_type, frame->stacks_done); + } + + /* Translate the hyp stack address to a kernel address */ + fp_kern_va = hyp_stack_kern_va(fp, info.type); + if (!fp_kern_va) + return -EINVAL; + + /* + * Record this frame record's values and location. The prev_fp and + * prev_type are only meaningful to the next hyp_unwind_frame() + * invocation. + */ + frame->fp = READ_ONCE_NOCHECK(*(unsigned long *)(fp_kern_va)); + /* PC = LR - 4; All aarch64 instructions are 32-bits in size */ + frame->pc = READ_ONCE_NOCHECK(*(unsigned long *)(fp_kern_va + 8)) - 4; + frame->prev_fp = fp; + frame->prev_type = info.type; + + return 0; +} + +/* + * AArch64 PCS assigns the frame pointer to x29. + * + * A simple function prologue looks like this: + * sub sp, sp, #0x10 + * stp x29, x30, [sp] + * mov x29, sp + * + * A simple function epilogue looks like this: + * mov sp, x29 + * ldp x29, x30, [sp] + * add sp, sp, #0x10 + */ +static void hyp_start_backtrace(struct hyp_stackframe *frame, unsigned long fp) +{ + frame->fp = fp; + + /* + * Prime the first unwind. + * + * In hyp_unwind_frame() we'll check that the FP points to a valid + * stack, which can't be HYP_STACK_TYPE_UNKNOWN, and the first unwind + * will be treated as a transition to whichever stack that happens to + * be. The prev_fp value won't be used, but we set it to 0 such that + * it is definitely not an accessible stack address. The first frame + * (hyp_panic()) is skipped, so we also set PC to 0. + */ + bitmap_zero(frame->stacks_done, __NR_HYP_STACK_TYPES); + frame->pc = frame->prev_fp = 0; + frame->prev_type = HYP_STACK_TYPE_UNKNOWN; +} + +static void hyp_dump_backtrace_entry(unsigned long hyp_pc, unsigned long hyp_offset) +{ + unsigned long va_mask = GENMASK_ULL(vabits_actual - 1, 0); + + hyp_pc &= va_mask; /* Mask tags */ + hyp_pc += hyp_offset; + + kvm_err("[<%016lx>]\n", hyp_pc); +} + +void hyp_dump_backtrace(unsigned long hyp_offset) +{ + struct kvm_nvhe_panic_info *panic_info = this_cpu_ptr_nvhe_sym(kvm_panic_info); + struct hyp_stackframe frame; + int frame_nr = 0; + int skip = 1; /* Skip the first frame: hyp_panic() */ + + kvm_err("nVHE HYP call trace:\n"); + + hyp_start_backtrace(&frame, (unsigned long)panic_info->start_fp); + + do { + if (skip) { + skip--; + continue; + } + + hyp_dump_backtrace_entry(frame.pc, hyp_offset); + + frame_nr++; + } while (!hyp_unwind_frame(&frame)); + + kvm_err("---- end of nVHE HYP call trace ----\n"); +} diff --git a/arch/arm64/kvm/stacktrace.h b/arch/arm64/kvm/stacktrace.h new file mode 100644 index 000000000000..40c397394b9b --- /dev/null +++ b/arch/arm64/kvm/stacktrace.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Stack unwinder for EL2 nVHE hypervisor. + */ + +#ifndef __KVM_HYP_STACKTRACE_H +#define __KVM_HYP_STACKTRACE_H + +#ifdef CONFIG_NVHE_EL2_DEBUG +void hyp_dump_backtrace(unsigned long hyp_offset); +#else +static inline void hyp_dump_backtrace(unsigned long hyp_offset) +{ +} +#endif /* CONFIG_NVHE_EL2_DEBUG */ + +#endif /* __KVM_HYP_STACKTRACE_H */ From patchwork Tue Feb 22 16:51:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 12755731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 55386C433EF for ; Tue, 22 Feb 2022 17:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ut7tpMJu2FEFvFYO9Q5H9YR66PwgNd7/1r4zmxukHxw=; b=2Cez3hxOKQrwb1 adDkXW/qyEL938XBk6iWhYJrLxEOKpmCVaoDDgVD3GcGD85tOj5VVnJHaewTwtpSz808Tn48oztUC pmsB53d69BnXtQodWQauejpKyQAzSMyVPf+OX/jWwBxRHrEpE8UgKXc9O37sFL87UGw5Y7Mbok2x8 Z5w/urY6Ll7Gi51rVgPmvwetVEJo3EE7ru0Mx6itKOqqvhuR0D2R+m7yre5vLNk+REY3jfCiIch9z 94RE8oYqGbR0Ye5CCxCAVzQIW9eCvCfWAKsgtKIR/PcnfVN0p6AyAKs546bUXBxnX0szoT0VlFQqR qNS1yuArvuW+f2rTyxgQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYnf-00Ave5-RF; Tue, 22 Feb 2022 17:18:08 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMYYs-00Aptr-Nu for linux-arm-kernel@lists.infradead.org; Tue, 22 Feb 2022 17:02:52 +0000 Received: by mail-yb1-xb49.google.com with SMTP id s22-20020a252d56000000b00624652ac3e1so9332341ybe.16 for ; Tue, 22 Feb 2022 09:02:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:cc; bh=beGtXg45IQ4dKCY/A8ewT2jAssjG8oCd/wil2F8VoF8=; b=Qmw8f9ngENw67mW7ZY926jso9fM6FSWsadBt2xuNCmEM/wL9yNNx+++6QAzW5Geysa X0CVpzvJkNzo1xClPSFGBfQoQ9e82eUu5kgF0cfza/NxsGQMydRbfBIhqKkBzLs5aCzK VBY/ce5OQiMkErnhinkSoy35Z7xGC9tfoOBy0pVxgyAmPZog1qhtESjBaWlKEIEeimgX bKnw6hLvfmjhuk5Ps7Kiuf9lL11CkBXEiVCRXEz1k0Pj8iOLK68P3f2mF/3G5iD/5IP2 26qsHRaykVbWjx15kawdg3HR402vdDmc9IFBtIz1CQ/1k/qSnS4QDbrQcuG21X7huvEk B9pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=beGtXg45IQ4dKCY/A8ewT2jAssjG8oCd/wil2F8VoF8=; b=YjsNK9ME0Zzlv98zcfmv5XnXDO4FHu/LGRmcrflUqI/Y9pxiER0zdicMMACWfIc0Tg S2ZPFCKMiJ++ZaSsNf+HcS56e0KzZ/3ucQwwO/JUsvzq8YyPLu7G3OutjdOLMQyvrIfT Uyd8fQXGV8nhl5sVD0PjcAaZT5GyBGldOb/MJg/bzMBguSoUEaFrlPENlm/lYaWl1IRf DvF+ogvkxowk10LXgTte6FDEXjb4vb/v+Xbm8SY4I9q0xPG+vne/p/ALRwdmruYWeVYj 7hVjqaVCatZKe/x3lLbQBCUfLXuI+VgmrbIfKUu67HCWV5iqmqes5Br8x0QlwR6EqbqX R+Lg== X-Gm-Message-State: AOAM5306D0DvQH4kR7C1JI2IMrYHBngxXM3SfrI7F3Gb2Ld0Esm4fuFC NqKMKBTNkmZ2/f8Le4I3rPBOSuqMXPJkTobDLA== X-Google-Smtp-Source: ABdhPJx9ya/dSuGbtyc2Drwg2dL1gTZ3RclR9peZ4NgVIVWBnHY+dyxNePng3kg7EOauiAGLQaMx/zndC7RygsTqtw== X-Received: from kaleshsingh.mtv.corp.google.com ([2620:15c:211:200:5db7:1235:b3dd:cfcb]) (user=kaleshsingh job=sendgmr) by 2002:a25:d60c:0:b0:610:dc8d:b3bd with SMTP id n12-20020a25d60c000000b00610dc8db3bdmr24399063ybg.561.1645549369167; Tue, 22 Feb 2022 09:02:49 -0800 (PST) Date: Tue, 22 Feb 2022 08:51:10 -0800 In-Reply-To: <20220222165212.2005066-1-kaleshsingh@google.com> Message-Id: <20220222165212.2005066-10-kaleshsingh@google.com> Mime-Version: 1.0 References: <20220222165212.2005066-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v2 9/9] KVM: arm64: Symbolize the nVHE HYP backtrace From: Kalesh Singh Cc: will@kernel.org, maz@kernel.org, qperret@google.com, tabba@google.com, surenb@google.com, kernel-team@android.com, Kalesh Singh , Catalin Marinas , James Morse , Alexandru Elisei , Suzuki K Poulose , Ard Biesheuvel , Mark Rutland , Pasha Tatashin , Joey Gouly , Peter Collingbourne , Andrew Walbran , Andrew Scull , Paolo Bonzini , Andrew Jones , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.cs.columbia.edu X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220222_090250_832031_708D13EA X-CRM114-Status: GOOD ( 15.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Reintroduce the __kvm_nvhe_ symbols in kallsyms, ignoring the local symbols in this namespace. The local symbols are not informative and can cause aliasing issues when symbolizing the addresses. With the necessary symbols now in kallsyms we can symbolize nVHE stacktrace addresses using the %pB print format specifier. Some sample call traces: ------- [ 167.018598][ T407] kvm [407]: nVHE hyp panic at: [] __kvm_nvhe_overflow_stack+0x10/0x34! [ 167.020841][ T407] kvm [407]: nVHE HYP call trace: [ 167.021371][ T407] kvm [407]: [] __kvm_nvhe_hyp_panic_bad_stack+0xc/0x10 [ 167.021972][ T407] kvm [407]: [] __kvm_nvhe___kvm_hyp_host_vector+0x248/0x794 [ 167.022572][ T407] kvm [407]: [] __kvm_nvhe_overflow_stack+0x20/0x34 [ 167.023135][ T407] kvm [407]: [] __kvm_nvhe_overflow_stack+0x20/0x34 [ 167.023699][ T407] kvm [407]: [] __kvm_nvhe_overflow_stack+0x20/0x34 [ 167.024261][ T407] kvm [407]: [] __kvm_nvhe_overflow_stack+0x20/0x34 . . . ------- [ 166.161699][ T409] kvm [409]: Invalid host exception to nVHE hyp! [ 166.163789][ T409] kvm [409]: nVHE HYP call trace: [ 166.164709][ T409] kvm [409]: [] __kvm_nvhe_handle___kvm_vcpu_run+0x198/0x21c [ 166.165352][ T409] kvm [409]: [] __kvm_nvhe_handle_trap+0xa4/0x124 [ 166.165911][ T409] kvm [409]: [] __kvm_nvhe___host_exit+0x60/0x64 [ 166.166657][ T409] Kernel panic - not syncing: HYP panic: . . . ------- Signed-off-by: Kalesh Singh --- Changes in v2: - Fix printk warnings - %p expects (void *) arch/arm64/kvm/handle_exit.c | 13 +++++-------- arch/arm64/kvm/stacktrace.c | 2 +- scripts/kallsyms.c | 2 +- 3 files changed, 7 insertions(+), 10 deletions(-) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index b038c32a3236..1b953005d301 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -296,13 +296,8 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_in_kimg = __phys_to_kimg(elr_phys); u64 hyp_offset = elr_in_kimg - kaslr_offset() - elr_virt; u64 mode = spsr & PSR_MODE_MASK; + u64 panic_addr = elr_virt + hyp_offset; - /* - * The nVHE hyp symbols are not included by kallsyms to avoid issues - * with aliasing. That means that the symbols cannot be printed with the - * "%pS" format specifier, so fall back to the vmlinux address if - * there's no better option. - */ if (mode != PSR_MODE_EL2t && mode != PSR_MODE_EL2h) { kvm_err("Invalid host exception to nVHE hyp!\n"); } else if (ESR_ELx_EC(esr) == ESR_ELx_EC_BRK64 && @@ -322,9 +317,11 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, if (file) kvm_err("nVHE hyp BUG at: %s:%u!\n", file, line); else - kvm_err("nVHE hyp BUG at: %016llx!\n", elr_virt + hyp_offset); + kvm_err("nVHE hyp BUG at: [<%016llx>] %pB!\n", panic_addr, + (void *)panic_addr); } else { - kvm_err("nVHE hyp panic at: %016llx!\n", elr_virt + hyp_offset); + kvm_err("nVHE hyp panic at: [<%016llx>] %pB!\n", panic_addr, + (void *)panic_addr); } hyp_dump_backtrace(hyp_offset); diff --git a/arch/arm64/kvm/stacktrace.c b/arch/arm64/kvm/stacktrace.c index cdd672bf0ea8..896c225a4a89 100644 --- a/arch/arm64/kvm/stacktrace.c +++ b/arch/arm64/kvm/stacktrace.c @@ -261,7 +261,7 @@ static void hyp_dump_backtrace_entry(unsigned long hyp_pc, unsigned long hyp_off hyp_pc &= va_mask; /* Mask tags */ hyp_pc += hyp_offset; - kvm_err("[<%016lx>]\n", hyp_pc); + kvm_err("[<%016lx>] %pB\n", hyp_pc, (void *)hyp_pc); } void hyp_dump_backtrace(unsigned long hyp_offset) diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c index 54ad86d13784..19aba43d9da4 100644 --- a/scripts/kallsyms.c +++ b/scripts/kallsyms.c @@ -111,7 +111,7 @@ static bool is_ignored_symbol(const char *name, char type) ".LASANPC", /* s390 kasan local symbols */ "__crc_", /* modversions */ "__efistub_", /* arm64 EFI stub namespace */ - "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */ + "__kvm_nvhe_$", /* arm64 local symbols in non-VHE KVM namespace */ "__AArch64ADRPThunk_", /* arm64 lld */ "__ARMV5PILongThunk_", /* arm lld */ "__ARMV7PILongThunk_",