From patchwork Wed Jan 10 19:07:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Punit Agrawal X-Patchwork-Id: 10155927 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 48B10601A1 for ; Wed, 10 Jan 2018 19:09:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B38D26B41 for ; Wed, 10 Jan 2018 19:09:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2FA6628595; Wed, 10 Jan 2018 19:09:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 921E3283CB for ; Wed, 10 Jan 2018 19:09:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Ufh8icQEjIyh5E9WyU/5nNKT8rWKzaP11ppxBRnRpZw=; b=Mc3EfSzwQOrZID3ZCWrTG7PM+N WOzjlmWUo8FQQFSCyx2MCQol9QJRBoPRK7rly7DoA6C0b+i7mx07kMhZ9m0m5ofMX/Lj2g9Gh6GDc rRIVQAO1kBFinL5nnYt7a9oM0nVxwDbXozKgmPQ9S1jLbDN8XDsqOhka0lnXff15Ot22D8t5PuTCR /NsGgY0L872LFiIR8kBB6DKykBReAEoOBcpQdhPmJk0aShShVd40RsxgagfKIhz+XxjVntk67SAVp 7BvDvzSbg8JCW8HGCIZ8ewcfO6IkM5TCv99UmVNNIoNkmP6UfKdxd2b4O3YtsfwCH4p+odpuIHGXy 5ff8HDTA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.89 #1 (Red Hat Linux)) id 1eZLkP-0003Ss-R2; Wed, 10 Jan 2018 19:09:13 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.89 #1 (Red Hat Linux)) id 1eZLjs-0002hX-BS for linux-arm-kernel@lists.infradead.org; Wed, 10 Jan 2018 19:08:58 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 678DA15A2; Wed, 10 Jan 2018 11:08:38 -0800 (PST) Received: from localhost (e105922-lin.cambridge.arm.com [10.1.207.29]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0B6CD3F581; Wed, 10 Jan 2018 11:08:37 -0800 (PST) From: Punit Agrawal To: kvmarm@lists.cs.columbia.edu Subject: [RFC 4/4] KVM: arm64: Add support for PUD hugepages at stage 2 Date: Wed, 10 Jan 2018 19:07:29 +0000 Message-Id: <20180110190729.18383-5-punit.agrawal@arm.com> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180110190729.18383-1-punit.agrawal@arm.com> References: <20180110190729.18383-1-punit.agrawal@arm.com> X-ARM-No-Footer: FoSSMail X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180110_110840_914979_2F370D22 X-CRM114-Status: GOOD ( 15.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: suzuki.poulose@arm.com, Marc Zyngier , Catalin Marinas , Punit Agrawal , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Christoffer Dall MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP KVM only supports PMD hugepages at stage 2. Extend the stage 2 fault handling to add support for PUD hugepages. Addition of PUD hugpage support enables additional hugepage sizes (1G with 4K granule and 4TB with 64k granule) which can be useful on cores that have support for mapping larger block sizes in the TLB entries. Signed-off-by: Punit Agrawal Cc: Marc Zyngier Cc: Christoffer Dall Cc: Catalin Marinas --- arch/arm/include/asm/kvm_mmu.h | 10 +++++ arch/arm/include/asm/pgtable-3level.h | 2 + arch/arm64/include/asm/kvm_mmu.h | 19 +++++++++ arch/arm64/include/asm/pgtable-hwdef.h | 2 + arch/arm64/include/asm/pgtable.h | 4 ++ virt/kvm/arm/mmu.c | 72 +++++++++++++++++++++++++++++----- 6 files changed, 99 insertions(+), 10 deletions(-) diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 3fbe919b9181..6e2e34348cb3 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -59,6 +59,10 @@ phys_addr_t kvm_get_idmap_vector(void); int kvm_mmu_init(void); void kvm_clear_hyp_idmap(void); +static inline void kvm_set_pud(pud_t *pud, pud_t new_pud) +{ +} + static inline void kvm_set_pmd(pmd_t *pmd, pmd_t new_pmd) { *pmd = new_pmd; @@ -230,6 +234,12 @@ static inline unsigned int kvm_get_vmid_bits(void) return 8; } +static inline pud_t stage2_build_pud(kvm_pfn_t pfn, pgprot_t mem_type, + bool writable) +{ + return __pud(0); +} + #endif /* !__ASSEMBLY__ */ #endif /* __ARM_KVM_MMU_H__ */ diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h index 1a7a17b2a1ba..97e04fdbfa85 100644 --- a/arch/arm/include/asm/pgtable-3level.h +++ b/arch/arm/include/asm/pgtable-3level.h @@ -249,6 +249,8 @@ PMD_BIT_FUNC(mkyoung, |= PMD_SECT_AF); #define pfn_pmd(pfn,prot) (__pmd(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))) #define mk_pmd(page,prot) pfn_pmd(page_to_pfn(page),prot) +#define pud_pfn(pud) (((pud_val(pud) & PUD_MASK) & PHYS_MASK) >> PAGE_SHIFT) + /* represent a notpresent pmd by faulting entry, this is used by pmdp_invalidate */ static inline pmd_t pmd_mknotpresent(pmd_t pmd) { diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index dbfd18e08cfb..89eac3dbe123 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -160,6 +160,7 @@ void kvm_clear_hyp_idmap(void); #define kvm_set_pte(ptep, pte) set_pte(ptep, pte) #define kvm_set_pmd(pmdp, pmd) set_pmd(pmdp, pmd) +#define kvm_set_pud(pudp, pud) set_pud(pudp, pud) static inline pte_t kvm_s2pte_mkwrite(pte_t pte) { @@ -173,6 +174,12 @@ static inline pmd_t kvm_s2pmd_mkwrite(pmd_t pmd) return pmd; } +static inline pud_t kvm_s2pud_mkwrite(pud_t pud) +{ + pud_val(pud) |= PUD_S2_RDWR; + return pud; +} + static inline void kvm_set_s2pte_readonly(pte_t *pte) { pteval_t old_pteval, pteval; @@ -319,5 +326,17 @@ static inline unsigned int kvm_get_vmid_bits(void) return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8; } +static inline pud_t stage2_build_pud(kvm_pfn_t pfn, pgprot_t mem_type, + bool writable) +{ + pud_t pud = pfn_pud(pfn, mem_type); + + pud = pud_mkhuge(pud); + if (writable) + pud = kvm_s2pud_mkwrite(pud); + + return pud; +} + #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index 40a998cdd399..a091a6192eee 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -181,6 +181,8 @@ #define PMD_S2_RDONLY (_AT(pmdval_t, 1) << 6) /* HAP[2:1] */ #define PMD_S2_RDWR (_AT(pmdval_t, 3) << 6) /* HAP[2:1] */ +#define PUD_S2_RDWR (_AT(pudval_t, 3) << 6) /* HAP[2:1] */ + /* * Memory Attribute override for Stage-2 (MemAttr[3:0]) */ diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index bdcc7f1c9d06..d5ffff4369d2 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -362,7 +362,11 @@ static inline int pmd_protnone(pmd_t pmd) #define mk_pmd(page,prot) pfn_pmd(page_to_pfn(page),prot) #define pud_write(pud) pte_write(pud_pte(pud)) + +#define pud_mkhuge(pud) (__pud(pud_val(pud) & ~PUD_TABLE_BIT)) + #define pud_pfn(pud) (((pud_val(pud) & PUD_MASK) & PHYS_MASK) >> PAGE_SHIFT) +#define pfn_pud(pfn, prot) (__pud(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))) #define set_pmd_at(mm, addr, pmdp, pmd) set_pte_at(mm, addr, (pte_t *)pmdp, pmd_pte(pmd)) diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index f02219a91b19..5362de098768 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -872,6 +872,32 @@ static pud_t *stage2_get_pud(struct kvm *kvm, struct kvm_mmu_memory_cache *cache return stage2_pud_offset(pgd, addr); } +static int stage2_set_pud_huge(struct kvm *kvm, struct kvm_mmu_memory_cache + *cache, phys_addr_t addr, const pud_t *new_pud) +{ + pud_t *pud, old_pud; + + pud = stage2_get_pud(kvm, cache, addr); + VM_BUG_ON(!pud); + + /* + * Mapping in huge pages should only happen through a fault. + */ + VM_BUG_ON(stage2_pud_present(*pud) && + pud_pfn(*pud) != pud_pfn(*new_pud)); + + old_pud = *pud; + if (stage2_pud_present(old_pud)) { + stage2_pud_clear(pud); + kvm_tlb_flush_vmid_ipa(kvm, addr); + } else { + get_page(virt_to_page(pud)); + } + + kvm_set_pud(pud, *new_pud); + return 0; +} + static pmd_t *stage2_get_pmd(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, phys_addr_t addr) { @@ -1307,6 +1333,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, kvm_pfn_t pfn; pgprot_t mem_type = PAGE_S2; bool logging_active = memslot_is_logging(memslot); + unsigned long vma_pagesize; unsigned long flags = 0; write_fault = kvm_is_write_fault(vcpu); @@ -1324,9 +1351,13 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, return -EFAULT; } - if (vma_kernel_pagesize(vma) == PMD_SIZE && !logging_active) { + vma_pagesize = vma_kernel_pagesize(vma); + if ((vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE) && + !logging_active) { + struct hstate *h = hstate_vma(vma); + hugetlb = true; - gfn = (fault_ipa & PMD_MASK) >> PAGE_SHIFT; + gfn = (fault_ipa & huge_page_mask(h)) >> PAGE_SHIFT; } else { /* * Pages belonging to memslots that don't have the same @@ -1393,17 +1424,38 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (mmu_notifier_retry(kvm, mmu_seq)) goto out_unlock; - if (!hugetlb && !force_pte) + if (!hugetlb && !force_pte) { + /* + * We only support PMD_SIZE transparent + * hugepages. This code will need updates if we enable + * other page sizes for THP. + */ hugetlb = transparent_hugepage_adjust(&pfn, &fault_ipa); + vma_pagesize = PMD_SIZE; + } if (hugetlb) { - pmd_t new_pmd = stage2_build_pmd(pfn, mem_type, writable); - - if (writable) - kvm_set_pfn_dirty(pfn); - - coherent_cache_guest_page(vcpu, pfn, PMD_SIZE); - ret = stage2_set_pmd_huge(kvm, memcache, fault_ipa, &new_pmd); + if (vma_pagesize == PUD_SIZE) { + pud_t new_pud; + + new_pud = stage2_build_pud(pfn, mem_type, writable); + if (writable) + kvm_set_pfn_dirty(pfn); + + coherent_cache_guest_page(vcpu, pfn, PUD_SIZE); + ret = stage2_set_pud_huge(kvm, memcache, + fault_ipa, &new_pud); + } else { + pmd_t new_pmd; + + new_pmd = stage2_build_pmd(pfn, mem_type, writable); + if (writable) + kvm_set_pfn_dirty(pfn); + + coherent_cache_guest_page(vcpu, pfn, PMD_SIZE); + ret = stage2_set_pmd_huge(kvm, memcache, + fault_ipa, &new_pmd); + } } else { pte_t new_pte = pfn_pte(pfn, mem_type);