From patchwork Wed May 1 09:44:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zenghui Yu X-Patchwork-Id: 10924747 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 361AC933 for ; Wed, 1 May 2019 09:48:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 24D7628BEB for ; Wed, 1 May 2019 09:48:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 16FCD28C35; Wed, 1 May 2019 09:48:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9BCB728BEB for ; Wed, 1 May 2019 09:48:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726427AbfEAJsq (ORCPT ); Wed, 1 May 2019 05:48:46 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:55026 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726224AbfEAJsR (ORCPT ); Wed, 1 May 2019 05:48:17 -0400 Received: from DGGEMS412-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id C4F005DC777B751C2BF8; Wed, 1 May 2019 17:48:14 +0800 (CST) Received: from HGHY2Y004646261.china.huawei.com (10.184.12.158) by DGGEMS412-HUB.china.huawei.com (10.3.19.212) with Microsoft SMTP Server id 14.3.439.0; Wed, 1 May 2019 17:48:07 +0800 From: Zenghui Yu To: , , , CC: , , , , , , , , , , Zenghui Yu Subject: [PATCH 1/5] KVM: arm/arm64: Introduce helpers for page table enties with contiguous bit Date: Wed, 1 May 2019 09:44:23 +0000 Message-ID: <1556703867-22396-2-git-send-email-yuzenghui@huawei.com> X-Mailer: git-send-email 2.6.4.windows.1 In-Reply-To: <1556703867-22396-1-git-send-email-yuzenghui@huawei.com> References: <1556703867-22396-1-git-send-email-yuzenghui@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.184.12.158] X-CFilter-Loop: Reflected Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Introduce helpers to manipulate stage2 page table entries - set contiguous bit in the entry and say whether this entry points to a contiguous block. The helpers are introduced in preparation for supporting contiguous hugepages at stage2. Signed-off-by: Zenghui Yu --- arch/arm/include/asm/kvm_mmu.h | 22 ++++++++++++++++++++++ arch/arm64/include/asm/kvm_mmu.h | 20 ++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 31de4ab..80d73ae 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -143,6 +143,28 @@ static inline bool kvm_s2pud_young(pud_t pud) return false; } +static inline pte_t kvm_s2pte_mkcont(pte_t pte) +{ + BUG(); + return pte; +} + +static inline bool kvm_s2pte_cont(pte_t pte) +{ + return false; +} + +static inline pmd_t kvm_s2pmd_mkcont(pmd_t pmd) +{ + BUG(); + return pmd; +} + +static inline bool kvm_s2pmd_cont(pmd_t pmd) +{ + return false; +} + static inline pte_t kvm_s2pte_mkwrite(pte_t pte) { pte_val(pte) |= L_PTE_S2_RDWR; diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index ebeefcf..4afdad9 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -295,6 +295,26 @@ static inline bool kvm_s2pud_young(pud_t pud) return pud_young(pud); } +static inline pte_t kvm_s2pte_mkcont(pte_t pte) +{ + return pte_mkcont(pte); +} + +static inline bool kvm_s2pte_cont(pte_t pte) +{ + return pte_cont(pte); +} + +static inline pmd_t kvm_s2pmd_mkcont(pmd_t pmd) +{ + return pmd_mkcont(pmd); +} + +static inline bool kvm_s2pmd_cont(pmd_t pmd) +{ + return !!(pmd_val(pmd) & PMD_SECT_CONT); +} + #define hyp_pte_table_empty(ptep) kvm_page_empty(ptep) #ifdef __PAGETABLE_PMD_FOLDED From patchwork Wed May 1 09:44:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zenghui Yu X-Patchwork-Id: 10924751 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18FFF92A for ; Wed, 1 May 2019 09:48:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0797428BEB for ; Wed, 1 May 2019 09:48:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F033928C2C; Wed, 1 May 2019 09:48:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 88A4828BEB for ; Wed, 1 May 2019 09:48:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726276AbfEAJsR (ORCPT ); Wed, 1 May 2019 05:48:17 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:55028 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726193AbfEAJsR (ORCPT ); Wed, 1 May 2019 05:48:17 -0400 Received: from DGGEMS412-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id CCCB721BE0C0F1547E36; Wed, 1 May 2019 17:48:14 +0800 (CST) Received: from HGHY2Y004646261.china.huawei.com (10.184.12.158) by DGGEMS412-HUB.china.huawei.com (10.3.19.212) with Microsoft SMTP Server id 14.3.439.0; Wed, 1 May 2019 17:48:08 +0800 From: Zenghui Yu To: , , , CC: , , , , , , , , , , Zenghui Yu Subject: [PATCH 2/5] KVM: arm/arm64: Re-factor building the stage2 page table entries Date: Wed, 1 May 2019 09:44:24 +0000 Message-ID: <1556703867-22396-3-git-send-email-yuzenghui@huawei.com> X-Mailer: git-send-email 2.6.4.windows.1 In-Reply-To: <1556703867-22396-1-git-send-email-yuzenghui@huawei.com> References: <1556703867-22396-1-git-send-email-yuzenghui@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.184.12.158] X-CFilter-Loop: Reflected Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP As we're going to support creating CONT_{PTE,PMD}_SIZE huge mappings in user_mem_abort(), the logic to check vma_pagesize and build page table entries will become longer, and looks almost the same (but actually they dont). Refactor this part to make it a bit cleaner. Add contiguous as a parameter of stage2_build_{pmd,pte}, to indicate if we're creating contiguous huge mappings. Signed-off-by: Zenghui Yu --- virt/kvm/arm/mmu.c | 81 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 57 insertions(+), 24 deletions(-) diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 27c9583..cf8b035 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -1616,6 +1616,56 @@ static void kvm_send_hwpoison_signal(unsigned long address, send_sig_mceerr(BUS_MCEERR_AR, (void __user *)address, lsb, current); } +static pud_t stage2_build_pud(kvm_pfn_t pfn, pgprot_t mem_type, bool writable, + bool needs_exec) +{ + pud_t new_pud = kvm_pfn_pud(pfn, mem_type); + + new_pud = kvm_pud_mkhuge(new_pud); + if (writable) + new_pud = kvm_s2pud_mkwrite(new_pud); + + if (needs_exec) + new_pud = kvm_s2pud_mkexec(new_pud); + + return new_pud; +} + +static pmd_t stage2_build_pmd(kvm_pfn_t pfn, pgprot_t mem_type, bool writable, + bool needs_exec, bool contiguous) +{ + pmd_t new_pmd = kvm_pfn_pmd(pfn, mem_type); + + new_pmd = kvm_pmd_mkhuge(new_pmd); + if (writable) + new_pmd = kvm_s2pmd_mkwrite(new_pmd); + + if (needs_exec) + new_pmd = kvm_s2pmd_mkexec(new_pmd); + + if (contiguous) + new_pmd = kvm_s2pmd_mkcont(new_pmd); + + return new_pmd; +} + +static pte_t stage2_build_pte(kvm_pfn_t pfn, pgprot_t mem_type, bool writable, + bool needs_exec, bool contiguous) +{ + pte_t new_pte = kvm_pfn_pte(pfn, mem_type); + + if (writable) + new_pte = kvm_s2pte_mkwrite(new_pte); + + if (needs_exec) + new_pte = kvm_s2pte_mkexec(new_pte); + + if (contiguous) + new_pte = kvm_s2pte_mkcont(new_pte); + + return new_pte; +} + static bool fault_supports_stage2_huge_mapping(struct kvm_memory_slot *memslot, unsigned long hva, unsigned long map_size) @@ -1807,38 +1857,21 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, (fault_status == FSC_PERM && stage2_is_exec(kvm, fault_ipa)); if (vma_pagesize == PUD_SIZE) { - pud_t new_pud = kvm_pfn_pud(pfn, mem_type); - - new_pud = kvm_pud_mkhuge(new_pud); - if (writable) - new_pud = kvm_s2pud_mkwrite(new_pud); - - if (needs_exec) - new_pud = kvm_s2pud_mkexec(new_pud); + pud_t new_pud = stage2_build_pud(pfn, mem_type, writable, + needs_exec); ret = stage2_set_pud_huge(kvm, memcache, fault_ipa, &new_pud); } else if (vma_pagesize == PMD_SIZE) { - pmd_t new_pmd = kvm_pfn_pmd(pfn, mem_type); - - new_pmd = kvm_pmd_mkhuge(new_pmd); - - if (writable) - new_pmd = kvm_s2pmd_mkwrite(new_pmd); - - if (needs_exec) - new_pmd = kvm_s2pmd_mkexec(new_pmd); + pmd_t new_pmd = stage2_build_pmd(pfn, mem_type, writable, + needs_exec, false); ret = stage2_set_pmd_huge(kvm, memcache, fault_ipa, &new_pmd); } else { - pte_t new_pte = kvm_pfn_pte(pfn, mem_type); + pte_t new_pte = stage2_build_pte(pfn, mem_type, writable, + needs_exec, false); - if (writable) { - new_pte = kvm_s2pte_mkwrite(new_pte); + if (writable) mark_page_dirty(kvm, gfn); - } - - if (needs_exec) - new_pte = kvm_s2pte_mkexec(new_pte); ret = stage2_set_pte(kvm, memcache, fault_ipa, &new_pte, flags); } From patchwork Wed May 1 09:44:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zenghui Yu X-Patchwork-Id: 10924741 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9021B933 for ; Wed, 1 May 2019 09:48:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7EB1228BEB for ; Wed, 1 May 2019 09:48:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7293928C2C; Wed, 1 May 2019 09:48:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0553028BEB for ; Wed, 1 May 2019 09:48:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726363AbfEAJsW (ORCPT ); Wed, 1 May 2019 05:48:22 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:37752 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726351AbfEAJsW (ORCPT ); Wed, 1 May 2019 05:48:22 -0400 Received: from DGGEMS412-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id CEDEF9F163E2777B5360; Wed, 1 May 2019 17:48:19 +0800 (CST) Received: from HGHY2Y004646261.china.huawei.com (10.184.12.158) by DGGEMS412-HUB.china.huawei.com (10.3.19.212) with Microsoft SMTP Server id 14.3.439.0; Wed, 1 May 2019 17:48:09 +0800 From: Zenghui Yu To: , , , CC: , , , , , , , , , , Zenghui Yu Subject: [PATCH 3/5] KVM: arm/arm64: Support dirty page tracking for contiguous hugepages Date: Wed, 1 May 2019 09:44:25 +0000 Message-ID: <1556703867-22396-4-git-send-email-yuzenghui@huawei.com> X-Mailer: git-send-email 2.6.4.windows.1 In-Reply-To: <1556703867-22396-1-git-send-email-yuzenghui@huawei.com> References: <1556703867-22396-1-git-send-email-yuzenghui@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.184.12.158] X-CFilter-Loop: Reflected Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When logging is enabled, we should keep tracking of normal size pages. If the memslot is backed by contiguous hugepages, we should dissolve the contiguous hugepages - clear the whole contiguous range and flush corresponding TLB entry. Move kvm_set_{pte,pmd} to the front of stage2_dissolve_cont_{pte,pmd}s to avoid one more declaration. Signed-off-by: Zenghui Yu --- arch/arm/include/asm/pgtable-hwdef.h | 8 ++++ virt/kvm/arm/mmu.c | 76 ++++++++++++++++++++++++++++++------ 2 files changed, 72 insertions(+), 12 deletions(-) diff --git a/arch/arm/include/asm/pgtable-hwdef.h b/arch/arm/include/asm/pgtable-hwdef.h index 8426229..41f4633 100644 --- a/arch/arm/include/asm/pgtable-hwdef.h +++ b/arch/arm/include/asm/pgtable-hwdef.h @@ -16,4 +16,12 @@ #include #endif +/* dummy definition */ +#define CONT_PTES (0) +#define CONT_PTE_SIZE (0) +#define CONT_PTE_MASK (0) +#define CONT_PMDS (0) +#define CONT_PMD_SIZE (0) +#define CONT_PMD_MASK (0) + #endif diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index cf8b035..36c863f 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -96,6 +96,58 @@ static bool kvm_is_device_pfn(unsigned long pfn) return !pfn_valid(pfn); } +static inline void kvm_set_pte(pte_t *ptep, pte_t new_pte) +{ + WRITE_ONCE(*ptep, new_pte); + dsb(ishst); +} + +static inline void kvm_set_pmd(pmd_t *pmdp, pmd_t new_pmd) +{ + WRITE_ONCE(*pmdp, new_pmd); + dsb(ishst); +} + +static void stage2_dissolve_cont_ptes(struct kvm *kvm, phys_addr_t addr, + pte_t *pte) +{ + phys_addr_t addrp; + pte_t *ptep; + int i; + + if (!kvm_s2pte_cont(*pte)) + return; + + /* Start with the first pte */ + addrp = addr & CONT_PTE_MASK; + ptep = pte - (addr - addrp) / PAGE_SIZE; + + for (i = 0; i < CONT_PTES; i++, ptep++, addrp += PAGE_SIZE) { + kvm_set_pte(ptep, __pte(0)); + kvm_tlb_flush_vmid_ipa(kvm, addrp); + put_page(virt_to_page(ptep)); + } +} + +static void stage2_dissolve_cont_pmds(struct kvm *kvm, phys_addr_t addr, + pmd_t *pmd) +{ + phys_addr_t addrp; + pmd_t *pmdp; + int i; + + /* Start with the first pmd */ + addr &= PMD_MASK; + addrp = addr & CONT_PMD_MASK; + pmdp = pmd - (addr - addrp) / PMD_SIZE; + + for (i = 0; i < CONT_PMDS; i++, pmdp++, addrp += PMD_SIZE) { + pmd_clear(pmdp); + kvm_tlb_flush_vmid_ipa(kvm, addrp); + put_page(virt_to_page(pmdp)); + } +} + /** * stage2_dissolve_pmd() - clear and flush huge PMD entry * @kvm: pointer to kvm structure. @@ -109,6 +161,11 @@ static void stage2_dissolve_pmd(struct kvm *kvm, phys_addr_t addr, pmd_t *pmd) if (!pmd_thp_or_huge(*pmd)) return; + if (kvm_s2pmd_cont(*pmd)) { + stage2_dissolve_cont_pmds(kvm, addr, pmd); + return; + } + pmd_clear(pmd); kvm_tlb_flush_vmid_ipa(kvm, addr); put_page(virt_to_page(pmd)); @@ -193,18 +250,6 @@ static void clear_stage2_pmd_entry(struct kvm *kvm, pmd_t *pmd, phys_addr_t addr put_page(virt_to_page(pmd)); } -static inline void kvm_set_pte(pte_t *ptep, pte_t new_pte) -{ - WRITE_ONCE(*ptep, new_pte); - dsb(ishst); -} - -static inline void kvm_set_pmd(pmd_t *pmdp, pmd_t new_pmd) -{ - WRITE_ONCE(*pmdp, new_pmd); - dsb(ishst); -} - static inline void kvm_pmd_populate(pmd_t *pmdp, pte_t *ptep) { kvm_set_pmd(pmdp, kvm_mk_pmd(ptep)); @@ -1289,6 +1334,13 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, pte = pte_offset_kernel(pmd, addr); + /* + * While dirty page logging - dissolve contiguous PTE pages, then + * continue on to allocate page. + */ + if (logging_active) + stage2_dissolve_cont_ptes(kvm, addr, pte); + if (iomap && pte_present(*pte)) return -EFAULT; From patchwork Wed May 1 09:44:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zenghui Yu X-Patchwork-Id: 10924737 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A08592A for ; Wed, 1 May 2019 09:48:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3810C28BEB for ; Wed, 1 May 2019 09:48:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2BFE528C2C; Wed, 1 May 2019 09:48:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B358128BEB for ; Wed, 1 May 2019 09:48:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726382AbfEAJsX (ORCPT ); Wed, 1 May 2019 05:48:23 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:37756 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726349AbfEAJsV (ORCPT ); Wed, 1 May 2019 05:48:21 -0400 Received: from DGGEMS412-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id D711E68B5D98E459F02A; Wed, 1 May 2019 17:48:19 +0800 (CST) Received: from HGHY2Y004646261.china.huawei.com (10.184.12.158) by DGGEMS412-HUB.china.huawei.com (10.3.19.212) with Microsoft SMTP Server id 14.3.439.0; Wed, 1 May 2019 17:48:10 +0800 From: Zenghui Yu To: , , , CC: , , , , , , , , , , Zenghui Yu Subject: [PATCH 4/5] KVM: arm/arm64: Add support for creating PTE contiguous hugepages at stage2 Date: Wed, 1 May 2019 09:44:26 +0000 Message-ID: <1556703867-22396-5-git-send-email-yuzenghui@huawei.com> X-Mailer: git-send-email 2.6.4.windows.1 In-Reply-To: <1556703867-22396-1-git-send-email-yuzenghui@huawei.com> References: <1556703867-22396-1-git-send-email-yuzenghui@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.184.12.158] X-CFilter-Loop: Reflected Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Extend the stage2 fault handling to map in PTE contiguous hugepages first. With this patch, we now support additional following page size at stage2: CONT PTE -------- 4K granule: 64K 16K granule: 2M 64K granule: 2M The PMD contiguous huge mapping is still not supported yet and will be simply fall back to a PTE mapping. Signed-off-by: Zenghui Yu --- virt/kvm/arm/mmu.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-) diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 36c863f..fdd6314 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -1104,6 +1104,27 @@ static pmd_t *stage2_get_pmd(struct kvm *kvm, struct kvm_mmu_memory_cache *cache return stage2_pmd_offset(kvm, pud, addr); } +static pte_t *stage2_get_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, + phys_addr_t addr) +{ + pmd_t *pmd; + pte_t *pte; + + pmd = stage2_get_pmd(kvm, cache, addr); + if (!pmd || pmd_thp_or_huge(*pmd)) + return NULL; + + if (pmd_none(*pmd)) { + if (!cache) + return NULL; + pte = mmu_memory_cache_alloc(cache); + kvm_pmd_populate(pmd, pte); + get_page(virt_to_page(pmd)); + } + + return pte_offset_kernel(pmd, addr); +} + static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, phys_addr_t addr, const pmd_t *new_pmd) { @@ -1270,6 +1291,54 @@ static bool stage2_is_exec(struct kvm *kvm, phys_addr_t addr) return kvm_s2pte_exec(ptep); } +/* This code is borrowed from arch/arm64/mm/hugetlbpage.c */ +static inline pgprot_t pte_pgprot(pte_t pte) +{ + unsigned long pfn = pte_pfn(pte); + + return __pgprot(pte_val(pfn_pte(pfn, __pgprot(0))) ^ pte_val(pte)); +} + +static int stage2_set_cont_ptes(struct kvm *kvm, struct kvm_mmu_memory_cache + *cache, phys_addr_t addr, const pte_t *new_pte) +{ + pte_t *pte, old_pte; + unsigned long pfn, dpfn; + int i; + pgprot_t hugeprot; + + /* + * Make sure we start manipulating the first pte + * within CONT_PTES ptes. + */ + addr &= CONT_PTE_MASK; + pfn = pte_pfn(*new_pte); + dpfn = PAGE_SIZE >> PAGE_SHIFT; + hugeprot = pte_pgprot(*new_pte); + + pte = stage2_get_pte(kvm, cache, addr); + VM_BUG_ON(!pte); + + old_pte = *pte; + + /* Skip page table update if there is no change */ + if (pte_val(old_pte) == pte_val(*new_pte)) + return 0; + + for (i = 0; i < CONT_PTES; i++, pte++, addr += PAGE_SIZE, pfn += dpfn) { + if (pte_present(old_pte)) { + kvm_set_pte(pte, __pte(0)); + kvm_tlb_flush_vmid_ipa(kvm, addr); + } else { + get_page(virt_to_page(pte)); + } + + kvm_set_pte(pte, kvm_pfn_pte(pfn, hugeprot)); + } + + return 0; +} + static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, phys_addr_t addr, const pte_t *new_pte, unsigned long flags) @@ -1824,7 +1893,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * As for PUD huge maps, we must make sure that we have at least * 3 levels, i.e, PMD is not folded. */ - if (vma_pagesize == PMD_SIZE || + if (vma_pagesize == CONT_PTE_SIZE || vma_pagesize == PMD_SIZE || (vma_pagesize == PUD_SIZE && kvm_stage2_has_pmd(kvm))) gfn = (fault_ipa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT; up_read(¤t->mm->mmap_sem); @@ -1918,6 +1987,11 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, needs_exec, false); ret = stage2_set_pmd_huge(kvm, memcache, fault_ipa, &new_pmd); + } else if (vma_pagesize == CONT_PTE_SIZE) { + pte_t new_pte = stage2_build_pte(pfn, mem_type, writable, + needs_exec, true); + + ret = stage2_set_cont_ptes(kvm, memcache, fault_ipa, &new_pte); } else { pte_t new_pte = stage2_build_pte(pfn, mem_type, writable, needs_exec, false); From patchwork Wed May 1 09:44:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zenghui Yu X-Patchwork-Id: 10924739 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DEA492A for ; Wed, 1 May 2019 09:48:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0CF7728BEB for ; Wed, 1 May 2019 09:48:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 00C5E28C2C; Wed, 1 May 2019 09:48:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 90A8C28BEB for ; Wed, 1 May 2019 09:48:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726376AbfEAJsX (ORCPT ); Wed, 1 May 2019 05:48:23 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:37748 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726336AbfEAJsV (ORCPT ); Wed, 1 May 2019 05:48:21 -0400 Received: from DGGEMS412-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id C6025F73C997A760BED0; Wed, 1 May 2019 17:48:19 +0800 (CST) Received: from HGHY2Y004646261.china.huawei.com (10.184.12.158) by DGGEMS412-HUB.china.huawei.com (10.3.19.212) with Microsoft SMTP Server id 14.3.439.0; Wed, 1 May 2019 17:48:10 +0800 From: Zenghui Yu To: , , , CC: , , , , , , , , , , Zenghui Yu Subject: [PATCH 5/5] KVM: arm/arm64: Add support for creating PMD contiguous hugepages at stage2 Date: Wed, 1 May 2019 09:44:27 +0000 Message-ID: <1556703867-22396-6-git-send-email-yuzenghui@huawei.com> X-Mailer: git-send-email 2.6.4.windows.1 In-Reply-To: <1556703867-22396-1-git-send-email-yuzenghui@huawei.com> References: <1556703867-22396-1-git-send-email-yuzenghui@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.184.12.158] X-CFilter-Loop: Reflected Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP With this patch, we now support PMD contiguous hugepages at stage2, with following additional page size at stage2: CONT PMD -------- 4K granule: 32M 16K granule: 1G 64K granule: 16G Signed-off-by: Zenghui Yu --- virt/kvm/arm/mmu.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index fdd6314..7173589 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -1125,6 +1125,66 @@ static pte_t *stage2_get_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache return pte_offset_kernel(pmd, addr); } +static inline pgprot_t pmd_pgprot(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + return __pgprot(pmd_val(pfn_pmd(pfn, __pgprot(0))) ^ pmd_val(pmd)); +} + +static int stage2_set_cont_pmds(struct kvm *kvm, struct kvm_mmu_memory_cache + *cache, phys_addr_t addr, const pmd_t *new_pmd) +{ + pmd_t *pmd, old_pmd; + unsigned long pfn, dpfn; + int i; + pgprot_t hugeprot; + phys_addr_t baddr; + + /* Start with the first pmd. */ + addr &= CONT_PMD_MASK; + pfn = pmd_pfn(*new_pmd); + dpfn = PMD_SIZE >> PAGE_SHIFT; + hugeprot = pmd_pgprot(*new_pmd); + +retry: + pmd = stage2_get_pmd(kvm, cache, addr); + VM_BUG_ON(!pmd); + + old_pmd = *pmd; + + /* Skip page table update if there is no change */ + if (pmd_val(old_pmd) == pmd_val(*new_pmd)) + return 0; + + /* + * baddr and the following loop is for only one scenario: + * logging cancel ... Can we do it better? + */ + baddr = addr; + for (i = 0; i < CONT_PMDS; i++, pmd++, baddr += PMD_SIZE) { + if (pmd_present(*pmd) && !pmd_thp_or_huge(*pmd)) { + unmap_stage2_range(kvm, baddr, S2_PMD_SIZE); + goto retry; + } + } + + pmd = stage2_get_pmd(kvm, cache, addr); + + for (i = 0; i < CONT_PMDS; i++, pmd++, addr += PMD_SIZE, pfn += dpfn) { + if (pmd_present(old_pmd)) { + pmd_clear(pmd); + kvm_tlb_flush_vmid_ipa(kvm, addr); + } else { + get_page(virt_to_page(pmd)); + } + + kvm_set_pmd(pmd, pfn_pmd(pfn, hugeprot)); + } + + return 0; +} + static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, phys_addr_t addr, const pmd_t *new_pmd) { @@ -1894,6 +1954,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * 3 levels, i.e, PMD is not folded. */ if (vma_pagesize == CONT_PTE_SIZE || vma_pagesize == PMD_SIZE || + vma_pagesize == CONT_PMD_SIZE || (vma_pagesize == PUD_SIZE && kvm_stage2_has_pmd(kvm))) gfn = (fault_ipa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT; up_read(¤t->mm->mmap_sem); @@ -1982,6 +2043,11 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, needs_exec); ret = stage2_set_pud_huge(kvm, memcache, fault_ipa, &new_pud); + } else if (vma_pagesize == CONT_PMD_SIZE) { + pmd_t new_pmd = stage2_build_pmd(pfn, mem_type, writable, + needs_exec, true); + + ret = stage2_set_cont_pmds(kvm, memcache, fault_ipa, &new_pmd); } else if (vma_pagesize == PMD_SIZE) { pmd_t new_pmd = stage2_build_pmd(pfn, mem_type, writable, needs_exec, false);