From patchwork Thu Jul 2 13:55:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11638989 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 89567618 for ; Thu, 2 Jul 2020 13:58:48 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 62F6020772 for ; Thu, 2 Jul 2020 13:58:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="yb3njlnC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 62F6020772 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=PjVQeh2GFgqJhQvstURyO0j68Bo4q0xpBCeo7kfvJKE=; b=yb3njlnCf+2PU2cEjV1nF7W1g qqDwoybLPghBX/EoG9Qu4wuEhc9EVd3/tJi/KA0JSlPQ8SbYC/A+RJjno0CiubMawlCGGuVIVbysF kpnkgzBryy26ABA3/k0xbIorXZ8ZwSKCQjfsovZdJC7V2BzZuHgQQIAOsZz/RW4rfPl1GaMvosWDL Lu3AV/V8H73ex1wxZUdGKCtTd9aE4zTNFe8gNPg8fh2hvn2WiicmeJ6qT2j71uzo3uDs8nraVoQxf 50QJWO2nV1VQfqzlBiMmkiG0ax3vfvVQMaXBw8sBHj3gTubsUi2fljuWzdbwZjQ3oUecuSwDN93eZ L8wV/dI1Q==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqziM-0002Lu-4s; Thu, 02 Jul 2020 13:57:22 +0000 Received: from szxga04-in.huawei.com ([45.249.212.190] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqziA-0002E7-HH for linux-arm-kernel@lists.infradead.org; Thu, 02 Jul 2020 13:57:11 +0000 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 8000126C23D4911ADAAC; Thu, 2 Jul 2020 21:56:10 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Thu, 2 Jul 2020 21:56:01 +0800 From: Keqian Zhu To: , , , Subject: [PATCH v2 1/8] KVM: arm64: Set DBM bit for writable PTEs Date: Thu, 2 Jul 2020 21:55:49 +0800 Message-ID: <20200702135556.36896-2-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200702135556.36896-1-zhukeqian1@huawei.com> References: <20200702135556.36896-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200702_095710_887279_2CDBF7CA X-CRM114-Status: UNSURE ( 7.69 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.190 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.190 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Catalin Marinas , Keqian Zhu , Sean Christopherson , Steven Price , liangpeng10@huawei.com, Alexios Zavras , zhengxiang9@huawei.com, Mark Brown , James Morse , Marc Zyngier , wanghaibin.wang@huawei.com, Thomas Gleixner , Will Deacon , Andrew Morton , Julien Thierry Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DBM bit is used by MMU to differentiate a genuinely non-writable page from a page that is only temporarily non-writable in order to mark dirty. Signed-off-by: Keqian Zhu Signed-off-by: Peng Liang --- arch/arm64/include/asm/kvm_mmu.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index b12bfc1f051a..2700442b0f75 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -169,6 +169,10 @@ void kvm_clear_hyp_idmap(void); static inline pte_t kvm_s2pte_mkwrite(pte_t pte) { pte_val(pte) |= PTE_S2_RDWR; + + if (IS_ENABLED(CONFIG_ARM64_HW_AFDBM)) + pte_val(pte) |= PTE_DBM; + return pte; } From patchwork Thu Jul 2 13:55:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11638975 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E75C618 for ; Thu, 2 Jul 2020 13:57:42 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6793220772 for ; Thu, 2 Jul 2020 13:57:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ifQ5LJRd" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6793220772 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GWZkB9VtRmYQeCoxouTyHpxIXu/MiemXmfNqPXLNdlM=; b=ifQ5LJRdistmqOz0CKYT2f2Jy 8yE03lZ2qJOtwTPLKZX0FMlExxxDLnEwLRMKBvU9PSO3gz/ydKOTvNSFNXIm7rgvZc5jvCwTe5yra HvWE0oVlRPi+02+Ku6PrUDOfjRLYk5v27Gf2LbZ5MgAAADWmoA1UQpJRlamUp5DMxAopZ45vGDRy3 Zvs0tWpE4JdQPJieqE2s4weaOiztz68pyzYlNRcRSLeALoj3Nio7iWCKE0psOUJHVoHxvdL/E4zgP N8QCTpK/Hq1qlUCbAH4DxUPSvwEBqPuWskU3s9avrk012FaWg9Q2Z6Y342CCo9NgXWuK1hHCtkkHo xvBNOvjtQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqzhK-0001sc-PZ; Thu, 02 Jul 2020 13:56:18 +0000 Received: from szxga07-in.huawei.com ([45.249.212.35] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqzhG-0001rA-K4 for linux-arm-kernel@lists.infradead.org; Thu, 02 Jul 2020 13:56:17 +0000 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id B3A8A379818FB6A0EAAB; Thu, 2 Jul 2020 21:56:10 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Thu, 2 Jul 2020 21:56:03 +0800 From: Keqian Zhu To: , , , Subject: [PATCH v2 2/8] KVM: arm64: Scan PTEs to sync dirty log Date: Thu, 2 Jul 2020 21:55:50 +0800 Message-ID: <20200702135556.36896-3-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200702135556.36896-1-zhukeqian1@huawei.com> References: <20200702135556.36896-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200702_095616_678981_D159D696 X-CRM114-Status: GOOD ( 14.22 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.35 listed in list.dnswl.org] 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.35 listed in wl.mailspike.net] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Catalin Marinas , Keqian Zhu , Sean Christopherson , Steven Price , liangpeng10@huawei.com, Alexios Zavras , zhengxiang9@huawei.com, Mark Brown , James Morse , Marc Zyngier , wanghaibin.wang@huawei.com, Thomas Gleixner , Will Deacon , Andrew Morton , Julien Thierry Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org For hardware management of dirty state, dirty state is stored in PTEs. We have to scan all PTEs to sync dirty log to memslot dirty bitmap. Signed-off-by: Keqian Zhu Signed-off-by: Peng Liang --- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/include/asm/kvm_mmu.h | 13 +++ arch/arm64/kvm/arm.c | 3 +- arch/arm64/kvm/mmu.c | 142 ++++++++++++++++++++++++++++++ 4 files changed, 158 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index c3e6fcc664b1..86b9c210ba43 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -480,6 +480,7 @@ u64 __kvm_call_hyp(void *hypfn, ...); void force_vm_exit(const cpumask_t *mask); void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot); +void kvm_mmu_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot); int handle_exit(struct kvm_vcpu *vcpu, struct kvm_run *run, int exception_index); diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 2700442b0f75..4c12b7ad8ae8 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -269,6 +269,19 @@ static inline bool kvm_s2pud_young(pud_t pud) return pud_young(pud); } +static inline bool arm_mmu_hw_dbm_supported(void) +{ + u8 hadbs = (read_sysreg(id_aa64mmfr1_el1) >> + ID_AA64MMFR1_HADBS_SHIFT) & 0xf; + + return hadbs == 0x2; +} + +static inline bool kvm_mmu_hw_dbm_enabled(struct kvm *kvm) +{ + return arm_mmu_hw_dbm_supported() && !!(kvm->arch.vtcr & VTCR_EL2_HD); +} + #define hyp_pte_table_empty(ptep) kvm_page_empty(ptep) #ifdef __PAGETABLE_PMD_FOLDED diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 90cb90561446..fefa5406e037 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1199,7 +1199,8 @@ long kvm_arch_vcpu_ioctl(struct file *filp, void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) { - + if (IS_ENABLED(CONFIG_ARM66_HW_AFDBM) && kvm_mmu_hw_dbm_enabled(kvm)) + kvm_mmu_sync_dirty_log(kvm, memslot); } void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 8c0035cab6b6..b3cb8b6da4c2 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -2411,6 +2411,148 @@ int kvm_mmu_init(void) return err; } +#ifdef CONFIG_ARM64_HW_AFDBM +/** + * stage2_sync_dirty_log_ptes() - synchronize dirty log from PMD range + * @kvm: The KVM pointer + * @pmd: pointer to pmd entry + * @addr: range start address + * @end: range end address + */ +static void stage2_sync_dirty_log_ptes(struct kvm *kvm, pmd_t *pmd, + phys_addr_t addr, phys_addr_t end) +{ + pte_t *pte; + + pte = pte_offset_kernel(pmd, addr); + do { + if (!pte_none(*pte) && !kvm_s2pte_readonly(pte)) + mark_page_dirty(kvm, addr >> PAGE_SHIFT); + } while (pte++, addr += PAGE_SIZE, addr != end); +} + +/** + * stage2_sync_dirty_log_pmds() - synchronize dirty log from PUD range + * @kvm: The KVM pointer + * @pud: pointer to pud entry + * @addr: range start address + * @end: range end address + */ +static void stage2_sync_dirty_log_pmds(struct kvm *kvm, pud_t *pud, + phys_addr_t addr, phys_addr_t end) +{ + pmd_t *pmd; + phys_addr_t next; + + pmd = stage2_pmd_offset(kvm, pud, addr); + do { + next = stage2_pmd_addr_end(kvm, addr, end); + if (!pmd_none(*pmd) && !pmd_thp_or_huge(*pmd)) + stage2_sync_dirty_log_ptes(kvm, pmd, addr, next); + } while (pmd++, addr = next, addr != end); +} + +/** + * stage2_sync_dirty_log_puds() - synchronize dirty log from P4D range + * @kvm: The KVM pointer + * @pgd: pointer to pgd entry + * @addr: range start address + * @end: range end address + */ +static void stage2_sync_dirty_log_puds(struct kvm *kvm, p4d_t *p4d, + phys_addr_t addr, phys_addr_t end) +{ + pud_t *pud; + phys_addr_t next; + + pud = stage2_pud_offset(kvm, p4d, addr); + do { + next = stage2_pud_addr_end(kvm, addr, end); + if (!stage2_pud_none(kvm, *pud) && !stage2_pud_huge(kvm, *pud)) + stage2_sync_dirty_log_pmds(kvm, pud, addr, next); + } while (pud++, addr = next, addr != end); +} + +/** + * stage2_sync_dirty_log_p4ds() - synchronize dirty log from PGD range + * @kvm: The KVM pointer + * @pgd: pointer to pgd entry + * @addr: range start address + * @end: range end address + */ +static void stage2_sync_dirty_log_p4ds(struct kvm *kvm, pgd_t *pgd, + phys_addr_t addr, phys_addr_t end) +{ + p4d_t *p4d; + phys_addr_t next; + + p4d = stage2_p4d_offset(kvm, pgd, addr); + do { + next = stage2_p4d_addr_end(kvm, addr, end); + if (!stage2_p4d_none(kvm, *p4d)) + stage2_sync_dirty_log_puds(kvm, p4d, addr, next); + } while (p4d++, addr = next, addr != end); +} + +/** + * stage2_sync_dirty_log_range() - synchronize dirty log from stage2 memory + * region range + * @kvm: The KVM pointer + * @addr: Start address of range + * @end: End address of range + */ +static void stage2_sync_dirty_log_range(struct kvm *kvm, phys_addr_t addr, + phys_addr_t end) +{ + pgd_t *pgd; + phys_addr_t next; + + pgd = kvm->arch.pgd + stage2_pgd_index(kvm, addr); + do { + cond_resched_lock(&kvm->mmu_lock); + if (!READ_ONCE(kvm->arch.pgd)) + break; + next = stage2_pgd_addr_end(kvm, addr, end); + if (stage2_pgd_present(kvm, *pgd)) + stage2_sync_dirty_log_p4ds(kvm, pgd, addr, next); + } while (pgd++, addr = next, addr != end); +} + +/** + * kvm_mmu_sync_dirty_log() - synchronize dirty log from stage2 PTEs for + * memory slot + * @kvm: The KVM pointer + * @slot: The memory slot to synchronize dirty log + * + * Called to synchronize dirty log (marked by hw) after memory region + * KVM_GET_DIRTY_LOG operation is called. After this function returns + * all dirty log information (for that hw will modify page tables during + * this routine, it is true only when guest is stopped, but it is OK + * because we won't miss dirty log finally.) are collected into memslot + * dirty_bitmap. Afterwards dirty_bitmap can be copied to userspace. + * + * Acquires kvm_mmu_lock. Called with kvm->slots_lock mutex acquired, + * serializing operations for VM memory regions. + */ +void kvm_mmu_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) +{ + phys_addr_t start = memslot->base_gfn << PAGE_SHIFT; + phys_addr_t end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; + int srcu_idx; + + if (WARN_ON_ONCE(!memslot->dirty_bitmap)) + return; + + srcu_idx = srcu_read_lock(&kvm->srcu); + spin_lock(&kvm->mmu_lock); + + stage2_sync_dirty_log_range(kvm, start, end); + + spin_unlock(&kvm->mmu_lock); + srcu_read_unlock(&kvm->srcu, srcu_idx); +} +#endif /* CONFIG_ARM64_HW_AFDBM */ + void kvm_arch_commit_memory_region(struct kvm *kvm, const struct kvm_userspace_memory_region *mem, struct kvm_memory_slot *old, From patchwork Thu Jul 2 13:55:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11638985 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0646413B4 for ; Thu, 2 Jul 2020 13:58:45 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D3F2F20772 for ; Thu, 2 Jul 2020 13:58:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ZXkp7AxD" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D3F2F20772 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vua+4WcNWD3mR2DT2mQpCpK52mAfDNG8llmzB+zQym8=; b=ZXkp7AxDsbq2AeeydbiXAw23V Twf23Jzg/0FXKQDLH5KIRYi5lwKq8P0/PX3R05DhFr0kPwVKLBP/WI5KV9wQ+EWGERMdNrEGtsTW6 1BhQEdVFSfJRr6igSsJbn6YA25Ad3Z0ceOWF4oUPNNa1ogGqIVIQse0BeQV+M1AOYX8YnkW8TbQdm fQREwD8CGAUEnAs3H4Xpv1BHmmEouZC4U1oH2A8wlUeV2YpAgBFAaDDxs0mfGEdoHiub0ZkeLhtl8 9Wz7z7s60fhBPwk3wHrrz6fW+Q+kOGbM5+rl5iEXSruP+7Bn6D+mKhQTtpNmha+ZbcFGhqgXV6Coj fmdMk+Pjw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqziC-0002Gl-Rj; Thu, 02 Jul 2020 13:57:12 +0000 Received: from szxga04-in.huawei.com ([45.249.212.190] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqzi8-0002Dp-Cw for linux-arm-kernel@lists.infradead.org; Thu, 02 Jul 2020 13:57:09 +0000 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 887BE1F41B7E81547404; Thu, 2 Jul 2020 21:56:10 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Thu, 2 Jul 2020 21:56:04 +0800 From: Keqian Zhu To: , , , Subject: [PATCH v2 3/8] KVM: arm64: Modify stage2 young mechanism to support hw DBM Date: Thu, 2 Jul 2020 21:55:51 +0800 Message-ID: <20200702135556.36896-4-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200702135556.36896-1-zhukeqian1@huawei.com> References: <20200702135556.36896-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200702_095708_738437_854FAB84 X-CRM114-Status: GOOD ( 12.38 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.190 listed in list.dnswl.org] 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.190 listed in wl.mailspike.net] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Catalin Marinas , Keqian Zhu , Sean Christopherson , Steven Price , liangpeng10@huawei.com, Alexios Zavras , zhengxiang9@huawei.com, Mark Brown , James Morse , Marc Zyngier , wanghaibin.wang@huawei.com, Thomas Gleixner , Will Deacon , Andrew Morton , Julien Thierry Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Marking PTs young (set AF bit) should be atomic to avoid cover dirty status set by hardware. Signed-off-by: Keqian Zhu Signed-off-by: Peng Liang --- arch/arm64/include/asm/kvm_mmu.h | 31 ++++++++++++++++++++++--------- arch/arm64/kvm/mmu.c | 15 ++++++++------- 2 files changed, 30 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 4c12b7ad8ae8..a1b6131d980c 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -219,6 +219,18 @@ static inline void kvm_set_s2pte_readonly(pte_t *ptep) } while (pteval != old_pteval); } +static inline void kvm_set_s2pte_young(pte_t *ptep) +{ + pteval_t old_pteval, pteval; + + pteval = READ_ONCE(pte_val(*ptep)); + do { + old_pteval = pteval; + pteval |= PTE_AF; + pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval); + } while (pteval != old_pteval); +} + static inline bool kvm_s2pte_readonly(pte_t *ptep) { return (READ_ONCE(pte_val(*ptep)) & PTE_S2_RDWR) == PTE_S2_RDONLY; @@ -234,6 +246,11 @@ static inline void kvm_set_s2pmd_readonly(pmd_t *pmdp) kvm_set_s2pte_readonly((pte_t *)pmdp); } +static inline void kvm_set_s2pmd_young(pmd_t *pmdp) +{ + kvm_set_s2pte_young((pte_t *)pmdp); +} + static inline bool kvm_s2pmd_readonly(pmd_t *pmdp) { return kvm_s2pte_readonly((pte_t *)pmdp); @@ -249,6 +266,11 @@ static inline void kvm_set_s2pud_readonly(pud_t *pudp) kvm_set_s2pte_readonly((pte_t *)pudp); } +static inline void kvm_set_s2pud_young(pud_t *pudp) +{ + kvm_set_s2pte_young((pte_t *)pudp); +} + static inline bool kvm_s2pud_readonly(pud_t *pudp) { return kvm_s2pte_readonly((pte_t *)pudp); @@ -259,15 +281,6 @@ static inline bool kvm_s2pud_exec(pud_t *pudp) return !(READ_ONCE(pud_val(*pudp)) & PUD_S2_XN); } -static inline pud_t kvm_s2pud_mkyoung(pud_t pud) -{ - return pud_mkyoung(pud); -} - -static inline bool kvm_s2pud_young(pud_t pud) -{ - return pud_young(pud); -} static inline bool arm_mmu_hw_dbm_supported(void) { diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index b3cb8b6da4c2..ab8a6ceecbd8 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -2008,8 +2008,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * Resolve the access fault by making the page young again. * Note that because the faulting entry is guaranteed not to be * cached in the TLB, we don't need to invalidate anything. - * Only the HW Access Flag updates are supported for Stage 2 (no DBM), - * so there is no need for atomic (pte|pmd)_mkyoung operations. + * + * Note: Both DBM and HW AF updates are supported for Stage2, so + * young operations should be atomic. */ static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) { @@ -2027,15 +2028,15 @@ static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) goto out; if (pud) { /* HugeTLB */ - *pud = kvm_s2pud_mkyoung(*pud); + kvm_set_s2pud_young(pud); pfn = kvm_pud_pfn(*pud); pfn_valid = true; } else if (pmd) { /* THP, HugeTLB */ - *pmd = pmd_mkyoung(*pmd); + kvm_set_s2pmd_young(pmd); pfn = pmd_pfn(*pmd); pfn_valid = true; - } else { - *pte = pte_mkyoung(*pte); /* Just a page... */ + } else { /* Just a page... */ + kvm_set_s2pte_young(pte); pfn = pte_pfn(*pte); pfn_valid = true; } @@ -2280,7 +2281,7 @@ static int kvm_test_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void * return 0; if (pud) - return kvm_s2pud_young(*pud); + return pud_young(*pud); else if (pmd) return pmd_young(*pmd); else From patchwork Thu Jul 2 13:55:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11638991 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E4918618 for ; Thu, 2 Jul 2020 13:58:50 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BE04820772 for ; Thu, 2 Jul 2020 13:58:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="QDcxjCFn" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BE04820772 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=LIDZIN9kY0V+4uEP+B+8gNbpapEsqITzcU6zpwh8XcU=; b=QDcxjCFnCN9mfdcrRUkSozKjO gQyZY+blUdqTUj9GLcQdg6en3pOM+3vXzyEhqnZl6/hVU9t1DI2V+OngYyeT6YbRdFn5Lyjdo1Gvc 326FtMtIHI6KxVhKH0WRtHI0lIEMVR353vSmRsk4vHwM7DONdBAv/jjbkPTv7DkUyF/lmXNHbqZ/9 9uhMCkYb9LBI5QCzN6ZnpFuAWRrloZB8BNgoBmyJEefujHXa/Ns5BNZMS5EUU3XTXS3/i787QdBYj 59CH3aj3kATnC5DHf5OOYh6g1YzwdMhPt/eSp06kb8wS0Adf7c661IDx3yqdc3wQoWU8JzTdGYjlt P48A7BDZg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqziV-0002P6-5J; Thu, 02 Jul 2020 13:57:31 +0000 Received: from szxga04-in.huawei.com ([45.249.212.190] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqziF-0002EK-NR for linux-arm-kernel@lists.infradead.org; Thu, 02 Jul 2020 13:57:18 +0000 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id C489A4599281558D6B9C; Thu, 2 Jul 2020 21:56:15 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Thu, 2 Jul 2020 21:56:05 +0800 From: Keqian Zhu To: , , , Subject: [PATCH v2 4/8] KVM: arm64: Save stage2 PTE dirty status if it is covered Date: Thu, 2 Jul 2020 21:55:52 +0800 Message-ID: <20200702135556.36896-5-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200702135556.36896-1-zhukeqian1@huawei.com> References: <20200702135556.36896-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200702_095718_077999_4379995C X-CRM114-Status: GOOD ( 12.97 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.190 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.190 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Catalin Marinas , Keqian Zhu , Sean Christopherson , Steven Price , liangpeng10@huawei.com, Alexios Zavras , zhengxiang9@huawei.com, Mark Brown , James Morse , Marc Zyngier , wanghaibin.wang@huawei.com, Thomas Gleixner , Will Deacon , Andrew Morton , Julien Thierry Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org kvm_set_pte is called to replace a target PTE with a desired one. We always do this without changing the desired one, but if dirty status set by hardware is covered, let caller know it. There are two types of operations will change PTE and may cover dirty status set by hardware. 1. Stage2 PTE unmapping: Page table merging (revert of huge page table dissolving), kvm_unmap_hva_range() and so on. 2. Stage2 PTE changing: including user_mem_abort(), kvm_mmu_notifier _change_pte() and so on. All operations above will invoke kvm_set_pte() finally. We should save the dirty status into memslot bitmap. Signed-off-by: Keqian Zhu Signed-off-by: Peng Liang --- arch/arm64/include/asm/kvm_mmu.h | 5 ++++ arch/arm64/kvm/mmu.c | 42 ++++++++++++++++++++++++++++---- 2 files changed, 42 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index a1b6131d980c..adb5c8edb29e 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -295,6 +295,11 @@ static inline bool kvm_mmu_hw_dbm_enabled(struct kvm *kvm) return arm_mmu_hw_dbm_supported() && !!(kvm->arch.vtcr & VTCR_EL2_HD); } +static inline bool kvm_s2pte_dbm(pte_t *ptep) +{ + return !!(READ_ONCE(pte_val(*ptep)) & PTE_DBM); +} + #define hyp_pte_table_empty(ptep) kvm_page_empty(ptep) #ifdef __PAGETABLE_PMD_FOLDED diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index ab8a6ceecbd8..d0c34549ef3b 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -194,10 +194,26 @@ static void clear_stage2_pmd_entry(struct kvm *kvm, pmd_t *pmd, phys_addr_t addr put_page(virt_to_page(pmd)); } -static inline void kvm_set_pte(pte_t *ptep, pte_t new_pte) +/** + * @ret: true if dirty status set by hardware is covered. + */ +static inline bool kvm_set_pte(pte_t *ptep, pte_t new_pte) { - WRITE_ONCE(*ptep, new_pte); - dsb(ishst); + pteval_t old_pteval; + bool old_logging, new_no_write; + + old_logging = IS_ENABLED(CONFIG_ARM64_HW_AFDBM) && + arm_mmu_hw_dbm_supported() && kvm_s2pte_dbm(ptep); + new_no_write = pte_none(new_pte) || kvm_s2pte_readonly(&new_pte); + + if (!old_logging || !new_no_write) { + WRITE_ONCE(*ptep, new_pte); + dsb(ishst); + return false; + } + + old_pteval = xchg(&pte_val(*ptep), pte_val(new_pte)); + return !kvm_s2pte_readonly(&__pte(old_pteval)); } static inline void kvm_set_pmd(pmd_t *pmdp, pmd_t new_pmd) @@ -260,15 +276,23 @@ static void unmap_stage2_ptes(struct kvm *kvm, pmd_t *pmd, { phys_addr_t start_addr = addr; pte_t *pte, *start_pte; + bool dirty_covered; + int idx; start_pte = pte = pte_offset_kernel(pmd, addr); do { if (!pte_none(*pte)) { pte_t old_pte = *pte; - kvm_set_pte(pte, __pte(0)); + dirty_covered = kvm_set_pte(pte, __pte(0)); kvm_tlb_flush_vmid_ipa(kvm, addr); + if (dirty_covered) { + idx = srcu_read_lock(&kvm->srcu); + mark_page_dirty(kvm, addr >> PAGE_SHIFT); + srcu_read_unlock(&kvm->srcu, idx); + } + /* No need to invalidate the cache for device mappings */ if (!kvm_is_device_pfn(pte_pfn(old_pte))) kvm_flush_dcache_pte(old_pte); @@ -1354,6 +1378,8 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, pte_t *pte, old_pte; bool iomap = flags & KVM_S2PTE_FLAG_IS_IOMAP; bool logging_active = flags & KVM_S2_FLAG_LOGGING_ACTIVE; + bool dirty_covered; + int idx; VM_BUG_ON(logging_active && !cache); @@ -1419,8 +1445,14 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, if (pte_val(old_pte) == pte_val(*new_pte)) return 0; - kvm_set_pte(pte, __pte(0)); + dirty_covered = kvm_set_pte(pte, __pte(0)); kvm_tlb_flush_vmid_ipa(kvm, addr); + + if (dirty_covered) { + idx = srcu_read_lock(&kvm->srcu); + mark_page_dirty(kvm, addr >> PAGE_SHIFT); + srcu_read_unlock(&kvm->srcu, idx); + } } else { get_page(virt_to_page(pte)); } From patchwork Thu Jul 2 13:55:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11638983 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C9BA813B4 for ; Thu, 2 Jul 2020 13:58:04 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A272920772 for ; Thu, 2 Jul 2020 13:58:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="rBXm3C6B" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A272920772 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=c4FzRVXHpdaH1i/1dXZS3e6khUQWvKjIbvxNW6AEm4w=; b=rBXm3C6BHeYTD4t1Zrr7Zsq0a JDt951uZxL5Z5d352dzbUdgBsmZPRNydJelfY3eUM9TKeIcUV8I7oVytncRA/ImE3OvTh5s7ub00s yDOfdvLiCqWTEljr2vKBTRkFv2w+tpJDlF/9Z9AzLuc8+NSxyzaW/HZOfNMLjeX8zL+9adfGRWeoJ fXRU+qoPqtQ6AC1Bh7PpQdis191aChcdgrVUXfne2muGPOO7Su4XHG3i/jcDaTtFSA82cJpCtKHgB zikUgQ37oUghSaaFpjty6iEIj3SyfUHOsMr7vPEe/hon3PnQxfkmUss36Da6DXR32awahN5OwLOv1 Rcydb57XA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqzhO-0001ts-D6; Thu, 02 Jul 2020 13:56:22 +0000 Received: from szxga07-in.huawei.com ([45.249.212.35] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqzhJ-0001rd-It for linux-arm-kernel@lists.infradead.org; Thu, 02 Jul 2020 13:56:18 +0000 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 8FA50E69E0B83810267D; Thu, 2 Jul 2020 21:56:15 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Thu, 2 Jul 2020 21:56:05 +0800 From: Keqian Zhu To: , , , Subject: [PATCH v2 5/8] KVM: arm64: Steply write protect page table by mask bit Date: Thu, 2 Jul 2020 21:55:53 +0800 Message-ID: <20200702135556.36896-6-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200702135556.36896-1-zhukeqian1@huawei.com> References: <20200702135556.36896-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200702_095617_874637_7A8C88D1 X-CRM114-Status: UNSURE ( 8.22 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.35 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.35 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Catalin Marinas , Keqian Zhu , Sean Christopherson , Steven Price , liangpeng10@huawei.com, Alexios Zavras , zhengxiang9@huawei.com, Mark Brown , James Morse , Marc Zyngier , wanghaibin.wang@huawei.com, Thomas Gleixner , Will Deacon , Andrew Morton , Julien Thierry Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org During dirty log clear, page table entries are write protected according to a mask. In the past we write protect all entries corresponding to the mask from ffs to fls. Though there may be zero bits between this range, we are holding the kvm mmu lock so we won't write protect entries that we don't want to. We are about to add support for hardware management of dirty state to arm64, holding kvm mmu lock will be not enough. We should write protect entries steply by mask bit. Signed-off-by: Keqian Zhu Signed-off-by: Peng Liang --- arch/arm64/kvm/mmu.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d0c34549ef3b..adfa62f1fced 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1703,10 +1703,16 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, gfn_t gfn_offset, unsigned long mask) { phys_addr_t base_gfn = slot->base_gfn + gfn_offset; - phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; - phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; + phys_addr_t start, end; + u32 i; - stage2_wp_range(kvm, start, end); + for (i = __ffs(mask); i <= __fls(mask); i++) { + if (test_bit_le(i, &mask)) { + start = (base_gfn + i) << PAGE_SHIFT; + end = (base_gfn + i + 1) << PAGE_SHIFT; + stage2_wp_range(kvm, start, end); + } + } } /* From patchwork Thu Jul 2 13:55:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11638981 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 16088618 for ; Thu, 2 Jul 2020 13:58:01 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DBDED20772 for ; Thu, 2 Jul 2020 13:58:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="U8MRctYs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DBDED20772 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=LFsnT/Nt/+k0T9DqTZEVYiZ2AGlCyMtw5bp3icblC4w=; b=U8MRctYs0swUzORDw2/fwIhk4 nvZt8V6WIx2aB/9FudWTuYRCIcqIJp0IEdjEQynr7M3dy61NJ4E48b6anpL/LAIRrknaDes5ZhbxE yrORXTwEyhTXx3H15jAw2IFgJ6CR/XzMnOG4+hoRQwdq298kis3tR+S9XyaWTwPQnVGG6Hdq4vnJ4 IkTK3TUuaq8x+WCTNs84AfOOH2nxqeBqzoSd1LJSGOxFAIUOtUgYUSGs9A28sT5mQtG1CWvcOX7TU A11W6PyWm8VJvTHVphU7uAl/N62tIEgJV2QgPEce06qMCPFQ70anQGkf1xMpMA4VhmoYArl1ngrpW BhnxfVsBQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqzhb-0001yL-4c; Thu, 02 Jul 2020 13:56:35 +0000 Received: from szxga07-in.huawei.com ([45.249.212.35] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqzhO-0001s1-90 for linux-arm-kernel@lists.infradead.org; Thu, 02 Jul 2020 13:56:23 +0000 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id AAE1856A3318634F4E03; Thu, 2 Jul 2020 21:56:15 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Thu, 2 Jul 2020 21:56:06 +0800 From: Keqian Zhu To: , , , Subject: [PATCH v2 6/8] KVM: arm64: Add KVM_CAP_ARM_HW_DIRTY_LOG capability Date: Thu, 2 Jul 2020 21:55:54 +0800 Message-ID: <20200702135556.36896-7-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200702135556.36896-1-zhukeqian1@huawei.com> References: <20200702135556.36896-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200702_095622_666192_9A7B4B18 X-CRM114-Status: GOOD ( 15.31 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.35 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.35 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Catalin Marinas , Keqian Zhu , Sean Christopherson , Steven Price , liangpeng10@huawei.com, Alexios Zavras , zhengxiang9@huawei.com, Mark Brown , James Morse , Marc Zyngier , wanghaibin.wang@huawei.com, Thomas Gleixner , Will Deacon , Andrew Morton , Julien Thierry Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org For that using arm64 DBM to log dirty pages has the side effect of long time dirty log sync, we should give userspace opportunity to enable or disable this feature, to realize some policy. This feature is disabled by default. Signed-off-by: Keqian Zhu Signed-off-by: Peng Liang --- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 35 +++++++++++++++++++++++++++++++ arch/arm64/kvm/mmu.c | 22 +++++++++++++++++++ arch/arm64/kvm/reset.c | 5 +++++ include/uapi/linux/kvm.h | 1 + tools/include/uapi/linux/kvm.h | 1 + 6 files changed, 65 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 86b9c210ba43..69a5317c7049 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -481,6 +481,7 @@ u64 __kvm_call_hyp(void *hypfn, ...); void force_vm_exit(const cpumask_t *mask); void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot); void kvm_mmu_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot); +void kvm_mmu_sync_dirty_log_all(struct kvm *kvm); int handle_exit(struct kvm_vcpu *vcpu, struct kvm_run *run, int exception_index); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index fefa5406e037..9e3f765d5467 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -78,6 +78,11 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap) { int r; +#ifdef CONFIG_ARM64_HW_AFDBM + int i; + struct kvm_vcpu *vcpu; + bool enable_hw_dirty_log; +#endif if (cap->flags) return -EINVAL; @@ -87,6 +92,36 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, r = 0; kvm->arch.return_nisv_io_abort_to_user = true; break; +#ifdef CONFIG_ARM64_HW_AFDBM + case KVM_CAP_ARM_HW_DIRTY_LOG: + if (!arm_mmu_hw_dbm_supported() || (cap->args[0] & ~0x1)) + r = -EINVAL; + + enable_hw_dirty_log = !!(cap->args[0] & 0x1); + if (!!(kvm->arch.vtcr & VTCR_EL2_HD) != enable_hw_dirty_log) { + if (enable_hw_dirty_log) + kvm->arch.vtcr |= VTCR_EL2_HD; + else + kvm->arch.vtcr &= ~VTCR_EL2_HD; + + /* + * We should kick vcpus out of guest mode here to + * load new vtcr value to vtcr_el2 register when + * re-enter guest mode. + */ + kvm_for_each_vcpu(i, vcpu, kvm) + kvm_vcpu_kick(vcpu); + + if (!enable_hw_dirty_log) { + mutex_lock(&kvm->slots_lock); + kvm_mmu_sync_dirty_log_all(kvm); + mutex_unlock(&kvm->slots_lock); + } + } + + r = 0; + break; +#endif default: r = -EINVAL; break; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index adfa62f1fced..1a48554accb0 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -2590,6 +2590,28 @@ void kvm_mmu_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) spin_unlock(&kvm->mmu_lock); srcu_read_unlock(&kvm->srcu, srcu_idx); } + +/** + * kvm_mmu_sync_dirty_log_all() - synchronize dirty log from PTEs for whole VM + * @kvm: The KVM pointer + * + * Called with kvm->slots_lock mutex acquired + */ +void kvm_mmu_sync_dirty_log_all(struct kvm *kvm) +{ + struct kvm_memslots *slots = kvm_memslots(kvm); + struct kvm_memory_slot *memslots = slots->memslots; + struct kvm_memory_slot *memslot; + int slot; + + if (unlikely(!slots->used_slots)) + return; + + for (slot = 0; slot < slots->used_slots; slot++) { + memslot = &memslots[slot]; + kvm_mmu_sync_dirty_log(kvm, memslot); + } +} #endif /* CONFIG_ARM64_HW_AFDBM */ void kvm_arch_commit_memory_region(struct kvm *kvm, diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index d3b209023727..a3be703dd54b 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -83,6 +83,11 @@ int kvm_arch_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = has_vhe() && system_supports_address_auth() && system_supports_generic_auth(); break; +#ifdef CONFIG_ARM64_HW_AFDBM + case KVM_CAP_ARM_HW_DIRTY_LOG: + r = arm_mmu_hw_dbm_supported(); + break; +#endif /* CONFIG_ARM64_HW_AFDBM */ default: r = 0; } diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 4fdf30316582..e0b12c43397b 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1031,6 +1031,7 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_PPC_SECURE_GUEST 181 #define KVM_CAP_HALT_POLL 182 #define KVM_CAP_ASYNC_PF_INT 183 +#define KVM_CAP_ARM_HW_DIRTY_LOG 184 #ifdef KVM_CAP_IRQ_ROUTING diff --git a/tools/include/uapi/linux/kvm.h b/tools/include/uapi/linux/kvm.h index 4fdf30316582..e0b12c43397b 100644 --- a/tools/include/uapi/linux/kvm.h +++ b/tools/include/uapi/linux/kvm.h @@ -1031,6 +1031,7 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_PPC_SECURE_GUEST 181 #define KVM_CAP_HALT_POLL 182 #define KVM_CAP_ASYNC_PF_INT 183 +#define KVM_CAP_ARM_HW_DIRTY_LOG 184 #ifdef KVM_CAP_IRQ_ROUTING From patchwork Thu Jul 2 13:55:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11638977 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CDB5B13B4 for ; Thu, 2 Jul 2020 13:57:51 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A57A72088E for ; Thu, 2 Jul 2020 13:57:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="xWV3w1pr" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A57A72088E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=YGNOwTDNs5wGhF1UcsxyUgNQ41/ArUPAm011dp4NdYs=; b=xWV3w1prfZJq5HTBA7xWF7uiG 28wCMpKzLf22Hk5OC5rZy1JhjB0u7KM4hyDOT40f+RH2QZTXnwmGdLgSC6VctMF1DQI0ZpCBNdtWw BSCEvZIHE6lc9HkA8MEvbPYsRt8CwZVKm/1iUDmw+xpx21VTF3OJ0toGEzS8T4vlYf5RY/3RuP7+0 e/buCxEXxrGiPLj7iVsdCcIxbKWfFwr84WnIc16tYi38vQrv+JehU0/Jrbv2Q1QHzfvFkOKNEl/zW RiVGZRBWitN/BokotGG+UcrCNZa/oDIODWF094085rjMtEOycfzkpdu+hJaBhOgCpqjSvwdAqp8ZM CFP/6S5PQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqzhR-0001v3-Ry; Thu, 02 Jul 2020 13:56:25 +0000 Received: from szxga07-in.huawei.com ([45.249.212.35] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqzhK-0001rf-LJ for linux-arm-kernel@lists.infradead.org; Thu, 02 Jul 2020 13:56:20 +0000 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 984256462E39DFB8901B; Thu, 2 Jul 2020 21:56:15 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Thu, 2 Jul 2020 21:56:07 +0800 From: Keqian Zhu To: , , , Subject: [PATCH v2 7/8] KVM: arm64: Sync dirty log parallel Date: Thu, 2 Jul 2020 21:55:55 +0800 Message-ID: <20200702135556.36896-8-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200702135556.36896-1-zhukeqian1@huawei.com> References: <20200702135556.36896-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200702_095618_989344_207DE5FE X-CRM114-Status: GOOD ( 14.88 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.35 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.35 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Catalin Marinas , Keqian Zhu , Sean Christopherson , Steven Price , liangpeng10@huawei.com, Alexios Zavras , zhengxiang9@huawei.com, Mark Brown , James Morse , Marc Zyngier , wanghaibin.wang@huawei.com, Thomas Gleixner , Will Deacon , Andrew Morton , Julien Thierry Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Give userspace another selection to solve high-cost dirty log sync, which called multi-core offload. Usersapce can enable this policy through KVM_CAP_ARM_HW_DIRTY_LOG. Signed-off-by: Keqian Zhu Signed-off-by: Peng Liang --- arch/arm64/include/asm/kvm_host.h | 3 ++ arch/arm64/kvm/arm.c | 9 +++- arch/arm64/kvm/mmu.c | 82 +++++++++++++++++++++++++++++-- arch/arm64/kvm/reset.c | 2 +- 4 files changed, 90 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 69a5317c7049..05da819f9adc 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -95,6 +95,9 @@ struct kvm_arch { * supported. */ bool return_nisv_io_abort_to_user; + + /* Sync dirty log parallel when hw dirty log enabled */ + bool sync_dirty_log_parallel; }; #define KVM_NR_MEM_OBJS 40 diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 9e3f765d5467..89614984831d 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -82,6 +82,7 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, int i; struct kvm_vcpu *vcpu; bool enable_hw_dirty_log; + bool enable_sync_parallel; #endif if (cap->flags) @@ -94,10 +95,14 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, break; #ifdef CONFIG_ARM64_HW_AFDBM case KVM_CAP_ARM_HW_DIRTY_LOG: - if (!arm_mmu_hw_dbm_supported() || (cap->args[0] & ~0x1)) + if (!arm_mmu_hw_dbm_supported() || (cap->args[0] & ~0x3)) r = -EINVAL; enable_hw_dirty_log = !!(cap->args[0] & 0x1); + enable_sync_parallel = !!(cap->args[0] & 0x2); + if (!enable_hw_dirty_log && enable_sync_parallel) + r = -EINVAL; + if (!!(kvm->arch.vtcr & VTCR_EL2_HD) != enable_hw_dirty_log) { if (enable_hw_dirty_log) kvm->arch.vtcr |= VTCR_EL2_HD; @@ -119,6 +124,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, } } + kvm->arch.sync_dirty_log_parallel = enable_sync_parallel; + r = 0; break; #endif diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 1a48554accb0..be360e0fd20b 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -2548,15 +2548,55 @@ static void stage2_sync_dirty_log_range(struct kvm *kvm, phys_addr_t addr, pgd = kvm->arch.pgd + stage2_pgd_index(kvm, addr); do { - cond_resched_lock(&kvm->mmu_lock); - if (!READ_ONCE(kvm->arch.pgd)) - break; + if (!kvm->arch.sync_dirty_log_parallel) { + cond_resched_lock(&kvm->mmu_lock); + if (!READ_ONCE(kvm->arch.pgd)) + break; + } next = stage2_pgd_addr_end(kvm, addr, end); if (stage2_pgd_present(kvm, *pgd)) stage2_sync_dirty_log_p4ds(kvm, pgd, addr, next); } while (pgd++, addr = next, addr != end); } +static struct dirty_sync_task { + struct kvm *kvm; + struct kvm_memory_slot *memslot; + u32 cpu_cnt; + u16 cpu_idx_map[NR_CPUS]; + u32 ite_npages; + u32 ite; + bool finished; +} sync_task; + +static void stage2_sync_dirty_log_smp(void *task) +{ + struct dirty_sync_task *t = task; + struct kvm_memory_slot *memslot = t->memslot; + unsigned long ite_idx, base_page, end_page; + gfn_t base_gfn; + + ite_idx = t->cpu_cnt * t->ite + t->cpu_idx_map[smp_processor_id()]; + + base_page = ite_idx * t->ite_npages; + if (base_page >= memslot->npages) { + t->finished = true; + trace_printk("stage2_sync_dirty_log_smp finished 1.\n"); + return; + } + + end_page = min(memslot->npages, base_page + t->ite_npages); + if (end_page == memslot->npages) { + t->finished = true; + trace_printk("stage2_sync_dirty_log_smp finished 2.\n"); + } + + base_gfn = memslot->base_gfn; + trace_printk("base_page 0x%lx, end_page 0x%lx.\n", base_page, end_page); + stage2_sync_dirty_log_range(t->kvm, (base_gfn + base_page) << PAGE_SHIFT, + (base_gfn + end_page) << PAGE_SHIFT); +} + /** * kvm_mmu_sync_dirty_log() - synchronize dirty log from stage2 PTEs for * memory slot @@ -2577,18 +2617,52 @@ void kvm_mmu_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) { phys_addr_t start = memslot->base_gfn << PAGE_SHIFT; phys_addr_t end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; + u32 ite_npages, cpu_cnt, this_cpu, cpu; + u16 cpu_idx; int srcu_idx; if (WARN_ON_ONCE(!memslot->dirty_bitmap)) return; + get_online_cpus(); srcu_idx = srcu_read_lock(&kvm->srcu); spin_lock(&kvm->mmu_lock); - stage2_sync_dirty_log_range(kvm, start, end); + ite_npages = (1 << 17); /* 512MB max for per iteration and CPU */ + cpu_cnt = num_online_cpus(); + + /* Use single CPU */ + if (!kvm->arch.sync_dirty_log_parallel || + memslot->npages <= ite_npages || + unlikely(cpu_cnt == 1)) { + stage2_sync_dirty_log_range(kvm, start, end); + goto out_unlock; + } + /* Use many CPUs through IPI */ + cpu_idx = 0; + this_cpu = smp_processor_id(); + for_each_online_cpu(cpu) { + if (cpu != this_cpu) + sync_task.cpu_idx_map[cpu] = cpu_idx++; + } + + sync_task.kvm = kvm; + sync_task.memslot = memslot; + sync_task.cpu_cnt = cpu_cnt - 1; /* Not include this CPU */ + sync_task.ite_npages = ite_npages; + + sync_task.ite = 0; + sync_task.finished = false; + do { + smp_call_function(stage2_sync_dirty_log_smp, &sync_task, 1); + sync_task.ite++; + } while (!sync_task.finished); + +out_unlock: spin_unlock(&kvm->mmu_lock); srcu_read_unlock(&kvm->srcu, srcu_idx); + put_online_cpus(); } /** diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index a3be703dd54b..4171d6c1d400 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -85,7 +85,7 @@ int kvm_arch_vm_ioctl_check_extension(struct kvm *kvm, long ext) break; #ifdef CONFIG_ARM64_HW_AFDBM case KVM_CAP_ARM_HW_DIRTY_LOG: - r = arm_mmu_hw_dbm_supported(); + r = arm_mmu_hw_dbm_supported() ? 0x3 : 0x0; break; #endif /* CONFIG_ARM64_HW_AFDBM */ default: From patchwork Thu Jul 2 13:55:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11638979 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9ED87618 for ; Thu, 2 Jul 2020 13:57:56 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7205920772 for ; Thu, 2 Jul 2020 13:57:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="dgBf9lhi" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7205920772 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=1t3kfxaSCNH/ptVpQiosJ9MRi83eH2naNxn2vqhkbLk=; b=dgBf9lhiCKr8sKVVgyvm/U34Z hPmFqqRgB54ZvwyDoyLSSb5VLSEvYOpudohIu+9Gb1fd2vP+l+qXY7P8mYJBMXtdPWHYchuL9YYOP GPplthJH2Z4/fsPBunWrM7I665SlAoxjt7F4oCiAjt/m10hPeWQHtVX7IpgEGNUP4yV+87RG4iKRU FiDF3oE2k04TssdbZ+ZAITd90B5HzmVMjLCG+83GCNx2EIWqHTZBzFyIHJRM9BAjGlas3ZrtdR/n7 I5AmuYfamPCOIIbxYXHf1IQflfJskz2LliVvjq5uKRRVXIFN9AKgjvl54a6VE2XopRx79Lo1Ehgqg Ejlpy8xGw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqzhV-0001wK-5b; Thu, 02 Jul 2020 13:56:29 +0000 Received: from szxga07-in.huawei.com ([45.249.212.35] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jqzhM-0001rw-FW for linux-arm-kernel@lists.infradead.org; Thu, 02 Jul 2020 13:56:21 +0000 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id A246244A5C93F08B323C; Thu, 2 Jul 2020 21:56:15 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Thu, 2 Jul 2020 21:56:09 +0800 From: Keqian Zhu To: , , , Subject: [PATCH v2 8/8] KVM: Omit dirty log sync in log clear if initially all set Date: Thu, 2 Jul 2020 21:55:56 +0800 Message-ID: <20200702135556.36896-9-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200702135556.36896-1-zhukeqian1@huawei.com> References: <20200702135556.36896-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200702_095620_872199_3C6BA8CA X-CRM114-Status: UNSURE ( 9.05 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.35 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.35 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Catalin Marinas , Keqian Zhu , Sean Christopherson , Steven Price , liangpeng10@huawei.com, Alexios Zavras , zhengxiang9@huawei.com, Mark Brown , James Morse , Marc Zyngier , wanghaibin.wang@huawei.com, Thomas Gleixner , Will Deacon , Andrew Morton , Julien Thierry Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Synchronizing dirty log during log clear is useful only when the dirty bitmap of userspace contains dirty bits that memslot dirty bitmap does not contain, because we can sync new dirty bits to memslot dirty bitmap, then we can clear them by the way and avoid reporting them to userspace later. With dirty bitmap "initially all set" feature, the above situation will not appear if userspace logic is normal, so we can omit dirty log sync in log clear. This is valuable when dirty log sync is a high-cost operation, such as arm64 DBM. Signed-off-by: Keqian Zhu --- virt/kvm/kvm_main.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index a852af5c3214..3f9e51d52b7a 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1554,7 +1554,8 @@ static int kvm_clear_dirty_log_protect(struct kvm *kvm, (log->num_pages < memslot->npages - log->first_page && (log->num_pages & 63))) return -EINVAL; - kvm_arch_sync_dirty_log(kvm, memslot); + if (!kvm_dirty_log_manual_protect_and_init_set(kvm)) + kvm_arch_sync_dirty_log(kvm, memslot); flush = false; dirty_bitmap_buffer = kvm_second_dirty_bitmap(memslot);