From patchwork Thu Apr 15 11:50:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yanan Wang X-Patchwork-Id: 12205229 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF3C5C433B4 for ; Thu, 15 Apr 2021 11:53:35 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3DA20610FA for ; Thu, 15 Apr 2021 11:53:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3DA20610FA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=HlV+L61T8ujLbGSpIoFuTAJXKQwmJOgltT/wapSuCUk=; b=Lg657C+Nmn/CaMorlT9dpc1bu LYI3pUopsgQByQVJyUcHFDg5qW50GC7huhEGoSZ9x8jwD1C4mmjeky4EQB7R/uPBF+p698ztOwle6 MWprfHMOZJu29YPOiBy0aK29cj2HndVDjt3tWA78kQubAFduY1aAZsBGn+4gF0Qv8Oj+5jst2uOMj 1bW7djY1gW7QVvo4Rbwiqu5Ky2ktsxuoqNuPFEWzx6/QIoi9GjGZkAuDv85o4EZDgdvUtZc/lqOU7 Tv4wlUhZHarPW/IOMX1ljIFRmKu/aLY3Am9R8nSbx9+CjvQ6Wl9hBnCoiFfkMaSGL/of2pMpbiqQf qrKp71opw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lX0XS-00G1NB-3U; Thu, 15 Apr 2021 11:52:02 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WQ-00G1Cm-U1 for linux-arm-kernel@desiato.infradead.org; Thu, 15 Apr 2021 11:50:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=V37cbmucqJxNcsgY7L4KZs4Ri2E2TBGjdLe9ygPFZ0o=; b=LWi1p4gM0vylisHRULXRZ3wJMk Z956SrUEkkg6UyWFYs4qWbpoR36d03kWc7oQMwgLE9ySGI/4yyhAUCOqHRMmiz3K3ASBDY40n6dz7 mGAl6YFSqo8DsRsesKE+fWiaFfvNP3YvkDo9BfGvttRgxupBEDc69U3bY0Fpd7DxIl5+wR+QzhMCG YH9mUsjp8ojTjpKy42099ykcqwkDoho2m/pwA25OzPG1h+F91aOukoYcldwJTgPah5xiSBfZD5b1q 0EQtVmF3lV8tRUUZM0jtKoGKCAw7oCqx4K3pG7iEcIusLpeum+1GRilT3d271uHOA0rGnkWpAo63w hj9c6UOg==; Received: from szxga04-in.huawei.com ([45.249.212.190]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WM-008Xbm-Oi for linux-arm-kernel@lists.infradead.org; Thu, 15 Apr 2021 11:50:57 +0000 Received: from DGGEMS407-HUB.china.huawei.com (unknown [172.30.72.60]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4FLcz76TfBzpYXf; Thu, 15 Apr 2021 19:47:47 +0800 (CST) Received: from DESKTOP-TMVL5KK.china.huawei.com (10.174.187.128) by DGGEMS407-HUB.china.huawei.com (10.3.19.207) with Microsoft SMTP Server id 14.3.498.0; Thu, 15 Apr 2021 19:50:34 +0800 From: Yanan Wang To: Marc Zyngier , Will Deacon , "Quentin Perret" , Alexandru Elisei , , , , CC: Catalin Marinas , James Morse , Julien Thierry , "Suzuki K Poulose" , Gavin Shan , , , , Yanan Wang Subject: [PATCH v5 1/6] KVM: arm64: Introduce KVM_PGTABLE_S2_GUEST stage-2 flag Date: Thu, 15 Apr 2021 19:50:27 +0800 Message-ID: <20210415115032.35760-2-wangyanan55@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20210415115032.35760-1-wangyanan55@huawei.com> References: <20210415115032.35760-1-wangyanan55@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.128] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210415_045055_173777_1A32AD93 X-CRM114-Status: GOOD ( 14.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The stage-2 page table code in pgtable.c now is generally used for guest stage-2 and host stage-2. There may be some different issues between guest S2 page-table and host S2 page-table that we should consider, e.g., whether CMOs are needed when creating a new mapping. So introduce the KVM_PGTABLE_S2_GUEST flag to determine if we are doing something about guest stage-2. This flag will be used in a coming patch, in which we will move CMOs for guest to pgtable.c. Signed-off-by: Yanan Wang --- arch/arm64/include/asm/kvm_pgtable.h | 38 ++++++++++++++++++---------- arch/arm64/kvm/mmu.c | 3 ++- 2 files changed, 26 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index c3674c47d48c..a43cbe697b37 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -61,10 +61,12 @@ struct kvm_pgtable_mm_ops { * @KVM_PGTABLE_S2_NOFWB: Don't enforce Normal-WB even if the CPUs have * ARM64_HAS_STAGE2_FWB. * @KVM_PGTABLE_S2_IDMAP: Only use identity mappings. + * @KVM_PGTABLE_S2_GUEST: Whether the page-tables are guest stage-2. */ enum kvm_pgtable_stage2_flags { KVM_PGTABLE_S2_NOFWB = BIT(0), KVM_PGTABLE_S2_IDMAP = BIT(1), + KVM_PGTABLE_S2_GUEST = BIT(2), }; /** @@ -221,12 +223,10 @@ int kvm_pgtable_stage2_init_flags(struct kvm_pgtable *pgt, struct kvm_arch *arch struct kvm_pgtable_mm_ops *mm_ops, enum kvm_pgtable_stage2_flags flags); -#define kvm_pgtable_stage2_init(pgt, arch, mm_ops) \ - kvm_pgtable_stage2_init_flags(pgt, arch, mm_ops, 0) - /** * kvm_pgtable_stage2_destroy() - Destroy an unused guest stage-2 page-table. - * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @pgt: Page-table structure initialised by function + * kvm_pgtable_stage2_init_flags(). * * The page-table is assumed to be unreachable by any hardware walkers prior * to freeing and therefore no TLB invalidation is performed. @@ -235,7 +235,8 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); /** * kvm_pgtable_stage2_map() - Install a mapping in a guest stage-2 page-table. - * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @pgt: Page-table structure initialised by function + * kvm_pgtable_stage2_init_flags(). * @addr: Intermediate physical address at which to place the mapping. * @size: Size of the mapping. * @phys: Physical address of the memory to map. @@ -268,7 +269,8 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, /** * kvm_pgtable_stage2_set_owner() - Unmap and annotate pages in the IPA space to * track ownership. - * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @pgt: Page-table structure initialised by function + * kvm_pgtable_stage2_init_flags(). * @addr: Base intermediate physical address to annotate. * @size: Size of the annotated range. * @mc: Cache of pre-allocated and zeroed memory from which to allocate @@ -287,7 +289,8 @@ int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, /** * kvm_pgtable_stage2_unmap() - Remove a mapping from a guest stage-2 page-table. - * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @pgt: Page-table structure initialised by function + * kvm_pgtable_stage2_init_flags(). * @addr: Intermediate physical address from which to remove the mapping. * @size: Size of the mapping. * @@ -307,7 +310,8 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size); /** * kvm_pgtable_stage2_wrprotect() - Write-protect guest stage-2 address range * without TLB invalidation. - * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @pgt: Page-table structure initialised by function + * kvm_pgtable_stage2_init_flags(). * @addr: Intermediate physical address from which to write-protect, * @size: Size of the range. * @@ -324,7 +328,8 @@ int kvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size); /** * kvm_pgtable_stage2_mkyoung() - Set the access flag in a page-table entry. - * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @pgt: Page-table structure initialised by function + * kvm_pgtable_stage2_init_flags(). * @addr: Intermediate physical address to identify the page-table entry. * * The offset of @addr within a page is ignored. @@ -338,7 +343,8 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr); /** * kvm_pgtable_stage2_mkold() - Clear the access flag in a page-table entry. - * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @pgt: Page-table structure initialised by function + * kvm_pgtable_stage2_init_flags(). * @addr: Intermediate physical address to identify the page-table entry. * * The offset of @addr within a page is ignored. @@ -357,7 +363,8 @@ kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr); /** * kvm_pgtable_stage2_relax_perms() - Relax the permissions enforced by a * page-table entry. - * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @pgt: Page-table structure initialised by function + * kvm_pgtable_stage2_init_flags(). * @addr: Intermediate physical address to identify the page-table entry. * @prot: Additional permissions to grant for the mapping. * @@ -376,7 +383,8 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, /** * kvm_pgtable_stage2_is_young() - Test whether a page-table entry has the * access flag set. - * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @pgt: Page-table structure initialised by function + * kvm_pgtable_stage2_init_flags(). * @addr: Intermediate physical address to identify the page-table entry. * * The offset of @addr within a page is ignored. @@ -389,7 +397,8 @@ bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr); * kvm_pgtable_stage2_flush_range() - Clean and invalidate data cache to Point * of Coherency for guest stage-2 address * range. - * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @pgt: Page-table structure initialised by function + * kvm_pgtable_stage2_init_flags(). * @addr: Intermediate physical address from which to flush. * @size: Size of the range. * @@ -428,7 +437,8 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, * kvm_pgtable_stage2_find_range() - Find a range of Intermediate Physical * Addresses with compatible permission * attributes. - * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @pgt: Page-table structure initialised by function + * kvm_pgtable_stage2_init_flags(). * @addr: Address that must be covered by the range. * @prot: Protection attributes that the range must be compatible with. * @range: Range structure used to limit the search space at call time and diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index cd4d51ae3d4a..2cfcfc5f4e4e 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -457,7 +457,8 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) if (!pgt) return -ENOMEM; - err = kvm_pgtable_stage2_init(pgt, &kvm->arch, &kvm_s2_mm_ops); + err = kvm_pgtable_stage2_init_flags(pgt, &kvm->arch, &kvm_s2_mm_ops, + KVM_PGTABLE_S2_GUEST); if (err) goto out_free_pgtable; From patchwork Thu Apr 15 11:50:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yanan Wang X-Patchwork-Id: 12205223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2551EC43462 for ; Thu, 15 Apr 2021 11:53:06 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 93C06610FA for ; Thu, 15 Apr 2021 11:53:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 93C06610FA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=DxVB6bVl6Sm1TiTbts7ubFyvmC+N2enxiDWrYoYc1P4=; b=kRPqyLxIc3x+P0u0RnHPB1fDq /jLIMFp1IIfXIK3PvmAfLzDOLX3NMWDW/V3ldP4F6rIGPQypiXSkm/2Dr/vUA2Eh4uQd1JihHGu+a Pkz4xRaXRAXkbct1kODzcagQkRBifwPPQU95JB6K4PC2vG41JQL5A8Vfyq5UdiH/WueKoup+cGqge rsN47chXZ3Wki0QN1nWQkLPP864z3NC3AHp72V9IW1UJQMEEREobGSoWETdj6DwnSBg4xToK+0Rah EiRQ2RxGme9K/T84TuOal4K3QThPiih0CVLHxU/8xVUy4iRE3/UA4LJDbb1k2j+bxS9bkXR8O7Oj0 ZznAyRpVA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lX0Wn-00G1EW-Bd; Thu, 15 Apr 2021 11:51:21 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WP-00G1Ci-ML for linux-arm-kernel@desiato.infradead.org; Thu, 15 Apr 2021 11:50:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=2sO2lVhYBEaqPVkDV5RbKW318G+7hGpZLxnTCo3YBhk=; b=RCEeDUaO6FxQOBBTYJcjJJtyxH 2X3EFKLy/Qa4dDoLi2OVG7U3DrGkGq7vlBGMov7f7nT2m65rtyV1atIbGFiLo0bzZljuJpehlspuF b7EXPYLwl5O2y7Hj2JxNSwizHhOfFeFpSDaTGW2jrpBhv8v723kxZq3am28aZx1Yx9rATDINHOd+E Q9f43EGxZfXWQDx4h5gvafiKOK7DppWwr7JYzTyu1Uo0jL8XRzmX2sARPGM8FicP9F4EMlLy6DZeI AyqtZOzrtA1fsEgHi+X8yVWBiIG1DWqAaV0F7Sv+fi+VvmP0TCuIaktKoARr5F/jeimEoIqpWkHqq xSrJai6w==; Received: from szxga04-in.huawei.com ([45.249.212.190]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WM-008Xbn-5s for linux-arm-kernel@lists.infradead.org; Thu, 15 Apr 2021 11:50:56 +0000 Received: from DGGEMS407-HUB.china.huawei.com (unknown [172.30.72.60]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4FLcz75fP7zpYWg; Thu, 15 Apr 2021 19:47:47 +0800 (CST) Received: from DESKTOP-TMVL5KK.china.huawei.com (10.174.187.128) by DGGEMS407-HUB.china.huawei.com (10.3.19.207) with Microsoft SMTP Server id 14.3.498.0; Thu, 15 Apr 2021 19:50:36 +0800 From: Yanan Wang To: Marc Zyngier , Will Deacon , "Quentin Perret" , Alexandru Elisei , , , , CC: Catalin Marinas , James Morse , Julien Thierry , "Suzuki K Poulose" , Gavin Shan , , , , Yanan Wang Subject: [PATCH v5 2/6] KVM: arm64: Move D-cache flush to the fault handlers Date: Thu, 15 Apr 2021 19:50:28 +0800 Message-ID: <20210415115032.35760-3-wangyanan55@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20210415115032.35760-1-wangyanan55@huawei.com> References: <20210415115032.35760-1-wangyanan55@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.128] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210415_045054_605840_9C5CE3B6 X-CRM114-Status: GOOD ( 17.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We currently uniformly permorm CMOs of D-cache and I-cache in function user_mem_abort before calling the fault handlers. If we get concurrent guest faults(e.g. translation faults, permission faults) or some really unnecessary guest faults caused by BBM, CMOs for the first vcpu are necessary while the others later are not. By moving CMOs to the fault handlers, we can easily identify conditions where they are really needed and avoid the unnecessary ones. As it's a time consuming process to perform CMOs especially when flushing a block range, so this solution reduces much load of kvm and improve efficiency of the page table code. This patch only moves clean of D-cache to the map path, and drop the original APIs in mmu.c/mmu.h for D-cache maintenance by using what we already have in pgtable.c. Change about the I-side will come from a later patch. Signed-off-by: Yanan Wang --- arch/arm64/include/asm/kvm_mmu.h | 16 ---------------- arch/arm64/kvm/hyp/pgtable.c | 20 ++++++++++++++------ arch/arm64/kvm/mmu.c | 14 +++----------- 3 files changed, 17 insertions(+), 33 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 25ed956f9af1..e9b163c5f023 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -187,22 +187,6 @@ static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu) return (vcpu_read_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101; } -static inline void __clean_dcache_guest_page(kvm_pfn_t pfn, unsigned long size) -{ - void *va = page_address(pfn_to_page(pfn)); - - /* - * With FWB, we ensure that the guest always accesses memory using - * cacheable attributes, and we don't have to clean to PoC when - * faulting in pages. Furthermore, FWB implies IDC, so cleaning to - * PoU is not required either in this case. - */ - if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) - return; - - kvm_flush_dcache_to_poc(va, size); -} - static inline void __invalidate_icache_guest_page(kvm_pfn_t pfn, unsigned long size) { diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index c37c1dc4feaf..e3606c9dcec7 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -562,6 +562,12 @@ static bool stage2_pte_is_counted(kvm_pte_t pte) return !!pte; } +static bool stage2_pte_cacheable(struct kvm_pgtable *pgt, kvm_pte_t pte) +{ + u64 memattr = pte & KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR; + return memattr == KVM_S2_MEMATTR(pgt, NORMAL); +} + static void stage2_put_pte(kvm_pte_t *ptep, struct kvm_s2_mmu *mmu, u64 addr, u32 level, struct kvm_pgtable_mm_ops *mm_ops) { @@ -583,6 +589,7 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, { kvm_pte_t new, old = *ptep; u64 granule = kvm_granule_size(level), phys = data->phys; + struct kvm_pgtable *pgt = data->mmu->pgt; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; if (!kvm_block_mapping_supported(addr, end, phys, level)) @@ -606,6 +613,13 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, stage2_put_pte(ptep, data->mmu, addr, level, mm_ops); } + /* Perform CMOs before installation of the guest stage-2 PTE */ + if (pgt->flags & KVM_PGTABLE_S2_GUEST) { + if (stage2_pte_cacheable(pgt, new) && !stage2_has_fwb(pgt)) + __flush_dcache_area(mm_ops->phys_to_virt(phys), + granule); + } + smp_store_release(ptep, new); if (stage2_pte_is_counted(new)) mm_ops->get_page(ptep); @@ -798,12 +812,6 @@ int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, return ret; } -static bool stage2_pte_cacheable(struct kvm_pgtable *pgt, kvm_pte_t pte) -{ - u64 memattr = pte & KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR; - return memattr == KVM_S2_MEMATTR(pgt, NORMAL); -} - static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 2cfcfc5f4e4e..86f7dd1c234f 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -694,11 +694,6 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask); } -static void clean_dcache_guest_page(kvm_pfn_t pfn, unsigned long size) -{ - __clean_dcache_guest_page(pfn, size); -} - static void invalidate_icache_guest_page(kvm_pfn_t pfn, unsigned long size) { __invalidate_icache_guest_page(pfn, size); @@ -972,9 +967,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (writable) prot |= KVM_PGTABLE_PROT_W; - if (fault_status != FSC_PERM && !device) - clean_dcache_guest_page(pfn, vma_pagesize); - if (exec_fault) { prot |= KVM_PGTABLE_PROT_X; invalidate_icache_guest_page(pfn, vma_pagesize); @@ -1234,10 +1226,10 @@ int kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte) trace_kvm_set_spte_hva(hva); /* - * We've moved a page around, probably through CoW, so let's treat it - * just like a translation fault and clean the cache to the PoC. + * We've moved a page around, probably through CoW, so let's treat + * it just like a translation fault and the map handler will clean + * the cache to the PoC. */ - clean_dcache_guest_page(pfn, PAGE_SIZE); handle_hva_to_gpa(kvm, hva, end, &kvm_set_spte_handler, &pfn); return 0; } From patchwork Thu Apr 15 11:50:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yanan Wang X-Patchwork-Id: 12205219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, UNWANTED_LANGUAGE_BODY,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39D69C433ED for ; Thu, 15 Apr 2021 11:53:04 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D91FA61153 for ; Thu, 15 Apr 2021 11:53:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D91FA61153 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=QVb3kV5H1I2x546mUV83lqYbEbHQ1hFTNRYq3TKw60U=; b=WZ75kQ2NrBDSPJFMIXaoHlUFw k4i2hvouRatd4K1/Z71oNqWfswJM/P15L+bY7XvdK746jbacvtLl6yYZDmvh7qsHlqRaVpR2MuBgC a8+NFVSjVUw4Soos2JHhXxinshc7D28JOxWadDeW5KDOew/GGj65epKKVZm8ecNQuHb7RR8jEfU8/ 9eWFXIBfrOZ+31yLv+Fz92ojlc1dT/pZxGigunKCbIIsX46UPrnNk9Na/Qo5xvzAzaMHrY5c8X0L6 dq5BtmaMPlqKPkUL4B1bMbIACzDK4uOxwM5QtTT20FZGi6bvD4zeMtq6t61qMecyx2hZT7Nj+wDJa uoUZ3y3iQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WT-00G1DP-JF; Thu, 15 Apr 2021 11:51:01 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WP-00G1CW-MG for linux-arm-kernel@desiato.infradead.org; Thu, 15 Apr 2021 11:50:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=2cdk1W2fRyQessHJDQMZUPl3pmyTa/MaxLU2xWhM4/E=; b=iCfl9QWlSAjM2S9nCJNcHCz2aC 9omvAqJQkbd+dSuMhWTlPQWUWFBan0egh+VK7zyAKfbQK8Hq/HGTNHwihYMBtvsu6wfJsDw7i5DM5 Igmbg15dX8Ctw86wtzKzx+1njAcmQ3r+ys1ve9Dn2UjbQv7SPqFdNtxKx3PSFeerL70NIIALUqDeg GnkhNDdaL4SZ7V7ebcLUs4OzplVx+mlZUuONAe1D4K5X33JvPT3ahnXeuh89fvGFiJqT1LNJ884LH GHSCpC4dfJX0R1PRkhoFHQXN1YdzVNtmemJDP1cUvTHQ2rAt4DL6oZv2TWYY2dJVQXZEYGmwi9A6q ewSxUdYA==; Received: from szxga07-in.huawei.com ([45.249.212.35]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WL-008XcP-AY for linux-arm-kernel@lists.infradead.org; Thu, 15 Apr 2021 11:50:54 +0000 Received: from DGGEMS407-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga07-in.huawei.com (SkyGuard) with ESMTP id 4FLczw6xrNzB178; Thu, 15 Apr 2021 19:48:28 +0800 (CST) Received: from DESKTOP-TMVL5KK.china.huawei.com (10.174.187.128) by DGGEMS407-HUB.china.huawei.com (10.3.19.207) with Microsoft SMTP Server id 14.3.498.0; Thu, 15 Apr 2021 19:50:37 +0800 From: Yanan Wang To: Marc Zyngier , Will Deacon , "Quentin Perret" , Alexandru Elisei , , , , CC: Catalin Marinas , James Morse , Julien Thierry , "Suzuki K Poulose" , Gavin Shan , , , , Yanan Wang Subject: [PATCH v5 3/6] KVM: arm64: Add mm_ops member for structure stage2_attr_data Date: Thu, 15 Apr 2021 19:50:29 +0800 Message-ID: <20210415115032.35760-4-wangyanan55@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20210415115032.35760-1-wangyanan55@huawei.com> References: <20210415115032.35760-1-wangyanan55@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.128] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210415_045053_550590_12CFE7E0 X-CRM114-Status: UNSURE ( 9.64 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Also add a mm_ops member for structure stage2_attr_data, since we will move I-cache maintenance for guest stage-2 to the permission path and as a result will need mm_ops for address transformation. Signed-off-by: Yanan Wang --- arch/arm64/kvm/hyp/pgtable.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index e3606c9dcec7..b480f6d1171e 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -869,10 +869,11 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) } struct stage2_attr_data { - kvm_pte_t attr_set; - kvm_pte_t attr_clr; - kvm_pte_t pte; - u32 level; + kvm_pte_t attr_set; + kvm_pte_t attr_clr; + kvm_pte_t pte; + u32 level; + struct kvm_pgtable_mm_ops *mm_ops; }; static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, @@ -911,6 +912,7 @@ static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr, struct stage2_attr_data data = { .attr_set = attr_set & attr_mask, .attr_clr = attr_clr & attr_mask, + .mm_ops = pgt->mm_ops, }; struct kvm_pgtable_walker walker = { .cb = stage2_attr_walker, From patchwork Thu Apr 15 11:50:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yanan Wang X-Patchwork-Id: 12205217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E680C433B4 for ; Thu, 15 Apr 2021 11:53:04 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 05C61610FA for ; Thu, 15 Apr 2021 11:53:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 05C61610FA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=QgU2vKMaJ70vBAG3gn1pAGZfredbyc8LiVVDhDoMKdY=; b=nK6X7ZoNbq1TFDCsky3sJVi/G HyBQm3DNyV4wKROZbK2p42EufzPmG/Opq9aRMbPcZ+RNx/YDVZ1z4SnimmBrPPyWkH+41ivxExJ6D X4JTKCOtNzVBNG8QHqCkhxLuZf5Q99AX2wJwAAjI/zk0u5bscbiCoT3DN/YWrplUMguB6okGtNEER T28qsuPgkPFePsFAwiNa3H1sV6YL8tXKUDHIGXHCijKGG0rYi0ou2XVVpvTVsyJ4vwgsyPFX2c6rN cYvRcS5sRWTr6J669bFWXfNASoa3DNLbaXwQ7Xw+xvPqkO+LrASjDk4H2cW1hoYVUqXPjRAUPwvjo P6Sp+ymZg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lX0Ww-00G1GC-BV; Thu, 15 Apr 2021 11:51:31 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WP-00G1Ch-MJ for linux-arm-kernel@desiato.infradead.org; Thu, 15 Apr 2021 11:50:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=S6Qgf4Dx0JASURe4gC0D5vnpQiOcqshxt1TUwy61dF8=; b=JbJb1x5Q/muVHac0qisVK104FG QzTLnr3nQyu4/yvOGT8hE4bRp+tVllrCb9UPgLLsc1oKmw5PCvJJGFnqK6KtWUmKR3DW1x/Q+ASuv gJS+gphNpwB3vO1UMEqlrz02sI2Rssg/Lf3T35mKlaBqCHIbVHG6KZUgA1iPC3m7t0MdEoXYdGaD2 +/Wa+xYcp8INlxIeTi8JHm/kDO23vTZEB+NQ/8H48a6fclB/wXANWjdWoDL55owDEZVjP8AEIpbPa SOj35PZCeIiC0gUbHUDhpowh2iNSP7l/c1GWlp7LrrqhnpbG3xWEc2UcMIA1oP4Hlk+opUyiyjNLw Kb9s2srw==; Received: from szxga07-in.huawei.com ([45.249.212.35]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WM-008XcQ-Pl for linux-arm-kernel@lists.infradead.org; Thu, 15 Apr 2021 11:50:56 +0000 Received: from DGGEMS407-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga07-in.huawei.com (SkyGuard) with ESMTP id 4FLczw63WhzB10g; Thu, 15 Apr 2021 19:48:28 +0800 (CST) Received: from DESKTOP-TMVL5KK.china.huawei.com (10.174.187.128) by DGGEMS407-HUB.china.huawei.com (10.3.19.207) with Microsoft SMTP Server id 14.3.498.0; Thu, 15 Apr 2021 19:50:38 +0800 From: Yanan Wang To: Marc Zyngier , Will Deacon , "Quentin Perret" , Alexandru Elisei , , , , CC: Catalin Marinas , James Morse , Julien Thierry , "Suzuki K Poulose" , Gavin Shan , , , , Yanan Wang Subject: [PATCH v5 4/6] KVM: arm64: Provide invalidate_icache_range at non-VHE EL2 Date: Thu, 15 Apr 2021 19:50:30 +0800 Message-ID: <20210415115032.35760-5-wangyanan55@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20210415115032.35760-1-wangyanan55@huawei.com> References: <20210415115032.35760-1-wangyanan55@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.128] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210415_045055_015851_B2E7B4B0 X-CRM114-Status: UNSURE ( 9.42 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We want to move I-cache maintenance for the guest to the stage-2 page table code for performance improvement. Before it can work, we should first make function invalidate_icache_range available to non-VHE EL2 to avoid compiling or program running error, as pgtable.c is now linked into the non-VHE EL2 code for pKVM mode. In this patch, we only introduce symbol of invalidate_icache_range with no real functionality in nvhe/cache.S, because there haven't been situations found currently where I-cache maintenance is also needed in non-VHE EL2 for pKVM mode. Signed-off-by: Yanan Wang --- arch/arm64/kvm/hyp/nvhe/cache.S | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/arm64/kvm/hyp/nvhe/cache.S b/arch/arm64/kvm/hyp/nvhe/cache.S index 36cef6915428..a125ec9aeed2 100644 --- a/arch/arm64/kvm/hyp/nvhe/cache.S +++ b/arch/arm64/kvm/hyp/nvhe/cache.S @@ -11,3 +11,14 @@ SYM_FUNC_START_PI(__flush_dcache_area) dcache_by_line_op civac, sy, x0, x1, x2, x3 ret SYM_FUNC_END_PI(__flush_dcache_area) + +/* + * invalidate_icache_range(start,end) + * + * Ensure that the I cache is invalid within specified region. + * + * - start - virtual start address of region + * - end - virtual end address of region + */ +SYM_FUNC_START(invalidate_icache_range) +SYM_FUNC_END(invalidate_icache_range) From patchwork Thu Apr 15 11:50:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yanan Wang X-Patchwork-Id: 12205221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D45F2C43460 for ; Thu, 15 Apr 2021 11:53:04 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2D2206124B for ; Thu, 15 Apr 2021 11:53:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2D2206124B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Qxc/MzTgJENfSakX15NSwrMNi7oDYfmD5i3UAm9+S6U=; b=GsP145edLbnq6552rqb6ZMsQD 1lwQ8hWdv48s0Ntz6uYQZY7fGMV2sbvX/ZRtqoVqFf4WSkQmWXtuI+X5HnYm9JIOqSI5OA22tq9hl fHFqOWxS1SMbUjFSz/+F+4lvPy92L4raO4wJoT8eLTdcgw1f6IJU1ts2jd4zaZupQ4I2FeGqzQKgo eU4/Xy7JBXgIeldirceifZVk432W2JDWH3CVcJbYNeptQfG/AX1SFUcEHQ9DOIHDiinnQjOsr+X6n 91jGsZrf8s5SrJzja9X/2TcbilOdG/Vt5h/44jyy803R/wh/F71QGrNM4TnRBhpIjSwUt+HEnjmiz i0AahzFRg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lX0We-00G1Dv-4A; Thu, 15 Apr 2021 11:51:12 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WP-00G1Cg-MK for linux-arm-kernel@desiato.infradead.org; Thu, 15 Apr 2021 11:50:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=PFvGCzteOSrgmUK655E5LOFf4hVGDh/P7ArpW8y/c9A=; b=c35ODS1uHI2WtWfjJTvnIyqxrZ tYXXdxESUrB92hqiMNoJPKbglCTw/s7mBgTiEDU6WzpoVJ5VSAQ0sfF+kKIyU5dCObTYL1LBMERL4 fdtjkm6MDPIeflc/m/qRpHG8MQAS1NHU4+Rcx1CFJjDKNnhN+4UFj/sxHaFbQfxFcwUpFmJrhdSIj dMKNfn7ADGCPguPfLBKRhXl6FbbZ5gWggqR+oKIGkeCfYSkfXvu8CB7a5dxkEjrBCfKBO3jutnztv cVnxOUmOp1QcdFK7mMh3MhaLArOTcVianGujH2I5NWJPRFOVrikVyW5tm69edDMkIk92mE+PSMni6 GeAUeF+Q==; Received: from szxga07-in.huawei.com ([45.249.212.35]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WM-008XcR-6b for linux-arm-kernel@lists.infradead.org; Thu, 15 Apr 2021 11:50:56 +0000 Received: from DGGEMS407-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga07-in.huawei.com (SkyGuard) with ESMTP id 4FLczw6VZLzB13T; Thu, 15 Apr 2021 19:48:28 +0800 (CST) Received: from DESKTOP-TMVL5KK.china.huawei.com (10.174.187.128) by DGGEMS407-HUB.china.huawei.com (10.3.19.207) with Microsoft SMTP Server id 14.3.498.0; Thu, 15 Apr 2021 19:50:40 +0800 From: Yanan Wang To: Marc Zyngier , Will Deacon , "Quentin Perret" , Alexandru Elisei , , , , CC: Catalin Marinas , James Morse , Julien Thierry , "Suzuki K Poulose" , Gavin Shan , , , , Yanan Wang Subject: [PATCH v5 5/6] KVM: arm64: Move I-cache flush to the fault handlers Date: Thu, 15 Apr 2021 19:50:31 +0800 Message-ID: <20210415115032.35760-6-wangyanan55@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20210415115032.35760-1-wangyanan55@huawei.com> References: <20210415115032.35760-1-wangyanan55@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.128] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210415_045054_585118_F248E24C X-CRM114-Status: GOOD ( 15.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In this patch, we move invalidation of I-cache to the fault handlers to avoid unnecessary I-cache maintenances. On the map path, invalidate the I-cache if we are going to create an executable stage-2 mapping for guest. And on the permission path, invalidate the I-cache if we are going to add an executable permission to the existing guest stage-2 mapping. Signed-off-by: Yanan Wang --- arch/arm64/include/asm/kvm_mmu.h | 15 -------------- arch/arm64/kvm/hyp/pgtable.c | 35 +++++++++++++++++++++++++++++++- arch/arm64/kvm/mmu.c | 9 +------- 3 files changed, 35 insertions(+), 24 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index e9b163c5f023..155492fe5b15 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -187,21 +187,6 @@ static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu) return (vcpu_read_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101; } -static inline void __invalidate_icache_guest_page(kvm_pfn_t pfn, - unsigned long size) -{ - if (icache_is_aliasing()) { - /* any kind of VIPT cache */ - __flush_icache_all(); - } else if (is_kernel_in_hyp_mode() || !icache_is_vpipt()) { - /* PIPT or VPIPT at EL2 (see comment in __kvm_tlb_flush_vmid_ipa) */ - void *va = page_address(pfn_to_page(pfn)); - - invalidate_icache_range((unsigned long)va, - (unsigned long)va + size); - } -} - void kvm_set_way_flush(struct kvm_vcpu *vcpu); void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index b480f6d1171e..9f4429d80df0 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -568,6 +568,26 @@ static bool stage2_pte_cacheable(struct kvm_pgtable *pgt, kvm_pte_t pte) return memattr == KVM_S2_MEMATTR(pgt, NORMAL); } +static bool stage2_pte_executable(kvm_pte_t pte) +{ + return !(pte & KVM_PTE_LEAF_ATTR_HI_S2_XN); +} + +static void stage2_invalidate_icache(void *addr, u64 size) +{ + if (icache_is_aliasing()) { + /* Any kind of VIPT cache */ + __flush_icache_all(); + } else if (is_kernel_in_hyp_mode() || !icache_is_vpipt()) { + /* + * See comment in __kvm_tlb_flush_vmid_ipa(). + * Invalidate PIPT, or VPIPT at EL2. + */ + invalidate_icache_range((unsigned long)addr, + (unsigned long)addr + size); + } +} + static void stage2_put_pte(kvm_pte_t *ptep, struct kvm_s2_mmu *mmu, u64 addr, u32 level, struct kvm_pgtable_mm_ops *mm_ops) { @@ -618,6 +638,10 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (stage2_pte_cacheable(pgt, new) && !stage2_has_fwb(pgt)) __flush_dcache_area(mm_ops->phys_to_virt(phys), granule); + + if (stage2_pte_executable(new)) + stage2_invalidate_icache(mm_ops->phys_to_virt(phys), + granule); } smp_store_release(ptep, new); @@ -896,8 +920,17 @@ static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, * but worst-case the access flag update gets lost and will be * set on the next access instead. */ - if (data->pte != pte) + if (data->pte != pte) { + /* + * Invalidate the instruction cache before updating + * if we are going to add the executable permission + * for the guest stage-2 PTE. + */ + if (!stage2_pte_executable(*ptep) && stage2_pte_executable(pte)) + stage2_invalidate_icache(kvm_pte_follow(pte, data->mm_ops), + kvm_granule_size(level)); WRITE_ONCE(*ptep, pte); + } return 0; } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 86f7dd1c234f..aa536392b308 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -694,11 +694,6 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask); } -static void invalidate_icache_guest_page(kvm_pfn_t pfn, unsigned long size) -{ - __invalidate_icache_guest_page(pfn, size); -} - static void kvm_send_hwpoison_signal(unsigned long address, short lsb) { send_sig_mceerr(BUS_MCEERR_AR, (void __user *)address, lsb, current); @@ -967,10 +962,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (writable) prot |= KVM_PGTABLE_PROT_W; - if (exec_fault) { + if (exec_fault) prot |= KVM_PGTABLE_PROT_X; - invalidate_icache_guest_page(pfn, vma_pagesize); - } if (device) prot |= KVM_PGTABLE_PROT_DEVICE; From patchwork Thu Apr 15 11:50:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yanan Wang X-Patchwork-Id: 12205227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAC87C433B4 for ; Thu, 15 Apr 2021 11:53:12 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3A3B2610FA for ; Thu, 15 Apr 2021 11:53:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3A3B2610FA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=uZFJIHe9OqRF5vF3DxlJY9oGIs8e+g6wiwy97RurCoY=; b=hIydvQFFx3Rin9jSVN3dbyOu8 0Sik+hDqt5Hy8pn7Y6VOVWGFMTx6KW7WDZ2mt/mt7fH41HPeDut/jSIIh9HBN+xyDZF1lqBNVJqqh FDQdY59bp+M1vBEMkfKJDp8RwLkGMt1eeeOVAWjzcQU92oWcBZ3SXR8ysaWhD1LdoYVhnZxlla6oR w49DF5OA5pE5B9siSgOJ1f+GsmsX5/VKXNlzJW1xkIz8lUVBPeFXLyFLZknlKZG7P2U3OOnJFgybO I6PpVUWmlCNIf6YV7mtbkGKg4ewshqHRqCrDfVT9EbsGhD1/XBHMbYyi+Vdt8cW+CzQpKwJilSm6s eOz35s9sQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lX0XH-00G1Jn-A2; Thu, 15 Apr 2021 11:51:51 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WQ-00G1Ck-NC for linux-arm-kernel@desiato.infradead.org; Thu, 15 Apr 2021 11:50:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=IukGuNTvROSjhe+Vy2C1r5vX+BRSkKYr1j7bl3RlkkQ=; b=rfh8cOST3HbU7JvlRLQ7eBFbuM E8NkH/KBt5m56AJZgd7vSoFa5r55oYrIaaeyvhNg83v2mo9mphv0UF7Qf2PZnAJ5aJbGFXVkqT2qj oyKB02vuIVRe0Pcd+VafGw0c5/dEAYZwrO1qh5X0SwH0xJ8jpeHzb8jQPd3kC6I62Kw/TAnR3+BNN nOzccLEIO13mlE1IAztQl5cdwKzsQ9FMoAKscIx5W2HAwZG4M+QUuG54Mll8XVwKHLzjOG6S3IauH Q319SLRdmdaeEka0kSKMtSmFuAQEAjVTb8KDNzcs+5Mjmq3o9XG0pj1zr7J2cD1UarOVanSSoQfD1 bwDqn/RA==; Received: from szxga06-in.huawei.com ([45.249.212.32]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX0WN-008Xcu-9g for linux-arm-kernel@lists.infradead.org; Thu, 15 Apr 2021 11:50:57 +0000 Received: from DGGEMS407-HUB.china.huawei.com (unknown [172.30.72.60]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4FLd0W4RGXzlXKt; Thu, 15 Apr 2021 19:48:59 +0800 (CST) Received: from DESKTOP-TMVL5KK.china.huawei.com (10.174.187.128) by DGGEMS407-HUB.china.huawei.com (10.3.19.207) with Microsoft SMTP Server id 14.3.498.0; Thu, 15 Apr 2021 19:50:41 +0800 From: Yanan Wang To: Marc Zyngier , Will Deacon , "Quentin Perret" , Alexandru Elisei , , , , CC: Catalin Marinas , James Morse , Julien Thierry , "Suzuki K Poulose" , Gavin Shan , , , , Yanan Wang Subject: [PATCH v5 6/6] KVM: arm64: Distinguish cases of memcache allocations completely Date: Thu, 15 Apr 2021 19:50:32 +0800 Message-ID: <20210415115032.35760-7-wangyanan55@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20210415115032.35760-1-wangyanan55@huawei.com> References: <20210415115032.35760-1-wangyanan55@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.128] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210415_045055_523478_192449A9 X-CRM114-Status: GOOD ( 17.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org With a guest translation fault, the memcache pages are not needed if KVM is only about to install a new leaf entry into the existing page table. And with a guest permission fault, the memcache pages are also not needed for a write_fault in dirty-logging time if KVM is only about to update the existing leaf entry instead of collapsing a block entry into a table. By comparing fault_granule and vma_pagesize, cases that require allocations from memcache and cases that don't can be distinguished completely. Signed-off-by: Yanan Wang --- arch/arm64/kvm/mmu.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index aa536392b308..9e35aa5d29f2 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -895,19 +895,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, gfn = fault_ipa >> PAGE_SHIFT; mmap_read_unlock(current->mm); - /* - * Permission faults just need to update the existing leaf entry, - * and so normally don't require allocations from the memcache. The - * only exception to this is when dirty logging is enabled at runtime - * and a write fault needs to collapse a block entry into a table. - */ - if (fault_status != FSC_PERM || (logging_active && write_fault)) { - ret = kvm_mmu_topup_memory_cache(memcache, - kvm_mmu_cache_min_pages(kvm)); - if (ret) - return ret; - } - mmu_seq = vcpu->kvm->mmu_notifier_seq; /* * Ensure the read of mmu_notifier_seq happens before we call @@ -970,6 +957,18 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC)) prot |= KVM_PGTABLE_PROT_X; + /* + * Allocations from the memcache are required only when granule of the + * lookup level where the guest fault happened exceeds vma_pagesize, + * which means new page tables will be created in the fault handlers. + */ + if (fault_granule > vma_pagesize) { + ret = kvm_mmu_topup_memory_cache(memcache, + kvm_mmu_cache_min_pages(kvm)); + if (ret) + return ret; + } + /* * Under the premise of getting a FSC_PERM fault, we just need to relax * permissions only if vma_pagesize equals fault_granule. Otherwise,