From patchwork Wed Mar 10 09:43:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yanan Wang X-Patchwork-Id: 12127443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4388CC433E0 for ; Wed, 10 Mar 2021 09:45:29 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BA29E64F96 for ; Wed, 10 Mar 2021 09:45:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA29E64F96 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7UDelYQDuwC/KzVLKlWGV918npPhYxTECVACOfja79s=; b=ju0nhs1ER+z43uVX9w+Hi5ZMB ajwRz1SdyBSipe2zVCyC07/i7+P/3R7FEyIosmf1IQiHQSxZXCAL45hlcw0nf0Yihwe9NsFsGPwvK B7bm3vOIGz43aRd3cT1DOoVuS06cYYod2dEjpc22dOVo8xIwUe45/f8x9vO3nRxNOL2JBzcZIIeP4 9mrcDVHVSyplZMVYsYD7y3voX1w3L0nlHd8l1pbPyH2hpNs9+Q4YrlXx898Cjk7de4Ta+NPwoOIqp T5ZMTzYVKCH9vE5E9XPHA9+CU713x5jsFr2nfzAAXo7PrQgzqboQHWt7oMtv+vHbQ0evCIoOR34sJ mszok3N+A==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lJvNu-006UBr-Dn; Wed, 10 Mar 2021 09:44:06 +0000 Received: from szxga06-in.huawei.com ([45.249.212.32]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJvNV-006U40-Ge for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 09:43:45 +0000 Received: from DGGEMS412-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4DwRtg0c5szkX1y; Wed, 10 Mar 2021 17:42:03 +0800 (CST) Received: from DESKTOP-TMVL5KK.china.huawei.com (10.174.187.128) by DGGEMS412-HUB.china.huawei.com (10.3.19.212) with Microsoft SMTP Server id 14.3.498.0; Wed, 10 Mar 2021 17:43:23 +0800 From: Yanan Wang To: Marc Zyngier , Will Deacon , "Catalin Marinas" , James Morse , "Julien Thierry" , Suzuki K Poulose , Gavin Shan , Quentin Perret , , , , CC: , , , Yanan Wang Subject: [RFC PATCH v2 2/3] KVM: arm64: Install the block entry before unmapping the page mappings Date: Wed, 10 Mar 2021 17:43:18 +0800 Message-ID: <20210310094319.18760-3-wangyanan55@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20210310094319.18760-1-wangyanan55@huawei.com> References: <20210310094319.18760-1-wangyanan55@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.128] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_094342_185940_BC0F4498 X-CRM114-Status: GOOD ( 18.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When KVM needs to coalesce the existing normal page mappings into a block mapping, we currently follow the following steps successively: 1) invalidate the table entry in the PMD/PUD table 2) flush TLB by VMID 3) unmap the old sub-level tables 4) install the new block entry to the PMD/PUD table It will cost a long time to unmap the old page mappings in step 3, which means there will be a long period when the PMD/PUD table entry could be found invalid (step 1, 2, 3). So the other vcpus have a really big chance to trigger unnecessary translations if they access any page within the block and find the table entry invalid. So let's quickly install the block entry at first to ensure uninterrupted memory access of the other vcpus, and then unmap the page mappings after installation. This will reduce most of the time when the table entry is invalid, and avoid most of the unnecessary translation faults. After this patch the steps can be like: 1) invalidate the table entry in the PMD/PUD table 2) flush TLB by VMID 3) install the new block entry to the PMD/PUD table 4) unmap the old sub-level tables Signed-off-by: Yanan Wang --- arch/arm64/kvm/hyp/pgtable.c | 44 ++++++++++++++++++++++++------------ 1 file changed, 30 insertions(+), 14 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 829a34eea526..b40b1f1615c7 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -434,6 +434,7 @@ struct stage2_map_data { kvm_pte_t attr; kvm_pte_t *anchor; + kvm_pte_t *follow; struct kvm_s2_mmu *mmu; struct kvm_mmu_memory_cache *memcache; @@ -545,6 +546,24 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, return 0; } +static void stage2_coalesce_tables_into_block(u64 addr, u32 level, + kvm_pte_t *ptep, + struct stage2_map_data *data) +{ + u64 granule = kvm_granule_size(level), phys = data->phys; + kvm_pte_t new = kvm_init_valid_leaf_pte(phys, data->attr, level); + + kvm_set_invalid_pte(ptep); + + /* + * Invalidate the whole stage-2, as we may have numerous leaf entries + * below us which would otherwise need invalidating individually. + */ + kvm_call_hyp(__kvm_tlb_flush_vmid, data->mmu); + smp_store_release(ptep, new); + data->phys += granule; +} + static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) @@ -555,15 +574,14 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, if (!kvm_block_mapping_supported(addr, end, data->phys, level)) return 0; - kvm_set_invalid_pte(ptep); - /* - * Invalidate the whole stage-2, as we may have numerous leaf - * entries below us which would otherwise need invalidating - * individually. + * If we need to coalesce existing table entries into a block here, + * then install the block entry first and the sub-level page mappings + * will be unmapped later. */ - kvm_call_hyp(__kvm_tlb_flush_vmid, data->mmu); data->anchor = ptep; + data->follow = kvm_pte_follow(*ptep); + stage2_coalesce_tables_into_block(addr, level, ptep, data); return 0; } @@ -616,20 +634,18 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) { - int ret = 0; - if (!data->anchor) return 0; - free_page((unsigned long)kvm_pte_follow(*ptep)); - put_page(virt_to_page(ptep)); - - if (data->anchor == ptep) { + if (data->anchor != ptep) { + free_page((unsigned long)kvm_pte_follow(*ptep)); + put_page(virt_to_page(ptep)); + } else { + free_page((unsigned long)data->follow); data->anchor = NULL; - ret = stage2_map_walk_leaf(addr, end, level, ptep, data); } - return ret; + return 0; } /*