From patchwork Mon Feb 6 16:58:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27105C636D4 for ; Mon, 6 Feb 2023 16:58:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230430AbjBFQ66 (ORCPT ); Mon, 6 Feb 2023 11:58:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36652 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230397AbjBFQ65 (ORCPT ); Mon, 6 Feb 2023 11:58:57 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B30812870 for ; Mon, 6 Feb 2023 08:58:56 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id 201-20020a6300d2000000b004ccf545f44fso5406895pga.12 for ; Mon, 06 Feb 2023 08:58:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=P3PXZO2PAZ/1nsWzQyVXqhwuxW1n83TTriFfsfUCsQA=; b=L4RNQJZL1MT/Aad1VHABZMMB22m0kayodkbnYbokIBzqNcAJmn4Y/NIBteI7b+9Qy2 H9f7LwtCQx/TA7Jfp+rYtQ6bsMT0UgKYOalNOwe5sJW1ydOJB0oHp6YQA/ALnpB73rAu f/YYAhop4pNSqMAC1Ezfe1UwI9xNmDL4GXM7WjXsUiqYCn5oWC0XwjWvJiO2HK1V2KFK zVPzkbruwA54lKSjGKnE6Phx56m7dp30o+XsWg24I3XD+1OawhNAAf4KMHW/QSUIWsGz FeC4ysjVn/BaeQSwRYh6IC6cpAD4biZoZNkY/yAHN7iYD0+UT3ZRq57AGjEP4p2X4RPm epCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=P3PXZO2PAZ/1nsWzQyVXqhwuxW1n83TTriFfsfUCsQA=; b=DucZ6/Yz1ocWSHroUgc5rV9xB2amqIB0GC8mDkSHM3TYuemf++dYykOR7JCzd82+6x y6dIbNk7j1txvNG0ZphbMpImH0fIF22U2QIDuHBnsUX87WMUC4L1WHS6oGXV63YCunrG fr+ZbxlAD0v38cvavzm3ShavyVKQmcv/dT7bVR3mkacrBisL5awwYjSVKJFGKg61OQB0 W5sje4vVVqXMK8OaWCHEvXmwk0I/rSxh1caHfbh40YznAm4oD5s10KNg6yw/nLkvU2fT 5ENXNxpmksuBfuwIBO3/6XDhqPcHRcwtYVO101X8ClEQH7TbjA/BCmaE6VNrtL7N/y8x /12w== X-Gm-Message-State: AO0yUKUNvFod6ntVkV5bD9IdSJxnMpgEzATINw7ceR1flM5ig+I+O2sL kAuhmI951tp1+p+KbUEbl0CkH4rETAbW0Q== X-Google-Smtp-Source: AK7set+90OrweG69CIxyYmFiQ3bdqgsDLbX1w8hVLWMGuRqtT43AVv2Q4Ruvd4/uGc9cuLGsQLYNjxtcfJrqdA== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a17:90a:195c:b0:230:c4f5:8577 with SMTP id 28-20020a17090a195c00b00230c4f58577mr100062pjh.20.1675702735889; Mon, 06 Feb 2023 08:58:55 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:40 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-2-ricarkol@google.com> Subject: [PATCH v2 01/12] KVM: arm64: Add KVM_PGTABLE_WALK ctx->flags for skipping BBM and CMO From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add two flags to kvm_pgtable_visit_ctx, KVM_PGTABLE_WALK_SKIP_BBM and KVM_PGTABLE_WALK_SKIP_CMO, to indicate that the walk should not perform break-before-make (BBM) nor cache maintenance operations (CMO). This will by a future commit to create unlinked tables not accessible to the HW page-table walker. This is safe as these removed tables are not visible to the HW page-table walker. Signed-off-by: Ricardo Koller --- arch/arm64/include/asm/kvm_pgtable.h | 18 ++++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 27 ++++++++++++++++----------- 2 files changed, 34 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 63f81b27a4e3..3339192a97a9 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -188,12 +188,20 @@ typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end, * children. * @KVM_PGTABLE_WALK_SHARED: Indicates the page-tables may be shared * with other software walkers. + * @KVM_PGTABLE_WALK_SKIP_BBM: Visit and update table entries + * without Break-before-make + * requirements. + * @KVM_PGTABLE_WALK_SKIP_CMO: Visit and update table entries + * without Cache maintenance + * operations required. */ enum kvm_pgtable_walk_flags { KVM_PGTABLE_WALK_LEAF = BIT(0), KVM_PGTABLE_WALK_TABLE_PRE = BIT(1), KVM_PGTABLE_WALK_TABLE_POST = BIT(2), KVM_PGTABLE_WALK_SHARED = BIT(3), + KVM_PGTABLE_WALK_SKIP_BBM = BIT(4), + KVM_PGTABLE_WALK_SKIP_CMO = BIT(4), }; struct kvm_pgtable_visit_ctx { @@ -215,6 +223,16 @@ static inline bool kvm_pgtable_walk_shared(const struct kvm_pgtable_visit_ctx *c return ctx->flags & KVM_PGTABLE_WALK_SHARED; } +static inline bool kvm_pgtable_walk_skip_bbm(const struct kvm_pgtable_visit_ctx *ctx) +{ + return ctx->flags & KVM_PGTABLE_WALK_SKIP_BBM; +} + +static inline bool kvm_pgtable_walk_skip_cmo(const struct kvm_pgtable_visit_ctx *ctx) +{ + return ctx->flags & KVM_PGTABLE_WALK_SKIP_CMO; +} + /** * struct kvm_pgtable_walker - Hook into a page-table walk. * @cb: Callback function to invoke during the walk. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index b11cf2c618a6..e093e222daf3 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -717,14 +717,17 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, if (!stage2_try_set_pte(ctx, KVM_INVALID_PTE_LOCKED)) return false; - /* - * Perform the appropriate TLB invalidation based on the evicted pte - * value (if any). - */ - if (kvm_pte_table(ctx->old, ctx->level)) - kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); - else if (kvm_pte_valid(ctx->old)) - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level); + if (!kvm_pgtable_walk_skip_bbm(ctx)) { + /* + * Perform the appropriate TLB invalidation based on the + * evicted pte value (if any). + */ + if (kvm_pte_table(ctx->old, ctx->level)) + kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + else if (kvm_pte_valid(ctx->old)) + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, + ctx->addr, ctx->level); + } if (stage2_pte_is_counted(ctx->old)) mm_ops->put_page(ctx->ptep); @@ -808,11 +811,13 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx, return -EAGAIN; /* Perform CMOs before installation of the guest stage-2 PTE */ - if (mm_ops->dcache_clean_inval_poc && stage2_pte_cacheable(pgt, new)) + if (!kvm_pgtable_walk_skip_cmo(ctx) && mm_ops->dcache_clean_inval_poc && + stage2_pte_cacheable(pgt, new)) mm_ops->dcache_clean_inval_poc(kvm_pte_follow(new, mm_ops), - granule); + granule); - if (mm_ops->icache_inval_pou && stage2_pte_executable(new)) + if (!kvm_pgtable_walk_skip_cmo(ctx) && mm_ops->icache_inval_pou && + stage2_pte_executable(new)) mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule); stage2_make_pte(ctx, new); From patchwork Mon Feb 6 16:58:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2AA4C63797 for ; Mon, 6 Feb 2023 16:59:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230465AbjBFQ7B (ORCPT ); Mon, 6 Feb 2023 11:59:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36696 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230432AbjBFQ67 (ORCPT ); Mon, 6 Feb 2023 11:58:59 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08F0B12870 for ; Mon, 6 Feb 2023 08:58:58 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id r17-20020a17090aa09100b0021903e75f14so4780433pjp.9 for ; Mon, 06 Feb 2023 08:58:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mdF2nBrDI9n7NEZ3J9+632BCKkHv+FRSzZtaOODcshg=; b=j2E/+YJw1xCD4yUATk8cjcM0VFhBlllfGB1fNbsbWLl1vwyXYAIUyNTUyK6TjEUUU7 8Uc/Wk2SFrnj/6LATHyacHGzVx4tGAiftpBrl6RwYe3iblKLFeTG6VOqSZO9w+otOF+q sHGBn6IF36GHSYavCF/6vqofi56NrHlRx1+njAXEedh28WbWmb7LPvqfuECtkkftRW1b GrBKy8/YTbPCAVIqWoQ9jRgt30bqeAEjR4g1qcG/xzTs172G7uQAItqRQF8tqUJtz+Zq nIJAz6EbMDmleE0vkTqrMhOhlE4z2JnY1nsD0yiJpe38ulooV57LXhqpZT1AYKUbEc68 YuOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mdF2nBrDI9n7NEZ3J9+632BCKkHv+FRSzZtaOODcshg=; b=FzRZBapMyTC+6iRm4uyGqeHDRF+IKYU8N+R7A+vNJpwn6BDH5djPBmzaa7KImGaywa 7JWc8O5dxuWFbDJPyO8TPKnLHNx2aWdTQJtvhKVTIRhmelBQAOWkt7iob6es99+cozFf Ai0hIeZYRH/JHslShEuuxsDaW/uDR3DThxf+NH2wj+f6i+q9HcK6yXqFZLOlp95Y3zmf J4PUAhv78ajLAPxhQnZeUYRcvRb3C6rvO4cwT2mlgomqvbsjT9tF6WcWPKDWSs27uuCr GvZE6ARhP91QpLKHt1s5hQdEDpRNVcdOaVrnVGSRMsQs/B0ARbrxlpFYZX0BMOqpSw6k PXCQ== X-Gm-Message-State: AO0yUKVU9FbxJtgHUD3FKPj6A0wVRa51rlevBoHaRlwCkBtJt4KgB/uz o3utbsKlUJo14nsyoXCWwhgVJxRf7+D9uQ== X-Google-Smtp-Source: AK7set/rkHoSyh75SWE6JNTuD7TTvZidZ641Oy+26NGZh4imEdWx1oL/rYP8K0hGHSbcG+/9Tp0PGyb0L4nthA== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a17:90a:688c:b0:230:a3b9:5dc7 with SMTP id a12-20020a17090a688c00b00230a3b95dc7mr1120581pjd.72.1675702737493; Mon, 06 Feb 2023 08:58:57 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:41 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-3-ricarkol@google.com> Subject: [PATCH v2 02/12] KVM: arm64: Rename free_unlinked to free_removed From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Make it clearer that the "free_removed" functions refer to tables that have never been part of the paging structure: they are "unlinked". No functional change intended. Signed-off-by: Ricardo Koller --- arch/arm64/include/asm/kvm_pgtable.h | 8 ++++---- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 6 +++--- arch/arm64/kvm/hyp/pgtable.c | 6 +++--- arch/arm64/kvm/mmu.c | 10 +++++----- 4 files changed, 15 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 3339192a97a9..7c45082e6c23 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -99,7 +99,7 @@ static inline bool kvm_level_supports_block_mapping(u32 level) * allocation is physically contiguous. * @free_pages_exact: Free an exact number of memory pages previously * allocated by zalloc_pages_exact. - * @free_removed_table: Free a removed paging structure by unlinking and + * @free_unlinked_table: Free an unlinked paging structure by unlinking and * dropping references. * @get_page: Increment the refcount on a page. * @put_page: Decrement the refcount on a page. When the @@ -119,7 +119,7 @@ struct kvm_pgtable_mm_ops { void* (*zalloc_page)(void *arg); void* (*zalloc_pages_exact)(size_t size); void (*free_pages_exact)(void *addr, size_t size); - void (*free_removed_table)(void *addr, u32 level); + void (*free_unlinked_table)(void *addr, u32 level); void (*get_page)(void *addr); void (*put_page)(void *addr); int (*page_count)(void *addr); @@ -450,7 +450,7 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu, void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); /** - * kvm_pgtable_stage2_free_removed() - Free a removed stage-2 paging structure. + * kvm_pgtable_stage2_free_unlinked() - Free un unlinked stage-2 paging structure. * @mm_ops: Memory management callbacks. * @pgtable: Unlinked stage-2 paging structure to be freed. * @level: Level of the stage-2 paging structure to be freed. @@ -458,7 +458,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); * The page-table is assumed to be unreachable by any hardware walkers prior to * freeing and therefore no TLB invalidation is performed. */ -void kvm_pgtable_stage2_free_removed(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level); +void kvm_pgtable_stage2_free_unlinked(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level); /** * kvm_pgtable_stage2_map() - Install a mapping in a guest stage-2 page-table. diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 552653fa18be..b030170d803b 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -91,9 +91,9 @@ static void host_s2_put_page(void *addr) hyp_put_page(&host_s2_pool, addr); } -static void host_s2_free_removed_table(void *addr, u32 level) +static void host_s2_free_unlinked_table(void *addr, u32 level) { - kvm_pgtable_stage2_free_removed(&host_mmu.mm_ops, addr, level); + kvm_pgtable_stage2_free_unlinked(&host_mmu.mm_ops, addr, level); } static int prepare_s2_pool(void *pgt_pool_base) @@ -110,7 +110,7 @@ static int prepare_s2_pool(void *pgt_pool_base) host_mmu.mm_ops = (struct kvm_pgtable_mm_ops) { .zalloc_pages_exact = host_s2_zalloc_pages_exact, .zalloc_page = host_s2_zalloc_page, - .free_removed_table = host_s2_free_removed_table, + .free_unlinked_table = host_s2_free_unlinked_table, .phys_to_virt = hyp_phys_to_virt, .virt_to_phys = hyp_virt_to_phys, .page_count = hyp_page_count, diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index e093e222daf3..0a5ef9288371 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -841,7 +841,7 @@ static int stage2_map_walk_table_pre(const struct kvm_pgtable_visit_ctx *ctx, if (ret) return ret; - mm_ops->free_removed_table(childp, ctx->level); + mm_ops->free_unlinked_table(childp, ctx->level); return 0; } @@ -886,7 +886,7 @@ static int stage2_map_walk_leaf(const struct kvm_pgtable_visit_ctx *ctx, * The TABLE_PRE callback runs for table entries on the way down, looking * for table entries which we could conceivably replace with a block entry * for this mapping. If it finds one it replaces the entry and calls - * kvm_pgtable_mm_ops::free_removed_table() to tear down the detached table. + * kvm_pgtable_mm_ops::free_unlinked_table() to tear down the detached table. * * Otherwise, the LEAF callback performs the mapping at the existing leaves * instead. @@ -1250,7 +1250,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt) pgt->pgd = NULL; } -void kvm_pgtable_stage2_free_removed(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level) +void kvm_pgtable_stage2_free_unlinked(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level) { kvm_pteref_t ptep = (kvm_pteref_t)pgtable; struct kvm_pgtable_walker walker = { diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index a3ee3b605c9b..9bd3c2cfb476 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -130,21 +130,21 @@ static void kvm_s2_free_pages_exact(void *virt, size_t size) static struct kvm_pgtable_mm_ops kvm_s2_mm_ops; -static void stage2_free_removed_table_rcu_cb(struct rcu_head *head) +static void stage2_free_unlinked_table_rcu_cb(struct rcu_head *head) { struct page *page = container_of(head, struct page, rcu_head); void *pgtable = page_to_virt(page); u32 level = page_private(page); - kvm_pgtable_stage2_free_removed(&kvm_s2_mm_ops, pgtable, level); + kvm_pgtable_stage2_free_unlinked(&kvm_s2_mm_ops, pgtable, level); } -static void stage2_free_removed_table(void *addr, u32 level) +static void stage2_free_unlinked_table(void *addr, u32 level) { struct page *page = virt_to_page(addr); set_page_private(page, (unsigned long)level); - call_rcu(&page->rcu_head, stage2_free_removed_table_rcu_cb); + call_rcu(&page->rcu_head, stage2_free_unlinked_table_rcu_cb); } static void kvm_host_get_page(void *addr) @@ -681,7 +681,7 @@ static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = { .zalloc_page = stage2_memcache_zalloc_page, .zalloc_pages_exact = kvm_s2_zalloc_pages_exact, .free_pages_exact = kvm_s2_free_pages_exact, - .free_removed_table = stage2_free_removed_table, + .free_unlinked_table = stage2_free_unlinked_table, .get_page = kvm_host_get_page, .put_page = kvm_s2_put_page, .page_count = kvm_host_page_count, From patchwork Mon Feb 6 16:58:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 872BAC05027 for ; Mon, 6 Feb 2023 16:59:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229648AbjBFQ7D (ORCPT ); Mon, 6 Feb 2023 11:59:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229924AbjBFQ7A (ORCPT ); Mon, 6 Feb 2023 11:59:00 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92E6D29E2D for ; Mon, 6 Feb 2023 08:58:59 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id g5-20020a62e305000000b00593dc84b678so6745928pfh.18 for ; Mon, 06 Feb 2023 08:58:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lwDAN8xTdbSeYamNw9rt6CbOQWpvlazD5x5calnCoVc=; b=mtXtryOgkR93yN+xk/TEeiHfhjAjCQ2JxYPvvCu6Qt9hItX1GUcPhFa4L0K89IKwrY diOoruEAxsfZ6uZv7v1BnwmHuhP4zQjg5dttXdKCJGezPraDeutt0OWeDyYVuLPNhl8O 2DdAI40mR8Tn1n+ztCr8ALnSS55XTkvfMXEMsezdzotFa/ifjdsTb7c9aDTJehYZrrqM qf5FGUZVkW0/AeXXONrTUqpCS41YKD7n3w5WrfOCWUXH0upfeLwET0UVZhct0vmVTrMT uEB+ikXwq65Ih/90/rzKpgwOwcZqSBhszqCS+/IwPDk/obaNNAc4BvemnYknYOpxARdr LHWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lwDAN8xTdbSeYamNw9rt6CbOQWpvlazD5x5calnCoVc=; b=xYkBX0ThusXp4jaNdOlByTqhGAraAKL9y6hIH6/A8MVHCSNaDckZ6PIkjbt8cN8wTi T2yNnW/1qF3dTy/35hbX0tzvWVZBkaIG9ksFcMZvJ0GzV5qotDgs7vfSt7roiRHOdzaL yW5GhAjLxpxcwIDltJGUYsFqtaugYH7EdVHtK6KX9HZe2H8lF1wDjC3pt6MuCzBPa3Kk zHQudDpzcOWXb0Ba5OFTLY2NB2T5ViM4YDskgFm1T21l9XPNJyFXU9HZqGSZYZdfsyV6 XjpoCvnSWcl+FjhqD8/5Ka76yebNiLJ/N5iTuaS/2K7rtEqe4lnsKhrOek8Q6i44eJMp oGIA== X-Gm-Message-State: AO0yUKU7y6UhBMqy4HR602tosLIqHGg6ZkQicUqa7jtIWrMcJxxm3BDz wd+5s6LkM/1AOcxMck8LhzKehvDAdlgmpQ== X-Google-Smtp-Source: AK7set8g/AffBQcEBR6L6ii4YneuMXWLgtgjdWccaVr51I+0oupThprfYtwfKSK6XTTZAjErbmHc+iJlJMWADA== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a17:90a:eb92:b0:230:cbb4:b6ff with SMTP id o18-20020a17090aeb9200b00230cbb4b6ffmr483899pjy.30.1675702739082; Mon, 06 Feb 2023 08:58:59 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:42 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-4-ricarkol@google.com> Subject: [PATCH v2 03/12] KVM: arm64: Add helper for creating unlinked stage2 subtrees From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add a stage2 helper, kvm_pgtable_stage2_create_unlinked(), for creating unlinked tables (the opposite of kvm_pgtable_stage2_free_unlinked()). Creating an unlinked table is useful for splitting block PTEs into subtrees of 4K PTEs. For example, a 1G block PTE can be split into 4K PTEs by first creating a fully populated tree, and then use it to replace the 1G PTE in a single step. This will be used in a subsequent commit for eager huge-page splitting (a dirty-logging optimization). No functional change intended. This new function will be used in a subsequent commit. Signed-off-by: Ricardo Koller --- arch/arm64/include/asm/kvm_pgtable.h | 29 +++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 47 ++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 7c45082e6c23..e94c92988745 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -460,6 +460,35 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); */ void kvm_pgtable_stage2_free_unlinked(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level); +/** + * kvm_pgtable_stage2_create_unlinked() - Create an unlinked stage-2 paging structure. + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). + * @new: Unlinked stage-2 paging structure to be created. + * @phys: Physical address of the memory to map. + * @level: Level of the stage-2 paging structure to be created. + * @prot: Permissions and attributes for the mapping. + * @mc: Cache of pre-allocated and zeroed memory from which to allocate + * page-table pages. + * @force_pte: Force mappings to PAGE_SIZE granularity. + * + * Create an unlinked page-table tree under @new. If @force_pte is + * true and @level is the PMD level, then the tree is mapped up to the + * PAGE_SIZE leaf PTE; the tree is mapped up one level otherwise. This + * new page-table tree is not reachable (i.e., it is removed) from the + * root pgd and it's therefore unreachableby the hardware page-table + * walker. No TLB invalidation or CMOs are performed. + * + * If device attributes are not explicitly requested in @prot, then the + * mapping will be normal, cacheable. + * + * Return: 0 only if a fully populated tree was created (all memory + * under @level is mapped), negative error code on failure. + */ +int kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, + kvm_pte_t *new, u64 phys, u32 level, + enum kvm_pgtable_prot prot, void *mc, + bool force_pte); + /** * kvm_pgtable_stage2_map() - Install a mapping in a guest stage-2 page-table. * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 0a5ef9288371..fed314f2b320 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1181,6 +1181,53 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) return kvm_pgtable_walk(pgt, addr, size, &walker); } +int kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, + kvm_pte_t *new, u64 phys, u32 level, + enum kvm_pgtable_prot prot, void *mc, + bool force_pte) +{ + struct stage2_map_data map_data = { + .phys = phys, + .mmu = pgt->mmu, + .memcache = mc, + .force_pte = force_pte, + }; + struct kvm_pgtable_walker walker = { + .cb = stage2_map_walker, + .flags = KVM_PGTABLE_WALK_LEAF | + KVM_PGTABLE_WALK_SKIP_BBM | + KVM_PGTABLE_WALK_SKIP_CMO, + .arg = &map_data, + }; + /* .addr (the IPA) is irrelevant for a removed table */ + struct kvm_pgtable_walk_data data = { + .walker = &walker, + .addr = 0, + .end = kvm_granule_size(level), + }; + struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops; + kvm_pte_t *pgtable; + int ret; + + ret = stage2_set_prot_attr(pgt, prot, &map_data.attr); + if (ret) + return ret; + + pgtable = mm_ops->zalloc_page(mc); + if (!pgtable) + return -ENOMEM; + + ret = __kvm_pgtable_walk(&data, mm_ops, (kvm_pteref_t)pgtable, + level + 1); + if (ret) { + kvm_pgtable_stage2_free_unlinked(mm_ops, pgtable, level); + mm_ops->put_page(pgtable); + return ret; + } + + *new = kvm_init_table_pte(pgtable, mm_ops); + return 0; +} int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu, struct kvm_pgtable_mm_ops *mm_ops, From patchwork Mon Feb 6 16:58:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130384 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD06AC05027 for ; Mon, 6 Feb 2023 16:59:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230447AbjBFQ7J (ORCPT ); Mon, 6 Feb 2023 11:59:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231134AbjBFQ7C (ORCPT ); Mon, 6 Feb 2023 11:59:02 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 25400298DA for ; Mon, 6 Feb 2023 08:59:01 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id h126-20020a636c84000000b004d31ad79086so5440280pgc.23 for ; Mon, 06 Feb 2023 08:59:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sycVjS/8vH1Yip3OrT5l3m3XOZFV09Y6IZ3loHvAqI0=; b=M2XimBj4AZUXOOI9YyAUluFWLJlskgmE+P0t5KNuwcUgUlCHGGtb/ammqMGFXzxscw Kyf8o1Le8kL706wk8+YzwI8iOnzoneMjzqtyKcOtn6U0P1O5Ej9PKa0T5M7k2VOJy1FY 4kTpEssoNSlYqvU+mVqse5RSZVjqhQoYK4EijNtDAOtLyzqgswtFv7cAB2ilh5kVH9KU yOepw04fgSu7PmaeRSn9YuoCOaW0JVDXZkPYrshYlvnmyX/ETkGIlyKgFunxVo8aKPrS VLjtOvnZuuCrtRbhURreDAalO3SFQjqY4HRas/JYYOjZ7s1CC6iIe9wtG42LEtD6s4eq HgTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sycVjS/8vH1Yip3OrT5l3m3XOZFV09Y6IZ3loHvAqI0=; b=nHrhy/eSSxK8Wd7KQ9rGLF0mQx9Txe32XNWyHYYzwQ2egWiaOkOrF8rdNHlBKeIyYk 69Alrnj7tX0nLoW3/23Gdv6gSorrl3GpCGvZd/KVX5LMAWM7sQKMOIDW8wzdPZ5buzZD OPKNTt746KXQ5U6KgWilzoZOLe7CTRbQdF4fQg58lziSfL14wb+y6GEpgxDzgDY38qea daxpvjaV/uCFZP16ATUTt+qVDmCjrKWkbuEquM6mszXW6/TOa0+EV5+E/0RHEonmySvX 2dZ9WsFxqzQBnFN3vo0RlDzy/ecaW6ddXOy62fXjcwQnqWHI9A9OnmJNaUdop0FOwLcS rHZA== X-Gm-Message-State: AO0yUKWG4Vj1XbIeZUUbf04fvFKDE/KUT09Q3TSMPNuM1UqHhEPDVz0A Z4JATTecBygrFxJM1fzZE4is48l1KFNvFg== X-Google-Smtp-Source: AK7set+vQXUrlZln3lkRbxF3dLJN90KsXVlnVSvAcGMPqFcONmeiLN0CaAeVKDi0CJUMtYsvCwUOnO4fFFSbMg== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a17:902:e545:b0:199:1458:6c67 with SMTP id n5-20020a170902e54500b0019914586c67mr1058514plf.28.1675702740582; Mon, 06 Feb 2023 08:59:00 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:43 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-5-ricarkol@google.com> Subject: [PATCH v2 04/12] KVM: arm64: Add kvm_pgtable_stage2_split() From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add a new stage2 function, kvm_pgtable_stage2_split(), for splitting a range of huge pages. This will be used for eager-splitting huge pages into PAGE_SIZE pages. The goal is to avoid having to split huge pages on write-protection faults, and instead use this function to do it ahead of time for large ranges (e.g., all guest memory in 1G chunks at a time). No functional change intended. This new function will be used in a subsequent commit. Signed-off-by: Ricardo Koller --- arch/arm64/include/asm/kvm_pgtable.h | 30 ++++++++ arch/arm64/kvm/hyp/pgtable.c | 105 +++++++++++++++++++++++++++ 2 files changed, 135 insertions(+) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index e94c92988745..871c4eeb0184 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -658,6 +658,36 @@ bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr); */ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size); +/** + * kvm_pgtable_stage2_split() - Split a range of huge pages into leaf PTEs pointing + * to PAGE_SIZE guest pages. + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init(). + * @addr: Intermediate physical address from which to split. + * @size: Size of the range. + * @mc: Cache of pre-allocated and zeroed memory from which to allocate + * page-table pages. + * @mc_capacity: Number of pages in @mc. + * + * @addr and the end (@addr + @size) are effectively aligned down and up to + * the top level huge-page block size. This is an example using 1GB + * huge-pages and 4KB granules. + * + * [---input range---] + * : : + * [--1G block pte--][--1G block pte--][--1G block pte--][--1G block pte--] + * : : + * [--2MB--][--2MB--][--2MB--][--2MB--] + * : : + * [ ][ ][:][ ][ ][ ][ ][ ][:][ ][ ][ ] + * : : + * + * Return: 0 on success, negative error code on failure. Note that + * kvm_pgtable_stage2_split() is best effort: it tries to break as many + * blocks in the input range as allowed by @mc_capacity. + */ +int kvm_pgtable_stage2_split(struct kvm_pgtable *pgt, u64 addr, u64 size, + void *mc, u64 mc_capacity); + /** * kvm_pgtable_walk() - Walk a page-table. * @pgt: Page-table structure initialised by kvm_pgtable_*_init(). diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index fed314f2b320..ae80845c8db7 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1229,6 +1229,111 @@ int kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, return 0; } +struct stage2_split_data { + struct kvm_s2_mmu *mmu; + void *memcache; + u64 mc_capacity; +}; + +/* + * Get the number of page-tables needed to replace a bock with a fully + * populated tree, up to the PTE level, at particular level. + */ +static inline u32 stage2_block_get_nr_page_tables(u32 level) +{ + switch (level) { + /* There are no blocks at level 0 */ + case 1: return 1 + PTRS_PER_PTE; + case 2: return 1; + case 3: return 0; + default: + WARN_ON_ONCE(1); + return ~0; + } +} + +static int stage2_split_walker(const struct kvm_pgtable_visit_ctx *ctx, + enum kvm_pgtable_walk_flags visit) +{ + struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops; + struct stage2_split_data *data = ctx->arg; + kvm_pte_t pte = ctx->old, new, *childp; + enum kvm_pgtable_prot prot; + void *mc = data->memcache; + u32 level = ctx->level; + u64 phys, nr_pages; + bool force_pte; + int ret; + + /* No huge-pages exist at the last level */ + if (level == KVM_PGTABLE_MAX_LEVELS - 1) + return 0; + + /* We only split valid block mappings */ + if (!kvm_pte_valid(pte) || kvm_pte_table(pte, ctx->level)) + return 0; + + nr_pages = stage2_block_get_nr_page_tables(level); + if (data->mc_capacity >= nr_pages) { + /* Build a tree mapped down to the PTE granularity. */ + force_pte = true; + } else { + /* + * Don't force PTEs. This requires a single page of PMDs at the + * PUD level, or a single page of PTEs at the PMD level. If we + * are at the PUD level, the PTEs will be created recursively. + */ + force_pte = false; + nr_pages = 1; + } + + if (data->mc_capacity < nr_pages) + return -ENOMEM; + + phys = kvm_pte_to_phys(pte); + prot = kvm_pgtable_stage2_pte_prot(pte); + + ret = kvm_pgtable_stage2_create_unlinked(data->mmu->pgt, &new, phys, + level, prot, mc, force_pte); + if (ret) + return ret; + + if (!stage2_try_break_pte(ctx, data->mmu)) { + childp = kvm_pte_follow(new, mm_ops); + kvm_pgtable_stage2_free_unlinked(mm_ops, childp, level); + mm_ops->put_page(childp); + return -EAGAIN; + } + + /* + * Note, the contents of the page table are guaranteed to be made + * visible before the new PTE is assigned because stage2_make_pte() + * writes the PTE using smp_store_release(). + */ + stage2_make_pte(ctx, new); + dsb(ishst); + data->mc_capacity -= nr_pages; + return 0; +} + +int kvm_pgtable_stage2_split(struct kvm_pgtable *pgt, u64 addr, u64 size, + void *mc, u64 mc_capacity) +{ + struct stage2_split_data split_data = { + .mmu = pgt->mmu, + .memcache = mc, + .mc_capacity = mc_capacity, + }; + + struct kvm_pgtable_walker walker = { + .cb = stage2_split_walker, + .flags = KVM_PGTABLE_WALK_LEAF, + .arg = &split_data, + }; + + return kvm_pgtable_walk(pgt, addr, size, &walker); +} + int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu, struct kvm_pgtable_mm_ops *mm_ops, enum kvm_pgtable_stage2_flags flags, From patchwork Mon Feb 6 16:58:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130386 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA282C63797 for ; Mon, 6 Feb 2023 16:59:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230472AbjBFQ7M (ORCPT ); Mon, 6 Feb 2023 11:59:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230342AbjBFQ7E (ORCPT ); Mon, 6 Feb 2023 11:59:04 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA0CA298E2 for ; Mon, 6 Feb 2023 08:59:03 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-517f8be4b00so120445737b3.3 for ; Mon, 06 Feb 2023 08:59:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8zrNUdMOQOUCJgLNSlqpWBpwPg1ysQqMfJW7ug17pk8=; b=OhogaDmxK2A4kV6+TTY9E1br3vwEnQWAXP4ItIL5FD1Y3YQ5dt5/DbUeN2hySyWDqS 9fLNvLxT88V3LYBi+kPyBlwcie/evnfpfJtMNUdhmIzve8VkYv6BRKl3pks0VuN9TfLA 5cREu96AzkOJpObRjU9/AoKblo5ztdPmcipREktbavYR3DrY7vg6fdHjQf8Hqsu8hmaK yvDcx+jU9yapfiu7bODzfejb2YyxXRzuG9pvwLSqreYZEKs8f4yT7IvCvX7nAOQhf3X3 dNIJHt65wWlTiwvNzcfT0DdBZp9V8AwPjH0TzAl+hYWnc0uz46bbjHQxVH2Lnvt9pSyr KGyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8zrNUdMOQOUCJgLNSlqpWBpwPg1ysQqMfJW7ug17pk8=; b=jjStMwUy4xp17ydscQKsm6TWisr39b4K85hNJoiwu1FtkGbvBbDVtQlr98OS24wvQl FBpEZsdNh/Hux69tTyViKwj+tHBjl5xYWp7AfwM1MYkq2YzlnavPpk60dhRDyeWmyuYN R8ZRzGwGzu0vQ9BRPIdPplQVHf4mrqPjwPLaUsENXVKQ8+/tXgkKrfikr5Sz43Gf4hrK mOIyz/46eNJ8/9H0cHhMHPlTkIPn5ZpgLS3iJFv2DnsUnFxDeOxtKeqzsOSQhRrFsDBF Rhu2sxyr/UKbwUoVdZsVhV8k4J5gXRv2Q3gkuQsxfOUgXlTL/BqG/gLAh0YaOgO9lNbd 66lg== X-Gm-Message-State: AO0yUKXGxD2mCZMh26g99uZU3eAn6gt+WcpfDZVhIEwbyCmgoaIIKRtT qL8/cSCNiO2m+U3nQ00faJ2Sj8xm4ZCPAg== X-Google-Smtp-Source: AK7set94430d3TW0rOSOGdcCMwmeKCK2QLWM9dIYTScifN3Lhaw6SVCQvu82gV70yChRGOpb9SbRNDBzgZ4dCA== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a25:13c6:0:b0:855:fdcb:4467 with SMTP id 189-20020a2513c6000000b00855fdcb4467mr0ybt.0.1675702742440; Mon, 06 Feb 2023 08:59:02 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:44 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-6-ricarkol@google.com> Subject: [PATCH v2 05/12] KVM: arm64: Refactor kvm_arch_commit_memory_region() From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Refactor kvm_arch_commit_memory_region() as a preparation for a future commit to look cleaner and more understandable. Also, it looks more like its x86 counterpart (in kvm_mmu_slot_apply_flags()). No functional change intended. Signed-off-by: Ricardo Koller --- arch/arm64/kvm/mmu.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 9bd3c2cfb476..d2c5e6992459 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1761,20 +1761,27 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, const struct kvm_memory_slot *new, enum kvm_mr_change change) { + bool log_dirty_pages = new && new->flags & KVM_MEM_LOG_DIRTY_PAGES; + /* * At this point memslot has been committed and there is an * allocated dirty_bitmap[], dirty pages will be tracked while the * memory slot is write protected. */ - if (change != KVM_MR_DELETE && new->flags & KVM_MEM_LOG_DIRTY_PAGES) { + if (log_dirty_pages) { + + if (change == KVM_MR_DELETE) + return; + /* * If we're with initial-all-set, we don't need to write * protect any pages because they're all reported as dirty. * Huge pages and normal pages will be write protect gradually. */ - if (!kvm_dirty_log_manual_protect_and_init_set(kvm)) { - kvm_mmu_wp_memory_region(kvm, new->id); - } + if (kvm_dirty_log_manual_protect_and_init_set(kvm)) + return; + + kvm_mmu_wp_memory_region(kvm, new->id); } } From patchwork Mon Feb 6 16:58:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BA4CC636D4 for ; Mon, 6 Feb 2023 16:59:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231149AbjBFQ7L (ORCPT ); Mon, 6 Feb 2023 11:59:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230472AbjBFQ7H (ORCPT ); Mon, 6 Feb 2023 11:59:07 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21A2229E20 for ; Mon, 6 Feb 2023 08:59:05 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id lx10-20020a17090b4b0a00b00230b26a46b6so2220274pjb.6 for ; Mon, 06 Feb 2023 08:59:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ned5ZH8F69GPFdj4cuN6WK5vFKhDB+aqB56zYhL8gAM=; b=V5/nDwgUHHF6CWVuMNYTpzf9GNUV0/IEDAjTALBiNrylNWFmIPY5WFXEJ3yqe4n1q4 PZP25IJMjYJRup8+5Tdgp6vitrhxT2tTiNjHi3YMp5/Q2T8ezkBUgYh5ixbzSyofSJOf h4mkHAOplb1MwZE/F5dNAxniXBuX/xolqd4SBRIPRx5x7Uu2rMw/VrZA/XH+8btGS0Aa aXvMadlh/D+Wyx+khVqbAVPVYU6aWRaMDVCZtQ9wdmazIb/jJV3rDc+eDsv59MlERKwm 9s+gPXnoA8Wt2yUBhHToh/6iflaIrX13sJaHyVLyVBLxoc6xNwVWTfS/dQ/Du74cDZO1 Ddrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ned5ZH8F69GPFdj4cuN6WK5vFKhDB+aqB56zYhL8gAM=; b=4n6YIUPf8yLfALT5lWfJKZyaxDvrdDfoDvQu8kWJPuS3cbH4FuO9AY+zjE8b/VRzac CkYyM0Hj05vLa5o4dliGMcMaDG8Y98l/cLUhRo6jSxZQFzgN8K0bNkxOUdgqZDUVB/uh iUY0hQuHF8xY0ufhKBqrjRgyJlf0D+tSKMFc8hpkTO64IcrXk4k1E+fJ4X3uyeGMSjp0 CP3H4V58jUYfp3R08U5k3gqHzvUhwMvljevLv+Ng8zYpAFO2GI4lTO1jgjo5pzPK3mQc qufjOhjAbo5UwsoCofi51q42wecn4iSD/b80nZDBM8ZaP3e0J865Uv9e7qXirJWnIvSz 6t4A== X-Gm-Message-State: AO0yUKVmIz4/54LKfgg+MEMmUuHTOlG32xJcg/G03grDUbuxvicYSbLo xuX1l8fDBnNmYlYneL5blm6YgRQqlIkDdQ== X-Google-Smtp-Source: AK7set9fNVVKWi71d4Mx5UI1FkBgjOML5W4imqZGeHvkXnc95/ruYP//2mmjpRMKpD7hf/wYCKXhTBjf6u6VnQ== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a05:6a00:1a:b0:592:4daf:20d6 with SMTP id h26-20020a056a00001a00b005924daf20d6mr7056pfk.43.1675702744554; Mon, 06 Feb 2023 08:59:04 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:45 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-7-ricarkol@google.com> Subject: [PATCH v2 06/12] KVM: arm64: Add kvm_uninit_stage2_mmu() From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add kvm_uninit_stage2_mmu() and move kvm_free_stage2_pgd() into it. A future commit will add some more things to do inside of kvm_uninit_stage2_mmu(). No functional change intended. Signed-off-by: Ricardo Koller --- arch/arm64/include/asm/kvm_mmu.h | 1 + arch/arm64/kvm/mmu.c | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index e4a7e6369499..058f3ae5bc26 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -167,6 +167,7 @@ void free_hyp_pgds(void); void stage2_unmap_vm(struct kvm *kvm); int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long type); +void kvm_uninit_stage2_mmu(struct kvm *kvm); void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu); int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, phys_addr_t pa, unsigned long size, bool writable); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d2c5e6992459..812633a75e74 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -766,6 +766,11 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t return err; } +void kvm_uninit_stage2_mmu(struct kvm *kvm) +{ + kvm_free_stage2_pgd(&kvm->arch.mmu); +} + static void stage2_unmap_memslot(struct kvm *kvm, struct kvm_memory_slot *memslot) { @@ -1855,7 +1860,7 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) void kvm_arch_flush_shadow_all(struct kvm *kvm) { - kvm_free_stage2_pgd(&kvm->arch.mmu); + kvm_uninit_stage2_mmu(kvm); } void kvm_arch_flush_shadow_memslot(struct kvm *kvm, From patchwork Mon Feb 6 16:58:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130387 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF677C05027 for ; Mon, 6 Feb 2023 16:59:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231165AbjBFQ7N (ORCPT ); Mon, 6 Feb 2023 11:59:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230505AbjBFQ7H (ORCPT ); Mon, 6 Feb 2023 11:59:07 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB55D29E33 for ; Mon, 6 Feb 2023 08:59:06 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id j18-20020a170903029200b00198aa765a9dso6645478plr.6 for ; Mon, 06 Feb 2023 08:59:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iOpp/S8Rtelv3h2rWZJqIJdgZzud+NaCm9qMEIp1MnI=; b=sKfiuMTpfyXaq12seZrzSDjLQ086gfmqtdv4cF7ki165PqRcS4JuSmFzlwgwD8G5xk UmH8vLp6oybFNKOXyHV8rZBBZuWWr/XEbTmtMZk4fa4it2miA63L77htcx57jNOkjjjC fjVc3EiLaA6BzAWlOd5aGzYvA87cTEexWkjcATvLpehYjZ/4ng5onMRqBqjTPXRibHU8 5d1hueXAYquDsbUKDj303ZwOwYZqTCZu5l/TS0/CdHSfNHiszcROETa7jQ3MTibzwOHM A9dz4+r0LX57xxL5TDp/i1AN5WGxB4UDieKQBexLxjfqzoEzPu6Hrg6Hd3iktEMwx/uT lBDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iOpp/S8Rtelv3h2rWZJqIJdgZzud+NaCm9qMEIp1MnI=; b=5O2BfTz5SkeCuZn6uc9E4wWcJxWNpOjf6fq70/rrV2fFJBKBej3vCFjR+V83AiqvKM QeGQPQiC43mghoq6nGjLYpFh2j0JvYW0rC/L3aT/1s+aYJwXCAV+Ae1qNJQo8HxvhCGk 6O1BG1tmSg/Gl2USZFvkaZoLFglwAQsuQRyQu5MUiamPbOuMASNxFWyJQ8GBz+rpB8dy xCxhzUddtICZ73Y3uVbQty4g5fJz4E12ijD51Rf3344dUgQlUjaVzFOtC5P6N7Z/NbeH dq6HNWrYMEoUIS1bVZHEAosGgRbVzF2wlQy96BAtZhZxehthplP2GnJdlV2LRnQXIo9h RbtQ== X-Gm-Message-State: AO0yUKVyi+Fz30E23e9GkspJcUYviu6uCcmLQfzutUUAskHook+jjz+J BsJk6Yqq0Yzd6ptGQo1NSOC5fBoDDRUu2g== X-Google-Smtp-Source: AK7set+9BMru5C7ajdZ/QFnYvf0vq6OsZXfNWB4URhHpoB1VV9dRbuBfGMCP/U4V0dUdFkxQcatVyOKOpOVTJQ== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a65:45c9:0:b0:477:31bc:348f with SMTP id m9-20020a6545c9000000b0047731bc348fmr3202793pgr.72.1675702746208; Mon, 06 Feb 2023 08:59:06 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:46 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-8-ricarkol@google.com> Subject: [PATCH v2 07/12] KVM: arm64: Export kvm_are_all_memslots_empty() From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Export kvm_are_all_memslots_empty(). This will be used by a future commit when checking before setting a capability. No functional change intended. Signed-off-by: Ricardo Koller --- include/linux/kvm_host.h | 2 ++ virt/kvm/kvm_main.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 4f26b244f6d0..8c5530e03a78 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -991,6 +991,8 @@ static inline bool kvm_memslots_empty(struct kvm_memslots *slots) return RB_EMPTY_ROOT(&slots->gfn_tree); } +bool kvm_are_all_memslots_empty(struct kvm *kvm); + #define kvm_for_each_memslot(memslot, bkt, slots) \ hash_for_each(slots->id_hash, bkt, memslot, id_node[slots->node_idx]) \ if (WARN_ON_ONCE(!memslot->npages)) { \ diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 9c60384b5ae0..3940d2467e1b 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -4604,7 +4604,7 @@ int __attribute__((weak)) kvm_vm_ioctl_enable_cap(struct kvm *kvm, return -EINVAL; } -static bool kvm_are_all_memslots_empty(struct kvm *kvm) +bool kvm_are_all_memslots_empty(struct kvm *kvm) { int i; From patchwork Mon Feb 6 16:58:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130388 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70599C636D4 for ; Mon, 6 Feb 2023 16:59:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231185AbjBFQ7O (ORCPT ); Mon, 6 Feb 2023 11:59:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231143AbjBFQ7J (ORCPT ); Mon, 6 Feb 2023 11:59:09 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19092298F8 for ; Mon, 6 Feb 2023 08:59:08 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id u3-20020a056a00124300b0056d4ab0c7cbso6681788pfi.7 for ; Mon, 06 Feb 2023 08:59:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zzlWyFCqYcyGZzCULhoOstPX7ibhNXtVEONcSJrj+Jc=; b=HLLwmkjhmxTQyNCpagTXJVAaLKpNx2dLM8cXIXmhfJJrAbw7EzlGzw9IIkSRHpWfB8 yZwe45G0Sz7zaQstlj05xgJ7A0wYxWQa4KELG/b8qLQqrtwFJP2oJX0cxvA+xtaW0UJd OJlWkCjV9c/dHAjZUXQsrgcYU2kqrCPF9CzUpK7Sn9ppmsadmTh7vnl8e/ZvlfCTthdu nMcMxOxHEQcfRGFDT6iWKL7Bcm4DZdsFoIkUe3aZNVb5wrBqcOVwzuEEc1VCvy3h7Jny pZkPEo7joA2IZE0YvtDqIFuoiXw8UQkOf8jTTtJwvL+4pC1CCTVyx3X0PEqycNfnXSQj IdSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zzlWyFCqYcyGZzCULhoOstPX7ibhNXtVEONcSJrj+Jc=; b=EN12PnqEt7WS5dK9DRmM9zvL7R6nlbTQmzHZLO/NtWCeTAnxltdMjrstQ05BA4GKLf jL6sYew2FxITJ4qKk62EA/Zrsa+e3ixX1kEsodBHpM6fGG+UGs+XjaFY2Ofp2lraWRhE hFQZxiu5XHsg2U5Ss0iOJ5HXZIRESbCosrsxGlZuA1gm2Q6VHdATCvX/L0zfqOajnBXb 2ADqXqu7F3WyN3xEqud0Hwg4FeGoXKw4JAs3H0KuH+e139dYj6lWV7Zoh3P9cOyw9w6f DOAHouVp1RdGWdnbkiN2Ig+qO3IaivBY19cEqVU+vC7mSddjZ07qUVY+7xJ0aEcTh695 BSvA== X-Gm-Message-State: AO0yUKVuU9M1LDqLG0cSNaAfvL3ZUpdY6zEnZRHm19hSPzxqvygXPt4K +0v3cEe/rNytSbA7rh1XJ8IF1abMRMl2EA== X-Google-Smtp-Source: AK7set9Sa9xH8cDy8qU28rf3Jsx4C/c70O1Yy7vzzH+DlFlqSo6pRcMhNRkBnC7LJI7dyxBps1pSsMsrhU2vDA== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a17:90a:2c05:b0:230:97d5:a224 with SMTP id m5-20020a17090a2c0500b0023097d5a224mr56108pjd.122.1675702747618; Mon, 06 Feb 2023 08:59:07 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:47 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-9-ricarkol@google.com> Subject: [PATCH v2 08/12] KVM: arm64: Add KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add a capability for userspace to specify the eager split chunk size. The chunk size specifies how many pages to break at a time, using a single allocation. Bigger the chunk size, more pages need to be allocated ahead of time. Suggested-by: Oliver Upton Signed-off-by: Ricardo Koller --- Documentation/virt/kvm/api.rst | 26 ++++++++++++++++++++++++++ arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/kvm/arm.c | 22 ++++++++++++++++++++++ arch/arm64/kvm/mmu.c | 3 +++ include/uapi/linux/kvm.h | 1 + 5 files changed, 54 insertions(+) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 9807b05a1b57..a9332e331cce 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -8284,6 +8284,32 @@ structure. When getting the Modified Change Topology Report value, the attr->addr must point to a byte where the value will be stored or retrieved from. +8.40 KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE +--------------------------------------- + +:Capability: KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE +:Architectures: arm64 +:Type: vm +:Parameters: arg[0] is the new chunk size. +:Returns: 0 on success, -EINVAL if any memslot has been created. + +This capability sets the chunk size used in Eager Page Splitting. + +Eager Page Splitting improves the performance of dirty-logging (used +in live migrations) when guest memory is backed by huge-pages. This +optimization is enabled by default on arm64. It avoids splitting +huge-pages (into PAGE_SIZE pages) on fault, by doing it eagerly when +enabling dirty logging (with the KVM_MEM_LOG_DIRTY_PAGES flag for a +memory region), or when using KVM_CLEAR_DIRTY_LOG. + +The chunk size specifies how many pages to break at a time, using a +single allocation for each chunk. Bigger the chunk size, more pages +need to be allocated ahead of time. A good heuristic is to pick the +size of the huge-pages as the chunk size. + +If the chunk size (arg[0]) is zero, then no eager page splitting is +performed. The default value PMD size (e.g., 2M when PAGE_SIZE is 4K). + 9. Known KVM API problems ========================= diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 35a159d131b5..a69a815719cf 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -153,6 +153,8 @@ struct kvm_s2_mmu { /* The last vcpu id that ran on each physical CPU */ int __percpu *last_vcpu_ran; +#define KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT PMD_SIZE + struct kvm_arch *arch; }; diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 9c5573bc4614..c80617ced599 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -101,6 +101,22 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, r = 0; set_bit(KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED, &kvm->arch.flags); break; + case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE: + mutex_lock(&kvm->lock); + mutex_lock(&kvm->slots_lock); + /* + * To keep things simple, allow changing the chunk + * size only if there are no memslots created. + */ + if (!kvm_are_all_memslots_empty(kvm)) { + r = -EINVAL; + } else { + r = 0; + kvm->arch.mmu.split_page_chunk_size = cap->args[0]; + } + mutex_unlock(&kvm->slots_lock); + mutex_unlock(&kvm->lock); + break; default: r = -EINVAL; break; @@ -298,6 +314,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_ARM_PTRAUTH_GENERIC: r = system_has_full_ptr_auth(); break; + case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE: + if (kvm) + r = kvm->arch.mmu.split_page_chunk_size; + else + r = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT; + break; default: r = 0; } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 812633a75e74..e2ada6588017 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -755,6 +755,9 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t for_each_possible_cpu(cpu) *per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1; + mmu->split_page_cache.gfp_zero = __GFP_ZERO; + mmu->split_page_chunk_size = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT; + mmu->pgt = pgt; mmu->pgd_phys = __pa(pgt->pgd); return 0; diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 55155e262646..02e05f7918e2 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1175,6 +1175,7 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224 #define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225 +#define KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE 226 #ifdef KVM_CAP_IRQ_ROUTING From patchwork Mon Feb 6 16:58:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 901BFC05027 for ; Mon, 6 Feb 2023 16:59:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229738AbjBFQ7P (ORCPT ); Mon, 6 Feb 2023 11:59:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231132AbjBFQ7K (ORCPT ); Mon, 6 Feb 2023 11:59:10 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B2DDF298E2 for ; Mon, 6 Feb 2023 08:59:09 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id p18-20020a170902ead200b0019909279002so2519936pld.3 for ; Mon, 06 Feb 2023 08:59:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TYyeERF14skYe60v/LD7u8bs5+ccW9y+uSSzeC9iljY=; b=NusBZRgSybZRl8Dgov/teHGL78b0fMqxMJ+c+TNlwfuo/lO60q/obPq47mtzqTnybF AftEhzbq8yTDSoX5x3rnceibzAzKVUmT658gH9/lf3LDVpruPP8MlCs7FBI/n461zAaF wuosNWlozCDgMhv+SH+elxLeyVt6nqxTGAw3kb/dLC9WDKlAd/muvpKJThJalMOUQtlT QxJzURL1GDN8szWBlKJ1UFvi5q06SR/3uagNFkdqz3uCyf9ZqQZprMkdoEBiEXAm0ehw wX5tnGTvhh/RCxYefKCwzkiqcEZkkujwb2TYhHnpqp+j4CWmUPBahvdAqJVWWFNC8Wup qKDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TYyeERF14skYe60v/LD7u8bs5+ccW9y+uSSzeC9iljY=; b=KBEC919fRKfq26yniXDAZlEOB352ry6ZPwcJPcCYIl2Ei8sQUqiLM4DpLNBZNzVb4F ohdi4sPi3AL5fEVQKdn+VN3uUExAhDMdfq4funYSzXUvmZ4oK9Ug9n4Cxg2bgbdIpkiM RJFNOJVzz8Uz/BnkD27ITh/spH4jCGxqNHix/LzOJM+6hoh9u4azJaNoOb3XGYhl/H9L N+ByON/4/YS8TY0umElizo2emjEKEc6KMKmqas6ZI3Wkm4ksSd1Oiq+9NM5hEad+mkjl Wx7YpdPN1rwLmmIkiYVsroYLaOArRZK7DP9FeDHTxqaoOXFdTQX0yByNZVZTVc5F7Q3M XZEg== X-Gm-Message-State: AO0yUKUQYh0yMB+hxqxTNeSY+P09IAzG7Ihxvr0fi2Yh17pOkGDAQfSm KNVebJ01ic8/j83ogFWr6PaSQ5x+/p3Baw== X-Google-Smtp-Source: AK7set9dCR+yppIiQAPDvfDaLr3FGKBm5/uLbHrkgFGOs/U1RnGLzjAV8ZhxGpaGompZxKBNZ01+V/gjb3Eb9A== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a17:90a:4f42:b0:230:d7dc:1fa9 with SMTP id w2-20020a17090a4f4200b00230d7dc1fa9mr80557pjl.58.1675702749259; Mon, 06 Feb 2023 08:59:09 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:48 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-10-ricarkol@google.com> Subject: [PATCH v2 09/12] KVM: arm64: Split huge pages when dirty logging is enabled From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Split huge pages eagerly when enabling dirty logging. The goal is to avoid doing it while faulting on write-protected pages, which negatively impacts guest performance. A memslot marked for dirty logging is split in 1GB pieces at a time. This is in order to release the mmu_lock and give other kernel threads the opportunity to run, and also in order to allocate enough pages to split a 1GB range worth of huge pages (or a single 1GB huge page). Note that these page allocations can fail, so eager page splitting is best-effort. This is not a correctness issue though, as huge pages can still be split on write-faults. The benefits of eager page splitting are the same as in x86, added with commit a3fe5dbda0a4 ("KVM: x86/mmu: Split huge pages mapped by the TDP MMU when dirty logging is enabled"). For example, when running dirty_log_perf_test with 64 virtual CPUs (Ampere Altra), 1GB per vCPU, 50% reads, and 2MB HugeTLB memory, the time it takes vCPUs to access all of their memory after dirty logging is enabled decreased by 44% from 2.58s to 1.42s. Signed-off-by: Ricardo Koller --- arch/arm64/include/asm/kvm_host.h | 16 +++++ arch/arm64/kvm/mmu.c | 113 +++++++++++++++++++++++++++++- 2 files changed, 127 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index a69a815719cf..eab62d8b3ad4 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -154,6 +154,22 @@ struct kvm_s2_mmu { int __percpu *last_vcpu_ran; #define KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT PMD_SIZE + /* + * Memory cache used to split + * KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE worth of huge pages. It + * is used to allocate stage2 page tables while splitting huge + * pages. Note that the choice of EAGER_PAGE_SPLIT_CHUNK_SIZE + * influences both the capacity of the split page cache, and + * how often KVM reschedules. Be wary of raising CHUNK_SIZE + * too high. + * + * A good heuristic to pick CHUNK_SIZE is that it should be + * the size of huge-page to be split. + * + * Protected by kvm->slots_lock. + */ + struct kvm_mmu_memory_cache split_page_cache; + uint64_t split_page_chunk_size; struct kvm_arch *arch; }; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index e2ada6588017..73f8b3953f6a 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -31,14 +31,21 @@ static phys_addr_t hyp_idmap_vector; static unsigned long io_map_base; -static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end) +static phys_addr_t __stage2_range_addr_end(phys_addr_t addr, phys_addr_t end, + phys_addr_t size) { - phys_addr_t size = kvm_granule_size(KVM_PGTABLE_MIN_BLOCK_LEVEL); phys_addr_t boundary = ALIGN_DOWN(addr + size, size); return (boundary - 1 < end - 1) ? boundary : end; } +static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end) +{ + phys_addr_t size = kvm_granule_size(KVM_PGTABLE_MIN_BLOCK_LEVEL); + + return __stage2_range_addr_end(addr, end, size); +} + /* * Release kvm_mmu_lock periodically if the memory region is large. Otherwise, * we may see kernel panics with CONFIG_DETECT_HUNG_TASK, @@ -71,6 +78,72 @@ static int stage2_apply_range(struct kvm *kvm, phys_addr_t addr, return ret; } +static bool need_topup_split_page_cache_or_resched(struct kvm *kvm, uint64_t min) +{ + struct kvm_mmu_memory_cache *cache; + + if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) + return true; + + cache = &kvm->arch.mmu.split_page_cache; + return kvm_mmu_memory_cache_nr_free_objects(cache) < min; +} + +static int kvm_mmu_split_nr_page_tables(u64 range) +{ + int n = 0; + + if (KVM_PGTABLE_MIN_BLOCK_LEVEL < 2) + n += DIV_ROUND_UP_ULL(range, PUD_SIZE); + n += DIV_ROUND_UP_ULL(range, PMD_SIZE); + return n; +} + +static int kvm_mmu_split_huge_pages(struct kvm *kvm, phys_addr_t addr, + phys_addr_t end) +{ + struct kvm_mmu_memory_cache *cache; + struct kvm_pgtable *pgt; + int ret; + u64 next; + u64 chunk_size = kvm->arch.mmu.split_page_chunk_size; + int cache_capacity = kvm_mmu_split_nr_page_tables(chunk_size); + + if (chunk_size == 0) + return 0; + + lockdep_assert_held_write(&kvm->mmu_lock); + + cache = &kvm->arch.mmu.split_page_cache; + + do { + if (need_topup_split_page_cache_or_resched(kvm, + cache_capacity)) { + write_unlock(&kvm->mmu_lock); + cond_resched(); + /* Eager page splitting is best-effort. */ + ret = __kvm_mmu_topup_memory_cache(cache, + cache_capacity, + cache_capacity); + write_lock(&kvm->mmu_lock); + if (ret) + break; + } + + pgt = kvm->arch.mmu.pgt; + if (!pgt) + return -EINVAL; + + next = __stage2_range_addr_end(addr, end, chunk_size); + ret = kvm_pgtable_stage2_split(pgt, addr, next - addr, + cache, cache_capacity); + if (ret) + break; + } while (addr = next, addr != end); + + return ret; +} + #define stage2_apply_range_resched(kvm, addr, end, fn) \ stage2_apply_range(kvm, addr, end, fn, true) @@ -772,6 +845,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t void kvm_uninit_stage2_mmu(struct kvm *kvm) { kvm_free_stage2_pgd(&kvm->arch.mmu); + kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache); } static void stage2_unmap_memslot(struct kvm *kvm, @@ -999,6 +1073,31 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, stage2_wp_range(&kvm->arch.mmu, start, end); } +/** + * kvm_mmu_split_memory_region() - split the stage 2 blocks into PAGE_SIZE + * pages for memory slot + * @kvm: The KVM pointer + * @slot: The memory slot to split + * + * Acquires kvm->mmu_lock. Called with kvm->slots_lock mutex acquired, + * serializing operations for VM memory regions. + */ +static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot) +{ + struct kvm_memslots *slots = kvm_memslots(kvm); + struct kvm_memory_slot *memslot = id_to_memslot(slots, slot); + phys_addr_t start, end; + + lockdep_assert_held(&kvm->slots_lock); + + start = memslot->base_gfn << PAGE_SHIFT; + end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; + + write_lock(&kvm->mmu_lock); + kvm_mmu_split_huge_pages(kvm, start, end); + write_unlock(&kvm->mmu_lock); +} + /* * kvm_arch_mmu_enable_log_dirty_pt_masked - enable dirty logging for selected * dirty pages. @@ -1790,6 +1889,16 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, return; kvm_mmu_wp_memory_region(kvm, new->id); + kvm_mmu_split_memory_region(kvm, new->id); + } else { + /* + * Free any leftovers from the eager page splitting cache. Do + * this when deleting, moving, disabling dirty logging, or + * creating the memslot (a nop). Doing it for deletes makes + * sure we don't leak memory, and there's no need to keep the + * cache around for any of the other cases. + */ + kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache); } } From patchwork Mon Feb 6 16:58:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130390 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE369C63797 for ; Mon, 6 Feb 2023 16:59:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231207AbjBFQ7Q (ORCPT ); Mon, 6 Feb 2023 11:59:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230342AbjBFQ7M (ORCPT ); Mon, 6 Feb 2023 11:59:12 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4F3D29419 for ; Mon, 6 Feb 2023 08:59:11 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id k14-20020aa7972e000000b00593a8232ac3so6789114pfg.22 for ; Mon, 06 Feb 2023 08:59:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bcuhZaZfjdzxoj++f8hFmgmqHpfnylwDUPS0Pw8clPQ=; b=MInBsXbldMdDJwWtvpfnfFyFCrS23cE+ASyvEQ8FoOoGcTWFygbgofdsH5q5La6PMZ VUWqyoFVE77VHQcWst4hCiz6l1nfDPStWD9OTCK3VXYMljearuxhg1of2GBVm0cEWbf8 Zk5LgdyXz0YsXeT0z+k9iibjRiqpfkDYWXhvJOFM2q7RrHVO1w5uiDlZUc7fXjMplsUQ 5/R9GyzZaGCoNmtVkxkpXZFm7pflvCVW6+5LkV9wT1qJzTUOvPEncfgLVhKxvYOJWsrX wBf1P4ylR+Rla/bxprWC4or6kE0lloewAgHSfDP4O6hCs9k59d9TNary4Nd2MLwJra0c l7SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bcuhZaZfjdzxoj++f8hFmgmqHpfnylwDUPS0Pw8clPQ=; b=GkbUN+Q+H1Yw4ByA6WWUyQ4swZkR1FAPnYL9pUOvDt24FoUJfTMmKjNa2F3+BMiSUn RwzFkj7b38ETUYPg6MDnv5fGVYrp2thEP7aP+RJHkFfD9vBvj4Yw+lr/OEOTKLtXrxyj WYLpgUw2m1acFTDC+MXM1h7T3ZeRpI7fKNucJvOD+tQ1/jF87KSTnaonmPgbCNytalSu Pz+RIi8iyJRec3YW+OeFdcSkNsUHTwz/vR6/2JPEsc+hr54srEr9/UknBlcHG3jA3EfL aihb2MYeGKmW1OHhp/PE32RgP8ScK6p42hclZp9l+h7r3rOr8rQYj0aQEYGce5VKxU8F nSaA== X-Gm-Message-State: AO0yUKWUoaQ4bwhwGIWTINsHD9o6PJLuQndoIEGfznpW1ksvWu4pBc3A zRkjXenKM9sv+W+kjVyR2cVmF3HUznPqYQ== X-Google-Smtp-Source: AK7set9Nl/4SZbvIylvDER1Y3w+aUZz91bVitx+VkfIqunHAOTb6TNcjgAKRaNOyA8sq9SV13N49sSXzl6b7tw== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a17:90a:ca11:b0:230:cff0:52ef with SMTP id x17-20020a17090aca1100b00230cff052efmr506917pjt.81.1675702751082; Mon, 06 Feb 2023 08:59:11 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:49 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-11-ricarkol@google.com> Subject: [PATCH v2 10/12] KVM: arm64: Open-code kvm_mmu_write_protect_pt_masked() From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Move the functionality of kvm_mmu_write_protect_pt_masked() into its caller, kvm_arch_mmu_enable_log_dirty_pt_masked(). This will be used in a subsequent commit in order to share some of the code in kvm_arch_mmu_enable_log_dirty_pt_masked(). No functional change intended. Signed-off-by: Ricardo Koller --- arch/arm64/kvm/mmu.c | 42 +++++++++++++++--------------------------- 1 file changed, 15 insertions(+), 27 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 73f8b3953f6a..f6fb2bdaab71 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1051,28 +1051,6 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) kvm_flush_remote_tlbs(kvm); } -/** - * kvm_mmu_write_protect_pt_masked() - write protect dirty pages - * @kvm: The KVM pointer - * @slot: The memory slot associated with mask - * @gfn_offset: The gfn offset in memory slot - * @mask: The mask of dirty pages at offset 'gfn_offset' in this memory - * slot to be write protected - * - * Walks bits set in mask write protects the associated pte's. Caller must - * acquire kvm_mmu_lock. - */ -static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, - struct kvm_memory_slot *slot, - gfn_t gfn_offset, unsigned long mask) -{ - phys_addr_t base_gfn = slot->base_gfn + gfn_offset; - phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; - phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; - - stage2_wp_range(&kvm->arch.mmu, start, end); -} - /** * kvm_mmu_split_memory_region() - split the stage 2 blocks into PAGE_SIZE * pages for memory slot @@ -1099,17 +1077,27 @@ static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot) } /* - * kvm_arch_mmu_enable_log_dirty_pt_masked - enable dirty logging for selected - * dirty pages. + * kvm_arch_mmu_enable_log_dirty_pt_masked() - enable dirty logging for selected pages. + * @kvm: The KVM pointer + * @slot: The memory slot associated with mask + * @gfn_offset: The gfn offset in memory slot + * @mask: The mask of pages at offset 'gfn_offset' in this memory + * slot to enable dirty logging on * - * It calls kvm_mmu_write_protect_pt_masked to write protect selected pages to - * enable dirty logging for them. + * Writes protect selected pages to enable dirty logging for them. Caller must + * acquire kvm->mmu_lock. */ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn_offset, unsigned long mask) { - kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask); + phys_addr_t base_gfn = slot->base_gfn + gfn_offset; + phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; + phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; + + lockdep_assert_held_write(&kvm->mmu_lock); + + stage2_wp_range(&kvm->arch.mmu, start, end); } static void kvm_send_hwpoison_signal(unsigned long address, short lsb) From patchwork Mon Feb 6 16:58:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130391 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C1D3C636D4 for ; Mon, 6 Feb 2023 16:59:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231214AbjBFQ7R (ORCPT ); Mon, 6 Feb 2023 11:59:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231184AbjBFQ7O (ORCPT ); Mon, 6 Feb 2023 11:59:14 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7026629E21 for ; Mon, 6 Feb 2023 08:59:13 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id z129-20020a256587000000b0089da1e9b65cso2580983ybb.22 for ; Mon, 06 Feb 2023 08:59:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lfnhNEO0qYqIhXOTY/DBVHuHgYeXN1mb64E/Y2nHvZo=; b=OY7lNGmecD6RBy6Mhr6OImmD0/FXI0fhdJXYXMiw0o1/dLG4IgaHT0JQPRHBTeUf58 5msr1aZj1lI9GcLS3wNywd8pPkEPwX7CktwYIK2exFcSNz3XIykgMYJa1y9i/sn4QfU3 uTyEyl2awA7X1QdbQctHSB+ITmUtfAp/b7mZkHE/76pFn/J7kOo5at3a5Jb/PHS7Rxe1 2hI8UeRYIsfuT2KoZ7FedmvXfz1NAL3CIqA+oKq6tSknbCcF2k3REZLEbHJlSy5fC6nL cyJLwUo5vCVDbSeX+DFhFUfNAR+U9E0+NFVNp9CGunZ4nImhdk1gSffO6QDHbCZbJfgu uziQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lfnhNEO0qYqIhXOTY/DBVHuHgYeXN1mb64E/Y2nHvZo=; b=2+dU4xC8GDJgMUBonda3AOfcaX9I0UQ2P+MF5Gj/LJK4bMDEyrKlxRo7UfYofzr32g at/sKBNg0zy9gR1nyyk+tmUSXQGLqOHp6XEgfnfG52i6VfKYeK90KGGEv4rZcWqLU6yd qd7cmXZvKSbz5F2d5qWvqKfSLtSB/Ib1m89sWQHM6NoH00Cdt1+eOgTdcGca6Iqqmgyh qDhYimqYsPzO/SrU/UyyidbQkP1hpq9KhDbnpxk2uGeLPvSVIHSWXqmXkKSNmaGkvV/R BuJe8gOuZkc90QAHGVzvg7+oJasKyvtjgdRdj2OFefnk3PMTQTNF3SbAtPs2VhgIO2M3 w1OA== X-Gm-Message-State: AO0yUKXNxltJLeQD8+NbhikmSvV/ZvjP1HvCPHPKTUMh81t8bCdTVzEK P3n6p7UjvIOySkxzzCKs44LSM7qCw/yuJw== X-Google-Smtp-Source: AK7set+7/xe5sxL/rI27WlC2RLSsQTQOrFJS8uuHs7s4BpRCKIiQCqgpucmxVo0v5s3NXepw3PwSRNz7QMdrRg== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a81:a046:0:b0:525:b3c5:f175 with SMTP id x67-20020a81a046000000b00525b3c5f175mr1462276ywg.453.1675702752713; Mon, 06 Feb 2023 08:59:12 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:50 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-12-ricarkol@google.com> Subject: [PATCH v2 11/12] KVM: arm64: Split huge pages during KVM_CLEAR_DIRTY_LOG From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This is the arm64 counterpart of commit cb00a70bd4b7 ("KVM: x86/mmu: Split huge pages mapped by the TDP MMU during KVM_CLEAR_DIRTY_LOG"), which has the benefit of splitting the cost of splitting a memslot across multiple ioctls. Split huge pages on the range specified using KVM_CLEAR_DIRTY_LOG. And do not split when enabling dirty logging if KVM_DIRTY_LOG_INITIALLY_SET is set. Signed-off-by: Ricardo Koller --- arch/arm64/kvm/mmu.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index f6fb2bdaab71..da2fbd04fb01 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1084,8 +1084,8 @@ static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot) * @mask: The mask of pages at offset 'gfn_offset' in this memory * slot to enable dirty logging on * - * Writes protect selected pages to enable dirty logging for them. Caller must - * acquire kvm->mmu_lock. + * Splits selected pages to PAGE_SIZE and then writes protect them to enable + * dirty logging for them. Caller must acquire kvm->mmu_lock. */ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, struct kvm_memory_slot *slot, @@ -1098,6 +1098,13 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, lockdep_assert_held_write(&kvm->mmu_lock); stage2_wp_range(&kvm->arch.mmu, start, end); + + /* + * If initially-all-set mode is not set, then huge-pages were already + * split when enabling dirty logging: no need to do it again. + */ + if (kvm_dirty_log_manual_protect_and_init_set(kvm)) + kvm_mmu_split_huge_pages(kvm, start, end); } static void kvm_send_hwpoison_signal(unsigned long address, short lsb) @@ -1884,7 +1891,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, * this when deleting, moving, disabling dirty logging, or * creating the memslot (a nop). Doing it for deletes makes * sure we don't leak memory, and there's no need to keep the - * cache around for any of the other cases. + * cache around for any of the other cases. Keeping the cache + * is useful for succesive KVM_CLEAR_DIRTY_LOG calls, which is + * not handled in this function. */ kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache); } From patchwork Mon Feb 6 16:58:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Koller X-Patchwork-Id: 13130392 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9174FC05027 for ; Mon, 6 Feb 2023 16:59:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231223AbjBFQ7T (ORCPT ); Mon, 6 Feb 2023 11:59:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231203AbjBFQ7Q (ORCPT ); Mon, 6 Feb 2023 11:59:16 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C771B29419 for ; Mon, 6 Feb 2023 08:59:14 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id b5-20020a170902d50500b00198f3be5233so3456515plg.16 for ; Mon, 06 Feb 2023 08:59:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=11GwySaCQQVZ76JQ0UYzoTUxZC142vqSXtu1AxugS1U=; b=X9EbiorDJ2mPwlJOvEJwg9dfZg6fJXBc2ulCX/1rCWNJj8kB+ltZMVHZ+Zk5t/3Tfi xsir60vX1m2vW2O77FI2c8V2eXwGa5iaDQ5n/M8tV7VaeNXHCRD39Utfyxzc6yx/EtW1 kZGRs6zYNajQ58Hl+R/yqHA0HoAxAmgYVjwRz1chI5lKg5QrkrjBhB0uvl9G1SvSNBZT 8YMu1r/eYHvbdAuwwVNtHiidVaiAHC82b/m9TAkexUhDahGvS6g/6EgblJWKNGBQTULE xR0OwqZGtt2Kh0ONp5BMz6H9I5MX3nR8/PXGphCpKTFhm2YZy2OwRFQaKLmnmBIOMvBs zKzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=11GwySaCQQVZ76JQ0UYzoTUxZC142vqSXtu1AxugS1U=; b=Z4MAZu0eR2xJw2VdlremyqYWvDN3jSGqIj3xLbsTCW1mXUcqkWjrNN0vzgTUU53Q4l jhOgE7BTRBvcTN/FW7ifEgC/ghn4zblXZnDLF2rJK2dk94LS3V29BSG5BR4zV3DxBm/J C1PjmrDQ+0FxKHs6FKiHy6hcL426Z5h5lbbOtkgTAH27qS/TvNVloe+1vqyi7edExPXG QXdN6MGuvSDmSGxsodiVAbod6r0EWLOhwo1cuSZ6ARsBOG6CBgYDA4Uv4HJ+n8fBtcFu udfxHmohmC2XrCMHhc/WoqnVlgBOHGbhlnfKQsvRlXVgtQz2K5SQnf2c5nrGSaxgKmny ob9g== X-Gm-Message-State: AO0yUKUjZsJRU3KyGTh6qBRYqeYjTjLftGQZP94RUEhJjy9kdZCy3+qD V40d3zQZ/La3b4L8s03LYP2+t7BeqyskAA== X-Google-Smtp-Source: AK7set9keklNXR8SX4dKBdKvJqQdvdXPL+xWMBOHWs0VbqbVjUxObY5U3++Tw9yIeP4j6wtI3cfKpL2y2VaYsg== X-Received: from ricarkol4.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1248]) (user=ricarkol job=sendgmr) by 2002:a63:ee0a:0:b0:4db:2ad4:5997 with SMTP id e10-20020a63ee0a000000b004db2ad45997mr3177794pgi.29.1675702754282; Mon, 06 Feb 2023 08:59:14 -0800 (PST) Date: Mon, 6 Feb 2023 16:58:51 +0000 In-Reply-To: <20230206165851.3106338-1-ricarkol@google.com> Mime-Version: 1.0 References: <20230206165851.3106338-1-ricarkol@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230206165851.3106338-13-ricarkol@google.com> Subject: [PATCH v2 12/12] KVM: arm64: Use local TLBI on permission relaxation From: Ricardo Koller To: pbonzini@redhat.com, maz@kernel.org, oupton@google.com, yuzenghui@huawei.com, dmatlack@google.com Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, qperret@google.com, catalin.marinas@arm.com, andrew.jones@linux.dev, seanjc@google.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, eric.auger@redhat.com, gshan@redhat.com, reijiw@google.com, rananta@google.com, bgardon@google.com, ricarkol@gmail.com, Ricardo Koller Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Marc Zyngier Broadcasted TLB invalidations (TLBI) are usually less performant than their local variant. In particular, we observed some implementations that take millliseconds to complete parallel broadcasted TLBIs. It's safe to use local, non-shareable, TLBIs when relaxing permissions on a PTE in the KVM case for a couple of reasons. First, according to the ARM Arm (DDI 0487H.a D5-4913), permission relaxation does not need break-before-make. Second, KVM does not set the VTTBR_EL2.CnP bit, so each PE has its own TLB entry for the same page. KVM could tolerate that when doing permission relaxation (i.e., not having changes broadcasted to all PEs). Signed-off-by: Marc Zyngier Signed-off-by: Ricardo Koller --- arch/arm64/include/asm/kvm_asm.h | 4 +++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 10 ++++++ arch/arm64/kvm/hyp/nvhe/tlb.c | 54 ++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 2 +- arch/arm64/kvm/hyp/vhe/tlb.c | 32 ++++++++++++++++++ 5 files changed, 101 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 43c3bc0f9544..bb17b2ead4c7 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -68,6 +68,7 @@ enum __kvm_host_smccc_func { __KVM_HOST_SMCCC_FUNC___kvm_vcpu_run, __KVM_HOST_SMCCC_FUNC___kvm_flush_vm_context, __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa, + __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa_nsh, __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid, __KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context, __KVM_HOST_SMCCC_FUNC___kvm_timer_set_cntvoff, @@ -225,6 +226,9 @@ extern void __kvm_flush_vm_context(void); extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, int level); +extern void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu, + phys_addr_t ipa, + int level); extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_timer_set_cntvoff(u64 cntvoff); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 728e01d4536b..c6bf1e49ca93 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -125,6 +125,15 @@ static void handle___kvm_tlb_flush_vmid_ipa(struct kvm_cpu_context *host_ctxt) __kvm_tlb_flush_vmid_ipa(kern_hyp_va(mmu), ipa, level); } +static void handle___kvm_tlb_flush_vmid_ipa_nsh(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); + DECLARE_REG(phys_addr_t, ipa, host_ctxt, 2); + DECLARE_REG(int, level, host_ctxt, 3); + + __kvm_tlb_flush_vmid_ipa_nsh(kern_hyp_va(mmu), ipa, level); +} + static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt) { DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); @@ -315,6 +324,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__kvm_vcpu_run), HANDLE_FUNC(__kvm_flush_vm_context), HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa), + HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa_nsh), HANDLE_FUNC(__kvm_tlb_flush_vmid), HANDLE_FUNC(__kvm_flush_cpu_context), HANDLE_FUNC(__kvm_timer_set_cntvoff), diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index d296d617f589..ef2b70587f93 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -109,6 +109,60 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, __tlb_switch_to_host(&cxt); } +void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu, + phys_addr_t ipa, int level) +{ + struct tlb_inv_context cxt; + + dsb(nshst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + /* + * We could do so much better if we had the VA as well. + * Instead, we invalidate Stage-2 for this IPA, and the + * whole of Stage-1. Weep... + */ + ipa >>= 12; + __tlbi_level(ipas2e1, ipa, level); + + /* + * We have to ensure completion of the invalidation at Stage-2, + * since a table walk on another CPU could refill a TLB with a + * complete (S1 + S2) walk based on the old Stage-2 mapping if + * the Stage-1 invalidation happened first. + */ + dsb(nsh); + __tlbi(vmalle1); + dsb(nsh); + isb(); + + /* + * If the host is running at EL1 and we have a VPIPT I-cache, + * then we must perform I-cache maintenance at EL2 in order for + * it to have an effect on the guest. Since the guest cannot hit + * I-cache lines allocated with a different VMID, we don't need + * to worry about junk out of guest reset (we nuke the I-cache on + * VMID rollover), but we do need to be careful when remapping + * executable pages for the same guest. This can happen when KSM + * takes a CoW fault on an executable page, copies the page into + * a page that was previously mapped in the guest and then needs + * to invalidate the guest view of the I-cache for that page + * from EL1. To solve this, we invalidate the entire I-cache when + * unmapping a page from a guest if we have a VPIPT I-cache but + * the host is running at EL1. As above, we could do better if + * we had the VA. + * + * The moral of this story is: if you have a VPIPT I-cache, then + * you should be running with VHE enabled. + */ + if (icache_is_vpipt()) + icache_inval_all_pou(); + + __tlb_switch_to_host(&cxt); +} + void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { struct tlb_inv_context cxt; diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index ae80845c8db7..b154404547d1 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1148,7 +1148,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level, KVM_PGTABLE_WALK_SHARED); if (!ret) - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, level); + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr, level); return ret; } diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index 24cef9b87f9e..e69da550cdc5 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -111,6 +111,38 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, __tlb_switch_to_host(&cxt); } +void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu, + phys_addr_t ipa, int level) +{ + struct tlb_inv_context cxt; + + dsb(nshst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + /* + * We could do so much better if we had the VA as well. + * Instead, we invalidate Stage-2 for this IPA, and the + * whole of Stage-1. Weep... + */ + ipa >>= 12; + __tlbi_level(ipas2e1, ipa, level); + + /* + * We have to ensure completion of the invalidation at Stage-2, + * since a table walk on another CPU could refill a TLB with a + * complete (S1 + S2) walk based on the old Stage-2 mapping if + * the Stage-1 invalidation happened first. + */ + dsb(nsh); + __tlbi(vmalle1); + dsb(nsh); + isb(); + + __tlb_switch_to_host(&cxt); +} + void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { struct tlb_inv_context cxt;