From patchwork Fri Apr 15 21:58:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6C30C433EF for ; Fri, 15 Apr 2022 21:59:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354390AbiDOWBn (ORCPT ); Fri, 15 Apr 2022 18:01:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244034AbiDOWBj (ORCPT ); Fri, 15 Apr 2022 18:01:39 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 135B637ABC for ; Fri, 15 Apr 2022 14:59:10 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id t190-20020a25c3c7000000b006410799ab3dso7542870ybf.21 for ; Fri, 15 Apr 2022 14:59:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Tyg2Q1CJQYh6kqJKn5I8WsZWhqynGf1zv3GZVnyVrPI=; b=bs/o3a1yI+YtQKTec7tfylXXQu4M6ad2QWFFL62JX+gXbwqO9FEus3pkH+ThX+JPAr JPEPreeCxgkfcEjZxuDSMTLLZUbJYuMNUMiPP28ZaBVQA8xAoGGJwyWWqjUTVOr9w+H1 scbsRRQ4oV/0Ko6nzOrL8D58Z+zUc2quNieeYqKnL4/2GddH6EUAGmW8TUmhdokw0buR c0G9PrlnzO6Zct/R0EMtBthEbtbkFluRFT3bcL6Benpi1sBm1BLEUooH5TD8OFDM9SNB FGX1+tPIQqSDfmOl4/bgqfhcbkyjAEMmpAUImOqM0PuNfNbC+i60/UI9Bsp0mp7hYMtn Jqnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Tyg2Q1CJQYh6kqJKn5I8WsZWhqynGf1zv3GZVnyVrPI=; b=kczd9ns9/U4CWq6ZSBPbq7eX/fTVaiRobOGdGtXtPG2RQku3YuzTdJ/XBouaF7o7Iv Zg0nWohPGGtvqcw8t6inzntyEwr07RE8TiNH/p0DOCZqXPMeNkTUyCm653ZN4X1ZcH24 cc/0tOiVDhCjevbqAEoGCY/WXrz6nkAi2jOhObmTZ2pwhE4N+BOa/5wr/sV0USEZnC0z tfw+WQip3nlYW/EmlIumo+n4csFKnnllslY2WiOLbP632pB1hVEWxLIEJm/ulsl/UfXP /OHg5W4EeWNuGrDaDYc+ibYu0cPO8YlVvkkg9DUZeIk7LZZWihE7wSjVZzKnx0LqLCkl XLXg== X-Gm-Message-State: AOAM530rHY8o7T2wN18sjAs3l0tTmXeebtwi6RL3FS2GcVhI44jW3A+c EkHuho8iv9OXc0Si5Jrpz/5gzon10Nw= X-Google-Smtp-Source: ABdhPJxEvl0U3CkvkcrBAn3HsWtsYmV0z/JZVUMkoVOPtU1p1JiWR679eaZiC7C8eGZtheHJa4oZ8Gi2BLk= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6902:544:b0:641:4007:1da8 with SMTP id z4-20020a056902054400b0064140071da8mr1070971ybs.195.1650059949262; Fri, 15 Apr 2022 14:59:09 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:45 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-2-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 01/17] KVM: arm64: Directly read owner id field in stage2_pte_is_counted() From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org A subsequent change to KVM will make use of additional bits in invalid ptes. Prepare for said change by explicitly checking the valid bit and owner fields in stage2_pte_is_counted() Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 2cb3867eb7c2..e1506da3e2fb 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -172,6 +172,11 @@ static kvm_pte_t kvm_init_invalid_leaf_owner(u8 owner_id) return FIELD_PREP(KVM_INVALID_PTE_OWNER_MASK, owner_id); } +static u8 kvm_invalid_pte_owner(kvm_pte_t pte) +{ + return FIELD_GET(KVM_INVALID_PTE_OWNER_MASK, pte); +} + static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64 addr, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag) @@ -679,7 +684,7 @@ static bool stage2_pte_is_counted(kvm_pte_t pte) * encode ownership of a page to another entity than the page-table * owner, whose id is 0. */ - return !!pte; + return kvm_pte_valid(pte) || kvm_invalid_pte_owner(pte); } static void stage2_put_pte(kvm_pte_t *ptep, struct kvm_s2_mmu *mmu, u64 addr, From patchwork Fri Apr 15 21:58:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E23DC433F5 for ; Fri, 15 Apr 2022 21:59:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355427AbiDOWBo (ORCPT ); Fri, 15 Apr 2022 18:01:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244037AbiDOWBl (ORCPT ); Fri, 15 Apr 2022 18:01:41 -0400 Received: from mail-oo1-xc4a.google.com (mail-oo1-xc4a.google.com [IPv6:2607:f8b0:4864:20::c4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14E02381B5 for ; Fri, 15 Apr 2022 14:59:11 -0700 (PDT) Received: by mail-oo1-xc4a.google.com with SMTP id g18-20020a4a6b12000000b003246ed86f00so4838933ooc.23 for ; Fri, 15 Apr 2022 14:59:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=33zitZnypC/5iqOPd40oIiPlXIxM1MYIm9nWTEFWX9o=; b=VeV72ZOKUV0992/61pLhySJu6tUdMC93GGMiIqLb9e42oF03C+E2MC6P0USn6WlWZL rRNNBiXDbdfTk5SekJB4zDXKX5HZQcP9L/L90QZ9Q75Zs43P4FnxEdhX0e9NjQNL70sv 2OPflS3JXbC6xVrB8AYJnRT4D6TEDtD80e3gDO4GetEaSf95eT3DYLJKy2GdkEoTECwt kBvgRBy6BE0rZ3f1wCMtyh7VurS1pRhhSYr6eZEbOvHISaR8hy4G1UMIcnlc/RFerVBo oXY5ygxtXok6qpcxi+MJT9/FZJVx/Zmp7+6uAsJiQSmMLB6jJ6cMJW4AyoLAa7e0QIHR F2Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=33zitZnypC/5iqOPd40oIiPlXIxM1MYIm9nWTEFWX9o=; b=be+PQ91cqdDs0ByBGK7l9fakSORx8Vs/lcRCa5sW0Uod3Ud2Q8TgEVmTTRqGe04Oq8 TyggKPMV++ppmieGyY58sGlYkKUT/j/7uxxHA5nLRhDkhUozMydZkjt45545dC4w7uwg 4HWk5IVrDiDAxvyVnVZLctobfDtTT1hoThr4LU5/K5etFF1U6bwsIunAQ9PBilhjQRwf CGm67navucov36TMTXBqOOvoyZ8ueBltJnAzvbBkT2w3IYNZ6Wds70k48et78Jqlx95l YS3BkivqoWhCV+mUZ7em48BuFoOzjfCU+b9iRBB9Ysf9LIU6xPSyksaoVEgFJwNRuYr/ GWFg== X-Gm-Message-State: AOAM531chp+BhwOiKbygZyg+Qz9QXnjGVHPoCVc/4SV4qork6vznP56P vK3nxcq+pmPg2ueGYe8UOiI+3TEKs3c= X-Google-Smtp-Source: ABdhPJzWYiHch/fvnCe1iFtXCe45SXo8QvSqLvIYoF5WCPSgovlAS0bf0qoZ0Uk3C+rcWH/OsMMHHUvyqdM= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6870:f29a:b0:de:eaa4:233a with SMTP id u26-20020a056870f29a00b000deeaa4233amr354373oap.137.1650059950378; Fri, 15 Apr 2022 14:59:10 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:46 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-3-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 02/17] KVM: arm64: Only read the pte once per visit From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org A subsequent change to KVM will parallize modifications to the stage-2 page tables. The various page table walkers read the ptep multiple times, which could lead to a visitor seeing multiple values during the visit. Pass through the observed pte to the visitor callbacks. Promote reads of the ptep to a full READ_ONCE(), which will matter more when we start tweaking ptes atomically. Note that a pointer to the old pte is given to visitors, as parallel visitors will need to steer the page table traversal as they adjust the page tables. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_pgtable.h | 2 +- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 7 +- arch/arm64/kvm/hyp/nvhe/setup.c | 9 +- arch/arm64/kvm/hyp/pgtable.c | 113 +++++++++++++------------- 4 files changed, 63 insertions(+), 68 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 9f339dffbc1a..ea818a5f7408 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -192,7 +192,7 @@ enum kvm_pgtable_walk_flags { }; typedef int (*kvm_pgtable_visitor_fn_t)(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg); diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 78edf077fa3b..601a586581d8 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -422,17 +422,16 @@ struct check_walk_data { }; static int __check_page_state_visitor(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct check_walk_data *d = arg; - kvm_pte_t pte = *ptep; - if (kvm_pte_valid(pte) && !addr_is_memory(kvm_pte_to_phys(pte))) + if (kvm_pte_valid(*old) && !addr_is_memory(kvm_pte_to_phys(*old))) return -EINVAL; - return d->get_page_state(pte) == d->desired ? 0 : -EPERM; + return d->get_page_state(*old) == d->desired ? 0 : -EPERM; } static int check_page_state_range(struct kvm_pgtable *pgt, u64 addr, u64 size, diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index 27af337f9fea..ecab7a4049d6 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -162,17 +162,16 @@ static void hpool_put_page(void *addr) } static int finalize_host_mappings_walker(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct kvm_pgtable_mm_ops *mm_ops = arg; enum kvm_pgtable_prot prot; enum pkvm_page_state state; - kvm_pte_t pte = *ptep; phys_addr_t phys; - if (!kvm_pte_valid(pte)) + if (!kvm_pte_valid(*old)) return 0; /* @@ -187,7 +186,7 @@ static int finalize_host_mappings_walker(u64 addr, u64 end, u32 level, if (level != (KVM_PGTABLE_MAX_LEVELS - 1)) return -EINVAL; - phys = kvm_pte_to_phys(pte); + phys = kvm_pte_to_phys(*old); if (!addr_is_memory(phys)) return -EINVAL; @@ -195,7 +194,7 @@ static int finalize_host_mappings_walker(u64 addr, u64 end, u32 level, * Adjust the host stage-2 mappings to match the ownership attributes * configured in the hypervisor stage-1. */ - state = pkvm_getstate(kvm_pgtable_hyp_pte_prot(pte)); + state = pkvm_getstate(kvm_pgtable_hyp_pte_prot(*old)); switch (state) { case PKVM_PAGE_OWNED: return host_stage2_set_owner_locked(phys, PAGE_SIZE, pkvm_hyp_id); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index e1506da3e2fb..ad911cd44425 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -178,11 +178,11 @@ static u8 kvm_invalid_pte_owner(kvm_pte_t pte) } static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64 addr, - u32 level, kvm_pte_t *ptep, + u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag) { struct kvm_pgtable_walker *walker = data->walker; - return walker->cb(addr, data->end, level, ptep, flag, walker->arg); + return walker->cb(addr, data->end, level, ptep, old, flag, walker->arg); } static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, @@ -193,17 +193,17 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, { int ret = 0; u64 addr = data->addr; - kvm_pte_t *childp, pte = *ptep; + kvm_pte_t *childp, pte = READ_ONCE(*ptep); bool table = kvm_pte_table(pte, level); enum kvm_pgtable_walk_flags flags = data->walker->flags; if (table && (flags & KVM_PGTABLE_WALK_TABLE_PRE)) { - ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, + ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, KVM_PGTABLE_WALK_TABLE_PRE); } if (!table && (flags & KVM_PGTABLE_WALK_LEAF)) { - ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, + ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, KVM_PGTABLE_WALK_LEAF); pte = *ptep; table = kvm_pte_table(pte, level); @@ -224,7 +224,7 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, goto out; if (flags & KVM_PGTABLE_WALK_TABLE_POST) { - ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, + ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, KVM_PGTABLE_WALK_TABLE_POST); } @@ -297,12 +297,12 @@ struct leaf_walk_data { u32 level; }; -static int leaf_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, +static int leaf_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct leaf_walk_data *data = arg; - data->pte = *ptep; + data->pte = *old; data->level = level; return 0; @@ -388,10 +388,10 @@ enum kvm_pgtable_prot kvm_pgtable_hyp_pte_prot(kvm_pte_t pte) return prot; } -static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, struct hyp_map_data *data) +static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, + kvm_pte_t old, struct hyp_map_data *data) { - kvm_pte_t new, old = *ptep; + kvm_pte_t new; u64 granule = kvm_granule_size(level), phys = data->phys; if (!kvm_block_mapping_supported(addr, end, phys, level)) @@ -410,14 +410,14 @@ static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, return true; } -static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, +static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { kvm_pte_t *childp; struct hyp_map_data *data = arg; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; - if (hyp_map_walker_try_leaf(addr, end, level, ptep, arg)) + if (hyp_map_walker_try_leaf(addr, end, level, ptep, *old, arg)) return 0; if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1)) @@ -461,19 +461,19 @@ struct hyp_unmap_data { struct kvm_pgtable_mm_ops *mm_ops; }; -static int hyp_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, +static int hyp_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { - kvm_pte_t pte = *ptep, *childp = NULL; + kvm_pte_t *childp = NULL; u64 granule = kvm_granule_size(level); struct hyp_unmap_data *data = arg; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; - if (!kvm_pte_valid(pte)) + if (!kvm_pte_valid(*old)) return -EINVAL; - if (kvm_pte_table(pte, level)) { - childp = kvm_pte_follow(pte, mm_ops); + if (kvm_pte_table(*old, level)) { + childp = kvm_pte_follow(*old, mm_ops); if (mm_ops->page_count(childp) != 1) return 0; @@ -537,19 +537,18 @@ int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits, return 0; } -static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, +static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct kvm_pgtable_mm_ops *mm_ops = arg; - kvm_pte_t pte = *ptep; - if (!kvm_pte_valid(pte)) + if (!kvm_pte_valid(*old)) return 0; mm_ops->put_page(ptep); - if (kvm_pte_table(pte, level)) - mm_ops->put_page(kvm_pte_follow(pte, mm_ops)); + if (kvm_pte_table(*old, level)) + mm_ops->put_page(kvm_pte_follow(*old, mm_ops)); return 0; } @@ -723,10 +722,10 @@ static bool stage2_leaf_mapping_allowed(u64 addr, u64 end, u32 level, } static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t old, struct stage2_map_data *data) { - kvm_pte_t new, old = *ptep; + kvm_pte_t new; u64 granule = kvm_granule_size(level), phys = data->phys; struct kvm_pgtable *pgt = data->mmu->pgt; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; @@ -769,7 +768,7 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, } static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t *old, struct stage2_map_data *data) { if (data->anchor) @@ -778,7 +777,7 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, if (!stage2_leaf_mapping_allowed(addr, end, level, data)) return 0; - data->childp = kvm_pte_follow(*ptep, data->mm_ops); + data->childp = kvm_pte_follow(*old, data->mm_ops); kvm_clear_pte(ptep); /* @@ -792,20 +791,20 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, } static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - struct stage2_map_data *data) + kvm_pte_t *old, struct stage2_map_data *data) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; - kvm_pte_t *childp, pte = *ptep; + kvm_pte_t *childp; int ret; if (data->anchor) { - if (stage2_pte_is_counted(pte)) + if (stage2_pte_is_counted(*old)) mm_ops->put_page(ptep); return 0; } - ret = stage2_map_walker_try_leaf(addr, end, level, ptep, data); + ret = stage2_map_walker_try_leaf(addr, end, level, ptep, *old, data); if (ret != -E2BIG) return ret; @@ -824,7 +823,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, * a table. Accesses beyond 'end' that fall within the new table * will be mapped lazily. */ - if (stage2_pte_is_counted(pte)) + if (stage2_pte_is_counted(*old)) stage2_put_pte(ptep, data->mmu, addr, level, mm_ops); kvm_set_table_pte(ptep, childp, mm_ops); @@ -834,7 +833,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, } static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t *old, struct stage2_map_data *data) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; @@ -848,9 +847,9 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, childp = data->childp; data->anchor = NULL; data->childp = NULL; - ret = stage2_map_walk_leaf(addr, end, level, ptep, data); + ret = stage2_map_walk_leaf(addr, end, level, ptep, old, data); } else { - childp = kvm_pte_follow(*ptep, mm_ops); + childp = kvm_pte_follow(*old, mm_ops); } mm_ops->put_page(childp); @@ -878,18 +877,18 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, * the page-table, installing the block entry when it revisits the anchor * pointer and clearing the anchor to NULL. */ -static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, +static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct stage2_map_data *data = arg; switch (flag) { case KVM_PGTABLE_WALK_TABLE_PRE: - return stage2_map_walk_table_pre(addr, end, level, ptep, data); + return stage2_map_walk_table_pre(addr, end, level, ptep, old, data); case KVM_PGTABLE_WALK_LEAF: - return stage2_map_walk_leaf(addr, end, level, ptep, data); + return stage2_map_walk_leaf(addr, end, level, ptep, old, data); case KVM_PGTABLE_WALK_TABLE_POST: - return stage2_map_walk_table_post(addr, end, level, ptep, data); + return stage2_map_walk_table_post(addr, end, level, ptep, old, data); } return -EINVAL; @@ -955,29 +954,29 @@ int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, } static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - enum kvm_pgtable_walk_flags flag, + kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct kvm_pgtable *pgt = arg; struct kvm_s2_mmu *mmu = pgt->mmu; struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops; - kvm_pte_t pte = *ptep, *childp = NULL; + kvm_pte_t *childp = NULL; bool need_flush = false; - if (!kvm_pte_valid(pte)) { - if (stage2_pte_is_counted(pte)) { + if (!kvm_pte_valid(*old)) { + if (stage2_pte_is_counted(*old)) { kvm_clear_pte(ptep); mm_ops->put_page(ptep); } return 0; } - if (kvm_pte_table(pte, level)) { - childp = kvm_pte_follow(pte, mm_ops); + if (kvm_pte_table(*old, level)) { + childp = kvm_pte_follow(*old, mm_ops); if (mm_ops->page_count(childp) != 1) return 0; - } else if (stage2_pte_cacheable(pgt, pte)) { + } else if (stage2_pte_cacheable(pgt, *old)) { need_flush = !stage2_has_fwb(pgt); } @@ -989,7 +988,7 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, stage2_put_pte(ptep, mmu, addr, level, mm_ops); if (need_flush && mm_ops->dcache_clean_inval_poc) - mm_ops->dcache_clean_inval_poc(kvm_pte_follow(pte, mm_ops), + mm_ops->dcache_clean_inval_poc(kvm_pte_follow(*old, mm_ops), kvm_granule_size(level)); if (childp) @@ -1018,10 +1017,10 @@ struct stage2_attr_data { }; static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - enum kvm_pgtable_walk_flags flag, + kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { - kvm_pte_t pte = *ptep; + kvm_pte_t pte = *old; struct stage2_attr_data *data = arg; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; @@ -1146,18 +1145,17 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, } static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - enum kvm_pgtable_walk_flags flag, + kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct kvm_pgtable *pgt = arg; struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops; - kvm_pte_t pte = *ptep; - if (!kvm_pte_valid(pte) || !stage2_pte_cacheable(pgt, pte)) + if (!kvm_pte_valid(*old) || !stage2_pte_cacheable(pgt, *old)) return 0; if (mm_ops->dcache_clean_inval_poc) - mm_ops->dcache_clean_inval_poc(kvm_pte_follow(pte, mm_ops), + mm_ops->dcache_clean_inval_poc(kvm_pte_follow(*old, mm_ops), kvm_granule_size(level)); return 0; } @@ -1206,19 +1204,18 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu, } static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - enum kvm_pgtable_walk_flags flag, + kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct kvm_pgtable_mm_ops *mm_ops = arg; - kvm_pte_t pte = *ptep; - if (!stage2_pte_is_counted(pte)) + if (!stage2_pte_is_counted(*old)) return 0; mm_ops->put_page(ptep); - if (kvm_pte_table(pte, level)) - mm_ops->put_page(kvm_pte_follow(pte, mm_ops)); + if (kvm_pte_table(*old, level)) + mm_ops->put_page(kvm_pte_follow(*old, mm_ops)); return 0; } From patchwork Fri Apr 15 21:58:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6087C43217 for ; Fri, 15 Apr 2022 21:59:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355699AbiDOWBp (ORCPT ); Fri, 15 Apr 2022 18:01:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37488 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244152AbiDOWBl (ORCPT ); Fri, 15 Apr 2022 18:01:41 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3CA57377F8 for ; Fri, 15 Apr 2022 14:59:12 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id n10-20020a25da0a000000b0064149e56ecaso7651087ybf.2 for ; Fri, 15 Apr 2022 14:59:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=rJSIRdYzlgGDLU7B2Sk5nACgnVsuidWq4rds0LFJzOM=; b=SkrQ1f8DZ8O/NnR7vbyJvJQIHE/etfmq3uFO690m3CAYnbo6y2Ixqb7gAvpAOTVELL 6wOSAg7n/y+F6WeMp05hmHl1fcyiWC8VoNMxEO4OeOYzRlFDvzzQ7jFGCTtm845QHSoG LccB7cLdd71eA0BI0Ru81TgjWmc8dS30Ot2pDg0Hy1uI1b5zb05v+/GrUfnWCauhOnPJ NySizl2te/E7Y0lMCQzjXVzUzPthbWsjhBBncJtixhl5pRXhKQ3taAldhzHbsTSuXMOE rm5tR4TfdW/tfw/Uf8GxBSY3l4Dd0OpWCNbMfnbvgCpEtN0cME8Q2ExGtfE12VB6U/GX +VFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=rJSIRdYzlgGDLU7B2Sk5nACgnVsuidWq4rds0LFJzOM=; b=G+3hRbYGjYRm2NKwWMIAm94QseQDBfgmWEkLbHK5qdiNSRAChfPODxHMFGIkyIpBAP SSunruvgX2mPh8WjhMSD//TL4lKoyRLzmCLVxTVL8mZQ+HzlVc+6oRtmpCzWCq74BlGG 1n+ZqUbEzoNuhNLz65w2TPXLguaV9lapVUTHASxmXjsTX8kRjcEdy4YL/cQgYM2PEw3e XuLKFAvTuH1sjVOiCGBv3u7z5dxgw4q6hO1X17EfZhPp4S3cUwlKBR2xGbCr+YVE6kdL lm+HKxdRXDRMROCEsMKniIDocIOi2G8kuTqaD8vUJMaXTBxq4xkmf88jksg0IK0cqlvz NIDQ== X-Gm-Message-State: AOAM531wfoMTh0KgHFSQOtA2OZBZ6CmVqo+SvZltjnSl4a0VXYlgwTh+ Sd6ovPowFCakB34FECbtcOBJ8JUulv0= X-Google-Smtp-Source: ABdhPJwz4sx5L43ob0Z3jNzyJVyA2Jzsl+sCY5ofO1Xxk9cu0zlH1igqV35RpwEAUlFfB8+y4wm6MGToMi4= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a25:6f55:0:b0:63e:7447:7c19 with SMTP id k82-20020a256f55000000b0063e74477c19mr1036666ybc.551.1650059951469; Fri, 15 Apr 2022 14:59:11 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:47 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-4-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 03/17] KVM: arm64: Return the next table from map callbacks From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The stage-2 and hyp stage-1 map walkers install new page tables during their traversal. In order to support parallel table walks, make callers return the next table to traverse. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index ad911cd44425..5b64fbca8a93 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -205,13 +205,12 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, if (!table && (flags & KVM_PGTABLE_WALK_LEAF)) { ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, KVM_PGTABLE_WALK_LEAF); - pte = *ptep; - table = kvm_pte_table(pte, level); } if (ret) goto out; + table = kvm_pte_table(pte, level); if (!table) { data->addr = ALIGN_DOWN(data->addr, kvm_granule_size(level)); data->addr += kvm_granule_size(level); @@ -429,6 +428,7 @@ static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte kvm_set_table_pte(ptep, childp, mm_ops); mm_ops->get_page(ptep); + *old = *ptep; return 0; } @@ -828,7 +828,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_set_table_pte(ptep, childp, mm_ops); mm_ops->get_page(ptep); - + *old = *ptep; return 0; } From patchwork Fri Apr 15 21:58:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815456 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89F99C433EF for ; Fri, 15 Apr 2022 21:59:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355842AbiDOWBq (ORCPT ); Fri, 15 Apr 2022 18:01:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229500AbiDOWBm (ORCPT ); Fri, 15 Apr 2022 18:01:42 -0400 Received: from mail-io1-xd49.google.com (mail-io1-xd49.google.com [IPv6:2607:f8b0:4864:20::d49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DEE6037ABC for ; Fri, 15 Apr 2022 14:59:12 -0700 (PDT) Received: by mail-io1-xd49.google.com with SMTP id y13-20020a5e834d000000b0064fc8cd0bf1so3626539iom.14 for ; Fri, 15 Apr 2022 14:59:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=tskkletjI1xqy2hmf+NUDTVo6kgDSU4EL5qdYTa3BLw=; b=QA94h9r9pTA8GiRYxYPRCc5lnTzXtNJwZmjCbvYN9o6CVFPh0a2tzWI0hkn2GGLweA Txpvs213Jksx0vyr2ngPBysyiEw8KC95EbN0Ju8DT94akH1bQ62gDiZBnBpVmOCl0cU/ aVzTHXbtTPOJrXqnXT7zTKFXplRLdZ7urB7/NYg8iYE+D1B5NQXQp/3h+iCMDQQGzAYs +i1QI9nvkwP3sApqy92wxAC2ykVI1oGcIkQRiYST9T63+AoQWeZ9+xAr1yQAoB7YwpSw L5Y0d4II8kHY6IRbv/oLjeQ9jnR7XErbOljztHLeXABbdjtB3zDvW2Om/oADovnpl9yM AUkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=tskkletjI1xqy2hmf+NUDTVo6kgDSU4EL5qdYTa3BLw=; b=U5vmIuN6Hr/jRvU+pyjJEmF8DJ4hx/k6CrwcRQG1Gjyvlqe+e7gMd3SfaQBS4jgbo/ 9zMivxY+IO07y+Oq5HKeY4ABOnoAruVGUOHmpN6sCh7+s2xg9wZ0Vk52wdKu3iglThfm FpxWIpzoo8qJfPuvhhbLuz8nD/4G5t4/cikfyXz/DIHwMbgnntrZEnx6T3n+vaA1REx5 OrwaTBeUUViXwWwTWUY9ajqEMlV4wyDo3HcnS6tT7aTm99V/+uswFIKJj4/QA37QL+kN HiHUub0y2CsZw54Gcmlj4aGQPIfiDDiZZt12nb/oT1J9dzUHF9y2/Hmax5OqTFIFhqjY yenA== X-Gm-Message-State: AOAM533pPHHfg0a5RlOdyaCBYmwYS/Ah7DJPmwvAz6FtEtBmvR/8gWVK 6NCIJm4P11MVghnKnIfav05U21IRQKc= X-Google-Smtp-Source: ABdhPJyVhtFMabjcfJB2grqI91o53/8kDGiTwPio15sftKb54bR0cux4tb8br2mZT5w8v0L5arzr0CpVpX8= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6e02:20cd:b0:2ca:c074:4ebb with SMTP id 13-20020a056e0220cd00b002cac0744ebbmr283965ilq.73.1650059952284; Fri, 15 Apr 2022 14:59:12 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:48 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-5-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 04/17] KVM: arm64: Protect page table traversal with RCU From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use RCU to safely traverse the page tables in parallel; the tables themselves will only be freed from an RCU synchronized context. Don't even bother with adding support to hyp, and instead just assume exclusive access of the page tables. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 5b64fbca8a93..d4699f698d6e 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -132,9 +132,28 @@ static kvm_pte_t kvm_phys_to_pte(u64 pa) return pte; } + +#if defined(__KVM_NVHE_HYPERVISOR__) +static inline void kvm_pgtable_walk_begin(void) +{} + +static inline void kvm_pgtable_walk_end(void) +{} + +#define kvm_dereference_ptep rcu_dereference_raw +#else +#define kvm_pgtable_walk_begin rcu_read_lock + +#define kvm_pgtable_walk_end rcu_read_unlock + +#define kvm_dereference_ptep rcu_dereference +#endif + static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte, struct kvm_pgtable_mm_ops *mm_ops) { - return mm_ops->phys_to_virt(kvm_pte_to_phys(pte)); + kvm_pte_t __rcu *ptep = mm_ops->phys_to_virt(kvm_pte_to_phys(pte)); + + return kvm_dereference_ptep(ptep); } static void kvm_clear_pte(kvm_pte_t *ptep) @@ -288,7 +307,9 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, .walker = walker, }; + kvm_pgtable_walk_begin(); return _kvm_pgtable_walk(&walk_data); + kvm_pgtable_walk_end(); } struct leaf_walk_data { From patchwork Fri Apr 15 21:58:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A9A7C433F5 for ; Fri, 15 Apr 2022 21:59:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356042AbiDOWBr (ORCPT ); Fri, 15 Apr 2022 18:01:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244034AbiDOWBn (ORCPT ); Fri, 15 Apr 2022 18:01:43 -0400 Received: from mail-io1-xd4a.google.com (mail-io1-xd4a.google.com [IPv6:2607:f8b0:4864:20::d4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B2A84377F8 for ; Fri, 15 Apr 2022 14:59:13 -0700 (PDT) Received: by mail-io1-xd4a.google.com with SMTP id ay41-20020a5d9da9000000b00653e75506b8so58861iob.2 for ; Fri, 15 Apr 2022 14:59:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=1Dvq0vSpuJ+B5i+lAUjot+b1S2UM0uuMiRyTOuKbozw=; b=f0Ct2KeIjmpzITrXuAlU/IXp9HqHaYUY14BbtuMlXzD5cMFehCeAkuIQZN9MFinFsR Qbp4qrLoUZqthvpvsvdzGjy8wLygbQ0WlDP6aTuRbHdO6gSsTKMD7flO7JZ2wK78u7pG wT+k3nbsihczJyCyCIw6RfoaQT/fGxDvVRDbU7dqJ8LveecvlvpXCtSrZ8FvWshbrqjr Ng8Gw6k3NN7OCQLHsMdOXloj+4FOOvcMCgra/n/WfgQ+hXQpA9D4ZmLf6aM2jgKzhggE p0G7rjmCjRnZn/8CS7n2NbTu/eKmMqhSDUDAs++6W0Jit3quykTz1+d+JiM2+53KSPXM XqQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=1Dvq0vSpuJ+B5i+lAUjot+b1S2UM0uuMiRyTOuKbozw=; b=lu8q2an8UiMEty2zgKqGq8PYwXvBK5wqGtQWmCW7sQN3OZ9DINaFSzEz3rG5VInF62 pr9s52Dwz8UrajEbI8p8zQDdMPmYWraNnksuXdsTVigZ+sxBdltaGU9N9rjUG5hiFrVb WR//8geIT/Pq3Y7MLpvBO/Xgzs1IoqgTTYtOcTtWpVnR6JenwFHhmZvPhcezijuTXrFD CSngKQuTEOyVrK/i0h605LnEZs/Ya3366caHrR5AndgQ+i4E2HNvT9q4YZevDZwjmpbc +Tc56QqJ47IPBNBEHCFT6drFEcv8STrHJ+s9Iz3ttfavfTnGFnq1NWwafk6vBYWwry4w Vc9A== X-Gm-Message-State: AOAM531POOA8WQpxGaloeztHhD8UIGCwzZlifSb2bjkDqy/qX60FEe/S n5lH8n2GM3KI/z0kgJvHyqcUm+RRVTM= X-Google-Smtp-Source: ABdhPJw+CGHjGO0ZS08PkgVlHelz0Jm4z2IXXsD/Vyzt5RIyBZDRwHzHFKVKAtaVcfL7OMy4ZxqJFar+bXU= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6e02:12e9:b0:2c9:f576:8951 with SMTP id l9-20020a056e0212e900b002c9f5768951mr355751iln.204.1650059953138; Fri, 15 Apr 2022 14:59:13 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:49 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-6-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 05/17] KVM: arm64: Take an argument to indicate parallel walk From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org It is desirable to reuse the same page walkers for serial and parallel faults. Take an argument to kvm_pgtable_walk() (and throughout) to indicate whether or not a walk might happen in parallel with another. No functional change intended. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_pgtable.h | 5 +- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 4 +- arch/arm64/kvm/hyp/nvhe/setup.c | 4 +- arch/arm64/kvm/hyp/pgtable.c | 91 ++++++++++++++------------- 4 files changed, 54 insertions(+), 50 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index ea818a5f7408..74955aba5918 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -194,7 +194,7 @@ enum kvm_pgtable_walk_flags { typedef int (*kvm_pgtable_visitor_fn_t)(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg); + void * const arg, bool shared); /** * struct kvm_pgtable_walker - Hook into a page-table walk. @@ -490,6 +490,7 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size); * @addr: Input address for the start of the walk. * @size: Size of the range to walk. * @walker: Walker callback description. + * @shared: Indicates if the page table walk can be done in parallel * * The offset of @addr within a page is ignored and @size is rounded-up to * the next page boundary. @@ -506,7 +507,7 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size); * Return: 0 on success, negative error code on failure. */ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, - struct kvm_pgtable_walker *walker); + struct kvm_pgtable_walker *walker, bool shared); /** * kvm_pgtable_get_leaf() - Walk a page-table and retrieve the leaf entry diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 601a586581d8..42a5f35cd819 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -424,7 +424,7 @@ struct check_walk_data { static int __check_page_state_visitor(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { struct check_walk_data *d = arg; @@ -443,7 +443,7 @@ static int check_page_state_range(struct kvm_pgtable *pgt, u64 addr, u64 size, .flags = KVM_PGTABLE_WALK_LEAF, }; - return kvm_pgtable_walk(pgt, addr, size, &walker); + return kvm_pgtable_walk(pgt, addr, size, &walker, false); } static enum pkvm_page_state host_get_page_state(kvm_pte_t pte) diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index ecab7a4049d6..178a5539fe7c 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -164,7 +164,7 @@ static void hpool_put_page(void *addr) static int finalize_host_mappings_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { struct kvm_pgtable_mm_ops *mm_ops = arg; enum kvm_pgtable_prot prot; @@ -224,7 +224,7 @@ static int finalize_host_mappings(void) struct memblock_region *reg = &hyp_memory[i]; u64 start = (u64)hyp_phys_to_virt(reg->base); - ret = kvm_pgtable_walk(&pkvm_pgtable, start, reg->size, &walker); + ret = kvm_pgtable_walk(&pkvm_pgtable, start, reg->size, &walker, false); if (ret) return ret; } diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index d4699f698d6e..bf46d6d24951 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -198,17 +198,17 @@ static u8 kvm_invalid_pte_owner(kvm_pte_t pte) static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64 addr, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag) + enum kvm_pgtable_walk_flags flag, bool shared) { struct kvm_pgtable_walker *walker = data->walker; - return walker->cb(addr, data->end, level, ptep, old, flag, walker->arg); + return walker->cb(addr, data->end, level, ptep, old, flag, walker->arg, shared); } static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, - kvm_pte_t *pgtable, u32 level); + kvm_pte_t *pgtable, u32 level, bool shared); static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, - kvm_pte_t *ptep, u32 level) + kvm_pte_t *ptep, u32 level, bool shared) { int ret = 0; u64 addr = data->addr; @@ -218,12 +218,12 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, if (table && (flags & KVM_PGTABLE_WALK_TABLE_PRE)) { ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, - KVM_PGTABLE_WALK_TABLE_PRE); + KVM_PGTABLE_WALK_TABLE_PRE, shared); } if (!table && (flags & KVM_PGTABLE_WALK_LEAF)) { ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, - KVM_PGTABLE_WALK_LEAF); + KVM_PGTABLE_WALK_LEAF, shared); } if (ret) @@ -237,13 +237,13 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, } childp = kvm_pte_follow(pte, data->pgt->mm_ops); - ret = __kvm_pgtable_walk(data, childp, level + 1); + ret = __kvm_pgtable_walk(data, childp, level + 1, shared); if (ret) goto out; if (flags & KVM_PGTABLE_WALK_TABLE_POST) { ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, - KVM_PGTABLE_WALK_TABLE_POST); + KVM_PGTABLE_WALK_TABLE_POST, shared); } out: @@ -251,7 +251,7 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, } static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, - kvm_pte_t *pgtable, u32 level) + kvm_pte_t *pgtable, u32 level, bool shared) { u32 idx; int ret = 0; @@ -265,7 +265,7 @@ static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, if (data->addr >= data->end) break; - ret = __kvm_pgtable_visit(data, ptep, level); + ret = __kvm_pgtable_visit(data, ptep, level, shared); if (ret) break; } @@ -273,7 +273,7 @@ static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, return ret; } -static int _kvm_pgtable_walk(struct kvm_pgtable_walk_data *data) +static int _kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, bool shared) { u32 idx; int ret = 0; @@ -289,7 +289,7 @@ static int _kvm_pgtable_walk(struct kvm_pgtable_walk_data *data) for (idx = kvm_pgd_page_idx(data); data->addr < data->end; ++idx) { kvm_pte_t *ptep = &pgt->pgd[idx * PTRS_PER_PTE]; - ret = __kvm_pgtable_walk(data, ptep, pgt->start_level); + ret = __kvm_pgtable_walk(data, ptep, pgt->start_level, shared); if (ret) break; } @@ -298,7 +298,7 @@ static int _kvm_pgtable_walk(struct kvm_pgtable_walk_data *data) } int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, - struct kvm_pgtable_walker *walker) + struct kvm_pgtable_walker *walker, bool shared) { struct kvm_pgtable_walk_data walk_data = { .pgt = pgt, @@ -308,7 +308,7 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, }; kvm_pgtable_walk_begin(); - return _kvm_pgtable_walk(&walk_data); + return _kvm_pgtable_walk(&walk_data, shared); kvm_pgtable_walk_end(); } @@ -318,7 +318,7 @@ struct leaf_walk_data { }; static int leaf_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag, void * const arg) + enum kvm_pgtable_walk_flags flag, void * const arg, bool shared) { struct leaf_walk_data *data = arg; @@ -340,7 +340,7 @@ int kvm_pgtable_get_leaf(struct kvm_pgtable *pgt, u64 addr, int ret; ret = kvm_pgtable_walk(pgt, ALIGN_DOWN(addr, PAGE_SIZE), - PAGE_SIZE, &walker); + PAGE_SIZE, &walker, false); if (!ret) { if (ptep) *ptep = data.pte; @@ -409,7 +409,7 @@ enum kvm_pgtable_prot kvm_pgtable_hyp_pte_prot(kvm_pte_t pte) } static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - kvm_pte_t old, struct hyp_map_data *data) + kvm_pte_t old, struct hyp_map_data *data, bool shared) { kvm_pte_t new; u64 granule = kvm_granule_size(level), phys = data->phys; @@ -431,13 +431,13 @@ static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *pte } static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag, void * const arg) + enum kvm_pgtable_walk_flags flag, void * const arg, bool shared) { kvm_pte_t *childp; struct hyp_map_data *data = arg; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; - if (hyp_map_walker_try_leaf(addr, end, level, ptep, *old, arg)) + if (hyp_map_walker_try_leaf(addr, end, level, ptep, *old, arg, shared)) return 0; if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1)) @@ -471,7 +471,7 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, if (ret) return ret; - ret = kvm_pgtable_walk(pgt, addr, size, &walker); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); dsb(ishst); isb(); return ret; @@ -483,7 +483,7 @@ struct hyp_unmap_data { }; static int hyp_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag, void * const arg) + enum kvm_pgtable_walk_flags flag, void * const arg, bool shared) { kvm_pte_t *childp = NULL; u64 granule = kvm_granule_size(level); @@ -536,7 +536,7 @@ u64 kvm_pgtable_hyp_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) if (!pgt->mm_ops->page_count) return 0; - kvm_pgtable_walk(pgt, addr, size, &walker); + kvm_pgtable_walk(pgt, addr, size, &walker, false); return unmap_data.unmapped; } @@ -559,7 +559,7 @@ int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits, } static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag, void * const arg) + enum kvm_pgtable_walk_flags flag, void * const arg, bool shared) { struct kvm_pgtable_mm_ops *mm_ops = arg; @@ -582,7 +582,7 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt) .arg = pgt->mm_ops, }; - WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); + WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker, false)); pgt->mm_ops->put_page(pgt->pgd); pgt->pgd = NULL; } @@ -744,7 +744,8 @@ static bool stage2_leaf_mapping_allowed(u64 addr, u64 end, u32 level, static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t old, - struct stage2_map_data *data) + struct stage2_map_data *data, + bool shared) { kvm_pte_t new; u64 granule = kvm_granule_size(level), phys = data->phys; @@ -790,7 +791,8 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - struct stage2_map_data *data) + struct stage2_map_data *data, + bool shared) { if (data->anchor) return 0; @@ -812,7 +814,7 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, } static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - kvm_pte_t *old, struct stage2_map_data *data) + kvm_pte_t *old, struct stage2_map_data *data, bool shared) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; kvm_pte_t *childp; @@ -825,7 +827,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, return 0; } - ret = stage2_map_walker_try_leaf(addr, end, level, ptep, *old, data); + ret = stage2_map_walker_try_leaf(addr, end, level, ptep, *old, data, shared); if (ret != -E2BIG) return ret; @@ -855,7 +857,8 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - struct stage2_map_data *data) + struct stage2_map_data *data, + bool shared) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; kvm_pte_t *childp; @@ -868,7 +871,7 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, childp = data->childp; data->anchor = NULL; data->childp = NULL; - ret = stage2_map_walk_leaf(addr, end, level, ptep, old, data); + ret = stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared); } else { childp = kvm_pte_follow(*old, mm_ops); } @@ -899,17 +902,17 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, * pointer and clearing the anchor to NULL. */ static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag, void * const arg) + enum kvm_pgtable_walk_flags flag, void * const arg, bool shared) { struct stage2_map_data *data = arg; switch (flag) { case KVM_PGTABLE_WALK_TABLE_PRE: - return stage2_map_walk_table_pre(addr, end, level, ptep, old, data); + return stage2_map_walk_table_pre(addr, end, level, ptep, old, data, shared); case KVM_PGTABLE_WALK_LEAF: - return stage2_map_walk_leaf(addr, end, level, ptep, old, data); + return stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared); case KVM_PGTABLE_WALK_TABLE_POST: - return stage2_map_walk_table_post(addr, end, level, ptep, old, data); + return stage2_map_walk_table_post(addr, end, level, ptep, old, data, shared); } return -EINVAL; @@ -942,7 +945,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, if (ret) return ret; - ret = kvm_pgtable_walk(pgt, addr, size, &walker); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); dsb(ishst); return ret; } @@ -970,13 +973,13 @@ int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, if (owner_id > KVM_MAX_OWNER_ID) return -EINVAL; - ret = kvm_pgtable_walk(pgt, addr, size, &walker); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); return ret; } static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { struct kvm_pgtable *pgt = arg; struct kvm_s2_mmu *mmu = pgt->mmu; @@ -1026,7 +1029,7 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, }; - return kvm_pgtable_walk(pgt, addr, size, &walker); + return kvm_pgtable_walk(pgt, addr, size, &walker, false); } struct stage2_attr_data { @@ -1039,7 +1042,7 @@ struct stage2_attr_data { static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { kvm_pte_t pte = *old; struct stage2_attr_data *data = arg; @@ -1091,7 +1094,7 @@ static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr, .flags = KVM_PGTABLE_WALK_LEAF, }; - ret = kvm_pgtable_walk(pgt, addr, size, &walker); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); if (ret) return ret; @@ -1167,7 +1170,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { struct kvm_pgtable *pgt = arg; struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops; @@ -1192,7 +1195,7 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) if (stage2_has_fwb(pgt)) return 0; - return kvm_pgtable_walk(pgt, addr, size, &walker); + return kvm_pgtable_walk(pgt, addr, size, &walker, false); } @@ -1226,7 +1229,7 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu, static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { struct kvm_pgtable_mm_ops *mm_ops = arg; @@ -1251,7 +1254,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt) .arg = pgt->mm_ops, }; - WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); + WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker, false)); pgd_sz = kvm_pgd_pages(pgt->ia_bits, pgt->start_level) * PAGE_SIZE; pgt->mm_ops->free_pages_exact(pgt->pgd, pgd_sz); pgt->pgd = NULL; From patchwork Fri Apr 15 21:58:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BB43C433FE for ; Fri, 15 Apr 2022 21:59:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356086AbiDOWBt (ORCPT ); Fri, 15 Apr 2022 18:01:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354780AbiDOWBo (ORCPT ); Fri, 15 Apr 2022 18:01:44 -0400 Received: from mail-io1-xd4a.google.com (mail-io1-xd4a.google.com [IPv6:2607:f8b0:4864:20::d4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E33937ABC for ; Fri, 15 Apr 2022 14:59:14 -0700 (PDT) Received: by mail-io1-xd4a.google.com with SMTP id z23-20020a6b0a17000000b00649f13ea3a7so5404376ioi.23 for ; Fri, 15 Apr 2022 14:59:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=dlVM2Fz2Y7ISzW+T03M67t0iCDhFZowMvNW5W95Ip88=; b=Diuw9S7MBxEBgzLHbnnt4GYfzxtBiJUB0zsGMYHDzpDoe3JhqJZWSJDdokUWM0zkha i8x/gC5nUrVYUs6hOdPDTL9nf6kIJU6VcYk1l6lDT4D+fYmAlc/RjfiLOv6Lbn4Qc5ix 72AehN7YLrkzrcdoZHRTEpT+RxysbKRUYxlWnpz0GrQzl8WPj5OUn6ZyuIr66cu77bE9 U74z9VvwrCW2I1jAaAY869FqA4+HHoDayX/3H+iVX/0ZTb6Hj4TiQ6JkdiuX1fC5i1P3 41rsk/YPkHQpsi2FPnvR+M5tehzMzixyFXLPJG6fOGRp/S1x48HLPLhSkHQm4o3yOoM5 DRZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=dlVM2Fz2Y7ISzW+T03M67t0iCDhFZowMvNW5W95Ip88=; b=tey9A75OQzBkkbgqNwZiS+gYGM/WAgF1VN3l4McOditX94ll8JMwlfXSOfn6RHOCoU vMbtJgE7uDrt0ntNqyVnjJosUWkz31oKzOrn9/COzlKXVof8GNwChhv5QcTkn53KBIlm uyYO1zuVD+i+V4zEZcCMz1pRXUEex7dBd1eRlf5e8xL5O5AQJ1LoouM3EtmKIyWW2iA0 4zhn9hsC0ruR010PMrkP5ztA96UXXD6Ca85E4jFyc/vZWayts9G2n8THHTdkkF43abxu 6p5iaVFbvVJIfJZc7PtTEMDuRhOqGCYvxGiMutDfuGDQmt70p4rG+lw8+e2ewMEz/XDh Iufg== X-Gm-Message-State: AOAM5324tXU7ow+ks2HUhxp4VZhOgDvqRcXWGVu9w+MYs11rNTS7dZU9 h+fhuWY9dHYF64SfUgtquw0DRYVv/us= X-Google-Smtp-Source: ABdhPJz3tHc7G5iWdHXhbRBfoyVUj3SKr1PVP3WtCMI3dyoPjw0ifVyDiqazQpsgWctE2YfwQt7/F4KvGHU= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a92:c542:0:b0:2cb:d371:1f3e with SMTP id a2-20020a92c542000000b002cbd3711f3emr288202ilj.323.1650059953946; Fri, 15 Apr 2022 14:59:13 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:50 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-7-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 06/17] KVM: arm64: Implement break-before-make sequence for parallel walks From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The ARM architecture requires that software use the 'break-before-make' sequence whenever memory is being remapped. An additional requirement of parallel page walks is a mechanism to ensure exclusive access to a pte, thereby avoiding two threads changing the pte and invariably stomping on one another. Roll the two concepts together into a new helper to implement the 'break' sequence. Use a special invalid pte value to indicate that the pte is under the exclusive control of a thread. If software walkers are traversing the tables in parallel, use an atomic compare-exchange to break the pte. Retry execution on a failed attempt to break the pte, in the hopes that either the instruction will succeed or the pte lock will be successfully acquired. Avoid unnecessary DSBs and TLBIs by only completing the sequence if the evicted pte was valid. For counted non-table ptes drop the reference immediately. Otherwise, references on tables are dropped in post-order traversal as the walker must recurse on the pruned subtree. All of the new atomics do nothing (for now), as there are a few other bits of the map walker that need to be addressed before actually walking in parallel. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 172 +++++++++++++++++++++++++++++------ 1 file changed, 146 insertions(+), 26 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index bf46d6d24951..059ebb921125 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -49,6 +49,12 @@ #define KVM_INVALID_PTE_OWNER_MASK GENMASK(9, 2) #define KVM_MAX_OWNER_ID 1 +/* + * Used to indicate a pte for which a 'make-before-break' sequence is in + * progress. + */ +#define KVM_INVALID_PTE_LOCKED BIT(10) + struct kvm_pgtable_walk_data { struct kvm_pgtable *pgt; struct kvm_pgtable_walker *walker; @@ -707,6 +713,122 @@ static bool stage2_pte_is_counted(kvm_pte_t pte) return kvm_pte_valid(pte) || kvm_invalid_pte_owner(pte); } +static bool stage2_pte_is_locked(kvm_pte_t pte) +{ + return !kvm_pte_valid(pte) && (pte & KVM_INVALID_PTE_LOCKED); +} + +static inline bool kvm_try_set_pte(kvm_pte_t *ptep, kvm_pte_t old, kvm_pte_t new, bool shared) +{ + if (!shared) { + WRITE_ONCE(*ptep, new); + return true; + } + + return cmpxchg(ptep, old, new) == old; +} + +/** + * stage2_try_break_pte() - Invalidates a pte according to the + * 'break-before-make' sequence. + * + * @ptep: Pointer to the pte to break + * @old: The previously observed value of the pte; used for compare-exchange in + * a parallel walk + * @addr: IPA corresponding to the pte + * @level: Table level of the pte + * @shared: true if the tables are shared by multiple software walkers + * @data: pointer to the map walker data + * + * Returns: true if the pte was successfully broken. + * + * If the removed pt was valid, performs the necessary DSB and TLB flush for + * the old value. Drops references to the page table if a non-table entry was + * removed. Otherwise, the table reference is preserved as the walker must also + * recurse through the child tables. + * + * See ARM DDI0487G.a D5.10.1 "General TLB maintenance requirements" for details + * on the 'break-before-make' sequence. + */ +static bool stage2_try_break_pte(kvm_pte_t *ptep, kvm_pte_t old, u64 addr, u32 level, bool shared, + struct stage2_map_data *data) +{ + /* + * Another thread could have already visited this pte and taken + * ownership. + */ + if (stage2_pte_is_locked(old)) { + /* + * If the table walker has exclusive access to the page tables + * then no other software walkers should have locked the pte. + */ + WARN_ON(!shared); + return false; + } + + if (!kvm_try_set_pte(ptep, old, KVM_INVALID_PTE_LOCKED, shared)) + return false; + + /* + * If we removed a valid pte, break-then-make rules are in effect as a + * translation may have been cached that traversed this entry. + */ + if (kvm_pte_valid(old)) { + dsb(ishst); + + if (kvm_pte_table(old, level)) + /* + * Invalidate the whole stage-2, as we may have numerous leaf + * entries below us which would otherwise need invalidating + * individually. + */ + kvm_call_hyp(__kvm_tlb_flush_vmid, data->mmu); + else + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); + } + + /* + * Don't drop the reference on table entries yet, as the walker must + * first recurse on the unlinked subtree to unlink and drop references + * to child tables. + */ + if (!kvm_pte_table(old, level) && stage2_pte_is_counted(old)) + data->mm_ops->put_page(ptep); + + return true; +} + +/** + * stage2_make_pte() - Installs a new pte according to the 'break-before-make' + * sequence. + * + * @ptep: pointer to the pte to make + * @new: new pte value to install + * + * Assumes that the pte addressed by ptep has already been broken and is under + * the ownership of the table walker. If the new pte to be installed is a valid + * entry, perform a DSB to make the write visible. Raise the reference count on + * the table if the new pte requires a reference. + * + * See ARM DDI0487G.a D5.10.1 "General TLB maintenance requirements" for details + * on the 'break-before-make' sequence. + */ +static void stage2_make_pte(kvm_pte_t *ptep, kvm_pte_t new, struct kvm_pgtable_mm_ops *mm_ops) +{ + /* Yikes! We really shouldn't install to an entry we don't own. */ + WARN_ON(!stage2_pte_is_locked(*ptep)); + + if (stage2_pte_is_counted(new)) + mm_ops->get_page(ptep); + + if (kvm_pte_valid(new)) { + WRITE_ONCE(*ptep, new); + dsb(ishst); + } else { + smp_store_release(ptep, new); + } +} + static void stage2_put_pte(kvm_pte_t *ptep, struct kvm_s2_mmu *mmu, u64 addr, u32 level, struct kvm_pgtable_mm_ops *mm_ops) { @@ -760,18 +882,17 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, else new = kvm_init_invalid_leaf_owner(data->owner_id); - if (stage2_pte_is_counted(old)) { - /* - * Skip updating the PTE if we are trying to recreate the exact - * same mapping or only change the access permissions. Instead, - * the vCPU will exit one more time from guest if still needed - * and then go through the path of relaxing permissions. - */ - if (!stage2_pte_needs_update(old, new)) - return -EAGAIN; + /* + * Skip updating the PTE if we are trying to recreate the exact same + * mapping or only change the access permissions. Instead, the vCPU will + * exit one more time from the guest if still needed and then go through + * the path of relaxing permissions. + */ + if (!stage2_pte_needs_update(old, new)) + return -EAGAIN; - stage2_put_pte(ptep, data->mmu, addr, level, mm_ops); - } + if (!stage2_try_break_pte(ptep, old, addr, level, shared, data)) + return -EAGAIN; /* Perform CMOs before installation of the guest stage-2 PTE */ if (mm_ops->dcache_clean_inval_poc && stage2_pte_cacheable(pgt, new)) @@ -781,9 +902,7 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (mm_ops->icache_inval_pou && stage2_pte_executable(new)) mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule); - smp_store_release(ptep, new); - if (stage2_pte_is_counted(new)) - mm_ops->get_page(ptep); + stage2_make_pte(ptep, new, data->mm_ops); if (kvm_phys_is_valid(phys)) data->phys += granule; return 0; @@ -800,15 +919,10 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, if (!stage2_leaf_mapping_allowed(addr, end, level, data)) return 0; - data->childp = kvm_pte_follow(*old, data->mm_ops); - kvm_clear_pte(ptep); + if (!stage2_try_break_pte(ptep, *old, addr, level, shared, data)) + return -EAGAIN; - /* - * Invalidate the whole stage-2, as we may have numerous leaf - * entries below us which would otherwise need invalidating - * individually. - */ - kvm_call_hyp(__kvm_tlb_flush_vmid, data->mmu); + data->childp = kvm_pte_follow(*old, data->mm_ops); data->anchor = ptep; return 0; } @@ -837,18 +951,24 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, if (!data->memcache) return -ENOMEM; + if (!stage2_try_break_pte(ptep, *old, addr, level, shared, data)) + return -EAGAIN; + childp = mm_ops->zalloc_page(data->memcache); - if (!childp) + if (!childp) { + /* + * Release the pte if we were unable to install a table to allow + * another thread to make an attempt. + */ + stage2_make_pte(ptep, 0, data->mm_ops); return -ENOMEM; + } /* * If we've run into an existing block mapping then replace it with * a table. Accesses beyond 'end' that fall within the new table * will be mapped lazily. */ - if (stage2_pte_is_counted(*old)) - stage2_put_pte(ptep, data->mmu, addr, level, mm_ops); - kvm_set_table_pte(ptep, childp, mm_ops); mm_ops->get_page(ptep); *old = *ptep; From patchwork Fri Apr 15 21:58:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815458 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FB99C43217 for ; Fri, 15 Apr 2022 21:59:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349834AbiDOWBv (ORCPT ); Fri, 15 Apr 2022 18:01:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37542 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1355435AbiDOWBp (ORCPT ); Fri, 15 Apr 2022 18:01:45 -0400 Received: from mail-il1-x14a.google.com (mail-il1-x14a.google.com [IPv6:2607:f8b0:4864:20::14a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D2A738D83 for ; Fri, 15 Apr 2022 14:59:15 -0700 (PDT) Received: by mail-il1-x14a.google.com with SMTP id m11-20020a056e020deb00b002cbde7e7dcfso4186132ilj.2 for ; Fri, 15 Apr 2022 14:59:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=JyIVyQg6UGD612cbxcwyNwFE+I6JblVf7Z7F7spTUWo=; b=LMqmd6P7Ue7Y9IYx3CzgSFOTxT7tT239JnYQyS2EtWQ3fN4qbQ0VflsOtSv+zaS5bk +bvfK/SmAwgiQklAVqKG84UBqOcNnfZBYDShTxLbmp4TCyU+thTgl31lI4p91fxWMOn/ dd2QO+9mQAqSvAt90OOkgKcjFxVpl7VjcraP/FuiseUcuGPj+F+x30DmaWdzLXajn575 2MLUXCslyvGHkc1BlYeAgCkKGGLpODBbWpAJnCW0QHKTJCA67zs859BL+KVaEhsWfoh0 j+lM1SBjp+89lbo+kBMwvVR9gQck0ZWmyt7cTKZrx3+vLXytC9/xjlO4AIrc/U0c/XjQ nkAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=JyIVyQg6UGD612cbxcwyNwFE+I6JblVf7Z7F7spTUWo=; b=CDESC0YGl6rgLDKQpiQtyD1OfIOdlk1nvvj5XcyfYCaIoc9KrmklFBwDrsrT1K7MAI Z+GWSPsvBxGfcChzto3tHVr/k7S8+RpvuevipjeW49JV3zWFei3L7uO/kCRf7QQB5kZJ Xp+gk1FoMmPuIIV6MM6XjDg2Wczw7dm9Qv6/jgDkaUacV9lnj7KGeyIQolFQURv8AVUD hKUqEcn5WPlYZDOByigdlJF5CxjTH8fQRXgCznhrq1ewPQKdbwNNAva13jLZcpD1yUcz 7fDoEUUVRw/rqch78nT/v29CFThAfnnYrIKlR/t/ZnFzufKm5CXUH5npEyTbKpudCV56 rBpg== X-Gm-Message-State: AOAM5338tvMCKv8cIouuCCdoY/0vCIcyRlYiYURa6NijNDXSB7fJPvjK KQJ+cPNELE+eRf6InNmD43zex2lWTdI= X-Google-Smtp-Source: ABdhPJwW/MzIxEmQ87y7GaINMs+YHxfeosd37KaEM3yVPDMpYVf1oEXa8efMhhfJEE88ApB9wQ629pQLxlQ= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6602:27cc:b0:5f0:876e:126b with SMTP id l12-20020a05660227cc00b005f0876e126bmr328774ios.129.1650059954862; Fri, 15 Apr 2022 14:59:14 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:51 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-8-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 07/17] KVM: arm64: Enlighten perm relax path about parallel walks From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org To date the permission relax path of the stage-2 fault handler hasn't had to worry about the paging structures changing under its nose, as map operations acquire the write lock. That's about to change, which means a permission relaxation walker could traverse in parallel with a map operation. If at any point during traversal the permission relax walker finds a locked pte, bail immediately. Either the instruction will succeed or the vCPU will fault once more and (hopefully) walk the tables successfully. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 059ebb921125..ff6f14755d0c 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1168,6 +1168,11 @@ static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_attr_data *data = arg; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; + if (stage2_pte_is_locked(pte)) { + WARN_ON(!shared); + return -EAGAIN; + } + if (!kvm_pte_valid(pte)) return 0; @@ -1190,7 +1195,9 @@ static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, stage2_pte_executable(pte) && !stage2_pte_executable(*ptep)) mm_ops->icache_inval_pou(kvm_pte_follow(pte, mm_ops), kvm_granule_size(level)); - WRITE_ONCE(*ptep, pte); + + if (!kvm_try_set_pte(ptep, data->pte, pte, shared)) + return -EAGAIN; } return 0; @@ -1199,7 +1206,7 @@ static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr, u64 size, kvm_pte_t attr_set, kvm_pte_t attr_clr, kvm_pte_t *orig_pte, - u32 *level) + u32 *level, bool shared) { int ret; kvm_pte_t attr_mask = KVM_PTE_LEAF_ATTR_LO | KVM_PTE_LEAF_ATTR_HI; @@ -1214,7 +1221,7 @@ static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr, .flags = KVM_PGTABLE_WALK_LEAF, }; - ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, shared); if (ret) return ret; @@ -1230,14 +1237,14 @@ int kvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size) { return stage2_update_leaf_attrs(pgt, addr, size, 0, KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W, - NULL, NULL); + NULL, NULL, false); } kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr) { kvm_pte_t pte = 0; stage2_update_leaf_attrs(pgt, addr, 1, KVM_PTE_LEAF_ATTR_LO_S2_AF, 0, - &pte, NULL); + &pte, NULL, false); dsb(ishst); return pte; } @@ -1246,7 +1253,7 @@ kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr) { kvm_pte_t pte = 0; stage2_update_leaf_attrs(pgt, addr, 1, 0, KVM_PTE_LEAF_ATTR_LO_S2_AF, - &pte, NULL); + &pte, NULL, false); /* * "But where's the TLBI?!", you scream. * "Over in the core code", I sigh. @@ -1259,7 +1266,7 @@ kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr) bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr) { kvm_pte_t pte = 0; - stage2_update_leaf_attrs(pgt, addr, 1, 0, 0, &pte, NULL); + stage2_update_leaf_attrs(pgt, addr, 1, 0, 0, &pte, NULL, false); return pte & KVM_PTE_LEAF_ATTR_LO_S2_AF; } @@ -1282,7 +1289,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, if (prot & KVM_PGTABLE_PROT_X) clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN; - ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level); + ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level, true); if (!ret) kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, level); return ret; From patchwork Fri Apr 15 21:58:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815460 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E5ACC4332F for ; Fri, 15 Apr 2022 21:59:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356188AbiDOWBx (ORCPT ); Fri, 15 Apr 2022 18:01:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37558 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1355788AbiDOWBp (ORCPT ); Fri, 15 Apr 2022 18:01:45 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 936BC396BF for ; Fri, 15 Apr 2022 14:59:16 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id i5-20020a258b05000000b006347131d40bso7580147ybl.17 for ; Fri, 15 Apr 2022 14:59:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=B6406tYui+5lJgRvZIHJ2dZYS6AUdUd0eyrsU1t7VRI=; b=RzttCkqDWkMAl2afGr2S9DJJm5Ry89zm9fK8vSWXxjh+Br5xy5GQOart2FDuwkrzu8 +3Iq4y2goOAEc/FMm6DlVztE2ytFr3oiz9PTrVTxIm5m3Agm5OTfDy0Zs4ZMjS6SjcTP Zb1tTD9KujpWLSKkx4Mqs5RJ6LI4Z/u/gq513Flk/dllCKb2U4eZ8URwcntAGL1GuOmq ufWltbnns/Dj/9u47iW81VO53GtXf0bBfExXn11hVKS6QGf6GMMShHapP0dxeuiycC4J XphrZoAoqeoctTsXEHQSr5ZLlJzNIHeMt45VK8ODHWJivQe27pVAh1K+W85kwYhJyLS2 rKpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=B6406tYui+5lJgRvZIHJ2dZYS6AUdUd0eyrsU1t7VRI=; b=UPa/hTV0Ihms7NJLIBHvg/RtWlxpY4RS+6cD9eamNB6B8BhwpZsxRRzFYKi6lW+7+U YTmdaNXdKefW5CARI/73PC54Tw8MKaGIVAkx70jLeoL+JTu8F+HW1lg2N6hZF/vhELoo iA7k0igPvoFuEdXLQzUCsvfzn544J2aLChR61mNAirxzWlBKC8wGs6NzUMNNlwqFT18C b9tRkwzjf0hnCrAixmyGuCZf0T9JXn7RI6tGxPG4WbIHkss6zqh7/Bxaqn1XAuYYVorB RT7WOfuliTz8SZBvTgwqmTidG2UidtJkPO1zhEMPGafEhQLdQfNIOgXgyW99X+m/HSgf XxFQ== X-Gm-Message-State: AOAM532D7/sCqsYh8cErC0akcgNn4qltz9YWKrGa1tWVBG19AgFeQGyc vFXscqIb0FuWWiGTQAk+Spejf6O5vxI= X-Google-Smtp-Source: ABdhPJyC6al0eeORHQcVaP84DLwWwiUHL9eCDBoXdDwFa7r/iAh2U4p6UOyrCV4QCls9mT6W0yMv3xIYId8= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a81:4989:0:b0:2f1:47b9:8ac2 with SMTP id w131-20020a814989000000b002f147b98ac2mr935486ywa.346.1650059955713; Fri, 15 Apr 2022 14:59:15 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:52 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-9-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 08/17] KVM: arm64: Spin off helper for initializing table pte From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org With parallel table walks there is no guarantee that KVM reads back the same pte that was written. Spin off a helper that creates a pte value, thereby allowing the visitor callback to return the next table without reading the ptep again. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index ff6f14755d0c..ffdfd5ee9642 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -167,14 +167,23 @@ static void kvm_clear_pte(kvm_pte_t *ptep) WRITE_ONCE(*ptep, 0); } -static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp, - struct kvm_pgtable_mm_ops *mm_ops) +static kvm_pte_t kvm_init_table_pte(kvm_pte_t *childp, struct kvm_pgtable_mm_ops *mm_ops) { - kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(mm_ops->virt_to_phys(childp)); + kvm_pte_t pte = kvm_phys_to_pte(mm_ops->virt_to_phys(childp)); pte |= FIELD_PREP(KVM_PTE_TYPE, KVM_PTE_TYPE_TABLE); pte |= KVM_PTE_VALID; + return pte; +} + +static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp, + struct kvm_pgtable_mm_ops *mm_ops) +{ + kvm_pte_t pte, old = *ptep; + + pte = kvm_init_table_pte(childp, mm_ops); + WARN_ON(kvm_pte_valid(old)); smp_store_release(ptep, pte); } @@ -931,7 +940,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, struct stage2_map_data *data, bool shared) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; - kvm_pte_t *childp; + kvm_pte_t *childp, pte; int ret; if (data->anchor) { @@ -969,9 +978,9 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, * a table. Accesses beyond 'end' that fall within the new table * will be mapped lazily. */ - kvm_set_table_pte(ptep, childp, mm_ops); - mm_ops->get_page(ptep); - *old = *ptep; + pte = kvm_init_table_pte(childp, mm_ops); + stage2_make_pte(ptep, pte, data->mm_ops); + *old = pte; return 0; } From patchwork Fri Apr 15 21:58:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F032C43217 for ; Fri, 15 Apr 2022 21:59:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356100AbiDOWBy (ORCPT ); Fri, 15 Apr 2022 18:01:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37604 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356124AbiDOWBr (ORCPT ); Fri, 15 Apr 2022 18:01:47 -0400 Received: from mail-io1-xd49.google.com (mail-io1-xd49.google.com [IPv6:2607:f8b0:4864:20::d49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80EA8396BF for ; Fri, 15 Apr 2022 14:59:17 -0700 (PDT) Received: by mail-io1-xd49.google.com with SMTP id v9-20020a5ed709000000b006530841a32cso1129582iom.21 for ; Fri, 15 Apr 2022 14:59:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0g9zJiooWDeKugRAQw4FN5FN0/UzyXcPgqHvAqWzKag=; b=oLCH+GSdJvXkd0Gi9+H7vA79g6hjv3mWJ8f1Ky8MSz9VGrXihtRm2Q2mQk94XB6R1V rI0BdGd8LpAzMP3bNbGXHpcJwONP3gy5q638j75OkVYuPzBzMikI3B/GpnP4ZHi6WYgC 1NnjFM6MAYPTaJtge/P7t3M0AOj4J/HW1tzhhrT32UjnHREO3sHO9Q6aRYB6eSSaw14y 1XJYEXTFyIROlajcWQONhn8qbXtAQymvl+UmDZ5XpOcMbLVur2O69+FsbWoSTYPTr05p 24KmSiJ75xiAF16xI5zuCM76equOlcpV45VjmXOI3Dw1o6nycZZCyMKAyu7F35txmRD/ JODA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0g9zJiooWDeKugRAQw4FN5FN0/UzyXcPgqHvAqWzKag=; b=BdPO0T38oKl73SkxrY8/XWoxFpiGBy/ZQK0f4dbx3P3iTf5WDcnh5wjcT6fvPnnT6i edOGbuG8Q4kPblQth9qI4LWi8FuNp8Snb4ESTaS6n1R/WxZzd0cL/ORd0I52qYgm1Ol/ O7y95edtXMjS+A9+WMeOZtgcs/U62F/13RzeoacbeY7J8hmVUWyUQUTrbBl4AlsDDTdQ 2JEEjZr5d2Q4iDMqF3lFd5CApIDVoHDQY1oXth1fvLRQV6kE87dqZInPhSvN6tVifQzk tzZRX+fliwOgYuoez85dCZSEZE3xkgGZ9Z1i58pzdG3bWByDKy68KZjT7mwgGHJGtH4M SPgw== X-Gm-Message-State: AOAM532HPWjVtxayUMxalu21BQdmzpc7gxZwkK5XPrpzittksQ6/GyXJ YAYq9D384GtXGLT0M0ctYagyywqg/4Q= X-Google-Smtp-Source: ABdhPJy38Wy2JIODUbPjdCPdcAqin2kc7u187gx+PAUVY0P1F7nHnp3qVfiaCtdGLADk1YLKoWq+Y853FYQ= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6638:218a:b0:326:6ed7:c011 with SMTP id s10-20020a056638218a00b003266ed7c011mr473714jaj.242.1650059956892; Fri, 15 Apr 2022 14:59:16 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:53 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-10-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 09/17] KVM: arm64: Tear down unlinked page tables in parallel walk From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Breaking a table pte is insufficient to guarantee ownership of an unlinked subtree. Parallel software walkers could be traversing substructures and changing their mappings. Recurse through the unlinked subtree and lock all descendent ptes to take ownership of the subtree. Since the ptes are actually being evicted, return table ptes back to the table walker to ensure child tables are also traversed. Note that this is done both in both the pre-order and leaf visitors as the underlying pte remains volatile until it is unlinked. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 56 +++++++++++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index ffdfd5ee9642..146fc44acf31 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -838,6 +838,54 @@ static void stage2_make_pte(kvm_pte_t *ptep, kvm_pte_t new, struct kvm_pgtable_m } } +static kvm_pte_t stage2_unlink_pte_shared(kvm_pte_t *ptep) +{ + kvm_pte_t old; + + while (true) { + old = xchg(ptep, KVM_INVALID_PTE_LOCKED); + if (old != KVM_INVALID_PTE_LOCKED) + return old; + + cpu_relax(); + } +} + + +/** + * stage2_unlink_pte() - Tears down an unreachable pte, returning the next pte + * to visit (if any). + * + * @ptep: pointer to the pte to unlink + * @level: page table level of the pte + * @shared: true if the tables are shared by multiple software walkers + * @mm_ops: pointer to the mm ops table + * + * Return: a table pte if another level of recursion is necessary, 0 otherwise. + */ +static kvm_pte_t stage2_unlink_pte(kvm_pte_t *ptep, u32 level, bool shared, + struct kvm_pgtable_mm_ops *mm_ops) +{ + kvm_pte_t old; + + if (shared) { + old = stage2_unlink_pte_shared(ptep); + } else { + old = *ptep; + WRITE_ONCE(*ptep, KVM_INVALID_PTE_LOCKED); + } + + WARN_ON(stage2_pte_is_locked(old)); + + if (kvm_pte_table(old, level)) + return old; + + if (stage2_pte_is_counted(old)) + mm_ops->put_page(ptep); + + return 0; +} + static void stage2_put_pte(kvm_pte_t *ptep, struct kvm_s2_mmu *mmu, u64 addr, u32 level, struct kvm_pgtable_mm_ops *mm_ops) { @@ -922,8 +970,10 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, struct stage2_map_data *data, bool shared) { - if (data->anchor) + if (data->anchor) { + *old = stage2_unlink_pte(ptep, level, shared, data->mm_ops); return 0; + } if (!stage2_leaf_mapping_allowed(addr, end, level, data)) return 0; @@ -944,9 +994,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, int ret; if (data->anchor) { - if (stage2_pte_is_counted(*old)) - mm_ops->put_page(ptep); - + *old = stage2_unlink_pte(ptep, level, shared, data->mm_ops); return 0; } From patchwork Fri Apr 15 21:58:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815462 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCDE8C433F5 for ; Fri, 15 Apr 2022 21:59:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356124AbiDOWBz (ORCPT ); Fri, 15 Apr 2022 18:01:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356170AbiDOWBr (ORCPT ); Fri, 15 Apr 2022 18:01:47 -0400 Received: from mail-il1-x149.google.com (mail-il1-x149.google.com [IPv6:2607:f8b0:4864:20::149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80211381B5 for ; Fri, 15 Apr 2022 14:59:18 -0700 (PDT) Received: by mail-il1-x149.google.com with SMTP id y19-20020a056e02119300b002c2d3ef05bfso5401442ili.18 for ; Fri, 15 Apr 2022 14:59:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=vop5IqucqvM+bii5zgbcIy+JPyxF0keenz7pMhv8m58=; b=jHJtFLgbxSnb6N3tLR+HWD/NrSJB2+DQfaJR6BV6mi7BHclXm3b5DYu3lRIqkK8ZCZ 9Nsc5qvQWjHWPk7VT1sJ3wsYP6XTOuBtzzol+Dyx7Y/G8XVKzTeOZ0wTCYISer2gacY1 1q7hF+P4I9XhHOxrOx0DwqCHp3nlNR7k1lS2WUuwKr6yrpl/wcv2QIQYdgxci9ERkbhN GjVhDkCkSw4jTSLDIZKUru4yoXAgzeVmoDz3uo26yb6h1LXdBYsBdQUdxT5prqHk01wm t/IKRZMZolVSXf5jGA0B71c/6Sed7CUrur83859QSKnETLM/2FiH2WcM4zCc5JAIz871 u5gA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vop5IqucqvM+bii5zgbcIy+JPyxF0keenz7pMhv8m58=; b=uURHcPAyxJBkguNrNYSEAL3ovr75IsrHS8IUscCM6Q/fXgZSAvMRVFV6DbJ7x7hsJ9 pvEmz/ILJnynQ1A6yFHgDIeP9ASiZBSrIF5+nLG1gu1JUVxF+edtg+KPd5cXPc2XPo61 LHyNYGPFqgK/udynEKSG9IT0id0Y54rhCDc1hDS0VYxSUloTRcQ1R8jZpFvue+YzR1JP 7dvC/vIBLZWB0fQH2PlVjrMDT+Rj/5RkbuaiMHBlcYOd+nTqaJl9RNHtsSmDTEfcfHDt 6oP+p/5dTqKzcotACeBlieMSrOsWblipE+mkwLu7x1avPn8T+f6ZNs28LhJAeO7jb/vt JUPQ== X-Gm-Message-State: AOAM533v/GuDAWP1cgILjZSvn33OftzuTCjMEcHqnqXahTIyt9UjFgka jo72ekshQei8ZLU/nI11iLMFPoRKE14= X-Google-Smtp-Source: ABdhPJxNjcu2tnVJ1+/YbC82bpFAmroc7SBNGMAr2Bd/O0LGqLCc49cJVFBPbSuomBfNsod9gY/vH0O4fqo= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6638:164b:b0:323:ac42:8d4b with SMTP id a11-20020a056638164b00b00323ac428d4bmr475237jat.75.1650059957848; Fri, 15 Apr 2022 14:59:17 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:54 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-11-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 10/17] KVM: arm64: Assume a table pte is already owned in post-order traversal From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org For parallel walks that collapse a table into a block KVM ensures a locked invalid pte is visible to all observers in pre-order traversal. As such, there is no need to try breaking the pte again. Directly set the pte if it has already been broken. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 146fc44acf31..121818d4c33e 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -924,7 +924,7 @@ static bool stage2_leaf_mapping_allowed(u64 addr, u64 end, u32 level, static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t old, struct stage2_map_data *data, - bool shared) + bool shared, bool locked) { kvm_pte_t new; u64 granule = kvm_granule_size(level), phys = data->phys; @@ -948,7 +948,7 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (!stage2_pte_needs_update(old, new)) return -EAGAIN; - if (!stage2_try_break_pte(ptep, old, addr, level, shared, data)) + if (!locked && !stage2_try_break_pte(ptep, old, addr, level, shared, data)) return -EAGAIN; /* Perform CMOs before installation of the guest stage-2 PTE */ @@ -987,7 +987,8 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, } static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - kvm_pte_t *old, struct stage2_map_data *data, bool shared) + kvm_pte_t *old, struct stage2_map_data *data, bool shared, + bool locked) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; kvm_pte_t *childp, pte; @@ -998,10 +999,13 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, return 0; } - ret = stage2_map_walker_try_leaf(addr, end, level, ptep, *old, data, shared); + ret = stage2_map_walker_try_leaf(addr, end, level, ptep, *old, data, shared, locked); if (ret != -E2BIG) return ret; + /* We should never attempt installing a table in post-order */ + WARN_ON(locked); + if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1)) return -EINVAL; @@ -1048,7 +1052,13 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, childp = data->childp; data->anchor = NULL; data->childp = NULL; - ret = stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared); + + /* + * We are guaranteed exclusive access to the pte in post-order + * traversal since the locked value was made visible to all + * observers in stage2_map_walk_table_pre. + */ + ret = stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared, true); } else { childp = kvm_pte_follow(*old, mm_ops); } @@ -1087,7 +1097,7 @@ static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_ case KVM_PGTABLE_WALK_TABLE_PRE: return stage2_map_walk_table_pre(addr, end, level, ptep, old, data, shared); case KVM_PGTABLE_WALK_LEAF: - return stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared); + return stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared, false); case KVM_PGTABLE_WALK_TABLE_POST: return stage2_map_walk_table_post(addr, end, level, ptep, old, data, shared); } From patchwork Fri Apr 15 21:58:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 133D5C4332F for ; Fri, 15 Apr 2022 21:59:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356179AbiDOWBz (ORCPT ); Fri, 15 Apr 2022 18:01:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356171AbiDOWBs (ORCPT ); Fri, 15 Apr 2022 18:01:48 -0400 Received: from mail-il1-x149.google.com (mail-il1-x149.google.com [IPv6:2607:f8b0:4864:20::149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89D5A3BBE0 for ; Fri, 15 Apr 2022 14:59:19 -0700 (PDT) Received: by mail-il1-x149.google.com with SMTP id i14-20020a056e020ece00b002ca198245e6so5445965ilk.4 for ; Fri, 15 Apr 2022 14:59:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ukSikfLZPySq7lKrWdQC3W4tZRZDdV+TXwwmzbO5VKE=; b=UFQNViFm5ywOL6fUUgUmBF2pMIllmvYSsEmxRVNh1gzD1izmJ3LtYUTbQoVRVd6R7/ jqzms2/UXzy9A1cKXaPCa6uGhs2jB4kYpL25R2bC00CDonStKGhvdJ4YVdoFWKmexTi/ QkfTtuq7/OrDuIAAqPw+yUALYUDmuCCrGRxB9F24yb275VyW2/5r9d7nu5//Jrfs1gyU bBbceIMSX7GVKPd/CqxGaga7ODuF8yOc8fr/uf+kbuofz35ev3lh9BLs/jGFbUXJ/kkj 2mjW+x4bwKEomG8LlUwv7ZO+oESMs9ddaaWpZvG/caTzJIQruvv70SZXMOewJ76h8qCh w+3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ukSikfLZPySq7lKrWdQC3W4tZRZDdV+TXwwmzbO5VKE=; b=RK8Yx1vQEoMYH7sBZnAKkHLXxQAPJj9yt6EuHP0xN/6uGXUFZQX9FRzQbJUu/f3lP4 Utl1E+cxXKuVz2cstHnf2oW1QY1+wDbzm2I+e+x+vuo8EOe8BOgVRZhDvffhAShmlhbw a+Yu9LVFZLSUnIZME2XrfmzAkl0T2q3LQnVAwILjLJLXZw2cwF1DeWUEk6VhQ+3bM5D8 mMmYTR5eiT60KpM3SMt/pIETXPXToyfE5HS8rlW/C5WZ6vIszIelx51dEAhs2p4jjjel 39VQrPUzWBPVyB+lsvBiivkvovO9EUN9vev/SBNNRLZuJJaiRFsK8Z3ZPVm54GVer0Ml Sgog== X-Gm-Message-State: AOAM53299hyYSQROZDJkjUSZM9thqk+CSYAcNRgL1GwQJmB3SYGH1ZRC vANKYDXw2jBME86YAVCs3gaE2PZleJw= X-Google-Smtp-Source: ABdhPJw3+rg28SXpnYzElP/Fh1mOEN5FS2KtRHWJcXruhJlFqtNNF+kNgwwNT/3GJfepUrZ+VWNiihmiJLs= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6e02:1a4f:b0:2c7:a4c8:25f5 with SMTP id u15-20020a056e021a4f00b002c7a4c825f5mr334502ilv.64.1650059958943; Fri, 15 Apr 2022 14:59:18 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:55 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-12-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 11/17] KVM: arm64: Move MMU cache init/destroy into helpers From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org No functional change intended. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_mmu.h | 2 ++ arch/arm64/kvm/arm.c | 4 ++-- arch/arm64/kvm/mmu.c | 10 ++++++++++ 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 74735a864eee..3bb7b678a7e7 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -172,6 +172,8 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu); phys_addr_t kvm_mmu_get_httbr(void); phys_addr_t kvm_get_idmap_vector(void); int kvm_mmu_init(u32 *hyp_va_bits); +void kvm_mmu_vcpu_init(struct kvm_vcpu *vcpu); +void kvm_mmu_vcpu_destroy(struct kvm_vcpu *vcpu); static inline void *__kvm_vector_slot2addr(void *base, enum arm64_hyp_spectre_vector slot) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 523bc934fe2f..f7862fec1595 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -320,7 +320,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.target = -1; bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES); - vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO; + kvm_mmu_vcpu_init(vcpu); /* Set up the timer */ kvm_timer_vcpu_init(vcpu); @@ -349,7 +349,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) if (vcpu_has_run_once(vcpu) && unlikely(!irqchip_in_kernel(vcpu->kvm))) static_branch_dec(&userspace_irqchip_in_use); - kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache); + kvm_mmu_vcpu_destroy(vcpu); kvm_timer_vcpu_terminate(vcpu); kvm_pmu_vcpu_destroy(vcpu); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 53ae2c0640bc..f29d5179196b 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1601,6 +1601,16 @@ int kvm_mmu_init(u32 *hyp_va_bits) return err; } +void kvm_mmu_vcpu_init(struct kvm_vcpu *vcpu) +{ + vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO; +} + +void kvm_mmu_vcpu_destroy(struct kvm_vcpu *vcpu) +{ + kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache); +} + void kvm_arch_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *old, const struct kvm_memory_slot *new, From patchwork Fri Apr 15 21:58:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815464 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CDEAC433FE for ; Fri, 15 Apr 2022 21:59:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232321AbiDOWB5 (ORCPT ); Fri, 15 Apr 2022 18:01:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356162AbiDOWBt (ORCPT ); Fri, 15 Apr 2022 18:01:49 -0400 Received: from mail-oo1-xc4a.google.com (mail-oo1-xc4a.google.com [IPv6:2607:f8b0:4864:20::c4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7479C60F2 for ; Fri, 15 Apr 2022 14:59:20 -0700 (PDT) Received: by mail-oo1-xc4a.google.com with SMTP id l2-20020a4ab0c2000000b00334cb56f0a5so2342553oon.17 for ; Fri, 15 Apr 2022 14:59:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=22i24DLcyXUieSnW3PAwQCkjoUnvyMWdp2sKJnAUF64=; b=VPtHgLB2eymVJQjQSElXFQ0Zv2gLUgxLry175CS0FCPH1QdUk/jXMMBoPOrHmGBwoq eWdmwHWefSkyKrTkoNNUmWeZAwfQOesyHQ/c4F+nNUh1RnLeH6LvowZHB8nX3nwQ6l3L GEUKEunrQWJsjT9PH9meExTOYNHH3hU7S6IfE1PLsiTpOqRRkTbel/mLpuqamcBbNfG9 GoFAe3BQFqD1GHm76/Rdvr36nNdmSstr6rfNRpwHof4KzfU+AjgBiLREY1NzMY207hNQ sxhnRfFngtq2kuNUX0v2DKU0e2znK8YIUUSoecH7YVoHWITRrIMv7LGu6/DrG+fBMRUD Iqsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=22i24DLcyXUieSnW3PAwQCkjoUnvyMWdp2sKJnAUF64=; b=egHPP79YGki9fwyBuRAT/he9wqjgQmsjyI/NQ0eYksI4c+uI3JuC9gubKPdv7xxV+k fqf/3HSG2CZ6gqHZN0LZTBQKJYmWvqbPxrfifti14/tHlzhT4xCkz/MFCAf6hj1ho9jN giYnWPx8h7hHK4O89OnDh7zkT91eSBm74HbyqEubseLpUthY5gmLzqDyNM/2P7Nig5lT 2v3o4Ue5ys7hDMEOGIBSjnZPwlQMTPAa3iWAhCsCgf7GvWmiWL/J7mP5eR5EQ8AZzOzw 417CRCJjJAEZApl6JYPUq2+dC0+tsjbwtIUPYBG+Ia3p/rqAQ3Ix3vvUbS0zm4rCdSrU 9/3A== X-Gm-Message-State: AOAM5330xfzgCpf/Kl1GB74fZNfo3zJ+Yb3/F75Pl1xRXXnsoasGLwDB U/rm60vXcrOvbfykGmy4m2vwDNLcRec= X-Google-Smtp-Source: ABdhPJya0hEPL3+v3IGSw82KDihmFFDLjFiBSt27i0G11hzjmjLYmOhkEY2+c79aK0bz4MZlGmwNEBfJ24k= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6870:639e:b0:e2:ab7c:d868 with SMTP id t30-20020a056870639e00b000e2ab7cd868mr373366oap.108.1650059959809; Fri, 15 Apr 2022 14:59:19 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:56 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-13-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 12/17] KVM: arm64: Stuff mmu page cache in sub struct From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org We're about to add another mmu cache. Stuff the current one in a sub struct so its easier to pass them all to ->zalloc_page(). No functional change intended. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_host.h | 4 +++- arch/arm64/kvm/mmu.c | 14 +++++++------- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 94a27a7520f4..c8947597a619 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -372,7 +372,9 @@ struct kvm_vcpu_arch { bool pause; /* Cache some mmu pages needed inside spinlock regions */ - struct kvm_mmu_memory_cache mmu_page_cache; + struct kvm_mmu_caches { + struct kvm_mmu_memory_cache page_cache; + } mmu_caches; /* Target CPU and feature flags */ int target; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index f29d5179196b..7a588928740a 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -91,10 +91,10 @@ static bool kvm_is_device_pfn(unsigned long pfn) static void *stage2_memcache_zalloc_page(void *arg) { - struct kvm_mmu_memory_cache *mc = arg; + struct kvm_mmu_caches *mmu_caches = arg; /* Allocated with __GFP_ZERO, so no need to zero */ - return kvm_mmu_memory_cache_alloc(mc); + return kvm_mmu_memory_cache_alloc(&mmu_caches->page_cache); } static void *kvm_host_zalloc_pages_exact(size_t size) @@ -1073,7 +1073,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, bool shared; unsigned long mmu_seq; struct kvm *kvm = vcpu->kvm; - struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache; + struct kvm_mmu_caches *mmu_caches = &vcpu->arch.mmu_caches; struct vm_area_struct *vma; short vma_shift; gfn_t gfn; @@ -1160,7 +1160,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * and a write fault needs to collapse a block entry into a table. */ if (fault_status != FSC_PERM || (logging_active && write_fault)) { - ret = kvm_mmu_topup_memory_cache(memcache, + ret = kvm_mmu_topup_memory_cache(&mmu_caches->page_cache, kvm_mmu_cache_min_pages(kvm)); if (ret) return ret; @@ -1273,7 +1273,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize, __pfn_to_phys(pfn), prot, - memcache); + mmu_caches); } /* Mark the page dirty only if the fault is handled successfully */ @@ -1603,12 +1603,12 @@ int kvm_mmu_init(u32 *hyp_va_bits) void kvm_mmu_vcpu_init(struct kvm_vcpu *vcpu) { - vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO; + vcpu->arch.mmu_caches.page_cache.gfp_zero = __GFP_ZERO; } void kvm_mmu_vcpu_destroy(struct kvm_vcpu *vcpu) { - kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache); + kvm_mmu_free_memory_cache(&vcpu->arch.mmu_caches.page_cache); } void kvm_arch_commit_memory_region(struct kvm *kvm, From patchwork Fri Apr 15 21:58:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46744C433EF for ; Fri, 15 Apr 2022 21:59:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356162AbiDOWB5 (ORCPT ); Fri, 15 Apr 2022 18:01:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37674 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356169AbiDOWBu (ORCPT ); Fri, 15 Apr 2022 18:01:50 -0400 Received: from mail-io1-xd49.google.com (mail-io1-xd49.google.com [IPv6:2607:f8b0:4864:20::d49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59DE7E80 for ; Fri, 15 Apr 2022 14:59:21 -0700 (PDT) Received: by mail-io1-xd49.google.com with SMTP id i19-20020a5d9353000000b006495ab76af6so5477936ioo.0 for ; Fri, 15 Apr 2022 14:59:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+7THeEZjrQkI4cL8qs69/ZVQNerdV4AViBcRVZLLx0E=; b=sgJWtGYVInFd1drSIPRRI4aezVykNPn0h3TZjKKiErpQM7sAIHOOKNGcQ0jWe9x2J3 4uorM5vPe/np9qjkIYA4dfw4gtmkWWOabaU8tGatq8w8LsDoUg9+FMK0LVF1fX+QFyw6 i9msKz6hKpt7r66C+xns81BxvuHbanyJZ610bbwgkWkEmS6Xl9AcIZ1z+pnbftljqbuV TZJnMR1dtR7pvNgtOgZtS1KwcCNK2XXzdflPkGu6sg9JhoQgHweTSx9tVIIm2620GETp v/lovjRa15qTioUPiP/HsnkVKkyBFJUAm8nxrDqS7jgG1Smlaj2NKciPmkptKeLE+kdI nEKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+7THeEZjrQkI4cL8qs69/ZVQNerdV4AViBcRVZLLx0E=; b=IMSASfOS5CzjGEJpc9TUTHICeMW5ihX1mPOTL9qfVnyFyTX1PDxAtRCIIIyWSX7ZDE L2jFvj7ZBVaNOXPWV4hy8WnrrNoKP8TmCQTJZeFXqJZVEsUguuwmpIGRIxOozgnWrucU hFB7GcVF3FBjV/6+4rXYssWRMxdTpbDCd5S++T06Bi9FSbJgAtGOoP6aN6LOSBFQXl/W SpyVBcUh+AkKzlQeWhenrfK5PzWD9zNWvDb8RqsDwdjnqDcHo07Hw0RSoT279TmMYlyd uin3HKAbiFqmIEsE2OpSK+DQtHvQ0n2nqxcwA8yR9PVzYhn57UGGnKzseLMYc7dFGPzK DL9g== X-Gm-Message-State: AOAM533Ez0YtCbUzsDundfxzDHKcBB1pb6h+BG3ijB8RG3ZQ545s61Ev y57q+wR+eOVQ2jIggvmXkBEEic4YCdk= X-Google-Smtp-Source: ABdhPJyV71KyX7wIk4SdepWHYsPDO7JEsMOtpP8Ew2zkdCBFEtHJx6sDDgdROjEilaiKBOX2bMTgAgDq5z4= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a5e:8a07:0:b0:64c:8b33:6d19 with SMTP id d7-20020a5e8a07000000b0064c8b336d19mr315910iok.170.1650059960675; Fri, 15 Apr 2022 14:59:20 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:57 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-14-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 13/17] KVM: arm64: Setup cache for stage2 page headers From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In order to punt the last reference drop on a page to an RCU synchronization we need to get a pointer to the page to handle the callback. Set up a memcache for stage2 page headers, but do nothing with it for now. Note that the kmem_cache is never destoyed as it is currently not possible to build KVM/arm64 as a module. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/mmu.c | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index c8947597a619..a640d015790e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -374,6 +374,7 @@ struct kvm_vcpu_arch { /* Cache some mmu pages needed inside spinlock regions */ struct kvm_mmu_caches { struct kvm_mmu_memory_cache page_cache; + struct kvm_mmu_memory_cache header_cache; } mmu_caches; /* Target CPU and feature flags */ diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 7a588928740a..cc6ed6b06ec2 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -31,6 +31,12 @@ static phys_addr_t hyp_idmap_vector; static unsigned long io_map_base; +static struct kmem_cache *stage2_page_header_cache; + +struct stage2_page_header { + struct rcu_head rcu_head; + struct page *page; +}; /* * Release kvm_mmu_lock periodically if the memory region is large. Otherwise, @@ -1164,6 +1170,11 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, kvm_mmu_cache_min_pages(kvm)); if (ret) return ret; + + ret = kvm_mmu_topup_memory_cache(&mmu_caches->header_cache, + kvm_mmu_cache_min_pages(kvm)); + if (ret) + return ret; } mmu_seq = vcpu->kvm->mmu_notifier_seq; @@ -1589,6 +1600,13 @@ int kvm_mmu_init(u32 *hyp_va_bits) if (err) goto out_destroy_pgtable; + stage2_page_header_cache = kmem_cache_create("stage2_page_header", + sizeof(struct stage2_page_header), + 0, SLAB_ACCOUNT, NULL); + + if (!stage2_page_header_cache) + goto out_destroy_pgtable; + io_map_base = hyp_idmap_start; return 0; @@ -1604,11 +1622,13 @@ int kvm_mmu_init(u32 *hyp_va_bits) void kvm_mmu_vcpu_init(struct kvm_vcpu *vcpu) { vcpu->arch.mmu_caches.page_cache.gfp_zero = __GFP_ZERO; + vcpu->arch.mmu_caches.header_cache.kmem_cache = stage2_page_header_cache; } void kvm_mmu_vcpu_destroy(struct kvm_vcpu *vcpu) { kvm_mmu_free_memory_cache(&vcpu->arch.mmu_caches.page_cache); + kvm_mmu_free_memory_cache(&vcpu->arch.mmu_caches.header_cache); } void kvm_arch_commit_memory_region(struct kvm *kvm, From patchwork Fri Apr 15 21:58:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815466 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C75F6C433F5 for ; Fri, 15 Apr 2022 21:59:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356230AbiDOWB6 (ORCPT ); Fri, 15 Apr 2022 18:01:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356173AbiDOWBw (ORCPT ); Fri, 15 Apr 2022 18:01:52 -0400 Received: from mail-io1-xd49.google.com (mail-io1-xd49.google.com [IPv6:2607:f8b0:4864:20::d49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B1D8E7C for ; Fri, 15 Apr 2022 14:59:22 -0700 (PDT) Received: by mail-io1-xd49.google.com with SMTP id w28-20020a05660205dc00b00645d3cdb0f7so5423442iox.10 for ; Fri, 15 Apr 2022 14:59:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Mb/OefWBn9RCKNoTmiaaE/VUMk03Unut8vex1/PGQS0=; b=fsNxIh3BEH+XlmSMjPVMtJeBZUk0FQpQlK3w9p+lwpUBGbcSk9U8YLOg1rOBKVY1uR viQStsQ0f4HjS7zw60Z7a9eHBjDc6y8KRSAjfz92x1WW108wRFezoo18fGyW9sSXBFjN 2njPnnWl4Sr2o3BvCsi7QBaMPCEPmdo7c66TYpshJHac92EsLJ7nryQaBB4BbzWGOmGE IYHOMX5S+Fo9qzysE1iP/+8Q31pXvGIBFwOUa9Uu4JmQUu3iidG5OczCVUkqWrYk1BLJ vJIDxqvp/F8fIVIgSNja8wv/C6uygQfysG5mpeJRy6sVuameHckrI6H2LQ9ArdcD5nwl 2BBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Mb/OefWBn9RCKNoTmiaaE/VUMk03Unut8vex1/PGQS0=; b=lq6nYystU9OUznPwyOTvfxPk2uqgVN7ePJHZYnPTE8RoIEUzT6QLok9HWWF82WToLt 1Bo18B6/fJg5dJw0GtytH3dSFNtLVD7bVIbqQWIllq6hOuhGJXh0YNeKGsxmQm3l92rU xH5ztZp08bXkk0NrhRuA23dpAFhc5ewizP448GKiTXdYM5WZ0sAjlKx5dBloz4L++mYY vTBlpNtR1T3Lnst3p2LGSN/+NDpAt8r092kOpfa9HZ7VQdxoqMAF9bFnLwWZPpEj6lAQ +1wsn8gmBnBAPIUxDTfUdkFH7Yc42yokAL7wjKIk9264IKINXZu8ErailvVNg76pvr+0 Vy8Q== X-Gm-Message-State: AOAM531DqwkvmAgz5fizEDr+S+0IMfbIqtrCZhnMDaTvEKru+f6E3p8+ i67/MdT3dZp/v0e2gAd8ATiY2B+KYns= X-Google-Smtp-Source: ABdhPJzuFSrhWKcatk7b24EN0VEAajIWqNloCuQY+463KUJzIaWadPCnXDRKjBU8YzbqADXAgMTcRMSd1kw= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6602:14cb:b0:646:3b7d:6aee with SMTP id b11-20020a05660214cb00b006463b7d6aeemr346421iow.178.1650059961830; Fri, 15 Apr 2022 14:59:21 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:58 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-15-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 14/17] KVM: arm64: Punt last page reference to rcu callback for parallel walk From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org It is possible that a table page remains visible to another thread until the next rcu synchronization event. To that end, we cannot drop the last page reference synchronous with post-order traversal for a parallel table walk. Schedule an rcu callback to clean up the child table page for parallel walks. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_pgtable.h | 3 ++ arch/arm64/kvm/hyp/pgtable.c | 24 +++++++++++++-- arch/arm64/kvm/mmu.c | 44 +++++++++++++++++++++++++++- 3 files changed, 67 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 74955aba5918..52e55e00f0ca 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -81,6 +81,8 @@ static inline bool kvm_level_supports_block_mapping(u32 level) * @put_page: Decrement the refcount on a page. When the * refcount reaches 0 the page is automatically * freed. + * @free_table: Drop the last page reference, possibly in the + * next RCU sync if doing a shared walk. * @page_count: Return the refcount of a page. * @phys_to_virt: Convert a physical address into a virtual * address mapped in the current context. @@ -98,6 +100,7 @@ struct kvm_pgtable_mm_ops { void (*get_page)(void *addr); void (*put_page)(void *addr); int (*page_count)(void *addr); + void (*free_table)(void *addr, bool shared); void* (*phys_to_virt)(phys_addr_t phys); phys_addr_t (*virt_to_phys)(void *addr); void (*dcache_clean_inval_poc)(void *addr, size_t size); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 121818d4c33e..a9a48edba63b 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -147,12 +147,19 @@ static inline void kvm_pgtable_walk_end(void) {} #define kvm_dereference_ptep rcu_dereference_raw + +static inline void kvm_pgtable_destroy_barrier(void) +{} + #else #define kvm_pgtable_walk_begin rcu_read_lock #define kvm_pgtable_walk_end rcu_read_unlock #define kvm_dereference_ptep rcu_dereference + +#define kvm_pgtable_destroy_barrier rcu_barrier + #endif static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte, struct kvm_pgtable_mm_ops *mm_ops) @@ -1063,7 +1070,12 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, childp = kvm_pte_follow(*old, mm_ops); } - mm_ops->put_page(childp); + /* + * If we do not have exclusive access to the page tables it is possible + * the unlinked table remains visible to another thread until the next + * rcu synchronization. + */ + mm_ops->free_table(childp, shared); mm_ops->put_page(ptep); return ret; @@ -1203,7 +1215,7 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_granule_size(level)); if (childp) - mm_ops->put_page(childp); + mm_ops->free_table(childp, shared); return 0; } @@ -1433,7 +1445,7 @@ static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, mm_ops->put_page(ptep); if (kvm_pte_table(*old, level)) - mm_ops->put_page(kvm_pte_follow(*old, mm_ops)); + mm_ops->free_table(kvm_pte_follow(*old, mm_ops), shared); return 0; } @@ -1452,4 +1464,10 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt) pgd_sz = kvm_pgd_pages(pgt->ia_bits, pgt->start_level) * PAGE_SIZE; pgt->mm_ops->free_pages_exact(pgt->pgd, pgd_sz); pgt->pgd = NULL; + + /* + * Guarantee that all unlinked subtrees associated with the stage2 page + * table have also been freed before returning. + */ + kvm_pgtable_destroy_barrier(); } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index cc6ed6b06ec2..6ecf37009c21 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -98,9 +98,50 @@ static bool kvm_is_device_pfn(unsigned long pfn) static void *stage2_memcache_zalloc_page(void *arg) { struct kvm_mmu_caches *mmu_caches = arg; + struct stage2_page_header *hdr; + void *addr; /* Allocated with __GFP_ZERO, so no need to zero */ - return kvm_mmu_memory_cache_alloc(&mmu_caches->page_cache); + addr = kvm_mmu_memory_cache_alloc(&mmu_caches->page_cache); + if (!addr) + return NULL; + + hdr = kvm_mmu_memory_cache_alloc(&mmu_caches->header_cache); + if (!hdr) { + free_page((unsigned long)addr); + return NULL; + } + + hdr->page = virt_to_page(addr); + set_page_private(hdr->page, (unsigned long)hdr); + return addr; +} + +static void stage2_free_page_now(struct stage2_page_header *hdr) +{ + WARN_ON(page_ref_count(hdr->page) != 1); + + __free_page(hdr->page); + kmem_cache_free(stage2_page_header_cache, hdr); +} + +static void stage2_free_page_rcu_cb(struct rcu_head *head) +{ + struct stage2_page_header *hdr = container_of(head, struct stage2_page_header, + rcu_head); + + stage2_free_page_now(hdr); +} + +static void stage2_free_table(void *addr, bool shared) +{ + struct page *page = virt_to_page(addr); + struct stage2_page_header *hdr = (struct stage2_page_header *)page_private(page); + + if (shared) + call_rcu(&hdr->rcu_head, stage2_free_page_rcu_cb); + else + stage2_free_page_now(hdr); } static void *kvm_host_zalloc_pages_exact(size_t size) @@ -613,6 +654,7 @@ static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = { .free_pages_exact = free_pages_exact, .get_page = kvm_host_get_page, .put_page = kvm_host_put_page, + .free_table = stage2_free_table, .page_count = kvm_host_page_count, .phys_to_virt = kvm_host_va, .virt_to_phys = kvm_host_pa, From patchwork Fri Apr 15 21:58:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8DAAC433FE for ; Fri, 15 Apr 2022 21:59:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231343AbiDOWCA (ORCPT ); Fri, 15 Apr 2022 18:02:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356185AbiDOWBx (ORCPT ); Fri, 15 Apr 2022 18:01:53 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93858FD3B for ; Fri, 15 Apr 2022 14:59:23 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 7-20020a250c07000000b0064137917a4eso7655633ybm.12 for ; Fri, 15 Apr 2022 14:59:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+g21tjl52haZEGL9PeLa8ZsXfugpwy0IPvVVK6CWGsg=; b=ZLsICKC3SmkNR+5tKzOjUbnQi/MrOWshrvj8ba9gD9ErKmF7bM9Ls3/73nj7AN223f 2SqgDq0mAv2Q+0NseF2Q6yJ9vAST2pHisfGt1aJ03ZR+yeXY4WCtFCLFnkoU0PMNx7bv GtmK0jCaxnoFBzIF482k+LXDT3QNYJ5NswQc3tSxLDzrxHjhk7kCcz/EML91XtB0TNBK Xm9rTMKrDR8cvHZo0rs4YCcBNImFSd/Zp6dYdWGcwa6fTHERGWIJByQqZFmWaHjjCdY8 3CDcSDTVe1yeF5TkHlWZQLhTDhaOUlu2ycQ37ChtdjS0d0bGJEyFflrGc5HuZQxrxKqT /V6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+g21tjl52haZEGL9PeLa8ZsXfugpwy0IPvVVK6CWGsg=; b=69tysHXBjhEpOyrAVy/kDTuidGmBpoel8itLXH8pS8AIptVMFqA4GO+hGiS+TcgVuF /DiupU7MLXGwTigb4FjQECqyFQ+r0ndbTm1lBrH6pFbK3+VugI4umisdwxg06u4+K8EH 8m2kWKsHZY/yef8TpHgoyxHVS8Jkyb87i3Jut4xzstBqFdY6CY2DXrVyYyZUSbRyCULU aNvnmUDn38MQCLoNAy8pCdmgTDqI+28YUvQNFz3J4GixPDM3j1+tYqtwJdHS6W45F92u JW7mDEu1A7wZhTCvJhhFvWrZWey6gFfIUCJk7FZ0ubJjkqoG8kzLlnsmM3kuorr39Qty u2HA== X-Gm-Message-State: AOAM532BbTEkGC25M/LKkNRKF9hOWscJQn3Aby6u8JncD4tKotXrTzEr Bfo5cVH5QFnLKlRadyCVcqZmDtPHb6w= X-Google-Smtp-Source: ABdhPJxd3FM6U0CPN7En0pTvibRCFHGaVeXWp61fOLSmlPBx3PvTF9e7RZlcLR5fnOduP4TGiTN80GlSguU= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a0d:e743:0:b0:2eb:3106:9b32 with SMTP id q64-20020a0de743000000b002eb31069b32mr940693ywe.512.1650059962733; Fri, 15 Apr 2022 14:59:22 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:59 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-16-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 15/17] KVM: arm64: Allow parallel calls to kvm_pgtable_stage2_map() From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The map walker is now appraised of how to walk the tables in parallel with another table walker. Take a parameter indicating whether or not a walk is done in parallel so as to relax the atomicity/locking requirements on ptes. Defer actually using parallel walks to a later change. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_pgtable.h | 4 +++- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 2 +- arch/arm64/kvm/hyp/pgtable.c | 4 ++-- arch/arm64/kvm/mmu.c | 6 +++--- 4 files changed, 9 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 52e55e00f0ca..9830eea19de4 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -328,6 +328,8 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); * @prot: Permissions and attributes for the mapping. * @mc: Cache of pre-allocated and zeroed memory from which to allocate * page-table pages. + * @shared: true if multiple software walkers could be traversing the tables + * in parallel * * The offset of @addr within a page is ignored, @size is rounded-up to * the next page boundary and @phys is rounded-down to the previous page @@ -349,7 +351,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); */ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot, - void *mc); + void *mc, bool shared); /** * kvm_pgtable_stage2_set_owner() - Unmap and annotate pages in the IPA space to diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 42a5f35cd819..53b172036c2a 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -251,7 +251,7 @@ static inline int __host_stage2_idmap(u64 start, u64 end, enum kvm_pgtable_prot prot) { return kvm_pgtable_stage2_map(&host_kvm.pgt, start, end - start, start, - prot, &host_s2_pool); + prot, &host_s2_pool, false); } /* diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index a9a48edba63b..20ff198ebef7 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1119,7 +1119,7 @@ static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot, - void *mc) + void *mc, bool shared) { int ret; struct stage2_map_data map_data = { @@ -1144,7 +1144,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, if (ret) return ret; - ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, shared); dsb(ishst); return ret; } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 6ecf37009c21..63cf18cdb978 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -832,7 +832,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, write_lock(&kvm->mmu_lock); ret = kvm_pgtable_stage2_map(pgt, addr, PAGE_SIZE, pa, prot, - &cache); + &cache, false); write_unlock(&kvm->mmu_lock); if (ret) break; @@ -1326,7 +1326,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize, __pfn_to_phys(pfn), prot, - mmu_caches); + mmu_caches, true); } /* Mark the page dirty only if the fault is handled successfully */ @@ -1526,7 +1526,7 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) */ kvm_pgtable_stage2_map(kvm->arch.mmu.pgt, range->start << PAGE_SHIFT, PAGE_SIZE, __pfn_to_phys(pfn), - KVM_PGTABLE_PROT_R, NULL); + KVM_PGTABLE_PROT_R, NULL, false); return false; } From patchwork Fri Apr 15 21:59:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3360C433F5 for ; Fri, 15 Apr 2022 21:59:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230008AbiDOWCC (ORCPT ); Fri, 15 Apr 2022 18:02:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356159AbiDOWBx (ORCPT ); Fri, 15 Apr 2022 18:01:53 -0400 Received: from mail-io1-xd49.google.com (mail-io1-xd49.google.com [IPv6:2607:f8b0:4864:20::d49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3BB77DE88 for ; Fri, 15 Apr 2022 14:59:24 -0700 (PDT) Received: by mail-io1-xd49.google.com with SMTP id d19-20020a0566022bf300b00645eba5c992so5453372ioy.4 for ; Fri, 15 Apr 2022 14:59:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=we5tUcZWUFX29TKi0tS/Yi5sEjxnPk5PaYrhMJdNw1U=; b=O1SErvMM/5NizynFN7dxIWF/wAOfb8Mq9ZfTSGeW0QBIGQpcAPGMZiYPK5b+AIE+yj 08zA/OxjxYQfyzgWCjGX98P8LIEuoV19f+IbszwnCQUQtevPI3XYw1re2k4fZZe6/vAf Qpexkx6ehQQAsh1RxGlFf+N5b4EPHjVdZkItOeRiv/yIkybMfQ6rKCCTjah2MjXE8ysA UfJUeFHEmDdjsqYhKAbeH01CdWClTD/JQdzWn2NRgu/MbALmGjnCzElEQCYlDtSQfUul EPbBWfGohELG4mrVx0BVrh06dZ/wmMGcdkJN6MSYmZEdZM8jKw5KOVWNj7WfK3JyNrde ComQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=we5tUcZWUFX29TKi0tS/Yi5sEjxnPk5PaYrhMJdNw1U=; b=o0XHWhKsaiztLT/N9YB2FpaQeS1g5xuPk0Arbb22Oj9Ow3EkiU3gXH3zLPF4LUOMVe OvBsN0+ce+pn6X1ACpSE0nuwG0UDW+x3argJXttIYHolf11S2pplyKsuOkdlMGqITPRZ 9betibpBMY4fT+1XadVQ/vwtqgCSNEgvcGtjhiFS/xHr8VbrMT6iXh3Oj0ULcJB9aGKQ ktnNaGeQdkBVpOCafrvsxrDMbH1f/E0Fy7pp8kA587mOScgng6NNeKlb9PnqJfQNgcIU 0PQS8m4nPks5Y6HCX4fRurakeeYL1TT0SqFJBlhvp9m7pNXQ4DPQ5Kjz+sUBO/+wRsoV TCwQ== X-Gm-Message-State: AOAM532FAXzi0vnNNB+g0VFTmLgrhVlncrAz+nA4PJN2o6qSz8fZo5nj Y3xiU67Xe3xx2rqX43jKyALRIju7+mQ= X-Google-Smtp-Source: ABdhPJzFhPu6MxIliWrYx7BlAqWDvC1LTp6uI8FxSJyQJeYO+JKxmOQwyxXOX0XwY8FpVX7WQSGMEYCJYH0= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6e02:1807:b0:2ca:4b88:1a42 with SMTP id a7-20020a056e02180700b002ca4b881a42mr312136ilv.258.1650059963514; Fri, 15 Apr 2022 14:59:23 -0700 (PDT) Date: Fri, 15 Apr 2022 21:59:00 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-17-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 16/17] KVM: arm64: Enable parallel stage 2 MMU faults From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Voila! Since the map walkers are able to work in parallel there is no need to take the write lock on a stage 2 memory abort. Relax locking on map operations and cross fingers we got it right. Signed-off-by: Oliver Upton --- arch/arm64/kvm/mmu.c | 21 +++------------------ 1 file changed, 3 insertions(+), 18 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 63cf18cdb978..2881051c3743 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1127,7 +1127,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, gfn_t gfn; kvm_pfn_t pfn; bool logging_active = memslot_is_logging(memslot); - bool use_read_lock = false; unsigned long fault_level = kvm_vcpu_trap_get_fault_level(vcpu); unsigned long vma_pagesize, fault_granule; enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R; @@ -1162,8 +1161,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (logging_active) { force_pte = true; vma_shift = PAGE_SHIFT; - use_read_lock = (fault_status == FSC_PERM && write_fault && - fault_granule == PAGE_SIZE); } else { vma_shift = get_vma_page_shift(vma, hva); } @@ -1267,15 +1264,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (exec_fault && device) return -ENOEXEC; - /* - * To reduce MMU contentions and enhance concurrency during dirty - * logging dirty logging, only acquire read lock for permission - * relaxation. - */ - if (use_read_lock) - read_lock(&kvm->mmu_lock); - else - write_lock(&kvm->mmu_lock); + read_lock(&kvm->mmu_lock); + pgt = vcpu->arch.hw_mmu->pgt; if (mmu_notifier_retry(kvm, mmu_seq)) goto out_unlock; @@ -1322,8 +1312,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (fault_status == FSC_PERM && vma_pagesize == fault_granule) { ret = kvm_pgtable_stage2_relax_perms(pgt, fault_ipa, prot); } else { - WARN_ONCE(use_read_lock, "Attempted stage-2 map outside of write lock\n"); - ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize, __pfn_to_phys(pfn), prot, mmu_caches, true); @@ -1336,10 +1324,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, } out_unlock: - if (use_read_lock) - read_unlock(&kvm->mmu_lock); - else - write_unlock(&kvm->mmu_lock); + read_unlock(&kvm->mmu_lock); kvm_set_pfn_accessed(pfn); kvm_release_pfn_clean(pfn); return ret != -EAGAIN ? ret : 0; From patchwork Fri Apr 15 21:59:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56571C4332F for ; Fri, 15 Apr 2022 21:59:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356159AbiDOWCE (ORCPT ); Fri, 15 Apr 2022 18:02:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1355788AbiDOWBy (ORCPT ); Fri, 15 Apr 2022 18:01:54 -0400 Received: from mail-il1-x149.google.com (mail-il1-x149.google.com [IPv6:2607:f8b0:4864:20::149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FB7DFD3B for ; Fri, 15 Apr 2022 14:59:25 -0700 (PDT) Received: by mail-il1-x149.google.com with SMTP id p10-20020a056e02104a00b002caa828f7b1so5432447ilj.7 for ; Fri, 15 Apr 2022 14:59:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=jzvc81PBGj4bV/jHUjtzPtFNqWylMffaehwRBxD6iIE=; b=aWkRKsBqnouPuu0iRgQFmbdZNwM6u65S9STgd8WzLazymunDNq+F/2LV041lFacxkX sH2au5CIL4U6btbR36TcgERHcBL5JA8T93pnN1zy/vCBpJyTdEx4s8aDOU+JwJD7FEgH EDuCMiJ2lKlmqCfo2HCyFA8YBpzh+0As1Im7YSg7MivXDgsyltO1HdVbeLv+yGFmNIPi O5p8sXbUH14m1SBo2J0fuW1TDCouol/p3zaCmN7Pb1tclO6RjKEsMyqqJ0eOLp38uD1h AbzgBBIOOsbBeQk1WFRiKNO/0k6ugAlDvMy5k+FqVLVzTLXvQl7QmS1J6MjDvDSfP9W4 9n1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=jzvc81PBGj4bV/jHUjtzPtFNqWylMffaehwRBxD6iIE=; b=2kYp3lXJQnCS/TUV5BYxYxtdLECtPFj4/a03o/57xYQd7oRAIfsB+o1hOohd8/2nwj c/n6FKpK0a2Isz4bNe8hySob7GVzzH2JhMDCJX+EeSQdY8SiCrDcQRADfv1uaAfxszri eX25F6OeuBxM3ht5H16Qu5Usj2gU0/IF2qXY7NxVpgxHNeecFrG4jVwyNu4Jl863Slln JJNRH04eNWsk0MmCLVdqx/befZqIfZUgRvYggDTCqNyRrERKZOuSuctpcnHRpMRosamW nINWz0RQS7tkDh2clabjaH2zqhpVEJorfQbORbH7NdtBYQvGb94RopivnsTwxtLD4pLX lKfw== X-Gm-Message-State: AOAM530WZaYoNfso+nnQm/jvMK6Q5uOmnkF3cHBWXJ6CPwvqXK9GrDEb IRV6RXA87OEaCn7tt831Cqmp7XTFoDo= X-Google-Smtp-Source: ABdhPJy0T0D9tPSOS2fkDjAWGdXFAsgYKiBOBw6Vv9SkS2qbIf2KOV78tziChH3+cjdCnnpihxhBubRLicI= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6e02:1b0f:b0:2c7:9a3f:6e84 with SMTP id i15-20020a056e021b0f00b002c79a3f6e84mr312271ilv.32.1650059964470; Fri, 15 Apr 2022 14:59:24 -0700 (PDT) Date: Fri, 15 Apr 2022 21:59:01 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-18-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 17/17] TESTONLY: KVM: arm64: Add super lazy accounting of stage 2 table pages From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Don't use this please. I was just being lazy but wanted to make sure tables are all accounted for. There's a race here too, do you see it? :) Signed-off-by: Oliver Upton --- arch/arm64/kvm/mmu.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 2881051c3743..68ea7f0244fe 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -95,6 +95,8 @@ static bool kvm_is_device_pfn(unsigned long pfn) return !pfn_is_map_memory(pfn); } +static atomic_t stage2_pages = ATOMIC_INIT(0); + static void *stage2_memcache_zalloc_page(void *arg) { struct kvm_mmu_caches *mmu_caches = arg; @@ -112,6 +114,8 @@ static void *stage2_memcache_zalloc_page(void *arg) return NULL; } + atomic_inc(&stage2_pages); + hdr->page = virt_to_page(addr); set_page_private(hdr->page, (unsigned long)hdr); return addr; @@ -121,6 +125,8 @@ static void stage2_free_page_now(struct stage2_page_header *hdr) { WARN_ON(page_ref_count(hdr->page) != 1); + atomic_dec(&stage2_pages); + __free_page(hdr->page); kmem_cache_free(stage2_page_header_cache, hdr); } @@ -662,6 +668,8 @@ static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = { .icache_inval_pou = invalidate_icache_guest_page, }; +static atomic_t stage2_mmus = ATOMIC_INIT(0); + /** * kvm_init_stage2_mmu - Initialise a S2 MMU structure * @kvm: The pointer to the KVM structure @@ -699,6 +707,8 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) for_each_possible_cpu(cpu) *per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1; + atomic_inc(&stage2_mmus); + mmu->pgt = pgt; mmu->pgd_phys = __pa(pgt->pgd); return 0; @@ -796,6 +806,9 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) kvm_pgtable_stage2_destroy(pgt); kfree(pgt); } + + if (atomic_dec_and_test(&stage2_mmus)) + WARN_ON(atomic_read(&stage2_pages)); } /**