From patchwork Fri Apr 15 21:58:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9FEDFC433F5 for ; Fri, 15 Apr 2022 22:00:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ynqN6Obptrkgz/ZqOcdeGY4ZvY0jvLFXY+NHlgK0olk=; b=TOmYzubbI8q0bltT5z0hM2Z+WJ zZYf2+sfqwUg6J7OOi0rWaMXvf8oPLxnQzdXuUq/R4zgw4NBciHluvIZoddid/NlJQkN40mVJJk77 rcr4A8ZiqVIERyuIin4C6vaC1//uWbJCHwzqq373Ly2m7WjEixlXODEqQRmfua/I07WVYhx0i0r6L GEXoXRDCzF0tSo90Kf3yTWehwSF4TCAJFZVMFQg3lPNCm+YuZMn/f8TuPU9UABizLdVJY6cmb27Dv yLQByDFUwXwZbF6ULpD3a2NbJSv1Kv0ydVi0HUxK+xnIRUS06hmVjK6Nw1Aq5dCLcRAD0yOf4vejq qEEfWMfQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyR-00BS9k-3I; Fri, 15 Apr 2022 21:59:27 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyB-00BRyx-Pd for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:13 +0000 Received: by mail-yb1-xb49.google.com with SMTP id q142-20020a25d994000000b00641d23dbeb9so7336150ybg.9 for ; Fri, 15 Apr 2022 14:59:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Tyg2Q1CJQYh6kqJKn5I8WsZWhqynGf1zv3GZVnyVrPI=; b=bs/o3a1yI+YtQKTec7tfylXXQu4M6ad2QWFFL62JX+gXbwqO9FEus3pkH+ThX+JPAr JPEPreeCxgkfcEjZxuDSMTLLZUbJYuMNUMiPP28ZaBVQA8xAoGGJwyWWqjUTVOr9w+H1 scbsRRQ4oV/0Ko6nzOrL8D58Z+zUc2quNieeYqKnL4/2GddH6EUAGmW8TUmhdokw0buR c0G9PrlnzO6Zct/R0EMtBthEbtbkFluRFT3bcL6Benpi1sBm1BLEUooH5TD8OFDM9SNB FGX1+tPIQqSDfmOl4/bgqfhcbkyjAEMmpAUImOqM0PuNfNbC+i60/UI9Bsp0mp7hYMtn Jqnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Tyg2Q1CJQYh6kqJKn5I8WsZWhqynGf1zv3GZVnyVrPI=; b=gQvYwX4vwvHf2SsMWnebRI/2dx3771eLvYxvi5nfnmRUzCQQvb0zoEEMtqKq2kunW8 VcZbfI0roExPudMPwBpa5oWwQ12Ym1jH4xMegAoN8Uhjn0ZcoOkYIeyFBST7WR6s+cOV BklfNOb8VjkOVhuw3L0VE8dF25SGt6GHge0SlRzY2JfdOcnuaH/+erRxiV/TDPHP/LDf 10HQg42FgvowT4/rJcsfC1slEwssLhutYiz433217vW1IJEosGTXNL50CjWVvMSZ1Uci lPVw7Ekh9f4nIsV7JZN2T2dU6msWi/tphkzRFhHM945cx06HNfxYDk/mrz2OQALTJp0u vUXQ== X-Gm-Message-State: AOAM533U3ODWFbGMw/V78nmkhEfM2VkeoQQkiWZnVfHhfDo8t221aaVN CrcpgsI6utcoWULH9FGixjzQUkVAIr8= X-Google-Smtp-Source: ABdhPJxEvl0U3CkvkcrBAn3HsWtsYmV0z/JZVUMkoVOPtU1p1JiWR679eaZiC7C8eGZtheHJa4oZ8Gi2BLk= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6902:544:b0:641:4007:1da8 with SMTP id z4-20020a056902054400b0064140071da8mr1070971ybs.195.1650059949262; Fri, 15 Apr 2022 14:59:09 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:45 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-2-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 01/17] KVM: arm64: Directly read owner id field in stage2_pte_is_counted() From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145911_867628_2771F17A X-CRM114-Status: UNSURE ( 9.62 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A subsequent change to KVM will make use of additional bits in invalid ptes. Prepare for said change by explicitly checking the valid bit and owner fields in stage2_pte_is_counted() Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 2cb3867eb7c2..e1506da3e2fb 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -172,6 +172,11 @@ static kvm_pte_t kvm_init_invalid_leaf_owner(u8 owner_id) return FIELD_PREP(KVM_INVALID_PTE_OWNER_MASK, owner_id); } +static u8 kvm_invalid_pte_owner(kvm_pte_t pte) +{ + return FIELD_GET(KVM_INVALID_PTE_OWNER_MASK, pte); +} + static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64 addr, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag) @@ -679,7 +684,7 @@ static bool stage2_pte_is_counted(kvm_pte_t pte) * encode ownership of a page to another entity than the page-table * owner, whose id is 0. */ - return !!pte; + return kvm_pte_valid(pte) || kvm_invalid_pte_owner(pte); } static void stage2_put_pte(kvm_pte_t *ptep, struct kvm_s2_mmu *mmu, u64 addr, From patchwork Fri Apr 15 21:58:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7A655C433F5 for ; Fri, 15 Apr 2022 22:00:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ci/12OyJGwP3QxbKp+IZG5BfWZbQt9cpFmCGUwUllqk=; b=q8S4YbfkOWb0rSiqoFGylbtZAi C7FD5OL6QOxissB8yze4aI97oG3LLcQf9dQEv1x9bEi2ec2iIBND/00+dhoutIFjgIlW32/pWjSjM zSmW5pq91N66StsOp8ZI0f6YBFOkb6NPZjg4V9VQ3jRYk2bHUJ7GE7HQgAuDbLO9Vv8C3AJHA88fj lro5OWZPYqQ+D4ysufqocOKStPbX6mkacoetBnE2jvljiOq3sKT9vJkbIXtv1CI/PpkfUmXLtuPOi KgAMfQZEVM2vlEsW/ofCJyEUO1OvqNidJMfvTdCzSuVXvyTFS4w4alEWwGKw2QSG9KSegTd4WAcGi UEwbSc3Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyb-00BSEs-Q2; Fri, 15 Apr 2022 21:59:37 +0000 Received: from mail-oo1-xc4a.google.com ([2607:f8b0:4864:20::c4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyC-00BRze-Cp for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:15 +0000 Received: by mail-oo1-xc4a.google.com with SMTP id k189-20020a4a4ac6000000b003353c23321fso2265362oob.18 for ; Fri, 15 Apr 2022 14:59:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=33zitZnypC/5iqOPd40oIiPlXIxM1MYIm9nWTEFWX9o=; b=VeV72ZOKUV0992/61pLhySJu6tUdMC93GGMiIqLb9e42oF03C+E2MC6P0USn6WlWZL rRNNBiXDbdfTk5SekJB4zDXKX5HZQcP9L/L90QZ9Q75Zs43P4FnxEdhX0e9NjQNL70sv 2OPflS3JXbC6xVrB8AYJnRT4D6TEDtD80e3gDO4GetEaSf95eT3DYLJKy2GdkEoTECwt kBvgRBy6BE0rZ3f1wCMtyh7VurS1pRhhSYr6eZEbOvHISaR8hy4G1UMIcnlc/RFerVBo oXY5ygxtXok6qpcxi+MJT9/FZJVx/Zmp7+6uAsJiQSmMLB6jJ6cMJW4AyoLAa7e0QIHR F2Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=33zitZnypC/5iqOPd40oIiPlXIxM1MYIm9nWTEFWX9o=; b=gcQ7uBsCM0u3dm88R+sMty8NTkcwL4LAMECUZf5AIQ8OGCjySjG2SRLshyaWCm74FH tanS83BayuLZnJjpmjoV+0lhoTWkHYYSmRyhitAqr2+Ue2zS+Ao+RXuIeCi8sE0qcdjJ 3YwFsR3xBSOtR3izYKgqRZtt4TIqHqBpy4DHJm/WYE2/z5h9RCkpo9dg8/A5ALJuz87h KchGPQYvczUlTmX7JezCF6os8qcIpoGOvNLZSJMIdecTcnP6a26QvfkcrrKJlxYuZqiv sV/sTjOnfBYFogPuQENqs3z7ELllGJXuuPJOldgCblSUpvtg4/jDxIUqJng8d1bA8Soe CSXA== X-Gm-Message-State: AOAM5308I28TcfdR/qR0Bh1U7ukl0D0BWkewrZ35D5eC9/0htomUiUFm /nWiZbVcrYc2bltPGbgTfTtptDazxek= X-Google-Smtp-Source: ABdhPJzWYiHch/fvnCe1iFtXCe45SXo8QvSqLvIYoF5WCPSgovlAS0bf0qoZ0Uk3C+rcWH/OsMMHHUvyqdM= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6870:f29a:b0:de:eaa4:233a with SMTP id u26-20020a056870f29a00b000deeaa4233amr354373oap.137.1650059950378; Fri, 15 Apr 2022 14:59:10 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:46 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-3-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 02/17] KVM: arm64: Only read the pte once per visit From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145912_487016_FE0964DE X-CRM114-Status: GOOD ( 19.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A subsequent change to KVM will parallize modifications to the stage-2 page tables. The various page table walkers read the ptep multiple times, which could lead to a visitor seeing multiple values during the visit. Pass through the observed pte to the visitor callbacks. Promote reads of the ptep to a full READ_ONCE(), which will matter more when we start tweaking ptes atomically. Note that a pointer to the old pte is given to visitors, as parallel visitors will need to steer the page table traversal as they adjust the page tables. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_pgtable.h | 2 +- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 7 +- arch/arm64/kvm/hyp/nvhe/setup.c | 9 +- arch/arm64/kvm/hyp/pgtable.c | 113 +++++++++++++------------- 4 files changed, 63 insertions(+), 68 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 9f339dffbc1a..ea818a5f7408 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -192,7 +192,7 @@ enum kvm_pgtable_walk_flags { }; typedef int (*kvm_pgtable_visitor_fn_t)(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg); diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 78edf077fa3b..601a586581d8 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -422,17 +422,16 @@ struct check_walk_data { }; static int __check_page_state_visitor(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct check_walk_data *d = arg; - kvm_pte_t pte = *ptep; - if (kvm_pte_valid(pte) && !addr_is_memory(kvm_pte_to_phys(pte))) + if (kvm_pte_valid(*old) && !addr_is_memory(kvm_pte_to_phys(*old))) return -EINVAL; - return d->get_page_state(pte) == d->desired ? 0 : -EPERM; + return d->get_page_state(*old) == d->desired ? 0 : -EPERM; } static int check_page_state_range(struct kvm_pgtable *pgt, u64 addr, u64 size, diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index 27af337f9fea..ecab7a4049d6 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -162,17 +162,16 @@ static void hpool_put_page(void *addr) } static int finalize_host_mappings_walker(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct kvm_pgtable_mm_ops *mm_ops = arg; enum kvm_pgtable_prot prot; enum pkvm_page_state state; - kvm_pte_t pte = *ptep; phys_addr_t phys; - if (!kvm_pte_valid(pte)) + if (!kvm_pte_valid(*old)) return 0; /* @@ -187,7 +186,7 @@ static int finalize_host_mappings_walker(u64 addr, u64 end, u32 level, if (level != (KVM_PGTABLE_MAX_LEVELS - 1)) return -EINVAL; - phys = kvm_pte_to_phys(pte); + phys = kvm_pte_to_phys(*old); if (!addr_is_memory(phys)) return -EINVAL; @@ -195,7 +194,7 @@ static int finalize_host_mappings_walker(u64 addr, u64 end, u32 level, * Adjust the host stage-2 mappings to match the ownership attributes * configured in the hypervisor stage-1. */ - state = pkvm_getstate(kvm_pgtable_hyp_pte_prot(pte)); + state = pkvm_getstate(kvm_pgtable_hyp_pte_prot(*old)); switch (state) { case PKVM_PAGE_OWNED: return host_stage2_set_owner_locked(phys, PAGE_SIZE, pkvm_hyp_id); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index e1506da3e2fb..ad911cd44425 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -178,11 +178,11 @@ static u8 kvm_invalid_pte_owner(kvm_pte_t pte) } static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64 addr, - u32 level, kvm_pte_t *ptep, + u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag) { struct kvm_pgtable_walker *walker = data->walker; - return walker->cb(addr, data->end, level, ptep, flag, walker->arg); + return walker->cb(addr, data->end, level, ptep, old, flag, walker->arg); } static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, @@ -193,17 +193,17 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, { int ret = 0; u64 addr = data->addr; - kvm_pte_t *childp, pte = *ptep; + kvm_pte_t *childp, pte = READ_ONCE(*ptep); bool table = kvm_pte_table(pte, level); enum kvm_pgtable_walk_flags flags = data->walker->flags; if (table && (flags & KVM_PGTABLE_WALK_TABLE_PRE)) { - ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, + ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, KVM_PGTABLE_WALK_TABLE_PRE); } if (!table && (flags & KVM_PGTABLE_WALK_LEAF)) { - ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, + ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, KVM_PGTABLE_WALK_LEAF); pte = *ptep; table = kvm_pte_table(pte, level); @@ -224,7 +224,7 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, goto out; if (flags & KVM_PGTABLE_WALK_TABLE_POST) { - ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, + ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, KVM_PGTABLE_WALK_TABLE_POST); } @@ -297,12 +297,12 @@ struct leaf_walk_data { u32 level; }; -static int leaf_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, +static int leaf_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct leaf_walk_data *data = arg; - data->pte = *ptep; + data->pte = *old; data->level = level; return 0; @@ -388,10 +388,10 @@ enum kvm_pgtable_prot kvm_pgtable_hyp_pte_prot(kvm_pte_t pte) return prot; } -static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, struct hyp_map_data *data) +static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, + kvm_pte_t old, struct hyp_map_data *data) { - kvm_pte_t new, old = *ptep; + kvm_pte_t new; u64 granule = kvm_granule_size(level), phys = data->phys; if (!kvm_block_mapping_supported(addr, end, phys, level)) @@ -410,14 +410,14 @@ static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, return true; } -static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, +static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { kvm_pte_t *childp; struct hyp_map_data *data = arg; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; - if (hyp_map_walker_try_leaf(addr, end, level, ptep, arg)) + if (hyp_map_walker_try_leaf(addr, end, level, ptep, *old, arg)) return 0; if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1)) @@ -461,19 +461,19 @@ struct hyp_unmap_data { struct kvm_pgtable_mm_ops *mm_ops; }; -static int hyp_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, +static int hyp_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { - kvm_pte_t pte = *ptep, *childp = NULL; + kvm_pte_t *childp = NULL; u64 granule = kvm_granule_size(level); struct hyp_unmap_data *data = arg; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; - if (!kvm_pte_valid(pte)) + if (!kvm_pte_valid(*old)) return -EINVAL; - if (kvm_pte_table(pte, level)) { - childp = kvm_pte_follow(pte, mm_ops); + if (kvm_pte_table(*old, level)) { + childp = kvm_pte_follow(*old, mm_ops); if (mm_ops->page_count(childp) != 1) return 0; @@ -537,19 +537,18 @@ int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits, return 0; } -static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, +static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct kvm_pgtable_mm_ops *mm_ops = arg; - kvm_pte_t pte = *ptep; - if (!kvm_pte_valid(pte)) + if (!kvm_pte_valid(*old)) return 0; mm_ops->put_page(ptep); - if (kvm_pte_table(pte, level)) - mm_ops->put_page(kvm_pte_follow(pte, mm_ops)); + if (kvm_pte_table(*old, level)) + mm_ops->put_page(kvm_pte_follow(*old, mm_ops)); return 0; } @@ -723,10 +722,10 @@ static bool stage2_leaf_mapping_allowed(u64 addr, u64 end, u32 level, } static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t old, struct stage2_map_data *data) { - kvm_pte_t new, old = *ptep; + kvm_pte_t new; u64 granule = kvm_granule_size(level), phys = data->phys; struct kvm_pgtable *pgt = data->mmu->pgt; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; @@ -769,7 +768,7 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, } static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t *old, struct stage2_map_data *data) { if (data->anchor) @@ -778,7 +777,7 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, if (!stage2_leaf_mapping_allowed(addr, end, level, data)) return 0; - data->childp = kvm_pte_follow(*ptep, data->mm_ops); + data->childp = kvm_pte_follow(*old, data->mm_ops); kvm_clear_pte(ptep); /* @@ -792,20 +791,20 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, } static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - struct stage2_map_data *data) + kvm_pte_t *old, struct stage2_map_data *data) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; - kvm_pte_t *childp, pte = *ptep; + kvm_pte_t *childp; int ret; if (data->anchor) { - if (stage2_pte_is_counted(pte)) + if (stage2_pte_is_counted(*old)) mm_ops->put_page(ptep); return 0; } - ret = stage2_map_walker_try_leaf(addr, end, level, ptep, data); + ret = stage2_map_walker_try_leaf(addr, end, level, ptep, *old, data); if (ret != -E2BIG) return ret; @@ -824,7 +823,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, * a table. Accesses beyond 'end' that fall within the new table * will be mapped lazily. */ - if (stage2_pte_is_counted(pte)) + if (stage2_pte_is_counted(*old)) stage2_put_pte(ptep, data->mmu, addr, level, mm_ops); kvm_set_table_pte(ptep, childp, mm_ops); @@ -834,7 +833,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, } static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, - kvm_pte_t *ptep, + kvm_pte_t *ptep, kvm_pte_t *old, struct stage2_map_data *data) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; @@ -848,9 +847,9 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, childp = data->childp; data->anchor = NULL; data->childp = NULL; - ret = stage2_map_walk_leaf(addr, end, level, ptep, data); + ret = stage2_map_walk_leaf(addr, end, level, ptep, old, data); } else { - childp = kvm_pte_follow(*ptep, mm_ops); + childp = kvm_pte_follow(*old, mm_ops); } mm_ops->put_page(childp); @@ -878,18 +877,18 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, * the page-table, installing the block entry when it revisits the anchor * pointer and clearing the anchor to NULL. */ -static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, +static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct stage2_map_data *data = arg; switch (flag) { case KVM_PGTABLE_WALK_TABLE_PRE: - return stage2_map_walk_table_pre(addr, end, level, ptep, data); + return stage2_map_walk_table_pre(addr, end, level, ptep, old, data); case KVM_PGTABLE_WALK_LEAF: - return stage2_map_walk_leaf(addr, end, level, ptep, data); + return stage2_map_walk_leaf(addr, end, level, ptep, old, data); case KVM_PGTABLE_WALK_TABLE_POST: - return stage2_map_walk_table_post(addr, end, level, ptep, data); + return stage2_map_walk_table_post(addr, end, level, ptep, old, data); } return -EINVAL; @@ -955,29 +954,29 @@ int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, } static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - enum kvm_pgtable_walk_flags flag, + kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct kvm_pgtable *pgt = arg; struct kvm_s2_mmu *mmu = pgt->mmu; struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops; - kvm_pte_t pte = *ptep, *childp = NULL; + kvm_pte_t *childp = NULL; bool need_flush = false; - if (!kvm_pte_valid(pte)) { - if (stage2_pte_is_counted(pte)) { + if (!kvm_pte_valid(*old)) { + if (stage2_pte_is_counted(*old)) { kvm_clear_pte(ptep); mm_ops->put_page(ptep); } return 0; } - if (kvm_pte_table(pte, level)) { - childp = kvm_pte_follow(pte, mm_ops); + if (kvm_pte_table(*old, level)) { + childp = kvm_pte_follow(*old, mm_ops); if (mm_ops->page_count(childp) != 1) return 0; - } else if (stage2_pte_cacheable(pgt, pte)) { + } else if (stage2_pte_cacheable(pgt, *old)) { need_flush = !stage2_has_fwb(pgt); } @@ -989,7 +988,7 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, stage2_put_pte(ptep, mmu, addr, level, mm_ops); if (need_flush && mm_ops->dcache_clean_inval_poc) - mm_ops->dcache_clean_inval_poc(kvm_pte_follow(pte, mm_ops), + mm_ops->dcache_clean_inval_poc(kvm_pte_follow(*old, mm_ops), kvm_granule_size(level)); if (childp) @@ -1018,10 +1017,10 @@ struct stage2_attr_data { }; static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - enum kvm_pgtable_walk_flags flag, + kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { - kvm_pte_t pte = *ptep; + kvm_pte_t pte = *old; struct stage2_attr_data *data = arg; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; @@ -1146,18 +1145,17 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, } static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - enum kvm_pgtable_walk_flags flag, + kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct kvm_pgtable *pgt = arg; struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops; - kvm_pte_t pte = *ptep; - if (!kvm_pte_valid(pte) || !stage2_pte_cacheable(pgt, pte)) + if (!kvm_pte_valid(*old) || !stage2_pte_cacheable(pgt, *old)) return 0; if (mm_ops->dcache_clean_inval_poc) - mm_ops->dcache_clean_inval_poc(kvm_pte_follow(pte, mm_ops), + mm_ops->dcache_clean_inval_poc(kvm_pte_follow(*old, mm_ops), kvm_granule_size(level)); return 0; } @@ -1206,19 +1204,18 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu, } static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - enum kvm_pgtable_walk_flags flag, + kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, void * const arg) { struct kvm_pgtable_mm_ops *mm_ops = arg; - kvm_pte_t pte = *ptep; - if (!stage2_pte_is_counted(pte)) + if (!stage2_pte_is_counted(*old)) return 0; mm_ops->put_page(ptep); - if (kvm_pte_table(pte, level)) - mm_ops->put_page(kvm_pte_follow(pte, mm_ops)); + if (kvm_pte_table(*old, level)) + mm_ops->put_page(kvm_pte_follow(*old, mm_ops)); return 0; } From patchwork Fri Apr 15 21:58:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91E41C433F5 for ; Fri, 15 Apr 2022 22:00:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=9OqTIowHW1OTnuKQQfDu3dhCzfA+AyAxYpRGqdMW6U4=; b=vbZKz2k04RLsVj9Y2g02AdRe45 THYt4UYC/1Qg+QIvQAbLHVKcQgwvpELmrhHk2rr4KWfflEGlbqsXx8wMwzmAaZzB4aO19p0id3guY 2mgb5GoVoRz2nSnplbJaLpt4FC9elqe9cVtr8ImT51UH4PHqKuevNG7a5Fe8SnCt03mdFtiTOYxUK Im020fsyzmFvOYA3VBpU4YGcypG5tIUOGN+nXu3/pt8fL0EzHioliEg0cxpy5dCEVwGUZ8IQz0vOG P3yLAPPE2yz66P7kzGm72y55nMNwoAnxDDpgbzMfqCWKKvX/vjC6xyQbwJFBy0CFN7tFTEVEh/9JW VBTZyKtA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyn-00BSLS-BN; Fri, 15 Apr 2022 21:59:49 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyE-00BS04-0r for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:15 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-2ebebe9f982so74996837b3.17 for ; Fri, 15 Apr 2022 14:59:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=rJSIRdYzlgGDLU7B2Sk5nACgnVsuidWq4rds0LFJzOM=; b=SkrQ1f8DZ8O/NnR7vbyJvJQIHE/etfmq3uFO690m3CAYnbo6y2Ixqb7gAvpAOTVELL 6wOSAg7n/y+F6WeMp05hmHl1fcyiWC8VoNMxEO4OeOYzRlFDvzzQ7jFGCTtm845QHSoG LccB7cLdd71eA0BI0Ru81TgjWmc8dS30Ot2pDg0Hy1uI1b5zb05v+/GrUfnWCauhOnPJ NySizl2te/E7Y0lMCQzjXVzUzPthbWsjhBBncJtixhl5pRXhKQ3taAldhzHbsTSuXMOE rm5tR4TfdW/tfw/Uf8GxBSY3l4Dd0OpWCNbMfnbvgCpEtN0cME8Q2ExGtfE12VB6U/GX +VFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=rJSIRdYzlgGDLU7B2Sk5nACgnVsuidWq4rds0LFJzOM=; b=Yf7TbSNRLwh8e/lJa0DqoHeL8k1Z/jSB6Dm4ASK41Fsl2EKy3WBc3q1m4jhR2H3jNH cagYjiQ54ikYqLQrWeOzb1g43iV+a8lmDsPleFSUm59JJ9gJJBH/paV6Kosenwfd9HP6 UQSaNnREWTgcbRZupqbzBk/htiT1napjK4QbBDuwfqPg5o6+wsSEnnvjHh9lOqN9gOIa SHbznaIHJ2Suqd1yvj3CLq7657+QCO2q7YPkxPtfZhBvmuIj+AqXM9SOD0I724fm16LA ssSkYl0+w76uZ3EwGBHbc+M6E/oFGpj0VEEiXj59SWzSvCN0b6ztr9qAktVMRub6WjtN jU7A== X-Gm-Message-State: AOAM530iTnLkR7tafTmhtJsh3RGnja+RgoL7RJ/6O7Z2Jro01bQr8mOu 2/f4q/4CwFGCTiX/nTuMXoottk+l3Hc= X-Google-Smtp-Source: ABdhPJwz4sx5L43ob0Z3jNzyJVyA2Jzsl+sCY5ofO1Xxk9cu0zlH1igqV35RpwEAUlFfB8+y4wm6MGToMi4= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a25:6f55:0:b0:63e:7447:7c19 with SMTP id k82-20020a256f55000000b0063e74477c19mr1036666ybc.551.1650059951469; Fri, 15 Apr 2022 14:59:11 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:47 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-4-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 03/17] KVM: arm64: Return the next table from map callbacks From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145914_127740_2FBBFC5C X-CRM114-Status: GOOD ( 11.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The stage-2 and hyp stage-1 map walkers install new page tables during their traversal. In order to support parallel table walks, make callers return the next table to traverse. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index ad911cd44425..5b64fbca8a93 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -205,13 +205,12 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, if (!table && (flags & KVM_PGTABLE_WALK_LEAF)) { ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, KVM_PGTABLE_WALK_LEAF); - pte = *ptep; - table = kvm_pte_table(pte, level); } if (ret) goto out; + table = kvm_pte_table(pte, level); if (!table) { data->addr = ALIGN_DOWN(data->addr, kvm_granule_size(level)); data->addr += kvm_granule_size(level); @@ -429,6 +428,7 @@ static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte kvm_set_table_pte(ptep, childp, mm_ops); mm_ops->get_page(ptep); + *old = *ptep; return 0; } @@ -828,7 +828,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_set_table_pte(ptep, childp, mm_ops); mm_ops->get_page(ptep); - + *old = *ptep; return 0; } From patchwork Fri Apr 15 21:58:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E0BEDC433EF for ; Fri, 15 Apr 2022 22:01:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=qtP92N6YqsjCU1iwE5vI1KgzL+Es+RThlLfO3VIBTC8=; b=uVdyxY8mnI43MH8kOW2O1xpYe9 ONWGrwXLDtVW+jeFn/IC/tuebJVQH2wjGr3seyXMy/lFyWtmOkE4XAvh8z9hrrpxQJvDUVfwf1HOS snki9flKaeUhmZ0lQ9U1MO3wMt8t7cD4wvHueWYAKyCRnTjD3BC+YSweIl2jg2pbmMtgo3JqGE6DZ w2eZYKwi5AnBUhC8L4k5dvBF+7n2Ba7ywvC0uNOtBXjJ4LR2eFPxC5n2ETXV2QBZ8dageXMHCDUNB PU5f7I8FGk796GYAXsqwXjR786j3GR8tZ+oVGdCSfNipKOSMJtfy4rPoYMOTcRJ0N2CBpie7LtPFV f65TU1vw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyy-00BSOO-Mo; Fri, 15 Apr 2022 22:00:00 +0000 Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyE-00BS0Y-No for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:17 +0000 Received: by mail-io1-xd4a.google.com with SMTP id z23-20020a6b0a17000000b00649f13ea3a7so5404304ioi.23 for ; Fri, 15 Apr 2022 14:59:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=tskkletjI1xqy2hmf+NUDTVo6kgDSU4EL5qdYTa3BLw=; b=QA94h9r9pTA8GiRYxYPRCc5lnTzXtNJwZmjCbvYN9o6CVFPh0a2tzWI0hkn2GGLweA Txpvs213Jksx0vyr2ngPBysyiEw8KC95EbN0Ju8DT94akH1bQ62gDiZBnBpVmOCl0cU/ aVzTHXbtTPOJrXqnXT7zTKFXplRLdZ7urB7/NYg8iYE+D1B5NQXQp/3h+iCMDQQGzAYs +i1QI9nvkwP3sApqy92wxAC2ykVI1oGcIkQRiYST9T63+AoQWeZ9+xAr1yQAoB7YwpSw L5Y0d4II8kHY6IRbv/oLjeQ9jnR7XErbOljztHLeXABbdjtB3zDvW2Om/oADovnpl9yM AUkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=tskkletjI1xqy2hmf+NUDTVo6kgDSU4EL5qdYTa3BLw=; b=nZzyx299/n6kJClTS6oovyqQSylB/dHvilkA6nuZ+NorAFUhF6K+j31e2RBlyldOOw eBgtlAXO2M18GlOm/alu1nN12DoT9+IpUCfC1SLdHmtm8PQ1uaFh55WuhwIdMoJ3cBg3 KtFYOaPPL4qaFB9Ah2AWaFDSCEDO04oWfaWCl0fKvWoEaOWhDpe4nsSJq0HPvqVMipCo D2by2fSBaQrJFZ1MXP7lYVzhYcpgdvsng5WBuGn8MCys8+0CI0/NaCdiQhcmqn8V92F4 2x+rj+XnkKlNoeLAmUIpBtLe2vusXPdk4lVwwfIAZt65jpk8ATrmmrDGQJsXobPYVjm8 rnCA== X-Gm-Message-State: AOAM530XHeNh1+uzkUvmxM6dVS5bJi8tRVaEBz0dt2oPXMdAOhx/pckR qayJabDAz0b4xzrDFUVF55hn3kPkZrI= X-Google-Smtp-Source: ABdhPJyVhtFMabjcfJB2grqI91o53/8kDGiTwPio15sftKb54bR0cux4tb8br2mZT5w8v0L5arzr0CpVpX8= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6e02:20cd:b0:2ca:c074:4ebb with SMTP id 13-20020a056e0220cd00b002cac0744ebbmr283965ilq.73.1650059952284; Fri, 15 Apr 2022 14:59:12 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:48 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-5-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 04/17] KVM: arm64: Protect page table traversal with RCU From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145914_838127_64B3F0DE X-CRM114-Status: GOOD ( 10.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use RCU to safely traverse the page tables in parallel; the tables themselves will only be freed from an RCU synchronized context. Don't even bother with adding support to hyp, and instead just assume exclusive access of the page tables. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 5b64fbca8a93..d4699f698d6e 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -132,9 +132,28 @@ static kvm_pte_t kvm_phys_to_pte(u64 pa) return pte; } + +#if defined(__KVM_NVHE_HYPERVISOR__) +static inline void kvm_pgtable_walk_begin(void) +{} + +static inline void kvm_pgtable_walk_end(void) +{} + +#define kvm_dereference_ptep rcu_dereference_raw +#else +#define kvm_pgtable_walk_begin rcu_read_lock + +#define kvm_pgtable_walk_end rcu_read_unlock + +#define kvm_dereference_ptep rcu_dereference +#endif + static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte, struct kvm_pgtable_mm_ops *mm_ops) { - return mm_ops->phys_to_virt(kvm_pte_to_phys(pte)); + kvm_pte_t __rcu *ptep = mm_ops->phys_to_virt(kvm_pte_to_phys(pte)); + + return kvm_dereference_ptep(ptep); } static void kvm_clear_pte(kvm_pte_t *ptep) @@ -288,7 +307,9 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, .walker = walker, }; + kvm_pgtable_walk_begin(); return _kvm_pgtable_walk(&walk_data); + kvm_pgtable_walk_end(); } struct leaf_walk_data { From patchwork Fri Apr 15 21:58:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815450 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B6519C433F5 for ; Fri, 15 Apr 2022 22:02:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=HV2ZxNOHPY3tn2UnggwWz+3FcTaiPOzPldAQ32bCrcU=; b=tbVdLQ4SkqftR7aSiOCYlc2JSn 4UksyrsLorZ82ggqOeQzXr87vTfjFjWyulu/G3AL77cNKL6NDrF/dPp/qbWD+hBffZWlUis9SrzoN fiD/2fVD88x2YW9No1VzxE7NzB/206O1GlW4FtuUnKBL9Nj0Md/ATirAiC83wJLAgYywGa0BHFqfX y4/GZJyfWAJwIuSLIW5rvjOmxMJ8mNhLC5SvFsNj8XNUSmsN18ludDnl/eGmPBDG0WMvMwc09IpkQ sovOrq5iAr95zjc5DdsE3EOvqHZa5NclZp8ILsAzsNeBRSyjrsrjalPz/hfIHeOq4Tl4meoxW4aGl iH4ozo0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU03-00BSoC-0e; Fri, 15 Apr 2022 22:01:07 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyE-00BS13-OW for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:19 +0000 Received: by mail-io1-xd49.google.com with SMTP id h14-20020a05660208ce00b00645c339411bso5434541ioz.8 for ; Fri, 15 Apr 2022 14:59:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=1Dvq0vSpuJ+B5i+lAUjot+b1S2UM0uuMiRyTOuKbozw=; b=f0Ct2KeIjmpzITrXuAlU/IXp9HqHaYUY14BbtuMlXzD5cMFehCeAkuIQZN9MFinFsR Qbp4qrLoUZqthvpvsvdzGjy8wLygbQ0WlDP6aTuRbHdO6gSsTKMD7flO7JZ2wK78u7pG wT+k3nbsihczJyCyCIw6RfoaQT/fGxDvVRDbU7dqJ8LveecvlvpXCtSrZ8FvWshbrqjr Ng8Gw6k3NN7OCQLHsMdOXloj+4FOOvcMCgra/n/WfgQ+hXQpA9D4ZmLf6aM2jgKzhggE p0G7rjmCjRnZn/8CS7n2NbTu/eKmMqhSDUDAs++6W0Jit3quykTz1+d+JiM2+53KSPXM XqQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=1Dvq0vSpuJ+B5i+lAUjot+b1S2UM0uuMiRyTOuKbozw=; b=mOknFtwQLZcBOTug+NrLYjYv8y4F7dtPa7ZYlSQO0CyZAVhXh5Cq+3hbcIDrtSxm/i /LtjJnVLtvnGNDRVrm9crtKP7fnEGvzggrrEwsDyLKqjGrsp8pz9s8FtZCEL7oqIYU4t 0IjP3gQ5LlXNmQue57Z6sx8WSMV0bvUfJbHEV1Jq2Hgnp78ZKgmo8PETYztdXHNkleeT YVQQpTXMADnvMEYFMDjBIp8DpjOnx1u2QKFKXEBsx/fdGi2FgFkqlU9P1g+aRxRwVQJh 4zJSnkOytR4VzqyCQnARhwXrQuoPa1PNBjDcTwgN5VX44RjK4SZuAJ/TEwopQnIX8URl f0sA== X-Gm-Message-State: AOAM531KZl+nY8FBtb1gLgeZtT9MDX8Y44JEkYPPw+PGxqcpwDf69XQA We7F2CVLVir0FRsY9M1o0gQhc7Y3Ovg= X-Google-Smtp-Source: ABdhPJw+CGHjGO0ZS08PkgVlHelz0Jm4z2IXXsD/Vyzt5RIyBZDRwHzHFKVKAtaVcfL7OMy4ZxqJFar+bXU= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6e02:12e9:b0:2c9:f576:8951 with SMTP id l9-20020a056e0212e900b002c9f5768951mr355751iln.204.1650059953138; Fri, 15 Apr 2022 14:59:13 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:49 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-6-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 05/17] KVM: arm64: Take an argument to indicate parallel walk From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145914_880832_DB9F4437 X-CRM114-Status: GOOD ( 19.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org It is desirable to reuse the same page walkers for serial and parallel faults. Take an argument to kvm_pgtable_walk() (and throughout) to indicate whether or not a walk might happen in parallel with another. No functional change intended. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_pgtable.h | 5 +- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 4 +- arch/arm64/kvm/hyp/nvhe/setup.c | 4 +- arch/arm64/kvm/hyp/pgtable.c | 91 ++++++++++++++------------- 4 files changed, 54 insertions(+), 50 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index ea818a5f7408..74955aba5918 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -194,7 +194,7 @@ enum kvm_pgtable_walk_flags { typedef int (*kvm_pgtable_visitor_fn_t)(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg); + void * const arg, bool shared); /** * struct kvm_pgtable_walker - Hook into a page-table walk. @@ -490,6 +490,7 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size); * @addr: Input address for the start of the walk. * @size: Size of the range to walk. * @walker: Walker callback description. + * @shared: Indicates if the page table walk can be done in parallel * * The offset of @addr within a page is ignored and @size is rounded-up to * the next page boundary. @@ -506,7 +507,7 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size); * Return: 0 on success, negative error code on failure. */ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, - struct kvm_pgtable_walker *walker); + struct kvm_pgtable_walker *walker, bool shared); /** * kvm_pgtable_get_leaf() - Walk a page-table and retrieve the leaf entry diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 601a586581d8..42a5f35cd819 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -424,7 +424,7 @@ struct check_walk_data { static int __check_page_state_visitor(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { struct check_walk_data *d = arg; @@ -443,7 +443,7 @@ static int check_page_state_range(struct kvm_pgtable *pgt, u64 addr, u64 size, .flags = KVM_PGTABLE_WALK_LEAF, }; - return kvm_pgtable_walk(pgt, addr, size, &walker); + return kvm_pgtable_walk(pgt, addr, size, &walker, false); } static enum pkvm_page_state host_get_page_state(kvm_pte_t pte) diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index ecab7a4049d6..178a5539fe7c 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -164,7 +164,7 @@ static void hpool_put_page(void *addr) static int finalize_host_mappings_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { struct kvm_pgtable_mm_ops *mm_ops = arg; enum kvm_pgtable_prot prot; @@ -224,7 +224,7 @@ static int finalize_host_mappings(void) struct memblock_region *reg = &hyp_memory[i]; u64 start = (u64)hyp_phys_to_virt(reg->base); - ret = kvm_pgtable_walk(&pkvm_pgtable, start, reg->size, &walker); + ret = kvm_pgtable_walk(&pkvm_pgtable, start, reg->size, &walker, false); if (ret) return ret; } diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index d4699f698d6e..bf46d6d24951 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -198,17 +198,17 @@ static u8 kvm_invalid_pte_owner(kvm_pte_t pte) static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64 addr, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag) + enum kvm_pgtable_walk_flags flag, bool shared) { struct kvm_pgtable_walker *walker = data->walker; - return walker->cb(addr, data->end, level, ptep, old, flag, walker->arg); + return walker->cb(addr, data->end, level, ptep, old, flag, walker->arg, shared); } static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, - kvm_pte_t *pgtable, u32 level); + kvm_pte_t *pgtable, u32 level, bool shared); static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, - kvm_pte_t *ptep, u32 level) + kvm_pte_t *ptep, u32 level, bool shared) { int ret = 0; u64 addr = data->addr; @@ -218,12 +218,12 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, if (table && (flags & KVM_PGTABLE_WALK_TABLE_PRE)) { ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, - KVM_PGTABLE_WALK_TABLE_PRE); + KVM_PGTABLE_WALK_TABLE_PRE, shared); } if (!table && (flags & KVM_PGTABLE_WALK_LEAF)) { ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, - KVM_PGTABLE_WALK_LEAF); + KVM_PGTABLE_WALK_LEAF, shared); } if (ret) @@ -237,13 +237,13 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, } childp = kvm_pte_follow(pte, data->pgt->mm_ops); - ret = __kvm_pgtable_walk(data, childp, level + 1); + ret = __kvm_pgtable_walk(data, childp, level + 1, shared); if (ret) goto out; if (flags & KVM_PGTABLE_WALK_TABLE_POST) { ret = kvm_pgtable_visitor_cb(data, addr, level, ptep, &pte, - KVM_PGTABLE_WALK_TABLE_POST); + KVM_PGTABLE_WALK_TABLE_POST, shared); } out: @@ -251,7 +251,7 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, } static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, - kvm_pte_t *pgtable, u32 level) + kvm_pte_t *pgtable, u32 level, bool shared) { u32 idx; int ret = 0; @@ -265,7 +265,7 @@ static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, if (data->addr >= data->end) break; - ret = __kvm_pgtable_visit(data, ptep, level); + ret = __kvm_pgtable_visit(data, ptep, level, shared); if (ret) break; } @@ -273,7 +273,7 @@ static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, return ret; } -static int _kvm_pgtable_walk(struct kvm_pgtable_walk_data *data) +static int _kvm_pgtable_walk(struct kvm_pgtable_walk_data *data, bool shared) { u32 idx; int ret = 0; @@ -289,7 +289,7 @@ static int _kvm_pgtable_walk(struct kvm_pgtable_walk_data *data) for (idx = kvm_pgd_page_idx(data); data->addr < data->end; ++idx) { kvm_pte_t *ptep = &pgt->pgd[idx * PTRS_PER_PTE]; - ret = __kvm_pgtable_walk(data, ptep, pgt->start_level); + ret = __kvm_pgtable_walk(data, ptep, pgt->start_level, shared); if (ret) break; } @@ -298,7 +298,7 @@ static int _kvm_pgtable_walk(struct kvm_pgtable_walk_data *data) } int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, - struct kvm_pgtable_walker *walker) + struct kvm_pgtable_walker *walker, bool shared) { struct kvm_pgtable_walk_data walk_data = { .pgt = pgt, @@ -308,7 +308,7 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, }; kvm_pgtable_walk_begin(); - return _kvm_pgtable_walk(&walk_data); + return _kvm_pgtable_walk(&walk_data, shared); kvm_pgtable_walk_end(); } @@ -318,7 +318,7 @@ struct leaf_walk_data { }; static int leaf_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag, void * const arg) + enum kvm_pgtable_walk_flags flag, void * const arg, bool shared) { struct leaf_walk_data *data = arg; @@ -340,7 +340,7 @@ int kvm_pgtable_get_leaf(struct kvm_pgtable *pgt, u64 addr, int ret; ret = kvm_pgtable_walk(pgt, ALIGN_DOWN(addr, PAGE_SIZE), - PAGE_SIZE, &walker); + PAGE_SIZE, &walker, false); if (!ret) { if (ptep) *ptep = data.pte; @@ -409,7 +409,7 @@ enum kvm_pgtable_prot kvm_pgtable_hyp_pte_prot(kvm_pte_t pte) } static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - kvm_pte_t old, struct hyp_map_data *data) + kvm_pte_t old, struct hyp_map_data *data, bool shared) { kvm_pte_t new; u64 granule = kvm_granule_size(level), phys = data->phys; @@ -431,13 +431,13 @@ static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *pte } static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag, void * const arg) + enum kvm_pgtable_walk_flags flag, void * const arg, bool shared) { kvm_pte_t *childp; struct hyp_map_data *data = arg; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; - if (hyp_map_walker_try_leaf(addr, end, level, ptep, *old, arg)) + if (hyp_map_walker_try_leaf(addr, end, level, ptep, *old, arg, shared)) return 0; if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1)) @@ -471,7 +471,7 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, if (ret) return ret; - ret = kvm_pgtable_walk(pgt, addr, size, &walker); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); dsb(ishst); isb(); return ret; @@ -483,7 +483,7 @@ struct hyp_unmap_data { }; static int hyp_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag, void * const arg) + enum kvm_pgtable_walk_flags flag, void * const arg, bool shared) { kvm_pte_t *childp = NULL; u64 granule = kvm_granule_size(level); @@ -536,7 +536,7 @@ u64 kvm_pgtable_hyp_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) if (!pgt->mm_ops->page_count) return 0; - kvm_pgtable_walk(pgt, addr, size, &walker); + kvm_pgtable_walk(pgt, addr, size, &walker, false); return unmap_data.unmapped; } @@ -559,7 +559,7 @@ int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits, } static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag, void * const arg) + enum kvm_pgtable_walk_flags flag, void * const arg, bool shared) { struct kvm_pgtable_mm_ops *mm_ops = arg; @@ -582,7 +582,7 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt) .arg = pgt->mm_ops, }; - WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); + WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker, false)); pgt->mm_ops->put_page(pgt->pgd); pgt->pgd = NULL; } @@ -744,7 +744,8 @@ static bool stage2_leaf_mapping_allowed(u64 addr, u64 end, u32 level, static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t old, - struct stage2_map_data *data) + struct stage2_map_data *data, + bool shared) { kvm_pte_t new; u64 granule = kvm_granule_size(level), phys = data->phys; @@ -790,7 +791,8 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - struct stage2_map_data *data) + struct stage2_map_data *data, + bool shared) { if (data->anchor) return 0; @@ -812,7 +814,7 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, } static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - kvm_pte_t *old, struct stage2_map_data *data) + kvm_pte_t *old, struct stage2_map_data *data, bool shared) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; kvm_pte_t *childp; @@ -825,7 +827,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, return 0; } - ret = stage2_map_walker_try_leaf(addr, end, level, ptep, *old, data); + ret = stage2_map_walker_try_leaf(addr, end, level, ptep, *old, data, shared); if (ret != -E2BIG) return ret; @@ -855,7 +857,8 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - struct stage2_map_data *data) + struct stage2_map_data *data, + bool shared) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; kvm_pte_t *childp; @@ -868,7 +871,7 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, childp = data->childp; data->anchor = NULL; data->childp = NULL; - ret = stage2_map_walk_leaf(addr, end, level, ptep, old, data); + ret = stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared); } else { childp = kvm_pte_follow(*old, mm_ops); } @@ -899,17 +902,17 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, * pointer and clearing the anchor to NULL. */ static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, - enum kvm_pgtable_walk_flags flag, void * const arg) + enum kvm_pgtable_walk_flags flag, void * const arg, bool shared) { struct stage2_map_data *data = arg; switch (flag) { case KVM_PGTABLE_WALK_TABLE_PRE: - return stage2_map_walk_table_pre(addr, end, level, ptep, old, data); + return stage2_map_walk_table_pre(addr, end, level, ptep, old, data, shared); case KVM_PGTABLE_WALK_LEAF: - return stage2_map_walk_leaf(addr, end, level, ptep, old, data); + return stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared); case KVM_PGTABLE_WALK_TABLE_POST: - return stage2_map_walk_table_post(addr, end, level, ptep, old, data); + return stage2_map_walk_table_post(addr, end, level, ptep, old, data, shared); } return -EINVAL; @@ -942,7 +945,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, if (ret) return ret; - ret = kvm_pgtable_walk(pgt, addr, size, &walker); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); dsb(ishst); return ret; } @@ -970,13 +973,13 @@ int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, if (owner_id > KVM_MAX_OWNER_ID) return -EINVAL; - ret = kvm_pgtable_walk(pgt, addr, size, &walker); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); return ret; } static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { struct kvm_pgtable *pgt = arg; struct kvm_s2_mmu *mmu = pgt->mmu; @@ -1026,7 +1029,7 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, }; - return kvm_pgtable_walk(pgt, addr, size, &walker); + return kvm_pgtable_walk(pgt, addr, size, &walker, false); } struct stage2_attr_data { @@ -1039,7 +1042,7 @@ struct stage2_attr_data { static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { kvm_pte_t pte = *old; struct stage2_attr_data *data = arg; @@ -1091,7 +1094,7 @@ static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr, .flags = KVM_PGTABLE_WALK_LEAF, }; - ret = kvm_pgtable_walk(pgt, addr, size, &walker); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); if (ret) return ret; @@ -1167,7 +1170,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { struct kvm_pgtable *pgt = arg; struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops; @@ -1192,7 +1195,7 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) if (stage2_has_fwb(pgt)) return 0; - return kvm_pgtable_walk(pgt, addr, size, &walker); + return kvm_pgtable_walk(pgt, addr, size, &walker, false); } @@ -1226,7 +1229,7 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu, static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, enum kvm_pgtable_walk_flags flag, - void * const arg) + void * const arg, bool shared) { struct kvm_pgtable_mm_ops *mm_ops = arg; @@ -1251,7 +1254,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt) .arg = pgt->mm_ops, }; - WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); + WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker, false)); pgd_sz = kvm_pgd_pages(pgt->ia_bits, pgt->start_level) * PAGE_SIZE; pgt->mm_ops->free_pages_exact(pgt->pgd, pgd_sz); pgt->pgd = NULL; From patchwork Fri Apr 15 21:58:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4FBECC433FE for ; Fri, 15 Apr 2022 22:01:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ziS0rq6xPPilMAk8ngkpdI5nfIKUV87VOL/WDS8gd4I=; b=KM/Tif7b9SFkkn9QY3QUwQ6qhW 0150Dw+5nAPYoXmiLsap2W7S1u3kgF5i4u8UjZqvgGAJbsfQ6dFkkrh88KzHN/ptKNtzXpqLN432g IPi5l/AK/CKXmPnbC4cxfKKHFgNc/u17wxNQ2thkGE2y4von8/U7INcxxnVWDLAiJbDfoeJmtYBib ANfLsHzvd1UfR3tIbbkiX7ai1NKdodnFiwpLBw6vpGhH9480B9gIMb75LunP0mcJbhGrXA3ED/dzD zlfRBVoudxiI3d5QFD7FwHlXIQJzKTIUZIhiSWtVOLDoU1ATvgpqaH7cHpBsq7yvRmG9GdtQKYTZR dXoV9QdA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTzb-00BScg-9i; Fri, 15 Apr 2022 22:00:39 +0000 Received: from mail-il1-x14a.google.com ([2607:f8b0:4864:20::14a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyF-00BS1Q-6H for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:18 +0000 Received: by mail-il1-x14a.google.com with SMTP id x6-20020a923006000000b002bea39c3974so5409180ile.12 for ; Fri, 15 Apr 2022 14:59:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=dlVM2Fz2Y7ISzW+T03M67t0iCDhFZowMvNW5W95Ip88=; b=Diuw9S7MBxEBgzLHbnnt4GYfzxtBiJUB0zsGMYHDzpDoe3JhqJZWSJDdokUWM0zkha i8x/gC5nUrVYUs6hOdPDTL9nf6kIJU6VcYk1l6lDT4D+fYmAlc/RjfiLOv6Lbn4Qc5ix 72AehN7YLrkzrcdoZHRTEpT+RxysbKRUYxlWnpz0GrQzl8WPj5OUn6ZyuIr66cu77bE9 U74z9VvwrCW2I1jAaAY869FqA4+HHoDayX/3H+iVX/0ZTb6Hj4TiQ6JkdiuX1fC5i1P3 41rsk/YPkHQpsi2FPnvR+M5tehzMzixyFXLPJG6fOGRp/S1x48HLPLhSkHQm4o3yOoM5 DRZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=dlVM2Fz2Y7ISzW+T03M67t0iCDhFZowMvNW5W95Ip88=; b=MHXlWEfYOinXvV3rrS7jFbCmGqvvkKF7+rXoeGPdoc2H7ELnjYCc28cZ8JkIw5QbbH GUUsAbRHdg0iy0yRjEqEUtJSJrj+xD7Ky1QvgDY0Bk7b4uDJTAMGJsW32SjT/wXdE8iA EFjCXHPPTVeOFWH+erS2h+w7oq6bnf5OKXCHsRNXk4BCDzYwnP6dng7HtD3H2B0A3Qk0 2QYhAn0AjG6hby69NR+qLcGMav9Kkibyp78XKZmzchdntJttV7gUXoOH2BfKyaVm0piX hti+72GhgTvoy9vTTMpIycGsn0yX7iFTqnUnPdXGzpixjuo4EKw8zuOZst/K0bjDYiP+ ol1g== X-Gm-Message-State: AOAM531HypTa+e7bqgqNMMp2+tZCngMMJCk1imNaywSQTwyv4xZzj8P7 ge4C6+bPww6MTN7SJcHIikuMsDhmqVg= X-Google-Smtp-Source: ABdhPJz3tHc7G5iWdHXhbRBfoyVUj3SKr1PVP3WtCMI3dyoPjw0ifVyDiqazQpsgWctE2YfwQt7/F4KvGHU= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a92:c542:0:b0:2cb:d371:1f3e with SMTP id a2-20020a92c542000000b002cbd3711f3emr288202ilj.323.1650059953946; Fri, 15 Apr 2022 14:59:13 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:50 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-7-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 06/17] KVM: arm64: Implement break-before-make sequence for parallel walks From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145915_335810_8895EE5D X-CRM114-Status: GOOD ( 28.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The ARM architecture requires that software use the 'break-before-make' sequence whenever memory is being remapped. An additional requirement of parallel page walks is a mechanism to ensure exclusive access to a pte, thereby avoiding two threads changing the pte and invariably stomping on one another. Roll the two concepts together into a new helper to implement the 'break' sequence. Use a special invalid pte value to indicate that the pte is under the exclusive control of a thread. If software walkers are traversing the tables in parallel, use an atomic compare-exchange to break the pte. Retry execution on a failed attempt to break the pte, in the hopes that either the instruction will succeed or the pte lock will be successfully acquired. Avoid unnecessary DSBs and TLBIs by only completing the sequence if the evicted pte was valid. For counted non-table ptes drop the reference immediately. Otherwise, references on tables are dropped in post-order traversal as the walker must recurse on the pruned subtree. All of the new atomics do nothing (for now), as there are a few other bits of the map walker that need to be addressed before actually walking in parallel. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 172 +++++++++++++++++++++++++++++------ 1 file changed, 146 insertions(+), 26 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index bf46d6d24951..059ebb921125 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -49,6 +49,12 @@ #define KVM_INVALID_PTE_OWNER_MASK GENMASK(9, 2) #define KVM_MAX_OWNER_ID 1 +/* + * Used to indicate a pte for which a 'make-before-break' sequence is in + * progress. + */ +#define KVM_INVALID_PTE_LOCKED BIT(10) + struct kvm_pgtable_walk_data { struct kvm_pgtable *pgt; struct kvm_pgtable_walker *walker; @@ -707,6 +713,122 @@ static bool stage2_pte_is_counted(kvm_pte_t pte) return kvm_pte_valid(pte) || kvm_invalid_pte_owner(pte); } +static bool stage2_pte_is_locked(kvm_pte_t pte) +{ + return !kvm_pte_valid(pte) && (pte & KVM_INVALID_PTE_LOCKED); +} + +static inline bool kvm_try_set_pte(kvm_pte_t *ptep, kvm_pte_t old, kvm_pte_t new, bool shared) +{ + if (!shared) { + WRITE_ONCE(*ptep, new); + return true; + } + + return cmpxchg(ptep, old, new) == old; +} + +/** + * stage2_try_break_pte() - Invalidates a pte according to the + * 'break-before-make' sequence. + * + * @ptep: Pointer to the pte to break + * @old: The previously observed value of the pte; used for compare-exchange in + * a parallel walk + * @addr: IPA corresponding to the pte + * @level: Table level of the pte + * @shared: true if the tables are shared by multiple software walkers + * @data: pointer to the map walker data + * + * Returns: true if the pte was successfully broken. + * + * If the removed pt was valid, performs the necessary DSB and TLB flush for + * the old value. Drops references to the page table if a non-table entry was + * removed. Otherwise, the table reference is preserved as the walker must also + * recurse through the child tables. + * + * See ARM DDI0487G.a D5.10.1 "General TLB maintenance requirements" for details + * on the 'break-before-make' sequence. + */ +static bool stage2_try_break_pte(kvm_pte_t *ptep, kvm_pte_t old, u64 addr, u32 level, bool shared, + struct stage2_map_data *data) +{ + /* + * Another thread could have already visited this pte and taken + * ownership. + */ + if (stage2_pte_is_locked(old)) { + /* + * If the table walker has exclusive access to the page tables + * then no other software walkers should have locked the pte. + */ + WARN_ON(!shared); + return false; + } + + if (!kvm_try_set_pte(ptep, old, KVM_INVALID_PTE_LOCKED, shared)) + return false; + + /* + * If we removed a valid pte, break-then-make rules are in effect as a + * translation may have been cached that traversed this entry. + */ + if (kvm_pte_valid(old)) { + dsb(ishst); + + if (kvm_pte_table(old, level)) + /* + * Invalidate the whole stage-2, as we may have numerous leaf + * entries below us which would otherwise need invalidating + * individually. + */ + kvm_call_hyp(__kvm_tlb_flush_vmid, data->mmu); + else + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); + } + + /* + * Don't drop the reference on table entries yet, as the walker must + * first recurse on the unlinked subtree to unlink and drop references + * to child tables. + */ + if (!kvm_pte_table(old, level) && stage2_pte_is_counted(old)) + data->mm_ops->put_page(ptep); + + return true; +} + +/** + * stage2_make_pte() - Installs a new pte according to the 'break-before-make' + * sequence. + * + * @ptep: pointer to the pte to make + * @new: new pte value to install + * + * Assumes that the pte addressed by ptep has already been broken and is under + * the ownership of the table walker. If the new pte to be installed is a valid + * entry, perform a DSB to make the write visible. Raise the reference count on + * the table if the new pte requires a reference. + * + * See ARM DDI0487G.a D5.10.1 "General TLB maintenance requirements" for details + * on the 'break-before-make' sequence. + */ +static void stage2_make_pte(kvm_pte_t *ptep, kvm_pte_t new, struct kvm_pgtable_mm_ops *mm_ops) +{ + /* Yikes! We really shouldn't install to an entry we don't own. */ + WARN_ON(!stage2_pte_is_locked(*ptep)); + + if (stage2_pte_is_counted(new)) + mm_ops->get_page(ptep); + + if (kvm_pte_valid(new)) { + WRITE_ONCE(*ptep, new); + dsb(ishst); + } else { + smp_store_release(ptep, new); + } +} + static void stage2_put_pte(kvm_pte_t *ptep, struct kvm_s2_mmu *mmu, u64 addr, u32 level, struct kvm_pgtable_mm_ops *mm_ops) { @@ -760,18 +882,17 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, else new = kvm_init_invalid_leaf_owner(data->owner_id); - if (stage2_pte_is_counted(old)) { - /* - * Skip updating the PTE if we are trying to recreate the exact - * same mapping or only change the access permissions. Instead, - * the vCPU will exit one more time from guest if still needed - * and then go through the path of relaxing permissions. - */ - if (!stage2_pte_needs_update(old, new)) - return -EAGAIN; + /* + * Skip updating the PTE if we are trying to recreate the exact same + * mapping or only change the access permissions. Instead, the vCPU will + * exit one more time from the guest if still needed and then go through + * the path of relaxing permissions. + */ + if (!stage2_pte_needs_update(old, new)) + return -EAGAIN; - stage2_put_pte(ptep, data->mmu, addr, level, mm_ops); - } + if (!stage2_try_break_pte(ptep, old, addr, level, shared, data)) + return -EAGAIN; /* Perform CMOs before installation of the guest stage-2 PTE */ if (mm_ops->dcache_clean_inval_poc && stage2_pte_cacheable(pgt, new)) @@ -781,9 +902,7 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (mm_ops->icache_inval_pou && stage2_pte_executable(new)) mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule); - smp_store_release(ptep, new); - if (stage2_pte_is_counted(new)) - mm_ops->get_page(ptep); + stage2_make_pte(ptep, new, data->mm_ops); if (kvm_phys_is_valid(phys)) data->phys += granule; return 0; @@ -800,15 +919,10 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, if (!stage2_leaf_mapping_allowed(addr, end, level, data)) return 0; - data->childp = kvm_pte_follow(*old, data->mm_ops); - kvm_clear_pte(ptep); + if (!stage2_try_break_pte(ptep, *old, addr, level, shared, data)) + return -EAGAIN; - /* - * Invalidate the whole stage-2, as we may have numerous leaf - * entries below us which would otherwise need invalidating - * individually. - */ - kvm_call_hyp(__kvm_tlb_flush_vmid, data->mmu); + data->childp = kvm_pte_follow(*old, data->mm_ops); data->anchor = ptep; return 0; } @@ -837,18 +951,24 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, if (!data->memcache) return -ENOMEM; + if (!stage2_try_break_pte(ptep, *old, addr, level, shared, data)) + return -EAGAIN; + childp = mm_ops->zalloc_page(data->memcache); - if (!childp) + if (!childp) { + /* + * Release the pte if we were unable to install a table to allow + * another thread to make an attempt. + */ + stage2_make_pte(ptep, 0, data->mm_ops); return -ENOMEM; + } /* * If we've run into an existing block mapping then replace it with * a table. Accesses beyond 'end' that fall within the new table * will be mapped lazily. */ - if (stage2_pte_is_counted(*old)) - stage2_put_pte(ptep, data->mmu, addr, level, mm_ops); - kvm_set_table_pte(ptep, childp, mm_ops); mm_ops->get_page(ptep); *old = *ptep; From patchwork Fri Apr 15 21:58:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 76342C433F5 for ; Fri, 15 Apr 2022 22:01:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=387+uCNYy/2lP02jUgyNJwOX/H7JMCr0cZtjFRBCuDg=; b=wDarZEMW7Ajij3bVo60I4tKMbt Uvqt2sQA21b5vwxH77jCCGPDb1JGi00BVrIoO82/OLlvUqAQrOULLs2Wx7fQbvs1MNjFHm59iofN0 DFF6GixjQPOSQR0PWNH05Gs/V/By3r+Ph7k9KLnbjbWN9rBRdL5G8jh0eFCkvw0/LmNYkSehHvLA9 e//2Dix3tLMstQnXhIooQAWffIHe/biq1HEY6DB56mGiN96Cg/CRQ5hoF/gVTOONN/0JrtdopxXI+ xFtUo9enf8t4cxmZf5BsLapSkoU6Keli+hRjhnkqe9e6HsiGZLn9TBUIsYuOmTa6RiYr5W3D3iK3N LPNxQfzQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTzE-00BSTa-L4; Fri, 15 Apr 2022 22:00:16 +0000 Received: from mail-il1-x14a.google.com ([2607:f8b0:4864:20::14a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyG-00BS2Z-7P for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:18 +0000 Received: by mail-il1-x14a.google.com with SMTP id i2-20020a056e021d0200b002cac9b3b46cso5434595ila.5 for ; Fri, 15 Apr 2022 14:59:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=JyIVyQg6UGD612cbxcwyNwFE+I6JblVf7Z7F7spTUWo=; b=LMqmd6P7Ue7Y9IYx3CzgSFOTxT7tT239JnYQyS2EtWQ3fN4qbQ0VflsOtSv+zaS5bk +bvfK/SmAwgiQklAVqKG84UBqOcNnfZBYDShTxLbmp4TCyU+thTgl31lI4p91fxWMOn/ dd2QO+9mQAqSvAt90OOkgKcjFxVpl7VjcraP/FuiseUcuGPj+F+x30DmaWdzLXajn575 2MLUXCslyvGHkc1BlYeAgCkKGGLpODBbWpAJnCW0QHKTJCA67zs859BL+KVaEhsWfoh0 j+lM1SBjp+89lbo+kBMwvVR9gQck0ZWmyt7cTKZrx3+vLXytC9/xjlO4AIrc/U0c/XjQ nkAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=JyIVyQg6UGD612cbxcwyNwFE+I6JblVf7Z7F7spTUWo=; b=NTrfUta372Av0TLeJiJ4cWBancpsTZ1vxn6l/be9Xun7m3XZTGcb4W3Aeg+NLYe2zZ My7XIJWAXmfhtWqM9roChRq1rHnaXZkw6mMinTfWMUB9g+rf9VTmyWu9xjJTja9eocq9 hianoU8eKEAq8VpXtm2xcdkdSYQ6Wgq7lf3KsJ5uNIvOwHj53Gbz47d1fj77NBfsYGFG Lsn+a0ffwEEobi+dlSbKfdxwmDbKKEV5lUTtmrdSozf5rH+ePHBn3daxdxnHlM+adrJJ Nv5mLYuvVr7LBWCewxVkxmgKKDA+nWXFjnaZJrg2W3fTgdTs67kpevc27rtWAsrMo9to KuAA== X-Gm-Message-State: AOAM530+cUyoCCc8eYdXHAc0Nmt+1SOjKJhMWqoVCK/Eaga3zNJec9fS 5dGss+StHJfb3xR7c7ToBR5OCScDVZE= X-Google-Smtp-Source: ABdhPJwW/MzIxEmQ87y7GaINMs+YHxfeosd37KaEM3yVPDMpYVf1oEXa8efMhhfJEE88ApB9wQ629pQLxlQ= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6602:27cc:b0:5f0:876e:126b with SMTP id l12-20020a05660227cc00b005f0876e126bmr328774ios.129.1650059954862; Fri, 15 Apr 2022 14:59:14 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:51 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-8-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 07/17] KVM: arm64: Enlighten perm relax path about parallel walks From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145916_370450_54AA9042 X-CRM114-Status: GOOD ( 13.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To date the permission relax path of the stage-2 fault handler hasn't had to worry about the paging structures changing under its nose, as map operations acquire the write lock. That's about to change, which means a permission relaxation walker could traverse in parallel with a map operation. If at any point during traversal the permission relax walker finds a locked pte, bail immediately. Either the instruction will succeed or the vCPU will fault once more and (hopefully) walk the tables successfully. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 059ebb921125..ff6f14755d0c 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1168,6 +1168,11 @@ static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_attr_data *data = arg; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; + if (stage2_pte_is_locked(pte)) { + WARN_ON(!shared); + return -EAGAIN; + } + if (!kvm_pte_valid(pte)) return 0; @@ -1190,7 +1195,9 @@ static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, stage2_pte_executable(pte) && !stage2_pte_executable(*ptep)) mm_ops->icache_inval_pou(kvm_pte_follow(pte, mm_ops), kvm_granule_size(level)); - WRITE_ONCE(*ptep, pte); + + if (!kvm_try_set_pte(ptep, data->pte, pte, shared)) + return -EAGAIN; } return 0; @@ -1199,7 +1206,7 @@ static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr, u64 size, kvm_pte_t attr_set, kvm_pte_t attr_clr, kvm_pte_t *orig_pte, - u32 *level) + u32 *level, bool shared) { int ret; kvm_pte_t attr_mask = KVM_PTE_LEAF_ATTR_LO | KVM_PTE_LEAF_ATTR_HI; @@ -1214,7 +1221,7 @@ static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr, .flags = KVM_PGTABLE_WALK_LEAF, }; - ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, shared); if (ret) return ret; @@ -1230,14 +1237,14 @@ int kvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size) { return stage2_update_leaf_attrs(pgt, addr, size, 0, KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W, - NULL, NULL); + NULL, NULL, false); } kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr) { kvm_pte_t pte = 0; stage2_update_leaf_attrs(pgt, addr, 1, KVM_PTE_LEAF_ATTR_LO_S2_AF, 0, - &pte, NULL); + &pte, NULL, false); dsb(ishst); return pte; } @@ -1246,7 +1253,7 @@ kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr) { kvm_pte_t pte = 0; stage2_update_leaf_attrs(pgt, addr, 1, 0, KVM_PTE_LEAF_ATTR_LO_S2_AF, - &pte, NULL); + &pte, NULL, false); /* * "But where's the TLBI?!", you scream. * "Over in the core code", I sigh. @@ -1259,7 +1266,7 @@ kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr) bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr) { kvm_pte_t pte = 0; - stage2_update_leaf_attrs(pgt, addr, 1, 0, 0, &pte, NULL); + stage2_update_leaf_attrs(pgt, addr, 1, 0, 0, &pte, NULL, false); return pte & KVM_PTE_LEAF_ATTR_LO_S2_AF; } @@ -1282,7 +1289,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, if (prot & KVM_PGTABLE_PROT_X) clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN; - ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level); + ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level, true); if (!ret) kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, level); return ret; From patchwork Fri Apr 15 21:58:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E5DBFC433EF for ; Fri, 15 Apr 2022 22:02:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=euK9vK48hsV1lVjdTZ4ZTw3BPKShUEDQTxTUdxAxv+M=; b=yJzTKlBSIN8QobNB0zjhL+2oZ3 kne5FxxgQkWV0iENjUSpeLVyynIFXP+mU4jaHbFdDjcyHq0RQJGiF3Uj4LSkNOEHoniOveLlAqC5L FzVEy8bcYkqDPfAOcGLyKisXixGEOy9DC9Xfq80CTQYi9mniy2ZoUC/TlfBYt+HWI+l4QFyeXPqWZ FRKOXt8+0GuMt/erjcA72BuQU4tFyGy4Sc9N+nBM4CvWy47dY2PJ6bsWoR+kX8gWl7bnCShDx9kzq CFJ/EhDpZpIga/E084a2eERf9o999usqOwdvsdnXJUISWm+ItFzJvsqqRHNxTu86TC4LPIa1mPQNX My4fPdnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU0O-00BT0W-3M; Fri, 15 Apr 2022 22:01:28 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyI-00BS3F-1o for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:19 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-2ec12272fb2so75111417b3.6 for ; Fri, 15 Apr 2022 14:59:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=B6406tYui+5lJgRvZIHJ2dZYS6AUdUd0eyrsU1t7VRI=; b=RzttCkqDWkMAl2afGr2S9DJJm5Ry89zm9fK8vSWXxjh+Br5xy5GQOart2FDuwkrzu8 +3Iq4y2goOAEc/FMm6DlVztE2ytFr3oiz9PTrVTxIm5m3Agm5OTfDy0Zs4ZMjS6SjcTP Zb1tTD9KujpWLSKkx4Mqs5RJ6LI4Z/u/gq513Flk/dllCKb2U4eZ8URwcntAGL1GuOmq ufWltbnns/Dj/9u47iW81VO53GtXf0bBfExXn11hVKS6QGf6GMMShHapP0dxeuiycC4J XphrZoAoqeoctTsXEHQSr5ZLlJzNIHeMt45VK8ODHWJivQe27pVAh1K+W85kwYhJyLS2 rKpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=B6406tYui+5lJgRvZIHJ2dZYS6AUdUd0eyrsU1t7VRI=; b=LvctxYKjNEXow3UjLQfu+ePR+Nhy/+m719TyUrbG6CL3Ca5vPn4JIB1t+vVNGnaWlG hZvhi8/2yoF5RaqdqZ5vPZJIv/UyQ/Y2UjplAvebMmasdkTrfsgI+f3RE/999CHVdvre 7VEpHNiUAZ7qsGdLYWaqe7hxAVE0M9mY/uJCzBpIACkesSEiMcptkEKohF5lK7djaML0 dgS57zEHcqwhSf2ZLFwdBA03VEoaIZU7WGK+I+1WcZXsn5vq+fkRzEVtXomPEyLStmCW 0qbc52RlFrYEYP7hZXg8pHb0fk81cVIjV30+AczAnJ22FtsEYuZMc+VHF0a/unHeVVMs qp8w== X-Gm-Message-State: AOAM5331obwSgBixoCwXP3aj6ut000SZrfN3P1sHzz7+vXxLNLWm0VCZ mS1H31J8UlVXDKoq3ncKsK1BGRt4K18= X-Google-Smtp-Source: ABdhPJyC6al0eeORHQcVaP84DLwWwiUHL9eCDBoXdDwFa7r/iAh2U4p6UOyrCV4QCls9mT6W0yMv3xIYId8= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a81:4989:0:b0:2f1:47b9:8ac2 with SMTP id w131-20020a814989000000b002f147b98ac2mr935486ywa.346.1650059955713; Fri, 15 Apr 2022 14:59:15 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:52 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-9-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 08/17] KVM: arm64: Spin off helper for initializing table pte From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145918_170646_89FB57B4 X-CRM114-Status: GOOD ( 12.01 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org With parallel table walks there is no guarantee that KVM reads back the same pte that was written. Spin off a helper that creates a pte value, thereby allowing the visitor callback to return the next table without reading the ptep again. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index ff6f14755d0c..ffdfd5ee9642 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -167,14 +167,23 @@ static void kvm_clear_pte(kvm_pte_t *ptep) WRITE_ONCE(*ptep, 0); } -static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp, - struct kvm_pgtable_mm_ops *mm_ops) +static kvm_pte_t kvm_init_table_pte(kvm_pte_t *childp, struct kvm_pgtable_mm_ops *mm_ops) { - kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(mm_ops->virt_to_phys(childp)); + kvm_pte_t pte = kvm_phys_to_pte(mm_ops->virt_to_phys(childp)); pte |= FIELD_PREP(KVM_PTE_TYPE, KVM_PTE_TYPE_TABLE); pte |= KVM_PTE_VALID; + return pte; +} + +static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp, + struct kvm_pgtable_mm_ops *mm_ops) +{ + kvm_pte_t pte, old = *ptep; + + pte = kvm_init_table_pte(childp, mm_ops); + WARN_ON(kvm_pte_valid(old)); smp_store_release(ptep, pte); } @@ -931,7 +940,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t *old, struct stage2_map_data *data, bool shared) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; - kvm_pte_t *childp; + kvm_pte_t *childp, pte; int ret; if (data->anchor) { @@ -969,9 +978,9 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, * a table. Accesses beyond 'end' that fall within the new table * will be mapped lazily. */ - kvm_set_table_pte(ptep, childp, mm_ops); - mm_ops->get_page(ptep); - *old = *ptep; + pte = kvm_init_table_pte(childp, mm_ops); + stage2_make_pte(ptep, pte, data->mm_ops); + *old = pte; return 0; } From patchwork Fri Apr 15 21:58:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815470 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EF51BC433EF for ; Fri, 15 Apr 2022 22:03:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=pf19ZlEPCfi7zh1fAamqENYwsLn8q26xAo83DxuzDek=; b=Aa8Aav5/Z+I7VYDRFehnqwS4Jp 3f/zUAU/iTCm86LcZQqrhgNLxpNQ+8rFZfmlC8GWZXzth1rrVNBLcFOdRC5bo92TsvO5phT6TUKZx H/qfuKuBt0fpjH8p5V2Ea5zqk7vBhvwB2fTMWbiN3Rg9QAlUeHWCvhAcDxl8yS7LGAEmANxweOo49 5o4JPiKhExy6tIjmHSmOs1wTXu8Sp3IVz05KF1ciYT/g9aRRl4z+mLvG375ov8wjcrfiCw8sbJv0G iSe1HIgLaWvO+eRADbvLtrr+orwj0SG7AICjicBUsZzB8b4wXrbJ93jZRtgFW7P1taT828Vzg+NBe gWJSJLtA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU0z-00BTMZ-VI; Fri, 15 Apr 2022 22:02:06 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyI-00BS3Z-Af for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:20 +0000 Received: by mail-io1-xd49.google.com with SMTP id f11-20020a056602070b00b00645d08010fcso5428982iox.15 for ; Fri, 15 Apr 2022 14:59:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0g9zJiooWDeKugRAQw4FN5FN0/UzyXcPgqHvAqWzKag=; b=oLCH+GSdJvXkd0Gi9+H7vA79g6hjv3mWJ8f1Ky8MSz9VGrXihtRm2Q2mQk94XB6R1V rI0BdGd8LpAzMP3bNbGXHpcJwONP3gy5q638j75OkVYuPzBzMikI3B/GpnP4ZHi6WYgC 1NnjFM6MAYPTaJtge/P7t3M0AOj4J/HW1tzhhrT32UjnHREO3sHO9Q6aRYB6eSSaw14y 1XJYEXTFyIROlajcWQONhn8qbXtAQymvl+UmDZ5XpOcMbLVur2O69+FsbWoSTYPTr05p 24KmSiJ75xiAF16xI5zuCM76equOlcpV45VjmXOI3Dw1o6nycZZCyMKAyu7F35txmRD/ JODA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0g9zJiooWDeKugRAQw4FN5FN0/UzyXcPgqHvAqWzKag=; b=6heaL0CI/VG0x8SH7lnS5Sbqpkowx2pwS6k9Giskfq7okT1Ax+qZwRosDioERomvZg F1Z+u7RWEKwu74SPCTz92Ah99zWs1l9nitCYtNlFIRFsvZ7aLLb7PTSL6whw1xqtyCck qfaqqrkWTWfd0weZZi0P2gF36qxs/vbPVC64cyGZxYIVZqqlIe/mEry7cEbhYwzzuRQQ Gnaf87y3sauJKkfTdYN2JNrvAist1PvpDVuKPe0xd8mFIo6dSbk6FC4vCyEHiOZ4skEU 1dLrTSHsOMGQvfIvr0lh9wwxn2S0BCxFGKG2Y+RP9UYRjF3/Qcb7wvlSV03c5rcU5+Uw k63w== X-Gm-Message-State: AOAM533g+vTsDJBM/zQm3WTubgszMNWJIwKmmBKGJ2QZa/B07WbCFJAr IfJqRSBEpoS8MgE/czb4Wio73Lj22KA= X-Google-Smtp-Source: ABdhPJy38Wy2JIODUbPjdCPdcAqin2kc7u187gx+PAUVY0P1F7nHnp3qVfiaCtdGLADk1YLKoWq+Y853FYQ= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6638:218a:b0:326:6ed7:c011 with SMTP id s10-20020a056638218a00b003266ed7c011mr473714jaj.242.1650059956892; Fri, 15 Apr 2022 14:59:16 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:53 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-10-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 09/17] KVM: arm64: Tear down unlinked page tables in parallel walk From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145918_493916_0AAC8CEF X-CRM114-Status: GOOD ( 15.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Breaking a table pte is insufficient to guarantee ownership of an unlinked subtree. Parallel software walkers could be traversing substructures and changing their mappings. Recurse through the unlinked subtree and lock all descendent ptes to take ownership of the subtree. Since the ptes are actually being evicted, return table ptes back to the table walker to ensure child tables are also traversed. Note that this is done both in both the pre-order and leaf visitors as the underlying pte remains volatile until it is unlinked. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 56 +++++++++++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index ffdfd5ee9642..146fc44acf31 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -838,6 +838,54 @@ static void stage2_make_pte(kvm_pte_t *ptep, kvm_pte_t new, struct kvm_pgtable_m } } +static kvm_pte_t stage2_unlink_pte_shared(kvm_pte_t *ptep) +{ + kvm_pte_t old; + + while (true) { + old = xchg(ptep, KVM_INVALID_PTE_LOCKED); + if (old != KVM_INVALID_PTE_LOCKED) + return old; + + cpu_relax(); + } +} + + +/** + * stage2_unlink_pte() - Tears down an unreachable pte, returning the next pte + * to visit (if any). + * + * @ptep: pointer to the pte to unlink + * @level: page table level of the pte + * @shared: true if the tables are shared by multiple software walkers + * @mm_ops: pointer to the mm ops table + * + * Return: a table pte if another level of recursion is necessary, 0 otherwise. + */ +static kvm_pte_t stage2_unlink_pte(kvm_pte_t *ptep, u32 level, bool shared, + struct kvm_pgtable_mm_ops *mm_ops) +{ + kvm_pte_t old; + + if (shared) { + old = stage2_unlink_pte_shared(ptep); + } else { + old = *ptep; + WRITE_ONCE(*ptep, KVM_INVALID_PTE_LOCKED); + } + + WARN_ON(stage2_pte_is_locked(old)); + + if (kvm_pte_table(old, level)) + return old; + + if (stage2_pte_is_counted(old)) + mm_ops->put_page(ptep); + + return 0; +} + static void stage2_put_pte(kvm_pte_t *ptep, struct kvm_s2_mmu *mmu, u64 addr, u32 level, struct kvm_pgtable_mm_ops *mm_ops) { @@ -922,8 +970,10 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, struct stage2_map_data *data, bool shared) { - if (data->anchor) + if (data->anchor) { + *old = stage2_unlink_pte(ptep, level, shared, data->mm_ops); return 0; + } if (!stage2_leaf_mapping_allowed(addr, end, level, data)) return 0; @@ -944,9 +994,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, int ret; if (data->anchor) { - if (stage2_pte_is_counted(*old)) - mm_ops->put_page(ptep); - + *old = stage2_unlink_pte(ptep, level, shared, data->mm_ops); return 0; } From patchwork Fri Apr 15 21:58:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2241C433F5 for ; Fri, 15 Apr 2022 22:04:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ljYxRGGyxvypdofqjPSGVPz7pRGJBivx1ebAy4SJRIo=; b=XA1r9KmpVMnfqxg9GJEv9usOYG Y/gGEU1wN1QJhTFRQNjwT5Oqrk9dYDkoGt+K2wkLIpNOlwaz602q6qq7M9mF6wc6zpko0v25q+qBe zMtXEJ9hS6T9aBC4gqH2F1PWymYYxYwkI7tfE0OHdohysQPBeGiaadABUV0zrLJlpSx+K4RmpjWGO yD0UuHlkppjVyXVaDXFhbcbc+vfGoQHtMruGOMFVodDEAkHmC7+7/jeBPyVVVjHo/lBlPYkC0Ghgp NLvOxPtBbAMuKH1lCXKIddkCeyavnqmA6flnPFJYgsEjhQqwex6BLKy0f+h0NCZHvJap/u/gfGpiu P90y+7QQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU1l-00BTgM-CE; Fri, 15 Apr 2022 22:02:53 +0000 Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyJ-00BS4F-6C for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:20 +0000 Received: by mail-io1-xd4a.google.com with SMTP id f11-20020a056602070b00b00645d08010fcso5429007iox.15 for ; Fri, 15 Apr 2022 14:59:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=vop5IqucqvM+bii5zgbcIy+JPyxF0keenz7pMhv8m58=; b=jHJtFLgbxSnb6N3tLR+HWD/NrSJB2+DQfaJR6BV6mi7BHclXm3b5DYu3lRIqkK8ZCZ 9Nsc5qvQWjHWPk7VT1sJ3wsYP6XTOuBtzzol+Dyx7Y/G8XVKzTeOZ0wTCYISer2gacY1 1q7hF+P4I9XhHOxrOx0DwqCHp3nlNR7k1lS2WUuwKr6yrpl/wcv2QIQYdgxci9ERkbhN GjVhDkCkSw4jTSLDIZKUru4yoXAgzeVmoDz3uo26yb6h1LXdBYsBdQUdxT5prqHk01wm t/IKRZMZolVSXf5jGA0B71c/6Sed7CUrur83859QSKnETLM/2FiH2WcM4zCc5JAIz871 u5gA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vop5IqucqvM+bii5zgbcIy+JPyxF0keenz7pMhv8m58=; b=KzcgaWa/3sF6mB6N6ouoj2eD3Tp8lhhRM/d2xE8mxPetVlgVCK4Nc5C2UHenmkI1qV qzjDjA4wCVQ2bZJe9nJirO5c7VHihuCk7PiEvmhCRGpStDh9ykJjoS7mS9rId4E+YTSq NLTnLGi/Nl1ZPp78hdPMzBgxFQ9mtJ8EZ8YCIN6k3Y6EfXAlEYmkAuVtn+8qiiQFGTLm XhYmx3pz1LwKkZMpKqGgu24oLqMvj71aT39LzUSl3IkdNDphXdCcWLUYrqF9hMt5JLAp vH1/cHNStHqKKkrWdZCE6d5dyWD8nINnoO+iBjtDuFj8T/xTN7wW640EtwQPLgWOzVkz vciw== X-Gm-Message-State: AOAM531/DCah9knYLMkjF1WOgtW4giVlEYvEMhBDc7tFUNu0zm/Vuj6H 66fidN2g6pt7Ce/9MFdWVry2DFimlT4= X-Google-Smtp-Source: ABdhPJxNjcu2tnVJ1+/YbC82bpFAmroc7SBNGMAr2Bd/O0LGqLCc49cJVFBPbSuomBfNsod9gY/vH0O4fqo= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6638:164b:b0:323:ac42:8d4b with SMTP id a11-20020a056638164b00b00323ac428d4bmr475237jat.75.1650059957848; Fri, 15 Apr 2022 14:59:17 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:54 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-11-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 10/17] KVM: arm64: Assume a table pte is already owned in post-order traversal From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145919_291354_2871B530 X-CRM114-Status: GOOD ( 15.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For parallel walks that collapse a table into a block KVM ensures a locked invalid pte is visible to all observers in pre-order traversal. As such, there is no need to try breaking the pte again. Directly set the pte if it has already been broken. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 146fc44acf31..121818d4c33e 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -924,7 +924,7 @@ static bool stage2_leaf_mapping_allowed(u64 addr, u64 end, u32 level, static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t old, struct stage2_map_data *data, - bool shared) + bool shared, bool locked) { kvm_pte_t new; u64 granule = kvm_granule_size(level), phys = data->phys; @@ -948,7 +948,7 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (!stage2_pte_needs_update(old, new)) return -EAGAIN; - if (!stage2_try_break_pte(ptep, old, addr, level, shared, data)) + if (!locked && !stage2_try_break_pte(ptep, old, addr, level, shared, data)) return -EAGAIN; /* Perform CMOs before installation of the guest stage-2 PTE */ @@ -987,7 +987,8 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, } static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, - kvm_pte_t *old, struct stage2_map_data *data, bool shared) + kvm_pte_t *old, struct stage2_map_data *data, bool shared, + bool locked) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; kvm_pte_t *childp, pte; @@ -998,10 +999,13 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, return 0; } - ret = stage2_map_walker_try_leaf(addr, end, level, ptep, *old, data, shared); + ret = stage2_map_walker_try_leaf(addr, end, level, ptep, *old, data, shared, locked); if (ret != -E2BIG) return ret; + /* We should never attempt installing a table in post-order */ + WARN_ON(locked); + if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1)) return -EINVAL; @@ -1048,7 +1052,13 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, childp = data->childp; data->anchor = NULL; data->childp = NULL; - ret = stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared); + + /* + * We are guaranteed exclusive access to the pte in post-order + * traversal since the locked value was made visible to all + * observers in stage2_map_walk_table_pre. + */ + ret = stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared, true); } else { childp = kvm_pte_follow(*old, mm_ops); } @@ -1087,7 +1097,7 @@ static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_ case KVM_PGTABLE_WALK_TABLE_PRE: return stage2_map_walk_table_pre(addr, end, level, ptep, old, data, shared); case KVM_PGTABLE_WALK_LEAF: - return stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared); + return stage2_map_walk_leaf(addr, end, level, ptep, old, data, shared, false); case KVM_PGTABLE_WALK_TABLE_POST: return stage2_map_walk_table_post(addr, end, level, ptep, old, data, shared); } From patchwork Fri Apr 15 21:58:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815472 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 04A96C433EF for ; Fri, 15 Apr 2022 22:04:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=rwuWSzmwhNSVL/TaO3aqY0danNIVLzS/W9vKqg0h/7w=; b=CUmqcl2xBtvxAmGi7WeYPzVBEZ rAw2GBTi5/8/Nvvx2RbnTWACBJ3Wtw05t7Y/fc7s97tKig+gP38k+RgwBxRlf9sx4oCSClSenripY UDoaaVJNHmqYEQ5xYva3gu/PNhWhl6PVDzKaaa7VPL67q5uMmu+UX8aYRww+b/3bfplqcj0HLvi/d KO+Y0WLn5JFANBwEvrUY07phmqRoKq8d42OTpZpxGqxwuOGLQgKhbeOrdlq/FBIWKwu4sTmOxFX1E KRgzDa8IlddWiXdxRu2a7MZwLh5PCTrGoanOUkQdkZBmidag1CXOfoPJuwlIVko6PWZhoKkCFWzvJ X3pEQIXA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU2E-00BTwN-9V; Fri, 15 Apr 2022 22:03:22 +0000 Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyK-00BS4w-5v for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:21 +0000 Received: by mail-io1-xd4a.google.com with SMTP id c25-20020a5d9399000000b00652e2b23358so1130402iol.6 for ; Fri, 15 Apr 2022 14:59:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ukSikfLZPySq7lKrWdQC3W4tZRZDdV+TXwwmzbO5VKE=; b=UFQNViFm5ywOL6fUUgUmBF2pMIllmvYSsEmxRVNh1gzD1izmJ3LtYUTbQoVRVd6R7/ jqzms2/UXzy9A1cKXaPCa6uGhs2jB4kYpL25R2bC00CDonStKGhvdJ4YVdoFWKmexTi/ QkfTtuq7/OrDuIAAqPw+yUALYUDmuCCrGRxB9F24yb275VyW2/5r9d7nu5//Jrfs1gyU bBbceIMSX7GVKPd/CqxGaga7ODuF8yOc8fr/uf+kbuofz35ev3lh9BLs/jGFbUXJ/kkj 2mjW+x4bwKEomG8LlUwv7ZO+oESMs9ddaaWpZvG/caTzJIQruvv70SZXMOewJ76h8qCh w+3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ukSikfLZPySq7lKrWdQC3W4tZRZDdV+TXwwmzbO5VKE=; b=J1vNwhaYvBejCSXy3oWmBqZ/a0UNCCKdcYYP5wDffeSi164v2n+C3AmmWz07Tif8ve fGQaKF64c5h+o1KmezPovk0jQny5M5//KpJfNHLQIKNJc1lDI2Ezrtn4xbL3oE68Ix6B SB/U4IZuNx3K5awDyUyfcJZv4w7X2bdbRZ1M7XSsthV0LhSNqwbqsO6SDk3GVPcoFsuW 6ZKGaHQKgbPQxqfzmJV19Nko/Amlj+jhduFoxVXrzMLbzC4SMysazbIpQWQFDocZW9kL B8kUHgON7h9hH9k4vyJBnN7YtLC1DyiPdaI2tf2Z5oZ3INlU28r+4RRHVg2uzE7EnfW8 XSLg== X-Gm-Message-State: AOAM5302zVM6y3B+KmvBn8lfF6LaCH3NIEvMRIprfNhyXaFKz6WDrVNC X3HM0L96yo7k1L2CvZxPffuu4EQ/VgQ= X-Google-Smtp-Source: ABdhPJw3+rg28SXpnYzElP/Fh1mOEN5FS2KtRHWJcXruhJlFqtNNF+kNgwwNT/3GJfepUrZ+VWNiihmiJLs= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6e02:1a4f:b0:2c7:a4c8:25f5 with SMTP id u15-20020a056e021a4f00b002c7a4c825f5mr334502ilv.64.1650059958943; Fri, 15 Apr 2022 14:59:18 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:55 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-12-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 11/17] KVM: arm64: Move MMU cache init/destroy into helpers From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145920_304076_2D684DF8 X-CRM114-Status: GOOD ( 10.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org No functional change intended. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_mmu.h | 2 ++ arch/arm64/kvm/arm.c | 4 ++-- arch/arm64/kvm/mmu.c | 10 ++++++++++ 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 74735a864eee..3bb7b678a7e7 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -172,6 +172,8 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu); phys_addr_t kvm_mmu_get_httbr(void); phys_addr_t kvm_get_idmap_vector(void); int kvm_mmu_init(u32 *hyp_va_bits); +void kvm_mmu_vcpu_init(struct kvm_vcpu *vcpu); +void kvm_mmu_vcpu_destroy(struct kvm_vcpu *vcpu); static inline void *__kvm_vector_slot2addr(void *base, enum arm64_hyp_spectre_vector slot) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 523bc934fe2f..f7862fec1595 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -320,7 +320,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.target = -1; bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES); - vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO; + kvm_mmu_vcpu_init(vcpu); /* Set up the timer */ kvm_timer_vcpu_init(vcpu); @@ -349,7 +349,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) if (vcpu_has_run_once(vcpu) && unlikely(!irqchip_in_kernel(vcpu->kvm))) static_branch_dec(&userspace_irqchip_in_use); - kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache); + kvm_mmu_vcpu_destroy(vcpu); kvm_timer_vcpu_terminate(vcpu); kvm_pmu_vcpu_destroy(vcpu); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 53ae2c0640bc..f29d5179196b 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1601,6 +1601,16 @@ int kvm_mmu_init(u32 *hyp_va_bits) return err; } +void kvm_mmu_vcpu_init(struct kvm_vcpu *vcpu) +{ + vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO; +} + +void kvm_mmu_vcpu_destroy(struct kvm_vcpu *vcpu) +{ + kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache); +} + void kvm_arch_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *old, const struct kvm_memory_slot *new, From patchwork Fri Apr 15 21:58:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A0A9CC433F5 for ; Fri, 15 Apr 2022 22:05:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=JGq0Vq+iHYYWdOMKGmZvWQRuzsrjhO3OVa9PP46YANw=; b=j/htdTe1aIbsIX3rMWTlQ0Xubj 2+m0pg/57ZLMLOGzmCcHsqNoOLPw/pP+vZD8crSKn0awORpWB+BFXWEPx3GNT8xT9IJ6GsjEdoRwx GANm0rkB3vK/1JsCfSzd8hVPD2Dl07gztZMatJHpAeW1R84+8026wGztVmuq5P9ZLZf/1bG+kmsgw z6WdhH42QLVKO4/T61Pnn0Zp/UfNxuJSOEly2VHojO6LFoTCrnnYzxfnvhJKJCd3pcujoPkfu0341 Vc5tShZNqPWdXwNSym9jpHHK0RRuNO2RWUuZsDZOu6VzNUWineMAXZdCvdUMElkXeHGSRwgR+p+/j cGczSWjw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU2v-00BUGQ-Of; Fri, 15 Apr 2022 22:04:06 +0000 Received: from mail-oo1-xc49.google.com ([2607:f8b0:4864:20::c49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyL-00BS5k-Lf for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:23 +0000 Received: by mail-oo1-xc49.google.com with SMTP id d3-20020a4aeb83000000b00324f07ebc76so4906799ooj.0 for ; Fri, 15 Apr 2022 14:59:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=22i24DLcyXUieSnW3PAwQCkjoUnvyMWdp2sKJnAUF64=; b=VPtHgLB2eymVJQjQSElXFQ0Zv2gLUgxLry175CS0FCPH1QdUk/jXMMBoPOrHmGBwoq eWdmwHWefSkyKrTkoNNUmWeZAwfQOesyHQ/c4F+nNUh1RnLeH6LvowZHB8nX3nwQ6l3L GEUKEunrQWJsjT9PH9meExTOYNHH3hU7S6IfE1PLsiTpOqRRkTbel/mLpuqamcBbNfG9 GoFAe3BQFqD1GHm76/Rdvr36nNdmSstr6rfNRpwHof4KzfU+AjgBiLREY1NzMY207hNQ sxhnRfFngtq2kuNUX0v2DKU0e2znK8YIUUSoecH7YVoHWITRrIMv7LGu6/DrG+fBMRUD Iqsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=22i24DLcyXUieSnW3PAwQCkjoUnvyMWdp2sKJnAUF64=; b=5FrP08VIUsvgeDBv8nqRrbM57GefKvHz4x3GZ3wFnCyPfLX6Y4pExQ5DB1ynffGgjn vQJLGTjvddJ5zXp6Pe413Hi0aeOU3sr+p9qd8ooSNuUNJacPjwj4B5A7isBnDJMU4ePV 4Gt/hIY9F0HDcSQ+ZEt0QEYEZXH2YI57W05+SQFX+fj00B4jQ7WcvoxyG+/ulVNrWPlz u3UyesERTxmIJ2DfR1YCQ2TGW280GJIJFyKti8mgPoY1U30cEPguPT8+LZ0queve4j9y RuYELBYxDS0MvlKzqNrS7MqcmqcZdO8yaHskVNxoZQrejVZFOoVArKZzekw77H/pqYZ6 +EDg== X-Gm-Message-State: AOAM5338zO6U2H+EQImVGVHXU6pUp5lzq/b4aXH9cvH8y1YTfD18ogQg LQDNmpmyfKF7GdDW2tR6N0nmnzat+Eg= X-Google-Smtp-Source: ABdhPJya0hEPL3+v3IGSw82KDihmFFDLjFiBSt27i0G11hzjmjLYmOhkEY2+c79aK0bz4MZlGmwNEBfJ24k= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6870:639e:b0:e2:ab7c:d868 with SMTP id t30-20020a056870639e00b000e2ab7cd868mr373366oap.108.1650059959809; Fri, 15 Apr 2022 14:59:19 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:56 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-13-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 12/17] KVM: arm64: Stuff mmu page cache in sub struct From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145921_751740_BFBF4B53 X-CRM114-Status: GOOD ( 15.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We're about to add another mmu cache. Stuff the current one in a sub struct so its easier to pass them all to ->zalloc_page(). No functional change intended. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_host.h | 4 +++- arch/arm64/kvm/mmu.c | 14 +++++++------- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 94a27a7520f4..c8947597a619 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -372,7 +372,9 @@ struct kvm_vcpu_arch { bool pause; /* Cache some mmu pages needed inside spinlock regions */ - struct kvm_mmu_memory_cache mmu_page_cache; + struct kvm_mmu_caches { + struct kvm_mmu_memory_cache page_cache; + } mmu_caches; /* Target CPU and feature flags */ int target; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index f29d5179196b..7a588928740a 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -91,10 +91,10 @@ static bool kvm_is_device_pfn(unsigned long pfn) static void *stage2_memcache_zalloc_page(void *arg) { - struct kvm_mmu_memory_cache *mc = arg; + struct kvm_mmu_caches *mmu_caches = arg; /* Allocated with __GFP_ZERO, so no need to zero */ - return kvm_mmu_memory_cache_alloc(mc); + return kvm_mmu_memory_cache_alloc(&mmu_caches->page_cache); } static void *kvm_host_zalloc_pages_exact(size_t size) @@ -1073,7 +1073,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, bool shared; unsigned long mmu_seq; struct kvm *kvm = vcpu->kvm; - struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache; + struct kvm_mmu_caches *mmu_caches = &vcpu->arch.mmu_caches; struct vm_area_struct *vma; short vma_shift; gfn_t gfn; @@ -1160,7 +1160,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * and a write fault needs to collapse a block entry into a table. */ if (fault_status != FSC_PERM || (logging_active && write_fault)) { - ret = kvm_mmu_topup_memory_cache(memcache, + ret = kvm_mmu_topup_memory_cache(&mmu_caches->page_cache, kvm_mmu_cache_min_pages(kvm)); if (ret) return ret; @@ -1273,7 +1273,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize, __pfn_to_phys(pfn), prot, - memcache); + mmu_caches); } /* Mark the page dirty only if the fault is handled successfully */ @@ -1603,12 +1603,12 @@ int kvm_mmu_init(u32 *hyp_va_bits) void kvm_mmu_vcpu_init(struct kvm_vcpu *vcpu) { - vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO; + vcpu->arch.mmu_caches.page_cache.gfp_zero = __GFP_ZERO; } void kvm_mmu_vcpu_destroy(struct kvm_vcpu *vcpu) { - kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache); + kvm_mmu_free_memory_cache(&vcpu->arch.mmu_caches.page_cache); } void kvm_arch_commit_memory_region(struct kvm *kvm, From patchwork Fri Apr 15 21:58:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815474 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 11DF0C433F5 for ; Fri, 15 Apr 2022 22:06:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=kNs0vOJevm5gDHTEwTXXtr22xLFwZdw8EN4+A0VPkBM=; b=yNcVzHzaRB8y9jo1ykwm5BLaux D0usJ1xxOvfOGvZCXdlLz3EPXnVBg9UI8Y3dpp1kZd8qwK7P+vajC9W3NT0wX+gY+JkeSHy69gHXJ 1zsPWL7QoKV7BDgmvDuIvxOI3lRMIPYV0rfmgYU9bxAPJENlZlfUQ6v7JquaUIMjP9fYeeuTaGFLL j6mF8OtViPpkad0nWEXZXYYkxqaItHST5Jw70p+qm8BUifB3z67xrnly/Quj2/RGLfodg4zVzdmqy mDub62PrqQe+G5hCIm5NhQqJb053Ervor0Vp1H4GQYVxwhpF29v/weR3NQfIOe2HcX5dO34wRgIgp 2uij6jVQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU3i-00BUbF-ED; Fri, 15 Apr 2022 22:04:54 +0000 Received: from mail-il1-x14a.google.com ([2607:f8b0:4864:20::14a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyL-00BS6K-Ve for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:24 +0000 Received: by mail-il1-x14a.google.com with SMTP id k2-20020a056e02134200b002caaa88e702so5454925ilr.0 for ; Fri, 15 Apr 2022 14:59:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+7THeEZjrQkI4cL8qs69/ZVQNerdV4AViBcRVZLLx0E=; b=sgJWtGYVInFd1drSIPRRI4aezVykNPn0h3TZjKKiErpQM7sAIHOOKNGcQ0jWe9x2J3 4uorM5vPe/np9qjkIYA4dfw4gtmkWWOabaU8tGatq8w8LsDoUg9+FMK0LVF1fX+QFyw6 i9msKz6hKpt7r66C+xns81BxvuHbanyJZ610bbwgkWkEmS6Xl9AcIZ1z+pnbftljqbuV TZJnMR1dtR7pvNgtOgZtS1KwcCNK2XXzdflPkGu6sg9JhoQgHweTSx9tVIIm2620GETp v/lovjRa15qTioUPiP/HsnkVKkyBFJUAm8nxrDqS7jgG1Smlaj2NKciPmkptKeLE+kdI nEKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+7THeEZjrQkI4cL8qs69/ZVQNerdV4AViBcRVZLLx0E=; b=KsvMDnllk2qQKZBvZoT8+Eq5JTjnTliZY+/Y/AwhLQ6fZ2w7L2ZdfsGwtequC2yhio bj/zh0H/fL/dlJpelX3yoYd34yffox21WJ5fQLtyocbLcZpn3WdvF3ez0mIw4Pi0x5XY 8NXGANThwYg1qVqhnja/fHbpgIP6MPr18xCyO7aru2Vh9Yri/+uDXCUbdywZEHdPi2wI l4eQcjtHm/o/FnL0C9IseGcrmjtYTk6iezQKx/AJ2w/NQ2NzqE/GiweCw/uKY4WZg7U8 ZD/Kb9JAXCcPbwP3OGoZvMbf4j6jv5RT0Xkerf7CjPq+oTGREjDT4wB/hr7FIVxyZ+23 Wk2A== X-Gm-Message-State: AOAM5315TuERX3/vOXqCAQfu8fJ7TwQH/Rl5xla5M1Ckl0nEXze8s9CE L01vGkYLdE0JHTO3m1JyiqZBKLpexlY= X-Google-Smtp-Source: ABdhPJyV71KyX7wIk4SdepWHYsPDO7JEsMOtpP8Ew2zkdCBFEtHJx6sDDgdROjEilaiKBOX2bMTgAgDq5z4= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a5e:8a07:0:b0:64c:8b33:6d19 with SMTP id d7-20020a5e8a07000000b0064c8b336d19mr315910iok.170.1650059960675; Fri, 15 Apr 2022 14:59:20 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:57 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-14-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 13/17] KVM: arm64: Setup cache for stage2 page headers From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145922_076724_6062C6EA X-CRM114-Status: GOOD ( 15.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to punt the last reference drop on a page to an RCU synchronization we need to get a pointer to the page to handle the callback. Set up a memcache for stage2 page headers, but do nothing with it for now. Note that the kmem_cache is never destoyed as it is currently not possible to build KVM/arm64 as a module. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/mmu.c | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index c8947597a619..a640d015790e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -374,6 +374,7 @@ struct kvm_vcpu_arch { /* Cache some mmu pages needed inside spinlock regions */ struct kvm_mmu_caches { struct kvm_mmu_memory_cache page_cache; + struct kvm_mmu_memory_cache header_cache; } mmu_caches; /* Target CPU and feature flags */ diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 7a588928740a..cc6ed6b06ec2 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -31,6 +31,12 @@ static phys_addr_t hyp_idmap_vector; static unsigned long io_map_base; +static struct kmem_cache *stage2_page_header_cache; + +struct stage2_page_header { + struct rcu_head rcu_head; + struct page *page; +}; /* * Release kvm_mmu_lock periodically if the memory region is large. Otherwise, @@ -1164,6 +1170,11 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, kvm_mmu_cache_min_pages(kvm)); if (ret) return ret; + + ret = kvm_mmu_topup_memory_cache(&mmu_caches->header_cache, + kvm_mmu_cache_min_pages(kvm)); + if (ret) + return ret; } mmu_seq = vcpu->kvm->mmu_notifier_seq; @@ -1589,6 +1600,13 @@ int kvm_mmu_init(u32 *hyp_va_bits) if (err) goto out_destroy_pgtable; + stage2_page_header_cache = kmem_cache_create("stage2_page_header", + sizeof(struct stage2_page_header), + 0, SLAB_ACCOUNT, NULL); + + if (!stage2_page_header_cache) + goto out_destroy_pgtable; + io_map_base = hyp_idmap_start; return 0; @@ -1604,11 +1622,13 @@ int kvm_mmu_init(u32 *hyp_va_bits) void kvm_mmu_vcpu_init(struct kvm_vcpu *vcpu) { vcpu->arch.mmu_caches.page_cache.gfp_zero = __GFP_ZERO; + vcpu->arch.mmu_caches.header_cache.kmem_cache = stage2_page_header_cache; } void kvm_mmu_vcpu_destroy(struct kvm_vcpu *vcpu) { kvm_mmu_free_memory_cache(&vcpu->arch.mmu_caches.page_cache); + kvm_mmu_free_memory_cache(&vcpu->arch.mmu_caches.header_cache); } void kvm_arch_commit_memory_region(struct kvm *kvm, From patchwork Fri Apr 15 21:58:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 20130C433F5 for ; Fri, 15 Apr 2022 22:06:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Ch6zrk9Kv9k1XiM7sDrodvMpu1MSXgQL/oftqRO9LIU=; b=zpCbw/08KHLPk+DLMxCrGwrxyS BM481/V8Ml8Osie4o1nxVn5RDS+bie2r2XBSWNGFZA+ot5FMtmEsNDIgAJ9BKaOs1Xpg6i/2dq2Zk iEokWhXZekHt6IgKZ2RUrfbSfQQkp7Fl4srIJU8njygyoOGcdAj7PTo9I0D3sLcQJeLyB143keg4m KtOAhAQBBuvHCCUeF2lsXby2hPTQrcQ/jh0yDNqXkkTiFhI1TaHXM704aYog30BIc2niA9rjqXrNY P8bs/DCFxmI3r3UPj6h1AzZjXQve38rP68+M5lkw05nkPbM/P70EIgiN4oeQTlQ4Frpzxs9Y2MknY m5h3S/JA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU4S-00BUsr-Jw; Fri, 15 Apr 2022 22:05:41 +0000 Received: from mail-il1-x149.google.com ([2607:f8b0:4864:20::149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyN-00BS6w-Pb for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:25 +0000 Received: by mail-il1-x149.google.com with SMTP id j4-20020a92c204000000b002caad37af3fso5408255ilo.22 for ; Fri, 15 Apr 2022 14:59:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Mb/OefWBn9RCKNoTmiaaE/VUMk03Unut8vex1/PGQS0=; b=fsNxIh3BEH+XlmSMjPVMtJeBZUk0FQpQlK3w9p+lwpUBGbcSk9U8YLOg1rOBKVY1uR viQStsQ0f4HjS7zw60Z7a9eHBjDc6y8KRSAjfz92x1WW108wRFezoo18fGyW9sSXBFjN 2njPnnWl4Sr2o3BvCsi7QBaMPCEPmdo7c66TYpshJHac92EsLJ7nryQaBB4BbzWGOmGE IYHOMX5S+Fo9qzysE1iP/+8Q31pXvGIBFwOUa9Uu4JmQUu3iidG5OczCVUkqWrYk1BLJ vJIDxqvp/F8fIVIgSNja8wv/C6uygQfysG5mpeJRy6sVuameHckrI6H2LQ9ArdcD5nwl 2BBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Mb/OefWBn9RCKNoTmiaaE/VUMk03Unut8vex1/PGQS0=; b=vZ4AgW7L5jjVCPL7SmVwhxc3BjzfxAtwAqRVuzTSzmqFFN8UrSzhTHx8bzXLWxD4my 2iFl5Ja1M40uy5An//zw8tFEL4pZTedMd4pUGMXaI+kAeuM4uiR4cyi/AvScGKAgRIPJ iqs0ECSEFWL2s1uSnc74ibqbsbYKeBT7lsaRMf6r77e4VUjOXu8aJEbE1/OBuvycGwsX 8NEoIKiM326vQU2ljonJVS0Biluxs8bUTy7Qv8dzLAH/yP3Su8Y4ybBpziK5TCJ0hgTK UocLvAbVDe0z555Q+CJVKSJt4RyM7S+Ll0lIkW8QSebfSBT7bB4XWFjRlWbtqosM8D4U 3epw== X-Gm-Message-State: AOAM531eTL1iAKUg2cZIUokOUqi5RJ99fra+giB+U18MctSowGdafy/M q5MZBwgad5PPaix+APfVdfkZh+uq78o= X-Google-Smtp-Source: ABdhPJzuFSrhWKcatk7b24EN0VEAajIWqNloCuQY+463KUJzIaWadPCnXDRKjBU8YzbqADXAgMTcRMSd1kw= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6602:14cb:b0:646:3b7d:6aee with SMTP id b11-20020a05660214cb00b006463b7d6aeemr346421iow.178.1650059961830; Fri, 15 Apr 2022 14:59:21 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:58 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-15-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 14/17] KVM: arm64: Punt last page reference to rcu callback for parallel walk From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145923_875768_CA106AE9 X-CRM114-Status: GOOD ( 20.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org It is possible that a table page remains visible to another thread until the next rcu synchronization event. To that end, we cannot drop the last page reference synchronous with post-order traversal for a parallel table walk. Schedule an rcu callback to clean up the child table page for parallel walks. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_pgtable.h | 3 ++ arch/arm64/kvm/hyp/pgtable.c | 24 +++++++++++++-- arch/arm64/kvm/mmu.c | 44 +++++++++++++++++++++++++++- 3 files changed, 67 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 74955aba5918..52e55e00f0ca 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -81,6 +81,8 @@ static inline bool kvm_level_supports_block_mapping(u32 level) * @put_page: Decrement the refcount on a page. When the * refcount reaches 0 the page is automatically * freed. + * @free_table: Drop the last page reference, possibly in the + * next RCU sync if doing a shared walk. * @page_count: Return the refcount of a page. * @phys_to_virt: Convert a physical address into a virtual * address mapped in the current context. @@ -98,6 +100,7 @@ struct kvm_pgtable_mm_ops { void (*get_page)(void *addr); void (*put_page)(void *addr); int (*page_count)(void *addr); + void (*free_table)(void *addr, bool shared); void* (*phys_to_virt)(phys_addr_t phys); phys_addr_t (*virt_to_phys)(void *addr); void (*dcache_clean_inval_poc)(void *addr, size_t size); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 121818d4c33e..a9a48edba63b 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -147,12 +147,19 @@ static inline void kvm_pgtable_walk_end(void) {} #define kvm_dereference_ptep rcu_dereference_raw + +static inline void kvm_pgtable_destroy_barrier(void) +{} + #else #define kvm_pgtable_walk_begin rcu_read_lock #define kvm_pgtable_walk_end rcu_read_unlock #define kvm_dereference_ptep rcu_dereference + +#define kvm_pgtable_destroy_barrier rcu_barrier + #endif static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte, struct kvm_pgtable_mm_ops *mm_ops) @@ -1063,7 +1070,12 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, childp = kvm_pte_follow(*old, mm_ops); } - mm_ops->put_page(childp); + /* + * If we do not have exclusive access to the page tables it is possible + * the unlinked table remains visible to another thread until the next + * rcu synchronization. + */ + mm_ops->free_table(childp, shared); mm_ops->put_page(ptep); return ret; @@ -1203,7 +1215,7 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_granule_size(level)); if (childp) - mm_ops->put_page(childp); + mm_ops->free_table(childp, shared); return 0; } @@ -1433,7 +1445,7 @@ static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, mm_ops->put_page(ptep); if (kvm_pte_table(*old, level)) - mm_ops->put_page(kvm_pte_follow(*old, mm_ops)); + mm_ops->free_table(kvm_pte_follow(*old, mm_ops), shared); return 0; } @@ -1452,4 +1464,10 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt) pgd_sz = kvm_pgd_pages(pgt->ia_bits, pgt->start_level) * PAGE_SIZE; pgt->mm_ops->free_pages_exact(pgt->pgd, pgd_sz); pgt->pgd = NULL; + + /* + * Guarantee that all unlinked subtrees associated with the stage2 page + * table have also been freed before returning. + */ + kvm_pgtable_destroy_barrier(); } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index cc6ed6b06ec2..6ecf37009c21 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -98,9 +98,50 @@ static bool kvm_is_device_pfn(unsigned long pfn) static void *stage2_memcache_zalloc_page(void *arg) { struct kvm_mmu_caches *mmu_caches = arg; + struct stage2_page_header *hdr; + void *addr; /* Allocated with __GFP_ZERO, so no need to zero */ - return kvm_mmu_memory_cache_alloc(&mmu_caches->page_cache); + addr = kvm_mmu_memory_cache_alloc(&mmu_caches->page_cache); + if (!addr) + return NULL; + + hdr = kvm_mmu_memory_cache_alloc(&mmu_caches->header_cache); + if (!hdr) { + free_page((unsigned long)addr); + return NULL; + } + + hdr->page = virt_to_page(addr); + set_page_private(hdr->page, (unsigned long)hdr); + return addr; +} + +static void stage2_free_page_now(struct stage2_page_header *hdr) +{ + WARN_ON(page_ref_count(hdr->page) != 1); + + __free_page(hdr->page); + kmem_cache_free(stage2_page_header_cache, hdr); +} + +static void stage2_free_page_rcu_cb(struct rcu_head *head) +{ + struct stage2_page_header *hdr = container_of(head, struct stage2_page_header, + rcu_head); + + stage2_free_page_now(hdr); +} + +static void stage2_free_table(void *addr, bool shared) +{ + struct page *page = virt_to_page(addr); + struct stage2_page_header *hdr = (struct stage2_page_header *)page_private(page); + + if (shared) + call_rcu(&hdr->rcu_head, stage2_free_page_rcu_cb); + else + stage2_free_page_now(hdr); } static void *kvm_host_zalloc_pages_exact(size_t size) @@ -613,6 +654,7 @@ static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = { .free_pages_exact = free_pages_exact, .get_page = kvm_host_get_page, .put_page = kvm_host_put_page, + .free_table = stage2_free_table, .page_count = kvm_host_page_count, .phys_to_virt = kvm_host_va, .virt_to_phys = kvm_host_pa, From patchwork Fri Apr 15 21:58:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815476 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 85A53C433EF for ; Fri, 15 Apr 2022 22:07:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ZCG3anlkamIUlIpgRUWDwSZAXnikRgoQpcpEXOcTPlo=; b=Q/G+qjUpro+ISl+yMRt/N7PCDE LkQWcpOq7dW4mrb2WV/kori6CTu9LFM927B/oCRHpDcpASL2774sLg7SQ6umuPCCz0EUzAOUxmT1L ai8Fa6ezDpnsBmSrUGYijUQ5XgWdx2zEmdP8XmvYMXb8ijPdrsa8rERfS5BNcw+PrmWtcENcIoBbi bomjiBmHVcZPfBffMGSTGymmuPe5do9kIMchH3ENcSrp2H9YVToZiPOBDpztqy1dwCcDc6jxJU85c S4H43H9PF+ZQSfRWg0paIeq+PRCW3Z6Zs+6oOIPn9XMfV6lNAIJ0RKVMjU/cTNftzboDq59UlN9br 6IocL9pA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU58-00BV8w-QH; Fri, 15 Apr 2022 22:06:23 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyO-00BS7L-1R for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:26 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-2ec0490dc1bso75812657b3.5 for ; Fri, 15 Apr 2022 14:59:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+g21tjl52haZEGL9PeLa8ZsXfugpwy0IPvVVK6CWGsg=; b=ZLsICKC3SmkNR+5tKzOjUbnQi/MrOWshrvj8ba9gD9ErKmF7bM9Ls3/73nj7AN223f 2SqgDq0mAv2Q+0NseF2Q6yJ9vAST2pHisfGt1aJ03ZR+yeXY4WCtFCLFnkoU0PMNx7bv GtmK0jCaxnoFBzIF482k+LXDT3QNYJ5NswQc3tSxLDzrxHjhk7kCcz/EML91XtB0TNBK Xm9rTMKrDR8cvHZo0rs4YCcBNImFSd/Zp6dYdWGcwa6fTHERGWIJByQqZFmWaHjjCdY8 3CDcSDTVe1yeF5TkHlWZQLhTDhaOUlu2ycQ37ChtdjS0d0bGJEyFflrGc5HuZQxrxKqT /V6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+g21tjl52haZEGL9PeLa8ZsXfugpwy0IPvVVK6CWGsg=; b=lboEB/m0aa3yvmcS0tU2vXzuM3fyTRiztcM9mrHOGjfsDggUpTfLu/wWtFis895/8w SDa2PXurufgnIBhFmDitoxVh1LdZejiJwoJf7vn4UpgmBvTOKHOIgIVWLb6fyNWgzc0h Lvmffho0ph2LGC25MzMj+eQX3gZ3cgZwZYF64vtn+oS03/+L9kRK5qPUaT7EyzJDAxSS mQe3ewGWQ/VKBob6ulr6giRynA3ZTonXaCBzBySa/xyPqh/uGzWT+p2/WChxgOfxo8Yp jHYn+jzVzxZpgcd40IOp2pyx0IViz5kBI0nbHj8K0vhJG+EFOGxS31j1KLtnZUKr9ggk dZrw== X-Gm-Message-State: AOAM53084NrK30MRtRJaPSpXywRWrw0k6klLMZXa5Prcbpldeqtei0Ae 6Azdqk0OxCF6ljUlTdfq0t9h+0PSq6E= X-Google-Smtp-Source: ABdhPJxd3FM6U0CPN7En0pTvibRCFHGaVeXWp61fOLSmlPBx3PvTF9e7RZlcLR5fnOduP4TGiTN80GlSguU= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a0d:e743:0:b0:2eb:3106:9b32 with SMTP id q64-20020a0de743000000b002eb31069b32mr940693ywe.512.1650059962733; Fri, 15 Apr 2022 14:59:22 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:59 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-16-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 15/17] KVM: arm64: Allow parallel calls to kvm_pgtable_stage2_map() From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145924_150826_51D58EDA X-CRM114-Status: GOOD ( 14.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The map walker is now appraised of how to walk the tables in parallel with another table walker. Take a parameter indicating whether or not a walk is done in parallel so as to relax the atomicity/locking requirements on ptes. Defer actually using parallel walks to a later change. Signed-off-by: Oliver Upton --- arch/arm64/include/asm/kvm_pgtable.h | 4 +++- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 2 +- arch/arm64/kvm/hyp/pgtable.c | 4 ++-- arch/arm64/kvm/mmu.c | 6 +++--- 4 files changed, 9 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 52e55e00f0ca..9830eea19de4 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -328,6 +328,8 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); * @prot: Permissions and attributes for the mapping. * @mc: Cache of pre-allocated and zeroed memory from which to allocate * page-table pages. + * @shared: true if multiple software walkers could be traversing the tables + * in parallel * * The offset of @addr within a page is ignored, @size is rounded-up to * the next page boundary and @phys is rounded-down to the previous page @@ -349,7 +351,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); */ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot, - void *mc); + void *mc, bool shared); /** * kvm_pgtable_stage2_set_owner() - Unmap and annotate pages in the IPA space to diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 42a5f35cd819..53b172036c2a 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -251,7 +251,7 @@ static inline int __host_stage2_idmap(u64 start, u64 end, enum kvm_pgtable_prot prot) { return kvm_pgtable_stage2_map(&host_kvm.pgt, start, end - start, start, - prot, &host_s2_pool); + prot, &host_s2_pool, false); } /* diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index a9a48edba63b..20ff198ebef7 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1119,7 +1119,7 @@ static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot, - void *mc) + void *mc, bool shared) { int ret; struct stage2_map_data map_data = { @@ -1144,7 +1144,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, if (ret) return ret; - ret = kvm_pgtable_walk(pgt, addr, size, &walker, false); + ret = kvm_pgtable_walk(pgt, addr, size, &walker, shared); dsb(ishst); return ret; } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 6ecf37009c21..63cf18cdb978 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -832,7 +832,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, write_lock(&kvm->mmu_lock); ret = kvm_pgtable_stage2_map(pgt, addr, PAGE_SIZE, pa, prot, - &cache); + &cache, false); write_unlock(&kvm->mmu_lock); if (ret) break; @@ -1326,7 +1326,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize, __pfn_to_phys(pfn), prot, - mmu_caches); + mmu_caches, true); } /* Mark the page dirty only if the fault is handled successfully */ @@ -1526,7 +1526,7 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) */ kvm_pgtable_stage2_map(kvm->arch.mmu.pgt, range->start << PAGE_SHIFT, PAGE_SIZE, __pfn_to_phys(pfn), - KVM_PGTABLE_PROT_R, NULL); + KVM_PGTABLE_PROT_R, NULL, false); return false; } From patchwork Fri Apr 15 21:59:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6FDF5C433EF for ; Fri, 15 Apr 2022 22:08:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=lRatxQllHOqeyb5oJoaM0Gntex1X+zS0kgC19qjnM5M=; b=FiJwVohgTl1dh6befLhR+tq769 GJv4qzItzRhr+dJXL2uo8e3LLDOcvYlU85MALbWxs8j/nqXr3l1A43KE+57XyUSEq6QwFSQel/ThN /KpsefoF/hJnD0gLm42cyuIUKicUoWaNVrZFHGG4kJpfNJg3RHP1R77sC9C3BrtAULZAV9uDkRmZF 35nbya3qiEyrUihQvWOLM4TsXtKvR2WFj+5Dygt7Y3TGRM35vB4fnge/DhdZ/46g1QXQ4aqoAKhMB md0QK4yzNc3X4p8p2UefX96dcH1o2IKJGZcTeIeFFKHA7L1d/NhxuLzMpCLRi+A1Jqhy8letkrgo+ 9lM9imSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU5m-00BVNY-Ot; Fri, 15 Apr 2022 22:07:03 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyO-00BS7b-R4 for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:26 +0000 Received: by mail-io1-xd49.google.com with SMTP id a9-20020a5d89c9000000b0064cb68a9ba6so5444794iot.11 for ; Fri, 15 Apr 2022 14:59:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=we5tUcZWUFX29TKi0tS/Yi5sEjxnPk5PaYrhMJdNw1U=; b=O1SErvMM/5NizynFN7dxIWF/wAOfb8Mq9ZfTSGeW0QBIGQpcAPGMZiYPK5b+AIE+yj 08zA/OxjxYQfyzgWCjGX98P8LIEuoV19f+IbszwnCQUQtevPI3XYw1re2k4fZZe6/vAf Qpexkx6ehQQAsh1RxGlFf+N5b4EPHjVdZkItOeRiv/yIkybMfQ6rKCCTjah2MjXE8ysA UfJUeFHEmDdjsqYhKAbeH01CdWClTD/JQdzWn2NRgu/MbALmGjnCzElEQCYlDtSQfUul EPbBWfGohELG4mrVx0BVrh06dZ/wmMGcdkJN6MSYmZEdZM8jKw5KOVWNj7WfK3JyNrde ComQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=we5tUcZWUFX29TKi0tS/Yi5sEjxnPk5PaYrhMJdNw1U=; b=seTYcuqJY/WVBHsMng9VhQPA7iU9Oa+7Nmmu01oUT2E/KRe9ZrE2gAd2VmkY92YZww JIxIlJLNHuQJPBarrqhhmPudXs7bsk+nbrkJhOXpN+xWE+xgPG6NxWwwiD85wcsXh2lh A3OHKkRxxDkGrJrLPtyTfGW4YAFZ3BH19lRoGjgri9L6+jUcfw2TITOtdRhJs8NTuPCa /zd97QVLqlesbdoMO20KKIWpPWocrRtVXeW8tZ9OUM8Y0/2szY2HWCEjOB+Mef2SZatX aphiq6ZVdc53t1XzGAEZ+3uTA0vdQ/HYP4paQIDU/ObPVMSNXiuY58E7HPGWHCBuZ4zm fcJA== X-Gm-Message-State: AOAM530wRILzxAMgywkgRH4yHwGPVk5lnOUZJ+hoK0hzhAVq4L8X3VY4 lGrww78C+0hImqeBN82CCyLt20KI/3w= X-Google-Smtp-Source: ABdhPJzFhPu6MxIliWrYx7BlAqWDvC1LTp6uI8FxSJyQJeYO+JKxmOQwyxXOX0XwY8FpVX7WQSGMEYCJYH0= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6e02:1807:b0:2ca:4b88:1a42 with SMTP id a7-20020a056e02180700b002ca4b881a42mr312136ilv.258.1650059963514; Fri, 15 Apr 2022 14:59:23 -0700 (PDT) Date: Fri, 15 Apr 2022 21:59:00 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-17-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 16/17] KVM: arm64: Enable parallel stage 2 MMU faults From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145924_929215_D8E08392 X-CRM114-Status: GOOD ( 13.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Voila! Since the map walkers are able to work in parallel there is no need to take the write lock on a stage 2 memory abort. Relax locking on map operations and cross fingers we got it right. Signed-off-by: Oliver Upton --- arch/arm64/kvm/mmu.c | 21 +++------------------ 1 file changed, 3 insertions(+), 18 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 63cf18cdb978..2881051c3743 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1127,7 +1127,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, gfn_t gfn; kvm_pfn_t pfn; bool logging_active = memslot_is_logging(memslot); - bool use_read_lock = false; unsigned long fault_level = kvm_vcpu_trap_get_fault_level(vcpu); unsigned long vma_pagesize, fault_granule; enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R; @@ -1162,8 +1161,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (logging_active) { force_pte = true; vma_shift = PAGE_SHIFT; - use_read_lock = (fault_status == FSC_PERM && write_fault && - fault_granule == PAGE_SIZE); } else { vma_shift = get_vma_page_shift(vma, hva); } @@ -1267,15 +1264,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (exec_fault && device) return -ENOEXEC; - /* - * To reduce MMU contentions and enhance concurrency during dirty - * logging dirty logging, only acquire read lock for permission - * relaxation. - */ - if (use_read_lock) - read_lock(&kvm->mmu_lock); - else - write_lock(&kvm->mmu_lock); + read_lock(&kvm->mmu_lock); + pgt = vcpu->arch.hw_mmu->pgt; if (mmu_notifier_retry(kvm, mmu_seq)) goto out_unlock; @@ -1322,8 +1312,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (fault_status == FSC_PERM && vma_pagesize == fault_granule) { ret = kvm_pgtable_stage2_relax_perms(pgt, fault_ipa, prot); } else { - WARN_ONCE(use_read_lock, "Attempted stage-2 map outside of write lock\n"); - ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize, __pfn_to_phys(pfn), prot, mmu_caches, true); @@ -1336,10 +1324,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, } out_unlock: - if (use_read_lock) - read_unlock(&kvm->mmu_lock); - else - write_unlock(&kvm->mmu_lock); + read_unlock(&kvm->mmu_lock); kvm_set_pfn_accessed(pfn); kvm_release_pfn_clean(pfn); return ret != -EAGAIN ? ret : 0; From patchwork Fri Apr 15 21:59:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815478 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 51F34C433EF for ; Fri, 15 Apr 2022 22:09:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=gamzmLV2uBZhRKMmOc6yIB/A2y3Wbujf/0hDhA8LKwY=; b=TGjimBNqusHIaG6cuU3+/CFTtW jPMtBPb4GbZdp3TnzF1q4JVrddwYFw+VqyKul4CS5UmDof4dnDFzhJW3JFrwULs1KST3r1FB3Z+gv 6E6OYFVfaZbfctfK4lShTCqCTuGGkDLhp8f+2npXAIeDJZBrlHQIlgE4+VBAk8vBudURpgn73HUyb mCHjSJ8ceTP4lv5OXRzPsNbmE2a/acpu+ZjQX8v5eYb9dGfXbWrw6lmPfxfhuLLFVUs9CpJ6EFSB1 grpS8b2FsTwDpFFhk9BGitkLaEZ8EVIw8mP8vrKql8qCtZcG5jmSCIPK1iJUVDDpS0rU8z7a172fN 9RXQAsSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU6V-00BVfJ-4k; Fri, 15 Apr 2022 22:07:47 +0000 Received: from mail-il1-x149.google.com ([2607:f8b0:4864:20::149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyP-00BS8a-NI for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:27 +0000 Received: by mail-il1-x149.google.com with SMTP id v14-20020a056e020f8e00b002caa6a5d918so5417574ilo.15 for ; Fri, 15 Apr 2022 14:59:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=jzvc81PBGj4bV/jHUjtzPtFNqWylMffaehwRBxD6iIE=; b=aWkRKsBqnouPuu0iRgQFmbdZNwM6u65S9STgd8WzLazymunDNq+F/2LV041lFacxkX sH2au5CIL4U6btbR36TcgERHcBL5JA8T93pnN1zy/vCBpJyTdEx4s8aDOU+JwJD7FEgH EDuCMiJ2lKlmqCfo2HCyFA8YBpzh+0As1Im7YSg7MivXDgsyltO1HdVbeLv+yGFmNIPi O5p8sXbUH14m1SBo2J0fuW1TDCouol/p3zaCmN7Pb1tclO6RjKEsMyqqJ0eOLp38uD1h AbzgBBIOOsbBeQk1WFRiKNO/0k6ugAlDvMy5k+FqVLVzTLXvQl7QmS1J6MjDvDSfP9W4 9n1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=jzvc81PBGj4bV/jHUjtzPtFNqWylMffaehwRBxD6iIE=; b=Jxpj2AFmZwekidIXMe5o4oxcgR4/DIs1II+plSzq3k9NWIuy8jln3LrTl82606u9Wv tETXGV3p/CQ4c2nwuKEvKgzTCxVA/i3O40bOmLKX0hZ6LjtLJ7XvSfQR2buwxRGiZgX2 TDAYLOqKIoA5IAfInlnlxrnJaD4TYEZh4kzce99suVAaG6PwUf47jSh/Zpc2ZAFQHTN3 9cKe6SVl2BR9WHwt3+4PasMnEuZ6WpbXrv2mqQW6OD1j097+uyh9xKB/Y6QIpQZcB9Fn nMPmqtfPb6e58h34rec1wz31eX6jj0TZzxCIUu/MTk9DgHsffPbXdpWGFWk5BrNaDN2y h4gQ== X-Gm-Message-State: AOAM532mkctJKxiWtH06SjJQ9gbkoZYo83P9R/389mROLfHOU2LL60HL ZuFXmsTWTT4qJLMBDCY132sbZ/xyAsU= X-Google-Smtp-Source: ABdhPJy0T0D9tPSOS2fkDjAWGdXFAsgYKiBOBw6Vv9SkS2qbIf2KOV78tziChH3+cjdCnnpihxhBubRLicI= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6e02:1b0f:b0:2c7:9a3f:6e84 with SMTP id i15-20020a056e021b0f00b002c79a3f6e84mr312271ilv.32.1650059964470; Fri, 15 Apr 2022 14:59:24 -0700 (PDT) Date: Fri, 15 Apr 2022 21:59:01 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-18-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 17/17] TESTONLY: KVM: arm64: Add super lazy accounting of stage 2 table pages From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145925_823052_9045C5F6 X-CRM114-Status: GOOD ( 12.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Don't use this please. I was just being lazy but wanted to make sure tables are all accounted for. There's a race here too, do you see it? :) Signed-off-by: Oliver Upton --- arch/arm64/kvm/mmu.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 2881051c3743..68ea7f0244fe 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -95,6 +95,8 @@ static bool kvm_is_device_pfn(unsigned long pfn) return !pfn_is_map_memory(pfn); } +static atomic_t stage2_pages = ATOMIC_INIT(0); + static void *stage2_memcache_zalloc_page(void *arg) { struct kvm_mmu_caches *mmu_caches = arg; @@ -112,6 +114,8 @@ static void *stage2_memcache_zalloc_page(void *arg) return NULL; } + atomic_inc(&stage2_pages); + hdr->page = virt_to_page(addr); set_page_private(hdr->page, (unsigned long)hdr); return addr; @@ -121,6 +125,8 @@ static void stage2_free_page_now(struct stage2_page_header *hdr) { WARN_ON(page_ref_count(hdr->page) != 1); + atomic_dec(&stage2_pages); + __free_page(hdr->page); kmem_cache_free(stage2_page_header_cache, hdr); } @@ -662,6 +668,8 @@ static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = { .icache_inval_pou = invalidate_icache_guest_page, }; +static atomic_t stage2_mmus = ATOMIC_INIT(0); + /** * kvm_init_stage2_mmu - Initialise a S2 MMU structure * @kvm: The pointer to the KVM structure @@ -699,6 +707,8 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) for_each_possible_cpu(cpu) *per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1; + atomic_inc(&stage2_mmus); + mmu->pgt = pgt; mmu->pgd_phys = __pa(pgt->pgd); return 0; @@ -796,6 +806,9 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) kvm_pgtable_stage2_destroy(pgt); kfree(pgt); } + + if (atomic_dec_and_test(&stage2_mmus)) + WARN_ON(atomic_read(&stage2_pages)); } /**