From patchwork Fri Apr 15 21:58:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12815470 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EF51BC433EF for ; Fri, 15 Apr 2022 22:03:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=pf19ZlEPCfi7zh1fAamqENYwsLn8q26xAo83DxuzDek=; b=Aa8Aav5/Z+I7VYDRFehnqwS4Jp 3f/zUAU/iTCm86LcZQqrhgNLxpNQ+8rFZfmlC8GWZXzth1rrVNBLcFOdRC5bo92TsvO5phT6TUKZx H/qfuKuBt0fpjH8p5V2Ea5zqk7vBhvwB2fTMWbiN3Rg9QAlUeHWCvhAcDxl8yS7LGAEmANxweOo49 5o4JPiKhExy6tIjmHSmOs1wTXu8Sp3IVz05KF1ciYT/g9aRRl4z+mLvG375ov8wjcrfiCw8sbJv0G iSe1HIgLaWvO+eRADbvLtrr+orwj0SG7AICjicBUsZzB8b4wXrbJ93jZRtgFW7P1taT828Vzg+NBe gWJSJLtA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfU0z-00BTMZ-VI; Fri, 15 Apr 2022 22:02:06 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfTyI-00BS3Z-Af for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 21:59:20 +0000 Received: by mail-io1-xd49.google.com with SMTP id f11-20020a056602070b00b00645d08010fcso5428982iox.15 for ; Fri, 15 Apr 2022 14:59:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0g9zJiooWDeKugRAQw4FN5FN0/UzyXcPgqHvAqWzKag=; b=oLCH+GSdJvXkd0Gi9+H7vA79g6hjv3mWJ8f1Ky8MSz9VGrXihtRm2Q2mQk94XB6R1V rI0BdGd8LpAzMP3bNbGXHpcJwONP3gy5q638j75OkVYuPzBzMikI3B/GpnP4ZHi6WYgC 1NnjFM6MAYPTaJtge/P7t3M0AOj4J/HW1tzhhrT32UjnHREO3sHO9Q6aRYB6eSSaw14y 1XJYEXTFyIROlajcWQONhn8qbXtAQymvl+UmDZ5XpOcMbLVur2O69+FsbWoSTYPTr05p 24KmSiJ75xiAF16xI5zuCM76equOlcpV45VjmXOI3Dw1o6nycZZCyMKAyu7F35txmRD/ JODA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0g9zJiooWDeKugRAQw4FN5FN0/UzyXcPgqHvAqWzKag=; b=6heaL0CI/VG0x8SH7lnS5Sbqpkowx2pwS6k9Giskfq7okT1Ax+qZwRosDioERomvZg F1Z+u7RWEKwu74SPCTz92Ah99zWs1l9nitCYtNlFIRFsvZ7aLLb7PTSL6whw1xqtyCck qfaqqrkWTWfd0weZZi0P2gF36qxs/vbPVC64cyGZxYIVZqqlIe/mEry7cEbhYwzzuRQQ Gnaf87y3sauJKkfTdYN2JNrvAist1PvpDVuKPe0xd8mFIo6dSbk6FC4vCyEHiOZ4skEU 1dLrTSHsOMGQvfIvr0lh9wwxn2S0BCxFGKG2Y+RP9UYRjF3/Qcb7wvlSV03c5rcU5+Uw k63w== X-Gm-Message-State: AOAM533g+vTsDJBM/zQm3WTubgszMNWJIwKmmBKGJ2QZa/B07WbCFJAr IfJqRSBEpoS8MgE/czb4Wio73Lj22KA= X-Google-Smtp-Source: ABdhPJy38Wy2JIODUbPjdCPdcAqin2kc7u187gx+PAUVY0P1F7nHnp3qVfiaCtdGLADk1YLKoWq+Y853FYQ= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6638:218a:b0:326:6ed7:c011 with SMTP id s10-20020a056638218a00b003266ed7c011mr473714jaj.242.1650059956892; Fri, 15 Apr 2022 14:59:16 -0700 (PDT) Date: Fri, 15 Apr 2022 21:58:53 +0000 In-Reply-To: <20220415215901.1737897-1-oupton@google.com> Message-Id: <20220415215901.1737897-10-oupton@google.com> Mime-Version: 1.0 References: <20220415215901.1737897-1-oupton@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [RFC PATCH 09/17] KVM: arm64: Tear down unlinked page tables in parallel walk From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Paolo Bonzini , Sean Christopherson , Ben Gardon , David Matlack , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220415_145918_493916_0AAC8CEF X-CRM114-Status: GOOD ( 15.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Breaking a table pte is insufficient to guarantee ownership of an unlinked subtree. Parallel software walkers could be traversing substructures and changing their mappings. Recurse through the unlinked subtree and lock all descendent ptes to take ownership of the subtree. Since the ptes are actually being evicted, return table ptes back to the table walker to ensure child tables are also traversed. Note that this is done both in both the pre-order and leaf visitors as the underlying pte remains volatile until it is unlinked. Signed-off-by: Oliver Upton --- arch/arm64/kvm/hyp/pgtable.c | 56 +++++++++++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index ffdfd5ee9642..146fc44acf31 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -838,6 +838,54 @@ static void stage2_make_pte(kvm_pte_t *ptep, kvm_pte_t new, struct kvm_pgtable_m } } +static kvm_pte_t stage2_unlink_pte_shared(kvm_pte_t *ptep) +{ + kvm_pte_t old; + + while (true) { + old = xchg(ptep, KVM_INVALID_PTE_LOCKED); + if (old != KVM_INVALID_PTE_LOCKED) + return old; + + cpu_relax(); + } +} + + +/** + * stage2_unlink_pte() - Tears down an unreachable pte, returning the next pte + * to visit (if any). + * + * @ptep: pointer to the pte to unlink + * @level: page table level of the pte + * @shared: true if the tables are shared by multiple software walkers + * @mm_ops: pointer to the mm ops table + * + * Return: a table pte if another level of recursion is necessary, 0 otherwise. + */ +static kvm_pte_t stage2_unlink_pte(kvm_pte_t *ptep, u32 level, bool shared, + struct kvm_pgtable_mm_ops *mm_ops) +{ + kvm_pte_t old; + + if (shared) { + old = stage2_unlink_pte_shared(ptep); + } else { + old = *ptep; + WRITE_ONCE(*ptep, KVM_INVALID_PTE_LOCKED); + } + + WARN_ON(stage2_pte_is_locked(old)); + + if (kvm_pte_table(old, level)) + return old; + + if (stage2_pte_is_counted(old)) + mm_ops->put_page(ptep); + + return 0; +} + static void stage2_put_pte(kvm_pte_t *ptep, struct kvm_s2_mmu *mmu, u64 addr, u32 level, struct kvm_pgtable_mm_ops *mm_ops) { @@ -922,8 +970,10 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, struct stage2_map_data *data, bool shared) { - if (data->anchor) + if (data->anchor) { + *old = stage2_unlink_pte(ptep, level, shared, data->mm_ops); return 0; + } if (!stage2_leaf_mapping_allowed(addr, end, level, data)) return 0; @@ -944,9 +994,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, int ret; if (data->anchor) { - if (stage2_pte_is_counted(*old)) - mm_ops->put_page(ptep); - + *old = stage2_unlink_pte(ptep, level, shared, data->mm_ops); return 0; }