From patchwork Thu May 6 16:52:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shameerali Kolothum Thodi X-Patchwork-Id: 12242777 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8B77C433ED for ; Thu, 6 May 2021 16:56:47 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5556E610A7 for ; Thu, 6 May 2021 16:56:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5556E610A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ZUx7uJw3GtpWMN9lRHnxu+/pT8yIeFEMb/MyuyteUIo=; b=O5lAqjwM+c4LAFCUCgOabg/fz TGblhej/WGRl+SbS3Y2a9yt7saaHUMmhN78Am7qTdHjwKcNBEF5XhB1TmGZAMCPWMjb8EKFjose9n oe4M/h2bslLoTkiEJrusGgm+BOYTXV+dOnBRRoK0f4EHqmNWFCb3lcJofh/8z3ik7EGcbfHw8fPEX xKfqokFcnfo/hW7VLEeLn5DMtRlHwTQIyNNUDYmXB6/pfeW+Yv2eEN69jmKUu9EWqvFpHIGP64BtY G6ecPCY0YA6oIw5Fkmf8C+V7ze/HG5ySfGWwOb757EuYLZI0zHVHjE69i1MzBGBdlvKkLaOI0oXGo DJxI821PA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lehGy-004rz0-PU; Thu, 06 May 2021 16:54:48 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lehGu-004ryQ-TX for linux-arm-kernel@desiato.infradead.org; Thu, 06 May 2021 16:54:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=aHUZroXXSSE8C8nyEfvC/P0qZw4OJlK/iOkVXIWzGGs=; b=lxpfXmAsLTZCM06yIqIxJR6vqb I5hDOM5gekKH1ugxQap3gnO6QmwnQXrirqn0aHr5g8ELHJNQDx5c9zPS11Srg5cs6MqsmRoInZlN0 FR28Ix1YtRIqy1E7h1Q1uxb7eoNokEhxQTuSA4m5Dlx2/ETx2zM6TUJj6Z/F9+gnax0yeYp/toHGt hMWEHMx5dmp4WjZNjw/y97qbCEeT2thq5KzF2+pP4XnTdI4YoRLTOHFHxGZd9Ggqn2UDhYkNbX/dV dW4dZNdYspJWPNUsLsmGhIcJQsDn8GNaYqoSgchmkAIzcDH+nHwJ0z8B2Bszxh5ViSudZypINfm+Z lWEGnegw==; Received: from szxga04-in.huawei.com ([45.249.212.190]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lehGr-006EVO-PX for linux-arm-kernel@lists.infradead.org; Thu, 06 May 2021 16:54:43 +0000 Received: from DGGEMS410-HUB.china.huawei.com (unknown [172.30.72.60]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4FbfkT4f4Vz1BHvj; Fri, 7 May 2021 00:52:01 +0800 (CST) Received: from A2006125610.china.huawei.com (10.47.85.115) by DGGEMS410-HUB.china.huawei.com (10.3.19.210) with Microsoft SMTP Server id 14.3.498.0; Fri, 7 May 2021 00:54:27 +0800 From: Shameer Kolothum To: , , CC: , , , , , , , Subject: [RFC PATCH 1/3] arch/arm64: Introduce a capability to tell whether 16-bit VMID is available Date: Thu, 6 May 2021 17:52:30 +0100 Message-ID: <20210506165232.1969-2-shameerali.kolothum.thodi@huawei.com> X-Mailer: git-send-email 2.12.0.windows.1 In-Reply-To: <20210506165232.1969-1-shameerali.kolothum.thodi@huawei.com> References: <20210506165232.1969-1-shameerali.kolothum.thodi@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.85.115] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210506_095442_025779_3338AFFF X-CRM114-Status: GOOD ( 12.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Julien Grall At the moment, the function kvm_get_vmid_bits() is looking up for the sanitized value of ID_AA64MMFR1_EL1 and extract the information regarding the number of VMID bits supported. This is fine as the function is mainly used during VMID roll-over. New use in a follow-up patch will require the function to be called a every context switch so we want the function to be more efficient. A new capability is introduced to tell whether 16-bit VMID is available. Signed-off-by: Julien Grall Signed-off-by: Shameer Kolothum --- arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/include/asm/kvm_mmu.h | 4 +--- arch/arm64/kernel/cpufeature.c | 9 +++++++++ 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index c40f2490cd7b..acb92da5c254 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -67,7 +67,8 @@ #define ARM64_HAS_LDAPR 59 #define ARM64_KVM_PROTECTED_MODE 60 #define ARM64_WORKAROUND_NVIDIA_CARMEL_CNP 61 +#define ARM64_HAS_16BIT_VMID 62 -#define ARM64_NCAPS 62 +#define ARM64_NCAPS 63 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 90873851f677..c3080966ef83 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -213,9 +213,7 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled); static inline unsigned int kvm_get_vmid_bits(void) { - int reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1); - - return get_vmid_bits(reg); + return cpus_have_const_cap(ARM64_HAS_16BIT_VMID) ? 16 : 8; } /* diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index e5281e1c8f1d..ff956fb2f712 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2203,6 +2203,15 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, .min_field_value = 1, }, + { + .capability = ARM64_HAS_16BIT_VMID, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .sys_reg = SYS_ID_AA64MMFR1_EL1, + .field_pos = ID_AA64MMFR1_VMIDBITS_SHIFT, + .sign = FTR_UNSIGNED, + .min_field_value = ID_AA64MMFR1_VMIDBITS_16, + .matches = has_cpuid_feature, + }, {}, }; From patchwork Thu May 6 16:52:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shameerali Kolothum Thodi X-Patchwork-Id: 12242781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE3E1C433ED for ; Thu, 6 May 2021 16:56:58 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5E3FA61090 for ; Thu, 6 May 2021 16:56:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5E3FA61090 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Mv7uaxxtcjs4hovSl0BuB/N3RvIc513RXWe1hQc6kks=; b=m446qtGLaNxmeYraPsvqPu0/Y Tm5Nv5dvNX7t3QzY+vofK3W3kdIdC+B6nckHJIKznsaoDDEMynbvlZeVFdmqyNeLpRwwNNcWK2nyy AruYEqbc44bieweOMbjCVXj1fi2B2AaYvThiD/kqfaQeuIjwtYhFq7jxu54oQUBHNlr4L6y/AcYq4 k/IwHW9Yun0q67gp1K6/TNfH/L348gTAZ/p+xwez7O/EKmBLTuwSpyU98uMZShIJvetEbMOb5IQjI 4ElVejTTuG3EfJFguOf29BpjtZN1OZzzErfS6PgjYcd5p1w/0Wl7EJaSGzr7yF7MO9532qITsDt3J q7gv9ZVvA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lehHI-004s57-Au; Thu, 06 May 2021 16:55:08 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lehH4-004rzK-8V for linux-arm-kernel@desiato.infradead.org; Thu, 06 May 2021 16:54:54 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=VyZEMCKeToeh9tbzAoZ3/So11mHXT1Y8wiNVzVHz7PE=; b=dpjcO5BO4100GvTdv/eDFqrwE0 4ArF7vwSwVlO4mw0v9YLned72W+7UZyqKhhp7hm0j3beZgKIQndoRyk+jSdnuxuQtRU1amJsJ+Pm6 RHJgXlcANtwzVa4ywhjxrfsaPblGHxWQG0HXdh43v3ulRDwTphTaJLhNsaH01Gg4lyjp6q43dCwmq PPoCxsb7lTMenxTJLuFnaDt9rrbbXVTdlI28MqWawfjVeqe+vBumji5BX2HVfFYd3Y1tDBQQIwxju KE4kDicngATpdY02U3kzYdf6Qw1z3LkP4Xm/qBA6oesfXnKdGrMQFeHBCFqfQ6HhQHvx4IjrbVVy6 j36GxvLA==; Received: from szxga06-in.huawei.com ([45.249.212.32]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lehH0-006EWC-Os for linux-arm-kernel@lists.infradead.org; Thu, 06 May 2021 16:54:53 +0000 Received: from DGGEMS410-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4FbflC6j1qzlc5G; Fri, 7 May 2021 00:52:39 +0800 (CST) Received: from A2006125610.china.huawei.com (10.47.85.115) by DGGEMS410-HUB.china.huawei.com (10.3.19.210) with Microsoft SMTP Server id 14.3.498.0; Fri, 7 May 2021 00:54:38 +0800 From: Shameer Kolothum To: , , CC: , , , , , , , Subject: [RFC PATCH 2/3] kvm/arm: Introduce a new vmid allocator for KVM Date: Thu, 6 May 2021 17:52:31 +0100 Message-ID: <20210506165232.1969-3-shameerali.kolothum.thodi@huawei.com> X-Mailer: git-send-email 2.12.0.windows.1 In-Reply-To: <20210506165232.1969-1-shameerali.kolothum.thodi@huawei.com> References: <20210506165232.1969-1-shameerali.kolothum.thodi@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.85.115] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210506_095451_169440_28CB5B0A X-CRM114-Status: GOOD ( 30.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This is based on arm64 asid allocator algorithm and has duplicated most of the code here. Signed-off-by: Shameer Kolothum --- arch/arm64/include/asm/kvm_host.h | 6 + arch/arm64/kvm/vmid.c | 285 ++++++++++++++++++++++++++++++ 2 files changed, 291 insertions(+) create mode 100644 arch/arm64/kvm/vmid.c diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 3d10e6527f7d..9d476f1f34af 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -670,6 +670,12 @@ int kvm_arm_pvtime_get_attr(struct kvm_vcpu *vcpu, int kvm_arm_pvtime_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr); +int kvm_arm_vmid_alloc_init(void); +void kvm_arm_vmid_alloc_free(void); +void kvm_arm_update_vmid(atomic64_t *id, refcount_t *pinned); +unsigned long kvm_arm_pinned_vmid_get(atomic64_t *id, refcount_t *pinned); +void kvm_arm_pinned_vmid_put(atomic64_t *id, refcount_t *pinned); + static inline void kvm_arm_pvtime_vcpu_init(struct kvm_vcpu_arch *vcpu_arch) { vcpu_arch->steal.base = GPA_INVALID; diff --git a/arch/arm64/kvm/vmid.c b/arch/arm64/kvm/vmid.c new file mode 100644 index 000000000000..14854c786e00 --- /dev/null +++ b/arch/arm64/kvm/vmid.c @@ -0,0 +1,285 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * VMID allocator. + * + * Based on arch/arm64/mm/context.c + * + * Copyright (C) 2002-2003 Deep Blue Solutions Ltd, all rights reserved. + * Copyright (C) 2012 ARM Ltd. + */ + +#include +#include + +#include +#include + +static u32 vmid_bits; +static DEFINE_RAW_SPINLOCK(cpu_vmid_lock); + +static atomic64_t vmid_generation; +static unsigned long *vmid_map; + +static DEFINE_PER_CPU(atomic64_t, active_vmids); +static DEFINE_PER_CPU(u64, reserved_vmids); +static cpumask_t tlb_flush_pending; + +static unsigned long max_pinned_vmids; +static unsigned long nr_pinned_vmids; +static unsigned long *pinned_vmid_map; + +#define VMID_MASK (~GENMASK(vmid_bits - 1, 0)) +#define VMID_FIRST_VERSION (1UL << vmid_bits) + +#define NUM_USER_VMIDS VMID_FIRST_VERSION +#define vmid2idx(vmid) ((vmid) & ~VMID_MASK) +#define idx2vmid(idx) vmid2idx(idx) + +#define vmid_gen_match(vmid) \ + (!(((vmid) ^ atomic64_read(&vmid_generation)) >> vmid_bits)) + +static void flush_context(void) +{ + int cpu; + u64 vmid; + + if (pinned_vmid_map) + bitmap_copy(vmid_map, pinned_vmid_map, NUM_USER_VMIDS); + else + bitmap_clear(vmid_map, 0, NUM_USER_VMIDS); + + for_each_possible_cpu(cpu) { + vmid = atomic64_xchg_relaxed(&per_cpu(active_vmids, cpu), 0); + /* + * If this CPU has already been through a + * rollover, but hasn't run another task in + * the meantime, we must preserve its reserved + * VMID, as this is the only trace we have of + * the process it is still running. + */ + if (vmid == 0) + vmid = per_cpu(reserved_vmids, cpu); + __set_bit(vmid2idx(vmid), vmid_map); + per_cpu(reserved_vmids, cpu) = vmid; + } + + /* + * Queue a TLB invalidation for each CPU to perform on next + * context-switch + */ + cpumask_setall(&tlb_flush_pending); +} + +static bool check_update_reserved_vmid(u64 vmid, u64 newvmid) +{ + int cpu; + bool hit = false; + + /* + * Iterate over the set of reserved VMIDs looking for a match. + * If we find one, then we can update our mm to use newvmid + * (i.e. the same VMID in the current generation) but we can't + * exit the loop early, since we need to ensure that all copies + * of the old VMID are updated to reflect the mm. Failure to do + * so could result in us missing the reserved VMID in a future + * generation. + */ + for_each_possible_cpu(cpu) { + if (per_cpu(reserved_vmids, cpu) == vmid) { + hit = true; + per_cpu(reserved_vmids, cpu) = newvmid; + } + } + + return hit; +} + +static u64 new_vmid(atomic64_t *id, refcount_t *pinned) +{ + static u32 cur_idx = 1; + u64 vmid = atomic64_read(id); + u64 generation = atomic64_read(&vmid_generation); + + if (vmid != 0) { + u64 newvmid = generation | (vmid & ~VMID_MASK); + + /* + * If our current VMID was active during a rollover, we + * can continue to use it and this was just a false alarm. + */ + if (check_update_reserved_vmid(vmid, newvmid)) + return newvmid; + + /* + * If it is pinned, we can keep using it. Note that reserved + * takes priority, because even if it is also pinned, we need to + * update the generation into the reserved_vmids. + */ + if (pinned && refcount_read(pinned)) + return newvmid; + + /* + * We had a valid VMID in a previous life, so try to re-use + * it if possible. + */ + if (!__test_and_set_bit(vmid2idx(vmid), vmid_map)) + return newvmid; + } + + /* + * Allocate a free VMID. If we can't find one, take a note of the + * currently active VMIDs and mark the TLBs as requiring flushes. We + * always count from VMID #2 (index 1), as we use VMID #0 for host. + */ + vmid = find_next_zero_bit(vmid_map, NUM_USER_VMIDS, cur_idx); + if (vmid != NUM_USER_VMIDS) + goto set_vmid; + + /* We're out of VMIDs, so increment the global generation count */ + generation = atomic64_add_return_relaxed(VMID_FIRST_VERSION, + &vmid_generation); + flush_context(); + + /* We have more VMIDs than CPUs, so this will always succeed */ + vmid = find_next_zero_bit(vmid_map, NUM_USER_VMIDS, 1); + +set_vmid: + __set_bit(vmid, vmid_map); + cur_idx = vmid; + return idx2vmid(vmid) | generation; +} + +void kvm_arm_update_vmid(atomic64_t *id, refcount_t *pinned) +{ + unsigned long flags; + unsigned int cpu; + u64 vmid, old_active_vmid; + + vmid = atomic64_read(id); + + /* + * The memory ordering here is subtle. + * If our active_vmids is non-zero and the VMID matches the current + * generation, then we update the active_vmids entry with a relaxed + * cmpxchg. Racing with a concurrent rollover means that either: + * + * - We get a zero back from the cmpxchg and end up waiting on the + * lock. Taking the lock synchronises with the rollover and so + * we are forced to see the updated generation. + * + * - We get a valid VMID back from the cmpxchg, which means the + * relaxed xchg in flush_context will treat us as reserved + * because atomic RmWs are totally ordered for a given location. + */ + old_active_vmid = atomic64_read(this_cpu_ptr(&active_vmids)); + if (old_active_vmid && vmid_gen_match(vmid) && + atomic64_cmpxchg_relaxed(this_cpu_ptr(&active_vmids), + old_active_vmid, vmid)) + return; + + raw_spin_lock_irqsave(&cpu_vmid_lock, flags); + /* Check that our VMID belongs to the current generation. */ + vmid = atomic64_read(id); + if (!vmid_gen_match(vmid)) { + vmid = new_vmid(id, pinned); + atomic64_set(id, vmid); + } + + cpu = smp_processor_id(); + if (cpumask_test_and_clear_cpu(cpu, &tlb_flush_pending)) + kvm_call_hyp(__kvm_tlb_flush_local_all); + + atomic64_set(this_cpu_ptr(&active_vmids), vmid); + raw_spin_unlock_irqrestore(&cpu_vmid_lock, flags); +} + +unsigned long kvm_arm_pinned_vmid_get(atomic64_t *id, refcount_t *pinned) +{ + unsigned long flags; + u64 vmid; + + if (!pinned_vmid_map) + return 0; + + raw_spin_lock_irqsave(&cpu_vmid_lock, flags); + + vmid = atomic64_read(id); + + if (refcount_inc_not_zero(pinned)) + goto out_unlock; + + if (nr_pinned_vmids >= max_pinned_vmids) { + vmid = 0; + goto out_unlock; + } + + if (!vmid_gen_match(vmid)) { + /* + * We went through one or more rollover since that VMID was + * used. Ensure that it is still valid, or generate a new one. + */ + vmid = new_vmid(id, pinned); + atomic64_set(id, vmid); + } + + nr_pinned_vmids++; + __set_bit(vmid2idx(vmid), pinned_vmid_map); + refcount_set(pinned, 1); + +out_unlock: + raw_spin_unlock_irqrestore(&cpu_vmid_lock, flags); + + vmid &= ~VMID_MASK; + + return vmid; +} + +void kvm_arm_pinned_vmid_put(atomic64_t *id, refcount_t *pinned) +{ + unsigned long flags; + u64 vmid = atomic64_read(id); + + if (!pinned_vmid_map) + return; + + raw_spin_lock_irqsave(&cpu_vmid_lock, flags); + + if (refcount_dec_and_test(pinned)) { + __clear_bit(vmid2idx(vmid), pinned_vmid_map); + nr_pinned_vmids--; + } + + raw_spin_unlock_irqrestore(&cpu_vmid_lock, flags); +} + +/* + * Initialize the VMID allocator + */ +int kvm_arm_vmid_alloc_init(void) +{ + vmid_bits = kvm_get_vmid_bits(); + + /* + * Expect allocation after rollover to fail if we don't have at least + * one more VMID than CPUs. VMID #0 is always reserved. + */ + WARN_ON(NUM_USER_VMIDS - 1 <= num_possible_cpus()); + atomic64_set(&vmid_generation, VMID_FIRST_VERSION); + vmid_map = kcalloc(BITS_TO_LONGS(NUM_USER_VMIDS), + sizeof(*vmid_map), GFP_KERNEL); + if (!vmid_map) + return -ENOMEM; + + pinned_vmid_map = kcalloc(BITS_TO_LONGS(NUM_USER_VMIDS), + sizeof(*pinned_vmid_map), GFP_KERNEL); + nr_pinned_vmids = 0; + max_pinned_vmids = NUM_USER_VMIDS - num_possible_cpus() - 2; + + return 0; +} + +void kvm_arm_vmid_alloc_free(void) +{ + kfree(vmid_map); + kfree(pinned_vmid_map); +} From patchwork Thu May 6 16:52:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shameerali Kolothum Thodi X-Patchwork-Id: 12242783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 295D7C433ED for ; Thu, 6 May 2021 16:57:12 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 94C0961090 for ; Thu, 6 May 2021 16:57:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 94C0961090 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=tLz85PYc4su9LMdC1kiElrVSgtzpBA5ZwBkkQIRZ3AE=; b=kPITBxQVuobAC3FILFHp09BHQ sMZOVDS7jsgIjuNDP0u3lj1YMK+FZcp/S/Rx2PvqcLKdNIVgDXgbf9d7lGpd6Q3u62Fv3kKhhdiN9 1SAEjwuiJvrJf97Ug/Rw9JiOy/V6eeFZyoxlO5Bh3TJjaPJBeXJ9X2IW6pEmJpOi6/KdsWe5jqG2f xJVnb4m9R/ZRgX69+f6l62+V2eTM/rUYNiEKeGblKn/ediwlhuHDloWWoPhgtc+vpGIHcu1xMiBln UI5BGRAdOKPUjhDAifV9c5+iPIb1ut09eoEsN1sMsHKNli48Gez/QbfGxHIxmrcB0NtFuh9TW1C9/ iLRk5PXfw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lehHS-004s76-Ak; Thu, 06 May 2021 16:55:18 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lehH9-004s0A-0X for linux-arm-kernel@desiato.infradead.org; Thu, 06 May 2021 16:54:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=SVja5xwkQWDDcB3oCohIZ1DxGnTE6yel98Hq66r48NE=; b=R2SnkGHur+cxtZt2oyMdFDjzgN 5gw0bZFU23GBf9APFNGLNgST493xhbPZx42vPw1Y6MhATH1Xzv9hR0btYLG3N1MsroOgU2ZpeNpQ4 hhHeAG7AdLUglT8QeGJKRtE4Pjlgry/494a3Kv+OU6yAjmhYzMFGW1nHliLP7GK6uCGhiYUS7nNDT bwM0NXtjzBl2nGp50vcFQQlWhywqWvwWJtnOoLXXYCuO+SXpkJBWU9johCW8+lO//ZhofzaXrhQbc O9JPDqHNSvMdHz/16FnldDkRYD3Xjr7FIqAouG/4HIalFOxAYgqs/SrIthwDjPdl6ldmF0P5Sucgh WQVs8HrA==; Received: from szxga05-in.huawei.com ([45.249.212.191]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lehH5-006EWt-CB for linux-arm-kernel@lists.infradead.org; Thu, 06 May 2021 16:54:57 +0000 Received: from DGGEMS410-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4Fbfjz391HzQjt7; Fri, 7 May 2021 00:51:35 +0800 (CST) Received: from A2006125610.china.huawei.com (10.47.85.115) by DGGEMS410-HUB.china.huawei.com (10.3.19.210) with Microsoft SMTP Server id 14.3.498.0; Fri, 7 May 2021 00:54:44 +0800 From: Shameer Kolothum To: , , CC: , , , , , , , Subject: [RFC PATCH 3/3] kvm/arm: Align the VMID allocation with the arm64 ASID one Date: Thu, 6 May 2021 17:52:32 +0100 Message-ID: <20210506165232.1969-4-shameerali.kolothum.thodi@huawei.com> X-Mailer: git-send-email 2.12.0.windows.1 In-Reply-To: <20210506165232.1969-1-shameerali.kolothum.thodi@huawei.com> References: <20210506165232.1969-1-shameerali.kolothum.thodi@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.85.115] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210506_095455_788877_B96FD826 X-CRM114-Status: GOOD ( 33.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Julien Grall At the moment, the VMID algorithm will send an SGI to all the CPUs to force an exit and then broadcast a full TLB flush and I-Cache invalidation. This patch use the new VMID allocator. The benefits are: - CPUs are not forced to exit at roll-over. Instead the VMID will be marked reserved and the context will be flushed at next exit. This will reduce the IPIs traffic. - Context invalidation is now per-CPU rather than broadcasted. - Catalin has a formal model of the ASID allocator. With the new algo, the code is now adapted: - The function __kvm_flush_vm_context() has been renamed to __kvm_tlb_flush_local_all() and now only flushing the current CPU context. - The call to update_vmid() will be done with preemption disabled as the new algo requires to store information per-CPU. - The TLBs associated to EL1 will be flushed when booting a CPU to deal with stale information. This was previously done on the allocation of the first VMID of a new generation. Signed-off-by: Julien Grall Signed-off-by: Shameer Kolothum --- arch/arm64/include/asm/kvm_asm.h | 4 +- arch/arm64/include/asm/kvm_host.h | 5 +- arch/arm64/include/asm/kvm_mmu.h | 3 +- arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 115 ++++++++--------------------- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 6 +- arch/arm64/kvm/hyp/nvhe/tlb.c | 10 +-- arch/arm64/kvm/hyp/vhe/tlb.c | 10 +-- arch/arm64/kvm/mmu.c | 1 - 9 files changed, 51 insertions(+), 105 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index a7ab84f781f7..29697c5ab2c2 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -44,7 +44,7 @@ #define __KVM_HOST_SMCCC_FUNC___kvm_hyp_init 0 #define __KVM_HOST_SMCCC_FUNC___kvm_vcpu_run 1 -#define __KVM_HOST_SMCCC_FUNC___kvm_flush_vm_context 2 +#define __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_local_all 2 #define __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa 3 #define __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid 4 #define __KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context 5 @@ -182,7 +182,7 @@ DECLARE_KVM_NVHE_SYM(__per_cpu_end); DECLARE_KVM_HYP_SYM(__bp_harden_hyp_vecs); #define __bp_harden_hyp_vecs CHOOSE_HYP_SYM(__bp_harden_hyp_vecs) -extern void __kvm_flush_vm_context(void); +extern void __kvm_tlb_flush_local_all(void); extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, int level); diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 9d476f1f34af..c06370f387cb 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -70,9 +70,7 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu); void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu); struct kvm_vmid { - /* The VMID generation used for the virt. memory system */ - u64 vmid_gen; - u32 vmid; + atomic64_t id; }; struct kvm_s2_mmu { @@ -631,7 +629,6 @@ void kvm_arm_resume_guest(struct kvm *kvm); ret; \ }) -void force_vm_exit(const cpumask_t *mask); void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot); int handle_exit(struct kvm_vcpu *vcpu, int exception_index); diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index c3080966ef83..43e83df87e3a 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -252,7 +252,8 @@ static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) u64 cnp = system_supports_cnp() ? VTTBR_CNP_BIT : 0; baddr = mmu->pgd_phys; - vmid_field = (u64)vmid->vmid << VTTBR_VMID_SHIFT; + vmid_field = atomic64_read(&vmid->id) << VTTBR_VMID_SHIFT; + vmid_field &= VTTBR_VMID_MASK(kvm_get_vmid_bits()); return kvm_phys_to_vttbr(baddr) | vmid_field | cnp; } diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 589921392cb1..717c4cbf557a 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -16,7 +16,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ inject_fault.o va_layout.o handle_exit.o \ guest.o debug.o reset.o sys_regs.o \ vgic-sys-reg-v3.o fpsimd.o pmu.o \ - arch_timer.o trng.o\ + arch_timer.o trng.o vmid.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 7f06ba76698d..4f58db358f72 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -55,11 +55,6 @@ static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); unsigned long kvm_arm_hyp_percpu_base[NR_CPUS]; DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); -/* The VMID used in the VTTBR */ -static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1); -static u32 kvm_next_vmid; -static DEFINE_SPINLOCK(kvm_vmid_lock); - static bool vgic_present; static DEFINE_PER_CPU(unsigned char, kvm_arm_hardware_enabled); @@ -486,85 +481,13 @@ bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu) return vcpu_mode_priv(vcpu); } -/* Just ensure a guest exit from a particular CPU */ -static void exit_vm_noop(void *info) -{ -} - -void force_vm_exit(const cpumask_t *mask) -{ - preempt_disable(); - smp_call_function_many(mask, exit_vm_noop, NULL, true); - preempt_enable(); -} - -/** - * need_new_vmid_gen - check that the VMID is still valid - * @vmid: The VMID to check - * - * return true if there is a new generation of VMIDs being used - * - * The hardware supports a limited set of values with the value zero reserved - * for the host, so we check if an assigned value belongs to a previous - * generation, which requires us to assign a new value. If we're the first to - * use a VMID for the new generation, we must flush necessary caches and TLBs - * on all CPUs. - */ -static bool need_new_vmid_gen(struct kvm_vmid *vmid) -{ - u64 current_vmid_gen = atomic64_read(&kvm_vmid_gen); - smp_rmb(); /* Orders read of kvm_vmid_gen and kvm->arch.vmid */ - return unlikely(READ_ONCE(vmid->vmid_gen) != current_vmid_gen); -} - /** * update_vmid - Update the vmid with a valid VMID for the current generation * @vmid: The stage-2 VMID information struct */ static void update_vmid(struct kvm_vmid *vmid) { - if (!need_new_vmid_gen(vmid)) - return; - - spin_lock(&kvm_vmid_lock); - - /* - * We need to re-check the vmid_gen here to ensure that if another vcpu - * already allocated a valid vmid for this vm, then this vcpu should - * use the same vmid. - */ - if (!need_new_vmid_gen(vmid)) { - spin_unlock(&kvm_vmid_lock); - return; - } - - /* First user of a new VMID generation? */ - if (unlikely(kvm_next_vmid == 0)) { - atomic64_inc(&kvm_vmid_gen); - kvm_next_vmid = 1; - - /* - * On SMP we know no other CPUs can use this CPU's or each - * other's VMID after force_vm_exit returns since the - * kvm_vmid_lock blocks them from reentry to the guest. - */ - force_vm_exit(cpu_all_mask); - /* - * Now broadcast TLB + ICACHE invalidation over the inner - * shareable domain to make sure all data structures are - * clean. - */ - kvm_call_hyp(__kvm_flush_vm_context); - } - - vmid->vmid = kvm_next_vmid; - kvm_next_vmid++; - kvm_next_vmid &= (1 << kvm_get_vmid_bits()) - 1; - - smp_wmb(); - WRITE_ONCE(vmid->vmid_gen, atomic64_read(&kvm_vmid_gen)); - - spin_unlock(&kvm_vmid_lock); + kvm_arm_update_vmid(&vmid->id, NULL); } static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu) @@ -728,8 +651,6 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) */ cond_resched(); - update_vmid(&vcpu->arch.hw_mmu->vmid); - check_vcpu_requests(vcpu); /* @@ -739,6 +660,15 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) */ preempt_disable(); + /* + * The VMID allocator only tracks active VMIDs per + * physical CPU, and therefore the VMID allocated may not be + * preserved on VMID roll-over if the task was preempted, + * making a thread's VMID inactive. So we need to call + * update_vttbr in non-premptible context. + */ + update_vmid(&vcpu->arch.hw_mmu->vmid); + kvm_pmu_flush_hwstate(vcpu); local_irq_disable(); @@ -777,8 +707,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) */ smp_store_mb(vcpu->mode, IN_GUEST_MODE); - if (ret <= 0 || need_new_vmid_gen(&vcpu->arch.hw_mmu->vmid) || - kvm_request_pending(vcpu)) { + if (ret <= 0 || kvm_request_pending(vcpu)) { vcpu->mode = OUTSIDE_GUEST_MODE; isb(); /* Ensure work in x_flush_hwstate is committed */ kvm_pmu_sync_hwstate(vcpu); @@ -1460,6 +1389,8 @@ static void cpu_hyp_reset(void) { if (!is_kernel_in_hyp_mode()) __hyp_reset_vectors(); + + kvm_call_hyp(__kvm_tlb_flush_local_all); } /* @@ -1635,9 +1566,26 @@ static bool init_psci_relay(void) static int init_common_resources(void) { + int err; + + /* + * Initialize the VMID allocator telling it to allocate a single + * VMID per VM. + */ + err = kvm_arm_vmid_alloc_init(); + if (err) { + kvm_err("Failed to initialize VMID allocator.\n"); + return err; + } + return kvm_set_ipa_limit(); } +static void free_common_resources(void) +{ + kvm_arm_vmid_alloc_free(); +} + static int init_subsystems(void) { int err = 0; @@ -1918,7 +1866,7 @@ int kvm_arch_init(void *opaque) err = kvm_arm_init_sve(); if (err) - return err; + goto out_err; if (!in_hyp_mode) { err = init_hyp_mode(); @@ -1952,6 +1900,7 @@ int kvm_arch_init(void *opaque) if (!in_hyp_mode) teardown_hyp_mode(); out_err: + free_common_resources(); return err; } diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 936328207bde..62027448d534 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -25,9 +25,9 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt) cpu_reg(host_ctxt, 1) = __kvm_vcpu_run(kern_hyp_va(vcpu)); } -static void handle___kvm_flush_vm_context(struct kvm_cpu_context *host_ctxt) +static void handle___kvm_tlb_flush_local_all(struct kvm_cpu_context *host_ctxt) { - __kvm_flush_vm_context(); + __kvm_tlb_flush_local_all(); } static void handle___kvm_tlb_flush_vmid_ipa(struct kvm_cpu_context *host_ctxt) @@ -112,7 +112,7 @@ typedef void (*hcall_t)(struct kvm_cpu_context *); static const hcall_t host_hcall[] = { HANDLE_FUNC(__kvm_vcpu_run), - HANDLE_FUNC(__kvm_flush_vm_context), + HANDLE_FUNC(__kvm_tlb_flush_local_all), HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa), HANDLE_FUNC(__kvm_tlb_flush_vmid), HANDLE_FUNC(__kvm_flush_cpu_context), diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index 229b06748c20..3f1fc5125e9e 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -138,10 +138,10 @@ void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu) __tlb_switch_to_host(&cxt); } -void __kvm_flush_vm_context(void) +void __kvm_tlb_flush_local_all(void) { - dsb(ishst); - __tlbi(alle1is); + dsb(nshst); + __tlbi(alle1); /* * VIPT and PIPT caches are not affected by VMID, so no maintenance @@ -153,7 +153,7 @@ void __kvm_flush_vm_context(void) * */ if (icache_is_vpipt()) - asm volatile("ic ialluis"); + asm volatile("ic iallu" : : ); - dsb(ish); + dsb(nsh); } diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index 66f17349f0c3..89f229e77b7d 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -142,10 +142,10 @@ void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu) __tlb_switch_to_host(&cxt); } -void __kvm_flush_vm_context(void) +void __kvm_tlb_flush_local_all(void) { - dsb(ishst); - __tlbi(alle1is); + dsb(nshst); + __tlbi(alle1); /* * VIPT and PIPT caches are not affected by VMID, so no maintenance @@ -157,7 +157,7 @@ void __kvm_flush_vm_context(void) * */ if (icache_is_vpipt()) - asm volatile("ic ialluis"); + asm volatile("ic iallu" : : ); - dsb(ish); + dsb(nsh); } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 8711894db8c2..4933fc9a13fb 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -390,7 +390,6 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) mmu->kvm = kvm; mmu->pgt = pgt; mmu->pgd_phys = __pa(pgt->pgd); - mmu->vmid.vmid_gen = 0; return 0; out_destroy_pgtable: