From patchwork Mon Jan 9 21:53:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13094416 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7F728C67871 for ; Mon, 9 Jan 2023 21:57:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=4SlbL+mhMENCkfr9VneKZBDEBpONWoeSdfO01kw7fVM=; b=tNp28LvRO2hJpOyuAnGE/o9gF+ sBsN2NKzhR+rFhk6JINkjzYyswsfbwCXvSQlELi//LqWzPMtQM99kDwqDmyinNMOBXch6DBrVaYCw cLB6JiqVBiYmUigUPlSJOcQwjRDu3gMM55pP/BtCK7Rj6ZFFC99jGBUHKlsPvv82+uoW9p9qGD3y4 DjI8mEkUBMqAEnR/DCv/5HU7sXAYvs/ljLoHNsQjdFEj+T+907g/AEiMBih23uBt7SssvQ+uxaO1H q9Ea2fZnbUWLHvDMHxRnjyE/XVB4HsYir8R+ZNmm6NccQZcTCnpmkQuHHoUXPvUPFqNgIiI4vs9FN J+tUmOjA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF083-004NgY-OY; Mon, 09 Jan 2023 21:56:28 +0000 Received: from mail-il1-x149.google.com ([2607:f8b0:4864:20::149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF05l-004MrL-0i for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 21:54:06 +0000 Received: by mail-il1-x149.google.com with SMTP id r6-20020a92cd86000000b00304b2d1c2d7so7054289ilb.11 for ; Mon, 09 Jan 2023 13:54:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/kazqzODthv1qfFn0x+pjMq50iNJTv/+JbuwMSlo7YE=; b=NKkixVE4ytkVG7mwZgERD13bjl2m+lUGpZMCqnWHry9zfsPflpQq1rVJc74OuU1ia2 2CEmJO3mSmu3vSxAL4xtBu8+TupJ6cZjnt9Y5E+7povPyx3w6iTFO2cTh11afuuKuNSl jZS6pJFmGXWkUcEKCpYGcRWJAOgmMxQ3jNhfcy0XL+0pxWPM0J8UaKlC+ipUUTXSL/SP feUeudKhQwCLOSAXy2HxarZze74aLnKBHhe5PIpoiHgJb5xy93TOwrjHhPU5cUB/dh4q MNh9KR5MmosQRkbyCQFryoGEM6urWXA/eEMzSR3/WxUAh/cNMaq6nv1oJrMWUmyxixcS qmzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/kazqzODthv1qfFn0x+pjMq50iNJTv/+JbuwMSlo7YE=; b=a6bq7cU5Y+HueI8hzU2YfWuN3oX55xFgAQteFtSJGX8wfj2e2LKDluWpDHarsUaDDl zTVGvEemYV3Bmrap2mpu/HJbq0Mhx6tX7mqFtaOA/M6wOaO6dJaYdfQpOtZAxd2/4tvb KOkzZYvErjUKt1f+Vex2Igab8FMVynDz4kyNZa+ku9thhpOUNXK/bL3RakFV8b21MFR9 C/HOcny9XdF5+VeTy+Tn0vaslDu0knaVUo7sxFiy/dtSGZtLNL/6vXeBBmbSlYMFnXTb w/XpxsJl6CaruIlG4AUMd7Uj/j23hXxbfvDYUDC8Uv0ytG3KZp9ICXl24AE7fZftPLKH hMtw== X-Gm-Message-State: AFqh2kqIpQCr4yZd3P8/eXjSIHR+MQg++rNzf/q/JlBAdPBIL9IhNK0W SrQy4oKt2A/NrJ8LwSHegSZJgo6SXRmM X-Google-Smtp-Source: AMrXdXvonqWAUpO95LxSDHvsAVVCuFauYrn8ITjqvbJGve5IrTypBrOzuR8Yn/DJ97ucl1mwY1BTfPeRWhSv X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a05:6e02:1c28:b0:30b:fe74:8e15 with SMTP id m8-20020a056e021c2800b0030bfe748e15mr6475266ilh.11.1673301242581; Mon, 09 Jan 2023 13:54:02 -0800 (PST) Date: Mon, 9 Jan 2023 21:53:42 +0000 In-Reply-To: <20230109215347.3119271-1-rananta@google.com> Mime-Version: 1.0 References: <20230109215347.3119271-1-rananta@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109215347.3119271-2-rananta@google.com> Subject: [RFC PATCH 1/6] arm64: tlb: Refactor the core flush algorithm of __flush_tlb_range From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , Ricardo Koller , Reiji Watanabe , James Morse , Alexandru Elisei , Suzuki K Poulose Cc: Paolo Bonzini , Catalin Marinas , Will Deacon , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_135405_093291_6686CFF8 X-CRM114-Status: GOOD ( 18.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the core TLB flush functionality of __flush_tlb_range() hardcodes vae1is (and variants) for the flush operation. In the upcoming patches, the KVM code reuses this core algorithm with ipas2e1is for range based TLB invalidations based on the IPA. Hence, extract the core flush functionality of __flush_tlb_range() into its own macro that accepts an 'op' argument to pass any TLBI instruction. No functional changes intended. Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/include/asm/tlbflush.h | 107 +++++++++++++++--------------- 1 file changed, 54 insertions(+), 53 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 412a3b9a3c25d..9a57eae14e576 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -278,14 +278,60 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, */ #define MAX_TLBI_OPS PTRS_PER_PTE +/* When the CPU does not support TLB range operations, flush the TLB + * entries one by one at the granularity of 'stride'. If the TLB + * range ops are supported, then: + * + * 1. If 'pages' is odd, flush the first page through non-range + * operations; + * + * 2. For remaining pages: the minimum range granularity is decided + * by 'scale', so multiple range TLBI operations may be required. + * Start from scale = 0, flush the corresponding number of pages + * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it + * until no pages left. + * + * Note that certain ranges can be represented by either num = 31 and + * scale or num = 0 and scale + 1. The loop below favours the latter + * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. + */ +#define __flush_tlb_range_op(op, start, pages, stride, asid, tlb_level, tlbi_user) do { \ + int num = 0; \ + int scale = 0; \ + unsigned long addr; \ + \ + while (pages > 0) { \ + if (!system_supports_tlb_range() || \ + pages % 2 == 1) { \ + addr = __TLBI_VADDR(start, asid); \ + __tlbi_level(op, addr, tlb_level); \ + if (tlbi_user) \ + __tlbi_user_level(op, addr, tlb_level); \ + start += stride; \ + pages -= stride >> PAGE_SHIFT; \ + continue; \ + } \ + \ + num = __TLBI_RANGE_NUM(pages, scale); \ + if (num >= 0) { \ + addr = __TLBI_VADDR_RANGE(start, asid, scale, \ + num, tlb_level); \ + __tlbi(r##op, addr); \ + if (tlbi_user) \ + __tlbi_user(r##op, addr); \ + start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; \ + pages -= __TLBI_RANGE_PAGES(num, scale); \ + } \ + scale++; \ + } \ +} while (0) + static inline void __flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) { - int num = 0; - int scale = 0; - unsigned long asid, addr, pages; + unsigned long asid, pages; start = round_down(start, stride); end = round_up(end, stride); @@ -307,56 +353,11 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, dsb(ishst); asid = ASID(vma->vm_mm); - /* - * When the CPU does not support TLB range operations, flush the TLB - * entries one by one at the granularity of 'stride'. If the TLB - * range ops are supported, then: - * - * 1. If 'pages' is odd, flush the first page through non-range - * operations; - * - * 2. For remaining pages: the minimum range granularity is decided - * by 'scale', so multiple range TLBI operations may be required. - * Start from scale = 0, flush the corresponding number of pages - * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it - * until no pages left. - * - * Note that certain ranges can be represented by either num = 31 and - * scale or num = 0 and scale + 1. The loop below favours the latter - * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. - */ - while (pages > 0) { - if (!system_supports_tlb_range() || - pages % 2 == 1) { - addr = __TLBI_VADDR(start, asid); - if (last_level) { - __tlbi_level(vale1is, addr, tlb_level); - __tlbi_user_level(vale1is, addr, tlb_level); - } else { - __tlbi_level(vae1is, addr, tlb_level); - __tlbi_user_level(vae1is, addr, tlb_level); - } - start += stride; - pages -= stride >> PAGE_SHIFT; - continue; - } - - num = __TLBI_RANGE_NUM(pages, scale); - if (num >= 0) { - addr = __TLBI_VADDR_RANGE(start, asid, scale, - num, tlb_level); - if (last_level) { - __tlbi(rvale1is, addr); - __tlbi_user(rvale1is, addr); - } else { - __tlbi(rvae1is, addr); - __tlbi_user(rvae1is, addr); - } - start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; - pages -= __TLBI_RANGE_PAGES(num, scale); - } - scale++; - } + if (last_level) + __flush_tlb_range_op(vale1is, start, pages, stride, asid, tlb_level, true); + else + __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true); + dsb(ish); } From patchwork Mon Jan 9 21:53:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13094417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E500AC5479D for ; Mon, 9 Jan 2023 21:58:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=HtG5/GIqsgu8KMiZM/X7R9WJh/J7ELTmrIigxcO7CfA=; b=MmgXMKmZp63YyMLdkoi0UfYtZb 2FN04qEx/OwESP3Zt2XLle3m4VY1vNrUb7qwkrBzHHM83wWXvdG+A+PloEiifttOaxldsBLBPy3P4 nGIQo6N//E8vep4gm9N5WfBw+106g+QJi1uCr2UwLbjYWDQmm0f0LGEjRIuIwRDCRewnWQeVQxAmO nL6c5aClGHSfAX27w7+Per60VGny0b3JGq6tR1UfkM99DqKCh1OMCOHfouXWjQ7MNOZLs+hvLBHB4 JKd3IzBDyqZ6TpfpKCQFfEOHVv8hqMuqsdWb7vVLrga6c7l7pTGSzFx8L3OWXSF8wkhHoMjoP4UZc p4fZodkQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF096-004O3Q-AH; Mon, 09 Jan 2023 21:57:32 +0000 Received: from mail-il1-x14a.google.com ([2607:f8b0:4864:20::14a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF05n-004MsW-EV for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 21:54:09 +0000 Received: by mail-il1-x14a.google.com with SMTP id i14-20020a056e020d8e00b003034b93bd07so7059253ilj.14 for ; Mon, 09 Jan 2023 13:54:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MF+FfAxmoC6dFnj4ddByp++95c4ioR7dqQUFBF7r1tA=; b=KAaLiFLNSM2c203YeUTO7jF6xO8EhU/ISPtEeD5TD+CwxmpXoDhWvszR4cpvw0hbov XgSW91l2fld0hCkSrFubd+0ZVKstaNXvS2OObcyoZMl1C0Q/3091JKrAbJ1xtAzLYSZn +12PoMfix+0cqHZ0FWT6jhefdQ7qbQWxgqKSWvcydwnkGpi36ex064jnqxQXGcoagQED wmFS0SmXhsFi/MWc5uTsq9bPcd8HRE/fBqDCIg0Uxld930nL0GHYNBsLIKFAL0Dh45mN ipSG795V+G3ctkaDMs/fS4Zs7WckCYmyk7VP/PsR7QuY2Merv00NdFHgdLDlsW7DufCR jcVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MF+FfAxmoC6dFnj4ddByp++95c4ioR7dqQUFBF7r1tA=; b=y4cHW/1/5KCBhM+PB4hVjHVhOf30bftxdt/ZnULOSX+cyyb5yf/i/P+OGhM+D7JBZ9 vJGLc6eaPeHeIcr2SuaJV53+CUoUa3jmsFtev3hJIrnvCAuU3glRSqYZjBMU47vvxi0h b/Ph92ARu7ThI+Jy0J4No3Qc7pgk+lOSZUNhk73IZPXGWTi0VHFjBpDS1EsGjFRKjbUL R6v6mkn/+MQDX1lRwCxVC8ZEF8LxJROFmrsKEYXEgVX1NbDvmUcO4elkJB6TYCbOYCOr +HNNN4uwHJhGYZASIPKL4H0Ybs8xx0yik778T1xZiHbnwzbY84gewlILP42CUGGbAT9g fFBg== X-Gm-Message-State: AFqh2kp1p16IxArhIHxpg29dr4fpJEkm7e7Y2aFA6tNUPl6R0WorFUps B7+cDkDJB7iVua64+ZDvWUr2IYGClcGW X-Google-Smtp-Source: AMrXdXvsMsw3ht7evpmsM3wj6mrDWGrXj8WzBYsxkxo6YHxdcq+zXrV8iCa7YF2HBD1+Zwf/eh64SHJ4FoDT X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a02:cc4b:0:b0:375:c16b:7776 with SMTP id i11-20020a02cc4b000000b00375c16b7776mr6011988jaq.54.1673301244565; Mon, 09 Jan 2023 13:54:04 -0800 (PST) Date: Mon, 9 Jan 2023 21:53:43 +0000 In-Reply-To: <20230109215347.3119271-1-rananta@google.com> Mime-Version: 1.0 References: <20230109215347.3119271-1-rananta@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109215347.3119271-3-rananta@google.com> Subject: [RFC PATCH 2/6] KVM: arm64: Add support for FEAT_TLBIRANGE From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , Ricardo Koller , Reiji Watanabe , James Morse , Alexandru Elisei , Suzuki K Poulose Cc: Paolo Bonzini , Catalin Marinas , Will Deacon , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_135407_542570_2CBDFB9A X-CRM114-Status: GOOD ( 15.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Define a generic function __kvm_tlb_flush_range() to invalidate the TLBs over a range of addresses. Use this to define __kvm_tlb_flush_range_vmid_ipa() (for VHE and nVHE) to flush a range of stage-2 page-tables using IPA in one go. If the system supports FEAT_TLBIRANGE, the following patches would conviniently replace global TLBI such as vmalls12e1is in the map, unmap, and dirty-logging paths with ripas2e1is instead. Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/include/asm/kvm_asm.h | 21 +++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 +++++++++++ arch/arm64/kvm/hyp/nvhe/tlb.c | 24 ++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/tlb.c | 20 ++++++++++++++++++++ 4 files changed, 76 insertions(+) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 43c3bc0f9544d..bdf94ae0333b0 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -79,6 +79,7 @@ enum __kvm_host_smccc_func { __KVM_HOST_SMCCC_FUNC___pkvm_init_vm, __KVM_HOST_SMCCC_FUNC___pkvm_init_vcpu, __KVM_HOST_SMCCC_FUNC___pkvm_teardown_vm, + __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_range_vmid_ipa, }; #define DECLARE_KVM_VHE_SYM(sym) extern char sym[] @@ -221,10 +222,30 @@ DECLARE_KVM_NVHE_SYM(__per_cpu_end); DECLARE_KVM_HYP_SYM(__bp_harden_hyp_vecs); #define __bp_harden_hyp_vecs CHOOSE_HYP_SYM(__bp_harden_hyp_vecs) +#define __kvm_tlb_flush_range(op, mmu, start, end, tlb_level) do { \ + unsigned long pages, stride; \ + \ + stride = PAGE_SIZE; \ + start = round_down(start, stride); \ + end = round_up(end, stride); \ + pages = (end - start) >> PAGE_SHIFT; \ + \ + if ((!system_supports_tlb_range() && \ + (end - start) >= (MAX_TLBI_OPS * stride)) || \ + pages >= MAX_TLBI_RANGE_PAGES) { \ + __kvm_tlb_flush_vmid(mmu); \ + break; \ + } \ + \ + __flush_tlb_range_op(op, start, pages, stride, 0, tlb_level, false); \ +} while (0) + extern void __kvm_flush_vm_context(void); extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, int level); +extern void __kvm_tlb_flush_range_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t start, + phys_addr_t end, int level); extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_timer_set_cntvoff(u64 cntvoff); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 728e01d4536b0..ac52d0fbb9719 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -116,6 +116,16 @@ static void handle___kvm_flush_vm_context(struct kvm_cpu_context *host_ctxt) __kvm_flush_vm_context(); } +static void handle___kvm_tlb_flush_range_vmid_ipa(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); + DECLARE_REG(phys_addr_t, start, host_ctxt, 2); + DECLARE_REG(phys_addr_t, end, host_ctxt, 3); + DECLARE_REG(int, level, host_ctxt, 4); + + __kvm_tlb_flush_range_vmid_ipa(kern_hyp_va(mmu), start, end, level); +} + static void handle___kvm_tlb_flush_vmid_ipa(struct kvm_cpu_context *host_ctxt) { DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); @@ -314,6 +324,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__kvm_adjust_pc), HANDLE_FUNC(__kvm_vcpu_run), HANDLE_FUNC(__kvm_flush_vm_context), + HANDLE_FUNC(__kvm_tlb_flush_range_vmid_ipa), HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa), HANDLE_FUNC(__kvm_tlb_flush_vmid), HANDLE_FUNC(__kvm_flush_cpu_context), diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index d296d617f5896..292f5c4834d08 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -55,6 +55,30 @@ static void __tlb_switch_to_host(struct tlb_inv_context *cxt) } } +void __kvm_tlb_flush_range_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t start, + phys_addr_t end, int level) +{ + struct tlb_inv_context cxt; + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + __kvm_tlb_flush_range(ipas2e1is, mmu, start, end, level); + + dsb(ish); + __tlbi(vmalle1is); + dsb(ish); + isb(); + + /* See the comment below in __kvm_tlb_flush_vmid_ipa() */ + if (icache_is_vpipt()) + icache_inval_all_pou(); + + __tlb_switch_to_host(&cxt); +} + void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, int level) { diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index 24cef9b87f9e9..2631cc09e4184 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -79,6 +79,26 @@ static void __tlb_switch_to_host(struct tlb_inv_context *cxt) local_irq_restore(cxt->flags); } +void __kvm_tlb_flush_range_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t start, + phys_addr_t end, int level) +{ + struct tlb_inv_context cxt; + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + __kvm_tlb_flush_range(ipas2e1is, mmu, start, end, level); + + dsb(ish); + __tlbi(vmalle1is); + dsb(ish); + isb(); + + __tlb_switch_to_host(&cxt); +} + void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, int level) { From patchwork Mon Jan 9 21:53:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13094418 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 09453C5479D for ; Mon, 9 Jan 2023 21:59:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ERaxEDTDMnmGnZAWWcAg89e2keGTeHaklG6WfFU82DI=; b=PGEW9SzLpg4zWOjnmrRYoABiYs mhi4e1ZouuckfhgfIgnOmPCxj/L0GdNI5JvFkIabKpjpHSKKVwUlvNYhb8r1M5pZ9Uzo8zhjVTPGV j6Zx+I1zS6UCjNoTxlms4WXvFyRMz6JgEfFvURS3dO8SYPRV9tkbOPYDFCrA5SeMjuhZ7WvbOOtO1 L+ZcpY3XqG7mDr33CqQDO+92L+WvyCoU3BIrZnnRHqrQjpRlW+34XSEptV1d+7Gnp0RvL4QgsF7Lk mVzC7rB/MXqScLpMy6juYcyp6yVdHLbBdFH7ErsKGQxer+t2lAr4kST01gv2TlQ6DIakGayG83QQT pMw1jGZQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF09y-004OOV-QG; Mon, 09 Jan 2023 21:58:27 +0000 Received: from mail-oi1-x24a.google.com ([2607:f8b0:4864:20::24a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF05o-004MtN-Hc for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 21:54:10 +0000 Received: by mail-oi1-x24a.google.com with SMTP id t25-20020a056808159900b0035ecfd3fa78so3103880oiw.4 for ; Mon, 09 Jan 2023 13:54:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SDYN0MRDIF+SnOeyd4pyYdGIKCrHRaNe9q+tU9g7tHk=; b=JpgU+lBoUCLbZ56Jvj7Pn/vdi78vU5U3QuKtvIBxucZgLmZuMLmd88ZuQTUzyBKvak 6URYiypCIW+TZt6tpj3vahCJkJfZ/bLR5/iBFmcgS2xJWZjSs5Km1XmDOCZFVyYNrNdF aQxMRNm9CoVdpDALfBC57x8+EFA1cfuvZqgDxyITXdGCIOEqTxQiAuVy9uJ1cxGO/3EV abfT9ocI5Otykw8Nadc9eXPbqcQZXXaZrNB0el9+yvluuWfZF1VWhrmpIVzBQ0tBz6CD +6oL0xnc06/xPuEhNIL+kfwdmFZFTaFbgKl3SlvFFlUqQ7wMRKAH+9CjY0D5URbPHfBd jhMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SDYN0MRDIF+SnOeyd4pyYdGIKCrHRaNe9q+tU9g7tHk=; b=TN9DcHI3ucKqX5wBF6LTGxZG3uJyrUgpeOsvnEqkHMpsfWN9CfrD+HL3mUSV5C3lDR 2ovFDknd+bEJkg6cGfA4rdYIBXqRe7VHlskzCWS4Qvc8l30F7MMR0ZHw/wooOq5bieNB vC7nutnnIa5ZGAHuMncVtwp+kJYRQ3PpuX0/LxyS40nGQGn/MBSBzWNmVJq1qGPKLd6E ZYOMP7V2CnTxjg2o5adWljAzUwjhsinbDlLH19KLZ62e2IyTbghsG5ylYgNOlvHrxho8 /waMsBMkK9Bu8Ss+X1fsRVoo9WMJE6IK+Syc9i9CE1AYcRfx4u0UdT/9SYVG7joBxx/c wG1A== X-Gm-Message-State: AFqh2kqPw5rElOdhnjgm3UBG3aSgT4M1VKSo+VAjJhMD6qpngIk/61xJ 7UvpAX255H6Fdp7hhyF6Vgj+BhpiRxY6 X-Google-Smtp-Source: AMrXdXtrkhTOwelbRPQw7yNs/JV3tuDLBxCLoimzlrlEKqCeF4IooRkE3J40GXTjhloRi+UlLLpIxIlpLZyo X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a05:6808:1a1d:b0:364:532d:215a with SMTP id bk29-20020a0568081a1d00b00364532d215amr195559oib.187.1673301246427; Mon, 09 Jan 2023 13:54:06 -0800 (PST) Date: Mon, 9 Jan 2023 21:53:44 +0000 In-Reply-To: <20230109215347.3119271-1-rananta@google.com> Mime-Version: 1.0 References: <20230109215347.3119271-1-rananta@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109215347.3119271-4-rananta@google.com> Subject: [RFC PATCH 3/6] KVM: Define kvm_flush_remote_tlbs_range From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , Ricardo Koller , Reiji Watanabe , James Morse , Alexandru Elisei , Suzuki K Poulose Cc: Paolo Bonzini , Catalin Marinas , Will Deacon , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_135408_645122_2494FF25 X-CRM114-Status: GOOD ( 12.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Define kvm_flush_remote_tlbs_range() to limit the TLB flush only to a certain range of addresses. Replace this with the existing call to kvm_flush_remote_tlbs() in the MMU notifier path. Architectures such as arm64 can define this to flush only the necessary addresses, instead of the entire range. Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/kvm/mmu.c | 10 ++++++++++ include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 7 ++++++- 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 39d9a334efb57..70f76bc909c5d 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -91,6 +91,16 @@ void kvm_flush_remote_tlbs(struct kvm *kvm) kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); } +void kvm_flush_remote_tlbs_range(struct kvm *kvm, unsigned long start, unsigned long end) +{ + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; + + if (system_supports_tlb_range()) + kvm_call_hyp(__kvm_tlb_flush_range_vmid_ipa, mmu, start, end, 0); + else + kvm_flush_remote_tlbs(kvm); +} + static bool kvm_is_device_pfn(unsigned long pfn) { return !pfn_is_map_memory(pfn); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index f51eb9419bfc3..a76cede9dc3bb 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1359,6 +1359,7 @@ int kvm_vcpu_yield_to(struct kvm_vcpu *target); void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool usermode_vcpu_not_eligible); void kvm_flush_remote_tlbs(struct kvm *kvm); +void kvm_flush_remote_tlbs_range(struct kvm *kvm, unsigned long start, unsigned long end); #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE int kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int min); diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 03e6a38094c17..f538ecc984f5b 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -376,6 +376,11 @@ void kvm_flush_remote_tlbs(struct kvm *kvm) ++kvm->stat.generic.remote_tlb_flush; } EXPORT_SYMBOL_GPL(kvm_flush_remote_tlbs); + +void kvm_flush_remote_tlbs_range(struct kvm *kvm, unsigned long start, unsigned long end) +{ + kvm_flush_remote_tlbs(kvm); +} #endif static void kvm_flush_shadow_all(struct kvm *kvm) @@ -637,7 +642,7 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, } if (range->flush_on_ret && ret) - kvm_flush_remote_tlbs(kvm); + kvm_flush_remote_tlbs_range(kvm, range->start, range->end - 1); if (locked) { KVM_MMU_UNLOCK(kvm); From patchwork Mon Jan 9 21:53:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13094419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 364B1C54EBD for ; Mon, 9 Jan 2023 22:00:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=NJCbH9WRwlV/dG2hTNKu3QFbGtsx4P0PqmVuO9iDxSk=; b=hn15H2bOHm4B7OirtwS2bLWs1k g/XF/HmyJEF1TYs6csSp5Zegm8aGtC4SXLA2QtPMfzQjNxwv+cb7grkdR8oiA9haBuWsyHUmwGkK3 8XaEBB4FBWmYqMBe6uRzc9JCPWGTbICAAE/Jrli+nhRTkHKcuDBgIumhcflCZFcw1B6dCzTKIBkcj AVXRBY26pg5rgPrcJM6MKaVJFBRiemzf6Ck1/K8R2Ek5tiWOYYx2CbnmdtKIShCKuqcvUrVqDB8KI /w91NSmIkDTXRU1WwQuFKpVUd326r8DjDnYz2pEgT3azIhUt7CoT4aWZ+TPpIMN/qPG2D4sOovtgx nUKwaBmw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF0Al-004Of3-OO; Mon, 09 Jan 2023 21:59:17 +0000 Received: from mail-ot1-x34a.google.com ([2607:f8b0:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF05q-004Mu7-EJ for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 21:54:11 +0000 Received: by mail-ot1-x34a.google.com with SMTP id x26-20020a9d6d9a000000b00672c2f06716so5114054otp.21 for ; Mon, 09 Jan 2023 13:54:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OYPMgEiPrxU+CobVMeoQJZAp/V2pR2BlGHT0msnI4jA=; b=gyqy+T4vgLafFMd0I72XmEQLYVlQTS+qhprWpdg3mEk77/0NG6F6Vo9alqxnGmqKdb WZvjupvvlPafMfSMJ2s+wpXfxi6b6ezq2somnPaIa5uD4SQ7TWF30I0YFqDDXo5mQ3bh 79QWb0qRKtavRZqsJV+vpoW2IclN8C/BLnslNlOGvS72sL3JOCAzgoDtI/GUtCetTbKM YHg5XDnlWZE66YvINnBXAwLe5dKRNCj7Ol15d31EPXsjblFs/A0QhZ2iW6tmwtEi98nt pwcuo1f6vJWFJMbxln1e1WzOxTvCZkL1qdGaMkN2BvtMAA13Fq5oW+Cscfw1n6sJ1KOq CPIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OYPMgEiPrxU+CobVMeoQJZAp/V2pR2BlGHT0msnI4jA=; b=H9VfX/lhP09H9bAm2wdm19onET4VuqrLkyZGLs9iNwGVeOuXNApdyH5whvqLksfsL/ PJltRsZRWRfE7QYL8361gAD2c7qb7yXd84oDkm6tjsLHH+0RKJlQzQ4e1OqhoG+A/51k bh2vgsDOPLN8SGXYoPovQwDD/YdbMQJnQ3d8BCUx2ArOKxXCRYzSn8/8cDKndKWlJ1+y PMFzfsUJOqhR0XaN3aqWV8H0E9WhDMoLsoaED++/N99WJu4S+qcWaar6bdqIwAePtg1a bdtWxv0fTLV9co+uF8DhnttuCMCXKg4Tjh3eiAT5gzI1kSaQc6OviW8UgegUwTmIyPvn 5t6g== X-Gm-Message-State: AFqh2koqdVsb9kkW/BBM5/p0d6XzZoFus/FjJLYV8zdaFEtsOxOJeA6a TfecXs03byP+69oOwrrZ6a324zn5eIEM X-Google-Smtp-Source: AMrXdXtbj9JAAHI73qaN+iD7WOKh4IAZdRGGbBS99shwFb5KWNNXJ3O7u4NPvmn0Z64VJQ7npOOB1EfCbLio X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a9d:193:0:b0:66e:be75:d63f with SMTP id e19-20020a9d0193000000b0066ebe75d63fmr2993190ote.294.1673301248299; Mon, 09 Jan 2023 13:54:08 -0800 (PST) Date: Mon, 9 Jan 2023 21:53:45 +0000 In-Reply-To: <20230109215347.3119271-1-rananta@google.com> Mime-Version: 1.0 References: <20230109215347.3119271-1-rananta@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109215347.3119271-5-rananta@google.com> Subject: [RFC PATCH 4/6] KVM: arm64: Optimize TLBIs in the dirty logging path From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , Ricardo Koller , Reiji Watanabe , James Morse , Alexandru Elisei , Suzuki K Poulose Cc: Paolo Bonzini , Catalin Marinas , Will Deacon , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_135410_528522_1CD9ED95 X-CRM114-Status: GOOD ( 11.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently the dirty-logging paths, including kvm_arch_flush_remote_tlbs_memslot() and kvm_mmu_wp_memory_region() ivalidates the entire VM's TLB entries using kvm_flush_remote_tlbs(). As the range of IPAs is provided by these functions, this is highly inefficient on the systems which support FEAT_TLBIRANGE. Hence, use kvm_flush_remote_tlbs_range() to flush the TLBs instead. Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/kvm/arm.c | 7 ++++++- arch/arm64/kvm/mmu.c | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 00da570ed72bd..179520888c697 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1433,7 +1433,12 @@ void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, const struct kvm_memory_slot *memslot) { - kvm_flush_remote_tlbs(kvm); + phys_addr_t start, end; + + start = memslot->base_gfn << PAGE_SHIFT; + end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; + + kvm_flush_remote_tlbs_range(kvm, start, end); } static int kvm_vm_ioctl_set_device_addr(struct kvm *kvm, diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 70f76bc909c5d..e34b81f5922ce 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -976,7 +976,7 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) write_lock(&kvm->mmu_lock); stage2_wp_range(&kvm->arch.mmu, start, end); write_unlock(&kvm->mmu_lock); - kvm_flush_remote_tlbs(kvm); + kvm_flush_remote_tlbs_range(kvm, start, end); } /** From patchwork Mon Jan 9 21:53:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13094422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF3C2C54EBD for ; Mon, 9 Jan 2023 22:01:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=xGsotIJdi1WSFJVywuzkcNFO0whpQCXW4vrqZcfHkQg=; b=M1VJn9CKdqVToYqQJIdmT2ziin ujqK6vixYto+Usxl6ucryfJVK+o5Dv6TCdoQgYa4ojCJMX1OC+2F2p55+oXAVFZfnqQp4vNE1QC51 FNwc5rtn3Wc2nc6ktOUzAok7pq5qbJ3L5gNLU8c2qyPGEXJhTFB4s9uSVWqny0hrv9GPuQhwGrf80 9vOel+MmzbehOA+cuO6H5rHQyvz4v4usVjMDMuccHLHm/IkLB0N4Uxr1hPfwRCOj0orxDD/AURXnM Mq4p3V1g8ftM1ZAvRx8DFIw6za7/HiZ0/jHGxuAHoxrplXvGmlnMHKqzKGXyKSNczUJIx+dAgi6HZ 7eS+7TZQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF0BY-004Oyb-40; Mon, 09 Jan 2023 22:00:05 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF05s-004MvE-R5 for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 21:54:14 +0000 Received: by mail-io1-xd49.google.com with SMTP id t15-20020a5d81cf000000b006f95aa9ba6eso5760817iol.16 for ; Mon, 09 Jan 2023 13:54:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BjMR5IMZCP3wwXGcrrOi/YjnHWAxQ/J3ysDtq7OdZIc=; b=QFiqc5zB4Ezhsqh0uJctnxBIZ06ZL1uMn/Q6xiIlBq5mgzNSVfWobcWpl+XMH2a1ZB dyTqEaOqhGBhJi7TVqx9StbjModEQJPT51Fp+zKoDaVSyWWwvkyuhXL0k9HyZODKS15w 9JlfwdWOMTUVfOEYqSMJLThOAGQxLoYB8ZW/Avt5hEjhYd0eK3JbsAvOQXV9NpGxcwRz lghVpihiMlnlLlYNqfMsVhO5Zpa6QT6bvsGvCsxj/WPp3DYK/2EnM1p3hEFKreLXtScE jGL0g9SEPsYvYkF7UxjUTJ9ghGVXrlOWYd3EPOYn4m5JgjcjbfoXOFdcf4QsUodhmm6m Uvhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BjMR5IMZCP3wwXGcrrOi/YjnHWAxQ/J3ysDtq7OdZIc=; b=F+N1nH2eM3gfJsEEYXIJMw9I9vDbBMvG0znTnP5+VSJJZzjJI0QIeWFVAVgsuZMb7S OBPWUNxXhcPPn+XxRpI+8xqnfOApz7m741mKNxXt+8RXwYtuq3wEwmUVKwD9Uhlx+0Sy CvLVGNWsgeXDSY3mQDXGgoqk7FBpMusoXPXMQ77pnOeteMBcXZTA4tU/2wADQAfBPkWk di9qdAuyqP9j/giJHSUixT9ZCZkpSptlGdmiPb8CeBjQ4e5JTfolSebnUJK1L4SReJyi uvoJ9zV5OjpV4kMiu9IB094EMLc4SBVTv/O8sR9kdMqdxaCTz03yTRg8lndxq7XC8hnw aTxQ== X-Gm-Message-State: AFqh2kqlXbcHImZqOysg+nkvJV+I7OMEoipA0DzWRDyH+QeJMj0Tz0We A77j9YaddEsWWarkNxloSFIfjuJkTGi7 X-Google-Smtp-Source: AMrXdXveLZ+wlH5ggK1nS8geCa3M7PI/cZSvzt5mjAy3d14vsfPB1eAi+vunpQHd5SoWZ396qOqlx95aAAQc X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a05:6e02:2206:b0:30c:3823:2907 with SMTP id j6-20020a056e02220600b0030c38232907mr4744567ilf.304.1673301250053; Mon, 09 Jan 2023 13:54:10 -0800 (PST) Date: Mon, 9 Jan 2023 21:53:46 +0000 In-Reply-To: <20230109215347.3119271-1-rananta@google.com> Mime-Version: 1.0 References: <20230109215347.3119271-1-rananta@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109215347.3119271-6-rananta@google.com> Subject: [RFC PATCH 5/6] KVM: arm64: Optimize the stage2 map path with TLBI range instructions From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , Ricardo Koller , Reiji Watanabe , James Morse , Alexandru Elisei , Suzuki K Poulose Cc: Paolo Bonzini , Catalin Marinas , Will Deacon , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_135412_960309_8B628C51 X-CRM114-Status: GOOD ( 17.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, when the map path of stage2 page-table coalesces a bunch of pages into a hugepage, KVM invalidates the entire VM's TLB entries. This would cause a perforamance penality for the guest whose pages have already been coalesced earlier as they would have to refill their TLB entries unnecessarily again. Hence, if the system supports it, use __kvm_tlb_flush_range_vmid_ipa() to flush only the range of pages that have been combined into a hugepage, while leaving other TLB entries alone. Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/kvm/hyp/pgtable.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index b11cf2c618a6c..099032bb01bce 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -686,6 +686,22 @@ static bool stage2_try_set_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_ return cmpxchg(ctx->ptep, ctx->old, new) == ctx->old; } +static void kvm_table_pte_flush(struct kvm_s2_mmu *mmu, u64 addr, u32 level, u32 tlb_level) +{ + if (system_supports_tlb_range()) { + u64 end = addr + kvm_granule_size(level); + + kvm_call_hyp(__kvm_tlb_flush_range_vmid_ipa, mmu, addr, end, tlb_level); + } else { + /* + * Invalidate the whole stage-2, as we may have numerous leaf + * entries below us which would otherwise need invalidating + * individually. + */ + kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + } +} + /** * stage2_try_break_pte() - Invalidates a pte according to the * 'break-before-make' requirements of the @@ -693,6 +709,7 @@ static bool stage2_try_set_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_ * * @ctx: context of the visited pte. * @mmu: stage-2 mmu + * @tlb_level: The level at which the leaf pages are expected (for FEAT_TTL hint) * * Returns: true if the pte was successfully broken. * @@ -701,7 +718,7 @@ static bool stage2_try_set_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_ * on the containing table page. */ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, - struct kvm_s2_mmu *mmu) + struct kvm_s2_mmu *mmu, u32 tlb_level) { struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops; @@ -722,7 +739,7 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, * value (if any). */ if (kvm_pte_table(ctx->old, ctx->level)) - kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + kvm_table_pte_flush(mmu, ctx->addr, ctx->level, tlb_level); else if (kvm_pte_valid(ctx->old)) kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level); @@ -804,7 +821,7 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx, if (!stage2_pte_needs_update(ctx->old, new)) return -EAGAIN; - if (!stage2_try_break_pte(ctx, data->mmu)) + if (!stage2_try_break_pte(ctx, data->mmu, ctx->level)) return -EAGAIN; /* Perform CMOs before installation of the guest stage-2 PTE */ @@ -861,7 +878,11 @@ static int stage2_map_walk_leaf(const struct kvm_pgtable_visit_ctx *ctx, if (!childp) return -ENOMEM; - if (!stage2_try_break_pte(ctx, data->mmu)) { + /* + * As the table will be replaced with a block, one level down would + * be the current page entries held by the table. + */ + if (!stage2_try_break_pte(ctx, data->mmu, ctx->level + 1)) { mm_ops->put_page(childp); return -EAGAIN; } From patchwork Mon Jan 9 21:53:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13094423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4C9A4C5479D for ; Mon, 9 Jan 2023 22:02:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=NaIhAgVDLjuRb3UoimDBUdZOY/82iVBt6yNe4qFhS7I=; b=Q1rWWlO03igHNzfoDZV+o/u7MH gufZACcZ6FXcjWZY+Rq78Wmr7CIofTklACCwxT91H5p729xhLmwE8opVXr18iJI77+WuYaDKyd+1U AZVLRRaOafZgxYepMg7oUoVM1foxCCnee2+RllbBNBsHSrqhtjLv3Mildbk8BBD4Dpm6Dbojf3LYN yxnzT/bN08lTUWh0mjPwckHlovDwc42h2KNE2CxX60kkO0tmNjodaC6ufGtJ2LqxSOnEmwZ5jP+Gt iwh4duAmJypM28GXwjBzQl+2CVC7IP+HhcK6pLVqIUDfFxm4K9sXHlWlYfwKcvfZUv2elyTwO0hT8 GLQ8KIPw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF0CT-004PJo-US; Mon, 09 Jan 2023 22:01:09 +0000 Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pF05v-004Mwg-Sc for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 21:54:17 +0000 Received: by mail-io1-xd4a.google.com with SMTP id 16-20020a5d9c10000000b00702de2ee669so5157786ioe.10 for ; Mon, 09 Jan 2023 13:54:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QnWQSYtdYbIpMuZp6LkKgpjLfcHlUZmLQRKHyPWiT+U=; b=SoQHKoORituEmv9rUcL664DtdyreklZiFGEKpPbMIqEEDHQRWBI7SRr3f7cEoqCsvZ 1e7IZZ836ejawXXum/N1EwK5B4IqD2gR6GsFJWCf92qXR9lLgyBN3G4c5l9JL/DOIgzd 5SoZlkxgAJogZrpvMLXlxuFB6zLuebhc5CnJ3Lvd6tWm5FooSJa7Awa5clf+HtqEP2aG eQESQnMpMQOrH2ch9tyF4Sn5bZVnHYL9cgG5M0HGBrBLO1SgEiQFRQ0FQspsxC0pC7hs ur+0YtUUCtgQF8i/Lg4Oe9SZDzJBgNK97UxeExA2fo2dWsRWc7G5+liOox0x9OsztI9W /J9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QnWQSYtdYbIpMuZp6LkKgpjLfcHlUZmLQRKHyPWiT+U=; b=cQwRCtGcSZQiegCC7LvTqvJYmNgrsfpAbmVQ/42nxYmjV4gEmecfxsRkkMaM/Na5n8 zlTIealS4n1Ru+0uj14LMePlgTXEVoUkvQhWvu43XuR2nJpEkNlyzioeDBEg+hM4G/Cl ePn6Mo8YhXpw+mUQLKIKyIsy2bJahDLm48CDLZWkcsWQzOVTEz168EAQXG0iKSIvknpR mxFWPTkN8GKyR925pD4C5w7vWPmT56863PJdoi+Wip657XlMdQwyQ84mDiHhBnsJLAMs GEykAkzhgMZ5tGWrRsqjQgj1OC+W2d3or4404Yek83rdT0ngF5c2dYCEaMh7dA9a4Zum 5VMA== X-Gm-Message-State: AFqh2kpoUsShy7iNv8PfKWzGcYO/LxgCHJKs8uaEf10pa6AnDsaUCrPj tpiknpaHS+VtvE4R7GwL2p8XQWu+CPQL X-Google-Smtp-Source: AMrXdXuEKpemaG/E8lRrZNDz8BRpwdnTj1XFG2QPwNU3UfWi6vqI2Uo/p3n228o5g0euwm8Gw3DaqOUCn3p1 X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a92:4a11:0:b0:300:e879:8094 with SMTP id m17-20020a924a11000000b00300e8798094mr6116771ilf.153.1673301252929; Mon, 09 Jan 2023 13:54:12 -0800 (PST) Date: Mon, 9 Jan 2023 21:53:47 +0000 In-Reply-To: <20230109215347.3119271-1-rananta@google.com> Mime-Version: 1.0 References: <20230109215347.3119271-1-rananta@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109215347.3119271-7-rananta@google.com> Subject: [RFC PATCH 6/6] KVM: arm64: Create a fast stage-2 unmap path From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , Ricardo Koller , Reiji Watanabe , James Morse , Alexandru Elisei , Suzuki K Poulose Cc: Paolo Bonzini , Catalin Marinas , Will Deacon , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_135416_006088_707ED6C4 X-CRM114-Status: GOOD ( 16.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The current implementation of the stage-2 unmap walker traverses the entire page-table to clear and flush the TLBs for each entry. This could be very expensive if the VM is not backed by hugepages. The unmap operation could be made efficient by disconnecting the table at the very top (level at which the largest block mapping can be hosted) and do the rest of the unmapping using free_removed_table(). If the system supports FEAT_TLBIRANGE, flush the entire range that has been disconnected from the rest of the page-table. Suggested-by: Ricardo Koller Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/kvm/hyp/pgtable.c | 44 ++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 099032bb01bce..7bcd898de2805 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1021,6 +1021,49 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, return 0; } +/* + * The fast walker executes only if the unmap size is exactly equal to the + * largest block mapping supported (i.e. at KVM_PGTABLE_MIN_BLOCK_LEVEL), + * such that the underneath hierarchy at KVM_PGTABLE_MIN_BLOCK_LEVEL can + * be disconnected from the rest of the page-table without the need to + * traverse all the PTEs, at all the levels, and unmap each and every one + * of them. The disconnected table can be freed using free_removed_table(). + */ +static int fast_stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, + enum kvm_pgtable_walk_flags visit) +{ + struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops; + kvm_pte_t *childp = kvm_pte_follow(ctx->old, mm_ops); + struct kvm_s2_mmu *mmu = ctx->arg; + + if (!kvm_pte_valid(ctx->old) || ctx->level != KVM_PGTABLE_MIN_BLOCK_LEVEL) + return 0; + + if (!stage2_try_break_pte(ctx, mmu, 0)) + return -EAGAIN; + + /* + * Gain back a reference for stage2_unmap_walker() to free + * this table entry from KVM_PGTABLE_MIN_BLOCK_LEVEL - 1. + */ + mm_ops->get_page(ctx->ptep); + + mm_ops->free_removed_table(childp, ctx->level); + return 0; +} + +static void kvm_pgtable_try_fast_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) +{ + struct kvm_pgtable_walker walker = { + .cb = fast_stage2_unmap_walker, + .arg = pgt->mmu, + .flags = KVM_PGTABLE_WALK_TABLE_PRE, + }; + + if (size == kvm_granule_size(KVM_PGTABLE_MIN_BLOCK_LEVEL)) + kvm_pgtable_walk(pgt, addr, size, &walker); +} + int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) { struct kvm_pgtable_walker walker = { @@ -1029,6 +1072,7 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, }; + kvm_pgtable_try_fast_stage2_unmap(pgt, addr, size); return kvm_pgtable_walk(pgt, addr, size, &walker); }