From patchwork Thu Aug 24 09:22:58 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Kuznetsov X-Patchwork-Id: 9919635 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B735C60327 for ; Thu, 24 Aug 2017 09:26:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AA41428BC8 for ; Thu, 24 Aug 2017 09:26:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9C97528BCB; Thu, 24 Aug 2017 09:26:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2179B28BCB for ; Thu, 24 Aug 2017 09:26:43 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dkoM1-0005fn-Q8; Thu, 24 Aug 2017 09:23:09 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dkoM1-0005fh-3A for xen-devel@lists.xenproject.org; Thu, 24 Aug 2017 09:23:09 +0000 Received: from [85.158.139.211] by server-1.bemta-5.messagelabs.com id 6F/A5-01993-CFA9E995; Thu, 24 Aug 2017 09:23:08 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrNLMWRWlGSWpSXmKPExsVysWW7jO6vWfM iDVo3alp83zKZyYHR4/CHKywBjFGsmXlJ+RUJrBmLenwK9ktVnH4S3MD4UqyLkZNDSGA2k8S3 7a5djBwcLAL2Er1XDUDCjAJlErfX7GeFsHMl9jzvB7K5gMrnM0ps3fWIGSTBJqAj8f3pKWaQX hEBIYmld+tAapgFWlklNnS2sIPUCAuYSCz+3QFmswioSnzbOpsFxOYVsJBYPfkUmC0hoC1xZP deNpBmCYE+Rokfuz+zTGDkXcDIsIpRvTi1qCy1SNdML6koMz2jJDcxM0fX0MBULze1uDgxPTU nMalYLzk/dxMjMBAYgGAH49QG50OMkhxMSqK8T6TnRQrxJeWnVGYkFmfEF5XmpBYfYpTh4FCS 4I2bCZQTLEpNT61Iy8wBhiRMWoKDR0mE9zBImre4IDG3ODMdInWKUVFKnDcXJCEAksgozYNrg 8XBJUZZKWFeRqBDhHgKUotyM0tQ5V8xinMwKgnzhoNM4cnMK4Gb/gpoMRPQ4kkn5oAsLklESE k1MIpUWbmE1gcUJdTcX/dut3C94tLDF2PEa0Jbf4cbbhO/x23cHm4ix5rW/l3DI/M3o6T0pbz Nb0qr4ly2LO7z/bDm+6ddpTMiuC9ZnT4cmiHHFGFy7JHsgfifh+qtmGfzdEaLm644ezVjwtm+ C4tC/cIrF8scapwUkPf1tvfZPqEft/LlNO8zKrEUZyQaajEXFScCAONjE0h+AgAA X-Env-Sender: vkuznets@redhat.com X-Msg-Ref: server-12.tower-206.messagelabs.com!1503566584!72359146!1 X-Originating-IP: [209.132.183.28] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMjA5LjEzMi4xODMuMjggPT4gNTQwNjQ=\n X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 6105 invoked from network); 24 Aug 2017 09:23:06 -0000 Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by server-12.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 24 Aug 2017 09:23:06 -0000 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7671B7E423; Thu, 24 Aug 2017 09:23:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 7671B7E423 Received: from vitty.brq.redhat.com (unknown [10.43.2.155]) by smtp.corp.redhat.com (Postfix) with ESMTP id A12B071C5E; Thu, 24 Aug 2017 09:22:59 +0000 (UTC) From: Vitaly Kuznetsov To: x86@kernel.org Date: Thu, 24 Aug 2017 11:22:58 +0200 Message-Id: <20170824092258.12375-1-vkuznets@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Thu, 24 Aug 2017 09:23:03 +0000 (UTC) Cc: Juergen Gross , Stephen Hemminger , Peter Zijlstra , Andrew Cooper , linux-kernel@vger.kernel.org, Steven Rostedt , Andy Lutomirski , Jork Loeser , Ingo Molnar , "H. Peter Anvin" , xen-devel@lists.xenproject.org, Thomas Gleixner , KY Srinivasan , Linus Torvalds , Boris Ostrovsky , "Kirill A. Shutemov" Subject: [Xen-devel] [PATCH v2] x86: enable RCU based table free X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP On x86 software page-table walkers depend on the fact that remote TLB flush does an IPI: walk is performed lockless but with interrupts disabled and in case the pagetable is freed the freeing CPU will get blocked as remote TLB flush is required. On other architecture which don't require an IPI to do remote TLB flush we have an RCU-based mechanism (see include/asm-generic/tlb.h for more details). In virtualized environments we may want to override .flush_tlb_others hook in pv_mmu_ops and use a hypercall asking the hypervisor to do remote TLB flush for us. This breaks the assumption about IPI. Xen PV does this for years and the upcoming remote TLB flush for Hyper-V will do it too. This is not safe, software pagetable walkers may step on an already freed page. Solve the issue by enabling RCU-based table free mechanism. Testing with kernbench and mmap/munmap microbenchmars didn't show any notable performance impact. Suggested-by: Peter Zijlstra Signed-off-by: Vitaly Kuznetsov Acked-by: Juergen Gross Acked-by: Kirill A. Shutemov --- Changes since v1: - Enable HAVE_RCU_TABLE_FREE unconditionally to avoid different code pathes for no reason [Linus Torvalds] --- arch/x86/Kconfig | 1 + arch/x86/include/asm/tlb.h | 5 +++++ arch/x86/mm/pgtable.c | 8 ++++---- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 323cb065be5e..b0bfc27d05a2 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -167,6 +167,7 @@ config X86 select HAVE_HARDLOCKUP_DETECTOR_PERF if PERF_EVENTS && HAVE_PERF_EVENTS_NMI select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP + select HAVE_RCU_TABLE_FREE select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if X86_64 && FRAME_POINTER && STACK_VALIDATION select HAVE_STACK_VALIDATION if X86_64 diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index c7797307fc2b..d43a7fcafee9 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -15,4 +15,9 @@ #include +static inline void __tlb_remove_table(void *table) +{ + free_page_and_swap_cache(table); +} + #endif /* _ASM_X86_TLB_H */ diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 508a708eb9a6..218834a3e9ad 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -56,7 +56,7 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { pgtable_page_dtor(pte); paravirt_release_pte(page_to_pfn(pte)); - tlb_remove_page(tlb, pte); + tlb_remove_table(tlb, pte); } #if CONFIG_PGTABLE_LEVELS > 2 @@ -72,21 +72,21 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) tlb->need_flush_all = 1; #endif pgtable_pmd_page_dtor(page); - tlb_remove_page(tlb, page); + tlb_remove_table(tlb, page); } #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); - tlb_remove_page(tlb, virt_to_page(pud)); + tlb_remove_table(tlb, virt_to_page(pud)); } #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); - tlb_remove_page(tlb, virt_to_page(p4d)); + tlb_remove_table(tlb, virt_to_page(p4d)); } #endif /* CONFIG_PGTABLE_LEVELS > 4 */ #endif /* CONFIG_PGTABLE_LEVELS > 3 */