From patchwork Sun Jan 12 15:53:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rik van Riel X-Patchwork-Id: 13936447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5648E7719C for ; Sun, 12 Jan 2025 15:55:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC7B46B009F; Sun, 12 Jan 2025 10:55:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E25B76B009C; Sun, 12 Jan 2025 10:55:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F0346B00A1; Sun, 12 Jan 2025 10:55:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4B9916B009C for ; Sun, 12 Jan 2025 10:55:43 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0F10140834 for ; Sun, 12 Jan 2025 15:55:43 +0000 (UTC) X-FDA: 82999250166.23.8E6BFE6 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf06.hostedemail.com (Postfix) with ESMTP id 7F57B180006 for ; Sun, 12 Jan 2025 15:55:41 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf06.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736697341; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dpGlWYXD8ytxJCUwIO6zQgRV3qwvcKaHGc7OrRtIz9E=; b=3OvQwpK04FD7KLcEDq4VhBMgysMm6WP8bqTZTBoOtaWDREmWIf1A+ZRL+ppbciENJthLt9 Gs+GswbIVaVY4aga6hYzhq0d0saDyr5hfMi3kT81ePuQlErk9u3YjpddZAQaSJgByZeFTz omjiS5KemTxauPICNODRGlCmTUi0Qn0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736697341; a=rsa-sha256; cv=none; b=mL3zMS4Q/liYQj7O+lu49PLGBWEps+OMSrW+0UjO6iNxlStyzqkhb3u6jpyohd7uoAi25w +Xi/EtAz6JWGE5KpOhq0QB/6MuWwXpfpEytf9C9gYXw/aBeeoO1O3IUpbDZV6ezPga+UX+ 12gJyQBDgMIerzo1b8dzNmJzI3GX508= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf06.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tX0Ii-0000000010W-2bLe; Sun, 12 Jan 2025 10:54:56 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, Rik van Riel Subject: [PATCH v4 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional Date: Sun, 12 Jan 2025 10:53:45 -0500 Message-ID: <20250112155453.1104139-2-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250112155453.1104139-1-riel@surriel.com> References: <20250112155453.1104139-1-riel@surriel.com> MIME-Version: 1.0 X-Stat-Signature: 6qq44x1kk76gd5phnx7tx4csbot6ok88 X-Rspamd-Queue-Id: 7F57B180006 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1736697341-41070 X-HE-Meta: U2FsdGVkX198NsjsJVlhcyA5sbECKkg5sVmTmjK/KA4aCywGJzIwRM6muah2hoeQtiO6BaAgsshTd08VAGl+8l530CKtQ60OSRRiFjB0oK8NeI3hMlBzhz4YAMVDmWFLVBrKLUT2AV43DdJQpRVWoktSYb0h0aI4BKtpI21cLPMrY5LGXhszhp0jeKnAabi+KYFibZBzNslxvOSLek0DlHJXlUVRjY7ArzbpQHbCWq9RYGvAcEDP6LwVC1U3rQvg8gY23riXKD+HbnHKC4uBriFJl41TkCz6K2Bqw9bBL49ih9n9nW12aLR7irPbfaRPDa99G8mfPoYIx/Aeo8uXWkNRH+jm1V+w0CO9Um8jnVjFP6SynHPdiJidG8vB9KJ3ybwqWzilIUPT+Q0ibgHyYQJlXDlc4m8bGpTE1jBqhykHX1nXhicZC93lJ91w2H3yEYxbxKVhSJKGhpxQsr7zWtcu7zoWSUNiEneuztDt8DjGMKAKSuQpQVOSB6lyau+WRAA2V+Ma3spwbkj8I1VOCDlmmvoG+jgRT6Sg/ygHl+AwknpCDUxJhGY1N9w5KgauhbeSRkFdIJd5ghrG5yvGPyJ0Fbk2mG7tOW7af6qi3JqT2LG3wY3CRsYfu8JeF8gNM/adEaq5mLLUrtuRXDSkV0+bFjiT9Ja7X18I11PXuG0lBTAc+Rx7qBNTjDpiOb7k0PiyVYA/EWcYKYmfgaLnz6YWH7OQQ+S3VwwM0v/lGdZ5h3PY7Jdgt5bPGoqtaDYIcySO9UgZ+rzDga4Q114U/v2Ri9AcctAS3az2lamUgKv5PwY6DwO/+XULqN8tSa02z9QA+PDfE1e2hwKJwcBt5TILhXWvvmutxjFQ4SCLatKBDdtDvxv/xXFSU3M4xWG6ujbLRPcUSNtswILq0Y5zIWKmHZnak0V8geugfGneYPL2fusTA0yHWPspAa43WUxaCyo0YPxD47/03n9fkxO 6klUfUoy xWG9cH7HQZpft3zbsG2jXDTL1ZRPiIhBx6lTPp2YMBCgSqwpiUN8HY66W1CNhoDwl61t38o4+Cnk9bUMu5CCaj1F8VNfiU6seGLoeR2QkSfU820OfY7hJa5ndpTXD/vrcsaM4CeFQUhVepn3MxRbW8GMaIFCj+JUt818vLioLbekmVe2diOpS70VajBw0dQhL1ACsDLWnTEMCqqz+SXHHC0yD1pJF7hOGbaYjSRT/so9pSC5fccnczYpjrckc5yYaci4RUKgLmDLM4NFym2Ir/GHCVtF9kZlHZti5uMYRyUPLh3sxKO7abB+YjB0hLn77G3uwjKA+KGMOS3weUzIW8Ee9EMV79HVkcuU4ia/a0kKDYnsUKRbNmUBQLsfLNMpUjKY7nH5JSP4IRwPTMOP6vDkwq9gYrPSaWIktDA/5K0c1BgnegwyLHOISncvQsQXsSFPRiwyAIqPsq18Qp0rtHUEFF2pV48J+serdgRJtph067T//nTaGdEkL/Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently x86 uses CONFIG_MMU_GATHER_TABLE_FREE when using paravirt, and not when running on bare metal. There is no real good reason to do things differently for each setup. Make them all the same. Currently get_user_pages_fast synchronizes against page table freeing in two different ways: - on bare metal, by blocking IRQs, which block TLB flush IPIs - on paravirt, with MMU_GATHER_RCU_TABLE_FREE This is done because some paravirt TLB flush implementations handle the TLB flush in the hypervisor, and will do the flush even when the target CPU has interrupts disabled. After this change, the synchronization between get_user_pages_fast and page table freeing is always handled with MMU_GATHER_RCU_TABLE_FREE, which allows bare metal to also do TLB flushes while interrupts are disabled. That makes it safe to use INVLPGB on AMD CPUs. Signed-off-by: Rik van Riel Suggested-by: Peter Zijlstra --- arch/x86/Kconfig | 2 +- arch/x86/kernel/paravirt.c | 7 +------ 2 files changed, 2 insertions(+), 7 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 9d7bd0ae48c4..e8743f8c9fd0 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -274,7 +274,7 @@ config X86 select HAVE_PCI select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP - select MMU_GATHER_RCU_TABLE_FREE if PARAVIRT + select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_MERGE_VMAS select HAVE_POSIX_CPU_TIMERS_TASK_WORK select HAVE_REGS_AND_STACK_ACCESS_API diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index fec381533555..2b78a6b466ed 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -59,11 +59,6 @@ void __init native_pv_lock_init(void) static_branch_enable(&virt_spin_lock_key); } -static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) -{ - tlb_remove_page(tlb, table); -} - struct static_key paravirt_steal_enabled; struct static_key paravirt_steal_rq_enabled; @@ -191,7 +186,7 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, .mmu.flush_tlb_multi = native_flush_tlb_multi, - .mmu.tlb_remove_table = native_tlb_remove_table, + .mmu.tlb_remove_table = tlb_remove_table, .mmu.exit_mmap = paravirt_nop, .mmu.notify_page_enc_status_changed = paravirt_nop,