From patchwork Fri Jul 26 15:07:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13742870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F151C3DA49 for ; Fri, 26 Jul 2024 15:08:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABD8F6B009C; Fri, 26 Jul 2024 11:08:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A6CF86B009E; Fri, 26 Jul 2024 11:08:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90E176B00A4; Fri, 26 Jul 2024 11:08:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 71F3E6B009C for ; Fri, 26 Jul 2024 11:08:01 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D90DFC18AC for ; Fri, 26 Jul 2024 15:08:00 +0000 (UTC) X-FDA: 82382233920.27.F95A3F6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 30F1EA0023 for ; Fri, 26 Jul 2024 15:07:59 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RZu8Weuq; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722006412; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RMo8fPcevHXuY+AOXboYttXTm4mmJMhIiWopBrN6CAw=; b=m/GdwJo9qfXl1xKk22HLFhOD8aEYlBiDyC2yG4ewWycxqyZD8KW9fAKw/owyHg0ZmRzbar wzI+w7x847yseqEpnuErCGsqwQkoapkZS0Y0cv+kd/6hVIvW9CpSA+Qgfka1xKErhDp1eH LsF6X8FB+K+a9c3++hKPBT9/o7lnS6E= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RZu8Weuq; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722006412; a=rsa-sha256; cv=none; b=WTkbwHjYuZ97IbxPn34WxF5H5WrJJTCUVx3FMmeFKgBNsANzPogGUir679J7SXG69djKla lRONLSig6Ln6j91dRkT9huyJX8KemU+98cbI24te8AEAdIrlegPimJsajFEphHmtivkHCJ kouqBK9Icea8h08e3ze0ejbSrDc8DJI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1722006478; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RMo8fPcevHXuY+AOXboYttXTm4mmJMhIiWopBrN6CAw=; b=RZu8Weuqb3hL24oTAFtChz91K+a59RIgiD+eKRjhlmydLUIL0zf+5GUhKPkk0TO+Y3n5eM X/gjjHnBaSX48nvfZk7TnM9BRcE5QW6oOGjXS/AUvHb9jf0BCEpNKsJqxgtREhC91HKodg vsTfOpvf++zXOCvU4Qs/2veMH/0fuYM= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-450-UZOl2F6gMmGTdFJuxLRNgg-1; Fri, 26 Jul 2024 11:07:54 -0400 X-MC-Unique: UZOl2F6gMmGTdFJuxLRNgg-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 911FC1956046; Fri, 26 Jul 2024 15:07:50 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.193.153]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 70DC71955D4C; Fri, 26 Jul 2024 15:07:42 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, David Hildenbrand , Andrew Morton , Oscar Salvador , Peter Xu , Muchun Song , Russell King , Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Naveen N. Rao" , Juergen Gross , Boris Ostrovsky , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Alexander Viro , Christian Brauner Subject: [PATCH v1 1/3] mm: turn USE_SPLIT_PTE_PTLOCKS / USE_SPLIT_PTE_PTLOCKS into Kconfig options Date: Fri, 26 Jul 2024 17:07:26 +0200 Message-ID: <20240726150728.3159964-2-david@redhat.com> In-Reply-To: <20240726150728.3159964-1-david@redhat.com> References: <20240726150728.3159964-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 30F1EA0023 X-Stat-Signature: 9rf3z68w56i9q1rpem55xcq5ctn34cqx X-Rspam-User: X-HE-Tag: 1722006478-449598 X-HE-Meta: U2FsdGVkX18pk3ZbVPTvaL1PraD6D1tGh3EZC/OWoYMiDUv3o3ZWfD7HurwR/6Kd7RAEnnSDsjDq6dwALdC4WY9ulGO6jhPwEG1tq5d7YeKvhqPidXDg+z+rIf7pedMa5W2n7u1YOuW7IFH7Xv7lrAHpmsM3tiDHgaQJ4pUDkqwwwsUlS+e+w+eV5z8CFPYXQ6PjKVQSCPXyziXlYmMLdZDEaIOO0lDWDA2i7Fz31PlPJrp+FOVVLbExcb8uxGsJhx9F0N2PjRibqv4VGfNCYeyv6+j24wW1g8P7/aeuQpwJ9F+2XieUOKaYoKwwE6EdDZ01E/ZiB4gpqUdo6nPbPVuoyS0+d+3z0MRr429NpvpkRjLeVmfGuvPqnhQg4ajE5OCGtJVtjz/OCzIPRdBL0JMHSSn0VhKRfrwlp7iJ5iqaGtSMrQsiTA5ONfw2lRWj8uGM1GaPhPx24tOnVpxBSp/0MixewbKK9cTu8rQbG2qSJkfnn7t5em3WilxTmTPrPF8GS0wHSJpPsOiiAmLoKp17Q9ToA6XUMNIko4ZpXMWuTL+356YEABpUHWOOU8OOSTdnDevLmc+5E5nW6scjE2z3tT/gqYMXCN4O57Fa5b937XceU1YOKYeoogld5h5W8PougXuvo2HTrxWkxc8TqyKvpyDa7JDuxrVtynqlGSt0Pk7YbekDTFVGgsYuVcYsMQLntFnc69wdyC65ssLW+mX1Du14fR6PhlqJnYthubSzaONxCJl1/dnHF9iunK2eGCMgf2+FPDSeiYIXLbo14lSAc1+fV/n/PzzFPmf20b9KPXusvmX1AOI9ztDRKiuY5JMZifB/quF4fJYZUeVnRzcBSHXFZqAcDdTFU5FyoWDW/EUhYfVHe0Ds8oXOlUuPIIDzpctd0+b9Rzyf3v75M78VRXg9lL8lumQkTOvOWle+9wrqowOgPkZuZ2kQSJPkEDby+kCiS+TyOV+o15L fznwrJkj wMd+pckGzyPPqoyLr/jUe2mGkQ/QEGkBLlxqRTmLazU5Ozt/91JB1QX04/DZpUsM0n4GNIXR8uy9DWJ1BJ9I5eFWHVZtXV4yZ1bROKP1dqZFrd5FXkkk4DhXkqK+ewrIPwmW/Kcrw2pfTak8dBpVhvrEjFsxrDAN995dg5AUioWJdZ4Awmy99ct4gFleJmft48OQDcy1U1uhVpuNDdfb8KdND/Z15ZYgPwEkZswSNKHQ9Xbx+LAuzC2yRWz0zxejzDVxHh2od/P8GSPxveqGKYdv/bQCZgI6IrEvAfPpTvLUHNNA5cJqeDMCWevCndY7SMyFFM/GGMzLs4SLk0b45EHemforbkTGkK1jR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's clean that up a bit and prepare for depending on CONFIG_SPLIT_PMD_PTLOCKS in other Kconfig options. More cleanups would be reasonable (like the arch-specific "depends on" for CONFIG_SPLIT_PTE_PTLOCKS), but we'll leave that for another day. Signed-off-by: David Hildenbrand Acked-by: Mike Rapoport (Microsoft) Reviewed-by: Qi Zheng Reviewed-by: Russell King (Oracle) --- arch/arm/mm/fault-armv.c | 6 +++--- arch/x86/xen/mmu_pv.c | 7 ++++--- include/linux/mm.h | 8 ++++---- include/linux/mm_types.h | 2 +- include/linux/mm_types_task.h | 3 --- kernel/fork.c | 4 ++-- mm/Kconfig | 18 +++++++++++------- mm/memory.c | 2 +- 8 files changed, 26 insertions(+), 24 deletions(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 2286c2ea60ec4..831793cd6ff94 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -61,7 +61,7 @@ static int do_adjust_pte(struct vm_area_struct *vma, unsigned long address, return ret; } -#if USE_SPLIT_PTE_PTLOCKS +#if defined(CONFIG_SPLIT_PTE_PTLOCKS) /* * If we are using split PTE locks, then we need to take the page * lock here. Otherwise we are using shared mm->page_table_lock @@ -80,10 +80,10 @@ static inline void do_pte_unlock(spinlock_t *ptl) { spin_unlock(ptl); } -#else /* !USE_SPLIT_PTE_PTLOCKS */ +#else /* !defined(CONFIG_SPLIT_PTE_PTLOCKS) */ static inline void do_pte_lock(spinlock_t *ptl) {} static inline void do_pte_unlock(spinlock_t *ptl) {} -#endif /* USE_SPLIT_PTE_PTLOCKS */ +#endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, unsigned long pfn) diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index f1ce39d6d32cb..f4a316894bbb4 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -665,7 +665,7 @@ static spinlock_t *xen_pte_lock(struct page *page, struct mm_struct *mm) { spinlock_t *ptl = NULL; -#if USE_SPLIT_PTE_PTLOCKS +#if defined(CONFIG_SPLIT_PTE_PTLOCKS) ptl = ptlock_ptr(page_ptdesc(page)); spin_lock_nest_lock(ptl, &mm->page_table_lock); #endif @@ -1553,7 +1553,8 @@ static inline void xen_alloc_ptpage(struct mm_struct *mm, unsigned long pfn, __set_pfn_prot(pfn, PAGE_KERNEL_RO); - if (level == PT_PTE && USE_SPLIT_PTE_PTLOCKS && !pinned) + if (level == PT_PTE && IS_ENABLED(CONFIG_SPLIT_PTE_PTLOCKS) && + !pinned) __pin_pagetable_pfn(MMUEXT_PIN_L1_TABLE, pfn); xen_mc_issue(XEN_LAZY_MMU); @@ -1581,7 +1582,7 @@ static inline void xen_release_ptpage(unsigned long pfn, unsigned level) if (pinned) { xen_mc_batch(); - if (level == PT_PTE && USE_SPLIT_PTE_PTLOCKS) + if (level == PT_PTE && IS_ENABLED(CONFIG_SPLIT_PTE_PTLOCKS)) __pin_pagetable_pfn(MMUEXT_UNPIN_TABLE, pfn); __set_pfn_prot(pfn, PAGE_KERNEL); diff --git a/include/linux/mm.h b/include/linux/mm.h index 0472a5090b180..dff43101572ec 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2843,7 +2843,7 @@ static inline void pagetable_free(struct ptdesc *pt) __free_pages(page, compound_order(page)); } -#if USE_SPLIT_PTE_PTLOCKS +#if defined(CONFIG_SPLIT_PTE_PTLOCKS) #if ALLOC_SPLIT_PTLOCKS void __init ptlock_cache_init(void); bool ptlock_alloc(struct ptdesc *ptdesc); @@ -2895,7 +2895,7 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) return true; } -#else /* !USE_SPLIT_PTE_PTLOCKS */ +#else /* !defined(CONFIG_SPLIT_PTE_PTLOCKS) */ /* * We use mm->page_table_lock to guard all pagetable pages of the mm. */ @@ -2906,7 +2906,7 @@ static inline spinlock_t *pte_lockptr(struct mm_struct *mm, pte_t *pte) static inline void ptlock_cache_init(void) {} static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} -#endif /* USE_SPLIT_PTE_PTLOCKS */ +#endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { @@ -2966,7 +2966,7 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd))? \ NULL: pte_offset_kernel(pmd, address)) -#if USE_SPLIT_PMD_PTLOCKS +#if defined(CONFIG_SPLIT_PMD_PTLOCKS) static inline struct page *pmd_pgtable_page(pmd_t *pmd) { diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 4854249792545..165c58b12ccc9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -947,7 +947,7 @@ struct mm_struct { #ifdef CONFIG_MMU_NOTIFIER struct mmu_notifier_subscriptions *notifier_subscriptions; #endif -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !defined(CONFIG_SPLIT_PMD_PTLOCKS) pgtable_t pmd_huge_pte; /* protected by page_table_lock */ #endif #ifdef CONFIG_NUMA_BALANCING diff --git a/include/linux/mm_types_task.h b/include/linux/mm_types_task.h index a2f6179b672b8..bff5706b76e14 100644 --- a/include/linux/mm_types_task.h +++ b/include/linux/mm_types_task.h @@ -16,9 +16,6 @@ #include #endif -#define USE_SPLIT_PTE_PTLOCKS (NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS) -#define USE_SPLIT_PMD_PTLOCKS (USE_SPLIT_PTE_PTLOCKS && \ - IS_ENABLED(CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK)) #define ALLOC_SPLIT_PTLOCKS (SPINLOCK_SIZE > BITS_PER_LONG/8) /* diff --git a/kernel/fork.c b/kernel/fork.c index a8362c26ebcb0..216ce9ba4f4e6 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -832,7 +832,7 @@ static void check_mm(struct mm_struct *mm) pr_alert("BUG: non-zero pgtables_bytes on freeing mm: %ld\n", mm_pgtables_bytes(mm)); -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !defined(CONFIG_SPLIT_PMD_PTLOCKS) VM_BUG_ON_MM(mm->pmd_huge_pte, mm); #endif } @@ -1276,7 +1276,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, RCU_INIT_POINTER(mm->exe_file, NULL); mmu_notifier_subscriptions_init(mm); init_tlb_flush_pending(mm); -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !defined(CONFIG_SPLIT_PMD_PTLOCKS) mm->pmd_huge_pte = NULL; #endif mm_init_uprobes_state(mm); diff --git a/mm/Kconfig b/mm/Kconfig index b72e7d040f789..7b716ac802726 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -585,17 +585,21 @@ config ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE # at the same time (e.g. copy_page_range()). # DEBUG_SPINLOCK and DEBUG_LOCK_ALLOC spinlock_t also enlarge struct page. # -config SPLIT_PTLOCK_CPUS - int - default "999999" if !MMU - default "999999" if ARM && !CPU_CACHE_VIPT - default "999999" if PARISC && !PA20 - default "999999" if SPARC32 - default "4" +config SPLIT_PTE_PTLOCKS + def_bool y + depends on MMU + depends on NR_CPUS >= 4 + depends on !ARM || CPU_CACHE_VIPT + depends on !PARISC || PA20 + depends on !SPARC32 config ARCH_ENABLE_SPLIT_PMD_PTLOCK bool +config SPLIT_PMD_PTLOCKS + def_bool y + depends on SPLIT_PTE_PTLOCKS && ARCH_ENABLE_SPLIT_PMD_PTLOCK + # # support for memory balloon config MEMORY_BALLOON diff --git a/mm/memory.c b/mm/memory.c index 833d2cad6eb29..714589582fe15 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6559,7 +6559,7 @@ long copy_folio_from_user(struct folio *dst_folio, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS */ -#if USE_SPLIT_PTE_PTLOCKS && ALLOC_SPLIT_PTLOCKS +#if defined(CONFIG_SPLIT_PTE_PTLOCKS) && ALLOC_SPLIT_PTLOCKS static struct kmem_cache *page_ptl_cachep;