From patchwork Wed Dec 4 11:09:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13893591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D02E2E7716D for ; Wed, 4 Dec 2024 11:11:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 632056B00A3; Wed, 4 Dec 2024 06:11:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E1B96B00A4; Wed, 4 Dec 2024 06:11:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45B876B00A5; Wed, 4 Dec 2024 06:11:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2791E6B00A3 for ; Wed, 4 Dec 2024 06:11:39 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D7E45160F4B for ; Wed, 4 Dec 2024 11:11:38 +0000 (UTC) X-FDA: 82857010824.04.1315E10 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf28.hostedemail.com (Postfix) with ESMTP id 7E51DC0010 for ; Wed, 4 Dec 2024 11:11:19 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=YlzE8FBa; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733310690; a=rsa-sha256; cv=none; b=3mTfPbkaMAyjLOJwx/5+HukQY8fQ6vxVlBW5WRDkmN0wD7JWgatl+YjzBRbUf1QR7+UUXM QXbfPfyEY9ZVHkbWJJPVRR6Bkbx3hNUZV6I2433Fvx0lzQEH40gsVl3y9IaoeNc3usUP0S Vr4kc0GLpiH3EuPJ2vBjekgQekq0S9U= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=YlzE8FBa; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733310690; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9fSWWcfeV8hzoFpFk7VaZVUYbnww7eiSUjs7tN8xr0c=; b=JT/y/Ri/z+6/G7kjcAxaUGXVh+lVnjyrb69/VDMPvrR9k4+bOXEaodVlsHUUVkNJ4fB186 jbUNUAhCeVClNzNj/oGjm6jtMqkBqlALsI1bJyZuXqWeI59AtHc5xDOtdtFjv1zMmI7aZ6 56WDCVdWGkIvZ1HrFbUH4yi/1+esT+w= Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-724f383c5bfso5158277b3a.1 for ; Wed, 04 Dec 2024 03:11:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733310696; x=1733915496; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9fSWWcfeV8hzoFpFk7VaZVUYbnww7eiSUjs7tN8xr0c=; b=YlzE8FBa/d9n1VxLgRP5yPcP6X9yh/0aoS8IdMD0U9pstDmnvGvTBDDVFJLc86JbWG AYhxTLTEYbgCpmeWYupNJSZDWmhGK22vR2Arz4YUsdliT1Esfox6Aw76HcK8Od4mOq1w IKoJwtQ2k+X7mzN/XLv+5qTY7YAfi3iTgKWDYpUZoaWYBvL2pMAGIUpIQYVtiMYRMa8o 7fclpUHgQV7zMZiWY6SrRXuuV3pT38CmikZ4s9M9D873NVG5BxY0+LwYO0E5BsoqfM5R jROumj0KsmZlqSkAkZwk+qld8Hj/0r5ZxGd0hv1Se5alY7ls/U5QiA9nDs+CshtO/PZb jtOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733310696; x=1733915496; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9fSWWcfeV8hzoFpFk7VaZVUYbnww7eiSUjs7tN8xr0c=; b=PBg48eamEV4CA+vvzR1BlfQiVDRnqKrXY1uaCoyb5ZPCKYFHXOElhYcbcB5FHbeoyb 0mCszVIPfWOVJvMkueriR20Xxv/VcSAcpskCExdn8kSh4gWjU5t6cULHGni+Stmf6Vzp 7gyUu0MG15MMkenRnJgvlQttdSIwOguRZym5czuat+8Zxma5CISPQEByQnX9OBNOfJw6 WXE6Zsoekwu6tAZGTzdVlQXGzOaTIB38EP0kbugI0/5eaZqhEf3srPIFU+y/+Ik7nWaw 3K0P4QPF/ux+3HDyKZ8naF9NlQWNd9TExWUrbxB/pw7IFnmhIJPhtKpWpQnK9viyaNos BQkg== X-Forwarded-Encrypted: i=1; AJvYcCWruMDpnY+7plUiYDB6p79o3AjGUGLiidPtLvnzuWQt+KU3n2Kqy5tptMYljCe3PgVjaQ4ZV9MMSw==@kvack.org X-Gm-Message-State: AOJu0YxFGKJtqpH6JZzIyjr2AChHKL7+nNiszz0eWw43UPN8lWQTSXkY o2ajq5CIm+y/+45GwB/oK2j359khM1ZwjBbFV2MA61nWX+L3OjaKkJqUvDj0SPY= X-Gm-Gg: ASbGncvSF8p8prCuFHb0sILgxe/71C0ZibtWMT6SuY2x2BjknS6PjUYCKG8xmAtXXF+ 8RtGysaibQhSd6UsDf6etQm7Q2ZivytE17U6euBzz+LZLNV/pXhAy/Z/ILCJj87kvha9/dl6RSF +m3HDJgcNNs+YQv6ry3RpIZApcXtOt8c4QJQxs2AKKQekyKUCpF8Gx+oyWC5bczNAFd3VsnhBks 4VRZp0p0mnOrkavPlpV9uGOfAXXQ+W5aJAj2OnCIJDVnKYKzaTq2U7v5Nr4c4vSCXon70g+DklZ Gk9HM7U3FEcRsnI= X-Google-Smtp-Source: AGHT+IFovq7hDlm4wLdDhm1VdKh1zZ19bET/1mJY6NLSrbW7cKbNUzk3vayQjzm+UlBEnc1Q8UAnEw== X-Received: by 2002:a17:902:ea10:b0:215:a190:ba28 with SMTP id d9443c01a7336-215d0041378mr59888075ad.22.1733310695765; Wed, 04 Dec 2024 03:11:35 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.148]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21527515731sm107447495ad.192.2024.12.04.03.11.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Dec 2024 03:11:35 -0800 (PST) From: Qi Zheng To: david@redhat.com, jannh@google.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, peterx@redhat.com, akpm@linux-foundation.org Cc: mgorman@suse.de, catalin.marinas@arm.com, will@kernel.org, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, lorenzo.stoakes@oracle.com, zokeefe@google.com, rientjes@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 10/11] x86: mm: free page table pages by RCU instead of semi RCU Date: Wed, 4 Dec 2024 19:09:50 +0800 Message-Id: <0287d442a973150b0e1019cc406e6322d148277a.1733305182.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 7E51DC0010 X-Stat-Signature: r16yz8bobab8jjtqbgyqf8xmspk1o7qa X-HE-Tag: 1733310679-266200 X-HE-Meta: U2FsdGVkX18Br+yUFfhvvf8ppidm4yYoQxG9THLZI3a3rgCiYMg0LrWsx/Q5KujffMPFoRdlr5BG4vBoqsr4I8FSoBlZbU2pUxygtaWbVrkFM3lbzQYoQoNHxuT4S0/NuEyrke7+SJJOeeBIIYd501m/zfsk5NnDs4oUmMpZTwfILLs0inAgABaJc3wvEKKHqKmb9/zIygI7EkQyOTY8tCfsrP1gfjkhV+GWatyMowoghZ6awluOsL0yJhycazoCEsRKmwTGSutRz7V6C00B6KundCY+FS3aham6Spue6KYBHjQvSNx4GyS82mHnxwu+/j0tpVMjED5AErzGAcDI+fXakgyRsXEF/PD7z9NAjjE6UtxqRSoeVeAJhXozvQj0TomF6RqpnT0+eEf2zcxI8TevkO0skwAIP0+CZS52a80Y4yCLGTHjJH9vop0voZlusMsY1SJFnhGGGPJFNzO/91weUA5qCg5uMEDlNQ863aKuaN9cllcFJ/LHnXnjrk9xGXfAcW6TLeEjVJxeVVMTmZJFYBgB4rw792U9/bHoBpSfzUvujUcV81pHNDLTdXaQm8a3cmqwfLzyotpI2IWVqhGrUh0MieFHywSDuJItyCKu3tojTs2V9CTEzmZjCjH9lOdtBdgtlkfZACk/rtS+nNoWrJg4UvdigX5pG0xjtC8hlFppm02cjnjM3+Z+0yIJq/C/QQsgQ2MYglNTLkbD6sLPLaCphp4Osh7GR6XHoMIIQNPJYfwOHeNQTKtl1ezO6pksMmZSuhtA2vjcY3J/NuBHNjBU3OWaBDDqgFB08xgE2ZH9Mxb9cYJvKWdC3k1pOI4+TC37xJnVwLFO5hzibMw08AuwXicmIzN06cX4bvbxgQsZu8gNIm3TXBs0uRfwWBkZe9gyfJJeCnZ+fcK+eza/eWMS86/Npie6Ikxzjq4tdoskIZ0XEHVlYe8JfsRHeTm/qF6qCKHv6I9RSb4 KJM2RRek P0m0+h8xi4BE+GZ8RNTgkBn8Bhb/le6yh40jha+UC6NliobBOc7HN9pOhvsSwn5UNXTkg2p5ZRCrHW03kz3n0s9xF2XNaou9+yspn0hnYq3QFO5GA8/ExnmhRSLgL9Gy1GCrAPpIovGybmbMjf5dCXjoSzNReukIW4gk8paMN2oe5Typbm94dE8SaVq6d87A8KrWewxopy45Bo8UFp0W4SPQ+vAZeo9Oq/8mtZU48q6CSsvZFJ13MBhUqlXT6kmpA6BsaD0YbkeDdlFwwRe2pZ54KhW5HZ+iuuTbFmcWb2HmchK68zEIDUhbrxPQJ1O1IdQuOAFtWTJzG6qIjHQV+P1fu9vG/2j9L5SG/qFmUQVtm+15ruTWV7TLM0L8lUipi4AOyv2SIJjhiTtDukAVlAV/OWqAy04tC8U+P X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now, if CONFIG_MMU_GATHER_RCU_TABLE_FREE is selected, the page table pages will be freed by semi RCU, that is: - batch table freeing: asynchronous free by RCU - single table freeing: IPI + synchronous free In this way, the page table can be lockless traversed by disabling IRQ in paths such as fast GUP. But this is not enough to free the empty PTE page table pages in paths other that munmap and exit_mmap path, because IPI cannot be synchronized with rcu_read_lock() in pte_offset_map{_lock}(). In preparation for supporting empty PTE page table pages reclaimation, let single table also be freed by RCU like batch table freeing. Then we can also use pte_offset_map() etc to prevent PTE page from being freed. Like pte_free_defer(), we can also safely use ptdesc->pt_rcu_head to free the page table pages: - The pt_rcu_head is unioned with pt_list and pmd_huge_pte. - For pt_list, it is used to manage the PGD page in x86. Fortunately tlb_remove_table() will not be used for free PGD pages, so it is safe to use pt_rcu_head. - For pmd_huge_pte, it is used for THPs, so it is safe. After applying this patch, if CONFIG_PT_RECLAIM is enabled, the function call of free_pte() is as follows: free_pte pte_free_tlb __pte_free_tlb ___pte_free_tlb paravirt_tlb_remove_table tlb_remove_table [!CONFIG_PARAVIRT, Xen PV, Hyper-V, KVM] [no-free-memory slowpath:] tlb_table_invalidate tlb_remove_table_one __tlb_remove_table_one [frees via RCU] [fastpath:] tlb_table_flush tlb_remove_table_free [frees via RCU] native_tlb_remove_table [CONFIG_PARAVIRT on native] tlb_remove_table [see above] Signed-off-by: Qi Zheng Cc: x86@kernel.org Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra --- arch/x86/include/asm/tlb.h | 20 ++++++++++++++++++++ arch/x86/kernel/paravirt.c | 7 +++++++ arch/x86/mm/pgtable.c | 10 +++++++++- include/linux/mm_types.h | 4 +++- mm/mmu_gather.c | 9 ++++++++- 5 files changed, 47 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 4d3c9d00d6b6b..73f0786181cc9 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -34,8 +34,28 @@ static inline void __tlb_remove_table(void *table) free_page_and_swap_cache(table); } +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct page *page; + + page = container_of(head, struct page, rcu_head); + put_page(page); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct page *page; + + page = table; + call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); +} +#define __tlb_remove_table_one __tlb_remove_table_one +#endif /* CONFIG_PT_RECLAIM */ + static inline void invlpg(unsigned long addr) { asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); } + #endif /* _ASM_X86_TLB_H */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index fec3815335558..89688921ea62e 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -59,10 +59,17 @@ void __init native_pv_lock_init(void) static_branch_enable(&virt_spin_lock_key); } +#ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { tlb_remove_page(tlb, table); } +#else +static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) +{ + tlb_remove_table(tlb, table); +} +#endif struct static_key paravirt_steal_enabled; struct static_key paravirt_steal_rq_enabled; diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 5745a354a241c..69a357b15974a 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -19,12 +19,20 @@ EXPORT_SYMBOL(physical_mask); #endif #ifndef CONFIG_PARAVIRT +#ifndef CONFIG_PT_RECLAIM static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { tlb_remove_page(tlb, table); } -#endif +#else +static inline +void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) +{ + tlb_remove_table(tlb, table); +} +#endif /* !CONFIG_PT_RECLAIM */ +#endif /* !CONFIG_PARAVIRT */ gfp_t __userpte_alloc_gfp = GFP_PGTABLE_USER | PGTABLE_HIGHMEM; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3a35546bac944..706b3c926a089 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -438,7 +438,9 @@ FOLIO_MATCH(compound_head, _head_2a); * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only. * @pt_rcu_head: For freeing page table pages. - * @pt_list: List of used page tables. Used for s390 and x86. + * @pt_list: List of used page tables. Used for s390 gmap shadow pages + * (which are not linked into the user page tables) and x86 + * pgds. * @_pt_pad_1: Padding that aliases with page's compound head. * @pmd_huge_pte: Protected by ptdesc->ptl, used for THPs. * @__page_mapping: Aliases with page->mapping. Unused for page tables. diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 99b3e9408aa0f..1e21022bcf339 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,11 +311,18 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) } } -static void tlb_remove_table_one(void *table) +#ifndef __tlb_remove_table_one +static inline void __tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); __tlb_remove_table(table); } +#endif + +static void tlb_remove_table_one(void *table) +{ + __tlb_remove_table_one(table); +} static void tlb_table_flush(struct mmu_gather *tlb) {