From patchwork Wed Feb 19 02:17:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luiz Capitulino X-Patchwork-Id: 13981428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E35B1C021AF for ; Wed, 19 Feb 2025 02:18:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 360252801CC; Tue, 18 Feb 2025 21:18:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 270612801CA; Tue, 18 Feb 2025 21:18:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C5132801CC; Tue, 18 Feb 2025 21:18:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E44D32801CA for ; Tue, 18 Feb 2025 21:18:22 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6BC1D160A47 for ; Wed, 19 Feb 2025 02:18:22 +0000 (UTC) X-FDA: 83135084844.16.0AFE2B1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id AA7D4180002 for ; Wed, 19 Feb 2025 02:18:20 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AxmZOGZs; spf=pass (imf06.hostedemail.com: domain of luizcap@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739931500; a=rsa-sha256; cv=none; b=mvG3byzxvWFRcz1RcnQNLlerE6EW9ccZkoPLYFXWY/5QQzxHmC205G12r7rD2ob8Lq6B94 24Xq9JPav008KSK0pb/HvboQgPPtyNCg7pPNYzlc0beMSP4KZWfBp8OCcTqw+nQO1mXBwa 8d21cx8jOvGbShGvXCgyP0xvCnrFO3s= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AxmZOGZs; spf=pass (imf06.hostedemail.com: domain of luizcap@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739931500; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GvFM6GZiRQiXqBZ3vCZFiH0Ays3za6p0+8us7Xm2FsA=; b=BcNl13gqi3GnTbbbgloaDBNNaGlYwOLhg81aheX1laVfy4C6s8r6jRN4UHk9OEVCpiAUFQ Uw224oSusSuw036Lc1x1UCP/aGpfrzn1QgS5bDj1402vWutIqHblxf1D13+0Udpa/nECj2 jlGQ0ZcxqbrTfd41VZBR3/lj1OZFI5o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739931500; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GvFM6GZiRQiXqBZ3vCZFiH0Ays3za6p0+8us7Xm2FsA=; b=AxmZOGZs2dhHxNycOG02YO2p/7RouMIIc2z4m20K2KX0aDn7B1fjcX67jB5MuolJo4n0HG Vwfp1cFjA9ZEfJ6yD2oNehoM/5UxmrBazDADXkOQEVh/IT8W1nxg0Gie1V82JDmGIPByZe mx/YjGRqtrvvPXsmcGDGn12OQBxGWU4= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-633-uF3_mlKrM2imXHAX3cLYTQ-1; Tue, 18 Feb 2025 21:18:16 -0500 X-MC-Unique: uF3_mlKrM2imXHAX3cLYTQ-1 X-Mimecast-MFC-AGG-ID: uF3_mlKrM2imXHAX3cLYTQ_1739931495 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 13B48180087A; Wed, 19 Feb 2025 02:18:15 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.65.50]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id BC6A719560B9; Wed, 19 Feb 2025 02:18:12 +0000 (UTC) From: Luiz Capitulino To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, david@redhat.com, yuzhao@google.com, pasha.tatashin@soleen.com Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, muchun.song@linux.dev, luizcap@redhat.com Subject: [PATCH 1/4] mm: page_ext: add an iteration API for page extensions Date: Tue, 18 Feb 2025 21:17:47 -0500 Message-ID: <3f0e058aef3951b39cf6bb4259c247352d4fe736.1739931468.git.luizcap@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Stat-Signature: yn7ee3u3m9uoxf3c634ro7kzn67kgwux X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: AA7D4180002 X-Rspam-User: X-HE-Tag: 1739931500-904722 X-HE-Meta: U2FsdGVkX1+gJjmWQOVQQAlOWiRdRkDpavpBLpC4dCB6ODw1D1fQKGwbtP3yxpPRyiK2FvmWqW+t8U4BWlLQus6A5hdWF4lETXZzaSB3n60lfF6cH8Fv4DmFWD3WSqfsGRZ/u4fpqfy1Ew+JTHAfDkkY3XuIGdiYzVKXQRr5UZp/0iMKUTB6uj2lNHcD9dJGy77y5gnb7mRJyjb9sk1ZCtINgzWABxcHe8bLNi5nLaM9qHM+WDmQTFwmjD6zZ1fIjQ5CqbOqgBykmecQ/1xJl3/VtuNSpXud1NoirjcTbE3dgVBZWRLSrguqBpMLW1Ut9Q6GZXt18ahBeJ6/L5ni8DqYkrIHKBpEa8q1JnI3zRdbaAtXE8GT33SWFIJntgQ5HtDKHG1PAK6NyChCs8XZft0vV1Ogmcga7DOutEymDLt9HapOhx85fUscP8xHxxkPdnkpCc7hDWmAo76l17P8XbHHCG+uAW3w+EdeMpsw8lKKSIQ85CnBdZzN1D2ExLi1PqHXNFGMi48FZMtJWHJKnyhnJ6d3rc/IWp1hxAodd+3UR5XGz1Ot2edDzAwUyLplwlYaNmREe3noMxr1bgW8Z5/5U/ay4t11e3RKqvDz90aznQFBh2c5AOhFTr2zT4dlE484T/5nB4O+L3NhnOWhnZoDlNRViGojyL+MzeCMPVG1K9Bwl8lnD1MIqEb1eAqyK5tEyQ1gnyY1VL2upYPEKY+hZ9IbELP+RW/1gYwLM4uBqFWw+zV1K8Ixn8PKjjWRvC4qwTohpkrZkSHYb3POZ0tChMxBXYBqOlJEHg0OLgA4VeR+BEjp0N8Qb/uYC2+tnMiKZy5kec4rANZn6H2QFB+0kyW+XN4PA4BL78uqg3WXYskIgTyHAoCTeAOlHJzxsUI7Pz4Jxz110O52o5M85SDTqdHykYvgKBKQ3PT/i+kS2OJ9/voU1daN9+BgH2jD56NWpLm336B1KSPBFQH 5DdlgJh6 fXfFNdcDqzgg62zjVyJ/RpLLVqev50Yz0NHkgW3846raoV9+jQe+hNYjcICo/XWvWiL6O8eYUJyCgRtR5EIBC0WEbveJ0B0HYSD8uekQM5zGODpn1Wl0wxfyF3s2soMsCnP4eOLueOK9BpaM0e3eyYYlJD9kuouEk3mRR4Er1fetI6hlXhKOwWrNmkPFVW7m1E2zPsAva54DXqW7UffxCQuPmiqFSYOELJY3dl2CayldgpojKLYaMuc/fIbB1YJMTd8q03REgSY244VmK/DT0WJiegB110mT0xB6ZZ+Jqir/i7Kpu2gqLUhwG0jBiXIL+37sc+XDir+hbAvOAe/o1ssGVR09kj51Dl16uWLEsCxKGfIWudP+UDRlc6g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The page extension implementation assumes that all page extensions of a given page order are stored in the same memory section. The function page_ext_next() relies on this assumption by adding an offset to the current object to return the next adjacent page extension. This behavior works as expected for flatmem but fails for sparsemem when using 1G pages. The commit cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") exposes this issue, making it possible for a crash when using page_owner or page_table_check page extensions. The problem is that for 1G pages, the page extensions may span memory section boundaries and be stored in different memory sections. This issue was not visible before commit cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") because alloc_contig_pages() never passed more than MAX_PAGE_ORDER to post_alloc_hook(). However, the series introducing mentioned commit changed this behavior allowing the full 1G page order to be passed. Reproducer: 1. Build the kernel with CONFIG_SPARSEMEM=y and table extensions support 2. Pass 'default_hugepagesz=1 page_owner=on' in the kernel command-line 3. Reserve one 1G page at run-time, this should crash (backtrace below) To address this issue, this commit introduces a new API for iterating through page extensions. The main iteration loops are for_each_page_ext() and for_each_page_ext_order(). Both must be called with the RCU read lock taken. Here's an usage example: """ struct page_ext_iter iter; struct page_ext *page_ext; ... rcu_read_lock(); for_each_page_ext_order(page, order, page_ext, iter) { struct my_page_ext *obj = get_my_page_ext_obj(page_ext); ... } rcu_read_unlock(); """ Both loop constructs use page_ext_iter_next(), which checks to see if we have crossed sections in the iteration. In this case, page_ext_iter_next() retrieves the next page_ext object from another section. Thanks to David Hildenbrand for helping identify the root cause and providing suggestions on how to fix and optmize the solution (final implementation and bugs are all mine through). Lastly, here's the backtrace, without kasan you can get random crashes: [ 76.052526] BUG: KASAN: slab-out-of-bounds in __update_page_owner_handle+0x238/0x298 [ 76.060283] Write of size 4 at addr ffff07ff96240038 by task tee/3598 [ 76.066714] [ 76.068203] CPU: 88 UID: 0 PID: 3598 Comm: tee Kdump: loaded Not tainted 6.13.0-rep1 #3 [ 76.076202] Hardware name: WIWYNN Mt.Jade Server System B81.030Z1.0007/Mt.Jade Motherboard, BIOS 2.10.20220810 (SCP: 2.10.20220810) 2022/08/10 [ 76.088972] Call trace: [ 76.091411] show_stack+0x20/0x38 (C) [ 76.095073] dump_stack_lvl+0x80/0xf8 [ 76.098733] print_address_description.constprop.0+0x88/0x398 [ 76.104476] print_report+0xa8/0x278 [ 76.108041] kasan_report+0xa8/0xf8 [ 76.111520] __asan_report_store4_noabort+0x20/0x30 [ 76.116391] __update_page_owner_handle+0x238/0x298 [ 76.121259] __set_page_owner+0xdc/0x140 [ 76.125173] post_alloc_hook+0x190/0x1d8 [ 76.129090] alloc_contig_range_noprof+0x54c/0x890 [ 76.133874] alloc_contig_pages_noprof+0x35c/0x4a8 [ 76.138656] alloc_gigantic_folio.isra.0+0x2c0/0x368 [ 76.143616] only_alloc_fresh_hugetlb_folio.isra.0+0x24/0x150 [ 76.149353] alloc_pool_huge_folio+0x11c/0x1f8 [ 76.153787] set_max_huge_pages+0x364/0xca8 [ 76.157961] __nr_hugepages_store_common+0xb0/0x1a0 [ 76.162829] nr_hugepages_store+0x108/0x118 [ 76.167003] kobj_attr_store+0x3c/0x70 [ 76.170745] sysfs_kf_write+0xfc/0x188 [ 76.174492] kernfs_fop_write_iter+0x274/0x3e0 [ 76.178927] vfs_write+0x64c/0x8e0 [ 76.182323] ksys_write+0xf8/0x1f0 [ 76.185716] __arm64_sys_write+0x74/0xb0 [ 76.189630] invoke_syscall.constprop.0+0xd8/0x1e0 [ 76.194412] do_el0_svc+0x164/0x1e0 [ 76.197891] el0_svc+0x40/0xe0 [ 76.200939] el0t_64_sync_handler+0x144/0x168 [ 76.205287] el0t_64_sync+0x1ac/0x1b0 Fixes: cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") Signed-off-by: Luiz Capitulino --- include/linux/page_ext.h | 66 ++++++++++++++++++++++++++++++++++++++++ mm/page_ext.c | 41 +++++++++++++++++++++++++ 2 files changed, 107 insertions(+) diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index e4b48a0dda244..a99da12e59fa7 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -3,6 +3,7 @@ #define __LINUX_PAGE_EXT_H #include +#include #include struct pglist_data; @@ -69,12 +70,26 @@ extern void page_ext_init(void); static inline void page_ext_init_flatmem_late(void) { } + +static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn) +{ + /* + * page_ext is allocated per memory section. Once we cross a + * memory section, we have to fetch the new pointer. + */ + return next_pfn % PAGES_PER_SECTION; +} #else extern void page_ext_init_flatmem(void); extern void page_ext_init_flatmem_late(void); static inline void page_ext_init(void) { } + +static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn) +{ + return true; +} #endif extern struct page_ext *page_ext_get(const struct page *page); @@ -93,6 +108,57 @@ static inline struct page_ext *page_ext_next(struct page_ext *curr) return next; } +struct page_ext_iter { + unsigned long pfn; + unsigned long index; + struct page_ext *page_ext; +}; + +struct page_ext *page_ext_iter_begin(struct page_ext_iter *iter, struct page *page); +struct page_ext *page_ext_iter_next(struct page_ext_iter *iter); + +/** + * page_ext_iter_get() - Get current page extension + * @iter: page extension iterator. + * + * Return: NULL if no page_ext exists for this iterator. + */ +static inline struct page_ext *page_ext_iter_get(const struct page_ext_iter *iter) +{ + return iter->page_ext; +} + +/** + * for_each_page_ext(): iterate through page_ext objects. + * @__page: the page we're interested in + * @__pgcount: how many pages to iterate through + * @__page_ext: struct page_ext pointer where the current page_ext + * object is returned + * @__iter: struct page_ext_iter object (defined in the stack) + * + * IMPORTANT: must be called with RCU read lock taken. + */ +#define for_each_page_ext(__page, __pgcount, __page_ext, __iter) \ + __page_ext = page_ext_iter_begin(&__iter, __page); \ + for (__iter.index = 0; \ + __page_ext && __iter.index < __pgcount; \ + __page_ext = page_ext_iter_next(&__iter), \ + __iter.index++) + +/** + * for_each_page_ext_order(): iterate through page_ext objects + * for a given page order + * @__page: the page we're interested in + * @__order: page order to iterate through + * @__page_ext: struct page_ext pointer where the current page_ext + * object is returned + * @__iter: struct page_ext_iter object (defined in the stack) + * + * IMPORTANT: must be called with RCU read lock taken. + */ +#define for_each_page_ext_order(__page, __order, __page_ext, __iter) \ + for_each_page_ext(__page, (1UL << __order), __page_ext, __iter) + #else /* !CONFIG_PAGE_EXTENSION */ struct page_ext; diff --git a/mm/page_ext.c b/mm/page_ext.c index 641d93f6af4c1..508deb04d5ead 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -549,3 +549,44 @@ void page_ext_put(struct page_ext *page_ext) rcu_read_unlock(); } + +/** + * page_ext_iter_begin() - Prepare for iterating through page extensions. + * @iter: page extension iterator. + * @page: The page we're interested in. + * + * Must be called with RCU read lock taken. + * + * Return: NULL if no page_ext exists for this page. + */ +struct page_ext *page_ext_iter_begin(struct page_ext_iter *iter, struct page *page) +{ + iter->pfn = page_to_pfn(page); + iter->page_ext = lookup_page_ext(page); + + return iter->page_ext; +} + +/** + * page_ext_iter_next() - Get next page extension + * @iter: page extension iterator. + * + * Must be called with RCU read lock taken. + * + * Return: NULL if no next page_ext exists. + */ +struct page_ext *page_ext_iter_next(struct page_ext_iter *iter) +{ + if (WARN_ON_ONCE(!iter->page_ext)) + return NULL; + + iter->pfn++; + + if (page_ext_iter_next_fast_possible(iter->pfn)) { + iter->page_ext = page_ext_next(iter->page_ext); + } else { + iter->page_ext = lookup_page_ext(pfn_to_page(iter->pfn)); + } + + return iter->page_ext; +} From patchwork Wed Feb 19 02:17:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luiz Capitulino X-Patchwork-Id: 13981429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24C16C021AD for ; Wed, 19 Feb 2025 02:18:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3D562801CD; Tue, 18 Feb 2025 21:18:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F87B2801CA; Tue, 18 Feb 2025 21:18:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 841D32801CD; Tue, 18 Feb 2025 21:18:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 655BF2801CA for ; Tue, 18 Feb 2025 21:18:26 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 219B01A0AF0 for ; Wed, 19 Feb 2025 02:18:26 +0000 (UTC) X-FDA: 83135085012.19.5F67D44 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 64E1440002 for ; Wed, 19 Feb 2025 02:18:24 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EQsvH8pX; spf=pass (imf01.hostedemail.com: domain of luizcap@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739931504; a=rsa-sha256; cv=none; b=8FBG95eKThaXw8CdMYiOb2YKTKqO2eepq4k9kz6BfcTkfOnRGD4k3BQQp56WZjGj4ViQGk nZPcWxMbJcaMZNFrB15O7A25mE13lwsFbZIHBYo5DkKa7x8dcTahPtBG8Qs6mP7X8OtNuR AyMfK/QXU06A0sN5c8EGCl2OzJx3OQc= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EQsvH8pX; spf=pass (imf01.hostedemail.com: domain of luizcap@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739931504; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NsF2KsmlxImjdJhL9OxCFldmZagOaePVCluv2R2e7Ow=; b=5xWkZP18TDbMaRHHOXM7m4I90p6fRgOQyCGH+rWKYgopad1f0eJjaUtYDmjS89mlswlOLs oVD421gpnzSQdFkLnwRv79O90P/8oC2lbBC9Sp96rAXUo8oCtpfViMq5qqCs1rB9pXoRu4 9UHeLEYqqFOID3p+uM5NApDzcUzPBeU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739931503; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NsF2KsmlxImjdJhL9OxCFldmZagOaePVCluv2R2e7Ow=; b=EQsvH8pXdPy8BwSszJe+fYwvS7BS7hLaUzZpCKZq+SFwA4499N8tpWrxqwPDjZkLTnoYQ1 LpfEGu2RodMiJYA0tlCrFeV3Z9SVtCe+lbGIDH/Fo8MjseiAhz2fLw6OM62DjqnHI+KwGM uQLq3HoGw/9jIB0sAOUQRqbtdF4NLvQ= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-599-8odB42S2OHSsiU1HfMlZmg-1; Tue, 18 Feb 2025 21:18:19 -0500 X-MC-Unique: 8odB42S2OHSsiU1HfMlZmg-1 X-Mimecast-MFC-AGG-ID: 8odB42S2OHSsiU1HfMlZmg_1739931498 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A6C1818EB2CF; Wed, 19 Feb 2025 02:18:17 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.65.50]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 59B1D19560B9; Wed, 19 Feb 2025 02:18:15 +0000 (UTC) From: Luiz Capitulino To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, david@redhat.com, yuzhao@google.com, pasha.tatashin@soleen.com Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, muchun.song@linux.dev, luizcap@redhat.com Subject: [PATCH 2/4] mm: page_table_check: use new iteration API Date: Tue, 18 Feb 2025 21:17:48 -0500 Message-ID: <85f11743d259d5e4a1f47456fbcda82ff6db9ab3.1739931468.git.luizcap@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Stat-Signature: jrcy9dbicw76915okcbrowuqh1kb3y9a X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 64E1440002 X-Rspam-User: X-HE-Tag: 1739931504-51527 X-HE-Meta: U2FsdGVkX1/FAmdi9HfTUU1KGVEU22gFp0SjLOv43ykY3INNT0RzsqzXeTxTQiRBVhpsLhmZOOVFVW8fO17nXUNQL0Cnjw+13z0hiPxYeJoz+KZrpg5iN1u7y01xosPvfWAsObRTU5JMKECpI9XXlGEimP/mGj3yfXFr+hDjYGANtkc4zcUKniHzSorHm8wVm1GtOEhyAb65UNZAYO7lbhcph3DOm03jfbW62fv6oAfw76UMYhCr3lzYSzCjDpXg9fDsBTfd2QlJaLw37w1L64/VPRDnQxRnPc/WHTlIkpzZHIRqQaARAaxgkwkxR6FYZdPtvCk33qhmmEnxtDnar9G6u2ixzkAiYrw+2frGRmBV1wX7mnigo1O683rNKDR/YuuGDIax/PxsI23psGDj/JQ2ocEg0quj+Fd6AQ626ArB5kxgviD6nc0U2rSO7bfUjrG7/loqpLVM5+xDUnhB1ICj5P2aScKK9FFFgOsWkQwrDe7CKoRHCbXqvD0gzOq7snFZytL8o33UvZ3Mxg+i0g4rRSzZASBbWCMYHXk3iaqL0qEYSHbAzNkNWfxI1Pg6dOS4XGf7UkMpjVAdJ3fMaSPlbtEvlDBAeHnr2SBL9JlIaP+VOSjn+Dza8OdhM8m9LsKPIOZMOwhQHfG6JEUNayy5rNdVpn2j7O6bOa3Oh3v9pKFXX1BEHxJVWvM2PjU8iehL1sXQcC/4qHQ7yZmY824dn20rOCHjXYlZzUN6+eUu8AkB0t2K0rtMqd/EZuhKBjp+sgjjb/9+MlArpOknFCaeceS7i70RomoIwZVXGn/xZ0FC56ksnakMH36+0ZKr/3n2qzA3zkWVyLOrgzM1TDoA4bU7pxYTxWJFoXTSMhKYeupa/4m/X5ifoPgL4mxmv8VbbS9wNCyc8GxQGPm0aKs8xCDZmb6uA5F7U9b1RXiUXpEe4T8mA5OUQMzvuhJd3yHL6F1c7mHQFmAb6II vBE+hSYm mO5JnWx0UaTEdChav1+7V7OJ0Zsa+MhjzkXLYmjM1Y0l+XX7OZY8DfFiNwvPxB0i8AaN5x8MA2IwsOM8vwXhLWePMg7inhFLEPi5N4sIIJkGuJ/8Z230g35C77PhlxUm0pCE9JFPTuqP8iyEoUweNhMBrn2gJDEnGIMevSmOm9r/gEBAU78ApVKHKKdop8p0i7fI8e3UWTsTP4FNOqw5Ik/GeEjiQW+3Gq2Kn4t4XaSsHSscV0dAK4TXAo9wSu4fEN2hN+dFS+X2CcnZg1QOSIdpT73Zz1wUbgf0bSDYdVfg8S9vDlK2HnCL3zEDr8nue6bI1gg7zoUrxC+jTprSZvXY/Y9b57FNwGwy0+zNPIK82B1PAj9UBkxh9qQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The page_ext_next() function assumes that page extension objects for a page order allocation always reside in the same memory section, which may not be true and could lead to crashes. Use the new page_ext iteration API instead. Fixes: cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") Signed-off-by: Luiz Capitulino Acked-by: David Hildenbrand --- mm/page_table_check.c | 39 ++++++++++++--------------------------- 1 file changed, 12 insertions(+), 27 deletions(-) diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 509c6ef8de400..b52e04d31c809 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -62,24 +62,20 @@ static struct page_table_check *get_page_table_check(struct page_ext *page_ext) */ static void page_table_check_clear(unsigned long pfn, unsigned long pgcnt) { + struct page_ext_iter iter; struct page_ext *page_ext; struct page *page; - unsigned long i; bool anon; if (!pfn_valid(pfn)) return; page = pfn_to_page(pfn); - page_ext = page_ext_get(page); - - if (!page_ext) - return; - BUG_ON(PageSlab(page)); anon = PageAnon(page); - for (i = 0; i < pgcnt; i++) { + rcu_read_lock(); + for_each_page_ext(page, pgcnt, page_ext, iter) { struct page_table_check *ptc = get_page_table_check(page_ext); if (anon) { @@ -89,9 +85,8 @@ static void page_table_check_clear(unsigned long pfn, unsigned long pgcnt) BUG_ON(atomic_read(&ptc->anon_map_count)); BUG_ON(atomic_dec_return(&ptc->file_map_count) < 0); } - page_ext = page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } /* @@ -102,24 +97,20 @@ static void page_table_check_clear(unsigned long pfn, unsigned long pgcnt) static void page_table_check_set(unsigned long pfn, unsigned long pgcnt, bool rw) { + struct page_ext_iter iter; struct page_ext *page_ext; struct page *page; - unsigned long i; bool anon; if (!pfn_valid(pfn)) return; page = pfn_to_page(pfn); - page_ext = page_ext_get(page); - - if (!page_ext) - return; - BUG_ON(PageSlab(page)); anon = PageAnon(page); - for (i = 0; i < pgcnt; i++) { + rcu_read_lock(); + for_each_page_ext(page, pgcnt, page_ext, iter) { struct page_table_check *ptc = get_page_table_check(page_ext); if (anon) { @@ -129,9 +120,8 @@ static void page_table_check_set(unsigned long pfn, unsigned long pgcnt, BUG_ON(atomic_read(&ptc->anon_map_count)); BUG_ON(atomic_inc_return(&ptc->file_map_count) < 0); } - page_ext = page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } /* @@ -140,24 +130,19 @@ static void page_table_check_set(unsigned long pfn, unsigned long pgcnt, */ void __page_table_check_zero(struct page *page, unsigned int order) { + struct page_ext_iter iter; struct page_ext *page_ext; - unsigned long i; BUG_ON(PageSlab(page)); - page_ext = page_ext_get(page); - - if (!page_ext) - return; - - for (i = 0; i < (1ul << order); i++) { + rcu_read_lock(); + for_each_page_ext_order(page, order, page_ext, iter) { struct page_table_check *ptc = get_page_table_check(page_ext); BUG_ON(atomic_read(&ptc->anon_map_count)); BUG_ON(atomic_read(&ptc->file_map_count)); - page_ext = page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } void __page_table_check_pte_clear(struct mm_struct *mm, pte_t pte) From patchwork Wed Feb 19 02:17:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luiz Capitulino X-Patchwork-Id: 13981431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16A25C021AA for ; Wed, 19 Feb 2025 02:18:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D9A302801CE; Tue, 18 Feb 2025 21:18:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CD13F2801CA; Tue, 18 Feb 2025 21:18:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B74892801CE; Tue, 18 Feb 2025 21:18:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8E67F2801CA for ; Tue, 18 Feb 2025 21:18:29 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4DE08B0CD0 for ; Wed, 19 Feb 2025 02:18:29 +0000 (UTC) X-FDA: 83135085138.06.2B305BF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 9BE161C0007 for ; Wed, 19 Feb 2025 02:18:27 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XBI6QU0E; spf=pass (imf20.hostedemail.com: domain of luizcap@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739931507; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sO1RKRz7AmW+mPQ8kuacqDxnDkJ26PvSRyCVK8dkxcg=; b=ISzWU2NQw1iJjOsf+yeR79Kj+FlhzvlranNHmCgyOa6ud08Y6dL0fET0pM3vf1VRpZ0db1 V+RsQzoDkir79OHnXNMtg7S4TYaP+nluOoJSO41CK/kuw0dXTdsQdBf4cxvrQzxlxIDhSc e8MMaW7WPxg1TRdFRf99OW4Evmd/+Ck= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XBI6QU0E; spf=pass (imf20.hostedemail.com: domain of luizcap@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739931507; a=rsa-sha256; cv=none; b=FCSrP/1QOwjTS57mQZi/CIJWINfJheyBYfbaSZchHbINuPGMIZt37mfn229+R74GfmUfVu aaPZX/XUdyvyegXzSKbMcusx7HzA/FAIK6cZWDh67c+mgkPDhSjRrMQuXxMHSzvKDRpMt6 98iWbu38Q/XLuS91ibrUPiCGG3/A/gg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739931507; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sO1RKRz7AmW+mPQ8kuacqDxnDkJ26PvSRyCVK8dkxcg=; b=XBI6QU0ENEuCwnc8j7ekfiLcmZTKpyIAcB47uNrYMWhxmXDFKXzE5+a3yZZovN6Xw/kLJY 6A8IFRqdYCjkBZqTJOe+UE0MTXgrheKec2GXiVW+aeldr8PEvE2veEMJUioziuSNowkKMi Y9dkDAmOVSpapFKu7Vp1oJmDQavyEiM= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-214-kACK9iz7N-iKjG_UQRlKJQ-1; Tue, 18 Feb 2025 21:18:21 -0500 X-MC-Unique: kACK9iz7N-iKjG_UQRlKJQ-1 X-Mimecast-MFC-AGG-ID: kACK9iz7N-iKjG_UQRlKJQ_1739931500 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id F35CB196E078; Wed, 19 Feb 2025 02:18:19 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.65.50]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id ECCAB1955F0F; Wed, 19 Feb 2025 02:18:17 +0000 (UTC) From: Luiz Capitulino To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, david@redhat.com, yuzhao@google.com, pasha.tatashin@soleen.com Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, muchun.song@linux.dev, luizcap@redhat.com Subject: [PATCH 3/4] mm: page_owner: use new iteration API Date: Tue, 18 Feb 2025 21:17:49 -0500 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9BE161C0007 X-Stat-Signature: ssqrfkamtstrhos3pmgbhboj4zor891f X-HE-Tag: 1739931507-26422 X-HE-Meta: U2FsdGVkX1+cDdd2Cv9VacaopXdsqpOLzMD5qyB3HxCUAx46KZl7dgwub6GdsM3uAc2SGD8U5n0XEchyQQ7hK0DM7oJBBptldkpRZNr+gwk+vqYv9GmnBMvdyZULyWdmO9hAC9HGHs8Xz61xZ38CalhYSA4/QKYAB1Q/t4mQq9BtNkJQod4BMp8rKZmBlt6mtyMcgdDgky2UO/CCe0UKOMivpPE7qFtI6bDtVRpmHZhlWHffykeR7xd0zsmFDmGAmdYn5X4HJ6IfR3ro4GBUXwzYnplN1fSFCc7GgUGNxdjc5CJiP/54A2AIGlK8P7lbp+iVxfgwukp9BlMeVAPAE22YznditybIfjwXA/QfhNJZYiLllVhoiYRz7Isutd/p+OkFIm1OIKeon9i9s56dyTiIM97VCYmndKXAYau48S5UJiWRjMxtNHGf331cYHgUVXDtIXvxisuszLkguWDg+uiaiDgNeLP1A/yvaJeFRfyx17A0v/e2z8dEgN8PiZ1YsxflpxTJTVDOL1v24mo6jTsve1NMW2lqpgmTZPRiEKimiptvVepWrwQiUrUtBVUmuoocFLu7sjPIK4XnNceiBfNWo99EjUKd1Q7qxpeuyZZMwXE0ygKl4Q3dzNTUMbT1ujiTOiZDuOmZiyR8RJLR+sMhpdxHcfwKcTy5Qs02ykGJ3+7X/Wmw4Bc+e53kIuTzp1v0tM7b7YZ8WGp/MEMjiAimtTG+2X++iqlITrBM6OY0q3ihMnd9BWPWXyD54tpDMDT97opyoRp81zxqiiZolc1fwchJQd5PxSvhc+81c6lE6pgiUK9MZqEdYdjH8ApCwqOK3kmOtAhtW504wXD9jbWw2wFsS5gTiihvjguii7cF9oCaGfp9n4QkCY85v3xIPJwSnYMu4Okl9Y3gK/17oMud1QUOszBJMSfk5KTsE829IrAhMCJH4QSf4spMPCq815iDCKLoO6kq+jrTOLe 5wiceaNf wDPk5lEtijLDBNCLfcHN9196S/v91rZ62LRCi1z7+OhO9g2VkIqw2oya4CBsrCpDuU5+MVoE3ggHx96J4XEjvZNsKbU61pe23JnZvN9vHJY82yrm9himXiOL/hEWvT2NPyzzlWWuan9Jc2jY1rnxII8eAs5XltfZpZ75qolEaHmC0lyWWTdAXXaEuhwhUXhANOZJj4Oq/g29u5XyxiwJiQc/p+l0nbzagkuZQA+h9prYmUEbZ6vHgb1ZcTM/7eeiyTqt+88APcRcDqA80zQ1ZBSDE9gozteX+kt2pe3AWvRL67ZsXaEG+aJes20znSBkLvFK+IYNnEemcXqHOWqyT+fX0j8SI4cV00HV46e7DXnjKyMA9KoFKMjIIhQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The page_ext_next() function assumes that page extension objects for a page order allocation always reside in the same memory section, which may not be true and could lead to crashes. Use the new page_ext iteration API instead. Fixes: cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") Signed-off-by: Luiz Capitulino --- mm/page_owner.c | 61 +++++++++++++++++++++++-------------------------- 1 file changed, 29 insertions(+), 32 deletions(-) diff --git a/mm/page_owner.c b/mm/page_owner.c index 2d6360eaccbb6..9afc62a882813 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -229,17 +229,19 @@ static void dec_stack_record_count(depot_stack_handle_t handle, handle); } -static inline void __update_page_owner_handle(struct page_ext *page_ext, +static inline void __update_page_owner_handle(struct page *page, depot_stack_handle_t handle, unsigned short order, gfp_t gfp_mask, short last_migrate_reason, u64 ts_nsec, pid_t pid, pid_t tgid, char *comm) { - int i; + struct page_ext_iter iter; + struct page_ext *page_ext; struct page_owner *page_owner; - for (i = 0; i < (1 << order); i++) { + rcu_read_lock(); + for_each_page_ext_order(page, order, page_ext, iter) { page_owner = get_page_owner(page_ext); page_owner->handle = handle; page_owner->order = order; @@ -252,20 +254,22 @@ static inline void __update_page_owner_handle(struct page_ext *page_ext, sizeof(page_owner->comm)); __set_bit(PAGE_EXT_OWNER, &page_ext->flags); __set_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); - page_ext = page_ext_next(page_ext); } + rcu_read_unlock(); } -static inline void __update_page_owner_free_handle(struct page_ext *page_ext, +static inline void __update_page_owner_free_handle(struct page *page, depot_stack_handle_t handle, unsigned short order, pid_t pid, pid_t tgid, u64 free_ts_nsec) { - int i; + struct page_ext_iter iter; + struct page_ext *page_ext; struct page_owner *page_owner; - for (i = 0; i < (1 << order); i++) { + rcu_read_lock(); + for_each_page_ext_order(page, order, page_ext, iter) { page_owner = get_page_owner(page_ext); /* Only __reset_page_owner() wants to clear the bit */ if (handle) { @@ -275,8 +279,8 @@ static inline void __update_page_owner_free_handle(struct page_ext *page_ext, page_owner->free_ts_nsec = free_ts_nsec; page_owner->free_pid = current->pid; page_owner->free_tgid = current->tgid; - page_ext = page_ext_next(page_ext); } + rcu_read_unlock(); } void __reset_page_owner(struct page *page, unsigned short order) @@ -293,11 +297,11 @@ void __reset_page_owner(struct page *page, unsigned short order) page_owner = get_page_owner(page_ext); alloc_handle = page_owner->handle; + page_ext_put(page_ext); handle = save_stack(GFP_NOWAIT | __GFP_NOWARN); - __update_page_owner_free_handle(page_ext, handle, order, current->pid, + __update_page_owner_free_handle(page, handle, order, current->pid, current->tgid, free_ts_nsec); - page_ext_put(page_ext); if (alloc_handle != early_handle) /* @@ -313,19 +317,13 @@ void __reset_page_owner(struct page *page, unsigned short order) noinline void __set_page_owner(struct page *page, unsigned short order, gfp_t gfp_mask) { - struct page_ext *page_ext; u64 ts_nsec = local_clock(); depot_stack_handle_t handle; handle = save_stack(gfp_mask); - - page_ext = page_ext_get(page); - if (unlikely(!page_ext)) - return; - __update_page_owner_handle(page_ext, handle, order, gfp_mask, -1, + __update_page_owner_handle(page, handle, order, gfp_mask, -1, ts_nsec, current->pid, current->tgid, current->comm); - page_ext_put(page_ext); inc_stack_record_count(handle, gfp_mask, 1 << order); } @@ -344,26 +342,24 @@ void __set_page_owner_migrate_reason(struct page *page, int reason) void __split_page_owner(struct page *page, int old_order, int new_order) { - int i; - struct page_ext *page_ext = page_ext_get(page); + struct page_ext_iter iter; + struct page_ext *page_ext; struct page_owner *page_owner; - if (unlikely(!page_ext)) - return; - - for (i = 0; i < (1 << old_order); i++) { + rcu_read_lock(); + for_each_page_ext_order(page, old_order, page_ext, iter) { page_owner = get_page_owner(page_ext); page_owner->order = new_order; - page_ext = page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } void __folio_copy_owner(struct folio *newfolio, struct folio *old) { - int i; struct page_ext *old_ext; struct page_ext *new_ext; + struct page_ext *page_ext; + struct page_ext_iter iter; struct page_owner *old_page_owner; struct page_owner *new_page_owner; depot_stack_handle_t migrate_handle; @@ -381,7 +377,7 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) old_page_owner = get_page_owner(old_ext); new_page_owner = get_page_owner(new_ext); migrate_handle = new_page_owner->handle; - __update_page_owner_handle(new_ext, old_page_owner->handle, + __update_page_owner_handle(&newfolio->page, old_page_owner->handle, old_page_owner->order, old_page_owner->gfp_mask, old_page_owner->last_migrate_reason, old_page_owner->ts_nsec, old_page_owner->pid, @@ -391,7 +387,7 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) * will be freed after migration. Keep them until then as they may be * useful. */ - __update_page_owner_free_handle(new_ext, 0, old_page_owner->order, + __update_page_owner_free_handle(&newfolio->page, 0, old_page_owner->order, old_page_owner->free_pid, old_page_owner->free_tgid, old_page_owner->free_ts_nsec); @@ -400,11 +396,12 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) * for the new one and the old folio otherwise there will be an imbalance * when subtracting those pages from the stack. */ - for (i = 0; i < (1 << new_page_owner->order); i++) { + rcu_read_lock(); + for_each_page_ext_order(&old->page, new_page_owner->order, page_ext, iter) { + old_page_owner = get_page_owner(page_ext); old_page_owner->handle = migrate_handle; - old_ext = page_ext_next(old_ext); - old_page_owner = get_page_owner(old_ext); } + rcu_read_unlock(); page_ext_put(new_ext); page_ext_put(old_ext); @@ -813,7 +810,7 @@ static void init_pages_in_zone(pg_data_t *pgdat, struct zone *zone) goto ext_put_continue; /* Found early allocated page */ - __update_page_owner_handle(page_ext, early_handle, 0, 0, + __update_page_owner_handle(page, early_handle, 0, 0, -1, local_clock(), current->pid, current->tgid, current->comm); count++; From patchwork Wed Feb 19 02:17:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luiz Capitulino X-Patchwork-Id: 13981430 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 606ABC021AA for ; Wed, 19 Feb 2025 02:18:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F4542801BB; Tue, 18 Feb 2025 21:18:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 002A02801CA; Tue, 18 Feb 2025 21:18:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE4832801CE; Tue, 18 Feb 2025 21:18:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BE91E2801CA for ; Tue, 18 Feb 2025 21:18:27 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3B22AC09FC for ; Wed, 19 Feb 2025 02:18:27 +0000 (UTC) X-FDA: 83135085054.16.E63C3B5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 7B6FB1A0010 for ; Wed, 19 Feb 2025 02:18:25 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KMeBtSc3; spf=pass (imf19.hostedemail.com: domain of luizcap@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739931505; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v8m9/U7uQSSFPcBA25A4fBqyiaCO54n8VIlpbaNJ5EU=; b=ZESi2ywx1MqTc1IGiBrJG3EjsZa1UaRF7ymPqPeImhe/PvvlFTErz6tbzZchLHTySO6pzy ZAh0qOOEuwLFDfmGSH76E/eUC62CoixbJyLMA9bf5r3z8uIvQhAOkzDUMkYyTQWltkF8NH R/joPv7bgO7pcUvTBxEWRz3YCyrnpYs= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KMeBtSc3; spf=pass (imf19.hostedemail.com: domain of luizcap@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739931505; a=rsa-sha256; cv=none; b=LUKuU1l3ABSiflXCQRC/0iYuXPC+KT+NhrYoIS3GSzvIIzrCG3D6+uMn7VFogWqQ0cPa+V QykDADtKbREaCvBPdjVx5IEF24TXDF1WjZhqU/x9umVAPGmyH3gH1OI27t65ikL3RKy+SK KQCngVjEjrVqO466rrJyuSM+6W8TkL8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739931504; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v8m9/U7uQSSFPcBA25A4fBqyiaCO54n8VIlpbaNJ5EU=; b=KMeBtSc31WCwMFOPm0p+VGo7RBKYgzK615glj7NEB4C8Js5F196Q08jEJxGHuJeX0pQzxj CPr/T/o5BlFwsb3Q1RjTtvfU4d/I65d1izgV9SySNP2WQGQANWUPMjfR4xHTG02HpqTFdw rhXte5kjvvXE5B8qe2Rv1a79Kf0atCo= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-338-rLhuEFm1MNKyofazmyW6_g-1; Tue, 18 Feb 2025 21:18:23 -0500 X-MC-Unique: rLhuEFm1MNKyofazmyW6_g-1 X-Mimecast-MFC-AGG-ID: rLhuEFm1MNKyofazmyW6_g_1739931502 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4186619373DC; Wed, 19 Feb 2025 02:18:22 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.65.50]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 44EAA1956094; Wed, 19 Feb 2025 02:18:20 +0000 (UTC) From: Luiz Capitulino To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, david@redhat.com, yuzhao@google.com, pasha.tatashin@soleen.com Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, muchun.song@linux.dev, luizcap@redhat.com Subject: [PATCH 4/4] mm: page_ext: make page_ext_next() private to page_ext Date: Tue, 18 Feb 2025 21:17:50 -0500 Message-ID: <5794ff5b322febd376728c8e22c802c15827dcc8.1739931468.git.luizcap@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7B6FB1A0010 X-Stat-Signature: 3t1w6whau38prj9rs8jcc3e83b6bowmu X-HE-Tag: 1739931505-233482 X-HE-Meta: U2FsdGVkX1/u+vMW8Xt0ib0HWPn1iBmwDOZl66HfbmqfbsLJZDfojfpPJm/BV04lueX8NeSbB4Afp54aMM6r72KT/b35QtsU9v9isYIlPoEjHkqhN2yBGBvZgZIJgDce9mnCu+QmleUOldVm2Eqom8tbanozJyPykdDi2kddFhrB5ww6NePcoI+J2ZeaGbl9lZOLhxgZIfGimltJ9D8+Hvhzc+LsWPxb3BqV8LBk+hXX3g8j7j5MDPcoIEYaYOcYABTV/u/Ma3Ulpk6hNSCtq+MAmaYENHvZQYxPKYxna5DAVgSWJYBtVVkX0l7i6VmeIoipvGk1fpSS5Nr+NWcvgi8KWhc7c9AUI4Ei8yemPje4ll2YF9EgrSd4lIUDdcykqyKLzoCeTKFVj0MTIXCg/zz+E/T+K8PUV/eIj/fdnaDkgE3zJumbORYBw8u6GxsKkOg6xUyPKPmcVrXrkR2h4r8zXF2ZIMZK1K8n3NH28CYG28oBB2sZgf5cqreAHshdXc3VAYVn3C8IutxmqTw2nGz1fOafaqE14L34fzLNCMUSpOJIwZsVrfLQ1DxHwbtZizvmmAzX49EYGDzu7sYlEVSMQeIgcAOO0P5Ec5dSEqeoP+1RCa01ZlCd23nAq7dfFt7FY+kiljmMoBJWValDM2UBpianrrq5GvaJ4KCGsSJuW6tPKjOqzFb4IJGfO6dX6ImoFtLZAKL6rCL2xo7zWUXCN3KEQCNQNNhZVWNM1iUm7zyZNwy063y75GQdH//FxfjkUMw5Lz75IoQ+XnGY6cuAt05b7MSCDZKOwiKCRgs877kTNYkF2sIYrMS68BcywneVvtBSelYkcjrSGg+7Jc7JP+CyoCe6BX7C8jmc814E1ifcx5M3etmrKvMpCD/hBIKACs21fgTrM4GB0jvo0N+e574hCLcVQvjNhVt9jUr7KI2QJONcrQ8eTGZGRlLz70BghaqR6jIaPEULZMI oRtv4ArO WO5QXtoLYpYlwfSKvnh1g2N7eWz1RvpjKLVVCM0SiA52ndfFuxoMlF+Zc9v9vjiugeHMnzM7E8inkhaR1h2W23KygPKYXAo4N9yJlt/utwmN2pVTsOcUxPQjUubKXh6YFLGODjtgYxD8GT23gtKB0j+SNm4Sdl8VMLykFfgURswdv2usMEcz2ja2fgDENCf4LVbiFZXqnugluChgQIq99AYX8lbP7KQBLyX+IeaB3EvxTs3nBWdp6NjRMj/fb52YhVeA7J2/Et+ouPi/aiWQBi9bxXTIerekx1giNl2uSycJqjAyMQeOHDOcp0a5L2AXEULJihu3gBbFVl6qSqZjNrfxyqnP8CUno22Vc5wZGXsdNZBVeTbvCIZaMfA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Previous commits removed page_ext_next() use from page_ext clients. Make it private. Fixes: cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") Signed-off-by: Luiz Capitulino --- include/linux/page_ext.h | 7 ------- mm/page_ext.c | 7 +++++++ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index a99da12e59fa7..3a4f0f825aa59 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -101,13 +101,6 @@ static inline void *page_ext_data(struct page_ext *page_ext, return (void *)(page_ext) + ops->offset; } -static inline struct page_ext *page_ext_next(struct page_ext *curr) -{ - void *next = curr; - next += page_ext_size; - return next; -} - struct page_ext_iter { unsigned long pfn; unsigned long index; diff --git a/mm/page_ext.c b/mm/page_ext.c index 508deb04d5ead..f9e515d353005 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -567,6 +567,13 @@ struct page_ext *page_ext_iter_begin(struct page_ext_iter *iter, struct page *pa return iter->page_ext; } +static struct page_ext *page_ext_next(struct page_ext *curr) +{ + void *next = curr; + next += page_ext_size; + return next; +} + /** * page_ext_iter_next() - Get next page extension * @iter: page extension iterator.