From patchwork Thu Apr 4 07:06:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oscar Salvador X-Patchwork-Id: 13617328 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E8D6CD1284 for ; Thu, 4 Apr 2024 07:05:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ACF2D6B0087; Thu, 4 Apr 2024 03:05:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A7F366B0089; Thu, 4 Apr 2024 03:05:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92B3D6B0087; Thu, 4 Apr 2024 03:05:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 71B616B0087 for ; Thu, 4 Apr 2024 03:05:53 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E790240F9F for ; Thu, 4 Apr 2024 07:05:52 +0000 (UTC) X-FDA: 81970964544.19.BAAD372 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf11.hostedemail.com (Postfix) with ESMTP id 0638640005 for ; Thu, 4 Apr 2024 07:05:50 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=b+JDHNpN; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=l19wa1e0; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf11.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.130 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712214351; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L9csULMalRTJ5bwNt2srfmcGplNPvj1BEqpu1yrZ8PM=; b=Q8mKySc2AX+gbgmGrdVoP7W1eQprp1Hsc9dOYm5eeu+b+l0TcDssycKl9DRaP/HTRZClP+ NRA1bs3rc3Xv4J/o/pffDm4lBRPxfWeHzveGPp9sRGQ8rhXC6pZxhuk7ZXSNBwzMyxWdLY 8jfI0bI4G2W2MKoXPNU1exZ8vzvmBgA= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=b+JDHNpN; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=l19wa1e0; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf11.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.130 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712214351; a=rsa-sha256; cv=none; b=Wf1POLCJTh12rwqDaCZ+cf8UdYXBwbEycwpL7j2R1U/gewUeONYJZbyHM2zceojp5xzUOk zjQyuvShXuDpIJMexU+BJMUJEysX4o1YxOggap8HmZ7kRQUsCk9vaAhp70L6umIThJ5gW/ plWzWC5M6PIMNiEi5kc+C6+JTnm9kxc= Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [10.150.64.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 7E193378EC; Thu, 4 Apr 2024 07:05:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1712214348; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L9csULMalRTJ5bwNt2srfmcGplNPvj1BEqpu1yrZ8PM=; b=b+JDHNpNj2CVttstyNqm814TITnISk4F7GULwqe+fIIP1mtyQ47GBToNvgBAC7H8/oGTfi Toh+8PzgcYrloVYdeg3thh3Tk5TlB1cInGEN2hML821UWErMTCw3kddxvluc+w0fX6Ra/B WPWbyqmxIhELcQ6nWa/BYZzbpHJUyeA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1712214348; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L9csULMalRTJ5bwNt2srfmcGplNPvj1BEqpu1yrZ8PM=; b=l19wa1e0HyXrXtipe5A2g71Pqed7khVCl1p+XMHjKo0UB/mwOIA+Q5UR0iRhXBjIjgaZXD VZRaK1iWwefbCQAQ== Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id E0607139E8; Thu, 4 Apr 2024 07:05:47 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap2.dmz-prg2.suse.org with ESMTPSA id SAvZMktRDmY6cgAAn2gu4w (envelope-from ); Thu, 04 Apr 2024 07:05:47 +0000 From: Oscar Salvador To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Michal Hocko , Vlastimil Babka , Marco Elver , Andrey Konovalov , Alexander Potapenko , Alexandre Ghiti , Oscar Salvador Subject: [PATCH v4 1/4] mm,page_owner: Update metadata for tail pages Date: Thu, 4 Apr 2024 09:06:59 +0200 Message-ID: <20240404070702.2744-2-osalvador@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240404070702.2744-1-osalvador@suse.de> References: <20240404070702.2744-1-osalvador@suse.de> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0638640005 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: gpf6cyexmafq64t4r7jzmbrrbzhgzzww X-HE-Tag: 1712214350-172461 X-HE-Meta: U2FsdGVkX1/z96bYW3o134C1xK4oLDT0ec5aMh+I40vvXHBXkO5JsIJrAJDWXOodjMZE+guHXDI1odxEphhRmElTtKD9cdtmrdO2BurG/y3lEygxMLrpmkNJC530XufmYE0QxW6eedJipAyo079OProbE5EfKGIKhFt6IFV8RifaX3anyoA0d5RXARY2gyr+JIja7CXtcdG4Szd9y5qBnt+ja2B8S/fed/yuOlhhrfK9STE6gSfw41fyJW4uoz1SafiEc+xaCQdhLu/VplDDOMWq/LBXE804rhrje7WF8BVTr6n/1pmSpmSdCvNfs+576dkCzyb/ETG/12jyALY/uQ+ZdVVGFf32hvjY1wErr4F9UD6icaJY5aM603pmEFSNFxwzSX9k1/vERLZTL6hqJB73gKIP2WnYaVJThvk99FtPVu0eSB5DeRywbs8wgRGNvpl723SlNFURZlxu1HfilqUDGymWpNp7T0AgZFkiy6ER53KD3UzcloWzkL3q718dcnRi4Q+f+d8LXU0m57NtXFeElJ70MNV69ksTpvJFjfG2kvHeQCjpge12lpQPbQUTkqCmXM8lF24CFSl67BC6RLg/WRxPz5MhYD7+X8J2OPSOesKF9/ACUYB0gt/AFUBMCcfFOeDOxti6hAVnVAIjs3D7LF/ubfg7CLR6dwU9ZPc00+Mvp+4BeRWg3yn545+01h7PqNJCAIK7RxbhnbQHkTWN+m3XIce7e7MxtE36icR8cQ9lIwiNJq81VSIX31vdNRxErGPjgmx+Ak8VS2vp7HvdWlqTiId+JBdae6F3MY+mZFxg5R47fW1FEYsDEquFSBxJd850NffIuEqcdeWbicrD1d28T4snBHqRFZOqhR89iwDUB2kjkFCM0jeOvWydg0gCERNUSqToZMHabuCb6k1eA1M6nXLIJJHkdd8fNDO2hLR1cRlPmS6XQkxswEGHdDSsrojF8hLEPN8DAKV GxumwFzG g4RFRS1nZwNKLl5WYVueZ7RVP3Q/58JXIZTEk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: __set_page_owner_handle() and __reset_page_owner() update the metadata of all pages when the page is of a higher-order, but we miss to do the same when the pages are migrated. __folio_copy_owner() only updates the metadata of the head page, meaning that the information stored in the first page and the tail pages will not match. Strictly speaking that is not a big problem because 1) we do not print tail pages and 2) upon splitting all tail pages will inherit the metadata of the head page, but it is better to have all metadata in check should there be any problem, so it can ease debugging. For that purpose, a couple of helpers are created __update_page_owner_handle() which updates the metadata on allocation, and __update_page_owner_free_handle() which does the same when the page is freed. __folio_copy_owner() will make use of both as it needs to entirely replace the page_owner metadata for the new page. Signed-off-by: Oscar Salvador Reviewed-by: Vlastimil Babka --- mm/page_owner.c | 137 ++++++++++++++++++++++++++---------------------- 1 file changed, 74 insertions(+), 63 deletions(-) diff --git a/mm/page_owner.c b/mm/page_owner.c index d17d1351ec84..52d1ced0b57f 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -228,9 +228,58 @@ static void dec_stack_record_count(depot_stack_handle_t handle) refcount_dec(&stack_record->count); } -void __reset_page_owner(struct page *page, unsigned short order) +static inline void __update_page_owner_handle(struct page_ext *page_ext, + depot_stack_handle_t handle, + unsigned short order, + gfp_t gfp_mask, + short last_migrate_reason, u64 ts_nsec, + pid_t pid, pid_t tgid, char *comm) { int i; + struct page_owner *page_owner; + + for (i = 0; i < (1 << order); i++) { + page_owner = get_page_owner(page_ext); + page_owner->handle = handle; + page_owner->order = order; + page_owner->gfp_mask = gfp_mask; + page_owner->last_migrate_reason = last_migrate_reason; + page_owner->pid = pid; + page_owner->tgid = tgid; + page_owner->ts_nsec = ts_nsec; + strscpy(page_owner->comm, comm, + sizeof(page_owner->comm)); + __set_bit(PAGE_EXT_OWNER, &page_ext->flags); + __set_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); + page_ext = page_ext_next(page_ext); + } +} + +static inline void __update_page_owner_free_handle(struct page_ext *page_ext, + depot_stack_handle_t handle, + unsigned short order, + pid_t pid, pid_t tgid, + u64 free_ts_nsec) +{ + int i; + struct page_owner *page_owner; + + for (i = 0; i < (1 << order); i++) { + page_owner = get_page_owner(page_ext); + /* Only __reset_page_owner() wants to clear the bit */ + if (handle) { + __clear_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); + page_owner->free_handle = handle; + } + page_owner->free_ts_nsec = free_ts_nsec; + page_owner->free_pid = current->pid; + page_owner->free_tgid = current->tgid; + page_ext = page_ext_next(page_ext); + } +} + +void __reset_page_owner(struct page *page, unsigned short order) +{ struct page_ext *page_ext; depot_stack_handle_t handle; depot_stack_handle_t alloc_handle; @@ -245,16 +294,10 @@ void __reset_page_owner(struct page *page, unsigned short order) alloc_handle = page_owner->handle; handle = save_stack(GFP_NOWAIT | __GFP_NOWARN); - for (i = 0; i < (1 << order); i++) { - __clear_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); - page_owner->free_handle = handle; - page_owner->free_ts_nsec = free_ts_nsec; - page_owner->free_pid = current->pid; - page_owner->free_tgid = current->tgid; - page_ext = page_ext_next(page_ext); - page_owner = get_page_owner(page_ext); - } + __update_page_owner_free_handle(page_ext, handle, order, current->pid, + current->tgid, free_ts_nsec); page_ext_put(page_ext); + if (alloc_handle != early_handle) /* * early_handle is being set as a handle for all those @@ -266,36 +309,11 @@ void __reset_page_owner(struct page *page, unsigned short order) dec_stack_record_count(alloc_handle); } -static inline void __set_page_owner_handle(struct page_ext *page_ext, - depot_stack_handle_t handle, - unsigned short order, gfp_t gfp_mask) -{ - struct page_owner *page_owner; - int i; - u64 ts_nsec = local_clock(); - - for (i = 0; i < (1 << order); i++) { - page_owner = get_page_owner(page_ext); - page_owner->handle = handle; - page_owner->order = order; - page_owner->gfp_mask = gfp_mask; - page_owner->last_migrate_reason = -1; - page_owner->pid = current->pid; - page_owner->tgid = current->tgid; - page_owner->ts_nsec = ts_nsec; - strscpy(page_owner->comm, current->comm, - sizeof(page_owner->comm)); - __set_bit(PAGE_EXT_OWNER, &page_ext->flags); - __set_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); - - page_ext = page_ext_next(page_ext); - } -} - noinline void __set_page_owner(struct page *page, unsigned short order, gfp_t gfp_mask) { struct page_ext *page_ext; + u64 ts_nsec = local_clock(); depot_stack_handle_t handle; handle = save_stack(gfp_mask); @@ -303,7 +321,9 @@ noinline void __set_page_owner(struct page *page, unsigned short order, page_ext = page_ext_get(page); if (unlikely(!page_ext)) return; - __set_page_owner_handle(page_ext, handle, order, gfp_mask); + __update_page_owner_handle(page_ext, handle, order, gfp_mask, -1, + current->pid, current->tgid, ts_nsec, + current->comm); page_ext_put(page_ext); inc_stack_record_count(handle, gfp_mask); } @@ -342,7 +362,7 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) { struct page_ext *old_ext; struct page_ext *new_ext; - struct page_owner *old_page_owner, *new_page_owner; + struct page_owner *old_page_owner; old_ext = page_ext_get(&old->page); if (unlikely(!old_ext)) @@ -355,31 +375,21 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) } old_page_owner = get_page_owner(old_ext); - new_page_owner = get_page_owner(new_ext); - new_page_owner->order = old_page_owner->order; - new_page_owner->gfp_mask = old_page_owner->gfp_mask; - new_page_owner->last_migrate_reason = - old_page_owner->last_migrate_reason; - new_page_owner->handle = old_page_owner->handle; - new_page_owner->pid = old_page_owner->pid; - new_page_owner->tgid = old_page_owner->tgid; - new_page_owner->free_pid = old_page_owner->free_pid; - new_page_owner->free_tgid = old_page_owner->free_tgid; - new_page_owner->ts_nsec = old_page_owner->ts_nsec; - new_page_owner->free_ts_nsec = old_page_owner->ts_nsec; - strcpy(new_page_owner->comm, old_page_owner->comm); - + __update_page_owner_handle(new_ext, old_page_owner->handle, + old_page_owner->order, old_page_owner->gfp_mask, + old_page_owner->last_migrate_reason, + old_page_owner->ts_nsec, old_page_owner->pid, + old_page_owner->tgid, old_page_owner->comm); /* - * We don't clear the bit on the old folio as it's going to be freed - * after migration. Until then, the info can be useful in case of - * a bug, and the overall stats will be off a bit only temporarily. - * Also, migrate_misplaced_transhuge_page() can still fail the - * migration and then we want the old folio to retain the info. But - * in that case we also don't need to explicitly clear the info from - * the new page, which will be freed. + * Do not proactively clear PAGE_EXT_OWNER{_ALLOCATED} bits as the folio + * will be freed after migration. Keep them until then as they may be + * useful. */ - __set_bit(PAGE_EXT_OWNER, &new_ext->flags); - __set_bit(PAGE_EXT_OWNER_ALLOCATED, &new_ext->flags); + __update_page_owner_free_handle(new_ext, 0, old_page_owner->order, + old_page_owner->free_pid, + old_page_owner->free_tgid, + old_page_owner->free_ts_nsec); + page_ext_put(new_ext); page_ext_put(old_ext); } @@ -787,8 +797,9 @@ static void init_pages_in_zone(pg_data_t *pgdat, struct zone *zone) goto ext_put_continue; /* Found early allocated page */ - __set_page_owner_handle(page_ext, early_handle, - 0, 0); + __update_page_owner_handle(page_ext, early_handle, 0, 0, + -1, local_clock(), current->pid, + current->tgid, current->comm); count++; ext_put_continue: page_ext_put(page_ext);