From patchwork Fri Sep 2 19:45:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964590 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C45CC6FA82 for ; Fri, 2 Sep 2022 19:48:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B00680121; Fri, 2 Sep 2022 15:47:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5604F80120; Fri, 2 Sep 2022 15:47:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38DAE80121; Fri, 2 Sep 2022 15:47:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 20BB380120 for ; Fri, 2 Sep 2022 15:47:27 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0658FC0410 for ; Fri, 2 Sep 2022 19:47:27 +0000 (UTC) X-FDA: 79868179734.27.7CF283C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 724C818004D for ; Fri, 2 Sep 2022 19:47:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=dhyA1+k3/oD84JmGsDqBjS76RHZcTcJfSFZtNWcKtjE=; b=T/WuGauJbv2n3MTO9GhTnKgWnm XMUAt+lAzgsgRpE4CmPiyUTeGXCSIvZY/c7xZkd2tANRbfNwxQWYvAkEye0OrKa6PgROPmUh5Iskf SrJpZZ/SbquDbrTgmiM3AbwVRSLaXHtKHyjoPjYKTUkT502fF5AHO6IeUojsMrjxyAkiRHH4RyCEJ I0E3Ci5fRaYGifeRjRdBER/sz0Y0apHAtEv16e3Yq0zhH3txS7lFIGVjp6HYYPJ68fixtTrXPN6cL OHwHQ+UWvumZ2eIUAlaG1hZe3R1xyXkofp4NR/+PDxtgyB3JdrhhOGpzCRwXnxamPmZFOYKwP6oLD s3NHkUxA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibd-0n; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 01/57] mm/vmscan: Fix a lot of comments Date: Fri, 2 Sep 2022 20:45:57 +0100 Message-Id: <20220902194653.1739778-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148046; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dhyA1+k3/oD84JmGsDqBjS76RHZcTcJfSFZtNWcKtjE=; b=cpq7WsRWsPBO+hqWSO/ziiDfkorU6sUoO6ipUwmeBG1SB4dcDNVGYKcCpaBur5V5/xJrOF lgSgJzFMH1p0+EUmUNFCeK3kxjQ6Z0AIKuJL6LoDSybjHxCbp7dGTgt7nVfReVGsdGtBWS veciVhJOOoqNAu1UVFjtXf6vNf2u58Q= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="T/WuGauJ"; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148046; a=rsa-sha256; cv=none; b=5Zj2RezIFgCWvBPFqHKdZG81qINVbT813ooEEieTKaAQ0IkhSYVBznKfTPY5sXRIB0oxs6 H0RIEsPq2RSZd9Q/CRaMEZmWaQNrmBWvOAyNSMAJ+ImdnwVGA65UGldRVKadUDhNdXhlXj 0gUlbaY/zNShKmYZJgDBKe/oTUW07tQ= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 724C818004D Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="T/WuGauJ"; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: xb5o7j5gib3k9n9ywifq8c5juomcc4yx X-HE-Tag: 1662148046-617261 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A lot of comments mention pages when they should say folios. Fix them up. Signed-off-by: Matthew Wilcox (Oracle) --- mm/vmscan.c | 252 ++++++++++++++++++++++++++-------------------------- 1 file changed, 125 insertions(+), 127 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 382dbe97329f..93b1087fc969 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -85,7 +85,7 @@ struct scan_control { unsigned long anon_cost; unsigned long file_cost; - /* Can active pages be deactivated as part of reclaim? */ + /* Can active folios be deactivated as part of reclaim? */ #define DEACTIVATE_ANON 1 #define DEACTIVATE_FILE 2 unsigned int may_deactivate:2; @@ -95,10 +95,10 @@ struct scan_control { /* Writepage batching in laptop mode; RECLAIM_WRITE */ unsigned int may_writepage:1; - /* Can mapped pages be reclaimed? */ + /* Can mapped folios be reclaimed? */ unsigned int may_unmap:1; - /* Can pages be swapped as part of reclaim? */ + /* Can folios be swapped as part of reclaim? */ unsigned int may_swap:1; /* Proactive reclaim invoked by userspace through memory.reclaim */ @@ -123,7 +123,7 @@ struct scan_control { /* There is easily reclaimable cold cache in the current node */ unsigned int cache_trim_mode:1; - /* The file pages on the current node are dangerously low */ + /* The file folios on the current node are dangerously low */ unsigned int file_is_tiny:1; /* Always discard instead of demoting to lower tier memory */ @@ -135,7 +135,7 @@ struct scan_control { /* Scan (total_size >> priority) pages at once */ s8 priority; - /* The highest zone to isolate pages for reclaim from */ + /* The highest zone to isolate folios for reclaim from */ s8 reclaim_idx; /* This context's GFP mask */ @@ -443,7 +443,7 @@ static bool cgroup_reclaim(struct scan_control *sc) * * The normal page dirty throttling mechanism in balance_dirty_pages() is * completely broken with the legacy memcg and direct stalling in - * shrink_page_list() is used for throttling instead, which lacks all the + * shrink_folio_list() is used for throttling instead, which lacks all the * niceties such as fairness, adaptive pausing, bandwidth proportional * allocation and configurability. * @@ -564,9 +564,9 @@ static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg, } /* - * This misses isolated pages which are not accounted for to save counters. + * This misses isolated folios which are not accounted for to save counters. * As the data only determines if reclaim or compaction continues, it is - * not expected that isolated pages will be a dominating factor. + * not expected that isolated folios will be a dominating factor. */ unsigned long zone_reclaimable_pages(struct zone *zone) { @@ -1039,9 +1039,9 @@ void drop_slab(void) static inline int is_page_cache_freeable(struct folio *folio) { /* - * A freeable page cache page is referenced only by the caller - * that isolated the page, the page cache and optional buffer - * heads at page->private. + * A freeable page cache folio is referenced only by the caller + * that isolated the folio, the page cache and optional filesystem + * private data at folio->private. */ return folio_ref_count(folio) - folio_test_private(folio) == 1 + folio_nr_pages(folio); @@ -1081,8 +1081,8 @@ static bool skip_throttle_noprogress(pg_data_t *pgdat) return true; /* - * If there are a lot of dirty/writeback pages then do not - * throttle as throttling will occur when the pages cycle + * If there are a lot of dirty/writeback folios then do not + * throttle as throttling will occur when the folios cycle * towards the end of the LRU if still under writeback. */ for (i = 0; i < MAX_NR_ZONES; i++) { @@ -1125,7 +1125,7 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason) * short. Failing to make progress or waiting on writeback are * potentially long-lived events so use a longer timeout. This is shaky * logic as a failure to make progress could be due to anything from - * writeback to a slow device to excessive references pages at the tail + * writeback to a slow device to excessive referenced folios at the tail * of the inactive LRU. */ switch(reason) { @@ -1171,8 +1171,8 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason) } /* - * Account for pages written if tasks are throttled waiting on dirty - * pages to clean. If enough pages have been cleaned since throttling + * Account for folios written if tasks are throttled waiting on dirty + * folios to clean. If enough folios have been cleaned since throttling * started then wakeup the throttled tasks. */ void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, @@ -1198,18 +1198,18 @@ void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, /* possible outcome of pageout() */ typedef enum { - /* failed to write page out, page is locked */ + /* failed to write folio out, folio is locked */ PAGE_KEEP, - /* move page to the active list, page is locked */ + /* move folio to the active list, folio is locked */ PAGE_ACTIVATE, - /* page has been sent to the disk successfully, page is unlocked */ + /* folio has been sent to the disk successfully, folio is unlocked */ PAGE_SUCCESS, - /* page is clean and locked */ + /* folio is clean and locked */ PAGE_CLEAN, } pageout_t; /* - * pageout is called by shrink_page_list() for each dirty page. + * pageout is called by shrink_folio_list() for each dirty folio. * Calls ->writepage(). */ static pageout_t pageout(struct folio *folio, struct address_space *mapping, @@ -1283,7 +1283,7 @@ static pageout_t pageout(struct folio *folio, struct address_space *mapping, } /* - * Same as remove_mapping, but if the page is removed from the mapping, it + * Same as remove_mapping, but if the folio is removed from the mapping, it * gets returned with a refcount of 0. */ static int __remove_mapping(struct address_space *mapping, struct folio *folio, @@ -1299,34 +1299,34 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); /* - * The non racy check for a busy page. + * The non racy check for a busy folio. * * Must be careful with the order of the tests. When someone has - * a ref to the page, it may be possible that they dirty it then - * drop the reference. So if PageDirty is tested before page_count - * here, then the following race may occur: + * a ref to the folio, it may be possible that they dirty it then + * drop the reference. So if the dirty flag is tested before the + * refcount here, then the following race may occur: * * get_user_pages(&page); * [user mapping goes away] * write_to(page); - * !PageDirty(page) [good] - * SetPageDirty(page); - * put_page(page); - * !page_count(page) [good, discard it] + * !folio_test_dirty(folio) [good] + * folio_set_dirty(folio); + * folio_put(folio); + * !refcount(folio) [good, discard it] * * [oops, our write_to data is lost] * * Reversing the order of the tests ensures such a situation cannot - * escape unnoticed. The smp_rmb is needed to ensure the page->flags - * load is not satisfied before that of page->_refcount. + * escape unnoticed. The smp_rmb is needed to ensure the folio->flags + * load is not satisfied before that of folio->_refcount. * - * Note that if SetPageDirty is always performed via set_page_dirty, + * Note that if the dirty flag is always set via folio_mark_dirty, * and thus under the i_pages lock, then this ordering is not required. */ refcount = 1 + folio_nr_pages(folio); if (!folio_ref_freeze(folio, refcount)) goto cannot_free; - /* note: atomic_cmpxchg in page_ref_freeze provides the smp_rmb */ + /* note: atomic_cmpxchg in folio_ref_freeze provides the smp_rmb */ if (unlikely(folio_test_dirty(folio))) { folio_ref_unfreeze(folio, refcount); goto cannot_free; @@ -1355,7 +1355,7 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, * back. * * We also don't store shadows for DAX mappings because the - * only page cache pages found in these are zero pages + * only page cache folios found in these are zero pages * covering holes, and because we don't want to mix DAX * exceptional entries and shadow exceptional entries in the * same address_space. @@ -1423,14 +1423,14 @@ void folio_putback_lru(struct folio *folio) folio_put(folio); /* drop ref from isolate */ } -enum page_references { - PAGEREF_RECLAIM, - PAGEREF_RECLAIM_CLEAN, - PAGEREF_KEEP, - PAGEREF_ACTIVATE, +enum folio_references { + FOLIOREF_RECLAIM, + FOLIOREF_RECLAIM_CLEAN, + FOLIOREF_KEEP, + FOLIOREF_ACTIVATE, }; -static enum page_references folio_check_references(struct folio *folio, +static enum folio_references folio_check_references(struct folio *folio, struct scan_control *sc) { int referenced_ptes, referenced_folio; @@ -1445,11 +1445,11 @@ static enum page_references folio_check_references(struct folio *folio, * Let the folio, now marked Mlocked, be moved to the unevictable list. */ if (vm_flags & VM_LOCKED) - return PAGEREF_ACTIVATE; + return FOLIOREF_ACTIVATE; /* rmap lock contention: rotate */ if (referenced_ptes == -1) - return PAGEREF_KEEP; + return FOLIOREF_KEEP; if (referenced_ptes) { /* @@ -1469,34 +1469,34 @@ static enum page_references folio_check_references(struct folio *folio, folio_set_referenced(folio); if (referenced_folio || referenced_ptes > 1) - return PAGEREF_ACTIVATE; + return FOLIOREF_ACTIVATE; /* * Activate file-backed executable folios after first usage. */ if ((vm_flags & VM_EXEC) && folio_is_file_lru(folio)) - return PAGEREF_ACTIVATE; + return FOLIOREF_ACTIVATE; - return PAGEREF_KEEP; + return FOLIOREF_KEEP; } /* Reclaim if clean, defer dirty folios to writeback */ if (referenced_folio && folio_is_file_lru(folio)) - return PAGEREF_RECLAIM_CLEAN; + return FOLIOREF_RECLAIM_CLEAN; - return PAGEREF_RECLAIM; + return FOLIOREF_RECLAIM; } -/* Check if a page is dirty or under writeback */ +/* Check if a folio is dirty or under writeback */ static void folio_check_dirty_writeback(struct folio *folio, bool *dirty, bool *writeback) { struct address_space *mapping; /* - * Anonymous pages are not handled by flushers and must be written + * Anonymous folios are not handled by flushers and must be written * from reclaim context. Do not stall reclaim based on them. - * MADV_FREE anonymous pages are put into inactive file list too. + * MADV_FREE anonymous folios are put into inactive file list too. * They could be mistakenly treated as file lru. So further anon * test is needed. */ @@ -1538,24 +1538,24 @@ static struct page *alloc_demote_page(struct page *page, unsigned long node) } /* - * Take pages on @demote_list and attempt to demote them to - * another node. Pages which are not demoted are left on - * @demote_pages. + * Take folios on @demote_folios and attempt to demote them to + * another node. Folios which are not demoted are left on + * @demote_folios. */ -static unsigned int demote_page_list(struct list_head *demote_pages, +static unsigned int demote_folio_list(struct list_head *demote_folios, struct pglist_data *pgdat) { int target_nid = next_demotion_node(pgdat->node_id); unsigned int nr_succeeded; - if (list_empty(demote_pages)) + if (list_empty(demote_folios)) return 0; if (target_nid == NUMA_NO_NODE) return 0; /* Demotion ignores all cpuset and mempolicy settings */ - migrate_pages(demote_pages, alloc_demote_page, NULL, + migrate_pages(demote_folios, alloc_demote_page, NULL, target_nid, MIGRATE_ASYNC, MR_DEMOTION, &nr_succeeded); @@ -1584,17 +1584,15 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask) } /* - * shrink_page_list() returns the number of reclaimed pages + * shrink_folio_list() returns the number of reclaimed pages */ -static unsigned int shrink_page_list(struct list_head *page_list, - struct pglist_data *pgdat, - struct scan_control *sc, - struct reclaim_stat *stat, - bool ignore_references) +static unsigned int shrink_folio_list(struct list_head *folio_list, + struct pglist_data *pgdat, struct scan_control *sc, + struct reclaim_stat *stat, bool ignore_references) { - LIST_HEAD(ret_pages); - LIST_HEAD(free_pages); - LIST_HEAD(demote_pages); + LIST_HEAD(ret_folios); + LIST_HEAD(free_folios); + LIST_HEAD(demote_folios); unsigned int nr_reclaimed = 0; unsigned int pgactivate = 0; bool do_demote_pass; @@ -1605,16 +1603,16 @@ static unsigned int shrink_page_list(struct list_head *page_list, do_demote_pass = can_demote(pgdat->node_id, sc); retry: - while (!list_empty(page_list)) { + while (!list_empty(folio_list)) { struct address_space *mapping; struct folio *folio; - enum page_references references = PAGEREF_RECLAIM; + enum folio_references references = FOLIOREF_RECLAIM; bool dirty, writeback; unsigned int nr_pages; cond_resched(); - folio = lru_to_folio(page_list); + folio = lru_to_folio(folio_list); list_del(&folio->lru); if (!folio_trylock(folio)) @@ -1733,7 +1731,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, folio_unlock(folio); folio_wait_writeback(folio); /* then go back and try same folio again */ - list_add_tail(&folio->lru, page_list); + list_add_tail(&folio->lru, folio_list); continue; } } @@ -1742,13 +1740,13 @@ static unsigned int shrink_page_list(struct list_head *page_list, references = folio_check_references(folio, sc); switch (references) { - case PAGEREF_ACTIVATE: + case FOLIOREF_ACTIVATE: goto activate_locked; - case PAGEREF_KEEP: + case FOLIOREF_KEEP: stat->nr_ref_keep += nr_pages; goto keep_locked; - case PAGEREF_RECLAIM: - case PAGEREF_RECLAIM_CLEAN: + case FOLIOREF_RECLAIM: + case FOLIOREF_RECLAIM_CLEAN: ; /* try to reclaim the folio below */ } @@ -1758,7 +1756,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, */ if (do_demote_pass && (thp_migration_supported() || !folio_test_large(folio))) { - list_add(&folio->lru, &demote_pages); + list_add(&folio->lru, &demote_folios); folio_unlock(folio); continue; } @@ -1785,7 +1783,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, */ if (!folio_entire_mapcount(folio) && split_folio_to_list(folio, - page_list)) + folio_list)) goto activate_locked; } if (!add_to_swap(folio)) { @@ -1793,7 +1791,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, goto activate_locked_split; /* Fallback to swap normal pages */ if (split_folio_to_list(folio, - page_list)) + folio_list)) goto activate_locked; #ifdef CONFIG_TRANSPARENT_HUGEPAGE count_vm_event(THP_SWPOUT_FALLBACK); @@ -1805,7 +1803,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, } else if (folio_test_swapbacked(folio) && folio_test_large(folio)) { /* Split shmem folio */ - if (split_folio_to_list(folio, page_list)) + if (split_folio_to_list(folio, folio_list)) goto keep_locked; } @@ -1870,7 +1868,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, goto activate_locked; } - if (references == PAGEREF_RECLAIM_CLEAN) + if (references == FOLIOREF_RECLAIM_CLEAN) goto keep_locked; if (!may_enter_fs(folio, sc->gfp_mask)) goto keep_locked; @@ -1983,13 +1981,13 @@ static unsigned int shrink_page_list(struct list_head *page_list, nr_reclaimed += nr_pages; /* - * Is there need to periodically free_page_list? It would + * Is there need to periodically free_folio_list? It would * appear not as the counts should be low */ if (unlikely(folio_test_large(folio))) destroy_large_folio(folio); else - list_add(&folio->lru, &free_pages); + list_add(&folio->lru, &free_folios); continue; activate_locked_split: @@ -2017,29 +2015,29 @@ static unsigned int shrink_page_list(struct list_head *page_list, keep_locked: folio_unlock(folio); keep: - list_add(&folio->lru, &ret_pages); + list_add(&folio->lru, &ret_folios); VM_BUG_ON_FOLIO(folio_test_lru(folio) || folio_test_unevictable(folio), folio); } - /* 'page_list' is always empty here */ + /* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_reclaimed += demote_page_list(&demote_pages, pgdat); - /* Folios that could not be demoted are still in @demote_pages */ - if (!list_empty(&demote_pages)) { - /* Folios which weren't demoted go back on @page_list for retry: */ - list_splice_init(&demote_pages, page_list); + nr_reclaimed += demote_folio_list(&demote_folios, pgdat); + /* Folios that could not be demoted are still in @demote_folios */ + if (!list_empty(&demote_folios)) { + /* Folios which weren't demoted go back on @folio_list for retry: */ + list_splice_init(&demote_folios, folio_list); do_demote_pass = false; goto retry; } pgactivate = stat->nr_activate[0] + stat->nr_activate[1]; - mem_cgroup_uncharge_list(&free_pages); + mem_cgroup_uncharge_list(&free_folios); try_to_unmap_flush(); - free_unref_page_list(&free_pages); + free_unref_page_list(&free_folios); - list_splice(&ret_pages, page_list); + list_splice(&ret_folios, folio_list); count_vm_events(PGACTIVATE, pgactivate); if (plug) @@ -2048,7 +2046,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, } unsigned int reclaim_clean_pages_from_list(struct zone *zone, - struct list_head *folio_list) + struct list_head *folio_list) { struct scan_control sc = { .gfp_mask = GFP_KERNEL, @@ -2076,7 +2074,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, * change in the future. */ noreclaim_flag = memalloc_noreclaim_save(); - nr_reclaimed = shrink_page_list(&clean_folios, zone->zone_pgdat, &sc, + nr_reclaimed = shrink_folio_list(&clean_folios, zone->zone_pgdat, &sc, &stat, true); memalloc_noreclaim_restore(noreclaim_flag); @@ -2135,7 +2133,7 @@ static __always_inline void update_lru_sizes(struct lruvec *lruvec, * * returns how many pages were moved onto *@dst. */ -static unsigned long isolate_lru_pages(unsigned long nr_to_scan, +static unsigned long isolate_lru_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct list_head *dst, unsigned long *nr_scanned, struct scan_control *sc, enum lru_list lru) @@ -2242,8 +2240,8 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan, * * Context: * - * (1) Must be called with an elevated refcount on the page. This is a - * fundamental difference from isolate_lru_pages() (which is called + * (1) Must be called with an elevated refcount on the folio. This is a + * fundamental difference from isolate_lru_folios() (which is called * without a stable reference). * (2) The lru_lock must not be held. * (3) Interrupts must be enabled. @@ -2315,13 +2313,13 @@ static int too_many_isolated(struct pglist_data *pgdat, int file, } /* - * move_pages_to_lru() moves folios from private @list to appropriate LRU list. + * move_folios_to_lru() moves folios from private @list to appropriate LRU list. * On return, @list is reused as a list of folios to be freed by the caller. * * Returns the number of pages moved to the given lruvec. */ -static unsigned int move_pages_to_lru(struct lruvec *lruvec, - struct list_head *list) +static unsigned int move_folios_to_lru(struct lruvec *lruvec, + struct list_head *list) { int nr_pages, nr_moved = 0; LIST_HEAD(folios_to_free); @@ -2341,7 +2339,7 @@ static unsigned int move_pages_to_lru(struct lruvec *lruvec, /* * The folio_set_lru needs to be kept here for list integrity. * Otherwise: - * #0 move_pages_to_lru #1 release_pages + * #0 move_folios_to_lru #1 release_pages * if (!folio_put_testzero()) * if (folio_put_testzero()) * !lru //skip lru_lock @@ -2398,11 +2396,11 @@ static int current_may_throttle(void) * shrink_inactive_list() is a helper for shrink_node(). It returns the number * of reclaimed pages */ -static unsigned long -shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, - struct scan_control *sc, enum lru_list lru) +static unsigned long shrink_inactive_list(unsigned long nr_to_scan, + struct lruvec *lruvec, struct scan_control *sc, + enum lru_list lru) { - LIST_HEAD(page_list); + LIST_HEAD(folio_list); unsigned long nr_scanned; unsigned int nr_reclaimed = 0; unsigned long nr_taken; @@ -2429,7 +2427,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, spin_lock_irq(&lruvec->lru_lock); - nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &page_list, + nr_taken = isolate_lru_folios(nr_to_scan, lruvec, &folio_list, &nr_scanned, sc, lru); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, nr_taken); @@ -2444,10 +2442,10 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, if (nr_taken == 0) return 0; - nr_reclaimed = shrink_page_list(&page_list, pgdat, sc, &stat, false); + nr_reclaimed = shrink_folio_list(&folio_list, pgdat, sc, &stat, false); spin_lock_irq(&lruvec->lru_lock); - move_pages_to_lru(lruvec, &page_list); + move_folios_to_lru(lruvec, &folio_list); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); item = current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; @@ -2458,16 +2456,16 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, spin_unlock_irq(&lruvec->lru_lock); lru_note_cost(lruvec, file, stat.nr_pageout); - mem_cgroup_uncharge_list(&page_list); - free_unref_page_list(&page_list); + mem_cgroup_uncharge_list(&folio_list); + free_unref_page_list(&folio_list); /* - * If dirty pages are scanned that are not queued for IO, it + * If dirty folios are scanned that are not queued for IO, it * implies that flushers are not doing their job. This can - * happen when memory pressure pushes dirty pages to the end of + * happen when memory pressure pushes dirty folios to the end of * the LRU before the dirty limits are breached and the dirty * data has expired. It can also happen when the proportion of - * dirty pages grows not through writes but through memory + * dirty folios grows not through writes but through memory * pressure reclaiming all the clean cache. And in some cases, * the flushers simply cannot keep up with the allocation * rate. Nudge the flusher threads in case they are asleep. @@ -2526,7 +2524,7 @@ static void shrink_active_list(unsigned long nr_to_scan, spin_lock_irq(&lruvec->lru_lock); - nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &l_hold, + nr_taken = isolate_lru_folios(nr_to_scan, lruvec, &l_hold, &nr_scanned, sc, lru); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, nr_taken); @@ -2586,8 +2584,8 @@ static void shrink_active_list(unsigned long nr_to_scan, */ spin_lock_irq(&lruvec->lru_lock); - nr_activate = move_pages_to_lru(lruvec, &l_active); - nr_deactivate = move_pages_to_lru(lruvec, &l_inactive); + nr_activate = move_folios_to_lru(lruvec, &l_active); + nr_deactivate = move_folios_to_lru(lruvec, &l_inactive); /* Keep all free folios in l_active list */ list_splice(&l_inactive, &l_active); @@ -2603,7 +2601,7 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_deactivate, nr_rotated, sc->priority, file); } -static unsigned int reclaim_page_list(struct list_head *page_list, +static unsigned int reclaim_folio_list(struct list_head *folio_list, struct pglist_data *pgdat) { struct reclaim_stat dummy_stat; @@ -2617,9 +2615,9 @@ static unsigned int reclaim_page_list(struct list_head *page_list, .no_demotion = 1, }; - nr_reclaimed = shrink_page_list(page_list, pgdat, &sc, &dummy_stat, false); - while (!list_empty(page_list)) { - folio = lru_to_folio(page_list); + nr_reclaimed = shrink_folio_list(folio_list, pgdat, &sc, &dummy_stat, false); + while (!list_empty(folio_list)) { + folio = lru_to_folio(folio_list); list_del(&folio->lru); folio_putback_lru(folio); } @@ -2649,11 +2647,11 @@ unsigned long reclaim_pages(struct list_head *folio_list) continue; } - nr_reclaimed += reclaim_page_list(&node_folio_list, NODE_DATA(nid)); + nr_reclaimed += reclaim_folio_list(&node_folio_list, NODE_DATA(nid)); nid = folio_nid(lru_to_folio(folio_list)); } while (!list_empty(folio_list)); - nr_reclaimed += reclaim_page_list(&node_folio_list, NODE_DATA(nid)); + nr_reclaimed += reclaim_folio_list(&node_folio_list, NODE_DATA(nid)); memalloc_noreclaim_restore(noreclaim_flag); @@ -2683,13 +2681,13 @@ static unsigned long shrink_list(enum lru_list lru, unsigned long nr_to_scan, * but large enough to avoid thrashing the aggregate readahead window. * * Both inactive lists should also be large enough that each inactive - * page has a chance to be referenced again before it is reclaimed. + * folio has a chance to be referenced again before it is reclaimed. * * If that fails and refaulting is observed, the inactive list grows. * - * The inactive_ratio is the target ratio of ACTIVE to INACTIVE pages + * The inactive_ratio is the target ratio of ACTIVE to INACTIVE folios * on this LRU, maintained by the pageout code. An inactive_ratio - * of 3 means 3:1 or 25% of the pages are kept on the inactive list. + * of 3 means 3:1 or 25% of the folios are kept on the inactive list. * * total target max * memory ratio inactive @@ -2732,8 +2730,8 @@ enum scan_balance { * Determine how aggressively the anon and file LRU lists should be * scanned. * - * nr[0] = anon inactive pages to scan; nr[1] = anon active pages to scan - * nr[2] = file inactive pages to scan; nr[3] = file active pages to scan + * nr[0] = anon inactive folios to scan; nr[1] = anon active folios to scan + * nr[2] = file inactive folios to scan; nr[3] = file active folios to scan */ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, unsigned long *nr) @@ -2748,7 +2746,7 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, unsigned long ap, fp; enum lru_list lru; - /* If we have no swap space, do not bother scanning anon pages. */ + /* If we have no swap space, do not bother scanning anon folios. */ if (!sc->may_swap || !can_reclaim_anon_pages(memcg, pgdat->node_id, sc)) { scan_balance = SCAN_FILE; goto out; From patchwork Fri Sep 2 19:45:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964582 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1653DC6FA83 for ; Fri, 2 Sep 2022 19:48:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 64B8A80118; Fri, 2 Sep 2022 15:47:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FAFB80119; Fri, 2 Sep 2022 15:47:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49D5280118; Fri, 2 Sep 2022 15:47:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2EBC780119 for ; Fri, 2 Sep 2022 15:47:09 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 191E3403C9 for ; Fri, 2 Sep 2022 19:47:09 +0000 (UTC) X-FDA: 79868178978.14.3CEF632 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id B06A618004D for ; Fri, 2 Sep 2022 19:47:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=D9OxPNakbQ7dXKQCeoqgLhnSNcTIRcyWrbUHuZ9BacA=; b=C9hzQIQYxRxO9E3u6NbkVaFzJl bPl4rX4I/YjP7q6oYtgXlF+spNpfnnNKp+aJP3hmVRnVobJT8wc4sFNnFfm09Mai0rsEsAebYchal E/qCfD3DEg40xlAfTTv/PvACmVsIE0FaNmpQhddJpXS0uFk0WQl5nTo8ZleU2kgBTYX8BFNZaPpgg fVdGrr/KTEcjct23RnMaLdVpiplLFnRQoggasky8S3KCzLxTWlZU5+K2d2LitP9FolRUWIKvaAR43 eDtCAu2lf94khoSndXK235u/ib72mV2oiGithjoWsCfIC95RqxzbeQq4KZ7RvW8hPg1MqzxzLTP5V /odqcHWg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibf-3P; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 02/57] mm: Add the first tail page to struct folio Date: Fri, 2 Sep 2022 20:45:58 +0100 Message-Id: <20220902194653.1739778-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148028; a=rsa-sha256; cv=none; b=hWJ+Th262ttw9h9LDbFVzdmSYgo76HwAKq/pFXmazV3xSyP17ILC1JfA5rZjNZBP0x+MuU NLY6r+tl5d2pF9uJkkvZibPRa/CCS9dGMRDFVAc4/VR/6MSOoF0UnPzQWCWbIf7WnjhUtv ZnfX0Ln3VyFBdrH9N8b31QSLSWDxg+8= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=C9hzQIQY; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148028; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D9OxPNakbQ7dXKQCeoqgLhnSNcTIRcyWrbUHuZ9BacA=; b=qdisPHXYF9xuNnDC8t4uJjovioB6pX7xMgwzCF64EKNFOmaT53hW9F+VFHJ3OEi1UEcFnf ysWPAPsNTlrgwKrMAnfTs+HCLb2AgIgBUe5ciu4pLpowvhH/6/7QcpQSxnan0c4L1jmMNv JoBRhy1WbsgafeOQyzYj9jUTDlf0Pjk= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: B06A618004D Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=C9hzQIQY; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 4tho95un6nibbjxetup8n7taxxy7tnw1 X-HE-Tag: 1662148028-692895 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some of the static checkers get confused by extracting the page from the folio and referring to fields in the first tail page. Adding these fields to struct folio lets us avoid doing that. It has the risk that people will refer to those fields without checking that the folio is actually a large folio, so prefix them with underscores and document the preferred function to use instead. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm_types.h | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index cf97f3884fda..8a9ee9d24973 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -244,6 +244,13 @@ struct page { * @_refcount: Do not access this member directly. Use folio_ref_count() * to find how many references there are to this folio. * @memcg_data: Memory Control Group data. + * @_flags_1: For large folios, additional page flags. + * @__head: Points to the folio. Do not use. + * @_folio_dtor: Which destructor to use for this folio. + * @_folio_order: Do not use directly, call folio_order(). + * @_total_mapcount: Do not use directly, call folio_entire_mapcount(). + * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). + * @_folio_nr_pages: Do not use directly, call folio_nr_pages(). * * A folio is a physically, virtually and logically contiguous set * of bytes. It is a power-of-two in size, and it is aligned to that @@ -282,9 +289,17 @@ struct folio { }; struct page page; }; + unsigned long _flags_1; + unsigned long __head; + unsigned char _folio_dtor; + unsigned char _folio_order; + atomic_t _total_mapcount; + atomic_t _pincount; +#ifdef CONFIG_64BIT + unsigned int _folio_nr_pages; +#endif }; -static_assert(sizeof(struct page) == sizeof(struct folio)); #define FOLIO_MATCH(pg, fl) \ static_assert(offsetof(struct page, pg) == offsetof(struct folio, fl)) FOLIO_MATCH(flags, flags); @@ -299,6 +314,19 @@ FOLIO_MATCH(_refcount, _refcount); FOLIO_MATCH(memcg_data, memcg_data); #endif #undef FOLIO_MATCH +#define FOLIO_MATCH(pg, fl) \ + static_assert(offsetof(struct folio, fl) == \ + offsetof(struct page, pg) + sizeof(struct page)) +FOLIO_MATCH(flags, _flags_1); +FOLIO_MATCH(compound_head, __head); +FOLIO_MATCH(compound_dtor, _folio_dtor); +FOLIO_MATCH(compound_order, _folio_order); +FOLIO_MATCH(compound_mapcount, _total_mapcount); +FOLIO_MATCH(compound_pincount, _pincount); +#ifdef CONFIG_64BIT +FOLIO_MATCH(compound_nr, _folio_nr_pages); +#endif +#undef FOLIO_MATCH static inline atomic_t *folio_mapcount_ptr(struct folio *folio) { From patchwork Fri Sep 2 19:45:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3753BC6FA82 for ; Fri, 2 Sep 2022 19:48:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C8FBF8011A; Fri, 2 Sep 2022 15:47:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BC88C80119; Fri, 2 Sep 2022 15:47:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A5BC8011A; Fri, 2 Sep 2022 15:47:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8574580119 for ; Fri, 2 Sep 2022 15:47:11 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 64598120F14 for ; Fri, 2 Sep 2022 19:47:11 +0000 (UTC) X-FDA: 79868179062.05.E4C7743 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id 0DBA040094 for ; Fri, 2 Sep 2022 19:47:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3kKnSNC3eA+xvk17r4oT0lcPJYYUylIipEepqwkc/TE=; b=DO11Gy0Y1vxX5/1GF/KG8vmfRm bk5VhWZ5CRFhMMiJpnu1XjuadN//zREOuXipwQWtkqT1Eb1Hr2QHGKQUjof0vDusFBSiLQdVGmmu0 GmZFdNVi63rj0ivn/xr2N3j7U7uU6vBZFHI6ek6BtCexXlsHV9lzKX2AerI7bKmTdkCpVl/70SvJN SAEBTxLeC5V1yS1ScPpIE8L1/8xwkIMKLdKoXQZc90GcuX8NxOsH9ca05Fr/+VQ6/jY3oxLcf9vSK bK2dsTm4m9rs3sWKQOQNWnr3I4XqixuosyhrjmxxIWagilPPQT/gUfSwGbMWnGcknW9OfkzLxWpmT vu6AdtdQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibh-5a; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 03/57] mm: Reimplement folio_order() and folio_nr_pages() Date: Fri, 2 Sep 2022 20:45:59 +0100 Message-Id: <20220902194653.1739778-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148031; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3kKnSNC3eA+xvk17r4oT0lcPJYYUylIipEepqwkc/TE=; b=Bdoo7FmTiINopbt+6t4+0tpLVZaUroGJqviCD1KJeBsNfabSZogZ7odJ/gRbSng/TV9pej LQgUXXy8SUWA9wdjHn5U36eOIAh7Y2bD1QaE/L2gN8QGqi2y6MuOw8V5tDEj+7E9ocSBsw Gp+HCHNzWzyj6aX1Cevo7EPA4BWP22w= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DO11Gy0Y; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148031; a=rsa-sha256; cv=none; b=PiU/XmjIVOBx0n11G/UdweEnA+qXYiKQyIzvDEMjyai2JsXFcvslpHPO+NYhD+svdvjkBp FW8DENZCEiGO6FyaOpOa/o/VuSv89RuR8oNxbjdHadrH7Yn84zfcS9afnjdGvIfJ6xZt/c mrRvYYPE9m/RGEpNmnlSZJMCSefxVsM= X-Stat-Signature: qyw3nt5hhgfo7ryjk5pj3zee3mod6p1o X-Rspamd-Queue-Id: 0DBA040094 X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DO11Gy0Y; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam01 X-HE-Tag: 1662148030-108952 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of calling compound_order() and compound_nr_pages(), use the folio directly. Saves 1905 bytes from mm/filemap.o due to folio_test_large() now being a cheaper check than PageHead(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 21f8b27bd9fd..dcab29e50cca 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -697,7 +697,9 @@ static inline unsigned int compound_order(struct page *page) */ static inline unsigned int folio_order(struct folio *folio) { - return compound_order(&folio->page); + if (!folio_test_large(folio)) + return 0; + return folio->_folio_order; } #include @@ -1597,7 +1599,13 @@ static inline void set_page_links(struct page *page, enum zone_type zone, */ static inline long folio_nr_pages(struct folio *folio) { - return compound_nr(&folio->page); + if (!folio_test_large(folio)) + return 1; +#ifdef CONFIG_64BIT + return folio->_folio_nr_pages; +#else + return 1L << folio->_folio_order; +#endif } /** From patchwork Fri Sep 2 19:46:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964588 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70D9CC6FA82 for ; Fri, 2 Sep 2022 19:48:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0A4C8011F; Fri, 2 Sep 2022 15:47:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ADEBD80119; Fri, 2 Sep 2022 15:47:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D2EC8011F; Fri, 2 Sep 2022 15:47:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 63A6E80119 for ; Fri, 2 Sep 2022 15:47:23 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3FD8A1A04CC for ; Fri, 2 Sep 2022 19:47:23 +0000 (UTC) X-FDA: 79868179566.09.64D84CF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 0BA314004E for ; Fri, 2 Sep 2022 19:47:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=POr6peHZOfBhUzVXg3Q+r28kRUt2++JqJXx/jXBS+8U=; b=T6hKIpNnWQcP3OwCJCPqepRbq2 rJCnkjee+zJH0/Ow+NzKHHjtQ+/dDR/VtAeazTtIEWGYKuxEAZK5ZCC4H808iwBwlubonzbh3CDun kHQF49jCHmUUpmjPejIRpNrHMH/1HVBTWdCOmYHxzdRJ1EmNVKwOM39YDYRP2A/Gq7HOd3436AMHf QOndF2oWyDZu+tpcIUsLkFMiHWu4zieP0LZxw1IDda/jnfGmLZ3dLrPLtfdltTwFwMqrqQroerhjY hCpWMsYjS3O6Q1e7Sn7mfIDfgN11Eb+Ka9NytZiG8XreSQIYVRavqR5Qwwew4hZEY7PJ3vw7Hn+SW +bNc+nhw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibj-7k; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 04/57] mm: Add split_folio() Date: Fri, 2 Sep 2022 20:46:00 +0100 Message-Id: <20220902194653.1739778-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148043; a=rsa-sha256; cv=none; b=xq9m8UASjR8xDy0Hy/FbLVdK/KixUTXyNU1ftocreUoY0olCR4d4QKgzvyQ50eILVaFcUI 12QvhavXLsQ3XP+CXcJHNO+uLL3zn93fBeVQpNtt7Xlyj8z/n9DLB9lqwqp7VHj3apT/wF lUZ58269TyVUyMvEMhwJKPZiDlQ8n3I= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=T6hKIpNn; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148043; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=POr6peHZOfBhUzVXg3Q+r28kRUt2++JqJXx/jXBS+8U=; b=hTAvcqWo6VtyxGIoI1lBWOR1deT97VGplsBpjW0vSoE0Bul+gqHDWRKVAUB32WfnbiAoV+ UUzpszB/djPgF+8EbAhNF2QKHPoKeTylPQunKKnZT4y53+gEi/Z+URSfk0CXaQJutoiBb9 XDqO21ID77SjzfCd91d3cgaCw1BinK8= X-Stat-Signature: 3k4z9pnj53kfk5yc78knpmre3f7rgpkp X-Rspamd-Queue-Id: 0BA314004E Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=T6hKIpNn; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1662148042-713084 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This wrapper removes a need to use split_huge_page(&folio->page). Convert two callers. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/huge_mm.h | 5 +++++ mm/shmem.c | 2 +- mm/truncate.c | 2 +- 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 768e5261fdae..aa0a427284aa 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -435,6 +435,11 @@ static inline int split_folio_to_list(struct folio *folio, return split_huge_page_to_list(&folio->page, list); } +static inline int split_folio(struct folio *folio) +{ + return split_folio_to_list(folio, NULL); +} + /* * archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to * limitations in the implementation like arm64 MTE can override this to diff --git a/mm/shmem.c b/mm/shmem.c index 42e5888bf84d..674bde8b3085 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -629,7 +629,7 @@ static unsigned long shmem_unused_huge_shrink(struct shmem_sb_info *sbinfo, goto move_back; } - ret = split_huge_page(&folio->page); + ret = split_folio(folio); folio_unlock(folio); folio_put(folio); diff --git a/mm/truncate.c b/mm/truncate.c index 0b0708bf935f..c0be77e5c008 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -240,7 +240,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) folio_invalidate(folio, offset, length); if (!folio_test_large(folio)) return true; - if (split_huge_page(&folio->page) == 0) + if (split_folio(folio) == 0) return true; if (folio_test_dirty(folio)) return false; From patchwork Fri Sep 2 19:46:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10675ECAAD5 for ; Fri, 2 Sep 2022 19:48:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26734800FF; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A937D80117; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E0F9800FF; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5237F80113 for ; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 28CB8403C9 for ; Fri, 2 Sep 2022 19:47:05 +0000 (UTC) X-FDA: 79868178810.31.5B6E8CB Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id AE096140064 for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=eQz2L5Sd+WJhCeFxF2qMEcGTWQTSZ58GjI77k+ag9os=; b=mjOhpv6Go0YRvlYId1vfhit+KW HgsTHobw/0hgzgnon7wvXjCVpNNNf541CB1r0sZg5cl2wizUDZn+dxf9qW5komXYDKz3LC5rIS1rX 6qKDn3b3SrwKgc4aL1oeQJr+V0xNghqcgCwtZXHQjMw8iNDYcSmyAuZACMCDTFo5H8sysYV0KyvWY +JJpBV/bB/okb2y6c0z24zHMbUpP7WDfaicvIs6tiX1qN/X5nvG8sU9+yALeT9sYA8Byy3ZM9yTKp P28Jefww4teCzdIGL9nf2qYRRpBIiHn8i5JljdeX9l5pxFtOPxb1jCawFET4QOxmZTxEA7GOuP2EX c06/JOuQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibl-AQ; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 05/57] mm: Add folio_add_lru_vma() Date: Fri, 2 Sep 2022 20:46:01 +0100 Message-Id: <20220902194653.1739778-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148024; a=rsa-sha256; cv=none; b=gMLK3Uyt7CjhFkNKgEPYlCMnfXdeCb/gxgEsNqgwZh2K90CKN3pN6r8ZIgiKrXGYkB2Zhl aCFLOD23KluZL2AKv25388mL/ZpH+5TgC24OfuEkfW4vUIw+GOdfY92Osim3MFfc0sYOxS HvaiYmnHGkNeA/Y0FhVHISUHY1KqNIg= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=mjOhpv6G; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eQz2L5Sd+WJhCeFxF2qMEcGTWQTSZ58GjI77k+ag9os=; b=LnfdZrlBkWLmhlA820zAjfduILy+z/kekLfbIqwT0W0QCg1eLEL/i+VXvLIuKWrPw3hzf+ jdHjAADijGewTh6Vmc8YtNp+Pa35iAraVRP3qNs8yg28Rc3M5dVRaiUf4wUIoMhc9CNFc3 ggS5TI4EY4k2+A5fMM23DRhGOCvxaPo= X-Stat-Signature: uxo8c48zkibumxwhf75s6cjjmee46qkk X-Rspamd-Queue-Id: AE096140064 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=mjOhpv6G; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1662148024-499517 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert lru_cache_add_inactive_or_unevictable() to folio_add_lru_vma() and add a compatibility wrapper. --- include/linux/swap.h | 10 +++++----- mm/folio-compat.c | 6 ++++++ mm/swap.c | 19 +++++++++---------- 3 files changed, 20 insertions(+), 15 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 43150b9bbc5c..333d5588dc2d 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -375,11 +375,11 @@ extern unsigned long totalreserve_pages; /* linux/mm/swap.c */ -extern void lru_note_cost(struct lruvec *lruvec, bool file, - unsigned int nr_pages); -extern void lru_note_cost_folio(struct folio *); -extern void folio_add_lru(struct folio *); -extern void lru_cache_add(struct page *); +void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages); +void lru_note_cost_folio(struct folio *); +void folio_add_lru(struct folio *); +void folio_add_lru_vma(struct folio *, struct vm_area_struct *); +void lru_cache_add(struct page *); void mark_page_accessed(struct page *); void folio_mark_accessed(struct folio *); diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 458618c7302c..e1e23b4947d7 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -88,6 +88,12 @@ void lru_cache_add(struct page *page) } EXPORT_SYMBOL(lru_cache_add); +void lru_cache_add_inactive_or_unevictable(struct page *page, + struct vm_area_struct *vma) +{ + folio_add_lru_vma(page_folio(page), vma); +} + int add_to_page_cache_lru(struct page *page, struct address_space *mapping, pgoff_t index, gfp_t gfp) { diff --git a/mm/swap.c b/mm/swap.c index 9cee7f6a3809..6525011b715e 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -493,22 +493,21 @@ void folio_add_lru(struct folio *folio) EXPORT_SYMBOL(folio_add_lru); /** - * lru_cache_add_inactive_or_unevictable - * @page: the page to be added to LRU - * @vma: vma in which page is mapped for determining reclaimability + * folio_add_lru_vma() - Add a folio to the appropate LRU list for this VMA. + * @folio: The folio to be added to the LRU. + * @vma: VMA in which the folio is mapped. * - * Place @page on the inactive or unevictable LRU list, depending on its - * evictability. + * If the VMA is mlocked, @folio is added to the unevictable list. + * Otherwise, it is treated the same way as folio_add_lru(). */ -void lru_cache_add_inactive_or_unevictable(struct page *page, - struct vm_area_struct *vma) +void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) { - VM_BUG_ON_PAGE(PageLRU(page), page); + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); if (unlikely((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) == VM_LOCKED)) - mlock_new_page(page); + mlock_new_page(&folio->page); else - lru_cache_add(page); + folio_add_lru(folio); } /* From patchwork Fri Sep 2 19:46:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964580 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61C87ECAAD5 for ; Fri, 2 Sep 2022 19:48:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C050C80101; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 58C0E8010A; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02EE480108; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 688DF80108 for ; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4CC02140503 for ; Fri, 2 Sep 2022 19:47:07 +0000 (UTC) X-FDA: 79868178894.22.AA4B8A3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id 07E5418005B for ; Fri, 2 Sep 2022 19:47:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=IuyZHerRhkwBNtRa8kIi1nFk+TiVzHuHYYMyLWOqSdo=; b=tein7UpAYOCpsZISUS0xaljYAC MMvxuf2rCi5TZxUvylVbVZJwixF1yzDxvqX823wonm6bqYhkajR3p/R3n2zx4sepAftkX2a9Y3og2 x3yJYaJcf7aIdUTYYu1/a1cJFfUpb3B4k/pWdiUhLF7a6AAqetdDeYE8UUAF1DNH7+yX/IQ9mHw+O inhMxMFz5ZU4hhoFUJBkjnefL72alTiumFTIzY4f5K8y/CMMaoWqBMzfIdnI8imNr/nZe++tcPSY3 pd8wEyGsNeOiGH6kjvkjDaCNe48UXiQEL/GATfIL2Vq/OTaJSpnd0U3czrdq+cBxFcogMJemJfXob Gfk/ZHSg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibn-DC; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 06/57] shmem: Convert shmem_writepage() to use a folio throughout Date: Fri, 2 Sep 2022 20:46:02 +0100 Message-Id: <20220902194653.1739778-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148027; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IuyZHerRhkwBNtRa8kIi1nFk+TiVzHuHYYMyLWOqSdo=; b=bu2cuJIwH6g4Lbq8EGM5Uorpeiyv9j5TCWer/wX1q8Nr/NVqdPNtmainyWJOJLJZ2P3SQf 8NfxM5/lIoDxV8+Fs7hWKg3/Bx4ExjEaPdZektZK6ddIrZxkWS542+6afupJ7Z6R2DVOhD MQK4gOlLGzi1c3Ya7zYwNiU2JFTKzPw= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=tein7UpA; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148027; a=rsa-sha256; cv=none; b=XEhXNQsup6kgXb+LQELgchxaLkUUSGw0yodi9/WReCbFxA9RMVi+rXig3w9dSflXcikFbR xWQ681NozFKYkUwqD18l84h4yO3cI6eJgtjMsj69hxKWF7i9mAlMCToZScz3mJbd86SlP2 EIa8Whu4jdCdwzNj8QE2FUVaLK9tAjA= X-Rspamd-Queue-Id: 07E5418005B Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=tein7UpA; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: ip96b1r1dpphpstsubpyma68g4hp5dtg X-HE-Tag: 1662148026-790788 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Even though we will split any large folio that comes in, write the code to handle large folios so as to not leave a trap for whoever tries to handle large folios in the swap cache. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 47 ++++++++++++++++++++++++----------------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 674bde8b3085..3d2d35728793 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1328,17 +1328,18 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) * "force", drivers/gpu/drm/i915/gem/i915_gem_shmem.c gets huge pages, * and its shmem_writeback() needs them to be split when swapping. */ - if (PageTransCompound(page)) { + if (folio_test_large(folio)) { /* Ensure the subpages are still dirty */ - SetPageDirty(page); + folio_test_set_dirty(folio); if (split_huge_page(page) < 0) goto redirty; - ClearPageDirty(page); + folio = page_folio(page); + folio_clear_dirty(folio); } - BUG_ON(!PageLocked(page)); - mapping = page->mapping; - index = page->index; + BUG_ON(!folio_test_locked(folio)); + mapping = folio->mapping; + index = folio->index; inode = mapping->host; info = SHMEM_I(inode); if (info->flags & VM_LOCKED) @@ -1361,15 +1362,15 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) /* * This is somewhat ridiculous, but without plumbing a SWAP_MAP_FALLOC * value into swapfile.c, the only way we can correctly account for a - * fallocated page arriving here is now to initialize it and write it. + * fallocated folio arriving here is now to initialize it and write it. * - * That's okay for a page already fallocated earlier, but if we have + * That's okay for a folio already fallocated earlier, but if we have * not yet completed the fallocation, then (a) we want to keep track - * of this page in case we have to undo it, and (b) it may not be a + * of this folio in case we have to undo it, and (b) it may not be a * good idea to continue anyway, once we're pushing into swap. So - * reactivate the page, and let shmem_fallocate() quit when too many. + * reactivate the folio, and let shmem_fallocate() quit when too many. */ - if (!PageUptodate(page)) { + if (!folio_test_uptodate(folio)) { if (inode->i_private) { struct shmem_falloc *shmem_falloc; spin_lock(&inode->i_lock); @@ -1385,9 +1386,9 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) if (shmem_falloc) goto redirty; } - clear_highpage(page); - flush_dcache_page(page); - SetPageUptodate(page); + folio_zero_range(folio, 0, folio_size(folio)); + flush_dcache_folio(folio); + folio_mark_uptodate(folio); } swap = folio_alloc_swap(folio); @@ -1396,7 +1397,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) /* * Add inode to shmem_unuse()'s list of swapped-out inodes, - * if it's not already there. Do it now before the page is + * if it's not already there. Do it now before the folio is * moved to swap cache, when its pagelock no longer protects * the inode from eviction. But don't unlock the mutex until * we've incremented swapped, because shmem_unuse_inode() will @@ -1406,7 +1407,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) if (list_empty(&info->swaplist)) list_add(&info->swaplist, &shmem_swaplist); - if (add_to_swap_cache(page, swap, + if (add_to_swap_cache(&folio->page, swap, __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN, NULL) == 0) { spin_lock_irq(&info->lock); @@ -1415,21 +1416,21 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) spin_unlock_irq(&info->lock); swap_shmem_alloc(swap); - shmem_delete_from_page_cache(page, swp_to_radix_entry(swap)); + shmem_delete_from_page_cache(&folio->page, swp_to_radix_entry(swap)); mutex_unlock(&shmem_swaplist_mutex); - BUG_ON(page_mapped(page)); - swap_writepage(page, wbc); + BUG_ON(folio_mapped(folio)); + swap_writepage(&folio->page, wbc); return 0; } mutex_unlock(&shmem_swaplist_mutex); - put_swap_page(page, swap); + put_swap_page(&folio->page, swap); redirty: - set_page_dirty(page); + folio_mark_dirty(folio); if (wbc->for_reclaim) - return AOP_WRITEPAGE_ACTIVATE; /* Return with page locked */ - unlock_page(page); + return AOP_WRITEPAGE_ACTIVATE; /* Return with folio locked */ + folio_unlock(folio); return 0; } From patchwork Fri Sep 2 19:46:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964586 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C142C6FA82 for ; Fri, 2 Sep 2022 19:48:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC6BA8011D; Fri, 2 Sep 2022 15:47:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D761680119; Fri, 2 Sep 2022 15:47:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3DFF8011D; Fri, 2 Sep 2022 15:47:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AC6A180119 for ; Fri, 2 Sep 2022 15:47:17 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 822071607F7 for ; Fri, 2 Sep 2022 19:47:17 +0000 (UTC) X-FDA: 79868179314.15.2631DE2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 175531C0067 for ; Fri, 2 Sep 2022 19:47:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=m/c5IH76eaSvTQjuEmr5q/GNmfcy3v7SI+fS5ZP5tJ0=; b=AsslQU7hlS5ImomAwi42+TRdq6 ZZZtUxZTXFqS7pQQHhrcSDNkN9OlENAmGyDkQDaFy4Pwyj59SRM9jMDtL41etBCcOZmknuyDdGQjT es6oL2Ob1GAvXa/M6COT8Ey0TIhbGs+7JhWW/oc5rLH/kn/4xk6ee2oqLnXuk0KxG1esvGLuCYXEm qlG9o4fMc71tx0VQQNLg0N2kMTzMc05KpZROX/cwEqoPh3HzgOz95Cxv0tCXfgU0VetiQztYqqhlT HOXjw9AlTn4jMAhyUuPFbcjUFvWOxGtU7ddfgQeY7WmU/DskMkF8EmWUqDf2LwlCQGV99eOAswN/y ZrrzKkCw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibp-Fq; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 07/57] shmem: Convert shmem_delete_from_page_cache() to take a folio Date: Fri, 2 Sep 2022 20:46:03 +0100 Message-Id: <20220902194653.1739778-8-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AsslQU7h; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148037; a=rsa-sha256; cv=none; b=JEoIxluuFCQC1Nc29SZ8nnTveEeK1TX9PIGgru+hZnnOml9+TS7K1+lxrcnRa3fdHRX1N+ wk4di9GDh8ezisNXZehKwexltIUKnEyXzlMt7xVQ0Jy9qgdEviGgBVT4E0CuNu0VH63sCE ejuaWfIRVmN46FH2ejVMyss4kAKZY+M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m/c5IH76eaSvTQjuEmr5q/GNmfcy3v7SI+fS5ZP5tJ0=; b=t4tWLKk+O1F420Tqm5MpqOSi/1tRBeNcAGg8eyHCu4FvtRmigrV1v+Aegv7V+A31vP+rWI zLk6m+NtP6BqsmhDvk7Uwss/jy6kEy1r0IRtFBFYhHzO/1xQOTr3E1JQN7Pv3/asSFuiGb BpuK7y1Yc7AqGQc/6uMeo1Q1w9ukeZA= X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 175531C0067 X-Stat-Signature: 95hq1ws5roac7ncccpcn9u5nsbucnxzm Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AsslQU7h; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1662148036-888611 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove the assertion that the page is not Compound as this function now handles large folios correctly. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 3d2d35728793..9e851fc87601 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -763,23 +763,22 @@ static int shmem_add_to_page_cache(struct folio *folio, } /* - * Like delete_from_page_cache, but substitutes swap for page. + * Like delete_from_page_cache, but substitutes swap for @folio. */ -static void shmem_delete_from_page_cache(struct page *page, void *radswap) +static void shmem_delete_from_page_cache(struct folio *folio, void *radswap) { - struct address_space *mapping = page->mapping; + struct address_space *mapping = folio->mapping; + long nr = folio_nr_pages(folio); int error; - VM_BUG_ON_PAGE(PageCompound(page), page); - xa_lock_irq(&mapping->i_pages); - error = shmem_replace_entry(mapping, page->index, page, radswap); - page->mapping = NULL; - mapping->nrpages--; - __dec_lruvec_page_state(page, NR_FILE_PAGES); - __dec_lruvec_page_state(page, NR_SHMEM); + error = shmem_replace_entry(mapping, folio->index, folio, radswap); + folio->mapping = NULL; + mapping->nrpages -= nr; + __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, -nr); + __lruvec_stat_mod_folio(folio, NR_SHMEM, -nr); xa_unlock_irq(&mapping->i_pages); - put_page(page); + folio_put(folio); BUG_ON(error); } @@ -1416,7 +1415,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) spin_unlock_irq(&info->lock); swap_shmem_alloc(swap); - shmem_delete_from_page_cache(&folio->page, swp_to_radix_entry(swap)); + shmem_delete_from_page_cache(folio, swp_to_radix_entry(swap)); mutex_unlock(&shmem_swaplist_mutex); BUG_ON(folio_mapped(folio)); From patchwork Fri Sep 2 19:46:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964592 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BE60C6FA82 for ; Fri, 2 Sep 2022 19:48:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E6D0A80122; Fri, 2 Sep 2022 15:47:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CDCB180120; Fri, 2 Sep 2022 15:47:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A93880122; Fri, 2 Sep 2022 15:47:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7A88A80120 for ; Fri, 2 Sep 2022 15:47:29 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5B1481601BA for ; Fri, 2 Sep 2022 19:47:29 +0000 (UTC) X-FDA: 79868179818.10.D45E09B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id E93A2100060 for ; Fri, 2 Sep 2022 19:47:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=G2a4/s/ZK1NLTC/qMKepXr39H7qLw37+HTU/fUXQE5c=; b=kChyukNnjZquXUjR8RJHZCmghw hC6qSWfIILzXbSs1mI0nb7tS8x7kBDJqbLW3BRKDLNCC5eke/mjRKOIzRjGDetoFn1HW0D+pEPxii tKTnzmiUdbJmHVd14hc1flNfoS5z43fMFCQbm14PmfbPL7oJSDIsYtwti5T3xlK3l82moYy3Mm6n3 CkOi9D8ayJPlbLGQzdwF1mRvVnnoZtO3s6hKc1owjDQJ/Vs/ojKh/D6JGg7u2K1hn86IaAZq50kUA yKRD96tlCgKHNaC6miv9tpv7nKM53wSWoXMLccerPo2g4VrijDWqYhlxMrTh7KqoPXnjdvMODDy+G SJModUyg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibr-IY; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 08/57] shmem: Convert shmem_replace_page() to use folios throughout Date: Fri, 2 Sep 2022 20:46:04 +0100 Message-Id: <20220902194653.1739778-9-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=kChyukNn; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148048; a=rsa-sha256; cv=none; b=Pjn4fwUePqDtKJDWr0DGpaivXilIeC96BfAObeG8mJluhfmHKHZ69SC4fUSwyORyNUxlsx I4kxEkDJRmhorklFSnu++tcAU60mtLetRqWUktGaE2qCVle0HevVSH52WS3tYEUQr95X1m fpzZ4GadJt9DwhoycShKfXdvgvVXl/s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148048; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G2a4/s/ZK1NLTC/qMKepXr39H7qLw37+HTU/fUXQE5c=; b=Z8WFln+I0tLl6BsHGso1OhmomNdeWq91ab/ybut6RqVhHVMTDft3QCaHw3uGQKbCXyUNHR GMYPBUNexIkC1BkasM0A+99KH6lvZve8zo+EPcxWYmA6u6geCgRzFLF3jL5q5AHXCWsL5c 2iPjRuC2OY7AnFT6yYP2G6Z1xB91LM8= X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: E93A2100060 X-Stat-Signature: bop8draocz7cd5z84kdixt47q6up6oje Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=kChyukNn; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1662148048-4289 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce folio_set_swap_entry() to abstract how both folio->private and swp_entry_t work. Use swap_address_space() directly instead of indirecting through folio_mapping(). Include an assertion that the old folio is not large as we only allocate a single-page folio to replace it. Use folio_put_refs() instead of calling folio_put() twice. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/swap.h | 5 ++++ mm/shmem.c | 67 +++++++++++++++++++++----------------------- 2 files changed, 37 insertions(+), 35 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 333d5588dc2d..afcb76bbd141 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -351,6 +351,11 @@ static inline swp_entry_t folio_swap_entry(struct folio *folio) return entry; } +static inline void folio_set_swap_entry(struct folio *folio, swp_entry_t entry) +{ + folio->private = (void *)entry.val; +} + /* linux/mm/workingset.c */ void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages); void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg); diff --git a/mm/shmem.c b/mm/shmem.c index 9e851fc87601..4113f1b9d4a8 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1560,12 +1560,6 @@ static struct folio *shmem_alloc_folio(gfp_t gfp, return folio; } -static struct page *shmem_alloc_page(gfp_t gfp, - struct shmem_inode_info *info, pgoff_t index) -{ - return &shmem_alloc_folio(gfp, info, index)->page; -} - static struct folio *shmem_alloc_and_acct_folio(gfp_t gfp, struct inode *inode, pgoff_t index, bool huge) { @@ -1617,51 +1611,49 @@ static bool shmem_should_replace_folio(struct folio *folio, gfp_t gfp) static int shmem_replace_page(struct page **pagep, gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { - struct page *oldpage, *newpage; struct folio *old, *new; struct address_space *swap_mapping; swp_entry_t entry; pgoff_t swap_index; int error; - oldpage = *pagep; - entry.val = page_private(oldpage); + old = page_folio(*pagep); + entry = folio_swap_entry(old); swap_index = swp_offset(entry); - swap_mapping = page_mapping(oldpage); + swap_mapping = swap_address_space(entry); /* * We have arrived here because our zones are constrained, so don't * limit chance of success by further cpuset and node constraints. */ gfp &= ~GFP_CONSTRAINT_MASK; - newpage = shmem_alloc_page(gfp, info, index); - if (!newpage) + VM_BUG_ON_FOLIO(folio_test_large(old), old); + new = shmem_alloc_folio(gfp, info, index); + if (!new) return -ENOMEM; - get_page(newpage); - copy_highpage(newpage, oldpage); - flush_dcache_page(newpage); + folio_get(new); + folio_copy(new, old); + flush_dcache_folio(new); - __SetPageLocked(newpage); - __SetPageSwapBacked(newpage); - SetPageUptodate(newpage); - set_page_private(newpage, entry.val); - SetPageSwapCache(newpage); + __folio_set_locked(new); + __folio_set_swapbacked(new); + folio_mark_uptodate(new); + folio_set_swap_entry(new, entry); + folio_set_swapcache(new); /* * Our caller will very soon move newpage out of swapcache, but it's * a nice clean interface for us to replace oldpage by newpage there. */ xa_lock_irq(&swap_mapping->i_pages); - error = shmem_replace_entry(swap_mapping, swap_index, oldpage, newpage); + error = shmem_replace_entry(swap_mapping, swap_index, old, new); if (!error) { - old = page_folio(oldpage); - new = page_folio(newpage); mem_cgroup_migrate(old, new); - __inc_lruvec_page_state(newpage, NR_FILE_PAGES); - __inc_lruvec_page_state(newpage, NR_SHMEM); - __dec_lruvec_page_state(oldpage, NR_FILE_PAGES); - __dec_lruvec_page_state(oldpage, NR_SHMEM); + __lruvec_stat_mod_folio(new, NR_FILE_PAGES, 1); + __lruvec_stat_mod_folio(new, NR_SHMEM, 1); + __lruvec_stat_mod_folio(old, NR_FILE_PAGES, -1); + __lruvec_stat_mod_folio(old, NR_SHMEM, -1); } xa_unlock_irq(&swap_mapping->i_pages); @@ -1671,18 +1663,17 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp, * both PageSwapCache and page_private after getting page lock; * but be defensive. Reverse old to newpage for clear and free. */ - oldpage = newpage; + old = new; } else { - lru_cache_add(newpage); - *pagep = newpage; + folio_add_lru(new); + *pagep = &new->page; } - ClearPageSwapCache(oldpage); - set_page_private(oldpage, 0); + folio_clear_swapcache(old); + old->private = NULL; - unlock_page(oldpage); - put_page(oldpage); - put_page(oldpage); + folio_unlock(old); + folio_put_refs(old, 2); return error; } @@ -2383,6 +2374,12 @@ static struct inode *shmem_get_inode(struct super_block *sb, struct inode *dir, } #ifdef CONFIG_USERFAULTFD +static struct page *shmem_alloc_page(gfp_t gfp, + struct shmem_inode_info *info, pgoff_t index) +{ + return &shmem_alloc_folio(gfp, info, index)->page; +} + int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, struct vm_area_struct *dst_vma, From patchwork Fri Sep 2 19:46:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFAC6ECAAD5 for ; Fri, 2 Sep 2022 19:47:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B7E9800E9; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 16768800EA; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00836800E9; Fri, 2 Sep 2022 15:46:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E64D7800DD for ; Fri, 2 Sep 2022 15:46:59 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B874380580 for ; Fri, 2 Sep 2022 19:46:59 +0000 (UTC) X-FDA: 79868178558.09.DD40B1B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id 0AEF020055 for ; Fri, 2 Sep 2022 19:46:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rZ9tiYok3CmkdoKC4u5YlyBJYtd4IGvFvqCkTZC9NJA=; b=NLAaWMQ3kbvkMLsNmzgQY/1ZZ3 jDprBftjPzcJ4/jZsFohyU9o8k/lGo6qmREWhgjd8OHxAhW+uqd1XjZfd2ZGYjKf5ee/eK/yUL0sK LjaE+iFT4SgcldJx0kYAe5ESWNLetyvaVNf7gYL02nuZdg3ACoUp2hbRM1tBDBir6PYKy/uUdkAEx 6ADo61JdA+t5Dx6F3KbP3PQ1QqmO1qDmzNFFWlqrH51T4jhRYtEgZXQNRHC4ZY2AEQgsEHcB1M4x6 sjDkBP4Gx8fk8ARFRNaBAwJTpIcUuxVDspjlYqhNu/TSl+8ppQqLqgHFYbRTgv22L0FdPazOZaYCv 7TeQaFSQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibt-LG; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 09/57] mm/swapfile: Remove page_swapcount() Date: Fri, 2 Sep 2022 20:46:05 +0100 Message-Id: <20220902194653.1739778-10-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148019; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rZ9tiYok3CmkdoKC4u5YlyBJYtd4IGvFvqCkTZC9NJA=; b=hL+maF08M280zrdPIgMoHq8XRfEKzeWmj8A58eI/wMfpW94hCkIJfBcS4GrJNe+sZTNzVF x2NBoSCR6mH335Hr+Wf3ZVB4hKiQmS7vJJfy3H4OT4vtGOXTmc30psd0ENLrVWbLrcWG3H SSUParU10Z4klhX57Rg3YK5ndZzeZXE= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NLAaWMQ3; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148019; a=rsa-sha256; cv=none; b=XS7KqZjP2BS6oUZFnIRyrLyTtXB3M4fHGvblZEEoxENE3tX5Wj84NBP49CHQ/n92NK/lwf jdTFxPTWaoMtwpO9SPh1Y2179AKqqNI+0h1svW+s7Z9Bv233QM1vji68mnDqPx75ccSD2O YKSk6J5fEicNnbnssUJZnXI9sKTRczg= X-Rspamd-Queue-Id: 0AEF020055 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NLAaWMQ3; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: r84s8qejq3wx3t3r3d19c6oj49zs8sih X-HE-Tag: 1662148018-706935 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: By restructuring folio_swapped(), it can use swap_swapcount() instead of page_swapcount(). It's even a little more efficient. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swapfile.c | 46 +++++++++++++--------------------------------- 1 file changed, 13 insertions(+), 33 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 1fdccd2f1422..c042fd71de02 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1427,30 +1427,6 @@ void swapcache_free_entries(swp_entry_t *entries, int n) spin_unlock(&p->lock); } -/* - * How many references to page are currently swapped out? - * This does not give an exact answer when swap count is continued, - * but does include the high COUNT_CONTINUED flag to allow for that. - */ -static int page_swapcount(struct page *page) -{ - int count = 0; - struct swap_info_struct *p; - struct swap_cluster_info *ci; - swp_entry_t entry; - unsigned long offset; - - entry.val = page_private(page); - p = _swap_info_get(entry); - if (p) { - offset = swp_offset(entry); - ci = lock_cluster_or_swap_info(p, offset); - count = swap_count(p->swap_map[offset]); - unlock_cluster_or_swap_info(p, ci); - } - return count; -} - int __swap_count(swp_entry_t entry) { struct swap_info_struct *si; @@ -1465,11 +1441,16 @@ int __swap_count(swp_entry_t entry) return count; } +/* + * How many references to @entry are currently swapped out? + * This does not give an exact answer when swap count is continued, + * but does include the high COUNT_CONTINUED flag to allow for that. + */ static int swap_swapcount(struct swap_info_struct *si, swp_entry_t entry) { - int count = 0; pgoff_t offset = swp_offset(entry); struct swap_cluster_info *ci; + int count; ci = lock_cluster_or_swap_info(si, offset); count = swap_count(si->swap_map[offset]); @@ -1570,17 +1551,16 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si, static bool folio_swapped(struct folio *folio) { - swp_entry_t entry; - struct swap_info_struct *si; + swp_entry_t entry = folio_swap_entry(folio); + struct swap_info_struct *si = _swap_info_get(entry); + + if (!si) + return false; if (!IS_ENABLED(CONFIG_THP_SWAP) || likely(!folio_test_large(folio))) - return page_swapcount(&folio->page) != 0; + return swap_swapcount(si, entry) != 0; - entry = folio_swap_entry(folio); - si = _swap_info_get(entry); - if (si) - return swap_page_trans_huge_swapped(si, entry); - return false; + return swap_page_trans_huge_swapped(si, entry); } /* From patchwork Fri Sep 2 19:46:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964548 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99872ECAAD5 for ; Fri, 2 Sep 2022 19:47:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25660800F6; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 14538800F9; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9097F800F9; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5FC5C800F7 for ; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 48A5B1A0222 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) X-FDA: 79868178684.29.CEEDFCF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf24.hostedemail.com (Postfix) with ESMTP id 02E7B180077 for ; Fri, 2 Sep 2022 19:47:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=oGSPn8NwOXRP4TYNP3MMOCC5rGBSouXy25gKhFOlJYA=; b=AeDxYZ6YE2uYvhS6quE6GV/iQy bRbSolOaeB6ovSswv9ufUahlo9XxGNxcsEn5AcAy5AL91XOpSeNIfZHnmksLMJyGImMWT0xw4a+VQ QHkyYuGc0tKkntXMUg6qVSSuH/OCwUHywY7cSV5x83KYcxcloTOHbC1ht8s5D7t08PKx8GpLhLh6Z uX9Zh2mWL+Cabv5T3bRgRm3pgJ/TKNoNh3wK4O5LltZYksOo5IfekHT22LGv7osf/Ae63zLLL0Y4+ FiBig999XnZ11KQqDdko+S9KfN1BaU7P5rUqMblrryUvELw7yp6VQ8DJyPJlJXag4SH4I7EJUkzej 1nse1fSw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibv-NS; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 10/57] mm/swapfile: Convert try_to_free_swap() to folio_free_swap() Date: Fri, 2 Sep 2022 20:46:06 +0100 Message-Id: <20220902194653.1739778-11-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AeDxYZ6Y; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148022; a=rsa-sha256; cv=none; b=lI88B701axEmPOikIAr2/BYLXwxy57h6+zNr5Nwb+yCnK+GTth+K0Djg9CyignI84/p68x BdhY/RFka1xjJR/86nJK3WsjkoxQg0bkiOehepoGMhYCqEvnAie6LyAOd1g1sK9zQ+zOdW WGoktSMNFSwDE8AoF1xfAvQhw5ZfqhI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oGSPn8NwOXRP4TYNP3MMOCC5rGBSouXy25gKhFOlJYA=; b=SUNLmVaKIr3y71Lp/vM8N6Z6QkGJ027oE4WBCcXPh+/BrAjp8dWsmfzpWO5Ah9dk+X46kn wYeqD4zqRwXsB1BVz/TBkIwxuD4UzJwIfC5W0GeIssvxdwQPneedEmSY9PV1852v/ppCLT pAswh/JNchDH5wunOMaz6zfjEHZDqBU= X-Rspamd-Queue-Id: 02E7B180077 X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AeDxYZ6Y; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Stat-Signature: xbmtth3jqaecqype3eqn864b9aw9gg8r X-HE-Tag: 1662148021-672562 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add kernel-doc for folio_free_swap() and make it return bool. Add a try_to_free_swap() compatibility wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/swap.h | 6 ++++++ mm/folio-compat.c | 7 +++++++ mm/swapfile.c | 32 ++++++++++++++++++-------------- mm/vmscan.c | 2 +- 4 files changed, 32 insertions(+), 15 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index afcb76bbd141..4595cbc1cb02 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -486,6 +486,7 @@ static inline long get_nr_swap_pages(void) extern void si_swapinfo(struct sysinfo *); swp_entry_t folio_alloc_swap(struct folio *folio); +bool folio_free_swap(struct folio *folio); extern void put_swap_page(struct page *page, swp_entry_t entry); extern swp_entry_t get_swap_page_of_type(int); extern int get_swap_pages(int n, swp_entry_t swp_entries[], int entry_size); @@ -602,6 +603,11 @@ static inline swp_entry_t folio_alloc_swap(struct folio *folio) return entry; } +static inline bool folio_free_swap(struct folio *folio) +{ + return false; +} + static inline int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page, unsigned long nr_pages, sector_t start_block) diff --git a/mm/folio-compat.c b/mm/folio-compat.c index e1e23b4947d7..06d47f00609b 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -146,3 +146,10 @@ void putback_lru_page(struct page *page) { folio_putback_lru(page_folio(page)); } + +#ifdef CONFIG_SWAP +int try_to_free_swap(struct page *page) +{ + return folio_free_swap(page_folio(page)); +} +#endif diff --git a/mm/swapfile.c b/mm/swapfile.c index c042fd71de02..880871f4c6d4 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1563,43 +1563,47 @@ static bool folio_swapped(struct folio *folio) return swap_page_trans_huge_swapped(si, entry); } -/* - * If swap is getting full, or if there are no more mappings of this page, - * then try_to_free_swap is called to free its swap space. +/** + * folio_free_swap() - Free the swap space used for this folio. + * @folio: The folio to remove. + * + * If swap is getting full, or if there are no more mappings of this folio, + * then call folio_free_swap to free its swap space. + * + * Return: true if we were able to release the swap space. */ -int try_to_free_swap(struct page *page) +bool folio_free_swap(struct folio *folio) { - struct folio *folio = page_folio(page); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); if (!folio_test_swapcache(folio)) - return 0; + return false; if (folio_test_writeback(folio)) - return 0; + return false; if (folio_swapped(folio)) - return 0; + return false; /* * Once hibernation has begun to create its image of memory, - * there's a danger that one of the calls to try_to_free_swap() + * there's a danger that one of the calls to folio_free_swap() * - most probably a call from __try_to_reclaim_swap() while * hibernation is allocating its own swap pages for the image, * but conceivably even a call from memory reclaim - will free - * the swap from a page which has already been recorded in the - * image as a clean swapcache page, and then reuse its swap for + * the swap from a folio which has already been recorded in the + * image as a clean swapcache folio, and then reuse its swap for * another page of the image. On waking from hibernation, the - * original page might be freed under memory pressure, then + * original folio might be freed under memory pressure, then * later read back in from swap, now with the wrong data. * * Hibernation suspends storage while it is writing the image * to disk so check that here. */ if (pm_suspended_storage()) - return 0; + return false; delete_from_swap_cache(folio); folio_set_dirty(folio); - return 1; + return true; } /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 93b1087fc969..d3e26712dbc1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2004,7 +2004,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (folio_test_swapcache(folio) && (mem_cgroup_swap_full(&folio->page) || folio_test_mlocked(folio))) - try_to_free_swap(&folio->page); + folio_free_swap(folio); VM_BUG_ON_FOLIO(folio_test_active(folio), folio); if (!folio_test_mlocked(folio)) { int type = folio_is_file_lru(folio); From patchwork Fri Sep 2 19:46:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AADBDC6FA83 for ; Fri, 2 Sep 2022 19:48:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3BAD680117; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D8E2980113; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81F5680105; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8803980108 for ; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 69C84801EA for ; Fri, 2 Sep 2022 19:47:05 +0000 (UTC) X-FDA: 79868178810.16.DFCBBEF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id 1278140099 for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=EtUfHusf3N70jYSPPPixSjqlm8bwcBxVI5pl2nsQ5N4=; b=bXh5tCuv48UZsMOa1PURnVCSvP ewYCOvObZfqU8VO8eOVO5f4iuBg6r34ziqf/AcVnTPtBHqgPVYqQaqmZh5Hj2hi98D0HVuL1XEu++ cZdsiLpLnOxBgeUwD1KWgDXgA4J83IruRUAP3towQ9nCJ/TClIXyKkyEC+6yhvHXY5+Ka/E3rb/VQ HtAjOwAsXnDX/l15P68RZ6Xh+ZhOaLVS58/jf8q69d9eUJu2EHqoqVE2uvUPvYJAZhiXq5XZPBDvW guyd1Nl8ciNk1H9ILNinz2mZv/+xVIrGMpZGP+8jt5jLr4wfUAUK/DAPJMkT87H/tNFLI4Vid4nBq MfWY/uLQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibx-PY; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 11/57] mm/swap: Convert __read_swap_cache_async() to use a folio Date: Fri, 2 Sep 2022 20:46:07 +0100 Message-Id: <20220902194653.1739778-12-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bXh5tCuv; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148025; a=rsa-sha256; cv=none; b=OyeZm+KCZra+UCnQm4ixbWowQnrPGtWtZfa3+oCQOV/92Cuh0n+Hnvdghb/bC14nRBYuRn D/cmGn2nO3GSMwtk7BpZibVcJvIf5tzSh+LARBWgp5VPUK959lYyX1V2PSGPCHEgn+wVGA XjvvCBJtQv4WdQ9TWTbqq7kVBUUmoZE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148025; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EtUfHusf3N70jYSPPPixSjqlm8bwcBxVI5pl2nsQ5N4=; b=lnGVDDaVFbe/ZiJRrzlmvCQyp9+2a0iwAwUy4kk8PpVL8zpAqMb7SF8eLCJZOjPMLnT8Ph SU6bl9V3N9l/qsmsDMohb9SNgLTdjzpUccQDxnbIWvfoR11d4zsb34WEjXLyAAzEe/44yY h/mJBFtclA4ZeY1U6bWT5vYBc8PyfCM= X-Rspamd-Queue-Id: 1278140099 X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bXh5tCuv; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Stat-Signature: gu7snjead5xo6u1axgsm8y6f46iisgaq X-HE-Tag: 1662148024-340296 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove a few hidden (and one visible) calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap_state.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index 41afa6d45b23..b1e181fc5268 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -411,7 +411,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, bool *new_page_allocated) { struct swap_info_struct *si; - struct page *page; + struct folio *folio; void *shadow = NULL; *new_page_allocated = false; @@ -426,11 +426,11 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, si = get_swap_device(entry); if (!si) return NULL; - page = find_get_page(swap_address_space(entry), - swp_offset(entry)); + folio = filemap_get_folio(swap_address_space(entry), + swp_offset(entry)); put_swap_device(si); - if (page) - return page; + if (folio) + return folio_file_page(folio, swp_offset(entry)); /* * Just skip read ahead for unused swap slot. @@ -448,8 +448,8 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, * before marking swap_map SWAP_HAS_CACHE, when -EEXIST will * cause any racers to loop around until we add it to cache. */ - page = alloc_page_vma(gfp_mask, vma, addr); - if (!page) + folio = vma_alloc_folio(gfp_mask, 0, vma, addr, false); + if (!folio) return NULL; /* @@ -459,7 +459,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, if (!err) break; - put_page(page); + folio_put(folio); if (err != -EEXIST) return NULL; @@ -477,30 +477,30 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, * The swap entry is ours to swap in. Prepare the new page. */ - __SetPageLocked(page); - __SetPageSwapBacked(page); + __folio_set_locked(folio); + __folio_set_swapbacked(folio); - if (mem_cgroup_swapin_charge_page(page, NULL, gfp_mask, entry)) + if (mem_cgroup_swapin_charge_page(&folio->page, NULL, gfp_mask, entry)) goto fail_unlock; /* May fail (-ENOMEM) if XArray node allocation failed. */ - if (add_to_swap_cache(page, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow)) + if (add_to_swap_cache(&folio->page, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow)) goto fail_unlock; mem_cgroup_swapin_uncharge_swap(entry); if (shadow) - workingset_refault(page_folio(page), shadow); + workingset_refault(folio, shadow); - /* Caller will initiate read into locked page */ - lru_cache_add(page); + /* Caller will initiate read into locked folio */ + folio_add_lru(folio); *new_page_allocated = true; - return page; + return &folio->page; fail_unlock: - put_swap_page(page, entry); - unlock_page(page); - put_page(page); + put_swap_page(&folio->page, entry); + folio_unlock(folio); + folio_put(folio); return NULL; } From patchwork Fri Sep 2 19:46:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 722BCC6FA83 for ; Fri, 2 Sep 2022 19:48:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AF7580124; Fri, 2 Sep 2022 15:47:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C59C80123; Fri, 2 Sep 2022 15:47:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6733280122; Fri, 2 Sep 2022 15:47:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 54DC580120 for ; Fri, 2 Sep 2022 15:47:29 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 38F1C1C6884 for ; Fri, 2 Sep 2022 19:47:29 +0000 (UTC) X-FDA: 79868179818.25.18D46AE Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 043CB180044 for ; Fri, 2 Sep 2022 19:47:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=W2O1MoDfgqdouCpzRUrjzlsQ5SFNdm1Z+9DnK8kpTE8=; b=ENafStdAhbU/LLBICAKV4v/kZo +NxyJLw1OiQbxfG8uXCbg6cKDrxjorFq2YhscSvzX6+RPTyYjYcWQlORv+2iRAo5nDtPBFcs+8zhZ ciBGnAzFUE4ESXKNzvKC9ikBJ1xbZGfvF3wVQ8ZuOA1PaFjDapYKDT/ZB9FjHfKX/YxqLGaCbRaYN spargd2CAdHVQBevqUiDMadDlwI4C0AWCMCIX821qSJ8iDBqSZQ4EHT8UHRRzT0PFpwtv/zzlhXiE +Fc7TvwFmv/pZXbjCQk6PTEOn8iZD+0WQHj8+LYpjOFpSPEVWFBE0v5P7x8sBDDjFoBTOuj6UyxX6 IwyFW2HQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ibz-RZ; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 12/57] mm/swap: Convert add_to_swap_cache() to take a folio Date: Fri, 2 Sep 2022 20:46:08 +0100 Message-Id: <20220902194653.1739778-13-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148049; a=rsa-sha256; cv=none; b=VA3y8vn4XKqkzdBvY+CveulA4OqLpHtzHgvtGZ8b+ICi1g8SPIt6IUE6mLu0Ni42F2r/EI Sp7UwULkT7NYBI0sgOG3YrZ0KWgWyzqlC9o0JzW3qHgAA5Ck0ZfS0rQO6N0ilQiNhdFyh8 gOqGSGvu+G8Tqpylrnu6B3uv01O8jM0= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ENafStdA; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148049; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W2O1MoDfgqdouCpzRUrjzlsQ5SFNdm1Z+9DnK8kpTE8=; b=X3LPjB66WmU36Q1SPBQ1d8B+R3qdgjaIiYzI4U0Ppv2usJD1hCJgwaOsqhtmixg9IyqcQy HsjkcciislZzaKTBB49vxtmrE4MDzEF3oN2kdorj3oNFWF3rH92UT1Qt0YoSsWx5HYqv+W KR6BaZxUMFSqkUYJHTRIq6ZnzZvMByU= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 043CB180044 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ENafStdA; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 4gwtpiip79jsdm99c1hutustpsxucep7 X-HE-Tag: 1662148048-690636 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With all callers using folios, we can convert add_to_swap_cache() to take a folio and use it throughout. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 2 +- mm/swap.h | 4 ++-- mm/swap_state.c | 34 +++++++++++++++++----------------- 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 4113f1b9d4a8..ced76c229b96 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1406,7 +1406,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) if (list_empty(&info->swaplist)) list_add(&info->swaplist, &shmem_swaplist); - if (add_to_swap_cache(&folio->page, swap, + if (add_to_swap_cache(folio, swap, __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN, NULL) == 0) { spin_lock_irq(&info->lock); diff --git a/mm/swap.h b/mm/swap.h index 17936e068c1c..0e023765e110 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -34,7 +34,7 @@ extern struct address_space *swapper_spaces[]; void show_swap_cache_info(void); bool add_to_swap(struct folio *folio); void *get_shadow_from_swap_cache(swp_entry_t entry); -int add_to_swap_cache(struct page *page, swp_entry_t entry, +int add_to_swap_cache(struct folio *folio, swp_entry_t entry, gfp_t gfp, void **shadowp); void __delete_from_swap_cache(struct folio *folio, swp_entry_t entry, void *shadow); @@ -124,7 +124,7 @@ static inline void *get_shadow_from_swap_cache(swp_entry_t entry) return NULL; } -static inline int add_to_swap_cache(struct page *page, swp_entry_t entry, +static inline int add_to_swap_cache(struct folio *folio, swp_entry_t entry, gfp_t gfp_mask, void **shadowp) { return -1; diff --git a/mm/swap_state.c b/mm/swap_state.c index b1e181fc5268..ecf1accc2fb1 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -85,21 +85,21 @@ void *get_shadow_from_swap_cache(swp_entry_t entry) * add_to_swap_cache resembles filemap_add_folio on swapper_space, * but sets SwapCache flag and private instead of mapping and index. */ -int add_to_swap_cache(struct page *page, swp_entry_t entry, +int add_to_swap_cache(struct folio *folio, swp_entry_t entry, gfp_t gfp, void **shadowp) { struct address_space *address_space = swap_address_space(entry); pgoff_t idx = swp_offset(entry); - XA_STATE_ORDER(xas, &address_space->i_pages, idx, compound_order(page)); - unsigned long i, nr = thp_nr_pages(page); + XA_STATE_ORDER(xas, &address_space->i_pages, idx, folio_order(folio)); + unsigned long i, nr = folio_nr_pages(folio); void *old; - VM_BUG_ON_PAGE(!PageLocked(page), page); - VM_BUG_ON_PAGE(PageSwapCache(page), page); - VM_BUG_ON_PAGE(!PageSwapBacked(page), page); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(folio_test_swapcache(folio), folio); + VM_BUG_ON_FOLIO(!folio_test_swapbacked(folio), folio); - page_ref_add(page, nr); - SetPageSwapCache(page); + folio_ref_add(folio, nr); + folio_set_swapcache(folio); do { xas_lock_irq(&xas); @@ -107,19 +107,19 @@ int add_to_swap_cache(struct page *page, swp_entry_t entry, if (xas_error(&xas)) goto unlock; for (i = 0; i < nr; i++) { - VM_BUG_ON_PAGE(xas.xa_index != idx + i, page); + VM_BUG_ON_FOLIO(xas.xa_index != idx + i, folio); old = xas_load(&xas); if (xa_is_value(old)) { if (shadowp) *shadowp = old; } - set_page_private(page + i, entry.val + i); - xas_store(&xas, page); + set_page_private(folio_page(folio, i), entry.val + i); + xas_store(&xas, folio); xas_next(&xas); } address_space->nrpages += nr; - __mod_node_page_state(page_pgdat(page), NR_FILE_PAGES, nr); - __mod_lruvec_page_state(page, NR_SWAPCACHE, nr); + __node_stat_mod_folio(folio, NR_FILE_PAGES, nr); + __lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr); unlock: xas_unlock_irq(&xas); } while (xas_nomem(&xas, gfp)); @@ -127,8 +127,8 @@ int add_to_swap_cache(struct page *page, swp_entry_t entry, if (!xas_error(&xas)) return 0; - ClearPageSwapCache(page); - page_ref_sub(page, nr); + folio_clear_swapcache(folio); + folio_ref_sub(folio, nr); return xas_error(&xas); } @@ -194,7 +194,7 @@ bool add_to_swap(struct folio *folio) /* * Add it to the swap cache. */ - err = add_to_swap_cache(&folio->page, entry, + err = add_to_swap_cache(folio, entry, __GFP_HIGH|__GFP_NOMEMALLOC|__GFP_NOWARN, NULL); if (err) /* @@ -484,7 +484,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, goto fail_unlock; /* May fail (-ENOMEM) if XArray node allocation failed. */ - if (add_to_swap_cache(&folio->page, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow)) + if (add_to_swap_cache(folio, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow)) goto fail_unlock; mem_cgroup_swapin_uncharge_swap(entry); From patchwork Fri Sep 2 19:46:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 774B7C6FA83 for ; Fri, 2 Sep 2022 19:48:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F6588011C; Fri, 2 Sep 2022 15:47:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 97DFA80119; Fri, 2 Sep 2022 15:47:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81F1C8011C; Fri, 2 Sep 2022 15:47:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 72A0080119 for ; Fri, 2 Sep 2022 15:47:14 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 56929ABC2A for ; Fri, 2 Sep 2022 19:47:14 +0000 (UTC) X-FDA: 79868179188.01.C4966C2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id 069021C0046 for ; Fri, 2 Sep 2022 19:47:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=cjV56JpqqZJHrrCZTL+O7ueYNmp07ZGgA4+6twm12Es=; b=BGY7J/QVC8EYkIPskfhbhK3phw b/LO145th5zFJ2uMPmxcNcRDoo+NP7MyLhhULqlXEplbEbSRswlRoOB70UGTcQJjKv3KfRJ7OiYcb TVybpZ0uHWq0gHOhzUQYLFLtlunYMMjwM9EeVLgW74UHNHupBwbuWcoFG5nIg7JfCddy2dWTqwBoI GlgyhuAutUwic8Zr2AxJKSxhU9i6JSuVn2DwuZwRCASw3YHL1UUQT9WTvVt9qIFfLdvrZA4vM87sd J2mgAir+c0YeLei9Fh9g/VD+wx4DqyWnlNuSLRarykUK/yPLU4t9U45QxI8WjjOEckwfeUWgnvpeg vMa3Y7ew==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ic1-Tj; Fri, 02 Sep 2022 19:46:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 13/57] mm/swap: Convert put_swap_page() to put_swap_folio() Date: Fri, 2 Sep 2022 20:46:09 +0100 Message-Id: <20220902194653.1739778-14-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148034; a=rsa-sha256; cv=none; b=54NEaejP7vaXRV5+sp0zMBVqtuUwg+CR4aidhAysMZDCpLbRPS+k7D3Kr6GSYyq3WdDFrc qMGX7J7W98JTV+ZTVmhh6RDLeclTCYpbUXeKcZfcR4eDexRsTs86ZjnB35glwT0QWCLaqI NqMG0CoKOzHlM0QwfrE7YWRwlQ1tCMc= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="BGY7J/QV"; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148034; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cjV56JpqqZJHrrCZTL+O7ueYNmp07ZGgA4+6twm12Es=; b=ePxQ/EJKi6/z7Xg2Oua6IL9+zeZWmaUfwN+imPO7I6MHJnC7ZD8EKBeJIV1seb7/9uh/9h j5da4yGja80a1p+p43xs3WUSxm0vwZgmgmi7MOCKKRz7vG3YUV3SJnqoaTLGVX+gegkzRD 2NAb4g/YvZEzq6pZn0QUZPsfjRAJS7s= Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="BGY7J/QV"; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspam-User: X-Stat-Signature: pp77phqe8iewjkecutfpf6hmkh6iq9e5 X-Rspamd-Queue-Id: 069021C0046 X-Rspamd-Server: rspam11 X-HE-Tag: 1662148033-219497 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With all callers now using a folio, we can convert this function. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/swap.h | 4 ++-- mm/shmem.c | 2 +- mm/swap_slots.c | 2 +- mm/swap_state.c | 6 +++--- mm/swapfile.c | 4 ++-- mm/vmscan.c | 2 +- 6 files changed, 10 insertions(+), 10 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 4595cbc1cb02..f16c9af6bf32 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -487,7 +487,7 @@ static inline long get_nr_swap_pages(void) extern void si_swapinfo(struct sysinfo *); swp_entry_t folio_alloc_swap(struct folio *folio); bool folio_free_swap(struct folio *folio); -extern void put_swap_page(struct page *page, swp_entry_t entry); +void put_swap_folio(struct folio *folio, swp_entry_t entry); extern swp_entry_t get_swap_page_of_type(int); extern int get_swap_pages(int n, swp_entry_t swp_entries[], int entry_size); extern int add_swap_count_continuation(swp_entry_t, gfp_t); @@ -572,7 +572,7 @@ static inline void swap_free(swp_entry_t swp) { } -static inline void put_swap_page(struct page *page, swp_entry_t swp) +static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/shmem.c b/mm/shmem.c index ced76c229b96..56cabf9bb947 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1424,7 +1424,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) } mutex_unlock(&shmem_swaplist_mutex); - put_swap_page(&folio->page, swap); + put_swap_folio(folio, swap); redirty: folio_mark_dirty(folio); if (wbc->for_reclaim) diff --git a/mm/swap_slots.c b/mm/swap_slots.c index 10b94d64cc25..0bec1f705f8e 100644 --- a/mm/swap_slots.c +++ b/mm/swap_slots.c @@ -343,7 +343,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) get_swap_pages(1, &entry, 1); out: if (mem_cgroup_try_charge_swap(folio, entry)) { - put_swap_page(&folio->page, entry); + put_swap_folio(folio, entry); entry.val = 0; } return entry; diff --git a/mm/swap_state.c b/mm/swap_state.c index ecf1accc2fb1..ea354efd3735 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -218,7 +218,7 @@ bool add_to_swap(struct folio *folio) return true; fail: - put_swap_page(&folio->page, entry); + put_swap_folio(folio, entry); return false; } @@ -237,7 +237,7 @@ void delete_from_swap_cache(struct folio *folio) __delete_from_swap_cache(folio, entry, NULL); xa_unlock_irq(&address_space->i_pages); - put_swap_page(&folio->page, entry); + put_swap_folio(folio, entry); folio_ref_sub(folio, folio_nr_pages(folio)); } @@ -498,7 +498,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, return &folio->page; fail_unlock: - put_swap_page(&folio->page, entry); + put_swap_folio(folio, entry); folio_unlock(folio); folio_put(folio); return NULL; diff --git a/mm/swapfile.c b/mm/swapfile.c index 880871f4c6d4..186511a8ef4f 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1328,7 +1328,7 @@ void swap_free(swp_entry_t entry) /* * Called after dropping swapcache to decrease refcnt to swap entries. */ -void put_swap_page(struct page *page, swp_entry_t entry) +void put_swap_folio(struct folio *folio, swp_entry_t entry) { unsigned long offset = swp_offset(entry); unsigned long idx = offset / SWAPFILE_CLUSTER; @@ -1337,7 +1337,7 @@ void put_swap_page(struct page *page, swp_entry_t entry) unsigned char *map; unsigned int i, free_entries = 0; unsigned char val; - int size = swap_entry_size(thp_nr_pages(page)); + int size = swap_entry_size(folio_nr_pages(folio)); si = _swap_info_get(entry); if (!si) diff --git a/mm/vmscan.c b/mm/vmscan.c index d3e26712dbc1..ac7f6f77e28a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1339,7 +1339,7 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, shadow = workingset_eviction(folio, target_memcg); __delete_from_swap_cache(folio, swap, shadow); xa_unlock_irq(&mapping->i_pages); - put_swap_page(&folio->page, swap); + put_swap_folio(folio, swap); } else { void (*free_folio)(struct folio *); From patchwork Fri Sep 2 19:46:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64B48C6FA83 for ; Fri, 2 Sep 2022 19:47:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB975800DD; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 985FA800EE; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 69739800EB; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 37417800DD for ; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D8DAB40802 for ; Fri, 2 Sep 2022 19:46:59 +0000 (UTC) X-FDA: 79868178558.03.F3C347E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id 2F0AD14006B for ; Fri, 2 Sep 2022 19:46:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=m1WsqyDeTWbIGRO38ZCMMnf2bX8TTKK3IemcukWSpSE=; b=SA2dQjv70ewTrKLiwMtlZsrp8W W8vp1Kj2gdNB5czpHX00sQrnqkL7O6o7HCSR14lB1iZNgzYgtHmYwlIntbyK/VlCdlf/Uqliqeu5w +dX/g1NrAVXBevd5wFILcu8gZLyomQYsrklwsmFJymCeM7iI1rUb3HtpSlyqK5AT1eyY+U2oaReZh D13cu2mCX2RqPeu1bpztA8WpI60L7eMGB7f5CtVY6bnHgYbHud+2cOlUuBG5ZrKUZqshKnx7DWHTo uCo6N7LC2MH60YcFNWHxUUymWwuS6E+SYDTHKlsl5VGtB7La2MSLZob7TsWiTMiw+hetJdnb9jngf Q+STxWEg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd0-007Ic3-Vt; Fri, 02 Sep 2022 19:46:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 14/57] mm: Convert do_swap_page() to use a folio Date: Fri, 2 Sep 2022 20:46:10 +0100 Message-Id: <20220902194653.1739778-15-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148019; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m1WsqyDeTWbIGRO38ZCMMnf2bX8TTKK3IemcukWSpSE=; b=xDlRi5Cpu+WqZItOkydHJiGBehImk5v6NEKaUh7S1LnF8sju3a1qC1C8KZnNyeqr/kx3gq D7j5pjHRLgH+gg6hdOE+nugf4rvYk2TAzCZzp72Vye0jmjx6DDLSzJNt/7/+i6xizd9a2U o2TvOGtnRBe+iI3qMkOzN67hTsESqGA= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SA2dQjv7; dmarc=none; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148019; a=rsa-sha256; cv=none; b=WUbzUtcC1P5Br4EIZVBqDdTfK+Uof/mOCkFb+M5/fkXoo8zGZwAfuAYPGMpQgvhBD37qj5 JZ8XX8h2ykvrMtHX7MdXzPwH1b+wKn2QF2G+xQ/o0zWaQPlE5c8nymEpg7uSP728x5YcbC AkiKxtODnC+PLifNRNtfDQ1I1A7+yPE= X-Rspamd-Queue-Id: 2F0AD14006B X-Rspam-User: Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SA2dQjv7; dmarc=none; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Stat-Signature: scdjri7p94sdnf3u86y9h8poywf7ipxy X-Rspamd-Server: rspam05 X-HE-Tag: 1662148019-542697 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes quite a lot of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 57 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 24 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 4ba73f5aa8bb..f172b148e29b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3718,6 +3718,7 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf) vm_fault_t do_swap_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; + struct folio *folio; struct page *page = NULL, *swapcache; struct swap_info_struct *si = NULL; rmap_t rmap_flags = RMAP_NONE; @@ -3762,19 +3763,23 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page = lookup_swap_cache(entry, vma, vmf->address); swapcache = page; + if (page) + folio = page_folio(page); if (!page) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { /* skip swapcache */ - page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, - vmf->address); - if (page) { - __SetPageLocked(page); - __SetPageSwapBacked(page); + folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, + vma, vmf->address, false); + page = &folio->page; + if (folio) { + __folio_set_locked(folio); + __folio_set_swapbacked(folio); if (mem_cgroup_swapin_charge_page(page, - vma->vm_mm, GFP_KERNEL, entry)) { + vma->vm_mm, GFP_KERNEL, + entry)) { ret = VM_FAULT_OOM; goto out_page; } @@ -3782,20 +3787,21 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) shadow = get_shadow_from_swap_cache(entry); if (shadow) - workingset_refault(page_folio(page), - shadow); + workingset_refault(folio, shadow); - lru_cache_add(page); + folio_add_lru(folio); /* To provide entry to swap_readpage() */ - set_page_private(page, entry.val); + folio_set_swap_entry(folio, entry); swap_readpage(page, true, NULL); - set_page_private(page, 0); + folio->private = NULL; } } else { page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); swapcache = page; + if (page) + folio = page_folio(page); } if (!page) { @@ -3838,7 +3844,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * swapcache, we need to check that the page's swap has not * changed. */ - if (unlikely(!PageSwapCache(page) || + if (unlikely(!folio_test_swapcache(folio) || page_private(page) != entry.val)) goto out_page; @@ -3853,6 +3859,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page = swapcache; goto out_page; } + folio = page_folio(page); /* * If we want to map a page that's in the swapcache writable, we @@ -3861,7 +3868,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * pagevecs if required. */ if ((vmf->flags & FAULT_FLAG_WRITE) && page == swapcache && - !PageKsm(page) && !PageLRU(page)) + !folio_test_ksm(folio) && !folio_test_lru(folio)) lru_add_drain(); } @@ -3875,7 +3882,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) goto out_nomap; - if (unlikely(!PageUptodate(page))) { + if (unlikely(!folio_test_uptodate(folio))) { ret = VM_FAULT_SIGBUS; goto out_nomap; } @@ -3888,14 +3895,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * check after taking the PT lock and making sure that nobody * concurrently faulted in this page and set PG_anon_exclusive. */ - BUG_ON(!PageAnon(page) && PageMappedToDisk(page)); - BUG_ON(PageAnon(page) && PageAnonExclusive(page)); + BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio)); + BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page)); /* * Check under PT lock (to protect against concurrent fork() sharing * the swap entry concurrently) for certainly exclusive pages. */ - if (!PageKsm(page)) { + if (!folio_test_ksm(folio)) { /* * Note that pte_swp_exclusive() == false for architectures * without __HAVE_ARCH_PTE_SWP_EXCLUSIVE. @@ -3907,7 +3914,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * swapcache -> certainly exclusive. */ exclusive = true; - } else if (exclusive && PageWriteback(page) && + } else if (exclusive && folio_test_writeback(folio) && data_race(si->flags & SWP_STABLE_WRITES)) { /* * This is tricky: not all swap backends support @@ -3950,7 +3957,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exposing them to the swapcache or because the swap entry indicates * exclusivity. */ - if (!PageKsm(page) && (exclusive || page_count(page) == 1)) { + if (!folio_test_ksm(folio) && + (exclusive || folio_ref_count(folio) == 1)) { if (vmf->flags & FAULT_FLAG_WRITE) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; @@ -3970,16 +3978,17 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) /* ksm created a completely new copy */ if (unlikely(page != swapcache && swapcache)) { page_add_new_anon_rmap(page, vma, vmf->address); - lru_cache_add_inactive_or_unevictable(page, vma); + folio_add_lru_vma(folio, vma); } else { page_add_anon_rmap(page, vma, vmf->address, rmap_flags); } - VM_BUG_ON(!PageAnon(page) || (pte_write(pte) && !PageAnonExclusive(page))); + VM_BUG_ON(!folio_test_anon(folio) || + (pte_write(pte) && !PageAnonExclusive(page))); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); - unlock_page(page); + folio_unlock(folio); if (page != swapcache && swapcache) { /* * Hold the lock to avoid the swap entry to be reused @@ -4011,9 +4020,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) out_nomap: pte_unmap_unlock(vmf->pte, vmf->ptl); out_page: - unlock_page(page); + folio_unlock(folio); out_release: - put_page(page); + folio_put(folio); if (page != swapcache && swapcache) { unlock_page(swapcache); put_page(swapcache); From patchwork Fri Sep 2 19:46:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BFD2C6FA82 for ; Fri, 2 Sep 2022 19:47:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56B53800EA; Fri, 2 Sep 2022 15:47:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EFBBF800EB; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDA11800F1; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 88BB8800F0 for ; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 530BF407BA for ; Fri, 2 Sep 2022 19:47:00 +0000 (UTC) X-FDA: 79868178600.05.46533B9 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id DF1D920053 for ; Fri, 2 Sep 2022 19:46:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=wGo7P/d9Kjs6SsT3lbiCaIEX7mjwZhGrACXpSAcHWxA=; b=Ylgh8XpT47snrXf7jnSx7VHfmB 9HE90Nm9GgwRYpM2JpG8mVDLegqUp/ZbJmr55Fb9ldJHAKwf1kio+yaCR2cQTxArhLE9yZUQ/Uj97 MJw/1YUHF5Z2vwvkL6LAKvHGGo45uKKHkcXjqe+5XdRQe2JoYk9jQDyHyRD1AYxz+AWMC3bmxcdVw i0XZdT1S80C8JORNh3efu8Ye2CiHT+uuX1x32FSpQrq7FezK0R/N4YDq4OGO6FtekHyJZc+PuedPD 00ubfqgO5wWwuxrfdUqD7x0gEQ6ICm6Fc6fGOS8DW2a5ej5RVnxGorG7hVn3g2pg1Ec3+mJrGLpXF oIpuWS0w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd1-007IcK-3k; Fri, 02 Sep 2022 19:46:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 15/57] mm: Convert do_swap_page()'s swapcache variable to a folio Date: Fri, 2 Sep 2022 20:46:11 +0100 Message-Id: <20220902194653.1739778-16-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148020; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wGo7P/d9Kjs6SsT3lbiCaIEX7mjwZhGrACXpSAcHWxA=; b=kWnVjqe3fANw2FIQ7sLJ3P9gpB0QTqEoyHuZMv1cUlI/BA1s+HKR3AogmR61L2FmwypHyv iIN8jB0k1a9GdnoEhFV1X0asgXxCwVPnhIByEceeLisdIuE5dBPLjdyJqNfFjgs3wdCzFl GkI+KeeZbW7JpoHdA/L0h3z0G2p4HrU= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ylgh8XpT; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148020; a=rsa-sha256; cv=none; b=gFiujHQl5nguFGk65D9X9uLhCL9XsmeO4YJxPbUUuuO7qHkpLs83I9aESmKpI/gdBtV2aF sQzUtvAbfphOLcfEqV522iELAxcTTu3VL9MgYFZrR2srDifGBB2lAKR7dWKielY0YBjnBc AFsDleCob+vkYfmOOp4Gure5aVGEV74= X-Rspamd-Queue-Id: DF1D920053 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ylgh8XpT; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: zrzsuxarqg1awazprtqdw1dwyyqbpjsh X-HE-Tag: 1662148019-403541 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The 'swapcache' variable is used to track whether the page is from the swapcache or not. It can do this equally well by being the folio of the page rather than the page itself, and this saves a number of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index f172b148e29b..0184fe0ae736 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3718,8 +3718,8 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf) vm_fault_t do_swap_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - struct folio *folio; - struct page *page = NULL, *swapcache; + struct folio *swapcache, *folio = NULL; + struct page *page; struct swap_info_struct *si = NULL; rmap_t rmap_flags = RMAP_NONE; bool exclusive = false; @@ -3762,11 +3762,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out; page = lookup_swap_cache(entry, vma, vmf->address); - swapcache = page; if (page) folio = page_folio(page); + swapcache = folio; - if (!page) { + if (!folio) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { /* skip swapcache */ @@ -3799,12 +3799,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } else { page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); - swapcache = page; if (page) folio = page_folio(page); + swapcache = folio; } - if (!page) { + if (!folio) { /* * Back out if somebody else faulted in this pte * while we released the pte lock. @@ -3856,7 +3856,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page = ksm_might_need_to_copy(page, vma, vmf->address); if (unlikely(!page)) { ret = VM_FAULT_OOM; - page = swapcache; goto out_page; } folio = page_folio(page); @@ -3867,7 +3866,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * owner. Try removing the extra reference from the local LRU * pagevecs if required. */ - if ((vmf->flags & FAULT_FLAG_WRITE) && page == swapcache && + if ((vmf->flags & FAULT_FLAG_WRITE) && folio == swapcache && !folio_test_ksm(folio) && !folio_test_lru(folio)) lru_add_drain(); } @@ -3908,7 +3907,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * without __HAVE_ARCH_PTE_SWP_EXCLUSIVE. */ exclusive = pte_swp_exclusive(vmf->orig_pte); - if (page != swapcache) { + if (folio != swapcache) { /* * We have a fresh page that is not exposed to the * swapcache -> certainly exclusive. @@ -3976,7 +3975,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->orig_pte = pte; /* ksm created a completely new copy */ - if (unlikely(page != swapcache && swapcache)) { + if (unlikely(folio != swapcache && swapcache)) { page_add_new_anon_rmap(page, vma, vmf->address); folio_add_lru_vma(folio, vma); } else { @@ -3989,7 +3988,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); folio_unlock(folio); - if (page != swapcache && swapcache) { + if (folio != swapcache && swapcache) { /* * Hold the lock to avoid the swap entry to be reused * until we take the PT lock for the pte_same() check @@ -3998,8 +3997,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * so that the swap count won't change under a * parallel locked swapcache. */ - unlock_page(swapcache); - put_page(swapcache); + folio_unlock(swapcache); + folio_put(swapcache); } if (vmf->flags & FAULT_FLAG_WRITE) { @@ -4023,9 +4022,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) folio_unlock(folio); out_release: folio_put(folio); - if (page != swapcache && swapcache) { - unlock_page(swapcache); - put_page(swapcache); + if (folio != swapcache && swapcache) { + folio_unlock(swapcache); + folio_put(swapcache); } if (si) put_swap_device(si); From patchwork Fri Sep 2 19:46:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 508A9ECAAD5 for ; Fri, 2 Sep 2022 19:47:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D533B800F0; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BC7D0800EF; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 782A0800EF; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 44C93800EA for ; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 25112140EE2 for ; Fri, 2 Sep 2022 19:47:00 +0000 (UTC) X-FDA: 79868178600.13.10556B6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id 6B13240099 for ; Fri, 2 Sep 2022 19:46:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=q+sLIiEyh4MFhP0h+T3uruqfc7/VTFU2JnuaGsEdOq0=; b=uvXGozCuH7KxJ6JJtGwJinyA1c I99PK7hsTD3t1NSCibQ8Qx5ZqBAth8Ind4mUORv5BJhwOebyeXAuzZ7EOu50K114pc35Hr9FeZ6Ot AuBlY4VGhYm5BsCQ4LmWi3hzyZYzqlGyTK0HWarBCaL8jcPS3a/Qy8QaNsgIQqdFSGM+3Z97HW1I7 olAiYik/ki1YxylJPlVybhRJpWLdc3E6MvRxrl8LT3BEHqHjEFpsMZRR87D9c7fWKAyKvOtVdiz5Z 6Chp4sAY5dkrGJ8f6V3GYubUq/OSu9+0BMqV2K+PGGe8RjOMa955HUfZ1H4JiXW8ksFDsxKPazv6k k680NpFQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd1-007IcQ-8K; Fri, 02 Sep 2022 19:46:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 16/57] memcg: Convert mem_cgroup_swapin_charge_page() to mem_cgroup_swapin_charge_folio() Date: Fri, 2 Sep 2022 20:46:12 +0100 Message-Id: <20220902194653.1739778-17-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148019; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q+sLIiEyh4MFhP0h+T3uruqfc7/VTFU2JnuaGsEdOq0=; b=GeOPK9zDLzI/XT3xXWp7jvu3nK7xHEZ15GqUbhFbY6OdoHyNj6rpbuzXl5j1hPecDfalCM gOM6Q2UMUJXB9cXZmlIX9DwNTCalttAJH0FooBFHHlXOePjG8JFPwKdQT9IrU3kqPr1oZ4 DJCDr6l+OXaTJWKDAhfOCL8fSj4zqe8= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=uvXGozCu; dmarc=none; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148019; a=rsa-sha256; cv=none; b=SWq13aQb9J3SNGmnoyMOAnKSGs100ZpzEhy9oJSlMCLT9DLfMl/v4VuDZokptzQiDOEPmp 06p1D+dZqpELB9WkQkj6Z4RaFYXBxo0YtC2H8LVo/6RfOFDeOreY6G7LUq23xYnkrfgbQg MdI8C/FU5cmFkM1kxTXEYGt35/26aNk= X-Rspamd-Queue-Id: 6B13240099 X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=uvXGozCu; dmarc=none; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Stat-Signature: 6fkt6jun4oed6nm76d58ti3a3fxpynhn X-Rspamd-Server: rspam05 X-HE-Tag: 1662148019-254292 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All callers now have a folio, so pass it in here and remove an unnecessary call to page_folio(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 13 ++++++------- mm/memory.c | 2 +- mm/swap_state.c | 2 +- 4 files changed, 10 insertions(+), 11 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 6257867fbf95..930c450cf024 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -689,7 +689,7 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, return __mem_cgroup_charge(folio, mm, gfp); } -int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm, +int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, gfp_t gfp, swp_entry_t entry); void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry); @@ -1238,7 +1238,7 @@ static inline int mem_cgroup_charge(struct folio *folio, return 0; } -static inline int mem_cgroup_swapin_charge_page(struct page *page, +static inline int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, gfp_t gfp, swp_entry_t entry) { return 0; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b69979c9ced5..f65833efe90d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6813,21 +6813,20 @@ int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp) } /** - * mem_cgroup_swapin_charge_page - charge a newly allocated page for swapin - * @page: page to charge + * mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin. + * @folio: folio to charge. * @mm: mm context of the victim * @gfp: reclaim mode - * @entry: swap entry for which the page is allocated + * @entry: swap entry for which the folio is allocated * - * This function charges a page allocated for swapin. Please call this before - * adding the page to the swapcache. + * This function charges a folio allocated for swapin. Please call this before + * adding the folio to the swapcache. * * Returns 0 on success. Otherwise, an error code is returned. */ -int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm, +int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, gfp_t gfp, swp_entry_t entry) { - struct folio *folio = page_folio(page); struct mem_cgroup *memcg; unsigned short id; int ret; diff --git a/mm/memory.c b/mm/memory.c index 0184fe0ae736..8fefb24567da 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3777,7 +3777,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) __folio_set_locked(folio); __folio_set_swapbacked(folio); - if (mem_cgroup_swapin_charge_page(page, + if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, GFP_KERNEL, entry)) { ret = VM_FAULT_OOM; diff --git a/mm/swap_state.c b/mm/swap_state.c index ea354efd3735..a7e0438902dd 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -480,7 +480,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, __folio_set_locked(folio); __folio_set_swapbacked(folio); - if (mem_cgroup_swapin_charge_page(&folio->page, NULL, gfp_mask, entry)) + if (mem_cgroup_swapin_charge_folio(folio, NULL, gfp_mask, entry)) goto fail_unlock; /* May fail (-ENOMEM) if XArray node allocation failed. */ From patchwork Fri Sep 2 19:46:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C09EFC6FA82 for ; Fri, 2 Sep 2022 19:47:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47207800EC; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DCB8800ED; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2EF4E800EC; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 08D9D800DD for ; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D8457140866 for ; Fri, 2 Sep 2022 19:46:59 +0000 (UTC) X-FDA: 79868178558.04.FCD8BD7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id 89C5818004E for ; Fri, 2 Sep 2022 19:46:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=LMCBKoyBWEa1JTiNxACv5KWjlvfrg4Sp3Qe2NN/LXpw=; b=Y6SBrNYV6mTK4gi22nVmF4Kjzr QWV2To3hJJWIYRvny9MsnLKuvwXlf0xcDyvNuQP60LWFfLyEZpN7suFB5QFPainemOmGxOlianCWt qhHEN9ZsBLNoDxN7B49+KkQKKonv8ZQ9lU89GQdEaQ0Q8vx26YQ2qavxOIoq5FQN8bUeIfTOhcAte qUqx1MNu98EgCbng2k/mTCdZnyi5l6q0v1eXFcl8Pj+R59DkYbwHapuny2x+WHd7eOKr4kaWQ+ZzG sDqKO7x75o+sALe92T9ZWxlnkgjOZCziC2g6CyhR9o0HGfpmSBJWb0mmjas6jhNVb5UWdM9kBce5/ Fk59e/dg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd1-007IcW-C8; Fri, 02 Sep 2022 19:46:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 17/57] shmem: Convert shmem_mfill_atomic_pte() to use a folio Date: Fri, 2 Sep 2022 20:46:13 +0100 Message-Id: <20220902194653.1739778-18-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148019; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LMCBKoyBWEa1JTiNxACv5KWjlvfrg4Sp3Qe2NN/LXpw=; b=A2Mk7AREMBCwPzanA0VG6Bk2IvgqTickX333MgUflEkOBtupEBF4g813GkyDVyQ7bTOKUP pJ9MnE9Y2cgVCqUTWPGbg2yn6V2dqxuAEbGNIEgS+9SAqw2c8tM/xNs1glyl/v1YdzOBp1 yNuns1fGweo8JlChdNmgny9svXxf1hg= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Y6SBrNYV; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148019; a=rsa-sha256; cv=none; b=INdicHriS37sHGc2hDH28MCnRxMujBm6sB+xT7lWW2cAHEHpQXcFL/ZtlHEvohowIxy/aY zHrx7L5AkeQ07Sbtw5ehqh1GJlUuJlUJLQPdE60VCFFFzjwfn7u8YZVequt9EQnSPNTc5p hXPHC9+pV3KouczA92R6G6HsrTPu9cM= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 89C5818004E Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Y6SBrNYV; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 1hss4uqoqgpi9mt3s1yajzobs9tgbs5r X-HE-Tag: 1662148019-908124 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Assert that this is a single-page folio as there are several assumptions in here that it's exactly PAGE_SIZE bytes large. Saves several calls to compound_head() and removes the last caller of shmem_alloc_page(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 45 +++++++++++++++++++-------------------------- 1 file changed, 19 insertions(+), 26 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 56cabf9bb947..8754e2b4800a 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2374,12 +2374,6 @@ static struct inode *shmem_get_inode(struct super_block *sb, struct inode *dir, } #ifdef CONFIG_USERFAULTFD -static struct page *shmem_alloc_page(gfp_t gfp, - struct shmem_inode_info *info, pgoff_t index) -{ - return &shmem_alloc_folio(gfp, info, index)->page; -} - int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, struct vm_area_struct *dst_vma, @@ -2395,7 +2389,6 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, pgoff_t pgoff = linear_page_index(dst_vma, dst_addr); void *page_kaddr; struct folio *folio; - struct page *page; int ret; pgoff_t max_off; @@ -2414,53 +2407,53 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, if (!*pagep) { ret = -ENOMEM; - page = shmem_alloc_page(gfp, info, pgoff); - if (!page) + folio = shmem_alloc_folio(gfp, info, pgoff); + if (!folio) goto out_unacct_blocks; if (!zeropage) { /* COPY */ - page_kaddr = kmap_atomic(page); + page_kaddr = kmap_local_folio(folio, 0); ret = copy_from_user(page_kaddr, (const void __user *)src_addr, PAGE_SIZE); - kunmap_atomic(page_kaddr); + kunmap_local(page_kaddr); /* fallback to copy_from_user outside mmap_lock */ if (unlikely(ret)) { - *pagep = page; + *pagep = &folio->page; ret = -ENOENT; /* don't free the page */ goto out_unacct_blocks; } - flush_dcache_page(page); + flush_dcache_folio(folio); } else { /* ZEROPAGE */ - clear_user_highpage(page, dst_addr); + clear_user_highpage(&folio->page, dst_addr); } } else { - page = *pagep; + folio = page_folio(*pagep); + VM_BUG_ON_FOLIO(folio_test_large(folio), folio); *pagep = NULL; } - VM_BUG_ON(PageLocked(page)); - VM_BUG_ON(PageSwapBacked(page)); - __SetPageLocked(page); - __SetPageSwapBacked(page); - __SetPageUptodate(page); + VM_BUG_ON(folio_test_locked(folio)); + VM_BUG_ON(folio_test_swapbacked(folio)); + __folio_set_locked(folio); + __folio_set_swapbacked(folio); + __folio_mark_uptodate(folio); ret = -EFAULT; max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); if (unlikely(pgoff >= max_off)) goto out_release; - folio = page_folio(page); ret = shmem_add_to_page_cache(folio, mapping, pgoff, NULL, gfp & GFP_RECLAIM_MASK, dst_mm); if (ret) goto out_release; ret = mfill_atomic_install_pte(dst_mm, dst_pmd, dst_vma, dst_addr, - page, true, wp_copy); + &folio->page, true, wp_copy); if (ret) goto out_delete_from_cache; @@ -2470,13 +2463,13 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, shmem_recalc_inode(inode); spin_unlock_irq(&info->lock); - unlock_page(page); + folio_unlock(folio); return 0; out_delete_from_cache: - delete_from_page_cache(page); + filemap_remove_folio(folio); out_release: - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); out_unacct_blocks: shmem_inode_unacct_blocks(inode, 1); return ret; From patchwork Fri Sep 2 19:46:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67C87ECAAD5 for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82231800ED; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75EE4800DD; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 510C8800EF; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2556B800EB for ; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EEA58C0529 for ; Fri, 2 Sep 2022 19:46:59 +0000 (UTC) X-FDA: 79868178558.25.507BBBE Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id A28CF4004F for ; Fri, 2 Sep 2022 19:46:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=thf7apzdSr3NOuHUHzgvwosLzIF/+7M4T316bLt6cLU=; b=OKLNtOnuI9kUo/Di5ldun07t07 Pnup/VwBgMkv6bB7tQ1Vzi1SF5jZK2fmr+zKvACX8OwO3JCMkA8Z1bpcCI0mJz8jprdue3iRRAIai dB2U3GdI+yQ29uL3KomHJ6lloOSiQ7TnkOMr6p/qGOoZxr96WQq/LGEuUR36JLXKZU4uyonQCnbKW TV9svXDXdorxuUQVx+F4p+bpLGFh/sKSYk9fD68mKOKdi5iFkBwroE+kjJqzZdu6uVkJ1xAxSTShW +RsXCDczQ2bWNA+3ozicn09AgdTRMBOuvdNUw4mf4kKP8ndDWJD/tpqoHs0F6RZUMptiuLkQv7/d7 M+WLXlFA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd1-007Icc-G2; Fri, 02 Sep 2022 19:46:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 18/57] shmem: Convert shmem_replace_page() to shmem_replace_folio() Date: Fri, 2 Sep 2022 20:46:14 +0100 Message-Id: <20220902194653.1739778-19-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148019; a=rsa-sha256; cv=none; b=czDisaXRFX9BgLlw6c19Fq99hR26kVtvMtfZuguoEUeND+GB5mz+EDVwHVYrNo3jDuTZLq v+0FTgvbWS1el407vo0ed2j4jr1skUNDYcnvKuzf4VQ5YrBm7q/c3SdfMIrG1O157lf8wU Snrqukm209A15Zy62l6qZWrKGWcN54w= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=OKLNtOnu; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148019; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=thf7apzdSr3NOuHUHzgvwosLzIF/+7M4T316bLt6cLU=; b=ltr/LweJ+tWDuVtUf2hQDR9fyMPO3wCytvX4NlsNpph5PjQ32xjzPICIiqFVzSVHrzNIf2 UNmkjGpvNkHBkbOOu5HN7zaYZ7LtVd8fWeIyU5eRaXx8dIbxMAi+wMepIOYcb/PDfHcpIy FUJ47Vu8FkjJf+h8u17D6O2c82tAO6c= X-Stat-Signature: qyp1ur5sw5wc151czdwtzuqwmphmgj8x X-Rspamd-Queue-Id: A28CF4004F Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=OKLNtOnu; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1662148019-207797 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The caller has a folio, so convert the calling convention and rename the function. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 8754e2b4800a..2bb6f5cfdc11 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1608,7 +1608,7 @@ static bool shmem_should_replace_folio(struct folio *folio, gfp_t gfp) return folio_zonenum(folio) > gfp_zone(gfp); } -static int shmem_replace_page(struct page **pagep, gfp_t gfp, +static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { struct folio *old, *new; @@ -1617,7 +1617,7 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp, pgoff_t swap_index; int error; - old = page_folio(*pagep); + old = *foliop; entry = folio_swap_entry(old); swap_index = swp_offset(entry); swap_mapping = swap_address_space(entry); @@ -1666,7 +1666,7 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp, old = new; } else { folio_add_lru(new); - *pagep = &new->page; + *foliop = new; } folio_clear_swapcache(old); @@ -1772,8 +1772,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, arch_swap_restore(swap, folio); if (shmem_should_replace_folio(folio, gfp)) { - error = shmem_replace_page(&page, gfp, info, index); - folio = page_folio(page); + error = shmem_replace_folio(&folio, gfp, info, index); if (error) goto failed; } From patchwork Fri Sep 2 19:46:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5C0BC6FA82 for ; Fri, 2 Sep 2022 19:47:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10D8F800EF; Fri, 2 Sep 2022 15:47:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D4E05800EE; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81E19800EA; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4B198800EE for ; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1CA53140866 for ; Fri, 2 Sep 2022 19:47:00 +0000 (UTC) X-FDA: 79868178600.17.45D482E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id CB3811A0048 for ; Fri, 2 Sep 2022 19:46:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=LV3i0kkJqsfpyICvVTbaOoAHHMn/zS4yYuJXoR6jl3w=; b=tM3HZb4r6eIyw+PSHEcSFNwffQ EBamlOXJODnIV9Lx7JAvX59JB9J+Ggot8hGnDYMOwZa07pj7MmHWlZ1s7UnJAsVhCCFSAccfvBvu+ faaLNYm15ryjua40sFjUQD4kyHVzTQyl0QAFhX0UfotjLLRvw0SNzSF54t66GtWy+u926VON1ZxUz K5GTs8kMQZs/kmYHC+285wB9TTFRnxgpipp5Wt7MFvAdKkBUAKZURdCFPhSKMC1hxlOdGoWRPWzWN K1AvJb2/ALxJkwVG+F6aeypfrjsCMtPwynUAZZ0pOp+/1eIEHyY+01oHJyRjQNnwCRYSti94STsWf D7a171yA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd1-007Ici-KG; Fri, 02 Sep 2022 19:46:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 19/57] swap: Add swap_cache_get_folio() Date: Fri, 2 Sep 2022 20:46:15 +0100 Message-Id: <20220902194653.1739778-20-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148019; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LV3i0kkJqsfpyICvVTbaOoAHHMn/zS4yYuJXoR6jl3w=; b=rhgJcuj31uIrWQ8r45V4eqAu31w20/HnVm0cDe0lN3l97zGCfW2rNS4YVtR0cNXoiHxZ0h /LKWm+DImXDvKx9wXUfwXd7dH7gDP1zPvHsruZZYpwyBU7EfqK5x9p9LyQqawoHabK+NYE TOHHkR8iZM1/G4hU3TwI2gIoJKfDv/o= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=tM3HZb4r; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148019; a=rsa-sha256; cv=none; b=Ha4jZeLs8pW+MURfdNAoeVG0Dc1itylUUknp3TH4Do2bgidvuDtqJ6wXL6/rq9U6t25ebk oCSgyxsQcOXT4IUguae3PGmuOYTWNTEdM33hzAj6O2Cv2O773lmCTwXFfBiicz69nz29Ie b+Ci2kPMyzrEM2Qltbacrd+xHr8x7jM= X-Rspamd-Queue-Id: CB3811A0048 X-Rspam-User: Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=tM3HZb4r; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Stat-Signature: 3fxtnspfqgcozypg6gjzbhitj6mzmkqw X-Rspamd-Server: rspam05 X-HE-Tag: 1662148019-444650 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert lookup_swap_cache() into swap_cache_get_folio() and add a lookup_swap_cache() wrapper around it. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap.h | 2 ++ mm/swap_state.c | 32 +++++++++++++++++++++----------- 2 files changed, 23 insertions(+), 11 deletions(-) diff --git a/mm/swap.h b/mm/swap.h index 0e023765e110..f70ff34dab82 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -41,6 +41,8 @@ void __delete_from_swap_cache(struct folio *folio, void delete_from_swap_cache(struct folio *folio); void clear_shadow_from_swap_cache(int type, unsigned long begin, unsigned long end); +struct folio *swap_cache_get_folio(swp_entry_t entry, + struct vm_area_struct *vma, unsigned long addr); struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr); diff --git a/mm/swap_state.c b/mm/swap_state.c index a7e0438902dd..b96bf4ec8b5b 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -317,24 +317,24 @@ static inline bool swap_use_vma_readahead(void) } /* - * Lookup a swap entry in the swap cache. A found page will be returned + * Lookup a swap entry in the swap cache. A found folio will be returned * unlocked and with its refcount incremented - we rely on the kernel - * lock getting page table operations atomic even if we drop the page + * lock getting page table operations atomic even if we drop the folio * lock before returning. */ -struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, - unsigned long addr) +struct folio *swap_cache_get_folio(swp_entry_t entry, + struct vm_area_struct *vma, unsigned long addr) { - struct page *page; + struct folio *folio; struct swap_info_struct *si; si = get_swap_device(entry); if (!si) return NULL; - page = find_get_page(swap_address_space(entry), swp_offset(entry)); + folio = filemap_get_folio(swap_address_space(entry), swp_offset(entry)); put_swap_device(si); - if (page) { + if (folio) { bool vma_ra = swap_use_vma_readahead(); bool readahead; @@ -342,10 +342,10 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, * At the moment, we don't support PG_readahead for anon THP * so let's bail out rather than confusing the readahead stat. */ - if (unlikely(PageTransCompound(page))) - return page; + if (unlikely(folio_test_large(folio))) + return folio; - readahead = TestClearPageReadahead(page); + readahead = folio_test_clear_readahead(folio); if (vma && vma_ra) { unsigned long ra_val; int win, hits; @@ -366,7 +366,17 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, } } - return page; + return folio; +} + +struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, + unsigned long addr) +{ + struct folio *folio = swap_cache_get_folio(entry, vma, addr); + + if (!folio) + return NULL; + return folio_file_page(folio, swp_offset(entry)); } /** From patchwork Fri Sep 2 19:46:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5541BC6FA83 for ; Fri, 2 Sep 2022 19:47:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E1EA80110; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 97289800FF; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E38B4800FF; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2F0EB80109 for ; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 11F4A140EE2 for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) X-FDA: 79868178768.24.8CF2406 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id AE98D140064 for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=L+G3lzPsb0oPAWAGOjNGo3IbzotK51wOth4YKCXiMyw=; b=j9rAl9Og+x3QoCKAlYTpCJS/Nc QavwzAwfbF5Wl/8JOzq9DoVHxA72cen4cu02q4vUM6toq1pON6Y63IAYgZu+tP1CPpe8O/le9mJQi e4MjQ4nMBrOfEqo5tdbN9dxPxNnvyNIoEAxFApdnjdWa3sr21SK6fj82JoDniUlkcIRTUGxPDe2OC zMUAs9FG3GIEN+G+i6ciPI53lQ4D05XAFHayM5n3/UUrYTJcQ81jNzRYLuDjLjnzmPDeVyWyeNWzD dMW/jrClqDhbLpVTjpmj9JrVOirM5oJ8NuvpLEvaR5RlLRn+509lSgQqH19BWNo48bhJIpEG8yJLk Va7rpyYA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd1-007Icq-ON; Fri, 02 Sep 2022 19:46:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 20/57] shmem: Eliminate struct page from shmem_swapin_folio() Date: Fri, 2 Sep 2022 20:46:16 +0100 Message-Id: <20220902194653.1739778-21-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148023; a=rsa-sha256; cv=none; b=OsrK/L1JfHkleeKs5EwiLxNM/JUa4x43or+yk9/+qFb8JEMIPAK5VSulvv4tHUnbMMzmfM /sKfufJ4JG3ZXll+prGUJDdUh7ZFQ5HNnsmu/+IWe9Q5WzOPLQaJgei5v2/xiAygXKHGJH r4R2uU8GuZcPQRG1D6Piz15OJ990LKw= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=j9rAl9Og; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L+G3lzPsb0oPAWAGOjNGo3IbzotK51wOth4YKCXiMyw=; b=HURh5KAGfeA0PxTq6kEK/FwbIOYrrzJIgmfzDBDSR2O4Po2nQ9TcP0Qny9BPFS2LKii6KW RyvaBpKoZZ8uLPj44cDRe5L0HRkod+MLGb1rDzWejtI5qmyGkmsfIpIL8XS9UX4gg7IU2z Lh0CMZfZQpPuVABaBHXwQOI9sGEgHTE= X-Stat-Signature: 7tuunac11sk97th3ig9hhnyhgdmmggrs X-Rspamd-Queue-Id: AE98D140064 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=j9rAl9Og; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1662148023-647427 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert shmem_swapin() to return a folio and use swap_cache_get_folio(), removing all uses of struct page in this function. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 2bb6f5cfdc11..b685acd9f149 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1486,7 +1486,7 @@ static void shmem_pseudo_vma_destroy(struct vm_area_struct *vma) mpol_cond_put(vma->vm_policy); } -static struct page *shmem_swapin(swp_entry_t swap, gfp_t gfp, +static struct folio *shmem_swapin(swp_entry_t swap, gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { struct vm_area_struct pvma; @@ -1499,7 +1499,9 @@ static struct page *shmem_swapin(swp_entry_t swap, gfp_t gfp, page = swap_cluster_readahead(swap, gfp, &vmf); shmem_pseudo_vma_destroy(&pvma); - return page; + if (!page) + return NULL; + return page_folio(page); } /* @@ -1721,7 +1723,6 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, struct address_space *mapping = inode->i_mapping; struct shmem_inode_info *info = SHMEM_I(inode); struct mm_struct *charge_mm = vma ? vma->vm_mm : NULL; - struct page *page; struct folio *folio = NULL; swp_entry_t swap; int error; @@ -1734,8 +1735,8 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, return -EIO; /* Look it up and read it in.. */ - page = lookup_swap_cache(swap, NULL, 0); - if (!page) { + folio = swap_cache_get_folio(swap, NULL, 0); + if (!folio) { /* Or update major stats only when swapin succeeds?? */ if (fault_type) { *fault_type |= VM_FAULT_MAJOR; @@ -1743,13 +1744,12 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, count_memcg_event_mm(charge_mm, PGMAJFAULT); } /* Here we actually start the io */ - page = shmem_swapin(swap, gfp, info, index); - if (!page) { + folio = shmem_swapin(swap, gfp, info, index); + if (!folio) { error = -ENOMEM; goto failed; } } - folio = page_folio(page); /* We have to do this with folio locked to prevent races */ folio_lock(folio); From patchwork Fri Sep 2 19:46:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC28FC6FA82 for ; Fri, 2 Sep 2022 19:47:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE08F800FD; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 388F1800FC; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B332D800F8; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E9173800F8 for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D11A71A0A06 for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.31.8AE8E0F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id 63AB540059 for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=v553qHyggFSxtMtAEhAmKoWjKn9V+gfLFROIZgqqBpw=; b=D2UsbHQ6kqA5oQwdnu/RDxuabI rx0GhLzfvIsMF1w1MRvxHDtVpPkYJzqYdtKN+kGNitzMSBVViwuKPJiAopODBH7JCNIZcQvYj+LwC Z93It7qA5M635qCXZjBosFdhQEQrwavbRBI3p3J2LlzOsCRzasNvzbkrl9k6eg16irO8g/rlx533X oahNwq7USbg3fsAR94dEXuIwwph4IDnfz8wBjfL/IgeTnkkvGjSqKV7F5Op16+MoyzQHyvipgPfiz t8eIFzefpZGNIj0eKS8mKEZTyXdmUHfQOE0nMMPIwFbgsKc8nMWStGPMdpilwVprfqC7BKgbD6dW1 84/MBlrw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd1-007Icx-Sw; Fri, 02 Sep 2022 19:46:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 21/57] shmem: Convert shmem_getpage_gfp() to shmem_get_folio_gfp() Date: Fri, 2 Sep 2022 20:46:17 +0100 Message-Id: <20220902194653.1739778-22-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148023; a=rsa-sha256; cv=none; b=TGOAODBeQ8lc6j4tB9qSDI1JbzK/A96EOOKRFS42yM/wVXnpLQ59xQI3AIFru3U2YKQ4JJ jz14qspC8RT9YmInY9ZuMX+62rpkw2NsaFMY0vn7F8Jn6Hrz9pQ6GbG9xsoGrEJC7cEDUA 7YorL5JCovxyig93HhrWMZenhd7Wl1w= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=D2UsbHQ6; dmarc=none; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v553qHyggFSxtMtAEhAmKoWjKn9V+gfLFROIZgqqBpw=; b=PA7H9mj0LxOWmHpKI1Nhae1ayXFnV+Q6ZpDW6nak5Xibgkjx5rVKGcm9Gj4FtrUBXMJS+T NQGiO72d+5Lu/f2/M3W2UwIrPsd+TODThtCAkyxMTct2tpkRl/PqZZts+I5i1dWghZqSCC Rgmdz8Zxz6qly9Mb3h+waIfQXd6pSng= X-Stat-Signature: dup5zhec5zrx8kce3o6qs94k93eu5m9m X-Rspamd-Queue-Id: 63AB540059 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=D2UsbHQ6; dmarc=none; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1662148023-920752 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a shmem_getpage_gfp() wrapper for compatibility with current users. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 70 ++++++++++++++++++++++++++++++++---------------------- 1 file changed, 41 insertions(+), 29 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index b685acd9f149..89536091928f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -139,17 +139,6 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, struct folio **foliop, enum sgp_type sgp, gfp_t gfp, struct vm_area_struct *vma, vm_fault_t *fault_type); -static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp, - gfp_t gfp, struct vm_area_struct *vma, - struct vm_fault *vmf, vm_fault_t *fault_type); - -int shmem_getpage(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp) -{ - return shmem_getpage_gfp(inode, index, pagep, sgp, - mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL); -} static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb) { @@ -1595,7 +1584,7 @@ static struct folio *shmem_alloc_and_acct_folio(gfp_t gfp, struct inode *inode, /* * When a page is moved from swapcache to shmem filecache (either by the - * usual swapin of shmem_getpage_gfp(), or by the less common swapoff of + * usual swapin of shmem_get_folio_gfp(), or by the less common swapoff of * shmem_unuse_inode()), it may have been read in earlier from swap, in * ignorance of the mapping it belongs to. If that mapping has special * constraints (like the gma500 GEM driver, which requires RAM below 4GB), @@ -1812,7 +1801,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, } /* - * shmem_getpage_gfp - find page in cache, or get from swap, or allocate + * shmem_get_folio_gfp - find page in cache, or get from swap, or allocate * * If we allocate a new one we do not mark it dirty. That's up to the * vm. If we swap it in we mark it dirty since we also free the swap @@ -1821,10 +1810,10 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, * vma, vmf, and fault_type are only supplied by shmem_fault: * otherwise they are NULL. */ -static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp, gfp_t gfp, - struct vm_area_struct *vma, struct vm_fault *vmf, - vm_fault_t *fault_type) +static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, + struct folio **foliop, enum sgp_type sgp, gfp_t gfp, + struct vm_area_struct *vma, struct vm_fault *vmf, + vm_fault_t *fault_type) { struct address_space *mapping = inode->i_mapping; struct shmem_inode_info *info = SHMEM_I(inode); @@ -1864,7 +1853,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, if (error == -EEXIST) goto repeat; - *pagep = &folio->page; + *foliop = folio; return error; } @@ -1874,7 +1863,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, folio_mark_accessed(folio); if (folio_test_uptodate(folio)) goto out; - /* fallocated page */ + /* fallocated folio */ if (sgp != SGP_READ) goto clear; folio_unlock(folio); @@ -1882,10 +1871,10 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, } /* - * SGP_READ: succeed on hole, with NULL page, letting caller zero. - * SGP_NOALLOC: fail on hole, with NULL page, letting caller fail. + * SGP_READ: succeed on hole, with NULL folio, letting caller zero. + * SGP_NOALLOC: fail on hole, with NULL folio, letting caller fail. */ - *pagep = NULL; + *foliop = NULL; if (sgp == SGP_READ) return 0; if (sgp == SGP_NOALLOC) @@ -1918,7 +1907,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, if (error != -ENOSPC) goto unlock; /* - * Try to reclaim some space by splitting a huge page + * Try to reclaim some space by splitting a large folio * beyond i_size on the filesystem. */ while (retry--) { @@ -1954,9 +1943,9 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, if (folio_test_pmd_mappable(folio) && DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE) < - hindex + HPAGE_PMD_NR - 1) { + folio_next_index(folio) - 1) { /* - * Part of the huge page is beyond i_size: subject + * Part of the large folio is beyond i_size: subject * to shrink under memory pressure. */ spin_lock(&sbinfo->shrinklist_lock); @@ -1973,14 +1962,14 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, } /* - * Let SGP_FALLOC use the SGP_WRITE optimization on a new page. + * Let SGP_FALLOC use the SGP_WRITE optimization on a new folio. */ if (sgp == SGP_FALLOC) sgp = SGP_WRITE; clear: /* - * Let SGP_WRITE caller clear ends if write does not fill page; - * but SGP_FALLOC on a page fallocated earlier must initialize + * Let SGP_WRITE caller clear ends if write does not fill folio; + * but SGP_FALLOC on a folio fallocated earlier must initialize * it now, lest undo on failure cancel our earlier guarantee. */ if (sgp != SGP_WRITE && !folio_test_uptodate(folio)) { @@ -2006,7 +1995,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, goto unlock; } out: - *pagep = folio_page(folio, index - hindex); + *foliop = folio; return 0; /* @@ -2036,6 +2025,29 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, return error; } +static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, + struct page **pagep, enum sgp_type sgp, + gfp_t gfp, struct vm_area_struct *vma, + struct vm_fault *vmf, vm_fault_t *fault_type) +{ + struct folio *folio = NULL; + int ret = shmem_get_folio_gfp(inode, index, &folio, sgp, gfp, vma, + vmf, fault_type); + + if (folio) + *pagep = folio_file_page(folio, index); + else + *pagep = NULL; + return ret; +} + +int shmem_getpage(struct inode *inode, pgoff_t index, + struct page **pagep, enum sgp_type sgp) +{ + return shmem_getpage_gfp(inode, index, pagep, sgp, + mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL); +} + /* * This is like autoremove_wake_function, but it removes the wait queue * entry unconditionally - even if something else had already woken the From patchwork Fri Sep 2 19:46:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75BA3C6FA82 for ; Fri, 2 Sep 2022 19:47:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE4C7800FA; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D784B800FD; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D92D80114; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D15B380108 for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B0A43801EA for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.12.54970B5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id 4177B140058 for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=AwmMvcWOAAeFfkqkqYj/393+BTU1yn9WJ+SkxVL14ks=; b=e1GAC1ApoL6WjcIvr0CY8Y5zhy oCVIJpIdIMlMc+BhTE+buhSgP4HrH4NfoLFuP/G3e9i8xBYEwfzP+gDjv2ZeBRiiRtK2DJSCy9GqF DbqD/lrjbXGqpayqtMxjxPdc/KT1L2oYoRt4+bcwZnDj3+D2iebTbFkAsjBTeLASBxyRKtvL+crbg 4uwYXUqGasthcZSLr69060s9yCx0vKbrWW6OP0Aomsue+7Eme3ugY30xsNcC8efEYQDbuMWkQt7mt bADmR/+EV98HqOHZGbBawKr+pV5684HphMvQBKSuzbc5nEyMWDYK8WFVmQQXiBsXQFB+RS6c33HlZ ipxYzsSw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd2-007Id5-1F; Fri, 02 Sep 2022 19:47:00 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 22/57] shmem: Convert shmem_fault() to use shmem_get_folio_gfp() Date: Fri, 2 Sep 2022 20:46:18 +0100 Message-Id: <20220902194653.1739778-23-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148023; a=rsa-sha256; cv=none; b=MRiT4l9UPFM4eobVj8o1Ua20K1ACf4zPHvOxcY2/Niji4hYDn7JcQYbW4wn20/tZ7mwahl NvyX5TKbZ0FVi5MiKc2nsvwXriFvi1oDfJcyIYxvTbhkwlfGAu6nLgq8JHbkDS3OT6TC+y vZMkSU/M8Gf3fDY996a5h9K7qziR7N8= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=e1GAC1Ap; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AwmMvcWOAAeFfkqkqYj/393+BTU1yn9WJ+SkxVL14ks=; b=KcjqlhuxnwDFzeCX8h5LNtIb1PfNqMrNrWFPcEjKUkvKLupmgGTHBIwWoxAdmZujRu5/D7 7ZeqlxPDL7/xMWktdaIz88fxiSYkEe1jLJyWaOoj91L7dcic79Wz7Whb4UT5FK5rruumXW tp/trD30YFX5jvPqAnjaHxh42usYCyA= Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=e1GAC1Ap; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: 7j5m89iuab6pjhwjqgdpn7d4iszwop83 X-Rspamd-Queue-Id: 4177B140058 X-Rspamd-Server: rspam04 X-Rspam-User: X-HE-Tag: 1662148023-481951 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: No particular advantage for this function, but necessary to remove shmem_getpage_gfp(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/shmem.c b/mm/shmem.c index 89536091928f..59184bf6731f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2065,6 +2065,7 @@ static vm_fault_t shmem_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; struct inode *inode = file_inode(vma->vm_file); gfp_t gfp = mapping_gfp_mask(inode->i_mapping); + struct folio *folio; int err; vm_fault_t ret = VM_FAULT_LOCKED; @@ -2127,10 +2128,11 @@ static vm_fault_t shmem_fault(struct vm_fault *vmf) spin_unlock(&inode->i_lock); } - err = shmem_getpage_gfp(inode, vmf->pgoff, &vmf->page, SGP_CACHE, + err = shmem_get_folio_gfp(inode, vmf->pgoff, &folio, SGP_CACHE, gfp, vma, vmf, &ret); if (err) return vmf_error(err); + vmf->page = folio_file_page(folio, vmf->pgoff); return ret; } From patchwork Fri Sep 2 19:46:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964564 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CF7AC6FA83 for ; Fri, 2 Sep 2022 19:47:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96258800FE; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D25580105; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4560980101; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C048F80107 for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9918CA121F for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.18.22CBE98 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 395E01C005F for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=5Lid3a1FhkSdqQODgXuXyckrWFdaSG2l+haewh8Zgwg=; b=oSRwhnyqtBGt8BcB4U7Ni/76eY 42ZgFzo2lW5z7UH107AwuTPUtzjJdZWmZ5un0/oF5A4CMA5/18h8Jx0f3QWTyEMAi2iPiOk9PV2f/ mGuJiwIQOjIr73LNQJB7K7QUEcmCD4mB1ov9kfGyXQfHyBOB0u+FWMQNgefwbmaiXkmQoupdbcXb+ o8Ofyo4PYir3BimLxfJectKMMbHrj+vwE/L1B2WMgFm5Ne57sCr+isKL4PkjZFBuTog1D+TKDceqf 1bLWpuM80SsuJHdrzwWbL53yHb19JKljBoch+i/cRxpZKhCzMrUcikuTrnZ3RKR0S6oYM2Igyau3M hKvXenlg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd2-007IdC-5M; Fri, 02 Sep 2022 19:47:00 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 23/57] shmem: Convert shmem_read_mapping_page_gfp() to use shmem_get_folio_gfp() Date: Fri, 2 Sep 2022 20:46:19 +0100 Message-Id: <20220902194653.1739778-24-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148023; a=rsa-sha256; cv=none; b=vwEcy9fgoDKiCJXXRMladFiSUIXaG9Ivqu5uWvx7ztNnNodbQae/OCAAWWTtEMEGNF1owD SclP9eBtxkjWDfbV+cU3XDvk57Mrh6DUcW4Ug5+0COTrobTh+3ZIFmCrZZ++SmrU5p10/v Xg0Sn8gfxTakU5I8rQF/Wan0yuz4POI= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oSRwhnyq; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5Lid3a1FhkSdqQODgXuXyckrWFdaSG2l+haewh8Zgwg=; b=Dj5Z205clJ+/nnRmOAXEvx0cgzkrEtEOnBlu37pfqk4YeQBwyVSxmlqyJJc0igEVzuT1aJ vf0MT3sg72QmZUmb+WaHatgENV7Ar0akouuby41pVkeCE7MOIkd0JvQFuV83s5f+O8GOpv QBI56iAx/Djiy4k3QGmO7q4znrREax8= X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 395E01C005F Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oSRwhnyq; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 66nb414hgf8dm89ibpnkroxjyguk77is X-HE-Tag: 1662148023-71754 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves a couple of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 59184bf6731f..107b8f019373 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -4269,18 +4269,20 @@ struct page *shmem_read_mapping_page_gfp(struct address_space *mapping, { #ifdef CONFIG_SHMEM struct inode *inode = mapping->host; + struct folio *folio; struct page *page; int error; BUG_ON(!shmem_mapping(mapping)); - error = shmem_getpage_gfp(inode, index, &page, SGP_CACHE, + error = shmem_get_folio_gfp(inode, index, &folio, SGP_CACHE, gfp, NULL, NULL, NULL); if (error) return ERR_PTR(error); - unlock_page(page); + folio_unlock(folio); + page = folio_file_page(folio, index); if (PageHWPoison(page)) { - put_page(page); + folio_put(folio); return ERR_PTR(-EIO); } From patchwork Fri Sep 2 19:46:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1566EECAAD5 for ; Fri, 2 Sep 2022 19:47:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 38852800EE; Fri, 2 Sep 2022 15:47:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E3822800EA; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5852800EB; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8E580800F1 for ; Fri, 2 Sep 2022 15:47:00 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6788D121032 for ; Fri, 2 Sep 2022 19:47:00 +0000 (UTC) X-FDA: 79868178600.03.EBDC67C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id F084318004E for ; Fri, 2 Sep 2022 19:46:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=+uCDeTNBCbU4U+AQYTZ3Vcf2Qm5iFtfWTDrhJbl7ndY=; b=rUoLON+EI0BKBrLyOGzrCBb3Lm njwQuDxDA3GiI1Q2mu0mn50M/rid24Cwu+n7zYc9JCnygN0eVcShjfh2CfEyPGu7A5beIDNU1TIL7 NjdzDNsULku3oHOkR5yDT2B0qE4xILN42XzSjfJoi8mbkFFMmnnn8ogPZQiCZBvaO4FPNuUd0CEP+ 94J+WwFDpfv6NH83cUvJI7QXmSiUMu/nobxExUseqYUuYQ/Cmwx9K2WE2SsoERaBYbypBw014LQ5F hRAXbWEgHfhqQGMfHvDVrl3UNXcPbGWzzatzajV6ekqyUIsTov4Njc57G/ifb2AEacBhWUQI8JvH2 A0Te0KCg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd2-007IdJ-8t; Fri, 02 Sep 2022 19:47:00 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 24/57] shmem: Add shmem_get_folio() Date: Fri, 2 Sep 2022 20:46:20 +0100 Message-Id: <20220902194653.1739778-25-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148020; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+uCDeTNBCbU4U+AQYTZ3Vcf2Qm5iFtfWTDrhJbl7ndY=; b=LC4QPF+0Vv7nLo0krx7VzOfB+F2zhMimr//r4MZFhZz4ZKc63p9an4GadadgtB2aMzkAW9 oSUjtN3pYYe1g8+UOpuvra8E0e9yr+3Rgg5BvM1uaJn9+fJYJuNLfWXwkWwUKh0me/wUyP Kn2QWowV0Ts594KhsPJNuXBGY1r0w08= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rUoLON+E; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148020; a=rsa-sha256; cv=none; b=iFlJ7rgRHznZBXiOY/kb/RSsbFiYuA2f1bkux2DXkgfsnSpYLTS61JUPNrPS4xkOtvZ7bO sXwznOw4eNHXKxSDMMSaouBB2MZDLhyL1RGO/iMDf0m84SRP7iDWHiaDUDRMzU3wMwNYCs oeIjDRuPm7ETcK10E79uksh8PZ8FWEo= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: F084318004E Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rUoLON+E; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: mhk8eea1yrmnudtgnm6ea5cqkr1zeeax X-HE-Tag: 1662148019-717100 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With no remaining callers of shmem_getpage_gfp(), add shmem_get_folio() and reimplement shmem_getpage() as a call to shmem_get_folio(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/shmem_fs.h | 2 ++ mm/shmem.c | 23 ++++++++++------------- 2 files changed, 12 insertions(+), 13 deletions(-) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index ff0b990de83d..f4bd50b08a91 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -113,6 +113,8 @@ enum sgp_type { extern int shmem_getpage(struct inode *inode, pgoff_t index, struct page **pagep, enum sgp_type sgp); +int shmem_get_folio(struct inode *inode, pgoff_t index, struct folio **foliop, + enum sgp_type sgp); static inline struct page *shmem_read_mapping_page( struct address_space *mapping, pgoff_t index) diff --git a/mm/shmem.c b/mm/shmem.c index 107b8f019373..eb9e024f7cf4 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2025,14 +2025,18 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, return error; } -static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp, - gfp_t gfp, struct vm_area_struct *vma, - struct vm_fault *vmf, vm_fault_t *fault_type) +int shmem_get_folio(struct inode *inode, pgoff_t index, struct folio **foliop, + enum sgp_type sgp) +{ + return shmem_get_folio_gfp(inode, index, foliop, sgp, + mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL); +} + +int shmem_getpage(struct inode *inode, pgoff_t index, + struct page **pagep, enum sgp_type sgp) { struct folio *folio = NULL; - int ret = shmem_get_folio_gfp(inode, index, &folio, sgp, gfp, vma, - vmf, fault_type); + int ret = shmem_get_folio(inode, index, &folio, sgp); if (folio) *pagep = folio_file_page(folio, index); @@ -2041,13 +2045,6 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, return ret; } -int shmem_getpage(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp) -{ - return shmem_getpage_gfp(inode, index, pagep, sgp, - mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL); -} - /* * This is like autoremove_wake_function, but it removes the wait queue * entry unconditionally - even if something else had already woken the From patchwork Fri Sep 2 19:46:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964562 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CE36C6FA83 for ; Fri, 2 Sep 2022 19:47:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D7FE800F8; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3D9F880111; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D9CA180101; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A7ED1800F5 for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 83D05ABC6D for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.29.9B1CB64 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 145794005B for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=YnWYtJTxyUkfJ5xzIKovXVn+G+9hgTCLzXw2yYLgy04=; b=Xy3RbiM0O1IA+cpsJUECFt74xN ZnV/jNtRHjIMA5qc4TlbXhqDNgO+MHR/N8eyRx0yZqppKsGS575WxDtqJDujfrP84lkWf/2k/NZgU BoDy5s56C5g9H7VCKjb7JWM+5UqdOi+4wmLzh+DkIT3P27Wy15meb/nJybSywSzfAIE5nioJOKNrY ihG03gelw6kwWDNkyh1H0zA4Pgi7I+/ANKcPbB20s7z0iwoD+qqZ4B7hVMdrdqNFg4IQKx8BHn66z ER0gO1J2zUl4XGh5MsDNALjdcwJuw3/VTm/NzbhjkuazVV13/JMyU8B0hA5SNZrbbSjBzvbb36/nj aM2Pv7FQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd2-007IdQ-CQ; Fri, 02 Sep 2022 19:47:00 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 25/57] shmem: Convert shmem_get_partial_folio() to use shmem_get_folio() Date: Fri, 2 Sep 2022 20:46:21 +0100 Message-Id: <20220902194653.1739778-26-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148023; a=rsa-sha256; cv=none; b=Lm+98Npx8HDIw16ey+xtJ2B9wVwn3ov1avsjhKeDyCe4TgFFItf5xOCtQ1JmQgVW+uUX4g YN/ZodrJMYliTdVQLQiduiTczfv5qAGbaFnFmOJvzK2ekDMKWD6/0ZyJhFHFaDK5njRB3d PxYeflsTCJvEAqtRLdnIMG0EhUF9+0c= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Xy3RbiM0; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YnWYtJTxyUkfJ5xzIKovXVn+G+9hgTCLzXw2yYLgy04=; b=2GpD9YqFx17R5BA1bG4OKvgRQcgfoO7TvyfZkSwHnfoUGQ3BEs+4fxmaewGkREAi9rgHPn eNhzKGUDYhBDQxLUJZJ0nDvZ0GJ7H/YzBdvI124I2AB7F5EZvKHGCd4A2Y/LXcujkE9WhE 0uAJ6bsCZboPhc3o0RB6R0jzq8lyCnA= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 145794005B Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Xy3RbiM0; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: snr9g3yeps81ro57zn6jeofit1c5pxpo X-HE-Tag: 1662148023-82264 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Get rid of an unnecessary folio->page->folio conversion. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index eb9e024f7cf4..a1cbee06b5cf 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -874,10 +874,9 @@ void shmem_unlock_mapping(struct address_space *mapping) static struct folio *shmem_get_partial_folio(struct inode *inode, pgoff_t index) { struct folio *folio; - struct page *page; /* - * At first avoid shmem_getpage(,,,SGP_READ): that fails + * At first avoid shmem_get_folio(,,,SGP_READ): that fails * beyond i_size, and reports fallocated pages as holes. */ folio = __filemap_get_folio(inode->i_mapping, index, @@ -888,9 +887,9 @@ static struct folio *shmem_get_partial_folio(struct inode *inode, pgoff_t index) * But read a page back from swap if any of it is within i_size * (although in some cases this is just a waste of time). */ - page = NULL; - shmem_getpage(inode, index, &page, SGP_READ); - return page ? page_folio(page) : NULL; + folio = NULL; + shmem_get_folio(inode, index, &folio, SGP_READ); + return folio; } /* From patchwork Fri Sep 2 19:46:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964560 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ABD3C6FA82 for ; Fri, 2 Sep 2022 19:47:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F88780109; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2068D800FC; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7B608010B; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A8A1C800FF for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 915B41A0A06 for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.23.A638152 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 0E3EB10005B for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=OYs7DKMOJ2CJVBO8NYQ1XKRVPPHMKNYSdlF+6wCOqj0=; b=t3VCfxgRI8wpNt/IDPIvKgidU2 bbmxSRfVJhqmvmYcSvXelIC0YhOAx/cDdc+qektHI7mv4NNUO7gDVdYoXWw0CeusNjC70EkMUNrk6 zibVXKo1+LUkYjZbScNQxutW/R1WeaNVAv/nuYHxFQ595EabvWdeDC9OfzbyhvOODA1bWiM1i+iFE oBB5laxrBhvaznzl2Kef4nN+ws2aTBXMmID730DOtzrXoPHncSV8UmWK6Nuc05gtUcKSkNlI55EOx 6RcgTs4ovwoAGqaAZO1X3Vq9oerSh8y8+OYvrUMsqVRg498HIL/xAt7QFagbQinHj1xQy2BjHzFrp mP4m6V5w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd2-007IdX-GG; Fri, 02 Sep 2022 19:47:00 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 26/57] shmem: Convert shmem_write_begin() to use shmem_get_folio() Date: Fri, 2 Sep 2022 20:46:22 +0100 Message-Id: <20220902194653.1739778-27-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148023; a=rsa-sha256; cv=none; b=T5uKcBcX9GVz2HkvJchA7vzWszKbvNZ6/UJygQsajyz1iZBDpnIT2yQpI2YQ5Md7xVxlyR 5NbsmNaxctL8biJrNgMjeVqVdQYy22oG7wKKRnN/FrfypP01JhDMObxoPWHVWuSXtR6P51 mVT/vlpBaSWBJ3Z+nAHxOG+GwajdCCg= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=t3VCfxgR; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OYs7DKMOJ2CJVBO8NYQ1XKRVPPHMKNYSdlF+6wCOqj0=; b=tXGyDlyjAOwFiefox8xgtm+sSVgxRfQh97PcMrxB9RkH/76Mia5/zJZjz8IjbKp89oSe5r HlE16ZDMIVResJRDmlLlmWB0fjUHCxO8gK4Ogc3HYqCitxgY8r0BpJxOKG1YwmUtO0DrVs zGW4IkKRBEjaVZqClmai1fXCctfNpWU= X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 0E3EB10005B Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=t3VCfxgR; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 75xszikqakxqr7qpc5sg64ju8p41rt8f X-HE-Tag: 1662148022-555326 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use a folio throughout this function, saving a couple of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index a1cbee06b5cf..44bb6ea11710 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2497,6 +2497,7 @@ shmem_write_begin(struct file *file, struct address_space *mapping, struct inode *inode = mapping->host; struct shmem_inode_info *info = SHMEM_I(inode); pgoff_t index = pos >> PAGE_SHIFT; + struct folio *folio; int ret = 0; /* i_rwsem is held by caller */ @@ -2508,14 +2509,15 @@ shmem_write_begin(struct file *file, struct address_space *mapping, return -EPERM; } - ret = shmem_getpage(inode, index, pagep, SGP_WRITE); + ret = shmem_get_folio(inode, index, &folio, SGP_WRITE); if (ret) return ret; + *pagep = folio_file_page(folio, index); if (PageHWPoison(*pagep)) { - unlock_page(*pagep); - put_page(*pagep); + folio_unlock(folio); + folio_put(folio); *pagep = NULL; return -EIO; } From patchwork Fri Sep 2 19:46:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964554 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88490ECAAD5 for ; Fri, 2 Sep 2022 19:47:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A719580103; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E14758010A; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A64480100; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0AEAF800F5 for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DCA7F1C5D79 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) X-FDA: 79868178684.22.B70AF72 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 750F74004E for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=zuR82lnHCE/hSq+W53qYh+bEg1qeXP700ShKnxXJjXg=; b=YOqmub1o5LRM4Od0ty368EWqPB VxPWumz5nQfh5nVqtJ7E4NdbQ1Uao1vEiOPi9qRtDAhMBGpFIyQupAChNHfUx7NBDzugxQ1Xxjvv5 0qHEh6V5uUUuOKMgLu6uWbDZxBsGICtDy+7MFNAKbY4JPBMjf2tp87X44cSWkig2QfRR7XC8AyEnZ pkhs51IBKDV9gQoOZbpTEv4ncp0RabEnsL9K2cMkrdGndiY9r9qZoE/5gW1zf78/c1vUibijc2QJz tMWWCbw7ABAFoKHTgvnxIKnZw26D/+rHcy27yWff39HtkjOXDTFaAV0X53UrYun9y8nxTPwRy9t1m p7XQrcgA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd2-007Ide-Jm; Fri, 02 Sep 2022 19:47:00 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 27/57] shmem: Convert shmem_file_read_iter() to use shmem_get_folio() Date: Fri, 2 Sep 2022 20:46:23 +0100 Message-Id: <20220902194653.1739778-28-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148022; a=rsa-sha256; cv=none; b=PKTnXmb6WPopbif9Z3f/A/g9xDVvCf3Ukh1G7/p6HUe0Q/uPEH67uoL8it4KC6J66RQ/wk q9DWtUQFPpj0V9s8Y0+iXBUztA+4lCogAXbcYq3pBdteTcX/+uhJyCVIevqFajcj7HKt7W CecAy9d9ZFHQKr3T4efNvbk5EJKOR6M= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=YOqmub1o; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zuR82lnHCE/hSq+W53qYh+bEg1qeXP700ShKnxXJjXg=; b=PKrTb4SVK/rkqQcCzPSoNAOArM421Je/Hmv/ow1ksd4Z6XzueJIZ98kov2xioFHHqMOa/T Fc8ii3SUAgw9ExaQygwm9m2p7EJPokpQRE+psWj2GeiJIA+zLJY0pZo9lDXGBKuU0oVZEg fU3ZT+JLuKrP7taPgbr8h5j1GeE9y04= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 750F74004E Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=YOqmub1o; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: dpgqqyr7r9i77pe99dyzd9xddyd73scz X-HE-Tag: 1662148022-143122 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use a folio throughout, saving five calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 44bb6ea11710..214887812adb 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2576,6 +2576,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) offset = *ppos & ~PAGE_MASK; for (;;) { + struct folio *folio = NULL; struct page *page = NULL; pgoff_t end_index; unsigned long nr, ret; @@ -2590,17 +2591,18 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) break; } - error = shmem_getpage(inode, index, &page, SGP_READ); + error = shmem_get_folio(inode, index, &folio, SGP_READ); if (error) { if (error == -EINVAL) error = 0; break; } - if (page) { - unlock_page(page); + if (folio) { + folio_unlock(folio); + page = folio_file_page(folio, index); if (PageHWPoison(page)) { - put_page(page); + folio_put(folio); error = -EIO; break; } @@ -2616,14 +2618,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) if (index == end_index) { nr = i_size & ~PAGE_MASK; if (nr <= offset) { - if (page) - put_page(page); + if (folio) + folio_put(folio); break; } } nr -= offset; - if (page) { + if (folio) { /* * If users can be writing to this page using arbitrary * virtual addresses, take care about potential aliasing @@ -2635,13 +2637,13 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) * Mark the page accessed if we read the beginning. */ if (!offset) - mark_page_accessed(page); + folio_mark_accessed(folio); /* * Ok, we have the page, and it's up-to-date, so * now we can copy it to user space... */ ret = copy_page_to_iter(page, offset, nr, to); - put_page(page); + folio_put(folio); } else if (user_backed_iter(to)) { /* From patchwork Fri Sep 2 19:46:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A92DC6FA83 for ; Fri, 2 Sep 2022 19:47:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 78618800EB; Fri, 2 Sep 2022 15:47:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BD18800F1; Fri, 2 Sep 2022 15:47:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A03D800F3; Fri, 2 Sep 2022 15:47:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 13FCF800F1 for ; Fri, 2 Sep 2022 15:47:01 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E4D5E120B36 for ; Fri, 2 Sep 2022 19:47:00 +0000 (UTC) X-FDA: 79868178600.11.9734A06 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id 902AB18004E for ; Fri, 2 Sep 2022 19:47:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=KHdw5Sh1ebNYRtUN9/JSFbx5sWfjgmqbdXN+cIr0Ikw=; b=ex27bDTNAsdr2lYDRVKroPTZKz Bj8I2yfEzR7FrOstpA18q3Wm8NvGiSQFQCByCeoqYe2kpu1KgwmVH9G0jg71Xcdlnm/1R4eYWlafA uyAakz4DiNG2ZzK/j04ThWt1WnTPFx9BIMfzB4OFZgjHnnbbuSgx48dGV590qlyQNFeLDzxNMCdE8 3q+3DLZZO5wjKluQq9I/291snmbUU1s0znYDJsm7HQhGoo6wqsp7lQEB5H+hdPi7BWYR9FF+dH+TL yi7QYtNRAmD61/3iq90ZINpENcOjTSsrUIEPnMi5X+oZ6alPjyDA5Lc5qPUPAUoOE+G5yBgR+qQTQ LJYHTwfQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd2-007Idl-Na; Fri, 02 Sep 2022 19:47:00 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 28/57] shmem: Convert shmem_fallocate() to use a folio Date: Fri, 2 Sep 2022 20:46:24 +0100 Message-Id: <20220902194653.1739778-29-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148020; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KHdw5Sh1ebNYRtUN9/JSFbx5sWfjgmqbdXN+cIr0Ikw=; b=Ck8G348qCsH22CQwJCfGiCi0facgHW2EQrsczBMb0CGY9mboqOsMrU28LLNDlITA22AiMf iYB1M4vJ/z8ZsG+KtPI1cQcJnM/opYnWap48H1YoXPkRL2gAqIbJXvuLEwm1RnDNhtyj5w PeWb5XeJi86rzqR021rplEqs5s+CUmc= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ex27bDTN; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148020; a=rsa-sha256; cv=none; b=GcUk0dYxiN2v5594k/6kNt6oTcKXJtoMwHAH2XWaZ9Y7Ic/dGTyewKyv2QSf1FXmk1fIIH 0MhwrlXxHmv+CW4X7LPR2jnNUnANHwt2GPgTqPBGFfdcBLx7X3ArETWHFlqoVSZRFJaYOc JwjhHy4TG7GTY7pylIJGF0LE6vAPLLI= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 902AB18004E Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ex27bDTN; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: fxfumhsqg1du16jisheims1nsfxwg3o7 X-HE-Tag: 1662148020-80621 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Call shmem_get_folio() and use the folio APIs instead of the page APIs. Saves several calls to compound_head() and removes assumptions about the size of a large folio. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 214887812adb..f251b026e83f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2786,7 +2786,7 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, info->fallocend = end; for (index = start; index < end; ) { - struct page *page; + struct folio *folio; /* * Good, the fallocate(2) manpage permits EINTR: we may have @@ -2797,10 +2797,11 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, else if (shmem_falloc.nr_unswapped > shmem_falloc.nr_falloced) error = -ENOMEM; else - error = shmem_getpage(inode, index, &page, SGP_FALLOC); + error = shmem_get_folio(inode, index, &folio, + SGP_FALLOC); if (error) { info->fallocend = undo_fallocend; - /* Remove the !PageUptodate pages we added */ + /* Remove the !uptodate folios we added */ if (index > start) { shmem_undo_range(inode, (loff_t)start << PAGE_SHIFT, @@ -2809,37 +2810,34 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, goto undone; } - index++; /* * Here is a more important optimization than it appears: - * a second SGP_FALLOC on the same huge page will clear it, - * making it PageUptodate and un-undoable if we fail later. + * a second SGP_FALLOC on the same large folio will clear it, + * making it uptodate and un-undoable if we fail later. */ - if (PageTransCompound(page)) { - index = round_up(index, HPAGE_PMD_NR); - /* Beware 32-bit wraparound */ - if (!index) - index--; - } + index = folio_next_index(folio); + /* Beware 32-bit wraparound */ + if (!index) + index--; /* * Inform shmem_writepage() how far we have reached. * No need for lock or barrier: we have the page lock. */ - if (!PageUptodate(page)) + if (!folio_test_uptodate(folio)) shmem_falloc.nr_falloced += index - shmem_falloc.next; shmem_falloc.next = index; /* - * If !PageUptodate, leave it that way so that freeable pages + * If !uptodate, leave it that way so that freeable folios * can be recognized if we need to rollback on error later. - * But set_page_dirty so that memory pressure will swap rather - * than free the pages we are allocating (and SGP_CACHE pages + * But mark it dirty so that memory pressure will swap rather + * than free the folios we are allocating (and SGP_CACHE folios * might still be clean: we now need to mark those dirty too). */ - set_page_dirty(page); - unlock_page(page); - put_page(page); + folio_mark_dirty(folio); + folio_unlock(folio); + folio_put(folio); cond_resched(); } From patchwork Fri Sep 2 19:46:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BB50C6FA82 for ; Fri, 2 Sep 2022 19:47:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6759F800F1; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 600D5800F8; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42891800F5; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 30CF6800F1 for ; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 192851A0A06 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) X-FDA: 79868178684.14.FA52526 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id A9B7818004E for ; Fri, 2 Sep 2022 19:47:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ezUYIihnvv+vElD4XkRV6U4Svwf+ms2+pO0xLEEtH10=; b=i0JP9pKetNzcKZrAiI+YZ0Y29G Mhz1SmjXX3jBwmKDRsIlZ4JsQqhU1rRfckEKaf2e8ueDE/RzXqYthnA5r+4v/c0Iu8+jBJCxF3adB /myM0wPF4vP4yfVxSesxXcgMJEQ3Jr6kXn+Q9JK9ptqDVOyAkfd2R0qq12sYkSdFpjKiw1CkJYBrY O6rJv19L2mRgbDXY7I+sTB+ch+r/21xgptmnvwS4FjpeK2RLeIV5ZilEjAvBFVWzWp7QYK4s727IE qzzYlasJIx7jZsBX9FoqXJpnBrxNmG2BuvvBeY0wE/1iEEA72QBWiEPYOfBh80TJUZqs7Lp7V4Y2/ bMZU6a4A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd2-007Ids-RP; Fri, 02 Sep 2022 19:47:00 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 29/57] shmem: Convert shmem_symlink() to use a folio Date: Fri, 2 Sep 2022 20:46:25 +0100 Message-Id: <20220902194653.1739778-30-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148021; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ezUYIihnvv+vElD4XkRV6U4Svwf+ms2+pO0xLEEtH10=; b=CfNf8/mx4M9KWUcWAT/YdJ0VEekL1dJROrlx5b1BmIkxMFj7QTB0lWK/JH9kG0+kGU4lFc 9fcLfanprtGkzmOc32RWEcnjHR3qRqaSrNUwh74xT15SpBvuTbc3heW/fWKFK8wsSiX7om +zVBp4S08DegHA4HCJjiRU9MCtAmOmQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=i0JP9pKe; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148021; a=rsa-sha256; cv=none; b=NFqoEsb/dMEHU04heD/+N0o2ndAXT8YsGyvLbq0zQuc+/A+G11H2OqYxKwlpnkFfth6eXP AZVACnbkFAOfHALo3FjsG4pdkMspxTnlUphwPdDATsg/KKdY43MMpPfP1iubgteyI60lKQ DbExLqhZQp63k403wvWYUtH7N2OTlGM= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A9B7818004E Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=i0JP9pKe; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 89j7byouoq87tszjpsyzi9g7f7nkjyb6 X-HE-Tag: 1662148021-153310 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: While symlinks will always be < PAGE_SIZE, using the folio APIs gets rid of unnecessary calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index f251b026e83f..a4251c66f317 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3092,7 +3092,7 @@ static int shmem_symlink(struct user_namespace *mnt_userns, struct inode *dir, int error; int len; struct inode *inode; - struct page *page; + struct folio *folio; len = strlen(symname) + 1; if (len > PAGE_SIZE) @@ -3120,18 +3120,18 @@ static int shmem_symlink(struct user_namespace *mnt_userns, struct inode *dir, inode->i_op = &shmem_short_symlink_operations; } else { inode_nohighmem(inode); - error = shmem_getpage(inode, 0, &page, SGP_WRITE); + error = shmem_get_folio(inode, 0, &folio, SGP_WRITE); if (error) { iput(inode); return error; } inode->i_mapping->a_ops = &shmem_aops; inode->i_op = &shmem_symlink_inode_operations; - memcpy(page_address(page), symname, len); - SetPageUptodate(page); - set_page_dirty(page); - unlock_page(page); - put_page(page); + memcpy(folio_address(folio), symname, len); + folio_mark_uptodate(folio); + folio_mark_dirty(folio); + folio_unlock(folio); + folio_put(folio); } dir->i_size += BOGO_DIRENT_SIZE; dir->i_ctime = dir->i_mtime = current_time(dir); From patchwork Fri Sep 2 19:46:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2673DC6FA83 for ; Fri, 2 Sep 2022 19:47:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3FCFC8010C; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 912E880104; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56BE280100; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D0D65800FA for ; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B2AB5801EA for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) X-FDA: 79868178684.30.A61AA14 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id 4A188180053 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=vz0t8JH6vL8+F/QHJWuBLbUJiPp5/CD23wS+rUMcnKA=; b=aRt19jJgIa8RgtL88FscQgLE9W P0KQDIjFKBAlX51t+CXueIpmtRBK/HLGGVIRBQ3UnX6Gulga4q6yt+L5uErQF3/ikcelk/M55d667 87cCGOfGDYqh9IJ0OzITHRjC+QZ+OCD4fqSUhI3V/yF/Popd5Si+nLlJZh3ZW2nAq62UjpRmN0Dfi 2EFLg/uuuP9HnTVG5H1ecqiHY7fGHDXXDt8Aj8ZJ6VDPl/f7d2J1TC3wJ66tkWePyg2YmqbYMh9Z8 yuFbMeJGdnWJB82krXWmsHIYWiIAHpgBKbJFk9piwskC3+70veHKtxfVcdHwbvYpjMV860PLJSge3 NkHWDSBw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd2-007Ie5-WD; Fri, 02 Sep 2022 19:47:01 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 30/57] shmem: Convert shmem_get_link() to use a folio Date: Fri, 2 Sep 2022 20:46:26 +0100 Message-Id: <20220902194653.1739778-31-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vz0t8JH6vL8+F/QHJWuBLbUJiPp5/CD23wS+rUMcnKA=; b=iMb5W1c+RPzV41jdiKY+1fSTI2p+Sw31zckSo5g/TL3NGoo2w9ulD3ha7R+mIYk5rTUmwA JhZQjsNHEFB18B/RdnhV+sGCUU61KbEkvmHrAvJ4rpQC7F14l1LAYQ1r6dQvmPVqdlcTJA hARe+fVRRwnbpCvFpAcWIzmHfYlBY+E= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=aRt19jJg; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148022; a=rsa-sha256; cv=none; b=KJD3BCi7d/RD9+ieKQ1MUuHmb7jRzuyt1qsl6LUvYSzoXL2o5wp38YA0+HW/s7d3nFLSxc UJqs+SdbwkNhQJ1zlT67XcyxlqtWx/joOe39aiR0N/Ap6fs2sqOj1Cq6GbV2myfM80RbJC QUKpvweIbl5ZLFNM3gP17mgDgo4bAPo= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4A188180053 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=aRt19jJg; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: idumfmpxunoreifmi3ww59usi3ijuykf X-HE-Tag: 1662148022-24101 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Symlinks will never use a large folio, but using the folio API removes a lot of unnecessary folio->page->folio conversions. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index a4251c66f317..ab39753064f1 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3142,40 +3142,41 @@ static int shmem_symlink(struct user_namespace *mnt_userns, struct inode *dir, static void shmem_put_link(void *arg) { - mark_page_accessed(arg); - put_page(arg); + folio_mark_accessed(arg); + folio_put(arg); } static const char *shmem_get_link(struct dentry *dentry, struct inode *inode, struct delayed_call *done) { - struct page *page = NULL; + struct folio *folio = NULL; int error; + if (!dentry) { - page = find_get_page(inode->i_mapping, 0); - if (!page) + folio = filemap_get_folio(inode->i_mapping, 0); + if (!folio) return ERR_PTR(-ECHILD); - if (PageHWPoison(page) || - !PageUptodate(page)) { - put_page(page); + if (PageHWPoison(&folio->page) || + !folio_test_uptodate(folio)) { + folio_put(folio); return ERR_PTR(-ECHILD); } } else { - error = shmem_getpage(inode, 0, &page, SGP_READ); + error = shmem_get_folio(inode, 0, &folio, SGP_READ); if (error) return ERR_PTR(error); - if (!page) + if (!folio) return ERR_PTR(-ECHILD); - if (PageHWPoison(page)) { - unlock_page(page); - put_page(page); + if (PageHWPoison(&folio->page)) { + folio_unlock(folio); + folio_put(folio); return ERR_PTR(-ECHILD); } - unlock_page(page); + folio_unlock(folio); } - set_delayed_call(done, shmem_put_link, page); - return page_address(page); + set_delayed_call(done, shmem_put_link, folio); + return folio_address(folio); } #ifdef CONFIG_TMPFS_XATTR From patchwork Fri Sep 2 19:46:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964550 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B658C6FA83 for ; Fri, 2 Sep 2022 19:47:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA23A80106; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49CA8800FA; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE19A800FD; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C7652800F5 for ; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A63A8A120F for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) X-FDA: 79868178684.02.F75B9D2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id 48F4240044 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=V6IK8ZCbhUmsYP1wTpuNkrgSWvBm2omWzgtLiPmn7BE=; b=evk3E6SJkidq/2owDk2JrQmZ1v wVMcNRW+fG2Y7lyWSs7Fa5XplcWe5SkSfiZDjaI52mCY4284GlVLpVE868CJ4gAqPi2Py8H4RezMX pic2eU9GAamDr6Ucc1haAVMF6bao8gzceGhaN92uMhnLTz/WTwsg35fA69Rme8q8U9ErjryNsbXMJ lW/YKNJUQpQmA/DEC8l8/jpFxW7UM1uH/O+UrbPgHMZAGItGPAI1o3YznWYrd9rCAqooErPG75NGF ICnUPcsM0ZZ4Rd/t1XJzyiP11QGvR7fB3MviAqHRKYJAyIG7tfkSxHrt7oO33IGeCTywXYXRv9nj1 xs18ZJEA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd3-007IeK-3x; Fri, 02 Sep 2022 19:47:01 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 31/57] khugepaged: Call shmem_get_folio() Date: Fri, 2 Sep 2022 20:46:27 +0100 Message-Id: <20220902194653.1739778-32-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V6IK8ZCbhUmsYP1wTpuNkrgSWvBm2omWzgtLiPmn7BE=; b=JGA2SwO4Gi+5eJuc1pgQXdhff9z69OoBXSMlL4fhPMC6a5OTTj3XVCZOtQYy8ShYE9LpyI qSHaKIZFh/lRD6WeTClroqGpaoIVyeOozhfHAu6dtORNBIhx47CDuYubQ/wlreYYHK3dOB kEW1aEOKaGfo3KHNFxGa+YzE055JMOw= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=evk3E6SJ; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148022; a=rsa-sha256; cv=none; b=hO12CQBR/dUMvLBLfocLY9XAJJFB1KUwKqVGsLBMswOrnbSW19dubm3YaPBWet0X/A7QpO iXt1pVyoIyTrQLuDbKyq6uxdoaATF4N6JJGDjWP41hLNGngUuCMGXsQAFLTT0LodZOlKXc n+cRYuaucuPGx9wsOFjbfyg1aaL013w= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 48F4240044 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=evk3E6SJ; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: myzk1ugkfwhyxy3aywo4pompjhp38j6q X-HE-Tag: 1662148022-501854 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: shmem_getpage() is being removed, so call its replacement and find the precise page ourselves. Signed-off-by: Matthew Wilcox (Oracle) --- mm/khugepaged.c | 7 +++++-- mm/shmem.c | 4 ++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..6cf4bbd22fc9 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1682,13 +1682,16 @@ static void collapse_file(struct mm_struct *mm, } if (xa_is_value(page) || !PageUptodate(page)) { + struct folio *folio; + xas_unlock_irq(&xas); /* swap in or instantiate fallocated page */ - if (shmem_getpage(mapping->host, index, &page, - SGP_NOALLOC)) { + if (shmem_get_folio(mapping->host, index, + &folio, SGP_NOALLOC)) { result = SCAN_FAIL; goto xa_unlocked; } + page = folio_file_page(folio, index); } else if (trylock_page(page)) { get_page(page); xas_unlock_irq(&xas); diff --git a/mm/shmem.c b/mm/shmem.c index ab39753064f1..e89a11bf83f6 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3157,7 +3157,7 @@ static const char *shmem_get_link(struct dentry *dentry, folio = filemap_get_folio(inode->i_mapping, 0); if (!folio) return ERR_PTR(-ECHILD); - if (PageHWPoison(&folio->page) || + if (PageHWPoison(folio_page(folio, 0)) || !folio_test_uptodate(folio)) { folio_put(folio); return ERR_PTR(-ECHILD); @@ -3168,7 +3168,7 @@ static const char *shmem_get_link(struct dentry *dentry, return ERR_PTR(error); if (!folio) return ERR_PTR(-ECHILD); - if (PageHWPoison(&folio->page)) { + if (PageHWPoison(folio_page(folio, 0))) { folio_unlock(folio); folio_put(folio); return ERR_PTR(-ECHILD); From patchwork Fri Sep 2 19:46:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93571ECAAD5 for ; Fri, 2 Sep 2022 19:47:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED7CC800F3; Fri, 2 Sep 2022 15:47:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E13CC800F1; Fri, 2 Sep 2022 15:47:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C63C2800F3; Fri, 2 Sep 2022 15:47:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B8B4A800F1 for ; Fri, 2 Sep 2022 15:47:01 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8E7B2A1228 for ; Fri, 2 Sep 2022 19:47:01 +0000 (UTC) X-FDA: 79868178642.12.5843949 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id 25A1918004E for ; Fri, 2 Sep 2022 19:47:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xdYsV4qprpXif04fQYKo4IaOxe4qBke54s/dI1MnI7k=; b=eRLEM7H0IR6nF0u6QqJP6NbUSn 0oQfkoS99QOiNOzo+1o7F/1L88ouegJNK4TnVqXSWSQJyfuzNQ2bHivb7XhKZibiNhmdcDdfB0PP3 bze6t1/PtMhhWBoQ9KjvvXQng/cor9+urgFtvZY2g2OAXsvW68PVCAM0IFhedkVDgXtBa7+p8krbm R9upNAY1s5/4cw/Y7KMpJqi8ilNOovr/em01VekOgjz69DZWs032fq3nUiSrClYEtVv1XhTFPC2lG jQTu4MYJN72AADmo0QvG0KfpdYH/BcamHZ4zfiw2GXBihFSa1fMnvou9io/Q3LncyyeTxWh3/6VzA ti2pf8zQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd3-007IeS-8h; Fri, 02 Sep 2022 19:47:01 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 32/57] userfaultfd: Convert mcontinue_atomic_pte() to use a folio Date: Fri, 2 Sep 2022 20:46:28 +0100 Message-Id: <20220902194653.1739778-33-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148021; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xdYsV4qprpXif04fQYKo4IaOxe4qBke54s/dI1MnI7k=; b=VsPc2PweW10ueugP8gQAECYaf877+oEiFd2njqHkCr8NWZDMCPVZTw2RsBw4N4CQvnILA0 EsDiFLlqbgaruagwSH0ozuWg3QWLBSlGNXnA9peUg++nLS6UZWkTxbBr4L7Hea1CRX0Nov OZAzG/PFvEGTAZJynDnq9/eHSmnJUgI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eRLEM7H0; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148021; a=rsa-sha256; cv=none; b=1QdoGUEGqmf8Nmy5hyhWMyVHyc3I+yKRZtyAiHxffJoyhMyCxY+kXak8kpzxgtbjm4+D3V wbeOXRAIYMavBxFGlViJiRfdjgBPgl/2YVIoOh4rxcrXpyswoPOZj7gL7S3z6AgdzhD8pq xzaGrUweQVBmeu681Ja5T3YkBim5L8o= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 25A1918004E Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eRLEM7H0; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 4memm4sd85au5hdos7a8b44w83csop7h X-HE-Tag: 1662148021-523934 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: shmem_getpage() is being replaced by shmem_get_folio() so use a folio throughout this function. Saves several calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/userfaultfd.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 7327b2573f7c..9c035be2148b 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -243,20 +243,22 @@ static int mcontinue_atomic_pte(struct mm_struct *dst_mm, { struct inode *inode = file_inode(dst_vma->vm_file); pgoff_t pgoff = linear_page_index(dst_vma, dst_addr); + struct folio *folio; struct page *page; int ret; - ret = shmem_getpage(inode, pgoff, &page, SGP_NOALLOC); - /* Our caller expects us to return -EFAULT if we failed to find page. */ + ret = shmem_get_folio(inode, pgoff, &folio, SGP_NOALLOC); + /* Our caller expects us to return -EFAULT if we failed to find folio */ if (ret == -ENOENT) ret = -EFAULT; if (ret) goto out; - if (!page) { + if (!folio) { ret = -EFAULT; goto out; } + page = folio_file_page(folio, pgoff); if (PageHWPoison(page)) { ret = -EIO; goto out_release; @@ -267,13 +269,13 @@ static int mcontinue_atomic_pte(struct mm_struct *dst_mm, if (ret) goto out_release; - unlock_page(page); + folio_unlock(folio); ret = 0; out: return ret; out_release: - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); goto out; } From patchwork Fri Sep 2 19:46:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964546 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09517C6FA83 for ; Fri, 2 Sep 2022 19:47:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BAB0800F7; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 89CDE800F6; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C4A8800F6; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 32C49800F4 for ; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 18C72407BA for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) X-FDA: 79868178684.16.2522E6B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id C0AED4004D for ; Fri, 2 Sep 2022 19:47:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=9NiKz/AgNrTLUig1S7htcFVARNbE9lqPRbDWQiFWth8=; b=OGdXO+WBrVyGfrFBL9/PU/ddAA r0Z2ezGHhb0jzmiTy9KO0YJpEcI90LL1Nh/cULcaoCEtJ6QCbNdru3BE2vqncln6uMMqKmiYe2RXl GS7thNNJZDiyEHqc8Ypk3Xu+rBtsMLrxs3XeBL65pL0Ae4lYXYhpjVzvqUY4IhjR8FCkWTvWB4lFs 4/0ovxWBTnSpVu30uYNYRJr/H+XOHbKVbCS6PH1VSFJXOo98Pl/gaAxANr7CFsTyYN2IS5nMB1bDi ihqw0qF188Cjl+/jLH3fDQgQPbGqKiPaSaN4yyDchrMaJf6Pd9G+uxuxteqNQJ1pPNkMh/XWvstmQ Cpe5KVXQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd3-007Iea-Dt; Fri, 02 Sep 2022 19:47:01 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 33/57] shmem: Remove shmem_getpage() Date: Fri, 2 Sep 2022 20:46:29 +0100 Message-Id: <20220902194653.1739778-34-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148021; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9NiKz/AgNrTLUig1S7htcFVARNbE9lqPRbDWQiFWth8=; b=Mz4BSldWtK9D6c7Q82QNDiedRibrWGMAp7/mJhgafLrTGNzBeM2Ww19uA9RL1jLJdA/QNH HYmAUZPi3b3pToc4sysNEt0E0+syXYBpVo66X/oUZbaK/l7xd9HrEglxImlivI2e9ASrzF W5FC8sOPm9w9OuucLRc2W47A/KHddTQ= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=OGdXO+WB; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148021; a=rsa-sha256; cv=none; b=mU3wAM8xPuAR6oVxbKFxMEpBDm9WHr08IVrZOqw80BbN+NJyA2vw4WpW9Rb93vpFE3jyi7 Hf2TJ02QzMyEFmMuPWDQcFCj8H/9RSkDbnUY+N88xVQW4wAGc7HmGjqZh+hyE8ub/vV80e lbka/Oxzot8h88Ic9aFeOfOGHZth/bg= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C0AED4004D Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=OGdXO+WB; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: bg1qdrzh6gau9xekcq6u8b3yoac95nip X-HE-Tag: 1662148021-251117 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With all callers removed, remove this wrapper function. The flags are now mysteriously called SGP, but I think we can live with that. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/shmem_fs.h | 4 +--- mm/shmem.c | 15 +-------------- 2 files changed, 2 insertions(+), 17 deletions(-) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index f4bd50b08a91..f24071e3c826 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -102,7 +102,7 @@ extern unsigned long shmem_swap_usage(struct vm_area_struct *vma); extern unsigned long shmem_partial_swap_usage(struct address_space *mapping, pgoff_t start, pgoff_t end); -/* Flag allocation requirements to shmem_getpage */ +/* Flag allocation requirements to shmem_get_folio */ enum sgp_type { SGP_READ, /* don't exceed i_size, don't allocate page */ SGP_NOALLOC, /* similar, but fail on hole or use fallocated page */ @@ -111,8 +111,6 @@ enum sgp_type { SGP_FALLOC, /* like SGP_WRITE, but make existing page Uptodate */ }; -extern int shmem_getpage(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp); int shmem_get_folio(struct inode *inode, pgoff_t index, struct folio **foliop, enum sgp_type sgp); diff --git a/mm/shmem.c b/mm/shmem.c index e89a11bf83f6..99b7341bd0bf 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -179,7 +179,7 @@ static inline int shmem_reacct_size(unsigned long flags, /* * ... whereas tmpfs objects are accounted incrementally as * pages are allocated, in order to allow large sparse files. - * shmem_getpage reports shmem_acct_block failure as -ENOSPC not -ENOMEM, + * shmem_get_folio reports shmem_acct_block failure as -ENOSPC not -ENOMEM, * so that a failure on a sparse tmpfs mapping will give SIGBUS not OOM. */ static inline int shmem_acct_block(unsigned long flags, long pages) @@ -2031,19 +2031,6 @@ int shmem_get_folio(struct inode *inode, pgoff_t index, struct folio **foliop, mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL); } -int shmem_getpage(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp) -{ - struct folio *folio = NULL; - int ret = shmem_get_folio(inode, index, &folio, sgp); - - if (folio) - *pagep = folio_file_page(folio, index); - else - *pagep = NULL; - return ret; -} - /* * This is like autoremove_wake_function, but it removes the wait queue * entry unconditionally - even if something else had already woken the From patchwork Fri Sep 2 19:46:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86F77ECAAD5 for ; Fri, 2 Sep 2022 19:47:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D0EB2800FB; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BC796800F8; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A210800F5; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 43805800F1 for ; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 20E06120B36 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) X-FDA: 79868178684.09.B6D29D6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id CAC96100059 for ; Fri, 2 Sep 2022 19:47:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=pI4aU5Niep/H4leC+h0AKItoqsaUlWZ93voTFp62Ql4=; b=MnuPhHFGfeVm8RGpUVPQtOCgYs EGtBYV61/vhHOBAuoS71So3Fc8dqa91OtxEFr7NV71tDPPdie6iNpn/Gt+W4SK+na80ei3YzLhUda TZkbmD6tqp9r2zm5F/CIyHcs+Insm7prZQLGoNRQm6rHDuqaq6FzCgocWZUgp4QbXHwJQOvqlftOr 3ES9BauA1gtRLiFAScqd0wEfjOuXgM2OVFmAMfXoT1bwlIukkEybPgMKHvd+LlvoYowu9FdPQHf1O GO8NaT/fKe98JwLA8QFVPlF2X/qFShvT5MhwRkH9Vo5CcOPHoXAIXohxPJwvX/uHQ0IKoEglYyK0Y NugeZJTA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd3-007Iel-Kc; Fri, 02 Sep 2022 19:47:01 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 34/57] swapfile: Convert try_to_unuse() to use a folio Date: Fri, 2 Sep 2022 20:46:30 +0100 Message-Id: <20220902194653.1739778-35-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148021; a=rsa-sha256; cv=none; b=Yg6x8ZTxjkxejgLdk9mvpTiPc7H91c8tMCYNJNDGqwg3/7xxfFwaBrt2FpmDSFNPLpNkqP KVQhBMfKjfti5X3+Nj/m+Fi+x7RConPOIUvyuyxdFT2dBFRUZ3WTMYVoRSpzbkhvhvKRwG 0aHV7L1Zrm7Lp0xXakgRWRE84JEbS2s= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=MnuPhHFG; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148021; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pI4aU5Niep/H4leC+h0AKItoqsaUlWZ93voTFp62Ql4=; b=5AD+g3cOKugN/3hIg2VyKtVfXcvf5hlTjSMJDE2fjDpiARxxzKtR13SVGEGrC7jrKOiU7x 9eMG30gPrLvrxtwenOSCYeog2XMEXcqfPEBFfUVrqWfVk1cQd+YCx5Cqkmf49+ujA+VIhg ZEBMjfZzPU1khxB7Szlgkwtq2hiWEFQ= X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: CAC96100059 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=MnuPhHFG; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: umoagko3984yn87h33srkgfd7hkadqx7 X-HE-Tag: 1662148021-856323 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves five calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/swapfile.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 186511a8ef4f..df85eb73f34e 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2026,7 +2026,7 @@ static int try_to_unuse(unsigned int type) struct list_head *p; int retval = 0; struct swap_info_struct *si = swap_info[type]; - struct page *page; + struct folio *folio; swp_entry_t entry; unsigned int i; @@ -2076,21 +2076,21 @@ static int try_to_unuse(unsigned int type) (i = find_next_to_unuse(si, i)) != 0) { entry = swp_entry(type, i); - page = find_get_page(swap_address_space(entry), i); - if (!page) + folio = filemap_get_folio(swap_address_space(entry), i); + if (!folio) continue; /* - * It is conceivable that a racing task removed this page from - * swap cache just before we acquired the page lock. The page + * It is conceivable that a racing task removed this folio from + * swap cache just before we acquired the page lock. The folio * might even be back in swap cache on another swap area. But - * that is okay, try_to_free_swap() only removes stale pages. + * that is okay, folio_free_swap() only removes stale folios. */ - lock_page(page); - wait_on_page_writeback(page); - try_to_free_swap(page); - unlock_page(page); - put_page(page); + folio_lock(folio); + folio_wait_writeback(folio); + folio_free_swap(folio); + folio_unlock(folio); + folio_put(folio); } /* From patchwork Fri Sep 2 19:46:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 714B2C6FA82 for ; Fri, 2 Sep 2022 19:47:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CCAF800F9; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DBAF800FF; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1C79800F4; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5FC63800F4 for ; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3C2F71607F7 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) X-FDA: 79868178684.01.44B5E02 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id EB9BB4005B for ; Fri, 2 Sep 2022 19:47:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=YNJmY6dWypJzW3KXnsYA0Vz2JLUMi482l/ybv6oKdxY=; b=Zl3Qvatp+5zZG3ZJhwBRrDLIOc HaiYjYW6uQrDP/AnMnZtE1WR3Ys+n1xLx7YJW5nOweKo1fYVVmTTW2yB9eVKh0Kc+BHQAeVkbbezV jGbTWi1g19JYqoes1f4ffBF0eEPBhYomdOPzgouq94KsyoJGiQbWSAE03TdKhzDchZgG+KRLYW3e4 q0UYQOlsAbXKNiNrEtuYUnZVVQhf+cYZBT2rEb6M9tQvZ/LX3/9qM6iqZsOnOjCHRH/dvDt0/0hwk DFwMfSw4UrV1sa4fNgIAGxP0ix3vWja6k+jdBi9GYxgfeL0RY9Zq0qhVMYBt4WisgxDhK/G81PDae ZHd3ODlA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd3-007Iev-P3; Fri, 02 Sep 2022 19:47:01 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 35/57] swapfile: Convert __try_to_reclaim_swap() to use a folio Date: Fri, 2 Sep 2022 20:46:31 +0100 Message-Id: <20220902194653.1739778-36-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148021; a=rsa-sha256; cv=none; b=qYMOSxoP8PEaj53aUAZSu/4Us8VBmZTP4VDesszApwNOAOUZFtggq5FTi6wOTEgPVFT6+u wK0faiDwoCVj/dT92+jEP/CuKd7POOQ0WDIpPSj+CFukzdKz65KTof20fffBonQrjWCo7z 3au60/ntzVSCwly01PrlJq15TthZ3d0= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Zl3Qvatp; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148021; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YNJmY6dWypJzW3KXnsYA0Vz2JLUMi482l/ybv6oKdxY=; b=AaED36cIcL5XwtIm6gVKgl3+vioVAvEnqPwkF0aJ4OzMw9/h4avfWvU/wkLj3Oqwcf9z/v 1ZJx0NGXkmRNXoIJHbeibm6PhrEk+2iqR4Dahx5c7gl/j9IIw4WNbc0EjOXqJTUuYuYd0S 3oOsvMcNNWo6O/XZALaKLIEIlBer3WI= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: EB9BB4005B Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Zl3Qvatp; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: tacf9g1zyhxtcsuu5j9nmwf5jrat7m9o X-HE-Tag: 1662148021-903387 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves five calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/swapfile.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index df85eb73f34e..ce538c3f9161 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -128,27 +128,27 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si, unsigned long offset, unsigned long flags) { swp_entry_t entry = swp_entry(si->type, offset); - struct page *page; + struct folio *folio; int ret = 0; - page = find_get_page(swap_address_space(entry), offset); - if (!page) + folio = filemap_get_folio(swap_address_space(entry), offset); + if (!folio) return 0; /* * When this function is called from scan_swap_map_slots() and it's - * called by vmscan.c at reclaiming pages. So, we hold a lock on a page, + * called by vmscan.c at reclaiming folios. So we hold a folio lock * here. We have to use trylock for avoiding deadlock. This is a special - * case and you should use try_to_free_swap() with explicit lock_page() + * case and you should use folio_free_swap() with explicit folio_lock() * in usual operations. */ - if (trylock_page(page)) { + if (folio_trylock(folio)) { if ((flags & TTRS_ANYWAY) || - ((flags & TTRS_UNMAPPED) && !page_mapped(page)) || - ((flags & TTRS_FULL) && mem_cgroup_swap_full(page))) - ret = try_to_free_swap(page); - unlock_page(page); + ((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) || + ((flags & TTRS_FULL) && mem_cgroup_swap_full(&folio->page))) + ret = folio_free_swap(folio); + folio_unlock(folio); } - put_page(page); + folio_put(folio); return ret; } From patchwork Fri Sep 2 19:46:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64917C6FA82 for ; Fri, 2 Sep 2022 19:48:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BE6E8010A; Fri, 2 Sep 2022 15:47:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E14E380118; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BEF6C80119; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A715480118 for ; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 897C2A09BE for ; Fri, 2 Sep 2022 19:47:08 +0000 (UTC) X-FDA: 79868178936.08.D309174 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 4A2DE18004D for ; Fri, 2 Sep 2022 19:47:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=RwARIESPVJRvPjdBwRi4bFYQLWzQ2fXgslMpG4V504o=; b=AKs8AOvBwDG8jU9p6FGeL2KKt5 S7UWiJ6nzMzsQBbSKk6npx3TRb08F1HNjKO2mRwxAhOyYFmRtVmae642IpW7m+VtLl31KBd4iZQQy o0GbahNSjnvCdE2uKKAiFpQPy6f4HzryFwr94iYyufAFJNHIsHdJlvm0TodHP2A7KDyMkGowwzqpP dBngILO0N4ONkFWT/wdRvlMQt82GLNpQhQFbfkELlWyvjKJoA3546qq6AVGjkDLfHEkDMhrRLZ3lw NtZ7O2CK41+7pkxJEC165DVsBjHtaeEPbG80rAIMsYtVLAK9O/0ElKMYn/63ivJ54kmgFFeMZMLBK G4CemPDA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd3-007If1-Sp; Fri, 02 Sep 2022 19:47:01 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 36/57] swapfile: Convert unuse_pte_range() to use a folio Date: Fri, 2 Sep 2022 20:46:32 +0100 Message-Id: <20220902194653.1739778-37-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148028; a=rsa-sha256; cv=none; b=mxOwWYOtroXWdU3boScQRx+efl8BRAcFIU4gP1STQQlpxz4R1Xt5OBCU3GSxGu3PQA4Xhe zR2BJDxtN5OKtAbCLp1FQSBxDbcK7yrfA8x9sJ0Zz99wmArSxWGJOCJxd1tIxBNUjn086e GQsa0Imqr2eIIVAq4y8/L9qusNDElHA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AKs8AOvB; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148028; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RwARIESPVJRvPjdBwRi4bFYQLWzQ2fXgslMpG4V504o=; b=zBneueNUHG1Qa6V2v9r5tDVdYuREJ4DtetooA4K3EBLgQCTjAQ+TzzlYPqeGzjMDK3rN8/ kZzEv7jBbIQJdisol9g4q5ww0nTx5DzvNSioeih3h+bptkujytBU4HHBk2QpcGOhK68pU/ y6xXkMGVdplUrYtRlCRY/UQkbb6o/pw= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4A2DE18004D Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AKs8AOvB; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: i1p5hrt4zs6as51qacbpuw7hdkd3czfb X-HE-Tag: 1662148028-862213 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Delay fetching the precise page from the folio until we're in unuse_pte(). Saves many calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/swapfile.c | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index ce538c3f9161..9ee42a12cffc 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1754,8 +1754,9 @@ static inline int pte_same_as_swp(pte_t pte, pte_t swp_pte) * force COW, vm_page_prot omits write permission from any private vma. */ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long addr, swp_entry_t entry, struct page *page) + unsigned long addr, swp_entry_t entry, struct folio *folio) { + struct page *page = folio_file_page(folio, swp_offset(entry)); struct page *swapcache; spinlock_t *ptl; pte_t *pte, new_pte; @@ -1827,17 +1828,18 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, unsigned int type) { - struct page *page; swp_entry_t entry; pte_t *pte; struct swap_info_struct *si; - unsigned long offset; int ret = 0; volatile unsigned char *swap_map; si = swap_info[type]; pte = pte_offset_map(pmd, addr); do { + struct folio *folio; + unsigned long offset; + if (!is_swap_pte(*pte)) continue; @@ -1848,8 +1850,9 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, offset = swp_offset(entry); pte_unmap(pte); swap_map = &si->swap_map[offset]; - page = lookup_swap_cache(entry, vma, addr); - if (!page) { + folio = swap_cache_get_folio(entry, vma, addr); + if (!folio) { + struct page *page; struct vm_fault vmf = { .vma = vma, .address = addr, @@ -1859,25 +1862,27 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, &vmf); + if (page) + folio = page_folio(page); } - if (!page) { + if (!folio) { if (*swap_map == 0 || *swap_map == SWAP_MAP_BAD) goto try_next; return -ENOMEM; } - lock_page(page); - wait_on_page_writeback(page); - ret = unuse_pte(vma, pmd, addr, entry, page); + folio_lock(folio); + folio_wait_writeback(folio); + ret = unuse_pte(vma, pmd, addr, entry, folio); if (ret < 0) { - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); goto out; } - try_to_free_swap(page); - unlock_page(page); - put_page(page); + folio_free_swap(folio); + folio_unlock(folio); + folio_put(folio); try_next: pte = pte_offset_map(pmd, addr); } while (pte++, addr += PAGE_SIZE, addr != end); From patchwork Fri Sep 2 19:46:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60AB6C6FA82 for ; Fri, 2 Sep 2022 19:47:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22FB88010F; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75C75800FE; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0119F800FA; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 340B8800FE for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0B3A7801EA for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.14.637A1EF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id B13B61C005E for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=pKrWJop16v+8v5/5KOWLWY4vzs1AU2fWJKOYcokBVuY=; b=TVm2RvvSb+J+BZx//mMrfjxZHp kXiOPsLEhNQ3zmNm8Y9bmL9/6PqM8bg+YajBT5VJNXnxrStmvVYb5BwLt1wJ9ii/kMiuvOufRBMuf +LX+HceA0klTEXvQtrwdsphoo0a9BWxzUfacf8qiH9i6Kq4E5Zhufl74GueRNTFhRNQhch9B1z0FZ 8gXafICLiUw8gb/CIUA5rRIzpfSDWq1pZMbFAOacHyd8kfADVpy4nTxsY+PXvuPpSnxxfiK2Od9MT bAWroP3RR5xVYAZ6noBTD5WkALmXyXSXgVdIQt42OzqrXQcKfDbk9YsPWldGoZ9SIaT8C5atUDPSS iKUcve4w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd4-007If7-0l; Fri, 02 Sep 2022 19:47:02 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 37/57] mm: Convert do_swap_page() to use swap_cache_get_folio() Date: Fri, 2 Sep 2022 20:46:33 +0100 Message-Id: <20220902194653.1739778-38-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148022; a=rsa-sha256; cv=none; b=13C2LMDkjz7sTMg/JPK+5TITZd8aDye4PQJKghR2ouDAJdatok1iZ8zuc3uQjlpOWYy+nc y9yIfsLLLhPQ0dsUEplBF96RpLQVJE8H79ybTUTY2Acv505nOn9E/tDCAnJbMsfdiPG1OO NoTkM5Ox7E7eDkBahCu5olKYdsCbbtw= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TVm2RvvS; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pKrWJop16v+8v5/5KOWLWY4vzs1AU2fWJKOYcokBVuY=; b=z0huPrcfMfNPiHNMc7rXzc/1JJwSTgpREX7YaXlXNtjN7cU6V/pGROWqO+4ux5pfy2Qpm5 gK1qwZSNnP2rSAAY29I5U44dc8slcip1aouvOZJEfzaIZDAboM+/txGpCKUu1qm6wu9GMh DypP7l5jTSnG29Fd+j3xSft3hvDp5pc= X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: B13B61C005E Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TVm2RvvS; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: zip48imx8dhurxgfwujsqij6gj9o9oty X-HE-Tag: 1662148022-557376 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves a folio->page->folio conversion. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 8fefb24567da..10834effb32d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3761,9 +3761,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (unlikely(!si)) goto out; - page = lookup_swap_cache(entry, vma, vmf->address); - if (page) - folio = page_folio(page); + folio = swap_cache_get_folio(entry, vma, vmf->address); + if (folio) + page = folio_file_page(folio, swp_offset(entry)); swapcache = folio; if (!folio) { From patchwork Fri Sep 2 19:46:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33E04C6FA83 for ; Fri, 2 Sep 2022 19:47:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 519A580112; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D85680104; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 25AD0800FE; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4AE61800FD for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2734C804AD for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.02.978FCA9 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id DD0E340059 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=wIlw5FNX3gdHLQFFPi6WZtVvC9jRyLN+M3rxcAEhghc=; b=j2151Z5u4kpjO46DO/6pVoo30m E4FLgDOSdy0zT+KesC1rPhtRwg1AoqYNVwfZDLAy7rG5KXjV3NjlZo8jf6qrSZN9pft9zg1Hu5bHF cZcvZYqXk2Uw37GqKG7k38N8iAibggdrUq/0MAwJqO6sVXJZv9LWbE4KyjHoDZVYbNOZAj5U4ipd2 LvhUJtk1IQgo0/HqSDLJvcMwI++QUPHEgaNpkPNLunj6Tfi/OCSI4uPPKJICQjQ4lMISHBDkxDC8o 7PAN99EnVpIEAEeMDbZQHV0j4/1HPxFzxjUN2r6eShxA8ceLNndd4AXPYO9pLo7FXqTr49rbL65g0 Lb9EMs7w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd4-007IfE-4f; Fri, 02 Sep 2022 19:47:02 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 38/57] mm: Remove lookup_swap_cache() Date: Fri, 2 Sep 2022 20:46:34 +0100 Message-Id: <20220902194653.1739778-39-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148022; a=rsa-sha256; cv=none; b=LoAxCh1SEVhrC5CJRHOOFP71DR1ZlIby4odlMUChfo9zkl+zi/cMNJ4St54FWGEwZSqLrN Zjfi0FQjviiVU/2NYbc80renI77+ZPWknfXuwzplFEoHJqija7Z37OYWRHYkzsR3YUEoTz S91a2lX1iYUfPmqcZWj1E/pksO1PaMw= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=j2151Z5u; dmarc=none; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wIlw5FNX3gdHLQFFPi6WZtVvC9jRyLN+M3rxcAEhghc=; b=Km5rrjjCT+gcxC1zaxvGkALJdJJoCpVwyQfnMUXm2GU8GDZmnKzxiJwIf+4DQe6Wk5PS/h ckHYGEQX4ytPgg9UDS9Sa785lVwhlMzlMaGhH2IKCcxvmqMJst/wVSLmArnbxTfWwm+mlj 66koG1jhH3Bg3DkzeThX3J/g9S2wI0Y= X-Stat-Signature: hqshskpe3krig8twe3g77buaz1hdggxg X-Rspamd-Queue-Id: DD0E340059 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=j2151Z5u; dmarc=none; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1662148022-363740 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All callers have now been converted to swap_cache_get_folio(), so we can remove this wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memcontrol.c | 2 +- mm/swap.h | 10 ---------- mm/swap_state.c | 12 +----------- 3 files changed, 2 insertions(+), 22 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f65833efe90d..3b0698faf1ee 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5561,7 +5561,7 @@ static struct page *mc_handle_swap_pte(struct vm_area_struct *vma, return NULL; /* - * Because lookup_swap_cache() updates some statistics counter, + * Because swap_cache_get_folio() updates some statistics counter, * we call find_get_page() with swapper_space directly. */ page = find_get_page(swap_address_space(ent), swp_offset(ent)); diff --git a/mm/swap.h b/mm/swap.h index f70ff34dab82..9551d7815c37 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -43,9 +43,6 @@ void clear_shadow_from_swap_cache(int type, unsigned long begin, unsigned long end); struct folio *swap_cache_get_folio(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr); -struct page *lookup_swap_cache(swp_entry_t entry, - struct vm_area_struct *vma, - unsigned long addr); struct page *find_get_incore_page(struct address_space *mapping, pgoff_t index); struct page *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, @@ -103,13 +100,6 @@ static inline int swap_writepage(struct page *p, struct writeback_control *wbc) return 0; } -static inline struct page *lookup_swap_cache(swp_entry_t swp, - struct vm_area_struct *vma, - unsigned long addr) -{ - return NULL; -} - static inline struct page *find_get_incore_page(struct address_space *mapping, pgoff_t index) { diff --git a/mm/swap_state.c b/mm/swap_state.c index b96bf4ec8b5b..4af135a7b53c 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -369,16 +369,6 @@ struct folio *swap_cache_get_folio(swp_entry_t entry, return folio; } -struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, - unsigned long addr) -{ - struct folio *folio = swap_cache_get_folio(entry, vma, addr); - - if (!folio) - return NULL; - return folio_file_page(folio, swp_offset(entry)); -} - /** * find_get_incore_page - Find and get a page from the page or swap caches. * @mapping: The address_space to search. @@ -430,7 +420,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, int err; /* * First check the swap cache. Since this is normally - * called after lookup_swap_cache() failed, re-calling + * called after swap_cache_get_folio() failed, re-calling * that would confuse statistics. */ si = get_swap_device(entry); From patchwork Fri Sep 2 19:46:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46D7EECAAD5 for ; Fri, 2 Sep 2022 19:47:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 021FA80100; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 736A4800FC; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22B07800F4; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CFECB800F6 for ; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B37951607F7 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) X-FDA: 79868178684.29.A6F273D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 682EB100059 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=6A6aPUSwkycrtSCt4V834moUq4uc1T4cliZhXfXil0c=; b=UkXmzrIoTuM0mbaCiqDp693GfM Fz8en0GmWP1YWqjBwToWD5AsIf+8sEt6PaNfVHUw2k3nSzQ7a/BawiiI7oKJy1YZa0NiyyxkRIjbk ira7N6gnAKdP0PjDzZY+cjN8JZGpASqBeIs7JfUriCCPFUy5PdS4mSoIXtmzNzd37W/YvegLMvbwD hw9zDnBPa/0RYx2FnAzoe/i3QcYQF/GvQOackiAecxa+O/kD3m6pQR30jdYXwylG8H8XrR6BgfnpB wDYhlkvZjwiL9KngwQouL+SBF+cJrXo/+RCrnbTuUm0m6Z3ILDOQLQLJAQgTnL6SOvM1ecV9DHbBe HnXgoY7w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd4-007IfO-Al; Fri, 02 Sep 2022 19:47:02 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 39/57] swap_state: Convert free_swap_cache() to use a folio Date: Fri, 2 Sep 2022 20:46:35 +0100 Message-Id: <20220902194653.1739778-40-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148022; a=rsa-sha256; cv=none; b=75mqMfUURPf2qzZofmHea+ddayG8pAKFM/VLlvu+zJlle8xafEAGkPm2FJtCxTIciGu8k7 +sqMHfSD1X17dLIhM+vIYZhxczF6rE00gcvpTbs6fUoHqCvQuyoAmu+l+1UGU4W8gfWVC1 AzzAt0Rg98a2dDSk+6CaUOZpmCfS9d0= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=UkXmzrIo; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6A6aPUSwkycrtSCt4V834moUq4uc1T4cliZhXfXil0c=; b=yYulD9t3XQt90MbjJaoaViP9Yil84r8YHEi8yG5oitGTAUHRzU9DpQFT64VElKcvzX4JqR dPBex2fuojqQ4qysWBA2ttfl/seM2SBfUHcH/fRw59VzWHiOwgOnVhuoXVyKA5RrgtdNtZ kZRfFPmw/bMYsXcdcXTFgbaj4N/DOH0= X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 682EB100059 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=UkXmzrIo; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 4te7kcc1tgmh8y3yr5z8p4w44hhwu7c9 X-HE-Tag: 1662148022-130294 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves several calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap_state.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index 4af135a7b53c..438d0676c5be 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -272,16 +272,19 @@ void clear_shadow_from_swap_cache(int type, unsigned long begin, /* * If we are the only user, then try to free up the swap cache. * - * Its ok to check for PageSwapCache without the page lock + * Its ok to check the swapcache flag without the folio lock * here because we are going to recheck again inside - * try_to_free_swap() _with_ the lock. + * folio_free_swap() _with_ the lock. * - Marcelo */ void free_swap_cache(struct page *page) { - if (PageSwapCache(page) && !page_mapped(page) && trylock_page(page)) { - try_to_free_swap(page); - unlock_page(page); + struct folio *folio = page_folio(page); + + if (folio_test_swapcache(folio) && !folio_mapped(folio) && + folio_trylock(folio)) { + folio_free_swap(folio); + folio_unlock(folio); } } From patchwork Fri Sep 2 19:46:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1736ECAAD5 for ; Fri, 2 Sep 2022 19:47:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3A73800F5; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D17F800FC; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D799C80109; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2CBA7800F8 for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 04C61140866 for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.10.B3D0221 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id B2B9C140058 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rTpYsQFP2H5ZjV6RoJPpO4sBbPRbz8lr5sbkwA+Wh7o=; b=g6k2kScRm9qwjFJ3F6qn+6SxM3 Zp+IRVOoMvuWhqxDD/03gHDI19Zw5wyHy94BGsfvhB5aWAt5muZRl/bfEHYLasydiA8bvb2/IHi8O RMYz6LC5VeW569BABzBrUeNiqs4tv2yMDwtpOHYZBF5TsxJNyzlQhj24BrKH901CtPOQ+c1NwFwjD vOyladBkHQaVkeQDt+7fE6nsHg4a+pqHlczcPFZYDZwqBp5k0VhZ6gygds3XGKghfmr2kAnwQYjZb W8zKM5AZAEj1EM9nS+MaGZM1Vvh/8QEVZK6VYmLm4tlocaPV080NpdZ/whZMkpJuKrbLba6eQtxHZ Clj8cViA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd4-007IfV-Es; Fri, 02 Sep 2022 19:47:02 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 40/57] swap: Convert swap_writepage() to use a folio Date: Fri, 2 Sep 2022 20:46:36 +0100 Message-Id: <20220902194653.1739778-41-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148022; a=rsa-sha256; cv=none; b=Jzg61I2bfibmLREDwPpzN4bBDdr4ZlHXsvXtngfGBDim3X8Kj++XoduY0HpTnfpnvS3RYE mZRI2JVlsfrR/Z7KC48WesuuU4yycIpuhQEZP93KNd2A9EqVon4nSESRvl9wiNTWECVjUu th/E+bcV6Mu+onJTfsIQc5/fIc4DL2k= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=g6k2kScR; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rTpYsQFP2H5ZjV6RoJPpO4sBbPRbz8lr5sbkwA+Wh7o=; b=zItU7onxzvZWNFFPv6bW8Cx/tWLrtkbo3VqMT419bJ8HxMAcOhH1YMugc5tVwTnbsZAJi2 5MyTcc7qwiCpR0pB4hs72ubcN601/QvHvwKaFfJx3bGCDHrit/KUPdk/69DDtRI9ZYCwqG 1XC31nfF4E4NeJBFCyiTa5v0eckWFxc= Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=g6k2kScR; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: hw3rbhaq97en6igf8n5oggqxh38rk585 X-Rspamd-Queue-Id: B2B9C140058 X-Rspamd-Server: rspam04 X-Rspam-User: X-HE-Tag: 1662148022-310074 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes many calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_io.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index 68318134dc92..2a3e3e0a6497 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -180,29 +180,30 @@ int generic_swapfile_activate(struct swap_info_struct *sis, */ int swap_writepage(struct page *page, struct writeback_control *wbc) { + struct folio *folio = page_folio(page); int ret = 0; - if (try_to_free_swap(page)) { - unlock_page(page); + if (folio_free_swap(folio)) { + folio_unlock(folio); goto out; } /* * Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret = arch_prepare_to_swap(page); + ret = arch_prepare_to_swap(&folio->page); if (ret) { - set_page_dirty(page); - unlock_page(page); + folio_mark_dirty(folio); + folio_unlock(folio); goto out; } - if (frontswap_store(page) == 0) { - set_page_writeback(page); - unlock_page(page); - end_page_writeback(page); + if (frontswap_store(&folio->page) == 0) { + folio_start_writeback(folio); + folio_unlock(folio); + folio_end_writeback(folio); goto out; } - ret = __swap_writepage(page, wbc, end_swap_bio_write); + ret = __swap_writepage(&folio->page, wbc, end_swap_bio_write); out: return ret; } From patchwork Fri Sep 2 19:46:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D65BBC6FA82 for ; Fri, 2 Sep 2022 19:47:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7826280102; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A1A7980105; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C0EB800F4; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E3ECB800FC for ; Fri, 2 Sep 2022 15:47:02 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D21F9A121F for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) X-FDA: 79868178684.25.A612EFC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf24.hostedemail.com (Postfix) with ESMTP id 7B67018007B for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=vb5v6SCiW0r2u9p6AY6WdMhWa7Kyh2K4USUZrofwcsc=; b=mOwvg7Pa/OqFYqAQxBjyL2UYo9 p57go/VwNybiz9BbPNEgYhih8D1o8hvAorMJOGsYb0A/LZcGvTnjPHkgt5ivUNTrmfZR1otaGT7PW 6V4oxOhmz+aPCM+UziLDitmHsJPxanHAMJ0saN1EawtIGHAEY3t0SIFrOdmUD8Jy6HDluf8ttw/RX bT99qUg9zICIzdCDGKyFHk6oOZFTujQXrNBSMwFAsOLcY+IP1Ls7J7oQmQc4yUwN2Cm0JUVt8DEE7 euRQiyS05rZ0/mMAsX++MFJPpU61VdfvdcWD4AMmvDYmh65zPYZJJrpVccMpx4CgJhD9SoVDrRezD pn87F23Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd4-007Ifc-IL; Fri, 02 Sep 2022 19:47:02 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 41/57] mm: Convert do_wp_page() to use a folio Date: Fri, 2 Sep 2022 20:46:37 +0100 Message-Id: <20220902194653.1739778-42-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=mOwvg7Pa; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148022; a=rsa-sha256; cv=none; b=cIJ9desLnyJ3TCrxzDhFyU6fQnys9nWoIdQbpAXCLQ6eyCELCV3QxzGCfkSRkOrBrhiEn9 WgSY4ASTPTTWUet6RM2NqmjrcECcg/3HLaT/9XVUPdZ5fZjeoWSCiEhL1Nw7jlv7ICJhaI 8TgE0jI6sjhelrcfcjrZlkcwFxxCpl4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vb5v6SCiW0r2u9p6AY6WdMhWa7Kyh2K4USUZrofwcsc=; b=rW0J1+jeiJrGXT5jcG9upm4MzrAtBMDWQmj5PKjztCPLIJbf/FWtkzB5F+rz4dnY1Y5D3T UEd5D2Cj6E9vG9X1TuG2mFNXYhDFh+41XVJdbNdlgXX6CeAaz/xtCv6TevQrxYegP5L7Oa 04693jBl6emi+RzwbDQVdLtWvspvABE= X-Rspamd-Queue-Id: 7B67018007B X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=mOwvg7Pa; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Stat-Signature: w4fup9hoxptosbitnabisx5qmdasck1g X-HE-Tag: 1662148022-701479 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves many calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 10834effb32d..4ec9827a2d37 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3362,6 +3362,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) { const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; struct vm_area_struct *vma = vmf->vma; + struct folio *folio; VM_BUG_ON(unshare && (vmf->flags & FAULT_FLAG_WRITE)); VM_BUG_ON(!unshare && !(vmf->flags & FAULT_FLAG_WRITE)); @@ -3408,48 +3409,47 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) * Take out anonymous pages first, anonymous shared vmas are * not dirty accountable. */ - if (PageAnon(vmf->page)) { - struct page *page = vmf->page; - + folio = page_folio(vmf->page); + if (folio_test_anon(folio)) { /* * If the page is exclusive to this process we must reuse the * page without further checks. */ - if (PageAnonExclusive(page)) + if (PageAnonExclusive(vmf->page)) goto reuse; /* - * We have to verify under page lock: these early checks are - * just an optimization to avoid locking the page and freeing + * We have to verify under folio lock: these early checks are + * just an optimization to avoid locking the folio and freeing * the swapcache if there is little hope that we can reuse. * - * PageKsm() doesn't necessarily raise the page refcount. + * KSM doesn't necessarily raise the folio refcount. */ - if (PageKsm(page) || page_count(page) > 3) + if (folio_test_ksm(folio) || folio_ref_count(folio) > 3) goto copy; - if (!PageLRU(page)) + if (!folio_test_lru(folio)) /* * Note: We cannot easily detect+handle references from - * remote LRU pagevecs or references to PageLRU() pages. + * remote LRU pagevecs or references to LRU folios. */ lru_add_drain(); - if (page_count(page) > 1 + PageSwapCache(page)) + if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio)) goto copy; - if (!trylock_page(page)) + if (!folio_trylock(folio)) goto copy; - if (PageSwapCache(page)) - try_to_free_swap(page); - if (PageKsm(page) || page_count(page) != 1) { - unlock_page(page); + if (folio_test_swapcache(folio)) + folio_free_swap(folio); + if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) { + folio_unlock(folio); goto copy; } /* - * Ok, we've got the only page reference from our mapping - * and the page is locked, it's dark out, and we're wearing + * Ok, we've got the only folio reference from our mapping + * and the folio is locked, it's dark out, and we're wearing * sunglasses. Hit it. */ - page_move_anon_rmap(page, vma); - unlock_page(page); + page_move_anon_rmap(vmf->page, vma); + folio_unlock(folio); reuse: if (unlikely(unshare)) { pte_unmap_unlock(vmf->pte, vmf->ptl); From patchwork Fri Sep 2 19:46:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964558 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0FF7ECAAD5 for ; Fri, 2 Sep 2022 19:47:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 799A980104; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A8FBB80105; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38B2F8010B; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 78F3D80101 for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 60631801EA for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.07.6E53BD1 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id E2693180057 for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=FT/VSNYzJ0BZ/8MYWflRb8frQ+3dgt9MONPtkWqWVfM=; b=rBg4UXezFrU28rQ2oaOnmVBZo2 8tjBpRCgvFLduED8+P/LYWFa4ZlVLfX6lHfoXhMqyl6WI4OB6Wctfw/GPMzHFw8fAayUaYOz+etyF VvB7nAaqYv3GUEzetSyMKksT2ou+86O4/Xxgqq7eoKt5dWfNXNpD9ijk4mU9zKtPTTZsxeXMzMstq wChkd2x/qDc/9y8C9wbuS70jkR9g+MgwT6xKnyxEfmDSB3aGTMpJ6QH2O2KDw9cmiPbhDF29nJP+Y WX4QpBUQJHP9Uw8Jv1v667KyUromRvc+t2RjtFUTZY4KYXEpJfN036FBeew3yrMIVKJNi5GtZh7hA DcYXvGWw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd4-007Ifi-MS; Fri, 02 Sep 2022 19:47:02 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 42/57] huge_memory: Convert do_huge_pmd_wp_page() to use a folio Date: Fri, 2 Sep 2022 20:46:38 +0100 Message-Id: <20220902194653.1739778-43-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FT/VSNYzJ0BZ/8MYWflRb8frQ+3dgt9MONPtkWqWVfM=; b=G2DtP5dBjw48OkKMUUmxe+KkrJwp2QSkqO63S/fcfH51o6EcHo7cq2sW3luWJOBexl9C2o neO9XGSDRHZbET893+zck9a5KDZwFm51JVtrEIcp/Qm/biF6280glAaTEuf5tcLLlp8aJn psr7Be354hwV2H1bGtFeK3l8TuZvO0U= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rBg4UXez; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148023; a=rsa-sha256; cv=none; b=5f1TadPuev2YRI0W288oHdWqJ/nzLPLU8WxKiRhlJ7WFRSWZfEz8m/mMC4INhY5QVsnPHe k/ojz2vEFOtFgC6w5bIVO7GqDimSmAbThTu2DJq542Y+eJtTaBAveTEjm2JvtkD6xTg7KC o2tFZyMwW8ZJ4y19JzXjdR/+IVXs1d0= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E2693180057 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rBg4UXez; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: pcbxmzsc7fgxqbmp6frmn6cmtir97afg X-HE-Tag: 1662148022-648896 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes many calls to compound_head(). Does not remove the assumption that a folio may not be larger than a PMD. Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e9414ee57c5b..ffbc0412be1b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1307,6 +1307,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) { const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; struct vm_area_struct *vma = vmf->vma; + struct folio *folio; struct page *page; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; pmd_t orig_pmd = vmf->orig_pmd; @@ -1328,46 +1329,48 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) } page = pmd_page(orig_pmd); + folio = page_folio(page); VM_BUG_ON_PAGE(!PageHead(page), page); /* Early check when only holding the PT lock. */ if (PageAnonExclusive(page)) goto reuse; - if (!trylock_page(page)) { - get_page(page); + if (!folio_trylock(folio)) { + folio_get(folio); spin_unlock(vmf->ptl); - lock_page(page); + folio_lock(folio); spin_lock(vmf->ptl); if (unlikely(!pmd_same(*vmf->pmd, orig_pmd))) { spin_unlock(vmf->ptl); - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); return 0; } - put_page(page); + folio_put(folio); } /* Recheck after temporarily dropping the PT lock. */ if (PageAnonExclusive(page)) { - unlock_page(page); + folio_unlock(folio); goto reuse; } /* - * See do_wp_page(): we can only reuse the page exclusively if there are - * no additional references. Note that we always drain the LRU - * pagevecs immediately after adding a THP. + * See do_wp_page(): we can only reuse the folio exclusively if + * there are no additional references. Note that we always drain + * the LRU pagevecs immediately after adding a THP. */ - if (page_count(page) > 1 + PageSwapCache(page) * thp_nr_pages(page)) + if (folio_ref_count(folio) > + 1 + folio_test_swapcache(folio) * folio_nr_pages(folio)) goto unlock_fallback; - if (PageSwapCache(page)) - try_to_free_swap(page); - if (page_count(page) == 1) { + if (folio_test_swapcache(folio)) + folio_free_swap(folio); + if (folio_ref_count(folio) == 1) { pmd_t entry; page_move_anon_rmap(page, vma); - unlock_page(page); + folio_unlock(folio); reuse: if (unlikely(unshare)) { spin_unlock(vmf->ptl); @@ -1382,7 +1385,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) } unlock_fallback: - unlock_page(page); + folio_unlock(folio); spin_unlock(vmf->ptl); fallback: __split_huge_pmd(vma, vmf->pmd, vmf->address, false, NULL); From patchwork Fri Sep 2 19:46:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07207ECAAD5 for ; Fri, 2 Sep 2022 19:47:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3EE680114; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB76D8010A; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95A318010D; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8333780103 for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 676A3C021D for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.13.24B716F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id 260A514005A for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=9mOmmVtZpFIWWo3nfTdSfePzyVXKHrnvdlc0DSY2URw=; b=GyfP+FznKUxvyGDD5TJYfA32kz EiXBytusMRpzctiaQEtAb8i9RPewAmGcf1E8WnijTlixhtyIvnAd9UGzCnnWzyA4mTy87xM6dKVVd 4Vzu6acZs3UdhN+iYiKrl0Y3dHol8U/Yk9oQyiARcRtSoYrZtX0Nak1IeRQu27pLXB64WF5LIsyEF dBbe8ZeHiz/japwjObfOR+1AbnPNnjonnR9GUj/dBwTb2meYgEk6QzMkQOpSIMrptFsca67M6NFGo V5aevBHZC8Sp0+BpwiS8SM2INtQCkJuGISAQcUBrwSXzEbjSnWbbNyrksQwpSIT10YLiQFdRXxIJI ZysyNvfg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd4-007Ifs-Ql; Fri, 02 Sep 2022 19:47:02 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 43/57] madvise: Convert madvise_free_pte_range() to use a folio Date: Fri, 2 Sep 2022 20:46:39 +0100 Message-Id: <20220902194653.1739778-44-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148023; a=rsa-sha256; cv=none; b=kOw2kYuSHTm5/FbdTKw0s5+A47pG8kY+UFTOLrxRLpLcb6kVph3cTcfZ5ZFk0jdwOo34oG IQ7f41XTjzYhgsLg4eqvkRsnsjjBVUtMbk+OAjqd/Svuy0PCKxHDliL85cCtY99uFDGZAl hew6dYQzR70xUlw69iSKRco+YQu6gQk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GyfP+Fzn; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9mOmmVtZpFIWWo3nfTdSfePzyVXKHrnvdlc0DSY2URw=; b=dX5z8+e9wsMO3BQ3rWVxTiUv+Rw0/BbDiyL5Ny8UUjIiYRpJc5hMwxXqCBNMqChk2uLpff CStIHllWvrmdCty7h6DuQh81pmSdfwg2S2Ku4rd4vGzBjk/zHhCmK0eVYYp7cLsfDhgZcO zfu8b4zuMSdqPTHMFkrpiSu8ZeT4/fU= X-Stat-Signature: 6a3zxgjyyfaxenqipuhyor4nyiwedfz3 X-Rspamd-Queue-Id: 260A514005A Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GyfP+Fzn; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1662148022-456877 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves a lot of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/madvise.c | 49 +++++++++++++++++++++++++------------------------ 1 file changed, 25 insertions(+), 24 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 5f0f0948a50e..05de79d269e8 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -597,6 +597,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, struct vm_area_struct *vma = walk->vma; spinlock_t *ptl; pte_t *orig_pte, *pte, ptent; + struct folio *folio; struct page *page; int nr_swap = 0; unsigned long next; @@ -641,56 +642,56 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, page = vm_normal_page(vma, addr, ptent); if (!page || is_zone_device_page(page)) continue; + folio = page_folio(page); /* - * If pmd isn't transhuge but the page is THP and + * If pmd isn't transhuge but the folio is large and * is owned by only this process, split it and * deactivate all pages. */ - if (PageTransCompound(page)) { - if (page_mapcount(page) != 1) + if (folio_test_large(folio)) { + if (folio_mapcount(folio) != 1) goto out; - get_page(page); - if (!trylock_page(page)) { - put_page(page); + folio_get(folio); + if (!folio_trylock(folio)) { + folio_put(folio); goto out; } pte_unmap_unlock(orig_pte, ptl); - if (split_huge_page(page)) { - unlock_page(page); - put_page(page); + if (split_folio(folio)) { + folio_unlock(folio); + folio_put(folio); orig_pte = pte_offset_map_lock(mm, pmd, addr, &ptl); goto out; } - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); orig_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); pte--; addr -= PAGE_SIZE; continue; } - VM_BUG_ON_PAGE(PageTransCompound(page), page); - - if (PageSwapCache(page) || PageDirty(page)) { - if (!trylock_page(page)) + if (folio_test_swapcache(folio) || folio_test_dirty(folio)) { + if (!folio_trylock(folio)) continue; /* - * If page is shared with others, we couldn't clear - * PG_dirty of the page. + * If folio is shared with others, we mustn't clear + * the folio's dirty flag. */ - if (page_mapcount(page) != 1) { - unlock_page(page); + if (folio_mapcount(folio) != 1) { + folio_unlock(folio); continue; } - if (PageSwapCache(page) && !try_to_free_swap(page)) { - unlock_page(page); + if (folio_test_swapcache(folio) && + !folio_free_swap(folio)) { + folio_unlock(folio); continue; } - ClearPageDirty(page); - unlock_page(page); + folio_clear_dirty(folio); + folio_unlock(folio); } if (pte_young(ptent) || pte_dirty(ptent)) { @@ -708,7 +709,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, set_pte_at(mm, addr, pte, ptent); tlb_remove_tlb_entry(tlb, pte, addr); } - mark_page_lazyfree(page); + mark_page_lazyfree(&folio->page); } out: if (nr_swap) { From patchwork Fri Sep 2 19:46:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2192C6FA83 for ; Fri, 2 Sep 2022 19:47:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4E6480115; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BFD0F800FF; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F24680101; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 82A7280102 for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 66EF8120F71 for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.12.468E923 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id E0F354004F for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=cDP+DNYsXXA+kb85wo7Nrqn0aeI4BRiMp8pyQ5b78Q0=; b=ThjhD8mBX3Pa2r2cz/OysqgC3D 71gcuLxPzfDYWlx98/+VZ0eUYAkYMqdmVRyAzibXNHVpveohUG1p3Ol7zteJKmGzPmple0rQcxf9m o5isgoe+TktVq3rCoGSb2iYDbUtb+NpbqYR7w0J2V2Eskojm6SibaUaQ9VquP8RxnFbC3rPzeRCUG F/a1FBlHG4M/DM48zi1mJ0+ohOzLWGQX2g7BHfKoMdJFPQc1i3OD/ZWBWQ0b2udgslmTqaQVYQdc+ q2XaKHG3V7VNDgpzS7lBK8tkZ7mqEi3xDISkwg2JBB1GxocaEEdMHmfqxummjqOhj2CKEomYtFhYD 2OIOYLyQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd4-007Ifz-Ua; Fri, 02 Sep 2022 19:47:03 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 44/57] uprobes: Use folios more widely in __replace_page() Date: Fri, 2 Sep 2022 20:46:40 +0100 Message-Id: <20220902194653.1739778-45-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cDP+DNYsXXA+kb85wo7Nrqn0aeI4BRiMp8pyQ5b78Q0=; b=uTdIOU8l0qCUW97pnuzWnJDRKLa6NG+DiKN/oo1cmvjwTMi944/oES/cAfExV6dgI0B+ke ec30l1MWbJJFj/vk3TIyXbv6t7UhKv9AU9R09nwZUa7MgCJQYxpe59uhZTKA/LASLX5cap DVg1LIeW9b0nx6GCENkJ8r+mjm9HND4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ThjhD8mB; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148023; a=rsa-sha256; cv=none; b=R+YYzrucCIb0O5MbwQo9dSgGmIv+aqE8xoY0DWHve/h7JNhNhECsrJcLp6/0PKM+fbYnlZ nXLjy0SFY6Gw3urVz+VUVn69hTFbIKZk6DoKvfB3pL7PEpXWpuxUXAcxVW6lzIGZ/dw0c8 fydw6FFWrGYK7PCfniXZN2Gf5c99S6k= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E0F354004F Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ThjhD8mB; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: dj1hs1de959y3ribyrzfczra6bozdwgo X-HE-Tag: 1662148022-82009 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove a few hidden calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- kernel/events/uprobes.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 2eaa327f8158..9722c4587c48 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -19,7 +19,7 @@ #include #include /* anon_vma_prepare */ #include /* set_pte_at_notify */ -#include /* try_to_free_swap */ +#include /* folio_free_swap */ #include /* user_enable_single_step */ #include /* notifier mechanism */ #include "../../mm/internal.h" /* munlock_vma_page */ @@ -154,8 +154,9 @@ static loff_t vaddr_to_offset(struct vm_area_struct *vma, unsigned long vaddr) static int __replace_page(struct vm_area_struct *vma, unsigned long addr, struct page *old_page, struct page *new_page) { + struct folio *old_folio = page_folio(old_page); struct mm_struct *mm = vma->vm_mm; - DEFINE_FOLIO_VMA_WALK(pvmw, page_folio(old_page), vma, addr, 0); + DEFINE_FOLIO_VMA_WALK(pvmw, old_folio, vma, addr, 0); int err; struct mmu_notifier_range range; @@ -169,8 +170,8 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, return err; } - /* For try_to_free_swap() below */ - lock_page(old_page); + /* For folio_free_swap() below */ + folio_lock(old_folio); mmu_notifier_invalidate_range_start(&range); err = -EAGAIN; @@ -186,7 +187,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, /* no new page, just dec_mm_counter for old_page */ dec_mm_counter(mm, MM_ANONPAGES); - if (!PageAnon(old_page)) { + if (!folio_test_anon(old_folio)) { dec_mm_counter(mm, mm_counter_file(old_page)); inc_mm_counter(mm, MM_ANONPAGES); } @@ -198,15 +199,15 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mk_pte(new_page, vma->vm_page_prot)); page_remove_rmap(old_page, vma, false); - if (!page_mapped(old_page)) - try_to_free_swap(old_page); + if (!folio_mapped(old_folio)) + folio_free_swap(old_folio); page_vma_mapped_walk_done(&pvmw); - put_page(old_page); + folio_put(old_folio); err = 0; unlock: mmu_notifier_invalidate_range_end(&range); - unlock_page(old_page); + folio_unlock(old_folio); return err; } From patchwork Fri Sep 2 19:46:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C86DBECAAD5 for ; Fri, 2 Sep 2022 19:47:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CF7880116; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 513C2800FE; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F6C78010E; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A7C46800F4 for ; Fri, 2 Sep 2022 15:47:03 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8998A407BA for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) X-FDA: 79868178726.24.DE664D2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf24.hostedemail.com (Postfix) with ESMTP id 1FAE018007B for ; Fri, 2 Sep 2022 19:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=YMk9TEVmL8q7WxjorjEAwIc7bPZKGMaknIib7W5PcUw=; b=gKZiybOdSZrgi7QYB7GzwgeIGq 0YPNvyRD+NGGPb+UZIxg3mGxQ1TI6Dcg8oAICeND9fYoswizmGYbJEHCm3A5CFTKOB0lN/2M+um25 6ApZ97FsS8Vy62YaY3pkaDtiVrfhgzvPc8OikAx6VIAuPpQYxIMDTwAXY/SdQ32P3wWbqdxOV/EgB ZQNzydO/uV5kei4l7/23MmtYd1sq3wECxuGZHCHTqgjugQEG2TpTTxv2Y4Wd0Do2Yay4F3Vk5YI/d 2BLq+HvKWJ415tfEFrWVc88wTvo0plObbYZ6v3bBSKEuwm3I6E4m87hMEQI3RztfNBgGeO/Y1JMHZ oqDT8fZw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd5-007IgM-5N; Fri, 02 Sep 2022 19:47:03 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 45/57] ksm: Use a folio in replace_page() Date: Fri, 2 Sep 2022 20:46:41 +0100 Message-Id: <20220902194653.1739778-46-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=gKZiybOd; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148023; a=rsa-sha256; cv=none; b=EcfvhJCWG6mL0Qz6GZOKTEmgIrl6vrXjb6NnFfdIxvpuT1RMvP2iM1WVK5ZL5VgPXYrBnS S8elCVmfzVZ6MfMvXossWi/zfE45O3VL9kBPu8Pl+xEPh2StUlZhiaYIYhBxfoto7njOlx q6VPk3/K9jWd8dRHTcjU/8EEVma7zwg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YMk9TEVmL8q7WxjorjEAwIc7bPZKGMaknIib7W5PcUw=; b=TOSWw52sbO4wMxHG7nKhlMxZmckxMBLWHmAjr2eLglIEQRPKg+u01CUSe05eo7WaTLUurf 4EtKKgo3Uh8dhmDohWY43V4s278SnrohA/3dzbNKrZ0JPbIFtM3e2Y+hcIcQZ804Stvt71 4HEKsAc8IlbTUwZwk4xPHSb1uMQIMvs= X-Rspamd-Queue-Id: 1FAE018007B X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=gKZiybOd; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Stat-Signature: 8r7ncdfsk4osqos31g7i9mrtx86rfpxc X-HE-Tag: 1662148022-814822 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace three calls to compound_head() with one. Signed-off-by: Matthew Wilcox (Oracle) --- mm/ksm.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 42ab153335a2..322652e7e6fe 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1133,6 +1133,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, struct page *kpage, pte_t orig_pte) { struct mm_struct *mm = vma->vm_mm; + struct folio *folio; pmd_t *pmd; pte_t *ptep; pte_t newpte; @@ -1191,10 +1192,11 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, ptep_clear_flush(vma, addr, ptep); set_pte_at_notify(mm, addr, ptep, newpte); + folio = page_folio(page); page_remove_rmap(page, vma, false); - if (!page_mapped(page)) - try_to_free_swap(page); - put_page(page); + if (!folio_mapped(folio)) + folio_free_swap(folio); + folio_put(folio); pte_unmap_unlock(ptep, ptl); err = 0; From patchwork Fri Sep 2 19:46:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E860ECAAD5 for ; Fri, 2 Sep 2022 19:47:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 03819800FC; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C62A80111; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C91C180116; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2EE10800FD for ; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0EF3D801EA for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) X-FDA: 79868178768.21.835B940 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id 9B9A34004D for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=bcSQc1s9XNVBQkJqTs0sr3KkK5kgMX7Yz0rxH26LS1Y=; b=LbdG8PpX5mVgW9mZTUDOGLVVaa jzKbf9V/BIbzm1qDEnMT+NCUd085sr5JU+xCVdJIj4ovWBQxKVtTp/ONg4haXynkkzaWI/gf9zgIt wqfPizYcUclFnWllBYJjZBvYc+LOC+GS05P2xs1K7GuI4C7GqRFvYegzLxozoInCOQodIH9jibtg1 fLzf5rg+RGYhQFlDor9ryy+hM1hORoD6R2Kxtr9P0Oed6piOC7BfFQpBrts5pvISkaUIhNrxjJr9Q Dxw4+byRhuC+pcHq+w5J3CrxLxlTWDvs/xICk7s2HMeXM7w/6Ep4mAYLBMxz6rsk3AZXC7KxgH0eF 1Ef5Fx7Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd5-007IgX-CA; Fri, 02 Sep 2022 19:47:03 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 46/57] mm: Convert do_swap_page() to use folio_free_swap() Date: Fri, 2 Sep 2022 20:46:42 +0100 Message-Id: <20220902194653.1739778-47-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bcSQc1s9XNVBQkJqTs0sr3KkK5kgMX7Yz0rxH26LS1Y=; b=WdigpfYPHDD78ikeLpHtGC83c26npmTuPfssMkGixIocBAr5/W2kMaWVpC3fD2WGoQBARF RnxKMVJIyhpYSKGP4YiIcQOx8YkBCJVYtvZd9HKfwi7aYT/wuFo+ihX+b3BW/7ctoMNsn6 zgbAsm3ixC7o1srTHHg4z/ZwqDoyKcc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LbdG8PpX; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148023; a=rsa-sha256; cv=none; b=QUh2VBEW3JQazxwSTEcp1ogCK3/PxfewwWXoVkdlqwkJpqJ8OWt19lhUmySdFoH8BRWRGK MGggrA5D2McA18/WR10L+jXaDq7ZKiAckZ4JuN1PwcWHJH1WWWU6zhFQqKa1V4EF/ZLMY7 4Z8QzvvGPZsAVEUX0mNwOtMUC7NnLTQ= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9B9A34004D Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LbdG8PpX; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: xfsmkpqnq4bbfpxie3og8w71ir1sr9oy X-HE-Tag: 1662148023-802996 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Also convert should_try_to_free_swap() to use a folio. This removes a few calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 4ec9827a2d37..f39dc23ddb69 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3635,14 +3635,14 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) return 0; } -static inline bool should_try_to_free_swap(struct page *page, +static inline bool should_try_to_free_swap(struct folio *folio, struct vm_area_struct *vma, unsigned int fault_flags) { - if (!PageSwapCache(page)) + if (!folio_test_swapcache(folio)) return false; - if (mem_cgroup_swap_full(page) || (vma->vm_flags & VM_LOCKED) || - PageMlocked(page)) + if (mem_cgroup_swap_full(&folio->page) || (vma->vm_flags & VM_LOCKED) || + folio_test_mlocked(folio)) return true; /* * If we want to map a page that's in the swapcache writable, we @@ -3650,8 +3650,8 @@ static inline bool should_try_to_free_swap(struct page *page, * user. Try freeing the swapcache to get rid of the swapcache * reference only in case it's likely that we'll be the exlusive user. */ - return (fault_flags & FAULT_FLAG_WRITE) && !PageKsm(page) && - page_count(page) == 2; + return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && + folio_ref_count(folio) == 2; } static vm_fault_t pte_marker_clear(struct vm_fault *vmf) @@ -3943,8 +3943,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * yet. */ swap_free(entry); - if (should_try_to_free_swap(page, vma, vmf->flags)) - try_to_free_swap(page); + if (should_try_to_free_swap(folio, vma, vmf->flags)) + folio_free_swap(folio); inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS); From patchwork Fri Sep 2 19:46:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5876C6FA82 for ; Fri, 2 Sep 2022 19:48:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E41B80107; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ABA198010B; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BCC1800FA; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 58940800FA for ; Fri, 2 Sep 2022 15:47:04 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3729A14064B for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) X-FDA: 79868178768.21.BAA86AE Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id E6C5D1C005F for ; Fri, 2 Sep 2022 19:47:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=7aj2gwVlj5VcsZXUySMc7S7T76WQ2vhifR2A0XWxgfs=; b=Zb+2mXeM2KGyaVFUI57piyTyce HSRh1ZebjjIyMEq2CUwzcFrktdi9Ob/Qm1Y+tOkLQdT2ZfBpRZ2ukzK+woXVOoGSCuvwKhfXBUEi8 VyuTeBzTJMF+FqpdDNUi0ThcFzpXAIzZSgu35otADCo7tRUH5tPBo/XTkB14YwtQTSttmEicJZBNg vzjbXTNrlt03rsXSfAXXeyCXYzuTCVcL13IJThmknzY00gGt8BRAKSdQDOclasjrWP9piSJgB6VJ3 ZW71dYThq39TA0dm624uPoY21lYtxBtKtQYWG62QgaKuaxtU/lmYRJnGeOUtRGZhAG5mkVsP9kExL X9i1/QRA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd5-007Ih5-JV; Fri, 02 Sep 2022 19:47:03 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 47/57] memcg: Convert mem_cgroup_swap_full() to take a folio Date: Fri, 2 Sep 2022 20:46:43 +0100 Message-Id: <20220902194653.1739778-48-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148024; a=rsa-sha256; cv=none; b=Pt13intQp6O44Vaqf3F811PqdPVBLwoh8Z/EC+c+kgKX3+/u4u7QgYFxNmFewky9kcrDDL Tza0/4V92hrqX/s5QSRH7NSatSatzZ6AGONOLg0ky2GTg8hqODMO4P35SWHcAC66xMSy6S DCi8PcDGqa4UajB8Plv0rp+rdtUcY8c= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Zb+2mXeM; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7aj2gwVlj5VcsZXUySMc7S7T76WQ2vhifR2A0XWxgfs=; b=XqpuMS/MhcOKAmJz6O61nSUp8AZ80S340l5ltraM9fa1FfKcjxW8LOf/mP3AqoRi6gm4Ei vShrAsSUHgt+OsHs9PCuQqaof4sDW+umCD2nE9jFJuGZNDpY1Md+qowfhGaI+YswB9Koyr UnyC0N7NVCKmLdVYJ3VBnAv9W6rw72o= X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: E6C5D1C005F Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Zb+2mXeM; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 8scmeuwuuzegxedxbrard5jgk1jj6yqg X-HE-Tag: 1662148023-862588 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All callers now have a folio, so convert the function to take a folio. Saves a couple of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/swap.h | 4 ++-- mm/memcontrol.c | 6 +++--- mm/memory.c | 2 +- mm/swapfile.c | 2 +- mm/vmscan.c | 3 +-- 5 files changed, 8 insertions(+), 9 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index f16c9af6bf32..8178ec471fe3 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -688,7 +688,7 @@ static inline void mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_p } extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg); -extern bool mem_cgroup_swap_full(struct page *page); +extern bool mem_cgroup_swap_full(struct folio *folio); #else static inline void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry) { @@ -710,7 +710,7 @@ static inline long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg) return get_nr_swap_pages(); } -static inline bool mem_cgroup_swap_full(struct page *page) +static inline bool mem_cgroup_swap_full(struct folio *folio) { return vm_swap_full(); } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 3b0698faf1ee..557579f43e7b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7375,18 +7375,18 @@ long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg) return nr_swap_pages; } -bool mem_cgroup_swap_full(struct page *page) +bool mem_cgroup_swap_full(struct folio *folio) { struct mem_cgroup *memcg; - VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); if (vm_swap_full()) return true; if (cgroup_memory_noswap || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) return false; - memcg = page_memcg(page); + memcg = folio_memcg(folio); if (!memcg) return false; diff --git a/mm/memory.c b/mm/memory.c index f39dc23ddb69..3c7b2f1dbf6e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3641,7 +3641,7 @@ static inline bool should_try_to_free_swap(struct folio *folio, { if (!folio_test_swapcache(folio)) return false; - if (mem_cgroup_swap_full(&folio->page) || (vma->vm_flags & VM_LOCKED) || + if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) || folio_test_mlocked(folio)) return true; /* diff --git a/mm/swapfile.c b/mm/swapfile.c index 9ee42a12cffc..74929a8cbf88 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -144,7 +144,7 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si, if (folio_trylock(folio)) { if ((flags & TTRS_ANYWAY) || ((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) || - ((flags & TTRS_FULL) && mem_cgroup_swap_full(&folio->page))) + ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio))) ret = folio_free_swap(folio); folio_unlock(folio); } diff --git a/mm/vmscan.c b/mm/vmscan.c index ac7f6f77e28a..85af57bbfd81 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2002,8 +2002,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, activate_locked: /* Not a candidate for swapping, so reclaim swap space. */ if (folio_test_swapcache(folio) && - (mem_cgroup_swap_full(&folio->page) || - folio_test_mlocked(folio))) + (mem_cgroup_swap_full(folio) || folio_test_mlocked(folio))) folio_free_swap(folio); VM_BUG_ON_FOLIO(folio_test_active(folio), folio); if (!folio_test_mlocked(folio)) { From patchwork Fri Sep 2 19:46:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964584 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 812A2ECAAD5 for ; Fri, 2 Sep 2022 19:48:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 626058011B; Fri, 2 Sep 2022 15:47:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 587EA80119; Fri, 2 Sep 2022 15:47:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DBC28011B; Fri, 2 Sep 2022 15:47:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2BF8F80119 for ; Fri, 2 Sep 2022 15:47:12 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0FADE1209C3 for ; Fri, 2 Sep 2022 19:47:12 +0000 (UTC) X-FDA: 79868179104.04.A267E50 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id 8B03140096 for ; Fri, 2 Sep 2022 19:47:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=mADqW9cScCqLSjZJPxMJC5WEKhPmFXzGFtDTW1Nq/Fw=; b=wHAHz9Hl5lRg1vkc9uiBdzYl3p 2dx0IINv+/WWSWnzzfYfyUysre0ahZ6bwzjqiXHIcIkgNNMZFse/Leo1zUxdgZqk4ApNyZzEIgHg4 hUcy0lToeQhiWT6SfTCy9pU/IH3roptv3ooKve3rkz3aqZAz52fX4uiIHNw4L8dHKKCEZJvgh+YET iCvNSTNt/y8xt7hQFgjLMLHHvQeE8d7tMFK5VW7vS5wZZxhZQNRItovHGn+nLeNGLUoWnnQVKP6HE 7RMH0UDdNLOaDH+273sVPmb51mHlLZ0czrLZFTuXUPRKTPe5fXtwOP0jJLR1VTNR04aeZHeU2v+Iu xaXvPDjA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd5-007IhG-Nl; Fri, 02 Sep 2022 19:47:03 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 48/57] mm: Remove try_to_free_swap() Date: Fri, 2 Sep 2022 20:46:44 +0100 Message-Id: <20220902194653.1739778-49-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148031; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mADqW9cScCqLSjZJPxMJC5WEKhPmFXzGFtDTW1Nq/Fw=; b=qb2xmHK+NDKABYvwLbGCVKPrQei7yEOZ4CgK6zEXglb9FYXLH7+W+yvddmkTb6h/MaK9ZE GZMQduFowZsiJgMVmLwNb7rUjfx19N+k20lqhkLK5GWt/XaUuHHdeQC7FVsyCOXrWlIYRJ IHtsN2JbNRRprw/tmc257UQzpRz1SnM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=wHAHz9Hl; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148031; a=rsa-sha256; cv=none; b=mBTgrhaJZeWa8IwLdEt8oNSilAH3cKGSzJkENWN9aR/ampsRCOQYvpWZe4fY/7FT9j0yLI T/0Qs6hCjcZkz9IEM1LH8MYBWegMWXY4dXhGV5UgpUfJb0Gs82pJ3YNXICCb3j8ivQyzLc ixFzF73stl47UoTXPkwkJ6CUGHM1JdM= X-Stat-Signature: p993ja9s3ywf49qj9r6n8d1i6c8tnf7i X-Rspamd-Queue-Id: 8B03140096 X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=wHAHz9Hl; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam01 X-HE-Tag: 1662148031-409902 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All callers have now been converted to folio_free_swap() and we can remove this wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/swap.h | 6 ------ mm/folio-compat.c | 7 ------- mm/memory.c | 2 +- 3 files changed, 1 insertion(+), 14 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 8178ec471fe3..3be59affca63 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -506,7 +506,6 @@ extern int __swp_swapcount(swp_entry_t entry); extern int swp_swapcount(swp_entry_t entry); extern struct swap_info_struct *page_swap_info(struct page *); extern struct swap_info_struct *swp_swap_info(swp_entry_t entry); -extern int try_to_free_swap(struct page *); struct backing_dev_info; extern int init_swap_address_space(unsigned int type, unsigned long nr_pages); extern void exit_swap_address_space(unsigned int type); @@ -591,11 +590,6 @@ static inline int swp_swapcount(swp_entry_t entry) return 0; } -static inline int try_to_free_swap(struct page *page) -{ - return 0; -} - static inline swp_entry_t folio_alloc_swap(struct folio *folio) { swp_entry_t entry; diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 06d47f00609b..e1e23b4947d7 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -146,10 +146,3 @@ void putback_lru_page(struct page *page) { folio_putback_lru(page_folio(page)); } - -#ifdef CONFIG_SWAP -int try_to_free_swap(struct page *page) -{ - return folio_free_swap(page_folio(page)); -} -#endif diff --git a/mm/memory.c b/mm/memory.c index 3c7b2f1dbf6e..0b5630dba38e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3838,7 +3838,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (swapcache) { /* - * Make sure try_to_free_swap or swapoff did not release the + * Make sure folio_free_swap() or swapoff did not release the * swapcache from under us. The page pin, and pte_same test * below, are not enough to exclude that. Even if it is still * swapcache, we need to check that the page's swap has not From patchwork Fri Sep 2 19:46:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85C1CC6FA82 for ; Fri, 2 Sep 2022 19:48:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA5EC800F4; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 563FF80113; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E1D480117; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 17CD9800F4 for ; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E701A120B36 for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) X-FDA: 79868178768.18.741895D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id 8BA7840044 for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=jffQg3X6ia+ruwQoAslti3TKgRaqtkf3QGIvjwSuigM=; b=iZC2Sq2q58FdU4Xs3fDG1I/TQQ j2F29iqowulLanfN/2JaTeBDuBkI7OyRODH23wSFK7jI/uJ/PRGgtEJG/S57+csrenGCttxa2FDZT IPkfjnMCZ2zmmbI/i9Oo6p4WHqExEci4Y2Zqlrz5r0/2kJ/MEPQHGmVSmc5RnBadWlkKMA/ouBwX6 fDGDoo2DMu4PkQBLQ/kEJrzxvQFWltO/VBQUN42BnD+IQZ9cxMl+7wVopZ1HRqyLfB9IDG5y5laSv wt6v//pjgKXxmEK3xtJSkDxxdkjNbwqS9xrP2x0EcqrQu03bT8PQ8fnLzSS/isNz6h835qCAimtAl ZZ14PwAw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd5-007IhQ-TJ; Fri, 02 Sep 2022 19:47:03 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 49/57] rmap: Convert page_move_anon_rmap() to use a folio Date: Fri, 2 Sep 2022 20:46:45 +0100 Message-Id: <20220902194653.1739778-50-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jffQg3X6ia+ruwQoAslti3TKgRaqtkf3QGIvjwSuigM=; b=wr7+TL8hZnc7E2aR1ypZUf/C5HlcW/m9SzmcDgYGpyJGeZ2gKy7WNWCDP/KlWuJrzOnzLq VudEbelRUeY0lZ8+C0w3oOYHsZNUOCfb3at57PZFPSMPtwkNvCapjU93NgN/VtY7/VS2dx zefIggRfS6rHqlQJeEIiYMAGaCNzmxU= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=iZC2Sq2q; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148024; a=rsa-sha256; cv=none; b=sqxMMtkGOxF+72FeAXbG6N8Bz1op3rWjQB1wUpUPZKCu3dA/6uoUGlN0bCFneoi5+deeoD x6Wd2ACgWtkR8iLVntzJqERwPN+OdNVx0VGF7dkAVk+A+Jwj0qxO4uY7TxhEpFxox/3iIw 9x9MAzdzDSTMW1eOb6AlAYO70GEUyI8= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8BA7840044 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=iZC2Sq2q; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 9qo46uco9q5cpu4zopqsbjriwhzog7pu X-HE-Tag: 1662148024-152661 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes one call to compound_head() and a reference to page->mapping. Signed-off-by: Matthew Wilcox (Oracle) --- mm/rmap.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 93d5a6f793d2..76ebfc8beb7e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1101,22 +1101,20 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, */ void page_move_anon_rmap(struct page *page, struct vm_area_struct *vma) { - struct anon_vma *anon_vma = vma->anon_vma; - struct page *subpage = page; - - page = compound_head(page); + void *anon_vma = vma->anon_vma; + struct folio *folio = page_folio(page); - VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_VMA(!anon_vma, vma); - anon_vma = (void *) anon_vma + PAGE_MAPPING_ANON; + anon_vma += PAGE_MAPPING_ANON; /* * Ensure that anon_vma and the PAGE_MAPPING_ANON bit are written * simultaneously, so a concurrent reader (eg folio_referenced()'s * folio_test_anon()) will not see one without the other. */ - WRITE_ONCE(page->mapping, (struct address_space *) anon_vma); - SetPageAnonExclusive(subpage); + WRITE_ONCE(folio->mapping, anon_vma); + SetPageAnonExclusive(page); } /** From patchwork Fri Sep 2 19:46:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0E25ECAAD5 for ; Fri, 2 Sep 2022 19:48:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8DFE780111; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D8B8800F4; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87047800F4; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 209248010B for ; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F3788120F71 for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) X-FDA: 79868178768.17.2DA8339 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 9E8491C005F for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=T5pqUNJVuk3AY3ms43palyFIe+URqU42hbhfDsQmVx0=; b=SGpOC2GYnj+IOpaYuG9F6lrJCp ON7fPJtJRIkGpxMeqOuKwUQjBlt6CJkRnx5GT4Znj9yFkyojVBctT/Fo1Cv20afxCblUcsyFNb4bP qKQCdItuSQMxgKm7jGU+8X+6ThDraJq69RrI9pXJIokaYi4hxk/ArbMi1Tw+j32HU23Ka+GKK2mY0 DxIR/Si+K5SVTQbqBOEj1j7Zg+m266gvNFN84rHYG1mtzihz18yrpF6IekY8Fnce8XzJysPc/UdAI WO2bCLyk0Mjiwb/PK+L207l+LAlK/GI7D7h19IEKLCprPvI137ULG4IZ0mz3Qm78LJfO8CYzueSmr jFZsEVbQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd6-007Ihd-1T; Fri, 02 Sep 2022 19:47:04 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 50/57] migrate: Convert __unmap_and_move() to use folios Date: Fri, 2 Sep 2022 20:46:46 +0100 Message-Id: <20220902194653.1739778-51-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148024; a=rsa-sha256; cv=none; b=u30Gxv0wBOUu2HRJV/VKC3pI3YVtZwAAXQFznJcZ1ozPei/wUQbzMMVsF8SmqTt5m1FSwT wyXd8f+YebWR8QmJVfOFun9ecAqQbmXQxoA5FmGt7gLRrkCU1BRvayeS4/u6H7BbIKZgII 5HlN6yMiU4yirw77kIrbHgYbqSSwp2E= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SGpOC2GY; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T5pqUNJVuk3AY3ms43palyFIe+URqU42hbhfDsQmVx0=; b=hYVy3MC3mJ9EXGSkiASKYUDu496zqq+h7ILNXQskQNDkHZ2uWUT0qtvvNX+uutwyu5SEHT srXJ4BkY8qQHgRIJaRnp0xwbLl1y4RTya5jrtY8B/VhTunuuPY9Kkjb1h7lENFcplQwwZD KG6wCwYyrJILE9scj+/3jfYdh6UQSmY= X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 9E8491C005F Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SGpOC2GY; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: d9ooq7ogaekxxanq5jujqy6csatwp7f9 X-HE-Tag: 1662148024-10056 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes a lot of calls to compound_head(). Also remove a VM_BUG_ON that can never trigger as the PageAnon bit is the bottom bit of page->mapping. Signed-off-by: Matthew Wilcox (Oracle) --- mm/migrate.c | 75 ++++++++++++++++++++++++++-------------------------- 1 file changed, 37 insertions(+), 38 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 6a1597c92261..3cd1fdb5f572 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -976,17 +976,15 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, return rc; } -static int __unmap_and_move(struct page *page, struct page *newpage, +static int __unmap_and_move(struct folio *src, struct folio *dst, int force, enum migrate_mode mode) { - struct folio *folio = page_folio(page); - struct folio *dst = page_folio(newpage); int rc = -EAGAIN; bool page_was_mapped = false; struct anon_vma *anon_vma = NULL; - bool is_lru = !__PageMovable(page); + bool is_lru = !__PageMovable(&src->page); - if (!trylock_page(page)) { + if (!folio_trylock(src)) { if (!force || mode == MIGRATE_ASYNC) goto out; @@ -1006,10 +1004,10 @@ static int __unmap_and_move(struct page *page, struct page *newpage, if (current->flags & PF_MEMALLOC) goto out; - lock_page(page); + folio_lock(src); } - if (PageWriteback(page)) { + if (folio_test_writeback(src)) { /* * Only in the case of a full synchronous migration is it * necessary to wait for PageWriteback. In the async case, @@ -1026,12 +1024,12 @@ static int __unmap_and_move(struct page *page, struct page *newpage, } if (!force) goto out_unlock; - wait_on_page_writeback(page); + folio_wait_writeback(src); } /* - * By try_to_migrate(), page->mapcount goes down to 0 here. In this case, - * we cannot notice that anon_vma is freed while we migrates a page. + * By try_to_migrate(), src->mapcount goes down to 0 here. In this case, + * we cannot notice that anon_vma is freed while we migrate a page. * This get_anon_vma() delays freeing anon_vma pointer until the end * of migration. File cache pages are no problem because of page_lock() * File Caches may use write_page() or lock_page() in migration, then, @@ -1043,22 +1041,22 @@ static int __unmap_and_move(struct page *page, struct page *newpage, * because that implies that the anon page is no longer mapped * (and cannot be remapped so long as we hold the page lock). */ - if (PageAnon(page) && !PageKsm(page)) - anon_vma = page_get_anon_vma(page); + if (folio_test_anon(src) && !folio_test_ksm(src)) + anon_vma = page_get_anon_vma(&src->page); /* * Block others from accessing the new page when we get around to * establishing additional references. We are usually the only one - * holding a reference to newpage at this point. We used to have a BUG - * here if trylock_page(newpage) fails, but would like to allow for - * cases where there might be a race with the previous use of newpage. + * holding a reference to dst at this point. We used to have a BUG + * here if folio_trylock(dst) fails, but would like to allow for + * cases where there might be a race with the previous use of dst. * This is much like races on refcount of oldpage: just don't BUG(). */ - if (unlikely(!trylock_page(newpage))) + if (unlikely(!folio_trylock(dst))) goto out_unlock; if (unlikely(!is_lru)) { - rc = move_to_new_folio(dst, folio, mode); + rc = move_to_new_folio(dst, src, mode); goto out_unlock_both; } @@ -1066,7 +1064,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage, * Corner case handling: * 1. When a new swap-cache page is read into, it is added to the LRU * and treated as swapcache but it has no rmap yet. - * Calling try_to_unmap() against a page->mapping==NULL page will + * Calling try_to_unmap() against a src->mapping==NULL page will * trigger a BUG. So handle it here. * 2. An orphaned page (see truncate_cleanup_page) might have * fs-private metadata. The page can be picked up due to memory @@ -1074,57 +1072,56 @@ static int __unmap_and_move(struct page *page, struct page *newpage, * invisible to the vm, so the page can not be migrated. So try to * free the metadata, so the page can be freed. */ - if (!page->mapping) { - VM_BUG_ON_PAGE(PageAnon(page), page); - if (page_has_private(page)) { - try_to_free_buffers(folio); + if (!src->mapping) { + if (folio_test_private(src)) { + try_to_free_buffers(src); goto out_unlock_both; } - } else if (page_mapped(page)) { + } else if (folio_mapped(src)) { /* Establish migration ptes */ - VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma, - page); - try_to_migrate(folio, 0); + VM_BUG_ON_FOLIO(folio_test_anon(src) && + !folio_test_ksm(src) && !anon_vma, src); + try_to_migrate(src, 0); page_was_mapped = true; } - if (!page_mapped(page)) - rc = move_to_new_folio(dst, folio, mode); + if (!folio_mapped(src)) + rc = move_to_new_folio(dst, src, mode); /* - * When successful, push newpage to LRU immediately: so that if it + * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will - * automatically build up the correct newpage->mlock_count for it. + * automatically build up the correct dst->mlock_count for it. * * We would like to do something similar for the old page, when * unsuccessful, and other cases when a page has been temporarily * isolated from the unevictable LRU: but this case is the easiest. */ if (rc == MIGRATEPAGE_SUCCESS) { - lru_cache_add(newpage); + folio_add_lru(dst); if (page_was_mapped) lru_add_drain(); } if (page_was_mapped) - remove_migration_ptes(folio, - rc == MIGRATEPAGE_SUCCESS ? dst : folio, false); + remove_migration_ptes(src, + rc == MIGRATEPAGE_SUCCESS ? dst : src, false); out_unlock_both: - unlock_page(newpage); + folio_unlock(dst); out_unlock: /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); - unlock_page(page); + folio_unlock(src); out: /* - * If migration is successful, decrease refcount of the newpage, + * If migration is successful, decrease refcount of dst, * which will not free the page because new page owner increased * refcounter. */ if (rc == MIGRATEPAGE_SUCCESS) - put_page(newpage); + folio_put(dst); return rc; } @@ -1140,6 +1137,7 @@ static int unmap_and_move(new_page_t get_new_page, enum migrate_reason reason, struct list_head *ret) { + struct folio *dst, *src = page_folio(page); int rc = MIGRATEPAGE_SUCCESS; struct page *newpage = NULL; @@ -1157,9 +1155,10 @@ static int unmap_and_move(new_page_t get_new_page, newpage = get_new_page(page, private); if (!newpage) return -ENOMEM; + dst = page_folio(newpage); newpage->private = 0; - rc = __unmap_and_move(page, newpage, force, mode); + rc = __unmap_and_move(src, dst, force, mode); if (rc == MIGRATEPAGE_SUCCESS) set_page_owner_migrate_reason(newpage, reason); From patchwork Fri Sep 2 19:46:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97617ECAAD5 for ; Fri, 2 Sep 2022 19:48:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6007480108; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F23D580118; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BB428010E; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 92E998010E for ; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 748E6ABC68 for ; Fri, 2 Sep 2022 19:47:05 +0000 (UTC) X-FDA: 79868178810.11.BEA927B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id 270EE40044 for ; Fri, 2 Sep 2022 19:47:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=HZ8b1C8NuYghJrw/IzYZl0rPHNemrZPs9QNaQ3LLfIg=; b=PcjOCSeaTgLyk7JbKoQG7II5hN /7QfC6mSUEQ6ezSF0LOUpnrOTmL/aYZ1avlV99qip2/de+x1+wQbWop5s5s5NhvcPuMZpXx/2jqSF hOhNslGW6DZEAErbj4yrJbxWK9x5VZqj6BJ+t4LoiivuVHFI/S7tvIvBWM8MdF50DKA6U96w7s3tR Ts9UcuQsSjZIy4oSa62B+p6NB4PulW3R7IvNeThKfY7bWOtYklhAG48hNzSRJYxqP0qSO0oYaNIka uZAqYec7O8GTmsaiW1/EYSmxkpP8GRkEtSt0DyU4EA4sADLCQ4KjaF6vnxxlXFFUu170XWJKaO/Kj 8g/S2PfA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd6-007Iht-6c; Fri, 02 Sep 2022 19:47:04 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 51/57] migrate: Convert unmap_and_move_huge_page() to use folios Date: Fri, 2 Sep 2022 20:46:47 +0100 Message-Id: <20220902194653.1739778-52-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148025; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HZ8b1C8NuYghJrw/IzYZl0rPHNemrZPs9QNaQ3LLfIg=; b=exn3bNerz2w+t1cAZflG1+LZe/hOYl3JXMeL3+v6RCCDVsMpSwmV22PQQsfZKnNlR7gZQ1 087aQRZisD8YCQJ3SP5OIYJVlQuz64Ka3MThtayf+GxpE/Qg7lZhxqtFme7jRQ3CIzLJTd GXObW60Oeial6vf9su9xFkcjgobAX5o= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=PcjOCSea; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148025; a=rsa-sha256; cv=none; b=j5FgD1myXql20poLm2ltqFOZLY66BvVYY+3EEuDHJWxg74c5vEeKI6dmUb/dSyC5vgbDmH rRjmexO8vjptAVWs2iUJjd7wKasxsVE7hcNamZ7c/i/HvbS/om3ecyV+HT17J7Ue2qs9Ee ss+NfAEY6JS0Zqp9+S3eJEn3C49mJGk= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 270EE40044 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=PcjOCSea; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 9u3mz47azrren7hx5cagjk15dxcomt9t X-HE-Tag: 1662148025-56560 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves several calls to compound_head() and removes a couple of uses of page->lru. Signed-off-by: Matthew Wilcox (Oracle) --- mm/migrate.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 3cd1fdb5f572..7b338ddd011a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1244,11 +1244,11 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, * kicking migration. */ if (!hugepage_migration_supported(page_hstate(hpage))) { - list_move_tail(&hpage->lru, ret); + list_move_tail(&src->lru, ret); return -ENOSYS; } - if (page_count(hpage) == 1) { + if (folio_ref_count(src) == 1) { /* page was freed from under us. So we are done. */ putback_active_hugepage(hpage); return MIGRATEPAGE_SUCCESS; @@ -1259,7 +1259,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, return -ENOMEM; dst = page_folio(new_hpage); - if (!trylock_page(hpage)) { + if (!folio_trylock(src)) { if (!force) goto out; switch (mode) { @@ -1269,29 +1269,29 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, default: goto out; } - lock_page(hpage); + folio_lock(src); } /* * Check for pages which are in the process of being freed. Without - * page_mapping() set, hugetlbfs specific move page routine will not + * folio_mapping() set, hugetlbfs specific move page routine will not * be called and we could leak usage counts for subpools. */ - if (hugetlb_page_subpool(hpage) && !page_mapping(hpage)) { + if (hugetlb_page_subpool(hpage) && !folio_mapping(src)) { rc = -EBUSY; goto out_unlock; } - if (PageAnon(hpage)) - anon_vma = page_get_anon_vma(hpage); + if (folio_test_anon(src)) + anon_vma = page_get_anon_vma(&src->page); - if (unlikely(!trylock_page(new_hpage))) + if (unlikely(!folio_trylock(dst))) goto put_anon; - if (page_mapped(hpage)) { + if (folio_mapped(src)) { enum ttu_flags ttu = 0; - if (!PageAnon(hpage)) { + if (!folio_test_anon(src)) { /* * In shared mappings, try_to_unmap could potentially * call huge_pmd_unshare. Because of this, take @@ -1312,7 +1312,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, i_mmap_unlock_write(mapping); } - if (!page_mapped(hpage)) + if (!folio_mapped(src)) rc = move_to_new_folio(dst, src, mode); if (page_was_mapped) @@ -1320,7 +1320,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, rc == MIGRATEPAGE_SUCCESS ? dst : src, false); unlock_put_anon: - unlock_page(new_hpage); + folio_unlock(dst); put_anon: if (anon_vma) @@ -1332,12 +1332,12 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, } out_unlock: - unlock_page(hpage); + folio_unlock(src); out: if (rc == MIGRATEPAGE_SUCCESS) putback_active_hugepage(hpage); else if (rc != -EAGAIN) - list_move_tail(&hpage->lru, ret); + list_move_tail(&src->lru, ret); /* * If migration was not successful and there's a freeing callback, use From patchwork Fri Sep 2 19:46:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964570 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D32EC6FA83 for ; Fri, 2 Sep 2022 19:48:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 572C58010B; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D54B180105; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FC918010D; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 06E888010D for ; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D948114064B for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) X-FDA: 79868178768.21.355379B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id 97E2640094 for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=kdoCdesbfuUTPNOWhOmgwG2UyPqEVn409F2dI96dWqc=; b=kApHbwzOb7mNkK5kTlJpcN0OKw frF4i6gptJP34ptVh0+puCGmVo6AARONHuMfg34JOTGngcAT5lG+cXu9npb2fVykHMctuxAmQNqBN QLPh8veUbqwTCYf/WwyKWuTHHYRlSOy50L7u8vOu1hgCVC/3mky9p6KvwV2fo0cnfP3t4WPEe8Sv3 TtCSszsHKawpGoQ/Clva8o/MLz6RN65MSJuMBb7ZCB6Ey4dBI1iWvXXJyg/t9XAXP63HFmfLspl/n qXitRjR8TKUMQtDXlcy5DG73LVAmdTp6OSLzpLJXuubBJqNELrw28lJ0E6modGRSK9H1cFyoCPbEY LARJAn5A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd6-007IiG-CK; Fri, 02 Sep 2022 19:47:04 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 52/57] huge_memory: Convert split_huge_page_to_list() to use a folio Date: Fri, 2 Sep 2022 20:46:48 +0100 Message-Id: <20220902194653.1739778-53-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=kApHbwzO; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148024; a=rsa-sha256; cv=none; b=HN+PfAp6AHVa0Abgomb3PvTaXIUENTX/6kYrv5SEPbaMH9r0/F0KgLzvX6/7ylfu4nX2Je yLLEbXnfbHSafrjRfC3CqhKlGBMLX32uM0RejCzKuODJK7V76Xr2csY8gSzmHwZ5YOdPtu +4oQ031GJ6BfWThOJ1T6eQ8XxNWhilI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kdoCdesbfuUTPNOWhOmgwG2UyPqEVn409F2dI96dWqc=; b=xmAjIlqjhTGk5bae9fkER3yqfhPsZHEV0hylkZoss7zOUG8BepwQjlT6JAXK/0FOMuFFiX BanUyyZP9SfXL3zGECyn515zUlJwQ8kAxrWbtPdf8LyuW3aqzUvPOl7SaqCjLY8QTkAEJ3 gEFIkfwve3Fk5nBvG4FYeedguNtm1JE= X-Rspamd-Queue-Id: 97E2640094 X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=kApHbwzO; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Stat-Signature: fm4yj6arbkrkomi3798sapr178rmykyy X-HE-Tag: 1662148024-671244 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves many calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 49 ++++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ffbc0412be1b..366519eb2af8 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2614,27 +2614,26 @@ bool can_split_folio(struct folio *folio, int *pextra_pins) int split_huge_page_to_list(struct page *page, struct list_head *list) { struct folio *folio = page_folio(page); - struct page *head = &folio->page; - struct deferred_split *ds_queue = get_deferred_split_queue(head); - XA_STATE(xas, &head->mapping->i_pages, head->index); + struct deferred_split *ds_queue = get_deferred_split_queue(&folio->page); + XA_STATE(xas, &folio->mapping->i_pages, folio->index); struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; int extra_pins, ret; pgoff_t end; bool is_hzp; - VM_BUG_ON_PAGE(!PageLocked(head), head); - VM_BUG_ON_PAGE(!PageCompound(head), head); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); - is_hzp = is_huge_zero_page(head); - VM_WARN_ON_ONCE_PAGE(is_hzp, head); + is_hzp = is_huge_zero_page(&folio->page); + VM_WARN_ON_ONCE_FOLIO(is_hzp, folio); if (is_hzp) return -EBUSY; - if (PageWriteback(head)) + if (folio_test_writeback(folio)) return -EBUSY; - if (PageAnon(head)) { + if (folio_test_anon(folio)) { /* * The caller does not necessarily hold an mmap_lock that would * prevent the anon_vma disappearing so we first we take a @@ -2643,7 +2642,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) * is taken to serialise against parallel split or collapse * operations. */ - anon_vma = page_get_anon_vma(head); + anon_vma = page_get_anon_vma(&folio->page); if (!anon_vma) { ret = -EBUSY; goto out; @@ -2652,7 +2651,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) mapping = NULL; anon_vma_lock_write(anon_vma); } else { - mapping = head->mapping; + mapping = folio->mapping; /* Truncated ? */ if (!mapping) { @@ -2660,7 +2659,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) goto out; } - xas_split_alloc(&xas, head, compound_order(head), + xas_split_alloc(&xas, folio, folio_order(folio), mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK); if (xas_error(&xas)) { ret = xas_error(&xas); @@ -2675,7 +2674,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) * but on 32-bit, i_size_read() takes an irq-unsafe seqlock, * which cannot be nested inside the page tree lock. So note * end now: i_size itself may be changed at any moment, but - * head page lock is good enough to serialize the trimming. + * folio lock is good enough to serialize the trimming. */ end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE); if (shmem_mapping(mapping)) @@ -2691,38 +2690,38 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) goto out_unlock; } - unmap_page(head); + unmap_page(&folio->page); /* block interrupt reentry in xa_lock and spinlock */ local_irq_disable(); if (mapping) { /* - * Check if the head page is present in page cache. - * We assume all tail are present too, if head is there. + * Check if the folio is present in page cache. + * We assume all tail are present too, if folio is there. */ xas_lock(&xas); xas_reset(&xas); - if (xas_load(&xas) != head) + if (xas_load(&xas) != folio) goto fail; } /* Prevent deferred_split_scan() touching ->_refcount */ spin_lock(&ds_queue->split_queue_lock); - if (page_ref_freeze(head, 1 + extra_pins)) { - if (!list_empty(page_deferred_list(head))) { + if (folio_ref_freeze(folio, 1 + extra_pins)) { + if (!list_empty(page_deferred_list(&folio->page))) { ds_queue->split_queue_len--; - list_del(page_deferred_list(head)); + list_del(page_deferred_list(&folio->page)); } spin_unlock(&ds_queue->split_queue_lock); if (mapping) { - int nr = thp_nr_pages(head); + int nr = folio_nr_pages(folio); - xas_split(&xas, head, thp_order(head)); - if (PageSwapBacked(head)) { - __mod_lruvec_page_state(head, NR_SHMEM_THPS, + xas_split(&xas, folio, folio_order(folio)); + if (folio_test_swapbacked(folio)) { + __lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -nr); } else { - __mod_lruvec_page_state(head, NR_FILE_THPS, + __lruvec_stat_mod_folio(folio, NR_FILE_THPS, -nr); filemap_nr_thps_dec(mapping); } From patchwork Fri Sep 2 19:46:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FFC6C6FA83 for ; Fri, 2 Sep 2022 19:48:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 05B428010D; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F0D680101; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03FF48010D; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 39D7080110 for ; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1B7A114064B for ; Fri, 2 Sep 2022 19:47:05 +0000 (UTC) X-FDA: 79868178810.07.F4340FC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id D58FF1C0049 for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=D+82vubFJc6QJAVdIZ51p+s82tMbkawAG6VALT75iy8=; b=hoGHrcbUJ8AVq95ISK5ChGP7AJ A1hzOlFO7lBV6m1Xux6UHy8ZxmnbgHGP0A95uJY5dFfgNMG6Qa4OnZLdBsPm9Am6CbAfbwZrlCjgQ vokjDbHe2yqMPxwqAgTGyMDjF/ZSvwPp5YAtjNqezp4ntfhsFz6/5ae4EXUYHEujx65acaxZrtAWh JFOJNsvxI+kUJ/fwSfJgkboui5gu6Qfk7GAGdhZfnYIzQLoMyvJ+vwPPFo79kBX+2DUTEOWVzmfoN aLBhsJqKvbFHno4O03wrqDLsQdzTBHxU8SEkhvaWnCIxe6yiejk3DQ40qAIa8T+KXLjVgLoR2U9h5 mz9/8xdA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd6-007IiR-GT; Fri, 02 Sep 2022 19:47:04 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 53/57] huge_memory: Convert unmap_page() to unmap_folio() Date: Fri, 2 Sep 2022 20:46:49 +0100 Message-Id: <20220902194653.1739778-54-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148024; a=rsa-sha256; cv=none; b=P2Rg7SwocqfNIewFgpWQYiW9p7qJOaeZRHJRadv8PcOgzSpHIXTyMNYiLlAhaESX2+ZJp9 zcBqcybEKMBxvYq+QKhfdqXS/lytjitPmdg7OEOqwRYvzvfIb/3Z4JJahUK4xUx1ChhBUS D2xJQxK/+hTUqf6nA95c7MPwi7pbFjo= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hoGHrcbU; dmarc=none; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D+82vubFJc6QJAVdIZ51p+s82tMbkawAG6VALT75iy8=; b=EN7Hyk7vq1mKcqsvU284rjrJi4B+wglZ9v4/IvKwqK9eUquWjgdX627ePtO6TgCe1BdrP2 E+nu13nwyJpVQ464Xzlb4bC0Haf/E9GPM/0XeYGoPYGC2M3szgq97LTgTiqJ7QPnFx5Ukm U32fHl2PlJPezt8ir4qWroKxpnHYK2k= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: D58FF1C0049 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hoGHrcbU; dmarc=none; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: 8pi18kbce1iescwo44c4r6x6fbcec3mc X-HE-Tag: 1662148024-640125 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove a folio->page->folio conversion. Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 366519eb2af8..4e76b3c760ac 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2348,13 +2348,12 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, } } -static void unmap_page(struct page *page) +static void unmap_folio(struct folio *folio) { - struct folio *folio = page_folio(page); enum ttu_flags ttu_flags = TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | TTU_SYNC; - VM_BUG_ON_PAGE(!PageHead(page), page); + VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); /* * Anon pages need migration entries to preserve them, but file @@ -2371,7 +2370,7 @@ static void remap_page(struct folio *folio, unsigned long nr) { int i = 0; - /* If unmap_page() uses try_to_migrate() on file, remove this check */ + /* If unmap_folio() uses try_to_migrate() on file, remove this check */ if (!folio_test_anon(folio)) return; for (;;) { @@ -2421,7 +2420,7 @@ static void __split_huge_page_tail(struct page *head, int tail, * for example lock_page() which set PG_waiters. * * Note that for mapped sub-pages of an anonymous THP, - * PG_anon_exclusive has been cleared in unmap_page() and is stored in + * PG_anon_exclusive has been cleared in unmap_folio() and is stored in * the migration entry instead from where remap_page() will restore it. * We can still have PG_anon_exclusive set on effectively unmapped and * unreferenced sub-pages of an anonymous THP: we can simply drop @@ -2682,7 +2681,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) } /* - * Racy check if we can split the page, before unmap_page() will + * Racy check if we can split the page, before unmap_folio() will * split PMDs */ if (!can_split_folio(folio, &extra_pins)) { @@ -2690,7 +2689,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) goto out_unlock; } - unmap_page(&folio->page); + unmap_folio(folio); /* block interrupt reentry in xa_lock and spinlock */ local_irq_disable(); From patchwork Fri Sep 2 19:46:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 345F9ECAAD5 for ; Fri, 2 Sep 2022 19:48:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D334A80105; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D3518010D; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2565F80111; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4704B80107 for ; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 298B8160802 for ; Fri, 2 Sep 2022 19:47:05 +0000 (UTC) X-FDA: 79868178810.02.ED70D41 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf02.hostedemail.com (Postfix) with ESMTP id E0B0F80065 for ; Fri, 2 Sep 2022 19:47:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=g+3prt31CnADlYcIOTknfhIStNKaTWbAAl4JikLJpTo=; b=HfPc/Lu6YeSBzdfWlnDcpLVPoc Kv8um7qb50dMZoShR7W0mnjqgQzF/Gf23frXViHXwcMEbJg5hb1uYxAisYP6P6GVOuKWfIX3PlW1g LpHIyYcA2NtVIgYobxBxzn9RGuGUN7jF6S+S4/1e7T65Ohym3hGijGERW3UdxwSiOL9yhhpkv6wyB gGat0HzllQZb0tG6Am6Dqn+8htCR0MziUr7GK1RXyVTgajrCvSdX3rn9OPRB3ttKcXK3Sm6Esiwjs lMSyKuJAXaFAsKWLBei+dnScVnmw6CWNaO6bpWgBM35r4vDAs32uy7NIpabLUiitAzZsLLisoLA26 hz4Fe6iw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd6-007IiZ-KZ; Fri, 02 Sep 2022 19:47:04 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 54/57] mm: Convert page_get_anon_vma() to folio_get_anon_vma() Date: Fri, 2 Sep 2022 20:46:50 +0100 Message-Id: <20220902194653.1739778-55-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148024; a=rsa-sha256; cv=none; b=WrUmaa21rVWukN1Wj5u2jL/f++NQh8XXItlFUiymldXRXPKB/STLdSjo6h14tzfJO8GEKJ z8iN+IDIOYoqajkPXIzK9ymQUxh4JHc3NEDhHERnEapulBQcQSMBsIPHyj9GSCHTJNstfg wOsE6YUs7IScB1mLlJ1IwQIwg60xq1w= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="HfPc/Lu6"; dmarc=none; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g+3prt31CnADlYcIOTknfhIStNKaTWbAAl4JikLJpTo=; b=yOY7aKD8sT8HuAnb/LwrRsElatHBA0WwNVb9VijjDBTRLYLejyK/RcXAFsdjleB186f3FZ 4PRKuCDs8R5BUtDHmMV6YZxBtW+qBFPl+XAXcW277Yn1HLIYfZwTai+lwCi17FlyIEgasU J1ifrzllUH+1ySIhUuWdMtNDHfqV1JY= X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: E0B0F80065 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="HfPc/Lu6"; dmarc=none; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: e4mz68uc7uifneaaqfu3ickekeaqft7s X-HE-Tag: 1662148024-582216 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With all callers now passing in a folio, rename the function and convert all callers. Removes a couple of calls to compound_head() and a reference to page->mapping. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 2 +- mm/huge_memory.c | 2 +- mm/migrate.c | 6 +++--- mm/rmap.c | 14 +++++++------- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b89b4b86951f..79974285a6f0 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -166,7 +166,7 @@ static inline void anon_vma_merge(struct vm_area_struct *vma, unlink_anon_vmas(next); } -struct anon_vma *page_get_anon_vma(struct page *page); +struct anon_vma *folio_get_anon_vma(struct folio *folio); /* RMAP flags, currently only relevant for some anon rmap operations. */ typedef int __bitwise rmap_t; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4e76b3c760ac..38213d06f595 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2641,7 +2641,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) * is taken to serialise against parallel split or collapse * operations. */ - anon_vma = page_get_anon_vma(&folio->page); + anon_vma = folio_get_anon_vma(folio); if (!anon_vma) { ret = -EBUSY; goto out; diff --git a/mm/migrate.c b/mm/migrate.c index 7b338ddd011a..c688319ffd46 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1035,14 +1035,14 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, * File Caches may use write_page() or lock_page() in migration, then, * just care Anon page here. * - * Only page_get_anon_vma() understands the subtleties of + * Only folio_get_anon_vma() understands the subtleties of * getting a hold on an anon_vma from outside one of its mms. * But if we cannot get anon_vma, then we won't need it anyway, * because that implies that the anon page is no longer mapped * (and cannot be remapped so long as we hold the page lock). */ if (folio_test_anon(src) && !folio_test_ksm(src)) - anon_vma = page_get_anon_vma(&src->page); + anon_vma = folio_get_anon_vma(src); /* * Block others from accessing the new page when we get around to @@ -1283,7 +1283,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, } if (folio_test_anon(src)) - anon_vma = page_get_anon_vma(&src->page); + anon_vma = folio_get_anon_vma(src); if (unlikely(!folio_trylock(dst))) goto put_anon; diff --git a/mm/rmap.c b/mm/rmap.c index 76ebfc8beb7e..e37e5377e5a9 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -489,16 +489,16 @@ void __init anon_vma_init(void) * if there is a mapcount, we can dereference the anon_vma after observing * those. */ -struct anon_vma *page_get_anon_vma(struct page *page) +struct anon_vma *folio_get_anon_vma(struct folio *folio) { struct anon_vma *anon_vma = NULL; unsigned long anon_mapping; rcu_read_lock(); - anon_mapping = (unsigned long)READ_ONCE(page->mapping); + anon_mapping = (unsigned long)READ_ONCE(folio->mapping); if ((anon_mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON) goto out; - if (!page_mapped(page)) + if (!folio_mapped(folio)) goto out; anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON); @@ -508,13 +508,13 @@ struct anon_vma *page_get_anon_vma(struct page *page) } /* - * If this page is still mapped, then its anon_vma cannot have been + * If this folio is still mapped, then its anon_vma cannot have been * freed. But if it has been unmapped, we have no security against the * anon_vma structure being freed and reused (for another anon_vma: * SLAB_TYPESAFE_BY_RCU guarantees that - so the atomic_inc_not_zero() * above cannot corrupt). */ - if (!page_mapped(page)) { + if (!folio_mapped(folio)) { rcu_read_unlock(); put_anon_vma(anon_vma); return NULL; @@ -526,11 +526,11 @@ struct anon_vma *page_get_anon_vma(struct page *page) } /* - * Similar to page_get_anon_vma() except it locks the anon_vma. + * Similar to folio_get_anon_vma() except it locks the anon_vma. * * Its a little more complex as it tries to keep the fast path to a single * atomic op -- the trylock. If we fail the trylock, we fall back to getting a - * reference like with page_get_anon_vma() and then block on the mutex + * reference like with folio_get_anon_vma() and then block on the mutex * on !rwc->try_lock case. */ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, From patchwork Fri Sep 2 19:46:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF1AEECAAD5 for ; Fri, 2 Sep 2022 19:48:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE31E80119; Fri, 2 Sep 2022 15:47:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D577480121; Fri, 2 Sep 2022 15:47:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A447D80120; Fri, 2 Sep 2022 15:47:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7E8A680119 for ; Fri, 2 Sep 2022 15:47:23 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5868612076A for ; Fri, 2 Sep 2022 19:47:23 +0000 (UTC) X-FDA: 79868179566.04.82D1FC0 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 1CFF7A005B for ; Fri, 2 Sep 2022 19:47:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=K9aNQdPTBGqHuzzNU3BfYzeBOFpSRHntQcYqOXH/VsM=; b=Sfa/0Z5OyL8xP1W8thbb2HIv/V CbZ1TtKsc/+zlO+ibNNFGLBRisNfuaekbz7/wW421+IHw5JLuPHDxxOwducebW32KtUk2fvFB6Suh prJQ8cVk3u21B368JoRtIug6fLJQmfs1kLrvtxtYmyWeWIBeELARcIROrMMO3GetUltLFzgToN7V7 PM0SmB0yJK6VdSlm9KupdgfVlxPIfPDGXNOEuRjC2LO7LEl+VPJ3JedypT8rIdwhL3lUnB0yzT30B iQStyrUif1X565R0kjELGcTAixG4VR4yQmZNpZABMHWHa17xffz6Tw8HjNBac0nEXCTGPEkNztTQh q3YNBP4A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd6-007Iit-R1; Fri, 02 Sep 2022 19:47:04 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 55/57] rmap: Remove page_unlock_anon_vma_read() Date: Fri, 2 Sep 2022 20:46:51 +0100 Message-Id: <20220902194653.1739778-56-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="Sfa/0Z5O"; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148043; a=rsa-sha256; cv=none; b=Kt86sWDfUrZTeHDbkqHlWMlaTSB5+BiVXE/W6rtikHFjRU+Ve4+vng9R4NhTOp1CfHcTUU 4PuCjkOwQ277B2TAtXBnayDm8xwb1RVUPAy9WW03WyZS/7/HzIhCx0wfcJCx73pr/nxyYa E2/s5BVig/kiPFUuc1zpDSG6GXOufyE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148043; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K9aNQdPTBGqHuzzNU3BfYzeBOFpSRHntQcYqOXH/VsM=; b=6yBanKBhtfJ1lvhfDZqXrk6ZfI47W4uIFynHtcJmNXjvB29+O32SegNIf8C6fO3IoZ1cJY 7HdaBLolJ1j7mGVsc5dSUryWw35Q9a9WU/Ge5RLpjeGlP9EXfZs9vJLAufdo/bxvzhgPqL SkWJ5YhMh9ZokeR7kjzQIUMGYsKNGs0= X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1CFF7A005B X-Stat-Signature: wxdk7hja1qfu4pm613h4uy83xdwc95fe Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="Sfa/0Z5O"; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1662148042-372900 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This was simply an alias for anon_vma_unlock_read() since 2011. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 5 ----- mm/memory-failure.c | 2 +- mm/rmap.c | 5 ----- 3 files changed, 1 insertion(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 79974285a6f0..f7ebbeed6a1b 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -405,13 +405,8 @@ struct rmap_walk_control { void rmap_walk(struct folio *folio, struct rmap_walk_control *rwc); void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc); - -/* - * Called by memory-failure.c to kill processes. - */ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, struct rmap_walk_control *rwc); -void page_unlock_anon_vma_read(struct anon_vma *anon_vma); #else /* !CONFIG_MMU */ diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 14439806b5ef..87b95dc59956 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -527,7 +527,7 @@ static void collect_procs_anon(struct page *page, struct list_head *to_kill, } } read_unlock(&tasklist_lock); - page_unlock_anon_vma_read(av); + anon_vma_unlock_read(av); } /* diff --git a/mm/rmap.c b/mm/rmap.c index e37e5377e5a9..d4450b1b79d1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -602,11 +602,6 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, return anon_vma; } -void page_unlock_anon_vma_read(struct anon_vma *anon_vma) -{ - anon_vma_unlock_read(anon_vma); -} - #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH /* * Flush TLB entries for recently unmapped pages from remote CPUs. It is From patchwork Fri Sep 2 19:46:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964578 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45BF6C6FA82 for ; Fri, 2 Sep 2022 19:48:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8593E80113; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C1338010E; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8DE3800FF; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B496B80101 for ; Fri, 2 Sep 2022 15:47:05 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 987451C63B9 for ; Fri, 2 Sep 2022 19:47:05 +0000 (UTC) X-FDA: 79868178810.02.8D5F0A6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 2AE791C005F for ; Fri, 2 Sep 2022 19:47:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=OLpZzsA000SaHs8Mq1xkTy2luxKoWvALbfqtapq3Q7E=; b=dYn8kYnxUfO17ID3rTdFzv3l4R zMkRDmGEfD5ZBUprKIZCC8HANLrWrJg86CxpjmDJwD8IoEobWKOH24vCAjWkvFNE4JigaZ0YgMhj+ 4f1ENgCmaFkD4ZDzWLUA6JVrxr6J5YZJ2glMs7dDaMhDnvWjU3BscYd9U4OluzRFG6WR9Tybk0Orl a5IwhDW+JZW98DnsnFXkcbSX17T6OuGymo0hbf+LjE8palU5QP2qYllRvsVw6JIpl3iXR4Kh/9XIl RIIz+UnIa+DweUCapCVqYbTcOOBO5d6ay9GopziTpFiokDKKgQEwoKCuxwHGWY1hQ0JEy+YZ97FVc TwZVrYXA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd7-007Ij3-0W; Fri, 02 Sep 2022 19:47:05 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 56/57] uprobes: Use new_folio in __replace_page() Date: Fri, 2 Sep 2022 20:46:52 +0100 Message-Id: <20220902194653.1739778-57-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148025; a=rsa-sha256; cv=none; b=5ceW7hAlF95WHkGwjacBVgxMe28n4uU2uOAGr1iEfN6DpuyYP1wnoYyvvzkiy3oaggkbGp RzfW5wSSAMkuO9XsCiiqoD/N7amOwTXvAKHzVqOVUip+x7+K74/9IMOCorX9x6oD0W2Aau z3XHn5B/5pbcyEV1NW72Iw/TT/0AVxY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dYn8kYnx; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148025; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OLpZzsA000SaHs8Mq1xkTy2luxKoWvALbfqtapq3Q7E=; b=O8OlS2T7/cc/tVsfa/1aNRNvUk4vCYROxxmT/EOStZWzoEKRkWzsq7GbK59oNQstsz5126 jp9CEy60H6ssTrzt4A+6zZEIs7+xZJCMSYFV7EvMBAzslIdYzRywgHys2vbEPIrtU1lrxO z5QWpRMPGygqUWeRL93zCfVCMSsdfEU= X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 2AE791C005F Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dYn8kYnx; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: n9spozimon5mt65swwieuw4wrma37iq1 X-HE-Tag: 1662148025-948240 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves several calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- kernel/events/uprobes.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 9722c4587c48..7179298808c2 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -155,6 +155,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, struct page *old_page, struct page *new_page) { struct folio *old_folio = page_folio(old_page); + struct folio *new_folio; struct mm_struct *mm = vma->vm_mm; DEFINE_FOLIO_VMA_WALK(pvmw, old_folio, vma, addr, 0); int err; @@ -164,8 +165,8 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, addr + PAGE_SIZE); if (new_page) { - err = mem_cgroup_charge(page_folio(new_page), vma->vm_mm, - GFP_KERNEL); + new_folio = page_folio(new_page); + err = mem_cgroup_charge(new_folio, vma->vm_mm, GFP_KERNEL); if (err) return err; } @@ -180,9 +181,9 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, VM_BUG_ON_PAGE(addr != pvmw.address, old_page); if (new_page) { - get_page(new_page); + folio_get(new_folio); page_add_new_anon_rmap(new_page, vma, addr); - lru_cache_add_inactive_or_unevictable(new_page, vma); + folio_add_lru_vma(new_folio, vma); } else /* no new page, just dec_mm_counter for old_page */ dec_mm_counter(mm, MM_ANONPAGES); From patchwork Fri Sep 2 19:46:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12964579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2210C6FA83 for ; Fri, 2 Sep 2022 19:48:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B4FA8010E; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B3C880101; Fri, 2 Sep 2022 15:47:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D27818010A; Fri, 2 Sep 2022 15:47:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 259368010A for ; Fri, 2 Sep 2022 15:47:06 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F098E160A5E for ; Fri, 2 Sep 2022 19:47:05 +0000 (UTC) X-FDA: 79868178810.26.BDC5130 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id 922A840094 for ; Fri, 2 Sep 2022 19:47:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=vuuXoETAHy7ZtGGNennLbGDUoHNI3qcuFecNGvbDrOQ=; b=SbkOyw+unc1MFLscax1qFiop5M YCl/V5vu+fbhKtM490WT8eujU/5Bz2j3B/OJluLLPhxxB2eABtrtLLSLOgeibOqG9BYq3LkldpEcv o02mvK3day4A+oIozkamohG7Wmn9rOiUt5pu+odJSzy+o8LzuK2cZ5eI3qZjiSdBQ7vPa3U2gwByN xjPvyTHCKHOJT3O+kV35gu+91xyEwY/XYFU93d+j3gT3Y576/XhjYVdy82G1mQVW2MKA3scUz72zy KVkIVxj9Rtmc9GSiD2h0mnyTJJbzGwpNtsuvGYBg2dE1ZS8vPo8mbwdX+UmU+erNlaNdF8i5tWwA6 DwIzFKIw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oUCd7-007IjJ-6Y; Fri, 02 Sep 2022 19:47:05 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 57/57] mm: Convert lock_page_or_retry() to folio_lock_or_retry() Date: Fri, 2 Sep 2022 20:46:53 +0100 Message-Id: <20220902194653.1739778-58-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220902194653.1739778-1-willy@infradead.org> References: <20220902194653.1739778-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SbkOyw+u; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662148025; a=rsa-sha256; cv=none; b=NMf8/l7CIbtShiM3kPbnVcWUZQ/iOTcxpD3KT6XUsBsqQ1hbN0Hr/5n2uzQqCF56+mSvPx xWewIlHK/IWhkpYEPzzZFxMH4YaDBfWsftbvyFFIMUDyDUJUSP6F6qj3+5DkYaDrx1n/zf lRo9OxUytmPXEav4WHn4hsIkO0Wwqqw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662148025; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vuuXoETAHy7ZtGGNennLbGDUoHNI3qcuFecNGvbDrOQ=; b=48MqIwA+zTuftkjNM8qvRUrVO2q4NO1vhbmNapvZsiyahFshJqF5NcCYuQeLIwlFdhd0hc Ooh0SIEe45pg5cMwPCbn1Fs76UH3ycoR/YY164blqib5ihZu5+MP2mOnusxw0L6UoTTd5+ NnQTGEWxszaKDoEIz9JWgJQbEiQe+j0= X-Rspamd-Queue-Id: 922A840094 X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SbkOyw+u; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Stat-Signature: xx1atp9dof99bupmcgzjufbgw4t6xf45 X-HE-Tag: 1662148025-527611 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove a call to compound_head() in each of the two callers. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 9 +++------ mm/memory.c | 10 +++++----- 2 files changed, 8 insertions(+), 11 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 0178b2040ea3..1816d74c08b7 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -989,19 +989,16 @@ static inline int lock_page_killable(struct page *page) } /* - * lock_page_or_retry - Lock the page, unless this would block and the + * folio_lock_or_retry - Lock the folio, unless this would block and the * caller indicated that it can handle a retry. * * Return value and mmap_lock implications depend on flags; see * __folio_lock_or_retry(). */ -static inline bool lock_page_or_retry(struct page *page, struct mm_struct *mm, - unsigned int flags) +static inline bool folio_lock_or_retry(struct folio *folio, + struct mm_struct *mm, unsigned int flags) { - struct folio *folio; might_sleep(); - - folio = page_folio(page); return folio_trylock(folio) || __folio_lock_or_retry(folio, mm, flags); } diff --git a/mm/memory.c b/mm/memory.c index 0b5630dba38e..a844ae67f7a0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3612,11 +3612,11 @@ EXPORT_SYMBOL(unmap_mapping_range); */ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) { - struct page *page = vmf->page; + struct folio *folio = page_folio(vmf->page); struct vm_area_struct *vma = vmf->vma; struct mmu_notifier_range range; - if (!lock_page_or_retry(page, vma->vm_mm, vmf->flags)) + if (!folio_lock_or_retry(folio, vma->vm_mm, vmf->flags)) return VM_FAULT_RETRY; mmu_notifier_range_init_owner(&range, MMU_NOTIFY_EXCLUSIVE, 0, vma, vma->vm_mm, vmf->address & PAGE_MASK, @@ -3626,10 +3626,10 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (likely(pte_same(*vmf->pte, vmf->orig_pte))) - restore_exclusive_pte(vma, page, vmf->address, vmf->pte); + restore_exclusive_pte(vma, vmf->page, vmf->address, vmf->pte); pte_unmap_unlock(vmf->pte, vmf->ptl); - unlock_page(page); + folio_unlock(folio); mmu_notifier_invalidate_range_end(&range); return 0; @@ -3829,7 +3829,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_release; } - locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags); + locked = folio_lock_or_retry(folio, vma->vm_mm, vmf->flags); if (!locked) { ret |= VM_FAULT_RETRY;