| Message ID | 20210617151705.15367-3-alex.sierra@amd.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Support DEVICE_GENERIC memory in migrate_vma_* | expand |
On 6/17/21 8:16 AM, Alex Sierra wrote: > From: Ralph Campbell <rcampbell@nvidia.com> > > ZONE_DEVICE struct pages have an extra reference count that complicates the > code for put_page() and several places in the kernel that need to check the > reference count to see that a page is not being used (gup, compaction, > migration, etc.). Clean up the code so the reference count doesn't need to > be treated specially for ZONE_DEVICE. > > v2: > AS: merged this patch in linux 5.11 version > > Signed-off-by: Ralph Campbell <rcampbell@nvidia.com> > Signed-off-by: Alex Sierra <alex.sierra@amd.com> > --- > arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- > drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- > fs/dax.c | 4 +- > include/linux/dax.h | 2 +- > include/linux/memremap.h | 7 +-- > include/linux/mm.h | 44 ----------------- > lib/test_hmm.c | 2 +- > mm/internal.h | 8 +++ > mm/memremap.c | 68 +++++++------------------- > mm/migrate.c | 5 -- > mm/page_alloc.c | 3 ++ > mm/swap.c | 45 ++--------------- > 12 files changed, 45 insertions(+), 147 deletions(-) > I think it is great that you are picking this up and trying to revive it. However, I have a number of concerns about how it affects existing ZONE_DEVICE MEMORY_DEVICE_GENERIC and MEMORY_DEVICE_FS_DAX users and I don't see this addressing them. For example, dev_dax_probe() allocates MEMORY_DEVICE_GENERIC struct pages and then: dev_dax_fault() dev_dax_huge_fault() __dev_dax_pte_fault() vmf_insert_mixed() which just inserts the PFN into the CPU page tables without increasing the page refcount so it is zero (whereas it was one before). But using get_page() will trigger VM_BUG_ON_PAGE() if it is enabled. There isn't any current notion of free verses allocated for these struct pages. I suppose init_page_count() could be called on all the struct pages in dev_dax_probe() to fix that though. I'm even less clear about how to fix MEMORY_DEVICE_FS_DAX. File systems have clear allocate and free states for backing storage but there are the complications with the page cache references, etc. to consider. The >1 to 1 reference count seems to be used to tell when a page is idle (no I/O, reclaim scanners) rather than free (not allocated to any file) but I'm not 100% sure about that since I don't really understand all the issues around why a file system needs to have a DAX mount option besides knowing that the storage block size has to be a multiple of the page size.
Am 2021-06-17 um 3:16 p.m. schrieb Ralph Campbell: > > On 6/17/21 8:16 AM, Alex Sierra wrote: >> From: Ralph Campbell <rcampbell@nvidia.com> >> >> ZONE_DEVICE struct pages have an extra reference count that >> complicates the >> code for put_page() and several places in the kernel that need to >> check the >> reference count to see that a page is not being used (gup, compaction, >> migration, etc.). Clean up the code so the reference count doesn't >> need to >> be treated specially for ZONE_DEVICE. >> >> v2: >> AS: merged this patch in linux 5.11 version >> >> Signed-off-by: Ralph Campbell <rcampbell@nvidia.com> >> Signed-off-by: Alex Sierra <alex.sierra@amd.com> >> --- >> arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- >> drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- >> fs/dax.c | 4 +- >> include/linux/dax.h | 2 +- >> include/linux/memremap.h | 7 +-- >> include/linux/mm.h | 44 ----------------- >> lib/test_hmm.c | 2 +- >> mm/internal.h | 8 +++ >> mm/memremap.c | 68 +++++++------------------- >> mm/migrate.c | 5 -- >> mm/page_alloc.c | 3 ++ >> mm/swap.c | 45 ++--------------- >> 12 files changed, 45 insertions(+), 147 deletions(-) >> > I think it is great that you are picking this up and trying to revive it. > > However, I have a number of concerns about how it affects existing > ZONE_DEVICE > MEMORY_DEVICE_GENERIC and MEMORY_DEVICE_FS_DAX users and I don't see this > addressing them. For example, dev_dax_probe() allocates > MEMORY_DEVICE_GENERIC > struct pages and then: > dev_dax_fault() > dev_dax_huge_fault() > __dev_dax_pte_fault() > vmf_insert_mixed() > which just inserts the PFN into the CPU page tables without increasing > the page > refcount so it is zero (whereas it was one before). But using > get_page() will > trigger VM_BUG_ON_PAGE() if it is enabled. There isn't any current > notion of > free verses allocated for these struct pages. I suppose init_page_count() > could be called on all the struct pages in dev_dax_probe() to fix that > though. Hi Ralph, For DEVICE_GENERIC pages free_zone_device_page doesn't do anything. So I'm not sure what the reference counting is good for in this case. Alex is going to add free_zone_device_page support for DEVICE_GENERIC pages (patch 8 of this series). However, even then, it only does anything if there is an actual call to put_page. Where would that call come from in the dev_dax driver case? > > I'm even less clear about how to fix MEMORY_DEVICE_FS_DAX. File > systems have clear > allocate and free states for backing storage but there are the > complications with > the page cache references, etc. to consider. The >1 to 1 reference > count seems to > be used to tell when a page is idle (no I/O, reclaim scanners) rather > than free > (not allocated to any file) but I'm not 100% sure about that since I > don't really > understand all the issues around why a file system needs to have a DAX > mount option > besides knowing that the storage block size has to be a multiple of > the page size. The only thing that happens in free_zone_device_page for FS_DAX pages is wake_up_var(&page->_refcount). I guess, whoever is waiting for this wake-up will need to be prepared to see a refcount 0 instead of 1 now. I see these callers that would need to be updated: ./fs/ext4/inode.c: error = ___wait_var_event(&page->_refcount, ./fs/ext4/inode.c- atomic_read(&page->_refcount) == 1, ./fs/ext4/inode.c- TASK_INTERRUPTIBLE, 0, 0, ./fs/ext4/inode.c- ext4_wait_dax_page(ei)); -- ./fs/fuse/dax.c: return ___wait_var_event(&page->_refcount, ./fs/fuse/dax.c- atomic_read(&page->_refcount) == 1, TASK_INTERRUPTIBLE, ./fs/fuse/dax.c- 0, 0, fuse_wait_dax_page(inode)); -- ./fs/xfs/xfs_file.c: return ___wait_var_event(&page->_refcount, ./fs/xfs/xfs_file.c- atomic_read(&page->_refcount) == 1, TASK_INTERRUPTIBLE, ./fs/xfs/xfs_file.c- 0, 0, xfs_wait_dax_page(inode)); Regarding your page-cache comment, doesn't DAX mean that those file pages are not in the page cache? Regards, Felix
On 6/28/21 9:46 AM, Felix Kuehling wrote: > Am 2021-06-17 um 3:16 p.m. schrieb Ralph Campbell: >> On 6/17/21 8:16 AM, Alex Sierra wrote: >>> From: Ralph Campbell <rcampbell@nvidia.com> >>> >>> ZONE_DEVICE struct pages have an extra reference count that >>> complicates the >>> code for put_page() and several places in the kernel that need to >>> check the >>> reference count to see that a page is not being used (gup, compaction, >>> migration, etc.). Clean up the code so the reference count doesn't >>> need to >>> be treated specially for ZONE_DEVICE. >>> >>> v2: >>> AS: merged this patch in linux 5.11 version >>> >>> Signed-off-by: Ralph Campbell <rcampbell@nvidia.com> >>> Signed-off-by: Alex Sierra <alex.sierra@amd.com> >>> --- >>> arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- >>> drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- >>> fs/dax.c | 4 +- >>> include/linux/dax.h | 2 +- >>> include/linux/memremap.h | 7 +-- >>> include/linux/mm.h | 44 ----------------- >>> lib/test_hmm.c | 2 +- >>> mm/internal.h | 8 +++ >>> mm/memremap.c | 68 +++++++------------------- >>> mm/migrate.c | 5 -- >>> mm/page_alloc.c | 3 ++ >>> mm/swap.c | 45 ++--------------- >>> 12 files changed, 45 insertions(+), 147 deletions(-) >>> >> I think it is great that you are picking this up and trying to revive it. >> >> However, I have a number of concerns about how it affects existing >> ZONE_DEVICE >> MEMORY_DEVICE_GENERIC and MEMORY_DEVICE_FS_DAX users and I don't see this >> addressing them. For example, dev_dax_probe() allocates >> MEMORY_DEVICE_GENERIC >> struct pages and then: >> dev_dax_fault() >> dev_dax_huge_fault() >> __dev_dax_pte_fault() >> vmf_insert_mixed() >> which just inserts the PFN into the CPU page tables without increasing >> the page >> refcount so it is zero (whereas it was one before). But using >> get_page() will >> trigger VM_BUG_ON_PAGE() if it is enabled. There isn't any current >> notion of >> free verses allocated for these struct pages. I suppose init_page_count() >> could be called on all the struct pages in dev_dax_probe() to fix that >> though. > Hi Ralph, > > For DEVICE_GENERIC pages free_zone_device_page doesn't do anything. So > I'm not sure what the reference counting is good for in this case. > > Alex is going to add free_zone_device_page support for DEVICE_GENERIC > pages (patch 8 of this series). However, even then, it only does > anything if there is an actual call to put_page. Where would that call > come from in the dev_dax driver case? Correct, the drivers/dax/device.c driver allocates MEMORY_DEVICE_GENERIC struct pages and doesn't seem to allocate/free the page nor increment/decrement the reference count but it does call vmf_insert_mixed() if the /dev/file is mmap()'ed into a user process' address space. If devm_memremap_pages() returns the array of ZONE_DEVICE struct pages initialized with a reference count of zero, then the CPU page tables will have a PTE/PFN that points to a struct page with a zero reference count. This goes against the normal expectation in the rest of the mm code that assumes a page mapped by a CPU has a non-zero reference count. So yes, nothing "bad" happens because put_page() isn't called but the reference count will differ from other drivers that call vmf_insert_mixed() or vm_insert_page() where the page was allocated with alloc_pages() or similar. >> I'm even less clear about how to fix MEMORY_DEVICE_FS_DAX. File >> systems have clear >> allocate and free states for backing storage but there are the >> complications with >> the page cache references, etc. to consider. The >1 to 1 reference >> count seems to >> be used to tell when a page is idle (no I/O, reclaim scanners) rather >> than free >> (not allocated to any file) but I'm not 100% sure about that since I >> don't really >> understand all the issues around why a file system needs to have a DAX >> mount option >> besides knowing that the storage block size has to be a multiple of >> the page size. > The only thing that happens in free_zone_device_page for FS_DAX pages is > wake_up_var(&page->_refcount). I guess, whoever is waiting for this > wake-up will need to be prepared to see a refcount 0 instead of 1 now. I > see these callers that would need to be updated: > > ./fs/ext4/inode.c: error = ___wait_var_event(&page->_refcount, > ./fs/ext4/inode.c- atomic_read(&page->_refcount) == 1, > ./fs/ext4/inode.c- TASK_INTERRUPTIBLE, 0, 0, > ./fs/ext4/inode.c- ext4_wait_dax_page(ei)); > -- > ./fs/fuse/dax.c: return ___wait_var_event(&page->_refcount, > ./fs/fuse/dax.c- atomic_read(&page->_refcount) == 1, > TASK_INTERRUPTIBLE, > ./fs/fuse/dax.c- 0, 0, fuse_wait_dax_page(inode)); > -- > ./fs/xfs/xfs_file.c: return ___wait_var_event(&page->_refcount, > ./fs/xfs/xfs_file.c- atomic_read(&page->_refcount) == 1, > TASK_INTERRUPTIBLE, > ./fs/xfs/xfs_file.c- 0, 0, xfs_wait_dax_page(inode)); > > Regarding your page-cache comment, doesn't DAX mean that those file > pages are not in the page cache? > > Regards, > Felix > I don't really understand the FS_DAX code. I can see the __wait_var_event() is being used when truncating or punching holes in files but I'm not quite sure if it is using the >1 to 1 reference count to know when a page has no "extra" references or if it means the page is actually "free" and no longer assigned to a file. I really think some FS_DAX expert needs to weigh in on these reference count changes.
diff --git a/arch/powerpc/kvm/book3s_hv_uvmem.c b/arch/powerpc/kvm/book3s_hv_uvmem.c index 84e5a2dc8be5..acee67710620 100644 --- a/arch/powerpc/kvm/book3s_hv_uvmem.c +++ b/arch/powerpc/kvm/book3s_hv_uvmem.c @@ -711,7 +711,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm) dpage = pfn_to_page(uvmem_pfn); dpage->zone_device_data = pvt; - get_page(dpage); + init_page_count(dpage); lock_page(dpage); return dpage; out_clear: diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouveau/nouveau_dmem.c index 92987daa5e17..8bc7120e1216 100644 --- a/drivers/gpu/drm/nouveau/nouveau_dmem.c +++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c @@ -324,7 +324,7 @@ nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm) return NULL; } - get_page(page); + init_page_count(page); lock_page(page); return page; } diff --git a/fs/dax.c b/fs/dax.c index 321f4ddc6643..7b4c6b35b098 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -560,14 +560,14 @@ static void *grab_mapping_entry(struct xa_state *xas, /** * dax_layout_busy_page_range - find first pinned page in @mapping - * @mapping: address space to scan for a page with ref count > 1 + * @mapping: address space to scan for a page with ref count > 0 * @start: Starting offset. Page containing 'start' is included. * @end: End offset. Page containing 'end' is included. If 'end' is LLONG_MAX, * pages from 'start' till the end of file are included. * * DAX requires ZONE_DEVICE mapped pages. These pages are never * 'onlined' to the page allocator so they are considered idle when - * page->count == 1. A filesystem uses this interface to determine if + * page->count == 0. A filesystem uses this interface to determine if * any page in the mapping is busy, i.e. for DMA, or other * get_user_pages() usages. * diff --git a/include/linux/dax.h b/include/linux/dax.h index 8b5da1d60dbc..05fc982ce153 100644 --- a/include/linux/dax.h +++ b/include/linux/dax.h @@ -245,7 +245,7 @@ static inline bool dax_mapping(struct address_space *mapping) static inline bool dax_page_unused(struct page *page) { - return page_ref_count(page) == 1; + return page_ref_count(page) == 0; } #define dax_wait_page(_inode, _page, _wait_cb) \ diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 79c49e7f5c30..327f32427d21 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -66,9 +66,10 @@ enum memory_type { struct dev_pagemap_ops { /* - * Called once the page refcount reaches 1. (ZONE_DEVICE pages never - * reach 0 refcount unless there is a refcount bug. This allows the - * device driver to implement its own memory management.) + * Called once the page refcount reaches 0. The reference count + * should be reset to one with init_page_count(page) before reusing + * the page. This allows the device driver to implement its own + * memory management. */ void (*page_free)(struct page *page); diff --git a/include/linux/mm.h b/include/linux/mm.h index c9900aedc195..d8d79bb94be8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1117,39 +1117,6 @@ static inline bool is_zone_device_page(const struct page *page) } #endif -#ifdef CONFIG_DEV_PAGEMAP_OPS -void free_devmap_managed_page(struct page *page); -DECLARE_STATIC_KEY_FALSE(devmap_managed_key); - -static inline bool page_is_devmap_managed(struct page *page) -{ - if (!static_branch_unlikely(&devmap_managed_key)) - return false; - if (!is_zone_device_page(page)) - return false; - switch (page->pgmap->type) { - case MEMORY_DEVICE_PRIVATE: - case MEMORY_DEVICE_FS_DAX: - return true; - default: - break; - } - return false; -} - -void put_devmap_managed_page(struct page *page); - -#else /* CONFIG_DEV_PAGEMAP_OPS */ -static inline bool page_is_devmap_managed(struct page *page) -{ - return false; -} - -static inline void put_devmap_managed_page(struct page *page) -{ -} -#endif /* CONFIG_DEV_PAGEMAP_OPS */ - static inline bool is_device_private_page(const struct page *page) { return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) && @@ -1196,17 +1163,6 @@ static inline void put_page(struct page *page) { page = compound_head(page); - /* - * For devmap managed pages we need to catch refcount transition from - * 2 to 1, when refcount reach one it means the page is free and we - * need to inform the device driver through callback. See - * include/linux/memremap.h and HMM for details. - */ - if (page_is_devmap_managed(page)) { - put_devmap_managed_page(page); - return; - } - if (put_page_testzero(page)) __put_page(page); } diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 80a78877bd93..6998f10350ea 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -561,7 +561,7 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) } dpage->zone_device_data = rpage; - get_page(dpage); + init_page_count(dpage); lock_page(dpage); return dpage; diff --git a/mm/internal.h b/mm/internal.h index 25d2b2439f19..d3e58746f2d2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -623,4 +623,12 @@ struct migration_target_control { gfp_t gfp_mask; }; +#ifdef CONFIG_DEV_PAGEMAP_OPS +void free_zone_device_page(struct page *page); +#else +static inline void free_zone_device_page(struct page *page) +{ +} +#endif + #endif /* __MM_INTERNAL_H */ diff --git a/mm/memremap.c b/mm/memremap.c index 16b2fb482da1..614b3d600e95 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -12,6 +12,7 @@ #include <linux/types.h> #include <linux/wait_bit.h> #include <linux/xarray.h> +#include "internal.h" static DEFINE_XARRAY(pgmap_array); @@ -37,32 +38,6 @@ unsigned long memremap_compat_align(void) EXPORT_SYMBOL_GPL(memremap_compat_align); #endif -#ifdef CONFIG_DEV_PAGEMAP_OPS -DEFINE_STATIC_KEY_FALSE(devmap_managed_key); -EXPORT_SYMBOL(devmap_managed_key); - -static void devmap_managed_enable_put(struct dev_pagemap *pgmap) -{ - if (pgmap->type == MEMORY_DEVICE_PRIVATE || - pgmap->type == MEMORY_DEVICE_FS_DAX) - static_branch_dec(&devmap_managed_key); -} - -static void devmap_managed_enable_get(struct dev_pagemap *pgmap) -{ - if (pgmap->type == MEMORY_DEVICE_PRIVATE || - pgmap->type == MEMORY_DEVICE_FS_DAX) - static_branch_inc(&devmap_managed_key); -} -#else -static void devmap_managed_enable_get(struct dev_pagemap *pgmap) -{ -} -static void devmap_managed_enable_put(struct dev_pagemap *pgmap) -{ -} -#endif /* CONFIG_DEV_PAGEMAP_OPS */ - static void pgmap_array_delete(struct range *range) { xa_store_range(&pgmap_array, PHYS_PFN(range->start), PHYS_PFN(range->end), @@ -87,16 +62,6 @@ static unsigned long pfn_end(struct dev_pagemap *pgmap, int range_id) return (range->start + range_len(range)) >> PAGE_SHIFT; } -static unsigned long pfn_next(unsigned long pfn) -{ - if (pfn % 1024 == 0) - cond_resched(); - return pfn + 1; -} - -#define for_each_device_pfn(pfn, map, i) \ - for (pfn = pfn_first(map, i); pfn < pfn_end(map, i); pfn = pfn_next(pfn)) - static void dev_pagemap_kill(struct dev_pagemap *pgmap) { if (pgmap->ops && pgmap->ops->kill) @@ -152,20 +117,18 @@ static void pageunmap_range(struct dev_pagemap *pgmap, int range_id) void memunmap_pages(struct dev_pagemap *pgmap) { - unsigned long pfn; int i; dev_pagemap_kill(pgmap); for (i = 0; i < pgmap->nr_range; i++) - for_each_device_pfn(pfn, pgmap, i) - put_page(pfn_to_page(pfn)); + percpu_ref_put_many(pgmap->ref, pfn_end(pgmap, i) - + pfn_first(pgmap, i)); dev_pagemap_cleanup(pgmap); for (i = 0; i < pgmap->nr_range; i++) pageunmap_range(pgmap, i); WARN_ONCE(pgmap->altmap.alloc, "failed to free all reserved pages\n"); - devmap_managed_enable_put(pgmap); } EXPORT_SYMBOL_GPL(memunmap_pages); @@ -361,8 +324,6 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid) } } - devmap_managed_enable_get(pgmap); - /* * Clear the pgmap nr_range as it will be incremented for each * successfully processed range. This communicates how many @@ -477,16 +438,10 @@ struct dev_pagemap *get_dev_pagemap(unsigned long pfn, EXPORT_SYMBOL_GPL(get_dev_pagemap); #ifdef CONFIG_DEV_PAGEMAP_OPS -void free_devmap_managed_page(struct page *page) +static void free_device_private_page(struct page *page) { - /* notify page idle for dax */ - if (!is_device_private_page(page)) { - wake_up_var(&page->_refcount); - return; - } __ClearPageWaiters(page); - mem_cgroup_uncharge(page); /* @@ -513,4 +468,19 @@ void free_devmap_managed_page(struct page *page) page->mapping = NULL; page->pgmap->ops->page_free(page); } + +void free_zone_device_page(struct page *page) +{ + switch (page->pgmap->type) { + case MEMORY_DEVICE_FS_DAX: + /* notify page idle */ + wake_up_var(&page->_refcount); + return; + case MEMORY_DEVICE_PRIVATE: + free_device_private_page(page); + return; + default: + return; + } +} #endif /* CONFIG_DEV_PAGEMAP_OPS */ diff --git a/mm/migrate.c b/mm/migrate.c index 20ca887ea769..8c2430d3e77b 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -376,11 +376,6 @@ static int expected_page_refs(struct address_space *mapping, struct page *page) { int expected_count = 1; - /* - * Device private pages have an extra refcount as they are - * ZONE_DEVICE pages. - */ - expected_count += is_device_private_page(page); if (mapping) expected_count += thp_nr_pages(page) + page_has_private(page); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 519a60d5b6f7..4612c457d0b0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6210,6 +6210,9 @@ void __ref memmap_init_zone_device(struct zone *zone, __init_single_page(page, pfn, zone_idx, nid); + /* ZONE_DEVICE pages start with a zero reference count. */ + set_page_count(page, 0); + /* * Mark page reserved as it will need to wait for onlining * phase for it to be fully associated with a zone. diff --git a/mm/swap.c b/mm/swap.c index 2cca7141470c..0a12af814065 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -114,12 +114,11 @@ static void __put_compound_page(struct page *page) void __put_page(struct page *page) { if (is_zone_device_page(page)) { - put_dev_pagemap(page->pgmap); - /* * The page belongs to the device that created pgmap. Do * not return it to page allocator. */ + free_zone_device_page(page); return; } @@ -878,29 +877,18 @@ void release_pages(struct page **pages, int nr) if (is_huge_zero_page(page)) continue; + if (!put_page_testzero(page)) + continue; + if (is_zone_device_page(page)) { if (lruvec) { unlock_page_lruvec_irqrestore(lruvec, flags); lruvec = NULL; } - /* - * ZONE_DEVICE pages that return 'false' from - * page_is_devmap_managed() do not require special - * processing, and instead, expect a call to - * put_page_testzero(). - */ - if (page_is_devmap_managed(page)) { - put_devmap_managed_page(page); - continue; - } - if (put_page_testzero(page)) - put_dev_pagemap(page->pgmap); + free_zone_device_page(page); continue; } - if (!put_page_testzero(page)) - continue; - if (PageCompound(page)) { if (lruvec) { unlock_page_lruvec_irqrestore(lruvec, flags); @@ -1142,26 +1130,3 @@ void __init swap_setup(void) * _really_ don't want to cluster much more */ } - -#ifdef CONFIG_DEV_PAGEMAP_OPS -void put_devmap_managed_page(struct page *page) -{ - int count; - - if (WARN_ON_ONCE(!page_is_devmap_managed(page))) - return; - - count = page_ref_dec_return(page); - - /* - * devmap page refcounts are 1-based, rather than 0-based: if - * refcount is 1, then the page is free and the refcount is - * stable because nobody holds a reference on the page. - */ - if (count == 1) - free_devmap_managed_page(page); - else if (!count) - __put_page(page); -} -EXPORT_SYMBOL(put_devmap_managed_page); -#endif