[095/131] mm: memcontrol: convert anon and file-thp to new mem_cgroup_charge() API

Message ID	20200603230204.Eftrdj5QW%akpm@linux-foundation.org (mailing list archive)
State	New, archived
Headers	show Return-Path: <SRS0=OQel=7Q=kvack.org=owner-linux-mm@kernel.org> DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 285B42166E Date: Wed, 03 Jun 2020 16:02:04 -0700 From: Andrew Morton <akpm@linux-foundation.org> To: akpm@linux-foundation.org, alex.shi@linux.alibaba.com, bsingharora@gmail.com, cai@lca.pw, guro@fb.com, hannes@cmpxchg.org, hughd@google.com, iamjoonsoo.kim@lge.com, kirill@shutemov.name, linux-mm@kvack.org, mhocko@suse.com, mm-commits@vger.kernel.org, shakeelb@google.com, torvalds@linux-foundation.org Subject: [patch 095/131] mm: memcontrol: convert anon and file-thp to new mem_cgroup_charge() API Message-ID: <20200603230204.Eftrdj5QW%akpm@linux-foundation.org> In-Reply-To: <20200603155549.e041363450869eaae4c7f05b@linux-foundation.org> User-Agent: s-nail v14.8.16 Sender: owner-linux-mm@kvack.org Precedence: bulk
Series	[001/131] mm/slub: fix a memory leak in sysfs_slab_add() \| expand [001/131] mm/slub: fix a memory leak in sysfs_slab_add() [002/131] mm/memcg: optimize memory.numa_stat like memory.stat [003/131] mm/gup: move __get_user_pages_fast() down a few lines in gup.c [004/131] mm/gup: refactor and de-duplicate gup_fast() code [005/131] mm/gup: introduce pin_user_pages_fast_only() [006/131] drm/i915: convert get_user_pages() --> pin_user_pages() [007/131] mm/gup: might_lock_read(mmap_sem) in get_user_pages_fast() [008/131] kasan: stop tests being eliminated as dead code with FORTIFY_SOURCE [009/131] string.h: fix incompatibility between FORTIFY_SOURCE and KASAN [010/131] mm: clarify __GFP_MEMALLOC usage [011/131] mm: memblock: replace dereferences of memblock_region.nid with API calls [012/131] mm: make early_pfn_to_nid() and related defintions close to each other [013/131] mm: remove CONFIG_HAVE_MEMBLOCK_NODE_MAP option [014/131] mm: free_area_init: use maximal zone PFNs rather than zone sizes [015/131] mm: use free_area_init() instead of free_area_init_nodes() [016/131] alpha: simplify detection of memory zone boundaries [017/131] arm: simplify detection of memory zone boundaries [018/131] arm64: simplify detection of memory zone boundaries for UMA configs [019/131] csky: simplify detection of memory zone boundaries [020/131] m68k: mm: simplify detection of memory zone boundaries [021/131] parisc: simplify detection of memory zone boundaries [022/131] sparc32: simplify detection of memory zone boundaries [023/131] unicore32: simplify detection of memory zone boundaries [024/131] xtensa: simplify detection of memory zone boundaries [025/131] mm: memmap_init: iterate over memblock regions rather that check each PFN [026/131] mm: remove early_pfn_in_nid() and CONFIG_NODES_SPAN_OTHER_NODES [027/131] mm: free_area_init: allow defining max_zone_pfn in descending order [028/131] mm: rename free_area_init_node() to free_area_init_memoryless_node() [029/131] mm: clean up free_area_init_node() and its helpers [030/131] mm: simplify find_min_pfn_with_active_regions() [031/131] docs/vm: update memory-models documentation [032/131] mm/page_alloc.c: bad_[reason\|flags] is not necessary when PageHWPoison [033/131] mm/page_alloc.c: bad_flags is not necessary for bad_page() [034/131] mm/page_alloc.c: rename free_pages_check_bad() to check_free_page_bad() [035/131] mm/page_alloc.c: rename free_pages_check() to check_free_page() [036/131] mm/page_alloc.c: extract check_[new\|free]_page_bad() common part to page_bad_reason() [037/131] mm,page_alloc,cma: conditionally prefer cma pageblocks for movable allocations [038/131] mm/page_alloc.c: remove unused free_bootmem_with_active_regions [039/131] mm/page_alloc.c: only tune sysctl_lowmem_reserve_ratio value once when changing it [040/131] mm/page_alloc.c: clear out zone->lowmem_reserve[] if the zone is empty [041/131] mm/vmstat.c: do not show lowmem reserve protection information of empty zone [042/131] mm/page_alloc: use ac->high_zoneidx for classzone_idx [043/131] mm/page_alloc: integrate classzone_idx and high_zoneidx [044/131] mm/page_alloc.c: use NODE_MASK_NONE in build_zonelists() [045/131] mm: rename gfpflags_to_migratetype to gfp_migratetype for same convention [046/131] mm/page_alloc.c: reset numa stats for boot pagesets [047/131] mm, page_alloc: reset the zone->watermark_boost early [048/131] mm/page_alloc: restrict and formalize compound_page_dtors[] [049/131] mm/pagealloc.c: call touch_nmi_watchdog() on max order boundaries in deferred init [050/131] mm: initialize deferred pages with interrupts enabled [051/131] mm: call cond_resched() from deferred_init_memmap() [052/131] padata: remove exit routine [053/131] padata: initialize earlier [054/131] padata: allocate work structures for parallel jobs from a pool [055/131] padata: add basic support for multithreaded jobs [056/131] mm: don't track number of pages during deferred initialization [057/131] mm: parallelize deferred_init_memmap() [058/131] mm: make deferred init's max threads arch-specific [059/131] padata: document multithreaded jobs [060/131] mm/page_alloc.c: add missing newline [061/131] khugepaged: add self test [062/131] khugepaged: do not stop collapse if less than half PTEs are referenced [063/131] khugepaged: drain all LRU caches before scanning pages [064/131] khugepaged: drain LRU add pagevec after swapin [065/131] khugepaged: allow to collapse a page shared across fork [066/131] khugepaged: allow to collapse PTE-mapped compound pages [067/131] thp: change CoW semantics for anon-THP [068/131] khugepaged: introduce 'max_ptes_shared' tunable [069/131] hugetlbfs: add arch_hugetlb_valid_size [070/131] hugetlbfs: move hugepagesz= parsing to arch independent code [071/131] hugetlbfs: remove hugetlb_add_hstate() warning for existing hstate [072/131] hugetlbfs: clean up command line processing [073/131] hugetlbfs: fix changes to command line processing [074/131] mm/hugetlb: avoid unnecessary check on pud and pmd entry in huge_pte_offset [075/131] arm64/mm: drop __HAVE_ARCH_HUGE_PTEP_GET [076/131] mm/hugetlb: define a generic fallback for is_hugepage_only_range() [077/131] mm/hugetlb: define a generic fallback for arch_clear_hugepage_flags() [078/131] mm: simplify calling a compound page destructor [079/131] mm/vmscan.c: use update_lru_size() in update_lru_sizes() [080/131] mm/vmscan: count layzfree pages and fix nr_isolated_* mismatch [081/131] mm/vmscan.c: change prototype for shrink_page_list [082/131] mm/vmscan: update the comment of should_continue_reclaim() [083/131] mm: fix NUMA node file count error in replace_page_cache() [084/131] mm: memcontrol: fix stat-corrupting race in charge moving [085/131] mm: memcontrol: drop @compound parameter from memcg charging API [086/131] mm: shmem: remove rare optimization when swapin races with hole punching [087/131] mm: memcontrol: move out cgroup swaprate throttling [088/131] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API [089/131] mm: memcontrol: prepare uncharging for removal of private page type counters [090/131] mm: memcontrol: prepare move_account for removal of private page type counters [091/131] mm: memcontrol: prepare cgroup vmstat infrastructure for native anon counters [092/131] mm: memcontrol: switch to native NR_FILE_PAGES and NR_SHMEM counters [093/131] mm: memcontrol: switch to native NR_ANON_MAPPED counter [094/131] mm: memcontrol: switch to native NR_ANON_THPS counter [095/131] mm: memcontrol: convert anon and file-thp to new mem_cgroup_charge() API [096/131] mm: memcontrol: drop unused try/commit/cancel charge API [097/131] mm: memcontrol: prepare swap controller setup for integration [098/131] mm: memcontrol: make swap tracking an integral part of memory control [099/131] mm: memcontrol: charge swapin pages on instantiation [100/131] mm: memcontrol: document the new swap control behavior [101/131] mm: memcontrol: delete unused lrucare handling [102/131] mm: memcontrol: update page->mem_cgroup stability rules [103/131] mm: fix LRU balancing effect of new transparent huge pages [104/131] mm: keep separate anon and file statistics on page reclaim activity [105/131] mm: allow swappiness that prefers reclaiming anon over the file workingset [106/131] mm: fold and remove lru_cache_add_anon() and lru_cache_add_file() [107/131] mm: workingset: let cache workingset challenge anon [108/131] mm: remove use-once cache bias from LRU balancing [109/131] mm: vmscan: drop unnecessary div0 avoidance rounding in get_scan_count() [110/131] mm: base LRU balancing on an explicit cost model [111/131] mm: deactivations shouldn't bias the LRU balance [112/131] mm: only count actual rotations as LRU reclaim cost [113/131] mm: balance LRU lists based on relative thrashing [114/131] mm: vmscan: determine anon/file pressure balance at the reclaim root [115/131] mm: vmscan: reclaim writepage is IO cost [116/131] mm: vmscan: limit the range of LRU type balancing [117/131] mm: swap: fix vmstats for huge pages [118/131] mm: swap: memcg: fix memcg stats for huge pages [119/131] tools/vm/page_owner_sort.c: filter out unneeded line [120/131] mm, mempolicy: fix up gup usage in lookup_node [121/131] include/linux/memblock.h: fix minor typo and unclear comment [122/131] sparc32: register memory occupied by kernel as memblock.memory [123/131] hugetlbfs: get unmapped area below TASK_UNMAPPED_BASE for hugetlbfs [124/131] mm: thp: don't need to drain lru cache when splitting and mlocking THP [125/131] powerpc/mm: drop platform defined pmd_mknotpresent() [126/131] mm/thp: rename pmd_mknotpresent() as pmd_mkinvalid() [127/131] drivers/base/memory.c: cache memory blocks in xarray to accelerate lookup [128/131] mm: add DEBUG_WX support [129/131] riscv: support DEBUG_WX [130/131] x86: mm: use ARCH_HAS_DEBUG_WX instead of arch defined [131/131] arm64: mm: use ARCH_HAS_DEBUG_WX instead of arch defined

--- a/include/linux/mm.h~mm-memcontrol-convert-anon-and-file-thp-to-new-mem_cgroup_charge-api +++ a/include/linux/mm.h @@ -501,7 +501,6 @@ struct vm_fault { pte_t orig_pte; /* Value of PTE at the time of fault */ struct page *cow_page; /* Page handler may use for COW fault */ - struct mem_cgroup *memcg; /* Cgroup cow_page belongs to */ struct page *page; /* ->fault handlers should return a * page here, unless VM_FAULT_NOPAGE * is set (which is also implied by @@ -946,8 +945,7 @@ static inline pte_t maybe_mkwrite(pte_t return pte; } -vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, - struct page *page); +vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct page *page); vm_fault_t finish_fault(struct vm_fault *vmf); vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf); #endif --- a/kernel/events/uprobes.c~mm-memcontrol-convert-anon-and-file-thp-to-new-mem_cgroup_charge-api +++ a/kernel/events/uprobes.c @@ -162,14 +162,13 @@ static int __replace_page(struct vm_area }; int err; struct mmu_notifier_range range; - struct mem_cgroup *memcg; mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); if (new_page) { - err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, - &memcg); + err = mem_cgroup_charge(new_page, vma->vm_mm, GFP_KERNEL, + false); if (err) return err; } @@ -179,16 +178,12 @@ static int __replace_page(struct vm_area mmu_notifier_invalidate_range_start(&range); err = -EAGAIN; - if (!page_vma_mapped_walk(&pvmw)) { - if (new_page) - mem_cgroup_cancel_charge(new_page, memcg); + if (!page_vma_mapped_walk(&pvmw)) goto unlock; - } VM_BUG_ON_PAGE(addr != pvmw.address, old_page); if (new_page) { get_page(new_page); - mem_cgroup_commit_charge(new_page, memcg, false); page_add_new_anon_rmap(new_page, vma, addr, false); lru_cache_add_active_or_unevictable(new_page, vma); } else --- a/mm/filemap.c~mm-memcontrol-convert-anon-and-file-thp-to-new-mem_cgroup_charge-api +++ a/mm/filemap.c @@ -2633,7 +2633,7 @@ void filemap_map_pages(struct vm_fault * if (vmf->pte) vmf->pte += xas.xa_index - last_pgoff; last_pgoff = xas.xa_index; - if (alloc_set_pte(vmf, NULL, page)) + if (alloc_set_pte(vmf, page)) goto unlock; unlock_page(page); goto next; --- a/mm/huge_memory.c~mm-memcontrol-convert-anon-and-file-thp-to-new-mem_cgroup_charge-api +++ a/mm/huge_memory.c @@ -587,19 +587,19 @@ static vm_fault_t __do_huge_pmd_anonymou struct page *page, gfp_t gfp) { struct vm_area_struct *vma = vmf->vma; - struct mem_cgroup *memcg; pgtable_t pgtable; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; vm_fault_t ret = 0; VM_BUG_ON_PAGE(!PageCompound(page), page); - if (mem_cgroup_try_charge_delay(page, vma->vm_mm, gfp, &memcg)) { + if (mem_cgroup_charge(page, vma->vm_mm, gfp, false)) { put_page(page); count_vm_event(THP_FAULT_FALLBACK); count_vm_event(THP_FAULT_FALLBACK_CHARGE); return VM_FAULT_FALLBACK; } + cgroup_throttle_swaprate(page, gfp); pgtable = pte_alloc_one(vma->vm_mm); if (unlikely(!pgtable)) { @@ -630,7 +630,6 @@ static vm_fault_t __do_huge_pmd_anonymou vm_fault_t ret2; spin_unlock(vmf->ptl); - mem_cgroup_cancel_charge(page, memcg); put_page(page); pte_free(vma->vm_mm, pgtable); ret2 = handle_userfault(vmf, VM_UFFD_MISSING); @@ -640,7 +639,6 @@ static vm_fault_t __do_huge_pmd_anonymou entry = mk_huge_pmd(page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); - mem_cgroup_commit_charge(page, memcg, false); page_add_new_anon_rmap(page, vma, haddr, true); lru_cache_add_active_or_unevictable(page, vma); pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable); @@ -649,7 +647,7 @@ static vm_fault_t __do_huge_pmd_anonymou mm_inc_nr_ptes(vma->vm_mm); spin_unlock(vmf->ptl); count_vm_event(THP_FAULT_ALLOC); - count_memcg_events(memcg, THP_FAULT_ALLOC, 1); + count_memcg_event_mm(vma->vm_mm, THP_FAULT_ALLOC); } return 0; @@ -658,7 +656,6 @@ unlock_release: release: if (pgtable) pte_free(vma->vm_mm, pgtable); - mem_cgroup_cancel_charge(page, memcg); put_page(page); return ret; --- a/mm/khugepaged.c~mm-memcontrol-convert-anon-and-file-thp-to-new-mem_cgroup_charge-api +++ a/mm/khugepaged.c @@ -1037,7 +1037,6 @@ static void collapse_huge_page(struct mm struct page *new_page; spinlock_t *pmd_ptl, *pte_ptl; int isolated = 0, result = 0; - struct mem_cgroup *memcg; struct vm_area_struct *vma; struct mmu_notifier_range range; gfp_t gfp; @@ -1060,15 +1059,15 @@ static void collapse_huge_page(struct mm goto out_nolock; } - if (unlikely(mem_cgroup_try_charge(new_page, mm, gfp, &memcg))) { + if (unlikely(mem_cgroup_charge(new_page, mm, gfp, false))) { result = SCAN_CGROUP_CHARGE_FAIL; goto out_nolock; } + count_memcg_page_event(new_page, THP_COLLAPSE_ALLOC); down_read(&mm->mmap_sem); result = hugepage_vma_revalidate(mm, address, &vma); if (result) { - mem_cgroup_cancel_charge(new_page, memcg); up_read(&mm->mmap_sem); goto out_nolock; } @@ -1076,7 +1075,6 @@ static void collapse_huge_page(struct mm pmd = mm_find_pmd(mm, address); if (!pmd) { result = SCAN_PMD_NULL; - mem_cgroup_cancel_charge(new_page, memcg); up_read(&mm->mmap_sem); goto out_nolock; } @@ -1088,7 +1086,6 @@ static void collapse_huge_page(struct mm */ if (unmapped && !__collapse_huge_page_swapin(mm, vma, address, pmd, referenced)) { - mem_cgroup_cancel_charge(new_page, memcg); up_read(&mm->mmap_sem); goto out_nolock; } @@ -1175,9 +1172,7 @@ static void collapse_huge_page(struct mm spin_lock(pmd_ptl); BUG_ON(!pmd_none(*pmd)); - mem_cgroup_commit_charge(new_page, memcg, false); page_add_new_anon_rmap(new_page, vma, address, true); - count_memcg_events(memcg, THP_COLLAPSE_ALLOC, 1); lru_cache_add_active_or_unevictable(new_page, vma); pgtable_trans_huge_deposit(mm, pmd, pgtable); set_pmd_at(mm, address, pmd, _pmd); @@ -1191,10 +1186,11 @@ static void collapse_huge_page(struct mm out_up_write: up_write(&mm->mmap_sem); out_nolock: + if (!IS_ERR_OR_NULL(*hpage)) + mem_cgroup_uncharge(*hpage); trace_mm_collapse_huge_page(mm, isolated, result); return; out: - mem_cgroup_cancel_charge(new_page, memcg); goto out_up_write; } @@ -1618,7 +1614,6 @@ static void collapse_file(struct mm_stru struct address_space *mapping = file->f_mapping; gfp_t gfp; struct page *new_page; - struct mem_cgroup *memcg; pgoff_t index, end = start + HPAGE_PMD_NR; LIST_HEAD(pagelist); XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); @@ -1637,10 +1632,11 @@ static void collapse_file(struct mm_stru goto out; } - if (unlikely(mem_cgroup_try_charge(new_page, mm, gfp, &memcg))) { + if (unlikely(mem_cgroup_charge(new_page, mm, gfp, false))) { result = SCAN_CGROUP_CHARGE_FAIL; goto out; } + count_memcg_page_event(new_page, THP_COLLAPSE_ALLOC); /* This will be less messy when we use multi-index entries */ do { @@ -1650,7 +1646,6 @@ static void collapse_file(struct mm_stru break; xas_unlock_irq(&xas); if (!xas_nomem(&xas, GFP_KERNEL)) { - mem_cgroup_cancel_charge(new_page, memcg); result = SCAN_FAIL; goto out; } @@ -1844,18 +1839,9 @@ out_unlock: } if (nr_none) { - struct lruvec *lruvec; - /* - * XXX: We have started try_charge and pinned the - * memcg, but the page isn't committed yet so we - * cannot use mod_lruvec_page_state(). This hackery - * will be cleaned up when remove the page->mapping - * dependency from memcg and fully charge above. - */ - lruvec = mem_cgroup_lruvec(memcg, page_pgdat(new_page)); - __mod_lruvec_state(lruvec, NR_FILE_PAGES, nr_none); + __mod_lruvec_page_state(new_page, NR_FILE_PAGES, nr_none); if (is_shmem) - __mod_lruvec_state(lruvec, NR_SHMEM, nr_none); + __mod_lruvec_page_state(new_page, NR_SHMEM, nr_none); } xa_locked: @@ -1893,7 +1879,6 @@ xa_unlocked: SetPageUptodate(new_page); page_ref_add(new_page, HPAGE_PMD_NR - 1); - mem_cgroup_commit_charge(new_page, memcg, false); if (is_shmem) { set_page_dirty(new_page); @@ -1901,7 +1886,6 @@ xa_unlocked: } else { lru_cache_add_file(new_page); } - count_memcg_events(memcg, THP_COLLAPSE_ALLOC, 1); /* * Remove pte page tables, so we can re-fault the page as huge. @@ -1948,13 +1932,14 @@ xa_unlocked: VM_BUG_ON(nr_none); xas_unlock_irq(&xas); - mem_cgroup_cancel_charge(new_page, memcg); new_page->mapping = NULL; } unlock_page(new_page); out: VM_BUG_ON(!list_empty(&pagelist)); + if (!IS_ERR_OR_NULL(*hpage)) + mem_cgroup_uncharge(*hpage); /* TODO: tracepoints */ } --- a/mm/memory.c~mm-memcontrol-convert-anon-and-file-thp-to-new-mem_cgroup_charge-api +++ a/mm/memory.c @@ -2645,7 +2645,6 @@ static vm_fault_t wp_page_copy(struct vm struct page *new_page = NULL; pte_t entry; int page_copied = 0; - struct mem_cgroup *memcg; struct mmu_notifier_range range; if (unlikely(anon_vma_prepare(vma))) @@ -2676,8 +2675,9 @@ static vm_fault_t wp_page_copy(struct vm } } - if (mem_cgroup_try_charge_delay(new_page, mm, GFP_KERNEL, &memcg)) + if (mem_cgroup_charge(new_page, mm, GFP_KERNEL, false)) goto oom_free_new; + cgroup_throttle_swaprate(new_page, GFP_KERNEL); __SetPageUptodate(new_page); @@ -2710,7 +2710,6 @@ static vm_fault_t wp_page_copy(struct vm * thread doing COW. */ ptep_clear_flush_notify(vma, vmf->address, vmf->pte); - mem_cgroup_commit_charge(new_page, memcg, false); page_add_new_anon_rmap(new_page, vma, vmf->address, false); lru_cache_add_active_or_unevictable(new_page, vma); /* @@ -2749,8 +2748,6 @@ static vm_fault_t wp_page_copy(struct vm /* Free the old page.. */ new_page = old_page; page_copied = 1; - } else { - mem_cgroup_cancel_charge(new_page, memcg); } if (new_page) @@ -3088,7 +3085,6 @@ vm_fault_t do_swap_page(struct vm_fault { struct vm_area_struct *vma = vmf->vma; struct page *page = NULL, *swapcache; - struct mem_cgroup *memcg; swp_entry_t entry; pte_t pte; int locked; @@ -3193,10 +3189,11 @@ vm_fault_t do_swap_page(struct vm_fault goto out_page; } - if (mem_cgroup_try_charge_delay(page, vma->vm_mm, GFP_KERNEL, &memcg)) { + if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL, true)) { ret = VM_FAULT_OOM; goto out_page; } + cgroup_throttle_swaprate(page, GFP_KERNEL); /* * Back out if somebody else already faulted in this pte. @@ -3243,11 +3240,9 @@ vm_fault_t do_swap_page(struct vm_fault /* ksm created a completely new copy */ if (unlikely(page != swapcache && swapcache)) { - mem_cgroup_commit_charge(page, memcg, false); page_add_new_anon_rmap(page, vma, vmf->address, false); lru_cache_add_active_or_unevictable(page, vma); } else { - mem_cgroup_commit_charge(page, memcg, true); do_page_add_anon_rmap(page, vma, vmf->address, exclusive); activate_page(page); } @@ -3284,7 +3279,6 @@ unlock: out: return ret; out_nomap: - mem_cgroup_cancel_charge(page, memcg); pte_unmap_unlock(vmf->pte, vmf->ptl); out_page: unlock_page(page); @@ -3305,7 +3299,6 @@ out_release: static vm_fault_t do_anonymous_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - struct mem_cgroup *memcg; struct page *page; vm_fault_t ret = 0; pte_t entry; @@ -3358,8 +3351,9 @@ static vm_fault_t do_anonymous_page(stru if (!page) goto oom; - if (mem_cgroup_try_charge_delay(page, vma->vm_mm, GFP_KERNEL, &memcg)) + if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL, false)) goto oom_free_page; + cgroup_throttle_swaprate(page, GFP_KERNEL); /* * The memory barrier inside __SetPageUptodate makes sure that @@ -3384,13 +3378,11 @@ static vm_fault_t do_anonymous_page(stru /* Deliver the page fault to userland, check inside PT lock */ if (userfaultfd_missing(vma)) { pte_unmap_unlock(vmf->pte, vmf->ptl); - mem_cgroup_cancel_charge(page, memcg); put_page(page); return handle_userfault(vmf, VM_UFFD_MISSING); } inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); - mem_cgroup_commit_charge(page, memcg, false); page_add_new_anon_rmap(page, vma, vmf->address, false); lru_cache_add_active_or_unevictable(page, vma); setpte: @@ -3402,7 +3394,6 @@ unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); return ret; release: - mem_cgroup_cancel_charge(page, memcg); put_page(page); goto unlock; oom_free_page: @@ -3607,7 +3598,6 @@ static vm_fault_t do_set_pmd(struct vm_f * mapping. If needed, the fucntion allocates page table or use pre-allocated. * * @vmf: fault environment - * @memcg: memcg to charge page (only for private mappings) * @page: page to map * * Caller must take care of unlocking vmf->ptl, if vmf->pte is non-NULL on @@ -3618,8 +3608,7 @@ static vm_fault_t do_set_pmd(struct vm_f * * Return: %0 on success, %VM_FAULT_ code in case of error. */ -vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, - struct page *page) +vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct page *page) { struct vm_area_struct *vma = vmf->vma; bool write = vmf->flags & FAULT_FLAG_WRITE; @@ -3627,9 +3616,6 @@ vm_fault_t alloc_set_pte(struct vm_fault vm_fault_t ret; if (pmd_none(*vmf->pmd) && PageTransCompound(page)) { - /* THP on COW? */ - VM_BUG_ON_PAGE(memcg, page); - ret = do_set_pmd(vmf, page); if (ret != VM_FAULT_FALLBACK) return ret; @@ -3652,7 +3638,6 @@ vm_fault_t alloc_set_pte(struct vm_fault /* copy-on-write page */ if (write && !(vma->vm_flags & VM_SHARED)) { inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); - mem_cgroup_commit_charge(page, memcg, false); page_add_new_anon_rmap(page, vma, vmf->address, false); lru_cache_add_active_or_unevictable(page, vma); } else { @@ -3702,7 +3687,7 @@ vm_fault_t finish_fault(struct vm_fault if (!(vmf->vma->vm_flags & VM_SHARED)) ret = check_stable_address_space(vmf->vma->vm_mm); if (!ret) - ret = alloc_set_pte(vmf, vmf->memcg, page); + ret = alloc_set_pte(vmf, page); if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); return ret; @@ -3862,11 +3847,11 @@ static vm_fault_t do_cow_fault(struct vm if (!vmf->cow_page) return VM_FAULT_OOM; - if (mem_cgroup_try_charge_delay(vmf->cow_page, vma->vm_mm, - GFP_KERNEL, &vmf->memcg)) { + if (mem_cgroup_charge(vmf->cow_page, vma->vm_mm, GFP_KERNEL, false)) { put_page(vmf->cow_page); return VM_FAULT_OOM; } + cgroup_throttle_swaprate(vmf->cow_page, GFP_KERNEL); ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) @@ -3884,7 +3869,6 @@ static vm_fault_t do_cow_fault(struct vm goto uncharge_out; return ret; uncharge_out: - mem_cgroup_cancel_charge(vmf->cow_page, vmf->memcg); put_page(vmf->cow_page); return ret; } --- a/mm/migrate.c~mm-memcontrol-convert-anon-and-file-thp-to-new-mem_cgroup_charge-api +++ a/mm/migrate.c @@ -2740,7 +2740,6 @@ static void migrate_vma_insert_page(stru { struct vm_area_struct *vma = migrate->vma; struct mm_struct *mm = vma->vm_mm; - struct mem_cgroup *memcg; bool flush = false; spinlock_t *ptl; pte_t entry; @@ -2787,7 +2786,7 @@ static void migrate_vma_insert_page(stru if (unlikely(anon_vma_prepare(vma))) goto abort; - if (mem_cgroup_try_charge(page, vma->vm_mm, GFP_KERNEL, &memcg)) + if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL, false)) goto abort; /* @@ -2832,7 +2831,6 @@ static void migrate_vma_insert_page(stru goto unlock_abort; inc_mm_counter(mm, MM_ANONPAGES); - mem_cgroup_commit_charge(page, memcg, false); page_add_new_anon_rmap(page, vma, addr, false); if (!is_zone_device_page(page)) lru_cache_add_active_or_unevictable(page, vma); @@ -2855,7 +2853,6 @@ static void migrate_vma_insert_page(stru unlock_abort: pte_unmap_unlock(ptep, ptl); - mem_cgroup_cancel_charge(page, memcg); abort: *src &= ~MIGRATE_PFN_MIGRATE; } --- a/mm/swapfile.c~mm-memcontrol-convert-anon-and-file-thp-to-new-mem_cgroup_charge-api +++ a/mm/swapfile.c @@ -1892,7 +1892,6 @@ static int unuse_pte(struct vm_area_stru unsigned long addr, swp_entry_t entry, struct page *page) { struct page *swapcache; - struct mem_cgroup *memcg; spinlock_t *ptl; pte_t *pte; int ret = 1; @@ -1902,14 +1901,13 @@ static int unuse_pte(struct vm_area_stru if (unlikely(!page)) return -ENOMEM; - if (mem_cgroup_try_charge(page, vma->vm_mm, GFP_KERNEL, &memcg)) { + if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL, true)) { ret = -ENOMEM; goto out_nolock; } pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); if (unlikely(!pte_same_as_swp(*pte, swp_entry_to_pte(entry)))) { - mem_cgroup_cancel_charge(page, memcg); ret = 0; goto out; } @@ -1920,10 +1918,8 @@ static int unuse_pte(struct vm_area_stru set_pte_at(vma->vm_mm, addr, pte, pte_mkold(mk_pte(page, vma->vm_page_prot))); if (page == swapcache) { - mem_cgroup_commit_charge(page, memcg, true); page_add_anon_rmap(page, vma, addr, false); } else { /* ksm created a completely new copy */ - mem_cgroup_commit_charge(page, memcg, false); page_add_new_anon_rmap(page, vma, addr, false); lru_cache_add_active_or_unevictable(page, vma); } --- a/mm/userfaultfd.c~mm-memcontrol-convert-anon-and-file-thp-to-new-mem_cgroup_charge-api +++ a/mm/userfaultfd.c @@ -56,7 +56,6 @@ static int mcopy_atomic_pte(struct mm_st struct page **pagep, bool wp_copy) { - struct mem_cgroup *memcg; pte_t _dst_pte, *dst_pte; spinlock_t *ptl; void *page_kaddr; @@ -97,7 +96,7 @@ static int mcopy_atomic_pte(struct mm_st __SetPageUptodate(page); ret = -ENOMEM; - if (mem_cgroup_try_charge(page, dst_mm, GFP_KERNEL, &memcg)) + if (mem_cgroup_charge(page, dst_mm, GFP_KERNEL, false)) goto out_release; _dst_pte = pte_mkdirty(mk_pte(page, dst_vma->vm_page_prot)); @@ -123,7 +122,6 @@ static int mcopy_atomic_pte(struct mm_st goto out_release_uncharge_unlock; inc_mm_counter(dst_mm, MM_ANONPAGES); - mem_cgroup_commit_charge(page, memcg, false); page_add_new_anon_rmap(page, dst_vma, dst_addr, false); lru_cache_add_active_or_unevictable(page, dst_vma); @@ -138,7 +136,6 @@ out: return ret; out_release_uncharge_unlock: pte_unmap_unlock(dst_pte, ptl); - mem_cgroup_cancel_charge(page, memcg); out_release: put_page(page); goto out;

[095/131] mm: memcontrol: convert anon and file-thp to new mem_cgroup_charge() API

Commit Message

Patch