[067/146] mm: page table check

Message ID	20220114220637.fzWDGOzsB%akpm@linux-foundation.org (mailing list archive)
State	New
Headers	show Return-Path: <owner-linux-mm@kvack.org> Date: Fri, 14 Jan 2022 14:06:37 -0800 From: Andrew Morton <akpm@linux-foundation.org> To: akpm@linux-foundation.org, aneesh.kumar@linux.ibm.com, corbet@lwn.net, dave.hansen@linux.intel.com, frederic@kernel.org, gthelen@google.com, hpa@zytor.com, hughd@google.com, jirislaby@kernel.org, keescook@chromium.org, linux-mm@kvack.org, masahiroy@kernel.org, mingo@redhat.com, mm-commits@vger.kernel.org, pasha.tatashin@soleen.com, peterz@infradead.org, pjt@google.com, rientjes@google.com, rppt@kernel.org, samitolvanen@google.com, songmuchun@bytedance.com, tglx@linutronix.de, torvalds@linux-foundation.org, weixugc@google.com, will@kernel.org Subject: [patch 067/146] mm: page table check Message-ID: <20220114220637.fzWDGOzsB%akpm@linux-foundation.org> In-Reply-To: <20220114140222.6b14f0061194d3200000c52d@linux-foundation.org> User-Agent: s-nail v14.8.16 Sender: owner-linux-mm@kvack.org Precedence: bulk
Series	[001/146] kthread: add the helper function kthread_run_on_cpu() \| expand [001/146] kthread: add the helper function kthread_run_on_cpu() [002/146] RDMA/siw: make use of the helper function kthread_run_on_cpu() [003/146] ring-buffer: make use of the helper function kthread_run_on_cpu() [004/146] rcutorture: make use of the helper function kthread_run_on_cpu() [005/146] trace/osnoise: make use of the helper function kthread_run_on_cpu() [006/146] trace/hwlat: make use of the helper function kthread_run_on_cpu() [007/146] ia64: module: use swap() to make code cleaner [008/146] arch/ia64/kernel/setup.c: use swap() to make code cleaner [009/146] ia64: fix typo in a comment [010/146] ia64: topology: use default_groups in kobj_type [011/146] scripts/spelling.txt: add "oveflow" [012/146] fs/ntfs/attrib.c: fix one kernel-doc comment [013/146] squashfs: provide backing_dev_info in order to disable read-ahead [014/146] ocfs2: use BUG_ON instead of if condition followed by BUG. [015/146] ocfs2: clearly handle ocfs2_grab_pages_for_write() return value [016/146] ocfs2: use default_groups in kobj_type [017/146] ocfs2: remove redundant assignment to pointer root_bh [018/146] ocfs2: cluster: use default_groups in kobj_type [019/146] ocfs2: remove redundant assignment to variable free_space [020/146] fs/ioctl: remove unnecessary __user annotation [021/146] mm/slab_common: use WARN() if cache still has objects on destroy [022/146] mm: slab: make slab iterator functions static [023/146] kmemleak: fix kmemleak false positive report with HW tag-based kasan enable [024/146] mm: kmemleak: alloc gray object for reserved region with direct map [025/146] mm: defer kmemleak object creation of module_alloc() [026/146] mm/page_alloc: split prep_compound_page into head and tail subparts [027/146] mm/page_alloc: refactor memmap_init_zone_device() page init [028/146] mm/memremap: add ZONE_DEVICE support for compound pages [029/146] device-dax: use ALIGN() for determining pgoff [030/146] device-dax: use struct_size() [031/146] device-dax: ensure dev_dax->pgmap is valid for dynamic devices [032/146] device-dax: factor out page mapping initialization [033/146] device-dax: set mapping prior to vmf_insert_pfn{,_pmd,pud}() [034/146] device-dax: remove pfn from __dev_dax_{pte,pmd,pud}_fault() [035/146] device-dax: compound devmap support [036/146] kasan: test: add globals left-out-of-bounds test [037/146] kasan: add ability to detect double-kmem_cache_destroy() [038/146] kasan: test: add test case for double-kmem_cache_destroy() [039/146] kasan: fix quarantine conflicting with init_on_free [040/146] mm,fs: split dump_mapping() out from dump_page() [041/146] mm/debug_vm_pgtable: update comments regarding migration swap entries [042/146] mm/truncate.c: remove unneeded variable [043/146] gup: avoid multiple user access locking/unlocking in fault_in_{read/write}able [044/146] mm/gup.c: stricter check on THP migration entry during follow_pmd_mask [045/146] mm: shmem: don't truncate page if memory failure happens [046/146] shmem: fix a race between shmem_unused_huge_shrink and shmem_evict_inode [047/146] mm/frontswap.c: use non-atomic '__set_bit()' when possible [048/146] mm: memcontrol: make cgroup_memory_nokmem static [049/146] mm/page_counter: remove an incorrect call to propagate_protected_usage() [050/146] mm/memcg: add oom_group_kill memory event [051/146] memcg: better bounds on the memcg stats updates [052/146] mm/memcg: use struct_size() helper in kzalloc() [053/146] memcg: add per-memcg vmalloc stat [054/146] tools/testing/selftests/vm/userfaultfd.c: use swap() to make code cleaner [055/146] mm: remove redundant check about FAULT_FLAG_ALLOW_RETRY bit [056/146] mm: rearrange madvise code to allow for reuse [057/146] mm: add a field to store names for private anonymous memory [058/146] mm: add anonymous vma name refcounting [059/146] mm: move anon_vma declarations to linux/mm_inline.h [060/146] mm: move tlb_flush_pending inline helpers to mm_inline.h [061/146] mm: protect free_pgtables with mmap_lock write lock in exit_mmap [062/146] mm: document locking restrictions for vm_operations_struct::close [063/146] mm/oom_kill: allow process_mrelease to run under mmap_lock protection [064/146] docs/vm: add vmalloced-kernel-stacks document [065/146] mm: change page type prior to adding page table entry [066/146] mm: ptep_clear() page table helper [067/146] mm: page table check [068/146] x86: mm: add x86_64 support for page table check [069/146] mm: remove last argument of reuse_swap_page() [070/146] mm: remove the total_mapcount argument from page_trans_huge_map_swapcount() [071/146] mm: remove the total_mapcount argument from page_trans_huge_mapcount() [072/146] mm/dmapool.c: revert "make dma pool to use kmalloc_node" [073/146] mm/vmalloc: alloc GFP_NO{FS,IO} for vmalloc [074/146] mm/vmalloc: add support for __GFP_NOFAIL [075/146] mm/vmalloc: be more explicit about supported gfp flags. [076/146] mm: allow !GFP_KERNEL allocations for kvmalloc [077/146] mm: make slab and vmalloc allocators __GFP_NOLOCKDEP aware [078/146] mm: introduce memalloc_retry_wait() [079/146] mm/pagealloc: sysctl: change watermark_scale_factor max limit to 30% [080/146] mm: fix boolreturn.cocci warning [081/146] mm: page_alloc: fix building error on -Werror=array-compare [082/146] mm: drop node from alloc_pages_vma [083/146] include/linux/gfp.h: further document GFP_DMA32 [084/146] mm/page_alloc.c: modify the comment section for alloc_contig_pages() [085/146] mm_zone: add function to check if managed dma zone exists [086/146] dma/pool: create dma atomic pool only if dma zone has managed pages [087/146] mm/page_alloc.c: do not warn allocation failure on zone DMA if no managed pages [088/146] hugetlb: add hugetlb..numa_stat file [089/146] mm, hugepages: make memory size variable in hugepage-mremap selftest [090/146] mm/vmstat: add events for THP max_ptes_ exceeds [091/146] selftests/vm: make charge_reserved_hugetlb.sh work with existing cgroup setting [092/146] selftests/uffd: allow EINTR/EAGAIN [093/146] userfaultfd/selftests: clean up hugetlb allocation code [094/146] vmscan: make drop_slab_node static [095/146] mm/page_isolation: unset migratetype directly for non Buddy page [096/146] mm/mempolicy: use policy_node helper with MPOL_PREFERRED_MANY [097/146] mm/mempolicy: add set_mempolicy_home_node syscall [098/146] mm/mempolicy: wire up syscall set_mempolicy_home_node [099/146] mm/mempolicy: fix all kernel-doc warnings [100/146] mm, oom: OOM sysrq should always kill a process [101/146] hugetlbfs: fix off-by-one error in hugetlb_vmdelete_list() [102/146] mm: migrate: fix the return value of migrate_pages() [103/146] mm: migrate: correct the hugetlb migration stats [104/146] mm: compaction: fix the migration stats in trace_mm_compaction_migratepages() [105/146] mm: migrate: support multiple target nodes demotion [106/146] mm: migrate: add more comments for selecting target node randomly [107/146] mm/migrate: move node demotion code to near its user [108/146] mm/migrate: remove redundant variables used in a for-loop [109/146] mm/thp: drop unused trace events hugepage_[invalidate\|splitting] [110/146] mm: ksm: fix use-after-free kasan report in ksm_might_need_to_copy [111/146] mm/hwpoison: mf_mutex for soft offline and unpoison [112/146] mm/hwpoison: remove MF_MSG_BUDDY_2ND and MF_MSG_POISONED_HUGE [113/146] mm/hwpoison: fix unpoison_memory() [114/146] mm: memcg/percpu: account extra objcg space to memory cgroups [115/146] mm/rmap: fix potential batched TLB flush race [116/146] zpool: remove the list of pools_head [117/146] zram: use ATTRIBUTE_GROUPS [118/146] mm: fix some comment errors [119/146] mm: make some vars and functions static or __init [120/146] mm/hmm.c: allow VM_MIXEDMAP to work with hmm_range_fault [121/146] mm/damon: unified access_check function naming rules [122/146] mm/damon: add 'age' of region tracepoint support [123/146] mm/damon/core: use abs() instead of diff_of() [124/146] mm/damon: remove some unneeded function definitions in damon.h [125/146] mm/damon/vaddr: remove swap_ranges() and replace it with swap() [126/146] mm/damon/schemes: add the validity judgment of thresholds [127/146] mm/damon: move damon_rand() definition into damon.h [128/146] mm/damon: modify damon_rand() macro to static inline function [129/146] mm/damon: convert macro functions to static inline functions [131/146] Docs/admin-guide/mm/damon/usage: remove redundant information [132/146] Docs/admin-guide/mm/damon/usage: mention tracepoint at the beginning [133/146] Docs/admin-guide/mm/damon/usage: update for kdamond_pid and (mk\|rm)_contexts [134/146] mm/damon: remove a mistakenly added comment for a future feature [135/146] mm/damon/schemes: account scheme actions that successfully applied [136/146] mm/damon/schemes: account how many times quota limit has exceeded [137/146] mm/damon/reclaim: provide reclamation statistics [138/146] Docs/admin-guide/mm/damon/reclaim: document statistics parameters [139/146] mm/damon/dbgfs: support all DAMOS stats [140/146] Docs/admin-guide/mm/damon/usage: update for schemes statistics [141/146] mm/damon: add access checking for hugetlb pages [142/146] mm/damon: move the implementation of damon_insert_region to damon.h [143/146] mm/damon/dbgfs: remove an unnecessary variable [144/146] mm/damon/vaddr: use pr_debug() for damon_va_three_regions() failure logging [145/146] mm/damon/vaddr: hide kernel pointer from damon_va_three_regions() failure log [146/146] mm/damon: hide kernel pointer from tracepoint event

--- a/arch/Kconfig~mm-page-table-check +++ a/arch/Kconfig @@ -1297,6 +1297,9 @@ config HAVE_ARCH_PFN_VALID config ARCH_SUPPORTS_DEBUG_PAGEALLOC bool +config ARCH_SUPPORTS_PAGE_TABLE_CHECK + bool + config ARCH_SPLIT_ARG64 bool help --- a/Documentation/vm/index.rst~mm-page-table-check +++ a/Documentation/vm/index.rst @@ -31,6 +31,7 @@ algorithms. If you are looking for advi page_migration page_frags page_owner + page_table_check remap_file_pages slub split_page_table_lock --- /dev/null +++ a/Documentation/vm/page_table_check.rst @@ -0,0 +1,56 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _page_table_check: + +================ +Page Table Check +================ + +Introduction +============ + +Page table check allows to hardern the kernel by ensuring that some types of +the memory corruptions are prevented. + +Page table check performs extra verifications at the time when new pages become +accessible from the userspace by getting their page table entries (PTEs PMDs +etc.) added into the table. + +In case of detected corruption, the kernel is crashed. There is a small +performance and memory overhead associated with the page table check. Therefore, +it is disabled by default, but can be optionally enabled on systems where the +extra hardening outweighs the performance costs. Also, because page table check +is synchronous, it can help with debugging double map memory corruption issues, +by crashing kernel at the time wrong mapping occurs instead of later which is +often the case with memory corruptions bugs. + +Double mapping detection logic +============================== + ++-------------------+-------------------+-------------------+------------------+ +| Current Mapping | New mapping | Permissions | Rule | ++===================+===================+===================+==================+ +| Anonymous | Anonymous | Read | Allow | ++-------------------+-------------------+-------------------+------------------+ +| Anonymous | Anonymous | Read / Write | Prohibit | ++-------------------+-------------------+-------------------+------------------+ +| Anonymous | Named | Any | Prohibit | ++-------------------+-------------------+-------------------+------------------+ +| Named | Anonymous | Any | Prohibit | ++-------------------+-------------------+-------------------+------------------+ +| Named | Named | Any | Allow | ++-------------------+-------------------+-------------------+------------------+ + +Enabling Page Table Check +========================= + +Build kernel with: + +- PAGE_TABLE_CHECK=y + Note, it can only be enabled on platforms where ARCH_SUPPORTS_PAGE_TABLE_CHECK + is available. + +- Boot with 'page_table_check=on' kernel parameter. + +Optionally, build kernel with PAGE_TABLE_CHECK_ENFORCED in order to have page +table support without extra kernel parameter. --- /dev/null +++ a/include/linux/page_table_check.h @@ -0,0 +1,147 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * Copyright (c) 2021, Google LLC. + * Pasha Tatashin <pasha.tatashin@soleen.com> + */ +#ifndef __LINUX_PAGE_TABLE_CHECK_H +#define __LINUX_PAGE_TABLE_CHECK_H + +#ifdef CONFIG_PAGE_TABLE_CHECK +#include <linux/jump_label.h> + +extern struct static_key_true page_table_check_disabled; +extern struct page_ext_operations page_table_check_ops; + +void __page_table_check_zero(struct page *page, unsigned int order); +void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr, + pte_t pte); +void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr, + pmd_t pmd); +void __page_table_check_pud_clear(struct mm_struct *mm, unsigned long addr, + pud_t pud); +void __page_table_check_pte_set(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +void __page_table_check_pmd_set(struct mm_struct *mm, unsigned long addr, + pmd_t *pmdp, pmd_t pmd); +void __page_table_check_pud_set(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, pud_t pud); + +static inline void page_table_check_alloc(struct page *page, unsigned int order) +{ + if (static_branch_likely(&page_table_check_disabled)) + return; + + __page_table_check_zero(page, order); +} + +static inline void page_table_check_free(struct page *page, unsigned int order) +{ + if (static_branch_likely(&page_table_check_disabled)) + return; + + __page_table_check_zero(page, order); +} + +static inline void page_table_check_pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t pte) +{ + if (static_branch_likely(&page_table_check_disabled)) + return; + + __page_table_check_pte_clear(mm, addr, pte); +} + +static inline void page_table_check_pmd_clear(struct mm_struct *mm, + unsigned long addr, pmd_t pmd) +{ + if (static_branch_likely(&page_table_check_disabled)) + return; + + __page_table_check_pmd_clear(mm, addr, pmd); +} + +static inline void page_table_check_pud_clear(struct mm_struct *mm, + unsigned long addr, pud_t pud) +{ + if (static_branch_likely(&page_table_check_disabled)) + return; + + __page_table_check_pud_clear(mm, addr, pud); +} + +static inline void page_table_check_pte_set(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + pte_t pte) +{ + if (static_branch_likely(&page_table_check_disabled)) + return; + + __page_table_check_pte_set(mm, addr, ptep, pte); +} + +static inline void page_table_check_pmd_set(struct mm_struct *mm, + unsigned long addr, pmd_t *pmdp, + pmd_t pmd) +{ + if (static_branch_likely(&page_table_check_disabled)) + return; + + __page_table_check_pmd_set(mm, addr, pmdp, pmd); +} + +static inline void page_table_check_pud_set(struct mm_struct *mm, + unsigned long addr, pud_t *pudp, + pud_t pud) +{ + if (static_branch_likely(&page_table_check_disabled)) + return; + + __page_table_check_pud_set(mm, addr, pudp, pud); +} + +#else + +static inline void page_table_check_alloc(struct page *page, unsigned int order) +{ +} + +static inline void page_table_check_free(struct page *page, unsigned int order) +{ +} + +static inline void page_table_check_pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t pte) +{ +} + +static inline void page_table_check_pmd_clear(struct mm_struct *mm, + unsigned long addr, pmd_t pmd) +{ +} + +static inline void page_table_check_pud_clear(struct mm_struct *mm, + unsigned long addr, pud_t pud) +{ +} + +static inline void page_table_check_pte_set(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + pte_t pte) +{ +} + +static inline void page_table_check_pmd_set(struct mm_struct *mm, + unsigned long addr, pmd_t *pmdp, + pmd_t pmd) +{ +} + +static inline void page_table_check_pud_set(struct mm_struct *mm, + unsigned long addr, pud_t *pudp, + pud_t pud) +{ +} + +#endif /* CONFIG_PAGE_TABLE_CHECK */ +#endif /* __LINUX_PAGE_TABLE_CHECK_H */ --- a/MAINTAINERS~mm-page-table-check +++ a/MAINTAINERS @@ -14387,6 +14387,15 @@ F: include/net/page_pool.h F: include/trace/events/page_pool.h F: net/core/page_pool.c +PAGE TABLE CHECK +M: Pasha Tatashin <pasha.tatashin@soleen.com> +M: Andrew Morton <akpm@linux-foundation.org> +L: linux-mm@kvack.org +S: Maintained +F: Documentation/vm/page_table_check.rst +F: include/linux/page_table_check.h +F: mm/page_table_check.c + PANASONIC LAPTOP ACPI EXTRAS DRIVER M: Kenneth Chan <kenneth.t.chan@gmail.com> L: platform-driver-x86@vger.kernel.org --- a/mm/Kconfig.debug~mm-page-table-check +++ a/mm/Kconfig.debug @@ -62,6 +62,30 @@ config PAGE_OWNER If unsure, say N. +config PAGE_TABLE_CHECK + bool "Check for invalid mappings in user page tables" + depends on ARCH_SUPPORTS_PAGE_TABLE_CHECK + select PAGE_EXTENSION + help + Check that anonymous page is not being mapped twice with read write + permissions. Check that anonymous and file pages are not being + erroneously shared. Since the checking is performed at the time + entries are added and removed to user page tables, leaking, corruption + and double mapping problems are detected synchronously. + + If unsure say "n". + +config PAGE_TABLE_CHECK_ENFORCED + bool "Enforce the page table checking by default" + depends on PAGE_TABLE_CHECK + help + Always enable page table checking. By default the page table checking + is disabled, and can be optionally enabled via page_table_check=on + kernel parameter. This config enforces that page table check is always + enabled. + + If unsure say "n". + config PAGE_POISONING bool "Poison pages after freeing" help --- a/mm/Makefile~mm-page-table-check +++ a/mm/Makefile @@ -112,6 +112,7 @@ obj-$(CONFIG_GENERIC_EARLY_IOREMAP) += e obj-$(CONFIG_CMA) += cma.o obj-$(CONFIG_MEMORY_BALLOON) += balloon_compaction.o obj-$(CONFIG_PAGE_EXTENSION) += page_ext.o +obj-$(CONFIG_PAGE_TABLE_CHECK) += page_table_check.o obj-$(CONFIG_CMA_DEBUGFS) += cma_debug.o obj-$(CONFIG_SECRETMEM) += secretmem.o obj-$(CONFIG_CMA_SYSFS) += cma_sysfs.o --- a/mm/page_alloc.c~mm-page-table-check +++ a/mm/page_alloc.c @@ -63,6 +63,7 @@ #include <linux/sched/rt.h> #include <linux/sched/mm.h> #include <linux/page_owner.h> +#include <linux/page_table_check.h> #include <linux/kthread.h> #include <linux/memcontrol.h> #include <linux/ftrace.h> @@ -1307,6 +1308,7 @@ static __always_inline bool free_pages_p if (memcg_kmem_enabled() && PageMemcgKmem(page)) __memcg_kmem_uncharge_page(page, order); reset_page_owner(page, order); + page_table_check_free(page, order); return false; } @@ -1346,6 +1348,7 @@ static __always_inline bool free_pages_p page_cpupid_reset_last(page); page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; reset_page_owner(page, order); + page_table_check_free(page, order); if (!PageHighMem(page)) { debug_check_no_locks_freed(page_address(page), @@ -2420,6 +2423,7 @@ inline void post_alloc_hook(struct page } set_page_owner(page, order, gfp_flags); + page_table_check_alloc(page, order); } static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, --- a/mm/page_ext.c~mm-page-table-check +++ a/mm/page_ext.c @@ -8,6 +8,7 @@ #include <linux/kmemleak.h> #include <linux/page_owner.h> #include <linux/page_idle.h> +#include <linux/page_table_check.h> /* * struct page extension @@ -75,6 +76,9 @@ static struct page_ext_operations *page_ #if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT) &page_idle_ops, #endif +#ifdef CONFIG_PAGE_TABLE_CHECK + &page_table_check_ops, +#endif }; unsigned long page_ext_size = sizeof(struct page_ext); --- /dev/null +++ a/mm/page_table_check.c @@ -0,0 +1,270 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* + * Copyright (c) 2021, Google LLC. + * Pasha Tatashin <pasha.tatashin@soleen.com> + */ +#include <linux/mm.h> +#include <linux/page_table_check.h> + +#undef pr_fmt +#define pr_fmt(fmt) "page_table_check: " fmt + +struct page_table_check { + atomic_t anon_map_count; + atomic_t file_map_count; +}; + +static bool __page_table_check_enabled __initdata = + IS_ENABLED(CONFIG_PAGE_TABLE_CHECK_ENFORCED); + +DEFINE_STATIC_KEY_TRUE(page_table_check_disabled); +EXPORT_SYMBOL(page_table_check_disabled); + +static int __init early_page_table_check_param(char *buf) +{ + if (!buf) + return -EINVAL; + + if (strcmp(buf, "on") == 0) + __page_table_check_enabled = true; + else if (strcmp(buf, "off") == 0) + __page_table_check_enabled = false; + + return 0; +} + +early_param("page_table_check", early_page_table_check_param); + +static bool __init need_page_table_check(void) +{ + return __page_table_check_enabled; +} + +static void __init init_page_table_check(void) +{ + if (!__page_table_check_enabled) + return; + static_branch_disable(&page_table_check_disabled); +} + +struct page_ext_operations page_table_check_ops = { + .size = sizeof(struct page_table_check), + .need = need_page_table_check, + .init = init_page_table_check, +}; + +static struct page_table_check *get_page_table_check(struct page_ext *page_ext) +{ + BUG_ON(!page_ext); + return (void *)(page_ext) + page_table_check_ops.offset; +} + +static inline bool pte_user_accessible_page(pte_t pte) +{ + return (pte_val(pte) & _PAGE_PRESENT) && (pte_val(pte) & _PAGE_USER); +} + +static inline bool pmd_user_accessible_page(pmd_t pmd) +{ + return pmd_leaf(pmd) && (pmd_val(pmd) & _PAGE_PRESENT) && + (pmd_val(pmd) & _PAGE_USER); +} + +static inline bool pud_user_accessible_page(pud_t pud) +{ + return pud_leaf(pud) && (pud_val(pud) & _PAGE_PRESENT) && + (pud_val(pud) & _PAGE_USER); +} + +/* + * An enty is removed from the page table, decrement the counters for that page + * verify that it is of correct type and counters do not become negative. + */ +static void page_table_check_clear(struct mm_struct *mm, unsigned long addr, + unsigned long pfn, unsigned long pgcnt) +{ + struct page_ext *page_ext; + struct page *page; + bool anon; + int i; + + if (!pfn_valid(pfn)) + return; + + page = pfn_to_page(pfn); + page_ext = lookup_page_ext(page); + anon = PageAnon(page); + + for (i = 0; i < pgcnt; i++) { + struct page_table_check *ptc = get_page_table_check(page_ext); + + if (anon) { + BUG_ON(atomic_read(&ptc->file_map_count)); + BUG_ON(atomic_dec_return(&ptc->anon_map_count) < 0); + } else { + BUG_ON(atomic_read(&ptc->anon_map_count)); + BUG_ON(atomic_dec_return(&ptc->file_map_count) < 0); + } + page_ext = page_ext_next(page_ext); + } +} + +/* + * A new enty is added to the page table, increment the counters for that page + * verify that it is of correct type and is not being mapped with a different + * type to a different process. + */ +static void page_table_check_set(struct mm_struct *mm, unsigned long addr, + unsigned long pfn, unsigned long pgcnt, + bool rw) +{ + struct page_ext *page_ext; + struct page *page; + bool anon; + int i; + + if (!pfn_valid(pfn)) + return; + + page = pfn_to_page(pfn); + page_ext = lookup_page_ext(page); + anon = PageAnon(page); + + for (i = 0; i < pgcnt; i++) { + struct page_table_check *ptc = get_page_table_check(page_ext); + + if (anon) { + BUG_ON(atomic_read(&ptc->file_map_count)); + BUG_ON(atomic_inc_return(&ptc->anon_map_count) > 1 && rw); + } else { + BUG_ON(atomic_read(&ptc->anon_map_count)); + BUG_ON(atomic_inc_return(&ptc->file_map_count) < 0); + } + page_ext = page_ext_next(page_ext); + } +} + +/* + * page is on free list, or is being allocated, verify that counters are zeroes + * crash if they are not. + */ +void __page_table_check_zero(struct page *page, unsigned int order) +{ + struct page_ext *page_ext = lookup_page_ext(page); + int i; + + BUG_ON(!page_ext); + for (i = 0; i < (1 << order); i++) { + struct page_table_check *ptc = get_page_table_check(page_ext); + + BUG_ON(atomic_read(&ptc->anon_map_count)); + BUG_ON(atomic_read(&ptc->file_map_count)); + page_ext = page_ext_next(page_ext); + } +} + +void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr, + pte_t pte) +{ + if (&init_mm == mm) + return; + + if (pte_user_accessible_page(pte)) { + page_table_check_clear(mm, addr, pte_pfn(pte), + PAGE_SIZE >> PAGE_SHIFT); + } +} +EXPORT_SYMBOL(__page_table_check_pte_clear); + +void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr, + pmd_t pmd) +{ + if (&init_mm == mm) + return; + + if (pmd_user_accessible_page(pmd)) { + page_table_check_clear(mm, addr, pmd_pfn(pmd), + PMD_PAGE_SIZE >> PAGE_SHIFT); + } +} +EXPORT_SYMBOL(__page_table_check_pmd_clear); + +void __page_table_check_pud_clear(struct mm_struct *mm, unsigned long addr, + pud_t pud) +{ + if (&init_mm == mm) + return; + + if (pud_user_accessible_page(pud)) { + page_table_check_clear(mm, addr, pud_pfn(pud), + PUD_PAGE_SIZE >> PAGE_SHIFT); + } +} +EXPORT_SYMBOL(__page_table_check_pud_clear); + +void __page_table_check_pte_set(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + pte_t old_pte; + + if (&init_mm == mm) + return; + + old_pte = *ptep; + if (pte_user_accessible_page(old_pte)) { + page_table_check_clear(mm, addr, pte_pfn(old_pte), + PAGE_SIZE >> PAGE_SHIFT); + } + + if (pte_user_accessible_page(pte)) { + page_table_check_set(mm, addr, pte_pfn(pte), + PAGE_SIZE >> PAGE_SHIFT, + pte_write(pte)); + } +} +EXPORT_SYMBOL(__page_table_check_pte_set); + +void __page_table_check_pmd_set(struct mm_struct *mm, unsigned long addr, + pmd_t *pmdp, pmd_t pmd) +{ + pmd_t old_pmd; + + if (&init_mm == mm) + return; + + old_pmd = *pmdp; + if (pmd_user_accessible_page(old_pmd)) { + page_table_check_clear(mm, addr, pmd_pfn(old_pmd), + PMD_PAGE_SIZE >> PAGE_SHIFT); + } + + if (pmd_user_accessible_page(pmd)) { + page_table_check_set(mm, addr, pmd_pfn(pmd), + PMD_PAGE_SIZE >> PAGE_SHIFT, + pmd_write(pmd)); + } +} +EXPORT_SYMBOL(__page_table_check_pmd_set); + +void __page_table_check_pud_set(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, pud_t pud) +{ + pud_t old_pud; + + if (&init_mm == mm) + return; + + old_pud = *pudp; + if (pud_user_accessible_page(old_pud)) { + page_table_check_clear(mm, addr, pud_pfn(old_pud), + PUD_PAGE_SIZE >> PAGE_SHIFT); + } + + if (pud_user_accessible_page(pud)) { + page_table_check_set(mm, addr, pud_pfn(pud), + PUD_PAGE_SIZE >> PAGE_SHIFT, + pud_write(pud)); + } +} +EXPORT_SYMBOL(__page_table_check_pud_set);

[067/146] mm: page table check

Commit Message

Patch