From patchwork Thu Feb 15 23:13:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13559256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 869D7C48BC4 for ; Thu, 15 Feb 2024 23:14:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B02F8D0006; Thu, 15 Feb 2024 18:14:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 03A028D0001; Thu, 15 Feb 2024 18:14:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF4D88D0006; Thu, 15 Feb 2024 18:14:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C96E58D0001 for ; Thu, 15 Feb 2024 18:14:42 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 77C45A27B9 for ; Thu, 15 Feb 2024 23:14:42 +0000 (UTC) X-FDA: 81795594804.08.6792E44 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.21]) by imf09.hostedemail.com (Postfix) with ESMTP id 75464140019 for ; Thu, 15 Feb 2024 23:14:40 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=M6wJ18vk; spf=pass (imf09.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708038880; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g/kJ77swLhvIcHLWexy2BoaMa/dtV5NhrzZv0PtiUPk=; b=jK/cI7dmUTM+moll0vFRW/lqrR3wL9mCkKf13ZUCUi9qAzikIE9p0eCeCR+6sC0FEwy3ij GulicIj/roODMV5I4V1qbpgAoRonKlEX5hqkKMTR6bSmHMPne81HX96THQUyG6AGcvNLqZ ncNdk5Kp45AJJcmUYebGZQ1GVBZOG1E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708038880; a=rsa-sha256; cv=none; b=554DuEv0WwRMxDDW0ZmJjF90FWncGlUTxE6ACoopfgcd0MaMny/dyYjMYAZQ0ebvJkmv/Z Gp/ot+aWcY4kmk+A1VLU+zlqgMIC3lwJGlkELm2a7+ScBJed1XBttJ5uCHEjb+JKKhKWfg ixFsPjPztZA8gNZdWRpU++B9mtFl2so= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=M6wJ18vk; spf=pass (imf09.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708038881; x=1739574881; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9ZHYd2M5Kk8D0EsHHIG1lih+Sz46XE6ZpNULUyzgA8g=; b=M6wJ18vk2hhZYHhEvtnPeSLNSpxbLngZVweUeps8JQ35WcDQ6JDpbg81 tp+2iMNWxloggIimCUDnEDImqaNnr6tDDa5wX8iYxKF9BD78FiYkNx66H wqIGEp/wIENP/sklkOKutwX+t6UEU/qSAN00fGigPXBNwGyi0HpIcAh38 GQILfANGbprm94vx1VfHi9stAVkmcnEpl9z9YGG7Pp5BCFBnmS4Mlm2rK prCYQUfoB4pUZ0Y8ECY2MnHG5bKKoo4lIFVCRl7tmvWeoboe4uLt5Kdmc 4Zgc6th64bYnvWFXVkMVIQQkTShSwt83PzsnMws3IuSff9YTPIV8PquWH g==; X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="2066307" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="2066307" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orvoesa113.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:38 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="912250188" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="912250188" Received: from yshin-mobl1.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.209.95.133]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:37 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [RFC PATCH 1/8] mm: Switch mm->get_unmapped_area() to a flag Date: Thu, 15 Feb 2024 15:13:25 -0800 Message-Id: <20240215231332.1556787-2-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> References: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 75464140019 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 7yg3xm1ekh474pyad47q8y77uw9oay3e X-HE-Tag: 1708038880-776163 X-HE-Meta: U2FsdGVkX1/ER9+smy2w5RsnzP4g0H2JBJe8YOlg3miOtXlD2oKFjqBl2UvcMUTQZWTxx85rKuXhOVM692Oe9cW55IRtIlbOFTnJyEEVYns7tijBxhuUnkUThhBb2nCrRX9v7clB+TpUErDEfg6i7dvtDxnmXU0SlZBPL4WB0lcJpmr9g0S0EFex8ucI4g4lT7+5+833iKLFnJ3qy7RxZZLaQnB2Zh1a+ORbwnAPo2BKl59s37x1jIod589ud3At1GhuAIF/o/u1hYmEsflLFCzD+VpRPiwm5fZbq6SCnkjTEDbAnwP8ZcW8aiEb7DJB/D3u0m4Z83L2F9wtDG+SLbZo1iVMxMomp7rrTaZ201i2R3vHuIqAvx69BlZ/2mPVILedfFkmdHci81JIklDWs2VWBdWYcVx0Q/DBC08HjthU7sYu1BS8J7wSRU+326oG5WungwnboSVxBL8GhD5/fTdlGhnWYlaqI0XJ3+oswAXZO7yhEA1Upmj9gRcN4/l2aoPjvcQwUrTYE8qyuOooinO9XbfPnxfz0HN6qCbKxClvFbovCqEcP3ta9g6zaP6G2ws2kN5EUqdQs/McYiGsYTQrIsmeSofgwBBDl4s+pIV7vHj0e6y8KCbWswOF2upj8WrI8OFr6SXcS8cLXrVOYs9rrV8KdgEm8ym67JMaHqK6w2ZKPjvBn5BWJokiXo0U3nj27bbdHlPEdY49KvWyiGY3n6VsMOh3pHz112ZdOiz1V6n8oew1oCX28BSeHfEO3YK/WC2ZFtyXNUFXq41IwWtwuFGVaLdNXRgszaLz/3/0gZ1BeOH7USV2zt50I27ncD3Zzl4iogKCcI1xWSBylYp7G0awGIfXvaG2faIzkbwSJ6tuF1PBdB/cksWdC8QOe2eBMHN3h0S1rqprQr1V91M4B8XkIStSaCJMzghp/IOBURXi30qGVA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The mm_struct contains a function pointer *get_unmapped_area(), which is set to either arch_get_unmapped_area() or arch_get_unmapped_area_topdown() during the initialization of the mm. Since the function pointer only ever points to two functions that are named the same across all arch's, a function pointer is not really required. In addition future changes will want to add versions of the functions that take additional arguments. So to save a pointers worth of bytes in mm_struct, and prevent adding additional function pointers to mm_struct in future changes, remove it and keep the information about which get_unmapped_area() to use in a flag. Introduce a helper, mm_get_unmapped_area(), to easily convert code that refers to the old function pointer to instead select and call either arch_get_unmapped_area() or arch_get_unmapped_area_topdown() based on the flag. Then drop the mm->get_unmapped_area() function pointer. Leave the get_unmapped_area() pointer in struct file_operations alone. The main purpose of this change is to reorganize in preparation for future changes, but it also converts the calls of mm->get_unmapped_area() from indirect branches into a direct ones. The stress-ng bigheap benchmark calls realloc a lot, which calls through get_unmapped_area() in the kernel. On x86, the change yielded a ~4% improvement there. (bogo ops/s (usr+sys time)) In testing a few x86 configs, removing the pointer unfortunately didn't result in any actual size reductions in the compiled layout of mm_struct. But depending on compiler or arch alignment requirements, the change could possibly shrink the size of mm_struct. Signed-off-by: Rick Edgecombe Acked-by: Dave Hansen Acked-by: Liam R. Howlett Reviewed-by: Kirill A. Shutemov --- arch/s390/mm/hugetlbpage.c | 2 +- arch/s390/mm/mmap.c | 4 ++-- arch/sparc/kernel/sys_sparc_64.c | 15 ++++++--------- arch/sparc/mm/hugetlbpage.c | 2 +- arch/x86/kernel/cpu/sgx/driver.c | 2 +- arch/x86/mm/hugetlbpage.c | 2 +- arch/x86/mm/mmap.c | 4 ++-- drivers/char/mem.c | 2 +- drivers/dax/device.c | 6 +++--- fs/hugetlbfs/inode.c | 2 +- fs/proc/inode.c | 15 ++++++++------- fs/ramfs/file-mmu.c | 2 +- include/linux/mm_types.h | 6 +----- include/linux/sched/coredump.h | 1 + include/linux/sched/mm.h | 5 +++++ io_uring/io_uring.c | 2 +- mm/debug.c | 6 ------ mm/huge_memory.c | 6 +++--- mm/mmap.c | 21 ++++++++++++++++++--- mm/shmem.c | 11 +++++------ mm/util.c | 6 +++--- 21 files changed, 65 insertions(+), 57 deletions(-) diff --git a/arch/s390/mm/hugetlbpage.c b/arch/s390/mm/hugetlbpage.c index 297a6d897d5a..c2d2850ec8d5 100644 --- a/arch/s390/mm/hugetlbpage.c +++ b/arch/s390/mm/hugetlbpage.c @@ -328,7 +328,7 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, goto check_asce_limit; } - if (mm->get_unmapped_area == arch_get_unmapped_area) + if (!test_bit(MMF_TOPDOWN, &mm->flags)) addr = hugetlb_get_unmapped_area_bottomup(file, addr, len, pgoff, flags); else diff --git a/arch/s390/mm/mmap.c b/arch/s390/mm/mmap.c index fc9a7dc26c5e..cd52d72b59cf 100644 --- a/arch/s390/mm/mmap.c +++ b/arch/s390/mm/mmap.c @@ -182,10 +182,10 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) */ if (mmap_is_legacy(rlim_stack)) { mm->mmap_base = mmap_base_legacy(random_factor); - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); } else { mm->mmap_base = mmap_base(random_factor, rlim_stack); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; + set_bit(MMF_TOPDOWN, &mm->flags); } } diff --git a/arch/sparc/kernel/sys_sparc_64.c b/arch/sparc/kernel/sys_sparc_64.c index 1e9a9e016237..1dbf7211666e 100644 --- a/arch/sparc/kernel/sys_sparc_64.c +++ b/arch/sparc/kernel/sys_sparc_64.c @@ -218,14 +218,10 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, unsigned long len, unsigned long pgoff, unsigned long flags) { unsigned long align_goal, addr = -ENOMEM; - unsigned long (*get_area)(struct file *, unsigned long, - unsigned long, unsigned long, unsigned long); - - get_area = current->mm->get_unmapped_area; if (flags & MAP_FIXED) { /* Ok, don't mess with it. */ - return get_area(NULL, orig_addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, NULL, orig_addr, len, pgoff, flags); } flags &= ~MAP_SHARED; @@ -238,7 +234,8 @@ unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, u align_goal = (64UL * 1024); do { - addr = get_area(NULL, orig_addr, len + (align_goal - PAGE_SIZE), pgoff, flags); + addr = mm_get_unmapped_area(current->mm, NULL, orig_addr, + len + (align_goal - PAGE_SIZE), pgoff, flags); if (!(addr & ~PAGE_MASK)) { addr = (addr + (align_goal - 1UL)) & ~(align_goal - 1UL); break; @@ -256,7 +253,7 @@ unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, u * be obtained. */ if (addr & ~PAGE_MASK) - addr = get_area(NULL, orig_addr, len, pgoff, flags); + addr = mm_get_unmapped_area(current->mm, NULL, orig_addr, len, pgoff, flags); return addr; } @@ -292,7 +289,7 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) gap == RLIM_INFINITY || sysctl_legacy_va_layout) { mm->mmap_base = TASK_UNMAPPED_BASE + random_factor; - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); } else { /* We know it's 32-bit */ unsigned long task_size = STACK_TOP32; @@ -303,7 +300,7 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) gap = (task_size / 6 * 5); mm->mmap_base = PAGE_ALIGN(task_size - gap - random_factor); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; + set_bit(MMF_TOPDOWN, &mm->flags); } } diff --git a/arch/sparc/mm/hugetlbpage.c b/arch/sparc/mm/hugetlbpage.c index b432500c13a5..38a1bef47efb 100644 --- a/arch/sparc/mm/hugetlbpage.c +++ b/arch/sparc/mm/hugetlbpage.c @@ -123,7 +123,7 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, (!vma || addr + len <= vm_start_gap(vma))) return addr; } - if (mm->get_unmapped_area == arch_get_unmapped_area) + if (!test_bit(MMF_TOPDOWN, &mm->flags)) return hugetlb_get_unmapped_area_bottomup(file, addr, len, pgoff, flags); else diff --git a/arch/x86/kernel/cpu/sgx/driver.c b/arch/x86/kernel/cpu/sgx/driver.c index 262f5fb18d74..22b65a5f5ec6 100644 --- a/arch/x86/kernel/cpu/sgx/driver.c +++ b/arch/x86/kernel/cpu/sgx/driver.c @@ -113,7 +113,7 @@ static unsigned long sgx_get_unmapped_area(struct file *file, if (flags & MAP_FIXED) return addr; - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); } #ifdef CONFIG_COMPAT diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index 5804bbae4f01..6d77c0039617 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -141,7 +141,7 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, } get_unmapped_area: - if (mm->get_unmapped_area == arch_get_unmapped_area) + if (!test_bit(MMF_TOPDOWN, &mm->flags)) return hugetlb_get_unmapped_area_bottomup(file, addr, len, pgoff, flags); else diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c index c90c20904a60..a2cabb1c81e1 100644 --- a/arch/x86/mm/mmap.c +++ b/arch/x86/mm/mmap.c @@ -129,9 +129,9 @@ static void arch_pick_mmap_base(unsigned long *base, unsigned long *legacy_base, void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) { if (mmap_is_legacy()) - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); else - mm->get_unmapped_area = arch_get_unmapped_area_topdown; + set_bit(MMF_TOPDOWN, &mm->flags); arch_pick_mmap_base(&mm->mmap_base, &mm->mmap_legacy_base, arch_rnd(mmap64_rnd_bits), task_size_64bit(0), diff --git a/drivers/char/mem.c b/drivers/char/mem.c index 3c6670cf905f..9b80e622ae80 100644 --- a/drivers/char/mem.c +++ b/drivers/char/mem.c @@ -544,7 +544,7 @@ static unsigned long get_unmapped_area_zero(struct file *file, } /* Otherwise flags & MAP_PRIVATE: with no shmem object beneath it */ - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); #else return -ENOSYS; #endif diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 93ebedc5ec8c..47c126d37b59 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -329,14 +329,14 @@ static unsigned long dax_get_unmapped_area(struct file *filp, if ((off + len_align) < off) goto out; - addr_align = current->mm->get_unmapped_area(filp, addr, len_align, - pgoff, flags); + addr_align = mm_get_unmapped_area(current->mm, filp, addr, len_align, + pgoff, flags); if (!IS_ERR_VALUE(addr_align)) { addr_align += (off - addr_align) & (align - 1); return addr_align; } out: - return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags); } static const struct address_space_operations dev_dax_aops = { diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index f757d4f7ad98..a63d2eee086f 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -242,7 +242,7 @@ generic_hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * If architectures have special needs, they should define their own * version of hugetlb_get_unmapped_area. */ - if (mm->get_unmapped_area == arch_get_unmapped_area_topdown) + if (test_bit(MMF_TOPDOWN, &mm->flags)) return hugetlb_get_unmapped_area_topdown(file, addr, len, pgoff, flags); return hugetlb_get_unmapped_area_bottomup(file, addr, len, diff --git a/fs/proc/inode.c b/fs/proc/inode.c index b33e490e3fd9..6f4c2e21e68f 100644 --- a/fs/proc/inode.c +++ b/fs/proc/inode.c @@ -454,15 +454,16 @@ pde_get_unmapped_area(struct proc_dir_entry *pde, struct file *file, unsigned lo unsigned long len, unsigned long pgoff, unsigned long flags) { - typeof_member(struct proc_ops, proc_get_unmapped_area) get_area; - - get_area = pde->proc_ops->proc_get_unmapped_area; + if (pde->proc_ops->proc_get_unmapped_area) + return pde->proc_ops->proc_get_unmapped_area(file, orig_addr, + len, pgoff, + flags); #ifdef CONFIG_MMU - if (!get_area) - get_area = current->mm->get_unmapped_area; + else + return mm_get_unmapped_area(current->mm, file, orig_addr, + len, pgoff, flags); #endif - if (get_area) - return get_area(file, orig_addr, len, pgoff, flags); + return orig_addr; } diff --git a/fs/ramfs/file-mmu.c b/fs/ramfs/file-mmu.c index c7a1aa3c882b..b45c7edc3225 100644 --- a/fs/ramfs/file-mmu.c +++ b/fs/ramfs/file-mmu.c @@ -35,7 +35,7 @@ static unsigned long ramfs_mmu_get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); } const struct file_operations ramfs_file_operations = { diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 957ce38768b2..f01c01b4a4fc 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -745,11 +745,7 @@ struct mm_struct { } ____cacheline_aligned_in_smp; struct maple_tree mm_mt; -#ifdef CONFIG_MMU - unsigned long (*get_unmapped_area) (struct file *filp, - unsigned long addr, unsigned long len, - unsigned long pgoff, unsigned long flags); -#endif + unsigned long mmap_base; /* base of mmap area */ unsigned long mmap_legacy_base; /* base of mmap area in bottom-up allocations */ #ifdef CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES diff --git a/include/linux/sched/coredump.h b/include/linux/sched/coredump.h index 02f5090ffea2..428e440424c5 100644 --- a/include/linux/sched/coredump.h +++ b/include/linux/sched/coredump.h @@ -74,6 +74,7 @@ static inline int get_dumpable(struct mm_struct *mm) #define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP) #define MMF_OOM_REAP_QUEUED 25 /* mm was queued for oom_reaper */ #define MMF_MULTIPROCESS 26 /* mm is shared between processes */ +#define MMF_TOPDOWN 27 /* mm is shared between processes */ /* * MMF_HAS_PINNED: Whether this mm has pinned any pages. This can be either * replaced in the future by mm.pinned_vm when it becomes stable, or grow into diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 9a19f1b42f64..cde946e926d8 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -8,6 +8,7 @@ #include #include #include +#include /* * Routines for handling mm_structs @@ -186,6 +187,10 @@ arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +unsigned long mm_get_unmapped_area(struct mm_struct *mm, struct file *filp, + unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags); + unsigned long generic_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9626a363f121..b3c4ec115989 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3569,7 +3569,7 @@ static unsigned long io_uring_mmu_get_unmapped_area(struct file *filp, #else addr = 0UL; #endif - return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags); } #else /* !CONFIG_MMU */ diff --git a/mm/debug.c b/mm/debug.c index ee533a5ceb79..32db5de8e1e7 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -162,9 +162,6 @@ EXPORT_SYMBOL(dump_vma); void dump_mm(const struct mm_struct *mm) { pr_emerg("mm %px task_size %lu\n" -#ifdef CONFIG_MMU - "get_unmapped_area %px\n" -#endif "mmap_base %lu mmap_legacy_base %lu\n" "pgd %px mm_users %d mm_count %d pgtables_bytes %lu map_count %d\n" "hiwater_rss %lx hiwater_vm %lx total_vm %lx locked_vm %lx\n" @@ -190,9 +187,6 @@ void dump_mm(const struct mm_struct *mm) "def_flags: %#lx(%pGv)\n", mm, mm->task_size, -#ifdef CONFIG_MMU - mm->get_unmapped_area, -#endif mm->mmap_base, mm->mmap_legacy_base, mm->pgd, atomic_read(&mm->mm_users), atomic_read(&mm->mm_count), diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 86ee29b5c39c..e9ef43a719a5 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -641,8 +641,8 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, if (len_pad < len || (off + len_pad) < off) return 0; - ret = current->mm->get_unmapped_area(filp, addr, len_pad, - off >> PAGE_SHIFT, flags); + ret = mm_get_unmapped_area(current->mm, filp, addr, len_pad, + off >> PAGE_SHIFT, flags); /* * The failure might be due to length padding. The caller will retry @@ -672,7 +672,7 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, if (ret) return ret; - return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags); } EXPORT_SYMBOL_GPL(thp_get_unmapped_area); diff --git a/mm/mmap.c b/mm/mmap.c index aa82eec17489..b61bc515c729 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1807,7 +1807,8 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { unsigned long (*get_area)(struct file *, unsigned long, - unsigned long, unsigned long, unsigned long); + unsigned long, unsigned long, unsigned long) + = NULL; unsigned long error = arch_mmap_check(addr, len, flags); if (error) @@ -1817,7 +1818,6 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, if (len > TASK_SIZE) return -ENOMEM; - get_area = current->mm->get_unmapped_area; if (file) { if (file->f_op->get_unmapped_area) get_area = file->f_op->get_unmapped_area; @@ -1834,7 +1834,11 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, get_area = thp_get_unmapped_area; } - addr = get_area(file, addr, len, pgoff, flags); + if (get_area) + addr = get_area(file, addr, len, pgoff, flags); + else + addr = mm_get_unmapped_area(current->mm, file, addr, len, + pgoff, flags); if (IS_ERR_VALUE(addr)) return addr; @@ -1849,6 +1853,17 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, EXPORT_SYMBOL(get_unmapped_area); +unsigned long +mm_get_unmapped_area(struct mm_struct *mm, struct file *file, + unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags) +{ + if (test_bit(MMF_TOPDOWN, &mm->flags)) + return arch_get_unmapped_area_topdown(file, addr, len, pgoff, flags); + return arch_get_unmapped_area(file, addr, len, pgoff, flags); +} +EXPORT_SYMBOL(mm_get_unmapped_area); + /** * find_vma_intersection() - Look up the first VMA which intersects the interval * @mm: The process address space. diff --git a/mm/shmem.c b/mm/shmem.c index 0d1ce70bce38..a8cac92dff88 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2242,8 +2242,6 @@ unsigned long shmem_get_unmapped_area(struct file *file, unsigned long uaddr, unsigned long len, unsigned long pgoff, unsigned long flags) { - unsigned long (*get_area)(struct file *, - unsigned long, unsigned long, unsigned long, unsigned long); unsigned long addr; unsigned long offset; unsigned long inflated_len; @@ -2253,8 +2251,8 @@ unsigned long shmem_get_unmapped_area(struct file *file, if (len > TASK_SIZE) return -ENOMEM; - get_area = current->mm->get_unmapped_area; - addr = get_area(file, uaddr, len, pgoff, flags); + addr = mm_get_unmapped_area(current->mm, file, uaddr, len, pgoff, + flags); if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) return addr; @@ -2311,7 +2309,8 @@ unsigned long shmem_get_unmapped_area(struct file *file, if (inflated_len < len) return addr; - inflated_addr = get_area(NULL, uaddr, inflated_len, 0, flags); + inflated_addr = mm_get_unmapped_area(current->mm, NULL, uaddr, + inflated_len, 0, flags); if (IS_ERR_VALUE(inflated_addr)) return addr; if (inflated_addr & ~PAGE_MASK) @@ -4757,7 +4756,7 @@ unsigned long shmem_get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); } #endif diff --git a/mm/util.c b/mm/util.c index 744b4d7e3fae..10c836a75e66 100644 --- a/mm/util.c +++ b/mm/util.c @@ -452,17 +452,17 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) if (mmap_is_legacy(rlim_stack)) { mm->mmap_base = TASK_UNMAPPED_BASE + random_factor; - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); } else { mm->mmap_base = mmap_base(random_factor, rlim_stack); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; + set_bit(MMF_TOPDOWN, &mm->flags); } } #elif defined(CONFIG_MMU) && !defined(HAVE_ARCH_PICK_MMAP_LAYOUT) void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) { mm->mmap_base = TASK_UNMAPPED_BASE; - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); } #endif From patchwork Thu Feb 15 23:13:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13559258 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D46DC48BEC for ; Thu, 15 Feb 2024 23:14:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4D606B0085; Thu, 15 Feb 2024 18:14:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D6C56B0087; Thu, 15 Feb 2024 18:14:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 829826B0089; Thu, 15 Feb 2024 18:14:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 70EF16B0085 for ; Thu, 15 Feb 2024 18:14:44 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 45699160250 for ; Thu, 15 Feb 2024 23:14:44 +0000 (UTC) X-FDA: 81795594888.25.684EE7C Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.21]) by imf16.hostedemail.com (Postfix) with ESMTP id 2EE06180003 for ; Thu, 15 Feb 2024 23:14:41 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jBom+MP4; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf16.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708038882; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GJ3WV1MyMnvFy0laLvbLjyXhu5lENNpTuhfkEsO4ulk=; b=u02u3N7sl/9vEvjDNMBvlwj5a3sflNDMhQSxbI75l6FmRhFSM8k9nmuwgka+SO0GL69B0B e7moQAoeopuziViI7pDZhioYlla9x0y+i1MglXWmhsD/bR8SKuuh17/jEK1eVCiYM8Ieom cCmkSXM0cZd9yBQ9mqjlCOzlQIJDlKU= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jBom+MP4; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf16.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708038882; a=rsa-sha256; cv=none; b=P0FIJJJ/lNEHPrhOURmIYY6RLP2UJk4TGTQStq6OBkjUhbNyMmo0TIjyOXjXz8UaCzqvEC +z7kP2Esiib/zMJCMYvGrPG1fTOA2x7Am9AsE9LK79qBW52R5z8g1cgi/pgn+16n0BixvD 6+v+hvj3iVRSFx0cdPc8vIxdE1/W5No= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708038882; x=1739574882; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=NmgdWGUKdUuPH56dpQmxlcj6RB+edyhe9JvhgeXosrc=; b=jBom+MP4s6hiY331pIalwviUYgR9uAD1q7y0nPUtZI+Jemzu2XXrhzHH KbTEs4bJjhREsfUxLhW3J2OfqFp9dzfPaWAuV2GPmcwBoQr95QIORpmMz fX0HorQW3YbRxGFcQ8z/MQljhPtODsih4/PcgehooumJLTXDs8wNmfBKC l/EeiA2vGcd00dKGUUhLCL1SbxVTeLwniDo9V0vMI9U2CMbJ9fcMF5RkP S7BI0073+pozYIdSypuM21dtEQbYrx6AAYY2gn1TgChg6WeLeLLbc90a/ vBRLNRsiXJLhhh0LN9q6HO59wyS81uiL4k5USJ9Rikyc8P3CGl4avu8wl Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="2066319" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="2066319" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orvoesa113.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:39 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="912250191" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="912250191" Received: from yshin-mobl1.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.209.95.133]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:38 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [RFC PATCH 2/8] mm: Introduce arch_get_unmapped_area_vmflags() Date: Thu, 15 Feb 2024 15:13:26 -0800 Message-Id: <20240215231332.1556787-3-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> References: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2EE06180003 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: z6pmki6nf93x1q1fiynoya9rwp1zwcf6 X-HE-Tag: 1708038881-788467 X-HE-Meta: U2FsdGVkX18yMQGSrwK4geSJHVI6frKGZVAZSYcd39cJll3PoltvtOdLSQwqIpZLor+em0387QyjIM4Os8laxs1fBUvjZtlWemiqUBv0366TWf7KNe/lEkpgCMqSpaJ56Y43A/slBvk8EmOruZO7kkf0fkOD6PI/RTCqM1A6TRakXEqQ3CvGPlCdXyKMNy86AWA4ZyF2IR5iuGKmVvSx/pAQlrk6+LpT/E1O3lPzcc0uGPjpDlXjOEAR85URDOUxIJnxvAd8Gf15uDaVGQCBxFiMwTvEEh+eDwGDpoS3A+nLkYGnOxnP58kcP+RHUVUlSTYSYMG128DHwwG/SYTD8P1euUU9UMGyDhXyAvjqwNzHcy3uhPQdD+LAIaqFC1/ZTdXl3jughbzZa/9swfyVFeXL9dAbp/QYcBY//bAambYBcTrEqOQ8hQ0L45QU8yikOAX8g9JSKwMREzcyBDXK3w2YpkRsTvSyQLmfodrb3ac/VKIcKdtjIeGkCFbkaM+j/UKu0hfRkCD8KK9qq3IZVp43dqzbV9trO+otciMd4BESLIBVSzlNaVujx2sVtlBkgwTzIOrpxESSGYuNLh893UJut7wGfB2mxf2EHl633C3YO5ZznzGUhWHopCbwkqtwYaFdSCx0qFgD05JJe5c6dtTki4Rv4ItJdRm2bzrZn5dBAtD9FyQCqiZ76Hkrwppq09ALzhkFo/zg3kgjCo34KuIRwOAy7xk8scQCHQipH3whDYG9THi5C7WlZi8zHSPj33dcVXu9/sOcRJoSvXh6i7y+omcnzMHdc5o9SqU2agdpXHwBtBLz4Nx/Q9hKOYF3acRSxF5WCg+j5S82biLlvqhaK+A6c2dUdPaUKcYzAy5Mqn2jx5ZGLI73NddHSBhMnX1/98sNR87YTI2VTmHU/F+ZQoFJAxfkxXpR8G66oMxobymazQTMNdkpI+2Lg12d9Qk7rUoYLip7Q8FSFvS JKbQeYwA JMIfrZGjLNlOc8xyNocK4VmP8Gaywv5pdL/2MaTT1pRVErfYQDEMeU13iT8vjao4KOYHqJ5Vvp5ebDlGPaPabfOFScSgeQ8RCY6K4FmXATI6Yns8sNGjacMyJKPZNOLqzwKaNkBaNGhtmNN+9M3r+dMh4X+bqdF10XiO2B7yTOt9AjphoYROt68fGAKqXN3RwIykrSReHfbMANn3+HaHZsy/rEOWTO9rgyrWwlBeWp4cvYY8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. In order to take the start gap into account, the maple tree search needs to know the size of start gap the new mapping will need. The call chain from do_mmap() to the actual maple tree search looks like this: do_mmap(size, vm_flags, map_flags, ..) mm/mmap.c:get_unmapped_area(size, map_flags, ...) arch_get_unmapped_area(size, map_flags, ...) vm_unmapped_area(struct vm_unmapped_area_info) One option would be to add another MAP_ flag to mean a one page start gap (as is for shadow stack), but this consumes a flag unnecessarily. Another option could be to simply increase the size passed in do_mmap() by the start gap size, and adjust after the fact, but this will interfere with the alignment requirements passed in struct vm_unmapped_area_info, and unknown to mmap.c. Instead, introduce variants of arch_get_unmapped_area/_topdown() that take vm_flags. In future changes, these variants can be used in mmap.c:get_unmapped_area() to allow the vm_flags to be passed through to vm_unmapped_area(), while preserving the normal arch_get_unmapped_area/_topdown() for the existing callers. Signed-off-by: Rick Edgecombe --- include/linux/sched/mm.h | 17 +++++++++++++++++ mm/mmap.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index cde946e926d8..7b44441865c5 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -191,6 +191,23 @@ unsigned long mm_get_unmapped_area(struct mm_struct *mm, struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +extern unsigned long +arch_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t vm_flags); +extern unsigned long +arch_get_unmapped_area_topdown_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t); + +unsigned long mm_get_unmapped_area_vmflags(struct mm_struct *mm, + struct file *filp, + unsigned long addr, + unsigned long len, + unsigned long pgoff, + unsigned long flags, + vm_flags_t vm_flags); + unsigned long generic_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, diff --git a/mm/mmap.c b/mm/mmap.c index b61bc515c729..2021bc040e81 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1802,6 +1802,34 @@ arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr, } #endif +#ifndef HAVE_ARCH_UNMAPPED_AREA_VMFLAGS +extern unsigned long +arch_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags) +{ + return arch_get_unmapped_area(filp, addr, len, pgoff, flags); +} + +extern unsigned long +arch_get_unmapped_area_topdown_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t vm_flags) +{ + return arch_get_unmapped_area_topdown(filp, addr, len, pgoff, flags); +} +#endif + +unsigned long mm_get_unmapped_area_vmflags(struct mm_struct *mm, struct file *filp, + unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags) +{ + if (test_bit(MMF_TOPDOWN, &mm->flags)) + return arch_get_unmapped_area_topdown_vmflags(filp, addr, len, pgoff, + flags, vm_flags); + return arch_get_unmapped_area_vmflags(filp, addr, len, pgoff, flags, vm_flags); +} + unsigned long get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) From patchwork Thu Feb 15 23:13:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13559259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B73DEC48BC4 for ; Thu, 15 Feb 2024 23:14:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D00026B0087; Thu, 15 Feb 2024 18:14:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C460A6B008C; Thu, 15 Feb 2024 18:14:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8EC2A6B0088; Thu, 15 Feb 2024 18:14:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 73F7F6B0087 for ; Thu, 15 Feb 2024 18:14:44 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4A71F1606E4 for ; Thu, 15 Feb 2024 23:14:44 +0000 (UTC) X-FDA: 81795594888.20.02D2996 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.21]) by imf06.hostedemail.com (Postfix) with ESMTP id 3D26C180019 for ; Thu, 15 Feb 2024 23:14:42 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="YM/GTh8e"; spf=pass (imf06.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708038882; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iXPoKvTibJTs5Gz/12bFcL7AnHyYRAgKArZxxbAmZoY=; b=TLk0tMsn/uFDFT6hKBm3uGFOmD9//Ht25ngyWTQmvGsIkUy1fQJfD0Q8ESE6y5wR0AG8Sa RL95H+OufBWcB6kVSqCVT6mNqPFJUrfPvzoRNHVI3TDq6stXTzP8AXpMMos+AoefHtDG8L 6fPgZepyLrKK6PJOkHQ8+kCAWUuPln4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708038882; a=rsa-sha256; cv=none; b=UIXFMH1hE9JtWyLGxsfmHrBvcf6jy/91DotQuMAnh67SSZjwAh1pH6QRrsW5N2TRYtQazI kp/83sauWhBwahEczApa2m8NAsWQyjQvJQYfpPdPE38CnrFbKipAuoC3AnozkEUqt5wNx/ rhf12fMvYOUgVAGii+HBNh3FASk8lNw= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="YM/GTh8e"; spf=pass (imf06.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708038882; x=1739574882; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=unKAy/CGTalz/NFNrhf3RgWUr6Aw4RHxK63+GyNg8+4=; b=YM/GTh8eK6mmi7U9ztodoNavv2KsLhrv/Sw40BG7ed/5Y/ArNFlMiQdt AhW1OAnRzD+XSqg6dVq8RnXL+IipxMwuQwAYURyYBG3zf3jgjuhj+wvz0 MuUNearUX/GMA0y0QSBUp5mi4/0IreoGqW5jWJo8zQKvGcPCF3FqI0dNT kvCkrS8Kc/CJNaqxDwqNCMpB9+W69FXpZHVlmGLY9gigJThblFJTyIta3 ZHd4h9FL3YPa2Z1P+9ktN5/M+8KUwLGNv98cGw5JgsXby+lBzJDubLQne 5xoie9YIb97VNby7QW8BuQtfaNuQLrq0Y2aHHCu3dqiZKtGWLzYYfPz/+ Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="2066335" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="2066335" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orvoesa113.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:40 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="912250194" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="912250194" Received: from yshin-mobl1.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.209.95.133]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:39 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [RFC PATCH 3/8] mm: Use get_unmapped_area_vmflags() Date: Thu, 15 Feb 2024 15:13:27 -0800 Message-Id: <20240215231332.1556787-4-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> References: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3D26C180019 X-Rspam-User: X-Stat-Signature: qon844hi5zdbudfb856jhqkjby5n1939 X-Rspamd-Server: rspam03 X-HE-Tag: 1708038882-313299 X-HE-Meta: U2FsdGVkX1+L3uJffOXEWZot9pKoKS8YZ4HD2UFuK3TtEovEFCnvZnbVQTB3Jan4Vbx7tPOnReiLCtWp6+gI9WoOZtwiFmD0LQHn6E6XvWNY455+R+2JDdgKZLr/A1xsjdEOK3B8mHLceCueamei6Du0cmFEAnXL/KCkXUr8VWAfIFt98NVXSY7CXAZauq9cvubwwmBZZWgoWWaw7oh+8LI9nNYwv3psfB+5FE9FYMpa/b0U5geWi2k+qb7MDWheNUnTsHK0h5tSPgxFWORwvwLBrz/G0dJtX4VJUwI2tnMn2le9dIyrjaLqm6SU2IaEFU2TVoclJb+iMStmaz4QSvTVfByN6W3iYaLdM1IkJfPRANtaBAEL/Eta98Qa8HbMsIycNKcawKUIcB1cgqeIrKaYj0VAXXHXbvWc69jmZfbzLJM1TLFu71XocorAErew1NfUP+byUJETWPt6c1OVkEi7nqnd9+Ifmvm0BihrY7dvlKKOxkS8z64kNrTCUqjsL8frIwFZ5X1OKOGQmSBNbzDCesQH/V/hZyDCvaSGy8IYBUmQ8bcQV6/pzBxICNNtHDidAKQy8J62U6bi8EwG41O3mpmnPMp7I3BuJyCokQPJFIw3F8inTMsaeQ3FlcuLZ7hjjLoQSbGYMf7Kgf1h+plLEW8tTsRKfCttdfIR6wg5wFJb2e/rFpvJAHGTFJHbi3vHAqo5ax4XMOrf2bDnAvmGNgrb12+5lyWWXN/QnwKFbHAgM/1XyRwmOL14iod7/gSdl+2HFQCqJQBQwvQvQQK2v1E+9+SQisAJYliFClXtrJOAy6SZeoVft5kdK94nvGJzmPmLC/CB8RHl+edEJYOCb7jBYh55NF2g0V69fPrStkAwzA678loled8Tqg/FEihk1BqfwcL6jZe7DiuZJ3e82iB9jjiBxjxQi/TMEPrMEssmiNrtyvA1NBBc+8ISHC+2w7B2YUPrIpj2RtO sjEv+a5c Flwh+lpBBRY70fjttMnHWPSYV7vmv1iNyjECSlsxKmaWCndl3ZRsSw85h5GO+kcUNAFwntl8ne+GNdOXdt0sukKSaZalQhlSNAORLVym5mUuL2NJEIN4xKXbKGV2HEGROQczo7PM2jglwz0hDZVMp4ewFe5kzREsujix9A9YAWsec5lVlpzvnffqDEG7AMFJU3XqSas6+7RnH9R2Z726fM6WaC5RiwXPH5NG719hR3fMaeg/4pjWXTkjRqg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. Use mm_get_unmapped_area_vmflags() in the do_mmap() so future changes can cause shadow stack mappings to be placed with a guard gap. Also use the THP variant that takes vm_flags, such that THP shadow stack can get the same treatment. Adjust the vm_flags calculation to happen earlier so that the vm_flags can be passed into __get_unmapped_area(). Signed-off-by: Rick Edgecombe --- include/linux/mm.h | 2 ++ mm/mmap.c | 38 ++++++++++++++++++++++---------------- 2 files changed, 24 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index da5219b48d52..9addf16dbf18 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3352,6 +3352,8 @@ unsigned long randomize_stack_top(unsigned long stack_top); unsigned long randomize_page(unsigned long start, unsigned long range); extern unsigned long get_unmapped_area(struct file *, unsigned long, unsigned long, unsigned long, unsigned long); +extern unsigned long __get_unmapped_area(struct file *, unsigned long, unsigned long, + unsigned long, unsigned long, vm_flags_t); extern unsigned long mmap_region(struct file *file, unsigned long addr, unsigned long len, vm_flags_t vm_flags, unsigned long pgoff, diff --git a/mm/mmap.c b/mm/mmap.c index 2021bc040e81..2723f26f7c62 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1249,18 +1249,6 @@ unsigned long do_mmap(struct file *file, unsigned long addr, if (mm->map_count > sysctl_max_map_count) return -ENOMEM; - /* Obtain the address to map to. we verify (or select) it and ensure - * that it represents a valid section of the address space. - */ - addr = get_unmapped_area(file, addr, len, pgoff, flags); - if (IS_ERR_VALUE(addr)) - return addr; - - if (flags & MAP_FIXED_NOREPLACE) { - if (find_vma_intersection(mm, addr, addr + len)) - return -EEXIST; - } - if (prot == PROT_EXEC) { pkey = execute_only_pkey(mm); if (pkey < 0) @@ -1274,6 +1262,18 @@ unsigned long do_mmap(struct file *file, unsigned long addr, vm_flags |= calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) | mm->def_flags | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC; + /* Obtain the address to map to. we verify (or select) it and ensure + * that it represents a valid section of the address space. + */ + addr = __get_unmapped_area(file, addr, len, pgoff, flags, vm_flags); + if (IS_ERR_VALUE(addr)) + return addr; + + if (flags & MAP_FIXED_NOREPLACE) { + if (find_vma_intersection(mm, addr, addr + len)) + return -EEXIST; + } + if (flags & MAP_LOCKED) if (!can_do_mlock()) return -EPERM; @@ -1831,8 +1831,8 @@ unsigned long mm_get_unmapped_area_vmflags(struct mm_struct *mm, struct file *fi } unsigned long -get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, - unsigned long pgoff, unsigned long flags) +__get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags) { unsigned long (*get_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long) @@ -1865,8 +1865,8 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, if (get_area) addr = get_area(file, addr, len, pgoff, flags); else - addr = mm_get_unmapped_area(current->mm, file, addr, len, - pgoff, flags); + addr = mm_get_unmapped_area_vmflags(current->mm, file, addr, len, + pgoff, flags, vm_flags); if (IS_ERR_VALUE(addr)) return addr; @@ -1879,6 +1879,12 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, return error ? error : addr; } +unsigned long +get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags) +{ + return __get_unmapped_area(file, addr, len, pgoff, flags, 0); +} EXPORT_SYMBOL(get_unmapped_area); unsigned long From patchwork Thu Feb 15 23:13:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13559260 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90F36C48BC4 for ; Thu, 15 Feb 2024 23:14:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 34ADA6B0088; Thu, 15 Feb 2024 18:14:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D1996B008A; Thu, 15 Feb 2024 18:14:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FFC66B0092; Thu, 15 Feb 2024 18:14:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C34EF6B008A for ; Thu, 15 Feb 2024 18:14:44 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8D1FE120842 for ; Thu, 15 Feb 2024 23:14:44 +0000 (UTC) X-FDA: 81795594888.17.7290AB7 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.21]) by imf09.hostedemail.com (Postfix) with ESMTP id 9D3EA140015 for ; Thu, 15 Feb 2024 23:14:42 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NfpFrd8Q; spf=pass (imf09.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708038882; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y0eyua5VLqarJkFbZIkOnrGFAFnMrr4VgPsmAogZaFc=; b=2RxfUBVltD3+Dxo+ieNdY5c443FXReGzUT1B45WSeBpNHr9WfvPyoa4mqESBKC58gyZ9ah foLkXTex0Z6r4kpQXMcIhMXAjbeIL/AZy0BXH5k71K7N7Mrj7bou56V4nxQt3GofjbK9G1 rejco0yX4PRjP60qsWzdbiwXIr4oVMc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708038882; a=rsa-sha256; cv=none; b=HPpdf9n0q5WMrvnQGgui3/tfqKWMqe782IV/xBg7Q7Z1TxGbn5McIW7ggRQ5lwhBXmCqsh 4dLrdgJdJCjsqoXUFraINywWeDPghay7T6aV7X+HPRQXVfz2urZ+WTQ9UQnGXzEOQSQY+w 2HLDdTTRbMMytMO2k4zozWNPmJmGJgA= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NfpFrd8Q; spf=pass (imf09.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708038883; x=1739574883; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XHhdhuxOKWz8EspMQDfPASDrP/vRFk3wVhxiIKVJyZE=; b=NfpFrd8QKeI9hUSsEoGQdlTQLTlezIbtpPp8Y4K6b2oPY9jS6+Wvohsm 2vvbo9RRbf9w7K80AvrYN+7sm+rvOYqOKCPuyhyzXKQBAynW5cstBsKNF OSIlrxc06ExNllCwHwuym+USb6RXSx8RBsjlgdvIqvYsA31ULgejFJ0SY XsL5ww3hdqlK+i27Qh1s9X4vkADpL81xt7EPzKOFTDYAI8+WHpKt0hh54 TxEWKe4CAu/IwDzTl3BFj1l8ufffzWSOd9k61b89Qd1YTW8d/3+R44VkM Zsb2vKgtuE4L+eeoNW1ckkVHMhYLvekTND0Fq9hE5rFaFNPY9W+ALI0TH w==; X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="2066347" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="2066347" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orvoesa113.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="912250197" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="912250197" Received: from yshin-mobl1.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.209.95.133]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:39 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [RFC PATCH 4/8] thp: Add thp_get_unmapped_area_vmflags() Date: Thu, 15 Feb 2024 15:13:28 -0800 Message-Id: <20240215231332.1556787-5-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> References: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9D3EA140015 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 3ew9zr6kw8cz7zo7a3535cxo8pqq743n X-HE-Tag: 1708038882-276667 X-HE-Meta: U2FsdGVkX1/j/RH3h/XHa2KhnOpSJLGwanXeUTGPFcKDGm50oU0bCTc572ij4+dUXcKYia0uVTTcsP/g099HXJciJHJFSExK7VI/G6m0y7vfGPRwmy5567frb/rXeGgHtwvVjseLrsVDJ+RRNvz0rieJUFSE2cHmW9c0mCODc3R0OVQta7cE6xZsZML7LHg7S4itkn6JUQIKSA3lpDI6ytqexroIGHPb7INf2BX6YjdfZvAOaMFWKqtjc1c2Oc/VupRreSqqSYL1yK4vNcAhZkM/ZovKIJIwsNC96BnioEQ6FT7tZ3/9edvRm6yPGhKk/PcZ9peqAOlv/3hkiYMGBG57Ja78p7ilLbHGBWuoqCJSHh2/0XxRwhF7nB+qRIaU5VGWOvhIltVIXFRjhRGASvuoB/5ZW++l9iyuYhBKkLAf2Or7M4/m+TYBhCCzM/ouavbewj8KIRvMi09wXwp+z5i9XjdJMvO3U3j3QleNLX24NiMJ0WUrB/YPrWQK2bmtzb+QPanZCB86e5lzICelQuiAVbBFyYMNcittqztJ36r6YgpQUE4MU9LkvdREDA7g13JZu8vdeZZvCDnZqnUW5Ec366hHFJv/Ee/8NsawHgeaYWJW3NegJiMc4vj85PLuckAg0CIpOHHy1Bp9gUCYKLRrjDlvVxgEarhIvg/f2UW1+cvL3ztdFpkN/YdhSXjfFQ5Q+4OFSoT6UpxNHwh9XJuPId+Jfr4sEVXy69Yi3K5S4sVfMyn8f/OIfgLL1hkU7gwqprRMyrCCbt2VHrs5VPjGWPZ1Tls/iaspw6IqVkvfkvec6s4gtzUBRcIQSw5psSwzXueukaTQ5XCKGfrVJFt1Gs9HOvj3UpyFj7nZlvQKBr2XSg0zuYdeKrtzr73Jh09pS35xrrE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. Add a THP implementations of the vm_flags variant of get_unmapped_area(). Future changes will call this from mmap.c in the do_mmap() path to allow shadow stacks to be placed with consideration taken for the start guard gap. Shadow stack memory is always private and anonymous and so special guard gap logic is not needed in a lot of caseis, but it can be mapped by THP, so needs to be handled. Signed-off-by: Rick Edgecombe --- include/linux/huge_mm.h | 11 +++++++++++ mm/huge_memory.c | 23 ++++++++++++++++------- mm/mmap.c | 12 +++++++----- 3 files changed, 34 insertions(+), 12 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index fa0350b0812a..ef7251dfd9f9 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -139,6 +139,9 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags); void folio_prep_large_rmappable(struct folio *folio); bool can_split_folio(struct folio *folio, int *pextra_pins); @@ -286,6 +289,14 @@ static inline void folio_prep_large_rmappable(struct folio *folio) {} #define thp_get_unmapped_area NULL +static inline unsigned long +thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t vm_flags) +{ + return 0; +} + static inline bool can_split_folio(struct folio *folio, int *pextra_pins) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e9ef43a719a5..f235f6d3ff62 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -628,7 +628,8 @@ static inline bool is_transparent_hugepage(struct folio *folio) static unsigned long __thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, - loff_t off, unsigned long flags, unsigned long size) + loff_t off, unsigned long flags, unsigned long size, + vm_flags_t vm_flags) { loff_t off_end = off + len; loff_t off_align = round_up(off, size); @@ -641,8 +642,8 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, if (len_pad < len || (off + len_pad) < off) return 0; - ret = mm_get_unmapped_area(current->mm, filp, addr, len_pad, - off >> PAGE_SHIFT, flags); + ret = mm_get_unmapped_area_vmflags(current->mm, filp, addr, len_pad, + off >> PAGE_SHIFT, flags, vm_flags); /* * The failure might be due to length padding. The caller will retry @@ -662,17 +663,25 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, return ret; } -unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, - unsigned long len, unsigned long pgoff, unsigned long flags) +unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags) { unsigned long ret; loff_t off = (loff_t)pgoff << PAGE_SHIFT; - ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE); + ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE, vm_flags); if (ret) return ret; - return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags); + return mm_get_unmapped_area_vmflags(current->mm, filp, addr, len, pgoff, flags, + vm_flags); +} + +unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags) +{ + return thp_get_unmapped_area_vmflags(filp, addr, len, pgoff, flags, 0); } EXPORT_SYMBOL_GPL(thp_get_unmapped_area); diff --git a/mm/mmap.c b/mm/mmap.c index 2723f26f7c62..936d728ba1ca 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1857,16 +1857,18 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, */ pgoff = 0; get_area = shmem_get_unmapped_area; - } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { - /* Ensures that larger anonymous mappings are THP aligned. */ - get_area = thp_get_unmapped_area; } - if (get_area) + if (get_area) { addr = get_area(file, addr, len, pgoff, flags); - else + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + /* Ensures that larger anonymous mappings are THP aligned. */ + addr = thp_get_unmapped_area_vmflags(file, addr, len, + pgoff, flags, vm_flags); + } else { addr = mm_get_unmapped_area_vmflags(current->mm, file, addr, len, pgoff, flags, vm_flags); + } if (IS_ERR_VALUE(addr)) return addr; From patchwork Thu Feb 15 23:13:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13559263 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62D0CC48BC4 for ; Thu, 15 Feb 2024 23:15:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B6DE6B0092; Thu, 15 Feb 2024 18:14:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 806946B0098; Thu, 15 Feb 2024 18:14:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14F0C6B0092; Thu, 15 Feb 2024 18:14:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B8EB76B008A for ; Thu, 15 Feb 2024 18:14:46 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 71844A27DE for ; Thu, 15 Feb 2024 23:14:46 +0000 (UTC) X-FDA: 81795594972.24.659D703 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.21]) by imf16.hostedemail.com (Postfix) with ESMTP id 69DAE180003 for ; Thu, 15 Feb 2024 23:14:44 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Erd4U6+U; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf16.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708038884; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Kh/0VZ1saSSmU90+yAC133lTBVMc2tEcq5B/zah2PU4=; b=29qNEvKKvpnRymCV9lRAhUNVDjmFJoIMVvEJtYuiZYQFAiSx+lMoAiyDsjGAc7oImeXL2T rA3wtEjpGLsdudOhOUbJY2Nf32ySQxd+UwwV2pVQTGwO/BX1wwV4AVhfC4opvUxUDhkA/V vyn7EIxVjZhzCJo8KmK8yUehTuu4nLo= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Erd4U6+U; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf16.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708038884; a=rsa-sha256; cv=none; b=SvFMilzDefhZm5m7d3q8/WTdX+b4DHXNETwpBKgVc2JZWCWMvpeXdid4v+2vpbdwpkNQ4L VrcBrXqaBiFtkjBiKKbBr+QIuK/mamJyaPn1QPbYE7SHpx9tEqaKuzFUMBFTwGyjhjFo8E 5799HNi+uG+vHYz2xJSXpBid3Sry2fo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708038885; x=1739574885; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Ib9DRZGDBCSUD4uXPmR+ijcvzy4h1J3AkaOgyP6jQrA=; b=Erd4U6+Uxpr4yI/S9dhEmoBJsTpQRrdoO6ADCgYVx+ZxM+4T94/KBqFX /xPk4qBOwMwBtabRMChHjFus3EdBXSzE60pF09IkBOh5iuGBKuj98pom2 HpO9fZF4q+ZjIgrgqFI0S5TxGLaCAiwEiS3DikuP9BRyD+Oqbu91YVUIe UwhnRJ+22bGnZFZ6VDoutuM40SPePcSGNiNOiU6nWOFvqCQWPa8gGJ0QM VMPUmxZ1GZvrZuT9nXMJj61wniAHmMt+1IObFWfQ73GJSD9CkJQ/mUPNf XCJ1dvlzTssMsdc0WyVRApwQ/bIQyWeJ4cTyEvBgMLqHowcbli/TjM0lO Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="2066360" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="2066360" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orvoesa113.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="912250200" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="912250200" Received: from yshin-mobl1.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.209.95.133]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:40 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [RFC PATCH 5/8] mm: Take placement mappings gap into account Date: Thu, 15 Feb 2024 15:13:29 -0800 Message-Id: <20240215231332.1556787-6-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> References: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 69DAE180003 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ip5t95zikr7xgndydyoqmwnpof5h8qmd X-HE-Tag: 1708038884-346097 X-HE-Meta: U2FsdGVkX19XP2hvwqbj7h7k9aeoDErCIxfw5zgwEBpDE9RwrGlN7EG5CG9WRclrWZmm3WWcbziyCIVli/xJNtt9kVpRSMsU+c9CpVGSu8Rpau86arTamv7RbQ7PMvX2uyGO/Z+L3+aiL+8wpLICak8xwdVtn+T8NGJd8J/Iyg1GXGSbZ8jAHaCBnhvDN/SugCRyvK2lXjaz3/LXKScaJVheiGpVyBrUqJj1lv8D+3pqTIjk+YTo/6SSTvqrFAKobhEkOBPt4N8VHkLGrnhZkUgPZrUpEJNw0umi0Qy0VvsmTPvI90RaAkqTvdAUWxwSETBReoRawbBNxBeP/NHl/GHqRGLdMCdamPGdjFm88mKiQzWlK4n7dq7g98WWT9UWFI0Lsa6f2WCL54OTxDwvJwToTwvofY0hND6dHf+QzDUJGaatoSCw2YS/XLGgiud75VjXF9gd/c194ONNTAtCS73QEocWUwTS+SGspBkPIdRMKUraSGwwoepGWBCgQpiZurZE/xuobGG3df+pS+kDs8ni8CW45sF3hKkd0zBFXRKDovSlxqbyesjTAfMRtZ6sQrbsBMAE16voQtGsJ8g23OTIk9pbpS75LiTPdqUKZXyDrDZ63WG9Qf0ZPf3OI903YNzC2ub1XCNhsXJ6MiaFpCVSSSCGJur3RlGcfwzybAtHH8eRfQLiGcHrhfwmAbmfLh1BraTbDw0StsAQXWUfmLCndaM5aB9TJ+fUJCSvMNSgGgN3i4sBBIZCl7HOF55RWTd9ZXqcC1u8zs7fJ8/sUrtk1ZFcbC7tPCEFej7ixOdgBbCjf6MhcxWXv3wbhMqUwPDyg+l1XnuBeacrzev36IzpajorM75GEStvOP/A+eYIrfrhEd1RGq4RtM+6sPAXjnA1ns/iJXzbujhdEZZLj/faYfGYjYUu7ZJ1e8fT7Ggtj1mFV/QQ6Y8+XBq8stBrQcYbNiCvpzLn9wxqvG7 OGE79g7w Elz+5YXLnO0dX7Dl9LEWOPB5DGwnZIOMN+2dyG1oNCHC2bODYsM/WuVl5MfnCIFR8zzaSLhmiT1fL/t2W6R/8O/iEtszongcRdet7+2eaZJxcevM0iZNj4VV2FcdJAyjDQsdFFyRkC7FyemVYs1AcNeODJH1v3ryJvcuyot0TaEoQnmMFd3+mgq5EbDfE4Xxh2RXl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. For MAP_GROWSDOWN/VM_GROWSDOWN and MAP_GROWSUP/VM_GROWSUP this has not been a problem in practice because applications place these kinds of mappings very early, when there is not many mappings to find a space between. But for shadow stacks, they may be placed throughout the lifetime of the application. So define a VM_UNMAPPED_START_GAP_SET flag to specify that a start_gap field has been set, as most vm_unmapped_area_info structs are not zeroed, so the added field will often contain garbage. Use VM_UNMAPPED_START_GAP_SET in unmapped_area/_topdown() to find a space that includes the guard gap for the new mapping. Take care to not interfere with the alignment. Signed-off-by: Rick Edgecombe --- include/linux/mm.h | 2 ++ mm/mmap.c | 21 ++++++++++++++------- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 9addf16dbf18..160bb6db7a16 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3393,12 +3393,14 @@ extern unsigned long __must_check vm_mmap(struct file *, unsigned long, struct vm_unmapped_area_info { #define VM_UNMAPPED_AREA_TOPDOWN 1 +#define VM_UNMAPPED_START_GAP_SET 2 unsigned long flags; unsigned long length; unsigned long low_limit; unsigned long high_limit; unsigned long align_mask; unsigned long align_offset; + unsigned long start_gap; }; extern unsigned long vm_unmapped_area(struct vm_unmapped_area_info *info); diff --git a/mm/mmap.c b/mm/mmap.c index 936d728ba1ca..1b6c333656f9 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1567,14 +1567,17 @@ static inline int accountable_mapping(struct file *file, vm_flags_t vm_flags) */ static unsigned long unmapped_area(struct vm_unmapped_area_info *info) { - unsigned long length, gap; + unsigned long length, gap, start_gap = 0; unsigned long low_limit, high_limit; struct vm_area_struct *tmp; MA_STATE(mas, ¤t->mm->mm_mt, 0, 0); + if (info->flags & VM_UNMAPPED_START_GAP_SET) + start_gap = info->start_gap; + /* Adjust search length to account for worst case alignment overhead */ - length = info->length + info->align_mask; + length = info->length + info->align_mask + start_gap; if (length < info->length) return -ENOMEM; @@ -1586,7 +1589,7 @@ static unsigned long unmapped_area(struct vm_unmapped_area_info *info) if (mas_empty_area(&mas, low_limit, high_limit - 1, length)) return -ENOMEM; - gap = mas.index; + gap = mas.index + start_gap; gap += (info->align_offset - gap) & info->align_mask; tmp = mas_next(&mas, ULONG_MAX); if (tmp && (tmp->vm_flags & VM_STARTGAP_FLAGS)) { /* Avoid prev check if possible */ @@ -1619,13 +1622,17 @@ static unsigned long unmapped_area(struct vm_unmapped_area_info *info) */ static unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info) { - unsigned long length, gap, gap_end; + unsigned long length, gap, gap_end, start_gap = 0; unsigned long low_limit, high_limit; struct vm_area_struct *tmp; MA_STATE(mas, ¤t->mm->mm_mt, 0, 0); + + if (info->flags & VM_UNMAPPED_START_GAP_SET) + start_gap = info->start_gap; + /* Adjust search length to account for worst case alignment overhead */ - length = info->length + info->align_mask; + length = info->length + info->align_mask + start_gap; if (length < info->length) return -ENOMEM; @@ -1832,7 +1839,7 @@ unsigned long mm_get_unmapped_area_vmflags(struct mm_struct *mm, struct file *fi unsigned long __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, - unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags) + unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags) { unsigned long (*get_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long) @@ -1883,7 +1890,7 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, - unsigned long pgoff, unsigned long flags) + unsigned long pgoff, unsigned long flags) { return __get_unmapped_area(file, addr, len, pgoff, flags, 0); } From patchwork Thu Feb 15 23:13:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13559261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DD3CC4829E for ; Thu, 15 Feb 2024 23:14:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 286D36B008C; Thu, 15 Feb 2024 18:14:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 210C96B008A; Thu, 15 Feb 2024 18:14:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6A566B0098; Thu, 15 Feb 2024 18:14:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C3EDC6B0092 for ; Thu, 15 Feb 2024 18:14:46 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A0CD28097F for ; Thu, 15 Feb 2024 23:14:46 +0000 (UTC) X-FDA: 81795594972.26.A27C81D Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.21]) by imf06.hostedemail.com (Postfix) with ESMTP id 7DE82180009 for ; Thu, 15 Feb 2024 23:14:44 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=MwAOkfEU; spf=pass (imf06.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708038884; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eepGtBS7D3bO89xNOPfpfUsP1AAKKSi19IceRmMHNEc=; b=mZREkw2HPtngiRVLJrwxc91L/hu/cAblQYEnMu/oiOOnPLl6oCtFfkzx/H2PrK9UizVB+C yX82J7BeDFcFIebPCeXnl3kvpINJTbZThYpxk45UpYGnnvolry2AO+xJXcxx6bXCQbPWTw nPeBZEsH9zSSBrONIgx2ZECoRlz0hhk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708038884; a=rsa-sha256; cv=none; b=fVevAyCcNJqJMGfCJEpINQV014eO9W7GugyrrajcJEl1lr2F9Sw8JQngyKRqYeSKFq/bWo LohJ/QalQajWbvcaBg2db612tk9U9YDT5zA7E+zIqfQYwXMnkzi3EYc5f60lnwqFiU61p2 v5OIFp7YaQA6iowbit2SM/1rH5PPYiw= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=MwAOkfEU; spf=pass (imf06.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708038885; x=1739574885; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=VO9DQY6lGtiG1eC5gi6w3K4FE6tMMi6+Q5ti33XNprc=; b=MwAOkfEU3MSNIrYf3wDj1WECyeOxkbFR4I8LE2qXwp74w1g1yqToULZ6 whMtMVN8Us/CTtDJcjJTmIz1uspWZvJ0Gsz1HbahbYT9I5OVVe0x8feer cpCDaLz1Zj8P6AT6+BSbkrfm+h6Buiw2tzL6ydUpkq0etjcsvMAPW1tbb W4zoURgAZWKZfx1jXdnNqnMGuidwCeWAIHountMXVUW+7fQGRyK74y6O7 /E9n2PhlZhNjv/7SPI6eagYz7G90iSLWmjrErJpDstl6dVgCTZ/xQNGeG X8rgdL8sa8n5eI9VGWJvw6AMqsT6XN6sGBxVaaHNuWhjsS+/M7rBxtwMw w==; X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="2066375" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="2066375" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orvoesa113.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:42 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="912250203" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="912250203" Received: from yshin-mobl1.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.209.95.133]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:41 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [RFC PATCH 6/8] x86/mm: Implement HAVE_ARCH_UNMAPPED_AREA_VMFLAGS Date: Thu, 15 Feb 2024 15:13:30 -0800 Message-Id: <20240215231332.1556787-7-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> References: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7DE82180009 X-Rspam-User: X-Stat-Signature: 9wjrc3tk4egmadqk7x1a85ynmz1kpwk4 X-Rspamd-Server: rspam03 X-HE-Tag: 1708038884-518179 X-HE-Meta: U2FsdGVkX1/gfQuLatR44mi/Fk5yq8AUk4YbN10cKLHA9uLNe80QSjdo4jsF2I1TcbhvEJBLwp5N5OVQ03aRUMg1XyZGP3pozdBwXGdKq7m7HXk1VzXTyZbVZqIvsrLo3h/RwozmAau4ueeUkoJeNHt6uvNYA8haeUpX5r85Or5j9CvyU2CBfC731b4XX5qEGZI3ca4EFuCaIOwC6X7Wpinp+1Pl8BWJeaECS7dRS3NpoOPhjZPMt+nnb2ydFyJt2+SdYzmyDRJSICVnTsq9Ma+JorJ/W5nEr3ZLHVXdPPWYNaHczJh4HBHqjcsTE4htWOgdj+FaYhVS0MfgfkHjIwWbtdDkb5zxrfrWQlXo2Hsbom3115kdpf9yswxgglaxBqSaAjH6BtE/jk74dfJShH9AKgM8wU3k9LtbL7G1RQaaFlTf5B/bLYBaMp2oCCJqLSo+GCE0EtwQ3Dp6OKVMDyC4elnP7l2oeHGQy2JfUULo9vfON0lxnDHV56i1RZ4nWJXtJNRIbRfF1+aZdrfqs3y+7Fsn//I4Lerr2yPbKQQFvTcL1nqSpJL3VrS6gIJ1V/oIRXsegtn2wxkDEh8OF7ozKG8bveGoEdwuuXk8uicje0VJ+V1Pst4xtVDyn+RJJesq2seFSEi0MjTjL37GB07kupR8c5E/IDWD9NTjCc7Zln0dyZ8M6Lox+F8r/xSqjLxAFRYYQOYRAK69+WOJiwbtLuviuoqQvBS8JU+yhTw8ebSC7OAb0baVfpkMe1ifLrv0h18j61JUW4e6/3EWyjA6/8ha2VKthZk5ZXgi8/T7Uw0Bi+i+RvLOFoBX6CQTX7vWbSlUFT7iQ/1fADyjjrDwzMVJBvBKflwwV+qzMvxowPc+b4XhA+X5i+gc3UtPFDn1jXChmfzNgPdrdcOT86rbyQXh2oebrYK1fhOrQMhHOZ6ikt4u07Upsu7D4ynbyInMqQtiWaJr1xw7cv9 CMpCzCU8 9zd/jt2tWCXT+FXcc3o54S7eGhuZQxt08Ldjx7i9SwAN2mczCflaSz8G3IGn6EnJOp3N80/MH3RiHZkm7DwD5ZLsaB4EYUkS9t7eiii23n+e5in2lP0cjfCg8E8tNsHtFlAKUQk/CNzwliaY0+zT+t84bUNMAQ5Cgg/1RwG0xw1GtE+ZreMzSIgLbT8WZYBbdJQ1ILawDZtQ0x5H4+sAIZn9nVA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. Add x86 arch implementations of arch_get_unmapped_area_vmflags/_topdown() so future changes can allow the the guard gap of type of vma being placed to be taken into account. This will be used for shadow stack memory. Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/pgtable_64.h | 1 + arch/x86/kernel/sys_x86_64.c | 29 ++++++++++++++++++++++------- 2 files changed, 23 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtable_64.h index a629b1b9f65a..eb09a11621ad 100644 --- a/arch/x86/include/asm/pgtable_64.h +++ b/arch/x86/include/asm/pgtable_64.h @@ -244,6 +244,7 @@ extern void cleanup_highmap(void); #define HAVE_ARCH_UNMAPPED_AREA #define HAVE_ARCH_UNMAPPED_AREA_TOPDOWN +#define HAVE_ARCH_UNMAPPED_AREA_VMFLAGS #define PAGE_AGP PAGE_KERNEL_NOCACHE #define HAVE_PAGE_AGP 1 diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c index c783aeb37dce..f92780cf9662 100644 --- a/arch/x86/kernel/sys_x86_64.c +++ b/arch/x86/kernel/sys_x86_64.c @@ -119,9 +119,9 @@ static void find_start_end(unsigned long addr, unsigned long flags, *end = task_size_64bit(addr > DEFAULT_MAP_WINDOW); } -unsigned long -arch_get_unmapped_area(struct file *filp, unsigned long addr, - unsigned long len, unsigned long pgoff, unsigned long flags) +extern unsigned long +arch_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags) { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; @@ -157,10 +157,10 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, return vm_unmapped_area(&info); } -unsigned long -arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, - const unsigned long len, const unsigned long pgoff, - const unsigned long flags) +extern unsigned long +arch_get_unmapped_area_topdown_vmflags(struct file *filp, unsigned long addr0, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t vm_flags) { struct vm_area_struct *vma; struct mm_struct *mm = current->mm; @@ -230,3 +230,18 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, */ return arch_get_unmapped_area(filp, addr0, len, pgoff, flags); } + +unsigned long +arch_get_unmapped_area(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags) +{ + return arch_get_unmapped_area_vmflags(filp, addr, len, pgoff, flags, 0); +} + +unsigned long +arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr, + const unsigned long len, const unsigned long pgoff, + const unsigned long flags) +{ + return arch_get_unmapped_area_topdown_vmflags(filp, addr, len, pgoff, flags, 0); +} From patchwork Thu Feb 15 23:13:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13559262 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 665FFC4829E for ; Thu, 15 Feb 2024 23:14:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 519EA6B008A; Thu, 15 Feb 2024 18:14:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 461156B00A2; Thu, 15 Feb 2024 18:14:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 015FE6B0093; Thu, 15 Feb 2024 18:14:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B957B6B008C for ; Thu, 15 Feb 2024 18:14:46 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7020280899 for ; Thu, 15 Feb 2024 23:14:46 +0000 (UTC) X-FDA: 81795594972.26.52BA8DF Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.21]) by imf09.hostedemail.com (Postfix) with ESMTP id 926CF140010 for ; Thu, 15 Feb 2024 23:14:44 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PVh08EVe; spf=pass (imf09.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708038884; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZVnx71WeH6HH6EqY5Lraa48e1cboxzcmNmmFJj9hDh4=; b=kpYQIx2i66Ofxa2wYLQ9WRwAnyV+5xlHD9GwAMldD6pMeiYgiifnuM/pOgF/B7T/Q7tofB 5PpDwbrkHo3dD9bmzqgaYK2gJ2FgFkjhRegQe/dYHRbREL/LOpnJbS+GLXmS3MigNH319c 8IvFRjVuh4CjZrKzkNQTrIQNs6yszME= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708038884; a=rsa-sha256; cv=none; b=345685U1hXUAX+VPVi3+KqgsnHCHclwx7uVhStbfh1n5nXgElNkvVGzLdp06ksK0vMZXWB qtTsLGfjJd9xL7+KMr+sDO7TiiabweeUlcw5AMPOkRK6eRbqp7Nqf4TDjVm1a7igoszc0s sKKS58zwwYrDzrQLtme3rUF6mrpcLJ4= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PVh08EVe; spf=pass (imf09.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708038885; x=1739574885; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fQLzGTXW5Y/EBT0eWnQYJ+GT+r/+mVoQa4wR7CvVXl4=; b=PVh08EVezN8JnrhzLkbq0lM+KeuoNBnEybXTIw8rqwcf5mwgy9AVdhxE vyRjbcM8wg1LqUif4GKWl6o3aQyK25+ji/te3cLD/K8kbvVe+eA4l25yf g4gVQQZeHdXVY5FINTevT6rbZAXB/Gi7sIPhoVQZGJUnVG/xKRpTsGDGq +GP5+HtTCHH0mzmEped4jGp3lcpD2Ye+Icd2mbsf+A45ed+UzZ9HbOhrY CxEzJFfOfzM3bVzrcYVUkGHGxcQDJ0b3Nzrpgh1YQpFWklZdOuKfH7156 7MHa+aHKiioT6vPPN+n+TQaKy9m57yl/FQqbPxD0rX149sCtWFpXiy34u w==; X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="2066389" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="2066389" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orvoesa113.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:43 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="912250207" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="912250207" Received: from yshin-mobl1.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.209.95.133]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:41 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [RFC PATCH 7/8] x86/mm: Care about shadow stack guard gap during placement Date: Thu, 15 Feb 2024 15:13:31 -0800 Message-Id: <20240215231332.1556787-8-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> References: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 926CF140010 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: qmmnhungxcmpdny61bw3u5y47wik67he X-HE-Tag: 1708038884-986920 X-HE-Meta: U2FsdGVkX18t6KulNokAkc3CmcT8mEDO10U0wn15gxTkI/bt+V/fqzEnL0tj3unYTVgGM2CNqTYn3aOwHZkl0kvvU+B6MfHRilyZkbKaou6RK3ryc0mmwaJOnbFeYnaQW0fVibgRJBwtvb3YvzlZ591vgD1yAxDvl+575/H+WuVPGCUjR4xwz9tknqmXG84vNMkbKeH1R/bqPmYSkUqosrFQ2TPlwZh6wl8DnlI3b/Y8Xm0eZVA/zvdXfXXjyjSNoZqdJc2zPPnROmZmhbG8JTo80VODhbKVzlpHb63fe12B0Cas4r/mOc+TEVn9yg+qDlBOkocyOA0tcHljlhvXzMYg20aWzymsOMp0cYnrgWIm4+1jVIRJt0PjQwwB1Cl09Jq1qvafr/10TaH0VfxN3ZSUPQUF9Em5pnoomyh9kS8ipLQJc91CbzC/RhQpEDXEHf5OuCvUvo8WGg8fgIhAmTfkLr3I0IbQ4TP1UUQ6Qie4ZzLNfnRjV4kMgwMA8Jz8buKTlEogmGkorj6fck8G3fknC2yS+G053pJqYnhcwSHSA4xR4MwDPMBz+oeAHh77Cq2BzdPNG0GJ0is8k91v/613kk2NKsWqOy52Eu66CwUibwliNa4g6UAjWjtdpoxSbr9D16vq1ArY7Q5N8/l2w5i7r4luWvXR9RTvC4wcGgJP2OtJobyjc73o/D4WoChzxQkdSTxIOS2whF1TVKtSCLBBXf3s9rWKPUF/0vsDP/WzSqb8x7MHfEzzsSPNGSNaY18Bbyb/i7LcAfVdn6UMi/tqABsd14a5viSOcDM+e5ysSB1wsyMJOJxNT3w4lnoZ2FAUzEr25kfljBajFm9PmBZrugnrB4Tm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. Now that the vm_flags is passed into the arch get_unmapped_area()'s, and vm_unmapped_area() is ready to consider it, have VM_SHADOW_STACK's get guard gap consideration for scenario 2. Signed-off-by: Rick Edgecombe --- arch/x86/kernel/sys_x86_64.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c index f92780cf9662..3b78fdc235fc 100644 --- a/arch/x86/kernel/sys_x86_64.c +++ b/arch/x86/kernel/sys_x86_64.c @@ -119,6 +119,14 @@ static void find_start_end(unsigned long addr, unsigned long flags, *end = task_size_64bit(addr > DEFAULT_MAP_WINDOW); } +static inline unsigned long stack_guard_placement(vm_flags_t vm_flags) +{ + if (vm_flags & VM_SHADOW_STACK) + return PAGE_SIZE; + + return 0; +} + extern unsigned long arch_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags) @@ -144,12 +152,13 @@ arch_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, unsigned l return addr; } - info.flags = 0; + info.flags = VM_UNMAPPED_START_GAP_SET; info.length = len; info.low_limit = begin; info.high_limit = end; info.align_mask = 0; info.align_offset = pgoff << PAGE_SHIFT; + info.start_gap = stack_guard_placement(vm_flags); if (filp) { info.align_mask = get_align_mask(); info.align_offset += get_align_bits(); @@ -191,7 +200,7 @@ arch_get_unmapped_area_topdown_vmflags(struct file *filp, unsigned long addr0, } get_unmapped_area: - info.flags = VM_UNMAPPED_AREA_TOPDOWN; + info.flags = VM_UNMAPPED_AREA_TOPDOWN | VM_UNMAPPED_START_GAP_SET; info.length = len; if (!in_32bit_syscall() && (flags & MAP_ABOVE4G)) info.low_limit = SZ_4G; @@ -199,6 +208,7 @@ arch_get_unmapped_area_topdown_vmflags(struct file *filp, unsigned long addr0, info.low_limit = PAGE_SIZE; info.high_limit = get_mmap_base(0); + info.start_gap = stack_guard_placement(vm_flags); /* * If hint address is above DEFAULT_MAP_WINDOW, look for unmapped area From patchwork Thu Feb 15 23:13:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13559264 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 654A1C4829E for ; Thu, 15 Feb 2024 23:15:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA99B6B0098; Thu, 15 Feb 2024 18:14:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D34B08D0007; Thu, 15 Feb 2024 18:14:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5F406B00A5; Thu, 15 Feb 2024 18:14:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9C8236B0098 for ; Thu, 15 Feb 2024 18:14:48 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 71682C0988 for ; Thu, 15 Feb 2024 23:14:48 +0000 (UTC) X-FDA: 81795595056.09.2AE0202 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.21]) by imf09.hostedemail.com (Postfix) with ESMTP id 91768140010 for ; Thu, 15 Feb 2024 23:14:46 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gFlSpREV; spf=pass (imf09.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708038886; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NFCSdAhrunIzMrgwpPIPHm0AfcuiyWMeHA0mu0m/29Q=; b=Sas7LsOK5vfAM42uJH7ES/ogqjLnDoj9fqk7CbFBqKLtSr0PdFJHq1bPQv7mGGa1cWGO9M peh8coQSu/rRY4P9jVtMfwb2ATmtOr+TkGv1i3Bbu9JccIrOatkTCghbH6k/+l+hnTbPkP hj24BX+PWfXLqQJIlWcIkXFg4C7IMJ0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708038886; a=rsa-sha256; cv=none; b=HuPZJ3u4ixlAzdnxv2684SGuTRMMVKjcUKgF56qg/K/AUhfY9TLscPlIbHvze3lagl1YGw 6LCe7gJykNu3pwKNbWhVIqdLEns5qhjbXYc4Za5ehW1fS7v5YMBdAt0lEjKHOjbARgIXvw IhlOm8cIjz+ThIZ/vsT1tkvud/4P8JQ= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gFlSpREV; spf=pass (imf09.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708038887; x=1739574887; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9Kz1B1g9YBWVQVBA7u5n8sBfRhBddgPHEWEy/INdrdY=; b=gFlSpREV4V4OV3GtTaTHHWJcwNPA9FKK7DuZDRFjZxpACQCrip7LB6fD sST/cAnckqbqE1vZerAO/fOXuQJ3L7mIWpJ62tSScdr878JH53TzwaAre byvcCoWn7OSojAtjQiwvIc/92l1VeUw0kZIIE6bnm1L+2DwfuAw9LRbQS 3+jYwx/938NpuVB1RFl6fUdU88McJTEvA6QZwJOrB0VIY3uenjSk6ZOzf EiPmrk8+QRy6iye2Wz8DXCJBjDG2bfX/recc2kH5PpDIEigpOzOVbS//k I2KHnEgTXTq4T9JnIcFQEvQHqtXyBJ1aUmNAiiheYrlWOlahD98783EKx w==; X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="2066403" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="2066403" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orvoesa113.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:43 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="912250210" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="912250210" Received: from yshin-mobl1.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.209.95.133]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:42 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [RFC PATCH 8/8] selftests/x86: Add placement guard gap test for shstk Date: Thu, 15 Feb 2024 15:13:32 -0800 Message-Id: <20240215231332.1556787-9-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> References: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 91768140010 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: war7wr65a8qdm14zkc5cu3cf789b9jt4 X-HE-Tag: 1708038886-473267 X-HE-Meta: U2FsdGVkX19ZqzluOdQrsl2RQPgJBDBjL8Pgecj7MgR5eiuW7ayGh8oVoupDprVfXCXxRswc6CKB4VpiNLiGq1k1s2nfelGjlagEIjqRV1YNpcJ5rvssW8pN6CYLtbUorJrWSR5EpK3fJGx8e2lrzaIHquPeAkSaYSHZtrjcEElXs3j7C7AdI5rG4g1fDZVFGOEVUHgaAOJCJRFikrCawRv1dvU94xwQCF8gB9L2bvK3qCXJJpC8EQlK9qI3mIM0TLQRjMURP2/4vyWDrYFU7wBfhT0SxIs7daW+toeeI3YJx50NVah0Gk7Cx2y5GvkpGZMbWxAEMTBWPPs6kRJZgI0XjE9msHq3lup3D1Gyug5gXDXUiLrCj0OROFXC7Mb9n/pRSATkxLdpgSEV+T4muggO43tuhPdgVH7lBAgU0mxdmEo4TR4Hv77ufEl/3dMhCus8FNzeYgdYWBYmC1Ej/HTt6SlmOyNexi2dNeUPufO/p8U/JVi6Gtc8XCn6SaCo53M5bYH7eW1YUO+w5er9/MGJKe45GwjhqzbmqIdcm6viaAxTPeZjopqdXbOJ1k6ZGHREjcGM3ANZluZa98S/9/cQoIZE2YNyne5uk47F69LHf6a+Rfu5T4gHbIZIj5Zu4nd0i/e8y7GNUoxlHBVY9I7kW+PH7AwUVN50Pq1fcE3+AmhCQIXzaGrzhwu0aeLuikQwVKqRl91N74bl3aGlCNFqO+1AlpGkm3oUvJE18fjfqRRpk0+sgBQzmyCFfLWP9vckwTS/7EdG6m44EwEqARR23si+RxAR2iGmKAPtVOCOAdVcKct3zn7kLNUJEoMpySjpbXiGytkc4/NKhkvMN3I87gL+aw7oO8kaWKwuEQrjgIGrIGP17myRjdP3Kc9x X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The existing shadow stack test for guard gaps just checks that new mappings are not placed in an existing mapping's guard gap. Add one that checks that new mappings are not placed such that preexisting mappings are in the new mappings guard gap. Signed-off-by: Rick Edgecombe --- .../testing/selftests/x86/test_shadow_stack.c | 67 +++++++++++++++++-- 1 file changed, 63 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/test_shadow_stack.c b/tools/testing/selftests/x86/test_shadow_stack.c index 757e6527f67e..ee909a7927f9 100644 --- a/tools/testing/selftests/x86/test_shadow_stack.c +++ b/tools/testing/selftests/x86/test_shadow_stack.c @@ -556,7 +556,7 @@ struct node { * looked at the shadow stack gaps. * 5. See if it landed in the gap. */ -int test_guard_gap(void) +int test_guard_gap_other_gaps(void) { void *free_area, *shstk, *test_map = (void *)0xFFFFFFFFFFFFFFFF; struct node *head = NULL, *cur; @@ -593,11 +593,64 @@ int test_guard_gap(void) if (shstk - test_map - PAGE_SIZE != PAGE_SIZE) return 1; - printf("[OK]\tGuard gap test\n"); + printf("[OK]\tGuard gap test, other mapping's gaps\n"); return 0; } +/* Tests respecting the guard gap of the mapping getting placed */ +int test_guard_gap_new_mappings_gaps(void) +{ + void *free_area, *shstk_start, *test_map = (void *)0xFFFFFFFFFFFFFFFF; + struct node *head = NULL, *cur; + int ret = 0; + + free_area = mmap(0, PAGE_SIZE * 4, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + munmap(free_area, PAGE_SIZE * 4); + + /* Test letting map_shadow_stack find a free space */ + shstk_start = mmap(free_area, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (shstk_start == MAP_FAILED || shstk_start != free_area) + return 1; + + while (test_map > shstk_start) { + test_map = (void *)syscall(__NR_map_shadow_stack, 0, PAGE_SIZE, 0); + if (test_map == MAP_FAILED) { + printf("[INFO]\tmap_shadow_stack MAP_FAILED\n"); + ret = 1; + break; + } + + cur = malloc(sizeof(*cur)); + cur->mapping = test_map; + + cur->next = head; + head = cur; + + if (test_map == free_area + PAGE_SIZE) { + printf("[INFO]\tNew mapping has other mapping in guard gap!\n"); + ret = 1; + break; + } + } + + while (head) { + cur = head; + head = cur->next; + munmap(cur->mapping, PAGE_SIZE); + free(cur); + } + + munmap(shstk_start, PAGE_SIZE); + + if (!ret) + printf("[OK]\tGuard gap test, placement mapping's gaps\n"); + + return ret; +} + /* * Too complicated to pull it out of the 32 bit header, but also get the * 64 bit one needed above. Just define a copy here. @@ -850,9 +903,15 @@ int main(int argc, char *argv[]) goto out; } - if (test_guard_gap()) { + if (test_guard_gap_other_gaps()) { ret = 1; - printf("[FAIL]\tGuard gap test\n"); + printf("[FAIL]\tGuard gap test, other mappings' gaps\n"); + goto out; + } + + if (test_guard_gap_new_mappings_gaps()) { + ret = 1; + printf("[FAIL]\tGuard gap test, placement mapping's gaps\n"); goto out; }