From patchwork Tue Apr 2 20:15:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxwell Bland X-Patchwork-Id: 13632237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2DB2C4345F for ; Tue, 16 Apr 2024 17:25:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1947A6B0085; Tue, 16 Apr 2024 13:25:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1453F6B0088; Tue, 16 Apr 2024 13:25:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00BF56B0089; Tue, 16 Apr 2024 13:25:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D070B6B0085 for ; Tue, 16 Apr 2024 13:25:08 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6A8A91A0A9B for ; Tue, 16 Apr 2024 17:25:08 +0000 (UTC) X-FDA: 82016070696.04.F302806 Received: from mx0a-00823401.pphosted.com (mx0a-00823401.pphosted.com [148.163.148.104]) by imf16.hostedemail.com (Postfix) with ESMTP id 8237E18000C for ; Tue, 16 Apr 2024 17:25:06 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=motorola.com header.s=DKIM202306 header.b="U XNGIZC"; spf=pass (imf16.hostedemail.com: domain of mbland@motorola.com designates 148.163.148.104 as permitted sender) smtp.mailfrom=mbland@motorola.com; dmarc=pass (policy=none) header.from=motorola.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713288306; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=Ktj2knpZkxEzepUnXbFCAHnzLfkAa26Odp8kjW5qVXw=; b=U1FV9jeSqYERZYPNU2f7PJPQWTnozeB1ifl8SAtnIQifFtNdj+bmvBf/6cSPvFknWTRksk uRHh+e3cskJmwkig4/pF3Wm8YQptunyjCUBTZ1CCuQ9WR7lgu2GtQvVmG3QukIIAm2vzLk yjKSGAxq4rU5+9BaMnvb+RBAMk4PSmU= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=motorola.com header.s=DKIM202306 header.b="U XNGIZC"; spf=pass (imf16.hostedemail.com: domain of mbland@motorola.com designates 148.163.148.104 as permitted sender) smtp.mailfrom=mbland@motorola.com; dmarc=pass (policy=none) header.from=motorola.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713288306; a=rsa-sha256; cv=none; b=OW0yr6BJamgIuKx3wVZQl0IzM1yLJPiNxqve/uOOhLUl+VtZm5Gl/BajL28P2a+WnhCOnz EyPI7Jx2v6+dpOtfIYSExLQ/b406SSsrtv6MyIrB3CI3tkX4rinPhDNvOhGLFpSwuDVjXH g8I2Ego9H4MOaiqMeaiJGmpt4SlOVwI= Received: from pps.filterd (m0355087.ppops.net [127.0.0.1]) by mx0a-00823401.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 43GFPN5k006602; Tue, 16 Apr 2024 17:24:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=motorola.com; h= message-id:in-reply-to:references:to:cc:from:date:subject; s= DKIM202306; bh=Ktj2knpZkxEzepUnXbFCAHnzLfkAa26Odp8kjW5qVXw=; b=U XNGIZCS9YmDgBUeEu5cofHA9mvaV6V1cdLYUPyD490DtaRc+ylTIkjPFzxuWV+4T 21f6DuwRlCKVXuIbxpUAP7zIVfKAOnpoOVrxFs50JPTCkWAv031LnE5jmeiZi1aA R4XwQPSs9ieKI8NnJOZWkmkEZnXk98ir5fZoWx004yfu9JRghgvTKQx8l5r46UH2 3Gxac+BT+Y/DdxHClOLeMtkCkpMCot4eJMbl5uT1eqEPYTiWpENBGPURZC3kfUqR JGoVqQIu/+3pYXSYSfFrBnBWrgCOkuJqt8EReXdQEsqrYza/cDg2Dejomjcze8ru uOgM14JDYDWyjwAaePTTg== Received: from ilclpfpp01.lenovo.com ([144.188.128.67]) by mx0a-00823401.pphosted.com (PPS) with ESMTPS id 3xhfb73r8r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 16 Apr 2024 17:24:58 +0000 (GMT) Received: from va32lmmrp01.lenovo.com (va32lmmrp01.mot.com [10.62.177.113]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ilclpfpp01.lenovo.com (Postfix) with ESMTPS id 4VJrWs1z69zfBZq; Tue, 16 Apr 2024 17:24:57 +0000 (UTC) Received: from ilclbld243.mot.com (ilclbld243.mot.com [100.64.22.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: mbland) by va32lmmrp01.lenovo.com (Postfix) with ESMTPSA id 4VJrWs0bYbz2VZS4; Tue, 16 Apr 2024 17:24:57 +0000 (UTC) Message-Id: <20240416122254.868007168-2-mbland@motorola.com> In-Reply-To: <20240416122254.868007168-1-mbland@motorola.com> References: <20240416122254.868007168-1-mbland@motorola.com> To: linux-mm@kvack.org Cc: Maxwell Bland , linux-kernel@vger.kernel.org, Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes From: Maxwell Bland Date: Tue, 2 Apr 2024 15:15:01 -0500 Subject: [PATCH 1/5] mm: allow arch refinement/skip for vmap alloc X-Proofpoint-GUID: m5q-_kUrWpAiWSiKGKl9KqD-wJ45KyMZ X-Proofpoint-ORIG-GUID: m5q-_kUrWpAiWSiKGKl9KqD-wJ45KyMZ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-04-16_14,2024-04-16_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 impostorscore=0 mlxscore=0 suspectscore=0 adultscore=0 lowpriorityscore=0 malwarescore=0 clxscore=1015 mlxlogscore=999 phishscore=0 priorityscore=1501 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2404010003 definitions=main-2404160108 X-Stat-Signature: wc8dygh6t9n7ao87goasjpdimorc75af X-Rspamd-Queue-Id: 8237E18000C X-Rspamd-Server: rspam10 X-Rspam-User: X-HE-Tag: 1713288306-341819 X-HE-Meta: U2FsdGVkX1/SpeSPJflTsUqqse1mZKpwkGfJEMlQj1AMy5t8CFsiM1oCUMy1U+QNvYSz4NzRm0tbSPyk4nYoGeknafy/wGe2b6je21zwv4dBmejBLcYjwue1yN1hPaZEKGL66RBWm0DhacZEVph47742EIqB0PlO9oqJ7staoYaOkVK8AFVzIfG8jNUCnscIlwhUB9oEeC6kShn0gZ4T9gbWn0HM04XjeKLH6RjHC6MlGvSkYReqT4400wKxJh9yOltT46TQJIO/jZE3zHzFjRDhVjbN6MYv4eX73ZKWCqvrEEX6gQh+ICIksRZbQajNtjX4Pd05GTOIHaaLHEG+VLyt8XfCNGfv53kGzibMg77m68qqeMbDOeYbNKvcduLlZDsutv3L5zePskRkL7pEjGHy2c7AaiLjiXIEhEfHRpb14o4iIJwweSfH6gY8EwjLRnogMpL9mjvKBNypXPhKti/nbmUvvhHQQV/msRKgG34CiSwOB8tl6WBYkHOBFCYlHdjD+m46E1Hokf2mapwUYWN7YmRcEoepuuy6d4D8CjYp9gl856OVzWstjF0BSPdNSh8OO++j7ViysLymEs8TJozqT2DQEcgrGE9Kl5OoFVJkCYRWBwWgxOkdcdxJc4/FzmWyfnU2kqWTAuJBHohJ+q8ttTlvB9CL4sTxOO0aefSWCFsXyz50uv/pG6eMPHbr5B1IRlTiyz8J9LDUxoI4Q1IhZvc/FkHbLwrAYngk1afQF8STTSVeJB6U5vL2VGHMURd4UMusnY0fWtHZuGF+x0XR97/eL1K10JIFxiMkUkzhZ0m9Lm/TcOTXWkySwiqt3XqTwUD5nfd+W0CIreiN4Nf0Ksb+v3mYrbPnFPfWJPvr5pbKYPyT0kgqGJGkO9flZbQ1AwhmXxmnaMl7gcCWlidT7dwlvp2XkztdHfARNQZ0x/1632/9uB1V821GyLY9FMnUjagcrJNPPA2Av3F cDWGNPca QDLrypHqR4oqXEMdJSQtxtBvQ/tgT7qdbNVOq36moCaAqk1UoVm8sCTiC2JdPYqbZA0NFLpWSJSoGmbw5Nae9v2uAAm5/uC5X6Evornhwq3LPJ2C5sK6rCDXUcLM1rYlsKSL+QtGyxOj+Xm2/CqswfEvtwqDf+tRQjAsau//h4A3zmZL7ba5hUqC1mPLXzTEs6L+QjayaxiGYuBTihWuG4tE3ZD1L/j/WDDZhL5w4XoGmAW+5ADNz4r9YsgH1mVyr/hgOHbY8TisN5HQCYZRNQXcbug== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Makes red black tree allocation more flexible on a per-architecture basis by introducing an optional hooks to refine the red-black tree structuring and exposing vmalloc functions for clipping vmap areas, finding vmap areas, and inserting vmap areas. With this patch, the red-black vmap tree can be refined to account for architecture-specific memory management operations, most notably address space layout randomization, as these features conflict with generic management of a single vmalloc_start to vmalloc_end range as given by mm/vmalloc.c. For example, x86 is forced to restrict aslr to 1024 possible locations, which is a very, very small number, and arm64 breaks standard code/data partitioning altogether, which prevents the enforcement of performant immmutability on kernel page tables. Signed-off-by: Maxwell Bland --- include/linux/vmalloc.h | 24 ++++++++++++++++++++++++ mm/vmalloc.c | 16 ++++++++++------ 2 files changed, 34 insertions(+), 6 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 98ea90e90439..3c5ce7ee0bea 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -12,6 +12,7 @@ #include +struct kmem_cache; struct vm_area_struct; /* vma defining user mapping in mm_types.h */ struct notifier_block; /* in notifier.h */ struct iov_iter; /* in uio.h */ @@ -125,6 +126,21 @@ static inline pgprot_t arch_vmap_pgprot_tagged(pgprot_t prot) } #endif +#ifndef arch_skip_va +static inline bool arch_skip_va(struct vmap_area *va, unsigned long vstart) +{ + return false; +} +#endif + +#ifndef arch_refine_vmap_space +static inline void arch_refine_vmap_space(struct rb_root *root, + struct list_head *head, + struct kmem_cache *cachep) +{ +} +#endif + /* * Highlevel APIs for driver use */ @@ -214,6 +230,14 @@ extern struct vm_struct *__get_vm_area_caller(unsigned long size, void free_vm_area(struct vm_struct *area); extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); +extern void insert_vmap_area_augment(struct vmap_area *va, struct rb_node *from, + struct rb_root *root, + struct list_head *head); +extern int va_clip(struct rb_root *root, struct list_head *head, + struct vmap_area *va, unsigned long nva_start_addr, + unsigned long size); +extern struct vmap_area *__find_vmap_area(unsigned long addr, + struct rb_root *root); struct vmap_area *find_vmap_area(unsigned long addr); static inline bool is_vm_area_hugepages(const void *addr) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 68fa001648cc..de4577a3708e 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -989,7 +989,7 @@ unsigned long vmalloc_nr_pages(void) return atomic_long_read(&nr_vmalloc_pages); } -static struct vmap_area *__find_vmap_area(unsigned long addr, struct rb_root *root) +struct vmap_area *__find_vmap_area(unsigned long addr, struct rb_root *root) { struct rb_node *n = root->rb_node; @@ -1322,7 +1322,7 @@ insert_vmap_area(struct vmap_area *va, link_va(va, root, parent, link, head); } -static void +void insert_vmap_area_augment(struct vmap_area *va, struct rb_node *from, struct rb_root *root, struct list_head *head) @@ -1501,7 +1501,7 @@ find_vmap_lowest_match(struct rb_root *root, unsigned long size, vstart < va->va_start) { node = node->rb_left; } else { - if (is_within_this_va(va, size, align, vstart)) + if (!arch_skip_va(va, vstart) && is_within_this_va(va, size, align, vstart)) return va; /* @@ -1522,7 +1522,8 @@ find_vmap_lowest_match(struct rb_root *root, unsigned long size, */ while ((node = rb_parent(node))) { va = rb_entry(node, struct vmap_area, rb_node); - if (is_within_this_va(va, size, align, vstart)) + if (!arch_skip_va(va, vstart) && + is_within_this_va(va, size, align, vstart)) return va; if (get_subtree_max_size(node->rb_right) >= length && @@ -1554,7 +1555,7 @@ find_vmap_lowest_linear_match(struct list_head *head, unsigned long size, struct vmap_area *va; list_for_each_entry(va, head, list) { - if (!is_within_this_va(va, size, align, vstart)) + if (arch_skip_va(va, vstart) || !is_within_this_va(va, size, align, vstart)) continue; return va; @@ -1617,7 +1618,7 @@ classify_va_fit_type(struct vmap_area *va, return type; } -static __always_inline int +__always_inline int va_clip(struct rb_root *root, struct list_head *head, struct vmap_area *va, unsigned long nva_start_addr, unsigned long size) @@ -5129,4 +5130,7 @@ void __init vmalloc_init(void) vmap_node_shrinker->count_objects = vmap_node_shrink_count; vmap_node_shrinker->scan_objects = vmap_node_shrink_scan; shrinker_register(vmap_node_shrinker); + + arch_refine_vmap_space(&free_vmap_area_root, &free_vmap_area_list, + vmap_area_cachep); }