From patchwork Wed Aug 14 16:19:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sidhartha Kumar X-Patchwork-Id: 13763700 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF9A9C52D7B for ; Wed, 14 Aug 2024 16:20:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10E036B009C; Wed, 14 Aug 2024 12:20:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 099596B009D; Wed, 14 Aug 2024 12:20:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDCB86B009E; Wed, 14 Aug 2024 12:20:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BA4206B009C for ; Wed, 14 Aug 2024 12:20:07 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6DEEBA824E for ; Wed, 14 Aug 2024 16:20:07 +0000 (UTC) X-FDA: 82451362854.01.D3A31DB Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf07.hostedemail.com (Postfix) with ESMTP id EC69B4001B for ; Wed, 14 Aug 2024 16:20:04 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=nZBIKsmq; spf=pass (imf07.hostedemail.com: domain of sidhartha.kumar@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=sidhartha.kumar@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723652334; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zOe6rYReOJ9R8LQkfQkW5rBoXsJ7vwbdrjrfz4Pketk=; b=SwOiYyxLRkIZkc4VzuKtbix1STIyejwh0UjN4THos0WG9ta1xHfxjDGKSVjDK/5kT4DWIZ erNzMJBFTq40p21SLlcrdzpcn54mTicWYpqzy2pDPVzQ6IL9P3NNNl9uy1S6klGyrYFnXI QzD62ma3lxYT0zWZ4NKUPxj4x4lbTgw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723652334; a=rsa-sha256; cv=none; b=WUtw6Uw2oWnBVpOGCyalmpnaKxPCBKdSnGrblU2Vke7ENNvSzni7STcdmtqi3tkKWhpNMR 4oE7Cm0pyxjMxIG8zRu10+AatBc3N9va2QYW1mJiy3ZkFWJko//inok+ryAV/oqunXQjIK W0fwjiq5NSNJbF+qoBTvZfHRskvHa6o= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=nZBIKsmq; spf=pass (imf07.hostedemail.com: domain of sidhartha.kumar@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=sidhartha.kumar@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 47EBtZ2p028354; Wed, 14 Aug 2024 16:19:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=corp-2023-11-20; bh=z Oe6rYReOJ9R8LQkfQkW5rBoXsJ7vwbdrjrfz4Pketk=; b=nZBIKsmq3qfVDaDOR NFFfs0Q1jRlhfa5GlaKoXfrdhdqeTn+TY3tZzmF2j5K5d6uungZgFxi6qMHhAMmQ RiKNhom+4tcnMagmGVicBeo30y8Fb7A1ldRREU8N2Y8ECVkHlYxn3ACEy+QYZ4yD u+fja3wHjXel0jgu37GTyaYjVQj/DOPVqp10MEawmBRZ7Z6H0DjfndUEm2Tv4CwB MLZzaEjFXpd8l1qt2aE3pGlBysVMoCA9AdaRD00hsm0Epqqa/2IpmL3vMbJorien xvwlLcqBzQfVgqyXYYsjOqtGEAmV+jdnzXtx2iKR+4VxGEerEa2jE/Zp5ZSKzAcw 9zwZQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40wxt10shw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Aug 2024 16:19:53 +0000 (GMT) Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 47EFqAgH021061; Wed, 14 Aug 2024 16:19:51 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40wxngn7ma-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Aug 2024 16:19:51 +0000 Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 47EGIvC4035951; Wed, 14 Aug 2024 16:19:51 GMT Received: from sidkumar-mac.us.oracle.com (dhcp-10-65-174-212.vpn.oracle.com [10.65.174.212]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40wxngn7gt-5; Wed, 14 Aug 2024 16:19:51 +0000 From: Sidhartha Kumar To: linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org Cc: linux-mm@kvack.org, akpm@linux-foundation.org, liam.howlett@oracle.com, willy@infradead.org, surenb@google.com, Sidhartha Kumar Subject: [PATCH v4 04/17] maple_tree: introduce mas_wr_store_type() Date: Wed, 14 Aug 2024 12:19:31 -0400 Message-ID: <20240814161944.55347-5-sidhartha.kumar@oracle.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20240814161944.55347-1-sidhartha.kumar@oracle.com> References: <20240814161944.55347-1-sidhartha.kumar@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-08-14_12,2024-08-13_02,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 bulkscore=0 phishscore=0 malwarescore=0 suspectscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2408140111 X-Proofpoint-GUID: 2sRgIeRR8qHOa4oJ-mYnfXbiem0eaLIF X-Proofpoint-ORIG-GUID: 2sRgIeRR8qHOa4oJ-mYnfXbiem0eaLIF X-Rspamd-Queue-Id: EC69B4001B X-Stat-Signature: xo8s5ktea5cjqrmgdrb13oy4i6rhzc75 X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1723652404-69057 X-HE-Meta: U2FsdGVkX19O6j10c4Ff9imPJZk5SkgjsfUgHZD5YKpaNdfB3Lj8tCsNxCRHbQSAofh8/nDh+S/tH3OwybsqMVN/edk2IDZQdANkKpZcJKvTnanXN9CFF2X6nl/Aq5A149BCAZF6z2pzeyvO20e8behZt3MBUgUZu7ipAL2cMch+xHH7/OzNQlUkvGzqVwd4FDUuB6lZTqMDJWWjVBKZu1BlON3FyES/8I1nMzW63GtE8PEllEjB04AmtsBcoBX4HbUGvCu/iONnQhOK8EfWBBAdIAXPyK33apwN7SVvLTL6Fc6BVIUqNP2a9rqJxmFRr3N3646Jqvn19Ae4f0SB8/WNV+oLyqfA2SvXkVX8oqSLPwAv28xG8U8DScDKB3fJVELLOJnFWsGHriOHf4slPTs4s+i21q+bSV0T91LSrq6Y38QKcRJBp8iNpqqMs0NSCJ3yMJ7JEpliZXxX7kqV68yHm8qgAByHQT16Ms/KPcXBFkx2cRIQJGTahSQBiee4waV55KpbUoXy/Dqc5dpRx1B8W9p/ib1BE7ZbGtAdHoRlNQl2xqCQjWyMuxQNizjvtZeKKc6+ebieaITC9knZsDdM+PBQGrFPyqXCavLmO890rcbEKfLudtGlKlmcWWELrYdEW3r3hreHha5U+b6x1pqw+DV/m3CfGnKTEP3ocbHks1Ey19c1a20fZLOwJQPVPwmDL9qGN1cyxbWJNjfoYkeAVVsQ1q2C1mna9DG35CI2uo8hk2Cji0dyT4uoiSykIqExPrscq+BqAiSqrVa0+6KAEQDmy9CnIlI5bIVHAKPnq+1yhyb7mJHyCD/9lrju8H9ZOWrtWJ0FwYCKLDRTvHIVDA1hgmWVOKoUPgXn9WczY1bIMeP9/KmyehpJzdivBFLW6hb9rPFqs5FcQHS7jB+rx/K1XD+07sDYtDYtWNI1gCbydPh/a7Vt0oXdOuxUOp79bxYY4HUF1SvnPl/ MjhfD7fZ F1PFIsOi+KnkUBMKeh7mqgafmIM23Kyh45oOQLlrGyCm/ORtPa2cHGgv+f+vQrTtcK3Dd7GQq3fu0e5r8DHNZwAy3tuWrWpXitLR9wtAWp/FCbn5Eespgv3dmMUbYyz+h00PZsPUR2jE7SRnFFasZlImzHr78bTKmtqOlGVT802Y7ieNTPSpYyubZsdmrhyBzWfW75mrEL+o2honyxXVEAjy94SuAz05PGuIuIddOQ8PGB/S/K9qxLNvKawnsyRsSrTJBwZsWQKq7N58= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce mas_wr_store_type() which will set the correct store type based on a walk of the tree. In mas_wr_node_store() the <= min_slots condition is changed to < as if new_end is = to mt_min_slots then there is not enough room. mas_prealloc_calc() is also introduced to abstract the calculation used to determine the number of nodes needed for a store operation. In this change a call to mas_reset() is removed in the error case of mas_prealloc(). This is only needed in the MA_STATE_REBALANCE case of mas_destroy(). We can move the call to mas_reset() directly to mas_destroy(). Also, add a test case to validate the order that we check the store type in is correct. This test models a vma expanding and then shrinking which is part of the boot process. Signed-off-by: Sidhartha Kumar --- lib/maple_tree.c | 217 ++++++++++++++++++++++--------- tools/testing/radix-tree/maple.c | 36 +++++ 2 files changed, 193 insertions(+), 60 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index de4a91ced8ca..d0b9b3795b96 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1372,9 +1372,9 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) return NULL; } + mas->node = NULL; /* empty tree */ if (unlikely(!root)) { - mas->node = NULL; mas->status = ma_none; mas->offset = MAPLE_NODE_SLOTS; return NULL; @@ -3890,7 +3890,7 @@ static inline bool mas_wr_node_store(struct ma_wr_state *wr_mas, bool in_rcu = mt_in_rcu(mas->tree); /* Check if there is enough data. The room is enough. */ - if (!mte_is_root(mas->node) && (new_end <= mt_min_slots[wr_mas->type]) && + if (!mte_is_root(mas->node) && (new_end < mt_min_slots[wr_mas->type]) && !(mas->mas_flags & MA_STATE_BULK)) return false; @@ -4275,6 +4275,146 @@ static inline void mas_wr_prealloc_setup(struct ma_wr_state *wr_mas) wr_mas->content = mas_start(mas); } +/** + * mas_prealloc_calc() - Calculate number of nodes needed for a + * given store oepration + * @mas: The maple state + * @entry: The entry to store into the tree + * + * Return: Number of nodes required for preallocation. + */ +static inline int mas_prealloc_calc(struct ma_state *mas, void *entry) +{ + int ret = mas_mt_height(mas) * 3 + 1; + + switch (mas->store_type) { + case wr_invalid: + WARN_ON_ONCE(1); + break; + case wr_new_root: + ret = 1; + break; + case wr_store_root: + if (likely((mas->last != 0) || (mas->index != 0))) + ret = 1; + else if (((unsigned long) (entry) & 3) == 2) + ret = 1; + else + ret = 0; + break; + case wr_spanning_store: + ret = mas_mt_height(mas) * 3 + 1; + break; + case wr_split_store: + ret = mas_mt_height(mas) * 2 + 1; + break; + case wr_rebalance: + ret = mas_mt_height(mas) * 2 - 1; + break; + case wr_node_store: + ret = mt_in_rcu(mas->tree) ? 1 : 0; + break; + case wr_append: + case wr_exact_fit: + case wr_slot_store: + ret = 0; + } + + return ret; +} + +/* + * mas_wr_store_type() - Set the store type for a given + * store operation. + * @wr_mas: The maple write state + */ +static inline void mas_wr_store_type(struct ma_wr_state *wr_mas) +{ + struct ma_state *mas = wr_mas->mas; + unsigned char new_end; + + if (unlikely(mas_is_none(mas) || mas_is_ptr(mas))) { + mas->store_type = wr_store_root; + return; + } + + if (unlikely(!mas_wr_walk(wr_mas))) { + mas->store_type = wr_spanning_store; + return; + } + + /* At this point, we are at the leaf node that needs to be altered. */ + mas_wr_end_piv(wr_mas); + if (!wr_mas->entry) + mas_wr_extend_null(wr_mas); + + new_end = mas_wr_new_end(wr_mas); + if ((wr_mas->r_min == mas->index) && (wr_mas->r_max == mas->last)) { + mas->store_type = wr_exact_fit; + return; + } + + if (unlikely(!mas->index && mas->last == ULONG_MAX)) { + mas->store_type = wr_new_root; + return; + } + + /* Potential spanning rebalance collapsing a node */ + if (new_end < mt_min_slots[wr_mas->type]) { + if (!mte_is_root(mas->node)) { + mas->store_type = wr_rebalance; + return; + } + mas->store_type = wr_node_store; + return; + } + + if (new_end >= mt_slots[wr_mas->type]) { + mas->store_type = wr_split_store; + return; + } + + if (!mt_in_rcu(mas->tree) && (mas->offset == mas->end)) { + mas->store_type = wr_append; + return; + } + + if ((new_end == mas->end) && (!mt_in_rcu(mas->tree) || + (wr_mas->offset_end - mas->offset == 1))) { + mas->store_type = wr_slot_store; + return; + } + + if (mte_is_root(mas->node) || (new_end >= mt_min_slots[wr_mas->type]) || + (mas->mas_flags & MA_STATE_BULK)) { + mas->store_type = wr_node_store; + return; + } + + mas->store_type = wr_invalid; + MAS_WARN_ON(mas, 1); +} + +/** + * mas_wr_preallocate() - Preallocate enough nodes for a store operation + * @wr_mas: The maple write state + * @entry: The entry that will be stored + * + */ +static inline void mas_wr_preallocate(struct ma_wr_state *wr_mas, void *entry) +{ + struct ma_state *mas = wr_mas->mas; + int request; + + mas_wr_prealloc_setup(wr_mas); + mas_wr_store_type(wr_mas); + request = mas_prealloc_calc(mas, entry); + if (!request) + return; + + mas_node_count(mas, request); +} + /** * mas_insert() - Internal call to insert a value * @mas: The maple state @@ -5508,69 +5648,25 @@ EXPORT_SYMBOL_GPL(mas_store_prealloc); int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp) { MA_WR_STATE(wr_mas, mas, entry); - unsigned char node_size; - int request = 1; - int ret; - - - if (unlikely(!mas->index && mas->last == ULONG_MAX)) - goto ask_now; + int ret = 0; + int request; mas_wr_prealloc_setup(&wr_mas); - /* Root expand */ - if (unlikely(mas_is_none(mas) || mas_is_ptr(mas))) - goto ask_now; - - if (unlikely(!mas_wr_walk(&wr_mas))) { - /* Spanning store, use worst case for now */ - request = 1 + mas_mt_height(mas) * 3; - goto ask_now; - } - - /* At this point, we are at the leaf node that needs to be altered. */ - /* Exact fit, no nodes needed. */ - if (wr_mas.r_min == mas->index && wr_mas.r_max == mas->last) - return 0; - - mas_wr_end_piv(&wr_mas); - node_size = mas_wr_new_end(&wr_mas); - - /* Slot store, does not require additional nodes */ - if (node_size == mas->end) { - /* reuse node */ - if (!mt_in_rcu(mas->tree)) - return 0; - /* shifting boundary */ - if (wr_mas.offset_end - mas->offset == 1) - return 0; - } + mas_wr_store_type(&wr_mas); + request = mas_prealloc_calc(mas, entry); + if (!request) + return ret; - if (node_size >= mt_slots[wr_mas.type]) { - /* Split, worst case for now. */ - request = 1 + mas_mt_height(mas) * 2; - goto ask_now; + mas_node_count_gfp(mas, request, gfp); + if (mas_is_err(mas)) { + mas_set_alloc_req(mas, 0); + ret = xa_err(mas->node); + mas_destroy(mas); + mas_reset(mas); + return ret; } - /* New root needs a single node */ - if (unlikely(mte_is_root(mas->node))) - goto ask_now; - - /* Potential spanning rebalance collapsing a node, use worst-case */ - if (node_size - 1 <= mt_min_slots[wr_mas.type]) - request = mas_mt_height(mas) * 2 - 1; - - /* node store, slot store needs one node */ -ask_now: - mas_node_count_gfp(mas, request, gfp); mas->mas_flags |= MA_STATE_PREALLOC; - if (likely(!mas_is_err(mas))) - return 0; - - mas_set_alloc_req(mas, 0); - ret = xa_err(mas->node); - mas_reset(mas); - mas_destroy(mas); - mas_reset(mas); return ret; } EXPORT_SYMBOL_GPL(mas_preallocate); @@ -5596,7 +5692,8 @@ void mas_destroy(struct ma_state *mas) */ if (mas->mas_flags & MA_STATE_REBALANCE) { unsigned char end; - + if (mas_is_err(mas)) + mas_reset(mas); mas_start(mas); mtree_range_walk(mas); end = mas->end + 1; diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c index ef5b83cf94ea..ad42a36231fb 100644 --- a/tools/testing/radix-tree/maple.c +++ b/tools/testing/radix-tree/maple.c @@ -36283,6 +36283,38 @@ static void check_nomem_writer_race(struct maple_tree *mt) mtree_unlock(mt); } + /* test to simulate expanding a vma from [0x7fffffffe000, 0x7ffffffff000) + * to [0x7ffde4ca1000, 0x7ffffffff000) and then shrinking the vma to + * [0x7ffde4ca1000, 0x7ffde4ca2000) + */ +static inline int check_vma_modification(struct maple_tree *mt) +{ + MA_STATE(mas, mt, 0, 0); + + mtree_lock(mt); + /* vma with old start and old end */ + __mas_set_range(&mas, 0x7fffffffe000, 0x7ffffffff000 - 1); + mas_preallocate(&mas, xa_mk_value(1), GFP_KERNEL); + mas_store_prealloc(&mas, xa_mk_value(1)); + + /* next write occurs partly in previous range [0, 0x7fffffffe000)*/ + mas_prev_range(&mas, 0); + /* expand vma to {0x7ffde4ca1000, 0x7ffffffff000) */ + __mas_set_range(&mas, 0x7ffde4ca1000, 0x7ffffffff000 - 1); + mas_preallocate(&mas, xa_mk_value(1), GFP_KERNEL); + mas_store_prealloc(&mas, xa_mk_value(1)); + + /* shrink vma to [0x7ffde4ca1000, 7ffde4ca2000) */ + __mas_set_range(&mas, 0x7ffde4ca2000, 0x7ffffffff000 - 1); + mas_preallocate(&mas, NULL, GFP_KERNEL); + mas_store_prealloc(&mas, NULL); + mt_dump(mt, mt_dump_hex); + + mas_destroy(&mas); + mtree_unlock(mt); + return 0; +} + void farmer_tests(void) { struct maple_node *node; @@ -36290,6 +36322,10 @@ void farmer_tests(void) mt_dump(&tree, mt_dump_dec); + mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | MT_FLAGS_USE_RCU); + check_vma_modification(&tree); + mtree_destroy(&tree); + tree.ma_root = xa_mk_value(0); mt_dump(&tree, mt_dump_dec);