From patchwork Fri Jan 27 19:40:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F01C7C54EAA for ; Fri, 27 Jan 2023 19:41:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8CDC66B007D; Fri, 27 Jan 2023 14:41:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 87CFA6B007B; Fri, 27 Jan 2023 14:41:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71E776B007D; Fri, 27 Jan 2023 14:41:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 61A096B0078 for ; Fri, 27 Jan 2023 14:41:24 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3F390A0FC0 for ; Fri, 27 Jan 2023 19:41:24 +0000 (UTC) X-FDA: 80401598088.11.1CF7849 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf25.hostedemail.com (Postfix) with ESMTP id 76C97A0023 for ; Fri, 27 Jan 2023 19:41:17 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=q++c4JbF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 33CjUYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=33CjUYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848477; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SRJKiHKrq/SvrK23nJ6WvAwYz6ftOZ3c7xtMrR/Zkmo=; b=M+Uv3xJtiqZNNiJnySVOigo4aQngqJnOzIb2LlujnaUK7LrGaZDfa6T9byRFYUlK/kY+i5 OJ2OIzE8vrCryMgYMV/RmGUHl3taC7i0jjhSV8LDe7aC+Bm0IQbj55/GMaNJZGyOeP3Frw qKfU8rE9g/mBBFathPzz5c8IQbcXUB8= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=q++c4JbF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 33CjUYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=33CjUYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848477; a=rsa-sha256; cv=none; b=4HpnhH4pcRgvYYmL/D82lQCFpBzRhEI4WXHlQeoWJyf9RK7RQIOmIIAu7icHiL/l75Ny7n FQCzKQ2qOGN/SqCZRJMc+J3kH8ZvkpClDG9KZBKBoTWzE6i7qlqtJo8YZWIruSKY0za/8r 2iiGrEzXIDDM6Jqxfs3W1eX/MuLmG3g= Received: by mail-yb1-f202.google.com with SMTP id p19-20020a25d813000000b0080b78270db5so6347135ybg.15 for ; Fri, 27 Jan 2023 11:41:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SRJKiHKrq/SvrK23nJ6WvAwYz6ftOZ3c7xtMrR/Zkmo=; b=q++c4JbFs1cULNmXHJ6C3DczcQ0hXzOKIr6+9Qx2yz39vZWA+rjLc18PKWJkWy6B+C UeBdui8K3sk0l0OIGcxNzLRt494xbxI7WVRAtP+VLTOL1/q+UNWJf0WDLHuQPthduco2 hKKj52t3wKu54Vbflyx5DWFngCga+0YCh2aWf3wXfSgRE76VHyLLWka1QvKlb5EEsVq+ yKsc5SnLc4E9cZlzwXYLALtVVR/qLY2NGEjlF8VIz3gJJ8VVrEVxdVCt2bqpvNVXSzLO bCSv1unz+7C8JA4tTmfPKXKgKgaiBEIUDtx7XKU+dM+xg3yKNZdzkdQzW0TW1MecPnG9 7xHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SRJKiHKrq/SvrK23nJ6WvAwYz6ftOZ3c7xtMrR/Zkmo=; b=tLu+7zWgEo6/BkyKS64h1IcQuQjaRni1jqUZ/ZMNnLuPP3+X7ikSlKb8ObUrlAh+XF eBe+qhf2ox0vA0Ht8GeDAYMlR4uNtWRmYW9IDJv1dVQWM4JZx1tirT57w3EdzOEpIZ4z +MP+SWFT1SqJhMprSMrF6DxOS2zUeHSkVaYFSM3a8TkUA5wAwi8oK0zV+RKbaJX5TFPz S4F67Ca356wx2u6ALkWkMrjj7bOy4057V36Jd0ozA7+xrd8IkDd//tcJT4iYlmCWiSGx QurKU8NODJXqszYOhkaZv9ZspiGgQOM8WHjW3affDBVst1ZkWdLRUDlp2tmHfRNkGRbc 0Z4w== X-Gm-Message-State: AFqh2kr1yayv+E7gdrZwSfDu1/CGcBX05UZZU1cSAVMXPBkuoxndNgYC IvwDpRGs5fXhchU1yoJgOoCwZaNlsQE= X-Google-Smtp-Source: AMrXdXvEqVwWOYAYjpqjq4VZxkQLftXilDeGRL6KyIgBgBs2rP6fdUqH1kwtp8PmmItyT8YCkU5DjbIMLFI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:4e54:0:b0:7fe:6d35:1a28 with SMTP id c81-20020a254e54000000b007fe6d351a28mr4351236ybb.534.1674848476456; Fri, 27 Jan 2023 11:41:16 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:38 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-2-surenb@google.com> Subject: [PATCH v2 01/33] maple_tree: Be more cautious about dead nodes From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 76C97A0023 X-Stat-Signature: hf5wpagfwd45gow5awj7fwmnjpryqeih X-HE-Tag: 1674848477-674551 X-HE-Meta: U2FsdGVkX1+JNOx7J3euxuxadNSEq2NXdaXMWtHiWXHfrbwsrg9mzbcM8wD5h/ORkQ5n3uCxNHG31o2OuZcH4ttAoxtSWpxMRtqwDoh0SjAPmIdrEdc2fJqjPVsvchh3YYz62jZhrJKngMolXwhlkolOoTduSNf6lPG8zifxHpsVPUFi7Xo1gx7+s70ct6yXHEA6V7RG1cMxirRB2HmWUreLOhS5zuXFZRK+nlyIxrFikwmqLLC+F4gAJhRr04WN/CxfqMxPhDXFfPwLsISGY13a/weJsDAFFiSBc9VfhmihcV4FiBKimgvT2CCE+F7od0WUzHWPy2bqD5Bf34e9sR6xpsvlgqPil7ZGjpFiczDTZkaNzsNO+tLil9HKQDOkjWdhN32nq9Xs/hnycj+qRIakio0OXFujvSHEVXh7lXtcFWejnrw5QtxmZpO6uj1Dvt8wq62hOfKqzhaisl0SxiK8c7Wbp+EylVPnEkrXiw3DKWE0148oRPW6LdsopQqsNevQ9KHSJ+3A7W0zyfuDge6E8fTETlqWOaNHiApzEU5eBWEW4NZ0BcoQyC5QJZ4CpzdYXJ+ZRgTZQ4vDv9LbqN2iBBg8xnqTifGVcwufcZc2NiyDfUfijparyq8CYgjp8OsNQaqlhwkj6m2vrM31LD5idFHtg9GLzJKXbMmx6UF+8VIfbG0YKOx73OZPi5DxnMMwa91/KNqzZvpZpEb6eCYVbQ4xzmSFYqVrWpGmVVZ2M5/PX1GMVwSv6OPtcJxd03J6M4Cv51KR8wTkWKichw5aJEp3XXjFltLnsBKc0NTXd74d5vSgOnpaD4AhcbXgUIpQPZ4cG9GjMbqzIKHZdWXXxVWe6IZq9Bh8NELtjBI8v+UXBoRjINy6zL7h0i4rS6bsumdtOtihpHMqwvspmb2GNg3GzVitIBRvTOy5EkZm+z6VG9rm3QkUfSOeAa62CfimOZsdEcL3pqeKCWf RRu3p/d8 nxy2AyU8cWZNebpfB49NTxBLte+y+HmY9yGyDC/fHYDyB5a0+LsVYgBOwjKntoXFFXUfJZRt1zs2f0EKb1H/xE+gxxO27HTv2tcznGn9clDHY7ZWddbvYnznNPXCWe03nQ52QC07BKRS+Hy+EvCdJEixpYYT2wx1VCfMHCPBkwgyjIWOG/G3IS0B8ixk4KLnJpQy3LyF34tYU6UVSxTk7w+JhmRNNe7Kv8u7LZhHXvV1CVcARwuXUnFGCjyG790Ty/aVHHzf++bW6i1uEtR67F8JvO8QSBxLHQWEPvW5D72v0HFaWxIkLA3q480yzebzWOm0FpbIol8r8JX8mY0qybTxgsG0cTiDDuCb0621K4vVBsSP6XMJmWTXBe1WAamRPwS4WJBNIH9wMhxyQIUBlPQcxRrIdoosphfGHmWVu78qoRAwF4Ntu8SEkkxXAdG+lBMQcNbKu8uaen1nsEb1babpgkHoNDUi1MP3bEQGI1SXIt0mUjbDkX3cdIyJQHIajzSs1NVBfs4l4oH3zlraV2WBuLXT1KA1zSgPJ7WbKeV/MZmiEd/Jb2sUtcNRuTxfYsClFyxjrJZ3PGWHNi1oRbCdO0w9Y7QHuIdeaaKwEwg2vQNUgpshWnccHiyEhNeEgLilld6RLLyUK4iwPgsqQ0OvC0XR1r1q+JUMY0mr6+zgvbMxdN6z5b7JPl3fUp35U3tTeCnOKFklFofjrSeJhUhzCEQpoR5DIuCESd3x3YFbx/deoQUDNCh7SOzhy9GDNEVQRf0eV37JIu/m8sHuKoi+ukw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett ma_pivots() and ma_data_end() may be called with a dead node. Ensure to that the node isn't dead before using the returned values. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 52 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 5e9703189259..e5eeecd14eee 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -537,6 +537,7 @@ static inline bool ma_dead_node(const struct maple_node *node) return (parent == node); } + /* * mte_dead_node() - check if the @enode is dead. * @enode: The encoded maple node @@ -618,6 +619,8 @@ static inline unsigned int mas_alloc_req(const struct ma_state *mas) * @node - the maple node * @type - the node type * + * In the event of a dead node, this array may be %NULL + * * Return: A pointer to the maple node pivots */ static inline unsigned long *ma_pivots(struct maple_node *node, @@ -1089,8 +1092,11 @@ static int mas_ascend(struct ma_state *mas) a_type = mas_parent_enum(mas, p_enode); a_node = mte_parent(p_enode); a_slot = mte_parent_slot(p_enode); - pivots = ma_pivots(a_node, a_type); a_enode = mt_mk_node(a_node, a_type); + pivots = ma_pivots(a_node, a_type); + + if (unlikely(ma_dead_node(a_node))) + return 1; if (!set_min && a_slot) { set_min = true; @@ -1394,6 +1400,9 @@ static inline unsigned char ma_data_end(struct maple_node *node, { unsigned char offset; + if (!pivots) + return 0; + if (type == maple_arange_64) return ma_meta_end(node, type); @@ -1429,6 +1438,9 @@ static inline unsigned char mas_data_end(struct ma_state *mas) return ma_meta_end(node, type); pivots = ma_pivots(node, type); + if (unlikely(ma_dead_node(node))) + return 0; + offset = mt_pivots[type] - 1; if (likely(!pivots[offset])) return ma_meta_end(node, type); @@ -4498,6 +4510,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) node = mas_mn(mas); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + mas->max = pivots[offset]; if (offset) mas->min = pivots[offset - 1] + 1; @@ -4519,6 +4534,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); offset = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + if (offset) mas->min = pivots[offset - 1] + 1; @@ -4567,6 +4585,7 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, struct maple_enode *enode; int level = 0; unsigned char offset; + unsigned char node_end; enum maple_type mt; void __rcu **slots; @@ -4590,7 +4609,11 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, node = mas_mn(mas); mt = mte_node_type(mas->node); pivots = ma_pivots(node, mt); - } while (unlikely(offset == ma_data_end(node, mt, pivots, mas->max))); + node_end = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + + } while (unlikely(offset == node_end)); slots = ma_slots(node, mt); pivot = mas_safe_pivot(mas, pivots, ++offset, mt); @@ -4606,6 +4629,9 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, mt = mte_node_type(mas->node); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + offset = 0; pivot = pivots[0]; } @@ -4652,11 +4678,14 @@ static inline void *mas_next_nentry(struct ma_state *mas, return NULL; } - pivots = ma_pivots(node, type); slots = ma_slots(node, type); - mas->index = mas_safe_min(mas, pivots, mas->offset); + pivots = ma_pivots(node, type); count = ma_data_end(node, type, pivots, mas->max); - if (ma_dead_node(node)) + if (unlikely(ma_dead_node(node))) + return NULL; + + mas->index = mas_safe_min(mas, pivots, mas->offset); + if (unlikely(ma_dead_node(node))) return NULL; if (mas->index > max) @@ -4810,6 +4839,11 @@ static inline void *mas_prev_nentry(struct ma_state *mas, unsigned long limit, slots = ma_slots(mn, mt); pivots = ma_pivots(mn, mt); + if (unlikely(ma_dead_node(mn))) { + mas_rewalk(mas, index); + goto retry; + } + if (offset == mt_pivots[mt]) pivot = mas->max; else @@ -6624,11 +6658,11 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, while (likely(!ma_is_leaf(mt))) { MT_BUG_ON(mas->tree, mte_dead_node(mas->node)); slots = ma_slots(mn, mt); - pivots = ma_pivots(mn, mt); - max = pivots[0]; entry = mas_slot(mas, slots, 0); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + max = pivots[0]; mas->node = entry; mn = mas_mn(mas); mt = mte_node_type(mas->node); @@ -6648,13 +6682,13 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, if (likely(entry)) return entry; - pivots = ma_pivots(mn, mt); - mas->index = pivots[0] + 1; mas->offset = 1; entry = mas_slot(mas, slots, 1); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + mas->index = pivots[0] + 1; if (mas->index > limit) goto none; From patchwork Fri Jan 27 19:40:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1517C38142 for ; Fri, 27 Jan 2023 19:41:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 76E096B0082; Fri, 27 Jan 2023 14:41:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 744136B0083; Fri, 27 Jan 2023 14:41:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E5476B0085; Fri, 27 Jan 2023 14:41:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 493DF6B0082 for ; Fri, 27 Jan 2023 14:41:42 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 279798075B for ; Fri, 27 Jan 2023 19:41:42 +0000 (UTC) X-FDA: 80401598844.29.A94C370 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf08.hostedemail.com (Postfix) with ESMTP id 65C5016001F for ; Fri, 27 Jan 2023 19:41:32 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ij3G6594; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 33ijUYwYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=33ijUYwYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848492; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6D7AVbhEFxAo5EKdbJ2HQfTq9cKDp9C6gMDUsIDn2rU=; b=CRhlGRdcqB02SFnz/mxamiiOQqV7kZAgrTKnc/ygRgL2KciusDhf4XhA/HI3WoODzaPrtV LW8y7rvI++R0/jMVZN1MgVbF8wFtBtkVcxFz0cZBF+l6rj9HMOh4qDB2Ts9keru/TRrHwt x6GhMnkhyBATAw5/v0TWawWJLyay7WQ= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ij3G6594; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 33ijUYwYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=33ijUYwYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848492; a=rsa-sha256; cv=none; b=fjoImR3L+EIrUh/LdgpLpmUPO0Uf9q4kiE+RXqwwID/401RMfHJVsVnioje+cD5Nwb+UT8 s3lbHJzpi9AB/2e2lE0eSXtQTGZvfofkGisJxowYJeSSu5Hk1G52Ufadnzno1PAGvkhGwr kgfTsRm4y7tJzMR3BcKK+pkKEZVOt2w= Received: by mail-pj1-f74.google.com with SMTP id gn18-20020a17090ac79200b0022bef1f49c9so4968053pjb.0 for ; Fri, 27 Jan 2023 11:41:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6D7AVbhEFxAo5EKdbJ2HQfTq9cKDp9C6gMDUsIDn2rU=; b=ij3G6594G9QMVIcaC7y7IotZlrRGSd461+VUEA/BMKS/DqdJRnleVk5IzJ5gil69bZ nXwMrZpHBfopZhY+dBOM8TntT+xgyMzx0lhn0c4XjFlDdq8hUc51rQLAJayIjAlxU0Vs tSDUCQts0+fWuoHVHrLcPeZOnjtBdS6uOzMiFPrkU4AVT++YMj0+eDXitszbX73gjVet Uj7on6Z68KT42I30IFAHZ8MnQyKVo+KH97KXkIJWbVMmYdISzeeP/HtVHdR/3Pzjnpiz r0YfwFvkG3mRoMMu6hvaQ0O0QqVpO+Mcl7G8itnKrK/kLoHBd5Ey6CcVhBTBp6nBJTd/ 24fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6D7AVbhEFxAo5EKdbJ2HQfTq9cKDp9C6gMDUsIDn2rU=; b=RGS8k/JzlBawOwYcIbOPUHAuSzxy+x1VmdH2cxuRQc0IpHGG0HG87OuGsSX9eBvVUM tskW1Q2ZF/oGDQh2HBwYMd7h+Y2ZmV8vlY/ZCOZxTBEZbqxvc4IcVijy3D+itByrW0Su h6pR4eKAsWrNJSkBXQm/RkgI5P/iSu1vUn1a+UQqJt2BvuUZzahU7Q7qr1wP6pkCqCuJ OBTGMwmUTw+uqJS0UK4ceWOYXPSU0LB6RKalaKUMD2oRy/jHKTKnDLFeLk2rnsWsREmZ 01J+/KUFLXhQo0NK85EwXZTTuhPmrfi/79INxPDgxj2hF6EA9NWIN8U5YCAIXbik3jTg Ynjw== X-Gm-Message-State: AO0yUKXDQ8wKVbO+exwGcrDs3SBLra96hbU2n+j3om4oIAVSrqpDt7bb Vmucu0DYSKE8i1xFIpC7LEyedyw7+Ks= X-Google-Smtp-Source: AK7set+8nPPCBQ/ToCHxQhHQnspD+CZKcqljSV2Aj5HMLvXZA8Wu8rpp2+AujJr6dncfulY9izqabQa2p3o= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a05:6a00:1990:b0:593:909f:ed45 with SMTP id d16-20020a056a00199000b00593909fed45mr72889pfl.0.1674848478930; Fri, 27 Jan 2023 11:41:18 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:39 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-3-surenb@google.com> Subject: [PATCH v2 02/33] maple_tree: Detect dead nodes in mas_start() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 65C5016001F X-Stat-Signature: ei1j8uhse7n95sdozxnwkzgsc4zmszrz X-HE-Tag: 1674848492-74319 X-HE-Meta: U2FsdGVkX1+oXg8ElVYqmHM8K73jgmNfB7edvplSgMcUgu2bDRhpFNfhxLoH1fihV4tVvgQAPk1xSw44/iRxxWOWyDVAnSyHxIIP6kDlwf3rv23mMzvMZXOeJsLQrvLq529iW/gofc0oYvFxHjEA0MRnyWz+162R0u2hPiBFq51LOiaf6wM+5cs6fwpAi0xs/4z36sxknRN+1gTuTiR8sJTjewESL0atAbV8czeFjdVM5CfcXfXg87VhLqXdDMW5WecVsUdthGK9otOmmpYjXml5kPnGpyb/E/xpdcSK/k4MWCJiJIJomtEjuD64o0iTtqLeAnucXagzn7Fdq9ZOlVLK+gheDlhvCDa57WNOpWnrDU2XmnDbsb8/YVkx11CQRawSOoO9qgqbU01A7Jq8K7fw0VJZKTipXgXipeznlkNOyQOQiCdIdrVq5f8C9JNmSuAdMwqP+3ujnVbAOAhKuclF7rcNOaxCncI8qIvW+2NlMel3DCgxOQ2W0VJQPTJibMMbdi8tsV5M9QkMyX4dbfOHb3oazeWru5DcgtBLQWtuyp25Kmifzz/3Te3UXd2QjrWLKVzX9Bgman9wgf3k16103FWKG/sqT53hLcFce9wNTXNU0OtBfkq0b/4eYfCAlrGviFTBhBguQ88pXrFoZtlYo1lO61/07kM2UxQSf+D2XU/s8rC/+Z0iyuvYoN3goUSFeCJu1dnrBHrz7tEsSRksfOunu1IjXPdxKG6iyrPj1cdSURS7ZXDVHWDV9TL6x9YN2Rk4qBnl6BbgUa9hZ1Icmu5EvlNUgUZ6xnboa29jFHftUAoazVIcOKuBN5uY68Sm8tLcRuKOL8w/K/zVdz+3mWBLzmiUjSauK/rebFqYudKJ4z/N4dIunL+KpQkArbGTTkDhkjb4nCwpP1MnL9gFJ4h3z6B8OkJA5xIPB4ciY6w8E1NJy12FGJhzrrw7n/Sl9yz/6mXRqp+SG7b aXndxrIx ia8N9qqiZQO6vhxllXkMqBh3N3os03co+fonxGCqw0lI9WKdCXzIFiuBxVp6JPZ0QlEGJYFiolX8WaM2HLQm1chzK8MOnSpasM8807DlXShgzoEb9bFoKlzFPnBcSzQIgZRu+bENXDSAcuQBLnvBIz8ExKwEqHWTJ08MOx4e4+9lDTWaZPTSy8ylhD+oXhyWaOQogeggtrY0CnrZiTE/ekksokf/vXGS9vhQzNWeeJ/KSDVbUd1K/qHOHXO7NeceSMG0jY1eMd2+Kgeq2WhAAAC4aA82gmEvxsFC2uZMO9iOIgiOT3G5RTw31V01rCZdjL31V9f2nZAdqec27thFyAnU/Alep9ZCC3aP0ydhfeABTEVXTR3y6eMedC+9SNEzor2lbbkYx3q2q85K0WXTmPOac/bPArHvg8djZaoiJQ0lcXkmUkIhuO9poABgulH0xoE2LFBDRWBoaJvyfcCpEdR86jjIvuDR8ooIfWxyB360r3tLF4GFl9tKKT+5AaAyCvLmdbqUBUHRMVHAz1gJMsqh4SxaO8gFtiXTADb8aEHCgkTCe4ZkrpZgAQvQllcWRCfMe0uK4NaxJYQr95qNmIqndUL8NTUNZWCoxwDsgcKUUyvCZwGjgejXLjMZeT1nMlZGip0K4+toCE7XJNPZOIe+pwuCTkJfpb30FsURba6doSQ8QdThsUpmY90b5dLoIMPOSSW9pkQ5BKDJ7GAPfXC/XLnModYJAmr0xuFiyr4DkYQzcvZVtwmaeHMe2saNVEp4ldMfnI0IMMSg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett When initially starting a search, the root node may already be in the process of being replaced in RCU mode. Detect and restart the walk if this is the case. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index e5eeecd14eee..482e17a460cb 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1353,12 +1353,16 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) mas->max = ULONG_MAX; mas->depth = 0; +retry: root = mas_root(mas); /* Tree with nodes */ if (likely(xa_is_node(root))) { mas->depth = 1; mas->node = mte_safe_root(root); mas->offset = 0; + if (mte_dead_node(mas->node)) + goto retry; + return NULL; } From patchwork Fri Jan 27 19:40:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD975C38142 for ; Fri, 27 Jan 2023 19:41:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 487A66B007B; Fri, 27 Jan 2023 14:41:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 410626B007E; Fri, 27 Jan 2023 14:41:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 214AD6B0080; Fri, 27 Jan 2023 14:41:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1281E6B007B for ; Fri, 27 Jan 2023 14:41:36 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DCFFCAB7D5 for ; Fri, 27 Jan 2023 19:41:35 +0000 (UTC) X-FDA: 80401598550.23.0591CF3 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf23.hostedemail.com (Postfix) with ESMTP id 98834140011 for ; Fri, 27 Jan 2023 19:41:26 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=S2HR++yb; spf=pass (imf23.hostedemail.com: domain of 34SjUYwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=34SjUYwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848486; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FUzzHzCQcxOurh9YuvudjXpNdZx02NIZtoLw3Jde/PE=; b=WTOeRPgNok1VoTsYIr7sAPAwpPM65nDiw6+K6bil18gBu2abILzRDNGF36Ec0ehd97FrgA NsRMR5HxWUNsVpnI1BRGczhPuLMHaVTIny7T4MZCHnNDMP/I8rjLvDMh9pNw9EZr7Rp9cg p6HCV6haSwsS7liQCdvzgFDb51Sn2b8= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=S2HR++yb; spf=pass (imf23.hostedemail.com: domain of 34SjUYwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=34SjUYwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848486; a=rsa-sha256; cv=none; b=F7UWF4hglFs2pE6LFUmTCzyIOlWzGIBkriX2SJP/7F49qqWR6ZMQGztpTDZPLe+JIgZ2qv iZZBLghUV0J6c1AMPPhLXynQLAhrdzgiyNSIMpPo+Jp9S2MXHjZX+/dCBipBY7UujadzaZ 1FyLSxLeqmlAYMSU2i23AmJ73GUKUrk= Received: by mail-yb1-f202.google.com with SMTP id r8-20020a252b08000000b007b989d5e105so6375702ybr.11 for ; Fri, 27 Jan 2023 11:41:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FUzzHzCQcxOurh9YuvudjXpNdZx02NIZtoLw3Jde/PE=; b=S2HR++ybc/ns4vYecXBlgBA5lh7rKCR5RBxCn2d6nLoproYmP4Y/TuK8A2ykL1TZCy PQi2lsVKc//h5m3TPGFUZRCScYCxf/iBRsPO4Le2L6kyLee01De7/gBctPL5x+AFYdbM +GjlXUavIluJZluW6t/CA9goOPJfDGVnM+Mgt+aMp9WxpRPbD/Tewkr6wZubL9JQbZ/Q D22ZQZJELsrUxamrbo349CKHJL642zBXbi0BO/uzmOjlGEGSeG1j/mwEO9F4yWD/yh+R Z4EUx0fbMYpLh0Co5d8CYSkewP6ymMJoH7lGt/nPXvwtHvtN4R6vwQVewCoOaVtJA9nQ 6Lhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FUzzHzCQcxOurh9YuvudjXpNdZx02NIZtoLw3Jde/PE=; b=B+WnXAGNwMTLuKhhcSmWqEeyP7pDREdy1C/D0L+ShPJMiu7YsSPL6/TLtOIVfPn465 zo48ITBJzfeG5GKTbuBsHjL4GT52OhqWUjyjiEjj3TXZu2uBgZWJJX7b+biRm1Q7XfPT /x/5sgbwxLfe+ONFLAA97kEzoMXOgPGWD/F/I84Neg89C+pIgBYYteiwuF1wqEqZtSqh 5g9faLgghfZr+Ez9OrEr2rhsAyBN0sm79grjxfA83+8XS/W1Yhrgtlis0tgfGnDQPvuT osnVXY8Q9lc/RWyMFcgZkyuxmlsEXKvzh9a/LiSN4Ix+4RbQiHsv5j77EAU5gZQar2LR 9h9Q== X-Gm-Message-State: AO0yUKXzh3TbFAEfDgoJkWgZWaJMyP512W+Ws7FjdVdORQ5Z2Ygpz5QL /IOXe8RH+Evi4BAtwgFokfDTTFb+W7Y= X-Google-Smtp-Source: AK7set8Zoaf7xb4TZLOBvvU/fOAKWYAOQ/jwdJrqulf4473/2u2hKvPvmXN7ahB+v/eEMKhFLKA99DnsrP0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:7e4b:0:b0:506:4f19:740c with SMTP id p11-20020a817e4b000000b005064f19740cmr1518489ywn.383.1674848481189; Fri, 27 Jan 2023 11:41:21 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:40 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-4-surenb@google.com> Subject: [PATCH v2 03/33] maple_tree: Fix freeing of nodes in rcu mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 98834140011 X-Stat-Signature: tesnnfdah8gys9kxo799tzmc6wahykjw X-Rspam-User: X-HE-Tag: 1674848486-389842 X-HE-Meta: U2FsdGVkX18U6fdSWUBtUaEaMkEuomZvLOiO40FyN6i7N52QwtKwIbeXijhjSPypDEjV5FsCnzeORv1kmOYCo4OnhHBJa5WxkqC4WZmxh9GA6xvJvZnnJy8FpUsM7aAT+ZIhEweeXZGrYct3A6GtEBNajF3dnz8wFMfEHaWk72NXtK7FpyeX9B9LQonn2+mw8LsSdK4mHCMOT7N+n3ataG4CobUCZlobrwkRit9K3Xb+niNPjmkg0aLgTJmNR8ZWTwg21m96WnIwZ74J4ch+lxy+X5VfH0DbNcyAASt/USEx8qOCiHLEaxCDPDrIoQcmIg8USMNrsazloDj33ExgwSZ2O232lOmVLHgSI337YkcFtG6srnSKFBNn4Dr37IFBJwjRxkVAU5x1Mi+jAX2aMxx83fW88ZdwL7gduD1MGdIcNuikuKpZvJGTf720jN/bjYp3qHIMbuoXZ8FqYvi0M5kjyZGYXT36Vfh2+ZJeZI+gYe7fpp74zGfsAHS8UX0k7gYMMJvJrBQNr7IPg4pAOXKWdqjMmwnxu/UQzbSWlesLEvpRei4NmfOg+FSjSol4eHbzAOs+rB17OU1egr7Mm3eB6j9MyYDyzIzN8bhZZgVsftrhCJKQbAKNukho2coYiWH6lhAzRZDVMyWAOcxCg8xazrBJqRaWc/D8VWDtusYftBJbgym58TW0RHGBrSA4chXNx4GlRJnJgJrp3ZIjdwbnRL+LA5nSgBpeNm8VBs1Z8pHTZhO+irOCwiJQ+64gaqmF7iXiv8E+JnOLBpPxW21toJ5Mh48R0UiHx4BAtr6CfE3l/CBi32GvvZOoHI83YojXgWqDZ5ticFql+Zt9khKnPG+9+i9ZV7pwznORWgwUvFKAjJllSIPC2O7IA7sbkJ0j5gQYqZ9G8POhIFbB0cfpZl3cw4aj5e1jwFw2qXy1kZdq8x9/DWxM/i0UQ8878sTZhK2v8N/UZ2EZ4bS X52wvbuW Zuj3gGggANW6u+fK6Ct3Hb7kaMR4Ezrb3SOIMF9VufR/k4t22D0DAcq8sTlXBfT8IoRf7s/oRIxmMET384Yo3TN7XeRtoBINpmhshtiOm2Fv5ORtK8C+4JNRzAHCrTOfJ9Tvvm+nR50cQIaJeGQ7+jXbzfeiHhkQ7ZIJ8F/m+Z2og3gqOBzB/cE2ZtB6oFst000zFdHEXSiCpTqfsHUZJOJ6KM7whrRf4mhKO7lhFNgarZmDoKzCEGYz7X4ne/8/nm630O11SH/Si/cFdGmfDHnc0AW2bDyLfUvx8ouCAVs+k0DDT7jr56RwDo2k82FGemEwf0JMg/5nxsk1bTIpCNWMHjtapEzY8Ey0YeCuyCScAFjWl0zP1+UzNKz90cC9wcqc6e4gJl0D5wUS9edKF1//9yEtQKvowB5VRF1AOMzjGF63uRJcEHCUWwzgxPKWQ2yg/jXwJGe1aVocXVCu7iOlojkFdtYnFMaNvIPuNS1e7EC6KbCtFB+7k4BHJMXRshh5FAHRH9ckZ5hPw4x3Cvfhkl4i2HmlWuBdUHOVHgGqmtmMu8eCHICzH4ZaCT23s/D/KuGZMFe57uMP2INO276xOg0YsbFQ8XU7RJj5J5SrzKwin0N3/QNCzm6gDxm/oHzgWm+dEO4NilUGga7abDzcCB6DH1booAZXolYOikliFjAX2aj6lULFAtDmGxHf8wXXdMJr4wF15bXPzrV+WC7cLDdWLUE5FaTo/rmDhRlTS53CMTU+3rPWJTxbTB1te6/8HyY9a0vSC5ns= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett The walk to destroy the nodes was not always setting the node type and would result in a destroy method potentially using the values as nodes. Avoid this by setting the correct node types. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 73 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 482e17a460cb..73917dd2c608 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -895,6 +895,44 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, meta->end = end; } +/* + * mas_clear_meta() - clear the metadata information of a node, if it exists + * @mas: The maple state + * @mn: The maple node + * @mt: The maple node type + * @offset: The offset of the highest sub-gap in this node. + * @end: The end of the data in this node. + */ +static inline void mas_clear_meta(struct ma_state *mas, struct maple_node *mn, + enum maple_type mt) +{ + struct maple_metadata *meta; + unsigned long *pivots; + void __rcu **slots; + void *next; + + switch (mt) { + case maple_range_64: + pivots = mn->mr64.pivot; + if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { + slots = mn->mr64.slot; + next = mas_slot_locked(mas, slots, + MAPLE_RANGE64_SLOTS - 1); + if (unlikely((mte_to_node(next) && mte_node_type(next)))) + return; /* The last slot is a node, no metadata */ + } + fallthrough; + case maple_arange_64: + meta = ma_meta(mn, mt); + break; + default: + return; + } + + meta->gap = 0; + meta->end = 0; +} + /* * ma_meta_end() - Get the data end of a node from the metadata * @mn: The maple node @@ -5448,20 +5486,22 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, * mas_dead_leaves() - Mark all leaves of a node as dead. * @mas: The maple state * @slots: Pointer to the slot array + * @type: The maple node type * * Must hold the write lock. * * Return: The number of leaves marked as dead. */ static inline -unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) +unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, + enum maple_type mt) { struct maple_node *node; enum maple_type type; void *entry; int offset; - for (offset = 0; offset < mt_slot_count(mas->node); offset++) { + for (offset = 0; offset < mt_slots[mt]; offset++) { entry = mas_slot_locked(mas, slots, offset); type = mte_node_type(entry); node = mte_to_node(entry); @@ -5480,14 +5520,13 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) static void __rcu **mas_dead_walk(struct ma_state *mas, unsigned char offset) { - struct maple_node *node, *next; + struct maple_node *next; void __rcu **slots = NULL; next = mas_mn(mas); do { - mas->node = ma_enode_ptr(next); - node = mas_mn(mas); - slots = ma_slots(node, node->type); + mas->node = mt_mk_node(next, next->type); + slots = ma_slots(next, next->type); next = mas_slot_locked(mas, slots, offset); offset = 0; } while (!ma_is_leaf(next->type)); @@ -5551,11 +5590,14 @@ static inline void __rcu **mas_destroy_descend(struct ma_state *mas, node = mas_mn(mas); slots = ma_slots(node, mte_node_type(mas->node)); next = mas_slot_locked(mas, slots, 0); - if ((mte_dead_node(next))) + if ((mte_dead_node(next))) { + mte_to_node(next)->type = mte_node_type(next); next = mas_slot_locked(mas, slots, 1); + } mte_set_node_dead(mas->node); node->type = mte_node_type(mas->node); + mas_clear_meta(mas, node, node->type); node->piv_parent = prev; node->parent_slot = offset; offset = 0; @@ -5575,13 +5617,18 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, MA_STATE(mas, &mt, 0, 0); - if (mte_is_leaf(enode)) + mas.node = enode; + if (mte_is_leaf(enode)) { + node->type = mte_node_type(enode); goto free_leaf; + } + ma_flags &= ~MT_FLAGS_LOCK_MASK; mt_init_flags(&mt, ma_flags); mas_lock(&mas); - mas.node = start = enode; + mte_to_node(enode)->ma_flags = ma_flags; + start = enode; slots = mas_destroy_descend(&mas, start, 0); node = mas_mn(&mas); do { @@ -5589,7 +5636,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, unsigned char offset; struct maple_enode *parent, *tmp; - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; @@ -5613,7 +5661,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, } while (start != mas.node); node = mas_mn(&mas); - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); @@ -5623,6 +5672,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, free_leaf: if (free) mt_free_rcu(&node->rcu); + else + mas_clear_meta(&mas, node, node->type); } /* From patchwork Fri Jan 27 19:40:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BC59C61DA7 for ; Fri, 27 Jan 2023 19:41:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01CC86B0081; Fri, 27 Jan 2023 14:41:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DD5666B0082; Fri, 27 Jan 2023 14:41:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD9676B0083; Fri, 27 Jan 2023 14:41:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 971E66B0081 for ; Fri, 27 Jan 2023 14:41:36 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 744B2161039 for ; Fri, 27 Jan 2023 19:41:36 +0000 (UTC) X-FDA: 80401598592.18.A6E88CF Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf08.hostedemail.com (Postfix) with ESMTP id A9D6A16000E for ; Fri, 27 Jan 2023 19:41:33 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=rnpUdmPc; spf=pass (imf08.hostedemail.com: domain of 34yjUYwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34yjUYwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848493; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DF9zqRO1prEDm4IKfYXh+Z69pFj8RdYSRvmvJpFQZKM=; b=s3G3Q/0W5PLcbidORPXwij5JJ/jE3C3p/xcKaBiBry84NejWTGob5OlHwJPilR69ZjemSo vS0gkdsh4ub8qNLZxsZfFNOZyWTEoBeCY4qiHyL4KgZxOh5vLWN+P4zXlHjaUj/iCpKaek 5AkuR4QjL2CmkBr4s2vPvb24q58M8Mo= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=rnpUdmPc; spf=pass (imf08.hostedemail.com: domain of 34yjUYwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34yjUYwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848493; a=rsa-sha256; cv=none; b=iJTZL5utHZzshUNogaUu1DhMHEqAy36YdYzgADX7Rwpq496fgS0qD+UaAEI/FF+eqrbaeB Ve9EXoDAo8vv6ni73qLAGbK68Su5FoQcrT5HWWVz846LBbH6a8Xig2D4hyuOrCsB+9pOoI rko3v2+usGxHoM8Rn41htm47mj/CifU= Received: by mail-yb1-f201.google.com with SMTP id k15-20020a5b0a0f000000b007eba3f8e3baso6326351ybq.4 for ; Fri, 27 Jan 2023 11:41:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DF9zqRO1prEDm4IKfYXh+Z69pFj8RdYSRvmvJpFQZKM=; b=rnpUdmPcBnY7plbcdmS2pwvwnctQIRIqHumfjs2guCuPsY+Dw6YQ2tfutal8HnrgXz edy5Yrgey9QhoRSbyw/SmRVhoDEfIcnSP0t0LCmLUom2gJSxBSb6RoRoQ6R6E1A4+8T0 XcTb3Fhtcz5YcundAJZb+WYoiQCLWWkiPl+/L7XZyu1tWh8b/ENfJXXDc0COCB4H0HVg yuAJnUBiF1z/T9ta5KOn1AHf3cQZUH/oUuoUg6oOo5mcl+MifhrO/XwhHeojlZjv/4Mm lccC2wtCBVxTksQLP5lorO6u35MCkxakWEmMvAgnG98JEeGI6fsdBcznmI1LtXlDE24U yV4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DF9zqRO1prEDm4IKfYXh+Z69pFj8RdYSRvmvJpFQZKM=; b=u1RvYtlLuOKaqX6mfqnSOb+bqk6w7Dhovuo9oQfe9m2wkL3v/Yq7CiQk1u5uY4NGxP QBgH9MaIYz3asBkFfZCPY0AwIhFANCQUAbtguQEFsB5pwbpVWlG2Lp/cm8HGLv4zav/Y /xgyCAlB4dtaJHrMgHv22EXAsK6eVNVCW+JjCUdqSRK5QYfX54Ji0uG423iJjn3OEFCb SeZuegCDxxmPu6Uf3qfM3sK0n0lByls5eigbWPRPrB6yG4yTva55c6Y0TWSh1g8kji9L F0msnKKWabt2PZtGjjykex+UvoEHnabtYXoFfaJ5VkxItHEcMHnMQ+MW7x7lYcDtuEZT wa1g== X-Gm-Message-State: AFqh2kp63OGQ0Q5sn2UcOuR3d5ATCAjASUwTmC6XCc1zcdPkhTVVCcca mbYoTizyWx+7t4C3lGN/41Tt4kVZr3U= X-Google-Smtp-Source: AMrXdXs3xZaVcR3p65iqnfy8qe0w+PmLMIhbeI9n6IEvkAzMLp6jSQOzi5T/oBy2kRvdJXFWY9xcFs4GkSg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:c8c5:0:b0:4f3:10b1:1351 with SMTP id k188-20020a0dc8c5000000b004f310b11351mr5009568ywd.516.1674848483749; Fri, 27 Jan 2023 11:41:23 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:41 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-5-surenb@google.com> Subject: [PATCH v2 04/33] maple_tree: remove extra smp_wmb() from mas_dead_leaves() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A9D6A16000E X-Stat-Signature: 94rbtk83uax8ugec5pi671cod3babib1 X-HE-Tag: 1674848493-707053 X-HE-Meta: U2FsdGVkX1+gwA/W1jGxdh4mWQFIsLjamFCHSydVwfCMUF30zyoVkj+ASGVJovqrzBHwc2A1Zyi1nyFj0eu4exVc4wRmfKrvCbgK6cPF1fAlhA+Rm+aagxLo342srneGlgsQpQT3zvJec6k1L3X55NMTv0Qtn58j7SJrnwtEwvTzNawAHpjefvMWUnX8DUiFvGlSJ4FQj8osvWzFqrp2RFwc4zI2q8rEif2tdlGn1nUTH3daWZGSRk6XbJj7lBqsXNxWz+snhjDIVFqzXreRQSiy9amo14MN0qhVS5tZSuq2TOUjRnGgVruTp1UbsuczpywkPQ8HBHwCeEoLh43JVKuS2z4o6W123wqUvboYst6ZiSqHgdUxWVcNXzmcYyjIYlqhyJQHSA/b/Mg6sYHBeE7WFINc+0/EeOA3fyd1ZygF0dxkSyU1aYwCSiYVBA5pchPTMC4zXT0ba+dgK8Vu9Y7tfPYlvU+cIc8oiV6d7gEppE62CFufzieG/bNd/WSmpdAC2lkibjNBcvrrIwz5gxeHmW2pfloEQdn5ms6BzoNe+tEPJMNHIlAmGNOe1pNDyYiVGemMfqFdK/WLObvyaz0olzFJvP53J3pi95bh8Mj0vCDBm0ISzn6yxjUwOyBm9rHCcImiLHSdwlh7g2LONrhvohipATBD1RHU3IC/0W6M+q1Dw+QmQqVRLCiMfsNJZ86CS3S0gifw8VeDJm7ksAUxyf/sQMBspQdF/NNpDv0KBBQbNM8rAzpdyycJFUQ934Edm2sI8Y+8CVk+2eYhhq1l23tP+hP/xSk+pzjMSviJiMWEXj3eyxCOf6piQqXBRTT0Ftyq/30kqMYpygYpSn9Gu5ty5H+I2BETyYsPMVhWTYLAfX46SCjaEUQD2VppSQ8Xz/Q0k0W36BRxper8h6b1a4/DOdF6VUNyo26w+xQ5RnBFs6++dawcrxkFtkCjPfJL8LtGzLfqpApy4i4 bXGffeYY lhIOg7eIqmmgaawguShfsBaKGkjCp7HO8s3iL2pZsTEJ/k7FN2HydTAG0gazahSpL8nGDFtg/jRLi4lP9yCaqHSHCDMJt4x4EbOQ0p6zAEtTb4AyP5QgLCWwQU2tXRjER48nqnYg1l5VhqcBxDmUxo9G06CEma2atsKjwL+kl8pJUMr4gbYqBOvgsiJiRvnD+BIO026Ls23Uv8vW5Y8KpcvYVdJdpXOBS9xiXQBP+VcH/4nhDq+HC99f+tEK4RE6e3SWIp8c932+O7MPALy0AgKmcUFyKOLB/eX7W74tANrQP7F3F5MoF2wAev1L6DihRTMYyU1Dj5TZLPRcDN5FO9PqdUyxxlaYsz1f+sjpg3xMDYHmLHIbww2yhR9w7qa1Oa6Tx+G9aDIb+HC8DaCpXBnqW187yZ6R1eQTE9I6MuHBM54MKnmQQhnsdzNBsjU/61O5qrVGlG415Gi+Qrkl+21nqwzr+2qF2GDrf7XIMp7mtzoONVW5IAcm1/ScY6ZLCkIviCBOc7Y8H0I0jeSdhR+8slEu7Rb6k13lvAcwXhac3wF2PVJU3Q1Kri91+iU43R72XGbWkj7ZKbm/+nqAc8BHVni5faKuGmLDBbYkuUWSUbXpX32xLAXPEanXH2HDpqNzKIg4XYTSJpldD0wL5LvZHoVuf7aFUvHgG5ZYXzMeK1KP+fO2B0LmIKNq+xlYeWFCZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett The call to mte_set_dead_node() before the smp_wmb() already calls smp_wmb() so this is not needed. This is an optimization for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 73917dd2c608..75cce2d4d5da 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5510,7 +5510,6 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, break; mte_set_node_dead(entry); - smp_wmb(); /* Needed for RCU */ node->type = type; rcu_assign_pointer(slots[offset], node); } From patchwork Fri Jan 27 19:40:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEF79C61DA7 for ; Fri, 27 Jan 2023 19:41:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0DDDD6B0089; Fri, 27 Jan 2023 14:41:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 068146B008A; Fri, 27 Jan 2023 14:41:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFD576B008C; Fri, 27 Jan 2023 14:41:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9E8976B0089 for ; Fri, 27 Jan 2023 14:41:51 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7FEA5AB7E5 for ; Fri, 27 Jan 2023 19:41:51 +0000 (UTC) X-FDA: 80401599222.13.A58535F Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf21.hostedemail.com (Postfix) with ESMTP id 2B8B91C000B for ; Fri, 27 Jan 2023 19:41:44 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=B0QFA0zT; spf=pass (imf21.hostedemail.com: domain of 35SjUYwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=35SjUYwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848505; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fluAUEldqgo6Mr/4Su/gpwJVTd/f+QKHo7F0iCGeqOg=; b=XSFsMFNcfmHupgz7Ht3Qra/z4cyWvwOu6wQA+dx4iOeEjNj59Wa8mboP2RuyWEwieZy8iu l2TL8NSQbNiN0N9QsH7c1Imf57fEkg99E+lUCFbjnCDr+1jA0NUvlPkl/p9U7F8T7Fkejt sBA9pOaV/UGOV381oEmAGJchKqMX5ho= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=B0QFA0zT; spf=pass (imf21.hostedemail.com: domain of 35SjUYwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=35SjUYwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848505; a=rsa-sha256; cv=none; b=5pMKbVsE9IXKjCOQyfkgJzxcimqq5mph/sHgSniIsRfSJrY9L7rGI2vSc7bSFPJwMeke7o 1tsCDUtMq2b+g3HJOVN2566t+cKtbPyzk7r5KRFLVo8jyMpjB0P5z8/XowsQRv4lJg42Ir Rj5DTi0cGy9pZVktgS7DqBQ9+lz+gfM= Received: by mail-pf1-f202.google.com with SMTP id c5-20020aa78805000000b0058d983c708aso2831864pfo.22 for ; Fri, 27 Jan 2023 11:41:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fluAUEldqgo6Mr/4Su/gpwJVTd/f+QKHo7F0iCGeqOg=; b=B0QFA0zTTAgJFhNgrdDgKRfLtInL5VmAHE1KFwNYXYIlcWvaJJq1PKzYA8ckdy7Kyn xAwTf5YPOblK6+TJj1tGvdojweaIBaHFiTxRDHhkDnK6qDnyvgtYbq5iiXWxBxOCwUuh WaET5h8CLSuaMj6adU6fJWH6EpE5SqyIKnzFJ/pir1JZwDIco5M5jQ2a/9VWhgwD3iWL QgWDwglZ0wa8AC1rJh0bk86XOm5Sb2wt2duuTU/yqZcdsgUl6ZFLCYWIeCgAmHx4ObRd hHIbD6pxz2ke/VukXpDhoYuOo0GEGrcZQn8DqZwlpC0FiE7wMl7MPAjekSdb7LenTMW0 255w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fluAUEldqgo6Mr/4Su/gpwJVTd/f+QKHo7F0iCGeqOg=; b=I4vdcfttTSIEQV1VE8xumi232mL80QLjrugXZQMap7ibveo5eTm3qZ+bC4BiPMXzQW WlgdJyxD4k0o5iFGuEkqHe7+dHcRpoB1fKCNCSPpzeljJEkzkoGHPpQkxDs79JyOcSF1 R1NVnaOmKHjBwLBfZ0zjCFhgC0C3SZx9WDkamYBmLnWLJNKpau7mw48AxygcDFJlObo6 E6/LVeWDQKbUAHGKE1klSzZYiGkCV8185hOsP5k/GFBl9gyjwGpo5irxFF6qsPthpusZ zY9zMYfpqdTSyAWoGlV1hUNLxPrDEVK0bUwYhm6LZIH1jRYgQxbPcmLdbn8BSnXB+8FS /M7g== X-Gm-Message-State: AO0yUKWVLQDTjGJVn6E01BXtxljCsKKVmwntODNbxg7OiPzcTfgyuAdu Bi5kbpAqO0YQwm85tX0MNoxNVpRd1fo= X-Google-Smtp-Source: AK7set+gvzIFHouo+ELhD7kNJY3eRZqYYIzPpjJr9+paWQDRkxxsh7BJRXlaLIfiaGLvKexw3FDEYkSiOQU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a17:90b:a16:b0:225:eaa2:3f5d with SMTP id gg22-20020a17090b0a1600b00225eaa23f5dmr12984pjb.2.1674848485998; Fri, 27 Jan 2023 11:41:25 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:42 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-6-surenb@google.com> Subject: [PATCH v2 05/33] maple_tree: Fix write memory barrier of nodes once dead for RCU mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 2B8B91C000B X-Stat-Signature: ns3wgkt888wr9kbm1h5ii6xwmjt6fsrx X-Rspam-User: X-HE-Tag: 1674848504-672410 X-HE-Meta: U2FsdGVkX18RDQuCBI/nFsaR1Ygy0O6X/Ix7PRtdJnUS9l+lyCDlv2AmU6QY3xwsTFtwwBmoKKxuKvf5n1+jc6Wewy+ZTvN0c0DK1lQLDIlTJ142tV2A+rvYo53Dgh9BQsFK4aVgvfbfTxxaiQmyTOIhc2z2odJJzicOh9Y1rlCT1wACtx6GJsxPSvifBXh+nZKnWEakNZD4caJHLTix9/qKklUeh2YZWTNWNKGISJvd0L3Jz4AI8j1up1K5vMkD0/G/7m8BqyQyQFp5cdwSw4YkySEmDHo9Gr1sctC0LILzumkLEpimXSOtJO/xEnwgfqzqwIV+zXz6CGASJ+eg4yH7WgNd6rxCdy89g3yy9PH4rYyYeymBh7h+LIiBIaY03w8zE/Mla9REjbdU64NiknZ8+Nr+e8I+rs5Ibl5UTKm35U8siXSieC2pmCb3F7OsYAL6slxxv1nzTq0ejdNfKCUCr60ykg3t3gy/VtJE0byEKDX9gXesPyHBp0YSyrp4prpHwBNzCTY6g63tBVUehzQAbfJelRr1uSTuqKJuL7XR4X24z0SuTvji/aRbHKrveuTlxz2giyqaKoGwDeQrFwVRigkb/kW3pUK+G/rX/x+QKNQU072hxie6Y5IRT6oK22ox91x1F6O2dTv9pjhq01RUSorbyLQ06d4vtr7oPb1Ycr2B22TNGPhbUIpCIxh/3JL/ezAYd2EDdSHhHFaNBo7gU3svuEfBYVIAMEK90Kz5GTtVGoMBXzE15xk0zyOylelgWoKknzXJmMP2Pr33qLoDsl6/QMi8Xutsdk3pdpDRzeL/6xZ9PW0onnlsI2cJr8uMdp+VXYB4tsfsOSeIuROjU8jOK5qleTM0a4gYUv+Mehr5Eulyx/7DQ/OXrFDBiw2dsTXh4EsIZPxCWtA8ZMIQ+i5TfaNWki5JPFThpnuJYQxAj0Vj/wHAH8D3fE4e4qjPUo7KcOHz8++MXeu uaVm9GEY e3tslG1C428Fnn6K7P0B6Bwj9pDJCxOsAIoJfCGLkRmE5Pa/2eKhUUeip1eN8xCd3NgLXd0lvc98hn/jN5xX20ncD9hKdB4uL7pOLJ/98Wo0aA+N5jTyt4hqcBCbIDJpqc2LrvvBd4qJFLEM/kVQgtATdPWC99zcZ/mFfnam8etRrjqXmcaVj/+MBwbJzW7FjbsImUWkImPwSrsQSEl1lMLIEEip8yEFVwu734ZH7xIQMVi79PabNz9ploIr7G8HIQSPFeDlztpV9HRMj88i/P1DZ5ouS8jArwN4bz8k1GBCGcr4cgxoOUQGEfHRr8w8Kyars1aBNlYGM+2wLk8u6k0/xqnHzn7hvP0IdXopPUnviQotZ4whFd4NXVTcEomM1dOwglNbEwOt140z/dH+C5kjbSXmX+NeAaLqZtfJDy6dunJ9xMTOY1Qczrue98h4NVtPkxMSqGO6uBMEbqYCNx4LLrejWPRJGsn5AlIAfgCSLk+G2LGXng3msqvxiOPRsEuE0u9F6XgVJe3wZ1S50JIh8RC7SUVel6c22Tqk3SeyC628gmnku6fJrnns+bxqkQOqJ+ougDJInX1W/MeZRDJBrpW8OfX2ui7rhISokex4uZ9yA55nb+5P+XQLeUoPWLVqBLHtG2UlfflbPXXvxoRx0qEozoaM9nDjWuZxfwUTRGqwc9kAdYLNT4NY/W9/l9DiA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" During the development of the maple tree, the strategy of freeing multiple nodes changed and, in the process, the pivots were reused to store pointers to dead nodes. To ensure the readers see accurate pivots, the writers need to mark the nodes as dead and call smp_wmb() to ensure any readers can identify the node as dead before using the pivot values. There were two places where the old method of marking the node as dead without smp_wmb() were being used, which resulted in RCU readers seeing the wrong pivot value before seeing the node was dead. Fix this race condition by using mte_set_node_dead() which has the smp_wmb() call to ensure the race is closed. Add a WARN_ON() to the ma_free_rcu() call to ensure all nodes being freed are marked as dead to ensure there are no other call paths besides the two updated paths. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 7 +++++-- tools/testing/radix-tree/maple.c | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 75cce2d4d5da..49e399e8afaa 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -178,7 +178,7 @@ static void mt_free_rcu(struct rcu_head *head) */ static void ma_free_rcu(struct maple_node *node) { - node->parent = ma_parent_ptr(node); + WARN_ON(node->parent != ma_parent_ptr(node)); call_rcu(&node->rcu, mt_free_rcu); } @@ -1771,8 +1771,10 @@ static inline void mas_replace(struct ma_state *mas, bool advanced) rcu_assign_pointer(slots[offset], mas->node); } - if (!advanced) + if (!advanced) { + mte_set_node_dead(old_enode); mas_free(mas, old_enode); + } } /* @@ -4211,6 +4213,7 @@ static inline bool mas_wr_node_store(struct ma_wr_state *wr_mas) done: mas_leaf_set_meta(mas, newnode, dst_pivots, maple_leaf_64, new_end); if (in_rcu) { + mte_set_node_dead(mas->node); mas->node = mt_mk_node(newnode, wr_mas->type); mas_replace(mas, false); } else { diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c index 958ee9bdb316..4c89ff333f6f 100644 --- a/tools/testing/radix-tree/maple.c +++ b/tools/testing/radix-tree/maple.c @@ -108,6 +108,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mn->slot[1] != NULL); MT_BUG_ON(mt, mas_allocated(&mas) != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas.node = MAS_START; mas_nomem(&mas, GFP_KERNEL); @@ -160,6 +161,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i); MT_BUG_ON(mt, !mn); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -192,6 +194,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != i - 1); MT_BUG_ON(mt, !mn); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -210,6 +213,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != j - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -233,6 +237,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i - j); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1); } @@ -269,6 +274,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -294,6 +300,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas2); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas2) != 0); @@ -334,10 +341,12 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) { mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -375,6 +384,7 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); @@ -382,10 +392,13 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); } @@ -35369,6 +35382,7 @@ static noinline void check_prealloc(struct maple_tree *mt) MT_BUG_ON(mt, allocated != 1 + height * 3); mn = mas_pop_node(&mas); MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); mas_destroy(&mas); @@ -35386,6 +35400,7 @@ static noinline void check_prealloc(struct maple_tree *mt) mas_destroy(&mas); allocated = mas_allocated(&mas); MT_BUG_ON(mt, allocated != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); @@ -35756,6 +35771,7 @@ void farmer_tests(void) tree.ma_root = mt_mk_node(node, maple_leaf_64); mt_dump(&tree); + node->parent = ma_parent_ptr(node); ma_free_rcu(node); /* Check things that will make lockdep angry */ From patchwork Fri Jan 27 19:40:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0274C54EAA for ; Fri, 27 Jan 2023 19:41:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A7C736B0080; Fri, 27 Jan 2023 14:41:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 98D806B0085; Fri, 27 Jan 2023 14:41:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65B386B0082; Fri, 27 Jan 2023 14:41:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 439086B0080 for ; Fri, 27 Jan 2023 14:41:36 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1A6591410D3 for ; Fri, 27 Jan 2023 19:41:36 +0000 (UTC) X-FDA: 80401598592.21.B65142A Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf19.hostedemail.com (Postfix) with ESMTP id D554A1A0020 for ; Fri, 27 Jan 2023 19:41:29 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=h4z95aWq; spf=pass (imf19.hostedemail.com: domain of 36CjUYwYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=36CjUYwYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848489; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A97N+ZL4sr7Moa7VA3MaIfeszXjKUidgl5sbUl4VE3M=; b=o7ewt+RO3I1mORsM9bcDbLAqVhmA02Y1gYmJSdlaZj66manGIDPuq2Cw7bYi7w84Vq3BZl QePTBRERysG7GoXvc+ehuQZ3MxONNugu0HFgYLiPp8ESW3xMJco2CFfn37wOJ3p+akA8GV ScxJi4CxbNwbXqY5JepJu6MMSgAGNC4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=h4z95aWq; spf=pass (imf19.hostedemail.com: domain of 36CjUYwYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=36CjUYwYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848489; a=rsa-sha256; cv=none; b=YP0N98FAP/j1tgVRLS5vPYM0UMCENHorbooc+tGdufitieB9sR250kK1WQkvyM64viGYYF sfHf3P08xgYCpQa9NLyaSRP0MhAMA24l12txqSGkX+OW4PXs5gTj/ENAVSQFIijOxrp6Vj O4d1lYNp84/lMOc1jXMuI91qHkI4R6U= Received: by mail-yb1-f201.google.com with SMTP id b201-20020a2534d2000000b0080b87d910a2so6434807yba.21 for ; Fri, 27 Jan 2023 11:41:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=A97N+ZL4sr7Moa7VA3MaIfeszXjKUidgl5sbUl4VE3M=; b=h4z95aWqlTijLuw2+pa/O521iwctvYJ/JE9Gk94Om8JNdm1J4qRBJ1EfNZDOaurfUb h0xAr5ZA79m4jhsd5KmptV8L5YTVXkJEqIIo06fTcAgDWyf2Iw1unXEy8R6f3F+h7n0/ eKAno27phPvhqEsCndY+cAQm1TD4BuL7gONpQNAXYFn8Kf2q+ZhvCxmCxJsT4hK4blKF lDmG8J5RutFJ9mwfq4DqWgfTIN8+TeptwgRlAF+qB5k0K+pf2NR8dkIUNhEjxCEKvwc/ GAm3buayP+cfA/HX9e16HQfg7j+WzzAIFSLNgZD3Zm/44+VjSwN/Bplr8AdUjUNKklkq c5Vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=A97N+ZL4sr7Moa7VA3MaIfeszXjKUidgl5sbUl4VE3M=; b=b7CInATUXI8aUORtO//R0JTI7+2/Zg1/XN7fNg9CrBTILO17oAi61TxLgAhhGAzmL9 bhDE23wfftkMKIZz4sc6i7KxjWvhpupniW8MoEMbGWWvqkVLhfkmdqAPsAbPpp0qmMsS YGQ3fATWk5w3cep3nhutPeUbO7fgFN7tY3kvfpHclAYP51BFytvTqg6IcosOZte5aInD 3sjGBg1L+mpwXdq8tbO0T2DTZV0RLdjhUmFE0bh/PMWqJ+yBUTomv+lUBJ0PiQUlpwJF pNYnkLF5Ke3SyQH37GubXO3O37rQnoh2/aZI3ySOLAQGbo5R8VPYut13AufTzWoyykPX qo5w== X-Gm-Message-State: AO0yUKVtGRYRWKunQbmgat4+u58WjYOf/ED5QsHEh44uJrFr/DdwA5P6 kaHmnT69bjgM6hdpFCUeiCALB8W8t20= X-Google-Smtp-Source: AK7set8r1BYS7Wv/oAXgdJ6+dIiQhgZ/QiambSorQvdFTePlCpS2jC0PmTUUP3dVFKS1ORR7IfLO9gpuOYg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:9bd6:0:b0:509:5557:c194 with SMTP id s205-20020a819bd6000000b005095557c194mr892323ywg.449.1674848488954; Fri, 27 Jan 2023 11:41:28 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:43 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-7-surenb@google.com> Subject: [PATCH v2 06/33] maple_tree: Add smp_rmb() to dead node detection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D554A1A0020 X-Stat-Signature: 98c94x73e676cy4sbb7m4fph4enyyiqc X-HE-Tag: 1674848489-653444 X-HE-Meta: U2FsdGVkX1+slT72ma373G2z9ucDu2OCoDs6d5ZngN8x1YNU/utGr1m6Gr7zMaQUfzTppKCTJpe1HgDdWvCHTp1pzf+xciDTAKY6GzRJGax2OfHnsgaL8sMuZXhwwT4gpN6Rmba+jg9uPxQuXNvqvCyOmSYwmZwpDmjw2TBIHuQQOCjDDeYQfEkGdMoDYEB/LLtgzf1bG6y0E9pBMDFE5rauE5ZhpVmiqtwP/bHiSLpuVou8bP5NGWkTKxdH4gfp7B6VsoVts4qsu7o+PXVSpTE+UpA4FUmEYB+nzimlg2vEsVViIHJ5P9BItILSa4nqruLC2WUWfN0BodEEYxSQds0mRaEd2Zo2/Ndb1X3BJ7mjJr51pVFyEsVMfGdJ33G5Yn6vwTg/Rh6dkOv6pYO2kA41WgOAunYHBHKBWYN4kBhx8F54TzPl0MwvHIeSlgTwJZaS8VsgL6H9b342q6Pn9fx7Y2Wm67/Kl+ZhkIKYaxvgALAhG3cj5N27S9ulw9HxvvU+ftFA329UulB5cFiKOfLdQ9fMBvY7HEIBSbP4y4kAtmVQmWU8U3026+z6LpvmfAC0KdeOD8/EnmqQYIcW3o/z+XoATijmflMzrKQhZhiI1kXCXQoxTH3ZnyLHW7dMn+Y5HlFJMbGv5mNpOeMoiOoV0qz/QV4zFjMv8n6pmfk2scaGSdhNkH0w1x5cZfSbQufFsGYqcWIc+18SqNUXMQm1XyBttWn21uM4LNgrZ5FVXetbhfjBjtegyu9ooqDPKG4rlSacaybqXCV1dhvWW9sLqdEBUItxP/G7eu0j8n9DQkQ/gdD3StVD287q1D+I5S0eCYGkbdAVJzn6JO7CSvUxlS1eRG1cJxuWEpYRkq3Vtknbih56LArLmEgXZMPz4SEsTRR4g2MGUTT0Xv3d+JJe3oCYFDtqjW7QtA8pWE6UXW1PupQR8kl/1Xf9+1I2of/4ROWoTaWW/9UAnRJ iZyI7N6d hj3wU16JJ+Budjrba9X64/WPhc/qnnEqLmXXpJJtJnudMmNvYsRK0bGDg6ondjkTBjT4P1MpeyQ92n0g1pDrN7tQVGydzL5zIOfq8Ncd00O8bUIbWY0w8N61RS3ojk6MtUWgc8LhjDsSz12X6hFzg78zY64cbOIMExq1mWpuCk6QNCPCOnO8qRGL4Pvl1KDsQH9GuBLAu5AM7M3VNMzVZ5912WwHS3+M7F4rEyKeWfwpVj1TNKgNorJtuzhz5lAvPTl9uKnp1+VyRzNIJ82PtJP0bvBLiFVhbEqgpG3lLrSuUfVJnFXwbqdt/Fa1/ymrxixffyHhC5OAr/rgoZrinAaf47EGCNFmHznmHapAW65z8v+JAfF9CAhbbEKMWeXy2DtMQVAd6F5oDTfKCH9WTM3qg9ar4EHql5qR53Nk3+l9RsE7d0UhLEshFtwFKojYvaDl6q8faZIvpfdOEBSE0CAT5H4ps/Jv00vxvYVLZwFnH/aolFcGkM85m/Y/h5tWxRS+vrbG6S9jk3rRcqspGdk5C4uiU+34NUjYOlAXDBXX2aDUe6IUG8LagftjaGKLPbNa8YJ2CTAffmqa5LJ/7W8EJ+RIubtqsIpn4t7Ca8jmcrx0+o3pgIGyw3y3BbpwheX2mPSlB11Xpl3Z1qlGDI1DDQInIXhTEFTzMG5s2wdP68aigh4WJyMMVqHzH38kpexTo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Add an smp_rmb() before reading the parent pointer to ensure that anything read from the node prior to the parent pointer hasn't been reordered ahead of this check. The is necessary for RCU mode. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 49e399e8afaa..859303d2da90 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -532,9 +532,11 @@ static inline struct maple_node *mte_parent(const struct maple_enode *enode) */ static inline bool ma_dead_node(const struct maple_node *node) { - struct maple_node *parent = (void *)((unsigned long) - node->parent & ~MAPLE_NODE_MASK); + struct maple_node *parent; + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); + parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK); return (parent == node); } @@ -549,6 +551,8 @@ static inline bool mte_dead_node(const struct maple_enode *enode) struct maple_node *parent, *node; node = mte_to_node(enode); + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); parent = mte_parent(enode); return (parent == node); } From patchwork Fri Jan 27 19:40:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A4A6C61DA4 for ; Fri, 27 Jan 2023 19:41:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71C9E6B007E; Fri, 27 Jan 2023 14:41:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C1306B0083; Fri, 27 Jan 2023 14:41:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39B116B0081; Fri, 27 Jan 2023 14:41:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 23F156B007B for ; Fri, 27 Jan 2023 14:41:36 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D4BDB8070F for ; Fri, 27 Jan 2023 19:41:35 +0000 (UTC) X-FDA: 80401598550.24.64C12E6 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 2E38C180029 for ; Fri, 27 Jan 2023 19:41:31 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=n77zu0oR; spf=pass (imf24.hostedemail.com: domain of 36yjUYwYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=36yjUYwYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848492; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=oAnUozwryKPJZYsOHPjaJAC3y+r/EEsur/orpYvjbdi/ql0nGwc4ADB64AKId6QFjNxkVL BdBydALDJcruuUPKsVjflqowpqYyVSNkV3jBYG43HvlQxFn1ts9rThpMoGjGzYnDHHZjz5 twVgeNsZ7XR7qPmmOwCQEYMsNHOEEl4= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=n77zu0oR; spf=pass (imf24.hostedemail.com: domain of 36yjUYwYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=36yjUYwYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848492; a=rsa-sha256; cv=none; b=Mm++BnXOKL10HFIYsuTnlz0QFGudYmNLVa4JwB+9T+toP6b5Kb+xIk7F1taq04hNPt19Nb 5JpEr74QnNot47Geiu5gdDVUqIMKRavzUYJE2w59i4oK1rSBK9Nxt71QDA3qXUPhfQvgjE i226nbPuPkNJqzsCobRuy0SDF4Vtqs4= Received: by mail-yb1-f201.google.com with SMTP id z17-20020a256651000000b007907852ca4dso6388189ybm.16 for ; Fri, 27 Jan 2023 11:41:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=n77zu0oRsJL7OvjwFNjS7T1FAOtu4hdvxpK7kq3XthnUipBv5szcKq0gW6Eh2qWZ+N VqA54npBtuI1r6BsRB8yCAVHYO39SuVCqNfhqCm/Ya4OMopwGqTAxzjzqXFTaLEmWefn G7d9A2dlsE9NEs9J39+IVmn9jXa2H7iuxd7eUv1r4SdiVN/BsV9E4rX7Fx/7SJnqIKhp 5pljM+LwjnmUAESpDExSAXXjwcvSQjwdpBA+epV9jtEBYonrLt6ilkcmvKbIgLrFIqhL dWwHvu2PRt1rquBCjgEA2IpAdLCPilrM/O5s952uzrtLUUUIX7VUyMYpZs8xK3jyFgDu 2ztw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=gQamu5T5hAgvtJEbwXuY/BJXk5VU7Y5CJyzZYWDHI2EoWqUU8ic/5e7bWQOTSPXa89 fkRL68745zaYFR/Bw/fEf4BO6LAur6MqUeYitDam4nq28zV8iR2/eVy+c6Iz17vNyRJG wDlzF10x+0tCgKaWogv6Zc3fXahE6ALpM+kqnUpcA1bu4vlhnKz1TKekNThlIRvbMkUO 3H69aFXThIbZiPl0rZLw6pOWyOh0s7vVsKt1RCyou5sgJXZspBOLdaDskqLb4R5LB4Zy 4fBERAwt8NmNHVv33VkJ00tygjhgzMItrTDiDgjtFgWAlR7Bh8VBsLGhPvMxmjXhiylA Swcw== X-Gm-Message-State: AO0yUKU4H6K6PdpfcrzdWQVULXUuqGWoo45chr9YH3pWSroFfU3lAjyj OMhmItvQTA053Eme8f1e/md62qSSBKY= X-Google-Smtp-Source: AK7set9vKZUOyTcPurWe/bFPJyhfVHuV/nLiUj+ueaBvjKNu+svIA1blUVC8f6GWjxXLSfMUdNgMQcXmD6M= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:354:0:b0:80b:543d:2106 with SMTP id 81-20020a250354000000b0080b543d2106mr1234365ybd.10.1674848491227; Fri, 27 Jan 2023 11:41:31 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:44 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-8-surenb@google.com> Subject: [PATCH v2 07/33] mm: Enable maple tree RCU mode by default. From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 2E38C180029 X-Rspam-User: X-Stat-Signature: rejdj743ft4zjx1qbghjjhsfnitxmzy5 X-HE-Tag: 1674848491-612304 X-HE-Meta: U2FsdGVkX18OWEg5cCWDUyKZkaGGhscUDl+bcZngrpt8WCTz9Jh5ULShlYuHsHReaCy1gya3r7HetSowcQzeX3U0kMzRn5EVTVw1aEFv2pKNleGhxd5cEKsKwCm3hcJn9v9o6K9xN215t+dTXN/Gy7n3owWYDIvFRiud5UKTfXFFzOq+42MkQqDkQuKbVCTy6PkrbNpZCM/MxPzWzDg4e7qqzFNr9nYpRS0UWFAHdvx4lks7P3kz+RS5nCFMik5PT2Fxv76aMNrAxHmJxYTKSyUPIxu52xWS1Vrr59j5jDEmsCtnk6VSvkq48Gcjj2pAS9EmCQN4udDswnal89D0jlO9KhZh9kTCCyI/XrEnWVpzc9xoak9MC8RrqyU/GxfAhSBvaCABTQtwK1cFoUvtKeH6A2piuo5MqQuPJstq4ykWYB4C1+MossLXRPpkKu6Xjj2RjDEE4TiMECBGOgs4tiW+BFdT+HIB44YuLauyOOXym2wlVNUCdLFRQ3xGIhmU1fhIijyi3DfTB6FzczO6cFKFt6WBwHIU372RquoEAQ1PGL29aty6huKDMbDy7P2WtjeCxTIGeoWQBYy3W2A2i2/3n6aj7IfAIvjMPKHzOjDk4A0S81S+cydcwsN95xCOx3voNgo05wZtgzX5Jad/nwfZotLHlmg7udfTvoj1Kv/Ph/oGj3YPUY8xfeXPz4C6mVVmm4smPPPvTbU2Mn5+QpUcCHZcDPQgR5IWAeNZ7kWRZ5uPA/ATvPkKgqocIVBeFZuf1l6C6LSRlAOKUmQuf+vamZtB2RnMoyKXLxwfRbaVKQI8EgXRC+UK+TudI5t4FH9P7dIzFj80VpXTmeiMeH7DFfgbsi/M973dogBrzbV+YzZqSEo2WXkDjHH8DBZqkQzLnSHK7IfiSDRNFkp5r/76dzVG+kzMxF8xl4mxa8qTVYtdyWDcNdwcWrIIpQ6zDSRBWlDjpzi5aaUM8Yu JxTjzgyc RWTUJW3zzclt1V+3H7F6+565wETw97zUP/PfPO1/6H24zhznHiqPcUZ1/0ph8lUDd9QhJs1RSsrKYKIZTLJcXqwTkT08jUZhJlPb7FtgYF3B+pYJMxMWdhls3mcElV+zpDOU5Zl/i3P6IoTnbIjQv92EBz48Lae7oqQwkwg3HP0KdF1xJV79i+ODkX9C10MRgkM0pR1cF0gOizDhZyk9o2L62AcLzHky4Q6Nxw5zHigFLRqH1BJfGkhAra0/06FGEK8E+sXQNM49osAwrCHQuImEEuBP8GkEf5qqhN8D1Cgc+EMy9+Ku/9IZtaf0Zw60QvN0OQHvQECFE8rBmIrylve3qLuh8unDSIbgD6ZhMcpTTvQkOOn0u7X6GI3tGwd8m1Z4gfH8rYjgo5qQf0wA28mZreFQLxel1FFoPM63sExLzie/Gn3R0jCBQ4frc2h0cQoQghLQ98+T5Tfa7KHuUnGaiTAZQNA8BzUnRQfFxEtkug037LFWL4ExgBaKCx+siluWo6zioSlBgggQoPtXbdXtBRgaQzu5nL/TDQNYM74FsOTTOFoBp2T+Ag+AevTQbF9Zl/5OTq5noh6er9HgxzhrK7e9QpwyGd/YSeCHoW4cF6cM0fxoEkUPFp0yFhyXFrgxzTTowiGzWBPT6Nh/UYg1AEg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Use the maple tree in RCU mode for VMA tracking. This is necessary for the use of per-VMA locking. RCU mode is enabled by default but disabled when exiting an mm and for the new tree during a fork. Also enable RCU for the tree used in munmap operations to ensure the nodes remain valid for readers. Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 3 ++- kernel/fork.c | 3 +++ mm/mmap.c | 4 +++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index da983aedb741..8410c3052148 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -768,7 +768,8 @@ struct mm_struct { unsigned long cpu_bitmap[]; }; -#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN) +#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | \ + MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; /* Pointer magic because the dynamic array size confuses some compilers. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5e3029ea8e1e..5f23d5e03362 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -617,6 +617,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, if (retval) goto out; + mt_clear_in_rcu(vmi.mas.tree); for_each_vma(old_vmi, mpnt) { struct file *file; @@ -700,6 +701,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, retval = arch_dup_mmap(oldmm, mm); loop_out: vma_iter_free(&vmi); + if (!retval) + mt_set_in_rcu(vmi.mas.tree); out: mmap_write_unlock(mm); flush_tlb_mm(oldmm); diff --git a/mm/mmap.c b/mm/mmap.c index 9d0dadb55e9a..9efe13d36df7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2277,7 +2277,8 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, int count = 0; int error = -ENOMEM; MA_STATE(mas_detach, &mt_detach, 0, 0); - mt_init_flags(&mt_detach, MT_FLAGS_LOCK_EXTERN); + mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & + (MT_FLAGS_LOCK_MASK | MT_FLAGS_USE_RCU)); mt_set_external_lock(&mt_detach, &mm->mmap_lock); /* @@ -3042,6 +3043,7 @@ void exit_mmap(struct mm_struct *mm) */ set_bit(MMF_OOM_SKIP, &mm->flags); mmap_write_lock(mm); + mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); tlb_finish_mmu(&tlb); From patchwork Fri Jan 27 19:40:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 615E4C38142 for ; Fri, 27 Jan 2023 19:41:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 487DD6B0085; Fri, 27 Jan 2023 14:41:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 437306B0087; Fri, 27 Jan 2023 14:41:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 269E86B0089; Fri, 27 Jan 2023 14:41:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F1DEA6B0085 for ; Fri, 27 Jan 2023 14:41:45 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B96B21C6084 for ; Fri, 27 Jan 2023 19:41:45 +0000 (UTC) X-FDA: 80401598970.04.6B42CED Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 041BDA001B for ; Fri, 27 Jan 2023 19:41:43 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=m4Wtor8v; spf=pass (imf25.hostedemail.com: domain of 37SjUYwYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=37SjUYwYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848504; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=V+XTWC06Mf+LwjKwe8r9fW3Lz9l7d3ZzkZag4w+uXD9UdqvfWjC2I4svwRSQzy33pE2buA tXDNvXFbDVMOCRyUOqtmzI3iijJk08gprYTOOpp8/vKJLnuQ7407vJLv5jqGV53cTwf2Kn acPYkz7qG/EPcDcwQ6GR0+2qfGtFI2I= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=m4Wtor8v; spf=pass (imf25.hostedemail.com: domain of 37SjUYwYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=37SjUYwYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848504; a=rsa-sha256; cv=none; b=zevJuxbX48bkzYxZLEr0sbIApPmqsZYWVmCP+Ev2CfIOxMLWY1fEcWpc7EKw9a4lb30Jy6 /U88+cdOKd7MdVlqg0ArtC50WqfLa4ntuiZfEKSijuxqG1f7vE2+dqrg0uIUzYd8ZO8xV0 b5bAQex3kejTzVt/heU/+Pn8r3RZzLg= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-509ab88f98fso56707977b3.10 for ; Fri, 27 Jan 2023 11:41:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=m4Wtor8vj3J0ycRC6QGu22LySA+n2MGlbf/sH76T8N8Pnp/d8SuPXb/kR+Hyj37+ad 48NkH1zhtbcV3yGPaiQ5KPXnB06jsyj37BRmFZaU4H+LJ2YSwMP7BoYjL006YrtxIZWR VsSyGQhSW7cL19AuEXNwVKmULWycfkr7kU4POPAgtbT6hDzlZniUKpxVu1Hu98jAvRgz bcxq0t9OKK2J7/3O0lRl4dNTg073Sp+ojK3rnWcxuyGW0RvPh9anMnsNc+W5/rLzSMsJ Ho/St/2m0p0zsIWZF5QsNOVQeuQXkfsBP43+XpjCQzZI9euWanUB/WzPnytDjlkSAgXd IXOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=GurcPao/o9qZTJQ7LuM1DBKCuwb+itq/xYnCpo6iPqR3DpDcNDcYvbaOtROqV9Ew5y F/bAj+JAnYanxu0AyAk/NzUxEvEDUNtT0qbzOkvazDmrw2GTjxkkeYvdVKfQc+l+hXHF mXZoVWRYGtCbYIctuQmhpWP0izeFwz/BtWoto92KyMZ5Jq4i2DcrAmeAvBus+LeJxzhZ /6droS83aISR17tQ8dCSq2lLt6Cjq4t9XWAgVHBAYswDRwF+h+3eckjl0IpSa69DnYuk FvDZ3+81CHUS/G+sr2s1G/CjCkfb1j7jASqw3e5OEbI4yePNYCtVloASZS6HfuPsm7EN yiuA== X-Gm-Message-State: AFqh2kpEvKPFgfyPGHByZgvD2/aRPhzXvRWdoJFcLAh0iCZZRihQitQt 35yK0emoksJKZ3tbEgJOPN+QGDWBq+w= X-Google-Smtp-Source: AMrXdXt7ky1YKmCHrEvdSmppryHuxdZmy/ExGSeixqws3R/QJJ6NNyUiC3cubRusLU6Amdx/dq1XzZif1cI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:1d47:0:b0:3f4:6c62:18d1 with SMTP id d68-20020a811d47000000b003f46c6218d1mr3383963ywd.491.1674848493581; Fri, 27 Jan 2023 11:41:33 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:45 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-9-surenb@google.com> Subject: [PATCH v2 08/33] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 041BDA001B X-Stat-Signature: uwzar5fzjjwmqomfpfmdsd7331ezkg8u X-HE-Tag: 1674848503-654478 X-HE-Meta: U2FsdGVkX19NMWqlN8qAU+NFtgaApl1rJUyEsBxnol+RoxBOLV0oR1Q8TYaibQAqWbqQ0zaNP3k8L87D/BG04Jji6A3RAhcKc/1gcOnPcx9AYUU1q+tS0thEPoFIP8h5hs06cTop2dnozoi1GykdGYWmux17ecdjx4h0SSfx2Vz5MXLb2NH6aqio7+4ciuf1rgYkeVUdIPirHtBrsaQ/mJHvQZdQ89FdX/GyMQTyp5aKFyNN0IkdKORKEFsgyqzSFxjrCaI8BywNwNevt84481P99NqymacLbgwa42BmQ+F4mmyfsPX3t1N0bQrRIoN9P/A1QwBHjkX/pL09vz+hsXDUhNKeGhCgrgr8xg+vYCs/7YASMjFOPOFJPlmx8/TkUj1Z6OoWxg3HQgVn6+JDl231pY0bS4Bno22z0K1Y2MMAcHVB8RnRNbSO5tmg1dzyHFPbU6NXkRyBdAHZBM+NlCI5pGvM10VE9grVZ7OVS5QsbIP1plUbZw52RuuGmMtdv0ViccdsaUORAk5ELnBHZ/J3B7Orr2zWmYwvSek2dbsCkkpoj0DPmBJ89scnxo1Ak9v5AMWhHjS/keoGHJI/zaJZIbUnifQl6Ma0qdhOdubbH+B3LtbQ7diKtluVjbg7CkCvcW9VMINbACUUBiq3DaCTtoly1GKqvpxyU2JJgVv/IRiura86KhsviTTMp/e3g0CILwIGNM5Ix3FrFo0ycnsGlmtSCSLfT5n52K9+bYQDAygqixUeEyQWvH9CZ9eIZ4z22cjicarWDpLJQD7ipVxz0KpFlOQGTt/9Gaf1RFHQtzSoqrEZaQ1wv+k5czW9BuX88uzUDxWkYUDvg3hzUu2imwcCoQXitfMCufSAD6f00EdTlQoiKfBArmwrN/5XqxAECbgOpBpTYPdElrxSLnWRwgHmT/8+fGHElIPF6BKNCTCkz93DfqTmVnkweOrhYOeN4MrxDk2zUcaF1oP Hv7vo3UI J/HH3YZEqce+FNRURbgKEJTNdd1uenftmVKPLDtkqWbcorTqOiFbGnxmMvH7NIt/xnQciGJtFHQVFaQnxhg/LOBRhXYOtSBZC1FLIYFHXi6C+kFnN6TyTSVy0qBPjmfBOJkayJnSdqAAIUjDM2czxnEaeaEt3z+8d5pP6soAgjPsiJOa+BYtNY2sTTuBVmuYEgwdp/wX5C/JSWAI6bQ61GC2ole9hHJc/PnyIT2BXcAqjTqPIChSNS1ojykd9axfDYd1B/AhM6AbesJ7onUA1OzDikJXPeK/9I6fJBcTEDd2kTnm8amYTIsAgHX15FdRHLJ/7L9lfU2H2kGbUN5nPdsNT4LQQH9hMYImUOXQwZCZRPmqlgCQKJz7NVDUItFxShunpzOzjAAFB1LAtmnwH8q9/ggTYzgUMp99934o7919dlru+CNhgi3hWJ1VVXKom29jkoEeMWbU47Vhqsj2ZdxWuj/3rMu14iPwdOKNK2HMrw6/bhRZsD6cTk0wZs5gQmMUFxN6bT2habPVLE2tmZmml2IAPJzyNlHnkIUdcglM1xoNQFI9su2K9/3DUu1LCe52uXzAxyQwRG7T5nnOsyPDwQKq0z/2Dg07q8OPq+BXBHGBunVlHr+5LSqAV1m0jRwb7RtgepRtrckVkdqbVJyUSS2jJ2hPAk8mcjYPYOv3zn0M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001286, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index ca98b2072df5..2e4a7e61768a 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1211,6 +1211,18 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + def_bool y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Fri Jan 27 19:40:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32160C54EAA for ; Fri, 27 Jan 2023 19:41:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C77766B008A; Fri, 27 Jan 2023 14:41:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C271A6B008C; Fri, 27 Jan 2023 14:41:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC7D76B0092; Fri, 27 Jan 2023 14:41:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 99D946B008A for ; Fri, 27 Jan 2023 14:41:55 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 70EED16039C for ; Fri, 27 Jan 2023 19:41:55 +0000 (UTC) X-FDA: 80401599390.05.F00E792 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf17.hostedemail.com (Postfix) with ESMTP id BBFAA40021 for ; Fri, 27 Jan 2023 19:41:44 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oFyOjlhE; spf=pass (imf17.hostedemail.com: domain of 37yjUYwYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=37yjUYwYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848504; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Nu9/mdu38G6WtBjXb6Do7xpUyt6CdrDbmAUDhoyp47k=; b=6cVtnSH1CYtPAOqYOhW2Jx9BE3eqWl+9Gn30UfIBeMJmfIdakZGShc09IflDAzzyXe1xXA PxdA1KYnwhEHvDgahHHyN7ZIkUS3KRzYunNTeyik1kOMb6sknYqYp1i9dV1w0ro9WKsjHy fFixl4XxLsKKHuPSDgS3pE1bMHdLj8A= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oFyOjlhE; spf=pass (imf17.hostedemail.com: domain of 37yjUYwYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=37yjUYwYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848504; a=rsa-sha256; cv=none; b=PRo+1smCHlqdmuCSQcHIe3kx7Dst87zWZDnNJco/6S4aAGooAAQhdatgNRYXJ/5TE4tlXR J0tyMbMToYs2qD6U44T8mxfTPBWlEIQiPrpwn3ZOdHB+dwzKVFRE+LTEe3jvFyHsVvQ5U6 SnCa2uWEo/i5MghUVf989ZF/CGiCFWI= Received: by mail-yb1-f202.google.com with SMTP id t13-20020a056902018d00b0074747131938so6388726ybh.12 for ; Fri, 27 Jan 2023 11:41:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Nu9/mdu38G6WtBjXb6Do7xpUyt6CdrDbmAUDhoyp47k=; b=oFyOjlhEa6jPD1d+pKzu8xqMu9sShcBJM0zkcdXAO9bWZHfs5hDegJ8sEKWFr4AQNl iGPCIH7iC8aR5RiIjRoH9W2KUncZPWl0b5vsJXQEB2UE60mYiP5ldYuNlEW3dM96CgJp +2KdvoCyVN7G6APzAMKWAfXHTUhAA+JaJZhJgfnGwlr68Zr0GLZrYj2YnOQkKo96sWGj Ng0DOWnRANnP+XQns5edhVeWyYl+9EdfDgkHbYKOkVF++Ku/0IaWanDTI7HqCQuL/w5R edw8ZGqmSQJCCmVgs5lV30HhGrOZqx2K3qMJgNbHCqjkT1CdH2dJNnKk0QEKLZZ9jCPS y7/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Nu9/mdu38G6WtBjXb6Do7xpUyt6CdrDbmAUDhoyp47k=; b=POMaYp8TtRMrfQf+Io5hO35N9V1vIlYX0xZBh54kZlCwtzJEAhiTH/0r/5LpiRhUT3 eARGXFKe4eoSt77RxE01vXCoaBUshLGkxu3ev6UBStUMZcoMs7UsWDLvjRcCRyynwJND +PqogY5Iiz5vyIgYSyiiPae0ky8D1EjJce7bWTK65Z3URS1/uC3sQqCGwyrgZmCjEGC6 FIYQGOucH9QauSbvcwJVlRjaqzkT3VCCaEaH76FXJn4YeX1Bi3xxLhytSiiu1EgyC3a8 NHnxlMob5fehAF3GbOqBedogMX+I4BYvWhTdQaX5fWvgrAPJZoWHVpPPG/Rdf4bxtwX3 s+0w== X-Gm-Message-State: AFqh2kpXBXwzeDnvTomSBzieL/jv+ysE77wWfcy0uTL7b/v1vtd+QHlm PyiyD+GyQc6dwJginNITL4JYwT+g1UA= X-Google-Smtp-Source: AMrXdXtMMT5r56SGjlxKGvDbQirEmfUIu1ilIQcv1/3wi/4TSk/ascWeVtYTcuwkoFwuriGWfFUnQoKWHlg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:cdc7:0:b0:4df:f712:5d57 with SMTP id p190-20020a0dcdc7000000b004dff7125d57mr3818080ywd.303.1674848495850; Fri, 27 Jan 2023 11:41:35 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:46 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-10-surenb@google.com> Subject: [PATCH v2 09/33] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: BBFAA40021 X-Stat-Signature: apqwounq4u4dskbpxo1xxff3tb6kn74x X-HE-Tag: 1674848504-329247 X-HE-Meta: U2FsdGVkX1+jwKOs3D01LhwrEHRTNSA1uoauwViHj6MgV7UaUqvklofED2zVNuU31Pi5hkfir5bISJgYSNlNeq1dOjo2BPT0PixiaJxBxzJXRcuId3vjYaF88HczXVt+9VeV0ejU635PPflfHwrxLyjPAhlxZSXIn5vUfbgkIlb2gBh43OcQCHwtwwgkgQ0RBWfZaIXj5H8eQdyZ3HhXiEf7OUG609lJYXdo9m+99pXQ956fkYFBQq6F/ebH7l7rbsu/8hcwKrebCI2J/EZLhPUmUllteDMtTx5su34jCcjhx/bmErzGrvJtxZjtOeVR7CR4IfIQ4YLQ2IgMKFm6TI4FUGURz6gPTKV0VtYDCJOsHX3nA35GiQnrVZutmkTeUeFVY3cLDH1PhtEMTvXN0PuO3cKZXBtXYV6ik/u81kA5IDBoT1CH0l4qVoAwj6jQKyeUzkE9TeaZKVItap1cJ5sn3S3DAULNVaBjgmE4PMwhnxwfSlUkLkdDOGt9aa/qmgY9hADZKJa0OlTjcx2pqr4KpXnJjzyW53s3tyOOaqENJqrHDV5Tkccr2ZQpfxYkkTEeWzcPZPQF0ZMc/NBia733mADMdIHGwNYqv4YJK1Y106M4MM8bPIb0Hr1x2771OXXT67r/UQbsu8yOvyZ2yfU28BqKv0qM4ZKaRhmeA2gRm1VccQ6hJRdZPGqC2UtsHGIuggKklYVsvaDv/mXXVZcBZjTlDsz2JBAyUNnWuPF/4S+6duvFr4SzXUwvBwMdmvz7VESDnVtewgAxDbmpfZ3GSK4plzNcpWPRHko6C72ywWiL0n07xObErdeZVl3an6Rn9QcaLGfc29Jz0lBJ6QU/5k2cxpoNYAyxamqMVnEnd2r8Pv+2oUv4i+EkSe8QA/M9Dl426ZVdlYOvpoDWt5kF0lRrvxNt7Fn+nRJ00qO1nkaLBpk7W9fRrb6myCyjGfIXLF7xHDYb+J9SYBK cSN34llk 7zN1gVreEpD/InV/s5rEKwef9q00Xcznbu3wtAodOqc03LQfEyHGOUtuhVyXdEbLOcvdYdiuD859Vn3+RZC4Yzfr8YPYc5xsoGWQ1R1v54MznE02sEekes5GuVGRIJn8VPXO0Tt5NQw0rm8K6L3EgmL0lGJT/QjhUgbs7VAMLW2Y46I853XD7Oz6WQJJvMz1J4nDULDjJFcR2dnpPIOpVtJ61f2dsoVmekx+9ropvHrXT9fJbutDTO6KN3XITPrrAgo0QwaQ60TxXmnEB4MnrB3QiTwQ5JPNmdAve0IAM/U9pb3ESVhnBZfkxauX2BUFxSU3I/ZGOKUtRJshnR0UWJatihBP/hIqIcl2pUCVjcZBuXGsQfRaenjJF5uunHXd4gG9NszkE3gjKFf5y81FW7DOb3SPS2jWS4FUnmf5twGj7sTwNzpOKx2CTeHSCP//VulGAO1+kpYCfBbaC1jrmSZiV38lw393o18Jek65TL6U+/V4F99L63sBG7ui4JoZnwB0k5xuxL4nxe6uSAmQzdDVJ04TvEvHa1zCGo9X4SPHsJdqXWaI9wfC2CF3h4t0PiJTdyCAN/H8Zv3KlUFC3Wof03u680qnwjlMy4PJ22+Wq5F3z7Pdwse1X1D18/saiXKucHZN8L7nQ2w9xcMv3XDeCVzFTCB8MZwlbN4vlQBSUyClKj2Ua3hPLW58kRNxDpUXRGu19L4dcbK5iFh03OgJLonHFMw4o/SoGC7XTBBsuwwD0/fw2KmzMiDpj0TZM6bHcB2H3fcKEf/JUrxXwAm1LbUC4MxkScHxU8eJ3yKLvZ74OOJ3VFatvA8GZ4sB1vh8SSy5SxTgKVYuvOt3brwNTmg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++++++++++--- kernel/fork.c | 20 +++++++++++++++++++- 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8410c3052148..62e413f84011 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -480,9 +480,16 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start; + unsigned long vm_end; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct mm_struct *vm_mm; /* The address space we belong to. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5f23d5e03362..314d51eb91da 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -479,12 +479,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void vm_area_free(struct vm_area_struct *vma) +static void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } +#ifdef CONFIG_PER_VMA_LOCK +static void vm_area_free_rcu_cb(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + __vm_area_free(vma); +} +#endif + +void vm_area_free(struct vm_area_struct *vma) +{ +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); +#else + __vm_area_free(vma); +#endif +} + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { From patchwork Fri Jan 27 19:40:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1596BC38142 for ; Fri, 27 Jan 2023 19:48:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC4176B0095; Fri, 27 Jan 2023 14:48:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A4CAE6B00A3; Fri, 27 Jan 2023 14:48:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8EE8D6B00A6; Fri, 27 Jan 2023 14:48:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7B7446B0095 for ; Fri, 27 Jan 2023 14:48:20 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 53A9F14062F for ; Fri, 27 Jan 2023 19:48:20 +0000 (UTC) X-FDA: 80401615560.24.7036B0F Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf09.hostedemail.com (Postfix) with ESMTP id 967F914001D for ; Fri, 27 Jan 2023 19:48:13 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Q7jceUjh; spf=pass (imf09.hostedemail.com: domain of 38ijUYwYKCLkrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=38ijUYwYKCLkrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848893; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=cvx8cT08QWx4L93ob5VkOTehQLwcmglqC4MgKj3O/xnsofRZ65bL/iNJKK1zMO8Q4CRZEr q/dUFRGZWz0fV3RRbkDzg/D1gfKCcerXFXviox51Db3x0+vPvXGO+N1LBze3989pRzBHCR LlcJYHnfD8dBFvE4c96xOlHAIgRqGmc= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Q7jceUjh; spf=pass (imf09.hostedemail.com: domain of 38ijUYwYKCLkrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=38ijUYwYKCLkrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848893; a=rsa-sha256; cv=none; b=RBwNLCYOJ3E901yJ2E89ZTjb/T/2ul8YDJfQ+sBuKJQufa+6Duk1Ei8452c+f/sjHf/KOA 7Iuds5zbM6ks3YDrKsPnAIXKagKIR93aPj7VZlTsfgns+CU7J06GVsbx7ReHhV303MB5WP lURyjJyzsU+9xRoCq6+0bpPlH4INWW8= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5073cf66299so66386407b3.17 for ; Fri, 27 Jan 2023 11:48:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=Q7jceUjhtfhNXIUFz0/MzwmS5ZgtWEAK+cBK0blt69F8+KIFs/AwOKnAqPB+IEDKT/ FyLbx1gLVqD34Rlv7gRilV5FNygR87+qF/41EC627UqwDLTg6NJEFnAj/1s6LMME1gBy h9D9urWoN0e32uyucIiiCDtt59fZ91UJGpShxrAxprHccHpeDkPGS432b78Iuw1rAi+Y PGxS0P/Csnmusy4TpNpOjbr38NhV2uRvkkx/I7vAEJVPhb/fpP3E3BB7l9N5Jq8Ddy4K G0deOaLN7bYZJlZOXJL3xM94KhFiD/71EdmjsD4THVYmj7eB4g9ZYLPfdvfcp6yLyVR8 Oqkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=iRjhdiH9WuYnNQT73qEuJYWYrUXZ3m3LxYvgwxiCBM9MM1kpUemy5iu7VeRivAp0BP ZMQ1ME5f0VQ3BNaY0kQ2l2zzlk844K7eAnD3AP1BuNwbiAEdYiokgdiovNhbkTqdqFI3 QqjJFdo/9+Z4DIACRqOWYam6xL+Kgg4Ny4ZETIoDeeOPF1v6SOqUKB6V27xxb0tHnTjp yV77Ok203rmoNIr8sn+tvSJQQz53J+AlsBoasTqT1vsTv22iss/CvCWwK4ZFdFjMXA4s 6EK83PzHCXqMD91Tsn1+dG1dOf0oxVzV/LatbZaxhF0KbXm9NDB6fDXLypQiaBF5QAq2 uz8Q== X-Gm-Message-State: AFqh2koxHTX0/D9omRJie9HHpEZ1VW1DwRGn4DONxEBrclJkERuRcw0I VQj8cMpZoJQ+pfvisPuCwPdUYA4lKuQ= X-Google-Smtp-Source: AK7set9LIYXWd5UhR4gu2cf1OYz0XpBzDlrTVxbQWEhDgYvTk1OPOqMByF3iTmnXKTsrl+3RDXXXUtjCRT8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:f2c3:0:b0:50a:468d:48d8 with SMTP id b186-20020a0df2c3000000b0050a468d48d8mr731943ywf.104.1674848498315; Fri, 27 Jan 2023 11:41:38 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:47 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-11-surenb@google.com> Subject: [PATCH v2 10/33] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: jrqbd6wi8t9szthgkey4kyd9f35ow7zf X-Rspam-User: X-Rspamd-Queue-Id: 967F914001D X-Rspamd-Server: rspam06 X-HE-Tag: 1674848893-774427 X-HE-Meta: U2FsdGVkX19i83FXHsSQLUiC+K+NQMB19EggQKmRWocOPBLwakEQOp8NghqL4pAq8oeKK2eTCHbybedBTghjqdVdNXK6stZTlOV2lA2l9JBuioy82uPyPpKR/YKkO8S/zQLC7/SuG8GMHccNXyyssHn2VEbGbMgAZBTS7nuX3me8bZFkZnm0ATMl9Gq4zrVHSBvJiX6qGAdTYFe/pHi3a/2VtqKyB5OleDnpXSEyE4Cieuiooa0LFl87Oo0xqHbQD/355MQxsCtUWXOq7bTWjLcsxoLRjei6KLNtBoODW45AVua+YU82iKeVQ7bdnixZbmw1uGIa3n1l80i4i87+vMc9WT+WU49m0GlJw6T7VBanPX1tPPUJdwxPmY+ft3iPqVoDd6/pSzthPDokhGwUmOBTyV0aC3PFJz2ZgU7wAcWZPnTq3Q2e9bBoU1/Ft2hCMB8qvkDprdF5tTHgqjC5s5od9NxXLVNzCngt4ZxIeV7sWP8ehAiDXcHG8ESkucRWsVX98EJoe1yO7icD8ero2ho5zmppgxHO2q6707qegwCwNYNAHjV2XS7aTToQ+TDnacdKjNPCcrIsGLFpBizOoHuOhddU/FUFzoXvvi8AqMgIRJ/5kdr0oIeB04h626XPAPRKYg2ip/EwsP/grytI0ep6QPRgfoKZskKq+d4B59HAP2Q5EXvrhZoVgL5dWkjJc2H2CA1RD8MGnhL40NKH7SUHSDo3iDOPs2yNHFyeXYE+ckOq9vMoG2tMIP3m2Tz1xARQ/oofgSqN9prWWgsQI+EeXV95BKljfK0x7+uwrGASgGQWqxh5/q87d8bC6rc5/foHDFkC13yWkI6ypfaohckOSpK1r/7E4KOzoN50HYfu9ZzaL0k2Gr4Rcc2Y3L0MQnk/p+wrS0IzRbgc8qme0w5EGkSIEXTeQ7eD87qEh41i0yYs9K8OgcblG0cV0j/dXT8DYUrfRohXIP48TQ+ nd+0dGT1 MlOUpI6smIUbWT9wXdZemaiov6gzzPkb7qtSqWdxktFjDZNL56aQwMVbAVRdd/ZkaHWJKP5FgisIXwQk7RyPJ8wptippEtblm++GdX7ZxlL11uzdIwY2ryAz+Yk0flYH75hkWn22UT0N1AOzmng5SuYs2R40BpEF+IZ7Un4xpsndjGGouuSXxj4R1di44sORat7MR7bR/tGrXe87wwFbnGdD4nTvXo/D2CRHhS5KO3g0S7T9JIqw2CWAr8X/pq2qoK7001kv25XRZvbeWC8tzIWuWCHeHiQGRvFrUOdeoPNnsPicG0VUzF1OUWeZ06e8UgrmR9OSRv8pdIDIAo75SGZl0K5Tg84x1Gzu6KGFORrvUxkngvJHg9Q/+AZloGe2AY/hgkYJR34/mmmIKp7GFQLgav/m4uVOpm/ojCzjl374EoE5xN9FIPqqlmWtzsdER7Jzj3PZyEF5UQXy7I9WQh/BxYG8+6utigH8YgtIfrZ1ZsXwwdH1GgCyy5KYIpE5kXNbeTEr3cjXkePonyLQFTRcpsBt8griuY70SOUx8lpfiiGkTbNhdGzog5nJ3RR8QMgQkzXXoTWhVxgMpGhpr/W7v/NSxEGex1JuUAr3gPP9E0p1ceACN23vDxBQj7Zsy3Dd0bVTLeKGTqZq5CPllw0DgIDjdIg7j0XPH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Fri Jan 27 19:40:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA676C61DB3 for ; Fri, 27 Jan 2023 19:41:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD7546B0087; Fri, 27 Jan 2023 14:41:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D87106B0088; Fri, 27 Jan 2023 14:41:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFF2E6B0089; Fri, 27 Jan 2023 14:41:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A71126B0087 for ; Fri, 27 Jan 2023 14:41:46 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7A462161019 for ; Fri, 27 Jan 2023 19:41:46 +0000 (UTC) X-FDA: 80401599012.26.5DA76E4 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf04.hostedemail.com (Postfix) with ESMTP id BC60940026 for ; Fri, 27 Jan 2023 19:41:41 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=dBDLlmFt; spf=pass (imf04.hostedemail.com: domain of 39CjUYwYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=39CjUYwYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XXZYVjn5Y9alAXdawo6LSRZijnPMVBvcI/Ib+FHEz28=; b=jCIf5LueUxJCamx04445UzF0Nvb+Vt697oG0YIwR7aODH0ssNPCzxvN094Mp5v8NBCC7Yf R2GujtufJWhDDlx9ZfsYweA/JOJMNtIl156nE7phhRsyHgt8bvbfgSnWOc9gspUInBV9li MUW2fqO1VVIFRYX2Op8m4kqCaGfTYLM= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=dBDLlmFt; spf=pass (imf04.hostedemail.com: domain of 39CjUYwYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=39CjUYwYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848501; a=rsa-sha256; cv=none; b=k8nnBNKlXAFx8Paq3PFJaMH2CZ+bPEk/otqKEIQuj/WpYMpxpOwDAuEofRbYTxMMxF27On Q097oIs+Ro5DiUTixNG8kox+ngJqph98G0BhnHqpo8CPf74vXpXZK+YZEIpGjRvwLVo34P XAffp5XWtkMFiVn43Ev3uuEy2SGUmfs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-482d3bf0266so65368347b3.3 for ; Fri, 27 Jan 2023 11:41:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XXZYVjn5Y9alAXdawo6LSRZijnPMVBvcI/Ib+FHEz28=; b=dBDLlmFtxDGaSIxnsKi7TuOlfLmJDoa6COq6/jSvBgh64Qerq1YmsLWwhnAAxCyVf/ g582eX0Eo4U8zmrxOFGYnkJmijlE67J9D9xBXGrS1NtU3RqxGXmBh2f947WBs0NX/Zll emlcCuJwSkFMTmaE18K/oCpx+uIqpIqH/tqes3ZD00h73pebtBtpRPZksR0g19LqKl4v xmDEdLHNhpj3CDXqX8SiQjapJ6l/dJ1IUEAjdkc/2LeqJcIWhkIf+A2Qxlo3WsGI6Yzg srHRwUq16G0bEIrEimCIRBcTpWLvsmh9iUkFM5cCzPZIfcbsD+d957wvwGuV97WpJ3D1 vEJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XXZYVjn5Y9alAXdawo6LSRZijnPMVBvcI/Ib+FHEz28=; b=0BxYFji6JRhB7J9kIcp/6ZCaQryl8Dz1jeSeWNVx++RErjfZvyhLErbxrYztt9Bzpa 2M20ZZnSXlCIYLgWUZD3hmNAR007STEagQxxI1kdgT+N+uR19k1zHt/CgY0WCuHD0iwN AVvsKhr3F57C/SKZ9x7QWUKDsSESsBMo+rUtvhf6y2gC3bMfwiwOPUpl5TvI/q4mmtMM ftNIAJuSdbZ7DTvpArP+f8wKCsxFNyVlLeaoiWnbQhHsLWRd1SMVB7f5xPxMvTMBurV7 +/wR+ABOfs2gzWjrvxEawTCHGy7a8TVrpcEvlb2Ee4UV0/DE4yGtjN9/ONZgYVNYHRYv Z9sg== X-Gm-Message-State: AFqh2kpWZGVCS+ImRpb64FgR9GCJKVbatoBnSiczYoWc1NCrEdz6CCzH DbVvMiIMiA1ciECMzbfA+bOe/u/GvJQ= X-Google-Smtp-Source: AMrXdXuJ9Lmn9omVCHOpi5PBakoG2f5tpLAKUWWlsexTwKMDDrEOz6rknp2w/KZIAkkkuCmWH0rkraaKink= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a05:6902:10cb:b0:7b8:16c:e66d with SMTP id w11-20020a05690210cb00b007b8016ce66dmr5496306ybu.85.1674848500886; Fri, 27 Jan 2023 11:41:40 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:48 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-12-surenb@google.com> Subject: [PATCH v2 11/33] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: BC60940026 X-Rspam-User: X-Stat-Signature: t8hr5kxiygd6rn9kptx656marbyfjyox X-HE-Tag: 1674848501-63522 X-HE-Meta: U2FsdGVkX1/7i8iWWMRVla0s4lCEPSqAGJLnjSAHClrGc6DbIrHERirteSo7v257zPl3TEbnAMBZZ+xXc0l/uFT4DRnnjvwcW+bIHuv0Se86rDJasKLA+E4921B0Jhv3h9qDUnYhAqr49LnMnALQW/B6RsiAGVWk7x/CHTC4TcBsOwheMmmDfj5VMXYRQSx6EqQIR32ICeIWZirGPsiLHuu7eAzMyiEtJcJDqZNLJJCIouKXjqIyjoCCA2tAYEgs6QOv+MPcf1TKim0VrMKDMLCy+K8htj6mul+K0ux2HuCS3yO8xrvui0frAYMxICrX4gp7PjgBq3ug+flh3wYIpL2gfoHy/EUcH6odKtnlOEgPcXJMyYXZQoWS5veI/bqTbDsir8e3LUzguukxbUZyKldXR4PAuozqH0sV0GdwwUnCjuQR9q1VAiwKnqu3At5EVoSIAvsiOTMHzX+2y/SV8QBB/xEuK81NfxgyJCMS70mlEZEOqDX2TczrtDttHlbEk0MuPkaRj4skESDOlHReE0Eh78pKtwj9zlZTAaKeXP6RPfvdfoMgXAilPrCtJEtCTiGmIkWaUHkLhAFHV4I23675vGbTfGCwLGYITg46ViBjHZIT1t7xGOBSR3tVnAeWhCqHGPvBsIwrbQZzB+HyKLsOn8Entu8HjHz4xEO9MxGY0CHalwSGEQnuysfpYYWWh3WhIHZzKO2z+9PgCkkj22x0z5twdgVGZ3jJS0UR1WZQqyE76iemdjuyWuwNs03P1A7sMGoExDCaeYcoB1+yi6SWQkd+lQFDYpzxX40Kp/8eOkozwBR4oN9fcgH51IAGeVs6sbzdl8Mqbx2WGeojd9NC71aepvUFkY81FSnvquU2PeMV7OVGBCff/Du8ICjbQ/ArWcl6dNTMkwH6dQP/ZMJTS4OC5jorTpqfP2rriyxM2xDJezb+79JN3e9BiC1NqWqbNImZh0yWDkGbh1C 3YHfO3hG N/J2nblnNRIHbm0Yju6QsBsBi189xzLNp5flsy/G8nqy2o/+GKFRxcGMdPGUbv+fJzQlBFnqLPYsrBJ04ERSDl67bhBlhthvjGMeYLFSloAixFaGUrR0Qeq320Ni8ch+IOivj1857xN7P3GgB/sauVPk4bWMNDvHBK1m8pYMNB2jR9zrefY1+dObZbXaXS+sEjPzXlYZf3QW0U6xiEsXIoUopSyMEVuMS3malW9uaGIuiztBV+8EVInK3T9zC6s2K6yEzopYeYWh4eutDSH4SonHBplSZTkOI1qzWiW84Pwvh96SWyJBCZtgRAaeb4FPex55bBLZys4I/FKNsvrfow5UJedYqL/tNg+Car2b7rAWxXAwZVSXagTgQdf+VsKMdE/VBgFMbDddlgo7diQYFNAOXolVZze6cOD6ZpDZt6J4i+tVLnb6ovTgsmGxYpPveZ13TSce7rZhyhjba5R9a+WGWiy+7TFHizDopR3bQA6b+fAAiNTiBnCiJUKyVnIXzbn9ijiuRTJs96k6RjgYzVeM9nPCDjE0swD3atyk9pf3uAymMvN4gnPsuInMMUoC9kHFUOJavIXUdfj+Dsj0WWKg01bJNIjpDbN8WsofjfoBICyfpJtoESXhYXD/oUT50PwGVZTN7wrSYq8l7zjoGQmLvmg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce per-VMA locking. The lock implementation relies on a per-vma and per-mm sequence counters to note exclusive locking: - read lock - (implemented by vma_start_read) requires the vma (vm_lock_seq) and mm (mm_lock_seq) sequence counters to differ. If they match then there must be a vma exclusive lock held somewhere. - read unlock - (implemented by vma_end_read) is a trivial vma->lock unlock. - write lock - (vma_start_write) requires the mmap_lock to be held exclusively and the current mm counter is assigned to the vma counter. This will allow multiple vmas to be locked under a single mmap_lock write lock (e.g. during vma merging). The vma counter is modified under exclusive vma lock. - write unlock - (vma_end_write_all) is a batch release of all vma locks held. It doesn't pair with a specific vma_start_write! It is done before exclusive mmap_lock is released by incrementing mm sequence counter (mm_lock_seq). - write downgrade - if the mmap_lock is downgraded to the read lock, all vma write locks are released as well (effectivelly same as write unlock). Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 82 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 8 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 110 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd295c020e85..fee08e8fdce7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -617,6 +617,87 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +/* + * Try to read-lock a vma. The function is allowed to occasionally yield false + * locked result to avoid performance overhead, in which case we fall back to + * using mmap_lock. The function should never yield false unlocked result. + */ +static inline bool vma_start_read(struct vm_area_struct *vma) +{ + /* Check before locking. A race might cause false locked result. */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) + return false; + + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result. + * False unlocked result is impossible because we modify and check + * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * modification invalidates all existing locks. + */ + if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_end_read(struct vm_area_struct *vma) +{ + rcu_read_lock(); /* keeps vma alive till the end of up_read */ + up_read(&vma->lock); + rcu_read_unlock(); +} + +static inline void vma_start_write(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline bool vma_start_read(struct vm_area_struct *vma) + { return false; } +static inline void vma_end_read(struct vm_area_struct *vma) {} +static inline void vma_start_write(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -625,6 +706,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 62e413f84011..88619c6a29a3 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -508,6 +508,11 @@ struct vm_area_struct { vm_flags_t __private __vm_flags; }; +#ifdef CONFIG_PER_VMA_LOCK + int vm_lock_seq; + struct rw_semaphore lock; +#endif + /* * For areas with an address space and backing store, * linkage into the address_space->i_mmap interval tree. @@ -633,6 +638,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..aab8f1b28d26 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_end_write_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_end_write_all(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_end_write_all(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_end_write_all(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index 314d51eb91da..9141427a98b2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -474,6 +474,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ data_race(memcpy(new, orig, sizeof(*new))); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); dup_anon_vma_name(orig, new); } return new; @@ -1147,6 +1148,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + mm->mm_lock_seq = 0; +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index c9327abb771c..33269314e060 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Fri Jan 27 19:40:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B8EDC61DA4 for ; Fri, 27 Jan 2023 19:41:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 06CA46B0083; Fri, 27 Jan 2023 14:41:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0190A6B0088; Fri, 27 Jan 2023 14:41:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFB8E6B0087; Fri, 27 Jan 2023 14:41:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CED0D6B0083 for ; Fri, 27 Jan 2023 14:41:45 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AFD6C14072B for ; Fri, 27 Jan 2023 19:41:45 +0000 (UTC) X-FDA: 80401598970.02.51E116D Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf20.hostedemail.com (Postfix) with ESMTP id CD38B1C0024 for ; Fri, 27 Jan 2023 19:41:43 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=SD5yhaj9; spf=pass (imf20.hostedemail.com: domain of 39ijUYwYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=39ijUYwYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848503; a=rsa-sha256; cv=none; b=wMQ6EHNo8xz7VwBB37DhDNtW8ZUlhfTnTo20S8A0Ai1IflbC3MnfhXCGvTl5Tv4KH6MulQ R2I9Y2xD2vdtzMyQrGnONDwG5XjPf/RmN3XOFGf5/dnXHCJEGJtlPOT6832m/YZU9XyVJd b/z3QQp/Go1IoosQvFl2NkwGqcpbPtU= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=SD5yhaj9; spf=pass (imf20.hostedemail.com: domain of 39ijUYwYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=39ijUYwYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Hq20e7KS4f4C48PKmYYknVSNimtVVHYzyV8VwmVTJ0Y=; b=ruUxmDykOFDhGnREthCVH18E/FjDSVwU6SgiBvotVWSpXB60jaoHeC6qI2ov60dwVfLInJ /TmlAq92/fl30R+JPIasbK9BvTyHe7vvhCiRlIqE5ep/32pT8vC+2GItJ2tmaJQhK7Si+B 5rPtwGSqSP81b0HAuZnpUdDxvHU9Klc= Received: by mail-yb1-f202.google.com with SMTP id a4-20020a5b0004000000b006fdc6aaec4fso6430931ybp.20 for ; Fri, 27 Jan 2023 11:41:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Hq20e7KS4f4C48PKmYYknVSNimtVVHYzyV8VwmVTJ0Y=; b=SD5yhaj9WNDyBmH2zVXxIQSAPvkdOOdT/eZZRKlGP77z2J9nVnn6Obfbxv5sauUEV/ LCtaz+vX4qip57892cEocWjVTNmQS4/TdyxbxT8xLbIWgkt6lFG5ZxopwUuxBJEnTkjr vQID9RMKTIQGW2sRQmcGmPfXjoMh0D1eDPyuL/b6rqwKxYdkfEYgEgHjQL0Tw8ZlMbjX ZH2/uoF+k+VRzud7oIym//RQdtO+lmKAHdfjocpDVWhkn9piFYOBRbFkKQo8sM+osjEQ FxFQ0vmeBJ9JfrQeSA2WMn51BintgFmdPrgL1EZjLw25vLNDWAlkic6I+BYJ/3xqK9CW +GQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Hq20e7KS4f4C48PKmYYknVSNimtVVHYzyV8VwmVTJ0Y=; b=KWEY6ALH3pGcLI2Fer3k6xlbYtuXz+pZyEvdsU4KhJ7oj8cfEZ3iWM33xD9Rs983h8 ej5SB6sNgyxgB1ePyWcYOmgsnyFMtlsmhLWmrNHhbY4oTE9JMydy/5hBaB4woQFeK+1u cWZeiOc2ymvRGadSJ70NsT6TdyBglTKmqxdNXC49/7eQFsHBILEKinzE0zG/cVDGnpIw uxZfSVJ04zO2MhEJQVANbmP+fo8ZCjVi4b93mtNZGj+F0Do3rbN3YR47YFsVjCmqVa0d bJMon2c+O0Pse5VLj76A2vaL/+e4Lz6gMZgaAFXRqlEdGx8lnnjq7nC9R5xB2yYm6sts JwiA== X-Gm-Message-State: AO0yUKWKWY2XPati7i3HFNJUHrbFhI3kN4eabWulAEDEbmOz0sVfxEJU lzSbwqBJTQ9xVpgRZhbyJOKFQBOkoqo= X-Google-Smtp-Source: AK7set9CI7/3kZjfJSWqixfLkcUok8XHsx1YNSreUFU581ZlI+jbpFxEiZ86BfATPj+NjPN8bnYvtim468k= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:b283:0:b0:80b:c7cb:6d1f with SMTP id k3-20020a25b283000000b0080bc7cb6d1fmr1156521ybj.298.1674848502930; Fri, 27 Jan 2023 11:41:42 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:49 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-13-surenb@google.com> Subject: [PATCH v2 12/33] mm: mark VMA as being written when changing vm_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: CD38B1C0024 X-Rspamd-Server: rspam01 X-Stat-Signature: 66gqdopscec9kceqjygbtqpf1ki7kfzr X-HE-Tag: 1674848503-840653 X-HE-Meta: U2FsdGVkX19SJZ+SbjOnUTFf75OTRiQfQSr1DcmZ9nMVUKMUvPxS10QpPe+kWEybrKl17bPC/K8G+fjmsyV3frDOkl1/jnjTcG4KSEBI0D/zaStIqGzWLF8p2b4cnr3C+rPEhYONSgG+PvTRPE62rONTSLBkp4bezpRqcTJbxgX0JFn5w7ZAd95y5B8ZG68WBEypiMP/YGxUQ4J+UZnqJzH2OblW8xzqfCPgjSAakyWlZZau12kIrkZy7foCiokL9bRp6b/21nwtvwOKunVgHpeIyNLZCbTliPwuKULUeDph3n0mkuSqww4/4gP3RuTR05iprMbfCeOPkmK97hrlUAQ49BUD+x70+gNcoaj4LicnVUEa1mla9abdG/9d5OgudU2MaD8nnZPvGaN4lBE1S7ddPF9FMr0prlK6xyL8fmzqXGQWMZPPu0xj66LgvqIUp3sJSxn6suSey4l2scOkpOudaZtzcn8o/27rKm3xPHqe1noWhE6l0usmlamEJO/nLdsApA4OxnX/E4CpYxye9gWN6nVCIUCuyBde8gQKjy97h/bXJpHql8ym/i08YtvRrfuRdfkfbnTcZCLFvbknF5GTX6kuvXSlxVqURR35nQuzzqcWu8t/CPZOQMacX6UBlF363iV0DGX3Ws3dRHlPVQjMv3vXKTIKRQKqzKfTXCGHdLyvZHI/hKzNc6KVD0vuGKfecUmA+6Zem/v9U1vrCy4mKlgl/nOAHB6u6jQ+Dr63ebUbN6uhit8ioZyfeL36UZ9ExPPKlW+OuQkWmXhVkdMwHXUsK/pONpqbpD+FjLe0J3s2fKGCrWqA19xBPfY/aRAnV9uRFqRQXQtzPShyIfXSAB5u/FqRG9U2qqZxh4as906mviIQGS4qCvNR+PxqDcrxqFHDkNT/aj1pnNbMKI7wB5KmT+RNsTBW/Rcu1F0fr1jGHyQIULbsYqSIYsKNSaWuELiiQccdWjz6o+q 9lwbABfu Pu91Sf/GHile37drDACz4YCNBmuRDdW2RPyyw0QoMD4BYHINQTdn6CL78llKvrTebxWcXk5KePwhQYwmYsvn7EsJ1d9beKiarbuMnwPSPNBLmm37qNtu0mmiMItJZPyRwYu56VRH66TQqO6ElyKD0/vgwR1IjwFBAuPVQ1GrcDeRcVy/9ZcbWuCSeANFr91vSsXqXzSYWT0cNi6vxuNQFg3UoVwfFrIL1j1C6okDr1ItRfHvFRNrrJ6KGYPe4UC0cpnRu72AYQ93brTzW2yPqGpXX/FRRlZtJrr9vP0JJ1r1Eh48g+DOrJOTCikv42gDIEaBpriXsjddfXeCV7VmdRhzkGDTEwdLSOcwyNh74TMO4MiiXLkJ8TlX6KTMNdL1AoPhUhz8yrt9FhqDgveHoeS5QW0yNnL7ImGnyEytRv7TbL1fuisI2uCmlcjYNiqVQQtS57lMip+931Ec94MU94peKNtq36RoI5Z2llyT9BklAXTkEvrFXsGZwrMQqbl18qsAj2dA6GKAyFYo1Y953hZUQiNZd0RHykJLmnP4ZIDSoBzhvaasKRZY7dSMMgH/6EEghQ2fAR08qgjlUFe/CwztlPS/EfLGuh+YqGTOSfbgr+nrvPpM1v+FtrQvnZuOO6uhmgdzION1cSb2MFQKli9aqCccORBIUzq5dYUn/NEV50Q4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000121, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Updates to vm_flags have to be done with VMA marked as being written for preventing concurrent page faults or other modifications. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fee08e8fdce7..66dca140695e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -720,21 +720,21 @@ static inline void vm_flags_init(struct vm_area_struct *vma, static inline void vm_flags_reset(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); vm_flags_init(vma, flags); } static inline void vm_flags_set(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) |= flags; } static inline void vm_flags_clear(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) &= ~flags; } @@ -755,7 +755,7 @@ static inline void __vm_flags_mod(struct vm_area_struct *vma, static inline void vm_flags_mod(struct vm_area_struct *vma, vm_flags_t set, vm_flags_t clear) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); __vm_flags_mod(vma, set, clear); } From patchwork Fri Jan 27 19:40:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E92C8C38142 for ; Fri, 27 Jan 2023 19:50:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 829C86B0072; Fri, 27 Jan 2023 14:50:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D9FE6B0095; Fri, 27 Jan 2023 14:50:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67BA66B00A3; Fri, 27 Jan 2023 14:50:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 57CCB6B0072 for ; Fri, 27 Jan 2023 14:50:02 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2F7DB80754 for ; Fri, 27 Jan 2023 19:50:02 +0000 (UTC) X-FDA: 80401619844.20.64C5553 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 68BF2100027 for ; Fri, 27 Jan 2023 19:49:52 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=nhjpt9Vb; spf=pass (imf14.hostedemail.com: domain of 3-SjUYwYKCMAy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-SjUYwYKCMAy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848992; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=icGEQDhVNg8AQkGt0Dau1cb43VWZWeb/kUxMjHSAwGA=; b=qOjhPsVfxzKtIPm/YUBVoeLvOXSU04bEWofQSdt47CcT/ZSM97+rHBSqNJOZM215iDkmCs 6KCJa6qvbtL69ICXYUmTW6vzlA1xbTw12r5BkvnOEvSCb7+9LIJ6xdF/D8EtczLI/2Ffjn IIJxv04LiGMMgzG5g15nmOfhm/MlLdM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=nhjpt9Vb; spf=pass (imf14.hostedemail.com: domain of 3-SjUYwYKCMAy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-SjUYwYKCMAy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848992; a=rsa-sha256; cv=none; b=SW3iBIy8feHYDH3nhXN9cNlxrNzx54dehbVgdw/Mom/b/u4msTfqU/Ly7tnYjh46f8TFEK sEIjaqi9KvMumTF6iIyF1dk61D2wQ6/cWnI4cCvwmC+nDzVlMJOh0OlX79fva0ap1SIuQY F0HQdpEmg9UIIWSlTrscMGYWOFr6jJk= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-482d3bf0266so65539887b3.3 for ; Fri, 27 Jan 2023 11:49:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=icGEQDhVNg8AQkGt0Dau1cb43VWZWeb/kUxMjHSAwGA=; b=nhjpt9VbHDPsHYg33OlJfuq5Iz35VmBZ3kiMxPcXRXZZYdOboFlR0spKtxfGWFFcLn eBkIzKgB8QHaANcEzDHfkdLN1fRlxg2tqUBsMTv/7fNevBLbu/43ueowkRgVsLx1vLRo uHdRFKLtGeWmcojM55WkW4Vbk+IWqqj37nvYWjKvQOLDRhcdlXynMiveqJDQteuGpWZz ETc6Sup2efRPDAlMWE74qe3dlQIJeV8JWIsag7a7AlDm28r/breAdd9LIL/XkRDxD2X3 yud8bBPMeadGbrZ7ftbCoegvJj4wUFszXo3BfVZSLdIr/XgKY9tYqvkkj2bzILo8Fvtn ItZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=icGEQDhVNg8AQkGt0Dau1cb43VWZWeb/kUxMjHSAwGA=; b=bt+913KojV/87BniCo5+RJbyIGZXh8gUrWB5vCzzhF3ZrQB2MxCH9kSfE6O7Ij1a9p Ouep7jpzGTPYPsIN/2yW1+33ejS/2w0wrGYrokEHD9eXn464aCFKvHqdg+19zzT+YfwU aDuVtg6a5GotWLjQ54mdskPqQtEWTrrMUKgf1zVCT2aMUfxG1c4qSAJtH3ZPex8eD/Ua wiWjO7cwfU65T6PyXe9GLJp78avnt+fQ+fiN3mbBz+uRxrqz18qLvn6SYJWRu3QMgFkH 9WJPwpU6EMmKhgd7GysF3+z2jpy+VYyp6AEF/Huf8p9weAFY8nnVu6/iDsYhAjRm9Xhm xfqA== X-Gm-Message-State: AO0yUKVALNKxs9nJrB63oR+6SL7N7+y3mQNRbQKNSDix/Liz1XxcbQAx BAC9rqDG2AAGdZbaqWbK6h9GepxsDH0= X-Google-Smtp-Source: AK7set+R/81FxpFmvdAm43AdDVGV+lMO8EmoQcyFeqmQ4bL2GAdYwzTWfE7122Mwrx91JEcHzv5APn9iCmY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:48cb:0:b0:506:4c8b:231e with SMTP id v194-20020a8148cb000000b005064c8b231emr1848392ywa.80.1674848505193; Fri, 27 Jan 2023 11:41:45 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:50 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-14-surenb@google.com> Subject: [PATCH v2 13/33] mm/mmap: move VMA locking before vma_adjust_trans_huge call From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 68BF2100027 X-Stat-Signature: q7rqfhqo89yt7uqxje3teu7urq4opcge X-HE-Tag: 1674848992-624314 X-HE-Meta: U2FsdGVkX1/IvlHXNQtv9WJJXWexYC4KbqCCVLuSdkOwcD/bzoXPgfJD1gibTkT1/IghoxBXe1oZyVSAkYY4stV3gJ8V0/YXqdzvpFUHTO0v7zu6iYFph4gCnskeELe1IUaWJj94qQXqw04tGECjAPwuFRWRgLtw5xXAXt4SsPmmXGGDdU47+tkCF3KPD0Pjahxx4KTsbwdmQ8TT0OM5nUwvby3rNPfS4Yz8jOrMcRvJ4+r5wz+XHIBn3Qzn9qg+QyLBnjWVZxECudkjlqQZrTjt4VrR6ZgFKH7XtFWkgcVsouiqfdtRIiz3rIWEtiLoS1LBlIYZ2z+691KH1sV9P04Icwqr1bmUgi3lB8Cg2BX+BI5aHkx4lwFej6uQ1UYHPycKlJpCSRsXXlWbWnDVFd4oc2OAU/cmDnb2yvpundT7SPb4H8AVHfQhLtDj48NrXoUxZf/XxRvci2k6ufX7eM7CO/0sTdhdPniJYGubkmbl858d4XxWTdcuSzA4ap+k9GruE/8yk2AuiNrNzhQXHIIfeUe2DWfPAPiG7vKHsNKNk8zcvDtfw2RQtVxUfYVPEn+tTNfmJZ08mhO7j63HpqlE51+4/T0xD9wwDhXZWvocvtmoP93PAgc9ooqro2BTnZOhdt9YYdN/b9J302/clSBlRNI/e6CWIGrcSKF4a3ws353nluNLYac9+QmhG0k7vvesaD+KI+3bpelnLZkyU2nIu2jvOInmVkMy9oAc/3L31u5ayqKzyM/Lt9xDnP4AneT4KE6dslKFWvD1qiaEzRg4+XEPU4zokfC/ozvMr2CLjeBGJhyL7rVGow4gUkmgIi435EnUj3qr48yvab9664Bo/wrex3pSPVeciqKBUqqQHjh/mdM6WKFi3BWwfj09aNlCAEtUX/5TN4K97Q+Ibezdvb+gl6k+XX9+RyX2Kw/dI/lJjdCGMSMZKPA2EfLIkM+LqIE9jhD+djRScZ1 IM38Rt9o ZA8d8u+0ZZT/FRhTjQWBz9Bux6wEDJQ5DZA7HwgZ55jtKPrJGkIZ9hW2cPxTD7LawyjNnsdUqQ8IcuPohShZecco0MPPD6KoNLB66Zdc+E1uSBBkr2oKmDtLl47t34zgw0CKcQAVEDtIgqQMh3uWsPZrHLJB9WN4KSnNz0Mm8RTGIm9ks66K0AXwItvmIwuBqL3G93xCxtRdq73qxR2Y8m8eAWkG7dCwT8RxBPVfNYGIWi8km55qzpxt+TCtsNHoB0B9Of2ZHfdGSnt26eHm+nH9phTuP14fR/kTn+bm1MzchQSjZkYS5AocUPxtWSqNWWGNztc9UrjjtZscUDzt2Gdpta4+FHV4j78acZKW4kW7lnlloowY8bzvICIvUxKq9g29lZXOtsbD8+39KeFpM84pwwO9frkvE40BoK1UEhikLgvYQUudbxXFa4IzaFP+wJKtip8sT1UP6Xwe0DB5aNLzNA0vt4GfCm6zJ2G7rWixX8cC70DsLyRJvoil9iZQ3rcUs9Q5Z1nKniFwzxgsj8HzsbomOibXnaGfQqjeBgxXgE3Ur5ePSJ0UEGXe+Gt08HiVn0RD9JuPyVXM/HOw5rqhAbhcrVQNO/QPh9qY+NPA5W/T1SN0QME4UQAm5xjhjjTym1sNv8ZGrW6merti7dwz3WTGshpHdZIssBeh/5+lIhN0gnSF4ZLIZtRHINg07OHg35TbdJiD4R8kw/EU3GI7Oa8cjnaEq1WCEdEF/Kw3JCL+2U7+uxtPPYA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma_adjust_trans_huge() modifies the VMA and such modifications should be done after VMA is marked as being written. Therefore move VMA flag modifications before vma_adjust_trans_huge() so that VMA is marked before all these modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index 9efe13d36df7..7467d691e357 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2910,11 +2910,12 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto unacct_fail; + /* Set flags first to implicitly lock the VMA before updates */ + vm_flags_set(vma, VM_SOFTDIRTY); vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0); init_vma_prep(&vp, vma); vma_prepare(&vp); vma->vm_end = addr + len; - vm_flags_set(vma, VM_SOFTDIRTY); vma_iter_store(vmi, vma); vma_complete(&vp, vmi, mm); From patchwork Fri Jan 27 19:40:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119253 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61704C54EAA for ; Fri, 27 Jan 2023 19:41:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D89AE6B0088; Fri, 27 Jan 2023 14:41:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C9C6C6B0089; Fri, 27 Jan 2023 14:41:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B17056B008A; Fri, 27 Jan 2023 14:41:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9A52D6B0088 for ; Fri, 27 Jan 2023 14:41:50 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7509B1A021F for ; Fri, 27 Jan 2023 19:41:50 +0000 (UTC) X-FDA: 80401599180.19.6DC8DCE Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id B85EA100029 for ; Fri, 27 Jan 2023 19:41:48 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=MwDUDcWN; spf=pass (imf14.hostedemail.com: domain of 3-yjUYwYKCMI02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-yjUYwYKCMI02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848508; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7Tx+qt9ZwCfzeziH+AxLsfIEHPwRgvIaxjFEfdwiAVs=; b=ABcX3uOPhd7pkYQMugCdK5SPrS54ReZ2sNtJv+mgjEJsNzFP+2Nm0OK8iJjoT3tXNanxQz NDso+Fr7dRKZ4TMppCLOen79pPvXqPioKKqxwc69IRgFepJNYpPi9P9/0nMc6OsqPmsBAD Yp925pRfqyF0I0Xnn2bPJwj8GUFwWGE= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=MwDUDcWN; spf=pass (imf14.hostedemail.com: domain of 3-yjUYwYKCMI02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-yjUYwYKCMI02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848508; a=rsa-sha256; cv=none; b=X8jrQQ7jNYZ61rnO2N2ULew+zquB9KrPytzs7aJfLnuknMUNfUoKwXRIAwSuM0+MiFALzj 0u+aE4YgTUtZLes+h1Idqpg8bcrwhbavo8PhN9CodnFBycx4aqsgLzH0VRZVdVaY/PorDS 5f9VaWN7hREN+QRa/MuPLIKpA7WPaCo= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-434eb7c6fa5so65874077b3.14 for ; Fri, 27 Jan 2023 11:41:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7Tx+qt9ZwCfzeziH+AxLsfIEHPwRgvIaxjFEfdwiAVs=; b=MwDUDcWNdtFYfmhAuC43tEYj+bx0/7zv78dJ0hpiWDqIAfrnPySCh31pwGtES+llPo g9aJ1BaJcPmj6QIFKuJtNB4+OKnKX4w74YBlWwFtrSbaxVsErI4QpqtlUvrQKvku3x0Y fv8bSpOOjUv/1+/JuPfz5M6ciXa9EK4OZHTaoyyN5QHozfr1J0btP+Cb6s+q8s6DTvgx Eb/YBgAB71iKrit1/2Z+f0K+0VaK6nNNkqEz24OPAMCTiIMwYOn1N+e1hh+02mLm2qhR X0SB7IZ09+W7stQU0DgYwMtwYdWCgN82KCjbLOZQQ4WgzpAt4BTc4SEqK+AY1F70NhR7 prsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7Tx+qt9ZwCfzeziH+AxLsfIEHPwRgvIaxjFEfdwiAVs=; b=XJptCtZ/GiHputVbhI18bNOqCv1it51G3IspbdUmEt7MQGlwIgbFJ7VGsbTKNeB0rJ /LHaKFnUNwJeVtjNLlRzrfcA3v3j7Jf6+CDJU/w3Rcbd/HSJvLPrTG/5zJXUQiKTPGcE WZad9jI7leG4dNW5/9Pt8n679uCVmtBa4TRFvHImxR6J/vhldulEGstDyi+HQOsKsIA4 iipYTG3sWK7psTXoFIc8a7RHB9kW1jrB+NU8IFhvNu1NXJ5aiwh0JAeXzYzR4PMB457t B5CdA2fPMvOU3Yk/CTwTOHM1zgfj14XMK8rSjfValX/4GMDKnim+ro7P5SD1PQHDeNzL 0ZZQ== X-Gm-Message-State: AO0yUKVVItzfH1u08Ydl4bTx1r1ygUaTVaHSSHx4Hwsm/mWkU7SZlsxz 7BjBINUYlQifGe6leEHq7DO/XGRJDxI= X-Google-Smtp-Source: AK7set+eTaGDRvBxQnOso6u8cp6LCvB6g6dz+di36ExTAmlNkOKvBeltyp9kLCTqVY9tlqnocHmCi479DpA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:6008:0:b0:506:833a:c32e with SMTP id u8-20020a816008000000b00506833ac32emr1175163ywb.50.1674848507730; Fri, 27 Jan 2023 11:41:47 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:51 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-15-surenb@google.com> Subject: [PATCH v2 14/33] mm/khugepaged: write-lock VMA while collapsing a huge page From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: B85EA100029 X-Stat-Signature: 7hce5qwzi3ytxmibudykhis5frdhhzi7 X-Rspam-User: X-HE-Tag: 1674848508-890052 X-HE-Meta: U2FsdGVkX1/3lFnh8pvTvbhNjtn9ch2JLRVko4SxG3JGFJVfpmBFUo2teM+9HupVBdo51pygTMUn0X2RYGefVQLq617yxmR4vECZ8aGmvjgxKpMmbn0iIQ3y9YAf1dNvIv65r8hjQQHVjKjBDq1tB3jydDHQTRrXaBbhap8AXRR2I9UxASLn5Yns6GoBX/C274Yd1uWcX9Hylaydv8Plwu3NmR//Sy0ydKn7pc0nJBuUoQCg1J9SCwomwnlw7e58QJPAT5Z3Io3yWRT0zH1IdpQ1Xlsh8m38X/eGLhZslEulauNWmGyX7I4y9d82wzN0dTE2RQv/2KSH+/ardnSyRITtc9+O0pMJ37D84F86Omj2a9WoXRIg7fDdrQMCKryCMd0KUAA1w4sO/fk3/o/p3DcdjBpmzRkfqv3WbuZ7gMi0UXlTqJKGXYKKLaHZkZ601FgCa06YFR0IsqEWsx2qcdv8N1XHf7ZdXODpQhMHyE9xHRNsOZm3CiiLWQBGTrJTzUWa9k7H0kYQFQ85AIv7FnuDFem2ylnEvdJ0o0ft0quufc1VDE81iuhC/6IX/VkQ6lI1qXQmKXaEc5U3jtAUAyG11KLS4XupP6bmufow2Awu0MqlfsbWU57xjUSKYg5a7yQEO95n+x4eYTJEnrNVtsimL98qbhENofDiTkVM7G7qcubCOY5U4kCQ06YIIJ/lDCBn4JL2kXQUtmio7wXmo6YjGv2ImFUvDI+5z2znv+0SNYi1Bq2RxVYPm6vMf/j5k64sGCnzABBHAPx97jAQ0upmIvIYtaAM1/U3LqBXoUShJfEpkW0WVZQ7QvCByxq2yTiZAZFvhaODJ1FDyoZPIv89sHYVGF/4zYIBuPypt/aWEPvMX11dEdwihjBEvSxQE22t7alz+ks0VyCG+dhlTytYj83yI61KfAxIkE72F8jyikczRhlbDumMZHnDFMjMesOVXjUEJWo97hjhjDo eBTGB2z7 yVpkAcPqWJhC1q/7T68+Z87Wtp2OjgQPMXZVtBPktqh48nOet7tiy1hHYUV8ftp8TE6xF3v7HB8l5GxUX06F2p8/YuUqbITFRhO6kKzowajw4Lt4qAbUSwLstSnebOi+e3AmLImsP65KThglXuDMGx7+6xTTkGzE3vxKXXFs5i7QWqq9se5PjssP3hnKgXf24hRE0ygBtsK0mwWkLaUf1Pk+4HLG6TFGeEXyBkVrw9IKON5+Kmsl+yZZBaLcWEFmlhRc7yiQSojzT3jOZPVqQwwHjcXpu4KU02F8gt53ZBE5R2hrDGm1B6ebUL0aB6KFnQULTe6j6/uUIWdEaAmvam+dOkS6E5+2Kl1tCCjo9YsDH7XpUsFp272IByxzwtx5A3zp8w/BWOjxuM8wc73Wcq1EaObBT3olbS462LvgfCkFUSZwYBFTP44eAno6+AdpyWSAXPE0tJ34VeZ4ndeAYNyvX66pB61cnqC+FSxOlhzX7+Fxe+s3UVQeTH00T9TQiJZ4lwhPkLMqp+MczJ5WH6M1ekWSOfaJUXo1BnSmpIF6MjHWjuFZTimV8mhzv/TS8O7/jEJaG7i00BK7/C7ntc5OCqXsljn6xE3o/hj6I638FSkFYjMmODKVdQA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Protect VMA from concurrent page fault handler while collapsing a huge page. Page fault handler needs a stable PMD to use PTL and relies on per-VMA lock to prevent concurrent PMD changes. pmdp_collapse_flush(), set_huge_pmd() and collapse_and_free_pmd() can modify a PMD, which will not be detected by a page fault handler without proper locking. Before this patch, page tables can be walked under any one of the mmap_lock, the mapping lock, and the anon_vma lock; so when khugepaged unlinks and frees page tables, it must ensure that all of those either are locked or don't exist. This patch adds a fourth lock under which page tables can be traversed, and so khugepaged must also lock out that one. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 5 +++++ mm/rmap.c | 31 ++++++++++++++++--------------- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index eb38bd1b1b2f..b938b286cdc3 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1134,6 +1134,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (result != SCAN_SUCCEED) goto out_up_write; + vma_start_write(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address, @@ -1601,6 +1602,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* Lock the vma before taking i_mmap and page table locks */ + vma_start_write(vma); + /* * We need to lock the mapping so that from here on, only GUP-fast and * hardware page walks can access the parts of the page tables that @@ -1806,6 +1810,7 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, result = SCAN_PTE_UFFD_WP; goto unlock_next; } + vma_start_write(vma); collapse_and_free_pmd(mm, vma, addr, pmd); if (!cc->is_khugepaged && is_target) result = set_huge_pmd(vma, addr, pmd, hpage); diff --git a/mm/rmap.c b/mm/rmap.c index 15ae24585fc4..8e1a2ad9ca53 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -25,21 +25,22 @@ * mapping->invalidate_lock (in filemap_fault) * page->flags PG_locked (lock_page) * hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share, see hugetlbfs below) - * mapping->i_mmap_rwsem - * anon_vma->rwsem - * mm->page_table_lock or pte_lock - * swap_lock (in swap_duplicate, swap_info_get) - * mmlist_lock (in mmput, drain_mmlist and others) - * mapping->private_lock (in block_dirty_folio) - * folio_lock_memcg move_lock (in block_dirty_folio) - * i_pages lock (widely used) - * lruvec->lru_lock (in folio_lruvec_lock_irq) - * inode->i_lock (in set_page_dirty's __mark_inode_dirty) - * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) - * sb_lock (within inode_lock in fs/fs-writeback.c) - * i_pages lock (widely used, in set_page_dirty, - * in arch-dependent flush_dcache_mmap_lock, - * within bdi.wb->list_lock in __sync_single_inode) + * vma_start_write + * mapping->i_mmap_rwsem + * anon_vma->rwsem + * mm->page_table_lock or pte_lock + * swap_lock (in swap_duplicate, swap_info_get) + * mmlist_lock (in mmput, drain_mmlist and others) + * mapping->private_lock (in block_dirty_folio) + * folio_lock_memcg move_lock (in block_dirty_folio) + * i_pages lock (widely used) + * lruvec->lru_lock (in folio_lruvec_lock_irq) + * inode->i_lock (in set_page_dirty's __mark_inode_dirty) + * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) + * sb_lock (within inode_lock in fs/fs-writeback.c) + * i_pages lock (widely used, in set_page_dirty, + * in arch-dependent flush_dcache_mmap_lock, + * within bdi.wb->list_lock in __sync_single_inode) * * anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon) * ->tasklist_lock From patchwork Fri Jan 27 19:40:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30726C61DA7 for ; Fri, 27 Jan 2023 19:41:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B8BBA6B0092; Fri, 27 Jan 2023 14:41:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B16726B0093; Fri, 27 Jan 2023 14:41:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B5866B0095; Fri, 27 Jan 2023 14:41:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 863366B0092 for ; Fri, 27 Jan 2023 14:41:57 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5D16780754 for ; Fri, 27 Jan 2023 19:41:57 +0000 (UTC) X-FDA: 80401599474.16.F2CC071 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 99C921A0019 for ; Fri, 27 Jan 2023 19:41:55 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=lE9qRR84; spf=pass (imf19.hostedemail.com: domain of 3_ijUYwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3_ijUYwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848515; a=rsa-sha256; cv=none; b=VRO1fW0feJlVLDn9X1BvxjxVJ5dcWbzybcmwG+fC0xQqZyEl3uDfKmZk/NxeSFlFClssh5 R2yY5Fd/ifsp1L7heo31P9ryqqE9Cg8nsCznBPrnzkcvkUpDvZwismEMVx2PudEw0WeS0W NqeiReBU0Fpo2mjBW4CZxYq5NHtSkC0= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=lE9qRR84; spf=pass (imf19.hostedemail.com: domain of 3_ijUYwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3_ijUYwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848515; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=BkKKyRvWSiVYzAknmwH1ejeEaB6UwUtGyo04ZzDJ5IhTJXJKyYIy4zJhE1C79waIgyw9Sq zMAZRDzE7O7wkL3LRcRyHOp7A2TfHF1GWpvneoGDJWyKLjOo2JPTtnYov6rANoQDTUipFO i6vMm5qWsODArS9GYSGdM1E4p2isM30= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4c11ae6ab25so65246377b3.8 for ; Fri, 27 Jan 2023 11:41:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=lE9qRR84U5+rYrRmzpPAw7Ud6uFSOEezLerrdXzskk/7N7IWEzIS+P8h33l5/GHLwm pzf0b8p0OfVWGzGLb/dMUWWv/4+TG3PN/VW/HgmJDf4GTYQxz4qhZvzGmSe6iwYXQbEv pmRWin5z3kTr/dQHGmOmQouL9X7gNxEvHMfX/Mw4+j3xs8bwN3ke4zv6/7Jpk/cXGw8a hnXfsAcVwtJnLNS//9cNJnJKu2XJduYEuopAyOxXqZTZsrcp23rvXM55Jk6t7ntayedf QFH2QchxeUajMjFYvRPLkdcrU5vfcJYPLkQ5Wb/mKP36Ds4BBv5V6MGo7zAGbp1SePYq xw1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=2HeZ8bHr+JYFuyc3IetC9nshd1Qat691URqw0fD61oxd15vULQYs7/3PXXtS/2H92Z uXBzV81OP/pOEiNnjnXIGnFrRpUO69YsoMtSIKS+Q0qxgZJuY0cUo5CBMN59WQu7gU+O Toqr5Zm6sbL3w5OTwLff77oJjyvK8JqqDGVqxpwsMvb7Go0AvP9FPcZxEPO/ciBkay0c daRo8xsNpwOB+oj70P6I0debN+pyHZrr7UUqHYo9nx14iZ5mPoqQMr3PMl6ElDx75VaI 2Aw07fRe4eTDVO4GRXI2rXmy+ErwVLNSZkeFt+6HG8fL9Cfee8SqnnHb2rygoYQvCUtT Nc0Q== X-Gm-Message-State: AFqh2kpfGCkIKwVlmfeAnITyxnsic0Fimt51weDvTWhMWkknc7IWlV+w aAoO+KoO539BhwJI2GlwZ+/QiIE/fe8= X-Google-Smtp-Source: AMrXdXsibCiIco7YlC7GUbshkOFJQAk8PpTkup371cElxWyegDeUsxNSE/XyOxE1JUpmHxGWPZOyG9ioimM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:b94d:0:b0:801:e503:dd0c with SMTP id s13-20020a25b94d000000b00801e503dd0cmr2559229ybm.384.1674848510225; Fri, 27 Jan 2023 11:41:50 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:52 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-16-surenb@google.com> Subject: [PATCH v2 15/33] mm/mmap: write-lock VMAs before merging, splitting or expanding them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 99C921A0019 X-Rspamd-Server: rspam01 X-Stat-Signature: 446kwd7h3yn9g19nfk3jeoybse8ngx9f X-HE-Tag: 1674848515-113936 X-HE-Meta: U2FsdGVkX18W5mzTBRd2xiN8UMce5m9Os8AQ6Au9UD4tHD67fYsllKnF9GGoz8rxCHMfzMLDpIecDHnmCOt6JVfiBJIMYM+azo4+SbhAlpYbMyI3TVLqoT/DU9TOoeaCwLs8zGE4Rok299VVp0sKAa3OEb5vgIMuJZJbd2uMBEbs6GXvpIduF/mASNXHYz1MY2MPi9FofPXefNvWpKuIVDUY+LZupubDpmD4i2cQ24WtpFRjkNM1T2sWcmhxEIV4g5FQqWb6fdx/RRP6oKgnZC00d39oedj7cHWVDYleVAuGXzB34m4kENTIJy0cB5C0CPSc6PjaxUsabLI1xRmhlEZh/u4CP5KDgzjGyZMrDc3oT7NYNkJWAY6O3wUSJTsoNYd3f47P+sqWcgH1cfJbxIjejAuC7ZM4g/gaKfoV8a720QhazZn1H7YC6A3mFwcPNI+MD3+E1MsOqpOgBGVpNZpUN4OvRiApdTS+TWotSIFA+xaKmP3P8DZEgwPxaQqrLv0MaaeLgbi28aewRfiLlWLbkf2PMq3W2GHsQ0sP0skTbTXZRiKdTJPj8DDMI8QBHS4DQU2s08tNkGsvPM12JhH7U9Nu7ZQ1BNAoLAI1ZX3yDWOccOBwGISvx6RV6j4LxD9AWalj0b3WFJ/cyaiB0cMB+l0MX2hr+sFYoGY1eMiu1ZbvgM8I4XMNY23D36NYOl/6g5itl5JH3ndYRMQ227TtuLoovVDbYwNSO1ohc44xDQMXChNNPOUG5U8Sa0zSwNnjuIrvrKxs8VKT/O5t1dxLZi7YflfLItSBRxg+RrR9s9cJvnVH3zWCWCpkW6bi3wBSZWwFKa//uwvwZHazl2FGtZNAjnZPzNTo/Un6zvzPOMchT41/SBjrB+HTNwuP1/ZhQ0ZK0T80hYv9dJkN4WGmnKDo9Dy9Zpn65yJCaRaTnmfeWs0xLYcGS2t8bENvt9Ge+iGiZlrXNWS2y/g j/7biOt/ T0re0a5UDvpKwzmqRkSMdUI25PpWPUVoXPcyM9XFW/9prmoZF29PZrWwrH0GS0SDbZR1etBD1703n+k459XAvBQv69R9DVux2QD8+AYfEpBClirgkP36iJUNypB15qXLGlSaO1y9wI/f7DrdLuW3HjDqyc1wWIYMTTC//xhIRaIWaaP6CruC0rDqRt9tUlBoMb2XUe8oRWZT0nBRwNZEBiuk2xQ2vk0I11aCYC0c3LE2C9fg+i4KohVONwKH8tNDCMcKZ+ZoJHl9Sd4oNHrnN8DeEXbPzo8UYVBwb0KA8qKaH/V0n2gKsMsI09T4eN9zXbwZQN3j5cgItg9YBGI9ChYxG7r3poicBQXNLRjF0Ameoi43uq17lYcPADxIqy2BVoUul5sqPg6iQVGn8+rmc6KVHZlL/QtMKC6myeEx1+hVesWY79DbMppucSpWL7U6iTuQmHWkRJadxMoYfKdBpBRyboYqv43yM4D8f0Y4DI3GiZ0d35rHXPwbdvDzajOnKwFf0gIWTO9u39AAeOeToaaZFxu5rJ+iPszYixZOfo+OzZs4Rm6XuOJXJtQf8BAeNYID/x4gp1Ym1B6Zud6WvuYOlGEctgY9solZshFSY1XhmpZRnON/C0vg2oQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Decisions about whether VMAs can be merged, split or expanded must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 7467d691e357..6fff76334177 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -269,8 +269,11 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) */ vma_iter_init(&vmi, mm, oldbrk); next = vma_find(&vmi, newbrk + PAGE_SIZE + stack_guard_gap); - if (next && newbrk + PAGE_SIZE > vm_start_gap(next)) - goto out; + if (next) { + vma_start_write(next); + if (newbrk + PAGE_SIZE > vm_start_gap(next)) + goto out; + } brkvma = vma_prev_limit(&vmi, mm->start_brk); /* Ok, looks good - let it rip. */ @@ -912,10 +915,17 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_start_write(prev); next = find_vma(mm, prev ? prev->vm_end : 0); + if (next) + vma_start_write(next); mid = next; - if (next && next->vm_end == end) /* cases 6, 7, 8 */ + if (next && next->vm_end == end) { /* cases 6, 7, 8 */ next = find_vma(mm, next->vm_end); + if (next) + vma_start_write(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2163,6 +2173,7 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, WARN_ON(vma->vm_start >= addr); WARN_ON(vma->vm_end <= addr); + vma_start_write(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) @@ -2518,6 +2529,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, /* Attempt to expand an old mapping */ /* Check next */ + if (next) + vma_start_write(next); if (next && next->vm_start == end && !vma_policy(next) && can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, NULL_VM_UFFD_CTX, NULL)) { @@ -2527,6 +2540,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } /* Check prev */ + if (prev) + vma_start_write(prev); if (prev && prev->vm_end == addr && !vma_policy(prev) && (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, pgoff, vma->vm_userfaultfd_ctx, NULL) : @@ -2900,6 +2915,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) return -ENOMEM; + if (vma) + vma_start_write(vma); /* * Expand the existing vma if possible; Note that singular lists do not * occur after forking, so the expand will only happen on new VMAs. From patchwork Fri Jan 27 19:40:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A2E2C61DB3 for ; Fri, 27 Jan 2023 19:41:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BDCC06B0095; Fri, 27 Jan 2023 14:41:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B64006B0096; Fri, 27 Jan 2023 14:41:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F36C6B0098; Fri, 27 Jan 2023 14:41:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5F3E76B0095 for ; Fri, 27 Jan 2023 14:41:58 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 35155161019 for ; Fri, 27 Jan 2023 19:41:58 +0000 (UTC) X-FDA: 80401599516.05.039C756 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf29.hostedemail.com (Postfix) with ESMTP id 737AC120028 for ; Fri, 27 Jan 2023 19:41:56 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Cb3p2QcN; spf=pass (imf29.hostedemail.com: domain of 3ACnUYwYKCMc574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ACnUYwYKCMc574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848516; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z2/pTj5C3cgEFiOJp2CeCn46OI8ibDfme0ccGGe2ykQ=; b=oulqlebZnwq5ydExtS4WNpJami1T69rhbXQ9itYVWoXbzXrbfyBEFNKRpwgGshW3r7ia/l 23j47A4TsLiuGtCEoClJ9hsuCmKAuISFdmulLN9xbu1Cq+m6NH4SBSOusSvXDPe/Cemrp/ 6HA35W5u11WdQlXIMwNzvg9r0AbVldc= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Cb3p2QcN; spf=pass (imf29.hostedemail.com: domain of 3ACnUYwYKCMc574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ACnUYwYKCMc574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848516; a=rsa-sha256; cv=none; b=TJwaFRZQkqoRF/Eh28EQckN9zsgyPK4/9m3E+l2roaPljim50eUSaIlyrC+0BKAzN59FYN HQMNXfRYgvcRPRSsNQJuKv9pK1UG/NMOhCmlyAkek2ejz0Q9F7v5yG498zO7NEPd5CEjOA ekEfe0PVVNKOQ1b/Slx3Dx/JT0aB064= Received: by mail-yb1-f201.google.com with SMTP id a14-20020a5b0ace000000b007bf99065fcbso6360922ybr.2 for ; Fri, 27 Jan 2023 11:41:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=z2/pTj5C3cgEFiOJp2CeCn46OI8ibDfme0ccGGe2ykQ=; b=Cb3p2QcNsUnk+wL6/K6P4ED35dG3ZUcB734zcAsddvwr/PVSkrRYzepHFsLMU82wqr DCByjFxSWTXuXxKpPzrEyhOes5tD65SfDW58DVPGzZ4bl/GM62EXvV72BYyh+FkQftRL 9ik0iporDuduJD370bXV4KniJvgp40Hs/4xlWBjuzB4w/OEbzQnMEDOcEM26lHODokML 801aFmGgsonG3Ea3SKObTIkdYEG/WUj/siwNQ/Elnoh6/xVR3ypDahdEx2v1JUXFD+l6 cbwz8ftG3Ry5zhC62QLkQy0TB2bZZjts/vu8EdbF1X6krxdoifCb7qgLuiaEVqlGXvIi PdsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=z2/pTj5C3cgEFiOJp2CeCn46OI8ibDfme0ccGGe2ykQ=; b=rFim4uMDz55rnJKNGldbw0aZQbld4qSQAdCEDtqpp7GFX5gmwK2aOCgZzbuCn36Vek 4fPBF9FA7BsfY2yIjs6K6S7qbOyIgascUS9rKugANA/oe+MGcJdE8Qpt2hSq/TZtm+/o KALbvx5EvY9yWFOVSpGgkiK0yM+pQ/Mu68TWzWfiNw9Isye7qIQsU0JL989cPjCFPcxw bTxOnC2wfEQ/jofRHiPJU7ASNDRZoENzneeydUBnfKDZxTS0d2SJ8xV/kZw3jxRvzfW1 LQ6doa4v3Y4BVxZbk2ESjn3JO/DcZcmfmFybRU2rIY935inmDoxUmUwvGMFanuGN5kl1 PlYA== X-Gm-Message-State: AFqh2kq3/7FXZUCXOTBiUTC/+vHkPBvPRz8dXXQQ9NQg4aMcWDl9SBpD wZqdNd1FGpRKWYB+2nqoia1X51E8Dfw= X-Google-Smtp-Source: AMrXdXvc0ioDltykhYGthi7quhcG5gFdvKFRYGkIf9fgUkYhxxt/rX6VwG8N7AhqKPLty1w08hNt6tf3mAM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:f6c2:0:b0:4dc:1241:cc0a with SMTP id g185-20020a0df6c2000000b004dc1241cc0amr6165876ywf.499.1674848512603; Fri, 27 Jan 2023 11:41:52 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:53 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-17-surenb@google.com> Subject: [PATCH v2 16/33] mm/mmap: write-lock VMA before shrinking or expanding it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 737AC120028 X-Rspam-User: X-Stat-Signature: ydbmekczi8ubsioqqgbcf68by5mo44du X-HE-Tag: 1674848516-462550 X-HE-Meta: U2FsdGVkX1/k5MVOqsWPBPDfmxViNlYju4Po+NdheB9V3oKhzIpUvEoDlRqCC0hDg/seRvoNOJG8FPJ/cVt7S2CkJl1sbEoEcSwMBzfD/HyouOs/nAwDrgtic/wGz3/L/vjRXsGOjR88KDPeAELFiCa1CckFlCLrsrQ8MZ1HWzfKXgXvs41e9ihmgRYxwEmSKy+LASb336NG65N3ljZ5hRT/Z7uuEMahsQKenU6ezcejKzts3U19dyle1LZijogwzZp8p/521hUvDdXeR+1hHwhI6w8bX3e0LSXjRDXf6NUBai79fhPHGOE8cY9v9D+KIOfvpaqW7UJxIBUnRH2grAMKFILBjZxTQ0OmApx1n0eI4C3eoRf9Exdapja5kRSYqAdK8Oq+E29kkoEgS/3fYSzTCA1udOPDg114Esq+em7AN7sRYBSWC61IbM0s1DyPxWpwwEyrv0ene2Sb1TUU09DpyQUGyxGgdnM7ZdfsoVFKxr/OrSyBJ9vF1Bg0Tcs16D5/WzELbLJamEVZBYjwW6mSdVLZzp1COzTeWF3ibCafnqvFKNu6Nm75kaGCwIQ175vbY0JfQDZG6NT5NRbc2o8obZAhBGaSxJ+KZ4NngzD1xO8hhG5YRugF5D+aXlsi+l3/DoLA3K/ctRLZkfPnjRG6lPfQFiBy3LCA9BpdudSrhkigUgu+22cOO4ta3N1CjBPWtIdy9VM14mIURJjh6MyDY9gPsqDZ10VtISDUDbXYvgu6BEcFuLHyNmYB8YoKnwY8LV7PlgPmyoFtptbUG0xdThkbrKr+i6lcfFj/cnEw2PWzdPw41ozqMhlFptjMdrFvM2V0dbGohdqvOdC8/BQnBpeNi3yHNiyGF1N3hcN1tYV1cV5SFK2c8dcAcgSX2+g17jC4u+2n2g//UqJ2HWABeklvTx1JbPW3GFIsQc80BBDFvUX9LvMtGlskjIET7tWw7H+AiqB43jWU1cC zQ0gpI2S jh2LNeudrw16aqbstG6fzClG5rOQcjkpfVPadNQS9Jyxn0Rw34eFl6JKpsvshJuQKNGXIMHN6vIAnUAG8pIwkVLPofryXTfksG0/HhpPLVQltmH6hcW1YPbN4zurDZhTD8vawpHjBj3/xBr1Q7gSbD3Ek5hAUnK3Ve+RL5bgM1bCFq4qwSLOHRSKorSs6HDpodueT2W+pczorOm87jfT51VnVb4jL1c1ZgZV3vuBLTRREG3VX40E0JES0ejeUPDtrWrb1w2mkHleLPCCRbpcz02TKrJZFNi2QZUAviZV2ny8Dc5nkOG57RDdWYkrgrmcG1Vhzltp6BlOnBXmX18HRH3NdgsFSyOP+8m25xQBHDWpKEVbVD2Hx8aKMg6Mn8FoQD7SumZ8qoCy1wmvGxMgolmhXLvv883a2jDPqFjzT26kNpU6brq7Il304qjZDpEkNXowNZwe5w6v2MkNAMBBszQiewzuGPlNbRrn+f2q3l3lbaV7spqXS24JCHbudQTmUDY1f+XFc1jEkRNxvVHu9C668JcqFphX/hDr5AcRLJ0b3dV0i/pjTDkNimWxiu0AIKozn1kNZwDLgZsoouMOrlm8OWR4e63eJyDZYdCOL+dV5js1vpnnMdM4HSw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma_expand and vma_shrink change VMA boundaries. Expansion might also result in freeing of an adjacent VMA. Write-lock affected VMAs to prevent concurrent page faults. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 6fff76334177..60038c24d836 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -674,6 +674,9 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, ret = dup_anon_vma(vma, next); if (ret) return ret; + + /* Lock the VMA before removing it */ + vma_start_write(next); } init_multi_vma_prep(&vp, vma, NULL, remove_next ? next : NULL, NULL); @@ -686,6 +689,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto nomem; + vma_start_write(vma); vma_adjust_trans_huge(vma, start, end, 0); /* VMA iterator points to previous, so set to start if necessary */ if (vma_iter_addr(vmi) != start) @@ -725,6 +729,7 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) return -ENOMEM; + vma_start_write(vma); init_vma_prep(&vp, vma); vma_adjust_trans_huge(vma, start, end, 0); vma_prepare(&vp); From patchwork Fri Jan 27 19:40:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40757C61DA4 for ; Fri, 27 Jan 2023 19:42:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 105996B0099; Fri, 27 Jan 2023 14:42:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 066486B009C; Fri, 27 Jan 2023 14:42:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C575E6B0099; Fri, 27 Jan 2023 14:42:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B0F436B0099 for ; Fri, 27 Jan 2023 14:42:06 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8D82D80619 for ; Fri, 27 Jan 2023 19:42:06 +0000 (UTC) X-FDA: 80401599852.02.7F30E73 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf22.hostedemail.com (Postfix) with ESMTP id CF430C002C for ; Fri, 27 Jan 2023 19:41:58 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GKKENfMf; spf=pass (imf22.hostedemail.com: domain of 3AinUYwYKCMk796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3AinUYwYKCMk796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L4ezA/DdKPqYrQ5cS+Ke2gtcDFDEWQ5+mTqJ9joAzRk=; b=CRigBxutiXKuxVvJElPEqYtvjVwfafGEWL9heiPsEZhJXuite898AJG4XGTPk+rCo8cMYQ kfkK0Di9B01UWLu/++eyCI6nFdFbsjGsawaW8Mu0e+a9ES/54wrKzuA2qBPb6AShK3Smld nUOHxHQkpFdDDFhOdd2UWVP9O9iivWM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GKKENfMf; spf=pass (imf22.hostedemail.com: domain of 3AinUYwYKCMk796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3AinUYwYKCMk796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848518; a=rsa-sha256; cv=none; b=YYli7h631IsAcJP3Jqly2a/PqtqAYeAflu3sSWlb6ytnNDXUOjR+UN1z4FPxQfqhpYdskD F6YQYw1Z+7geYROvtRsSy4BViUKRDPN4WSji5Hk+6caM6Qh5Dh84Zk3otR21ZjFihv0vxO t6iXRi84au2lVq28EHH38t9W/PcfXDs= Received: by mail-yb1-f201.google.com with SMTP id d21-20020a25add5000000b007ff8112e8b8so6429496ybe.17 for ; Fri, 27 Jan 2023 11:41:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=L4ezA/DdKPqYrQ5cS+Ke2gtcDFDEWQ5+mTqJ9joAzRk=; b=GKKENfMfMG4N0pZCcl9OmPXOIcATJjfZsJmDxgzcwEK7KGqwCsad+AWtcOXvaoC0br qvO01w86MwzjSygGqi8PA6cojX9dXaSV7GQ7IPy4D+7hoAEyyoJoE4vESGbmz//KocjS XInPKVcNcxUU/7v9tYz8Eq3N+worhIjUrPeLT3iGbzfEm71MChhnFLrlat60IF4TyfFg cT/sx9cN16Dq7Csnqgd8AQwbf9+2MwGgrR42iD3xwMMpBpotxh+QKt5ySOsx98KoL0iq FNYiUMi+Hv+PIq3dZA5NAAwxArkgkUL/euE7T+pJZglO6ldDOiv8hb2KxyDrVWXCBL/Q MOaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=L4ezA/DdKPqYrQ5cS+Ke2gtcDFDEWQ5+mTqJ9joAzRk=; b=GhX3GmRJJTPWoCEHkbkPmRSQnIjHRG3/SFX3hIJmAhgdKTgTEfrnq3smIolBe4ZyoL y1Jt/fsbgWdDS7be/ODkSBiVnC/NYu25cTLuRqiG7JJxG5GmnwZxVhGl31ZoeHLbNbVv 8JZNFVMjoM7sQ3HJye/tlQujEd5YbBKM1Eq8pdCjNOrPmEzBuMl7ulJoF7+MAS8wjziq 6EXaX+qCa4xrMiHWScXYjGGGX6g7pULhCmCLXwaBw3ZLxMYgIiHAR3KfijzXR7oHQPCz K8fUkAzSzxxKILN6Anufc+q5tS/knVd6n/k1nIWS8std3al9vEDA/EoQZy0KY8XThXDz W3Rw== X-Gm-Message-State: AFqh2kooOkVM4+974u883mBD9RBnCrQXTeeID52kKtkhNztDZ8uMffDF k7MDFKUDhKRRCgMDK2M09ESrDYTyIFI= X-Google-Smtp-Source: AMrXdXtaMEzriXRGDDaPyWp6cI2qFr9C55J9V7x8prQAmgawr911NF2wOHqTeINFz8M7FkUN1SViGFe49Jc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:3c9:0:b0:803:3696:c473 with SMTP id 192-20020a2503c9000000b008033696c473mr2168756ybd.224.1674848514947; Fri, 27 Jan 2023 11:41:54 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:54 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-18-surenb@google.com> Subject: [PATCH v2 17/33] mm/mremap: write-lock VMA while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Laurent Dufour X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: tx18ru4zozsz18t9ad66ry9oy9gtpup9 X-Rspamd-Queue-Id: CF430C002C X-HE-Tag: 1674848518-701587 X-HE-Meta: U2FsdGVkX1+agbV0LYvoyCrRg93WY+lTLod7xZowuj2NaPTGtPPUHq5qBWhs2yeVXpEcuX5/tJdXyE6L+k5AtQ5bKxSQjrpNlZuWoV9AccMDSxCXyVZsl3Sy22Ij1S5UP3BJ1xFlqMtTpVrkFVvlBMwxwDx3mVxatkqztFIB0ZtYFeAfAUSY7kWDN0MV/TgJrR6UvHly79JR9mGy7ArdqGTi4U093Darfg0JTFZ/AnYFowpv+f0IInRRh788lCy6/NFk6LX1tWueq5nWK1akIhBD9v6DkCsVKJVOfus4z8FgTbJVmilSVoMZp7Az5w8/cAy64Vhiu9a4oAikYsFP2KS+2I82HgpTYqwB0xJN602Eo3WUE4cAxD5+M2YIgZ61cwRIJ0RxcQ58JKLgLjC+9bfxm1ItFfESTJusQPNAekd15wPMizldWSjaPWLrvdfc6Fdme6AqiRZp7SGM/K/OAn5CFqcCSPjTyYcJjjO3EOWoLSU/xXhQAinYt7UlLvGO7tCAV/qa3Qb5EqjCO7uAfAvhpnh6vts9n3CIenu9M1h7fTdpbdLbufqGFwJjGGkmqp/15/5Rt79ngWzi5CMPIF89Y0kNrX7V+ri6wZ6umYKsg40nFVjJmUzFr7PFhQab+8kGzfCgTpAPVdSWiAqvNbVxoNC4YWUrd1P2jR0mHMohCJPgPa3Ar8PIH55qFIpI5WI3nlPPHofnsGs3uNxde0LhVeHjMXDEcbG/+I2SRH5gzKDvkHJduUpIGogS1VBf+pdDmh5Ubg19YNqmHB34pHiZDMCLRvPjfmVCh6xErIYwNXeOcCgfiNIA36dgxHFid0YEadMjdHyxkmD4H8nUlTR8v4MGJt570muUJloW6IqB0MoPqrv8fQbEqnCvpsclqcpscOvmpKqIxDRUqz3hg2JeHzByXDbiFxfiN8saiagCFtbA/nP3jKl4nT6Go1twmE7nfQry9ZPI5HkywyW KGIwrMkj 2dUhPFF6u47bItiJwZUsg0Owz/5VHMUNywmz6yTxaPVGv+40jZTIFeIMgSIogLT+hZ2aTNVTd2jnMmnHSTvjH2BELyD1fV3TEr3eH5ObQ8okMqcqkkaSet37FF6sjvidg6RmbAxhf5XS181PbDR6skXZubf3D1sXmuCmuz52uiq033f2hRpA9NKOKsXjMAecbshXqfyLWTu+YcFKvxyY3FqYb8/WIlsTK0ZHqxkdbJjUjSlqYj+0QAEYp4VpdPqEQTOCHTgDIPqncR4wmrosBsYvmEI105k6ZQozN5Ewqi/Ykp7LsKrb4MWsIQ4+OXIiAIvOSYfEq+DyppOJVn1Jdk1cY9sjwY1HPGMGbwDSQtO7Oj0TqkYd4mJej2Pv/VCDO6mr8zMUau7OYtSzB4Rq53kI2d9MxCihksCMBdta2Y1J4ml1mYe42N76l5sLpRVfDdXi8bEzfkjp2WuU7m6sO7athvwaDxS/5YFOUHnkTGuLkXSLBHph8CAw1Pzw1sE/HtOADFsEr1SHDkANqPcQaqBHiv5UmgrLa7lnoASPLW98NOtjHIZAzXNKKuLIxqKQAa7qbjpLpsZr9uM9JlxCq2BBZwZylrE7KYait+lZOtD+X8AlB0NvUqRoDBseNNvJiXtJKaudvTaITWpNITBSxVMzO0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Write-lock VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 60038c24d836..b3c247073aa0 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3202,6 +3202,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_start_write(new_vma); if (vma_link(mm, new_vma)) goto out_vma_link; *need_rmap_locks = false; diff --git a/mm/mremap.c b/mm/mremap.c index 411a85682b58..dd541e59edda 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -623,6 +623,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_start_write(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Fri Jan 27 19:40:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119258 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 948E0C38142 for ; Fri, 27 Jan 2023 19:42:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D9806B0096; Fri, 27 Jan 2023 14:42:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 289446B0098; Fri, 27 Jan 2023 14:42:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12A4E6B0099; Fri, 27 Jan 2023 14:42:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0297B6B0096 for ; Fri, 27 Jan 2023 14:42:03 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C83A18089E for ; Fri, 27 Jan 2023 19:42:02 +0000 (UTC) X-FDA: 80401599684.29.7E44228 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 1E8EB100014 for ; Fri, 27 Jan 2023 19:41:57 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=iI2a7Mep; spf=pass (imf14.hostedemail.com: domain of 3BSnUYwYKCMwAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3BSnUYwYKCMwAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vIuOo+WD7hhMM8dbfJD4mGHBD08/Cqf6l0kIi97gWpI=; b=3l8T5PZ5qz+2jTwY7g06rAUZYykZRsJis5iufKWEskQdufK3uw2MPszQ+gG5GSfLayRfEo nx9oCd4IuSxlBlcFRScF+RnlaicchmEmKOo4ChsvpRevio1dRbagdDeBeqbE1tET5YEFCT cbpig9SiaJmdD1pnUubqYb3YEw68nUM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=iI2a7Mep; spf=pass (imf14.hostedemail.com: domain of 3BSnUYwYKCMwAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3BSnUYwYKCMwAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848518; a=rsa-sha256; cv=none; b=tDQe8nGWSDZ5yZ+h43ixMNDBlfVjs1UUtHsUiEPlNuyYEt/u/hPcDqJbHlHAkRinZs/5RI dUyk+EdkYfyk0z/m+c1tjFw0AWfD2YSf2gWlD348TkOk+RFjwINP23DYYLGENx8f+MGz88 8DYzkuPtrPg9JsFXCGgC6qhYa+EPDNo= Received: by mail-yb1-f202.google.com with SMTP id p19-20020a25d813000000b0080b78270db5so6348716ybg.15 for ; Fri, 27 Jan 2023 11:41:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vIuOo+WD7hhMM8dbfJD4mGHBD08/Cqf6l0kIi97gWpI=; b=iI2a7MepfOBNDqB7wu+/83gKbYOOOUW8grtwa3CiYxFM9JfwE6yM70Ne3Bb3nWcBVw Yu75uY4jTmGekedOn8ngeIwpB83dCuwgdGAmo3PP8/5H9WQZ6zUPSiR4iOLBbFN7215j t5fY8PD6baQLdLtkjh6u6ca2lGsT7fHu5oxzuRWhyEjLoCc6wJIYtxHPj4OPz1R/r154 +v9JTIHiwr1S1nQG1EzjVKOQzPqMVCb1Q8Qi6vYQuYU/8I9rd1kQnHGC5kdzCbnlOWk7 FPBIbkrQ5+p0qEBCYmhl8CWtOia+KC/VA/z2TgOLyA8KXzLYreAkUsa906GcFZG1qYyr SCmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vIuOo+WD7hhMM8dbfJD4mGHBD08/Cqf6l0kIi97gWpI=; b=3EJ5InoaDMAIZKEkAusn0xzS5hvx4XLsRlAgSHGuplw1xFlcn1karfQFL/HDnsqHNe 46C2/Lbc6XQC4HooH/P2rEjmqoKItiAHJ1n0e2wMmnRaZPMqO+JIJZF7LrjZYXzjFxWp jFz3/PFkJ89wnvryy7f9AfCU60INe4PcfVUrjtrvJlKdjOgt8X7HR3s03MCSNIB/mPKj AXI2vmJNJOh+GoUeh2XGra/rgBC2+JUZCKvbSj/SOv9bCkoxyaO2nmAdeYIE5CR+7OTI pJ8jhD5BXksu0btBKA1kpiNfBWO0NUXxhTkSJA52tvbQKVJhvSPtzzqA6LXh4FZRPgMf lipw== X-Gm-Message-State: AO0yUKUitaIxCu1EZuUI55L2QLbOd4+67c5ikBgbpShRUNdMCFlJE/ji SoDNlY1SGkWa+1gZQTyrN6oEkcavuBs= X-Google-Smtp-Source: AK7set91XCOXqw0Dg5UJpzZX/mqq3ueYFNiPcfvNNp2V4Tl7t8riuyYgZdoB/8yFEdiYxfiOzkRBBcWpyqU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:3e15:0:b0:506:653c:db97 with SMTP id l21-20020a813e15000000b00506653cdb97mr1675918ywa.201.1674848517274; Fri, 27 Jan 2023 11:41:57 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:55 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-19-surenb@google.com> Subject: [PATCH v2 18/33] mm: write-lock VMAs before removing them from VMA tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 1E8EB100014 X-Stat-Signature: 98sstob16rsbig5ehsthukukz5uuoy1x X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1674848517-788668 X-HE-Meta: U2FsdGVkX18by8xYAPHoCq/mAFJWx8LhRGGhc9awm9MrIbLf05bDz37EA1dYTHQTVZrejL5nLywguqT5zQsxZvO7yq/8a7am/DKAjmilxs4JLQspZFSiDiKPyuOB//o4BX+kOJWGekA9El5Q3zUltebNAnkyM7gk3bdHg/loowi4vXAA4JQrmNO/18ZPCo1uvWzbQyj5UgSnWgigXk73rzVo61fOd2yenbSgx8YXzAvXEO6yyM6ZD1XtFtil4exb0yEIY780UxPdQcCW1Ep3I7Ff9mo1xDq5id4c+vSzI0mpM6lh/kY8QvsrpZggkSx45pmeGA9vq9Yn4tB/WQCyy7XLdvY9YwhkS0IuRoNK2tZYxMLF6RRQ0uR+B4GL8H/JqvP27MRLZ0auuQ66TDnbrJPH2/1Obb8EBTGUESclYmHwL6aQecCpKErRekvDuGR0Olt2ZNhqGfpCJplRzZB4yPkzwy/09avXyNkUGrNf0SDMbvH3WxhGBE7fB1fKz16zDqMyuW5plSYHtGuCyElW+ymYr5DEJZ4xNn9I0ZwhBT3ShC1OGffJ74Oq8nxz9f61TlXkJOu03o7swTtDrKBLII9s0leSgetGdqLrie3VrR7K8ct+NCCpoCusTIFCF0nRdxVXGYHT5raQ8wXtiK+o8ntJZ1S9uCcxPX0qB2ak9OUNgCF7J6k3d5W7x+DQzyrwhlxfgDwjiLFi0dukuYSCOry/OOKq8wRiLejwl6pUPkQnHX1xJiWA7OlizsvYYrqAaI0qiwfuAMMO+RKmgNljB2bzok82ffv1DOpemXR5HCYlZhtQaJ8f6wLnco9J39UnIhZk6C6seF0PWUuP+uRm4AmT4/mq3X9npqvTdN2n1XcgVe+NElNf9BrK/RnjNTqaMBMFyu+5eI1o/DhgkSMDJUOIXAId2btvxsepmD27WFZ8rFPCM7EDrQD4GnESKWVrdk8pxOHTNyImnhCzD5Y guQdibgj pZ5fBGlf67gy/QKBfeUUxPfPpFSlMO8JqsNcKpMZJWLaYVnFEIWH2Xasn59pS6WNeuALxYXPRSSeRFpNJuRnZxZOSm8KP5KdBQYgPsJmHAG7n17Saf+9ibxhuXn8arpQnbpynXZjb2YBMZa5fG9bcTcYwSsZm8iPH+VSrz45tCBm1XL4KWCu50cB3ZWt8PPQqVM0509juTwpIXsWTxDFa1cEgLgGKFWp7uvakKV35jdzlGC3wvk/YHhy1jZxR3WsGsWb8Qa8EBo3a66a04+79IpvgZQpn0oIE21EX9uo+GG+ggKFkGUtYP21tcuyXJO98Y1bdCi3Q2Ill/A56Fe9oYR6PckIMa4JeMY/3BCQ4ValSWnJ5KBPI8V+P2vQbgHFG+Fdm3cdIgHHf+1oVtujTpgpZ0GLGxX2Tb/jXRrPggDZAwoxfyXLIutpAr95PzNmjNN62y3eku8m8gXW4CO9d/XjQczPgm78cVMn0nk9UZiGVDNnB7b8NEJXospJzKxALwzMTnI6AgVxxyJiO5jYLH+PQJVJvcD+s2hB0uDKdm52NUc9fumD+ZlE6UMpXp8TWTphu+S3+SGAoVCBjYVzPYe7++IdEqF0IbZ/HLhQjIOMyHzv35Ck2v2r8GA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Write-locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 1 + mm/nommu.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index b3c247073aa0..5bdfd087b632 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2261,6 +2261,7 @@ int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, static inline int munmap_sidetree(struct vm_area_struct *vma, struct ma_state *mas_detach) { + vma_start_write(vma); mas_set_range(mas_detach, vma->vm_start, vma->vm_end - 1); if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; diff --git a/mm/nommu.c b/mm/nommu.c index 57ba243c6a37..2ab162d773e2 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -588,6 +588,7 @@ static int delete_vma_from_mm(struct vm_area_struct *vma) current->pid); return -ENOMEM; } + vma_start_write(vma); cleanup_vma_from_mm(vma); /* remove from the MM's tree and list */ @@ -1519,6 +1520,10 @@ void exit_mmap(struct mm_struct *mm) */ mmap_write_lock(mm); for_each_vma(vmi, vma) { + /* + * No need to lock VMA because this is the only mm user and no + * page fault handled can race with it. + */ cleanup_vma_from_mm(vma); delete_vma(mm, vma); cond_resched(); From patchwork Fri Jan 27 19:40:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A719C54EAA for ; Fri, 27 Jan 2023 19:42:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEFDD6B0098; Fri, 27 Jan 2023 14:42:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AC70D6B0099; Fri, 27 Jan 2023 14:42:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 966A46B009A; Fri, 27 Jan 2023 14:42:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 880AD6B0098 for ; Fri, 27 Jan 2023 14:42:03 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6391B41014 for ; Fri, 27 Jan 2023 19:42:03 +0000 (UTC) X-FDA: 80401599726.27.8EC6F76 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf04.hostedemail.com (Postfix) with ESMTP id 945CC4000C for ; Fri, 27 Jan 2023 19:42:00 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=dIXUHHJy; spf=pass (imf04.hostedemail.com: domain of 3BynUYwYKCM4CEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3BynUYwYKCM4CEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848520; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PbZvlTacNRMtTYgLZEFyk5N8KO4rHMcD13osXtjRKQc=; b=77+figJFhK4BeQg66HUzCKtLZCl6HD+0P/vDyDAnBEzHDsvC3gAyXDvtwS2okYM53LqXEc p8zHtbTj+eAMeUxkxt6b+iaOnH45EVcPpgmFmboUkfm60CK0YmgrS3vixfARBbGsFsx5pw Elx8Wlyd9lVuH4SVVkzicfrildPRvJ8= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=dIXUHHJy; spf=pass (imf04.hostedemail.com: domain of 3BynUYwYKCM4CEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3BynUYwYKCM4CEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848520; a=rsa-sha256; cv=none; b=DSVZKfSlFi5N/jcL8BMccvQOt4uFb4sRgksJvlUd4S83bZLY+7hbWU1m3oB48IzyeKxyiy ysLosVPAGdXOgOyxTG65y7bH3FgwD6Wdr8qgUMWDJOeRDcGBXUUlcFirRzXzgXeOQgVkiv kaO7ZgY4uvr2HRBW0f9XtH8NsAU5xQU= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4d4b54d0731so65443587b3.18 for ; Fri, 27 Jan 2023 11:42:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PbZvlTacNRMtTYgLZEFyk5N8KO4rHMcD13osXtjRKQc=; b=dIXUHHJyt3zldvcJ25OA8M/N2ekAkSadxrM4Vc86G1vAxHUN7FQNbSzlWzbH3EF3Ay Np/mC7lgd8yNrbdx+fJZdOiPs5Ax7Y+isaOSeA0W3slHJVunqRE0R5gvKOYJUJLOV6vy eusS7uxTBSUHthbBE/M+AcF029iuITRiXxkeDAE1bHTcC4qSxwHlMi32ImCbpO9h9EMV +wvjzOskJitBDhg128jcOLxlukATYCWhVPwB6m3ynKWMcyTvnD5SZLfuK2oGD5H+Rl4/ hAABUVAQozzvBCIG2lhb5atfzqUlxx9w7Rjl6rs3/T9zVK0hATJevSe9VLcJ9uJd0aXA HxXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PbZvlTacNRMtTYgLZEFyk5N8KO4rHMcD13osXtjRKQc=; b=fHHrXOQdMHAuAzNwE/+IuDVzbGPOGI9mUy3dIy0TTQxqSJ/DA75EKc9HtO/uQ5uHYN JNFQK8a8LcBJVuIcpDxc7waDcmnQK3oHOhPHuftRHu6hj88W75TFF9PWZzlmcsSFTLwG 9PfFX6ta5doHzbzHpKyEVqQhPC8X90vPqkzevnBSsjoHVc5lTe6F9Zi5wn0ccfc+LWHi B1LFsQvYxImAAJrE9neSIHSgCxjQFBp3xs/+5clkO7xEfCWuwsz9r45wNr4p+bU4d2ov bgmJe6jPDxEyBlrObDQAWuS4HWJLMC9v7knB5rndPGhm2C4Gkc3MskV++D9pNLw4Da/A ofJw== X-Gm-Message-State: AFqh2ko59eXdo444AzCWBtYSfVBigmWVWKgwsTWv9Q3+TjxtxaoTmx18 /GPlvGoexLxipWjPqvaJU4Zmv/r9cZA= X-Google-Smtp-Source: AMrXdXtxOfNby/WJjyD0PPf8WyFmfovtGs47EqfediusuKgjOEAtqZl1SXyt3bJWMQsRbQEhpFz5LQpAf70= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:cd4:0:b0:4dc:e929:40fa with SMTP id 203-20020a810cd4000000b004dce92940famr5633708ywm.331.1674848519684; Fri, 27 Jan 2023 11:41:59 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:56 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-20-surenb@google.com> Subject: [PATCH v2 19/33] mm: conditionally write-lock VMA in free_pgtables From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: 8juc3ih8ump356fdz45si4ou5fqjehzh X-Rspam-User: X-Rspamd-Queue-Id: 945CC4000C X-Rspamd-Server: rspam06 X-HE-Tag: 1674848520-320982 X-HE-Meta: U2FsdGVkX18PEYw9l9mp9Q0tvDJNoJ8prQWMDPq8JQacNN02E0noQZoVnVZ1WskxK1NdlJqTt6loluil2tBkTwS5lk+2wKN3y9iwa/7KhAxd1DtpL3hf/GM9jF0IzxNHmkkRbkN4h87hz2mB8uYQaopH/1WHfyGsqSaK+k4jG7o8ZRVo/heAFUKyKAsYTYw7O7k4DGLhvBkjy9F3OI7umbdY6oxL4QDGWVD/k5xbMPlnHzyRdOvIWr1sjA0eSrVrrGenujKjkm00Fy3BefbpQW173Kx0OVZMlAOSh+HCFul3D/9p2uWMGH4t5CnNPUkHV25a30HH4jnV+ktsoaVO40PfDd5diM6KAqCEWsNgZ4u14umm+FQZrsHmfzis5KvM6GzJWDB/HHMhb/ht5lNzYGpQW/1z+cW/Nx9ARCljYSnL9w3Qq+nkauAtsMUCEhBrPI1decynyAf8QBTqYuoMij4JbjHGXRwfL1SICAKDDuEYJ4lcBofCqj94OYKSy8MhzjmrGbp11uYftihiQP09pFG8ogj2b+iYJlew+4Ks+QKC0Nc1EIuXX4zpUkB43R9Fnp3mZnHVWs5FDsEQNhn0I5AevnggLDjSs5DDGfxA5pjzib0DcBM2HNnV7TxJIdUnKFZ5+qus0iHjDEzxK8b6E+/+F7ODpVZzfH2CTlDBB7vVC/m+HFZM7Ep4DflIB5Qwk9nZ8AntZHH6Pmu4hd9wJVlUM2j37KtgvVNdq/qLJhu4JBxetM1Uhn0d7k0EpO8pneQGDdeGF6I6MPKejhzu+V7VvdatjMytyFC8aVfZrvOtKTiCOyJMOEX6l/eQd52pFwngVLWFd2zBv+R8QBXmxsS1AWAdgS4sYg64yd0Rd1JnfU2mAZh/8vwJSR9sKpe3eCLd4+XO6pNr+lzZkg2VSax/EebykVzZb4mQQvt/+oDVJT/VfXqvLgEW74cgHX4jggiUCNSL+JPJJ8iJUtw awZUSG4T Eq3ntYyGN9sdzMVPhF3UcmOJr79MSyFpBWqL+AsGOv8ZWM0PY16sW0WComL043aXrUnXOM0GZuW4XctI+X9CxpOr3M8w4cmKfcEdFXEH5VzMXOFVvGLp8twy2LcOboTTqGTZPpqZqLdz8NXdQhGXFhp/1q9CGWW5BwdJA/Tnqn3Jvid7Ksk4GxgAGK2HcImjNc2Wf+fVpXkh27wVwcar9WSctT5lsaKojrt/biT4+Q3DY8NO8ZUGUXUX8HGiTM3Fcs+QGer8w6MH10UW53v4LTraEv4Q81+Jxobj930cGR0ByLPLK5K1x2AMSn7Mjm6+pksO9tM4N4EVnHLMw1j5ebRNT71aI+RIg31Tbkzw7brNwoOMMx1z04JJg72H4yIMdVFLeNcvC9wdywxsXXBzD2FjQtXkhT2cgjc6F6u69jV1Xz6lFedHNx4+HKIX2GPf06V2RKOqBPsWrJxaUdIAyjQXs0z36WU5bQ50RwqOGehDJOqvQePm7Itzk34qnQZlQiG5hrZz+u+RwugwY0SZ1XXMY51eMwW8ThHN0cNqtfMyEUMx/NrdQuc/r/8Kd0FzAOWy1Ldn329nNM4O4sQQ1GpLrwV10xDQwQx6NeM5X9fm4ndrnZEJvuBi2Sp/9wCeGx/x6UvuAbMH1VtzQij4o/vpQXe98rYnhs4iN2WZKK/LttJU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Normally free_pgtables needs to lock affected VMAs except for the case when VMAs were isolated under VMA write-lock. munmap() does just that, isolating while holding appropriate locks and then downgrading mmap_lock and dropping per-VMA locks before freeing page tables. Add a parameter to free_pgtables for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +++++- mm/mmap.c | 5 +++-- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 90bb2078444c..52d7e9c2e58f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -105,7 +105,7 @@ void folio_activate(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *start_vma, unsigned long floor, - unsigned long ceiling); + unsigned long ceiling, bool mm_wr_locked); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; diff --git a/mm/memory.c b/mm/memory.c index 7a04a1130ec1..d48c76e9fa57 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -348,7 +348,7 @@ void free_pgd_range(struct mmu_gather *tlb, void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *vma, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, bool mm_wr_locked) { MA_STATE(mas, mt, vma->vm_end, vma->vm_end); @@ -366,6 +366,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -380,6 +382,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, && !is_vm_hugetlb_page(next)) { vma = next; next = mas_find(&mas, ceiling - 1); + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } diff --git a/mm/mmap.c b/mm/mmap.c index 5bdfd087b632..57cb3a2ac9b1 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2157,7 +2157,8 @@ static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, update_hiwater_rss(mm); unmap_vmas(&tlb, mt, vma, start, end, mm_wr_locked); free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + mm_wr_locked); tlb_finish_mmu(&tlb); } @@ -3069,7 +3070,7 @@ void exit_mmap(struct mm_struct *mm) mmap_write_lock(mm); mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, - USER_PGTABLES_CEILING); + USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* From patchwork Fri Jan 27 19:40:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119260 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35A6AC38142 for ; Fri, 27 Jan 2023 19:42:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C027A6B009A; Fri, 27 Jan 2023 14:42:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BB3126B009C; Fri, 27 Jan 2023 14:42:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2A666B009B; Fri, 27 Jan 2023 14:42:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8FA326B0099 for ; Fri, 27 Jan 2023 14:42:06 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 296CA1410D3 for ; Fri, 27 Jan 2023 19:42:06 +0000 (UTC) X-FDA: 80401599852.04.1B884D2 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf28.hostedemail.com (Postfix) with ESMTP id E16C9C0004 for ; Fri, 27 Jan 2023 19:42:02 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kcxorgjD; spf=pass (imf28.hostedemail.com: domain of 3CinUYwYKCNEFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3CinUYwYKCNEFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848522; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1VGgj1TmnjNrGwceq1Y9mwtgHdomU/ZJ3zMBBp7twvM=; b=qJR3t697eTQeesYOysfl58+Y7eA1sKFDGZ4Qdexgsq8nwxaMnpWxM5xlcQWNzsHDbeDDy6 LK+fj5oS0MgV0pObQvu9bFEyvGVIggH8ZBqaY8J6+30i0MoOIDE40kiod5BROD5JTNI8Nc 5Fl6bx5SoGevQAAwVdkducXItU4mogM= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kcxorgjD; spf=pass (imf28.hostedemail.com: domain of 3CinUYwYKCNEFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3CinUYwYKCNEFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848522; a=rsa-sha256; cv=none; b=uFSzxfUWUO+VHhl3XUX8BTb5TYKXF/ReV4i8r7Xr72DzTUzuk1uoHOggfXZRjPsdpNb/aR 46NPPzt6gGzs24bUXDdIvxc1S4N/1MJtmdHSva+VVdqb4+xE1Wf+Z+RmMpyOsMJ1sgxf8J TP6xCricn4pNDEhganU2lk/jNyb0XrM= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-4ff7c6679f2so65762467b3.12 for ; Fri, 27 Jan 2023 11:42:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1VGgj1TmnjNrGwceq1Y9mwtgHdomU/ZJ3zMBBp7twvM=; b=kcxorgjDvieL4H0gsZ4o6ev4StdVYsVDA7GZVrIORQdjZKoFiGI1KoZQdzzvMshMrE snI0AzaYSvDLC9zEn2IBQumHO6CKLkrL/i3wJ9ueET6bm8oRn7QEPBX+BM7UODPO3hzZ U4MGRqXmViJMsS4HDNU6vnAmwplZNvnqe2D+qu3fGFBMwOJOM+QBc0aicDuQnfczJKX+ vPW/+j1Kiez+csRZc1jn4fVsV2fgBwTNZpRX4guJZVX8gsy3dFik2ggDGPcziixMOIIZ ZlSkjqSVG2I0FEWcD7z9JqxBzo/zAlN1m8/Qkt+nlr6qzU3SYltCmGn1Rp59psmpvaAD Fzjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1VGgj1TmnjNrGwceq1Y9mwtgHdomU/ZJ3zMBBp7twvM=; b=cBG3n7UeR+1WtIxXw9ZWLZDVj6qimQcGIsqLVvwhSKvZX16OFJ59nYsTwk24qUwM0P wTIT4sy1u4rfiaA4jIIEhzkRm6Wqb6ymrv+o7ub0fyE1HSPuBY3DpxOoy3mIOxlmV98t jNBSlBDuSSMACXBEvlRT68iRVPah1RBDlpS9mRg4cpnFVyfaxA+omaj7Y3LkqLPSyvJQ d+YmTGbfjbGYVa6DQtth6Gbpu9ej85Vc/eiybn+FNhQJZIbE6z/X3XOQ0AtNtqtcsPZ/ 4cEXjW3f8tiEp2EtVjWimv+uuejr85LqRaatiUQxFiM1dcC4kkbTv8gx/jIkaMBduPs4 CJqg== X-Gm-Message-State: AO0yUKXq7O/gG5OljyuxPBNVy7Ey+//BeRyeNggqtK6jDBD7wO1Nz+bV 4ujtbr+Ra33r6ZBTGSwmBZIBa05YZaM= X-Google-Smtp-Source: AK7set/jPsOwzlm+kUPz+CABbySieLCP3cL/8jypOFocWiEQOE8fcbH4Z91B+4ltIKP12TkxYenF+FhrAT0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:cb8b:0:b0:50b:1b97:f19a with SMTP id n133-20020a0dcb8b000000b0050b1b97f19amr434148ywd.164.1674848522027; Fri, 27 Jan 2023 11:42:02 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:57 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-21-surenb@google.com> Subject: [PATCH v2 20/33] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E16C9C0004 X-Stat-Signature: dgf5tjoxu4o3gguc71jhqgfgpbqarrk7 X-HE-Tag: 1674848522-298337 X-HE-Meta: U2FsdGVkX1+ZPkPUBwtEm2GA9bLDdLNAg4VPYFnU3YPnaeq+UesWDArpW0xgarZ8hJLlMIUoXFxB226c4e3UwvKh0jSgb5lT/OqUFuH/G9bdstqGihOzYPTjHmAkO67FhYb9gA7iIn1vtLb2x82I3Q+BPhSSvitynbjn9LDai9pD3t5inqhXJjmG6t13NAdgRYRs1OG+PCLyvXrqOqveVhU2CP5VSz9avcD/zs3PEPYn/OmwN+CDbFQPPKnCfe0WWvOIsV8lthEmxVbJSe4AI/WNPnjCg0FAzbGFEws3Ue8VM0GIMplwLYJ2iKUvUlhN8MsZJtArcCfu9uhwQW3EfHk2BO4vDsF8rnYinCcnn5FobsqM97h/2H5LNn6kMm0wFhifBs9RLwOv17d1FW7KGU6fE0rn8NgJr35qEjCG0vfsVaO8VTMNSl7Zo0hLAtTVNef+Glc5ZMhD3krGoLusvi1Z7tNE7+TB2jG9DdoLV0VKRnaOM2Sx4A0XpisZp8RXf7CmNun8E/rI384ep1KQ4JWY4w5Sp7IEXOjYV+L80/rhhIOekfN7VWDfB9vn0aYatiy0unOzzC+D/G3NSErIX+tQdcFd8w4GZCWE3kLtXQlVjCWk9b1hRw5A4um6tZ1RkmrQ6/I28fdXpd/h2M528EV4Z6rT75K0wHN0+aIDWY7GOcrdKhkXkIxG5wxN6r33ldjWMQ4neTq6VhA4QQeap2GqRfCVbaRt/F86lkL4Rm1QIKE/N+ifjU487iA7qD36YObozcrwwgEfBbr3ZRgoQqGXe/jbvmdb2JH3hfAUxRGIgnsybpvhQIDpHQgwJi/84C9LF3t3ChSmPJJqD/2iGXBOlPlYSfyi5+s6UU7Ph1jPS+XaHxzJRLtBmsP8MUsI08gfcGBoTrBpZ9p/AMF2bGOHB1gFuSqNZcOastJrzRnpJsP006MOlPBsqVm+2Szt1unETJak7VlLtiBVUog bQ+w5nzq T89oUv2+JZ/2sy+R0RwG+SCDtdDUmWSC6xcK5wodWihCkQl5CIi600MZTwHkaN85yrN/pFLFgDfdlkNQi18PdH1koOY1qTgmgyx/aT+GAQUl/DMktW0msOOoOh5BqpNAhyngwlxbxDVcQHEhbZYiBLqwh1elCfRIjLg6AN4weihqKApsc+d7McwXfDI4MCvzgOFohnI50XvEa3Umf7KDTVN3Llap4Bwu2oDV4m0k+DYAGDJu4WsUEFD0Qxtj9N10VoBd+v6xB94OBCtDcYQXIeIdiTPmRH85uSq0qyhBGFuJPyOMqmlQRDcBjmijKEAgJohOEwm/TF4S0JeCS52fOOBYZ7fF1C5BXYZ23tgbvk5A9DGOCRTYd7NDAiBM1MGuValG5K3CoG8E9KTJWdBtkfpbIeBgv7YYVlfdPY27EWPpP+PzjPPmgJSnRVNJorIVPksTH4709avZ0m9MRhPmOx26lvlmDtRSFOOgboav0aGMtfPCQ2bMngJgWhmh9TQm94FQFm3+7RYeu1jG28TR/w7mP4QxHaNF/iQGcQQXBHE7tn3JdL1f/VmcFBfVinorIUelqh6BICXNZXWS/mkUtwdZ48Vk0iyft9MZ0JlfDleYdtt/c1edXhpxxUg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.011723, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases write-lock adjacent VMAs to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 57cb3a2ac9b1..3baf218836bb 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2399,11 +2399,13 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ if (downgrade) { - if (next && (next->vm_flags & VM_GROWSDOWN)) + if (next && (next->vm_flags & VM_GROWSDOWN)) { + vma_start_write(next); downgrade = false; - else if (prev && (prev->vm_flags & VM_GROWSUP)) + } else if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_start_write(prev); downgrade = false; - else + } else mmap_write_downgrade(mm); } From patchwork Fri Jan 27 19:40:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119268 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E53EDC54EAA for ; Fri, 27 Jan 2023 19:42:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80A666B0098; Fri, 27 Jan 2023 14:42:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7978A6B009A; Fri, 27 Jan 2023 14:42:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 597226B00A3; Fri, 27 Jan 2023 14:42:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 47CF96B0095 for ; Fri, 27 Jan 2023 14:42:27 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 288021C6084 for ; Fri, 27 Jan 2023 19:42:27 +0000 (UTC) X-FDA: 80401600734.01.F971AC5 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf08.hostedemail.com (Postfix) with ESMTP id 57151160023 for ; Fri, 27 Jan 2023 19:42:17 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oI0LJkxW; spf=pass (imf08.hostedemail.com: domain of 3DCnUYwYKCNMHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3DCnUYwYKCNMHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848537; a=rsa-sha256; cv=none; b=rx1NMh6YP/vy9R8v5My589eTEuDG4+KnM42TL9LRaJdU7w342awu+R37Tsug12jvbBNdhP Z9N+pJ7te5zunQ0+tzTxBIX5rJHfYeGTK6oOjbdaizTXr9Jx5kpXrWSLgvwBkDjzfuZjnR UpSLL3z9gfxDd0p7YjVrLOj09Ytq4WQ= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oI0LJkxW; spf=pass (imf08.hostedemail.com: domain of 3DCnUYwYKCNMHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3DCnUYwYKCNMHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848537; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=K6v+2msp1t3wcI7Y++u8Ot7ePMFGPZVFhsB4FFAkTu8fArjpeA/2hURr4u7d1NVgBihrSO xp7jMOP0eUh+uld58EeIeMNTFbYAELD5kgnuBKpeRbgcQKicOctpriJ4dcoGTQdctcOlXf msH60TBwJ0fqcbt9NP7AoiBXL4KP9LM= Received: by mail-pf1-f201.google.com with SMTP id p51-20020a056a0026f300b0058df209da03so2813825pfw.15 for ; Fri, 27 Jan 2023 11:42:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=oI0LJkxWJWb4FwrADuA4kk3Z1VuG/PqmB+wfudzVRf1FutZSssy7j1ZDpJRsJeC2x6 AK+2fBcgvA6Z1yebvss3n+ODruOq7UlamXlS7DfsbRapPdytBdlVYqH5VQCwBS0bW5Cs 14NyRgqd5c0dS96F3WtaSc6UYIsY2Q866sZsgDc9kakHc8RR5OiJX50DOvF/naV3sW35 K5NQwDrUGeBVt7kutcZKVO5xLo4MSKoGaFWDyWGZ3caTDR6ZBpyZR8N/O+LnlyPsEjZa UEkxm7VSV5agQqzuF3YgYMg0fZBeMdeXvS4WgRk9vqB/9OKyGK8SEZXO7u7cVN0V9JOR wUhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=Ye0JF9ic48o4d1NnfPm9wxwL8u8+HWqkIi9TL9VoSY5k1bLJEON1rEWNHjMHRNSGH6 r/HABSO2Mk2QtUXcqAJ+56CGkv/OBk8LIFsWMYFBYiiv8lahZK8arnu0GAkyqo5QG/cp dFxA7Q2DWUFn49k0NMXEXFWJBCOvkgYI6Put+pulSziHjsrdX7IKT5erESK7iqqFzWNi b7VasN2k1ZQDG0WsxGtIWLw6jGPQcEUBC6V75RFKDtt7aaAeRWyv/oERrOfbccthekxk coATY7Lx526foYw4YoY0NvaL359pXxwz7dQI+NUTOren3VTsGeyBNUV0peMNvSl3flvv vQaQ== X-Gm-Message-State: AO0yUKUEwNeiIpzqZAhnR/7CkJf0csWzGx0KTPgtfHQz7GeYpx8otufu WSHsxKjoTBXc1erMbAnSelBMQWNCXNU= X-Google-Smtp-Source: AK7set/yd1hcYRvbMqSOfXS0bVRoPLBLaG1qWLcI1OwZ+QZcbDYXkFqGvC1aaqenThgkEbN36zZqWHoF630= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a63:ba1a:0:b0:4dc:3b99:6f67 with SMTP id k26-20020a63ba1a000000b004dc3b996f67mr1091417pgf.47.1674848524282; Fri, 27 Jan 2023 11:42:04 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:58 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-22-surenb@google.com> Subject: [PATCH v2 21/33] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 57151160023 X-Rspamd-Server: rspam01 X-Stat-Signature: tcwnnnkiadz777hno6ap1byq9fhjzq4y X-HE-Tag: 1674848537-573439 X-HE-Meta: U2FsdGVkX19fzwGA6QaDvabSCwU8a9h1GcyaEhEaNyjfeIVNGyNhyDLTM/4VsFHpbIHc1a5fGfLa9xo7tc77iAOw4mBEBMk/kTjvnE0SgRwNZIrJexAI9fRRrcv8YXhbW/1pm3Lv/fI1TD0ULsulpZr7ROvkRYjOZWLqwOKgUPV2TxeBikoqQr8j1VBAqe6kmQ04SbNiDEc9yvp2bvPDwtpHkGz+od1aIpyzLc8kqVjKejlnT3xxmr6FyQQLmIruGBGoFO5o+hqiPCXU8hvnyeHPTZha/4YSyJLhD2860IePBB6me5mBlmqS8s7fEq4/geK/2DRPHvDox1KQhJxSJlt/iXNxJ9iARC2pAoY5laRdH4rY8+oCLpGXvXnMaTkVrt5ekvXXarFQT/LroHlnq7uMAYVHR2GA0maOIv7+9ULA32C06TM1VUQTD1sAMCIu9WtoNJGjVvSptH89a6hnX47Cf2HHdtIBFAIrPdafcF9tq+ku4IXFu24+CwfoFXZlvrClRANP4I+1CVeJcC1eeaTdoBtqfd+co4B58qF7sOjzJVhxfblbvlrEiK79TY9oYbl2yvZEu+C20EjYiTJWRqvxk3SgSdENJRAbomFqvdF3ykFwqYTuDqEDY3oBcfEWLHMetmynxQm7dk42X/wME8MsAPvTmmm0/ax9cVRxiDE4BzZdk5ARyqmUvl96TPFEi8vI2IW0tNeJaqpqmXRGOeS/8aRJ3dcXpFGkBh31lhIx/5lO4DXh4FGUNi4VHS9SENnwCZ8StRYqHKIzGNGe1Cj66vgNKEj/y4Ei0LCJZL5CkZtJsJfZJ6A25b0SD0FqgBg+vYGtkBjgYqp06p1h4aHXLo2D2FnDpix/atbKiD7M9C+abZuT5vj3Lkrz0XaofIAEGGyB7DxkumHkCYjW+HFZ+M0ks+14zM+qKxzvq0HwRblv1yqjF4jfZR7LFcfiJ0Jh4TzpF3fozDtw2WA dEE5Vqqm IAOK8UqFQmTSqdwSxhXBJeOBH3+3horElgp5vzcQCGUdddPzVH8a19X5r9/GtgnM5k/Gnct5Jgum46qvBgbtcu8pNyFxScxzHH/ULgwSsf9bp1MOwBE2t13TM08KlGJAwicRiZ+ZoHPVKmVXlhnYQQSqSQ+gfVhu9UBK12WT6qU0LFVHkNOzuCm2Iv3AkVvXi4wcEN9NQ7w6aWPQdYvkD079YOJf/3bGuVWkMFtn6bD65DaH0hPamJisGCd63xaY99WghYzJ9AvzC+itbWMZd2PUHQZCly2rRAf97XooyB3Fm7GyETTYKhnA3iHeBdDtNYL407v4BgGXSWtTDOEMF4jv9i3x1bQsVfd/FFrbR8gHHBN6MtrQ7JmrOhn6HnnLSZOkSmMzuL+19SGhDYJjuM8vDurv7UO9lQ7vTjHJPsP+4BD6F27uYcI9o89bm1BUjrVuK9sIeAlFyixXV+qQEjwLyLm4elHXJEZpYa4PRWbI+GkYctwVYMkQkCLG20nwWEKWjgrUSbNjQYRTrSFKTWHZT+YDP84iZHUtHemXJy1SRWFdtPEIVDsuQC6+tbnPqTnXK6cSzBDayCzjDGajRfrS2uYPqAqzKcYncJJ8Xuq2WHKjTmxVao+OAOE8TUkTyXoZCB+k45bw1DYpuP5trboSZbWwYOzt2so8wq2mbyBhkr+Nv4voRjsTykeA327Eb8q69 X-Bogosity: Ham, tests=bogofilter, spamicity=0.001513, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/fork.c b/kernel/fork.c index 9141427a98b2..a08cc0e2bfde 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -491,6 +491,9 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + + /* The vma should not be locked while being destroyed. */ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); __vm_area_free(vma); } #endif From patchwork Fri Jan 27 19:40:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119262 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26B95C54EAA for ; Fri, 27 Jan 2023 19:42:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B169E6B009B; Fri, 27 Jan 2023 14:42:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AC6FC6B009C; Fri, 27 Jan 2023 14:42:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 98EBE6B009D; Fri, 27 Jan 2023 14:42:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8960C6B009B for ; Fri, 27 Jan 2023 14:42:12 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5788D121044 for ; Fri, 27 Jan 2023 19:42:12 +0000 (UTC) X-FDA: 80401600104.22.0AA3C16 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf25.hostedemail.com (Postfix) with ESMTP id 98A6DA0020 for ; Fri, 27 Jan 2023 19:42:07 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mdQhCODm; spf=pass (imf25.hostedemail.com: domain of 3DinUYwYKCNUJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3DinUYwYKCNUJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848527; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mMW5WtXlwQRVw5jLv0uMPkgP2YoyRT4XQf8L1G2g0sM=; b=cerlaqD3P48VUsrBMxZfW1vHTOQGl16Ec5twFwUaFGBRWuYDFbDakeiUUl0s5LyGQqaSdk OyER9gh6XlsQymwYe05FbUDHPfb0lTfJ8hVUY8/qxr4NKLZw1ti0q1BJFPKBcVgzqNbKVR sZDhIxVv8Mtm/SIEIlE6mACDdWvZsmE= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mdQhCODm; spf=pass (imf25.hostedemail.com: domain of 3DinUYwYKCNUJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3DinUYwYKCNUJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848527; a=rsa-sha256; cv=none; b=3cHWVJ/RJwiYzxxxXB/XnuHDBNKrc3Aa7i8Ovl9m30vWdItEZQiwVKgzYZWRufINRex2js 2KYArlel6NgEqSkgyOJdXiufkSRaJKQK38BP1uQsijOR7lUIBwuIEkwJp3dsS2PsvqBjkj BoIAZQcj9k/YBUkzqrtMhQn0Xk+AqXE= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5063c0b909eso64918227b3.7 for ; Fri, 27 Jan 2023 11:42:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mMW5WtXlwQRVw5jLv0uMPkgP2YoyRT4XQf8L1G2g0sM=; b=mdQhCODmER/WcuXnOwjGOPGsluXP611R3K8yAAqGfZO+eojPR0iLkDWsNYoGq20Pxc pLGndC0gNc0k/+qBk7J/uFwYPbCjsW7XmlHDcZCSFndQyDIXmYjY6yxKamOAoeP6UD5R 10GuF5qfONyUFHlhJRc27yht8CtL7+xZMg96kBN6doFn/8DSebIrGsIvYM1XfrVRdrIT Vzd4VicNtC2Hx0ebeUSEC751adwD7ZH4dP9Tg/rHSx32WXBUDV2TcmMYgTc4y3CEq8j+ 2IuM945ICEGJ6/3bquYB/9nBx6UvS9o2Tl6iob7NJEFRIs2YxCbcwKtUsh6O7/MdtiEH lFnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mMW5WtXlwQRVw5jLv0uMPkgP2YoyRT4XQf8L1G2g0sM=; b=VW0kyCuZvud6c4L8rQJYLf27PUiN1EJed0Yco49W7ODS1+1w9x+HA5ueyYTbvP2Vs/ f3VKABbz2U2sZSccqqEY0dCP/ueKAeb4TP4CXXW1TWLsTUcBfW9OjOhHIl5GB+c9s6c7 pVekRS1kawwpL550fqhg7fACx/Qei/umw55yu6LN/7w9JcuMY1Hc2tKo+yGhp+ljz2M5 FwqvH+gMe/ktgaDoISfqPCe9OrqIuuq/Ud7YJR0fJtNU4sYxqxBcqnrOFVs37bdhVayO dpWlefbCDSLzAVAvMS8QeBqllBENNjQTkkE06Xp+HUf+m5sjvnXVg4wWW2KVkjUJ7GxQ 2e0Q== X-Gm-Message-State: AFqh2kpDaS+pmANJs3QNOuxyr6m1Kp/9mEBzXm4aOtC28NMghksICJ8k 2o4ho3+IVswW2QQ6lCKEl68AjWhqy+U= X-Google-Smtp-Source: AMrXdXvm/6CR5hD1SSLODW5O6UqAFkTGTyEvJF4A+5Su+1PwVS4k4WIa/asYlNCGXFd3Q9zHSTloGdPuiKg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:1fd4:0:b0:7d4:712e:2852 with SMTP id f203-20020a251fd4000000b007d4712e2852mr4289060ybf.628.1674848526673; Fri, 27 Jan 2023 11:42:06 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:59 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-23-surenb@google.com> Subject: [PATCH v2 22/33] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 98A6DA0020 X-Stat-Signature: i1j7pyqjqjxdyfgatzns4cmwuh3iupws X-Rspam-User: X-HE-Tag: 1674848527-835555 X-HE-Meta: U2FsdGVkX1/VRdywUeS+Son15aCkxbR25Q8Hg4I7Gsi3DKauZvD1k/f+nVQ/qZekTo1b68Edc8Nx8ItDooWr4OJZ09xvx7pSK/MnR75gmR1z4ZhO5LoK4oxTGb45Z4hKzXHkfxJlrltJbcLQ1nq4JjzAmSmTPyKPw0ZOnKV276qnNhsCyW2suTrrA1K/aFmcxeDYLGfziW3QiXuEcFSErDaAg5YjW9/pik7Ul8/okhk3uXA5kGeKboqnFcXzXe5FKd1WCdXyL7lQP69eytBYC/tERUYWM9shuk7e8VsqmQsbWek+vVhiXNyeq5gYs7U26LnzES+W7F05ZIhG433cKUA2cDheKgkd5tyeqoTqPldi0PYy9GQMAMLMcMacrPouOWO7nDMfYDpQ7XoMDRtwcn2G1N3/FJBzo+m7omFbBrqNIAaPlDm+Dehz7QYuqaGkLxvyxEEEcvCRRsQc7WdmuKHukXPW3G5MGHYCdhaLGRQlK9XA5uDkResVi3CF6QYtgqqvHPINGMay/BkSTQ7uDNc8os2Ocb8WU3r2tA6zfeF7wGKuMfV8Av7QvxD2t+sLo/e/JuxA79RUUSulRn9E5wHHzoqC55QzQdWoxSFNB2LiGhGcGcWwkiQB3hhkRSq3OaDi1lSfX5v3atHb10xeLYmkBkuECgSKAU5QlzkX91P7/MXO1phhpzoqc/D74D3QltPa6INV6G5bzX3VhzmpznZzt/tAkP8frvNnFZy2X6a1VtQs/M37Q/vNvZpTO9wSh0Ne/QN6hLE1PSwUj49pDlBJ/NlghQe7vq+/dev+gzjz3H/fl4iUBls3ROwxJngoynWqFjXvuBaqqzpq446RJKDAAIpqje8EhONWQiQFNVGEM8qy4Tt7fC/PsEGfmi3KSnTRVvABP2uCXvoMA8lIQVh0ib8/ZnMtHytjLwsZk5QUyA6yVn1kU/p/JyshmJcWYUwMFP7vG+9emxUhLhJ 7MOT4g0W GRBIckLXbMxOyGWzOsxB+W4v9jYZpzNrnKV5kh9wAP0Ff5TgwY2cMKtfRm6epLmzA7z54rClzNnAZq4dxghdry3SLqzRoE3mPdsjtnAisb17RyFAM9bDyJ3aG0+eeCAP+SspPpBQF9L3hW23UZ0vdX/zTHhSeQXGffiAp9/Ph8lMw9WnsVznmeXSwjAh2crwzJz0Rt7xJb5Po1/0TdW8ale/wP0O4gwVN7PAgt9UYcNusXEDoB02ohEEmVX/xuo2abT4Ib3pRxCzNKGthaX8KWkm0FUQ1k/cl2t7/OCwLcyXJiPcbv7zMd46xiJxZ545DfwC5N++XrNKLUOK91IgI5EEU3Z1z8JdXta89vPg/cUAAh74nZLCL/43LYBmuaB2pIcI0y0TR2OVyVHDxNl+QqNx2kag9CIpvIP9YYPitrLipriN7ehI8SJemT4aDUnl6cpmdeCEWkiQsvPH1AzYDUcAJeXdf31jqnXxKW4QkeB/pFxNSDq8c0RLe3MoPMyBdlro+xcy/CQMoZgbkUhUXf5I1r8Fo0SPtR80StR0/jNTPVZX6SKmR3sjx+O5q//CQTN/E8yMjRSHmroTn3b9unWKYhMar0OXIG0bOnv+GlyL/S2FN+klqG7u/RgvCXIo5GctHHJk376A3nicXexG6n+herjXTY8sWjudoE72lqX+XAv0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Page fault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to write-lock all VMAs and prevent this race with page fault handlers that would hold VMA locks. VMAs are locked before i_mmap_rwsem and anon_vma to keep the same locking order as in page fault handlers. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 3baf218836bb..3d0cfbc92745 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3501,6 +3501,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * of mm/rmap.c: * - all hugetlbfs_i_mmap_rwsem_key locks (aka mapping->i_mmap_rwsem for * hugetlb mapping); + * - all vmas marked locked * - all i_mmap_rwsem locks; * - all anon_vma->rwseml * @@ -3523,6 +3524,13 @@ int mm_take_all_locks(struct mm_struct *mm) mutex_lock(&mm_all_locks_mutex); + mas_for_each(&mas, vma, ULONG_MAX) { + if (signal_pending(current)) + goto out_unlock; + vma_start_write(vma); + } + + mas_set(&mas, 0); mas_for_each(&mas, vma, ULONG_MAX) { if (signal_pending(current)) goto out_unlock; @@ -3612,6 +3620,7 @@ void mm_drop_all_locks(struct mm_struct *mm) if (vma->vm_file && vma->vm_file->f_mapping) vm_unlock_mapping(vma->vm_file->f_mapping); } + vma_end_write_all(mm); mutex_unlock(&mm_all_locks_mutex); } From patchwork Fri Jan 27 19:41:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119264 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D74DC54EAA for ; Fri, 27 Jan 2023 19:42:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12CD86B009D; Fri, 27 Jan 2023 14:42:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B5376B009E; Fri, 27 Jan 2023 14:42:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E28146B00A0; Fri, 27 Jan 2023 14:42:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CDFD16B009E for ; Fri, 27 Jan 2023 14:42:14 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9F5E0AB7D5 for ; Fri, 27 Jan 2023 19:42:14 +0000 (UTC) X-FDA: 80401600188.06.FF659EC Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf29.hostedemail.com (Postfix) with ESMTP id E4E5A12002C for ; Fri, 27 Jan 2023 19:42:12 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kp92vFUO; spf=pass (imf29.hostedemail.com: domain of 3ESnUYwYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ESnUYwYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848532; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=UBbEa3tIwjL9ZOQe0pgcqyKqlKpi3TpjCV7vjHK5EEDHYL7+zjcXQKAnrOAf4zWdISuU14 JUV9rCl4YfQeXTnxz8doBDN4cjSWlTneukoUvmGP6b4Kj6aoSQXqLM3rDv5zUtDjqwQFn0 YW4Ninz7pl/jmL+DV6QeHNMxoh6yDng= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kp92vFUO; spf=pass (imf29.hostedemail.com: domain of 3ESnUYwYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ESnUYwYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848532; a=rsa-sha256; cv=none; b=OvQTFCR+gjERuBSHzH5MkTaw0TnmKZm5f08lCr29a5WzUkqpmIfmYw1Fea0033KFtH9LJ6 1XZFMsB1/nAu3HhZI8sBvjkdZQ7Ui79e0QuhDyW5QzQimNCxSb4F5DQhYWizgcPwsVLc1e 5S760Av2lvSAfl9WFG8ZtJWd3SFjHh4= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4bdeb1bbeafso65149737b3.4 for ; Fri, 27 Jan 2023 11:42:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=kp92vFUO2N4rCYUenom2PgB/dU7qqaylHAcH6HVVMYT5QMpL/oqQ/DdOm4bVNlYLNw 9rNSkse+dfgOJKwj09yKdpzV2qc7upA8mjHmEpxt3RKsaa3pDRks1QGh3Nb2hG4bp/jU 3eSArvNsdqtxNb6zpVi6gl2vf0iZr8y+PtaslVXMeV9Pu9xS/EmrXvCiajKJa9iCZe4S LpK/JO6vbmew4JxbSzbZQyffZL4sOt9sBlPWbT9WmPY7+pp+Ff0i+9BT7l4WdUaIrrgA btwhF2XkX8IUP7VpoyJ9OAYm06ddrZftAFhV0gBzofVRmYYEJ8dT0O9gMwk7XMh0b2Ct qw3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=0GPNSI6XBVIKO1Mymuzldkrdrp/yVyDhB1MLdgcnUEoO4wGBsdI8baV5l3wEP0DZrr lB0sFUGDK6Ot0Z+sG7/uq5dqJsnoEnEwDXvjNaTY0D/Vk2hsKltnPMH8c0yH/pnp8Edb gSM828eaZpA7GjlEqxvfGEorRA5SwKaqq1JT8JMRvQTZiPndqjypbCukIyFBViJFpDns 2kPG47yKSFC7TWcY65Qi1T/6UdMH2MTQmqSmspkVaugeTlwghtX63tjJhk928/DCz2gQ IcpZRM0/BGxIekqhGfswHaMovcx18qKjKkAQMZtK3zl/8eq1e+fe/VDtxS+1ET88aBAf U8Gg== X-Gm-Message-State: AO0yUKXk9shJvk2LPrwbYgf/s8HgaGXmwIQBmw2ZZTK/l169d7pvY/PC IpV63/9IHnh97vj8U/cYmyPLNU2G3ew= X-Google-Smtp-Source: AK7set8zsvT5wkP/7moSYA7dIHSCpHQad2RzXlV5c0Wh0zEZrq5yaOQZsQ7iykW9DnqHdXZz8PLHMfkZLc8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:6d2:0:b0:50e:e91d:f47 with SMTP id 201-20020a8106d2000000b0050ee91d0f47mr27189ywg.61.1674848529047; Fri, 27 Jan 2023 11:42:09 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:00 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-24-surenb@google.com> Subject: [PATCH v2 23/33] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: gw9khnpqb6sq7gt7gn3pixh11g4adut7 X-Rspamd-Queue-Id: E4E5A12002C X-HE-Tag: 1674848532-188511 X-HE-Meta: U2FsdGVkX1+CDrHj03saT6xeeD3Kb66E/+Q27wuigL6bv/8H5xZuccRrLwaDrUx5lXrQPHAuhs76nFQb/ik7SQerDAwbAuar6Yi3mMoqK1UqiN0aQ9JClpNhNZScWNh7QQLF5HiPgSxo63zZcS4C9EX5Db5LEYCDphp3o8ZoEBkwI7v3MNWHQPoEz2OLCyrJWeFkKSOL0RIYTgNQPxZFwABBsV7ba50UA/Hnxm7oWa9Q3Tw0rAvDg9s6AC4fepsBK0O1EeLqOzlnB401UrwTjBWxrKJ1ElRO8d4TinfT/Ck+oZxJSsa22801uUYmVVT71U3pWNfFvgA2dfQZCRi7yjt9nc9l5Gzjn4rtaQzhqEnGvkRBIhrTMKWlUNtUjT0w1y/FN9J/E74/bhuzC9oQhuivDmSUe4pGXb+PcC6BogNWqNDu2jvOsolWPCryjk28A6E6eDpPMKngfv2wQhHB0EeTOOddEmFc/+J+Ol769ncYBwxiSUqoWnK59GloavOQpj0yJq7eCpXtOR+QVIdG+RCyPimNGgktiocXsuV3uBC3sCJJV6X3oq5XfK9gqhWpF3/L2IGWGX1ukylr+GvqTgD5KsCxwj5rtw3Kmlz+VI6/n9ed1BvdtyIFvlzhxsGY1CES6NJBuCERMNjb+oJ+f+xC4QV674XtF85fvKtx5KOR9qZJDPNo3nGm0pAAnOkeVRQrVPrGjGBZKXcXV5qmGoGSCWDVrPW+bVWQt7Z2bkWhS10bxQLuRo+Y+Te0D5rVC+h/jYLiSrjdA1LKjCzCMjmuJZaAV87joRViwB/gIzD+wuDtvEFaAgARnpZ54q/a6zQ9JHr7qINp6vzyLrjxqAzFEyvHnpyDD2WVOkdb3tku/4L+uJsj21BSbM1oBD6D2ZN+It9QiwGXKYyKIRWJr2lohQ5xwjY27FFTjsRljoZksNjiG26Ua5bGZkQjYe18n66uWI20vdQ8vn6SlXv Dy9C3ptQ vlKQAOoqYMmiM2cwMAh6a3OMK3OG8XK+Kz1RDsDCfhasNi59b3PxBX9TDVSi3DAk53S7JaLiQ75eYjVyF0jpLcwclZtmmFEBjDiU5Bu+02NTeZ4yFwVSKahxABwusfmetSshDBjTCDIoRiVocdZ3bY4kzgp9MU3B+qfjCgVgJMDQlxux71lIj/vzTFDbNn5lDQVN+7zXuv5bpXosaVErnRCzR4DW00kpdd3BIMcwGQcCH5ve/A/N9Z0eQoKBEhWlr7UtbLx0Oxo0BOGp5ztCMKmLe4wo3Vl2X2gy2MB2D+PGPugA+kun1OaAo+Vjepfa+Ou3MdtqwnFQto3EcllZyj6JAGy9vlQDUiwUA8Vs9rnz/hkFR0bkXeKgW5xWAyWVuyRkrYFfAnEGZ/TtQp0fFLHNol+BNBg1FTmLh/V8QjdSDyIdlDsEEirXGJSE89a+hlNTXDuJyElBV0gAMAgQMAjZesPpDbOuv29TkoufxYYQaeNju9bLG1qc+CXJxRpVA6kmAL98kJD0z+RYUitY4OphoYLZP/3V32hxJh0bOXdPF5xzkUsKTQJzB4uJzPi/Jmx7xs858noHr3MSIZkVs31eHojlDpyPhctSBWp5aT7URNsaXISoicxoFEv/jsC7APY0ugFtgpD6Q1InP1G8c4KzHc2IPPMCUSlZ1gs/HI5ljMzo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 66dca140695e..fa2b9d6e665e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -687,6 +687,9 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index d48c76e9fa57..5568fcb0a46b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5222,6 +5222,57 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma, *validate; + + rcu_read_lock(); + vma = mas_walk(&mas); +retry: + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_start_read(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_end_read(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + mas.index = address; + validate = mas_walk(&mas); + if (validate != vma) { + vma_end_read(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + /* The area was replaced with another one. */ + vma = validate; + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Fri Jan 27 19:41:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119263 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39278C38142 for ; Fri, 27 Jan 2023 19:42:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C00AE6B009C; Fri, 27 Jan 2023 14:42:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BD9876B009D; Fri, 27 Jan 2023 14:42:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A7A126B009E; Fri, 27 Jan 2023 14:42:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 963626B009C for ; Fri, 27 Jan 2023 14:42:14 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 621921410A6 for ; Fri, 27 Jan 2023 19:42:14 +0000 (UTC) X-FDA: 80401600188.07.E934F69 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf03.hostedemail.com (Postfix) with ESMTP id A2E1520027 for ; Fri, 27 Jan 2023 19:42:12 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=rcBVm8w9; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3EynUYwYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3EynUYwYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848532; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gSaEMik01Wf1+Bdt2RmdUMjgYY7lDp9hLxJUYqXXyyM=; b=oUMKD7GJRkPMmu4Nh20AIcyOPCN92Rt0U5jb5RJbABpWyz8rQqY7KsfV2ApgFnlVCUXrDI MSYhUm5/mP6s81bzezKBmhMzNwDvO9E0HxNcSSFBstR7nzodgB5lryPuW+WypJWOhCCaBO jF094/arHlQeZ0tF9e3h47iowsgf18E= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=rcBVm8w9; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3EynUYwYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3EynUYwYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848532; a=rsa-sha256; cv=none; b=0RyT0Y7670keStWZ/tsAUVZa6wRFaMqEDPdsim4SCZNi7kXjFRiFat5RIbER/ccpAv7DkS kc7pbp/DYMe1BF3zkaeviB3Lp4umiv9+LPB/kGymrWE98iFMMXwedul4Xjv/bCMbG+NMrT 5ggVXsq4Spou5YSSqs/BAYUgR12NTqQ= Received: by mail-yb1-f202.google.com with SMTP id z9-20020a25ba49000000b007d4416e3667so6420750ybj.23 for ; Fri, 27 Jan 2023 11:42:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gSaEMik01Wf1+Bdt2RmdUMjgYY7lDp9hLxJUYqXXyyM=; b=rcBVm8w95my/f0ebkBHOBSdk/g3dXIBnrvyKhE+0lfp8SBBL9UDohndGfm9prz69Nb 1VsGdk5hVjbZdd99cfs1kRzqndkhM72YtTrleRtcoKwDWYV7T956W4hi9P14DNVnfcTE 4dzd8voX/MwzX+niWcYh6zC2gtPII1RsQu2W+o1SzbH/VUzyZkZ+K0mbi6bCuvIO2r/9 q8JUw2FMeYJjbLjibdrizaV+CI77RGRlLfF03fxNpzuIRE1ymkIpYCAH7a7Vh5p4LTWM djOGRW9GCrBrxw2Z+lpBSKPiwjL/b62fESDSi0yRTpPNMfCaBkk0gvqmlpNjM1674+ZN MkCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gSaEMik01Wf1+Bdt2RmdUMjgYY7lDp9hLxJUYqXXyyM=; b=LZgSwosTaTvqmpTEt24ZDsDR/+Ki6ZRHXd28b8seS8ToBIB7PkuE4vDRRgBaNnhw8K Or/x4A5uJj3fTZj1pOt5w/GHME4G5AiPmUqo2o7z64s2hUtdkRsupUk6LpEymlU14eBV h7AwUC9RsmIBTOjYcNBOGDD+Zqup0YtahkRSglyWEGO4rDVrOjiWGej0fO8P68C3j31x zx5YCkcBhH2x1sJDBdyaurTyexFq5dZIc/8NAOX/LqsBHpRA12rv63tr4uORtPlqGnJy +ZKgfLwjlmZSwxEjbcsbkDRmzCJVP9wsvfJFSe/Oc11TlXlq7qHQL46XcRVPlHTTLQs9 X3JQ== X-Gm-Message-State: AO0yUKU/0KnsaNnlt2qOZwTtb7TmU7xEaaapsg6uH7Fnj3fpnwPfUtLo SFt1/UzxklQzfarjQXHph9vmc06b0g4= X-Google-Smtp-Source: AK7set9ZuZbDGpVLmgenU71+ffHh46VvnF49YmmPOU0QZTkBBursiC5h0isQQWJJC0fjjZLxRphOeOdStg0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:d8d5:0:b0:508:1348:3e54 with SMTP id a204-20020a0dd8d5000000b0050813483e54mr1003857ywe.440.1674848531551; Fri, 27 Jan 2023 11:42:11 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:01 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-25-surenb@google.com> Subject: [PATCH v2 24/33] mm: fall back to mmap_lock if vma->anon_vma is not yet set From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: A2E1520027 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 943fhnh45myrf99cg85y95s84ja8z5t9 X-HE-Tag: 1674848532-467022 X-HE-Meta: U2FsdGVkX18RP6pqBFIWRuHkCqyoClvEGCF/XzAsSHIN6n9HGlSy8nXRbJIekYGTLyAPysd/bVGYzLHi9CEocrIBxEHybUjsHUjUd73dmYpff4ug39nkFo7gOtK3cFQrW1YdRv8Gu+AHhkUPYWSo3s/v1zIZWvpblfJJOeIz+mJPlxXIexmy7Fzolcd6FXCYUPEOsYE6yGFCu3lL2MYfASfKh9AUWbTXByAfRWgC0mOU5SVtxrtJUZDzUiZ8egUdSl9L8TLaMjE2uisS2w1dlO0tvVDbZiI1DJI+en2/w0QJ/Yc2NPoHWKsOiuEPcSUApp2aj+3FT6kE8nlAposecVmpuIniM6xnsg6SW8rQf0pJ635u+fQ+jmblytllRDuNgOOzasVCud3UHb7JXbBuBrIqmEZYPB+oj+hlSCPvLK58r2x7x/RLHPs7phtq3CaTYAyc3gpOriXNdHW31QUknVNIIcYAsVNk/WPZ5H921HqTmMcZlos8tS6e8Iu9YM3t5RyNHciThiKcrgRT42fOghnEKEUjYQRaXGsYzszp8Jlqz9+Y+A6iv0C7wI5gMSwOPU3BM9xlpm0E5l7gva5M6rGVyngeAyDT6PDqW2fUjU/W5dWoGQy1oVbDAp0Q5F3H740Evn23WAVO6jqspoDmVTavOyniNSEN2ke+ZWnf/944Io+s9nAdyU2JLjNHHBQaBpIAOSCpPC0VbW76BQEFSG4nnPNhugUp+cwjNyLarJtGamWwqyp960wm2ekCSHuz7hfbAdkbMkOQVanaoDthnpEm/dXGTV/MJJJJ/SdK2gYbpHpEblXAaafnw6+xk0bWQdzmiYKx0O0frVOkkCMQquQbQ4Uf21HXz+gTI96B/+Qo6QvpmvCtdVJrzJ62ASvMtq3bYLqtTroLfvDUmMECYG/pAyGhXU7hBiblrx4PwDsolOTdK3HQqAdEHKknpMAsHcS255RyVAuHHL9cs0G 22CC05RE cI42/1NgITlgmloydRQYbg/zJhvp9nfIv0fI3VdFhElagDp6hyyDsmLUdnOXJdGcvZz1A6obcVveqYLZ/V69j7TYdTleIk/G1Vztj8iS0eLLSeFwf0rVvb+syc2Btf6pkZRa+FpgB7VXdTinczcWMMWK6yiPwsmDBgj9VOoYEWnD7MW8y/9uZ60rcvdhxrM0OAiFNyAt52UrjDqlHydr7jg0oPGVqu6iZaoKwp/gQYP9hjRYSVG9p0U/ZqmZTIboHRJ1ayadNgLMXbY8KKZtAfbu2vKKRSl1K2FIBPzkjbCVsa6kKx+kHFFB2FqAFclGssKTWAtizBmzuQuQxeLCTeHDiBomzN51GuVhED1Zr3/YkmjHZIfTf3JjlutN2gpE/iO/FEEgr/p2BxVuoj5ln9z7oVCZf2LKLWmySWtPWS7WXjWS5V5ajjD3945coMcJK7hBcOI3VKglk81K/qFO3/h4JLK7rzyvQM9C3Yo1+UQEG1/7BD42VUP9TjKWcjbxxzSVd38j6SxOwDmHm3SK/dWZPpTEqp9YZfCNkyTPGMcdUCN3efa2qAshl9eVeiEsY0T/Vk/pEcFZEfyyEqQezBfoI7iP0cjy9xT5CiCZ14BE9klMGLgXSDRhFQJiEP4yuZr2asxshSxHBpBF+IVDJO0BgjxnEyoFcgCQm5Nrf2BFDlrA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When vma->anon_vma is not set, page fault handler will set it by either reusing anon_vma of an adjacent VMA if VMAs are compatible or by allocating a new one. find_mergeable_anon_vma() walks VMA tree to find a compatible adjacent VMA and that requires not only the faulting VMA to be stable but also the tree structure and other VMAs inside that tree. Therefore locking just the faulting VMA is not enough for this search. Fall back to taking mmap_lock when vma->anon_vma is not set. This situation happens only on the first page fault and should not affect overall performance. Signed-off-by: Suren Baghdasaryan --- mm/memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 5568fcb0a46b..593548f24007 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5244,6 +5244,10 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_is_anonymous(vma)) goto inval; + /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ + if (!vma->anon_vma) + goto inval; + if (!vma_start_read(vma)) goto inval; From patchwork Fri Jan 27 19:41:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 243CBC38142 for ; Fri, 27 Jan 2023 19:42:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B644C6B009F; Fri, 27 Jan 2023 14:42:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B14806B00A0; Fri, 27 Jan 2023 14:42:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9DC806B00A1; Fri, 27 Jan 2023 14:42:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8E39A6B009F for ; Fri, 27 Jan 2023 14:42:21 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 6DE99A0570 for ; Fri, 27 Jan 2023 19:42:21 +0000 (UTC) X-FDA: 80401600482.18.46A5B65 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf22.hostedemail.com (Postfix) with ESMTP id AFC01C0007 for ; Fri, 27 Jan 2023 19:42:14 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mLwUPJ0n; spf=pass (imf22.hostedemail.com: domain of 3FinUYwYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3FinUYwYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2v/Evza9dkU4Ne98Lm6HAWxL8/YazIwD41pmx7joRkE=; b=wXDXLMkrbvBwaXAThFYDmc6E/UPY4DXcJYCQTdIKBSyRt55Znvukcz/sb3hksL0Ah1Y4ph h18hVFpmhmcvj8x5fU4OxyixrQtuGIBiU02RA3YdUc0mxmMlwCciaIy6vCrRX1CliSQQkF DuLcmJEmN2lGEsV7kGD7WZCsaVyJIGA= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mLwUPJ0n; spf=pass (imf22.hostedemail.com: domain of 3FinUYwYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3FinUYwYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848534; a=rsa-sha256; cv=none; b=LiwmYv7X8F9k+TXzVqfF1H9l1MunZe1XuqBoR6ZIl7rbIUY/vKinMt5DsfU/OfLLTkkHoR f1jlgqaVOHisFK6OEcPgs0+k9d9yjKhZSz2jTy525QT8VoY6ArYvJkMJVdvBouG2a+ImNY cLGKRlprwLF8hfV+6ESLSp+9dHzUjiY= Received: by mail-yb1-f201.google.com with SMTP id d21-20020a25add5000000b007ff8112e8b8so6430317ybe.17 for ; Fri, 27 Jan 2023 11:42:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2v/Evza9dkU4Ne98Lm6HAWxL8/YazIwD41pmx7joRkE=; b=mLwUPJ0nc0ulmmwFoyR9amwtEYn4GV75pogWN7ijAK31MqrZCBgfhijsONjhepzopP 2wYY2SkQatHOd+vKpDSxYxHFT1IvhzZTwDWuTcrFWygojUbcZe7xPlEiO9BQnq0w9fKe z4cHjJJqSwoele+PetnaYF8lewRv+be+h1Bgt+SNNsmNqcyEHhiyYbNRieovHRIQnY7Q KgzOHSrbk4QlZX3GRmHUucd8wrsetfG2ey7u2AbUsjqg+O7jZuWXERmenSwIYYomTRZT 0AcxNZ8Sif0L8fvqzJHXb63zST/Gg9+vMF9oOa1KQZBohhw7ZSFfQ9eLw7FWbw/nE4Qf wodg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2v/Evza9dkU4Ne98Lm6HAWxL8/YazIwD41pmx7joRkE=; b=gZyeB09WEjsGN5CKnG0/uy1qMAEiyzoQAnOvtD6XjVHgfbMQjFXJADh7nVqpCqyhTb Q/gcJOCcBiaGUCJoyUCrgIf7ZUy/Y0VAGikjh8hS90hhvBlnqa31gCtHHse8HgPjazNm euo4UOOp2dUZi1k7+lkxJYK2/t2Gk0rIKPPDV6fKJZqk1EvaULpwua+R/Qd6wBZPldTl hlQqOcgIoIRJm+lpnHedNhOPQ4khyejTa7eyDK4gUE6XdPGXFaOUSEXWOqjb8PM8bmqb Nx/nFOQBwspSHPl2vzRUnCvivTLnPud3EEIJxIM3C/SxCbwHFT8UY0r6757Xg5s3PXfP pfZw== X-Gm-Message-State: AO0yUKW1EmyglXTaqWR2zYsp34lWyx8/N+HuMY37CuuvPWqPhnTvVre3 LwA2yh7hHOzobojTHMfEYs/NdsYQLcM= X-Google-Smtp-Source: AK7set9kKqDiJe5U7EOGuYIrwgRntRqTHzt+zr2t5qOdc5+YeQS/lL3tZ/xnNoww1dZdAcqE9B1Pln9KCVk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:b2c6:0:b0:509:5b20:ff22 with SMTP id q189-20020a81b2c6000000b005095b20ff22mr917640ywh.328.1674848534136; Fri, 27 Jan 2023 11:42:14 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:02 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-26-surenb@google.com> Subject: [PATCH v2 25/33] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Laurent Dufour X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: p81p3stjadjgxtmtgsozrt57oirzwegz X-Rspamd-Queue-Id: AFC01C0007 X-HE-Tag: 1674848534-537163 X-HE-Meta: U2FsdGVkX1/AcyRy4ExcehmvQ2fgUHu25eCgsFCq6j8NqA6HTJnKqet6h3ZF3gcJSendxEtgxI6AiHbZTW3ewU8PGOLGfD2V83Cy0QWNk2rFVSisBKANmwNbtQWxCXFr/pEZOfC/nRqXKu583gKRa61J+9aub7xFJ+z5YSDBJr5yfr95At40fZrpimOgT0KhuHf97NKjzW/VEaCab9pWqMsNRZz/8SEdrcRkSNa/vCD0G4Ysi7ipILHZHV8LNRlEYbcvLopal1sitOZ0LP9XhukoIaj9RMD4j/CWK8DcICgU3XEGCjX9oIgDWPyKcT5hoFOBDfHfTyrfddEOZE1SaExA4zbSmjhzhW2qSVS6vRKyceOaBJkkSQ8URDRb5/ZU4WgnErzLXCwaPCIscRPn08tveRRwzFvLEsgi07lSj4upJMBCIhAT9gfdvmyHuxmb5gf/SFKCOqUNk0ava23CVIroFQFcclP4yFDTjXmIyfJ9LDy9Jykb0uMf0Qa2yhcn502rbju21FHD6lwEjMNbMb4xUCQQ1PmlLrD2VL9VQkOVbOCWEzvDFnwmkeavcpGMSWoQGgjFEjeg9Ne+a+dfsKYMjmJvR++mPgggg8Rwi7fXJ72EK870hiAauPhLgEc2BJ/WBXGHKs3F3LNQR8eQoUchnieHFYghNHzgWkbujSMBDc+0okTW1CHKFWpLFqkUGux2uV1qRfx1VzlWlDjP/lY0oeZtXWl4DFi1LMVx0GBoeYCXAPfK9f+1MobUJiGgOfoaoCnh5JPPtC5Lj0KUB1qFFwKoKSfnV1dg3dWNRqiEwjOb46t2GySOShPTK9s+F6koJ2XmjCCUk726N40AGEeiteD7Ylm7WoffH/83LJCR2PZvSliuNGwWa2bZ+96CD8eFy9BepaT487uheQ+qM1iOge2ZTkFaxQ4IsAnY1MdkWRBkuIwL0i73YD4hFh7DRC4v0Tgdr8KSz2DxyOG hUpzwBB+ m1gdHwFMByDxqht6bcdTSHGIHxFgHZx1Q6Ah3Zgcnn3hJgdI2rBD6A6Btp01ps2bnO4oLPX790mCKBOHaAij96YwOiFDz2AsgqJmROcs9QnoxKrnO9zk7xuisl1rj0ZXbyPhGhymiWXb4wAJD/lEWLCXp5boKNHi5LLDirOlVKfLrFKRgst6KJqB/s0OhxOOw2k9qEoRl+sUdZj4oFMAdvUKqciCXbtR3bb8pfOjqxYe5ao1zuBLFrJMNbylbUVrgPye2QX8Jr8b1YL+cXn83dZpuJCd/7D+7OtOosswV6Oujwb6BQglxwzowjW2VLZ5l9wTm+grYadVtaLFcx48rOz9gDsya82az+5dWbPmjzsBIrhLOSoh4Pf8a0RaZWuMztcXpQxyXT2M8ExHy229qFoZvTuyU+S4Ua1Uvus9rjDSpyAtxN45B+58NfNyT2sN4HlA3KNDQTw0yUDwM1AdJ00toLeUw42VeVexPn4Mq+jpzfjwGJ2TWhtTN7E9dZLa2ZuDe7z8OmQP/sWMEbh6TfXcI6yfedCK7uoLU7t+n+qUhDeIBai6guQFTk1Wuo9K3tR6e+tlssc6Q+NBgjoI4o3cuh7lVu8v8nYiMtAJhvp+o1ENV6rjtBwYRFsuEB16Aqvzdh8Vs0eRn9W0tA9kxG7tFCdRtbdcx+LU8kjuX94Dxn6X3Hn0i440RGFiSY01rVNszDfrO/mI4d4c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000018, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fa2b9d6e665e..a6de58bb40c7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -472,7 +472,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 88619c6a29a3..c4c43f10344a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1053,6 +1053,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Fri Jan 27 19:41:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA8A6C38142 for ; Fri, 27 Jan 2023 19:42:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6345C6B0078; Fri, 27 Jan 2023 14:42:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 598026B00A3; Fri, 27 Jan 2023 14:42:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 410F26B00A4; Fri, 27 Jan 2023 14:42:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2EBE36B0078 for ; Fri, 27 Jan 2023 14:42:25 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id ECF29A03F6 for ; Fri, 27 Jan 2023 19:42:24 +0000 (UTC) X-FDA: 80401600608.14.4E4ED4C Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf23.hostedemail.com (Postfix) with ESMTP id ADF5A140020 for ; Fri, 27 Jan 2023 19:42:21 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="KWtfJ/7x"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3GCnUYwYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3GCnUYwYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848541; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sBLmwmiYmTj+OGGVUcLxhx0PU/jBUoUEJpANG8soAfU=; b=i0NoHPm0+EiqB86XK9A2n436t2A2AVKfc0+Poud9EW9fC/l9yeUKKPu3hG0McZ11K2PQ1L GJXJmE2iulK05YHOETFqLV02e/BSupunZKRaTAhg2wfaBiYFmpYMv6fZ1o5g9Ijn51GXiJ p6GLFV7sUfKS6XbsszOKebBxamfn6S4= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="KWtfJ/7x"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3GCnUYwYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3GCnUYwYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848541; a=rsa-sha256; cv=none; b=pFZ3nR3t2CK3MEbkriy5QhKXZWbZSuaTxz1f/TZeYFXglAbibMY6DliBGfyYbsUNsu7Fh1 q8kJeIl1IRwAMSwW83JdZoKp0f3gdVibl0jHqGnacvLhDmlfgA1oDd9legoWy4Vwtd7sXQ XLYYt6RspZV1XujiGHFbjInWwAgQkdU= Received: by mail-yb1-f202.google.com with SMTP id n194-20020a2540cb000000b008038647d9ebso6356617yba.5 for ; Fri, 27 Jan 2023 11:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sBLmwmiYmTj+OGGVUcLxhx0PU/jBUoUEJpANG8soAfU=; b=KWtfJ/7xIlPNyMbJgMly5Sn2+77Pi3kT3bIKcXhSEYdAbccLPUNTncstNEX89dvAhW NM9ImIflxrHOdJ1Mj5fWozRK69MXYGw+crQDfh9Oftap/iRU1kAVPiBZhr/+UuvLj4+U 4LLcR3mdcWgS75oxY+Q2vGwsrXQ6jEpwNWpRZ0m4MKHrCT8iGAtEZyFhrSxdE7CjPr88 djL5aWWS3cMBdV/JlcmaRg0LM4Y33l8LwLfoSQoWCllE6I7RGCcpBFSNwKknGwVlLQii i5R/yajGZAULuwac7zP2AeGWc0Ux4Tm1rMqU81PAQU3nWhlQVK4py4JD6fzjGBwa6RFE gFWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sBLmwmiYmTj+OGGVUcLxhx0PU/jBUoUEJpANG8soAfU=; b=Efg6UtUKWs/4WJLqDbFb8UEXh7KJ0ihB5c77doGmS+KR0RCPM56IRSqdK6x6eq0Xoa 2DNFWnYx85CfROCy25gC5kuV7PSi8pyAmvKZIuG3ySnXkVwniHZzhuwABM4VZelmvfem fdO8xKJ2xjmay3f8atn3W2emUcY4quZsTVek1g01A5uzYzncEviOfSDbhz3Co+Fuwl/5 Uz1Vq4vg0Ra+k+jhJj+IIexdGRYneIxvx8H1NMLGbEpfPj5zMqKy0/bsrzj18DvZtx0E R15HMFBN8mdWa6nM5ozpYvccY5aidwvForD1RLRKf53o9v5lhEOh48FqIGcTYVMvfAPk ZN6Q== X-Gm-Message-State: AFqh2kr11PTQB7+nVz21uQecmLSUYViZhpCPTyoH/xqwKQS50qHG7XuW JOrU83S2mQN/ZdXs5lrVdOqsX2mcBL0= X-Google-Smtp-Source: AMrXdXtj44VmRN6kQLB49pXglCfjpmg9iXCqYgU9gl6+Dl4v+hAma4/0PMDtIg4t8IMM3q4b9OalQWjETEg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:cd7:0:b0:4dd:9ed1:fba0 with SMTP id 206-20020a810cd7000000b004dd9ed1fba0mr5011323ywm.265.1674848536283; Fri, 27 Jan 2023 11:42:16 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:03 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-27-surenb@google.com> Subject: [PATCH v2 26/33] mm: prevent do_swap_page from handling page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Laurent Dufour X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: ADF5A140020 X-Stat-Signature: ri1ko7f7t7bqwqx4zd8u737hdacn3z9r X-HE-Tag: 1674848541-437479 X-HE-Meta: U2FsdGVkX1/+2jmO9FS69/hZTCfZwnlSuDOFsx47uvAJMoozAtHahe39ZatEEPYMsi97v29p/Xg8FMtiHgSwFnblRopdgsmRC+U2LTJgA4gOsWpzrtEYyJP0lIb6ojIWtv0yx0WpMI4h0F/xs74ydxdj5qJa52e1V69ZzDfJQBn+fISdaU3QgBwPaGcKk6SzvqpVh/lhtYtY1ZNMkpdXK+K6Pfqi4HB/fuZUON3MhLqPwlyVHL73cr2nSMcB7120J77bnW5nz/5V4jssnOnO7WutlldJAsgNOZC1m8HdqPy84G0DsXygVjDTt6vXgAkQDJ8TtMyKjxzXuxfw6YJTpJ4k7N8BoAfytF9FYpVzXc2IEa73pGuv/ZDxg9zLXz8iP5ScOTqFyTXGVFP/mJ92VLTKp/kuWPgLtelW7P05lHHFaKceuUfpAx8zJxhyRdYUygvuo1zzgsICbzDY8XI/cS+RaAD2uB8Bfwni1RLAQGeGCrpMZ/F77p1bz1snF9YVaKBpCm8DUR4VrbLMViy2n/OBDZY1wd0w8Gpy78qYBZJwDkiW8jcmycIQSsFfWG7vpp1/vAVFrI0PgE2W1eAo7d2DxZqpQgmGSeWpC7KM+Lhko3AvmC6BG84BgMb6scK7c6SrSspa5ZoHD9mZS2ajluGH+VQTcW60jJFIs5zMy40VrNgP670Uap2dckuHzmnFbomPc9c7I84rN9F6mBwUr3wyWxTEn1D1TBHZ2PWOcJhX1XhjBL1tCT300LykyX5QOenzPFQSHp6H+BWL3GhODwlAgkqhv+SpBsVmx8P/wN5V7I07XloBVbKFi/RKwPzhDFK5UFIBVot8hERDAlCdfw+7x9dP1V1gWoG97l79d6NTPidZSg8vVmlQpMR7gx/Ev+SaTEexnPnHYv2lOasapcxNPviHCkLu0ys0MdHOVUZ/vFyrp48rbi7RfkZBpdmxREnNQ8rbkAQQsjO9p62 VrM/Up9W jg1J7DDR17PtqdVf/7IP3E2vLUMEZaPo3MGSmaWx/0vVHRP6p30X1iU/OQ7eCDDm5nMqnd3PpZngtO3bzBWUecVlMlHQOMJDni4aPr1FLZexJR1Xo3nXQ1ZyDQea0CK1TZPKYldLF+unvYY6UJ5EgGMf1mkrOTs4MtNbHSt5psophBbfkuLZE+mAuMCrIPjzALrxLlIx4HGTyfnVWvOJJUURTn+UVVyRuM4HKV84ASLrFNVz08f0KFigoY7poAtgm6tN6/z9MNill7OM9IYojNBxZZzQ45H8XBrkaxYaTjX51CQ7vpcYbUSc+slIM6dnnPLu2ZUBwmA1dHJQDTOcA0An3O48Vb5p5E8WogxiGmIBlCd56Br/Lxkqb163fvVNbpPW5pwNItJ3IZuGhkeg4ijonjlfLvQYYeXQSiG9MzWnKHNcp0D7DDvESc+0z2RHIm42nqtTPoxo5WLMHZ8wdwMJxy2CIcPBnW4PMefe6zTX0WX8yNLxqHyRBoc5L7Fdvce1vZkKVOY/s/iafRpKDIZVNfIfz3xYVbRQtM2i28w8KyLHwGxIwMfvvFMGI8pE5MD7IduFl4EJtJSk9SOKzIThKFPfW17u0P8aQyltDlQAeIJJRNyDMTKJYjL/zRiGqBmMPDJ9LJdx2bmt2kdRygrhZP4z0K9alcv5oLf6Y2Y/bIMymISoop0qXPICrcr+4rZO9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 593548f24007..33ecc850d3cb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3690,6 +3690,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { From patchwork Fri Jan 27 19:41:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E4C7C61DA7 for ; Fri, 27 Jan 2023 19:42:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B8D226B0095; Fri, 27 Jan 2023 14:42:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B3F0B6B009A; Fri, 27 Jan 2023 14:42:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 856A56B00A4; Fri, 27 Jan 2023 14:42:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6DD176B0095 for ; Fri, 27 Jan 2023 14:42:27 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 44284140F81 for ; Fri, 27 Jan 2023 19:42:27 +0000 (UTC) X-FDA: 80401600734.29.0CA99DF Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 739B618002B for ; Fri, 27 Jan 2023 19:42:19 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GSbfUJd9; spf=pass (imf06.hostedemail.com: domain of 3GinUYwYKCOEVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3GinUYwYKCOEVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848539; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dtbb/rgJDVPG5hb8ydKzjEL35WMZYcFrpyL4j03oF8s=; b=vBfutJ+vn3qT23zOK0lER+gEysUvSY8GviFQg6C73f1lCPo4CMV1MRLMXfrWsUkvH2DlDk F8myZ4TmSDWc0rgy+21mE57TctTV+8lT0Twk+WUxwNxP7RZrPpefuNK9nI0zz95S5+l26r TdvgW5dg4xlM6DwlGptmjS8fif5dUf4= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GSbfUJd9; spf=pass (imf06.hostedemail.com: domain of 3GinUYwYKCOEVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3GinUYwYKCOEVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848539; a=rsa-sha256; cv=none; b=KbLg4F7TE8OXa8annrcYIB4mLBbRT4Mk20qrM0qdB7Rku5LDtz0kO2R89VQ/HwrKCN0+9u tdfYg1Ih1fSpJ/Gy4SVwVfODHW6OxOVOyqiYKm/jvuVpcVLNWadgKhznQk8gHVDQnQBOGB P2jMtzvBo209Joj8uz/OpcjAJA30V0M= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-4d5097a95f5so65088617b3.1 for ; Fri, 27 Jan 2023 11:42:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dtbb/rgJDVPG5hb8ydKzjEL35WMZYcFrpyL4j03oF8s=; b=GSbfUJd90yi2xMLpxJwkNIQoO7AYaRuo1p0W3egBrsgFnTlql/QObCOWtQyWcyCJri L9IkPEewPotUja2mT2dWXCpgA9vVmlCVPbr2KFEVViPcTyej+SHFX8+fhAEaOP6xtrM/ x5eCKOKsLW06Glc5jxntDrg/hIgANLyaiKO4UzofC0FOHh/fDzJT0Gg9GuJf/nEgmEZV hix6GjSSYXSWndo4QypjFazRowiaazjUscTvm2uHMbtPJDABgUnEX51b4PI7vURt3LgP X4xLvpH0mH+5Sv/7D+0hQE/rAXv1nGKdGusu8zdXbWRFOMqJp4iPgA6p21w37iQT/eTP bctw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dtbb/rgJDVPG5hb8ydKzjEL35WMZYcFrpyL4j03oF8s=; b=uN61KzduXT+9Lww1zT6ElPK+0hOTTEJ8mNACtZ66Q3bN1Reff/kni0CIGLwLy8E8ws mtCkHFh7duElDtJvvG+XVCv08JJDjZ1F7Dcd8Xa27q2HtvQPjNo0G0xaPlowl252uopG sWnruJSnlslY4rjW6RcIO+Yc2+A3QkF0IvVUgBxFCnJwluWxquVz/mNBchRNQ4V0ifhk a0xLxcCoTHWI+vx/WUnwEAFVzk92satjfF1LbaQz0tp7ZdzYAuFiSV5OvCCNLuplEnTT ukQPFOoPAeXXeF3geixWxs0t6Wj7gaLJE/AHWAKBjJEwBac0TLpr3pLXYZ1Ypel+SuZl zZrg== X-Gm-Message-State: AFqh2kom9Jyq+hWi7+TQ2fGdtUqDgVZ53PmZdsnfhQ/4U9wdcXt03xoE OmUDWMjcRC1FVHSJy/ZOCxllvpKfgIs= X-Google-Smtp-Source: AMrXdXszwUWrk3+ezT3vpCagEl0mvWZa9sEHax26SYGf7QapJl4fBzpxwL0/6TcVCB7ktvgIaczCkSx6r1s= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:fc86:0:b0:4db:3ac0:78a3 with SMTP id m128-20020a0dfc86000000b004db3ac078a3mr5702708ywf.266.1674848538548; Fri, 27 Jan 2023 11:42:18 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:04 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-28-surenb@google.com> Subject: [PATCH v2 27/33] mm: prevent userfaults to be handled under per-vma lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: 4q9bs888c17ao5o8aztwr1mysdiwanhb X-Rspam-User: X-Rspamd-Queue-Id: 739B618002B X-Rspamd-Server: rspam06 X-HE-Tag: 1674848539-535180 X-HE-Meta: U2FsdGVkX19+bd3qknzYw8G1SAVycXQ2j1VSAhpThlhIIKY1dACZpkKb9C5j1AaKHO4UPV6aaEUrYAEQ4gZkYyWPbtHqvldCwTRnnIsnVFn5HcnXRfquT8XVmdkYELa3hQLdc20/K09OORP3FhnXnIPDyYZOnlyuUrtexpY5SVUvRereXxk3cgtvb0Y1SNZUQeghhcq87AzjLFrcoZ8rqvKu7rZklqVSsIrQYdpZE5QbbfCr6UyiPn8AhdrGk2+DsvDdJ5D7arUrA2CJZQsF/yPpPNPKeCpgyCcLTsYwoD1BMbCtbBTgdhUMLUWyBqVJKvj4+BNLRlrqTDdvzkevEtJTi2nc9Yci1C0IyX8vNEFs9+o4y1hKNcEYrphJFIaMwElNni3g8PgnDyw31db+iEwoBOn3pU/4KzQsODJ6uLk4+KGuJIJlzUaC34LsDrPm8cyLNq0sj2azjXXYHrmUUoTqSCs+P873YcWT9wxqj3wXn1/7OXhU3mijUOgiVGnJMGNbjYC4OfvVxzqgOO5Y8D695jCDhVecKT9cplI96EUuRa/ek5SJ/HqrVJvh/SmbFUMwWNxPh7eXM9Ry96iS/RqRYL7cQaGJpG93dY2Wq6sZM9dx22ZTInFr+C5LChKP+k8gAGzKsgq0OiwrOLqqAy20OxGzT2HwukHlUgv7pVj1DKlDBUVIjV2Scb8uNveOOnoBizrbk4B5GuMSU1UN/j5OdMEOBZaOqfrstGC/FwDGHkxc16mIVhUSNhfuE0hi2kCtG2K8SYWLCYFbIdEaGvKVsRtt66wDIfmli1ITNFA4kqlhWE1iLXO4eovW87STLVf0R38VOoBNuIaQ2/7r46mpFLJfnNrih+Fi3k2c5Sr6pn+arvQCJ4a+Oxi6+Lr/J0Z6AP7D0UASUaG4W/2ouOSKOwyNuNopmGspgCu+69GfVjeojH9YXlVfC2TtgDX/orDqsAmom0vNCk6EVru RqSPDudy kXXMa6meqopEqcCjAXH2Crtdg+DQK/eaB5a89KHo9XGNaKhhS1n6wOPiRuhXxDG/2xAbcthljyDwODkJZVk4ou79v3zNGq/IFARvXRE93ARWL3ED6fFWj0KUsDwUaCk4BIJacz0/Ci8ZRCh2gdZ8PEeoFiWQof17gSzUUGZ1qQ3Ml3FL7fwvQ/rZqEkR3WLL0yf7b+QdO+duX1/c8hoVUQt5EjmzpmLfpTgA9n/pf/OaTUGWaiKmQVGqGCt9aWVMQgnO+iu3HM+KSqVnXccjHGxH3GAv/RUFDK6dngbac6mrylw1cEFftcTuapLuaFNuWxxnJZ2ipj0Th5Q5qCf88LnW/nywSxa3HruKpUL5X/g4HkQk+D/z1/+OLplknG7EVI+eLgB+oL82trTaC6mTLmXkejegAmCu56uisTyBO04hbMoAlYMsrEGzgYj07gWiK3hT+BtqyEl2npX1sI6wRsILxXaJ+RE01ovFY+hbaDEoY4D1TehA+YyTiAjvgTbTafo1ZBA4W+rNribH3ONIS+zOBEv87+39ijiv7+MKvViox2t3Is0fNaIY4mlPIfmLchW04nmjUERy035hm311UgChHomV21u6JW1zszmT9+Uz+8KrpL986Nzk1Us83acSwZP5rEu6v7g0oUVnnerEI2Qj86A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000007, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Due to the possibility of handle_userfault dropping mmap_lock, avoid fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Suggested-by: Peter Xu --- mm/memory.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 33ecc850d3cb..55582c6fa2fd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5256,6 +5256,15 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_start_read(vma)) goto inval; + /* + * Due to the possibility of userfault handler dropping mmap_lock, avoid + * it for now and fall back to page fault handling under mmap_lock. + */ + if (userfaultfd_armed(vma)) { + vma_end_read(vma); + goto inval; + } + /* Check since vm_start/vm_end might change before we lock the VMA */ if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { vma_end_read(vma); From patchwork Fri Jan 27 19:41:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119266 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21AEBC61DA4 for ; Fri, 27 Jan 2023 19:42:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B16B86B00A1; Fri, 27 Jan 2023 14:42:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AEED36B00A2; Fri, 27 Jan 2023 14:42:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B5186B00A3; Fri, 27 Jan 2023 14:42:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 88A046B00A1 for ; Fri, 27 Jan 2023 14:42:23 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4C893161014 for ; Fri, 27 Jan 2023 19:42:23 +0000 (UTC) X-FDA: 80401600566.29.7C90527 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf19.hostedemail.com (Postfix) with ESMTP id 8B0911A0010 for ; Fri, 27 Jan 2023 19:42:21 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="mU8gu/pW"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3HCnUYwYKCOMXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3HCnUYwYKCOMXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848541; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QTC+ocJD/ZK0d8AWah1kQLEh3QM8J68abwl+vLpX/MY=; b=em85GNuKVnhY0lsraMzU85uG9fQQgn3eCxh20kZskXFfKgYrJrjbgtR3Bq4CjAr136IVUu eTswcpek/IRB8nmjVVW9t/B1BLpgiBEwcTtgzIs5pwp0wYcI8zKeUMO/5aCUoZ7yuAgaIo 5h6/hM2HWLmRWb+w/Us2WfDdU/99W3Y= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="mU8gu/pW"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3HCnUYwYKCOMXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3HCnUYwYKCOMXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848541; a=rsa-sha256; cv=none; b=ss1Oi3/yDKJjkQcS9VrIxXn3BhLxc+QA9S0u3okdeBh0eaYCFXCTbpOfjIhiE47oHe5Leg 8jUiOtpQMhGZYi7SriIvHXe4FPYvdSPQmNJQyblDX0a/eNOd9zmTcQmtty0YkZ/7LCmXCL uNwhIY/dV62QmMT8txUwOf3IvdBTA/s= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-4fa63c84621so66298147b3.20 for ; Fri, 27 Jan 2023 11:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QTC+ocJD/ZK0d8AWah1kQLEh3QM8J68abwl+vLpX/MY=; b=mU8gu/pWf6zIsQFo9qZJmzO+C9Rz2zywIzJAt0hiLsBb0sfMScn0ga1WT0lJP4+cuU jnlE49/hYrnAMYh/RLEYzocL5Jy7mfCziMb33lJyjegHt186LkbV/WCwUSBxzV0FGK1I xrqhc33jx9MJLtdss02+APSdeRf2XUnFG09IUKfXCIr+hkTGBQo7qLvcD8/EZZarD0c6 9RiTDx0vtERha59paJtaKyAUCno0Vzl4l7lvqpJzE2nTjhpBkjOXcTvShnV68JU8Wm5n IoRxYa+gg9A7UVOaBxUJpIYlD0HzhsBk7iegMRGVNF5IQVYNK1FSZGz91UjWgsue8k5N kYLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QTC+ocJD/ZK0d8AWah1kQLEh3QM8J68abwl+vLpX/MY=; b=RJ0nu+cqo3premCryvMz2ezBxKmh+aTuPJoUg99OzrtTQAQaJ8vtMq2Dt8UNtoGEt1 yipVIDByQzL+5NfhHlJg7ttnPBPKTRhQF4eR0iFmPoYrdMXxVItFzNnEG0uXiyea9gMA W94nwJvR/aGnUTbUpanNOTux6HMHDSxL9Jov+SBnCOP6EH3ufGsicJwExs8wg6KQ9ivW wKHllPg/CIapx2/IHkc0Tpt/gdZfz4tS4F9UjmMAC7TMTKyk0l5Q/vQYJbOKdodOoivx refw8XzyAYheLHzvmYjYyDLnxFobwd4tuKpoV1M8HK151puDoiXM4aACX8y+ixzYKKEn 0HYw== X-Gm-Message-State: AO0yUKUUqgyRWOPT0bKI0xtLwSQS9tpwKrdv2ys8gW7sIw/jQl5oFBfT jxuq++3cImO6uDqJfwNlm8Thdn9lvE4= X-Google-Smtp-Source: AK7set93uG4w1kFwIB1zyvlDAWxZcHXpTslQnEHpzoDHLwlIEt0PkMVZGLSp9rtobNPjvtDDJk1UuyFV7ts= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:e705:0:b0:80b:bdc5:5060 with SMTP id e5-20020a25e705000000b0080bbdc55060mr930533ybh.27.1674848540634; Fri, 27 Jan 2023 11:42:20 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:05 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-29-surenb@google.com> Subject: [PATCH v2 28/33] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 8B0911A0010 X-Stat-Signature: j6t43d5imzed7suwyxh99sk75sejwyi5 X-HE-Tag: 1674848541-924395 X-HE-Meta: U2FsdGVkX18fqOGgr/BM/N2TfOuBingfUL5GA7md3RwuIbEsF4wzluBScUYDcQrSyR4lYxD812oYnHD7uU7ohM8UFSv+EE69N29qvGChnOycS6mZoLM+M1HRctPCp8zhczWVt4Ew4gWn4iBxnpKKJcbcli9cDDq+QQwdNW/0CYw5RCuegsDfQ58YgfB0fz84upEVE8nrF9JgTOrgqICn7QZ68IO+zwV/hMFEICu1UGiv3Z9Dy1mFsuSYV4RJPrpEOwVx0WNpvvGobfIsfkCIF67/D3qbjk1qoUbwTiETOGEJKRcsC2lbvBQLA3YMsBg7marZH7zhGUB1FzC4fjGGej6+6Bzxd/jqJ6IW+2CAjt8QnU37GNM1M+2uCpSI2zMJYQxZHwL2v9AswNc8X/dPDkY2AGeb+88Q4sRWFu0WZbXTXbMuQgaxDhbFWa1cOPAQIlvnO1HokHk527sgMj2lQKk+kcoGbg/mHzhyKw3HKbwUlS9p6DamqnO8Q6Q5Bh62XpgbmzKJw1P9RVkq5xh9FQmInhGu2YhgeYWzYmZuPojRYntzee/8YSW+Zg4iGPHAjKiIcSASCY+OLfYcY339MgllxwiyVJEGxaUdYbfCr3hECgXJjB7COySWuyReNpSBPrE1nbuFlCA16lny1BNGrORqITcSSO2R8IUjTO06tsWPE9iR3TUGkKBAlFl5s2Y5vPRsWAi6zCgK3Jlat5tPtsVRaQ1Dz4cJtVwU8ChURxphM8WzBq5mHZHsnPAQnhsF58C68jnUq5NkMuoHWUIa9HIbg7zO1ewjF7d4u5hvivqSywkZ5j0cRAJ5c7UFatSo2djkFjGsQ+dgE7TWkOnar9xKyZCO2ZpWyOP+vM1pRL73x5veB/txYJQO7ld5qlsoxIhLYx0b040QNTgS/uOWFqNp4/bKxb54HmTRoqlCxgYYfuCUy/4Tro0CnD6zQEIdTw4Mea2KhF2Za85xs80 HVZidvSy TJzab6zZxvXe4/V2IsLPFPoBqoCiHS6n0yiiuN1GR1v3dlnpwQznSPoOyUs/cr3tRqtqi6oh4wp653lkOwwUPv4Q0Sp5TpnkJ+vOcR1IIkhpJNSlWUgFaxmkt/OTc5xRuhCjno/T/2kWX/jhBUHB6wlk1h4dox7F2WQkvZ6R7IE1EwL6lMwFlf/02xkEwP4henYGJamP8GgWEvFGnLu8shYUDe/TbPMyJ+D7u3oHaPnef+QcAvZ+xwdPiKw6xRrQZbnl4Xa54hChBaVGCXYlglJEYoaiW7MRU7ZbUmRMRdGvhCUi+pb+2q7D07r/BOG66cFUOoPBcYtwsyiRW6u90WdmgagNpwC4nLJwtOCi842wwUE0igZ/zeOavGMr06NxPXMdL59XguM0rJaM3h9pb7u49bB3K+cr64bt+1kv8cwEjFc5YY8yYLCOW/HdfXgz5A0TgAeQ3xoadC01fhEJQTcqlnlz6HvTtPmTraVpDgadB5u2/Gnj0zLx8+i4Sd9FV7TUVkcg34vynX8G23aGsqVYp3akXYk8AzM8XeQo10e4fqwfQlbJAyDmM+T9+k4i85X3hcQVsSQqoAt8kl0G975RwEcKCtlP+d0FY1yT7qLrXKkb/PoZ7d0kHpg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 7 +++++++ mm/vmstat.c | 6 ++++++ 4 files changed, 25 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 7f5d1caf5890..8abfa1240040 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -149,6 +149,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 19cf5b6892ce..fed855bae6d8 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -125,6 +125,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_tlb_events(x, y) do { (void)(y); } while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 9191897e76af..4965a7333a3f 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -278,3 +278,10 @@ config DEBUG_KMEMLEAK_AUTO_SCAN memory leaks. If unsure, say Y. + +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + default y + help + Statistics for per-vma locks. diff --git a/mm/vmstat.c b/mm/vmstat.c index 1ea6a5ce1c41..4f1089a1860e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1399,6 +1399,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Fri Jan 27 19:41:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119298 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F090C54EAA for ; Fri, 27 Jan 2023 19:49:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D81826B0072; Fri, 27 Jan 2023 14:49:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D32546B0073; Fri, 27 Jan 2023 14:49:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD3156B0095; Fri, 27 Jan 2023 14:49:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AD87E6B0072 for ; Fri, 27 Jan 2023 14:49:14 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7DBB080864 for ; Fri, 27 Jan 2023 19:49:14 +0000 (UTC) X-FDA: 80401617828.30.A72F99E Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 3A7D41A0015 for ; Fri, 27 Jan 2023 19:49:09 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=I2P3olZB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3HinUYwYKCOUZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3HinUYwYKCOUZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848950; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zr26Tmb2M01sb1afZfnklHdnYeWn0/HWZzrpjXJYooI=; b=KxuP0WiqaOY/fPEgPiMSbFWovjpC8UuGaBBUm3NUHBsdUc5P+Os/41JsRof/eeZFPQIBmV D4RiOpDhSzqNJDNDj4/1nbSr0+uAmpDb+W3rXeYuVbeb8qNhfFlA7kA+KRuynbIZzq1AKD 4a4q9ijXR8AXH5rXuMTxPUGFLmqUA5k= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=I2P3olZB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3HinUYwYKCOUZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3HinUYwYKCOUZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848950; a=rsa-sha256; cv=none; b=OS0svs+olo/uxAk/O/ilbS4NcBSnkLWIAvjgn4rNFaxtVEpEuhEr6kCuP8g2BndBu6Qep/ XmqMtwe0PD9zaDvoFh3cvkbiRHmpqM+pSfJJBOFtLfM2Quqn7181S+rMHAhCk6QYtGpvbD c34ACB3mRxvcec4X62pHdmVSN3VS4Oc= Received: by mail-yb1-f202.google.com with SMTP id t13-20020a056902018d00b0074747131938so6405669ybh.12 for ; Fri, 27 Jan 2023 11:49:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zr26Tmb2M01sb1afZfnklHdnYeWn0/HWZzrpjXJYooI=; b=I2P3olZBrDoCFDP1jHcaHXqtsXyUzxQr/ws4wX6bBu3CUipyuxA31yRpWa3MnFa/SU pcf5+CCx+WnMbozlzDdIg2ALFwq/OFj8s/q24FIJwL7Tizh8I6eeGtbBFib4zJzLB5md /zjFFlwtWXGghsSwviKTLUSRcwBRmcJaXn6b7nEXFweXpvU3ZyCKtcIxikLC8+UQ9pUQ JujyWegWeeB7qz6qsENh+jKg1foSfeMqF01FuGu+Yb/GyFYoIS6vq3ynrguPkj0pOoLx BYJ8cK1I9Z+jOMo/TqTeh+dlSH9FVfZfZ84FkijhY572Ij8GlcP0ii8VORVWe0sgl8Ck Cp6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zr26Tmb2M01sb1afZfnklHdnYeWn0/HWZzrpjXJYooI=; b=iZiOh6ybeNxyqA/x+QNUSPzkQ1mEtenG8zM8p3QrypsOzJSPyQNSPTg77J3j3z4tif dy462f4sCcVKpLVSTKOueWsXBS/UNqOJtq4gubJW+u/HeTEfkSX90jAvbg8kOOtg49Ia b94TqdiCrS/Z9OUkZCP3rkD+/eXcoZMzsiKfoh/5hIRtM2y+6r9ij20lGcIg0bdn2iBm evewzRLHJQy7J3RR/mssnOP0xe5zveYw5ObK+ogbdKMmb7oFwqMWU+PtqD33eMbLCMWc jXuH+DcJM8Yrr62+LSjxzLgzHQLACDqWA0WmPJv9o4G+aDzWtQ/krov8xqW8AJ+mG1MA rfyQ== X-Gm-Message-State: AFqh2kqhb5qA4vwBWRGiWNVQkcPE1PTnfj780yn01rZbINf12YjAlfs8 TVchhjM2zJdumxvMrzP0RuSl+s8U0xw= X-Google-Smtp-Source: AK7set8yRf6g7ktuBKP0CQzBeFyUCv/mUJgTqrv00u2WfcKZqCCOgg0GVy3Vw6WGmXU8JBT03ehy/weY7OA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:eb09:0:b0:506:391c:6d37 with SMTP id u9-20020a0deb09000000b00506391c6d37mr2021270ywe.88.1674848542935; Fri, 27 Jan 2023 11:42:22 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:06 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-30-surenb@google.com> Subject: [PATCH v2 29/33] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3A7D41A0015 X-Stat-Signature: aythu115c47fjmxtriquj4wtqkpia3k8 X-HE-Tag: 1674848949-120599 X-HE-Meta: U2FsdGVkX1+OlP7Vb/4bdRH3obDya3jj1wOB0kBuFtmdjqEIvSTwGDybTq8f1c0Saw0riE0rA0xyY8IYKMwBKfHwdqUmpP6tZysO7cd164gm1YkX5HqTHVhop/wOsVxYvaxKQtUzGp9q5oZK48trHMaEUY7Ax6b6WN1cIxjTaD9HvxVvDhzKFfh9jdkZCAUKrK32ib9e4kkInEAqwzupReYszJ4h/5mJ8qNPa8otngoDL6XuslXv3NmRYkDM+gnRrz9EOw/1P7BN8SEz5jsYNE8ae11NtsyZcAhB0bNEPXR86X/+wbdFBxBSixAKZGagTwcMrpWI23g328H1oLizPdQj2k49jSxQDO1n1hnPtLtJ+fCMFl9/aulJydzUnzkXDwb/v3BMn3C1c7lwc4aIbzl0BIC2uH9EZf3sVOkTJTDn0aFa7QPE11SJUSvjge26PTgHu/gemJVXto3v9ic9f4zqKi2o9+UbraMDnKVq64VySZ6zegVDj6g+hpkiBxElmMdfVvWQNCX9k8dlTssT7xCQRqcyNMZtNR063ecOgHO6iChoi5DUz5T/tZbK4UT3+7VQCtQfBjHQnMvrTcG45MmXNsWQBpphHzP6T1mne3kjBPKwLtg2fwC6gjg0Z2NAvB2tZ9y9jEeex4+0S6pMi3ktlhBofIpfDoPygE6/f8Fcv2yTbdb3e8GKNBsnJnRp07xfTKgRAlBf2PE1lWZOgdezIeQm1zCEdSZr7uRgRpr//fLrLXlXnCWfg0/od1nSfZFlvOZtwcEA2+YhXoISvk3TVF3hy7sqWRZO04sVNuoTPbq6uF94LvU6DPQrfj4zNjOjZSMrjQc96bjU6BkePEBof3OLLAnXpYTx20bNglQxMRzuMmavB/frtR9O9pReVSC6CBdGxpwzAskmCnLkGnF7bCB22BTHHb8aGra5Pu/7AoSzY7wVDxoH9dbcWDmbx9HRO4u1DxmIl4q/D1C TaK/w6wb zf3o0rDbQ0wKUAoxCiWtWNIKKVEVI9cx/U874yRLlqk9Ehz/LhIgU0esgMF2J0mvCB9Ua9Fc0IfgKwLP8DG5zEaoUV9t6cRoJ65C2QuY0mcdQDj2ecLZj1zq/Br89SwRG4baFrpMKifjl5I5ec+gR6OtSrMbfpkZZ3df9UByPHpUG/tOyUSL/uYtzTd6Ibo9t82uTJNML4vcDh62O0+LzB+n4p0Xn2e5/HeI8f96HfXgWzwu7TCzGqwpDf1B0HReDNROXzJLn9glm/l0iISnbGv47T3/s+Biq6yqNvJA8yQ4U9a+PY8RuwmEreg2Xj4fJuMw70RgGSoCisvwbUn96m1UD5p0EBPXT5lQL0j1gG8nb8gfQ0qCax0ULkU80h1f5eT2oVtaHNDctYFhnoSxldW3goxmoeUYZadLN/q7x8Qo8wJTahmYmZAJ0rcwXo0Sc9EgY/1iWqil7xRDZxdG2+j5ZkaaLTIegIVSZCPFf2vcrthgl1gyAG2eXYWLsL3zFVaWHNYCnlLF7K7uPLM+N7wDM30D7y7u5iW1IzGECjuc6OFUd7PuEP9tmedMnP4JWRaBeCn6Irrlbqj9eQuVkFOmScIanfbrFOLTJqUSLhnyWYrhCtWXtv2A0VFzak8yT5k8UTwhhV0aFgdcJZ8CCvd3ckaDGyBUoxPuu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3604074a878b..3647f7bdb110 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a498ae1fbe66..122c8fe4af12 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1333,6 +1334,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1433,6 +1466,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Fri Jan 27 19:41:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B8ADC61DA4 for ; Fri, 27 Jan 2023 19:49:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B11086B0073; Fri, 27 Jan 2023 14:49:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AC1726B0095; Fri, 27 Jan 2023 14:49:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 961C46B00A3; Fri, 27 Jan 2023 14:49:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 897B26B0073 for ; Fri, 27 Jan 2023 14:49:21 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6F34C1C688D for ; Fri, 27 Jan 2023 19:49:21 +0000 (UTC) X-FDA: 80401618122.27.8127DAB Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf02.hostedemail.com (Postfix) with ESMTP id A28BC8002B for ; Fri, 27 Jan 2023 19:49:19 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JxxlO558; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3ISnUYwYKCOgcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3ISnUYwYKCOgcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848959; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MNwo0rhG4y8C+tQG7LPM4nLJWTbIHprc8uVQsZA2GSc=; b=aOiB7RFxAd3eSjiLOoZT04NsP6X3ExNjKA1jvvyJibPW7WbimYiWYQtKpQGaKeCjR/5cX+ CWCfwjOuM/xg+cj/q5mzyJ7lzr2XMNK0BG+zqp8FDqxzo+rr+I9YdgANP9qwjAHRo2LSRX +U5Erk3CiQRERYFvOtG05f5V9QBKxTc= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JxxlO558; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3ISnUYwYKCOgcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3ISnUYwYKCOgcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848959; a=rsa-sha256; cv=none; b=WjajM4TOsIoA8vZtMDWXzbNWf2RuHMqPkvPKhR5py7UR6x46uLkpfpdrVSHWnnZDhGPohM KeJ3W3esmTuyfy249UdnQdodJcrXdNCvTcMx6IrbNwlMZAUkDwavhp4YC+sUzgAPrqt8uw ba37FsnNtq9stk2/sQ5xEDIw4+zIlo8= Received: by mail-yb1-f202.google.com with SMTP id b201-20020a2534d2000000b0080b87d910a2so6452694yba.21 for ; Fri, 27 Jan 2023 11:49:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MNwo0rhG4y8C+tQG7LPM4nLJWTbIHprc8uVQsZA2GSc=; b=JxxlO5586M0z4sVDSxHugqEEADB1cAjEHo2xluVRFI46WqHTOUsszCaJjysGoPqCq8 ztKkJGmntymAGuEGxdM2A7EX8Sykw/reCFrQFVsgm7kaRoHFsG9BNF/PmmjyzPElNA3A u2zVx465PKwxuuDkWTqvf70dFDsT4sE76+Q1i+nyNMyErq//frz9WqLOph65UMZVM1Q3 1JnBBBOhnZpIDf2DrbbDSdhADbiUH2oov3flacw7sejkvu/YDNPNLxViqgxSV8aAiDpN WEe8RN23e0S32q4tATcZ7ZhVfhZ7+NOeU1bzW/fjTQ6HzW6TJlQUp4mHpwvwuOr4rhfO QkKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MNwo0rhG4y8C+tQG7LPM4nLJWTbIHprc8uVQsZA2GSc=; b=SGbbwAgZBCuPLpz7R8NQqQlix4ICRW+f+7P+aUWHitCZJyHjSJN7Zvwvq1VGU+SPYX w1QYLGpVLC3EHpunEA53fzP92jpxkzVcPg7PEk6ZwhXSTuxBLOjSJMYgUgUn1QzQUW2f khsX02TUwSKW4m6bMDrNjrhgaJ8KZ49bxzCool7pMvIBq3xTmvfR5ElFZXZJd2nvJk41 pKx9MXCAVpbOhneRpO122Pq3tAzWfNec+m8MQIX7Uyq6OndeiYPTX0nfgTnoKdIHJAgA vs9W99tB2Dbsz0e6csJf8Re6fyF2SS7LIlyU7Hf+oyxMDtubaPyErny8SuUob19jGxzI /rZg== X-Gm-Message-State: AO0yUKW2cFgkRxI9YUYGTXtMqcfnLFRCKIXApRlmy2VRFEyWEkQwk6sp yrSCnEycl7VfbHqC9AaVGfgOpv6t/gU= X-Google-Smtp-Source: AMrXdXv6cPUTMY06wj7LHpX7SofIKh9xmcnJ1P5+6wR9m3ToK9ikAh4HiQMQbcnL8It7MYI0W5hAcMnIURQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:5b88:0:b0:462:e278:1a08 with SMTP id p130-20020a815b88000000b00462e2781a08mr5361156ywb.243.1674848545116; Fri, 27 Jan 2023 11:42:25 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:07 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-31-surenb@google.com> Subject: [PATCH v2 30/33] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: A28BC8002B X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: h97zqnww79yxusa7wpbf5agm7wxeptgx X-HE-Tag: 1674848959-910212 X-HE-Meta: U2FsdGVkX1/ANR1PXn7OOH5pmv9lOZaqyDM4hzKQFFt/KMG1Qm6abhBI8cVvLkTFB8++PBERUpB0HeqbWL2N9vr34L9SMlf3SC1mf6V60J8AqffGNbX1E8zejOTPV91fMdjqFHbN2JzCl9UKtbLjJEaPedPY5GRuf/83LtX2p5FCuHmvDayg6zfS7nK6gTD9hJhEaPRADTW+d8Jf2McrTTGo7+8Ym4CMoXVyFDnV9JFouR2a4xM8xgarWhcB7LhgnjtpVv3NmpCY0S855cMRU9dAuOQcjpXOhhZ0vmGP3sGhR9atPuMNXIOw3FWNhz07hmejYXrujmPCDn3OUEpX+GVtYqZ9B3Rsk6BonTcx6q0J4lU58yQg2JosuibkA1+6R7MA6fPoqinj5nUIjw5TXq7ORE8T00X/YOpe5OTbnIanXcI2woEr3U4llA7e9+eBVFgO9gqszTF87I6OgLheMlv8Jg4Y+fjEzvrMOEmuamASI5TBoZzAauKm8wZ3fYMRnYeMzDxYxP4dIbhOuJcottM9iAZnmFVJYK1e0BSIUaMkSr4dc7okLkH64v1B3gwIkYHRNKY8wlK+d76M2JKw2npaul6j1xE2x3ya1oOJRnbN9B+jPFE4cb+kuLgYe+kIHWLPItWJypHD8Rw4B2O3ej4syxEqCceR4Vf06Dk31YtbwK9O9VyOWFNvZrVMaqH0uh3OLuT70JLOTe/9v1f9/OSLw190DhCibGllbMvR1rcR7O7cUV3nGTCO+ZYD0FyHGmZu6tXRY/u0Ti+BOAb0SBbN95CotOTWiDRHU/x5l21i+1fte5C3oTQTKkv3PVgrjd6XToclb6tjU5hZU/ZMOWBnjJbIxAJ9ZMWuQ8OjCti+j7lwYBAtjH5DLEixqIZl1b4/5t2ZBaADc+HsBkOkz6gPlHoH1q2xOOInLe2wdfJi/WCLgXizVdOtJntCrxzNiJ8o4U/HHAf4iQgbXot oqKMolp8 DhAjPX/JGeG3ujklb9pDT6g5cz2QP0rS0EMX+FpaGmN66q5XGGsHCdJrr96TZNjXIgcvNl/64sosNauzokS3iWon69s2dOP/DVW7assWDwDMA+0xFP79w6gMTfLP3I0WHbhq8Fdq1P3HMILkiuCCQmTmhLRIF7oIJN9dlgxd1cyhh0JR+D1Vc9a9/JkDSRC+JYwuV1jy3LRdHrtufmR9/vbLUoT/732dNTI/8R41mzQNhd6ObkMk5SG2a6UQTQE1ykZbk0aIVYvnOBBzti2wHB+q8M0rx7/Xio9/Rn5WaVA/aoifxSMqb3GWVPjBNDVYfMuEJEhJTLItJUuMzqVAzbqFCALlo72448FNtBCX/UpKwg2vWAjW1NZWlxO+JAl9GO9HtAvL4YeaHgYa3xGoq9maKVUP6VDJAln/S/GXTwx0Zkfx9XwPb5DBmTvqev0gKLww9jXV6faqIfaSUHLZyCbJUNaN4noRe1gON0rCVOfXS0uUgLUQI6cBqI1l4FJpuXR2avcSX9mB6V9M6NHKb/F9OJgBjYQcKN4ylPSbP3+3n3I1i2kFCZuoO2VF13qQ46uVNhhIvYxol+hgEz+av/9i1SrK2RmHnb7JLjtUvTRr4EjtwkAXPPFpv0w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c5ccca26a408..9f2c0e352da3 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -95,6 +95,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f4cb0f85ccf4..16588dda9258 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -535,6 +535,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -585,6 +588,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -628,6 +661,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Fri Jan 27 19:41:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECC22C636BD for ; Fri, 27 Jan 2023 19:42:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BB166B00A3; Fri, 27 Jan 2023 14:42:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 76BDA6B00A4; Fri, 27 Jan 2023 14:42:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 632FD6B00A5; Fri, 27 Jan 2023 14:42:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 50B646B00A3 for ; Fri, 27 Jan 2023 14:42:36 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2D9F516045D for ; Fri, 27 Jan 2023 19:42:36 +0000 (UTC) X-FDA: 80401601112.17.13AB1F1 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf14.hostedemail.com (Postfix) with ESMTP id 681C4100002 for ; Fri, 27 Jan 2023 19:42:28 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="Qw/TzqOb"; spf=pass (imf14.hostedemail.com: domain of 3IynUYwYKCOoegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3IynUYwYKCOoegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848548; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kzxeHm8qCtyYeQADp+m05TsFOqarDXtVrIERGy2x4bE=; b=Ro0+YL/h5yDP9TGhMHrE167/FK20s8MFqQwoTBjM2EDHfYZcVe8H7pi5z2I7gvaunRuZYN +KJqE9KbxBmksulEjqUs5EtCfWzYh4dJIW0uiNwMZDegbro2xItklBqfFrPzu+h6A/33U2 TJp4iznMtVJCuTyOHFeIhjWnvq9S2Uo= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="Qw/TzqOb"; spf=pass (imf14.hostedemail.com: domain of 3IynUYwYKCOoegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3IynUYwYKCOoegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848548; a=rsa-sha256; cv=none; b=iQXgfhBp9LqkdZkRzaQvM0uEPO+dMRPdG2AGOXGx5iURt16GZ/0cdd9e7NOiTn++y5Y2Hy 2VGsgk/5R051VHe9zKLWNQykt4cqqKCaBCIwXXNunNchgCSGibhbrpYvkdVst+c2ezb4PP 6rfGi21dDGlaMtk1Pxu7If23XIC4sdo= Received: by mail-yb1-f201.google.com with SMTP id r8-20020a252b08000000b007b989d5e105so6378438ybr.11 for ; Fri, 27 Jan 2023 11:42:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kzxeHm8qCtyYeQADp+m05TsFOqarDXtVrIERGy2x4bE=; b=Qw/TzqObuxKUrxtkILJdFMJrnRXDl7IhHsfqWYsg15Fht0zJymOStyW6umwGn696mL /vz0L3L+AxNuyNMgLOdykFyo3ffxujr1NJ9PZ/3TCzeykyM3g1PjhFRGoo/e9FoO8fmJ 0gJOei12QFUXaa/Oa6hhy8ENMFQY5aEUa+SKaoSJiwpGU68x3ruM/ipRGHnS2jScV2XA EPWUY+rZwtRmlqA25G6oa8LHcJDQM3bZQ4diO+PFB7fofYZtcMVMGn5EMXtqF1YwdUnI Yur83PIWndlmZU2J2xMV4at0clCg0WXiP4a0wm/tFrIYOxaiWzueFhV10wBQ62TvKtYz 5Efg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kzxeHm8qCtyYeQADp+m05TsFOqarDXtVrIERGy2x4bE=; b=vUtQB9b+HdMj3/QH8YTC4NjYGEybJ3x8HPzZ55BZrL6rlAUSFr4XFi/Qmzfo5eLWOS 9MEruw5ZTuuJPsW0MEQdneaTqyXkN5hdObToB0phCgCTEa4KAZJKutXFISF2ff/Khj/S ZsnoN8b0dKzayfjnkASSYRny9pUpJyUwTXe6CJ1Y2OlcWatnrUEBO4upCzHMI8JhI3mG qvPFc2MvbK/nyYrd2BrIk9VqXV1KeBJpVAMFIDL6SES6UO3RoHQCTVFyFZk7oTJ9Ie6i TgyhlEe8fFHO83lnIYnjIiEmoxFmNlgl9ChalZ3+we2zWgkuRHREyL8bNv0mMn/08NIf 998w== X-Gm-Message-State: AFqh2kq2WV0H7A0G+2/j60h7A+98OzXJBSBji0P4DzLoJfV2uVDjAC2W LDLG1N3FxNHpdf0c/lBuxB25D9kZkcY= X-Google-Smtp-Source: AMrXdXvuIF+ftjjZVhAq6LLQlzBTotqsaAzARNS7Nb+n90q796c/W7VGIXChRGDMsXog66D2ETjcV07ygFs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:1754:0:b0:4fd:ff18:de6c with SMTP id 81-20020a811754000000b004fdff18de6cmr2568712ywx.227.1674848547503; Fri, 27 Jan 2023 11:42:27 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:08 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-32-surenb@google.com> Subject: [PATCH v2 31/33] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 681C4100002 X-Stat-Signature: u3gk563ro1da8u597dmz4e8gw48jsifq X-Rspam-User: X-HE-Tag: 1674848548-490825 X-HE-Meta: U2FsdGVkX1/hGQOR0mOTi2cGwb5kZuvc6ttc55mwCUN09EQtIXqlEdtDPMCqgr3fGswRpSaD+3DOlNZsb2nbgilIB7OGpnQAarNydO2JiLkGHHVyDV0e9dB6SlH8xukb/IVKFKdbUULRrjxiT6k6HwSLIyZxpofI9W/vH1hVyFp4oZtc9nSeoPIxuYLCGlDhT5Ra56bqxZr4BsNUXZOM7xqryCR76XPOhxtaG2t0G5pBDKPLckai/z6JYjpxMefE9E/r8uL1GPqtVxzn47IScL+8k9VdGuX4/fojRRloDwhSduWU2gLQoo8ZCaJkZ9b2DB3NReJkW2OhKG3tQR9Cu1/g680bZvfB9c8BjDGsyR9OE8gTTxoe5+Pm84/63Wgk6TRavInBSt+9wdwly3lTAA2rxmsgpfR/t+JpA2FOjGimCqsRA1sFk3teasksYJ9V33uDEZQ2iohsmMaQv3nBeMQ8lTZUvKzsc5HQcBxyqOODDbrNjOMLP4RCGHixVbGdcEDyKLogSH4AV5ZmfTo55mO3l2FovCSXeba64YW/VP4OozbyYLURdx796BYeSWwk1XLuR+i18hKJvDQ6Eh5RwbByhP+QwNGHGRR8ubIsp6MNEMsE0/I1YiAmxNadRxas5zZFDSmIIXJSpCPKyqfVq4yzEWBMmnmEZtzgGiwBU2gYCK90xQ/lCiskMKKPqOeJP/YxDHfxgMeOcy37RnCVGoZFXepv3cKnkF2umHP6ReHLlHU6D87SnV6a7hg0vunUwEZx3AF9yFbqID7/DqNfapWCnN1fpaWOefp9b5uEeDyPXVP9g7o5NUyqkyxai9F/EVyWONG3NY6ZmPr4DoRH2YCDQQsVwNK0/VuS4ESYJXrkSzUK7H0Mpy/VprEsHhZ6zr61qv4Zm6NxQgHz5Gn9n1qHjJwaFeqGFDYVJgC1LdvE9FAPWDYlDNoqKeIuQ06/3jnhkSs7399TJw4jpKV 9cOVQzCt 0tBPHPu4/40d4o9QgpJMRQl4sI74v/uk/VTEVE7/0M7sdskDSOCHhMnfvh7xMRzMyX5P5ryCetebYzULMVCS8cAhhBcpZxI0h3TCSFURcSl1PQEBrvK/OockkCYfAKwJRerzaNWgG8vVSBE8FTvGDtPh+RPAIxu7i0anem+v/qdPMgn0CYXHgnn+VAPkpG1fvaNtQkyr6Cmx5x9XASVoVNK2JQMMqihSItFxeUCqjS5zqxPKc+TMBKQZD5wROrvpksGwYUIHrsKLsrVUPrX0TLE/wG9ZxjaDSwT7Oj8eTZhOmZZBcOn/0ZuETJXsxPkv/OJGcx6YsJm/BRT7+rF0blrBXC5MyQx94HusmSOuVfWYNPQs3Tv6eWWonS7eAabPziIkTKy8oeSknDM4OLWsgQOTdybltEaunYcRANMhPksbWEelXQvx6UBxbX1dOhaw2o7Io+V/fzfz0lH0A/jKsb3Fbf0fAHP7dLihHGNM5StUbTO/hCJY+7MYB4oq3cKvTzRbMme+q9q4TID8fb73z1hdbbzxWrNBHcOp3oIHn5jcna66EK7XWC+yA6xZQQ/krzjUlJ2Ac1u7iUJHhxaOtWh0ncAnJkueotH2w9uYLEUDQ3JdnbOzaNEyWB/iCTsDESED7Si9RW2BDt3yRrNcy1+lZKg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/fault.c | 41 ++++++++++++++++++++++++++ arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 3 files changed, 43 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 2bef19cc1b98..cab229e75018 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -469,6 +469,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_end_read(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_end_read(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -545,6 +583,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index a3b4d99567cb..e036a04ff1ca 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT From patchwork Fri Jan 27 19:41:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DBB99C54EAA for ; Fri, 27 Jan 2023 19:50:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 740266B0095; Fri, 27 Jan 2023 14:50:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C8DD6B00A3; Fri, 27 Jan 2023 14:50:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56A176B00A6; Fri, 27 Jan 2023 14:50:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 49D676B0095 for ; Fri, 27 Jan 2023 14:50:08 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 279B91C688D for ; Fri, 27 Jan 2023 19:50:08 +0000 (UTC) X-FDA: 80401620096.29.8724904 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf08.hostedemail.com (Postfix) with ESMTP id E32B4160031 for ; Fri, 27 Jan 2023 19:50:03 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=R8vm5sGe; spf=pass (imf08.hostedemail.com: domain of 3JSnUYwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3JSnUYwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674849003; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=aM9TOObWyxI9lZa2RiM06JqdLQX6NmXnIDz0gFwyKgPPXxq1oeBNgYzaSk21m1ZVN6oExr dzRdZHtzVW2sSfn3GzLruC0KK1wmIY6ublN8UITPwthkXfde5c9LFnwyCgtDIKakwjY0Du 8UVutTs05QI6zZdek+C29yWAUA2kBgs= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=R8vm5sGe; spf=pass (imf08.hostedemail.com: domain of 3JSnUYwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3JSnUYwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674849003; a=rsa-sha256; cv=none; b=JpbjCg4CLoGEnE8JssLi8uL260qb1ibBZtsZmru14Bp8le0eVURCXmHPbx2z1iWb5edyil 6FvydpfuF2VFEMN/Z59f+itaQZYb1URN9tdhv9uleM3ZnpmD4YKsdGoV6IX2rV7lnPBgjt d3YAFc5pPNljivuv8+pzh5sfxpKe0V8= Received: by mail-yb1-f201.google.com with SMTP id r8-20020a252b08000000b007b989d5e105so6395108ybr.11 for ; Fri, 27 Jan 2023 11:50:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=R8vm5sGePmLU1Pe4mKnKd7BNiKDqaMQRSE2Uxaf9jHVe/UX+8wZaZnli/NAPrvHh5D Gp2YVdh1UhOUz23uxWg1VsM9bUD5cup3/ZbXShp0DX2DQS4OYSwXrDDdrwmKhIAFH0Hw 74ENSW8vtZuV6jUaLsmkcaEy+yDT+z+95V7siTwW55EWIBuns5aniQb/TRczPRcRq3g4 VhtuB5yMK58BxiemGZ2CqsONqT2FuthDA4119d3zQhl7jpXryXulujWcH7o4rSRWH8Tl w610sqrsMHevwvr509uliYrTdDC7Uzh2VKUBma4LjNdxi7kycc6HPZnQZbUV0crFM3wd Zbgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=3VZAxCiVSpDAU72O6mEUewZmTvhi/kqOr11MFQEgpJm5s1x7ekijyD05e4tHzy/qJJ brprENsxxJ+LDtzZkozfOMvyV7v2s89yDsTo4epcin/QX9yskur/uq7/g+Yfh3D9jMA9 6rVGVc3L7bBKPag2+EtWykVefd/NyRjCOURwrj3aAky8/N+Gd6/oCTggprWJ7MYe7g1z N6BdoIqankUCFv9zJuKbbRXZyRfW/5QvRUfn0wthZzSgPcuPjw/a0NJEThcAUwa3H3MZ khdxddnsqcXvSfF7qG7FkYON9mC0uOoDEv3oH1/CLULcn1GInfHDek4P4JOfWy54LbXc 2zfw== X-Gm-Message-State: AO0yUKWPCUKyNHHwEcztdI6W7/kuTHaM7n3kA3Y5rC6NdMHsc4rF0cRP +MPeImQpaxFJeBYqwbZcvVy9pCIl6a8= X-Google-Smtp-Source: AK7set9ZwBhHaiq8PgHnHZpi30bVFoqemTYpH4CqkWTR3ntjpG4ZYW0UY93Gsvx11KtEAuMPsZpv9FPCPgs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:db07:0:b0:80b:8dd0:7b35 with SMTP id g7-20020a25db07000000b0080b8dd07b35mr1614607ybf.322.1674848549978; Fri, 27 Jan 2023 11:42:29 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:09 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-33-surenb@google.com> Subject: [PATCH v2 32/33] mm/mmap: free vm_area_struct without call_rcu in exit_mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: oy43wcwkhna518zgwc51o59g3yoik341 X-Rspam-User: X-Rspamd-Queue-Id: E32B4160031 X-Rspamd-Server: rspam06 X-HE-Tag: 1674849003-212394 X-HE-Meta: U2FsdGVkX18h9Jr8jNYoerN0aNCCD45g+9A3AO00y+C5dcPOalqOep8iP+PbPhUHcjnV4CBhK+v0H20TiAUmHp0nwVsr7MOV/lnZgs6To6Phz8jA632SlbKhlZdDWZt3OOoR+jEiv3Iflm52oqsr5MttH1r3vdVnI1Y6L2G+qhwbvek7bd60od5pJXk0OJITCYz7OGsuxFNNkzl+Tc+9UEpXDeHIYpBs3yYBFXq/FE9FWbYPPduvsoDjHlTGOTILkrNmxudpgW8xVM5uRzlot/b6E42ByBwmQQPRZtcq0xQDxHCN4K3xh4i+MRfAhg8cw2K0hAFIzPbjANjLMTeBqKGP56LgKfI46krXsDwVpuLlIq5yP7H2N54QDLqMoGtvZcyGSci4sCJDqoos4z4/PbLgiuSgNrI6XLU87Io/gsxa4D3A33jjzYwa2qPP30QkxDkh/1b9vNc5U1nkJCcVfMPvcszttEsbEAntzRQvJL7ZEO79ZT3CKBlpevmLKiOBrcixJOgOQ4hDC4R1YADfIdwDT16b4dYZ6pzABtDlLAbDm0RCmmf1SyNSejB3vlqTLEnDWmnLiCiUYOVZyx6KbQUuhbYdVaCYIvG4SUrCsLebJwxaXduPgwC/mX5vOMAqdUIxeZ5xrINNKl/R1xvNhH2OSBQxmvQByY3GiO2lTw5aMx/9dTVkFtJAHdu7nJOmkaC3vfZAb9gtIjk8TtCtEPoX2hebNm/iB2NVyjgJSjjumG06edMa7KACBBrzYGd2H5d2pChPZn3qZac8Lq0s9tJBB4oCFJdmnnCEH6Ja4hBihQy60S38Yp8OX4P3MzOKEFnKD6wSh9r4nRv3Z0IbF9Gi2uaMdwshZEKkXQBo2am59vXVI58YdJLlXquKzNSi7lFrpOYs7dSlLJtiB7vTTywmcYqJK+A5Swm8j+g1JVCNT+cVDEkAAW+QPwIc1jqhW/WTRhIBILyfFb8i3fX lJrLPaps KgR2hbjC2yR0zgJ29LZ1Nn3c0Yx317FBX3Kxm/IHAofG2IsVCAxolSuH9FMSQlTix5xX6wBqRwo/2N6RI8gZyguNKbTb3AnMj3/IT1BQTAAZV/hFMsqL8i3k6iNl9u0wbUJY8dHJFkZWoTzdOVzxKPVkXe+t9fYOew5ghoybK1NGjG0pP3LS4nMVwuqFBBqxX/2WWhI04DyxYEMv1qfjzG7AjGgMNl8UTS+8949oTQmWWBjR66VhwfQOI/cukB5NNiVjsc/4SQinmqyMupHgA7ioZn7LRim9BdIjp9A4SYtG1Ib2bfVV0nQkwZFyyJu474Lk7H01IPqwj3vnIi3XRVHzSsxJXG6bJJ1+IiOVmQCkQ4JGbhGNyOLgZ6owhoeVt84B9SBIsvZQh3rr3r8j0tGwuXC9Nc00Jo9YrViBR1qMeVTfGrBIP0DHUucGknD8jlhqHvWsDU4D5lpPfo4I5y8+Vn/bE+Vfov8S/0M1m1lqeqhs1qPb+Lib15OYhPPJ2rkOtS1K30b44V77iQ3KV6DOyxuFyoxPmSDqTmbffg8ZZWRE+2B1OflYWs/QzXRMtP17qHrhMjF6f834zVx51tSB378xfPB2wXnaIrixV2+uWgkSrUS+zMpmhm0wX+8dEOHRe0v4eETxaEnWXDjWcX0PNBt0PS+qh3gnaB0xv8seExfw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. Because exit_mmap() is called only after the last mm user drops its refcount, the page fault handlers can't be racing with it. Any other possible user like oom-reaper or process_mrelease are already synchronized using mmap_lock. Therefore exit_mmap() can free VMAs directly, without the use of call_rcu(). Expose __vm_area_free() and use it from exit_mmap() to avoid possible call_rcu() floods and performance regressions caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++++++---- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a6de58bb40c7..1c4ddcd6fd84 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -250,6 +250,8 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +/* Use only if VMA has no other users */ +void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/kernel/fork.c b/kernel/fork.c index a08cc0e2bfde..d0999de82f94 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -480,7 +480,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -static void __vm_area_free(struct vm_area_struct *vma) +void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/mm/mmap.c b/mm/mmap.c index 3d0cfbc92745..1028fe131bb7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -133,7 +133,7 @@ void unlink_file_vma(struct vm_area_struct *vma) /* * Close a vm structure and free it. */ -static void remove_vma(struct vm_area_struct *vma) +static void remove_vma(struct vm_area_struct *vma, bool unreachable) { might_sleep(); if (vma->vm_ops && vma->vm_ops->close) @@ -141,7 +141,10 @@ static void remove_vma(struct vm_area_struct *vma) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - vm_area_free(vma); + if (unreachable) + __vm_area_free(vma); + else + vm_area_free(vma); } static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi, @@ -2134,7 +2137,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; vm_stat_account(mm, vma->vm_flags, -nrpages); - remove_vma(vma); + remove_vma(vma, false); } vm_unacct_memory(nr_accounted); validate_mm(mm); @@ -3083,7 +3086,7 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma); + remove_vma(vma, true); count++; cond_resched(); } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL); From patchwork Fri Jan 27 19:41:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119270 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C620CC61DB3 for ; Fri, 27 Jan 2023 19:42:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5BEE06B009A; Fri, 27 Jan 2023 14:42:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 571596B00A3; Fri, 27 Jan 2023 14:42:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 410BA6B00A4; Fri, 27 Jan 2023 14:42:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2FA346B009A for ; Fri, 27 Jan 2023 14:42:35 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0B90E1606B7 for ; Fri, 27 Jan 2023 19:42:35 +0000 (UTC) X-FDA: 80401601070.29.7476709 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf13.hostedemail.com (Postfix) with ESMTP id 4D34F20028 for ; Fri, 27 Jan 2023 19:42:33 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FNDN2+9L; spf=pass (imf13.hostedemail.com: domain of 3KCnUYwYKCO8jliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3KCnUYwYKCO8jliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848553; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ud3T37IGHP0g1lmdTa/tq8X3xMhLxr1AXmnGvd1U4CI=; b=EbbLI1ytiMR6JrLVHaD7yjvmz3jVZtcOJ9QL0qnxDP8nHnLOv2Gw6lOYUfX9OkDLKxEezF FAJsoSAnGA0DP76ZugxG5cDdb68aJp6hSXLTFkjYtC7ZTxruFjcCstDa5vnPxc1UBe6sMo 6cBuxkcrfXMG3BrctwK63OdnZeGInWI= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FNDN2+9L; spf=pass (imf13.hostedemail.com: domain of 3KCnUYwYKCO8jliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3KCnUYwYKCO8jliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848553; a=rsa-sha256; cv=none; b=AVqsILdNBS2K9zPL+iu+SufvJQa3HMe1IO8O5DVXrdnnsivJwnttUPlLwRudzjo0URM1Mi PKmXnZLFImz6Uz3YYL0UbEx8OMuG6rssHB5uWkPAdXW8c6I5T0Lp1Ex0WhsluhUiUoZjK3 cCaBmKpk65mPTc051XExEhqWeNdSubw= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-507aac99fdfso65550597b3.11 for ; Fri, 27 Jan 2023 11:42:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ud3T37IGHP0g1lmdTa/tq8X3xMhLxr1AXmnGvd1U4CI=; b=FNDN2+9L1Vmxr+/mMF6eML5JcVJbNLvQW3fVgvwdePzy3lehe5YuSr9HWsNaMC4a8J FUhI3Xak6vLO90aUXXpIFs6adKFN80x7s3bJgLZLYfmaGEKhkkZLbtXJLX0N/ZyPrSOD vT5YWUMa6D2YWUL+XcKLQxhxuZ8Qre6a0DkbEZ1ct6ULdZqWxPdE1UEo8jeDeLLPMaSB b3W+W/qkByIa+a5SR1bxuJVPYAEoRSrded/1J6aiwU0I/kursScnVvXrI1MYVCgC0Qld xz7h6VUcKpIFgOOniSgMFSpFb+kGVYjmhq6wkneKC/yuMYuw9ACcdFQ59agxDWycdAy0 DJpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ud3T37IGHP0g1lmdTa/tq8X3xMhLxr1AXmnGvd1U4CI=; b=WGLENiBU9XZWfIEN/ZBzfD3FnbyThKh0+4cKOKIJ0dgFdjlWn5aSs+/MboR90gPce4 00UNxB5s7A2k9M0bHXdfAty7141dK//A2Mv+epeDosw6eMcjYNl9wz9YJEWcODd4DCcO I4fyRVFaHeUkjju4jTCYny3YHD0vFR73M6O9ll6lDKm8UNTpZ1Qm1XjeGLWEpUDW8ZbH iA0MgFzwrFQHQVskkTLdBT75XMm9/6s2CGY5aCRv5CCj0DF7PIJm01j9CZguK/vHPrPs oTU9EaRMd89FQt2PNpN1iEr+at229z0KPRljorCdBlOEh331rYtpkPAN1/3Kxle37WKe 06Sw== X-Gm-Message-State: AFqh2kqlG9vRtT9HLc9wp7joLRdXqJ9Tqh8H7k7YXcsGVAKpfm7v1S7I 58Q3L/h7H2XVbAEkHpm5omTzgivOJtE= X-Google-Smtp-Source: AMrXdXuWd7O6+z302DT+NWEpQuMGblny5VWrJhlz+cYeowewBMKBJvug6bhWSAlG5JyuNtY5PTSWQDkfixM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:be06:0:b0:4b2:fa7c:8836 with SMTP id i6-20020a81be06000000b004b2fa7c8836mr5004644ywn.195.1674848552353; Fri, 27 Jan 2023 11:42:32 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:10 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-34-surenb@google.com> Subject: [PATCH v2 33/33] mm: separate vma->lock from vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4D34F20028 X-Rspam-User: X-Stat-Signature: 31izyak4w9bndf74szkd8rudeqcwzeks X-HE-Tag: 1674848553-637510 X-HE-Meta: U2FsdGVkX1+Bd+L4VDTPWf3DUzQPegbNPSoJKd4v8gq4JMO1qzPNJfX5IxIvdmnaUlaLVgt8MLcftONtca7E4Sctm1VCMxyQ815x/GeW9X6Lf/5YhSBPtXUV1xAqWIXFkwBzik3xl4GM8CCYwt9iU0+IxchE9cKkQUAcKwiP02/5ImS6DCOveKIMFsc/rDjwtb7qwbN0AddvfletqA9us5soNgUAxA32oKjVTbI+cyzjYs6wqDPR5b9XbJDa0dIJCbSeR3NFEUY9f/fBHzFYkXzw6Vml+Cu3zXfuM021AjDVJvlEyIP/vL5JB2X2eF/wuIcTqtE34kzPD7N6Rs+XE47PyiPYU3xO15rqdUeSp+MsvkJXH7DnjMyfwOoO6HpCO3tLfPe5qhdvyJucSgxo9pn7umsJVMAfPSGAhiDgROBAvCXMed+nLn6ywqPwoi5CTWq91IZySKqWUzC0KiTC8U9RBpCGQxU7P0vS9Ashn04syeM3u7Z6s18Q3nWCzdimhWIKsDghNLkiJEWUTc/911SM80MG+Oik8Y2uQKKqLkP102srhb2GHwZqT78Sr/HdCOC9MtFd2RecY9fboHHFu7mx87rQLgmsfdxZs/CdbAEoXvbDup14aUp+3m3Iv8qFB6eZ4uc8nNAzwBs5DK3J8buUkcIwudzmPxj+2CpX0Wod4lhfBZpk/Dw6eOkqXcJA83h7g4gLfzStlXP097BRCyTXAI9oKdkkLoipYwR+cEWg5A/KFDkWnheJOpWxvRv3fxckDcFO10gsMRpjHCX749bvJheqnAKzG9CiyVugSFhf1/+b8wHqCVtC5R54ebw6/quLuhgWXcz+Jno+A8yRiXw5Vq56rviU3X616JSpgCMSeKCNyocGFOJZ9WjrqYjwbZO6OLn/qoflzPRiBh0HpZuKkssDB6+c7WAGew6TmSCKu33yycVHZTzCGAa3tHlP2g8XfAFn8ukDNqRP3tT uN3E2T+2 rmPkvK5Za+5R/agCFtGlgMnVa4FbiMTwni7Rzx66aL5r0e2eSeZLrknRdzalbiUqd6n+OI/v7DBOEHLLZQTbVtFL/GjmSibQ7mxN38onFppYxrxdl4tjPrMROG+M1t0CHHTC0UOCwI4iuE8oOqRMl68gx75a7/NSEX7UKW+wC2VjmZFOsuK7VnjI075+2xtODy6pOws5QCD7TIqrnpXzkoo5GZOhRV7Xr5iclYeYDbwERVozPoIp1uwxY0oEv9OH6Us9WR5bSi7tpBxtW0l5pUoM0meXxrBlxitUgVwIj+eLv9MAtP/JV5V8IyPH9ZQ6Qj2jI2UxItbf/CLzmk0C8/UsaU1RuOhcrkIINryHHhOVmZHapGlTNnjvMKN8gT+3ulm47VzqqLVcNhEu9Rx7HL9ovjFOweYG1/RDogJFn3+MaddeguLrSTGBlWWRbmWvEpAezbs8b1TALVbh3HFWEgLuYXISoHLTqGcVQVY6EjlPRuut/Gg95d4Y8rDwaQm3oOcE5gMJ8MX4+1MddZDju1SG2kwPVOCmUpUZUFAnQxcSqKj/c2F0fpNgYYEuuR4if4wom4CttH+Lq8xySfJl0fAZvhHOkzHkYs9l2qjQQxWhweY8aLGBcXp+LdEOTa3LRtCKoiYHOrEKT6FeH+13ucdQb6F4t5HHz7UZg/NGDBBrQw3o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma->lock being part of the vm_area_struct causes performance regression during page faults because during contention its count and owner fields are constantly updated and having other parts of vm_area_struct used during page fault handling next to them causes constant cache line bouncing. Fix that by moving the lock outside of the vm_area_struct. All attempts to keep vma->lock inside vm_area_struct in a separate cache line still produce performance regression especially on NUMA machines. Smallest regression was achieved when lock is placed in the fourth cache line but that bloats vm_area_struct to 256 bytes. Considering performance and memory impact, separate lock looks like the best option. It increases memory footprint of each VMA but that can be optimized later if the new size causes issues. Note that after this change vma_init() does not allocate or initialize vma->lock anymore. A number of drivers allocate a pseudo VMA on the stack but they never use the VMA's lock, therefore it does not need to be allocated. The future drivers which might need the VMA lock should use vm_area_alloc()/vm_area_free() to allocate the VMA. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 23 ++++++------- include/linux/mm_types.h | 6 +++- kernel/fork.c | 73 ++++++++++++++++++++++++++++++++-------- 3 files changed, 74 insertions(+), 28 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1c4ddcd6fd84..52e048c31239 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -621,12 +621,6 @@ struct vm_operations_struct { }; #ifdef CONFIG_PER_VMA_LOCK -static inline void vma_init_lock(struct vm_area_struct *vma) -{ - init_rwsem(&vma->lock); - vma->vm_lock_seq = -1; -} - /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -638,17 +632,17 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) return false; - if (unlikely(down_read_trylock(&vma->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) return false; /* * Overflow might produce false locked result. * False unlocked result is impossible because we modify and check - * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq * modification invalidates all existing locks. */ if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); return false; } return true; @@ -657,7 +651,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); rcu_read_unlock(); } @@ -675,9 +669,9 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (vma->vm_lock_seq == mm_lock_seq) return; - down_write(&vma->lock); + down_write(&vma->vm_lock->lock); vma->vm_lock_seq = mm_lock_seq; - up_write(&vma->lock); + up_write(&vma->vm_lock->lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -704,6 +698,10 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} #endif /* CONFIG_PER_VMA_LOCK */ +/* + * WARNING: vma_init does not initialize vma->vm_lock. + * Use vm_area_alloc()/vm_area_free() if vma needs locking. + */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -712,7 +710,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c4c43f10344a..1e97bb98197c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -471,6 +471,10 @@ struct anon_vma_name { char name[]; }; +struct vma_lock { + struct rw_semaphore lock; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -510,7 +514,7 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; - struct rw_semaphore lock; + struct vma_lock *vm_lock; #endif /* diff --git a/kernel/fork.c b/kernel/fork.c index d0999de82f94..a152804faa14 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -451,13 +451,49 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; +#ifdef CONFIG_PER_VMA_LOCK + +/* SLAB cache for vm_area_struct.lock */ +static struct kmem_cache *vma_lock_cachep; + +static bool vma_lock_alloc(struct vm_area_struct *vma) +{ + vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); + if (!vma->vm_lock) + return false; + + init_rwsem(&vma->vm_lock->lock); + vma->vm_lock_seq = -1; + + return true; +} + +static inline void vma_lock_free(struct vm_area_struct *vma) +{ + kmem_cache_free(vma_lock_cachep, vma->vm_lock); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline bool vma_lock_alloc(struct vm_area_struct *vma) { return true; } +static inline void vma_lock_free(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; vma = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (vma) - vma_init(vma, mm); + if (!vma) + return NULL; + + vma_init(vma, mm); + if (!vma_lock_alloc(vma)) { + kmem_cache_free(vm_area_cachep, vma); + return NULL; + } + return vma; } @@ -465,24 +501,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) { struct vm_area_struct *new = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (new) { - ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); - ASSERT_EXCLUSIVE_WRITER(orig->vm_file); - /* - * orig->shared.rb may be modified concurrently, but the clone - * will be reinitialized. - */ - data_race(memcpy(new, orig, sizeof(*new))); - INIT_LIST_HEAD(&new->anon_vma_chain); - vma_init_lock(new); - dup_anon_vma_name(orig, new); + if (!new) + return NULL; + + ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); + ASSERT_EXCLUSIVE_WRITER(orig->vm_file); + /* + * orig->shared.rb may be modified concurrently, but the clone + * will be reinitialized. + */ + data_race(memcpy(new, orig, sizeof(*new))); + if (!vma_lock_alloc(new)) { + kmem_cache_free(vm_area_cachep, new); + return NULL; } + INIT_LIST_HEAD(&new->anon_vma_chain); + dup_anon_vma_name(orig, new); + return new; } void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); + vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); } @@ -493,7 +535,7 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) vm_rcu); /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); __vm_area_free(vma); } #endif @@ -3089,6 +3131,9 @@ void __init proc_caches_init(void) NULL); vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); +#ifdef CONFIG_PER_VMA_LOCK + vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); +#endif mmap_init(); nsproxy_cache_init(); }