From patchwork Mon Feb 27 17:36:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153964 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AF49C64ED6 for ; Mon, 27 Feb 2023 17:36:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C34F76B007B; Mon, 27 Feb 2023 12:36:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BE56B6B007D; Mon, 27 Feb 2023 12:36:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A86DB6B007E; Mon, 27 Feb 2023 12:36:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9A8266B007B for ; Mon, 27 Feb 2023 12:36:44 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4BA60140B06 for ; Mon, 27 Feb 2023 17:36:44 +0000 (UTC) X-FDA: 80513776728.25.DE941AA Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf22.hostedemail.com (Postfix) with ESMTP id 7D7D2C0013 for ; Mon, 27 Feb 2023 17:36:41 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=J1npsbv1; spf=pass (imf22.hostedemail.com: domain of 3J-r8YwYKCBYEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3J-r8YwYKCBYEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519401; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9oOfd5ioBGFgnDThXBfBlaouYrbK90Ol7CTAINZYLi8=; b=ZVYDCot2UrUDAPpM/ZQoK+tlYoZfIkkTNRFbTs5y25AmLhWZUbVNSnmaa3lUdPxsE0KTd2 BiCe3+eUcw/oVtzoA0ChqOZKwhOSUYEEz/awkfnKOWOx7aGIJI/sqFHEGtGzPOW/FlvewV QxsbyaNSHCTi0y3tHX1JvVxFMa55tsE= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=J1npsbv1; spf=pass (imf22.hostedemail.com: domain of 3J-r8YwYKCBYEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3J-r8YwYKCBYEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519401; a=rsa-sha256; cv=none; b=S4g5JdoRAQuih8i+wPkeATTJV8XQXEQGnymixpd6te0IwXqLKzldAScZq3//4aqryv2esb ezAB/WZRmn1cboEPzwPVnICwt5nNaCeGquHvf3Y0uM/4Va+nQJMyfE/AhZ0aYAtuuAZ/zl +iWVmonFRv4PM2kjxSneqxkJXty0iW4= Received: by mail-pj1-f74.google.com with SMTP id u8-20020a17090ae00800b00237e4f46c8bso1449230pjy.7 for ; Mon, 27 Feb 2023 09:36:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9oOfd5ioBGFgnDThXBfBlaouYrbK90Ol7CTAINZYLi8=; b=J1npsbv1gUS6Xx9BiEsQjP7XeeP4aS0zilvFmnv/Rz+GOr5UTDVWeJrWl0j62csJqw pABdpewltkyMDTz5x9er0/tmyJAZh7x/KhM7c8duF9xadmsMayPg+H71fGIGq8oPKWOj MgbVwv2JgaJHWGglF3BAvt4DqnEv/mXmkg9toQmUgHu4O8sWHghp6pe4qMmBQxfPPa9Z PixCDkJQkwa7ilKkjAeCuQR96NDmtVqp0ZmLhbk++NpcJ1RgmTRpPPKm+6LBvLnGJc6j 5RqWVbaNSaafI5Uk5daP/3bMtRsXgSv4TZmCrUrgFU7+20OuWfc4Q8qOQnmnQqR2Eexd RlJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9oOfd5ioBGFgnDThXBfBlaouYrbK90Ol7CTAINZYLi8=; b=caZL6bODe6krlYqrvnxwhVUiUsXcgjchBrAyRNKlPQn9bRZ5lzFnD5l27oURQiLRlK sDVk80ZKmHT/Zjp8JlcESYwX3oBVtuEtwKP5ipBKz3IL5jjeVe6G7MIaFyOKwv/s51gL x0O8Dw2zYZotSGA9WRSWNEnO5gU6T3t/3wOiBDo0Fl6tfm29yM2KPm1XQ5cIT8EkNJBh CoUPIvoKk39GfyOK2xwkuiExst3L3aRKykfoosr0vCp+rSBVccrWQUyI5vG5MtORyseE DDnOqqlQyMjUBUFn70sWu32/JkZz22i4zziPCB1gavPV2Cdgqor/1pT4svCucq8H4S57 5I4g== X-Gm-Message-State: AO0yUKVotPQQnolC1IB1/Wbkd6t3F9a752CQ5NfRP0oZmzVlm/kM8ARz LXBC39iOuL+xdAF8IBQB3qu75iB7+oY= X-Google-Smtp-Source: AK7set8H0aV6OfunNhl/Vx76nzg5qpE5z0RrBhYwaC/okN28yJslkLbmkGTcNV4WqgwtQAFy3/Xgq0PVYIo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a17:90b:17c9:b0:237:ae7c:1595 with SMTP id me9-20020a17090b17c900b00237ae7c1595mr187914pjb.2.1677519399986; Mon, 27 Feb 2023 09:36:39 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:00 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-2-surenb@google.com> Subject: [PATCH v4 01/33] maple_tree: Be more cautious about dead nodes From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: m95qi4yhjqm51jrx56ir3ameqkqi5nma X-Rspamd-Queue-Id: 7D7D2C0013 X-HE-Tag: 1677519401-214590 X-HE-Meta: U2FsdGVkX18kU9CP3qT09h8QcPvAQGr3Bb8GFjCcs2oDDjQ2swYvXCfKCeSBmEPgvfbY2VFR0b2X6jSvqS+/rWjKE8RjyYPktySNDm2WjSKR9pmULGLmm1Ekz4XSX7JxZYvJkFALhhAxZ12jbFVZWtMHDgw2JWzYcCW51p5lqLeE5ouQusxy+F4tXmCRymmIdy44JIxX0dwXXUD7tzDJLmVTBSjN+L8snghEmMWny5Apyo2gTU/1GApvvk/R4IJw1/TRnc6YtDMW490CRLm18uwJtRdQ7/y47XWyondboDlXe7XMD9Q0vQfbZdfQTJnxmg88ke6v6lvIgpKFZMgQQnVWKTjTrtACS4gVrk2iq4lmXbRzQU0lXg2o6rzAf5UtDEem8MfPDZTdKq1v1g/c9CZOPx+GxHUFQaJ+30IGEj6Tl6a1LsW0HEaG5JjhyHDiktKTXB9ijkqSYwXtuYKks7hqFfvDg2srMmlSUfQ2JSaUEbwEfQUNwB30fPTRhoyvXOrNrGukUyfJmpb8vRyqoZ3SDgoQ174YFsC4k43Vh+Yw8lHxsqN/xmVzcQLD+b2EJYmedz2PiaauasbCwkaASvYkaZWGtUBc/81lSM3rMqDeGILNCi2Ozy2N0RFUcijiXcr+xk60yK8SM8yTZ+ntOfKMZ6CCXyy2OEIQnFR0dZAt89x2uy4tKDiYw9Fe3E/sqiuilsYC2IjxM+5YT0WRf+6EabHlaiD0kbMciW55biMPVWFLcUrFolbbkoj/4nx9jLzP+hhmzMRT9AXxig5pJdXtlJ2uVT0Vjv8LOy6C/y0inAmCA4T5b09MrTuJXixc456AOfTSqUHO/nefqPASHFY51B3aA68fvoVgqaP+e7wVzqbHMs82R2bvhjxiVu6VJmQmegLmV+PeiDx/SWW5A66i+RhPbP4Ak8HWStCiFYgd3nkKLXpXfGQFfl82R3XLjJMUHuJxSHrbefqN8GE s5MSJ8Ve Amxgr7aN3UD3g8sl8Rx8xJRejjC293TylKsOBtThVwkL/OAG8FfT2Xju2/WkZEbST//Lh3mSl/g09GPCXtmoSyq1pYDNPiVZpIS4X6ZrFfUzfa8Ht/1Y4FCtcEUjsIklzgPnh+JS3qP2RkO7n4exavbHEBCHK/qQb0vhid0myOuY9xaptQp7vuhH7f0WDdbPaVTGrRcdP9pIqELHjNZo5djyPRuHvbb+uEtNHe1Gkyf4He9nYA78CMHsGYOXCxwR/cGQfVXdI0tsEYzNmlQsBfNJdgNwBLz1frO3yzCaQP5VxaLne/IU6kdGoGDqKmAFjSHaBXDEB2bWHPrWCtRVF4XSBA5UTfl351Y2OJwbci/xUicRXxVzBX44v+yZ8i4CQeqUAv/bvAvIUio+zYlFi99VaysuyGfWYNyZAGS5sm66aiUZ6Su2pK2OpyYpyIq2DaUGqXu6Ef7JleePiJAKvSfENPcfn+AHFOkgS2aQAM9FGQomK5xgMuVFRhiYSlvMcDsjD8KYDFcBM1XLpJV2KWyCWkZsFnv3KpVISIX8I+XopTehmFFVEbrvdCAKj0y1muZ9Vre7QL1asfmOzBdOtebVZRxkFwanlaD/L4VJB4fB2jgV50KxcHnwWOxof10BcfWFr7+sHlC9s58rf3tVtrEjLA61fhAz/D53UWDUjjINmdsdTFarg+IjMEj7kL5RuyQNx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett ma_pivots() and ma_data_end() may be called with a dead node. Ensure to that the node isn't dead before using the returned values. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 52 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 646297cae5d1..cc356b8369ad 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -544,6 +544,7 @@ static inline bool ma_dead_node(const struct maple_node *node) return (parent == node); } + /* * mte_dead_node() - check if the @enode is dead. * @enode: The encoded maple node @@ -625,6 +626,8 @@ static inline unsigned int mas_alloc_req(const struct ma_state *mas) * @node - the maple node * @type - the node type * + * In the event of a dead node, this array may be %NULL + * * Return: A pointer to the maple node pivots */ static inline unsigned long *ma_pivots(struct maple_node *node, @@ -1096,8 +1099,11 @@ static int mas_ascend(struct ma_state *mas) a_type = mas_parent_enum(mas, p_enode); a_node = mte_parent(p_enode); a_slot = mte_parent_slot(p_enode); - pivots = ma_pivots(a_node, a_type); a_enode = mt_mk_node(a_node, a_type); + pivots = ma_pivots(a_node, a_type); + + if (unlikely(ma_dead_node(a_node))) + return 1; if (!set_min && a_slot) { set_min = true; @@ -1401,6 +1407,9 @@ static inline unsigned char ma_data_end(struct maple_node *node, { unsigned char offset; + if (!pivots) + return 0; + if (type == maple_arange_64) return ma_meta_end(node, type); @@ -1436,6 +1445,9 @@ static inline unsigned char mas_data_end(struct ma_state *mas) return ma_meta_end(node, type); pivots = ma_pivots(node, type); + if (unlikely(ma_dead_node(node))) + return 0; + offset = mt_pivots[type] - 1; if (likely(!pivots[offset])) return ma_meta_end(node, type); @@ -4505,6 +4517,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) node = mas_mn(mas); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + mas->max = pivots[offset]; if (offset) mas->min = pivots[offset - 1] + 1; @@ -4526,6 +4541,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); offset = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + if (offset) mas->min = pivots[offset - 1] + 1; @@ -4574,6 +4592,7 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, struct maple_enode *enode; int level = 0; unsigned char offset; + unsigned char node_end; enum maple_type mt; void __rcu **slots; @@ -4597,7 +4616,11 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, node = mas_mn(mas); mt = mte_node_type(mas->node); pivots = ma_pivots(node, mt); - } while (unlikely(offset == ma_data_end(node, mt, pivots, mas->max))); + node_end = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + + } while (unlikely(offset == node_end)); slots = ma_slots(node, mt); pivot = mas_safe_pivot(mas, pivots, ++offset, mt); @@ -4613,6 +4636,9 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, mt = mte_node_type(mas->node); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + offset = 0; pivot = pivots[0]; } @@ -4659,11 +4685,14 @@ static inline void *mas_next_nentry(struct ma_state *mas, return NULL; } - pivots = ma_pivots(node, type); slots = ma_slots(node, type); - mas->index = mas_safe_min(mas, pivots, mas->offset); + pivots = ma_pivots(node, type); count = ma_data_end(node, type, pivots, mas->max); - if (ma_dead_node(node)) + if (unlikely(ma_dead_node(node))) + return NULL; + + mas->index = mas_safe_min(mas, pivots, mas->offset); + if (unlikely(ma_dead_node(node))) return NULL; if (mas->index > max) @@ -4817,6 +4846,11 @@ static inline void *mas_prev_nentry(struct ma_state *mas, unsigned long limit, slots = ma_slots(mn, mt); pivots = ma_pivots(mn, mt); + if (unlikely(ma_dead_node(mn))) { + mas_rewalk(mas, index); + goto retry; + } + if (offset == mt_pivots[mt]) pivot = mas->max; else @@ -6631,11 +6665,11 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, while (likely(!ma_is_leaf(mt))) { MT_BUG_ON(mas->tree, mte_dead_node(mas->node)); slots = ma_slots(mn, mt); - pivots = ma_pivots(mn, mt); - max = pivots[0]; entry = mas_slot(mas, slots, 0); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + max = pivots[0]; mas->node = entry; mn = mas_mn(mas); mt = mte_node_type(mas->node); @@ -6655,13 +6689,13 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, if (likely(entry)) return entry; - pivots = ma_pivots(mn, mt); - mas->index = pivots[0] + 1; mas->offset = 1; entry = mas_slot(mas, slots, 1); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + mas->index = pivots[0] + 1; if (mas->index > limit) goto none; From patchwork Mon Feb 27 17:36:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153965 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2931DC7EE23 for ; Mon, 27 Feb 2023 17:36:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B728F6B007D; Mon, 27 Feb 2023 12:36:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B229C6B007E; Mon, 27 Feb 2023 12:36:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C35A6B0080; Mon, 27 Feb 2023 12:36:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8DDD56B007D for ; Mon, 27 Feb 2023 12:36:45 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 35A8812039A for ; Mon, 27 Feb 2023 17:36:45 +0000 (UTC) X-FDA: 80513776770.15.4A81A39 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 44BA140004 for ; Mon, 27 Feb 2023 17:36:43 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="Rtawk/5q"; spf=pass (imf04.hostedemail.com: domain of 3Kur8YwYKCBkHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Kur8YwYKCBkHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519403; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lArWv6/BKpLmlLXC6GszXodHEO09dxzXTjGiFRBLB5o=; b=3ve/8mTg8UTxhUBjp4TBAVBq+ZJvjkZSYFjQF9WU6U37gxhX7gre/PSQAehquVR61AOc3F 4o3BMjF3HYTOTenkqKj3625WfDAlMZP9cO64q+BxWPNsKeTWQ3wplHSYZ053AouR4OIfhp fULCVpZ2nBuYD3XfroPVdcBem4dAxzU= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="Rtawk/5q"; spf=pass (imf04.hostedemail.com: domain of 3Kur8YwYKCBkHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Kur8YwYKCBkHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519403; a=rsa-sha256; cv=none; b=M+U6tCTBSSC8TDTBXEsi2bRKTqrh7qgd5fZZYgoregsDJzcpVi+/cGOwyGpcPMo/oQOqAL t93RCMQtyepMg9OqCCXB8Zcg3+LhOSz/0H/A9B1GzlHcqjIdnfNkpzEh6xxWDjx/PDme6J mnxeygqbkEhq/+xtkmmMbIy0O+vRmcI= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536a5a0b6e3so155576817b3.10 for ; Mon, 27 Feb 2023 09:36:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lArWv6/BKpLmlLXC6GszXodHEO09dxzXTjGiFRBLB5o=; b=Rtawk/5qB+wQAwcC1iD4VXuhoYDvFfWXoR8TLDjNU0mxzXG+OcIc5M1YDwWsJSdzc1 5ov0BA8i2CNkZg28B06fzU5xhMSs1uI1giDFkRO+7bYBZMhhkWnMlBAHldMBIWHj/nI6 Se6XX7qnInZ409hEUwR8XAgKIrmPJD89Fz59bDJnlZVSabti6XILWdKtE9EozNXeoXJo jeE4G4VS8SUdBqPqEoVdM/8Bov6AoMRWvMj0a5NQOXvkupkTbiI158Odu2A7RSMPsE77 crrvpWC4xPR29dzTvNq/UerrAnu3X0id5UHZqYhq5AWR42pdWVuHy+UOicXnyhpFqhVJ jDJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lArWv6/BKpLmlLXC6GszXodHEO09dxzXTjGiFRBLB5o=; b=vNP8p/0y+y3ysPjgvN9rERABo1cdz9RtNTg7hBz2aWz9zvcpZX2o4vTKdu59BWR20J JS1Qy659V6VU5j8OHW6v7Y4XFRpTJhEbaJcBJGURai0Y4mUxK4GlG28JgyDzbjDrVVLd oa/JZt0GQbUdS2sT0N2eALzQfBoCaby3xZhX/xJxRSlsQPimp9LHuKfYEfV68n65IhLA HR4dpV7bkYDfb4xWu5WVHZ946zEn1hItGIdI6zC5z9E5VVde0aHVOpDRtOSYuVOERrfh c6w+rWGioHYC5UUBtgOvIudHve2q16iSNcsg/+6/dEthCOqEShCMlVBcIqLQi75ohf7T 2N1w== X-Gm-Message-State: AO0yUKVJOzcykulzLh0S3M/90qY5TjJ6fncuIqZmVikzNm7XC3XXlHN3 95+Xb63oBvPn+qmWLSEFrODRNsWLs2I= X-Google-Smtp-Source: AK7set/6BFahylhHoQQLwR3PqP3k7QFJ6TbZ44w6th+v/3oGX1dR6WIuOL9QS1wjUZYjoh9QnZVdIWwPxxI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a25:8890:0:b0:a0d:8150:be04 with SMTP id d16-20020a258890000000b00a0d8150be04mr7185754ybl.13.1677519402335; Mon, 27 Feb 2023 09:36:42 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:01 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-3-surenb@google.com> Subject: [PATCH v4 02/33] maple_tree: Detect dead nodes in mas_start() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 44BA140004 X-Stat-Signature: jnshk1irazsxsxn38fy8estgsapnhczi X-Rspam-User: X-HE-Tag: 1677519403-856423 X-HE-Meta: U2FsdGVkX18ND5ZXd+YuVt5BsNtMQJR+Uv9LlZf9fbiWvO/+0h8xITpggvqvHK0lTjFdjyWYz8UXivf+8sttEn8HD6++/dfKvm9tBgdRRXn9zaiBTwk36CYv06EImrAhoafp2na7JgrDRcD2E5rAemT7OnyL+nnTkzJGwc3Sh6gJSfpZy1uKlFFnmTVB3nLS+CExrhS162Gosukq4EGhI08TbTeuzLpcd7Ctr9yJV+0k4gIoQ6UMLM0HI1R9ax809bZALNBf+KfzhntxCNXf2ZAG/JpUkoHBZAWu2Nq5P2d4KbfPmfvZSvsBEqFtLOOgwyB26BRWHiq6+nmOH2xJcNf6i3TAXvzmDQk2Mpuwaioj9yD9Jjm7Daslo7fxbsqwBAuF6UUB+LlJpT9HhkLnlpLFpaQ16MIj+hdCBwC1oG5Iqm/3nE4RiuYLGWt/lMvLDFQdBG6y1HO4hbwiDsn7/O2kajcGkH3UJbmf05FbZ+xWKZ8uQyl6nYmggFFxFJs/g58zXUsSvEHScBGeTYQtWqV4SVehrgjiESKSJKi6Ks8TK71Op+HM/4s0oE7NEmCEUTBlYWiFSjKTBHoXXdFsCe6avp+x4wVCA2BZqoVnye0x3dOzT856oJkLf6CyFtrfAantOz9/KEi0eMWkWRSXW/ZqW9vrA6ta44JQh/MQORxdpwJWrVsSXx17C+7f6rMzt87OSh+JwHEIjqxrH3UMW6BzwEJ/sygXHlVAWxBL8btTZM88Ov3bW26K13v00+KQZCrfRaLvsK62FkDkBvuAf3KaXH8zyKhXL/6BLO5xabuzMoNGJECXh0K+/cA7d7Uw7AFgHIDzuVy6jXw02ePKTX38cAWwqzg3ksqSzlv4Nw5nidaPydA89AqeBJOppmLM7fWt82Ohxx0UIiGezUd9tTQTlWqjSg/QWvlMh/McQ80A0Lr4YvA0Sv/dDAz/HXgXpY/S4EntXNTklO1QHJ3 gd/+6bJI F6PlOO+XDHiaExIEsyVGp94ld+UG3OvLF/nbnOLTsczZv2vmy1KvQy8EZHkqPx9FQuEfv5t0osf/ItPN1jKKBrcNFc4SGG34LfgQevlsHSMfiPDT9MbaMMYlxD268L8PDaxH8zozrgxhzDKiByLvBuVhVGh3AuH9h01/xtlJtRWUvtdjRqmBfsrRZClKnV+ejwSXb+ttEIIkkwM/19cyCTUrTI8TJL34xvaVVMPdQO9FDWQ6A7cBZssZyQiwri3rYwJO21XCNGPNHJA7UgQ0Rqy94w6KKYek14kazXaujUdI9bIZcqi8P9yKfQwFZJAH4VH19jPVkj/nNZ+CycGsAQ557aa/TWNhO4ErwbvYb4PX8EAwxR+jT86bwsED8K6JUfk4mfL2YMu/0OsSrn8VP92ShmGQCl9zm/a8mFh/AB4/N5Nb8m/Dg/jSGnDFjrWOBA6/yBgTMVdSmAdDhEjVaqh9x/SyMgyFItFxIU0XFnStPOtsqwJ6zVPsoBL7KZQhA5fZQgDHjaDJTZeg6vI63x1TSidqcyrLxEiFryEjSBG4E0fqp2LZNntbbKvUIszmX9hcsSMA8F44pL1zm4Asg+cxw0GrVjBHAWpO+cSzfRT49EVuOA7zhO8TsEOzsDfEjKDOQXxcIDqfYv6wHgJqEvaJGvt4fjAE8lw8fSKyNS1cZXHDxAeSZqsgnEO8q/jRxIPsP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett When initially starting a search, the root node may already be in the process of being replaced in RCU mode. Detect and restart the walk if this is the case. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index cc356b8369ad..089cd76ec379 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1360,12 +1360,16 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) mas->max = ULONG_MAX; mas->depth = 0; +retry: root = mas_root(mas); /* Tree with nodes */ if (likely(xa_is_node(root))) { mas->depth = 1; mas->node = mte_safe_root(root); mas->offset = 0; + if (mte_dead_node(mas->node)) + goto retry; + return NULL; } From patchwork Mon Feb 27 17:36:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153966 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2006FC64ED6 for ; Mon, 27 Feb 2023 17:36:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B9A896B007E; Mon, 27 Feb 2023 12:36:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B4BC46B0080; Mon, 27 Feb 2023 12:36:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A11F06B0081; Mon, 27 Feb 2023 12:36:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 90D436B007E for ; Mon, 27 Feb 2023 12:36:47 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 444CD40A69 for ; Mon, 27 Feb 2023 17:36:47 +0000 (UTC) X-FDA: 80513776854.20.8AE6E90 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 8064340019 for ; Mon, 27 Feb 2023 17:36:45 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Am83ro2Y; spf=pass (imf04.hostedemail.com: domain of 3LOr8YwYKCBsJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3LOr8YwYKCBsJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519405; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=orUi5wHnC++aji9CdkMMFCsYDd1RwNIxCQdLrt+cFY0=; b=Dov5T2X/3139z9CzgdEx2SW6Akzl6uUjUyBV/YbIun9+b740wLLXz0vAQ7Vq067IRpNZK8 z9SItfvagn4FI6Hr81aqNKnUF8SFGh+xmw3rNtHUhzFlR8yrhbuc9ap3pSzuNSSe2LsS9Q Ig4Cm2QR8Gx2b/af8Xr4h9SYiD9Manw= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Am83ro2Y; spf=pass (imf04.hostedemail.com: domain of 3LOr8YwYKCBsJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3LOr8YwYKCBsJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519405; a=rsa-sha256; cv=none; b=CNymojD73XzL34lLYTnWuUb9IdJqUvQJlCE8PkAuEvDRaUQJxABhiU4XuI3lchqJ/tKopF +O/MD+c9CxLLo0UwqoevUO8X9YsvJtnMQ/QBBAEm7C7MC24hLtex3lp1gcpih+vXb5slIT 3zW44JQeeKtPqOC9Yg9a4Ukh0b014fw= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536a5a0b6e3so155578887b3.10 for ; Mon, 27 Feb 2023 09:36:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=orUi5wHnC++aji9CdkMMFCsYDd1RwNIxCQdLrt+cFY0=; b=Am83ro2YmRzZ4VgyIBhDoTNrPllqfw/5UTqTUL7yZCncC06+SMYT4eArFva+POwT7g 6A/0/pFhHTsDxJdieAltqSAe6HtiFFVlFItush+XbcRSxs2FetQjqrV9RzDHQ5xUXxy+ EFVFqHW9b/Vfsu1fjnNuXuTzPWG0kNjoL02ASewZm4aTLFi207WQmvOLdMhmY0rXGsBi NgW5j6vPb4Yb97AotYvdQXAzm4p61g/PV5t8r9ru9nwtbYld830L0Sd8TDb9hbnZaf2S Co5g3ZCxIRR9yw3YwCJpCKA97OC2cOIFxPWxIRbJAgeIMHg6EAO/atx61gUeYr5Wxasb jnNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=orUi5wHnC++aji9CdkMMFCsYDd1RwNIxCQdLrt+cFY0=; b=3dwbYgVG9uzO33qfGrgn7tcSVKWGfp+EmHBsiCO3+fDpGAA5Ibl6XNY6UP0fqZL7xd QJtN2/Kdu7csWphzysU30T5LSwVrfSg3AOzaXQgmOHef4dVqEMBJk4Jnj1U78eBqhJCW JDWQPMdfP4jU2/zOzAUvLgxS7DuwS5AE970pybSa7euLaTMsoj0KsOm2SdT+KXcooWPp QYiGEp4t7J6on/I0Qt8xUWknHyfAJfpLgTpvBwip44KV40UwK22ADpNPFjUXM0dbYIh3 d944aBvS4+MJIOKQbGq8LRsBaOHCnKP3mRGW8aXPrtrBO+0AuAo2vW4eRkc9dFKi6KcQ t2og== X-Gm-Message-State: AO0yUKWAEubdi51T+dTcfAcoU7Bi+ecjVQP68P72/Nh441UTyhVheMcL x0VmnSua4WbnrVY1p7xQpqg/9jQocM8= X-Google-Smtp-Source: AK7set8ivfKmLT2u3vpwWqjPXs4Ue9yElb1Co2/gr63/ZMmCdo5ef9MT+bBbe/btFlir1Y+dX6RcjVs74cU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a5b:ecb:0:b0:a03:da3f:3e68 with SMTP id a11-20020a5b0ecb000000b00a03da3f3e68mr9395474ybs.12.1677519404853; Mon, 27 Feb 2023 09:36:44 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:02 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-4-surenb@google.com> Subject: [PATCH v4 03/33] maple_tree: Fix freeing of nodes in rcu mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8064340019 X-Stat-Signature: t6nbufx7c1s5ssebc644ofakthy6e6iy X-Rspam-User: X-HE-Tag: 1677519405-427106 X-HE-Meta: U2FsdGVkX1+7bRyRLW6b3PKvgKvckWSADbVJyjnNGcI7i+D8MqLd89U0qmfnioa34NT+ArI8fZrYo/u2ocFW95OYn5U1xIpUw1E3tDNay0t5kpSu1XhYBBdl0pYxwUKlhAc3Glm1/r+FYoZsFzw1ced4UGIdpdeqyYVktxtsQXS38kgL1XEcPVy8QkknMC78iX8iPq9y/eaHcnOw0KGhgi+UtMzWddY9BwngmJqvRMx2EbRvDpqXlwfDMGp9iz4PcEMPvp/6GmQZXooOH4iP44l3lrICpEtwJTeY0vjvklz51gFXGfm3EcNbLVYC6Ot6rJtqqyyy/ZdNSWjuYRF+PmEGbaXHnMsD2Hp1/6TUM6e2v13+IvNCLIhYvh7tLDNHbSCsQXX+9iGPzBLQVuKStmxkApnd/oM8r2Z0NqNS572vRUH1deR99xz9gQlaXX0DnKKyXAapTXTifZK272Do7EMf4Qy6SUmM0Wakxl6syzmAmt81kSDVqvV2SlMlqtS4XGlhzr5MIN8Ih1bOhYxktD2yHifvAyuukpHLnOaUVsasYYA2LQtdRd5Zs1hP5kz8wzHO8InGTMIoK317FHJZbVWjxsJ2cwk+eph3l7w5YM1mGRJalL7H9NYQvBfhb289U2FUVPBEEA+ua92GWOv+CSI80n+oTis33DulMtc0d3EyR9SFqccN2SLtgVZXkIeF3+vAIG8vd3Dg3e9RqZp3KnExwTD9N9VgLJa4ZpEC5gLdNmFvC8C9oIDL94Zo70oCi0YhqBd/0/opgNfRpCj57tNH0sLtqQSUnbfGdw/Au/CpzsQKeSG0fMLZWHYsqbFxjD3Tfm9w6ii36P8UkEhnkuYoo0u3kimvbLgAe/utt0vnu/E74+XbswWg6R5bxlALQrgyYH1sB5VTJdifwJbfV13Q6mBNb20PlIhavy4lWxYe86bV0M/WQM3hQtwLb5VtG9gM3PGtBdzXbK6XGn+ FG2edh2D 7hKxUsLSeb6OMSc+S0KWMkURx/+wIyR2g78X7IDZJ2Lazr5KWIWWcLQCqy1QxxlNr2HkPhv6aNEo1v2YaMUSdcYD7b0DFCDwxTl9piuEF7vnYLLWQVVHTESf/B1pgNk1wmMWNEPksbdpoQkp+31iTauvknbRGEe7vRol0YuCIh8wlQv2wYsJI6yM0GgebzYA9SYUWY9JoQW6l0zUEXBzXbJ7B+UCLAhg4NeD6LCcFWcEr6tfGEJyBKTsDo+ahEh/P0O8AImW4s1tJr/aORj8srSXdlvCSY0OX099VDTmeklIRjd2S+tkbWMGwtUUcsDBhgigdvW/Se+l9fQ5rQfszPabxwSsnahAgzYMU+cKi+GPVODuexMkf29Kn9IeT2wz3TPO0KQCs/j9jSM6YxFtGXwTrFvsdl7Fef8aiQMkt/sYJY39oU458yDVXMJzCJo6cnG21woTArMDy8o99C4H/aU/kDy+xy+Ww5VFPWvLdB9Cgfp4ipdepkLzfU4IJNbT37CNOKHeDFgvafNOb3S1L89sc6+QCnrmkEeiUihzn66bvkH3Rxfh0ap7Y2NSigWiciSXw3xXjHlU65VZQJMGVsE5mKEykND0VW0Jw+iGjBUrY6IbXc0vLeK1W9KiB8E35nwaYvnQ5KHlHCMTy/J9Zs2sdccr9MsrT/4vKRBiel2DmAuRe4WdgnofpUxFX9a6hYq+o X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett The walk to destroy the nodes was not always setting the node type and would result in a destroy method potentially using the values as nodes. Avoid this by setting the correct node types. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 73 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 089cd76ec379..44d6ce30b28e 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -902,6 +902,44 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, meta->end = end; } +/* + * mas_clear_meta() - clear the metadata information of a node, if it exists + * @mas: The maple state + * @mn: The maple node + * @mt: The maple node type + * @offset: The offset of the highest sub-gap in this node. + * @end: The end of the data in this node. + */ +static inline void mas_clear_meta(struct ma_state *mas, struct maple_node *mn, + enum maple_type mt) +{ + struct maple_metadata *meta; + unsigned long *pivots; + void __rcu **slots; + void *next; + + switch (mt) { + case maple_range_64: + pivots = mn->mr64.pivot; + if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { + slots = mn->mr64.slot; + next = mas_slot_locked(mas, slots, + MAPLE_RANGE64_SLOTS - 1); + if (unlikely((mte_to_node(next) && mte_node_type(next)))) + return; /* The last slot is a node, no metadata */ + } + fallthrough; + case maple_arange_64: + meta = ma_meta(mn, mt); + break; + default: + return; + } + + meta->gap = 0; + meta->end = 0; +} + /* * ma_meta_end() - Get the data end of a node from the metadata * @mn: The maple node @@ -5455,20 +5493,22 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, * mas_dead_leaves() - Mark all leaves of a node as dead. * @mas: The maple state * @slots: Pointer to the slot array + * @type: The maple node type * * Must hold the write lock. * * Return: The number of leaves marked as dead. */ static inline -unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) +unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, + enum maple_type mt) { struct maple_node *node; enum maple_type type; void *entry; int offset; - for (offset = 0; offset < mt_slot_count(mas->node); offset++) { + for (offset = 0; offset < mt_slots[mt]; offset++) { entry = mas_slot_locked(mas, slots, offset); type = mte_node_type(entry); node = mte_to_node(entry); @@ -5487,14 +5527,13 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) static void __rcu **mas_dead_walk(struct ma_state *mas, unsigned char offset) { - struct maple_node *node, *next; + struct maple_node *next; void __rcu **slots = NULL; next = mas_mn(mas); do { - mas->node = ma_enode_ptr(next); - node = mas_mn(mas); - slots = ma_slots(node, node->type); + mas->node = mt_mk_node(next, next->type); + slots = ma_slots(next, next->type); next = mas_slot_locked(mas, slots, offset); offset = 0; } while (!ma_is_leaf(next->type)); @@ -5558,11 +5597,14 @@ static inline void __rcu **mas_destroy_descend(struct ma_state *mas, node = mas_mn(mas); slots = ma_slots(node, mte_node_type(mas->node)); next = mas_slot_locked(mas, slots, 0); - if ((mte_dead_node(next))) + if ((mte_dead_node(next))) { + mte_to_node(next)->type = mte_node_type(next); next = mas_slot_locked(mas, slots, 1); + } mte_set_node_dead(mas->node); node->type = mte_node_type(mas->node); + mas_clear_meta(mas, node, node->type); node->piv_parent = prev; node->parent_slot = offset; offset = 0; @@ -5582,13 +5624,18 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, MA_STATE(mas, &mt, 0, 0); - if (mte_is_leaf(enode)) + mas.node = enode; + if (mte_is_leaf(enode)) { + node->type = mte_node_type(enode); goto free_leaf; + } + ma_flags &= ~MT_FLAGS_LOCK_MASK; mt_init_flags(&mt, ma_flags); mas_lock(&mas); - mas.node = start = enode; + mte_to_node(enode)->ma_flags = ma_flags; + start = enode; slots = mas_destroy_descend(&mas, start, 0); node = mas_mn(&mas); do { @@ -5596,7 +5643,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, unsigned char offset; struct maple_enode *parent, *tmp; - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; @@ -5620,7 +5668,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, } while (start != mas.node); node = mas_mn(&mas); - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); @@ -5630,6 +5679,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, free_leaf: if (free) mt_free_rcu(&node->rcu); + else + mas_clear_meta(&mas, node, node->type); } /* From patchwork Mon Feb 27 17:36:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC845C64ED6 for ; Mon, 27 Feb 2023 17:36:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 752516B0080; Mon, 27 Feb 2023 12:36:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 703126B0081; Mon, 27 Feb 2023 12:36:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CB3F6B0082; Mon, 27 Feb 2023 12:36:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 507F86B0080 for ; Mon, 27 Feb 2023 12:36:50 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 22BB9809ED for ; Mon, 27 Feb 2023 17:36:50 +0000 (UTC) X-FDA: 80513776980.22.BE68289 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf16.hostedemail.com (Postfix) with ESMTP id 3F08A180020 for ; Mon, 27 Feb 2023 17:36:48 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cLE4KbCr; spf=pass (imf16.hostedemail.com: domain of 3L-r8YwYKCB4MOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3L-r8YwYKCB4MOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519408; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fnkDgHrJ0cqyk4yVnPbndf59jVemOaMLlO/iS92oZbE=; b=INGh4Diu/uS7KzfSYxGLADmDUS1aau4IJ6r/GbiHeO6XF8ffEpy6GxOnEAuz3nnoPE5eLi WtW5ecixfU+5Z8cxPsRNaUgD/uUvPDUqPfZseJYmWYzrVLjosq2Uu0lSaJeUieSfwoDjbd GM9S5MgZ5CHSRpbyWAs1kVqiv49ZjBk= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cLE4KbCr; spf=pass (imf16.hostedemail.com: domain of 3L-r8YwYKCB4MOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3L-r8YwYKCB4MOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519408; a=rsa-sha256; cv=none; b=kUHTDCx8r0FKgLSdFesqivjBk2vbHgTR3JzNiU0eeY/JCmW5z5CuGx8EGQcOLW5AFgAyGK RYWugLlBXQzSv0IYUC8A3u4fmoyDQeFGaiVNWOj5oUryH/chNcGzmo/ZcDiSIV8bT5Huyo sGhLDnvZZ74PXN7vha88SlEefOlR5YA= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536bbaa701aso152698917b3.3 for ; Mon, 27 Feb 2023 09:36:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fnkDgHrJ0cqyk4yVnPbndf59jVemOaMLlO/iS92oZbE=; b=cLE4KbCrSo3Ipu4g+yzfkRmcncyjiA+PVjyxhSlE0CxpbROsbbX5H0DHpPWYcZcUvL IgBczq5t3IK+5ladr1TbWOx/+SU8LSbf8Yx9jr8PkelvEstvvhZXeX43s6JTSESoyEye 3OJvntBP7bSxCk5pBrTwRFBtDhh/o6VDU9KVAovEJZ9n4EduivQuGfxt0pt6fskiyQUB zHwyjGzqyAJgiciJUq8aC5hSo+rK3+lmuiggplyhk0zrZZEyghcsffT9pFo2TSF/swmL j04i+/EYbcR8DTa/IGgGuaUtWO9uflgS4pwivCZBPdFnqw+L6jKJzSVJqswEg+Tmf1nw 9kkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fnkDgHrJ0cqyk4yVnPbndf59jVemOaMLlO/iS92oZbE=; b=vYVltP9ej/TtyB8M19ijqmK7iG5zCrWhg59tyhkwcfLweVL+j6hz0ZNGKzVz9Gi0NH BJpCjI++gcIXz41/N3m2A5k0xTxXvMTCe+rm6Npn7juL7SVnkz2XxYRRl32Ukz4bW1c+ Eo1IFMS9W9S6wQOk6fy6/AkxBvYx19f089U08NWag0KGsB/FdRBl9ITU+71D4huKN42x /kxOCoYAkyUf1hxVYd5vPfJQmSGvwWDn28bEgflXXDMmFNoERjL+4adUdRxOhJ5TPcsm KhuX9Yr8bo8YoMF2zc4AdTmLNyheip3yfy1NORzlhB6wrV7sJVmv3eIXEQ4rpyzbgR9F k1Pw== X-Gm-Message-State: AO0yUKU7o8xovMeD6RF0o38HhPzDHKjjdeqbFYYK9gK33qmFRLh2f0b4 EJwIVnHd/x8HWaRox4E/dbMxrvCdRK4= X-Google-Smtp-Source: AK7set9OxVnGQxF1LzHrZxP47amBuaMy7ZRonNPJ1uJTtUoNYZIGL83gP4W8p3rUFsPEboBNOudLGUD3TJ0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a05:6902:34f:b0:a35:f3f2:7deb with SMTP id e15-20020a056902034f00b00a35f3f27debmr14636ybs.250.1677519407298; Mon, 27 Feb 2023 09:36:47 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:03 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-5-surenb@google.com> Subject: [PATCH v4 04/33] maple_tree: remove extra smp_wmb() from mas_dead_leaves() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3F08A180020 X-Stat-Signature: 79tec1fj4zxpsordp4hp1n16nog9gnmg X-Rspam-User: X-HE-Tag: 1677519408-539506 X-HE-Meta: U2FsdGVkX1+fdv35zmdf1bQR+nNGTpXFEqfKCYBgrDYCrhRuGpFn8GLIaH/rBdHCCaHV5Bm/EjM22lStJTJDpEIkiEU05XoAuecIdx4HDt31AwlPaWT5OT/U7cPV0LvHlwOFvpHT8B3Z6uRg0b9W9LN8Gcj2DaEZGbX0TkVrBEoxpITuzf1/D/aJTpPvYlZ+9rU2b3gKSm0TFZAMgWwIw39S6Xm9QDfOEbGeNy8+v64f8sraUA/chYJvevd/aJ/nMG69RuK3JyQR3upU2oE85TqWQV0Cs+FXThr6wDKUzvHFY39YTXnjzlQqKRHUneTM7s95drCc/j5SzmeUvN1Zo2BvFdczaWC2tnVchQuGtPowPfmKzSvUOh01Y3oJGTSMm3HOUGBlTlMj1E3IHPtnq7yrnVsMs53cwI6eITPlUEDnKibTSWUDe6kdKpXEbHWPyGI6tKpDa3WzZWvLawALlAFYBH5VKHIpPRSe+XA43YHk9SJiABdWgrfPTVMrQDAoJ57e9sGKpkhY9Wfpr/j76RyUWs2efNOoMVKSetSjTQque/BzBy+2nC5/HDD1F8K0F3JvB4Pr7NBGcgfcbWr01coqXnudCPR53u1FcuXUXKBqMBR+qUcaN5xLlAaYGJutbN5XkKHbFIItBioo6JfCOcsvtWgYHwU8cR36YA6rJO56PiXQ0Hf+3gJNVVnYR14m78SGNYuf+eP9S0FKOrsOjJkjlYcJ/kKJGP8y1whJmaWhz+NQ30TgHqWOAZWtA/2VwGotD5IrLdtpoNoKCR/nzLaMRXKDhfxT4JTRhS2FctvkGvkRy74oXL4qk7wUCoWnXZ8VINwXjxDVeBZhiZOQGJ6mhyiXrU/yg0vGlebE1rJ2WDH4yyfIO1wDX7hvVuf+Twveh0Am5aMd8LPqOz+/5qydCFJ+0jTCpnMC/cg5V5+4aHQIDL1c7OMn7/E2g/d17/z9SCEBUTi6usekqvF LLfx1CUA r7thTYBFv4rBhkU7sYwsYkMLcqZfY5Y4il1vIe44I0CgDT3ZBPeEiBQzNkgRKpAWgXWH5pkmZH/j+6VZWg/QHEdQgBZlxxrg4YzY076oXE2gfStEiSM+Byk7bzCwZlVteWzctNIVUMgDANkheYZA83BKdmFVxo9pL4Fvs1wsqODW22DUZbvtgF4OW78rh5pgmGy8okZRPwVi5aXuhelQaqcJZV6TQC04Os6Z1TEhKJsjgQmVdMC6u5SJjBVxzdtSzWouMIzD8DFv4hLMQ0HmqTroPyyMHuCeZe/SYaF6vdb55Ws6XqCcdiPn8cbRhKXfVfXtIW3tEV1brjSrU9FfscmUMdWiPSZErJlbke803Y9yD3xbSkRAXt8/G0QGGoRVQXUZbojeIbqNwPltSebrVz9X9tOT56pJcfMTVm9k0PxmoVT8K8bf8hvgdm1CDGK3gypIjWIYMkIrEWDT76VcnrYIg38oICU7ItnZsRLde1n3RwXQhZOxyzAgeDBiBWdVhmp7WwDm/hKegV9HqmWmGjcQIXp6BlIOFpVyqG0frZNRMU+Ow3f+LX0rmEHbhgaBIeRbuxgnZFoCIBiucsQCcXsa6rKw4CJwnl8Ev9QMHzJumEPFY4jTrqu5TBFOf1BpdLSWPmGw5lc0zdqxnx6/aBjxT5HA/z2vJB9k8d/TEbtxmieY2KFyIIntjnazUDsGSULoK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett The call to mte_set_dead_node() before the smp_wmb() already calls smp_wmb() so this is not needed. This is an optimization for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 44d6ce30b28e..3d5ab02f981a 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5517,7 +5517,6 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, break; mte_set_node_dead(entry); - smp_wmb(); /* Needed for RCU */ node->type = type; rcu_assign_pointer(slots[offset], node); } From patchwork Mon Feb 27 17:36:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153968 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7C6AC64ED8 for ; Mon, 27 Feb 2023 17:36:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5303B6B0081; Mon, 27 Feb 2023 12:36:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DF8B6B0082; Mon, 27 Feb 2023 12:36:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 359676B0083; Mon, 27 Feb 2023 12:36:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2769E6B0081 for ; Mon, 27 Feb 2023 12:36:54 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D0BD1AAF4F for ; Mon, 27 Feb 2023 17:36:53 +0000 (UTC) X-FDA: 80513777106.12.70FB05E Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf29.hostedemail.com (Postfix) with ESMTP id 052B2120022 for ; Mon, 27 Feb 2023 17:36:50 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mjTDCe3u; spf=pass (imf29.hostedemail.com: domain of 3Mer8YwYKCCAOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3Mer8YwYKCCAOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519411; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9Ei1aTodbgKgkLCHkAcrfXghqfC4aFVkwG1NwHfqe18=; b=PJ3Y3L8cxGecGgkKHufPAleZh0vJRKX8TI92Tmj1wtcByh7wJ7sfkvGRoC41Nww28tWEQk d/2i4FIh/dgKiIfLqbBy2xgV2dhaG8DAUQVr0P3AkdVHhSJ/Lnid+PhtAFo29Q5bRt8dsR /elF7MOsCnYJuQ4dndwg7Aw7xKPBWEU= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mjTDCe3u; spf=pass (imf29.hostedemail.com: domain of 3Mer8YwYKCCAOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3Mer8YwYKCCAOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519411; a=rsa-sha256; cv=none; b=x8jCEULJ2+4514FMCTb+L/D5/g3cQD495KJXNJw//TrCb6336h/UzEdi10eo3KoUnVgNG/ Mp/jp3pOSTCHwBh91g0vdUzivVsYPCaBUK+C/K6CUpcSc/F1oZfGjiCuF8bFolCyJxaz/k Ym5W49Mnge3/mQVpc0OuH3lnf/Rmmlc= Received: by mail-pg1-f201.google.com with SMTP id h19-20020a63f913000000b00502f20ab936so2202836pgi.20 for ; Mon, 27 Feb 2023 09:36:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9Ei1aTodbgKgkLCHkAcrfXghqfC4aFVkwG1NwHfqe18=; b=mjTDCe3um0lc/xjVfVjgd+Npe9HefebnR6MJBHLvf3YF1uIBfRxVZUnFRJLGFeQ5Qn 8RjZv63JPHIbjxEVYDv4CrjZbJnnn4Mt2fDeT1LOlyG88MFvPDh3biXldQxmJDhFfhkL MeulbY1wD03auJFox9o7eR+S24BryOLDDsTTZGQeZ4URN1aDTRabL9SThEbzRf62X937 1Tp2+9t6o/KfoCPMJmoRWi+E8Rd5yccUzXxzvOPdrAcdSgNX0Ahh/fjHi6t+VXE449Oh mngGw09qy20HWkPARPiqIZAx0mZi+xWfyiKlsc96M1s7LVT+aejv+nyRL84gn5iOqmCZ H2mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9Ei1aTodbgKgkLCHkAcrfXghqfC4aFVkwG1NwHfqe18=; b=YN5MklP1jPYv3wK5ekssP8S17/h0MonZQ9cZ0+YQSkp+5J5n9RjtenUbVT90TBheAd 1wVNF3azxzt5vXtIz9mJRm0nTWKP1NpuK369Cg/cPzB1K0aPBAXO5vjAQNM9MXJF/F3F xbiNPPeropUFKtGVM3AX8UJ5jjNy0dfS2FeAFDT6tctom+8cR3SU5BGAcBbSHLWG0Qli AkCBgiL2uBbRcECxWl5L+jc4FpGvG4Ce/PFg50URhSjPzWdKYafwXJbPYAtZite0MN9/ N7m5sr6ecWwn0OuYSNymrcfeG5Iurtv9im8dyFyzV0FFZSSbQkGjFnSzTaLWACv13riN rypg== X-Gm-Message-State: AO0yUKXOd0F3WUAwZ7HshKTG+roKhmMfxjeQ+QQdkgREcTR9odjeyoUi d6LNbipNSrZQRExCDytaZ3m1SIgB/W0= X-Google-Smtp-Source: AK7set8gSrJb7C7al4vp6knawTGaEfFTfMkKKVc0fZqqef6Oc8nGI6YkUfCL0eSaXZ77p/UxVJxZTTA+0vM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a63:714f:0:b0:4fc:2058:fa2b with SMTP id b15-20020a63714f000000b004fc2058fa2bmr6283546pgn.3.1677519409843; Mon, 27 Feb 2023 09:36:49 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:04 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-6-surenb@google.com> Subject: [PATCH v4 05/33] maple_tree: Fix write memory barrier of nodes once dead for RCU mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 052B2120022 X-Rspam-User: X-Stat-Signature: 3teut3cf9i49zn5asw4sxcsdnbdqu9dp X-HE-Tag: 1677519410-133002 X-HE-Meta: U2FsdGVkX18pSJ2Jrls0FjeDuyyYHV7IyFn5IsQeI0nGZOkRZ+6iCCs9so6kMVHOzZKU9Ysae/yh27DQfRdRBe74SldYyRVEHrIqI7+2iDoa2X0jc8/oorMe/Vo+jdeEiB7YousfHU/WSTku1om2Hox6LmM0JjY4INXhMOTK8qZt5DMN6hbAgatGDO9PvIH4Wiylzn0KmvtFDoUyji1I42xmvd8uHzB/W7zu352XcfPD4vjf0QGvWOx2m5fqUjUg4+QBJhgAzXkozkyqSsSH6x6YllLaXC/bw1mTMQmUBcZqirlQMsUzabeDPcjdmWPq/P5CAk7U1XYQvAUb+BazqQ8/qzR5tT+12fGEIfQ++Mn4jQ7PjERSqTdvCTLUakw6py3mz9b+gjYt5w111mB0H+0VRXk6AXFu+2jo1gfIJSv1e0nOdC24mzQQgssgWdy5zO+iJv04bebgeT20JLCDsxDxUZj8Y7DpASeVS42w1rtETWJSNlc9imSFXCkw135B0JGn4+pDjGyV4PCM/fh84aS/VDq29ar22cWBk7XqBgzNpoxVWhFrMqi5S8p6aKh7FYV1a6U1Cv/e09KGSfDSAjtMe6j1NmBQVmefHYZRG524Bro4KbxgpOZ1Fw/KEUHShnTSnHhuiGLMj+Txnh3VA5IprCXiPxCaiOkSASlNxeARFmnS0LB0CGSI0tfump2E5jiRjtvhMroNsA4HvMS/0o+tGDTBfJaICk0R9YMKpGIRDf5LsSsj2i8PNhKR3EFtJJhV8/oq+pAYdMEILOhZB/ylQb9I1RDC09pxRyXZEISIqPlcTAxYV46NpVTcFVRzt4GY0E2RCERz2s6pEekwJtMfcsB+zuy2/tMI+K73OTD4/kJkXzYKCCiJvoYdzY+ngmQbTY4UqxbsdMZqPA3FBgDDAVbTYo934IBgg90V5/YYy1hNLckBu1R1M+5/uFR2j6UCCCJmwG2ecZSsft1 /mBP2HgN MzzAWPn/zUGRPGm5aqLa8adzvVZqFBniuyoXS2znz5yopATjrYf5dC3Zl/7vZhIug9p/Nac1cElyrOdWHQqr2u1LR+Tsm6teJlZmL8MrpnFP7TVXGpVOr7lwO+mpew1ZMBlXQ8tywNRO/htaH7ZoDIc40T35yZFhrAmKT+k6OW4lo49bbH/S33fLgQ4yn2v8fDKVgK/3vFOIJuGyjDgEiuhbL1s2wMm74w2f0MQ3P+EZN3QtRObe1KR8BT6lqdB/a5ty/fGfBpdTDItyVT0Ja0yQgAyDzR1slWcUymKR6pc7ogvlCdKuWylbTJKCQdXMCEpyzq98R2VLh3NOycn3ADyWD6bSaxzPfam7MMMzszi3s6bCVxlOmu4XnhrEK7AvJTV3yz+k6rti0CuZWVQScMqSOF7s1A0HQIYZXOyDpGAPX80jDVGDT7yTtfpvzhATvZQG3L8rIObtY9C/0H9/OxT9VALKrtleWuebCVFxrEcVKEdCh1BEqg6WXWl2kpN+V7WCRCmtbZvXc6BM488/51kqzFvdr9HgSmgerx3hnAzlx20tvSdV67QPC6KAFgUHOY1UJHlOi448fFxD5yd24cJbgBkGE7O+WCYX39l1PeHtv9eh9wcq99pZF0Y84UlpLphiOhzPuNSr5WIRRRVPgll7vJp+gs1A09Iqk+B3JNdDs/ll0Qqzq5dN/ki9iPZeh5ET8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" During the development of the maple tree, the strategy of freeing multiple nodes changed and, in the process, the pivots were reused to store pointers to dead nodes. To ensure the readers see accurate pivots, the writers need to mark the nodes as dead and call smp_wmb() to ensure any readers can identify the node as dead before using the pivot values. There were two places where the old method of marking the node as dead without smp_wmb() were being used, which resulted in RCU readers seeing the wrong pivot value before seeing the node was dead. Fix this race condition by using mte_set_node_dead() which has the smp_wmb() call to ensure the race is closed. Add a WARN_ON() to the ma_free_rcu() call to ensure all nodes being freed are marked as dead to ensure there are no other call paths besides the two updated paths. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 7 +++++-- tools/testing/radix-tree/maple.c | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 3d5ab02f981a..6b6eddadd9d2 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -185,7 +185,7 @@ static void mt_free_rcu(struct rcu_head *head) */ static void ma_free_rcu(struct maple_node *node) { - node->parent = ma_parent_ptr(node); + WARN_ON(node->parent != ma_parent_ptr(node)); call_rcu(&node->rcu, mt_free_rcu); } @@ -1778,8 +1778,10 @@ static inline void mas_replace(struct ma_state *mas, bool advanced) rcu_assign_pointer(slots[offset], mas->node); } - if (!advanced) + if (!advanced) { + mte_set_node_dead(old_enode); mas_free(mas, old_enode); + } } /* @@ -4218,6 +4220,7 @@ static inline bool mas_wr_node_store(struct ma_wr_state *wr_mas) done: mas_leaf_set_meta(mas, newnode, dst_pivots, maple_leaf_64, new_end); if (in_rcu) { + mte_set_node_dead(mas->node); mas->node = mt_mk_node(newnode, wr_mas->type); mas_replace(mas, false); } else { diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c index 958ee9bdb316..4c89ff333f6f 100644 --- a/tools/testing/radix-tree/maple.c +++ b/tools/testing/radix-tree/maple.c @@ -108,6 +108,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mn->slot[1] != NULL); MT_BUG_ON(mt, mas_allocated(&mas) != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas.node = MAS_START; mas_nomem(&mas, GFP_KERNEL); @@ -160,6 +161,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i); MT_BUG_ON(mt, !mn); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -192,6 +194,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != i - 1); MT_BUG_ON(mt, !mn); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -210,6 +213,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != j - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -233,6 +237,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i - j); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1); } @@ -269,6 +274,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -294,6 +300,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas2); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas2) != 0); @@ -334,10 +341,12 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) { mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -375,6 +384,7 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); @@ -382,10 +392,13 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); } @@ -35369,6 +35382,7 @@ static noinline void check_prealloc(struct maple_tree *mt) MT_BUG_ON(mt, allocated != 1 + height * 3); mn = mas_pop_node(&mas); MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); mas_destroy(&mas); @@ -35386,6 +35400,7 @@ static noinline void check_prealloc(struct maple_tree *mt) mas_destroy(&mas); allocated = mas_allocated(&mas); MT_BUG_ON(mt, allocated != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); @@ -35756,6 +35771,7 @@ void farmer_tests(void) tree.ma_root = mt_mk_node(node, maple_leaf_64); mt_dump(&tree); + node->parent = ma_parent_ptr(node); ma_free_rcu(node); /* Check things that will make lockdep angry */ From patchwork Mon Feb 27 17:36:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB550C7EE2E for ; Mon, 27 Feb 2023 17:36:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 077096B0082; Mon, 27 Feb 2023 12:36:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E7DDF6B0083; Mon, 27 Feb 2023 12:36:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA7BB6B0085; Mon, 27 Feb 2023 12:36:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BE2106B0082 for ; Mon, 27 Feb 2023 12:36:54 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 82440A0ABE for ; Mon, 27 Feb 2023 17:36:54 +0000 (UTC) X-FDA: 80513777148.04.C26B60D Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf16.hostedemail.com (Postfix) with ESMTP id 9F72B18000B for ; Mon, 27 Feb 2023 17:36:52 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=J5wmcAzy; spf=pass (imf16.hostedemail.com: domain of 3NOr8YwYKCCMRTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3NOr8YwYKCCMRTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519412; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SQNoSLNi/WgstuG55kXE7+m1dHKl+fKKDPrkCWl/DCo=; b=gusseYEb66HRvhGrNWw/KT5rRKgvter5whv/6zCe3ZnMSsED0UAX6lu73U9hg72eIs290P ceotOeZL3gaJQg7N3rjciBf78H23QdXwwnUkQAP9zGI8pXm64SnvIoB2H71BXCrj8BEuRy rnV26CzqLcUyaD6xzz9mZMEPyR9drHs= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=J5wmcAzy; spf=pass (imf16.hostedemail.com: domain of 3NOr8YwYKCCMRTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3NOr8YwYKCCMRTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519412; a=rsa-sha256; cv=none; b=0Jf4AwW0XzxNcoYzSTUy5WqyWU6B7KoaH9kWYbiZE3T/LeRXgmPp9fFHnL1QtC5ya8bn3V Zbw7Ouoc4kKnKroAyRKDad12JKE8Pb7JV2ioGMWGmrE+cag4QfoSkYqxeq+wNDPaOZ0XrT dCQ8SLrq0B/iGMOitZmwFIP5nNVWyXY= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536bbaa701aso152703017b3.3 for ; Mon, 27 Feb 2023 09:36:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SQNoSLNi/WgstuG55kXE7+m1dHKl+fKKDPrkCWl/DCo=; b=J5wmcAzyDZcatSwztTuxgzaYtke6kriIvYIqPT/BFwSZWL+3VmHsJK6pm6uqTXXs0k hQBQeTTqtZMXoydWH2EraS1BwJQMWApBWO4fGke8/Jq0OvUOu4cfdsRQ8dDM68VxgizQ ztRQr+1fezpVtAc0lp78wCnDR/PHSs/vBVo6WXbM19HIw+eJAf+OKImpXbbC44F1OdAA e9ipoEkHKuhTw98suQdXekAHWIPjjRBuGbOkaPOXnZpP2dJYchLjizYibUTfDcgNqsor BtPgKlPExQUuIZmGBr+b23UCrNVwj01MSEdyN/KRD26rWer10z6tKSB96K/CBsjdr3nr Dy4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SQNoSLNi/WgstuG55kXE7+m1dHKl+fKKDPrkCWl/DCo=; b=eLiK6XXxCFok9zQY/Z0p0ANbfc2NL4+bxk2kOtDTkEwcvIvDt2/Ly3hZ/igZqnYXFR Yoe5Vl7iOZG2Iy2qyNFlcNvC2pOYhqlNKO7qxPt9PyI5WfygfW9yzJIxRoNnso2S7ik2 GNt/fWHRGhp2QGjrbUsxNz2qQ86CDBr2pggHB+Snfn4SNdfTBBgWrbS7gfJJqFnawpeD F9PChhGst2yRECMFtGjO826p5TJnxvw6g2T+YXgbWutUgE/I+DcgaXgMgub5C26Wec3f 6XwJncI7qFiBl02bEAU/w0wtDyt1VW1VXzzJM20EwhHqvXP9MG1IrlNyh+xxqfJgiw00 WhNA== X-Gm-Message-State: AO0yUKU2vIxO0uk8I++LF3iDyp+RyxPBP+8Jp46YoPy4wz2KtYfkr7Qj MJenTgkzB1iBn547hD41YpIrMaI3QyM= X-Google-Smtp-Source: AK7set9MzLoXjLMA6aezNIGpaaorfvcTajbjU3PQe8ye5Rv3C1xsvBgDWbku9hWg2h6JXcwKW/yoJy8kVAo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a25:e805:0:b0:a5f:0:bf12 with SMTP id k5-20020a25e805000000b00a5f0000bf12mr3637071ybd.13.1677519412049; Mon, 27 Feb 2023 09:36:52 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:05 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-7-surenb@google.com> Subject: [PATCH v4 06/33] maple_tree: Add smp_rmb() to dead node detection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9F72B18000B X-Stat-Signature: 9utrq9b1hh95acrybxx74tawumyyn3qs X-Rspam-User: X-HE-Tag: 1677519412-999791 X-HE-Meta: U2FsdGVkX1/IPHMXCLcxJoIKEHIid4YV5RlOu+QprsIVZcUOrBvx3C4cewVxv1hdnkivvWMSUu5YEbO3t8DNSop6GHo+MVVH6u3ybV7wrrht59KZqJzBTzPA8P6nhis1V4svcTBNdDztlyYePOM/J5rdGkAxg04I97BVU01lUtzYwhgyUrG8rYMMmjR35IpVtwDUfEJIw5YvbP+C1d6+dDFUpnx+TT1ux+8ogXnl6T3E95SkmNc/3TWfJsM5mRNm0EY2usigCytsbJH2nbJoXM+vESb6v5pK0VpSBrVS8FRvjOBLNFHy8yQxRiyt6XFelOYbzoeipeuj6wN6oPrBzYaSXAJTmscEXPfr4pkQnhGySq09/rqKYZW5p6Hw8/RILiFGc77bQQOhXGh5ZXA6FhfM5uRZkO+COBroqce3muTSEFMV52p3Z6riqfJmiyfMZoEk49CH3kl4I0+zu2nDFwijpGujk8IvQrlqAbBYn6366C4CKEo18K7xsb5VlYtoEyuFWRcrHLTEUuaIp+1dTYrKyQbG3q9i9k1ZFOkNNBoaWUjXhztu4I48gqZ0EZQjeiS7xd6N97wgea6QQHs6+3Yv4kMclhlKLU/lSkFZs5wsxpEtXMFwzkgN3ck+PB1SFMaFGvuE/FdBsTcPxnnhdNSTIopX3j1UW0IAsbDOE19xjMLQ03nhymr3CsJUKXmvdb34ZHJGYWgEwjtay2IXUOaXXErD+/nYEBpVN0fwsITyre2xHFw80YAjj3r3VTwMLEHsCpWZ4ztvBGyScoFRPq8lvXuQQ16QGiVnQKb92YePxuuL20GToihrCzBpxK1CsnZqYezucConuEYC8yi4r13LEdxG8S4WR1B14YNptsKOwnAaYGv6T/5gmfXkpYPNEix5dZRPDAs5AXTUBS6pzzWX02XZ8fBRuhaB77nMQnDqoIlUL09zhJrwf5fgJxrxM3ZVJ6GRJSnQG4fkvkN 2TtSu31C DhoC5lBAfFsBvceOQuIKVw7rqD4KdPd0vlx+Kj/hcvEje5sMsJJQOhBLjlaXAcwsnSuQUf/hxgeHZmRt1F95l5x5/xzfw+kQDp2gw+Gjfjen9C0fMmqsv3DmnXS4GqWrbbtQPb2TjZFbhKkwHMKAIfPoZ2hU02r+dbS1zmSZHcm1N7VB+cor6pfVH6dI6uwgtzSqqTatnauF2J3lwCG9u+4di2xhCOy5pc1KzJ1VAzyFBjO0Z8J7WMxybta1fee00DisWWFtokfP8pAj9LF3DYpC/TQ5rKtclTo+gd9+YpiAKzO93KWpAcAG1iP10dvL+Qvo2cOudNcZG4Na6YsYijbcAtT+NNpoNnjgUqnyPsEYJHzxslOc+2KTJ9/AKkiV+6gysKqklz9smPrKGqSZKh19QltGb2rd+F2tDw9dwAOR3uwjPob7bfTRs4QLGp3G6mcnw6m1wRSqqq/0/pyxgTMoO8kzHyNw0ufPQXOhGMOjPwZ7GIqAP6XP9mLurLAJ4iyqlEGth9QuE0wEZaGbuVT4xpZ7kxXBjNkB8RKOw1RmY/+Dt9mgtLTYZmljsmCvnmomD/FjNfA6uHkuWlatcYYMH4wHBIwAEltfXT90kEqUCHIZrQ8aflyFNopqbpD/EccUY2Ne5/eR1FDQLc9XT7U7Zj6Y/fPtrqSpbmSfmrr+qY7vPX2cbqpyyfMIbsEiKGIET X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Add an smp_rmb() before reading the parent pointer to ensure that anything read from the node prior to the parent pointer hasn't been reordered ahead of this check. The is necessary for RCU mode. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 6b6eddadd9d2..8ad2d1669fad 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -539,9 +539,11 @@ static inline struct maple_node *mte_parent(const struct maple_enode *enode) */ static inline bool ma_dead_node(const struct maple_node *node) { - struct maple_node *parent = (void *)((unsigned long) - node->parent & ~MAPLE_NODE_MASK); + struct maple_node *parent; + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); + parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK); return (parent == node); } @@ -556,6 +558,8 @@ static inline bool mte_dead_node(const struct maple_enode *enode) struct maple_node *parent, *node; node = mte_to_node(enode); + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); parent = mte_parent(enode); return (parent == node); } From patchwork Mon Feb 27 17:36:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE756C64ED6 for ; Mon, 27 Feb 2023 17:36:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 706786B0083; Mon, 27 Feb 2023 12:36:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 618AE6B0085; Mon, 27 Feb 2023 12:36:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4445C6B0087; Mon, 27 Feb 2023 12:36:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 340A76B0083 for ; Mon, 27 Feb 2023 12:36:57 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0485EAAF96 for ; Mon, 27 Feb 2023 17:36:56 +0000 (UTC) X-FDA: 80513777274.14.CDE9A62 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf15.hostedemail.com (Postfix) with ESMTP id 346E8A0017 for ; Mon, 27 Feb 2023 17:36:54 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aK47QRtP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3Nur8YwYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Nur8YwYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519415; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fUqwZdzGNzQ24dn0iye+/d9qpIg3e2K+gtFu7zl18dg=; b=8Xum4TOW0tWQ2lfVZ4NKqjQ4s7rFRFzb98l56gBm6PZhQr6Ge61yGJbqM86Xn58g58H2G2 izMaAxofpTbgLHyV5uQXAXIvWmaxeRw7BtOtEdoSluV+YuL+dj7PiY6ZR4AM0U9OBPnj6w H+uhLfx4mIWwYpaU90w/BzLjW5bxmSc= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aK47QRtP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3Nur8YwYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Nur8YwYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519415; a=rsa-sha256; cv=none; b=pKEVJEfWeB5tbfe9pOn6RYj3xQuCfBPSPw+eO+a4EwK/9UPdsW+db81MObgQfDEdz9jf06 bXJpe4qXYkgREfdfolcRX38rCRLkjnbtl/WagH2xsabnGqvRk5Ggv7PfI1DEiIskP1TzNT HmpKrXSAKESH0NHoUUmB62Cy33KiXkY= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536be78056eso152373267b3.1 for ; Mon, 27 Feb 2023 09:36:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fUqwZdzGNzQ24dn0iye+/d9qpIg3e2K+gtFu7zl18dg=; b=aK47QRtPPpBCpM3JfuxQQgTdprUxWBxefyFdQCAQov/N683tdXupMjqGdDKaSCwwNE r/feh6muJswYFO3gz8+8hlh8fJ46mk6ytxKiHTyHUDkeqkCmRcZ5ILVSZrf82rniQwJS bwSBoSpbInW4fIYbWSg9wRUr2lr7BPbm97H4qJPgmiiSyOeTA46yxA3PrsE4fyck24P9 RYYIOxHb/Qq+7z+3uM5jAsGNi6XPyJ8SL/Fk6/nH9s0kq3HjTQmlwXVgHF3zawoBuzen oXraX9gQmV6bGYIwSDanLksc10lz5lKlqDY5O8IyYB6Zq/Hucf+Jh7yTBppqP+cHK8p4 s1gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fUqwZdzGNzQ24dn0iye+/d9qpIg3e2K+gtFu7zl18dg=; b=oTvg3OpzmJgQ5jG5sx9jHDOf8WuJmLxQT65RS2uj/9j+5+tq5sdBDRDv7wF1pElehD uELUVRfsWxmscNK2nepDmZ/eXaMnwa2JXjN3rt3FZB3bD8VbDEv4WHEwJy56A0PeIVrQ MrgJyagYiM5qHlw2GusRMVWyhlA5o4wmWTzYPQ9i2DnK1BXuTBPuf0iuDhJmDctVaHfY V3sVtKH8qdzL9aMi2CK56sV119whceyvpRc+PrxneLsxACEJu9OSmdCvj/ZAXIwWcL4F u0oAm6JugI/PsPzU/b45Oao+RFeVMvuJ5IMTcykgRQzaGcOSie5Zt+QOlrEjWWlgjRLS IkEw== X-Gm-Message-State: AO0yUKX+PahLXWgyM5wGp6ZQFBWdGjYwPFEmNMpX+fNOoils4N4M8vFc C7dRw5NkbC/2VbcMvK1l6N77s+WF0qc= X-Google-Smtp-Source: AK7set+xp/kCact/o4a/8QM+9CxJRaFgqFXpY/QxYftINt6FbP584Urjuu8wRQFTT7DzY73bdvjS3xZy3nM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a05:6902:1388:b0:855:fdcb:4467 with SMTP id x8-20020a056902138800b00855fdcb4467mr285964ybu.0.1677519414274; Mon, 27 Feb 2023 09:36:54 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:06 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-8-surenb@google.com> Subject: [PATCH v4 07/33] maple_tree: Add RCU lock checking to rcu callback functions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Rspamd-Queue-Id: 346E8A0017 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: jw7fjyw9379tjrzbfgkr7747aiknzt44 X-HE-Tag: 1677519414-748410 X-HE-Meta: U2FsdGVkX1/OLjeVEPtHe/j58a7FrJ3rxFXS3uMs9dqbLAen8nvoxcUKj7Fgreph2UVW9MH9S48onvsY7dGeFuqTTcHSjbMBXueNikoReZXSlDnE3qXA/Oe6rFRvwtlS5Mi6pfFt7ZURUSWpwWAOBM8Bg1eOWf3j8mUrDKazdWQtjAX9AUHliW+Vyw9kOkJemkeymcdlz3M3sF2sb04gzW0eRQgsFRvzq5miL+zRVmNGf3h6yQdzjggmO6F4zLJLZ4DpZiI94SFG+X4H1n120QXR/ICX0Rk4LrZhCIUSNPbD/7fG0NK+pL1Eh0oLW1aXWuAXU9CxJU29sKM8q/GSjwNAv6S9MrtCQHaCU0rIwySEVqSh5XBPOI5U4NOQ+ZDdEi1/hTrG/iKmSETjhrDvoOhf+nehDiP4n8Cy7sEo6HBT5Z7Mz0fl4o9QaLRo3dGLnaDZT8KhyGRErtTrTHAwriO+J/fOxY7dmnWZcS/+YqaF3QPoD1hswJvxpb4jyCLU4VsNbNvevBCPOb/drwlcZwEqThWyvYZisQEkyI1cgC76OfGnA/rhutLDqUbff/CusIM9f7LgWQdn2yxV1xESnAzOh3QkZG4KhpMZcRcQx+l2HIcxpMCpZ6S3D/QrJSdjUSTs8uj9G3MTI83pyNxTUYXfp785jZhUNDo2SUlL8JClsDft1VEVG74o8LiGtje3uAeDuuvxtY4PSQxwHSmodIxaR1wGsj6WHwTfaur1TudjbmF+Zn38boqk0KN92MicCtSfbs/622KYvN+kDsyKTKjJtz32tYd4oUQkszhPZq7lkYwxymlkhoxkUErHVzWyFkjLNuxb6HejiRw/HS1w/YzpdUZmz1PZ5hdc2UBGO/XlZ74nFMEGTyOOLEg13Qi7Sq4ovPZon5lKfEVSRK8s+9cNE0F+HWpkDJ/Fv8/7XQL0UFJWwCSVizY1wnPHe2J0VWPpJdVrLFhPAy9MZA+ hngVHV93 1e16jMRsd3mWx7d5NZw6jc/zqvvccX1fGi3lskECCqhCtypC8VY8byHut5+GpueGIJV7QDLiYfYsKLEP4MwTTcEKN3Tje2PA1uOo2autxg6tglBakXsDYOy674ZyfJ05w6un/nkHFDGh2TorQIQx0JjuSb+rbHzofyKx1KlFcN7tH3JyIsllYcgkyqFecerjLs2cYYJwScJfrQDWOWseWuliWgOaOU2L9Gx4PUybLF5Y0jQgR7ZzjQSfFymib2Nlplvhdp01B8ZyUnTVkKnwo9b0SMff/ScIBPWtFTZDZ9xClKQv/MP2zA+4M7fGRevAhCTqbDEPSqqCOXoo9ZZeqM/o0z23j+eIQVVENjgKAYzpMz8ZVE9QPUgnS8hKNTzFMKlwIFFIKYHxzznb9pZ6a6V959jkkfuzKXTWgu0QgoxOe4YqJ2dIPYcKDpypVzyOK0IVwt9C5wAlsA/xkpyj5+/gcjPo2v9icEfrcLiD2BuS3JzFggAsjsX06na/B3IgB7nN1RaXJ1iStLr5jV7/ZEOfbG8FaP8vgMWqbXmKwzK6JEatk1HBSUIP8QhQq1p4JkHoM+Xc+TGoUgtS1U+lAXvmJrqrTkNpSXbpe34zVMYAB4Ziwzn/QcpwR1HsX9uGn6yuVTrygRhbDzaC4bfXSTJ2Rk0OmAbbhQQGqEA9Uhx9k7S5adVne4Udaqu97ftuR0+td X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Dereferencing RCU objects within the RCU callback without the RCU check has caused lockdep to complain. Fix the RCU dereferencing by using the RCU callback lock to ensure the operation is safe. Also stop creating a new lock to use for dereferencing during destruction of the tree or subtree. Instead, pass through a pointer to the tree that has the lock that is held for RCU dereferencing checking. It also does not make sense to use the maple state in the freeing scenario as the tree walk is a special case where the tree no longer has the normal encodings and parent pointers. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Reported-by: Suren Baghdasaryan Signed-off-by: Liam R. Howlett --- lib/maple_tree.c | 188 ++++++++++++++++++++++++----------------------- 1 file changed, 96 insertions(+), 92 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 8ad2d1669fad..2be86368237d 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -824,6 +824,11 @@ static inline void *mt_slot(const struct maple_tree *mt, return rcu_dereference_check(slots[offset], mt_locked(mt)); } +static inline void *mt_slot_locked(struct maple_tree *mt, void __rcu **slots, + unsigned char offset) +{ + return rcu_dereference_protected(slots[offset], mt_locked(mt)); +} /* * mas_slot_locked() - Get the slot value when holding the maple tree lock. * @mas: The maple state @@ -835,7 +840,7 @@ static inline void *mt_slot(const struct maple_tree *mt, static inline void *mas_slot_locked(struct ma_state *mas, void __rcu **slots, unsigned char offset) { - return rcu_dereference_protected(slots[offset], mt_locked(mas->tree)); + return mt_slot_locked(mas->tree, slots, offset); } /* @@ -907,34 +912,35 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, } /* - * mas_clear_meta() - clear the metadata information of a node, if it exists - * @mas: The maple state + * mt_clear_meta() - clear the metadata information of a node, if it exists + * @mt: The maple tree * @mn: The maple node - * @mt: The maple node type + * @type: The maple node type * @offset: The offset of the highest sub-gap in this node. * @end: The end of the data in this node. */ -static inline void mas_clear_meta(struct ma_state *mas, struct maple_node *mn, - enum maple_type mt) +static inline void mt_clear_meta(struct maple_tree *mt, struct maple_node *mn, + enum maple_type type) { struct maple_metadata *meta; unsigned long *pivots; void __rcu **slots; void *next; - switch (mt) { + switch (type) { case maple_range_64: pivots = mn->mr64.pivot; if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { slots = mn->mr64.slot; - next = mas_slot_locked(mas, slots, - MAPLE_RANGE64_SLOTS - 1); - if (unlikely((mte_to_node(next) && mte_node_type(next)))) - return; /* The last slot is a node, no metadata */ + next = mt_slot_locked(mt, slots, + MAPLE_RANGE64_SLOTS - 1); + if (unlikely((mte_to_node(next) && + mte_node_type(next)))) + return; /* no metadata, could be node */ } fallthrough; case maple_arange_64: - meta = ma_meta(mn, mt); + meta = ma_meta(mn, type); break; default: return; @@ -5497,7 +5503,7 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, } /* - * mas_dead_leaves() - Mark all leaves of a node as dead. + * mte_dead_leaves() - Mark all leaves of a node as dead. * @mas: The maple state * @slots: Pointer to the slot array * @type: The maple node type @@ -5507,16 +5513,16 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, * Return: The number of leaves marked as dead. */ static inline -unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, - enum maple_type mt) +unsigned char mte_dead_leaves(struct maple_enode *enode, struct maple_tree *mt, + void __rcu **slots) { struct maple_node *node; enum maple_type type; void *entry; int offset; - for (offset = 0; offset < mt_slots[mt]; offset++) { - entry = mas_slot_locked(mas, slots, offset); + for (offset = 0; offset < mt_slot_count(enode); offset++) { + entry = mt_slot(mt, slots, offset); type = mte_node_type(entry); node = mte_to_node(entry); /* Use both node and type to catch LE & BE metadata */ @@ -5531,162 +5537,160 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, return offset; } -static void __rcu **mas_dead_walk(struct ma_state *mas, unsigned char offset) +/** + * mte_dead_walk() - Walk down a dead tree to just before the leaves + * @enode: The maple encoded node + * @offset: The starting offset + * + * Note: This can only be used from the RCU callback context. + */ +static void __rcu **mte_dead_walk(struct maple_enode **enode, unsigned char offset) { - struct maple_node *next; + struct maple_node *node, *next; void __rcu **slots = NULL; - next = mas_mn(mas); + next = mte_to_node(*enode); do { - mas->node = mt_mk_node(next, next->type); - slots = ma_slots(next, next->type); - next = mas_slot_locked(mas, slots, offset); + *enode = ma_enode_ptr(next); + node = mte_to_node(*enode); + slots = ma_slots(node, node->type); + next = rcu_dereference_protected(slots[offset], + lock_is_held(&rcu_callback_map)); offset = 0; } while (!ma_is_leaf(next->type)); return slots; } +/** + * mt_free_walk() - Walk & free a tree in the RCU callback context + * @head: The RCU head that's within the node. + * + * Note: This can only be used from the RCU callback context. + */ static void mt_free_walk(struct rcu_head *head) { void __rcu **slots; struct maple_node *node, *start; - struct maple_tree mt; + struct maple_enode *enode; unsigned char offset; enum maple_type type; - MA_STATE(mas, &mt, 0, 0); node = container_of(head, struct maple_node, rcu); if (ma_is_leaf(node->type)) goto free_leaf; - mt_init_flags(&mt, node->ma_flags); - mas_lock(&mas); start = node; - mas.node = mt_mk_node(node, node->type); - slots = mas_dead_walk(&mas, 0); - node = mas_mn(&mas); + enode = mt_mk_node(node, node->type); + slots = mte_dead_walk(&enode, 0); + node = mte_to_node(enode); do { mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; - mas.node = node->piv_parent; - if (mas_mn(&mas) == node) - goto start_slots_free; - - type = mte_node_type(mas.node); - slots = ma_slots(mte_to_node(mas.node), type); - if ((offset < mt_slots[type]) && (slots[offset])) - slots = mas_dead_walk(&mas, offset); - - node = mas_mn(&mas); + enode = node->piv_parent; + if (mte_to_node(enode) == node) + goto free_leaf; + + type = mte_node_type(enode); + slots = ma_slots(mte_to_node(enode), type); + if ((offset < mt_slots[type]) && + rcu_dereference_protected(slots[offset], + lock_is_held(&rcu_callback_map))) + slots = mte_dead_walk(&enode, offset); + node = mte_to_node(enode); } while ((node != start) || (node->slot_len < offset)); slots = ma_slots(node, node->type); mt_free_bulk(node->slot_len, slots); -start_slots_free: - mas_unlock(&mas); free_leaf: mt_free_rcu(&node->rcu); } -static inline void __rcu **mas_destroy_descend(struct ma_state *mas, - struct maple_enode *prev, unsigned char offset) +static inline void __rcu **mte_destroy_descend(struct maple_enode **enode, + struct maple_tree *mt, struct maple_enode *prev, unsigned char offset) { struct maple_node *node; - struct maple_enode *next = mas->node; + struct maple_enode *next = *enode; void __rcu **slots = NULL; + enum maple_type type; + unsigned char next_offset = 0; do { - mas->node = next; - node = mas_mn(mas); - slots = ma_slots(node, mte_node_type(mas->node)); - next = mas_slot_locked(mas, slots, 0); - if ((mte_dead_node(next))) { - mte_to_node(next)->type = mte_node_type(next); - next = mas_slot_locked(mas, slots, 1); - } + *enode = next; + node = mte_to_node(*enode); + type = mte_node_type(*enode); + slots = ma_slots(node, type); + next = mt_slot_locked(mt, slots, next_offset); + if ((mte_dead_node(next))) + next = mt_slot_locked(mt, slots, ++next_offset); - mte_set_node_dead(mas->node); - node->type = mte_node_type(mas->node); - mas_clear_meta(mas, node, node->type); + mte_set_node_dead(*enode); + node->type = type; node->piv_parent = prev; node->parent_slot = offset; - offset = 0; - prev = mas->node; + offset = next_offset; + next_offset = 0; + prev = *enode; } while (!mte_is_leaf(next)); return slots; } -static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, +static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt, bool free) { void __rcu **slots; struct maple_node *node = mte_to_node(enode); struct maple_enode *start; - struct maple_tree mt; - - MA_STATE(mas, &mt, 0, 0); - mas.node = enode; if (mte_is_leaf(enode)) { node->type = mte_node_type(enode); goto free_leaf; } - ma_flags &= ~MT_FLAGS_LOCK_MASK; - mt_init_flags(&mt, ma_flags); - mas_lock(&mas); - - mte_to_node(enode)->ma_flags = ma_flags; start = enode; - slots = mas_destroy_descend(&mas, start, 0); - node = mas_mn(&mas); + slots = mte_destroy_descend(&enode, mt, start, 0); + node = mte_to_node(enode); // Updated in the above call. do { enum maple_type type; unsigned char offset; struct maple_enode *parent, *tmp; - node->type = mte_node_type(mas.node); - node->slot_len = mas_dead_leaves(&mas, slots, node->type); + node->slot_len = mte_dead_leaves(enode, mt, slots); if (free) mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; - mas.node = node->piv_parent; - if (mas_mn(&mas) == node) - goto start_slots_free; + enode = node->piv_parent; + if (mte_to_node(enode) == node) + goto free_leaf; - type = mte_node_type(mas.node); - slots = ma_slots(mte_to_node(mas.node), type); + type = mte_node_type(enode); + slots = ma_slots(mte_to_node(enode), type); if (offset >= mt_slots[type]) goto next; - tmp = mas_slot_locked(&mas, slots, offset); + tmp = mt_slot_locked(mt, slots, offset); if (mte_node_type(tmp) && mte_to_node(tmp)) { - parent = mas.node; - mas.node = tmp; - slots = mas_destroy_descend(&mas, parent, offset); + parent = enode; + enode = tmp; + slots = mte_destroy_descend(&enode, mt, parent, offset); } next: - node = mas_mn(&mas); - } while (start != mas.node); + node = mte_to_node(enode); + } while (start != enode); - node = mas_mn(&mas); - node->type = mte_node_type(mas.node); - node->slot_len = mas_dead_leaves(&mas, slots, node->type); + node = mte_to_node(enode); + node->slot_len = mte_dead_leaves(enode, mt, slots); if (free) mt_free_bulk(node->slot_len, slots); -start_slots_free: - mas_unlock(&mas); - free_leaf: if (free) mt_free_rcu(&node->rcu); else - mas_clear_meta(&mas, node, node->type); + mt_clear_meta(mt, node, node->type); } /* @@ -5702,10 +5706,10 @@ static inline void mte_destroy_walk(struct maple_enode *enode, struct maple_node *node = mte_to_node(enode); if (mt_in_rcu(mt)) { - mt_destroy_walk(enode, mt->ma_flags, false); + mt_destroy_walk(enode, mt, false); call_rcu(&node->rcu, mt_free_walk); } else { - mt_destroy_walk(enode, mt->ma_flags, true); + mt_destroy_walk(enode, mt, true); } } From patchwork Mon Feb 27 17:36:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA749C7EE23 for ; Mon, 27 Feb 2023 17:36:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 863E26B0085; Mon, 27 Feb 2023 12:36:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 814566B0087; Mon, 27 Feb 2023 12:36:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B40B6B0088; Mon, 27 Feb 2023 12:36:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5D0F06B0085 for ; Mon, 27 Feb 2023 12:36:59 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1FC711209C0 for ; Mon, 27 Feb 2023 17:36:59 +0000 (UTC) X-FDA: 80513777358.28.B7E2D6D Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf25.hostedemail.com (Postfix) with ESMTP id 3EB0BA000D for ; Mon, 27 Feb 2023 17:36:57 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="D/TbQt2c"; spf=pass (imf25.hostedemail.com: domain of 3OOr8YwYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3OOr8YwYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519417; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9aVmQ+gMlQiQzO0BzppdTJNpvPFfXe9SF4GFtX+8IdI=; b=mdKqik5ZEgYW1tSqFMeWzb0SolM/XuFY3hTZ5HRRN1PR617tM5otambJ+2SMv4fzf3sFg7 7ryz/j5ZJNHRpc+Eh5CpyCMeSttFTYhXawejAUMTg7gmT3SNGUa1n8Mr+ihOzcbrSTpJqh 2MUctLExCivjcho6sWXYw6jMMuM9gfE= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="D/TbQt2c"; spf=pass (imf25.hostedemail.com: domain of 3OOr8YwYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3OOr8YwYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519417; a=rsa-sha256; cv=none; b=VJdEiEEl6h31pdajF9wPxMvd0JQA9TvKNYnHfM6DPciSYVxsGMfyInKecFRXgO+7Ml2W9y UalfSqVcbJocw4Zs7n6BAEHVvC1mukxI6wFC/BKgZ0XhzDIp3sjzyk09d0w/qDn1qdlX4I 7OUZ/VD/MJ9zGOp5JDEY/aNL+ruZup4= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-536e8d6d9ceso153557647b3.12 for ; Mon, 27 Feb 2023 09:36:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9aVmQ+gMlQiQzO0BzppdTJNpvPFfXe9SF4GFtX+8IdI=; b=D/TbQt2c2awJMi6HOd9NzTPN8Wh6HIDVRf22VcVpKnoo2iG9BW3cveeUmkm8PKhRJw e8yR0ZbMCGkBSW5fis3LsBzhCw5i+IpUgPXTff0+yi++DCnoiZhErYzM/F+7mHgcikXH QOit+1nZssfs8RFlG5fSiSPYICEv1yz50+pHU8s0ps/vZirUhJuz0Ulrl1hrsOJOpcE3 znNNgdhtvb5jKMhdVyGD/B9paUJjeax2LqQ02QvRkavKjrbvwoC4jih7/m+HHIs6ILLw YfzP5gfYx0HByoaWOc2LKVichxZvwcYhuSo6etH4BNZtHefRK1x1pNfZ+dtw2h02k/FS yoOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9aVmQ+gMlQiQzO0BzppdTJNpvPFfXe9SF4GFtX+8IdI=; b=hksm+TKOXQEuoEkOpiaFZYbdksZy0IDU5q+VmBAiWlZ48MhrqIDm1ID0bPAuSZazkm IZznVwjVyYXfdGiTeFuR2p+71S1VCR5/Gh7pGeldCgxWxy/7gHKTit40b/KRQ5lQeWBo FvneRS+D2PJrlWuIe6o+4Wz42fuVhVugiPW0kYV0Qb1hMUvYRT+RzasJu9oEMCrg8TBl 2qAFCT/dOFyRlXKJv3ufmTP71qcpTOaVtt0+v1xLv4xmYbPIOlqd7qwCUJCWK1G3QDfR hWzm8joHjjWoCFrW4f7i3ullENedLB7E7GT9TFgn5jxPFduqjpINhx7RAJ0ZTwcZefy3 vZew== X-Gm-Message-State: AO0yUKUJuYAbKWe4Mw/gtf/GCHGC8dYU8k+EODOZsirKh6aPLEkvYI8h hcREFPolQ2hAjMNdbODAQ6PdKxg06BU= X-Google-Smtp-Source: AK7set8Y6D5rMaDNffZrErpNBudkJshlslCM/7lUMkyK7fj0ZkdAWXrQW4GsW7RGl7OL0nGeq+RMkIj/EqM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a05:6902:1205:b0:a09:314f:9f09 with SMTP id s5-20020a056902120500b00a09314f9f09mr214802ybu.6.1677519416349; Mon, 27 Feb 2023 09:36:56 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:07 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-9-surenb@google.com> Subject: [PATCH v4 08/33] mm: Enable maple tree RCU mode by default. From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Rspamd-Queue-Id: 3EB0BA000D X-Stat-Signature: bzz56gxxk6e1x7itibpg1y8fy66sgkhm X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1677519417-967362 X-HE-Meta: U2FsdGVkX1+4/C7OwEft3fwf8KnXhhzZIcJyWMH4i4lQ4SXyq9ybr3axU2jC6R21uZeLwxQ0Wutm7SJA3lel4hM2EH/eL5ntA0H/A/dJsQkKwovTy5zaQnJUhkCly8/0mjcb/Irc5Mu8GGY1gyHFz0mcagym8b48V+dEOaPBaDZZPALcCDkj88gRMWU3x1cObBEZyY7GFQexhqVS+sbzeJozpBFWbbDI4wCvzZrtAU8K3npN9ecno+wQ4J+ywE2JoLHkkMAadq6CaUY+HxuagsmytYacsRQCvwcMZHP07l4uebPEcyrR4w2rQ3yrr0Ebi/z2F6mnACbQrGbZTLKu7XPMG6wLl9JrK7wbujF1vLcC5ets7s75AqW/4OCathXhbsvhruJVqquDW0FFMwMB2XyVT8qIr00VBG4HibMUYpqmlczYGRBUC7fSogpgcNxDmIKvWSl6o1Hqfwmf2OQLYRnyxc3TcruXdQxU/Ltqfs51tsp7avsKSHMBPQLpER1VyuEyaaZjWZ+m4QQHUJtEYkRIL2RP4YY+TIrFfvMr6Y9VlWQJ3kSMrExQJhsbIE8dN0GNq4NCER3cnwicbdqk4092ENc+Tzs5Ecy+gHjhV2fm6c+MdVFOnrZkn3Lx/7f4xfwDepmaJcvtIQbr1zMoy+cniwLmDZojneWZO031l8RdyEDrcmaQP/4rX2SQeNJEg6PyIszvOwC8BCe2pijSw3Z0si0Bs8J6Z0bhunFGUnmzPkKqBthDm2e7/OzSs/lhLd5toHqgU7y6t/irLOMAZ39EZQqfAO0SwLSKmPGnjrydX97V/fjJc+PcnmnJ2NixI8+f03+PJ0c7wfPDYuzDZ/WeTsN6nNZ4IZtK1yogcEFPcS9meoH5uSpvDWVLke5C6hezQDAj8nGqPAZjAJsaB+xQAFAVwNeOD8vb1Sp/g04P9X9CyyzdoETkqf16P0Wxiu1JI74ztmVdnkH4We8 wqomBlFC Lt+HLuTBtGpl0eLKSsXWn2SGRBnnGO2XBJHTkEX8aFYVP+m/HDKqusQl2SwLBd70Hdz2lGIiiojHcEIVCnlA3YneTwuPb4QlhWXka0HC572pdGaVuMF957+bZpFMGqBScJ2uGb7unv6fMgn+uIexs8ZTsBEClbLIzkJ4ziZA/wwQxOb3DfgIy0d8XAfIrtqqBYAu0Dhivf+ZpOvrQNhblpJTUY7NWhJHA4CJ5c+jhgpmt5f7+Hc0h2q2yokM774UyuZpwPhKswDLkyjo8W8TWpzX1cO+oEftNv6tURSSLahDypfFbrt+g96eSUd72GPLF2wcePAK2jZsCc6btxhRxo9B2V/RQSx9DNAbWzMuWuaSpSlu4sk88XYzns3jD/EbiEnJ7H20Ni7/4GU34UpQcqrzZgvwXLx45/wBYtrt0vlY8jeQGyJQ658FoQXAog5YIyOkgPd32/sdKYWmvLL3QdeoP+9eWJD0BXtXZL9uq2gnI3aVxYZRHzp+RPNz0tMKFlstDW2Z13F8HnX57wNHfMKlGDGV+YJjsfsMBeO12yvVBUwZtP8EY/P4ZKQGol+RgWfXHOqoVp1F7jSWKFq95A/N8Kl4h1YVKmAW9S5yfx5uWuE6PIyOv28QzK/ppm0RuTufNABKhBC9uzy/lvQJbYF4Xtw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Use the maple tree in RCU mode for VMA tracking. This is necessary for the use of per-VMA locking. RCU mode is enabled by default but disabled when exiting an mm and for the new tree during a fork. Also enable RCU for the tree used in munmap operations to ensure the nodes remain valid for readers. Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 3 ++- kernel/fork.c | 3 +++ mm/mmap.c | 4 +++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 417d25c6a262..22b2ac82bffd 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -779,7 +779,8 @@ struct mm_struct { unsigned long cpu_bitmap[]; }; -#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN) +#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | \ + MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; /* Pointer magic because the dynamic array size confuses some compilers. */ diff --git a/kernel/fork.c b/kernel/fork.c index 0cbfdc4b509e..abfcf95734c7 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -617,6 +617,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, if (retval) goto out; + mt_clear_in_rcu(vmi.mas.tree); for_each_vma(old_vmi, mpnt) { struct file *file; @@ -700,6 +701,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, retval = arch_dup_mmap(oldmm, mm); loop_out: vma_iter_free(&vmi); + if (!retval) + mt_set_in_rcu(vmi.mas.tree); out: mmap_write_unlock(mm); flush_tlb_mm(oldmm); diff --git a/mm/mmap.c b/mm/mmap.c index 740b54be3ed4..c234443ee24c 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2277,7 +2277,8 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, int count = 0; int error = -ENOMEM; MA_STATE(mas_detach, &mt_detach, 0, 0); - mt_init_flags(&mt_detach, MT_FLAGS_LOCK_EXTERN); + mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & + (MT_FLAGS_LOCK_MASK | MT_FLAGS_USE_RCU)); mt_set_external_lock(&mt_detach, &mm->mmap_lock); /* @@ -3042,6 +3043,7 @@ void exit_mmap(struct mm_struct *mm) */ set_bit(MMF_OOM_SKIP, &mm->flags); mmap_write_lock(mm); + mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); tlb_finish_mmu(&tlb); From patchwork Mon Feb 27 17:36:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153972 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D400BC64ED8 for ; Mon, 27 Feb 2023 17:37:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 72B816B0089; Mon, 27 Feb 2023 12:37:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 702F36B0088; Mon, 27 Feb 2023 12:37:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C9756B0089; Mon, 27 Feb 2023 12:37:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 509476B0087 for ; Mon, 27 Feb 2023 12:37:01 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 347C0AAF8B for ; Mon, 27 Feb 2023 17:37:01 +0000 (UTC) X-FDA: 80513777442.11.4DA5EE7 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf09.hostedemail.com (Postfix) with ESMTP id 5FA5814000B for ; Mon, 27 Feb 2023 17:36:59 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jatZw+va; spf=pass (imf09.hostedemail.com: domain of 3Our8YwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Our8YwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519419; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ycuncMhySgAqKB0JKjWQ87T3vvxMS9R2K+oLMhIEJ8Y=; b=EwDLLGHnCneHANfZNdAV0RrenU/6ilK2I5eC11ESPVDBWo03GC749XGM5awgz+gDGzvIzg XFreSfdIDITGeq6jSqopeWPDiaLJG16WsGdNz9oJYCn0oGI+PhvlSzITnCsT7etIQ5LwDd 9ixXLcDh9T3pGBH6IMomlZM8DYoZiOA= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jatZw+va; spf=pass (imf09.hostedemail.com: domain of 3Our8YwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Our8YwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519419; a=rsa-sha256; cv=none; b=Xw6Vw1aqvnETQv2FILGiLl1FS90GXalIXpf1d43AkdC1p1n1AYbJcHWuX3hYiYygEnzaWO G4eje7DwVTkSFU+o6YV3qzx58DLwO6ZVS542xh1jRd4OAqm9bAcsWzaHmPhnPz5iIqTbuO NKX/S2dFiSx69erGyZIauyRQ2m5Ytr0= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-536cb268ab8so154420367b3.17 for ; Mon, 27 Feb 2023 09:36:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ycuncMhySgAqKB0JKjWQ87T3vvxMS9R2K+oLMhIEJ8Y=; b=jatZw+vavHEldR/zWx4fIqW8wAPU7kuskm2fNsdTmUbEoYYRUK1QgmfSQwty0F7vyh In/5k0Ybu4OL/7+U8XoT4kd5dwj6nkMsSYSfl49SjuU4fWHkZmC7EvE/NA5jUKV5mijt tEPaHMFzYqt7Ecgvi/GqjzXDl7rzKrsdjxUTz3IPQXQFpGFBi+tum9hpbaiwuAQcs7YP AcTg6xqgjkoCojGsNiAv0DLTEBbzq13kS5i64Oy3wTmze25CUcKHKdl0RIiSjO/ZdJhS rd+Ro59v2yktpIygy3VtQ7oiDpWrjQ+HLII3XoEVuOsktTglELO3PMbQ5C9t6zyu7z8h 3t1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ycuncMhySgAqKB0JKjWQ87T3vvxMS9R2K+oLMhIEJ8Y=; b=bWHtoz26BJGuNwkNbFtr3tY/D4Uf3hVrhShBVspY0yfb3lDLIRAahqelWrk3TVsj7j s/30LPKvSFbwO5ZkwPRx3yE7+VhS8NB0tXC7hDyTpRzKNVYTRDQNz188krSi/iq86qzc Oor052vbU9x0MCf+T/1hvxEypkC3T78gyTFFAjFbofW81w7FzVJ01sSzsuquCArYXk4+ 9hVsMXUIpBLfIu2mkKC/qL/OOW8eW/U2Km3LYJz2RdeNo8q4v2J6AqyNO8coUSw2img5 ikRbm8ZxrCtGvE+mVRLsRc1amkZTMB+00I6MVvF30i4w5Fm+TnJcuu48RpHzGSwZtfo/ k5lw== X-Gm-Message-State: AO0yUKUnM1+haV7e4tBlVC7f5653nSUyU01pZHa49bgCYUVD1xWZ7MFr fl0ppfPVokHbpyfgwgSfzW4DKcjCwLk= X-Google-Smtp-Source: AK7set+sQnwo5grHRclokRwp0C+SftMLAAuN7gXfh+MOPKXwAKaiWwA4Ongifn/fr7bxEOF3Yk7Tuxt7Hwo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a81:af1f:0:b0:52f:1c40:b1f9 with SMTP id n31-20020a81af1f000000b0052f1c40b1f9mr10776885ywh.7.1677519418452; Mon, 27 Feb 2023 09:36:58 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:08 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-10-surenb@google.com> Subject: [PATCH v4 09/33] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5FA5814000B X-Stat-Signature: ese91ow9dc3wtk8eijkoqc5ysakdob1x X-HE-Tag: 1677519419-846243 X-HE-Meta: U2FsdGVkX18qzcITy5+fyKWC7qv2yo/TrjH4JMCW+jFyvIPTVqcoHaG4KPEY2puhgF2ydYk9+0NtHK3qCXLpVJZSZ5B+Rqg6VpGUUjBWjWA9DtfwPauGkhOIx8F44i4KtkbYGojNQDoGrYYHYa33kNp8DDEewMTLSQ5aHcLbBJusYHBVZJ74GQVE4DWEYVkkjP0k7cOtl6GNvlmUasiz4yOvvn5yisStaNO7C42PfaYsmhNAmTTLXKC0Qbn2uaL8GwlhBa8F92FlvmHyUNGQ+wIwKX7rf5GIkybF4gpvRqfrcC8NKKOyy1yt5Pckv2gj6PUnwbG1ZCGClr1A6SQ4M4HbSRYVV9pLh2mp3eg9QPdGjv1Hf6aTymUJAE7Bwoay+2oVKwOwudK1AMJD4X2ZCFVuwvmF2porxd8bPfAe4skZieGseLAg/GltrubMWi9pukYU4KHnI7D1tttFvYLYlcaKcCSck3Hiv8mUPeFZTMrf0g2uSuik1fPjxLfRZoJ++v+TpizjQ945kFKTv0aY4Rwyslg0C/dDDnYaPaP3jW/1v8pNxNAZW862Dsn2lb+RbrRYwfKt64hf+4c25hYCo1csl5NrjRW9AGUbEJIUy2hxVExYzRPA4Ta9Aq2J3i7zcVaQZUc8zgd/rk/VLwTyw6nCa7dewCLUPmThqXelqTvLt0HtlQ2vMHMpnT5dFj21bqqyG9ujDsnjMOHp7Ra0JHkc4Nxjy4LT1opRO19joNOOrt77Uc3X3EOJiquMepghN5uO30hxODU0WdfIOqxhuOeoqBx1hJrD/e4VMdOP97rW45Rdw8lODdMLUDCZynJh2eCmENQDpXRH9oTD1nQdQLsPAnXspLzSVcrTn0rRFd67KG4WhKAJylh2xfolS11MAGeH3NPloKqi0RkTE7MrpqKabTOt+MiQP1PIiTnT+51ei9kBvS3XIhguVArpnPA2+Shc9U0BRq5BhxcL6/6 +0zafySP o/J/ItNlRRdsYV2BEOcZr5fNkqzPqQhEy7Chunc0zIVwbWggzQKKVjDvziHZxUEbmmx+OGsHp/OgqnncXQ0CxGfjfOjv0QHE2zhoUHNsuglzk8mLc9wRpdC6TXP6kKOTnlEy2MzTbwb009etNOA7tOrh6MN39hJZR/OyQ6zD2P1SGJrcEZ/oDA5g6Soc3svrC1XpWSj/IA4oVs8i0pNifOD2n+19rcXKeoIrS0GVhbcO3y30rKuW6uTP3JCp3Bx3iS+QAJz2hFO+1P16YA9op3ODFOwbkfU9sh21XPo0dsOKdjOprHCY3mgxjVrmA8/mRYSQJAng/QNT72N5wOQi+kZOnIlFKYM9Wl+Qj2PgSR4bYk6rFalI6NDc/ykdPJQHM7lIYvYxZxQtZ4HCwDHt9fdqfVRa3s9OAOJNZ2cfJ0WUiJXx3jYvt+szuu0II6mLEGpchXf8rZfBX/RKQqn5iENBY7X/D7n/GX9VU9D/qbjAEgp+drIwWcDVV7krVNIw65EFTRkgIywAzx4bAatjUGDOl4s4xg12yAEb4fmokGSSWJPTuJxREmmhzHxH82nA0v4gI9YJ60BPKvMPO6qI68i/TvBxrWlwe98KubCAJlvhE6eTDnxwmJcMq+vAPpZgSPk4gCyVUlhJsTi7cZMhPOmLe7gIvwI5HzUdCtLXx1oZtR/c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index ca98b2072df5..2e4a7e61768a 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1211,6 +1211,18 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + def_bool y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Mon Feb 27 17:36:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153973 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B757AC64ED8 for ; Mon, 27 Feb 2023 17:37:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52D626B0087; Mon, 27 Feb 2023 12:37:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DDBF6B0088; Mon, 27 Feb 2023 12:37:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A6B16B008A; Mon, 27 Feb 2023 12:37:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2A6A26B0087 for ; Mon, 27 Feb 2023 12:37:04 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B29451A0B02 for ; Mon, 27 Feb 2023 17:37:03 +0000 (UTC) X-FDA: 80513777526.20.F4A44FD Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf23.hostedemail.com (Postfix) with ESMTP id E6D77140004 for ; Mon, 27 Feb 2023 17:37:01 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="jX0E/sUV"; spf=pass (imf23.hostedemail.com: domain of 3Per8YwYKCCwacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Per8YwYKCCwacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519421; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=29YuznebDbhH7LJ7uqA8Selv98qAa02JmRVPnHaqVO4=; b=dHsuHQYPoPWhhke5nkxdq/i6ANxO6bpzi+k7ql1Fd8+fjPUiSmZFSmiH4Peh0cMlCPNmo9 lbo/bLU9rCDdevD0fxl2RWAscDWSfql8bsnUbgRMVG5b8snQLpOBQ6ma9IW/1ZWlVXC+Eq sEDA9t75O1xbHMknqks0i04Jav3bQdE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="jX0E/sUV"; spf=pass (imf23.hostedemail.com: domain of 3Per8YwYKCCwacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Per8YwYKCCwacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519421; a=rsa-sha256; cv=none; b=VuELfeVvFmEz8cFbyWYHQUqnP4xG78QLj8R5dvC8J8rgAP5mv9ZoP7MLCiFqr0kX9vWZLX QjsQ358JYwmxSssIrwzu897RMyng+tNajECYHSMpyWXeJkanUXBKcajwwGZ2jXUdAhJqgS P8eZ9Cn+4PgE1fdJnpZyAE+xoO3901Q= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-536bf649e70so153078417b3.0 for ; Mon, 27 Feb 2023 09:37:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=29YuznebDbhH7LJ7uqA8Selv98qAa02JmRVPnHaqVO4=; b=jX0E/sUVT2RKYWg5Tqa21wWzSsitbAZllQiaBV4eWJXbUNmmhHROoBx1qNz1dBdUuE WfwseHSvUNs7qQdK+oiHVNrKDTVUoZgUvG4zzPJtFXGpgjghqMQHZP4+pvFGIZY9K8t8 yTb/HpAgn0jfVzcS4IzWrxVVhGHhJpK4EoMdSwUP/MFG5D0Q4nJDsYGVVLA1YMlIDxWr /VFBixWLHZCmRtwCrTiJJEDxVSpInKYA+7RDdnLaY6ZEJ31POIP0qLghcCdmVf5ptWUG sRbVaoL3AhiMndAmHz8lepZWlnznP3Urg+vUr3DV+pv2Gj3JIWJnnqJV/02ALHT1NaZc S6wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=29YuznebDbhH7LJ7uqA8Selv98qAa02JmRVPnHaqVO4=; b=7Habno1kLU4pMALaTvq4/Fg45L4qTnEhBZ/Y9UPhkjqrl48I9O/EHwhIP2FCzZHF+A wkbOH5zu9+Yy9wV5Wo1B5mo3ZtyeuNp6cB9FwyihJW4EQlmDox2Qevslk3e/KBe7SCW9 XBtlbEVbgFdzkvdgUz1nsXgVsCO4+oQzL5eVoMErkEo7Dgn2q5nRqquC9Sd6W5Vn5hYN exFVMRoTkJl82Haontx3gwOXsAt0r9rVEdISF+FJR+fMn2wYwVrYukODxexJtqVQZJng q1ylbDis5nzDi07bh4Y0klIbUn/IkkgvjSlJr2gcOYWp1Ugp8lsYjfvoSk8wjhoLgT8H FU7A== X-Gm-Message-State: AO0yUKUPVqK4caL44pn9PSk4l9wLmpqr4VBeEoDM9tZqp6wOJ+Ik4Yc4 UPPVfg1C/4OXrpurGyUPKY5nR8UdEYs= X-Google-Smtp-Source: AK7set+1Pq3oRMVl5cZH8khR0Bvwd8LGMD5WFRsqtwPODZNGlHBw37TZClpeLubRs3LjGXy9DC4x47Xc4Uo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a25:9390:0:b0:a53:5825:f25b with SMTP id a16-20020a259390000000b00a535825f25bmr6018468ybm.0.1677519421007; Mon, 27 Feb 2023 09:37:01 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:09 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-11-surenb@google.com> Subject: [PATCH v4 10/33] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E6D77140004 X-Stat-Signature: n4x1n17jpywqyth7gx5p9ji8ogdd3r43 X-Rspam-User: X-HE-Tag: 1677519421-548033 X-HE-Meta: U2FsdGVkX19qUyqDTTyIJEiD6YhZ2uh22Nz9UPsg/qjl5fNvsXEUY3nfx35m4kWJjUvWAe+DUgBIVpmG0wB8VT5JTf9KuViCNLJ3E+atyAgp8I/j3Jc2FRtEfkaV3bQoD1kFu896jnf3/35uttqYkZR4JZkt1e5MQGURVAG6O1P+MjKsG2KJAByMEXhMWy8PRoLF+dO/MU7VSY4rhQbJ1rfzDfbHQi5y8Hs3UNSG4UoRU2Vk8zNORMMgl0B4EBNxp4ZlTrUJ6qoGTHol7TiucVz2A8vRuCubvPKR6v4RdLnykWDOkKWsVnOaYgtK8sqD3n7Wxd0xJctgU68gwzBX9cI5wcB7XyTroPsVcyFHm1KnL79FfoHXJMmTaifaL2bzYf11UP1bd/44Bd1tBHdvRyb8LhXFoB6QQ4PP3Xu2iU53yXy118AuO0ZhuCULZaYLNUElqHQa/tkyC/6TmTh6I+mSDRJl4kNLicXZShYmxlXLCNzFHyN7eAovDSleIA3tw+WoiV5Q1kbJdyS4Fsp/WCk8JznfQZUGjkUmtE17Lf7UMD53qzKUbNISgoHpjxl6Asyh7nH/p9BYTB95f65GGl3bvg+fYbv9r/9Hf/OpHaf/r0ScqwsG2tzu1wwi/0cYABbgrNNG6pidHpjo4puXCMWEYgim9XI/JA8bNtlIIpmg4Z8k05ePDXhwkqaaM33YwctG7TI0GM0JQFz1nSXM+FY5Q0PEg9Z1H3HWWISh0TphZRJ6rVzEVnQOw/XvgM331KSkaqz7RE5MDLnaYXPxzSWoYEibGdnbnCth+kpIPLaggBb2giIeYWB5cFR1kFrK9HuIhHw7K/3lQFM51BfMrzYqqVQiIJVdQZBNXf7C4kWDImGC6jpjNmekCl9ZdDxPP+DYyM+Fer0uC6+4FGC1Qfjso9SNYx0Hr4zqtXN9PyVaA8HnfgXL1JDaDMyYLt94EOdq/MNpQNfp8z/p0IO klIjTlDD Oqy3UJ9qztgJWoaEpPbxOX5ywji/YQNYZ8DvSlcnGZeeN7eM6FDw6RCaybUj5VbruK6ZsZdW0A07vULQdm/GfxNkJSyw/QffWLdXwCIvLEscVy7zIS5dv7AcKpD1x32cLS3lmjLJgC3by0jB3781K/ixJ0hFBpZXnL43hDEnbPFluSdXCQjgUkp9ST5jsk1yvqZtATkooT2NQGXNMBLBn3Gb3v9XHIT0tITbHfbJsTj1rWbfE1mUN84uiwhc9/jQ87+j85P+NmSjlw6CXrfVZqNUSQhNiRb+hZKDXo7nnYZbo+0ctmH7KPNN71WzVK0imKjFqJtmNHiShWjWtKGeYle58vFUP+PBSQlTc0uvlZ4JC44hggFCUIn4blndgcSqNmC5GxLHQBrMhFWOntwgPLwr4BtSgS8b9+5n3yLkKhv1ZPdWffDT1wJNYmIvhOh2sQ2CVO4kGACKhNDwYjMhzf7j30Y1vJHAs0UgxcXijna3X0+OHqAVMrXj7reBNrVjqnaTU35D343KGMLHvoVRxAeLzhkBIGMcnZEVJ9O9jwc8Id7iqiLNnSXWSVor+nphkglcxqHD4Y2hrvzpu86JT2Lb6fZ21kpW3os3UhjAMJv/xP970rU7HkRWArpViXOUGAxX718CYgaMmmcbgKv6cfh6LXZf/JD0JzJa3HAg2KJNBBR8LnItp2VjyUA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++++++++++--- kernel/fork.c | 20 +++++++++++++++++++- 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 22b2ac82bffd..64a6b3f6b74f 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -480,9 +480,16 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start; + unsigned long vm_end; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct mm_struct *vm_mm; /* The address space we belong to. */ diff --git a/kernel/fork.c b/kernel/fork.c index abfcf95734c7..a63b739aeca9 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -479,12 +479,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void vm_area_free(struct vm_area_struct *vma) +static void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } +#ifdef CONFIG_PER_VMA_LOCK +static void vm_area_free_rcu_cb(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + __vm_area_free(vma); +} +#endif + +void vm_area_free(struct vm_area_struct *vma) +{ +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); +#else + __vm_area_free(vma); +#endif +} + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { From patchwork Mon Feb 27 17:36:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153974 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADFA8C64ED6 for ; Mon, 27 Feb 2023 17:37:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 491856B008C; Mon, 27 Feb 2023 12:37:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 444BA6B008A; Mon, 27 Feb 2023 12:37:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F9DD6B008C; Mon, 27 Feb 2023 12:37:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CE92E6B0088 for ; Mon, 27 Feb 2023 12:37:05 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7C6AFAAEA7 for ; Mon, 27 Feb 2023 17:37:05 +0000 (UTC) X-FDA: 80513777610.06.8B99643 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf25.hostedemail.com (Postfix) with ESMTP id A5C95A0024 for ; Mon, 27 Feb 2023 17:37:03 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=XsYGyrr6; spf=pass (imf25.hostedemail.com: domain of 3P-r8YwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3P-r8YwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519423; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ls18it+007PHyc+1gU+R0B2Ah0edgQPBYFo5Fg80PJ0=; b=CJapzgghQCFSEFULiQZBbSVcynoSj+ljvw6JNeesf3TcgkTI1zRed6o2iGUa/UjLm4JptQ JD3gv85sAfaYKUA7/JwtvgvMBmfS+7F70g2hCFa4vhAm5BYR2JZanrYkjKSNkZCLQ13oR3 ZVHIzNJQE44cTj2TWMA0czChTB9kHh8= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=XsYGyrr6; spf=pass (imf25.hostedemail.com: domain of 3P-r8YwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3P-r8YwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519423; a=rsa-sha256; cv=none; b=HapTmj0ZlcPRCFL15SfRtpqODgke2oHGmxTEm9fQ1uaTG6I3JY82qfU5lVGM9p+AyY2nUL sHiaVvcTX8slG2svzKmH2294dz8HeFLrYYNt4GsCs2gbMVCCJA3jNiWhJoky9jblkBU04F Oqb/u/mc+phTakDM6VO9m8Db181cgzk= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-536e8d6d9ceso153563077b3.12 for ; Mon, 27 Feb 2023 09:37:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ls18it+007PHyc+1gU+R0B2Ah0edgQPBYFo5Fg80PJ0=; b=XsYGyrr6UscqFf/3E/PqXH+0brCoiV/AniBzcmo5DJxAd/4Bmzv4DyD8sM4NuAQxi8 dZWrcG1/ZDBfEbfkvX95mB7A+YCMYHAJEmom82/9Nzaqf1YzLljv1hrT1/pnhQY+fnL+ IOvvFbrnVTXpiEotFDmf2wc0ksmnu2CR25KjblrUK/fkJWLyctUr7YoQUoBBpu2dXgAS BlvIHfsMMuqia7hwWHF3vceBwA8k9G+49DeoR5tUXJAWoHB1hsAUuDBJJWuX8cJ0Z90U 6Vd32qha+psgDSdv++LPDOcFAXLUVGYeIPM/jmszF46HDebsTawQRm0lgk8RRbgTNhtx sLHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ls18it+007PHyc+1gU+R0B2Ah0edgQPBYFo5Fg80PJ0=; b=vjWMfBqbiDfVNXVVnjz+NL53U6oqYi8noiRgJhN23B3xaPdp7XUsLZWx+Gv+Rebw7L dDU8qtg1NbAOx/51LN1pEI4x/1OwSOQpCvkarESJVmG5t3/Z6gP5zHcoLrrO+mhU7Bfe Ymfgu3QCskOF/9ZIxdqWQhY9CGDO27f4AUAOnzEN0OCWtMLA3eunrFKpRfnj7xAw9Udj 6ZUtIT/7keV9PtBN2V5+9emlkjjJW/RrCicAoOn7A4QEHbOUmySO1wr+XMUCaiesNJ5y qZfQXNo+hEBL742YVtZUBK8quK7UjqzW2vInfENplkSe5zyTcG77mm4rEzLI81ZJhcAh 9pMQ== X-Gm-Message-State: AO0yUKWGiDuyEF+9q0kJLOQWluFacbgZVDnOosRDd4DdhuptdZ8EKRdY DZIQhqkBJ2F1DMmz9bZtA2zXhR4JHxM= X-Google-Smtp-Source: AK7set84bNGznxOoxVaVlILHB4VZYiPLd+BYNZO1PonJXcuULbgT0KQ1/CW8aHys4CM8FC8mlmVpy5ibqO4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a05:6902:1007:b0:8f3:904a:2305 with SMTP id w7-20020a056902100700b008f3904a2305mr261507ybt.2.1677519423051; Mon, 27 Feb 2023 09:37:03 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:10 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-12-surenb@google.com> Subject: [PATCH v4 11/33] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Queue-Id: A5C95A0024 X-Stat-Signature: t6yhg4fz7qyy6zggdib548x554jkb3ji X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1677519423-383144 X-HE-Meta: U2FsdGVkX1/NjhoA98FY+P714en/Emlbfmo/iHx3n9/Lm03LccauGZLDvxJZrFtl0NPyw8kb3jB77iSzY6aOyzeON7z4e2JkiPP39eGb6n0wSRoCWRpy6QwtIqUma7LY0tT2VBTEuf4Uln4e2/KcVrLpzeNlGx8wlaveTX/XURcaw/B5EVqiKTSmBURQ+iCLxx3vlS1ArL9njUpFzECTamRSX87o8DyuuOxbxgSslGFNRvjS7wpJCjvI9wfHfiG54tvr3FzPocD2EdyXiJMy2c33lpXbAODQUCVC07rjxI7fvcBg9+syW3Qg2ChYgqq625gLhRwdxJG4yK0FLzbTsCSaJ9DAjuOVe9M8ZUMps6wwsZsc41EkdsyBNjyUF2Nt/E/DBJQhd5/xjTUkT4OFVNdLM6cHejLTUPAvN6qbLasJxoviQCGbzeXnXef29s7nwUVvaPuTX88yHqMPUSKELXUKhI86dMeMJU7pHnSbsZfepOZ/laCJdRIxa3BL3s0rE0Q94v1oMJTVUT2D8IOMcS5W+cj9n5bThPHbZpKEfLspSTysL0vAGR/fP7WeoCWMnUEN1B51qfl0k5jA1EOLBsxQvr4DpbJQsiZ6u5dA6kmVmLAYBvp7gBpb2GQKlicmddOZ+Jda8UCB6LilGfKto7pN7lZPmer4prmbZ2wN1IIyExWdY57MNtoL+q8v2KAT/3k5/8wbt2fJv99hae7NWn/zbn+qIeXe00O+4WMpbGyeLsNI4HDpJ70xdu+y5BbsELBCbMCTkCoHvBe+w5ZYfeYCR1cVKuFONxro+elFgIInJpyMHPZg8CK/8yQczHtjNUCYtk0eR2mR39QaKmiyJmMhp0BkzxnuiKx1K9J+Fe8VsqObOmV6I883R4cFOa6jfahYfcFBO8sk6JeUS9wbHZy/KFQBSru/1mAZIMe2QreLx8ooz2XLRwat6tzTymZNBVdYGeX9MrxAFVGQo6K WXuJuuka /K+ba2mn4AwxStVV6WjucwWKX8Nj8lfa9QsiKfUrZ8X06QvXn6Fo51Jt6/7re5LPzTg/88tABcRerKBmMYFjOyuR+zAzEfe9Gdnl+gnRY27xMq0D7+E/eWpPi28zsu1vYQ9KQLFWH51doz/R109EDjvc2fSoeK4zrZcqWl/5nKJqfoK4M1JDDqmQSsAw2226lieJkRxmjPjTZZGCMxgiTsnMk9TpjEjDZdvPlE4+SjuWK62duuwQ1oMEwzAtUlcAGlSafGRgxIcesROotpqOpynlPFAetHvi+4joSmQ3NgEmOgFtKIcxZfu9N2k4UqtgfbgeLujnUJLZ6oqM26Frf/EIN/ABkhsNiGFyZu/yM5Cz08XHaHcKke04S8wkDYizJ2YmrZiBdqRuNJVDTjVzbRmNDXMNR5EgEKN8swE/NZhSa/qLjwTpUVxxabOCKgbTKslcwSXJDnSqp0jPzehjXt97VPAKJr3KPpymjg68cSBrle6V/ZljlUcrOFLX0CISKFrtlklXstUtPeSGHgLPr0Z/JTvB3uzBb04C3bq5u0AALANBwwVKkoYBi/Xy1U0xTrgMYgW7zSCJx9RnFo9cwdx0nbTsYSQiUgngTXdlWSOqWi8T0090m7F/rcHc1JJcEW3eLuM4grHWMXjo4aNYi1aW9/Kjm0mOBbBh4zgoc5sM61Z4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Mon Feb 27 17:36:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153975 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 231FCC64ED8 for ; Mon, 27 Feb 2023 17:37:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B707E6B0088; Mon, 27 Feb 2023 12:37:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B45916B008A; Mon, 27 Feb 2023 12:37:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0DA06B0092; Mon, 27 Feb 2023 12:37:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8FE336B0088 for ; Mon, 27 Feb 2023 12:37:08 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6AAA440616 for ; Mon, 27 Feb 2023 17:37:08 +0000 (UTC) X-FDA: 80513777736.05.E5A1AB7 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf23.hostedemail.com (Postfix) with ESMTP id 9AEC3140002 for ; Mon, 27 Feb 2023 17:37:06 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NrffhdAm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3Qer8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3Qer8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519426; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q/GJoky+VbSuk6dzXc8JP5Q5/FC6q697TPLIvz4vCWo=; b=ZIxPj6b3DS/EUW7nJSLBtPBkYoEIOoFmsJvgJ1fc/Kp0yvlj5TYXaYymnG2xp7n1dYCGUS ImVZ2C51edMl3jpO+sv44nMF7PFrII/o4JhOT24AYdGKWzyUU4mjbnWxwpg73WT9ivZlqV ExzVCyZemzKV9VdkIPlZ+U4lylo/azs= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NrffhdAm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3Qer8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3Qer8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519426; a=rsa-sha256; cv=none; b=2VSdZYPPbgtdpFetLwXoTko6gQA3fLzeRS0VOXn6s8eP+U0cHwe2X5JdunXobzKLwlG2FD TmeDdew53O9gCybx4KUTR/rXrN4UOuZMTcaClGtmkTIT5AbpLN4SvpRRlk7cSO5Y19ZP9j O1bhnPyW1d5qNT/GoZS7zpwOTNjGSJo= Received: by mail-pg1-f201.google.com with SMTP id g13-20020a63f40d000000b005015be7b9faso2178534pgi.15 for ; Mon, 27 Feb 2023 09:37:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Q/GJoky+VbSuk6dzXc8JP5Q5/FC6q697TPLIvz4vCWo=; b=NrffhdAmMk53hmq587bWVqAcVwUoFMyrGdI1dcdSP5rFWcjv1b2dxTOEh/45qUYAPi h/0PdE/bzVHMe1/toIQLnI1PA1/EP/QcNn95krcTC4R7+IMiZJdtCRCtMs9jcf3/2ANu i0kT+E3X8ZM+V+p/+ReYSUc0stVg3Z855SoJ4cBQAEK0+2hyKtIS1nwe4GOth2yQHLfU w8Fk0BJXAVVwtUoVe1w9Qpj9BUCfZjuVzX1iYFGwYp+nbDSsHPi+lTKw+kw0AGwaK95T wMmibkxVEN+68L0K6akco8ImKzGRBceFctt146aukjldpyDGZk2PpWKH7Fdr+teexTnY XHzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Q/GJoky+VbSuk6dzXc8JP5Q5/FC6q697TPLIvz4vCWo=; b=C1LPkgV1IP6bzx5snlKjzgkQraqiISFMDu3YTI9rlWXN1Hi30Xnl+9e47e1cVXg5PI QSaigPygeZmZQ+hJpQTq5HvBb6l5o1v5A23FovwB/6Mb2wf4oMOLhqiFxEAFBPHmGHF9 xyxNypsmNTeu7Q/uLLNwK1f6KN1x4gIE2ThvrjIYzt3DsOMjv/vLV0Y9CzHs1Bgcll00 pXa/dSzbOAecJA2MBW93sjrj8f4H7XMWrtjBDy7l++uUM+29i2U3OWXMXVG4z9rSyKmf X8Aeu6A4tNL7lsEIO24uNGOTafsqgr5AnKte/m8MDXRaaT10AlWcsS3Xzit0kG6nEIKx C84Q== X-Gm-Message-State: AO0yUKX+x9eu06u+pHTf0BFmv1gZQrqBWEbK2eg4Fs9OgNaFXgHPZ/DV rzHbxScH3VeZtVALrQjwMZTIT9KiLMM= X-Google-Smtp-Source: AK7set/svLuIVJzjBHat6rB8sP8zcMP/ZnW0bz7V1s9w04rE5j+sFd4oXDqXKYJ9s+zwLVwyfpWMezpHbsk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a17:90a:1081:b0:237:3b68:e50c with SMTP id c1-20020a17090a108100b002373b68e50cmr30137pja.1.1677519425412; Mon, 27 Feb 2023 09:37:05 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:11 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-13-surenb@google.com> Subject: [PATCH v4 12/33] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9AEC3140002 X-Stat-Signature: zatzcsbdypsra8s1fgn1f58gnhrcxzd1 X-HE-Tag: 1677519426-549607 X-HE-Meta: U2FsdGVkX18sO0OpCCS8/XO8GlBbQkP4Ka40b9FRMk6Nrg2+GGkvrzQHMUeRbUF4cGzVlxvcZynoVuH0j0KRqXk7r5wIiUZrfTdc4h8CPkin0FiME62VDdWnxJ+bGSLT+mNvUpJEPB4RbxIw7eNLI8P72ud0APvGwH5p0e7kuCRXtkTX4IvyW4Ae0FKtx0+WLgQAwYcYN53+EkAoQlZVypE03ABa592mTckxC5SikwMzvkGf1cbaVKyqNpXmit++Lk2GhizmijqvXB1UL2298ur7DCz30MAvjgUH980M+Yb5q/X2wK2qMIusLslqO9hAAAOD0ztF4+14e+48otOZL5xg1GU4D01l12n0WNi1KooQAbIPpsJftnTw0b1VWMD4pERzLyjhzlQdYAsTc+CFb11S5YZ5SDKUCRZUQ7jCDSWiJ+uP9gWNlpH+4GpV1m8ABlcPpwq/uUQiX19nHGSGKQSJFQAOZUElfHhOJ6pjQzO6s7FoE73lCFx1IuCWvLlPriF+EHoA4R8miL0lCMBfXS1m9EoVbJOSjIi6qFb4qdUJWtZJ3ZMudtIGD+2/Rr24MVqqhbKlLGPJ/+4EvKamIDcxLU9XrwWh4+3rgdQl1Sfm/El/thsY3/bx/4yu2IWAqc2eho1Bfep1ifmB/ik+bz95QcZs0R2kcKjX0ALKh6j8R9JszicbihMpOW2JP/ZbgRaI4ihy9KXKuWmYY7otE4R2WANicUlTC3aHMAtMKoZpQnK+qrtpYgJu/crvlosntLG9bBiD2rCPbX5li/9uqAal5jo3kaEHXb7iXZ7I/ylB/YN3fXDhe9nNSqxdf4D/XsZ/k0czGWHLo7hJGBAiIxcBKZAeQbOVZK3ZqUxcorc+ZTOKuZV9DBSaUy9XAyJnnLPTGip7MYEONHBpwZ23q5dGvWCDQEpSyy4alFET5X4eReSs7ndsFgO19zVNyuk6vu5cUzGat9SZlzWRsQr 7SsiapRK NgR+YfREgO1bdyoebYcaYD8yAUuSEX+pAnyuyf/vuAt24ukklzjfy2V0JuouhjfeuJu6nTsY84X/hKHPK4cj6BjS0rl681/djLlarYyD6nQZ+ak04TCH7QVv6NyOmqGOl2BRoAIt9HsLV0suyR946wIMvY46fqpIgyy04YSE8AJ8YZ0vYgkIufGyfpj/gq4TgDA/zkEIm9M6MR9tBote/FJ0KzMyWJycb6g6s8EVef6tgGV+EI3V5P2xeHcSDjWaj1POFBD47Tbba5EWljVtCerWlgu9ngNU24pYXTt5QCWaWnDcx2T6tusfs05AbRLnnG8J4bRTfp9KbI5EHchJvFn1MxodL/21lVMKQL/UYhQ8X+1EZuE6dD+PymSziplKNcX9pSrpm1DOj31qjjeco0xMvJqHrS9QZuV3iEJc4z4RoeMft0Ao/9kp4F7FSfFxBOIXHqCD48mPMdZYnJL1gH4QZTtXk/Ibq1mEwKR0D9a4Q7TpVFoZ/qL/8SnSZpNS+jt3BWe7QZV/546P27GorY7C5jw26LZtdbqpeLU45whgQWeDdYavjGTn+2TQNk/BgJUqoFtahFVEAE08u86NJXhwP8INjyAhQIYv9y41kF0f8NvCPPpyaao9vih7qjQo0/QcQf7LapvOtWpN4vycKz+5Crg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce per-VMA locking. The lock implementation relies on a per-vma and per-mm sequence counters to note exclusive locking: - read lock - (implemented by vma_start_read) requires the vma (vm_lock_seq) and mm (mm_lock_seq) sequence counters to differ. If they match then there must be a vma exclusive lock held somewhere. - read unlock - (implemented by vma_end_read) is a trivial vma->lock unlock. - write lock - (vma_start_write) requires the mmap_lock to be held exclusively and the current mm counter is assigned to the vma counter. This will allow multiple vmas to be locked under a single mmap_lock write lock (e.g. during vma merging). The vma counter is modified under exclusive vma lock. - write unlock - (vma_end_write_all) is a batch release of all vma locks held. It doesn't pair with a specific vma_start_write! It is done before exclusive mmap_lock is released by incrementing mm sequence counter (mm_lock_seq). - write downgrade - if the mmap_lock is downgraded to the read lock, all vma write locks are released as well (effectivelly same as write unlock). Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 82 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 8 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 110 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1f79667824eb..bbad5d4fa81b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -623,6 +623,87 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +/* + * Try to read-lock a vma. The function is allowed to occasionally yield false + * locked result to avoid performance overhead, in which case we fall back to + * using mmap_lock. The function should never yield false unlocked result. + */ +static inline bool vma_start_read(struct vm_area_struct *vma) +{ + /* Check before locking. A race might cause false locked result. */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) + return false; + + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result. + * False unlocked result is impossible because we modify and check + * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * modification invalidates all existing locks. + */ + if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_end_read(struct vm_area_struct *vma) +{ + rcu_read_lock(); /* keeps vma alive till the end of up_read */ + up_read(&vma->lock); + rcu_read_unlock(); +} + +static inline void vma_start_write(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline bool vma_start_read(struct vm_area_struct *vma) + { return false; } +static inline void vma_end_read(struct vm_area_struct *vma) {} +static inline void vma_start_write(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -631,6 +712,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 64a6b3f6b74f..a4e7493bacd7 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -508,6 +508,11 @@ struct vm_area_struct { vm_flags_t __private __vm_flags; }; +#ifdef CONFIG_PER_VMA_LOCK + int vm_lock_seq; + struct rw_semaphore lock; +#endif + /* * For areas with an address space and backing store, * linkage into the address_space->i_mmap interval tree. @@ -644,6 +649,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..aab8f1b28d26 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_end_write_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_end_write_all(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_end_write_all(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_end_write_all(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index a63b739aeca9..e1dd79c7738c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -474,6 +474,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ data_race(memcpy(new, orig, sizeof(*new))); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); dup_anon_vma_name(orig, new); } return new; @@ -1216,6 +1217,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + mm->mm_lock_seq = 0; +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index c9327abb771c..33269314e060 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Mon Feb 27 17:36:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153976 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38708C64ED6 for ; Mon, 27 Feb 2023 17:37:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5B1E6B008A; Mon, 27 Feb 2023 12:37:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B95456B0092; Mon, 27 Feb 2023 12:37:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A352A6B0093; Mon, 27 Feb 2023 12:37:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 95E576B008A for ; Mon, 27 Feb 2023 12:37:10 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 69E4E140B01 for ; Mon, 27 Feb 2023 17:37:10 +0000 (UTC) X-FDA: 80513777820.19.FEAF4AB Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf12.hostedemail.com (Postfix) with ESMTP id ACCDF40023 for ; Mon, 27 Feb 2023 17:37:08 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=XBhlBuuW; spf=pass (imf12.hostedemail.com: domain of 3Q-r8YwYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Q-r8YwYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519428; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I212iwzKiM4RAh4/x5atSdllFVTlqkGJNP08f3JbnWQ=; b=R/IwkxRpbpX2L3L2tyMs9Vsu8TfQg63i+O9mVSmm7i9h0nFDZR6H2wrH3UAB5sbHgbVOks SbcBRGwl0ki+RwvOS+ym46vc9m4TQNIvta/eeJU6T5ovy92GOEB4bSGp38rPJSV2GdSXIw vfeK4o5gUyYWg4xVlL2S1Q8I2JLhYDs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=XBhlBuuW; spf=pass (imf12.hostedemail.com: domain of 3Q-r8YwYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Q-r8YwYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519428; a=rsa-sha256; cv=none; b=0lPq25YniqNQYLw1tNNeMACqCqoXJVZ8YMkge9WQ41mVr7doLHET6lYWPGkyIsZx3HXs0e kid3Gh4Ieke+K+wwQfRK8uP4ecz6DQ0ohvN0XOYgLj6b1k/qduNli/537NiikCuWzjDrLX 5e8dNUW+qlhhXISXW917OU40zAJ2JFI= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536d63d17dbso154505707b3.22 for ; Mon, 27 Feb 2023 09:37:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=I212iwzKiM4RAh4/x5atSdllFVTlqkGJNP08f3JbnWQ=; b=XBhlBuuWfB0rAIGd6hmSTNeCzmJSeNc8sCljBThxSUCF4xl/MSw7wD8HJnBQqXsjUJ 57kJQANXg6fNdWd8Pt4eiSrUjDCyUw1I4L84NqCzMgqp7jq/T8VmsbuNmrFrLa4O4XYU GqAQJ+b6Oca+hy9pxgzol0kd/HIC9ER2wtl/KEZHOhFYwY8HxJnljLvK30eOyvBMk653 HhqyRDcfvr4BXt/19NWrR6/WQesJSjC/tN0jtggmEFB14E/QwVd/f4szWEolKHRDFF2I b8hGsHDba//nCM47ED9hrsor6X6kCcWnUTvy8GzRTm8e9DVF7x8YXaJ1E8nG+/bnJf9E 7yEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=I212iwzKiM4RAh4/x5atSdllFVTlqkGJNP08f3JbnWQ=; b=wFQA6wHPvYFcQOzC2xuzKo3CZ6x+gSAdxcp6EGKIRpTPx8bw1dbFd2IOgGQpn1JYf3 mst9GFZrRvmREFI6Oww25d+Xy07LvMWWTg45Ima2B6ODnH2rK9XzewvE69gwnL3j/Nhz PJIo5zehvFUZM7I76qXormA0zl0S22AAC8XLZqH9nTTcGhxhMwlKAeS5cdfofUbmiTZ1 pEwiBkVus09dEOKd6vd6FcWZkOB9RpyBo54dR4up8/vQE+81oTkN9HUV8H698mZWTRzv tWg+zv8M9PaUuoRu725jEXJQ0RxdY9Op1nKXvQ/Su+wWaIS1lEuGY0LqRuIPmLTtv3R5 A/GA== X-Gm-Message-State: AO0yUKUN53d26EhEYuxL8Pat0Go5h4d4Nn1Jx1f7FSi/i+7zpnDIo9bk SM0oawL1unFQP8bw5ELdRd2bb3h7fzk= X-Google-Smtp-Source: AK7set92Gr75A5uaM9VLoSN3m2+g8CdJzSD2poMBi0hZE65OwRnSrd/3zaVdEb4P76h2ChlMWirB5h4UQZs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a5b:a4b:0:b0:a06:5ef5:3a82 with SMTP id z11-20020a5b0a4b000000b00a065ef53a82mr7473143ybq.5.1677519427726; Mon, 27 Feb 2023 09:37:07 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:12 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-14-surenb@google.com> Subject: [PATCH v4 13/33] mm: mark VMA as being written when changing vm_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: ACCDF40023 X-Rspam-User: X-Stat-Signature: pfkp1rda9j4uzup6f1j45dscsxdkz8jo X-HE-Tag: 1677519428-466450 X-HE-Meta: U2FsdGVkX1+aPsXnPwe9fPHE5wCBxfrgQmWFw52+vJxjA0qhBAxMA487YqUWiEwn9NMLx/m5siWhOS5rVol+7JF/geFsyPx+GpXAIPRLzhITs6oJmgMVWvKcXqdbvbfGFEPt9jYL18wAGeaNhhuVOpOYw4dR6xru1BZ8xpGAaZoVLaKvdHlG2jpA17kNHOb/gdE6uaJNdTqcrHPFZa4yDMtl5OgWD+VQqFYOIDg8i9hC6l38w59FG3RoSHsNmyb9j1As9eSpDMengt6r30UwAQqffa7iF0EIMYjp1CIfJE9ogyKHDd2FpqzqEbb3qnWG8MXsI/0BS8bH2iHkZMEgCYJguP2hZ2vJVXDOkC18Esqx1QVQQhj7FrM8cpPAUGcctws1h0UwoqchEGZ0G/82gEdbVBR/Gid3WkDjZeqdLlZb7NPgRz1d8gDcywWhgvCzX+ckpLfyaeqNSgVMLX1M7pYpp8CBkbwiu0VzsmT/v4v60mfq1rlYhkURrdoCKn1bJdo9mbfwZ1y+Tjip3hlw/sXJItw2H02MFtM08jzoRfA0sUvHJxJVvntd/zswo1nFRj1jGOmOkvZfH6dC+DB+Uhv7FrDLzTPAnCj1KC7PDJUPGCpuZye6i/r1KemEavq/gFrMTCAQKbcPsAZFNEjBYVoxxZaCD0rzryNXrHthtA/8//T1UupdRIofBwXVFVF+vKW9ySN2RoaebhpzYVzNksxRER7szoGNDRvdntBoYtyiAmODpIIpTPSXY58WQ+K4AooqjKZEbz3KGuTaHyF5kZoRA8AVKdx5LsilDUvyYQ35soWzElxvSPhi+9oiejX/ENeYCAIssgmRBKQkjZX8f7hhaOVXSCp0GAOPL7u/ngK1Ih43Pl7ZTlGdiasbqKDCHPUNYrKrX8o3ordXUR0xh4aThx/c3N8hJy9UHqcNcmMcok8k5JCEWkRbvBUWMZ6cn3E/Sv3+UBkhiwcEojF R+dzr4OJ rEAkF3UdoBxtZFRMci1gJIyarBLZaFP3zbU0mBaiTvBbiLRzcwAUTsNAuMdrsWHUpttHL7VJnk3htf3tVxLUloR/zfIvNObprwddHcSdnwCBCTEES7pyZYoM/gRbNXGBH0/VGzWMe86sHWL5Pmc8wdz9zrpgL69IrnDDw++ZbZoYsAJo+dW2b0mdAXyZizf3N3GvPSAYeblr0joFyFMAh6/T6lIQn6MZFHOudAdgXx5LCtFd4a0zun4jWcZsPWuwYPsJWeFOOcUUadMU029WPM4eR/HxcOlhzqcy56qG0tSdMbhpX++g72Fo7pioS9dUODIyqhxUbDsqKhd8w8m29gYMz5WHv1TqoYcxY+lReO/i9wPtxMLI7nD2oxa4ySwjY3ICRzjtDxum90ptFcRQpwlcaetdJwQrDBlDrFJdGbaSldcYyF7nigRo/Vw0KPmvaMhugzJWICHFJtBl0hhFWGE52p6mvFAErB3VAQxCkZ7Dsn0wQpksVLnYnz3DVgvy8ipVvGxTpgA8eC0iqSNcMo8tTK0vHAb2FRrsOLRZif9rIgBz364e5DurMyuEYUUgOdcXZC5p+Eu78QuuH3D/ixdXH3PXZt351P2qsurU/iGM06NE0MurkF2P6oCFU2wLSwUEA5zLiK84n31jd/2Isw9u0IHlPsQYQbwT5bryD7XACkks= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000028, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Updates to vm_flags have to be done with VMA marked as being written for preventing concurrent page faults or other modifications. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index bbad5d4fa81b..3d5e8666892d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -726,28 +726,28 @@ static inline void vm_flags_init(struct vm_area_struct *vma, static inline void vm_flags_reset(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); vm_flags_init(vma, flags); } static inline void vm_flags_reset_once(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); WRITE_ONCE(ACCESS_PRIVATE(vma, __vm_flags), flags); } static inline void vm_flags_set(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) |= flags; } static inline void vm_flags_clear(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) &= ~flags; } @@ -768,7 +768,7 @@ static inline void __vm_flags_mod(struct vm_area_struct *vma, static inline void vm_flags_mod(struct vm_area_struct *vma, vm_flags_t set, vm_flags_t clear) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); __vm_flags_mod(vma, set, clear); } From patchwork Mon Feb 27 17:36:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0AD8AC7EE23 for ; Mon, 27 Feb 2023 17:37:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E7B36B0092; Mon, 27 Feb 2023 12:37:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 970366B0093; Mon, 27 Feb 2023 12:37:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 839136B0095; Mon, 27 Feb 2023 12:37:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 73A296B0092 for ; Mon, 27 Feb 2023 12:37:13 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3D68E80A55 for ; Mon, 27 Feb 2023 17:37:13 +0000 (UTC) X-FDA: 80513777946.21.215B5B0 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf08.hostedemail.com (Postfix) with ESMTP id 5E57616001E for ; Mon, 27 Feb 2023 17:37:11 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qq9ST5KY; spf=pass (imf08.hostedemail.com: domain of 3Rur8YwYKCDUjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3Rur8YwYKCDUjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QWCYeH3rb3vSAFp54Mmrv1+eix/+tyqClZQn+3lumNA=; b=eJl/Kf3d80MFfonKldym6G7f0K1tEn6qZyOVkc1HHwR/tnYfxMQJOx7eb87xsKPcPRwgwr nlJgpAEVtkGoHJ+x9MZk5DJFyOus0Ss36tdyhwYBfIL2HCgBkGk6OfMMCIlxHjpmW264iN gF2Sx9MdEpyIKAc+pkQsJtoa8eT2qcA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qq9ST5KY; spf=pass (imf08.hostedemail.com: domain of 3Rur8YwYKCDUjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3Rur8YwYKCDUjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519431; a=rsa-sha256; cv=none; b=w+jRE75eC6/TdhObovOBtHmft9TbwFRdRzLOncDr1qJKs+t6ynO0L6lXZL9HW3nCDIGjJZ yF4GasRPGYf6m8nsHIJmx9TYUlrR6NR5m1AWUreggUvyZpfzShTkxz+IFBuUIdFIsEVeDI 3vsSZ+QX+Snc1A5zBe6B5/Guuvz36x0= Received: by mail-pg1-f202.google.com with SMTP id e127-20020a636985000000b004bbc748ca63so2166410pgc.3 for ; Mon, 27 Feb 2023 09:37:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QWCYeH3rb3vSAFp54Mmrv1+eix/+tyqClZQn+3lumNA=; b=qq9ST5KYd7H0i08HiNSL8c29Xrjl3aIZNo0vqd33ohY7IlOekjqUFmHX1pCbc/0CIB UJYRScXv/uQI5WgiH6m5n3NDzNqXvXc+EBT3Nr8sb84051cUA5rcu7NsZAUPZLQbAT+k L1+D52rxNhib5vRaYFXHr0l0OBrPbhYiKHs0HG8E+v4FWKdNvWFhRWhiPM7+2blQtGKB AQqKpecJF/PSLRs/Iu3OWc/90nRfqSr4OY1HTu201oV1jp1/8gN3EH/qUSvYipkLdMaB KA1JreUYCECV/wYDbpy6/KFPjd6XlpEX5mJczzYRoBrzVCeQlTDt7q+I0gm0goYKe+3A HbFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QWCYeH3rb3vSAFp54Mmrv1+eix/+tyqClZQn+3lumNA=; b=oNGmomknAu071I29g4qitrfrfw1EpBZ7jMNsRLN6vkCLRvOaSa9osJiQO7O0ZBX3K4 J744vRcGWQhKEDEeoBWbYZELGgKFzIpmngYFCU3Zx4W6qhl315pN+7Zw4tc+vRkF0uN1 wBKNuQqsWrw096fevMb7X9UAw7AZh9tH604AZaXgs2OKl6xuWdUZF+pzsHzyUcRv2rpp VQphqTRhER9CusvHd9fBiuvuCXFwXjVSEaX8qPcTjyBniQQfswJvsRMmz4axg/ISIONO UUS+65E1VB7lreaysRTyvupR2xvwjmx8Hlj3lTV4/RWbPS7Uooy7ak0VMTGPdH82BaEt 1MPw== X-Gm-Message-State: AO0yUKXxOSHbID6nNwZL2gVgbfvMpzV9gLwVDBfSOovo9NEff2PGZKCz r5ibLBKT8uuFOx425leONYeP3dXrwt0= X-Google-Smtp-Source: AK7set8jO949nwq1qPTfiOwF5d/NxOGl9PwUBICmy0J7iLD+CPMrX0ZITRyhhJB1FHKtenUKzwzPrWF2iBQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a17:903:2682:b0:199:56f2:3fc4 with SMTP id jf2-20020a170903268200b0019956f23fc4mr6277864plb.8.1677519430200; Mon, 27 Feb 2023 09:37:10 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:13 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-15-surenb@google.com> Subject: [PATCH v4 14/33] mm/mmap: move vma_prepare before vma_adjust_trans_huge From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Queue-Id: 5E57616001E X-Stat-Signature: jokfnrewmx9mgzhx31qf87htsjh9cuz8 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1677519431-580107 X-HE-Meta: U2FsdGVkX19jyDlgABG6r/LxnJ1Dozo1CVc35tgQ2OjO+u2v50yfKIe7/D+/z2wofHpbNoUZcd6tm2bSjRse7gpG88OAWMiQFqxPRwJveG3OppV1UoHCgXR6WrYMLLeLPUIUVN1tuB5LHph6G65VPsUsPpCE30dvLfkkynX+MTn65o2RvMktKNyfOKfPrjpwLfGbBEzcBet2FIXqvRGyeTdGdCabULQ2lW4Mrl5oOKpC9CqVfs9WuDFRwN9uD9xhoBrHQ76crbJ5atuQDU1GQXKboRKkCtJPycSwhCz7VPHYY2EjuhlGgsq1L/FGg8eLbcuomJ4/pRwsVTPzKZD94tAEyB8ydWGCbmJd57a2KUtVaj/Ok/bkkbAKi7WNIrbtuuOthKI4pJjZmLGdkR0Atm0LBAwQmArtMeOD6xfY0YaDPMili6YLvgOpABN8yMWgJhQ+hkjo5f+2OIUhgH1RgSXlzMgYRPJAHeFN8uA+/O+mDOp5umNi4LzIWDHXk/dBE6YkWxkzJbyVMK4fMvkhbzN6oZJlULi4geDZLMFvls8jMkgrxxj4HardS8TN7DrxL2toGCrcmi7kde+Be3WUWUdtUjDvKF2fSuu8mL1UHF1MHftuM4i38C23UF//GqC91uNlEl/JuK+PffFO5JEYdM+z53rrBrD6Fv0I8qWTEak+92AhemeURdSz+kaWRpl1zlGxufZkEwmHs82S2Tqf6gCElHBwfycPaiQMYL4frvNali9lZoW0jDB7KoA6UeQqQIHb+WQBhTxACDBG7v+IGts6Iqo6B4uP9NQlF55czrO1MM7QX+SD6Zn2V6WWmPXftyVJM2hL78EN0c4HG0eS/AWdiKKNPacvOLOeO+Z0gNCrj8fb2PQ6KRg0ixKimAoF2neOWHurJnZdUFynxyimwAZd4HbtmQdLYemsK7Otl8AiCZKZbTwealKqetXtFEMkk3xJeibc/WuhASdazPc P8I/6jjI ZMx0kNGaOYlDhrpBF2bLjY27ZRho2A+4VcRNCCCi+2XKYtQ7Lpz1YMd8zK2DEKz910Vh0nAFJNVsvDD+s1b7pGvTgfBTs+EsC9abn9ZEXcZy0nS7DxsTCFjny3gW3GFLJTJ0qntiLqYZfKQjfYh9YxocgrR29UiJjpMmSSGpeoSASKtCx75VymQCzXUqBMzLx3tmkkKz2V5GPr2fWIboPbMX5U+mfQo3PJYGVqoELgBy/WX0vpB4yQkIPXcuVrMIQqANxj/IvssW7+Wx31CJ+yTOYoi3gWnPgQ4DAUtyeVzpUxGRi88v3myGElAAKV0+vr7uIa4dcvku8eeTsSNM3lvyoXFcUwajoruBOHqkg4dHxpx1mvfALXsAaZuO5kGq4Ov6UF0otUUaCnFVVvho9HWk1WJCjlJiueod9BG3fq3EjnM6AtVTjvUbiLcKA0/g+Puf1c0lLPFxffoQQTabBz8ArAAf7qYYJsysNoA2JnKqZHo7PgeFsA+67k20jsPF8Dud+TtlO/P2VGmRunCi+9RY6b5my2vfz++YJb0OCv5DIqMwoW+XTyl/ytQtjRpywZXn5StP2Q16YiJGQNGSRjVN+f+XHeifT24CpusGN/Z0/EgHyYmhef0248JA3Ig1RmK3pO0bhbffGZg7/1ZprbEypgdWbxDrFvnFCE2MuVR+JIks= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma_prepare() acquires all locks required before VMA modifications. Move vma_prepare() before vma_adjust_trans_huge() so that VMA is locked before any modification. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index c234443ee24c..92893d86c0af 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -683,12 +683,12 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto nomem; + vma_prepare(&vp); vma_adjust_trans_huge(vma, start, end, 0); /* VMA iterator points to previous, so set to start if necessary */ if (vma_iter_addr(vmi) != start) vma_iter_set(vmi, start); - vma_prepare(&vp); vma->vm_start = start; vma->vm_end = end; vma->vm_pgoff = pgoff; @@ -723,8 +723,8 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, return -ENOMEM; init_vma_prep(&vp, vma); - vma_adjust_trans_huge(vma, start, end, 0); vma_prepare(&vp); + vma_adjust_trans_huge(vma, start, end, 0); if (vma->vm_start < start) vma_iter_clear(vmi, vma->vm_start, start); @@ -994,12 +994,12 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (vma_iter_prealloc(vmi)) return NULL; - vma_adjust_trans_huge(vma, vma_start, vma_end, adj_next); init_multi_vma_prep(&vp, vma, adjust, remove, remove2); VM_WARN_ON(vp.anon_vma && adjust && adjust->anon_vma && vp.anon_vma != adjust->anon_vma); vma_prepare(&vp); + vma_adjust_trans_huge(vma, vma_start, vma_end, adj_next); if (vma_start < vma->vm_start || vma_end > vma->vm_end) vma_expanded = true; @@ -2198,10 +2198,10 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, if (new->vm_ops && new->vm_ops->open) new->vm_ops->open(new); - vma_adjust_trans_huge(vma, vma->vm_start, addr, 0); init_vma_prep(&vp, vma); vp.insert = new; vma_prepare(&vp); + vma_adjust_trans_huge(vma, vma->vm_start, addr, 0); if (new_below) { vma->vm_start = addr; @@ -2910,9 +2910,9 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto unacct_fail; - vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0); init_vma_prep(&vp, vma); vma_prepare(&vp); + vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0); vma->vm_end = addr + len; vm_flags_set(vma, VM_SOFTDIRTY); vma_iter_store(vmi, vma); From patchwork Mon Feb 27 17:36:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153978 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D88CC64ED8 for ; Mon, 27 Feb 2023 17:37:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B21C16B0093; Mon, 27 Feb 2023 12:37:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AAACA6B0095; Mon, 27 Feb 2023 12:37:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94D556B0096; Mon, 27 Feb 2023 12:37:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 856176B0093 for ; Mon, 27 Feb 2023 12:37:15 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4F3FA1209CC for ; Mon, 27 Feb 2023 17:37:15 +0000 (UTC) X-FDA: 80513778030.02.005A652 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 84D1C18001A for ; Mon, 27 Feb 2023 17:37:13 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=eYNrKBSi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3SOr8YwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3SOr8YwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wnWzHyaDa4xKEsusiA17Gd0QXhxkYtF9yictDdgKzrI=; b=YiniyJBkKXpdPw/x685v7SuBn9ONu+FYh83Npt4ge6a0/PVCd29kWpDa1MQpl01+e2ALuv X0Pa/NFiQcoakHxc9ET60VGTnYqrfb1YyiiQdrQB3waWHv1Og+NGQIYdQ+naSVBUhxGGWe 9O7UXZAoKa+0yoa2B5sUmuCFDOhQBTM= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=eYNrKBSi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3SOr8YwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3SOr8YwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519433; a=rsa-sha256; cv=none; b=sBZ3MFsETD2Pu9pnFCf6Wgfq5mQmODSNH/yPCqD7vXbAh+KUr78zgCDA8V/nE4Qhg3mB5c 6QoQeCJeS3NhgCiNKD846mfYPzPz+dTc2RtL2TcdpSYWyDMFQEHuGhfgJRUGUAWXa1cJVK aVvGkbddseytEnyvZvUhbQjiTztvmm4= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536be78056eso152390237b3.1 for ; Mon, 27 Feb 2023 09:37:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wnWzHyaDa4xKEsusiA17Gd0QXhxkYtF9yictDdgKzrI=; b=eYNrKBSiv4pp3yC/iTCyZADelL0yGuzmaY6iqmhnRhO2ZC4AwjZBlMwQ60T7+F6v5V W9pXVV/VzgegCAMywuIjmCwiOA+90SLBeJW+MLSiosTMfuWNrFFPzi/DpMcpFnT2QAIz CS8A9pF/mDc6TBfXFulN3NvXyigohh18r+RbFrpjgpOYnpRGGrUngztXOUUKHZu+O84j vZr6z0eLJ0rZFHfCy+yhp6M25rLJHKBYEd4jK+R2iR3coHxysAy7kDI+ez956k4JFPra ihhR22fhGEVzhhNLMgR5SxjkhXirQ5xjL8arQw+dXDtDF4Wmy1/3QOzDKcZaxxbfGAP+ YaEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wnWzHyaDa4xKEsusiA17Gd0QXhxkYtF9yictDdgKzrI=; b=v/3gQJk1fBydZxxMhSrtNFgQFxtVkuB9xUs3HljKXt/aYRu4Uh5OwmlAU+1jcUwNzt wrTofZrZTBS5CLG8ZpAJuADA9JjRVWe4KtlH4lgcjB3ez68gTc1PzFo/rFhyOoXeixUi T6HR0yZts/zbr1jKNreuSFKTg9uW1s03aC91HQPGI4sqzaZKW+bCyeVf1gPyizhFsydD CHCEXFFi5O9PwACOx1dsBoOI4Zj7NoqRxD1HdpQjtqN2vWOJJFt/x8lNho7hfmHK5F80 L+fnhKDeo6zBW8VSeyP78ytk5hJ5j9iXj+w6blmYX+kZENBzEFO9zHqdH9z/YbSC+Tro xqhA== X-Gm-Message-State: AO0yUKWJv06JJAuX5k0RkpgDtJrXJMl6XSvBbV5qL6oXzqBRKepw6CgZ 73zx/OWAg1eyHhhtLIQlpeuc3QHE0Lc= X-Google-Smtp-Source: AK7set85RoEyMLPPVhWwtVXRmKlr9rEFzrfKcusuRBnfvQixZvHj1lhFnYPTkzBCMOeQSYlYKbHYaDSIuDs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a05:6902:118c:b0:a06:538f:265f with SMTP id m12-20020a056902118c00b00a06538f265fmr8111185ybu.4.1677519432589; Mon, 27 Feb 2023 09:37:12 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:14 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-16-surenb@google.com> Subject: [PATCH v4 15/33] mm/khugepaged: write-lock VMA while collapsing a huge page From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Queue-Id: 84D1C18001A X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: j19jqmpajub3h6ui7bmdkxf6a3kugtj8 X-HE-Tag: 1677519433-463233 X-HE-Meta: U2FsdGVkX18qVt9DdrxK4cdL9p+R1CBwf4AulOTDVFWSgc6UCEe3Iuja9rNMPJQLmRJvHuCdycFn+cVxk5OQyK8lbaoInBne+5A4Hnzw71oFyBAP9zfswn20n+UtNNMV9ctU9FZeII/dJx/mQ5jLoD/zKVI34mMi5WBC+Tq5pIJBYGoMvkkRWiY04aBOAvxQrQw5gXcbT04d4XFHMG6ASJfoYupaCD+RKIM2vuBNAFVeJbTn5AIj4walpqrUUfazjSjNuqq0KbSbAo0unDxiB+jpSKxVY+eV04fAFUDKcWMETCnh6YZONzkpM7XkvWWC6emWYGmxIy3/qVGdsjN8rEkUVH9VHGQSvTGXb8fIOA5p47Roy4EtUqEFOI+tYu++oOCak8OObhdnKPp+1tZHm5zg9/fsH+ZonNwwjHPUcroJO6Nf9Fqa0/yy0URkc1HBABT65i3YXhcsGa6VReesEAoZ1JIzC0PJ4MgstleRBDMk/76HomJNv96BOa2gewXbZPWgF/D1SYq1JKPqPHj5Pzr+t20RyML9akqz2KtNkCRdSk7rq+UaxTJzWP5fK+e32DIsJpQMS71YyISvejixxAHTP9zwIwAScc9B6xLybBhGaUJ2dzXOnXCkONfM35KsDNvukGKHmKMSs5TjkzH9vYZJ1Zlzs1olSuaiZezXxLatcqQJmCQZoLkJoQegOhCaU5cAeuUA86BlPi+GtfE8wSkQBxMOEqOAL3HhYoCwexLDSS0vcyIfb5HXEccKw4L9dMsi0LKJF6ZHuiuXI2LuTSdVFs+fPxfyoOkunf9D1xDt2SMA/pBy4pA1biCoaGjc/Dk0uOVQNg4r1Lvf3ptS9FqeAgb7sQilMR6sOFYO6FR1YZMRTrSiIA83EDo8o6EKJGUkaqIW5n2APU6jlKlvb6gKA7vNk6HhlQT4+POTsL5WiZo5s/IpEsKYHNgqW6kpzB3E0ku8Cra7GDdasiY peq4yhdF BqEBZaYoUUXNPadj3cMYe8KY8NcmeRTOkAGDN1j5wtPXRotzDrlPCDH4C2epw6KVsE0DjpKUA4jWxZOrTpPjbvGKzdnqBRqEZ1Y5CQukbmQyCjZchXs6pAxPRKyC9D/7PevROYs0KrnwUrL2mnAmqcrwVZcPEYV6Ino2LW7URRt+k4HhMPqW0bevbuJS8GT5HaM6NG7YNWiBtTo8Q7b3LlNSm9Lulz0eP1VnUo9CvxB2B5Zfbn5U4EOLfJG+EL7bzvFG96CSAhZikHQ5Oz/FVgHvMj9xbfx8v8oYVNxt032HSQRJ2Y6TEH155/alQOCQNX1FUZ7jXu8WXqxwfwpbnplctECTBjXZky/g2i30eAP5P8CLWQBMgsq/RRZhSfqwhxDP+Qd0QWpwB3nc/CGQegZ3i5RXgiaVziCmftDJOu26OTZI0KhvAGmViFAxmyETycT/EP9zNXellYf21M8PDM1nJQJF1Ru//pJ3ZEWsWOOuzV9qen5Iy6WJ2oibWjzkfZGm9wpqPik2um+IMaPVLVOlc1NzzVAcGnSYAbi1gvfx4H4OMOb+Q7SU7q6KFPzR0VsqqayxqynKewmfIToYy8z7m3FHYcpZIEgwRee+dJMHEUw1eBOYa8PGeBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Protect VMA from concurrent page fault handler while collapsing a huge page. Page fault handler needs a stable PMD to use PTL and relies on per-VMA lock to prevent concurrent PMD changes. pmdp_collapse_flush(), set_huge_pmd() and collapse_and_free_pmd() can modify a PMD, which will not be detected by a page fault handler without proper locking. Before this patch, page tables can be walked under any one of the mmap_lock, the mapping lock, and the anon_vma lock; so when khugepaged unlinks and frees page tables, it must ensure that all of those either are locked or don't exist. This patch adds a fourth lock under which page tables can be traversed, and so khugepaged must also lock out that one. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 5 +++++ mm/rmap.c | 31 ++++++++++++++++--------------- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 941d1c7ea910..c64e01f03f27 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1147,6 +1147,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (result != SCAN_SUCCEED) goto out_up_write; + vma_start_write(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address, @@ -1614,6 +1615,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* Lock the vma before taking i_mmap and page table locks */ + vma_start_write(vma); + /* * We need to lock the mapping so that from here on, only GUP-fast and * hardware page walks can access the parts of the page tables that @@ -1819,6 +1823,7 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, result = SCAN_PTE_UFFD_WP; goto unlock_next; } + vma_start_write(vma); collapse_and_free_pmd(mm, vma, addr, pmd); if (!cc->is_khugepaged && is_target) result = set_huge_pmd(vma, addr, pmd, hpage); diff --git a/mm/rmap.c b/mm/rmap.c index 8632e02661ac..cfdaa56cad3e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -25,21 +25,22 @@ * mapping->invalidate_lock (in filemap_fault) * page->flags PG_locked (lock_page) * hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share, see hugetlbfs below) - * mapping->i_mmap_rwsem - * anon_vma->rwsem - * mm->page_table_lock or pte_lock - * swap_lock (in swap_duplicate, swap_info_get) - * mmlist_lock (in mmput, drain_mmlist and others) - * mapping->private_lock (in block_dirty_folio) - * folio_lock_memcg move_lock (in block_dirty_folio) - * i_pages lock (widely used) - * lruvec->lru_lock (in folio_lruvec_lock_irq) - * inode->i_lock (in set_page_dirty's __mark_inode_dirty) - * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) - * sb_lock (within inode_lock in fs/fs-writeback.c) - * i_pages lock (widely used, in set_page_dirty, - * in arch-dependent flush_dcache_mmap_lock, - * within bdi.wb->list_lock in __sync_single_inode) + * vma_start_write + * mapping->i_mmap_rwsem + * anon_vma->rwsem + * mm->page_table_lock or pte_lock + * swap_lock (in swap_duplicate, swap_info_get) + * mmlist_lock (in mmput, drain_mmlist and others) + * mapping->private_lock (in block_dirty_folio) + * folio_lock_memcg move_lock (in block_dirty_folio) + * i_pages lock (widely used) + * lruvec->lru_lock (in folio_lruvec_lock_irq) + * inode->i_lock (in set_page_dirty's __mark_inode_dirty) + * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) + * sb_lock (within inode_lock in fs/fs-writeback.c) + * i_pages lock (widely used, in set_page_dirty, + * in arch-dependent flush_dcache_mmap_lock, + * within bdi.wb->list_lock in __sync_single_inode) * * anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon) * ->tasklist_lock From patchwork Mon Feb 27 17:36:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153979 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1146C64ED6 for ; Mon, 27 Feb 2023 17:37:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DDF96B0095; Mon, 27 Feb 2023 12:37:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48D3C6B0096; Mon, 27 Feb 2023 12:37:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 37D336B0098; Mon, 27 Feb 2023 12:37:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 275DD6B0095 for ; Mon, 27 Feb 2023 12:37:18 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 024D6140B0C for ; Mon, 27 Feb 2023 17:37:17 +0000 (UTC) X-FDA: 80513778156.28.17CBDD0 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf01.hostedemail.com (Postfix) with ESMTP id DECA040024 for ; Mon, 27 Feb 2023 17:37:15 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=MCQ0c4ia; spf=pass (imf01.hostedemail.com: domain of 3Sur8YwYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Sur8YwYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519435; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h8ePWaoI68GEISPuC9Ge+4qXkO8FW2G00MOJpn9vfbw=; b=MLq3IRd8hQndVs68d4E+ElzHQcfNtrB5gEPCodP8rKogZfmxyPBzjzja4FgZLzjo8HTB7s wjrgk/7aIY4w0gVJnifmC+Nz3TSAAuv7zK1L7MxUE5tV+W6Q7iiLaRUqdC77Bu+YSVKK5o DrmVHuUPaubcuN+cftP/IEJ1EYo3j2k= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=MCQ0c4ia; spf=pass (imf01.hostedemail.com: domain of 3Sur8YwYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Sur8YwYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519435; a=rsa-sha256; cv=none; b=mdol6TwMm2memvJ/02m4wTgCgRGA0VNCRvCHoossGg3v61BCxizGh9oRG0PCb5V37IDAnm XQyQRVzgzuv2ErVz8bTvzdu0/ul/71I5MTo5z7kCFVrMql6sVuvaZYxpPkdDrdFoRVWmTQ m8+gquoM4yw9YgCCSpKUkMOhYfgXLBo= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536c8bcae3bso154382897b3.2 for ; Mon, 27 Feb 2023 09:37:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=h8ePWaoI68GEISPuC9Ge+4qXkO8FW2G00MOJpn9vfbw=; b=MCQ0c4ia6IOTYMEFJR7GcbSkx2aCS+J3n3kLqdT/eCneA5RnZIfOSsLVqqbhg8oV3C z6a/cjMHL2s70DKPWTui5sWqdtSoRO2Sa3X/ZDXnFAdUlSIwaWZO5o28PzA9YRicQEsL lrmb7xLSeO5Upy8PojNo03zIknEUlOaIGEk3GY2vDeeMswBskS1b7cfH1kyMJjq2FiTa EPOOWdtU+2jcY+efHi/NzqtHu6KIrndJCcMf9jiuRTBG5FPDsaXOxEi35OK/QcXUq7OH bWHqrLVOAQNuSwHdqjlfEqvb31BygBYoGvuY/kisE56Bi1ONNPXA+NnMTkKqBCgcNMjA jbEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=h8ePWaoI68GEISPuC9Ge+4qXkO8FW2G00MOJpn9vfbw=; b=AxaLS60Kd+LSMuPO07pQTIZqVOvcsziTo4rq9DbR/Z4VyeBnDrUNIVEHPuq5EavCVc O5HJYK1eLsAm8YGa0FYKC8/Pu4L48oYRLgziIjAhzBrQi5jqlGfZuvHc4CEO7PIW49mP jpYcqMIIzgzIl4y/IA2HHo68nfau3ymnL7gg5dLWLbbR81E/wJDT7j7BVffArPhc/F3j BNPMRh45YM2dnmdQKkJO1Y9ydBlMYlFQFM6qDsyHDofAQqgDIF0iJqJpzWbTxUnD84vV JUUmwgskjtqKOAVvZL/T3x1Tq/aPH6kMR5soFTJjBrMKRIbGLINfJGJGEfHV7w0A3i9G 5yKA== X-Gm-Message-State: AO0yUKU672NWN83XcmV4zyP7ee7sa8LYBBZjXVjDexhgG6YqdnCCX2v0 HFO1tWgla6LKAJndDZBuYsh0+XVCqFs= X-Google-Smtp-Source: AK7set9am1xybr1j78qWLG/uhzYsiBlXA8SkEZQ4RtcwObl9A6wj5gHEFg0gqcEhjrM4bOMvHErmcMX4UGI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a5b:84b:0:b0:a09:32fb:be73 with SMTP id v11-20020a5b084b000000b00a0932fbbe73mr9490924ybq.0.1677519434899; Mon, 27 Feb 2023 09:37:14 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:15 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-17-surenb@google.com> Subject: [PATCH v4 16/33] mm/mmap: write-lock VMAs in vma_prepare before modifying them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: q1sn358muewhi71jo3jiua3ft79ds7cd X-Rspamd-Queue-Id: DECA040024 X-HE-Tag: 1677519435-88976 X-HE-Meta: U2FsdGVkX1+wxxiNGgKk+ScGujvsQQDktmLZ3/D1MntSr07jzR2Y/hHH59Yn/eUou5tATrcwRAlGC5Jrx53OUazi/NZUayvJu3W3x0MDPgiVbjG7+/hWMT6fQtD28mhdSw38540cXg5o/otsKU+M47XyAngKkGdDvg/Tupu9LlJ+emKcgFk80S5D3SQdLB3BUPXvIn4Xiqwfuk1GuVYc+UH9tO9FfCs527hJsvbpbVK5fUC1WNgO3uSkNpT9MZUy0rY1jKVUtyjyrMohXCDCbdaHH/ubyyf8XljQgBlI7Wrjx/x0cjgpxnjFMTgHb+MxoJcFe9K85JnnK1AZcLprno1GBn81uizBQ3OvT2pRIXlFlV4iqo5PA0HZAEVTyIqYkNKYc8aBSz6asgq3SMlOjK83lszTCBNktjW75gslP/GB2Y+1+43zV0Hc3l+jUDlnbKxEOZx8WqcRgjBoHpaeAcVz+EJu8oZO5RzuyEXRQElKaU1+e9GKn36HOLQDevaYxF1vxM6XRzRmDODagH80TmHTiIQFeBG+QjbujimCnDlnyG3o3wtQ9xbZs9UgD/Qgcn5tq1IcNsRH+Faal8+XD44K8TZ7liPvTbG7YRrg7dB5NSkTZDT8xMqOxMRA0dmdxnMC6nST/wI7HU62l/m73YMVDoHgGUamfYuF1RkuR8nYL16qzm47LPPhplpnI1/APuJEvxRHFbSHo1A3p+QVyAq/m8QkvEIUb4Va145ATo8/MYxTScw6GmOIcaI1l/XgyxYAdJNnNzIQma4jsmRqvuH54sKOb8o+hSAhn5+mIoBNg3VAMFZNM4NdvV9lFN28gZt6RtPcFzCo9+n7DRU74PfCbpMzEQAQv5DcWSC1OWsQI6wY1S38KwD4Hd+jA0pxL7FUApZUSrkfQFx5ia/T/+/IQgmPFmWxlol0rsdAzQywZY9VVACkytc7fkOc7Cuss6p36ia6T2TFoo/nL9m ME06o+JM llshxKYJKLnjllbBVUSJoedTUxOVPChKifC0v20b6KKAV26yCWmtzD2eB4wb17qCyLlZTSwm5+LHFq/2NyED+zuRGz0JiJw5aYS1zZy/RNstAzjeXyKEr3v2eAygDDCN3yTLrICIprrBzhA0gXzR8R/HG4MBB5VW+n0L6igkM32qirN3fnfmX4oRyx6TYHs0nfit0tcd8WeiFIBp6ze7NU+23Hh1XVrgbPMyDySNTgxJPTJhT0esmW7Qw8MVVyHvWAsPY6HS4HVwjZDNIcSvNURHxUuRN1YvUbRQBU0HbZtq6ZHfKvjI355gVXnOPaS2Yg7aU6gTiapCyg7jWEZce2/ePBjZOYhznW74hCk6laZcQC6qgRWILI42IVQTlSJtlZCk6K2/bwT40nySKLttX/PQGlfDstvOwG6SLcMZvr5OlAqMzg4s2fw4TkozKREtHUdAzmrssgkRoAJbu5ex7MG4wk/2d8k2pNcU03Bwr3aEYbbuyh5Wz48WGwtd0YzqBorgMoV3bn8nIB6I5+/jcA2i7jLBpX/zA2dYQxpZQWRpQRBowPTa51vTw5t2V8ZNDLUIA8xNCBjiu1RtJZ4KCLrkCFiIrf+zmiY+G1ViCLIBWpu/qivOHs8Bfk/bT3GagUIelsoj2r0nQ8hXdk1KKnKNKbkUdEffma9Th+oSDwx2E61o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Write-lock all VMAs which might be affected by a merge, split, expand or shrink operations. All these operations use vma_prepare() before making the modifications, therefore it provides a centralized place to perform VMA locking. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 92893d86c0af..e73fbb84ce12 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -502,6 +502,16 @@ static inline void init_vma_prep(struct vma_prepare *vp, */ static inline void vma_prepare(struct vma_prepare *vp) { + if (vp->vma) + vma_start_write(vp->vma); + if (vp->adj_next) + vma_start_write(vp->adj_next); + /* vp->insert is always a newly created VMA, no need for locking */ + if (vp->remove) + vma_start_write(vp->remove); + if (vp->remove2) + vma_start_write(vp->remove2); + if (vp->file) { uprobe_munmap(vp->vma, vp->vma->vm_start, vp->vma->vm_end); From patchwork Mon Feb 27 17:36:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153980 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF585C64ED8 for ; Mon, 27 Feb 2023 17:37:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8AE8E6B0096; Mon, 27 Feb 2023 12:37:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 85D726B0098; Mon, 27 Feb 2023 12:37:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6FE976B0099; Mon, 27 Feb 2023 12:37:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 614236B0096 for ; Mon, 27 Feb 2023 12:37:20 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3CFB4809ED for ; Mon, 27 Feb 2023 17:37:20 +0000 (UTC) X-FDA: 80513778240.04.79415C5 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 7461D18001A for ; Mon, 27 Feb 2023 17:37:18 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=QlvGB3sb; spf=pass (imf16.hostedemail.com: domain of 3Ter8YwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Ter8YwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519438; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VWwKpXjDg6WAvmp/os9+Adpl/ccsxZtoSX3Q5XOWyQ0=; b=asuCFyvZSgAapwIlVrefX7pYWa4k2ZMV8L+A4vtiZlU5BzUr1MnuOQSzc6KS1dujxytcRs LUdRKoKkWDQ3HOrahxBavut3fIfNcySltijlnIZNr3TdaZKm0/vyYn/ZihfHt52GL3a2sh Fha4nZFLQUiK1+Q5THU8kTixIPzndNY= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=QlvGB3sb; spf=pass (imf16.hostedemail.com: domain of 3Ter8YwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Ter8YwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519438; a=rsa-sha256; cv=none; b=xaizBthyXy5PmIGBRm858o5qaaumSlzHQHC7RVXVBYYc5pDQPaVlVdUWJtsQQn7vJ0mogu Mhx/zorSA5/zgCI7QPZVS80R5dhjvVNosN5g2EkokWFMbs5gpIBiVjYeBpgrHLACWHNf8T P22cUikDkeYqPamZFqBqMMEBxdhm7kI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-536cad819c7so153268587b3.6 for ; Mon, 27 Feb 2023 09:37:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VWwKpXjDg6WAvmp/os9+Adpl/ccsxZtoSX3Q5XOWyQ0=; b=QlvGB3sbxoYWlBS31TzcGu38FuNf7Lhjhtq2VlYCxWOSVfHgnAjaRBrb1w97aEacbe K6advecLicCSUfFEo9GDjw6ni0z0vYUO24YPFOrOT6PGrUxiFDeAMZRvmYa7tXn8nDcV jQQpuW50daFJ8WQmqyVE7pXe4jO0h72Ly1mymbXPSaJk3hcoxxEHUhmsYcU7Utdafwgh EoU7EkscyBq5l+R+tPrIdVU+apzyZHKYcAM5J8frjMBd1EpUZg/bzw2FKzGiNCJc/yPF OX60CUqczCC42EwMz0GH4Ohwd96NL51ADp4JavDsEQkoG6zGf4C60SM+ihxfLXuX3u7Q RNJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VWwKpXjDg6WAvmp/os9+Adpl/ccsxZtoSX3Q5XOWyQ0=; b=c+aFavWTjNbIlwXlW9bm1fqV+V3f1/5Rd579OHmFSyLHz4HbItwoEofp4cNN3WgGwE m81hBOqssJVAtrEhG5YPYrjbXP2DbsfaWEM/kIQ1k3Sg0rrDJdiRWxw6j6RxFkdGeSqF tav9UdTBw/bZQBTiOqAUN2/Xd7PNTEhzD4QyNEo364mWWjK7Zo8Tr8B7oP6IuggkcnCZ JqwgAnrGRj/O6MP+FP9ogMEm2MmRjPefZpwIyloBBJ1UFVLQlGQpXYm5GF4rf9qSOLdj K+XyuCcKdeD4lFcGIc2NDYEESW7qkWEXLu4NYiRgcMq/GfBfe+UxQ3uPOcCMf/eJVxC7 f2fQ== X-Gm-Message-State: AO0yUKX1f67PvFoAtCw8X40i/UIY1K3MBjtqlX/Rq/d82I1lXykk71Cx DyyAV6HUFCzSLuO0h+fn6RNKEa0c48E= X-Google-Smtp-Source: AK7set/Ab7tdetucxhQI7Y0d4CepnVZrRoikg6zNi8DSW4Q8ICogb5HvJ+8Ost/w6wLIBwUIyCkVrkJznGc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a05:6902:10c:b0:a9f:3ee4:51a0 with SMTP id o12-20020a056902010c00b00a9f3ee451a0mr10ybh.505.1677519437283; Mon, 27 Feb 2023 09:37:17 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:16 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-18-surenb@google.com> Subject: [PATCH v4 17/33] mm/mremap: write-lock VMA while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Laurent Dufour X-Stat-Signature: mq7w9j3mk6ifeiazpjdhy4t89aaqhafd X-Rspam-User: X-Rspamd-Queue-Id: 7461D18001A X-Rspamd-Server: rspam06 X-HE-Tag: 1677519438-710185 X-HE-Meta: U2FsdGVkX1+v2TsrZCZPa4W3wajoQINmE/ghPqVfNEuXafwt9SaKOyyHfxOnPQKC/DyfdVxwowL8wtvgH1z0OgGSi5Ley4S+XJVJpIA+dO7+ZXdqZZBM75qAzVhNSg8M8fn853eHNW65ulWWNs0sQmmE1Gej6ww1hblA04A4tWWViVcbBHxoH8m/gICN2Lk7GY2U9PE1rHhXl0NR3906DynROdJ3oSlegBDocmR2TPY8Ss89GhoU+nW4rm8YS/71iNwWG/GOo+Qp6dswTjCgn58g8g4gh6tt12edszZOtwTpAYA+Q8D7qLz89Ru2rudlMekvg5018vMbMy9MJOQosLjT4Y5watOwqnnlEmbJd16utO357w/1BoASLdu/1pxLHxAbPrdDPBF9wd9yd+N4jlersEIW9qvh0V9HGc/eY0Wq88xBOwE/18mmfNENbYH/94VEBQWWui45Rqa91iA567Pr0jLJzoDRFbKLU76p1LnGZerd1LDu5OT79Qde9EkzzYgMT1CqQmRRZPMQcxwjOhVWQd0lFpTvtVGvPJKi89SKI2PivWj5nvcWPlM0fwV+NNOInrKdYTFs/yXTt0FAEopndayMfJhD4r2+kUmPHwEPMjYZx/mKTf3PqdyrXtXeWks6DdAs/c6uO4CKvEyiQmIv2WQazH/jgJjM+7jn40oc1KEXBcLy5GJ531BRf1C4HdSYS/mYS714ACiE5Q1SVn/5BLlabwPk/MOfJJ9c32NAM2ULLKFyHejLNLYoaUI4XDr0uw+KUGltfRgjVltsCMm17FNp678aTWR4ORokzZbTd44RPW7OY1+XolKE510D3ugA4pw1UDOZBde661HpctZOc93A9wiwlse6oUy31MC8riHX5Sh7qVVRPd132XQoj5UPZyxH+BTI8fF2i9TJLgIgVwtNQM2p1NLWOtcgvEHCQITwi1BfcO5AUdRKs2pnbTG8e/2/zuORhxOmhek wGBhnHzc c9rTdoBP1Ne9N6R4qR2VaCbrXfMKf4Dasy/Mx8M1nk2w/jKjpHmEc37OVth/CVjW8m1lYh5CgePYp2VkilXsgbW3oqgcVbZKwLbV2Oqd7ZN5wmjID5xDKgOtJXMzcrelnhNikLxU/t/xLmLayjjy7OmRG5q0xPcLHHFJsj48XkjEkIngG75YWWesutmHdBnEIYVkX9REAyCQBxYqeB5t6d5MmEtKR8qs4yjXr7BNCULxPh5G0QkZnncxqQ0OcqmcKtkHyxJ4N4VugaReDd/m4AasEyHm3Km0YQw0SYgmgZXshFh/hWIY4JNHdJgtX/CXrku8KQ66kGsPZ/Ozj8S66yQzlYfLnNfTQU3P2wfFn9nc3R6escXnHdAe8rVHYGarRWMH5d6tVqVVCMA/XhVAjxzFWtoqCQ8jgN84JQ9W9aD+RPLPrSXVvs9byf0p5ZGukPfpOrzf8NWZoExBHpdUw/vQ2Fgvo2iqrSm7AsFEDSviTB51biPAkPd9Q8yxDy1wAwAFjN9yXbMe/JZFnTVxj+7j+UzcNGDDHrUy19fihb4vdM+savcyqTPmO+gJKNI1SwG6xkanOD6/hKiQjizRPXrr4ms0oS0KRRPrsVdsuDZ6qwi91h+r9ZwaXurxgFqz4oGz86BjvBa+yc3b4CY+8APDpmA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Write-lock VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index e73fbb84ce12..1f42b9a52b9b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3189,6 +3189,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_start_write(new_vma); if (vma_link(mm, new_vma)) goto out_vma_link; *need_rmap_locks = false; diff --git a/mm/mremap.c b/mm/mremap.c index 1ddf7beb62e9..327c38eb132e 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -623,6 +623,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_start_write(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Mon Feb 27 17:36:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C88F9C7EE23 for ; Mon, 27 Feb 2023 17:37:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DDF0280002; Mon, 27 Feb 2023 12:37:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48DD5280001; Mon, 27 Feb 2023 12:37:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3091A280002; Mon, 27 Feb 2023 12:37:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1E647280001 for ; Mon, 27 Feb 2023 12:37:23 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E0182140B09 for ; Mon, 27 Feb 2023 17:37:22 +0000 (UTC) X-FDA: 80513778324.03.0E3E2C3 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 0FDFA40016 for ; Mon, 27 Feb 2023 17:37:20 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=J6HTSW3i; spf=pass (imf01.hostedemail.com: domain of 3UOr8YwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3UOr8YwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519441; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r0m7Mx7yExEfpE40BLjkT9H7Bv79s8byLhYJ0wRxTC4=; b=CDAvUI09PT66jQoAqU8SAlG21YoWsJIa8V6+nCgJrU9tzNIKCDPWqBvArF8OCuFeN6jfv5 W0ZZ7ZYaitl4RxPuu2w+sknvX2GOPbAnCpIhxzsS8Z1d6YA5O99RiVwN4NNy2ifSQ/Zosd vJSz69515L2tHs+p/+ySNcMzL3FVLso= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=J6HTSW3i; spf=pass (imf01.hostedemail.com: domain of 3UOr8YwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3UOr8YwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519441; a=rsa-sha256; cv=none; b=4+uR3PC3MVxqxdG5RUTUOmR1muxc1mJXEVscOqsrKaMtw+m+GBUTo1FauLRLfSwVhfxAZH moQ87FuWNGg/gi7ABJal0Faa7+w61uGHoHh6g27rhLMnDZB5nfTs32zasAD1UaRRuUGaMC hB1MAmhB/SKXWbUkF/BeGxts0BCwXfo= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-536af109f9aso154254607b3.13 for ; Mon, 27 Feb 2023 09:37:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=r0m7Mx7yExEfpE40BLjkT9H7Bv79s8byLhYJ0wRxTC4=; b=J6HTSW3ije3IDV45f4imLJua2F9c9fG/LezyaDIF0k2VV4fuwgH+fku2QXSjWP2auL d+6+O+xNSm6Xj7FlWbcn2IucG0PD3/gJW2mH/MURmde3Q+9WMeczT04Csr++aQNQAB4v uOclz5rts/w0oMZxgj6JdiddKEJJwswsgJAAZvi48OAzGBi1BuSBhTNF8+d2d3+YbCDm fwaoPEsSaG9C8KiMPsrgpeVhqU2cm1pHUxTYDzrWvN6667+vmOEZ293x3rsB2i9qTzir LhGya0SRahx81MeDy4vFauftQ8Z+i+VjBO+uhvPnINIKKKPeRWHZkWIWZS9shPaay/sM /HGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=r0m7Mx7yExEfpE40BLjkT9H7Bv79s8byLhYJ0wRxTC4=; b=nQ9xuy58ezQWkkagrKakuhZbe9ywdDdfaXIbmFP9hNOuwMWNwFAlfHNuPf6GpKXB49 r+oBkte482wvxrzQlk/KUbhfAoVs62117sSZA1NJD7AJFa8BRrmMH0NIyO7Oa3yiOCyq Dh4vHDU9wX8RC2GOlvfTG1yN7/IVBmdXPvfEsOtJTWmTScE9ucd7qx+4jje8UZuXnG6D dAnnUhuYEXFWVZbB3aNxphBt38IMsTB9xStY/Mmf88eT95a5Yt58zdiPhXSQYMrS+yKp 2ZzB08N7tKuCHfZW7Y5Y0t2w73hKY6+3c2k/e1PrG+NNIVxATITTqmZ2O5obwwmyBMGx nJzQ== X-Gm-Message-State: AO0yUKWP/7Oy0OJWwPDYqV07DJeynWd+Z/ThhmWe4yaaQnxHYGczzI6k DpHD9fm9nNusgWiOipOy5qZNxt6Q4IU= X-Google-Smtp-Source: AK7set8EV36Dr/ESiL80fu1d1SaW9bmdiDECZfWEbGGMOjKuqTD7GEuVWa5jynsNqQ2pFS6J79ENwuY0pY0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a25:938e:0:b0:a60:c167:c056 with SMTP id a14-20020a25938e000000b00a60c167c056mr4911453ybm.9.1677519440118; Mon, 27 Feb 2023 09:37:20 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:17 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-19-surenb@google.com> Subject: [PATCH v4 18/33] mm: write-lock VMAs before removing them from VMA tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0FDFA40016 X-Stat-Signature: hmjtbda9fx1e4kzfufa578setopgbrhz X-HE-Tag: 1677519440-693381 X-HE-Meta: U2FsdGVkX181jhuSJahHvGvscz9PUmGHmYkzgKMhPnIjtUHEQ4dS16unwWqaFgcaJj6/hLd3/6UlwELYLfj8/KYrbSJ8H6sR3PN/RRiPdM7KaFzVPnX/6/zGTgqFVvOOznc8YQdcWclAtlUpx/xSiGlu0mJyYu5t8ePQBZ1XXI3yVh6QI94G3vrVaVLQDJLeU2IGZFhSIxb/KO8FO/jgQBIk0xFZyOw2OS3rk7Az9VLWnG7BOkr8kQxYs9IhB1VZ2OnfRGMASlKzvXKu3oYPTnH6hbFp0xUvEfG8//k64Dhc9e22gxGO9Rt+AvsiFDEGGyTtMBQAarNfmKB8DJ20a+nkw8EKbP0wpISIq6wdy+ruw+7w/KRzV7fGyVFjq82LeKx8ybnnT+pqL4N5KIFJCVh4WeTqqJMRMxCqDcA5c9G0HxyIzWdQVWtIFiCIUZCNNyGm71dJXHY31foEsa7P9Q/zCDD5fdkK0D3YICKgi7vPejw2b+pr0+ByAeQzOAsmjkIZz+bQFIqo/Jdl1WcUdN4z6XwjgReYAU2sOfKvJeIsxeRP4g79jiQ8IOMIkuhWgyt8/f1TMvcs60DY2cTggm/iLA6YeVlTjiy0GAwlWeBqJIVQVmYv5VBUgH9/gt7szQZKUFX2MrEb6VVIxM0Eyy0ud0620PP4pFphFwtWa///X9VgyObrEExcOX0V47ekNwbofYxkR1Ge3tk5VJ6YyPvNhp8W7LsfK3q4BzxW35Yn2lWZpN7NmTBRGCvud7Zmhqhi3EDSV9Bu7rpAo41CPnLurUJwx1iaYnORkvkj2KfXtyEopTf1TQbMp9WJS/NToeBm60aW79JQ9D9OYyf8+zYIpVAV1JRdBXKbWAe/ihvFNnaO6OIVNgSjN8+MbdyFCCOsueK7V7flfD0tMF7euFX22H2AFLACzCvCP70NJGAto+CEb6E8J0gFUldni1UvCcMibFO+CBzthhMmOm1 bPcfUdap /wkyQRnkxDXvmwLE+hGLzWEL3H41IhjlMGgbMiGX0rFqkJVkP0fkleQS2QW952mvYRV09nuISu7YW7W5sdJ1Jei35UkjAIwhvl2vX3WLi3jQgyNpFe5yl/4cpeQceclzVZ37/5LGOxAn1XGYVXZ2YX4obDBjzjC1/kvxUF8blk/btxXQxnT9Rlis+QEgZArBm8csulqV1lLzavHFzf7hcTAoRIDTKQuPJI8LdDDYGHgLAjmNLXG5WTnW2KO1NBgcaZ6utXddFEarmzOROK6Owaqm50IvjeIfd+QSjOotZluLagao0yFZlQH8O0ype7/EHXCw5Ji9pJggrTJeGEDfNGIUJI12uJYEhHnBo76PB3U55t+2t8pU5nBx0nr67dlqIO7mrCGBF4pgDoGZZ/1PKUSlUwz1n2hfGLS3TNtEgKjjlJfIPYlX7t18LE1G+c7G0LU+4WWJXRWMZbcawcVkTBNM0EtnKQMQCoW3moa9bfuPEnhnyuGDvV+RDPYpyk3woDxjiBrV541I3fh9OwIb377ub8R85D8fW0XpI+qFILuBpFxGnSR17etN6rpfszJB36h8nJLVLCW8qXna4N2xYafVHnVBXbGQg3Q95ULwMDk1OzEGmNF8R01rlAA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Write-locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 1 + mm/nommu.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 1f42b9a52b9b..f7ed357056c4 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2255,6 +2255,7 @@ int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, static inline int munmap_sidetree(struct vm_area_struct *vma, struct ma_state *mas_detach) { + vma_start_write(vma); mas_set_range(mas_detach, vma->vm_start, vma->vm_end - 1); if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; diff --git a/mm/nommu.c b/mm/nommu.c index 57ba243c6a37..2ab162d773e2 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -588,6 +588,7 @@ static int delete_vma_from_mm(struct vm_area_struct *vma) current->pid); return -ENOMEM; } + vma_start_write(vma); cleanup_vma_from_mm(vma); /* remove from the MM's tree and list */ @@ -1519,6 +1520,10 @@ void exit_mmap(struct mm_struct *mm) */ mmap_write_lock(mm); for_each_vma(vmi, vma) { + /* + * No need to lock VMA because this is the only mm user and no + * page fault handled can race with it. + */ cleanup_vma_from_mm(vma); delete_vma(mm, vma); cond_resched(); From patchwork Mon Feb 27 17:36:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153982 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F33F8C64ED8 for ; Mon, 27 Feb 2023 17:37:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8FB8E280003; Mon, 27 Feb 2023 12:37:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8ACE3280001; Mon, 27 Feb 2023 12:37:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72497280003; Mon, 27 Feb 2023 12:37:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5CF93280001 for ; Mon, 27 Feb 2023 12:37:25 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D9210A0ABD for ; Mon, 27 Feb 2023 17:37:24 +0000 (UTC) X-FDA: 80513778408.15.103A128 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf19.hostedemail.com (Postfix) with ESMTP id 1333B1A001D for ; Mon, 27 Feb 2023 17:37:22 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=bS7cuWST; spf=pass (imf19.hostedemail.com: domain of 3Uur8YwYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Uur8YwYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519443; a=rsa-sha256; cv=none; b=sTrL25nQOnAzWmknSRZU9MUy/mUKY2o5y0Ax9xfoJMskWKzxQ1kRb4rXgsAICLE9T5Vyr9 fCivKl04Xf7+6BiFFYMf5pscX3F2Nz337+fCcII7b4UT41rbySJmw2cTWg6oUDl1W5sYOC 8iCtDO6UqvacVwpFQg6fa+5S07+ONH0= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=bS7cuWST; spf=pass (imf19.hostedemail.com: domain of 3Uur8YwYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Uur8YwYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519443; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q4IoNs8/2s63po158n4pCM20IldvB6LsORlyoWZYhDA=; b=q9LKX+bSF+y0eipIz0Q/UFYVTgCPSmSkhmLoLyW55UbWkCTVhDJ85Qusv/1Vami5aKFiPv IGDyPibnrzTbHvzVMS3jHTpp7I9DI7AOXF23ay7pCs51oeeSdnpcou4yIeBjCRdwP/qoMY XOn6Vg8lVfZbRGkGz6OfsePCik6LDis= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536bbaa701aso152729547b3.3 for ; Mon, 27 Feb 2023 09:37:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=q4IoNs8/2s63po158n4pCM20IldvB6LsORlyoWZYhDA=; b=bS7cuWSTvikNO5/1/KRQJLPoIX97MSDopLQjzIYhd9M4ATiilp+2nnzhJYwm+xkfAw CA4PkRjLFErdI6iJpWSrCW/0zzxhXa3aDXwe+0SrD6SM1VCA8VjhRHuY2QcHPeKNakQW P+DawdVxSLtrg/I5seo5dukO8m39Wj2s4hqiK8xve+bjTmkIkwxAaDpb3/Zo6UgfUlrQ EPB7WXnu31e6AaMIPOJ6t49a2aYPUJ8xG3eJeremwZc8bnMi2k5b+Q5ZfPziQ2FlHzzp w2I2hG7tsvd97Sj45eHKWDA+0nQshMfUHld0I+kGvTMYTZlSoT7VYdn14KEIlMTUGdtE e10A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=q4IoNs8/2s63po158n4pCM20IldvB6LsORlyoWZYhDA=; b=d+DQcxHPGj974esEPFS7dbuIs49mzv3PKpB6WFECXY5HQjBLg5b+rUkJYZ0HjYCFtt 7KF6fJHIjEp9Q9rRULyMsZ/bc2tAgOjfaDHGTin1q4nmMg0ywmdJBKX7SmZFfJd73bah eslqOx6U/IW5rUPosuZxV4Ib00xhQmLbG0G9X8HLpjpC0jw7nZyg+XV3EEN43lC2ifH/ MrqWnLwfBfX0qlRKz6w0dyNdVsXMnpA/Cn2udazc28/nWSgyuf3pA9wjV+vZZyh2DwfI 0N4hhL/NjT3ZfnVWYcn8m0labaQ+NOBGwT4JcjJ6PqzeGW3S+WQL4cJIZJOcLl7FJH4X VB/g== X-Gm-Message-State: AO0yUKWHcqFkzCuhQfnHQMmWgpDGdrVdj0I9Rw8ATfhgupfVUmjQFZoX D9kaBY3R8WeHeBmNH+1slWUTiYm4Xwc= X-Google-Smtp-Source: AK7set/PpfdmNkoTcymoja/27Ir9z6PS18iSX+0Y7AEWmCWJmOIOQsn+Wv7zMECojEbGHKP0cPXfu+hFpqg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a05:6902:1388:b0:855:fdcb:4467 with SMTP id x8-20020a056902138800b00855fdcb4467mr287176ybu.0.1677519442170; Mon, 27 Feb 2023 09:37:22 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:18 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-20-surenb@google.com> Subject: [PATCH v4 19/33] mm: conditionally write-lock VMA in free_pgtables From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Queue-Id: 1333B1A001D X-Rspamd-Server: rspam01 X-Stat-Signature: yb865zhget3ckzhj8znz69jbft43z8yh X-HE-Tag: 1677519442-704983 X-HE-Meta: U2FsdGVkX1/rFRFPuhB9hzJrnXF8p0c7abMcuN0IgdSWYV9ZUuiNbC4ttD8avccf58+NK4Sn9SzeWL5IzFPIwyuaaupytp1CcjjZQDjIDe6nfIZmmj9k6VGORXn0+po/bhwHwM75cPNyARS044mirFkUQdM0QD7yzET4rOG6x3RMm6ha6xKUvvk3S9kwKk9htm2J20tZdpwu/DUDalZX0rLjJvLCKXbkO41FpCf1Reutnes+hXhAkyEgjlOeAHfLvOqrL5F8AZYTn0SLEFdvdld/d5n1mBPzHdDv4Alo0VjXwfbi2aX7Ixj+7yrbccnKHfXPvNTVkQmIwIiKILFWYu6xwTkzNeNpOoEJBfx4+9+3Oz5CtFn7n9Q93jcd++SFt92twLTgsGLH4+/810LPcqG8e1COey2h2zHmSH87Hlqu4udMhYGhrQ/q9fnoco1v9kEuIix/FQHbEAsGYwkqF2vJGmDnAALGd/+wlTQRPoaocpF2y9jNCYFN+BKsEemlKPqmJyiaZ4D7MYy3DSB71T5APgJFhqZ0oQbt8PZBAO/Jta+HbL51m7WdFYzBWQhJIHVqk5jmsct+lm1JdfvdpRAn1BB3A1gndCa14uqR0pMplowhHADSZ92ZH35E8eqwECICF0HWk/45zbWsvUBfdLXlyZUvkulVu/9VU7ZtxivjD+b3G4/lP7HQGDxGYni4y+bKgq3BHzHZMfdf/4mPIW5jVAqi5r/BUaDBGRdAXJ3t9DWvUCIKbey9wWBgkZW1iChy1IWEX4oIvO/ZaIFv/XqyrxQYFLyLd5qfZ3dPvHMnGuu37A3nTcqtOarERwO2jHgPca/YJPd5eRDEV+Zmowe1oB3iJlpoxMqa8UVqa2vC/72zoEYvxPgrHPB2DeDqKz000g+ZBvYfU17xqo0z7SiseWyqmj309Sr0Z4QowD5RRZooONItAgKjy9FSUAmxqNvzC7ugqHhfzEAa3mC fxqRdOBP GdgKd63TCovbP/jXCM177SOOV/Sg6EvpEwjYnQ4lhVPI7sZ7JeWEy0ZaW9Dbr0b5bqyMuGyYcXsQhppzbgTebORkYfzc6O2L42JRuk19ZPW1+x3scA4VyRWHv0O4TIee7KAgKl7NuywvDL48AJ/gl9mbpXg5CIjisC97Obk/beWM/c0goj4rtNr0KTVnqeXPXAfAkO04tHZSF+Tlv7DFSQkIYPHgK7Vf1BceoGsgXggWr1Jlw2M2Eg6I9mgFCOMl6q4o3LpD4+VvZqD3hosy7u4EqnGy1hqE2uxBmGz1UYK4mw8Mg86pUqZxJFAeItFs+zSC4jyiYLRuZ/41LoPvgX0MBDduefPVfFc2x1iE2ReG6I5ztb8wsdl7vs/n/cCjDiwDYSJHcM9gVks/0KeLomTks6wZ+1/rs/clO85wlOo9JmZo8OF+QlEEuEhUsJjNxZvd/YOozxHyS7g2OoDMnt/FYUWZVRXUuqZ47cWRWqrpV7NRoOGjsk1TM6wz50NVyL1S+HoR7kqj8Q7sSAPMUxca/9hw8UrwrX7rvbIBTpizxmE2DGD2jiOAXqGwKvPnzR8Yc9gx11QuSlaHAups6gHapug6icgHj3+bmHmN5ygfJichC7Ex/3kKzujXRVtq6bYYkSYDmTGd9lN+tgkNoU8F1GEt3FBbIYynXna0TmuHTSoY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Normally free_pgtables needs to lock affected VMAs except for the case when VMAs were isolated under VMA write-lock. munmap() does just that, isolating while holding appropriate locks and then downgrading mmap_lock and dropping per-VMA locks before freeing page tables. Add a parameter to free_pgtables for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +++++- mm/mmap.c | 5 +++-- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 08ce56dbb1d9..fce94775819c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -105,7 +105,7 @@ void folio_activate(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *start_vma, unsigned long floor, - unsigned long ceiling); + unsigned long ceiling, bool mm_wr_locked); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; diff --git a/mm/memory.c b/mm/memory.c index bfa3100ec5a3..f7f412833e42 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -348,7 +348,7 @@ void free_pgd_range(struct mmu_gather *tlb, void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *vma, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, bool mm_wr_locked) { MA_STATE(mas, mt, vma->vm_end, vma->vm_end); @@ -366,6 +366,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -380,6 +382,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, && !is_vm_hugetlb_page(next)) { vma = next; next = mas_find(&mas, ceiling - 1); + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } diff --git a/mm/mmap.c b/mm/mmap.c index f7ed357056c4..ec745586785c 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2152,7 +2152,8 @@ static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, update_hiwater_rss(mm); unmap_vmas(&tlb, mt, vma, start, end, mm_wr_locked); free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + mm_wr_locked); tlb_finish_mmu(&tlb); } @@ -3056,7 +3057,7 @@ void exit_mmap(struct mm_struct *mm) mmap_write_lock(mm); mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, - USER_PGTABLES_CEILING); + USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* From patchwork Mon Feb 27 17:36:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D07C6C7EE2E for ; Mon, 27 Feb 2023 17:37:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F128280004; Mon, 27 Feb 2023 12:37:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 59DDB280001; Mon, 27 Feb 2023 12:37:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 418A4280004; Mon, 27 Feb 2023 12:37:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 30416280001 for ; Mon, 27 Feb 2023 12:37:27 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E5AE21C36A9 for ; Mon, 27 Feb 2023 17:37:26 +0000 (UTC) X-FDA: 80513778492.24.5F37C94 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 2070918001C for ; Mon, 27 Feb 2023 17:37:24 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kE2fXrc4; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3VOr8YwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3VOr8YwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519445; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q7njm8z68LI6RSv1P8+cvUE7J2LNjoWsKIE5FW5UVRU=; b=e4qnCnXQx2tclCaXa1BHewujPOXPPUI9YtyuMxee6E9S4tID74tqioGnuSjpkzsrsduVlL wHtedbXZyYTTb3yB3VF2hNwFNBir6KOGHl4qcEIhQG3JRGtzmLdiKyIa6XH+2R1ZU8sGLd W9Q6q4FyHC1oRQ7y4yTD4B/d9uZTipw= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kE2fXrc4; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3VOr8YwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3VOr8YwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519445; a=rsa-sha256; cv=none; b=6l8uyCGTv18esC0J9wzsskakDpVwHZ7EvtTJXwZp3VZjn9e/jTX0eu14izYcKF/MmRePqR YTnFWxojj9HhXpZVTnb8kVkBacmLX8J6WExSMxPjI1YsATvD/y1iu+wBPXrXAdMN/dlGe6 ua/w4XYEUo4iQ5a4L4/zLC4FVJVokPA= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536be78056eso152400017b3.1 for ; Mon, 27 Feb 2023 09:37:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Q7njm8z68LI6RSv1P8+cvUE7J2LNjoWsKIE5FW5UVRU=; b=kE2fXrc4CKxmWF4clr80N741Ci0xoX5G8lQJgmu9s0G2B1Dy6UfiGdmy87bkmQVz06 s0jNeoaCkSHSjTyAuaLFgp289qeTagMwone0ldYih81KbxqFw8/SoaDnJa0SNDiiUGH5 C2cHTDPFHiMhA/p+opto/zgvEpmwF0hSnXtUorxCz5wzOWx7pG82xGb9d1FeLnZgcBn6 7W1alMlgUF/qdYBONkVV4Ipuqjxo8jbsNMGdOFgbr+6bIzoXMj9Q7PsKDQfKRgxbc6VN kYkd8hTRqF7X5fVDHYhOgBPe6TXfun1LU5P2SsjoAl6axSlRSprAw6j/yFhpk0MslB1+ 5m8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Q7njm8z68LI6RSv1P8+cvUE7J2LNjoWsKIE5FW5UVRU=; b=Ky8iqhpf+UIA7LPXtg7Pz2L70ZwVe2iq+cJHZwW5s1nkPjkYbgVfF20JmRePErl/wg IHhxrK1khTbM4QpEdzYOpvj0UOwczgo0e2rE70KPYCSsyt74r0tyuisu9pgHNI2y4Mzc M4xMU3J9RJ1zrmm1PGBdgNf9HXkAXncg69qYTCA2G7NDwUfl7Xqcs9kZ/Cjt6kXWKmKz JcI9A3NNIziaR9lSDMuQs85bBlWuu1sLJkAu6atnexmj5+/fBrTi9vbdRSH/09PRuucs l0gaZhq06YiZOf8rVA1j1sXxNMtx1Yfx2SEMG05Zb5eXlx48NpmUrV9KZLFLGtn3WWOm fyfQ== X-Gm-Message-State: AO0yUKWkSgI+bBPIX02/DwyFIMvrsGKZOuBIwaPyyRwgA2fzNvdXLP12 Lv+he9SmPlQ+TwBOAVKp9xFaAE7x0yI= X-Google-Smtp-Source: AK7set8BLaXPA95+t3odvktVZcgw7j4RuC8orgkmVPEnGvQUgHaO3AYXobp8iNv8pTyoR4t4ofqJKVALLsQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a05:6902:4d4:b0:91d:d6ed:6bd2 with SMTP id v20-20020a05690204d400b0091dd6ed6bd2mr10048648ybs.1.1677519444556; Mon, 27 Feb 2023 09:37:24 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:19 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-21-surenb@google.com> Subject: [PATCH v4 20/33] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Queue-Id: 2070918001C X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: iocsro8j1e9jp79jp9xt49rpudz589jn X-HE-Tag: 1677519444-915799 X-HE-Meta: U2FsdGVkX1/XlnzdO1pACuukWjvcKvYB4s5RJkb+x76MX9WBnX/zK9cnrF+WVwqJSrFl0eDbT2BQWeBFqbagOcpjEGWnQzEN5qTCG+5hUGmzj79E0FvNaWg1jQb8qIee8CEh8BPdesLLlunltwBFsBIvShm8V+wkv0v/iRP2ELftRF31rTq25Squ+3YORtbWNMdma0m/Dcz9/1mm1gKe4uHp2CPWuF1Gj+qaLx9Gcxp4CM2dNhzNaiiqMbqxRnpBSlNu4iGuuMYg/N4XiSAszECJOAMcx7GSC9hBTo5mubuvwV4Oqsq/zKSexmbnbnjxAltP9PvU9Nm6320sauRUrkvYRTw3c2dTzyeL9pPeQ7OXyItlejMlzupFKCBiEA0yqRSIXGiCqXckNpsQAY+9uIxluRqLY9acziD4bXyNpXH1DDyu25gngOuhQcejjjysgqmKlLH2/bNYwK5NiP7vdadzg0EPgz3VjlpKF+4ngUN+LrKzzbEtQEPoD9tsi/qWFmYbLnG/TmG4iF1Ol1vM5xWtzUhg2oEX0xckQBrlt14kUBvrLjULXg1SnqvFn3PodPKxHvXDF3NE9qljYiS6S0uFHngnYTiLC45Z3aNMpsCgkvx0050Syroc3YIvBz5U50LLxjl+gSHTbOyqgC7YbritqgKdo6AdZ6Ar5Qlmk+g6dYYWme1t5BhpaGoEtYrkWO8LtIJRZVZoWbQOq0+1EizHTlOUZIkn26jXxaNtg8ZBV0AbvES/X0g0cADGjafU9Cyy6H6Zey7A6JhY6Hdw35rVu43jJIWFRRlwMt2huGT8H0uAA3S9pIoaP334S2XUHrjTvnhvEHzoHBHcXCk20+WJ3p5nKqP7NHrf6JAK4WP+oXMbzOO+rQ4ras9eqvDiyoMwGMseYi0uiX4V1DV4oywnECFWcc6b9m4AlWEWgAc7j0GFZnQj/mw5roqPf80uyVwse1ehBX9taWSFdX3 xJMO8isT fBW0zWXauNjV/5mYDVxQNzv8D59RwOKgkH9kzChxRsX0SIWMb9BhQBry0slCKIvonKGLsbIKVM0YjrTPdsBN29+VTVNSNRZCLSt7tdgjSyvVh50PgqterY2UooPkG51mXjjrFpyq5tq7Hi6ChXttDiEE0HZ0oNa1JTomucqsOvSPHrJlwOT1wLynQS3/z/lVkE48tBKLvimg279PpvKVaKZkEhziQsQ/nFvPPgxccmAJEwoXLEZ6Ir1WXxkPMbbXT1gCleYdM++YnRuhbply/Au/7l5l43vcqLz3Wl8o9U6HSJ1br/PxSNMzCYEEYy/nrBG5b9j4ZRe1Q6oYs6QVwc7llT9Eq3jHNkFfKIodpm2xcCEZf7Zi1wSLpjHU6GiD36N9NGGwAKZYOZQmak2OQEjUejIw+MFsqiYMw8IrIW5C9y2HMlU9W5Vl9P48ny6H9enYL31bxY7TIHmAAxLyI0emuvFsqx3QyDDvfTf62EOBy4/8gsf8XXavtnOmsAwy30Q27geWflrnXGev2M4EUMO3UBGYT8BhZaVHyqfoGCA1LKMEYi0W4ubee8NgFk3A1a7n+4pYiO1uhBQQiplivjNIyztBkiRQgBNsdEtG3F56aDNlSEQuDFQy6HA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000061, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/fork.c b/kernel/fork.c index e1dd79c7738c..bdb55f25895d 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -491,6 +491,9 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + + /* The vma should not be locked while being destroyed. */ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); __vm_area_free(vma); } #endif From patchwork Mon Feb 27 17:36:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153984 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACEE3C64ED6 for ; Mon, 27 Feb 2023 17:37:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B73C280005; Mon, 27 Feb 2023 12:37:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 43E0B280001; Mon, 27 Feb 2023 12:37:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30730280005; Mon, 27 Feb 2023 12:37:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1DB1F280001 for ; Mon, 27 Feb 2023 12:37:30 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C5E7AC0A4F for ; Mon, 27 Feb 2023 17:37:29 +0000 (UTC) X-FDA: 80513778618.23.3B1ACFD Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf11.hostedemail.com (Postfix) with ESMTP id F242940019 for ; Mon, 27 Feb 2023 17:37:27 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oOzu68rh; spf=pass (imf11.hostedemail.com: domain of 3Vur8YwYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3Vur8YwYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519448; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2RRQaU516ZwpTlJ/IPA+DGjHt1op6pQWjJsnPIcYfuA=; b=1/1XCHJvZ7BtY4ajBdLTrLfXuQ4sWVyFRxXEUCHZFF4c53fTzsCC9JyyO7+Azz2nTwoEDh cjzE8OH8RyJdEAdsk0b/nxmf1cFmPIJNJ23ZLYPm5JwjJ1ID7gzWPjCidSZzFgavcjhxHl oNzTcgIvyuo8wx8M+ddSCdPuVtP0hRk= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oOzu68rh; spf=pass (imf11.hostedemail.com: domain of 3Vur8YwYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3Vur8YwYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519448; a=rsa-sha256; cv=none; b=LrNr1Ou9VnFe01/rQeJdnEDNV3yYjD8bE43UM3KLeNejRZEJ4JNtsUD3HZu2SoFnZoP8br /O0v+VZi8MuDIiJNCe1vXUc2EgtimVHWypeT5ftYtPJWcuPOyCa2kXn+UbpGFbkD6Ayjb2 0+7AiXXa02pXzdhTzW4GjikAriKG7U4= Received: by mail-pg1-f201.google.com with SMTP id s187-20020a635ec4000000b00502f5c8f5eeso2175097pgb.14 for ; Mon, 27 Feb 2023 09:37:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2RRQaU516ZwpTlJ/IPA+DGjHt1op6pQWjJsnPIcYfuA=; b=oOzu68rhqFvQUE3xPXRgC65GI9mEeLvvYxOtiIzpOqVj22Kyf5G6v2nWjVGklLxMvP HmGHsiBCF1KS106jCim6ns3MdM+isHFIB2cW4BN/lGs8dwWcRP4ETfa+ag2t0c8ooDBq IbM6nmgazseNTIyYCuuyg4z6GzI60o3WZCSdFyA+NgXZEZsvKFhXlvHjg7yVejPrN9tm lKdZz9wSXLCbvII2MlT1JIIh3E3Pzy0qhZRWTGIGZxssznyrIga0ZMQq7hZXO4CZPVOT 3YmHDK/l7fYrKAyqOPLY4x/QliUuYuXk7UtTVeDvzP6SJegWWyEijbowKlqNMovXBpqB mtGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2RRQaU516ZwpTlJ/IPA+DGjHt1op6pQWjJsnPIcYfuA=; b=AV3Gin21mnCJcqMpL1VbuqHlcyvgHR26W2n9ym5OHAxusak/MFS1p+KtlK0TJT5chr ZerNtkF8DIQ1kMNC+ktSqH8cMhl6N0vYHAw+TjBs9D/L8uM7mHcNDeOkXC1Jbcxd5E4i PEzCuaygHS5puge3dpnxrq8/75U0zFavn2nIw9By0HBLfs6BL+bjjaeHXAMO/Prpm2I7 evKRPW/GOllZ+eZuNvKo65yJWQe2uNTwaA0RncCEqo+R6a9Q0h3vDrz2U9/mZX70CI6e FCQw4h8ORVxW3+exxvDhmX5eyiKKx9jqc2xtCXN0rGLrO30jSuJGsUiiayKtafZMKlid orvw== X-Gm-Message-State: AO0yUKXzgZy7mQ55ChigMk7LKN56Q6BpGgnGPHilKGEgwn4yd02/4SHD ZYT34mSJIq9qA9xAdM7b0rtwDS5+dKw= X-Google-Smtp-Source: AK7set/2J8KUumsEZ/0orhhZ+Y6s/zP6QMsk4a+yqlkwHuIWJVWfr/HftGogZhR0NE5sVdiuRfgOT6Lze5Y= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a62:8387:0:b0:593:974c:cba7 with SMTP id h129-20020a628387000000b00593974ccba7mr6413186pfe.5.1677519446727; Mon, 27 Feb 2023 09:37:26 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:20 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-22-surenb@google.com> Subject: [PATCH v4 21/33] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: F242940019 X-Rspam-User: X-Stat-Signature: orfbi1epaze6jbidjpzdiijt7qpoij5t X-HE-Tag: 1677519447-944288 X-HE-Meta: U2FsdGVkX1/9wbWRuLj4OE6iKRv9LJb99qEoWAPcgCo0zxMgIe+Q02OZbH7ihdYDAagJA023Qkv4vwkNMqAcW/gQwYP2zkFOTmNrjaSkZ+SKXJW/IL29SPkxppMtHW/qLu4yPUBBJtK7l+FHwhVT1nUO5IAs/ZLmpqSCK7GCxezjECNPnN4z78dUg/wR7T5FXoviT/JqdO2oOjt2erq1cwZSi7tOUcdW2eB+u8L8lPRwnKQYF4cOBmKGcq4ppeO9/lVih1Ylc9BohzECTbMXEQ55YaB+I1IOEtDm/gLWL3ON1vGQyRr/ueTX2vx17wtOaKdCPuVVWcZLBtTCL3UOCWQ5tun0ryCwR+yzgDyF8EhUTO5W4JAPnH4hh+guu9G5dXf/Oid/DcK1fXHWRTaX/U5Ux1OR7ZB7llGxagGK1NUoBrmH9F1d47trIjYJJNsDUQiO06RJ3IWRnp+NZ5oMU0WM1enwk1vMdSFJPZYZbdl7DlFSyFuRjzjk7zHR45xoMzptm6it2VqYHybJ0Q+6W01C6rxoOcpB+PyCou18Rdkp+k73VJsWdRUB8wWGdf57NhyzemikyN7h88DYvGD2DpTE0uWM44IrR5lVVwhDzhJ3bzNHeU5KuMoQihPNUXZUTQz8IgwtVps7qkW4ZZCWKtHk63QNts4XxzJT/RqAc6ooOF4wo/VuLTbQZq6e/xQDzlmRQrmYYLXH14V2LmzOtNVntf2ikV/FRE4EFX7p50/8T1QkDaUtB5BuIrsLgubtiaD8JXPkH+1+cVaD1wO9WbWRVHZBR/43zuSpvb85eRGB3mH6Zff+MyAtX9rl2r5/pUjNI4GTtnhXMyc86LiQTFX0n9ugh6ffi0McCw9c2a2EYAhon9MwA0gCxDIzL/XagRaEu5HMF4H0F0ZBeP9GlhpesVAz0+HGNxWYS2x9cBQvOCUygctug7fI1x8P531cVvEImalEbfNLx5bakFB y/UU5vje 7UOrN6cJ6evzB9zqE1kKTlzZ5Hj5r0wnbP56ImydBzaAvPcCEiqzMTIxT7z6fmYXWRMmWiXDof+Pqdk+aRpUoKAX2EOtTyVI3QVyoffE3ORg+peprL7az9cUmh4kJ8Jub7BeZ0jKKn+j/3rUrp3SER/ALs1a0vtvXldI0W8cJcyVyfH5ZCZPf8U5h+AfSR2ZIo/dWfMA85yHGxahUZrJ3ZoUWB0y2q8VzxOUSSl5fA2w3bBn3Go0Si5c5lZa7Ac+2BwyKEHmkB2aEIamHuYWitJQtWNsvBfIhzvg6Rk46FOHkzqjWVCHXJuVpEy21pkgrjxG6/CNa2WZzG48AOGUlMBP4nUZA21lHO47ghg+NTWePiIRGql9RsNm9DY/k5ZieVRMMi+xFT6XPIH4eFV80iF81LhelPMj14sMA8FWSF9Mvnzv93oHEMfWtGQD09qQ93pkfKYTnA4lTNFONwr7illYuklfiNHSPg5jhQXdXWo8sz0QJjafBEcRtwpJ3o0tfgOcYMxUCnMYKy8Dpm2B0XuIwKN3wXX05euEydB7rDWjOYP/jsCudTAxw0jHfBgxL20KPObCVZ3lWly6fezm9jn4ibJYfHfvFcaGXDBUrLuinR+mp67rgtzexRF4N+faPtMJ4aPwjXjr+jgz59XFWnYmqNaITszMb9AgWKazFcv9RtQ4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Page fault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to write-lock all VMAs and prevent this race with page fault handlers that would hold VMA locks. VMAs are locked before i_mmap_rwsem and anon_vma to keep the same locking order as in page fault handlers. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ec745586785c..b947d82e8522 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3486,6 +3486,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * of mm/rmap.c: * - all hugetlbfs_i_mmap_rwsem_key locks (aka mapping->i_mmap_rwsem for * hugetlb mapping); + * - all vmas marked locked * - all i_mmap_rwsem locks; * - all anon_vma->rwseml * @@ -3508,6 +3509,13 @@ int mm_take_all_locks(struct mm_struct *mm) mutex_lock(&mm_all_locks_mutex); + mas_for_each(&mas, vma, ULONG_MAX) { + if (signal_pending(current)) + goto out_unlock; + vma_start_write(vma); + } + + mas_set(&mas, 0); mas_for_each(&mas, vma, ULONG_MAX) { if (signal_pending(current)) goto out_unlock; @@ -3597,6 +3605,7 @@ void mm_drop_all_locks(struct mm_struct *mm) if (vma->vm_file && vma->vm_file->f_mapping) vm_unlock_mapping(vma->vm_file->f_mapping); } + vma_end_write_all(mm); mutex_unlock(&mm_all_locks_mutex); } From patchwork Mon Feb 27 17:36:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5E08C64ED8 for ; Mon, 27 Feb 2023 17:37:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 503E8280006; Mon, 27 Feb 2023 12:37:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48C81280001; Mon, 27 Feb 2023 12:37:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 353DE280006; Mon, 27 Feb 2023 12:37:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 25641280001 for ; Mon, 27 Feb 2023 12:37:32 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 06327A09C4 for ; Mon, 27 Feb 2023 17:37:32 +0000 (UTC) X-FDA: 80513778744.07.EDDBFA3 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf28.hostedemail.com (Postfix) with ESMTP id EF54EC001A for ; Mon, 27 Feb 2023 17:37:29 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FEvdPkue; spf=pass (imf28.hostedemail.com: domain of 3Wer8YwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Wer8YwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519450; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=arZZ82aLoZGzTODI4V3Ds+XnXRSv5CvpGtFg8O7kSHU=; b=WAJlDBXKeqQxXFZZIfE1OvYQnyGA71jg5GUYH+MqMlvgHbGjhNCOzal0RPTJuJYx3GTNgq QovPevyIeRmUAMIdLHDZ0gR9leb9BgFLKsrrspQHOL3z9pBj57mieNpy5sUE8wXhUsvtQN OpVUTSSLA8Jt8CStIL/rYdKSoCVh+Ho= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FEvdPkue; spf=pass (imf28.hostedemail.com: domain of 3Wer8YwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Wer8YwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519450; a=rsa-sha256; cv=none; b=VOq7RmfR8eLDnhsx+eZXZ2a5rJO2XrtrG6OBn4yYCjIuecrVNR+a4xf1axuVS10YaHcrWp KyB7wZjbx8wF9X2KMElu4NJSkMy8Sp7kkBNWmQHXzyvo+ad9U0DGy9HpoxuQCcNDth1cOT hid7Yp3uJ/r6cAfgS5XQSLCB05rkNGk= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5376fa4106eso153568777b3.7 for ; Mon, 27 Feb 2023 09:37:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=arZZ82aLoZGzTODI4V3Ds+XnXRSv5CvpGtFg8O7kSHU=; b=FEvdPkuedKMzzGsmheGWPizHKb0nGjYHVNnx+9kAk6JWaGUBiX99d1Bv/O2GM6XSSl gB8+ecHQ4IjEhzwJPfml0bM5L6LYZ2SwUORgwjGkuiwqlRjsYJT4KPxwzw1UlC60m4aZ A90w9fKe9n+yvmErALp7FrbdpDMG/bmb9rYVdsVvHwUyt0atMOUCv41ig6x8R+tdHGZS c/nc1AnKLbHyXNdSxCybAre4lyR5J1UNPqZOtvyqt/5yJckeBvBSk6OPvMXjo6UK2C1R s7BFEuIEED1IHyyqkGYjy7EtZvl+QTa1y4yJouRGAzmHG8rSxuivFT0pWoYd4Spinq1k BQ2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=arZZ82aLoZGzTODI4V3Ds+XnXRSv5CvpGtFg8O7kSHU=; b=amIogAhpEmwXwy527EkGnyAhTeLhIdxE8lFXoFEEW4zYUwWezUW3JuThy5sOYv6zVP czJCStRHqm8gGYaH46G171IKYwBnHDA6F1vbQCXdakpLnRFMX3RPaz117S47oHN13yqG 6di/sgoujAQf+ArxVv0+pZreG8YUlYR2UlNQ61080yCfBB2o2mxqpDJJnE5qpFd+J5fn CCBFkZOqATom02KL/9vEu5W1BipuqlS201maa4r1F70dQ0K/1ea2C9frQAX8iYmTaOKq /Wjg1XQwXOq6uFCV/ovVe16OnrVQgxnFkpEFqEE6E6MkFbmg2i02OJLjHb2OMY4tObLr +Asw== X-Gm-Message-State: AO0yUKVDjzL5xG0bi4sMDLmKD6p+aAd+ujjz61jKMZU+NKdIjKttqeZE RnMCaRmpnZNAmbY16qfHKAnLioDTkpo= X-Google-Smtp-Source: AK7set/UVPuqJo669/ldHphDhViQWe/utENdYlZ/T7NeJi51KLkDK1GElSVGwaBDs6B97rQ0tmz39GqO0ZQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a81:ac68:0:b0:535:18be:4126 with SMTP id z40-20020a81ac68000000b0053518be4126mr10693031ywj.6.1677519449031; Mon, 27 Feb 2023 09:37:29 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:21 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-23-surenb@google.com> Subject: [PATCH v4 22/33] mm: introduce vma detached flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: urcc3tskid8fmoyr59dzhwtd3coxxs8j X-Rspamd-Queue-Id: EF54EC001A X-HE-Tag: 1677519449-376355 X-HE-Meta: U2FsdGVkX18P2KPGWT9RUTirivBo35UbGIQyrU6/OPYPtaizODdPWMBRzehcXq8KDnbKmGSSray5/clyxxcM+MO2Jucbo6SAlXYcZkCjOPq+ZkazE0RjeDSUgUEJOt6uQnlC8OtuGy5/cWj2DB4LeSVUB4nMEqjsPC6N05l77pCKJPluDm9qqCU3WhpAqj/tm0PZz9jzY2xxrvhVvSrt3xgbYB4dz1lguxaKHRGNbesr/GF+EOnd0W/havGzLE1tg6rMk7tOoq9Z9Vxq8FPSChtJImQr86+fBvV20xQif9Al+Y8OzpUB+06FBpVa/nAZ42K78MN/PgbWJK/19UPoApWQhx3CmQLNxbP92SSGxAgEILF3AMcKdNGylV/xCC7KGuIVltCfnSrsLDFDU8exRvfb9hszydIlmSwrxawTUItdNqK0zZIWgEBZw577fgN1h6D9sgO1xBoFvXk4pX6fYULwyp6fb2AZCjshn+pNavO8hayZODXntACr01LT+5vYP01zrayoXqXE4ExnvMfo1zotSUyYiU1eh+N5Mti4KpEzR7ikL26veO2jQlNzNP4ipJnuAfleowkZM2hssXJh6fX03y0zTdIADGg2bW75BWaNVp/nahCXVEnrC5pQB2JUtGkHQC9mZr9uzi9NIfDcrzu7ZJm9pKkRr19LfV5dD5Z0w2RexONwMRraTgWQP2KXATCqIPZK8j9zceTGlK14lq3KWwVFhu7wzot24yIePeeAAGX72mWMnaP8lOceNWtzMMJ8R/htGKCF4Epv1yZF/6yRN5YPNsuQuSyUOj8tQprLBn9qWd89aZYnQYlxMYgXIc7wGeLE4yitjPIHMB25Qv8ps7ZWotKvR5bweZ0INGFEblKAnCtrUe+fKuMyUPzqkfxfUSatTPLUtZUWTDSdGhxYQyLzU1PPPkarvb5eqpZjmBVx33yVhXqrbtkyTH2x3dGSImk2LjdBYp48E0b Kxqlbiby EhT7yGvm8yTmQU9BIDE9f7+BPV4Km/5T8Uu4exevbvkuwX1ZufWvMQI/40KqRE8fSUVfxQCrdYrAXgGXu+f3/8O0w3+fd8P2x6D/2mo/La3ZU8wqPm2SDA+gYQX+fvc3HoxZMHpvCioEP/hOjKYmm1Y7OpZsz/WwLmSl7fkNe4NtP6YbYcRbMTB4oO/ciWzsweRM4uTaKuTa41PFY/BvgCdYn6yDVVU8mJl9/iuWfyRszQqpaWH6Nn/MIHy2GSExEVhKlALEydTQEXtXCmMpnCwILo4f5+6RHk/Hachg8j8DuyKOKPRoNtP6jbYin3oLhr9pIFU3at//aRwYW4hNfoEn7B9oe3Gofk4j0J9dIMyok+u6gGeeWPMnLt63PlmjS4ctKB+6Cy5R2PBM6aVkSo0w2X69UUOeHKnEsOMrUuob0qI94n2TdroMUwYpKM7t7IPeP/9nK/0yA8bXXLMgojtCjRAiNeN65Qo3uY2zfzZtmMUTriz2my44YB5m+uV9EicnwstdBNzFjpXtIRHegWhvH+9WroZ+/iFmGZFcJfCbYGZB14AKVpScG1Z4p6VM/HtnUOctvZrZwpIpFoDEQL43q1+rT1qPbEWgNPi6QQ0hRCgUgt41Ok/AZI30mnUa5tmW4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Per-vma locking mechanism will search for VMA under RCU protection and then after locking it, has to ensure it was not removed from the VMA tree after we found it. To make this check efficient, introduce a vma->detached flag to mark VMAs which were removed from the VMA tree. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 11 +++++++++++ include/linux/mm_types.h | 3 +++ mm/mmap.c | 2 ++ 3 files changed, 16 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3d5e8666892d..895bb3950e8a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -693,6 +693,14 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +{ + /* When detaching vma should be write-locked */ + if (detached) + vma_assert_write_locked(vma); + vma->detached = detached; +} + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -701,6 +709,8 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma, + bool detached) {} #endif /* CONFIG_PER_VMA_LOCK */ @@ -712,6 +722,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_mark_detached(vma, false); vma_init_lock(vma); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index a4e7493bacd7..45a219d33c6b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -511,6 +511,9 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; struct rw_semaphore lock; + + /* Flag to indicate areas detached from the mm->mm_mt tree */ + bool detached; #endif /* diff --git a/mm/mmap.c b/mm/mmap.c index b947d82e8522..df13c33498db 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -600,6 +600,7 @@ static inline void vma_complete(struct vma_prepare *vp, if (vp->remove) { again: + vma_mark_detached(vp->remove, true); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -2261,6 +2262,7 @@ static inline int munmap_sidetree(struct vm_area_struct *vma, if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; + vma_mark_detached(vma, true); if (vma->vm_flags & VM_LOCKED) vma->vm_mm->locked_vm -= vma_pages(vma); From patchwork Mon Feb 27 17:36:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32862C7EE23 for ; Mon, 27 Feb 2023 17:37:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA812280007; Mon, 27 Feb 2023 12:37:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A0E50280001; Mon, 27 Feb 2023 12:37:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88512280007; Mon, 27 Feb 2023 12:37:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 78F76280001 for ; Mon, 27 Feb 2023 12:37:34 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2177C80A2C for ; Mon, 27 Feb 2023 17:37:34 +0000 (UTC) X-FDA: 80513778828.13.C65C844 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf24.hostedemail.com (Postfix) with ESMTP id 44A6618001D for ; Mon, 27 Feb 2023 17:37:32 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=M2xu6PJm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3W-r8YwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3W-r8YwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519452; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iRRoEkIKRwl2QQHnq1OJVyPp4L6+mQc5CWukcTSQqHI=; b=zjMrW5reaThVuk2dhZY4+w8TctGplRAeDE53t8jwuL7tJQQwPVfxYm6SDhUQopZ+XH/+/6 SYjE4QQUG9r0QMV3dR6eAXa/d566Ndwq4yvfLAj8K8VwW9ZGZ5LVvWym4/cHLPGkP58XVx 6tEawo4EdV/wJdhatDolA/4mAILTS98= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=M2xu6PJm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3W-r8YwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3W-r8YwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519452; a=rsa-sha256; cv=none; b=m0ejVuPJtDDdgkkAr1TNoJZp0RoniaesRwVo4qmLiSoXc5kESCCbRAN6GyLXDiadHW7mRy uRuYRSfUDJTOvQRzL1DMDs6zFrDKxtSh4YcwAseDZDMiacCQqTh7bfDD5OeVMzqycX0UKB CmT1Y9LZ2D4obePsbSQoYzevgI9ABNc= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-53700262a47so150876027b3.4 for ; Mon, 27 Feb 2023 09:37:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iRRoEkIKRwl2QQHnq1OJVyPp4L6+mQc5CWukcTSQqHI=; b=M2xu6PJmgtYFCFjZtLFUAhDJwsRj1PNvn9tpCCynDkmhHKgKWgNMPKAbFeMB0d7vzQ aKVeG6bXcHl5yxZ6JEXjbkVB2BvokKvbI4LCTQCQPJySbeEhyr0HdzjFd7UV7XC52Nqd IM/Y4Y2oz4P4pwkohu9Bqul5qghmRgidkrq66aJWcYZe5Ah2frrlViyJFqWif5WyIuln lwhBindA8HTOjteANHg0VaUlc+wXF2i0bjESG2yKoDnx3/qzuFoEw7K8TYty79uXoeAT DG0JzJNCfRj4E5rbsNquxz9O5F/H/zOUYZHgnCU0K2ioTwSn7zpOFzVjAn3t3qu4Rbvy YYMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iRRoEkIKRwl2QQHnq1OJVyPp4L6+mQc5CWukcTSQqHI=; b=WZEP7LhTOrS/NYuK8cQSqPtTYnhNjfDTBAC0G6HZvX0B3doLQV8BmwjrPqRvg2Z+Rr xu239Xvd3vOpxzg45DffpMFA+IKTxIB8zrwgseE5mOjlAAT5diEglV0v0iWtUW9K6JXS mDw9aCeSdckV8SLSzwJB9fzJZ8U60vlD5j0EM9JKB/SRHrGv5M1eGV1GdJZLSyg6td0c XKu//xCuSs4dufnial8AzXqp0gC1/ijgF5oSWoS4ZiNCbleo2Rfmvn9mvD7IL8FZX8Yu NzEP1nH3I5O5r/5k3Glar0l75aWvDB4Vb8quA+OACaMkr0prQWt2omEFmQpV9TKH+kIC 05lA== X-Gm-Message-State: AO0yUKU7M3Y0IM94v0tIJY8GnfFxamvX6LNU70vt/JkhP0Nl5Y8wpULe meVW3FeTELNSnALaqAk8Tv2OQ1sS3k4= X-Google-Smtp-Source: AK7set8ux+VAfJlkMlZWU+S8XCIwu1HZ1wVSlOGqhGsiL0aWWxFXwk2KW/a7dOnnyUwvPtqegO11CuQKxxM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a5b:6cf:0:b0:a03:da3f:3e65 with SMTP id r15-20020a5b06cf000000b00a03da3f3e65mr9154608ybq.6.1677519451327; Mon, 27 Feb 2023 09:37:31 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:22 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-24-surenb@google.com> Subject: [PATCH v4 23/33] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 44A6618001D X-Stat-Signature: nk94po8o33958is6wwy54t3pxb9ofrd6 X-HE-Tag: 1677519452-589118 X-HE-Meta: U2FsdGVkX19GYoRBdAKUBfX8bBnpXN1kwHs3Ao5bbHHFP6rbCd8m3Sr0ce3gFe+zcdnbOJUmwdmsV3ybG+K13Aj7+07jla3TjLamOS/npJSCeRSiA6Bv9OSlLZYtOkJHopd3J/z76CmnDS7orEtB0TaeD27C1P1CXZiet9nzkl5l/1BpwHSFdMIRlZpil17k75whhQLl0FxYARLCsvyBO12TCxUTUoGR2xt8OmzgT4rDVsak82vS9OKNYte5BZE6+jLI38eF5MBkHsNQq5puXJs0KvFs/kLB3bJl0oOfmwtBKxkJE4kFpV6rgJFw4SwD4XW6GdDLYdPfqRlYd/hLms4508xXTExyzSeMCgGFKyUeFQr0U15AFsfLmTd9KBAgeDZuTzujKns4MzU5pMI7t+TFM3SAkj9ktqOkCXlMsYLFGGaV0Znl0JuwS1pXzfLfasVlIVoJHopQxwd4N5k9ljLzEG0mN8E2ctNtqwA+JzGfdToYamWW0aF8NMZ8O3yZr15Gb8UApTWITEhdNzWdni3ixaK2VoziYo1hxkbGQu01dUW2xPQ0d+FqGqQaQ8F08or9U7Uhy2tnzeklZH1oTCnkqxI02BZlPNjdiR+ZXUukTJSob+nTG0QXRwGS7CRTQCXqQP0+QKRpj1i28f7ynblyDS3gV+aMGy61JmoYoGNMpYD8Y4DiU4nah4AynhTsOFsyvzbNK+7rd7w5F7wlJb4jaOmBM2GjLUjve+vkNVNQV5JE3S6TKbjlHHgRXm8c0VI4rBXEPbWwBu2CAufAYkUVcP3usQF/arwWFtX2Cfk8jxik988RNwkyuaku6Pbc+/HjgiXlqYWHjT4SaqE8+LKZxO+mSKosbe2AfTB6i3ot8sHschs9lt0deOCx0mU8JzByIzTFs1xpSBtBP7nDBOAvKEi72IUCDwbnfXZfHlhRQzfyPei/ntbHudeg81A1ZZIEbTmXtvAcfDaPRmu 9vZMWlZx 3LTsQJUj9XIMroMYgJQmZYJREQFxhhVraBVHNmxbzwqNetLtKLBqe+5EfKzUySkSU7ERTIo+WOAUEqWG5G9ZJt7/oqEmBa7j+T0obSImXkJ+kHMkwKlt+pCbsYUmOYYjXxcW67n38l+5s5Av7b/AL4lVDC7wHpU31+njKI9RhME0elGQP5Q/lEzLQzJG1lyaLaLrMTrizGwAr/33BimGUL0ZrnKWNSX3vqzO0JfOp0sdKKCLyfjqyG7HPEvOr7+ufDM+BzbX7cFpxYTenPHoKGnKPLdFsCnks1d8zn69e6RIw/yD0Wzkx1HY+9sgBpT13fHGecIpFYFWN0Qp64gnustOMqK9PWsbQTuKc7SlgsO+UfeS9AT/nTZX7kmFgif+xOfI94s3TZAMruYs4KKbiByK50eACvk24vyITETrBWj1UQfwiM24YODimWVjoS8VbhRAmlC/03dvsIGVx7vfR6ansdPuB0+T4iIPmWdm58W2MKxgOtqikYEnvxG5Rc28F4URX50OuJpLFAUnY+VS1bhzPu9aIx7OhuX2MZSkzzf10a3ubGFM5kHvVfgS6OOXS4MN633Al1q/EQtIm+a1lqZjfW/RZmjnZSLS9vUedMTKCFo3HomfxNJOVsdKzKJeGHXj/oe7UsL1IeOt+SAWYaV0ZK1evMnGL1FtQWdq87HbyYWE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 895bb3950e8a..46d2db743b1a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -701,6 +701,9 @@ static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) vma->detached = detached; } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index f7f412833e42..bda4c1a991f0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5221,6 +5221,52 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma; + + rcu_read_lock(); +retry: + vma = mas_walk(&mas); + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_start_read(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_end_read(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + if (vma->detached) { + vma_end_read(vma); + /* The area was replaced with another one */ + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Mon Feb 27 17:36:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5531C7EE2E for ; Mon, 27 Feb 2023 17:37:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8375D280008; Mon, 27 Feb 2023 12:37:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E68F280001; Mon, 27 Feb 2023 12:37:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D53D280008; Mon, 27 Feb 2023 12:37:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5CD46280001 for ; Mon, 27 Feb 2023 12:37:37 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 09E39AAF88 for ; Mon, 27 Feb 2023 17:37:37 +0000 (UTC) X-FDA: 80513778954.09.CB88A74 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf05.hostedemail.com (Postfix) with ESMTP id F3D0210000C for ; Mon, 27 Feb 2023 17:37:34 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ehrse2Vd; spf=pass (imf05.hostedemail.com: domain of 3Xer8YwYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3Xer8YwYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519455; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qrP8wdv0q5g6TZoHkkTnkJI1bRVcCya3euyXWiQCR7k=; b=KsmtMi85L0zhCtD0dljVihiPZ6ptwGhGMh+WtQPK6fdGaJ/57fzwEbKKxm9pEWCmlamQ6z x/urUl4T5RZzbb3hRfU0tT6aUho4cHWVuV0S43TGSBpejbnvc/NSAt+r5+oISjnyulRPwW AoLoOMIvwrYGS4KXYrjczPQPG/mHZks= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ehrse2Vd; spf=pass (imf05.hostedemail.com: domain of 3Xer8YwYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3Xer8YwYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519455; a=rsa-sha256; cv=none; b=FxOztobs2mG40ILq3tqgNY4/2ekI+1uWIYf2596fS3WYeAXSzliqoCXOPVe3XNwQaOvytI ZcOkVsGaUC4rd1zuxi0FBhp/+eb8lUgyS9sv5QWeFNnbE9sPhmDMcI+7LkUNFZanxTlcHM XxxbWKxVTm04FtCVK5CsQFkCn+FX98o= Received: by mail-pg1-f201.google.com with SMTP id b21-20020a63cf55000000b004393806c06eso2170651pgj.4 for ; Mon, 27 Feb 2023 09:37:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qrP8wdv0q5g6TZoHkkTnkJI1bRVcCya3euyXWiQCR7k=; b=ehrse2VdjDnpUSRaJ3cu0jOOkJfFGFsUNPVo557shDqmppFkfxAsvzUOYaJjQ0PijB EblMOtm5g5NRXaAPcMrHVwQAv8s/x+GnRlxruVHPkK5jtQN3yNt/EBweE+tKCGKaism7 U3HUjXQbto1B7qoo3unz00bS4zA1LfeWfOlzZkorv9RHy9hKyHRYaEA4i2TEzaJJuMBg IRIyIDCAlrqJadscsfwaMyXl67M96LYiFdG5WP3t9rIaF13TbFMOYIsSXYvDmmcAe7mT R89y28xjRtFN9b/E58zHscfzaCRWqNFydygKz6NtUn93MmZ0Jv953fv4FHoISujugl1t udmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qrP8wdv0q5g6TZoHkkTnkJI1bRVcCya3euyXWiQCR7k=; b=RVDIjc+iD/6ao+VwR41WbP9PHjRs0454Nb46oDtgP/VvLQrQm1PzOCQ10ZV5cu1XxS ytTRg9WaUojQc+/bFoVNkzp0JjZFrQISmERN6FtumsjVwlPNQQoO8P4+SdLchUECPskg mu0nCq9bQLaJT4eKsbOM12dlI73I/BVhGZEarUItju7zoYc4EN5jx2MZOkJSIrOoToQz Bp4IEa5c4dl/0Oli7F/wzsshTrUOFhkd/9iP1FpFKbhcVsamD4k1oWQA8Bni6CIDb8qH sBoKB9tuAotHZfr2LTw34xJgO4oT0flhJHMChh7LwH9trpFgFD5yNklC9ENBdNuxF+rS 5fiw== X-Gm-Message-State: AO0yUKWvk6Y+n0FT352NcdlnRILWSETHirDjLWheg5Te5WgfAjl47Ozn sguSNUZmkIKPd0KjYE3X4wBKJdRNO0s= X-Google-Smtp-Source: AK7set/aNutcI5JPOH6nKZSnuzXLxYVujD/4TtRCmI167JUO0duhDMkhOBXuPDJivfwxfw4Ct9inq/o370Q= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a17:902:ab58:b0:19b:e4c:2039 with SMTP id ij24-20020a170902ab5800b0019b0e4c2039mr6578302plb.4.1677519453804; Mon, 27 Feb 2023 09:37:33 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:23 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-25-surenb@google.com> Subject: [PATCH v4 24/33] mm: fall back to mmap_lock if vma->anon_vma is not yet set From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: F3D0210000C X-Rspam-User: X-Stat-Signature: 7jijtff69ezux74krek9psnw3jperd8a X-HE-Tag: 1677519454-994377 X-HE-Meta: U2FsdGVkX18WujNJ3Mg+yJ7XyKmBNq/XjRtzkC1JVWNNVuXObkxi69u2ScwpwAznIIdba/AQZOszLccZRTlYu3LklUdiRguIegWzSI9sqSlS8o9jRUOQUFt7VNbd+4/pb5APp/iNARBKePDnhK3P+Iaa4gxCkWRab4d5JjWV4j42E51wYNZ/fPUTinkQgX9uthjl8yg9yMhlBtoLgbXijVSCIdij0waxifPajXi/VMO0LalNidBtcAAArlkfCt2z8/v7ngsMaECPP+ZLgKjG0ssUUxunp0fXPNXBadglwnkhAx9HlcNgvxGZZfKBNaPF2iLQcxzNdPmp2d8nTGlwSpspW0WZO7SL0moLznPsOABkcMB+4NaYQYxjsK/E0qK21Elab4KK7FLKxWbe168a0J76Uayo3Iszq93QkwgSKfVlN5DQMEMGrYAsSCtAjyheIWdoWySv81KzGPiC2/F2m1mhTI6S0mgXTQ7o7DKJ58Aoq+wCL5SFZWpXwMJA7ZBbwY4eZGVSN/MHTYrdaTyaEWjopgEzonUv1ikSNNMofBvKILjxWVxUzd23XNjSvzwVoV/7QTG+EABblSqIapxybZtdQK+e3jGtI1IXP2rwIyLgATl0z8eAMWGcuMpXRUN/e+YkU19lK/GrYHgqGup4lbm005WcZ+BuTGKvwZs/PdOtDpsAKcDuokmkdxd+Br7RZy+Jd0sJ4p9P6ZZB8W6QTSonZHMxBqOX27ceb10DiJtFrAg7c5fvi3tAXumMB7uDv/VsL1rmSNiioClZKsF587rHP697L7P+aePrN4p090vz8C8lzGCFSixnI9f9zEuh+X18qpMbP8/QKTYXBrdT8hSjsA2V2BML4Oo3wKVGFbgs5Vt9kEFqqHwjldy1xqCkq1xqY5pUMOQu6iZTWntPyZPXbc4S8Q5ttKYafWqL1Foc2RjJEWHNuaHmJqOL4MKixA4AYswNbt8oMnxSc6s 79P88Sgc Au0LtASBoSCTPaDhQhnC7ZvSw6g4MfnaUfynsM5+zwVrQqf2GVOFYY+kD7HBkbTlLXNPbq8yo9VsmAH481QoL0hliZMMDt/jXXsP6FJ73W7sj8IcHZerongxVvpOdYxEsTbtzU0h14fVjaagn5N14C4hAnKJDwNtXw9Hf2jDVk9keDzy1AJ1pHP2nzKkpYGXDqYo85TzfG1/ekYDzBk0Igw3ZG6LCLZsKXpBuhHUt5slBxetDtpFxGjLg77dlVzYG5bKM4+R0EdXDH3+78uacPxA8TARW+XYgMNcQF4q5/B0es1e5eFNk+w7+kr3s+cEf4wjPPT8tEoY0X08GMDyQDq0rYZTuy5R9oR5zFOTCTfM4uJOD/Sc41saq2S7p/D4qI2WeMB7ugs2q/n6gnzFhRuRKVsWXHp0NHDcWvL1LqkTymSxIK1ini2+TcvLHLS43mHAt3nWRAkIrsp8lDm7TaTOr2ULhs2EI89eVcpO3vO7cQ9NuWVICBWAlOrLFB89oNHlEJS2ctx5Cnfs1S3dPBZitUhZnNc+Qt0aStFgTsM3xIZ7qaGDLy7nloe74XFkywKjZGgi11aSlOA9gljaE75VLZdd0xMr8Ci8HrsdZ5DV4ZuNLPO2683bPyEEq8tFA+Ot5Dkv3zyeUWSbYyZHQX2YQfuAcxqMDAyNlK8i18CrV3ls= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When vma->anon_vma is not set, page fault handler will set it by either reusing anon_vma of an adjacent VMA if VMAs are compatible or by allocating a new one. find_mergeable_anon_vma() walks VMA tree to find a compatible adjacent VMA and that requires not only the faulting VMA to be stable but also the tree structure and other VMAs inside that tree. Therefore locking just the faulting VMA is not enough for this search. Fall back to taking mmap_lock when vma->anon_vma is not set. This situation happens only on the first page fault and should not affect overall performance. Signed-off-by: Suren Baghdasaryan Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index bda4c1a991f0..8855846a361b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5243,6 +5243,10 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_is_anonymous(vma)) goto inval; + /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ + if (!vma->anon_vma) + goto inval; + if (!vma_start_read(vma)) goto inval; From patchwork Mon Feb 27 17:36:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153988 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F5CCC64ED6 for ; Mon, 27 Feb 2023 17:37:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 13161280009; Mon, 27 Feb 2023 12:37:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 10A5D280001; Mon, 27 Feb 2023 12:37:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EED60280009; Mon, 27 Feb 2023 12:37:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DFD9D280001 for ; Mon, 27 Feb 2023 12:37:38 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B95494034C for ; Mon, 27 Feb 2023 17:37:38 +0000 (UTC) X-FDA: 80513778996.18.4738611 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 065804001D for ; Mon, 27 Feb 2023 17:37:36 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="rSn/HhKu"; spf=pass (imf17.hostedemail.com: domain of 3YOr8YwYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3YOr8YwYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519457; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7ZDLcNGctmKqQEDQSm/x/HJRE1p1urqN7vMq9ze3IhQ=; b=BFhMqNpO3EEmkZiX3jL975DGW5xaBRNkM2+OdU9BtlQ4ZAlSwVt6ZkUUtvYtFSYpY+g1BA vKCpLoi62Jr9J+U731GDH1W868rKrOOimO5C050dPR8scv2oBp7RfDDnmCVfWgREAziKl4 W2Mr9ISTrqSCcJzaAP4wT2+xSp4CShs= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="rSn/HhKu"; spf=pass (imf17.hostedemail.com: domain of 3YOr8YwYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3YOr8YwYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519457; a=rsa-sha256; cv=none; b=BI8fYJ4LdhcuyksgfSS2xahVLI6YxiiRPFEn3IQSE9kE/VXm11YH2a06OkU2MtfGXm5RWS 6T3Jo89qPsrrdmf7rlTgPmoDE+Bbmposk9ogl3tkZBvvub8p8cxU3ygxIMoWFCoIoDJvmR uPvljIovuwSMS1Lgzdtpzd7n084WDkA= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536bbaeceeaso152745687b3.11 for ; Mon, 27 Feb 2023 09:37:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7ZDLcNGctmKqQEDQSm/x/HJRE1p1urqN7vMq9ze3IhQ=; b=rSn/HhKulcAt/dXRGMHQok87P5M5E270I0oCS3gZJEWkLWidOpnej1dxaWrgMzWbdZ hRegAssYtkAXj68NCwN4NlvBTUXN9bj0pr+BM9FESPqsUtzNJdLfxpMOuaf3C4zcz8AG QQ9sSRnDU7Hb4QEga3Ttu8/AwCSxDxNjWb2i9IxegWBslOk4oTQU94dc0rtucLzduQ3Q 4ozKaX/E5D/Dfa902LgPaXoXtxfZ+pNftUMyHyvMV2xCazmDlSOzCrSiriiDBvuGe/Gm 12h/Fv6wWsZva2wBOKCzc+/nhceTE6OG2eBLFHaxFB9ziU5glhrshChbsmrwR3lUlmIy 6zUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7ZDLcNGctmKqQEDQSm/x/HJRE1p1urqN7vMq9ze3IhQ=; b=uJ6wDPep06ur/DCWcNxSchlM7gamLqEI31RkCrmDuWlMKYCXimijLyc+vAZ3/hggfa Xz+6DXSr6Iqx8o4j/z9JJwzbfUq8mxsgk0hmWWw1yG1t+itmeUd2tQoMdp8m8zDVx5dk Wk3p9eB75CzaJzPPkr0ABjJQ3FzbY6IFdIQia5EpozUczArhxhZzauTPyiW0PxjV+JJ+ P6/FJMDDjjX3Nm40BuZyMv+VATKDRyd8izAIXkACAmJ9USh3rRqZqcp5IXR9xRf5zbtS rDXBeHUl3yNfDFBGRE/dWmkvz1QuUD6vi+HzUwRmQvFQa1uRn68LmsayHIaDYm6wQQEj x2EQ== X-Gm-Message-State: AO0yUKUUQIYjfzHXlbQbOPMamYOiu8QCitAa/bOO9hAAwD2cB96glCQd 4OIk81thh630VVeT2xSJNa3lhObm2aE= X-Google-Smtp-Source: AK7set8grN4JIndM+ly9gt1j/cdj3SHv32UwM9Nv4EN/4JzKqYBKorAEBnSg1fP76/edXWaaLOwjV1mMS3c= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a5b:406:0:b0:904:2aa2:c26c with SMTP id m6-20020a5b0406000000b009042aa2c26cmr9545732ybp.5.1677519456128; Mon, 27 Feb 2023 09:37:36 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:24 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-26-surenb@google.com> Subject: [PATCH v4 25/33] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Laurent Dufour X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: b4f5s3y4jm3axs5nyezn1gxwe9de3iih X-Rspamd-Queue-Id: 065804001D X-HE-Tag: 1677519456-633202 X-HE-Meta: U2FsdGVkX18shNmkzc4v2MY+rN2RT4FoiXzldcJSVt7+Q0lm7vjTQk1S1kQlypfvqYmGRPLXDeAHHwVReMLgKbGfhjmVANd8AMSW8V/CacVi3uOqpsUGKPuxQYRy5FBo+eORVAIjIVD/A9ChlngJLoMGuHLXGBPYtUlwxHN6RhW16ZH22FkAm5UyLkJznNpUgQIlUUUsGlke3z2p+1ucRxYHilEmDD3SpfyrbMc7LP06XJDLpVijGV2WwY7TfhaN6noPx3t6IALTyFsmJnSynCM9vgwCVlSu/kQB+BQUsgqVw4YKyGRoo37zi/U0IHUi16P5aMjJP4vys4cKVVLLfbEBG9DSsCCdkLybmgDCBc+p/tGJQhFuXWHqfnMtDyjHGkrCgQqtOOYQl9SZuTaNTJv1hQhBamsogmXS9irHBBLtVeeI51SZhgXTneCWLGpdiCTYacKRs4QSZBPG0f7uH++NjATIYEdY2SOthLkW62QlCXOXb8jRrB338YpbhvSWuJiB/hazIw1nUpjcol79g4pcsbfpmPFOnA1pLw/j1qXUdrGcz2WIMBadbfTCr4bozAxe0Y5t7dhZdEn/jogl/8SRsp3uGC6e32b5ze/0CS0U4yT3enp4/JWK01WJMSRejWbON1qAG23Quy3e/BTARgrjICsV5XTxtVOa6EVSuEZ5HaE0YfAI6wPnMRZ8OBRf86exOdPqOw9D/6LUe0CXDrTWzYyo+vGBzSHNC8WelLIvgC9MV6vXqo55CTFEzijjowb8vqhqjlyzCGvIPRoOafcfPkoGebgZ0O5DS8RA+/RaOebhMeYi7115nFg65MWE8X0Xe5BnlHux9j0I0ro00r+6nLf3JmgOZuRBefJHvha63N6CFMR3CCcepHEOzxESO6atzUR9wjP9f9GuyrGPSVfeRvX0+5TZc7y6IwTSCUQ2YsJiNnpblFLRIp1bfE3nxHZmBlGVzcgSDu24vk4 6zvqXP+/ 9JjBiRHLJd49L2ZodrSV/bC2i/SlCkgLqyGI4uNYuXsDRP+hnMAJpFzpZKPMXtDzt2T+EaAGGhT0sX5I9OXdiRK6IkyRZ475i0xwYan4euP/TA5jYWbdnFjFsWWTcRwFATtGov+0O2YlInSKrWFb5H10H+aOVxWB5RRwO+e7j8S1LcgFPBCtto5bn2TN1W86QjB5vSzl5TZ24XLzgmnK5buWeYh2vhHTR73OPxDaoXH2MeT4eoGRMCo982fAnTaK4pPuKyWTO8Jqo08ROIuSszMW+WgcnFD0Qmg2vF7ShwGSrUZ/qtLkhEn8XWeAVZ1CFUGPrESrWsTtZ+BehDNCTbIGoaQMxCUoHTHsqveyUBEZgYDM6R3YEcHUqUhMymMbI5PeXiIGHvyA0yRYlyyCCxeLjQCFG6aX2NQMUhnAx0olKtYfqPYfBXcNWT6+6mnOqGYOdn754mRhzrQ1YiSoK3meWH99m9PgYHKaq0S7PpPcMFKvJR3AC5kyE9S1G7zAl0rGHJyP+/v03HqsxBRDF74J3/A5JYSUAIWnG4KvlO1mSnwUCpPaht+o4wLxvXgQskPpwBzwKWvw7c2dbykQVIGZyAfacm6ovpeLqUCP2rGkrTfNH4BlcQ7QqubWg1sHIqjsQh7nu6s4MBN98BUhJMzs5CX6Z4CJXc3FIDu/xnHEoZXm3+Xfo71o6D8OkXtTLxZVl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000011, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 46d2db743b1a..d07ac923333f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -478,7 +478,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 45a219d33c6b..6768533a6b7c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1097,6 +1097,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Mon Feb 27 17:36:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153989 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E826C7EE23 for ; Mon, 27 Feb 2023 17:37:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A333C28000A; Mon, 27 Feb 2023 12:37:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E33A280001; Mon, 27 Feb 2023 12:37:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A9A628000A; Mon, 27 Feb 2023 12:37:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7A787280001 for ; Mon, 27 Feb 2023 12:37:41 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5DE64AAE1D for ; Mon, 27 Feb 2023 17:37:41 +0000 (UTC) X-FDA: 80513779122.21.93DECFA Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf30.hostedemail.com (Postfix) with ESMTP id 8CC6180010 for ; Mon, 27 Feb 2023 17:37:39 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oVVsUrq9; spf=pass (imf30.hostedemail.com: domain of 3Yur8YwYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Yur8YwYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519459; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u/Oe5vIeahqwi5sYk+6bCjyqP36jyrRz0eAbwTkYv10=; b=iAMW1KWFXFSv9RtMClLiuYIkS5kexbzwqbylk5LotLn3F/VApsiReCuxzpzI8t7goj5Bim zwcHB6jQQ3wmZMFh/UUx+FNjxfwBOnpIte6C9YBPZHtMPSYxCNUl5slNkuVogTPBW6H9Ow MrWv1CmY/g7Smq1ohKQ6N7R3z2lx7Ss= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oVVsUrq9; spf=pass (imf30.hostedemail.com: domain of 3Yur8YwYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Yur8YwYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519459; a=rsa-sha256; cv=none; b=pz9QsCKoPBxxfCC5O2pQbqlDov4NixWmtBki0wDhnqZbWpoMMaQ1zYRyJCxIJLIzEgbffF zmswJDQIXJaI7golG8jeyXvP2Mi2s5kEcYbe/rHaT2KcqdqC2Ov/heiTdkvdy5g7FEonWm nkZBp+O3YR74/MjeUs4XduHkxcUI/ds= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-536c039f859so154201887b3.21 for ; Mon, 27 Feb 2023 09:37:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=u/Oe5vIeahqwi5sYk+6bCjyqP36jyrRz0eAbwTkYv10=; b=oVVsUrq9J7ftO0x2n5rOfx5Vuu5nhzvFsm0M1/orxQW1l03nlBX1oBt9Z3PeMUwxB8 EXB0rSNNqcp6UVy0odo1Y2qU2vLsCHExXTtZRuAA28OmedCWSwEPwJ/vb5UPfrQv5PJW ecTpj0oKN79TWuriZXNGVkql1JYRLuNigyKob+qED9T9sBsRaNyBmXPE+YrDxEJo7iH1 ubKjcF24jylzmHXYDSJCXBFRNZiczfW6Yvvr4W8sLGpkz89otOdurivgyE6QG9AxV0ch vbIsOJhdZ8Z0YDKh5k0bAgyDf+j5LXFUYUSVRKs5nZ5bUnBRVL57+VXMgXm8R+LTRb+i YnrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u/Oe5vIeahqwi5sYk+6bCjyqP36jyrRz0eAbwTkYv10=; b=mkBW2zlX7da7MqJGHchM/Gi6jJVazSkr8ondorcAvacpg0dBodDhiUdYM65FNRwYUk nI42PgQhhTDLzPM/ajv4zPSNwaIsM34L+AB0tA4SZiU3dYHv8Q/O0tL9whzg07OI3Osz VKSl74INV622s3LIB8X/RKVwG4l+YtaIPUfc5zPtfPq5BO5nQHdmdcI27e2siIgCtHki HvQHkWPzQ1HswOenEfEYDc4cwwF/ll9yb+q+KGsVZmEayXWodwbYzaIrWLXl82Go2vnY SL0ig96BtDMJwk/d05/HTb3pypmLlCOY2vprdu7fXT47Y7xqbP+tZ+ApsT1l0uh/oi6A PLFg== X-Gm-Message-State: AO0yUKWaMqaNyxDHewuHHtZFYNcLyCGdOxCRKu6zE3wUntIhDeClPc1G /gpbmoSE1OYOHiw49zE/XS8PYNv4Elw= X-Google-Smtp-Source: AK7set+K4/t0896rH5imu1aS5MRSoxg+vS9UwsZogQaAvLfU1/2OHkdV+UAhVM9dEQWx1b4yJz5LZ0tC6/0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a81:3e21:0:b0:533:cf4e:9a80 with SMTP id l33-20020a813e21000000b00533cf4e9a80mr10559241ywa.6.1677519458641; Mon, 27 Feb 2023 09:37:38 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:25 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-27-surenb@google.com> Subject: [PATCH v4 26/33] mm: prevent do_swap_page from handling page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Laurent Dufour X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8CC6180010 X-Rspam-User: X-Stat-Signature: zfw1dc7usb3mrga1m93bod1cy7xw1akd X-HE-Tag: 1677519459-864979 X-HE-Meta: U2FsdGVkX1+MG3g+/467QIP0/oGxe1OH8698p4Vk8lc7kL36Gnx4y88FoJwEshMOaFzGnPcwK1qCZwldhaJsPHAubyjpKTa63UZ0/Ehpd6/hspQ38RWBfvOh7xIO+vO94fdTe9sbRPlo60N7sQmeDRA41dynqdrudn/bJvbPAIMqBPJhPx8sPzxFZHW2YiQDkVz/jAYN8X4VAvAo4+F4IwEvAlryXB1CFSEYbVvo4Rxb0+oASZpCP3bR8+WHZQcRz2XSvWM6oJb3eJXzaGgBmyBUedJLdA/WIsaCx1KN/LcFhe/VqXbx7ExuZORDgcfflvaVDyqNDxApURw3LsuYKSv842wF50ROX48yGTxmdvqLOWBps6A1NJKwmfOkM1HwCsmCXQhg56gLpXd5hZJLFBDqEYiYxm+8RSjttZvbWO8QKWM46VGog2jKSppoC8cR5fJQ7w8OihyWSha/hjyb6hfKnlceMF44WkIxkOlePMYFWg0T5CxJOdvG3e/FrBpa9vrI1TaPmiTraHAw2uQsfJ8LxifKnEqlAOFBLdgt8E9tgvGzkw0LSlRDV/uvjUYrPlHU0vJ/BEmUDOum0ngnwOXvceMn/6EIeiUwNwaTpglk9+ym80lGRcdVc9OlQc5pZVaB2wLXmgo+G62l1e36dTqb5vjxLdaMwaI7LneJrN1q/zd2eBEywJBFmvm0UjK3z9260TWh7Pe13wzO/iWxS8EeECK5DKxOvhzcd8zCFE0e2YrSNtxgtN4buQUuoH8ju+zJoVt0iLZlSBHTBYFCq3+GNzu8GumudWFDZa/5GVjppwY+9sznwpnReki57Jy9nOEwBlz2M7UYD/c/Yf9ETC+slDbF+Rgq+t75NKx8QwKAFtm+cJb5qQUwAx1jYeAwyT932YfMHWtlBqqahr2Ix0BEb4NiSxK85bd4hXPob7ZfetPhx4TwhXgi7YDOCpw6fsI3oU7f8a0y5GGf7tu TRWc/dgp mPxOfAEpIZUKqcUvRmuwCiXzesQixc/ThJSCl7gh4wBI2qtP/eoMpFCKXvlU/Sk270j5+umPbf8yipe4WF+Cm9tcjceucqs3eWGaIiqwh4ptl47RO3pUKtNQU7UDiPwSYOnxEGfA4+fydBSNbPpcUWn2Sw8rh3GCmOLgGimzKMQSs9huISDYnNy7/i7OAo0tpnJtZfbA5nZ38OoeFVwxTWmxzlDXnL19HOGghH7xFVsMA754rSJL+3zdAZ3RQf1QfrJ0BrLB9zF3PjNfPMfQmCr5nmHTQprQAeQFzj0YA8zYq2E/g3xf8lBjZw7Te1gfGBeTKiW4Rxx6TH37YDufoB7sEEyJKCiRi+Z589RQBJ3W2iWe1Jm5Jl0aWcADZqUGlQt9l+oW7L9T+OVK1eXiYKxCsHdQkU1w0ql1Do8krDbE+UE2/j8X3acXlrq2lG0mIxpMGy3xJst52kRxAtXO5zCEV2aIF8KPQLYHOE5L93SBdYkoHs8BDPTZz9Xa2ZmRX3FoQJYgVAnuVpayJsDXXvn+slG5dHYI84C2I7+qY4mP6fi00E56t1UB5OCurgoxCTn9ux5lX4FhjIMOCDixJ8KmVevz+SHIOyz3zWXgjiZjTAUYmSMTRTrxNlpRWPsgT/HD8oWAc7RXFyasQXBJj8MkXrgqrsPZZyG0usZtIs9NgoG54Aw61l241IeNvANz0XVbz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 8855846a361b..af3c2c59cd11 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3689,6 +3689,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { From patchwork Mon Feb 27 17:36:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1558C64ED6 for ; Mon, 27 Feb 2023 17:37:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6AEAE28000B; Mon, 27 Feb 2023 12:37:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 65FAA280001; Mon, 27 Feb 2023 12:37:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 526B228000B; Mon, 27 Feb 2023 12:37:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 44C4B280001 for ; Mon, 27 Feb 2023 12:37:44 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 15F3C1A0AF3 for ; Mon, 27 Feb 2023 17:37:44 +0000 (UTC) X-FDA: 80513779248.24.D118886 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf12.hostedemail.com (Postfix) with ESMTP id 470EB40015 for ; Mon, 27 Feb 2023 17:37:42 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ltU27wOg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Zer8YwYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Zer8YwYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519462; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GQTFxYC9vfA9jf0qnv7gyPX3j3jACtIk3rCkUF/FJDY=; b=peinB+J8lGzTjuei8KTNZHEWXGSf7WUmrH2w6lusATvoHoXFePyUH90R3U0f1IvGVhS2D9 MySljc64troWJFQXebogJshdigR/MHObnAtZTkm7E0h9LX8rCF/6aXmXaMkdM/oBnD7IUq CvjtVC2hsrOfsuRQMzEI2MJ/1xqVq2o= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ltU27wOg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Zer8YwYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Zer8YwYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519462; a=rsa-sha256; cv=none; b=dGi2dVwF4NogdC2BJZroop85+o529wkXV+g5IVmSv+j0xDrlEY+lYUJLRwUwdi47Yv7I5F YrIuhpfaQWqvRWfqjEzxn4JXm3IjV1ywvq8I130nuQVektRvuWUVDzXvepxi/nui+qyhoF 2vYUK56/LJV9CJ3+8aVws7tLQ8vo6O8= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536e8d6d9ceso153598797b3.12 for ; Mon, 27 Feb 2023 09:37:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GQTFxYC9vfA9jf0qnv7gyPX3j3jACtIk3rCkUF/FJDY=; b=ltU27wOgGhATSjAgpKpsKxT2cYCgBXmJjPP7Vlf66xgvCPyy9Y+Q9/fs6W0SzgF7Jz AXhDZAKO1i4O/EUJVs8bawuizzatT9I4VypR9NCPeHEKAKGbhmDhab1/2DAQYWqdJDRL SlVcHXqIAb6AFOFtWd+XMHyyClJx99DPXWU2Oc6Hu8D9kLp3zMzECqCZ3BlXkfECm7ag vCaq44G6you6yfY7K1dBcUrSymqL5XjWSs2bzGHgp5mJNuIzmcdPLIfUbaNclxIn5eWy D+XKgbxlwvwdAgDoeqhoNS8X6MSQFy0GYxkz6nxN9qX46TIoCwJZF4rSw2+QWBYvnyX8 AYFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GQTFxYC9vfA9jf0qnv7gyPX3j3jACtIk3rCkUF/FJDY=; b=tGA93NK92jq9OGjibvIsI/XLOZ2jfR6sxJUSbJXXd9KEF7yNVb4jNwKgLDBrXjmhrl UhqYMumSHmBGtgT+tzTzTm7XxogQc08qjX4OYr9rZxsPqanXNbfReZfpuTa/y/Q470yQ kBrFHVskDyHxtAX4JlQ2PXHRfcPu6WeUccq3FBgvw3/Vjd7apWba9kzaFAFtl2dzcqui P9raAc13nWZKbOxbOd/7axyZ0McfdOO52VsfQgidhTJYywqTlztTpG/M+Wy4/1EtC0zJ ZKT9zJt80CJIOi5FOre4yNHDIlLfh2CoQmfXeZNu77L9/4JOT8wQJqBDtaqlUhggzm7z mmAA== X-Gm-Message-State: AO0yUKWvmKp6Hxf5U46Vpeo54FcyWzkkf+XFWnG+7NZ0zyKVA9nL8F6z X70pFXn8g31g84IRwbB57ELNiRS6or8= X-Google-Smtp-Source: AK7set9z2RIB1+MnUTCI+d6tD2+X/XlUvgLJksvCvzE5szG4B6uXrDxC8Bbk9Da1F9i9cjnu9YsjeU4IwLs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a0d:ca10:0:b0:52e:e6ed:3094 with SMTP id m16-20020a0dca10000000b0052ee6ed3094mr727498ywd.532.1677519461069; Mon, 27 Feb 2023 09:37:41 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:26 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-28-surenb@google.com> Subject: [PATCH v4 27/33] mm: prevent userfaults to be handled under per-vma lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 470EB40015 X-Stat-Signature: coe5c87dacc1o16se71c8umrr5bjynf4 X-HE-Tag: 1677519462-891358 X-HE-Meta: U2FsdGVkX1+t0/yzyUhB476znZP+r9T3GW/MABmqhbD+RMYUmp72pHOKZ4n2Iuxv8pKNvK39yZiRhEqVzOoqKN7MlVn43S3Un2en7Wrw52iH3h3DNF/t6QGKTjhDTUmbhzirJdn4cAZoiUoaO2P3wHC8v1ox4fEPH8H40pJlKy43Mrs305l6raRg2/ehBUnrjoc8nWVfjpMMGHUznXYiGHKr2wAkiNgyL/crHo/xlPsccyKZ0WWe6ZFnWAggTRL47mejFgJB0f1MGfLXM94ZjORE023ioRFiUiUbPok2a1IBu6ddh5bIMGkmKCCvhKVob0sYOGzLkQtVePSqp5knflK8iWS1f64VsjfRzclcQxHsLRVwmN56Uba0zHDPLfPnprlgSCWr3oYIVL/BYgZMLXgVpVZZ/chqQ0P3D6KN1rwS6r6PqHTBNRJa8G8C+W3ixAgX6NHwxEXp/7ljJXRUu2GUV67rwjkOg1PADY/Z16xjRWnp91korpNg1kbCCn2esDcIATbigGQnTYvQlj0fcxNs4ZixP4ufnNkTHLgy5Q0mEaP94UXq37r0d3MgFdDFL/5SDPqEQc+s/4wYGIE9KZE/z2LyxhTigNAN5050zixbhjb9an76wfDqpOGOFfzY/I16cO4FDVH6lZdEA6JaOPqMjq3KC86n4XkoRdFizvIS/msuexx2GZXBM3H3JXrSavicMVYONdw4zG3Gky3/qBCxJ4HZYPQbYwDXXPdYVXYnFqB0k2ev0WTpSSCdAE85o0oZlDHjQ3dsF4FIstceJ/0gaRopBiUqX96P6PNHBMF/U1wx0P2sbeabTmwOMlT1Wqt/HsYBWXdrS9vOYcx7/4hZh6/nSuVLPS3dv7TK1XiieeP/I2T+4bhtdjCbIQ54i2MxQr95iQOP62jhJgqzEsed9my5BCXtnR+4eEPQQzUKS7WBCoVkaf0mszU6/8Ye0TgoniIvv+gkCRwC4i4 BM5GeoqX ExHJ53u3AMYtlyyu4VM8ZOeWrJwLPQ7xH6cuD4ZGxGkJ9pYT09EecBnk5QH4rI+bIenbhxHliRZTJZAk9kxUiJhi/gsodpYvvCPZpYSu7bqPJyb6iimpm1OJHN3w17NPARipsX1Um+4uHQQCbvbPcon2WIpaSagmPR7U9YohDNoeA7Urn1BqpiExiHn20kmyK8+S3sgnlwfS/YWDSvyEwZRvHi6jFVe20uqquFaeW72qscTunxylh/KeEQs02EOfDxEnR2Ybnq2UqLXfOefGO9S+eJygGygl6aobk2Ny1s3Z3AoPnynDpSUwSfU7HeuBSxiqYy8nzJ6Cxi0w0nHJg6x2y97TapYawBi7yiWBcCbFMqdGmoVEaYhmMjrlKV7GgKGZs9cjdH/X14BsNood9lID1YkIFh7ULodTMI40J5V7ak82eubFVTTmenMhaqoQRNcxkkHNq9zVmUHsGpjGvstWiNDRHrUspIP0NohAvcblcULLHYwlwoVq2QIIoECC6duA6JmSx0QWXbzWFRjKFUKKDTJ61IOOd7DLN5bjpJmvO4iiFAnc/aNesW0PaRqxhDwm4t/akzD8dN/uIRn50nDqovKaJ/QtGywt6Xcnfx44rsvBh2HXtibGKvEn9GtT+7QduiCsxFHXkJPn5H4Q6lHe0bQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Due to the possibility of handle_userfault dropping mmap_lock, avoid fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Suggested-by: Peter Xu --- mm/memory.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index af3c2c59cd11..f734f80d28ca 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5255,6 +5255,15 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_start_read(vma)) goto inval; + /* + * Due to the possibility of userfault handler dropping mmap_lock, avoid + * it for now and fall back to page fault handling under mmap_lock. + */ + if (userfaultfd_armed(vma)) { + vma_end_read(vma); + goto inval; + } + /* Check since vm_start/vm_end might change before we lock the VMA */ if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { vma_end_read(vma); From patchwork Mon Feb 27 17:36:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACB5FC64ED8 for ; Mon, 27 Feb 2023 17:37:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DD786B007E; Mon, 27 Feb 2023 12:37:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 489686B0093; Mon, 27 Feb 2023 12:37:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 378DF6B0098; Mon, 27 Feb 2023 12:37:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 287F36B007E for ; Mon, 27 Feb 2023 12:37:48 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7C4C71A0B13 for ; Mon, 27 Feb 2023 17:37:46 +0000 (UTC) X-FDA: 80513779332.23.E7F3B8C Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 869C040011 for ; Mon, 27 Feb 2023 17:37:44 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="IGWknCp/"; spf=pass (imf17.hostedemail.com: domain of 3Z-r8YwYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Z-r8YwYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519464; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UPo8VnsZti2aeY7Ox0qRgfNdvo1gTkzbfBEHZWtvw/0=; b=mJMyvFdzhzEvFtjFIiaCBMG9kivOkMVqWqIpjRMikHnVLOeF9vbqOvqlrACV8vqCrC3Lko zoN2uG9F52PE/YztmntkTnkrGO1urokynOKPWEZqlGWckO6JbNPxtoG6HbAqop6864j4p1 03bW+zR1+uBObCi+XT5CZLHWYWqLnIQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="IGWknCp/"; spf=pass (imf17.hostedemail.com: domain of 3Z-r8YwYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Z-r8YwYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519464; a=rsa-sha256; cv=none; b=tv+PVppdXqcPIyxsPR8Le187x+w21guyVgfZjLhPz1FwF6qUz8K89/xsbvf+HdfubGpWZX zJ1nixUtN1q3A5tcdXykjeK+GmcHti1BV62Ap+ej14lMxq85Rez3rH/JyWzO+hxHjuTc3/ wHGUjkl6V02LzGj16owHf+BAlRavvhc= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536bbaeceeaso152752677b3.11 for ; Mon, 27 Feb 2023 09:37:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UPo8VnsZti2aeY7Ox0qRgfNdvo1gTkzbfBEHZWtvw/0=; b=IGWknCp/kisFoGwJNllVFcIXkGMt1odnhFRD9r74JJb+hle3qs+rysH7Cj7LxfSYl3 UW6Inlffa9zMrz18VMYTmm0mCybg9uuWP8GtlbTG3g9pftAlYiYDxCOgQ9yUPTQuNgUQ 7e5+Qgd8nKAvibx0XVaMl/6g6MRebcr67sOkD0X7DIV3NuMKmZV7fQCQn8tLNb6YhgpG HIOQasloSO/5ARX6XcV70hiI9ZtHPTCNioAp7z9/k/Sfdb6q0fhN5YCiCRhR+ImXzmes BXS9C1iWs23SeFAwT6rFElobGU5I0Xf+MXK56mAXr/Sbo1qNfZi6LCRNAeWAjz+RO9FM dTLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UPo8VnsZti2aeY7Ox0qRgfNdvo1gTkzbfBEHZWtvw/0=; b=JZZ4uHCdGHGkrLYbuokbN8JQJwryS9N2fa0sdEUzmOhb/QS5v5zgaWFVIY6ueHFqZ5 GJexbLwU147VIHKokj/gF0+8CSHQmTym4yfUVYBq2tH0FRcsgZpBhUYAYkVAM5BcjmjY HyZYuLg3R14da1fgbOdfaSl4xkgUNzXSFDZSFrWe5w80BQ23I1ZgLLL6cG3ywHJxXrfS wvSGudog9Yn2mdlC0oxwdb+ytz6nTaMTVmvqjXwELofE2+dUSMIMT6NM153OQ7pmeHbK k4/sqimVDHFloHdNoM96hc5xds6d1OX1ZyUzYWJlfYEyo7k9+k6xTNdB7I73SNLNAAmf dxkg== X-Gm-Message-State: AO0yUKW5KfM2aJgPAhE9a3HOOUi/0qxnwuGgCKqBvlJAmI6geb5bzK3i Z/w1b8AH2xFY/RrWLzALMVqvUtzPzqE= X-Google-Smtp-Source: AK7set9Qr99NowsWp6cjcp1aWhoq+Zdc9tJrNQG4gE1aZ9h1lXPmI6kCw99Bwq0EMQ8NeKIO2QVO54vN80s= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a0d:d002:0:b0:527:ae97:e8fe with SMTP id s2-20020a0dd002000000b00527ae97e8femr357186ywd.42.1677519463716; Mon, 27 Feb 2023 09:37:43 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:27 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-29-surenb@google.com> Subject: [PATCH v4 28/33] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: fesk1mksurn857hwbwa49zwpdn4u1i39 X-Rspamd-Queue-Id: 869C040011 X-HE-Tag: 1677519464-562775 X-HE-Meta: U2FsdGVkX189VMie2B+pJ5Pckxr/3lpeFEJrOdq9dz7cO1B0SYrP03mG5CsmdKkvYW13bfDUZBh7bt83TwnvCq9Be2NtEsMnPQigkaLsqxfpPU66ha29JZtbbvAqyH8OK+UgyJJMPUqnrk+lct5EZ4j0y0Y8UbxxuJgp2BZK2YLlov5fSn26Xn39no5LjldxKRYJpJA1cUE1kLfliW6IYjohUYRYooJv/IPpWiFY7jDMLRAqjBYLCufBaWDO4M7hndJ5+6MqCgECxlB4K7+olfUCpcTQusKCDgFtpjZuBEMGOirPh5xf9nPvnkws2sNIYQdkXjLxPEMWiazfjsBpR3Oc4E0lfeKP7atytaBsWM2SkT3Bnfs4Ve6RF8X44vBNo/bYKvLvuGX4Dbh6GQn2UIr8GbmbtIvXmelvGzAHNE/rYJwsDiYFsUDB+V7PO5RFIJUDNfl6EBNJMzc8pjTySEC7Ym3UJz7BGo3yDUp+8mxHh33bAkj7kBiEwEF6cVCLzMGiwd1QJmiF2AyHYr/S/U0Pw/gz2vYjR5BylbeExYkHpnmie4wzG3KYqz6FNGd6+/qBydUqaZoEqJ2daOR5/NoofwnkHYfpul0AfYtgRXppNfa0vCrUsFDeT1hIipYTEcsTENL9H+MnuRvSMzVI8NJAWkkw4BR9DO1z/9GZUMlo0Mf9Sj6itx27HNcJNYU/cSR9DVW1Pob5wu8uKk+o1HOQ3+pm0QKEdRQWkpM1I5RHFhrH2xtOwgvJLzcO96Y0oegjF2aCRV7Dn9HUOEIrzf6OcgPb6XFjD1dAFBgwvbyoxqZGDwrxO+0F7hPjRxQJ2TrZD+kdyJrk9NscBhUVtqE/4TCD+ocbb77Hq7Pz8iBvFulQLDy+2fjXDn52L3oATDTAu2IsnuVf6HujvAzgBzGx9otqFd9xbJ7pPfoWFzvW2yThQJi6g2zZpvOWSe3UmKHKC+9NQuB0PhrD5lN gLAY7A+D Lj2PvRc2pNv9enbwUiyZJUUuLIu6WinvKcb1gZ5+NJhaTSAf9SJbUwwdUnn0Acnf9gyYqCg7XPiixxuBVXoR68JGnNkNlDmlAVYNA6nKuTWW2NL4avS9VazJfCWh+1/v+9Y5m0n9m8j2/WOedTtOkgJI32frTY5EbnWZd0g9xmmLNejvBwcsAb/CcE7wWJ21YvzrTU+LR17brby0HJysHpwVFbgY1O3HrYCFGoe0DmtTwkuQ+8viKvcAUoAF/KPERF1rVBVQ9eqcHFB+YRSZq+I/NmhP7fhBp7qm6qi+Pu2uJBEamqLOZmzy0dVF391A6mzAOZfc1xxdt1ax82bWuKQUfhEAcQBn7IBEPKnRFLiN+4JrIkShQ9hX1CL2dyXA9dI9P+fFQa8nKVp4k4vykpapBHqYWFit6PAWGijMvYumsfhUYZaDi/SYQ3jBtQntqMbvE63mIfvT04OWeIMG7qPPz9nO8DRpMzNhQe5cJlOTMgPWnLamIk9DJIB8egw2HXDVGU6feNCdh88XV61bA4Am5cPT6rcFAPZ6zzt4gClOu3PaU9Kfo8OB4P1qjjb+RBDedrmffjyYOE5jsFCFVbra5EEZjrRehCrRILUi6XpaYFb3oy6v3yqmCOw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 6 ++++++ mm/memory.c | 2 ++ mm/vmstat.c | 6 ++++++ 5 files changed, 26 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 7f5d1caf5890..8abfa1240040 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -149,6 +149,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 19cf5b6892ce..fed855bae6d8 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -125,6 +125,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_tlb_events(x, y) do { (void)(y); } while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index c3547a373c9c..4965a7333a3f 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -279,3 +279,9 @@ config DEBUG_KMEMLEAK_AUTO_SCAN If unsure, say Y. +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + default y + help + Statistics for per-vma locks. diff --git a/mm/memory.c b/mm/memory.c index f734f80d28ca..255b2f4fdd4a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5273,6 +5273,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, /* Check if the VMA got isolated after we found it */ if (vma->detached) { vma_end_read(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ goto retry; } @@ -5281,6 +5282,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, return vma; inval: rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); return NULL; } #endif /* CONFIG_PER_VMA_LOCK */ diff --git a/mm/vmstat.c b/mm/vmstat.c index 1ea6a5ce1c41..4f1089a1860e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1399,6 +1399,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Mon Feb 27 17:36:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153992 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 251C4C64ED8 for ; Mon, 27 Feb 2023 17:37:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C04886B0093; Mon, 27 Feb 2023 12:37:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BB52928000C; Mon, 27 Feb 2023 12:37:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5591280001; Mon, 27 Feb 2023 12:37:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 96B366B0093 for ; Mon, 27 Feb 2023 12:37:51 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4DAB2A0ABE for ; Mon, 27 Feb 2023 17:37:49 +0000 (UTC) X-FDA: 80513779458.22.4E6772B Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf14.hostedemail.com (Postfix) with ESMTP id 7AF2410001E for ; Mon, 27 Feb 2023 17:37:47 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="S2z/SSZH"; spf=pass (imf14.hostedemail.com: domain of 3aur8YwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3aur8YwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519467; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8qbswzDezEhzstnVvKcWG8k4mqqShtR5CvkIjaL/Ams=; b=cyrPF+vGgveDr4aLv2OgQ29jxs4uCyDQ6grJUgIY3YWQrJMCvzNn3ug/JZ3JbdDRW8cK1K K40shIWIuaKEt7rQtrMSSCB3y8mqIgwUtlnGYBsNCxzkfW/66JwZVPWGB67hisn3Ye/gyG 1LAsaeXvZrlRZK0FdNjtkOwxWo08l2g= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="S2z/SSZH"; spf=pass (imf14.hostedemail.com: domain of 3aur8YwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3aur8YwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519467; a=rsa-sha256; cv=none; b=Zg6lsD/ihbYjyDE2p2UrCWWMvSP3Wefu06+20dyNWk7F/T9TzNjte2oD9q6z7KoLbaoaRB kKVjYIYxqskMA300pS0fYs/zGWN42aYbPVv5KJjJahYZL0PRZClhM0gDJBWM1YspfEclE9 MbpmQYNB/PvUL6C51Y1BPGr5g3oe0U0= Received: by mail-pl1-f201.google.com with SMTP id j20-20020a170902759400b0019ace17fa33so3919324pll.7 for ; Mon, 27 Feb 2023 09:37:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8qbswzDezEhzstnVvKcWG8k4mqqShtR5CvkIjaL/Ams=; b=S2z/SSZHK7ga7fn7K3WxQ1Z9Tp5cfnBCSQx6U3NwmV1gsUS+bGmINQRxa2j1mxprJj TlUS9zejozFI94Me4SRmYnZJhHoCJ1T/yWkWmu9QLBmKTP9UVKEKrUis4+6aHa3Th3fJ h+Y+1Jmq5YaDOU6wdiux0RRTdqhbgVjK/U97ZWeloQvvOddB9ya0oDdfoI+0RwHp4MBY fv4OUGk3ZqItB135iDjzRMrjgTvWp+7rVjVBq1LRXg1IvF5Kyuov7vPWbX/2nDhfcOko v7S1G6CGTLyAoqFG7bUJ2+HrxmmeBgvMQ0WZGMska7TydHy4EV5ohTcRFh6YPtqaKDjC j2tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8qbswzDezEhzstnVvKcWG8k4mqqShtR5CvkIjaL/Ams=; b=lwPpvifmkOXZYAkUwBxRiEvpmbCbwx+xppw5pvfrva9rgWEwjU9npZnci4CZxhj+RS upJqTeq2nSdykWpatpgebSkWmoxN0LDtTE0W1qgLB2+bUeY8D/qxDGIfq+8EQDiQXtT0 /1Rq4noKSuRkMnK5oqItWIF+/AMZKZ45KPwR2VCb+57oBscJSr1EPI9YANDI9MgCuKIl /iCVDbAh3U6hKprdIdJwWYmvia3RJbnp+wz6eUhoqe0zhz5aS4wYFUbv95eqbtvkS57N DgtYIkT2F0WujHdMF3drTe22kMsLcnda61iAGJ7u1lpDVcxvtl0VQCLUJXGx8DTOx1qt o9jw== X-Gm-Message-State: AO0yUKW+cJ+0ouIuKwp9IpQINFu5WEts4f/2C8NA5mlmvv8MyI/UBWQD xOzs0Vq8cAwIA49lHw8FWaINXg4PCxY= X-Google-Smtp-Source: AK7set9keW13BRbYPHEfojmqkqSUJl+DCAHGSXNxIKLRjiYP92gcRxEVFUNpHyaWkuSA+coR6ukP0Mdxl38= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a63:7a56:0:b0:4fb:b88f:e98a with SMTP id j22-20020a637a56000000b004fbb88fe98amr6296782pgn.7.1677519466345; Mon, 27 Feb 2023 09:37:46 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:28 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-30-surenb@google.com> Subject: [PATCH v4 29/33] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Queue-Id: 7AF2410001E X-Stat-Signature: p9z7hnuudk4xy885oxoyxhhgdwg7qy98 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1677519467-86738 X-HE-Meta: U2FsdGVkX19BoslYCvaWs3vgaR3SjoM1gV0Cqumom8NJoHZ5iVE0dpN9DZS3hKUnZw9dz0lWCzaKfTGF7a+3rtPVhMN6sjD6RVOttNu3Jt4YePYTRX1KOoYULKNPkIB6ZwPNasNT4xnLTLmCpfdgyz5HDeEIgVI2cFIukD9eW5May+rmOWfn0c18MV8xeUViejrfJl/MSOo1LjTddwLOMWBo1uV2oPNWpfhz5F+lBR4LvhLTdVB/D8hc7sL5GFfpa6ijBZp3/dWpVaVPKv0eHefXzRc1/nJnisLiwJpsqvBA0mVRxNIoT7bPNwA2urawXeVdgrGRdS7fl6LRLJ+p6DgN/sA+XwBYTqtCcN0rZvOm2z5ikptHS4vvIbisaLZFJ4bBMClojxphlgHzY4NyxcE9Q/F0S4bkHgScSB4QjTV8rd9rBwh+HWp+lRPci010xca9Hu++YbgZ+KhwDYXTZa/L9xi5cIHcNYjYYZNnzJlXuhbpwdt6wQ6M0noEN7ePeGit0lw1Kgty+Ex2GO3QIhidCStBnmJWbQbszfdVb8Hj6phYZ3Oec90mJAK0C2rA/Ee27E5PymjUbOyzUsvIVzptNklMa6uK5HNK+W8ZWqytETLHcwE0K024QfRSM6WwA4lTsI5HE5+qD6ZJL9Xxo0aRgBFzi0a/8uSZ1wW3clu9TlqAWBc9gbwqr0XW1mR/N+Ma3SuY4OtfMHxa4/VrUi2Nx6u0+r/88cv8RzUiTHavXHaoxzWXfb14m3FQsVRpB0amfUh+/mUdiB9ECZVXR+ZVjRuB3M2+eJ+F4xk0KPuYaLHtEgLxoaC1XUSfrrkEjxd9evn9NUbWUKhmvXZrUqY/wMbmlaPZjPSCk6/Qrtqtc4MUF2A2+0H4V1irYgYxS2JJL4yqTmZb8gdjrD8KQLFvfdWyqlIufhAdBTv/+6RBXpDQzXla01e1/oztW7GATPUUSCcOWQSMHJd2Hyj YBLO75kt hVm9Fq5C+3zqVZzUFzCZgM4LgPxA7hoygX7cyMiAbvtpVrrMsjB16i0Wk/6taABvvkkWoZiELWIvL+0ig26m1yiTN1Pts2QhbgREQJYiyWse7Ft2Tcy4RDczmhK+JzxkSlOpCumSSLAc4FkkadvxZnBi9O8QMJc0NCF3MqT8T1IgozoRWNZWrBjYVBmn0fYaaN4tHYhOvTE8M9LEAb3ScgyRDxqB+uCElKvWqKlZNYh0sT1/FG2AgbgBSz2g+PMi4DjzC6Kd8CQGPxO1G5aZyctuALVfRqDyEWJri03Cr/NIEkpYvoVpvErnF311JRCn+VMQsUrBSfYsWZpIvt3qr3mft08GAEUgDYH0MkjRhVwudppj9XKfqEjJ7dHS7GaKVEYqMunukBJ9ZBTR0ioklNCbxthtwTwUEk6C8awc6TnmTGVMNF2CasmMgi8lRQn5civ3SV7vUVN+iUFlYkmiWzmJ1bHBfzBlhIKfmZhMKMGIrs6reTBkn5bdSnyMrwMqFsuQ0Jeuhbwlqu/bYAy6P4AA2RPDH3mOkIoBJZMZ7XbKc4gZI0jgh/JFwZOj7B6IDxunvaj0rz2GBgBsmjqyZYP4hJPtiD3C5uTeYsTBn7JXfALdgl5ruiaPTEGjEVyAWOsoJc+6ojvzcQQnYbPjXMvWOYXG3RYM6wppZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index a825bf031f49..df21fba77db1 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a498ae1fbe66..e4399983c50c 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1333,6 +1334,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER)) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1433,6 +1466,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Mon Feb 27 17:36:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153993 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 556C9C7EE23 for ; Mon, 27 Feb 2023 17:37:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 130586B0098; Mon, 27 Feb 2023 12:37:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 107C128000C; Mon, 27 Feb 2023 12:37:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE894280001; Mon, 27 Feb 2023 12:37:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DF3BE6B0098 for ; Mon, 27 Feb 2023 12:37:51 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8ADC540A38 for ; Mon, 27 Feb 2023 17:37:51 +0000 (UTC) X-FDA: 80513779542.01.6AA25F5 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf05.hostedemail.com (Postfix) with ESMTP id BCD59100008 for ; Mon, 27 Feb 2023 17:37:49 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=axT7gIHS; spf=pass (imf05.hostedemail.com: domain of 3bOr8YwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3bOr8YwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519469; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MLyuEeEynE8KONkERvpFgsKF3fJfrVlAamWKGgJMnRA=; b=CclnYsyXijp7kETvCnQGgh8geGOcQz5n10pJmYzOa5e+wG5C0HdDIW1GlVKIwHkYD1yQU4 pBSJlWY2CQU9OqYXikJ4+UXnMijGov8BJnCzBU0GV8wfm6DN4zVHk2gD0KI4HNBMoemUos yTeXRcjkGIrxPwAI83DzkV5ni22Ui8s= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=axT7gIHS; spf=pass (imf05.hostedemail.com: domain of 3bOr8YwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3bOr8YwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519469; a=rsa-sha256; cv=none; b=J7fPkqqxejzN094VU4D0WuAqaMCrGPTS3vEQ+/IG8ggr6IpDqEavjfNpxSb49PGIQOyRnS RhOekxk2P9mmBSrkqqgzmH3gh+yoGHpx95B99eIhQZif7TQrUHh16hu8CZy+byb/c/tZJ+ uvO+GYVE7cfvLka4L+SCe8LD9IlEJAE= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-536b7eb9117so153613277b3.14 for ; Mon, 27 Feb 2023 09:37:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MLyuEeEynE8KONkERvpFgsKF3fJfrVlAamWKGgJMnRA=; b=axT7gIHSM9vrvAhwKVjKksPBBorC/hEiqwRtJY/ABl81gKVCntmX74iTRhXvXgAnUV Um80sGKa9BQlweiuzQTKDk//t9klBC4AuVFnP3CTk2uBlo6XhUaWjWY5Blg8c/od1wB2 kGKuk0XbNvhKWTaK071eLxFocsyNDE7thDMFx60EqbJKuNaJXP1uEzxTROegPPIy1HGi /hu0FB66yrsQ6uwQNFSWYEafLSSxfymT27BvoyNTtd8BgVaAOG8u+Vs4WjN1pPd4fScE c5yJ6zCfiGKeo2DRNILtL3E3ThjPFtXt9gJX7nMmqjJa0MWklX0CZPjvb9UiR6pRtDTU u9Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MLyuEeEynE8KONkERvpFgsKF3fJfrVlAamWKGgJMnRA=; b=D1TdaOIQsnfq3XoLoQuRAwIs/fQqPxTBOgsObZru5FBnB5FfWlA7DHvAYHaFZRoOBM VacNznoS4e/oNcgsiRUdfzoTjRnZlWxFcV+ic2I5DdUyVhZKp9AnJhVl1PvZjxnTwzLA gehXVaLSwtVhlHhMBlf8hZtz5J5ZGCxMhraU5cV7odKpd/rSsoKKaVsyC4Fv29bNZOuX B20Sha1vmd/uKxmvFy33mJ2oH5sEccD8KUr/0J7Zg8CdNkwOY3w87Dz4LJCGBd1q/sue rQ1dEew25peJvqHRcEb1yFd2/GrpoLIf1KgepUOYJttX/LaxM2fj1MCeKfs2I33Q0HUZ Lxow== X-Gm-Message-State: AO0yUKXJUlFYoPv/h04LU2wSC/vNITanL5dD8hnc3klcKzZBN7f7jcND pVvFMMTBaWAASO2lxvQw6dKkQUqVgGI= X-Google-Smtp-Source: AK7set/sV1mOrmQyWkQwKfhY9l7Yn97smkcjzbwoSpl4Lx6r+hbm0S79lHOvGLHJAd/kE6zZDkM+t31MhHQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a05:6902:140c:b0:88a:f2f:d004 with SMTP id z12-20020a056902140c00b0088a0f2fd004mr219395ybu.5.1677519468856; Mon, 27 Feb 2023 09:37:48 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:29 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-31-surenb@google.com> Subject: [PATCH v4 30/33] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Queue-Id: BCD59100008 X-Stat-Signature: iknjcuou4qyp6yxx4znx1suostnut6qt X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1677519469-145888 X-HE-Meta: U2FsdGVkX18wUL9MU3BLPwuxg2NrUuEbkjcOY52qReR4Q15xyDh6ydLQUTI3YGMrx7UXC9Rv0Mlc3cbnIUyRsZLzfNn0fxtb2Qj5SxazJpXy2f5GJdHi0NyRRhU1HNixGX9cvaMH8TATG1OuUOIQGGTZd6qGBOxXiPE2/qNowJDwZcCNjVzPR9BN/4QOAfgyqQZ0R6PFp5qEmd10Iro0hFEDszVvnZsOlieAnMXQIAno3luxAs4srkUJfjL5RzbfUZ45vGUW1EGWPPfTxBDPiTZf5FkAk3iTsyECZkR9s8QQn7ec9EQuAriy4UmM1MSlOXXE49mWUn6WY9yn0oVC9UGv6QjHICOBgTIcKhq+r5Z/nGpVoE14Ly3fh5FKsUNzovyCtwATxBsmOK+SR4RM2Ph/G1ygtQ9WK1GsMHqcUoVnNIMYPcZ0W2s6MR/kiyyWJtp0HSadGW8qszkHD2YBCWHzQ/Ddl1xfzDgC4nWMOiHjlwXfege12FqnQWBNRx3C2GQv2su7yBUNzhDJV7pLRauJs7RiygR2kwlkkEwfcqZ2DByl330bhtnZ9+iIxJ/i5Qo/mx6ZkxLZc5rSrIp9FmaAsYI7OPonpvSRn7v7leuHIphEKeKELpWOGRH386ftSHavdKMw+wgyDYZXYmKDxqmv5ZlJOI35t+L2RivN4XWkbDifrdfHj2709WvUWTw3XWNU+GgOW8NkkF5xRFO0sTWWuVhBG7Wt1KukrB/DR31EnTE3MU/xzGXBxchSK6Tz0bD2GGwu41tYvgEBtY8dFD4136XCmqCq5I1VxG1ZB/uz8TmrDmkqFkqSAZLXd37dsM4DfJtN0/d09YAIl0Tksx24MsdNhrrZ/Ml6cKJ9myPPvV+ZdRs8Tb9Lj7OKx+C9+UGEJ/4GPEyGqybFbgjXytbniazRPDh9LDS8BTcjnUV3DCjFj3uR5c77y1UAZ+46yGCHpsPPX5Qxj/VlByS vif1md7x LYLaAvBfSjv+l/i6iSuXtR84wHKwAUL3mNhdbA8fQcNRbb779Xq7AdbN4zhnuuV6P6p0gxOpkPrTtkQY8IGuohKSZyE7zyaQw17a9JdEyMP10UfSELKcUDxpyvzC5dbL/nBurfthilTGMxnbV5n68P2qo6dPyAZCk7dSYS5z++zoECRHx2TSZ3o97gOk1ZW98vRSxD9T6QgrXpxeVSYYiUhxX4Y20Jr2pfI8a/KI0fCOKxT6TsJ+pRDgiduTaE27xmvWxyegylTRXSV6V5kzk9cJJHWXIJeHYd9JZ8l88rLz/pV1fc/XDvwbfP8dNAK/X1rH3jsNT2iB/5z8okzUqFdTlpX9yHMMJPnW+r5r8n6zmZ5mnxOnqd3J4nrVbhbmy8WG8Z2KWsrVkJ0EFhm0luACDrELgG962avLW9/8gH/TbXf7KcG6tMjp2Q9ubaj6+7BU9tIqlmpBW0XyiCeFlucYNKngqQMKhJRhzrvJgubYZX/Hk0F0Bsx1pcfhsICS1gQOyZkzMWQsDxhLZx/5mzKX0VO6rt8OVb8ycvlt15TewEzD5OKDQdoIC7xrNe1WvtKy4Cu5eHBGJIbYhtDbO2OAO2LwqDPtqh+jtvrFcsiKK+tw8Xe2NnXIXnQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 27b2592698b0..412207d789b1 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -95,6 +95,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f4cb0f85ccf4..9e0db5c387e3 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -535,6 +535,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -585,6 +588,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER)) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -628,6 +661,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Mon Feb 27 17:36:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BD73C64ED8 for ; Mon, 27 Feb 2023 17:37:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 54EE26B0080; Mon, 27 Feb 2023 12:37:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D8AC28000C; Mon, 27 Feb 2023 12:37:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21A15280001; Mon, 27 Feb 2023 12:37:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F3A956B0080 for ; Mon, 27 Feb 2023 12:37:53 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id CD79AA0ABE for ; Mon, 27 Feb 2023 17:37:53 +0000 (UTC) X-FDA: 80513779626.06.B7A392D Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 0C46E180019 for ; Mon, 27 Feb 2023 17:37:51 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=lbYIqJZd; spf=pass (imf16.hostedemail.com: domain of 3b-r8YwYKCF4OQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3b-r8YwYKCF4OQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519472; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Hcfd1e1CYNsfSesf73569pOSKya0WHju+cd+LlOfzq8=; b=ISYTT7DLSjW7ECr9+97U3de4gYMpvKBFNOaeX448zC/DvBrC5Wzeh+7/bvIaVa9Vntqhrc gGSjbWsZcNmrYPV++v786VszzvZMeLdIxEnT2lIv66LAnWG3pw7W0YCemz+ecTLSddsC0I NGWQsyBqiIYeG9/rPr1vtETqHe0WNVQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=lbYIqJZd; spf=pass (imf16.hostedemail.com: domain of 3b-r8YwYKCF4OQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3b-r8YwYKCF4OQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519472; a=rsa-sha256; cv=none; b=5HiHTb772c1cod6WdwyZeEWoIcsYPPGRWTzl5HATIAzvjLP2hL6Ux067WxHA5UJShKvfVx /mBa8WFUMKm4CDRs4SVD0Cdgj03N1hm5L73MOWkoiUnX96yuq2TzfpVCgbZZMNXUt0p+Gh eyL3sb5vvSRJOl0Qh0OZRmwdcD0nxxw= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-536bbaa701aso152756227b3.3 for ; Mon, 27 Feb 2023 09:37:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Hcfd1e1CYNsfSesf73569pOSKya0WHju+cd+LlOfzq8=; b=lbYIqJZdRbbdDZdXc3KPyecDlcIgvcbfb+DcxfEk3pJ0tkCKREu4kmt7prlFLJ1+FQ T7SrZzz7AESmHbsebVB/dcsUJ3sSPqxUDap8J4TM2PMm/f0fKj2d5agwEF1dRo2i14i7 2Tm7gxYItOsxZ6/P6dejoIKGIeHHxYKwoxOS0Hzf8d46xNc4trKpV4bsx+tVItCky7z5 aUXOUsg8Vx6avsDKLvMNoEFIIJZIXtu9ciloUm8OADOlD0yqFEBOdNzn/R9lnJBdqTeu YgUEQUdg6StQbFTiY+Ix3uwxS+O8mpUohpFOuENl70pA6HrQlHtgGjdfJeJSJ8v+j9zf qj8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Hcfd1e1CYNsfSesf73569pOSKya0WHju+cd+LlOfzq8=; b=gAegZDvIUiMjR8C0JxZ775Zv7mEbA4CUGe8ULJnu4UqQvnJqpRaGGYfGNXlK4s3yKA rNlQGO8KB96D1r14yMLQWFx71I1uvoywXUQpJFw9ROaVGUBe3PDE58IuKcdVoTfHUuSo XiW/U61ODRCok/8n/xuGgKQ81b6Z04YweY5JK7pXD9NlRHMxruv4qqaKXJZbo4cdHSkp oymV7PgR/h0X6edBJaRe+CYj7DPdg/FsX6gy9I/8GcSKoseM9FpCTIC/yj6guOo1as2o +0mfyaCrMXhnJhvmEBjrpGN3mXLEXOBKPN4NrlLDK1HwsLBLOacijzNOqh0Q1qld/IZc EFQA== X-Gm-Message-State: AO0yUKUUgfZ/5y3xx4iwOgJCuq3ehZUzAcdWygx7hg2rZElxmWqg9OK+ CcS1/eySPhRpMjB1gqU2idjPYB/vk+I= X-Google-Smtp-Source: AK7set/pwIZDrdxVaNh2CuYGKu4UYoRP2oxQl2zbmcloOQnN3ea3mqmxriAC+2y4Q2nBddYhpyeOIu/z0Zw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a05:690c:31c:b0:52f:184a:da09 with SMTP id bg28-20020a05690c031c00b0052f184ada09mr265273ywb.2.1677519471088; Mon, 27 Feb 2023 09:37:51 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:30 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-32-surenb@google.com> Subject: [PATCH v4 31/33] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 0C46E180019 X-Rspam-User: X-Stat-Signature: 3jq4pnz8kohsph5bzw6qe31ges1x764n X-HE-Tag: 1677519471-176902 X-HE-Meta: U2FsdGVkX1+9mjr1tV/2/He2iq1BTXEHDWh7VnGNyWKYbQ4pmzXZl6juOEUNv+10heqrvPVK6Xh3uwlYNJUt3Eji91JBhUaECW+cdHK2wGdHkrBax+Wk0toO7nmJwXEvqIkp53BP9Ev7t1HZFIEMUFlVJDv2hgVk6/u4MerCqZIZTaDfH0D+h1fP5V36ghxcDoNnnvhDqkeeK+F+QVDTL7rTahUZUG9EyHkqmvfvev3UWQGyw+9G6VOxU1Y+KN7eQ6pGH6jjolZp5b8yReUV1cuxX5J8nqvjjt/+pkNmhWXXT9pOPBYGrmIUraN9rpX6VvMjFZjTfPimFJMxyfvFgQzJheIIpLmWzgRNR8YypMnZ8zDsujjcgqQiEnZ1jG4ErfeVEmXKCkCpMaJkIIZN9xE3eQjwpaqqYlcfV9FxiRR4mH/fsC5xnqRBQzF+ht76wxtSv651gTACq/xQIn//RxYteZ6IxvPClUXqEjYasnvWPk2onbIH0aeSjhjPId6RWE+W1LN8Z5yfciKp8naEERYwACM99NS1YjszYNOiLRZ3s349c9a6d5RIX5HUHtPcoH4JrWM2bv34irP0YdO4GF+rmDkshOKkYEf/jAa48ENrSDTdpkYbhU2b6Z444qDPi37dp+hrL959yMypAB9fiR5oUq8JV4N53s4rPpKD11w8jHDFSUH7NHNDPOasVvhhWn1Kv6Jno+kBfQtr+LL2ONtSSmdrXAAVha9H2AyryKtxuEQUTJU3UvHtRk7/YQ1HgOfn8o3t6CKy73BEbTsW3TZ3zEUFz3XMIMmmLyDvfC7bQNlYUY4pfvnCr+1lFg+svDRyQVOzdUf0VFBv87vY2uqGgrEIloJ+wWFUAXZLKFlQMSwksxtL0fiPAxgT8gmtlpkItvu56lekPKmzwjjWL9ZvINqu4OpjLAk4SZyoYrMLDkhgMQWHIeYXhzuESRYkMvI1y7dsjeNP067Sc5i iXMuwbJR jbCzHn+uK0/t6Opq1cCvIwOx4Os8HRncYb0yS7F4iWf/LR+2YtoU7SvjzPZVmFI7ambtqp7wvUl/qfeAHSbFnxUVueIIeYbfABZf85RrjoDeg7I9sxbIJ76oJ24eQr9RTShLPdU933iqUDGq6Dr7aI7ExZiLvxQTeKKKRbvanS4hX6GioSSqPyan+b0MRzfJRETqbGd9pgENO4mPwh/rCYuDwlZZE9jJJetI67v1CjqbbrT4DQO8NLcNJi1xzHmBWQzTOvHULMq8kk4kAJA3/0oUZOAdTHhhyRBM8Asmrm8DTWmfskC0oqGUMJLkJ31b5qsg9f1nMzIthpeFAEW9ffMN4bbxRIHGxhEI5eJpBwxrKotcH/0mbOzhw1AKRVaXKzN8Q+SV+hx/X43Rvod6CGXHBLRLdlAIixakh0NE6fQJd9C9RqoxAwXSqWdTVlfRYW5jkwRYMs3VAZ2FNAdqQqXLmN8mps2UuyrgUUv8zryxvdXrf+bGM2vonTu9zcb+XIOu7AE4sT0n3tWnnL2nFm0cGZ9pk1hs9Olzghwo5ZuDyfI9oFeky5BdvAiQeXPOPE+KdxWvx18TMROLsUT3OQIFUHJoUebI0AH/TC9qNi4FaXAFyOhwcnN/eT0cP8Jm+AYSzc9OTS5cDCCk2D2FSfl1P6Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/fault.c | 41 ++++++++++++++++++++++++++ arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 3 files changed, 43 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 2bef19cc1b98..c7ae86b04b8a 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -469,6 +469,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER)) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_end_read(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_end_read(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -545,6 +583,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index b481c5c8bae1..9c205fe0e619 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT From patchwork Mon Feb 27 17:36:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 683D6C7EE2E for ; Mon, 27 Feb 2023 17:37:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2F7128000C; Mon, 27 Feb 2023 12:37:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 96AA9280001; Mon, 27 Feb 2023 12:37:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80B4628000C; Mon, 27 Feb 2023 12:37:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 63C41280001 for ; Mon, 27 Feb 2023 12:37:56 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4C20B120408 for ; Mon, 27 Feb 2023 17:37:56 +0000 (UTC) X-FDA: 80513779752.08.5A36DEC Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf04.hostedemail.com (Postfix) with ESMTP id 7C3DC4001C for ; Mon, 27 Feb 2023 17:37:54 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=AIld2qfh; spf=pass (imf04.hostedemail.com: domain of 3cer8YwYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3cer8YwYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519474; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F3L0KDnzHjfWSabmEfm0QwnSpYouz37Z4kQXdx4wyok=; b=P9g1GR+q9Ob7BzbtQodHdZeQfujPzQueZrgt+4y2eF64RTK5U8DfMyNuT8J+avQnyH0a2c OhJQVwiAdyT0LZBJ+3tYrlDquLkja7hIPCepzvPSJdSyLImvm1HWbeLI4s19v3B8Yzk2lw VpufjPhWrjmUrfFTgy2Ra+vEMzL8PCs= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=AIld2qfh; spf=pass (imf04.hostedemail.com: domain of 3cer8YwYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3cer8YwYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519474; a=rsa-sha256; cv=none; b=eVxrfFjGO6kHJy3cRbfwiR1AYNDu8ytWSEaNpOzKHuuYRnwipAiz+dak/pmmCKSnbZy1DF vt7aB6u7ckFJeXjH/Fj8TAtnREaoglhAQK+BXNf6ifiWNGDCooNP3cmVMvfvECvl3fWECB RMRoRiZDMGFIiubO/y4ofV1ObzAfzrU= Received: by mail-pj1-f73.google.com with SMTP id i6-20020a17090a650600b002372da4819eso1871270pjj.0 for ; Mon, 27 Feb 2023 09:37:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=F3L0KDnzHjfWSabmEfm0QwnSpYouz37Z4kQXdx4wyok=; b=AIld2qfh1hJ81pjBHIB2k6hHEBq4B9cQ7unRZ/ajU5zus4NStrM4vKSoC+D3xf5i2V I3lWxazmPww6pY3JnOd6ILpcq/3gMzUiBxySkT7gq+qA5zIjvIc98kXlCsV1T3OkgNoC wkB+0CjdoEsCW0SjfCSG1j8AzXTBtq+xLwxdRi9trKrWP6pzQ/z8RPgzdw1aTSBFWEFp tuWqf3fIoKXV2B5eA1r5q/mXlFY1b1mqVSiItBYfUpJVFKPT1XTPlAbMteQM7dFl9Yt7 rI/aJtYFzH6GTH9qBEbz2dvU/L39seWUdch/8Eafr+7mGpiemwVm0woR+UJD5ik1qt28 1vdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=F3L0KDnzHjfWSabmEfm0QwnSpYouz37Z4kQXdx4wyok=; b=OOpHVuxZsgWUfB+3/hm/e2lIYtbJuRzqBskWy5wF4nnVBtjm9N1kHZCgTxz7LizWJN uHrZc497jfKfroevGTWG20T85m5Kij+MGNkmdk1Ibp7wYPGGTacl7tqpcLwrK39rUBmz /Mtk88G/qKmCjuL0APdPR88JD7JSCSAJT0rv4XlkdWup9TP23AcdxTa4LH1Qa0o2yEC3 usU1f41hnXFBWcCb2ac9SJdfuFKChuG7ClHlyQJ4nWBmWX8zKrYAdTciuYVnlDQM6ofL VR85EDffl16WX1FdHA3PlheW/uqZpN4EckRuHerYwtmSldRoGGFvliYVEtczcZd1YPmc PIAA== X-Gm-Message-State: AO0yUKUmOOKOcvDVkb3/7rOfSm0y69IXmbijaa1njvCX+dOKWqGYmFSi tbzI4pYIbASTCTzwbd4L+gkvH16Gp2U= X-Google-Smtp-Source: AK7set/h2/2/QuyVN0yF0KgPZgsVR0g2a/0YsQFw2JhIAgW+zKPw0n5+assM7s0mQSWtQbHur04pQimjpmw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a17:90a:b388:b0:234:ba6f:c97a with SMTP id e8-20020a17090ab38800b00234ba6fc97amr24135pjr.3.1677519473325; Mon, 27 Feb 2023 09:37:53 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:31 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-33-surenb@google.com> Subject: [PATCH v4 32/33] mm/mmap: free vm_area_struct without call_rcu in exit_mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Stat-Signature: 3awegoiagdegh699w8xg1smsrkejgpz4 X-Rspam-User: X-Rspamd-Queue-Id: 7C3DC4001C X-Rspamd-Server: rspam06 X-HE-Tag: 1677519474-133506 X-HE-Meta: U2FsdGVkX18XBIwYen47h3ZvomU8YCb0O3W7oV974l9cLYr7X6hIhYDnPJdT5XnPMxI4csTB2H7qLg1cXDELNYtOZjHWmSPuIehOWHY4GOisAD263X4+5UQbugVItfYJupz/JAw9yfHkF2pIi11szbLG0tk9QtdLfz9zlL+CuLII9lZeMyxebhe2143bO1IAyBA1KDrREEU2PfDGCC1DyAlXZ+WlqWwJejMuPN3VQinfAHTQo/2W1JdXuzOMq1+KivGkP8vGmM6KTVMNR57nRzop1Dbo1HfU3NXjdCKAm+jh553Z0QZohnr5iDTAotdZ6NiDIY73t6/hm73qgi67TeNhDABqOIcaiqOL7uTtVVZ5ysTEmFsisW/bVOlw91paligopER1OYtH5aN+qnj/1OIj9CORzAN/fhNQQpDFPJT2YC5hMkg5OXgF8B24RUPvBhdpGRy0xURGcP/vl7FK/Zvcd+FZWsOwI62n3f2QXMEw3difXTNknbYB4jKp0XsNXnygee2R+YFJTVvv7hpSQn8RDB6k38FBOyw1DysSJUksGcf/2RtuAbxpmc7Xvmxizw3oNR/qXqjhZFPJDtPW36s0DEwhmZ4f2SqKWmbhP3GGIFHqW2sHuZ80k3Lqy7I8gdcs46+DfWXWYDxxmbW1Va/BoibZi6dVpwBra4+00H/erFZDJwjj7VjOqt6TacXFpUGfgW8tMpQNWTLrAmyUCaUULYM4hVSbsRDAWFPxHUOHIw5RE/LhZtnsiPsxT9uYei0FmEThm2Cb+uOfGFvuVP4zsTPayhMCWUKEl4ubotLeWFGwoWS9k91qQthJg9bEGgWlAE9ojWFvyr/QEegrApY4JJ0q+WFtCCBLnam9qN1toK8RU0L1rEFRzT/fioXSacgEEmBoWQQeD8b6qGsNeqkwT+NLe7lEhuW8NXVWk+8mhR6/5uDOOZkjCuMuArAMrimfXFE0a7CutwFlT0G S/mnnBq8 SaxgMkGLF9TRBVXm8M3k9qPe7ZlpmoY4S+wJgaK2dXkOQ/TdTW9yGuhWd7539hQFEnvQqppFU6I+KGG5zOTa/OtXIUrJwneazttwEAoqCsKJZGLCNCz7ycTYc8yYJFS4HCoovDGTFkYYG27qSMlzvfzHti5Z9FvVMb6NmLiEBv/lIj+eSmazK90RjwTSH9leYD0v0lC4I53X6tsDSZBomV5NjtMk+2RSF6v1mcoEk5hXxBBwd5QRu2CAu48slwjyfjjIC8/4PM17wjaXeTjrRqS92LnUuupNBr9cyYnoqwecPukIXHzeD5yYdZ6gBDgSkEDB6htFlDK3zEoTDfifyg9zNHd9UH/HdTPIszUInzfA05W3pbqL7pOVvGUkLP8bCUUsPakfqOM0XcD5zwg+NuaryH3ngyXS9lQ/Zo3nJXPAHlBIX7S4HXqzdHHcNEtmamBLnlIFAnwoQqrje3oFlhPFJn3qfBytQyYU94rLgpxks8Uayi4v1avcGHiFwqLwbEayU7pymTPF2xtV+BEXfPgcCstTMSoL8K84o5wEvSqQ11CNwmLUpC3lFCc1xW56otSWmkc1gP0gFnrrS3fRhv+00MwaHA+Iv/tdyxNIhTCk96VPuQ4UayBYMyaz8uLzUZ0C8Bx5ZLTUyfJZLER0l1NgmylEaRaPHvlZ2H9vC2tYbyZU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. Because exit_mmap() is called only after the last mm user drops its refcount, the page fault handlers can't be racing with it. Any other possible user like oom-reaper or process_mrelease are already synchronized using mmap_lock. Therefore exit_mmap() can free VMAs directly, without the use of call_rcu(). Expose __vm_area_free() and use it from exit_mmap() to avoid possible call_rcu() floods and performance regressions caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++++++---- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d07ac923333f..5e142bfe7a58 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -256,6 +256,8 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +/* Use only if VMA has no other users */ +void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/kernel/fork.c b/kernel/fork.c index bdb55f25895d..ad37f1d0c5ab 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -480,7 +480,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -static void __vm_area_free(struct vm_area_struct *vma) +void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/mm/mmap.c b/mm/mmap.c index df13c33498db..0cd3714c2182 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -133,7 +133,7 @@ void unlink_file_vma(struct vm_area_struct *vma) /* * Close a vm structure and free it. */ -static void remove_vma(struct vm_area_struct *vma) +static void remove_vma(struct vm_area_struct *vma, bool unreachable) { might_sleep(); if (vma->vm_ops && vma->vm_ops->close) @@ -141,7 +141,10 @@ static void remove_vma(struct vm_area_struct *vma) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - vm_area_free(vma); + if (unreachable) + __vm_area_free(vma); + else + vm_area_free(vma); } static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi, @@ -2130,7 +2133,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; vm_stat_account(mm, vma->vm_flags, -nrpages); - remove_vma(vma); + remove_vma(vma, false); } vm_unacct_memory(nr_accounted); validate_mm(mm); @@ -3070,7 +3073,7 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma); + remove_vma(vma, true); count++; cond_resched(); } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL); From patchwork Mon Feb 27 17:36:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98EFBC64ED8 for ; Mon, 27 Feb 2023 17:37:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB58128000D; Mon, 27 Feb 2023 12:37:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B3F8B280001; Mon, 27 Feb 2023 12:37:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9440F28000D; Mon, 27 Feb 2023 12:37:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6DF8B280001 for ; Mon, 27 Feb 2023 12:37:58 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2D7BE160541 for ; Mon, 27 Feb 2023 17:37:58 +0000 (UTC) X-FDA: 80513779836.21.71E65E8 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf03.hostedemail.com (Postfix) with ESMTP id 6406520011 for ; Mon, 27 Feb 2023 17:37:56 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VvMTeoUx; spf=pass (imf03.hostedemail.com: domain of 3c-r8YwYKCGISURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3c-r8YwYKCGISURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519476; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qWTCVIi1iYe9BSyPLwARD6DqoqWW7R5amHwkRUYAz6E=; b=IvPbp9uW1C9rYI7XUUEeSogaudydyDxQw/OKJxLfC0gLb/QVshimzTnjl02NfLqyt+r4JT BC125RwLv7KLYQb6mh1U15kL4xSIzOe1iX4ju1biMDhCMaTR8rz0fsRLK88+4aEnMgFyN7 wwAt0oSQNz7iyctxu4Qt43vcokIqrbM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VvMTeoUx; spf=pass (imf03.hostedemail.com: domain of 3c-r8YwYKCGISURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3c-r8YwYKCGISURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519476; a=rsa-sha256; cv=none; b=Yr5j4k7nS+ez7H33yG75ImE9cxOasaMRN4MUhwyHpQCOBGyS8rgnlhTxR8MFHefKQL/O0u wzx+c/5Qi3bpA7wuxfaRnA49WbZHjclZSoFC8XRpvhRdy+ngmDsqUnuJGRJQfjXyj6GQvq g0wrBCS6hudhprEM2M3AHV3r2nH8zUk= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536bf635080so155072157b3.23 for ; Mon, 27 Feb 2023 09:37:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qWTCVIi1iYe9BSyPLwARD6DqoqWW7R5amHwkRUYAz6E=; b=VvMTeoUx5wjaqp/SSzBUAaS+m27RVNRJG9xutBinKMu13MVGWXfijU0pH0JIlVsRC1 1Qknjm7glaZd1lyCyg2w4XYR8Ujs0gzzwHLsVcLBT5JFXNg2+dcUo8mzjU56oHRaJiRQ Altiy52UpcBlGHU2tonuXHP1Zm9pZ7kEpRw7jPUHYcNU+RFCFPTwsB2NLxILr5m4FxUQ wXg6Va2RO6AqeAKOrEiDyanAl+KkHyrgZ3Z09+ORgyuXVSSi/uNsL4P3Dex0so3Z5yvz p07UqCIp1qtZGFtqF1JFTZwh/gE+2skLmmiHZO8ONAXTewlE9Wn0WJhWMtveGR25I/cT TS9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qWTCVIi1iYe9BSyPLwARD6DqoqWW7R5amHwkRUYAz6E=; b=gE1elZQGrkXKaclxPo/kScycddHaIcyYWWZFo0EeeoWODCjlNmeMQzDzii28Yi+xdr WaD5YyuWWNLxayHnmmi/Yjy616FYhFDEKO0DBJtkKAVh+q2gh/4oupmp+efzER1/poQt 9XHP89hDG8MoCdY/F+CJAAOu8UiCmN43kXnzpGjphBkS6CTS1J409PXMV6SivxAEs9EN IJpEkk2paAYVi2e7Er8xmwaODLwaq0s2y+xHd1wC9upV3bAtiTMWyl2HUnFmUWBVNSta cOuudV8hKCwdQ113AS/UX7yJJy7bxPVkvN8yYkvk2mTti+9QQxpOr1j3tBYySXI59smG ELHw== X-Gm-Message-State: AO0yUKWgFxs1tUIqEz3I96BtKOq2LB/pBwILQgRYNB+9yD3eiOmdBJBR Al28q42IwfEj8YM5nLCUnIiFN6Akpaw= X-Google-Smtp-Source: AK7set/c7Ur53LQtByhAksTFTJWraikr5DWovedZqPtoMuJRVU+wfT7Y+cDjXpRHOc4B8/glLU8UK1K2Jrk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a5b:ecb:0:b0:a03:da3f:3e68 with SMTP id a11-20020a5b0ecb000000b00a03da3f3e68mr9397156ybs.12.1677519475458; Mon, 27 Feb 2023 09:37:55 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:32 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-34-surenb@google.com> Subject: [PATCH v4 33/33] mm: separate vma->lock from vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Queue-Id: 6406520011 X-Stat-Signature: c49gtrdz5mk7kac3i6ar5ifj6iq19dbo X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1677519476-697977 X-HE-Meta: U2FsdGVkX1/TNdENgSxK7JbW1Kp6BbGKUB6Iy3ZtAAFKOOHQ6ThwHGqMlYQwMjGSD0OMoh1KrXhdp1QwjO3jIlkd8e//Rf+lZHcpwipOcLIPcMjl1ReQ5pW6II7XCNwYrwjWOh5BtBII9k/5AxOasmsVGAqR8DlgrXKILJQWGIgsblEyXRu+tTEbklP/VBQ1hKzPFfutj3cNfC6LkhvZbtN7a4w2EPwVRFMLt7KCpEVaQurGBkQAp44LuK8TXdNfHfPwkYCQhAUUgJtHg2+MAPbE0+y98uie4hYY/Jk3WA9bZCptQXXYarP7wQL9KbjBE0+SebqjISIShnLJdLiC3CbaYaCT1D3a/CoMrUz/+7JWwzOfeJdsfhKYIoHjr8Z1Tolu/C2+lB2ITSACykiaACMiZd+mBB7MtJKK2gmg09FGCi0jaN6J7kuZC9fX8JxpeUSOhoIREBFylmjKpSOvvfhxmM5xsRLbB6jiQdzClrUJSEApTB7lFUJYuxSHegCMaaI6l3AOAocDD81Qn4UeM/DsKnIsDj9AXjOPoO35oTB18nbB/GCdOYb9aLX4PTL+GH6UX5x9CnuzUYTPJFN6oT7OWkS34Q8jojZmPQ+7CPPU6ooJFkyGW8gAehoSWTRxqoC4T3F2lEvCmjFKpoVbqUu7i2EN2viCmSoTf9Qgv2hmVTxileZkMMtopicO8szx2Rpe12+9LtPNeMJcbFcSCEPtBzw8OL5+iCh3B0iYe5CkVJyzCqeGTpR3r901++SFSnLdGOCgBDPqUBCsA0+5UkgRYXlwjPxZdtfAelGB5qvyQ3j4eteXMwnzEsh6wE9F/jhKCnxNiJhdaBQzBqOKOVxqJMASrz8iVWSKvgByuQY91ZSvaDCiG0n7FLUdVUJrTYEnU5ydEBBC6KwE2vL7oVlR1M49jTEv1jgiWxHNsZzzUNvvr1hLLgXHXx2zDkm+0WMGm5XcJ+dQKWS+Irw 6IlutoJ/ CTc0CAWphREnhrnwUjewntKXCKZc5F3mrEMB1+FcvuplObHotX//D8BaGFZhgHErV7hHCx+oeBALwXtXNWOgBa+FtZDtNCQCLqU3R/8pWgVMq5Dbp0k/mkPq2mV5djHJncowr4kqR68+4qjMisMjUQZ3tKe8T/lIZzsG7naKegF3VsA8pfccmjFQyjNxziIsIhTSeCSVQGSfQoLMwsbKbxNFduk3oSEZ0TqMQcr8ZkiXf0x45FZYQ28849VuFNluQn1lrhgqWLLSEJi8M0QYuIZ3uJuFTROIisLBB5SdMnides7L692WGb6XvzVQW53C28OWQ8HY0BR0BIFOtGQWwIb5GYeuAYXixKmvUcg+hapOdDK+hNgxlCBt0/vEWtdUo48Li3ff7URFoprieGYrJigbH/pQpq5BZcQFh/7Hu1TTv03lmPkrAAbuPuEkGNhRBEaqJ8uwyszHatTeTYwMIYF6hPJgMYMluaN/UxBBOy451jZyXcgaVQb2nvLcfUbPmVIxiqTyfJLDKnBPllaRsSoDpr8GDl/MkGqoZM3ACRaGQELB5qO6SZr3JUmPsbrnfskTtuVHiTvIY4vrfnzUF98YJAVWVj+5VY7rSgL58XfiSax6JEiGCZZuAq6kR4alayfc48a4iVPoZdA+SSwLpoBs2Rzo+qHHqBuNWQRIuGd1239E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma->lock being part of the vm_area_struct causes performance regression during page faults because during contention its count and owner fields are constantly updated and having other parts of vm_area_struct used during page fault handling next to them causes constant cache line bouncing. Fix that by moving the lock outside of the vm_area_struct. All attempts to keep vma->lock inside vm_area_struct in a separate cache line still produce performance regression especially on NUMA machines. Smallest regression was achieved when lock is placed in the fourth cache line but that bloats vm_area_struct to 256 bytes. Considering performance and memory impact, separate lock looks like the best option. It increases memory footprint of each VMA but that can be optimized later if the new size causes issues. Note that after this change vma_init() does not allocate or initialize vma->lock anymore. A number of drivers allocate a pseudo VMA on the stack but they never use the VMA's lock, therefore it does not need to be allocated. The future drivers which might need the VMA lock should use vm_area_alloc()/vm_area_free() to allocate the VMA. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 23 ++++++------- include/linux/mm_types.h | 6 +++- kernel/fork.c | 73 ++++++++++++++++++++++++++++++++-------- 3 files changed, 74 insertions(+), 28 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5e142bfe7a58..3d4bb18dfcb7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -627,12 +627,6 @@ struct vm_operations_struct { }; #ifdef CONFIG_PER_VMA_LOCK -static inline void vma_init_lock(struct vm_area_struct *vma) -{ - init_rwsem(&vma->lock); - vma->vm_lock_seq = -1; -} - /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -644,17 +638,17 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) return false; - if (unlikely(down_read_trylock(&vma->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) return false; /* * Overflow might produce false locked result. * False unlocked result is impossible because we modify and check - * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq * modification invalidates all existing locks. */ if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); return false; } return true; @@ -663,7 +657,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); rcu_read_unlock(); } @@ -681,9 +675,9 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (vma->vm_lock_seq == mm_lock_seq) return; - down_write(&vma->lock); + down_write(&vma->vm_lock->lock); vma->vm_lock_seq = mm_lock_seq; - up_write(&vma->lock); + up_write(&vma->vm_lock->lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -720,6 +714,10 @@ static inline void vma_mark_detached(struct vm_area_struct *vma, #endif /* CONFIG_PER_VMA_LOCK */ +/* + * WARNING: vma_init does not initialize vma->vm_lock. + * Use vm_area_alloc()/vm_area_free() if vma needs locking. + */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -729,7 +727,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); - vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6768533a6b7c..89bbf7d8a312 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -471,6 +471,10 @@ struct anon_vma_name { char name[]; }; +struct vma_lock { + struct rw_semaphore lock; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -510,7 +514,7 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; - struct rw_semaphore lock; + struct vma_lock *vm_lock; /* Flag to indicate areas detached from the mm->mm_mt tree */ bool detached; diff --git a/kernel/fork.c b/kernel/fork.c index ad37f1d0c5ab..75792157f51a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -451,13 +451,49 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; +#ifdef CONFIG_PER_VMA_LOCK + +/* SLAB cache for vm_area_struct.lock */ +static struct kmem_cache *vma_lock_cachep; + +static bool vma_lock_alloc(struct vm_area_struct *vma) +{ + vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); + if (!vma->vm_lock) + return false; + + init_rwsem(&vma->vm_lock->lock); + vma->vm_lock_seq = -1; + + return true; +} + +static inline void vma_lock_free(struct vm_area_struct *vma) +{ + kmem_cache_free(vma_lock_cachep, vma->vm_lock); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline bool vma_lock_alloc(struct vm_area_struct *vma) { return true; } +static inline void vma_lock_free(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; vma = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (vma) - vma_init(vma, mm); + if (!vma) + return NULL; + + vma_init(vma, mm); + if (!vma_lock_alloc(vma)) { + kmem_cache_free(vm_area_cachep, vma); + return NULL; + } + return vma; } @@ -465,24 +501,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) { struct vm_area_struct *new = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (new) { - ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); - ASSERT_EXCLUSIVE_WRITER(orig->vm_file); - /* - * orig->shared.rb may be modified concurrently, but the clone - * will be reinitialized. - */ - data_race(memcpy(new, orig, sizeof(*new))); - INIT_LIST_HEAD(&new->anon_vma_chain); - vma_init_lock(new); - dup_anon_vma_name(orig, new); + if (!new) + return NULL; + + ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); + ASSERT_EXCLUSIVE_WRITER(orig->vm_file); + /* + * orig->shared.rb may be modified concurrently, but the clone + * will be reinitialized. + */ + data_race(memcpy(new, orig, sizeof(*new))); + if (!vma_lock_alloc(new)) { + kmem_cache_free(vm_area_cachep, new); + return NULL; } + INIT_LIST_HEAD(&new->anon_vma_chain); + dup_anon_vma_name(orig, new); + return new; } void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); + vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); } @@ -493,7 +535,7 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) vm_rcu); /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); __vm_area_free(vma); } #endif @@ -3160,6 +3202,9 @@ void __init proc_caches_init(void) NULL); vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); +#ifdef CONFIG_PER_VMA_LOCK + vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); +#endif mmap_init(); nsproxy_cache_init(); }