From patchwork Wed Feb 21 09:49:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13565302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8964CC48BC3 for ; Wed, 21 Feb 2024 09:50:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 709506B00A1; Wed, 21 Feb 2024 04:50:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 690B76B00A3; Wed, 21 Feb 2024 04:50:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44A266B00A2; Wed, 21 Feb 2024 04:50:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 27D486B00A0 for ; Wed, 21 Feb 2024 04:50:09 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 01411809FC for ; Wed, 21 Feb 2024 09:50:09 +0000 (UTC) X-FDA: 81815340138.13.9A12E7F Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf27.hostedemail.com (Postfix) with ESMTP id EDF8E40018 for ; Wed, 21 Feb 2024 09:50:06 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf27.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708509007; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=cd3ZnFHecMOsRIK23m1rWydcUQD/H5gTyI9z25ao2R8=; b=aX18BGWwQeJV5AwF2ZXoPLWTBJ4ZuABh5sDom9SgxpkD5JN8MJrfrXl5+6Es/iUZS0yg4O eyCgv0+W6eIu2LyUFkimICvt2wer2LqXAuNDD1u7BqvYpYS9dxcNC5qJBqXjM6QoP1WRQd H5uWdgJYSeTSky6R1L6HLDpMNA3DM3s= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf27.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708509007; a=rsa-sha256; cv=none; b=G7dzxqIBca0JfxR/FH9kSMBFo9v+NNHIsnqMEsDgjQBaqHEJkC5xA0HqzWCToiO9VhLDuK Gie1MjNuDlfHyFLbhdq1aLIyigj2KIW/IQWnHkzWEtY2fewSevOrZ/naSncUj56SiA/oGJ 1j3hWXkP8sXRfF6qT+Tb3y2WvXnQy2Q= X-AuditID: a67dfc5b-d85ff70000001748-08-65d5c73b32cc From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v12 24/27] dept: Track PG_locked with dept Date: Wed, 21 Feb 2024 18:49:30 +0900 Message-Id: <20240221094933.36348-25-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240221094933.36348-1-byungchul@sk.com> References: <20240221094933.36348-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSfUxTVxjGPefee+6lWHfXGXem0ZkmxsVFRaPba9HFZEZPnMYtmm2Zy1gj d6MZoGmliMli1aKOD4PGggI6QFM72vlRxKlQ0kH4KASlUhAZEm1kk1isAiXyMbXV+M+bX54n z++vV+I0NcJsyZC+WzGm61O1RMWrhqaXL05s7lISDnR9CsfyEiAyeoSH0osuAh0XnAhcV/Zj GGzcAHfGQggm229xUGTrQFD+4B4HV5r6EXgcBwh0PpwBgUiYgM+WS+Dg2YsE/I+nMPQVHsfg dG+GtoIKDN7x/3goGiRQUnQQR88jDOP2ShHslgUQdBSLMPVgGfj6uwXw9H4Mp870Eaj1+Hho uhbE0HmjlEC/66UAbU0tPHQcyxfgzycVBB6P2TmwR8Ii3PaWYbhkjYoOjbwQoDnfi+HQucsY AndrENQduY/B7eom0BAJYahy2ziYON+IIHh0SITsvHERSvYfRZCbXcjDrf+bBbD2rYTJ56Vk rY41hMIcs1ZlMs9YGc9aKyi7XnxPZNa6XpGVuTNYlWMRO1s7iFn5cERg7srfCHMPHxdZzlAA syc3b4qs5eQkzx4GivCXc75TrU5WUg1mxbj0sx9VKcF8O9lVu22Pze8RLWhkXQ6Kk6i8go5M TPBv+UTwqhhjIi+kPT3jXIxnyvNpVf6/Qg5SSZx8OJ46nraTWPGerKN3LOdxjHl5AR29/+w1 q+VPaPWNbvxG+iF1XvK+FsVF8z9KQkKMNfJK2uWv5mJSKh+Ooy0hP3kz+ID+7ejhC5C6DE2r RBpDujlNb0hdsSQlK92wZ8mOnWluFH0p+69T26+h4Y6t9UiWkHa6OuWvgKIR9GZTVlo9ohKn nanmM6OROlmftVcx7kwyZqQqpno0R+K176uXj2Uma+Sf9buVXxRll2J822IpbrYFfbP2+y0X drR3mr8tdFw+PRBef1dnS3y3rnL11aXbuzZxxrlUN1qS+HXry1z/ugSuYbHFJGm/oJM/ZLu2 tWbEb1pVU23+yTerX/e7yZnX4vVpv0IbI42fs4/2bTzBkgam2gqodc28p2rn1iRf76z4zf/U 2ZPMZ1bBs4GTTeF39l13aXlTin7ZIs5o0r8C33x9sE4DAAA= X-Brightmail-Tracker: H4sIAAAAAAAAAzWSf0yMcRzHfb/P83yf63TtUY1nMezMGvmRcfaZMDatR0b+sJkfjZOHbtXV 7irFWCmhZGquUHHKrtRRLj+ry6npB+uXrvxKyzGkEtU1p/y4Y/5577X3e3v99ZZQngWMj0Sl jhU1amWknEhp6ZaAlMUBjV2if321F2Sd8Qf72Cka8suNBNpvliEw3k7G0P84CJ6PDyKYaGmj IFfXjuDq2zcU3G7oRWAuOU6g870HWO3DBJp1GQRSisoJdAxMYujJycZQZtoMT88VYrA4PtKQ 208gLzcFO+MTBoehlAVD0nywlVxiYfLtMmju7WagvqCZAfMrP7h4uYdAjbmZhob7NgydVfkE eo2/GXja0ERDe1YmAze+FBIYGDdQYLAPs/DMosdQkeq0pY3+YqAx04Ih7dotDNaX1QhqT/Vh MBm7CdTbBzFUmnQU/Ch+jMB2doiFE2ccLOQln0WQcSKHhrafjQyk9ihg4ns+WRcg1A8OU0Jq 5SHBPK6nhSeFvPDg0htWSK19xQp6U5xQWbJQKKrpx8LVETsjmEpPE8E0ks0K6UNWLHxpbWWF pgsTtPDemou3ztopXb1fjFTFi5qla/dKw22ZBhJTsy1B12Fmk9DohnTkJuG5Ffx5213WxYTz 5V+8cFAu9ubm8pWZH5h0JJVQ3MmpfMnXFuIavLhV/POkYuximpvPj/V9+8sybiV/p6ob/5PO 4csqLH9Fbs7+et4g42JPTsF3ddyhziGpHk0pRd4qdXyUUhWpWKKNCE9UqxKWhEVHmZDzNIaj k1n30VhnUB3iJEjuLgu/ZxU9GWW8NjGqDvESSu4tow85K9l+ZeJhURO9RxMXKWrr0EwJLZ8h C94u7vXkDipjxQhRjBE1/1cscfNJQjNDPVbps/23LliTXTetI8riE92v2BcSVDUaksGGLP+a 8fDDlgHdcOCtmCPqd8mhisBWFBe43n22cZFPU1q8se3K9Hmnj8UkO1436XYwscS3JWzTO13w RjgaEXqx+HO6+7OhXdW/RyrWhRVZE3bbfHMKtX41+8bWzhlZ0de19NGBqXo5rQ1XLltIabTK P+dXv+4wAwAA X-CFilter-Loop: Reflected X-Rspam-User: X-Stat-Signature: wbymbxaxbxjimkb6u5ncdkb7iidodcgx X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EDF8E40018 X-HE-Tag: 1708509006-975026 X-HE-Meta: U2FsdGVkX19+guzjEJAfw9ElstZ0Hn7MNKb0kgkzGKbn6SHVYn7Sf1amjCkOkrUw9LnBJ2R2+y6hXNVmyllE4rk23FNSPUoFP1qvyY2IzGOd0L88KiUHvPWDiT8swD38eQsDn2vT3GUFT6k8f+RF4+Sf6apc31Qs2Lb8qSIAuBCs2V8vFSv5IPtS59DoLAQTgbqrpYmDP8QChmBMKC13cwH2JPaDnOWKTwRkg/o7yhqILhSVfA+RPI0/KIEPbYEuJKpNtok6p7VRd7aFqghoFXrqoHP8stB7Gg3hc5OKuwBc28Y3jJ8vHT9reYy4HXAPssCF2prYK0M4vOwDNQeDZw0TBbYRpAw6jSbphSN6uo86iMH43mZtM3AoZtz2wVUR5o2Y9ct/E++FoqcR950LQJoNQG1bZ63fo90qIa1LrO9QDRX2Dgm7UzH4g5Aq7+I/EwOUV9ww+mVrY2dQ/oscdfZUUan0A+WybBAlEfD1NW7BmnuQly7qG+Z+hPpPmXDKVviyu2LmAjINpd2b9qzBbxsPr3+7X2GscHrlEMal8/NgHTgjgzOILfNJFjTd9KQPvmkCvcOVUnlEsrKyIVCYl2qyI5BRIZQxazcuYIfidp7x+hGdXfVVYETe4jGR1SqUkyJGH+liBTaNc4YYt+2CikidbhypNWzSANhUa0cRQ5NBMqk/zxcLuEFtrfsxJ0RQJZLvyRrxUDFf0Eq6XHQNzrmIG4/3/rkvCwYqH7dIjnqXiKLfYXPSD01DrtBACh4kV+cYpF08ktBhbYyWsQAcKCQ2zGd1RdOujG+EcyHun8dwLpN8yDfQ2Kn6Xrmw465cokcxdDiO0Z6dLIJr9rD2aR7LhJ9HZrlRrqMLLjMAgfya2VhNBa77S3T/7sVTiCYyUwpBdVb2qqYlUxuz+qenVlFGVIJshEKUms9Op4R/7W0RyoIegrKFzlG7OCYFwJ1cMsv3ltTX178VA9xNMKF e2y4Aa7h xiSJN9Ce96Xth0sjilZ1QMHxz7fTARpv3PGpv3alb+Y7ZQFRqXmQBd2O1m/IA7xRGKr51X5vgsjCHxY9P3/MpBuOdzx1xPaJqX6PMBt0jNjnU32a8ihhXRhv18lB5YI/CJLGnc1u74ygOzn4Ciu73qs8EOg3RRosx9u6CWW98fsfD6hGGHcfkhFFYVOndPSatUBg2D8RBBFz2jsdb5OvHoQM8TK12y1LJf0RY0pKeaE0X8qs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Makes Dept able to track PG_locked waits and events. It's going to be useful in practice. See the following link that shows dept worked with PG_locked and can detect real issues: https://lore.kernel.org/lkml/1674268856-31807-1-git-send-email-byungchul.park@lge.com/ Signed-off-by: Byungchul Park --- include/linux/mm_types.h | 2 + include/linux/page-flags.h | 105 ++++++++++++++++++++++++++++++++----- include/linux/pagemap.h | 7 ++- mm/filemap.c | 26 +++++++++ mm/mm_init.c | 2 + 5 files changed, 129 insertions(+), 13 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 957ce38768b2..5c1112bc7a46 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -19,6 +19,7 @@ #include #include #include +#include #include @@ -203,6 +204,7 @@ struct page { struct page *kmsan_shadow; struct page *kmsan_origin; #endif + struct dept_ext_wgen PG_locked_wgen; } _struct_page_alignment; /* diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a88e64acebfe..0a498f2c4543 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -198,6 +198,43 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H +#ifdef CONFIG_DEPT +#include +#include + +extern struct dept_map PG_locked_map; + +/* + * Place the following annotations in its suitable point in code: + * + * Annotate dept_page_set_bit() around firstly set_bit*() + * Annotate dept_page_clear_bit() around clear_bit*() + * Annotate dept_page_wait_on_bit() around wait_on_bit*() + */ + +static inline void dept_page_set_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_request_event(&PG_locked_map, &p->PG_locked_wgen); +} + +static inline void dept_page_clear_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_event(&PG_locked_map, 1UL, _RET_IP_, __func__, &p->PG_locked_wgen); +} + +static inline void dept_page_wait_on_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_wait(&PG_locked_map, 1UL, _RET_IP_, __func__, 0, -1L); +} +#else +#define dept_page_set_bit(p, bit_nr) do { } while (0) +#define dept_page_clear_bit(p, bit_nr) do { } while (0) +#define dept_page_wait_on_bit(p, bit_nr) do { } while (0) +#endif + #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); @@ -379,44 +416,88 @@ static __always_inline int Page##uname(struct page *page) \ #define SETPAGEFLAG(uname, lname, policy) \ static __always_inline \ void folio_set_##lname(struct folio *folio) \ -{ set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); \ + dept_page_set_bit(&folio->page, PG_##lname); \ +} \ static __always_inline void SetPage##uname(struct page *page) \ -{ set_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + set_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_set_bit(page, PG_##lname); \ +} #define CLEARPAGEFLAG(uname, lname, policy) \ static __always_inline \ void folio_clear_##lname(struct folio *folio) \ -{ clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); \ + dept_page_clear_bit(&folio->page, PG_##lname); \ +} \ static __always_inline void ClearPage##uname(struct page *page) \ -{ clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + clear_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_clear_bit(page, PG_##lname); \ +} #define __SETPAGEFLAG(uname, lname, policy) \ static __always_inline \ void __folio_set_##lname(struct folio *folio) \ -{ __set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + __set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); \ + dept_page_set_bit(&folio->page, PG_##lname); \ +} \ static __always_inline void __SetPage##uname(struct page *page) \ -{ __set_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + __set_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_set_bit(page, PG_##lname); \ +} #define __CLEARPAGEFLAG(uname, lname, policy) \ static __always_inline \ void __folio_clear_##lname(struct folio *folio) \ -{ __clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + __clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); \ + dept_page_clear_bit(&folio->page, PG_##lname); \ +} \ static __always_inline void __ClearPage##uname(struct page *page) \ -{ __clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + __clear_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_clear_bit(page, PG_##lname); \ +} #define TESTSETFLAG(uname, lname, policy) \ static __always_inline \ bool folio_test_set_##lname(struct folio *folio) \ -{ return test_and_set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + bool ret = test_and_set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy));\ + if (!ret) \ + dept_page_set_bit(&folio->page, PG_##lname); \ + return ret; \ +} \ static __always_inline int TestSetPage##uname(struct page *page) \ -{ return test_and_set_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + bool ret = test_and_set_bit(PG_##lname, &policy(page, 1)->flags);\ + if (!ret) \ + dept_page_set_bit(page, PG_##lname); \ + return ret; \ +} #define TESTCLEARFLAG(uname, lname, policy) \ static __always_inline \ bool folio_test_clear_##lname(struct folio *folio) \ -{ return test_and_clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + bool ret = test_and_clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy));\ + if (ret) \ + dept_page_clear_bit(&folio->page, PG_##lname); \ + return ret; \ +} \ static __always_inline int TestClearPage##uname(struct page *page) \ -{ return test_and_clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + bool ret = test_and_clear_bit(PG_##lname, &policy(page, 1)->flags);\ + if (ret) \ + dept_page_clear_bit(page, PG_##lname); \ + return ret; \ +} #define PAGEFLAG(uname, lname, policy) \ TESTPAGEFLAG(uname, lname, policy) \ diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 06142ff7f9ce..c6683b228b20 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -991,7 +991,12 @@ void folio_unlock(struct folio *folio); */ static inline bool folio_trylock(struct folio *folio) { - return likely(!test_and_set_bit_lock(PG_locked, folio_flags(folio, 0))); + bool ret = !test_and_set_bit_lock(PG_locked, folio_flags(folio, 0)); + + if (ret) + dept_page_set_bit(&folio->page, PG_locked); + + return likely(ret); } /* diff --git a/mm/filemap.c b/mm/filemap.c index ad5b4aa049a3..241a67a363b0 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -45,6 +45,7 @@ #include #include #include +#include #include #include #include "internal.h" @@ -1098,6 +1099,7 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, if (flags & WQ_FLAG_CUSTOM) { if (test_and_set_bit(key->bit_nr, &key->folio->flags)) return -1; + dept_page_set_bit(&key->folio->page, key->bit_nr); flags |= WQ_FLAG_DONE; } } @@ -1181,6 +1183,7 @@ static inline bool folio_trylock_flag(struct folio *folio, int bit_nr, if (wait->flags & WQ_FLAG_EXCLUSIVE) { if (test_and_set_bit(bit_nr, &folio->flags)) return false; + dept_page_set_bit(&folio->page, bit_nr); } else if (test_bit(bit_nr, &folio->flags)) return false; @@ -1191,6 +1194,9 @@ static inline bool folio_trylock_flag(struct folio *folio, int bit_nr, /* How many times do we accept lock stealing from under a waiter? */ int sysctl_page_lock_unfairness = 5; +struct dept_map __maybe_unused PG_locked_map = DEPT_MAP_INITIALIZER(PG_locked_map, NULL); +EXPORT_SYMBOL(PG_locked_map); + static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, int state, enum behavior behavior) { @@ -1202,6 +1208,8 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, unsigned long pflags; bool in_thrashing; + dept_page_wait_on_bit(&folio->page, bit_nr); + if (bit_nr == PG_locked && !folio_test_uptodate(folio) && folio_test_workingset(folio)) { delayacct_thrashing_start(&in_thrashing); @@ -1295,6 +1303,23 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, break; } + /* + * dept_page_set_bit() might have been called already in + * folio_trylock_flag(), wake_page_function() or somewhere. + * However, call it again to reset the wgen of dept to ensure + * dept_page_wait_on_bit() is called prior to + * dept_page_set_bit(). + * + * Remind dept considers all the waits between + * dept_page_set_bit() and dept_page_clear_bit() as potential + * event disturbers. Ensure the correct sequence so that dept + * can make correct decisions: + * + * wait -> acquire(set bit) -> release(clear bit) + */ + if (wait->flags & WQ_FLAG_DONE) + dept_page_set_bit(&folio->page, bit_nr); + /* * If a signal happened, this 'finish_wait()' may remove the last * waiter from the wait-queues, but the folio waiters bit will remain @@ -1471,6 +1496,7 @@ void folio_unlock(struct folio *folio) BUILD_BUG_ON(PG_waiters != 7); BUILD_BUG_ON(PG_locked > 7); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + dept_page_clear_bit(&folio->page, PG_locked); if (folio_xor_flags_has_waiters(folio, 1 << PG_locked)) folio_wake_bit(folio, PG_locked); } diff --git a/mm/mm_init.c b/mm/mm_init.c index 077bfe393b5e..fc150d7a3686 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -564,6 +565,7 @@ void __meminit __init_single_page(struct page *page, unsigned long pfn, page_mapcount_reset(page); page_cpupid_reset_last(page); page_kasan_tag_reset(page); + dept_ext_wgen_init(&page->PG_locked_wgen); INIT_LIST_HEAD(&page->lru); #ifdef WANT_PAGE_VIRTUAL