From patchwork Mon Feb 19 06:04:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13562185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA870C48BC3 for ; Mon, 19 Feb 2024 06:04:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 127168D0008; Mon, 19 Feb 2024 01:04:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 03E398D0007; Mon, 19 Feb 2024 01:04:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 988F68D0005; Mon, 19 Feb 2024 01:04:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7DF258D0003 for ; Mon, 19 Feb 2024 01:04:29 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 449CB16020B for ; Mon, 19 Feb 2024 06:04:29 +0000 (UTC) X-FDA: 81807513858.01.4C69539 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf08.hostedemail.com (Postfix) with ESMTP id 272D716000B for ; Mon, 19 Feb 2024 06:04:26 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708322667; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=sYa6ZVoMA5/dASQ+t66FETLgCEz/UfospKSHXnWzbLI=; b=oICuyKKYelD3Zj/ruwSFsipSABSHXyCjU5dYDKKSr9vequreUW4u2rzr6ndnZFydRL5nKz BAQjlS0XO4tmQygTP6JCUmAAI7OZzkCaxQTzb6x4pu6egy1QnzxHWBh0Q4MfuS67AKYS+3 0URxwnA3sLUCROhOY2cyBJZ4C7DTUaM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708322667; a=rsa-sha256; cv=none; b=eLq3d33PUkrT4WUR4SWG5wRKnXd4Zn9UcftjpUhwKMgAOptDo32bb0EVbEfjW/n6fokrP9 NiIyp/8p0LbTUzJCuTBb8ywJUi4mreD9kZaqKZ2mDvu72721qfxDM3XCBPiifhAewVma5A fkiTaxW93gXCgRUqr5w+g2TI5Mw4xxY= X-AuditID: a67dfc5b-d6dff70000001748-d6-65d2ef61a9e5 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v8 6/8] mm: Add APIs to free a folio directly to the buddy bypassing pcp Date: Mon, 19 Feb 2024 15:04:05 +0900 Message-Id: <20240219060407.25254-7-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240219060407.25254-1-byungchul@sk.com> References: <20240219060407.25254-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrPLMWRmVeSWpSXmKPExsXC9ZZnkW7i+0upBouuWFjMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK+PU/UXsBddkK/6//cfawDhZoouRk0NCwERiwaOrzDD23CeX2EBsNgF1iRs3 foLFRQTMJA62/mHvYuTiYBb4yCSx+nsHC0hCWCBC4u69LUAJDg4WAVWJx5OCQMK8AqYSC/5+ YYKYKS+xesMBZpASTqA5uw8KgYSFgEpOXJ3MBDJSQqCdXeJO12qoGyQlDq64wTKBkXcBI8Mq RqHMvLLcxMwcE72MyrzMCr3k/NxNjMBoWFb7J3oH46cLwYcYBTgYlXh4M0QupQqxJpYVV+Ye YpTgYFYS4XVvupAqxJuSWFmVWpQfX1Sak1p8iFGag0VJnNfoW3mKkEB6YklqdmpqQWoRTJaJ g1OqgTHuwW3DhYWpH8t3zGNwaJe1WtX7ac7P6a+T27pFbyzj+MScWJbtknK4PuaooNij78eW xx5T1u5aqzi54+b2xvCJU0PLHr17O/26bNT/U2rSD768S72X4XH6SaTaTxexm836gvzTf828 ZlEWumj7xCtXuEuVkh7KP3sZkJCy/3u3+t/2vKb0bx5KLMUZiYZazEXFiQBUey43ggIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrGLMWRmVeSWpSXmKPExsXC5WfdrJv4/lKqwdsDehZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgyTt1fxF5wTbbi/9t/rA2MkyW6GDk5JARMJOY+ucQGYrMJ qEvcuPGTGcQWETCTONj6h72LkYuDWeAjk8Tq7x0sIAlhgQiJu/e2ACU4OFgEVCUeTwoCCfMK mEos+PuFCWKmvMTqDQeYQUo4gebsPigEEhYCKjlxdTLTBEauBYwMqxhFMvPKchMzc0z1irMz KvMyK/SS83M3MQJDe1ntn4k7GL9cdj/EKMDBqMTDmyFyKVWINbGsuDL3EKMEB7OSCK9704VU Id6UxMqq1KL8+KLSnNTiQ4zSHCxK4rxe4akJQgLpiSWp2ampBalFMFkmDk6pBkZO3uRjO9V9 duxKY4vpvCy18q6k+dy3AmbiM/O8rfsFLIVnOvBdkXFw2GOwZ5HNybOXn66N3X7liv/cuKk3 Flx8+rzbWPLtstWTJU9O+qy4dvZmRwGmZ3meGoIXhLUWGSR/3WGdF84jxvJodcFir0Um0u6O HXG3cxsDqn5bmdw8pnzNKtiFXUCJpTgj0VCLuag4EQD1ji3paQIAAA== X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 272D716000B X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: akdh9mqewgc4cuuodg7csusku7jafdsf X-HE-Tag: 1708322666-409486 X-HE-Meta: U2FsdGVkX18ea+izr0XKkWYO2nWCmufa9lMD5kTldvGAdHqCjw3z5NQ0ELiOanYYiwnDki2RCTm+h43cSwfEDswkQ5J+zB+ryVka7sXfEU1Fc/wYpHqx59PWO4p1gL8FZpiL+Xva68tXVQ7HlLQoTjTqSWP//u+NQHEpMYktfuBGO48co1YNWonOSqwDOwby04dVPJtu5fY3y0JXRKSPm/LdMMusY6Ug2TFANENy9WqEW9DCfRUV+1QtPaTcDAeWsxmt/pH18jMTviDlJ0e438TBbukyce0uekaR08yUVlHGvvS4aPlTrp7wr+uXPbH/wkvq+UYe2uF+RYaSefhbmmotyAOHO+L56vi8cPjc22SkbwCFXVRy/7BRTKwNAgY4WYINIPiXMpFk6BWDlMFIKB02UYaL1hXvD77nmgcVORxl7tvw92zmxXtB9F2BzQRARwp+g024RqHcKyH/JZOwQbqHFYzn1iJVlE9CaTEwDAB4YDSZq9sVvEGbltZRtVaWAaM0nsMiZ2f9bcqgOKSUKhSqZNEP8qY233+dblLn0iQqCrWYzxDB9jrkQqT0aDVXkES6jKjzEWn7XL/tucQ7i4Up9C2/DYEBWFbkze4K4mhwL2HN2cMBJd5GjsZHk5mD/BtLgFU1DlYIg8H0rXRqNPHLDHZD5NvpvZ8IHb/aqpr8GWDOztuV53wBomw81nCTiOz8i77tsubhM+E/QhFD7qQtTnkhshCE55sxH9hrXR8LiFzlcC/Z4+YuCAFrx+9Y8nqU4eu+Gdm+xp1c2DLMcsOesilrfgVzqtdP+haDwmfQN2SUSxmrVSCEnX9MdjmuIx1NnrwnyLt2SHu3fL48tuV9aWQdp46hH5TNprdT6rksQw9vuhdV1e1I8T6RrqIoJzjcKTKipKGPkWQjkj+NCcY2mOfQjFOrqPVWIjQ1Yj0PIy01tV+oJrAaUx+vSkZO9XDHafBS4RVLnKNaGE8 idfFZoJP ChCCOQGOysAF4AF5B9brYRALzWg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that frees folios at a better time. The mechanism will defer the use of folio_put*() for source folios of migration, that are unlikely to be used and a group of folios will be freed at once at a later time. However, this will pollute pcp so as to inexpectedly free_pcppages_bulk() fresher folios and make pcp get unstable. To facilitate this new mechanism, an additional API has been added that allows folios under migrc's control to be freed directly to buddy bypassing pcp. Signed-off-by: Byungchul Park --- include/linux/mm.h | 23 +++++++++++++++++++++++ mm/internal.h | 1 + mm/page_alloc.c | 10 ++++++++++ mm/swap.c | 7 +++++++ 4 files changed, 41 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index da5219b48d52..fc0581cce3a7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1284,6 +1284,7 @@ static inline struct folio *virt_to_folio(const void *x) } void __folio_put(struct folio *folio); +void __folio_put_small_nopcp(struct folio *folio); void put_pages_list(struct list_head *pages); @@ -1483,6 +1484,28 @@ static inline void folio_put(struct folio *folio) __folio_put(folio); } +/** + * folio_put_small_nopcp - Decrement the reference count on a folio. + * @folio: The folio. + * + * This is only for a single page folio to release directly to the buddy + * allocator bypassing pcp. + * + * If the folio's reference count reaches zero, the memory will be + * released back to the page allocator and may be used by another + * allocation immediately. Do not access the memory or the struct folio + * after calling folio_put_small_nopcp() unless you can be sure that it + * wasn't the last reference. + * + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. + */ +static inline void folio_put_small_nopcp(struct folio *folio) +{ + if (folio_put_testzero(folio)) + __folio_put_small_nopcp(folio); +} + /** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio. diff --git a/mm/internal.h b/mm/internal.h index b880f1e78700..3be8fd5604e8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -451,6 +451,7 @@ extern int user_min_free_kbytes; extern void free_unref_page(struct page *page, unsigned int order); extern void free_unref_page_list(struct list_head *list); +extern void free_pages_nopcp(struct page *page, unsigned int order); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 733732e7e0ba..21b8c8cd1673 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -565,6 +565,16 @@ static inline void free_the_page(struct page *page, unsigned int order) __free_pages_ok(page, order, FPI_NONE); } +void free_pages_nopcp(struct page *page, unsigned int order) +{ + /* + * This function will be used in case that the pages are too + * cold to keep in pcp e.g. migrc mechanism. So it'd better + * release the pages to the tail. + */ + __free_pages_ok(page, order, FPI_TO_TAIL); +} + /* * Higher-order pages are called "compound pages". They are structured thusly: * diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..3f37496a1184 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -106,6 +106,13 @@ static void __folio_put_small(struct folio *folio) free_unref_page(&folio->page, 0); } +void __folio_put_small_nopcp(struct folio *folio) +{ + __page_cache_release(folio); + mem_cgroup_uncharge(folio); + free_pages_nopcp(&folio->page, 0); +} + static void __folio_put_large(struct folio *folio) { /*