From patchwork Mon Feb 26 03:06:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13571190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2823C54798 for ; Mon, 26 Feb 2024 03:06:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9ED56B014D; Sun, 25 Feb 2024 22:06:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B895C6B014E; Sun, 25 Feb 2024 22:06:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D9F96B014F; Sun, 25 Feb 2024 22:06:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8F33A6B014D for ; Sun, 25 Feb 2024 22:06:35 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 575D84067B for ; Mon, 26 Feb 2024 03:06:35 +0000 (UTC) X-FDA: 81832467150.13.1D11DC9 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf01.hostedemail.com (Postfix) with ESMTP id 9D29340005 for ; Mon, 26 Feb 2024 03:06:33 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708916794; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=sYa6ZVoMA5/dASQ+t66FETLgCEz/UfospKSHXnWzbLI=; b=fZ20++BpYY5/jqF0Tci/J+Wtenf1wrmRMcxRhGW1w5IcHIejRnH+UQXA0ILZF4EtPbVDc5 LJ+SsxLY1SLUvKXPZiRmiPKZ4w2liVhKywUO43TNVtnVQpesr9OUTPF7CboafuPEIQMb3s OQvqPi0vuPtXnDcK9gYXu6UgyAbypjc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708916794; a=rsa-sha256; cv=none; b=cP+RMgYnWTBqVCK20foD1Q1cJn8fRrIjzxMI7vgiXn8OgFUIUqvfqBC30hnI1Sg5NA41oq UHfGH4mz/dIyXtv69nWinha1MNDrOxP5MOi5G+4NOQKjZqPoef5Jqo2eRvh7W92V+t85Wy hRiaruPlZELuYGX3OpKgpCIW5fgPKlc= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d6dff70000001748-41-65dc002f0738 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 6/8] mm: Add APIs to free a folio directly to the buddy bypassing pcp Date: Mon, 26 Feb 2024 12:06:11 +0900 Message-Id: <20240226030613.22366-7-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnoa4Bw51Ug/Pn2CzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG qfuL2AuuyVb8f/uPtYFxskQXIweHhICJxNZHGV2MnGDm0zntLCA2m4C6xI0bP5lBbBEBM4mD rX/YQWxmgbtMEgf62UBsYYF4if2X9oDVsAioSux98QOsl1fAVGLG58lsEDPlJVZvOMAMsooT aM77/4YgYSGgkr+9Vxm7GLmASj6zSWzZepgdol5S4uCKGywTGHkXMDKsYhTKzCvLTczMMdHL qMzLrNBLzs/dxAgM+2W1f6J3MH66EHyIUYCDUYmHd8GH26lCrIllxZW5hxglOJiVRHjDZW6m CvGmJFZWpRblxxeV5qQWH2KU5mBREuc1+laeIiSQnliSmp2aWpBaBJNl4uCUamBs/fwpueHU +a9tKw+2sUW6Xbrza+XE5pfOx8KLlTklC3pe7Dl5W/xH0iLN+GtZapmOzp+F9zow/WKaJquq t/Td3RgtJZXY6nUBOjITfu//bnl8389p/695R/1hzOyZG64y9XTs34X8TNqa66csEjSdH9Zm u73wwNTLs17mLVOZ/P3rydCGyGUflViKMxINtZiLihMBNdP+sncCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDLMWRmVeSWpSXmKPExsXC5WfdrKvPcCfVYOcJS4s569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZZy6v4i94Jpsxf+3/1gbGCdLdDFyckgImEg8ndPOAmKzCahL3LjxkxnEFhEw kzjY+ocdxGYWuMskcaCfDcQWFoiX2H9pD1gNi4CqxN4XP8B6eQVMJWZ8nswGMVNeYvWGA0A1 HBycQHPe/zcECQsBlfztvco4gZFrASPDKkaRzLyy3MTMHFO94uyMyrzMCr3k/NxNjMAwXlb7 Z+IOxi+X3Q8xCnAwKvHwLvhwO1WINbGsuDL3EKMEB7OSCG+4zM1UId6UxMqq1KL8+KLSnNTi Q4zSHCxK4rxe4akJQgLpiSWp2ampBalFMFkmDk6pBkadtKp9a39EpfhfkeG/yl7+RWGKU8Ar h7QetZ0O31+/at9+Y9qPHSdCr54z+NdVrbB9evJcp3jNm4/Z5V709a477P5QvGKyx1WtmQw9 zwzOBq15tV137Ycfy9IvnZa4HZEYYGsSVuHG98zS5ZL1gbK2ZSzHJ3fMO6gpEPyQ5yTnZY24 2xMfZzEosRRnJBpqMRcVJwIAuYNFwF8CAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 9D29340005 X-Rspam-User: X-Stat-Signature: tpm56n6nk6fbnw99fohyb4mubp3bawje X-Rspamd-Server: rspam03 X-HE-Tag: 1708916793-235151 X-HE-Meta: U2FsdGVkX19PTGhVDAK538BcYZJhgvK25bU3xBjwq7Whcx1JAa8iP0nH2YqD8Y8rU2fwSj1osnDZSGpqKddiO2oKlSw57q4QVO1me4LkkSBMscBxtFkoUWrR1oXA3Xo6HFp1VAy5QVH/gC4iy42+sJl6LmQGKqz00+3gtRXYb5yKVeM1wK7OTt8TSj3gw09gj8jLLbnsQx508M0S7jE+lDiAWTnvdkG61N/MOE5V/39/BQLfuzsdiqsrb72gML0U5WsnpFjYISORb5Zc8aXJ+kHcqxFtwlG8Ov0bj9msk2nAHxPqCmPR4ws4EbgKX3U8FkM8QFZbaN5OsfNM/xrUijlStRCtRYRE3OyQh0nn6TetF2V3PyHdfHMVmfwqA46W4hcdbf0hCzeTNVROIFUN3Jk6mqNtav7NuGADAGoooqEpPECe9dav3KrVz16cImNkWVbnLsXesmLT/apgBHkyzwKtv7kc0kwwrGtLMi9JupOR83lOY1tjlur0u/fcI6LeJBKnQqhCsWoQzkmagFAXKuAhGHsuXeqQiv5+68j5gGSaTdpxlQOQh309W1zaAqLNhSw42e1XFQ0k0IDTb2BJhySfJZT/zgwYNbQUFwpM4I4O5pSabO2ESRGfgzmwSQ1rwZekkx5g37znN7bao6Vu3NewOoYu2wRLikTCKXoVBlNPdwjRsUl5k1HiQxan1Ul8P+u+NIP9RJJlDD68XgILDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that frees folios at a better time. The mechanism will defer the use of folio_put*() for source folios of migration, that are unlikely to be used and a group of folios will be freed at once at a later time. However, this will pollute pcp so as to inexpectedly free_pcppages_bulk() fresher folios and make pcp get unstable. To facilitate this new mechanism, an additional API has been added that allows folios under migrc's control to be freed directly to buddy bypassing pcp. Signed-off-by: Byungchul Park --- include/linux/mm.h | 23 +++++++++++++++++++++++ mm/internal.h | 1 + mm/page_alloc.c | 10 ++++++++++ mm/swap.c | 7 +++++++ 4 files changed, 41 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index da5219b48d52..fc0581cce3a7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1284,6 +1284,7 @@ static inline struct folio *virt_to_folio(const void *x) } void __folio_put(struct folio *folio); +void __folio_put_small_nopcp(struct folio *folio); void put_pages_list(struct list_head *pages); @@ -1483,6 +1484,28 @@ static inline void folio_put(struct folio *folio) __folio_put(folio); } +/** + * folio_put_small_nopcp - Decrement the reference count on a folio. + * @folio: The folio. + * + * This is only for a single page folio to release directly to the buddy + * allocator bypassing pcp. + * + * If the folio's reference count reaches zero, the memory will be + * released back to the page allocator and may be used by another + * allocation immediately. Do not access the memory or the struct folio + * after calling folio_put_small_nopcp() unless you can be sure that it + * wasn't the last reference. + * + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. + */ +static inline void folio_put_small_nopcp(struct folio *folio) +{ + if (folio_put_testzero(folio)) + __folio_put_small_nopcp(folio); +} + /** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio. diff --git a/mm/internal.h b/mm/internal.h index b880f1e78700..3be8fd5604e8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -451,6 +451,7 @@ extern int user_min_free_kbytes; extern void free_unref_page(struct page *page, unsigned int order); extern void free_unref_page_list(struct list_head *list); +extern void free_pages_nopcp(struct page *page, unsigned int order); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 733732e7e0ba..21b8c8cd1673 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -565,6 +565,16 @@ static inline void free_the_page(struct page *page, unsigned int order) __free_pages_ok(page, order, FPI_NONE); } +void free_pages_nopcp(struct page *page, unsigned int order) +{ + /* + * This function will be used in case that the pages are too + * cold to keep in pcp e.g. migrc mechanism. So it'd better + * release the pages to the tail. + */ + __free_pages_ok(page, order, FPI_TO_TAIL); +} + /* * Higher-order pages are called "compound pages". They are structured thusly: * diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..3f37496a1184 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -106,6 +106,13 @@ static void __folio_put_small(struct folio *folio) free_unref_page(&folio->page, 0); } +void __folio_put_small_nopcp(struct folio *folio) +{ + __page_cache_release(folio); + mem_cgroup_uncharge(folio); + free_pages_nopcp(&folio->page, 0); +} + static void __folio_put_large(struct folio *folio) { /*