From patchwork Fri May 31 09:19:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13681402 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD036C25B75 for ; Fri, 31 May 2024 09:20:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8251B6B009D; Fri, 31 May 2024 05:20:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 762336B009F; Fri, 31 May 2024 05:20:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 541A56B00A1; Fri, 31 May 2024 05:20:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1EDC26B009D for ; Fri, 31 May 2024 05:20:20 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D18FEA0DFD for ; Fri, 31 May 2024 09:20:19 +0000 (UTC) X-FDA: 82178144958.18.B9C7963 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf10.hostedemail.com (Postfix) with ESMTP id EECF3C0020 for ; Fri, 31 May 2024 09:20:17 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717147218; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=+o6xo0C6ae0FoaIC20MRcVPPBKvho+cxd0i5byPrumw=; b=15ryczKgmm57iyShWitgqkEvm9IHQKAqNzmHssnWEBxNpXNllJQ2z5N0+rWo/cZLFmFo7+ SJvTj/qAPFoXY9Ox9nedgLfEn9b61R/ludtO/OOk4GzRW7ijSkyBTbPvysYTXd5KTQuiwz /8vjkeDbz7YANGw9w/C79oRh4zOxzJ8= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717147218; a=rsa-sha256; cv=none; b=BjbCzANCaftNpmpF9twcYmG/Fx5ih+JDxTu7NPuYK87ee4TetzY+oydtidM29hZW1mmHgJ ncF1L2f07xFeOQqgKhfo7B0/MZRxgmIsI8k8Nw2ch5eGxHhGnhzSDWXvhe7q9YulGDW1GZ P1CYsqo3c0zpzDT6AmhfXzelbrnZdjU= X-AuditID: a67dfc5b-d85ff70000001748-57-6659964c6c58 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v11 05/12] mm: buddy: make room for a new variable, ugen, in struct page Date: Fri, 31 May 2024 18:19:54 +0900 Message-Id: <20240531092001.30428-6-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240531092001.30428-1-byungchul@sk.com> References: <20240531092001.30428-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrGLMWRmVeSWpSXmKPExsXC9ZZnoa7PtMg0g3nvuC3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG k4+/WQuuqlQcn/6VuYGxTbaLkZNDQsBE4tO5A8ww9qXnqxhBbDYBdYkbN36CxUUEzCQOtv5h B7GZBe4ySRzoZwOxhQUiJDZd6wSLswioSmz93MwEYvMKmEqsWrWNBWKmvMTqDRDzOYHmHPh7 B2y+EFDNov+9QDYXUM17Nolfr06xQTRIShxccYNlAiPvAkaGVYxCmXlluYmZOSZ6GZV5mRV6 yfm5mxiBob+s9k/0DsZPF4IPMQpwMCrx8AZURKQJsSaWFVfmHmKU4GBWEuH9lQ4U4k1JrKxK LcqPLyrNSS0+xCjNwaIkzmv0rTxFSCA9sSQ1OzW1ILUIJsvEwSnVwCjryvz40vvJyrdqnzfJ uk1n9lu8Oz9HT9XU+5VggRRf1NFJz/cf6bA4d3p2nGbTqs6jog+Cv4YKzRWsOPtnztx9s/lM 3WPOSR6o2XYh2+DPZ1VB/eQ0YUNzAVmFstMcoe9qA581nuue6PLi6etLpmLn1y58PjNUO/Zq 3Pf20s0WV59Jf5LhW6bEUpyRaKjFXFScCABLWebVeQIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLLMWRmVeSWpSXmKPExsXC5WfdrOszLTLN4OlFRos569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZTz5+Ju14KpKxfHpX5kbGNtkuxg5OSQETCQuPV/FCGKzCahL3LjxkxnEFhEw kzjY+ocdxGYWuMskcaCfDcQWFoiQ2HStEyzOIqAqsfVzMxOIzStgKrFq1TYWiJnyEqs3HACb wwk058DfO2DzhYBqFv3vZZzAyLWAkWEVo0hmXlluYmaOqV5xdkZlXmaFXnJ+7iZGYCAvq/0z cQfjl8vuhxgFOBiVeHgDKiLShFgTy4orcw8xSnAwK4nw/koHCvGmJFZWpRblxxeV5qQWH2KU 5mBREuf1Ck9NEBJITyxJzU5NLUgtgskycXBKNTA+fnZK+2ia54ZJppuPr1j2tetYx4HIo7wt NTtTVeWLnj/4M2Frv76rQt9fw5IPSv+6F9sfTZ1Tz6uipmwYnnNZrlJl9qxI37DK0qJYFlPv niL30xeuX5nvV95jsN5zoc7Wvy5v5Pw8pzbsXj8vXpLjZHzer9fvW89tqz/kvdYmcYKtFGNH yUQlluKMREMt5qLiRADuneMJYAIAAA== X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: EECF3C0020 X-Stat-Signature: 3h1pcw3h5nquymc8j9pakwtpy8cmsixi X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1717147217-599206 X-HE-Meta: U2FsdGVkX1/3d13/A7xoNKOj1wOzucrPKePWryytQHPHTaieL/Q4fQfFAKv3xxGhs0YKf5xzQMlT3uCKqXFz3RgwUbf8C3s6WcGdNcfTq+DUP4D3B0u6adLU2tFZjNMaf7RDIQZpX3TGaRBH9atrAaTu2XK00edmb60roarBms11cKYCUuHz65GEXCzBlFqjGpkhrWtMwKhAXFsDA20/12kNu3EmFv1HVdUpoOfipTXM7Z4y1x2M6Qa9gI65GdDxab7gc3s+asmO6scSdnRAKAsgejAA5IRF84hOwKyVoJKj49uosu9xG1CnRRw3wq+xrXexoZN+EHEd6PO/3klDtFeXudS2rQ7CmQ2bzVlBM07triihu1mk8Az6c/7+gTWVYz/glpuibVtp1pR3/wusvjc64Vpx29VL0ghO2xSftgfDbU9OPYI8I0A9EI26KA76q+t/K21ICVME4oJrtn89jmQDAB+wCtxlaJ4Do8xpVFdRymQKMAqDh0aZgphkNUjdJxh2HioU5e6quM8HOYpniZufr+ZmbGbRZwwIaUThwlRQabjQ/iCVUqR2Z756TMMfhIExeUQMkDbpa3+CkZGkkARVUMEAV/B3KgdjI9JXL0w98pFS8jUHVmtl7hWec+gUYeqU6Y2HlhWnC/KkODg2tcxNg4xGDuVEx5QEJAzX/YQ8MwGzK0MctX/5DlMhfSCfBrfE44PTLTbMcRuhG3Gcf0+kKsmSrLoYqiAiqP/Os7hxxI6RqPODsFRllI4HecbeimvFtQZDmzHMY8Hbp4/KSGAMmT5ZzB4BPrcyipcfwmFGeODS9X4RX8gY+2OrNHNlmryhbDZ0ik/LK398+Ut7y/S0iO6M9Zh57mFyy7VXrL29YJ5SGLXLss8MCbmuOCQUOU+o1tXz0qM1Xvq9auq1cza7++g/b13zdvDnSAOiWcOS+hI3+g8guQuEySA9t6CuQCXU+MJIeTOiZoD4iR7 sIg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functionally, no change. This is a preparation for luf mechanism that tracks need of tlb flush for each page residing in buddy, using a generation number in struct page. Fortunately, since the private field in struct page is used only to store page order in buddy, ranging from 0 to MAX_PAGE_ORDER, that can be covered with unsigned short int. So splitted it into two smaller ones, order and ugen, so that the both can be used in buddy at the same time. Signed-off-by: Byungchul Park --- include/linux/mm_types.h | 40 +++++++++++++++++++++++++++++++++------- mm/internal.h | 4 ++-- mm/page_alloc.c | 13 ++++++++----- 3 files changed, 43 insertions(+), 14 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 24323c7d0bd4..37eb3000267c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -108,13 +108,25 @@ struct page { pgoff_t index; /* Our offset within mapping. */ unsigned long share; /* share count for fsdax */ }; - /** - * @private: Mapping-private opaque data. - * Usually used for buffer_heads if PagePrivate. - * Used for swp_entry_t if PageSwapCache. - * Indicates order in the buddy system if PageBuddy. - */ - unsigned long private; + union { + /** + * @private: Mapping-private opaque data. + * Usually used for buffer_heads if PagePrivate. + * Used for swp_entry_t if PageSwapCache. + */ + unsigned long private; + struct { + /* + * Indicates order in the buddy system if PageBuddy. + */ + unsigned short int order; + /* + * Tracks need of tlb flush used by luf, + * which stands for lazy unmap flush. + */ + unsigned short int ugen; + }; + }; }; struct { /* page_pool used by netstack */ /** @@ -521,6 +533,20 @@ static inline void set_page_private(struct page *page, unsigned long private) page->private = private; } +#define page_buddy_order(page) ((page)->order) + +static inline void set_page_buddy_order(struct page *page, unsigned int order) +{ + page->order = (unsigned short int)order; +} + +#define page_buddy_ugen(page) ((page)->ugen) + +static inline void set_page_buddy_ugen(struct page *page, unsigned short int ugen) +{ + page->ugen = ugen; +} + static inline void *folio_get_private(struct folio *folio) { return folio->private; diff --git a/mm/internal.h b/mm/internal.h index bbec99cc9d9d..552e1061d36d 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -461,7 +461,7 @@ struct alloc_context { static inline unsigned int buddy_order(struct page *page) { /* PageBuddy() must be checked by the caller */ - return page_private(page); + return page_buddy_order(page); } /* @@ -475,7 +475,7 @@ static inline unsigned int buddy_order(struct page *page) * times, potentially observing different values in the tests and the actual * use of the result. */ -#define buddy_order_unsafe(page) READ_ONCE(page_private(page)) +#define buddy_order_unsafe(page) READ_ONCE(page_buddy_order(page)) /* * This function checks whether a page is free && is the buddy diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b1e3eb5787de..ae57dd8718fe 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -565,9 +565,12 @@ void prep_compound_page(struct page *page, unsigned int order) prep_compound_head(page, order); } -static inline void set_buddy_order(struct page *page, unsigned int order) +static inline void set_buddy_order_ugen(struct page *page, + unsigned int order, + unsigned short int ugen) { - set_page_private(page, order); + set_page_buddy_order(page, order); + set_page_buddy_ugen(page, order); __SetPageBuddy(page); } @@ -826,7 +829,7 @@ static inline void __free_one_page(struct page *page, } done_merging: - set_buddy_order(page, order); + set_buddy_order_ugen(page, order, 0); if (fpi_flags & FPI_TO_TAIL) to_tail = true; @@ -1336,7 +1339,7 @@ static inline void expand(struct zone *zone, struct page *page, continue; __add_to_free_list(&page[size], zone, high, migratetype, false); - set_buddy_order(&page[size], high); + set_buddy_order_ugen(&page[size], high, 0); nr_added += size; } account_freepages(zone, nr_added, migratetype); @@ -6801,7 +6804,7 @@ static void break_down_buddy_pages(struct zone *zone, struct page *page, continue; add_to_free_list(current_buddy, zone, high, migratetype, false); - set_buddy_order(current_buddy, high); + set_buddy_order_ugen(current_buddy, high, 0); } }