From patchwork Mon Dec 16 15:04:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 13909884 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 577A3E7717F for ; Mon, 16 Dec 2024 15:05:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D4E846B009B; Mon, 16 Dec 2024 10:05:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C7B6F6B009D; Mon, 16 Dec 2024 10:05:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ACF046B009E; Mon, 16 Dec 2024 10:05:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8519D6B009B for ; Mon, 16 Dec 2024 10:05:30 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1DD261C8142 for ; Mon, 16 Dec 2024 15:05:30 +0000 (UTC) X-FDA: 82901145684.28.255B35B Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf13.hostedemail.com (Postfix) with ESMTP id 5D04920024 for ; Mon, 16 Dec 2024 15:04:59 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BtOojs9O; spf=pass (imf13.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734361497; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8CPvGJ/cvPTyS5UK7p6y0gXUdUDzIEZS5wzgg9UIGeA=; b=3C2jvBsqw/SmoZCHStw9bZ3C33BENOQULS5DP125cG/NOFolVqxnU5F9XluuAKqfC7ziPy TyYQzKbAt/XXsqvfaBP6mtA/Kus3a5Zb3wlFUhmWpsw+Knc27booEzcm0joV+hLN6Od2GH m9RvsFjQGichGVNiAv69khH4hHN3rgQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BtOojs9O; spf=pass (imf13.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734361497; a=rsa-sha256; cv=none; b=HPe1Af6Dd+M0nbUsxGjYUI2DvILsu5NRJWzS1DnMJnXPxQJHbTrXwdPixXrYHaJ0uQztJg nKxORVqVux10gQ4NE3/yNjrt9lMZzH0ePC3uER+Y3c7uQ5fyYTm6I9B0XrjN5XrH5XN3qV f6GloROi4tFCeiDQIkGyeV5hZRZeWp8= Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-728ea1573c0so3612370b3a.0 for ; Mon, 16 Dec 2024 07:05:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734361527; x=1734966327; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8CPvGJ/cvPTyS5UK7p6y0gXUdUDzIEZS5wzgg9UIGeA=; b=BtOojs9OlbeVDVPEGYwRNJ9kPnp0ADcZQIFHiBwY7V1rMQW7as4vih1iPl1JMBukle dZrja2/YZ4ZDJdbGKMW0ahsaVQk84u0CM+tKCLUwpuHSz0NR9ECwu/tl+KznXJ263F8C 79RCvr0V1R2corR1fVIVS+6MMcHf++eqpBI5MkSyBafWDAJmirGouypLth7l3BAJrmXR 0vPwMBM4kChS1d5bRgaAWXxGBUYNQjUPR/jukoKWBxIWX9aDQNg7p3G8uqKDbEsMUB8i 3pw395s17Ea9KlctETZ+soG0Wy2t1l1DxeHe28BilW3iBTVX6yYnI///AAAmt0AZXdk/ yT9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734361527; x=1734966327; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8CPvGJ/cvPTyS5UK7p6y0gXUdUDzIEZS5wzgg9UIGeA=; b=f/dDNg00DMFkG20VXEkfXZjDfdaG9e1BTKgEShPTa3g2GRK6/GZ81aFp8EVpMBXKq4 MWghwDVFF9RaDWIOyaw0KaUKt9AqseQeo23GtE/fcpolL9tp5zITTn31OoQrE3Iyzxq2 txbavmA5ZHTu3LHxQCRopk8nBdqVwyNJObRVJZNS0ovqiX9FPzGddiNic3ghNKriQKH4 7ZDPjoOwBNuu5jVSXhbeffgaR+LcIjY/r4BMm5Z/aMR/ZyV668UVFtVxgi+lPGMREABe xQEJNLPRWh1AydH7VdAtlP6JDXzV9xngDOdIpS/j06WvrQJ7dZOeM7wXz5EPytrh6rTf brHQ== X-Forwarded-Encrypted: i=1; AJvYcCUSMheXKOMXfU4VoYxsXBZkSemh2GnhspMp+qTOK3HZIUNU6Nk4jWgRVqIAb3Q3RhjwLN+tLa4vRg==@kvack.org X-Gm-Message-State: AOJu0Yx5b8+wTQglL+Z0YOISkDsKe3WHOzx9kSvRIqC9Ue12oDjegp0D 5mcHJKDhBFL7sqTcU26sO/cjQPooPEHf9U+ZOMCnu0RWnoGLRdxi X-Gm-Gg: ASbGncuTAB3BQIrnbb/wUKhLeDqBSovpKh2esjGg9hQCpXS/lvN9DCF67XL07+goHyN 5pH0qFgpAOcPcajihiYs2usVsDD4Hk5uPJmLEC+pEzYgdES+AH+7gHppt2qmB7NhlvaZg/JTjM7 qEOXIqByXcFas8Ole9gQ5zeWk+mE1Z5CDAoBZeb+gTiIcgNTxtcJ0Ps/iHEGu+H7REi53MX9kbO Q5pvc4o9g9t5u0x9mxlLblNSkxQxfNNt4cTTpSH/aJqHs80QaM8M7Oy97jTI2aQ1i4= X-Google-Smtp-Source: AGHT+IF0IuixUtLePxUF06qWaeSul3ZHb6NgVZn5KBzAPn8kgm5kB3cDfnzi0TIgos2HRYxqrYA2Dg== X-Received: by 2002:a05:6a00:3d52:b0:725:df1a:285 with SMTP id d2e1a72fcca58-7290c182322mr13615035b3a.12.1734361526735; Mon, 16 Dec 2024 07:05:26 -0800 (PST) Received: from localhost.localdomain ([1.245.180.67]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72918ad5ccdsm4845518b3a.74.2024.12.16.07.05.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2024 07:05:26 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Minchan Kim , Sergey Senozhatsky Cc: Andrew Morton , linux-mm@kvack.org, Matthew Wilcox , Vishal Moola , Alex Shi , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alex Shi Subject: [PATCH v9 mm-unstable 06/18] mm/zsmalloc: convert create_page_chain() and its users to use zpdesc Date: Tue, 17 Dec 2024 00:04:37 +0900 Message-ID: <20241216150450.1228021-7-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241216150450.1228021-1-42.hyeyoo@gmail.com> References: <20241216150450.1228021-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 5D04920024 X-Rspam-User: X-Stat-Signature: 37sgib6bykzfu4e9wmuxfhkx4ucf8dkk X-HE-Tag: 1734361499-102528 X-HE-Meta: U2FsdGVkX1+yaJ/oUB6/dLUOaNP+l4KENjTgV96nhhWohjY7YTBM0Jw5XTEN9e0g+11clYneLud0WajESnIyD/dgnXsravkwiRgHAP+RlY36cVqKYmmJPBY5+r9vZ1Xk+mtmx5i+82koPHry3uphv18BAAHAJ5aVY+UQJtp9wXluPPuy3u/+ixDGl9Sj6rgNKBCyC6DtoFJu8DgWFU7h+9CSNZDdM/al7TZ+Jy17FHaSzKxsAtVh0PBZb5h8nQ1oQMakskRqWnHWnTLn9ZZivUOg/W0VYyj+25rvjnC6zffal5G9uf2BbhyINKFkB7a0PYXw/n4vWaTMME8xCh8uLbekZm3L8nM8jVePSn5x0E7p4fsLD2rLYP7u28M2PkkUWtwIR5HsQpudv0/PZUQzxcovq62VbjC+vwjQweNP60guA5nQdqUinzWdXX6/2QpUFr/XYzNqA2cyFdOKJ3XBN0KvMyHP+z6+gLqABm+18lBmtYthTLZ+h2sFWoZdRYpiExQIk9k/nlrqXrDvUxdBflkzNhfOUkE3WSzWswmRw4yfZaVsVG0ST/6fospBDlHLIhcfbaIy+kB8wFM57srGgkbqog7vWgSt9uU4TLQT1ppGQ3LemD1K1ReQ3CSNWNIoZpVvuLjk9pwoRoNJf3tCBw2liDyQKRi3ZGq5Mcqyztd6MJcIzHfYk6SLpeR+twVOo516PHXJCzAxdXqrdue+jcHaw8Se21rJYUZQytpLqyipmYCbPNgGUPGr3FRsLCtsJq1KlbgG6L3uqjtwVU4CmnTY1ZuPbjnr/4EKczqov6ZgPy7YWdM34/tDiGnMjmOJuvrfIioSONjblvdAFxthFiUm9UWjqve75nZEMjC4dkkCjAR9JX8cP8vGFqtRHCP3aVhh4bcQ2amM/jReEwnFTT5mob15FApT1UYJs2gBJYzp7MaiNJCFjM7DAlJpqKf82I1NI9ksRBYwnJXFq+c T5bmP7/W KlF6EYORVLOuWk6hjPJS57zcckgRDiRHHsYDsdedMZ9wKcr2brwTnPswNq6k4zX7L8fFLjoorf4wr62CGXaQtAg429q9mfwcbsbooVw8mPniwaPXa2RLRMD5rdnGFJ8jZqBF+m7TKcAnEQNSxzUVZdJm+djAmlcywnnC1B5TpA/8IwAHubAt8VIjTHJqoca2/pwbOzGBUD8nfcNyaEvDpyMl7g3tsZNrLc5lzA4iE9oAUJNruuPLPoTnrW/sMc7Zy9o2iHj+bmz3bgMXmtu43fIYIykdbZh7io5DrTO1fsVcPFSsDH1nMN+xm2VendZMeU+94RJfkpwlWloG2u7b6zRrhWO69lceEWjV/H/Ncx+x6whXchDzQCCQy79QiRvs1/4vP/i/eCPSmsUvMAherIUyKYhR6xz9Np0GzTisiNhJuhBEGlUneuysRpOJj6nkTNa1fENGPS/LpxAckqXuiyf1klvyf5KgSSHEbaKO3ORJgBetdW0xDU5+15n1UjIgYKslsLhWM17gDfrA7iudetOyvIBBzfu0djUYBUqiU6K5CLa8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.228511, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alex Shi Introduce a few helper functions for conversion to convert create_page_chain() to use zpdesc, then use zpdesc in replace_sub_page(). Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/zpdesc.h | 6 +++ mm/zsmalloc.c | 109 ++++++++++++++++++++++++++++++++------------------ 2 files changed, 76 insertions(+), 39 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 223d0381a444..9aca8d307796 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -148,4 +148,10 @@ static inline struct zpdesc *pfn_zpdesc(unsigned long pfn) { return page_zpdesc(pfn_to_page(pfn)); } + +static inline void __zpdesc_set_movable(struct zpdesc *zpdesc, + const struct movable_operations *mops) +{ + __SetPageMovable(zpdesc_page(zpdesc), mops); +} #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index b7fab2e28d87..59a30c61160f 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -228,6 +228,35 @@ struct zs_pool { atomic_t compaction_in_progress; }; +static inline void zpdesc_set_first(struct zpdesc *zpdesc) +{ + SetPagePrivate(zpdesc_page(zpdesc)); +} + +static inline void zpdesc_inc_zone_page_state(struct zpdesc *zpdesc) +{ + inc_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES); +} + +static inline void zpdesc_dec_zone_page_state(struct zpdesc *zpdesc) +{ + dec_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES); +} + +static inline struct zpdesc *alloc_zpdesc(gfp_t gfp) +{ + struct page *page = alloc_page(gfp); + + return page_zpdesc(page); +} + +static inline void free_zpdesc(struct zpdesc *zpdesc) +{ + struct page *page = zpdesc_page(zpdesc); + + __free_page(page); +} + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -937,35 +966,35 @@ static void init_zspage(struct size_class *class, struct zspage *zspage) } static void create_page_chain(struct size_class *class, struct zspage *zspage, - struct page *pages[]) + struct zpdesc *zpdescs[]) { int i; - struct page *page; - struct page *prev_page = NULL; - int nr_pages = class->pages_per_zspage; + struct zpdesc *zpdesc; + struct zpdesc *prev_zpdesc = NULL; + int nr_zpdescs = class->pages_per_zspage; /* * Allocate individual pages and link them together as: - * 1. all pages are linked together using page->index - * 2. each sub-page point to zspage using page->private + * 1. all pages are linked together using zpdesc->next + * 2. each sub-page point to zspage using zpdesc->zspage * - * we set PG_private to identify the first page (i.e. no other sub-page + * we set PG_private to identify the first zpdesc (i.e. no other zpdesc * has this flag set). */ - for (i = 0; i < nr_pages; i++) { - page = pages[i]; - set_page_private(page, (unsigned long)zspage); - page->index = 0; + for (i = 0; i < nr_zpdescs; i++) { + zpdesc = zpdescs[i]; + zpdesc->zspage = zspage; + zpdesc->next = NULL; if (i == 0) { - zspage->first_zpdesc = page_zpdesc(page); - SetPagePrivate(page); + zspage->first_zpdesc = zpdesc; + zpdesc_set_first(zpdesc); if (unlikely(class->objs_per_zspage == 1 && class->pages_per_zspage == 1)) SetZsHugePage(zspage); } else { - prev_page->index = (unsigned long)page; + prev_zpdesc->next = zpdesc; } - prev_page = page; + prev_zpdesc = zpdesc; } } @@ -977,7 +1006,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, gfp_t gfp) { int i; - struct page *pages[ZS_MAX_PAGES_PER_ZSPAGE]; + struct zpdesc *zpdescs[ZS_MAX_PAGES_PER_ZSPAGE]; struct zspage *zspage = cache_alloc_zspage(pool, gfp); if (!zspage) @@ -987,25 +1016,25 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, migrate_lock_init(zspage); for (i = 0; i < class->pages_per_zspage; i++) { - struct page *page; + struct zpdesc *zpdesc; - page = alloc_page(gfp); - if (!page) { + zpdesc = alloc_zpdesc(gfp); + if (!zpdesc) { while (--i >= 0) { - dec_zone_page_state(pages[i], NR_ZSPAGES); - __ClearPageZsmalloc(pages[i]); - __free_page(pages[i]); + zpdesc_dec_zone_page_state(zpdescs[i]); + __ClearPageZsmalloc(zpdesc_page(zpdescs[i])); + free_zpdesc(zpdescs[i]); } cache_free_zspage(pool, zspage); return NULL; } - __SetPageZsmalloc(page); + __SetPageZsmalloc(zpdesc_page(zpdesc)); - inc_zone_page_state(page, NR_ZSPAGES); - pages[i] = page; + zpdesc_inc_zone_page_state(zpdesc); + zpdescs[i] = zpdesc; } - create_page_chain(class, zspage, pages); + create_page_chain(class, zspage, zpdescs); init_zspage(class, zspage); zspage->pool = pool; zspage->class = class->index; @@ -1725,26 +1754,28 @@ static void migrate_write_unlock(struct zspage *zspage) static const struct movable_operations zsmalloc_mops; static void replace_sub_page(struct size_class *class, struct zspage *zspage, - struct page *newpage, struct page *oldpage) + struct zpdesc *newzpdesc, struct zpdesc *oldzpdesc) { - struct page *page; - struct page *pages[ZS_MAX_PAGES_PER_ZSPAGE] = {NULL, }; + struct zpdesc *zpdesc; + struct zpdesc *zpdescs[ZS_MAX_PAGES_PER_ZSPAGE] = {NULL, }; + unsigned int first_obj_offset; int idx = 0; - page = get_first_page(zspage); + zpdesc = get_first_zpdesc(zspage); do { - if (page == oldpage) - pages[idx] = newpage; + if (zpdesc == oldzpdesc) + zpdescs[idx] = newzpdesc; else - pages[idx] = page; + zpdescs[idx] = zpdesc; idx++; - } while ((page = get_next_page(page)) != NULL); + } while ((zpdesc = get_next_zpdesc(zpdesc)) != NULL); - create_page_chain(class, zspage, pages); - set_first_obj_offset(newpage, get_first_obj_offset(oldpage)); + create_page_chain(class, zspage, zpdescs); + first_obj_offset = get_first_obj_offset(zpdesc_page(oldzpdesc)); + set_first_obj_offset(zpdesc_page(newzpdesc), first_obj_offset); if (unlikely(ZsHugePage(zspage))) - newpage->index = oldpage->index; - __SetPageMovable(newpage, &zsmalloc_mops); + newzpdesc->handle = oldzpdesc->handle; + __zpdesc_set_movable(newzpdesc, &zsmalloc_mops); } static bool zs_page_isolate(struct page *page, isolate_mode_t mode) @@ -1817,7 +1848,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, } kunmap_local(s_addr); - replace_sub_page(class, zspage, newpage, page); + replace_sub_page(class, zspage, page_zpdesc(newpage), page_zpdesc(page)); /* * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock.