From patchwork Mon Jan 27 07:59:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13951054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6B5CC0218F for ; Mon, 27 Jan 2025 08:03:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 382BE280133; Mon, 27 Jan 2025 03:03:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3349E28012F; Mon, 27 Jan 2025 03:03:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D2F2280133; Mon, 27 Jan 2025 03:03:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F263228012F for ; Mon, 27 Jan 2025 03:03:19 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A3CB41A20C8 for ; Mon, 27 Jan 2025 08:03:19 +0000 (UTC) X-FDA: 83052491718.23.A48D733 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf26.hostedemail.com (Postfix) with ESMTP id AA04114001C for ; Mon, 27 Jan 2025 08:03:17 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=MfbDQpOf; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737964997; a=rsa-sha256; cv=none; b=jj8TiDjkHmS9WccNUALmS6DZqgpSErCGFVw4RumDtIW5W9hooLJtORxuqaoNQPB5TcATNG M4jzdmhdNex06lTxpRwLJcL5W3K1tWxnRRPeWcn8Gl43/hva2/d9Qfu3f3W8TLWSnVhUpn 0hYYTbA4X4nPhYP0YmWEN6LZ89W/8ck= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=MfbDQpOf; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737964997; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sJOiK/AZ7SDP8ZEH670moHkVzGkxeiUJkhPpwvQZYoo=; b=cO8V19Fm+AqJ2VjsxgtuWKkJjINkhgzuvyb3zW5XSYsCP6OEs3kZt31PrqVWdlM/Lbrqmt d8Nw68uezE6Dzmbl7psyZW24J48c2AoHrZWiXtHUJZ7Uqyp3GTo7PiNQ3E6CZooP0qkSqy 7SvyxDjpiTugmN1qP2NTBeluyfeoLUE= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-21649a7bcdcso67393805ad.1 for ; Mon, 27 Jan 2025 00:03:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737964996; x=1738569796; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sJOiK/AZ7SDP8ZEH670moHkVzGkxeiUJkhPpwvQZYoo=; b=MfbDQpOfHl990ESNbRf8GnITkxztjBX2akTsaxJB7kSvQl2pt2v7kWJnE88/GSCFfr 2exbl8wjZJUTQcei0NY4aWtqEoPt4GAcuuucyAsj2IYgodRRcvTFGPO4xPzIYhof3JJg U1ugyKkqs6RBA4jcj1iLgDwUwCptK2Io48ZNw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737964996; x=1738569796; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sJOiK/AZ7SDP8ZEH670moHkVzGkxeiUJkhPpwvQZYoo=; b=FrxfSuUljsrRxrgtEfz0+qL9xLTZAFKo0IU1YR+E9K6aZ/YSgYBJzyM8tPjgAinfEP GtWPdK+phsSVNUc6CPUzKLBXxVobaRmy/TaqA0w+K/I/S12xCUwc9BjeW3NdUPiidiEA WhB/xal6nuUcfegpix9S42B0spL7D8owKYN1rZz8nrZ1cZehoRn4bOSmvEubeverzy4S dRJyB/bHyx4waN2zXzLp8w+4oZE4Ce6oxZaTZpzVWGTdI+ZYwUm0UfO4LKV741phgMgU 2Z/UvjIb0TFy2VDCeqsPj6fwZL9NOvI+ABQCpofY/ftLIaZ2NfHTp6QofHR1VYSFbKMu I9Iw== X-Gm-Message-State: AOJu0Yz9VcqfGjE4h+S3wH+5nVIxRd4mu1QRP9pa6SZSBC0DeMs4Db2I TFYTTjMYfHFyr/5PWD4SEtcQhVIULYHPnZ9fZOT+de3z0qf+3quOBrmZ6R4TAA== X-Gm-Gg: ASbGncvW131ITnbJ5h9YYaEga/mhZsOk8P9AbHn51GDdmihXjUyD4DsvfugBtBiXlOy 95fX5IRIoP2xWjkZ/BslWUeatdnwp5AaBFGBeOya4gO4QrAvqUjE61rMWObegpdVSPqUycjPvA9 AdLkLzwLrrLkKG/6f833NH0APRL/pOj3xUL6NYaKMBnZlJ95+PQqNeYYAFxdFDCTe2hBzsteL12 ZYy9cfq98pA3lyCFlVNMxLEHaYUzgWPUb9+F9K5r52uMDnk28QGk2wiPmBkAKyzAHmsVJnYHSlf YTEq2SU= X-Google-Smtp-Source: AGHT+IHumB/yavt3u5eoYPAQOgJWZc93KAkbBNAvMwvcgxL7JyG5xBcpCGjcvKI+xBBcHvQIx6IveQ== X-Received: by 2002:a17:902:d2c8:b0:216:5e6e:68ae with SMTP id d9443c01a7336-21c3558c76bmr601303625ad.31.1737964996493; Mon, 27 Jan 2025 00:03:16 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:566d:6152:c049:8d3a]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21da3d9dd80sm57265565ad.33.2025.01.27.00.03.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 27 Jan 2025 00:03:16 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 2/6] zsmalloc: make zspage lock preemptible Date: Mon, 27 Jan 2025 16:59:27 +0900 Message-ID: <20250127080254.1302026-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250127080254.1302026-1-senozhatsky@chromium.org> References: <20250127080254.1302026-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AA04114001C X-Stat-Signature: bbxnxdex7po866e1g6w4rxjmdpdx1pgo X-Rspam-User: X-HE-Tag: 1737964997-91772 X-HE-Meta: U2FsdGVkX1/iqVdD3Ekhkxb1RWPir4xaf6tutRLKEV11EIHLQv5MLscTbQyI126LypjD/mptVZpocBuZyFHZ+U7RWxkhe/Y6M0KNQNqjzOOMSES5+/sGV8m4culKr5R6+xHmnN6/MF5IAgQcSrC0weTDkzlMoZB55zvQXFJgTjl1BUfLPmbTvq/JMQqyU4HtA5WBySr10V34zY+lssyCunJ3mi/0yZ+Q5ZW/y2tUu55iQgx5N5OZRZ2WgdlPNqkmyONK2G9bxeyOO/ki8lRTizK4DcnkRLJUKAsAPvD4yPQ4tfXTa7WEgnGJUQMfo8/kK1fM+5D5PIrQYlttMXBhw9pL8dm048FOLRl9089fvJ5dysxo3NBVbf2b+cBbTj+gTWrslAJayOv76nNIMRNRU7V46hx+NvnSQiOhLMWXWT3UmiuAH9Tk3P3SlsE5lO8HVI9SjWF3r8cnjszU/8I0k59zgfL3bp/xm9W10EArtED2fhokr4Kv7ggFD/JU0W2Igr5igXPqYJskoI2AJEJEJYSlG7js0eL3MYDunxZ+pWWuciWGwrUZUvosm34F9YqW9SGG+llH/X0vhGZYK5alrdzPK/F5bu0TBHVuxAVPZppid5zrba5LgJ44tiC4TOCeEhggh/+lpn00xK2wQ+0402fZvR1uqJFTfY4zR+gt1ccygTu9S/mxMsgAGs46NayNV/CmOyHS1mJFxC43jnmpHCfdaq9vGsD7GXDTorhmiOjh7dlXkHCIm842clnDnueu3hozZgyNswJHTADAO40ywLC6y/CczIPaATYSuh0DfhPNKzHFGRMQG/nE1PvjXD73KdfIIteUh8LvzuG25/Bv+Hj2WlyLP6gg+CKZKP4oUk8w3pRL5ffPMIwvdc1TpmyRv6F8WwQxf5zY1kUOYd8vpJbRRaeY4QJp+pVSe3ntcqrCW/PtC/FXoVG/sViR6AM0wxQO+w5QExqzoMCGrS2 vjIHsBec oviLX5UgGioRTzedZWxLc/DnleH3AxtgyeAEe+98iLeQE3XFanCaambgivOuI3nmL8sdSfAKeq1jtuygnubiMLFxsDA+dPrceX6LB8c9NQqekHgXUSCoXJzQwmxb452rxVwSDyFrXc5ijjdUhh0Ff2vI1S0GsgEB1iC59VumRDx3ewjI5dLaZyOb+c5YZayo+b1UmIwjEAOYKPIHMG1Uq6L1Ejqix4jX9aLyN9fQU0bnavnnsSVvW44DwtnFFEs945UAgbe8W+etOr1DWH80Feg4FnV31p95l3rlkf3ILe+9XZUtZpSQ/kIZguw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Switch over from rwlock_t to a atomic_t variable that takes negative value when the page is under migration, or positive values when the page is used by zsmalloc users (object map, etc.) Using a rwsem per-zspage is a little too memory heavy, a simple atomic_t should suffice, after all we only need to mark zspage as either used-for-write or used-for-read. This is needed to make zsmalloc preemtible in the future. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 112 +++++++++++++++++++++++++++++--------------------- 1 file changed, 66 insertions(+), 46 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 817626a351f8..28a75bfbeaa6 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -257,6 +257,9 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -269,7 +272,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + atomic_t lock; }; struct mapping_area { @@ -290,11 +293,53 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } -static void migrate_lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); +static void zspage_lock_init(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); +} + +static void zspage_read_lock(struct zspage *zspage) +{ + atomic_t *lock = &zspage->lock; + int old; + + while (1) { + old = atomic_read(lock); + if (old == ZS_PAGE_WRLOCKED) { + cpu_relax(); + continue; + } + + if (atomic_cmpxchg(lock, old, old + 1) == old) + return; + + cpu_relax(); + } +} + +static void zspage_read_unlock(struct zspage *zspage) +{ + atomic_dec(&zspage->lock); +} + +static void zspage_write_lock(struct zspage *zspage) +{ + atomic_t *lock = &zspage->lock; + int old; + + while (1) { + old = atomic_cmpxchg(lock, ZS_PAGE_UNLOCKED, ZS_PAGE_WRLOCKED); + if (old == ZS_PAGE_UNLOCKED) + return; + + cpu_relax(); + } +} + +static void zspage_write_unlock(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); +} #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); @@ -992,7 +1037,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, return NULL; zspage->magic = ZSPAGE_MAGIC; - migrate_lock_init(zspage); + zspage_lock_init(zspage); for (i = 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1217,7 +1262,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); read_unlock(&pool->migrate_lock); class = zspage_class(pool, zspage); @@ -1277,7 +1322,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } local_unlock(&zs_map_area.lock); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); @@ -1671,18 +1716,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc = get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1693,41 +1738,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc = zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ -static void migrate_lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->lock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage->lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION static const struct movable_operations zsmalloc_mops; @@ -1803,8 +1823,8 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * the class lock protects zpage alloc/free in the zspage. */ spin_lock(&class->lock); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage_write_lock protects zpage access via zs_map_object */ + zspage_write_lock(zspage); offset = get_first_obj_offset(zpdesc); s_addr = kmap_local_zpdesc(zpdesc); @@ -1835,7 +1855,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, */ write_unlock(&pool->migrate_lock); spin_unlock(&class->lock); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) != zpdesc_zone(zpdesc)) { @@ -1971,9 +1991,9 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (!src_zspage) break; - migrate_write_lock(src_zspage); + zspage_write_lock(src_zspage); migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); fg = putback_zspage(class, src_zspage); if (fg == ZS_INUSE_RATIO_0) {