From patchwork Thu Jan 30 04:42:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13954299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F8B2C0218A for ; Thu, 30 Jan 2025 04:45:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA5832800BC; Wed, 29 Jan 2025 23:45:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B55822800B9; Wed, 29 Jan 2025 23:45:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F8002800BC; Wed, 29 Jan 2025 23:45:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7E8D22800B9 for ; Wed, 29 Jan 2025 23:45:12 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 03E5545A48 for ; Thu, 30 Jan 2025 04:45:11 +0000 (UTC) X-FDA: 83062878864.01.E883730 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf23.hostedemail.com (Postfix) with ESMTP id 1F77B140007 for ; Thu, 30 Jan 2025 04:45:09 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=d4K3Ezrt; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738212310; a=rsa-sha256; cv=none; b=xPxZdyBD6/B12d5YniCqbDnAmvcRdFxrG27wTWhzlUQpZowoprqLdcJh/CddSmE4oGFLRj dDn1tKz0/FBJ7efihW0uHbOvjLc+YJUoFivXORwVH5z6BNnZ6ByzVGZs5QfOef2DpC5Eb/ S64l5NI70GBVa5LGBASvx3CEjlPEZ6E= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=d4K3Ezrt; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738212310; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MkZn1kC+dMtAGDYo9TtmpyB1pyvIkC5NDATu7QnxsYw=; b=QNxp0Pif0nVGMWA6Nds9ep+ZWJqwcrPp9SrHBcJO5yGjO9qQJEfFG42C6Dhz/GiXjBGn92 WR2uk4Q/QhpcSXlyka3xTL0SC2Lijitpuj8kl69Lgc3RYDLnceiL6i857OmZWiX8ubBarf PYURKJrl1vVU8LxZdbmg0WN9qsUN2dc= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2161eb94cceso3836085ad.2 for ; Wed, 29 Jan 2025 20:45:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212309; x=1738817109; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MkZn1kC+dMtAGDYo9TtmpyB1pyvIkC5NDATu7QnxsYw=; b=d4K3Ezrtk4pLGvle7PAVPVqCupeoPJxqC7Hk2zoiYGgcZJrzqA+eGKNxRYknJzK7DW jmUTVKtSy5Req67nZd/AQ67pLNZy8+oylQRst+YEqOmeBu6BeysXLmyou3pPo5KDnxZA KccBaSI+26vobfCr9RNeO4EIvxAieW07yrcBo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212309; x=1738817109; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MkZn1kC+dMtAGDYo9TtmpyB1pyvIkC5NDATu7QnxsYw=; b=fq+LB/5E1pgeuXS1b5gR714ABZ33PQ7rylX13trroB/JUneY7KFwuu4FLbhR2DEISG Crrk972/UxjehmZEqLbPGEj4wx8oof9a62wHd/8dtFhHxHHskkCRKIc9fCgHqNSlwDr6 3DwzVqY2wgA00QHn2gnHMIDgUTKZnNCEt8/vrPgJHvJw1v6b5W8QPKivYaw1AhJudRzq 7si/cmG9dZ6G7nNb4JR6A813EWtqOXeoZ1AkpxUMKHG3daxjL5Gs5zGEeIPdZwqqkqI5 GF8ZRU2TgwYCienMOfLpK9tkKdVhNojtem4IuhcDx2UeAhjVgBShuYzEFeUz+JXRJo/M qdXg== X-Forwarded-Encrypted: i=1; AJvYcCUPIPkbZAbqiIrO6k1sNBSSgqV2qinzVW5n0kxsA8A5szeNDedN2saIcY+sAcZNfrUFriJRMjE5lA==@kvack.org X-Gm-Message-State: AOJu0YxYQGksbo8PbYG4f2VivbtBMKz7i9IjfmL8O2GG+dyQllUWPJM2 xZKzdRyr39egiXsMKFv5NPJ4EOEs0QmN8oBXGOATOukUB7NtHqTPIFZJYshBpw== X-Gm-Gg: ASbGncvn4Ty1V1SOzdj/mq3yi1Y1ChTIHhIzc6Shbb+pKMpD9scbf73IfordW6pFJMn 3xCyswmYdlB4Fzrtd1NuL1SGMH2zCbsTw6t/DmqVCorjwB+RSwtEhXBR8MQj8KwMVdc/wjo8y3J TcwhRN85Myhi2h14lg2ztYXB0U1lMM+ytybmEoUKXHua5F1eUzQcVq48hoT/eQvCZV0XpVQdmI2 9q16Xox1tQ8rhJcA8Ph7c77B2CsyXZMHAocL4/0V6vcp6z7akKUVtW94MoyneA/d+0yTqy7K8wZ GK66PamJarpUIRrnyA== X-Google-Smtp-Source: AGHT+IGZ3YC1dM+qQG3bGdWrkOZcidEoUlnu3m+rti9TiEBtmbAsSMQiEuLAx2BRY18SnHSZlarSww== X-Received: by 2002:a05:6a20:1584:b0:1e3:cf57:5f5f with SMTP id adf61e73a8af0-1ed7a6b290dmr9388630637.27.1738212308930; Wed, 29 Jan 2025 20:45:08 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-acebe8568fesm402564a12.40.2025.01.29.20.45.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:08 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 1/7] zsmalloc: factor out pool locking helpers Date: Thu, 30 Jan 2025 13:42:44 +0900 Message-ID: <20250130044455.2642465-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1F77B140007 X-Stat-Signature: puhu6bai7up3n343nfex9ec53opfy98r X-Rspam-User: X-HE-Tag: 1738212309-162281 X-HE-Meta: U2FsdGVkX1+lj1s449bkoFM3GC2Gw0rn4iTCCtDIoMUi8itHL7V0Lfvp2UxxXpJ2M4XOlhWcr+WPEH2MUY7Rz6K826MHyFyXUJ7uoSIRUiHfUWxABCdf+ovbvUmi04pmH43zCmyx4ycPB17GlfwWsCZyS+LsgH6TdYITsqu2bTNzoV76nk12gZE+ZS8Ta1XgOTJp7eP21JQSTkUTg6at0lmCPI+OnLb14+32nSfMw8b2FDNtjqckP1/hv5J2+iXfC/D7TaP2JD0BnPTdng1VTod2pfxwY8jJBG17VJTl8+JkPBrw0oujymI4Lg4thZmv4KzIe1hK4T3cY2HdsGUyUS7jkNTayBeJDsX5ByG6aT2Oy8VaWJ8DLo/rWUFfRsAm7EbJ6Q/Pi/lJ1WqvCTy9mgitQUDiijtHfqKL07g33qCjhFT6F6oiEn/BMlZNnRFurZRF6YUOitZvSwi1w6O62czxSe2nlEfW+8RQOrqoEhqHSH5EoLPpP5X6oWFCrxtuqI89OebBiemQzowlr74yIrz+79B7rUjD22aAz4PXirvVr1893v/mx2wVRbRzB9wOJRArKtFk4Quu20eNqi9tBQpMnAgJv9/xnRGdm2//iT73Yxknj/LKGbad0HWDUQOuhLnoI44BAV3x6fjHYFPOthUor7npgLCAiO/xQIiH2Qpkon2WRnMyU9Qp3y9sDVUkCXNvEkrPZBLcam13yMPJPPuvUEM4VydWtLHR2w3GgPB7RvWApkFU/jUg7wMcidAbzIrBbRkUDWYZEffEKCUmdjcQSx7qcbhLN1Nu3rAjTEIshjjzUMUOFpS+izeL2R3bD77QN0WlRu9Vfn9xNp+hQmLWzL1YxBaX4tcBHPFY9YVMtR4I5sPk3WOoVKXJFs99DXx/BP2UqM0TQultmV1ztUKgeIB8NX6/NjmIs1IvAqcKvxvaOsgooMnKhbkDAQBFcQGhw/gYS116gK9dpBd szgUTv5t SlQ1qjsPy5L3hNN+GgZHEhP9wyBxjSJQHtqeWvAk3L2YrU+Q11awOjSd5eaLJvU+FowK294rAXHxOa6Ns/npu624zPywq9IA5lwF32gaV41pThRQ5PwPoeqEmTEH8xMhIONIR1p4sVFmA/DilFPx2oqGJVuUOAgtdt7G1Xrs8Dy7rxseTPPF29lnNmD5wBWBTbVx+iqjK7Zo716TMDNGusQq6mqWrudXSzmirlRXmcUQ12nctYsJXfzIq/o771RGPq1MHwzj6/Qv3B6Wi9k5KRbgewSAzDzi8rItQmc1qopAgJJ9BRKJbl6DtMQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We currently have a mix of migrate_{read,write}_lock() helpers that lock zspages, but it's zs_pool that actually has a ->migrate_lock access to which is opene-coded. Factor out pool migrate locking into helpers, zspage migration locking API will be renamed to reduce confusion. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 6d0e47f7ae33..8972b4b56cd3 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -228,6 +228,31 @@ struct zs_pool { atomic_t compaction_in_progress; }; +static void pool_write_unlock(struct zs_pool *pool) +{ + write_unlock(&pool->migrate_lock); +} + +static void pool_write_lock(struct zs_pool *pool) +{ + write_lock(&pool->migrate_lock); +} + +static void pool_read_unlock(struct zs_pool *pool) +{ + read_unlock(&pool->migrate_lock); +} + +static void pool_read_lock(struct zs_pool *pool) +{ + read_lock(&pool->migrate_lock); +} + +static bool pool_lock_is_contended(struct zs_pool *pool) +{ + return rwlock_is_contended(&pool->migrate_lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -1206,7 +1231,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, BUG_ON(in_interrupt()); /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj = handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage = get_zspage(zpdesc); @@ -1218,7 +1243,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); class = zspage_class(pool, zspage); off = offset_in_page(class->size * obj_idx); @@ -1453,13 +1478,13 @@ void zs_free(struct zs_pool *pool, unsigned long handle) * The pool->migrate_lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj = handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1796,7 +1821,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * The pool migrate_lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); class = zspage_class(pool, zspage); /* @@ -1833,7 +1858,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); spin_unlock(&class->lock); migrate_write_unlock(zspage); @@ -1956,7 +1981,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +2008,14 @@ static unsigned long __zs_compact(struct zs_pool *pool, src_zspage = NULL; if (get_fullness_group(class, dst_zspage) == ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || pool_lock_is_contended(pool)) { putback_zspage(class, dst_zspage); dst_zspage = NULL; spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); cond_resched(); - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); } } @@ -2002,7 +2027,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); return pages_freed; }