From patchwork Thu Jan 30 04:42:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13954299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F8B2C0218A for ; Thu, 30 Jan 2025 04:45:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA5832800BC; Wed, 29 Jan 2025 23:45:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B55822800B9; Wed, 29 Jan 2025 23:45:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F8002800BC; Wed, 29 Jan 2025 23:45:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7E8D22800B9 for ; Wed, 29 Jan 2025 23:45:12 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 03E5545A48 for ; Thu, 30 Jan 2025 04:45:11 +0000 (UTC) X-FDA: 83062878864.01.E883730 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf23.hostedemail.com (Postfix) with ESMTP id 1F77B140007 for ; Thu, 30 Jan 2025 04:45:09 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=d4K3Ezrt; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738212310; a=rsa-sha256; cv=none; b=xPxZdyBD6/B12d5YniCqbDnAmvcRdFxrG27wTWhzlUQpZowoprqLdcJh/CddSmE4oGFLRj dDn1tKz0/FBJ7efihW0uHbOvjLc+YJUoFivXORwVH5z6BNnZ6ByzVGZs5QfOef2DpC5Eb/ S64l5NI70GBVa5LGBASvx3CEjlPEZ6E= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=d4K3Ezrt; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738212310; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MkZn1kC+dMtAGDYo9TtmpyB1pyvIkC5NDATu7QnxsYw=; b=QNxp0Pif0nVGMWA6Nds9ep+ZWJqwcrPp9SrHBcJO5yGjO9qQJEfFG42C6Dhz/GiXjBGn92 WR2uk4Q/QhpcSXlyka3xTL0SC2Lijitpuj8kl69Lgc3RYDLnceiL6i857OmZWiX8ubBarf PYURKJrl1vVU8LxZdbmg0WN9qsUN2dc= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2161eb94cceso3836085ad.2 for ; Wed, 29 Jan 2025 20:45:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212309; x=1738817109; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MkZn1kC+dMtAGDYo9TtmpyB1pyvIkC5NDATu7QnxsYw=; b=d4K3Ezrtk4pLGvle7PAVPVqCupeoPJxqC7Hk2zoiYGgcZJrzqA+eGKNxRYknJzK7DW jmUTVKtSy5Req67nZd/AQ67pLNZy8+oylQRst+YEqOmeBu6BeysXLmyou3pPo5KDnxZA KccBaSI+26vobfCr9RNeO4EIvxAieW07yrcBo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212309; x=1738817109; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MkZn1kC+dMtAGDYo9TtmpyB1pyvIkC5NDATu7QnxsYw=; b=fq+LB/5E1pgeuXS1b5gR714ABZ33PQ7rylX13trroB/JUneY7KFwuu4FLbhR2DEISG Crrk972/UxjehmZEqLbPGEj4wx8oof9a62wHd/8dtFhHxHHskkCRKIc9fCgHqNSlwDr6 3DwzVqY2wgA00QHn2gnHMIDgUTKZnNCEt8/vrPgJHvJw1v6b5W8QPKivYaw1AhJudRzq 7si/cmG9dZ6G7nNb4JR6A813EWtqOXeoZ1AkpxUMKHG3daxjL5Gs5zGEeIPdZwqqkqI5 GF8ZRU2TgwYCienMOfLpK9tkKdVhNojtem4IuhcDx2UeAhjVgBShuYzEFeUz+JXRJo/M qdXg== X-Forwarded-Encrypted: i=1; AJvYcCUPIPkbZAbqiIrO6k1sNBSSgqV2qinzVW5n0kxsA8A5szeNDedN2saIcY+sAcZNfrUFriJRMjE5lA==@kvack.org X-Gm-Message-State: AOJu0YxYQGksbo8PbYG4f2VivbtBMKz7i9IjfmL8O2GG+dyQllUWPJM2 xZKzdRyr39egiXsMKFv5NPJ4EOEs0QmN8oBXGOATOukUB7NtHqTPIFZJYshBpw== X-Gm-Gg: ASbGncvn4Ty1V1SOzdj/mq3yi1Y1ChTIHhIzc6Shbb+pKMpD9scbf73IfordW6pFJMn 3xCyswmYdlB4Fzrtd1NuL1SGMH2zCbsTw6t/DmqVCorjwB+RSwtEhXBR8MQj8KwMVdc/wjo8y3J TcwhRN85Myhi2h14lg2ztYXB0U1lMM+ytybmEoUKXHua5F1eUzQcVq48hoT/eQvCZV0XpVQdmI2 9q16Xox1tQ8rhJcA8Ph7c77B2CsyXZMHAocL4/0V6vcp6z7akKUVtW94MoyneA/d+0yTqy7K8wZ GK66PamJarpUIRrnyA== X-Google-Smtp-Source: AGHT+IGZ3YC1dM+qQG3bGdWrkOZcidEoUlnu3m+rti9TiEBtmbAsSMQiEuLAx2BRY18SnHSZlarSww== X-Received: by 2002:a05:6a20:1584:b0:1e3:cf57:5f5f with SMTP id adf61e73a8af0-1ed7a6b290dmr9388630637.27.1738212308930; Wed, 29 Jan 2025 20:45:08 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-acebe8568fesm402564a12.40.2025.01.29.20.45.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:08 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 1/7] zsmalloc: factor out pool locking helpers Date: Thu, 30 Jan 2025 13:42:44 +0900 Message-ID: <20250130044455.2642465-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1F77B140007 X-Stat-Signature: puhu6bai7up3n343nfex9ec53opfy98r X-Rspam-User: X-HE-Tag: 1738212309-162281 X-HE-Meta: U2FsdGVkX1+lj1s449bkoFM3GC2Gw0rn4iTCCtDIoMUi8itHL7V0Lfvp2UxxXpJ2M4XOlhWcr+WPEH2MUY7Rz6K826MHyFyXUJ7uoSIRUiHfUWxABCdf+ovbvUmi04pmH43zCmyx4ycPB17GlfwWsCZyS+LsgH6TdYITsqu2bTNzoV76nk12gZE+ZS8Ta1XgOTJp7eP21JQSTkUTg6at0lmCPI+OnLb14+32nSfMw8b2FDNtjqckP1/hv5J2+iXfC/D7TaP2JD0BnPTdng1VTod2pfxwY8jJBG17VJTl8+JkPBrw0oujymI4Lg4thZmv4KzIe1hK4T3cY2HdsGUyUS7jkNTayBeJDsX5ByG6aT2Oy8VaWJ8DLo/rWUFfRsAm7EbJ6Q/Pi/lJ1WqvCTy9mgitQUDiijtHfqKL07g33qCjhFT6F6oiEn/BMlZNnRFurZRF6YUOitZvSwi1w6O62czxSe2nlEfW+8RQOrqoEhqHSH5EoLPpP5X6oWFCrxtuqI89OebBiemQzowlr74yIrz+79B7rUjD22aAz4PXirvVr1893v/mx2wVRbRzB9wOJRArKtFk4Quu20eNqi9tBQpMnAgJv9/xnRGdm2//iT73Yxknj/LKGbad0HWDUQOuhLnoI44BAV3x6fjHYFPOthUor7npgLCAiO/xQIiH2Qpkon2WRnMyU9Qp3y9sDVUkCXNvEkrPZBLcam13yMPJPPuvUEM4VydWtLHR2w3GgPB7RvWApkFU/jUg7wMcidAbzIrBbRkUDWYZEffEKCUmdjcQSx7qcbhLN1Nu3rAjTEIshjjzUMUOFpS+izeL2R3bD77QN0WlRu9Vfn9xNp+hQmLWzL1YxBaX4tcBHPFY9YVMtR4I5sPk3WOoVKXJFs99DXx/BP2UqM0TQultmV1ztUKgeIB8NX6/NjmIs1IvAqcKvxvaOsgooMnKhbkDAQBFcQGhw/gYS116gK9dpBd szgUTv5t SlQ1qjsPy5L3hNN+GgZHEhP9wyBxjSJQHtqeWvAk3L2YrU+Q11awOjSd5eaLJvU+FowK294rAXHxOa6Ns/npu624zPywq9IA5lwF32gaV41pThRQ5PwPoeqEmTEH8xMhIONIR1p4sVFmA/DilFPx2oqGJVuUOAgtdt7G1Xrs8Dy7rxseTPPF29lnNmD5wBWBTbVx+iqjK7Zo716TMDNGusQq6mqWrudXSzmirlRXmcUQ12nctYsJXfzIq/o771RGPq1MHwzj6/Qv3B6Wi9k5KRbgewSAzDzi8rItQmc1qopAgJJ9BRKJbl6DtMQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We currently have a mix of migrate_{read,write}_lock() helpers that lock zspages, but it's zs_pool that actually has a ->migrate_lock access to which is opene-coded. Factor out pool migrate locking into helpers, zspage migration locking API will be renamed to reduce confusion. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 6d0e47f7ae33..8972b4b56cd3 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -228,6 +228,31 @@ struct zs_pool { atomic_t compaction_in_progress; }; +static void pool_write_unlock(struct zs_pool *pool) +{ + write_unlock(&pool->migrate_lock); +} + +static void pool_write_lock(struct zs_pool *pool) +{ + write_lock(&pool->migrate_lock); +} + +static void pool_read_unlock(struct zs_pool *pool) +{ + read_unlock(&pool->migrate_lock); +} + +static void pool_read_lock(struct zs_pool *pool) +{ + read_lock(&pool->migrate_lock); +} + +static bool pool_lock_is_contended(struct zs_pool *pool) +{ + return rwlock_is_contended(&pool->migrate_lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -1206,7 +1231,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, BUG_ON(in_interrupt()); /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj = handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage = get_zspage(zpdesc); @@ -1218,7 +1243,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); class = zspage_class(pool, zspage); off = offset_in_page(class->size * obj_idx); @@ -1453,13 +1478,13 @@ void zs_free(struct zs_pool *pool, unsigned long handle) * The pool->migrate_lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj = handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1796,7 +1821,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * The pool migrate_lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); class = zspage_class(pool, zspage); /* @@ -1833,7 +1858,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); spin_unlock(&class->lock); migrate_write_unlock(zspage); @@ -1956,7 +1981,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +2008,14 @@ static unsigned long __zs_compact(struct zs_pool *pool, src_zspage = NULL; if (get_fullness_group(class, dst_zspage) == ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || pool_lock_is_contended(pool)) { putback_zspage(class, dst_zspage); dst_zspage = NULL; spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); cond_resched(); - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); } } @@ -2002,7 +2027,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); return pages_freed; } From patchwork Thu Jan 30 04:42:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13954300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 102CBC0218A for ; Thu, 30 Jan 2025 04:45:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C1212800BD; Wed, 29 Jan 2025 23:45:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8794F2800B9; Wed, 29 Jan 2025 23:45:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 710EA2800BD; Wed, 29 Jan 2025 23:45:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 51DAC2800B9 for ; Wed, 29 Jan 2025 23:45:18 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E126C1A0889 for ; Thu, 30 Jan 2025 04:45:17 +0000 (UTC) X-FDA: 83062879074.15.B5C7654 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf30.hostedemail.com (Postfix) with ESMTP id 0FCEB8000B for ; Thu, 30 Jan 2025 04:45:15 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Lg2+Qz26; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf30.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738212316; a=rsa-sha256; cv=none; b=OkcR+bQcrv7YJz5LOXOpk01q0tMWxz4iKj1hAhFEmYDjREh5xDDEuax+krDYjHslWx5+ht f7lbozdTG6i5bce7cAl1GZss0R8LqRw+T2CuKLg8hMVOYFIe80FD2JFnobPymhjmrgdRbt P0+JMUl+XkI34iYJ7bxLbnkybT2qrO4= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Lg2+Qz26; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf30.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738212316; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qoanfGA+iKM2lkr04fEBeNmdyi5bnUfmaHtYhcyT7Yw=; b=BXn+WfshK5SnQ+xJb4uGC1megubgGoztcooIcL95PFMEUo1dClFf1TihalkUfUbbSe0Qjl RpUJoTgFCaWyYfxr32DSgRE9vA3RwYb5klQoEYnXf7uKtOOKk+XddheKP1acr/0tqd+60/ LEN/0J8fTzHa2HwNMtmIJifYIWLT5Bk= Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2f4448bf96fso454576a91.0 for ; Wed, 29 Jan 2025 20:45:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212315; x=1738817115; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qoanfGA+iKM2lkr04fEBeNmdyi5bnUfmaHtYhcyT7Yw=; b=Lg2+Qz26jPG20CX64oJyAxZQGQzY4GspdE2Sxd67CKfoyeGoUkO01LUZJbOvLnQxP3 xJSfX+y45TXHGVmFkIlOCKlZVeghbS0KPfUp/xR2OF8Nb6AYnqUtyLc/24ROKLsh600y 1s67JnCciNGT9dWiyfTgFZHw8y2jkBYDHCznE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212315; x=1738817115; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qoanfGA+iKM2lkr04fEBeNmdyi5bnUfmaHtYhcyT7Yw=; b=HLJ/Lz/n5de1UsrRB0JwbuknX30kTLzeq4JEIpHA7S5sGOOrpjv1mpN572GIswBPSq 0OFUGyadtZ8PABOZEzjfkU3qFSC7PacppWZORDTXTq4MEyDQyE/GsfPBqqHy9Pd+qggD 5XNP8lq7fMxP+fTL88Kx8xc6CHHytIopOeUoB0+Di4lug15xOlFMV3IaaVCheXDqBVL+ jqEvwXzefJPO0V25jI32kO5XOxDA/9a0xcdm1HtviOSZOoBlPAMgszc6Bar8JUvX4+LA af4u9twS9n1qDBVORkaMyX+2WXedOfcMQyfyP7+oP5A/6TdzWCwXVK+7rYseMYbAxNrv +c4w== X-Forwarded-Encrypted: i=1; AJvYcCWAfIo4gIvT/i8fa6p9q6vobQebJbZRNl4fXhZtVLJaz/FOpux8tGU96ul/9ESH/Y3VkV+iLqHl5Q==@kvack.org X-Gm-Message-State: AOJu0YzwIToSqzxNeYLaURTMcKj4riBAQ/7mXkagxXNjV4EZTPF5VwUF tLkpBmE3vsDUwoD7DoIcrlC0XG/Xqiu5rm0MWiDPa0O7FYGeQmd4MReFtU6Euw== X-Gm-Gg: ASbGncvF8GhX9A1sZ3bdC0RSMchXgfgXlDFNkeLLBrbEtvaRv1NvGekWfqJbGkjuVnE DdLuExT4Mj9i5R/y9kmt/CCMdn2CRspesBaYVl2gtlgLOGSjiPuhcwAOnBycJo6mT8b5QvjQSmS vr4l9hweCUvCm4di7cW5R2V9ZAR/EDUdGMv/qCKoL57gWDr8yB3PZJKa5C8niiwjeUSpSDzTY++ YdbQAMJXGfKu2f1ok9f0ZPDqm5oMoBcAaCa7QFqXVh4ZnSQyv9m1e8O5teqOcqrjOAcFRY08D3m i8gezrshZjrQYfHJuQ== X-Google-Smtp-Source: AGHT+IF2UbQJQ89YEqryPhFxoeZX28FVlFBfWDCACr57M7BnO6sGVckgAr34ZiwemLJQw2ugQkpK1Q== X-Received: by 2002:a17:90b:5190:b0:2ee:ad18:b30d with SMTP id 98e67ed59e1d1-2f83abbd886mr7881534a91.6.1738212314683; Wed, 29 Jan 2025 20:45:14 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83bc98ae9sm3012541a91.10.2025.01.29.20.45.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:14 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 2/7] zsmalloc: re-shuffle zs_pool members Date: Thu, 30 Jan 2025 13:42:45 +0900 Message-ID: <20250130044455.2642465-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 0FCEB8000B X-Rspamd-Server: rspam10 X-Stat-Signature: 43uxzszyajhrdjbp6mw7ob39hdjhcmuh X-HE-Tag: 1738212315-917157 X-HE-Meta: U2FsdGVkX181+1XOihExfj5eP2Osiov5XARz1nQDFTpdJr+xsXTD5OHKlwpAkkfARG+41FRM56dzw5gsRp8ets38IZmvXA8L/APxRVcB+dsYx0KquA2XrTvLVcQ2i6vOAgcAGuz2MuzrrIQteZWtD5LETwKfS0hJh4VjCiU71LGGmi+2AHWCGBQp0gwsl0uhqDRrAyLRUF4BGoV93qGcnijDAM5F3jIJEaMsYxigR3J98X66ClK68sD/R7y+I6P3BaTtc/ON4P95HnlIuVlkfjiLae7XnK58h4aQTNKbn0ab45VSCF5vkXoAW17bVPXUH7+Fu/LGNsrk+mIh7CYzE5cALbMOWh+DH2jhIYKAgLJW6cvj82sNsz2m2bbOtO8PGmfI/te5hzDaKYCatK6PGbYay21EEG/E8hYj3xkHKzYBEHi1d7PjH929FNj6TQJbY0pzT/l3lIJHKq3Z8VD0ML68TQFu+me70nGiuQw+D/swkh7K5k1sEd5x2BKzVDFcv21B6oE0HuQ3Sk1F4aYnyy8rgmvLO1N/WgX1LUQobxDP4dSOPsx1x8pmgj4ihBx3R4kYW1pn8DA+Y0JLW/DxIam6sNwWEUwRAM8q5l6MHH1YMtAn4K885fGqp+qKaOj7AoG/FuSXY7l+jLgUSsKUloIuY1eaQJK8en7YA18Rr+TEy9KUMzfFoFpmyQXpMS6yoXXHctNiRL0MM8OFB/dlqEyoOZ9Q3IJclFOY78ifYvRkDYhyJAGWZ3BzI06gLwo6Gvt6z2oQXr/J7CwR/XieEFEv0ZRytmVNkJoNfPq4oDhv0u1DUahzdJGbfIA5lOAtn/qtFUMtq4S49ULfBXsUGI5h3zTDlD81ikd5I6ArEScF4lTNJQ4AT16iFsKTNtNq7PaisZTNxFMv5ZlkhO9UOSOr/yaB9uo8NrA+JkV5rIb5Q5Up5n9pslQIuzMtwR31/vESiA7YyRhHCsHYggq Yw0Pu/W/ rAVeV9SUzEpu2p53ZjGreHNurySW9lEOs0jF4e/1JOBsuXoyp9xdKZ0QP20rPzBo9+doJnfP2KzGMvTCMAgmkDtZoCWN4oeJqXN4/YoeGR/ZZtPslQUJHF05uXDh9vrgBb9Jf9ThEbG/bPLwiy93/+Jq5YNqwf6/geFKVNDphlzYU8TkLrv2IzP1zEv55Sbixddhl/ljvuoye6oHflwXo/d8YX9B05RJFnCg2TDadoIX1drBrvXXJgvrPvU1FhwR6+sKJQeU+fguce5VcLdZzEo/RlVruPOzBhUq3hrKm5WQS+MjyUQDi495Ir7NgrVIrMiQCuYEJhmueMNwV3XDWZ9KWvaPY3r9kokXdX2PNluGtP1g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is no need in having ->name at offset 0, it's never user and is being followed but a very large RO (after init) array of size classes, so each time we need to lock pool->migrate_lock (which happens relatively often) we need to jump over that RO region. Swap ->migrate_lock and ->name. Move ->pages_allocated is modified relatively often, move it up right after RO size_classes arrays. We modify ->compaction_in_progress and ->stats only from compaction (defragmentation), which can run only on one CPU at a time and, depending on the case, may or may not be common. Move both to down. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 8972b4b56cd3..2280ea17796b 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -204,18 +204,15 @@ struct link_free { }; struct zs_pool { - const char *name; + /* protect page/zspage migration */ + rwlock_t migrate_lock; struct size_class *size_class[ZS_SIZE_CLASSES]; - struct kmem_cache *handle_cachep; - struct kmem_cache *zspage_cachep; atomic_long_t pages_allocated; - struct zs_pool_stats stats; - - /* Compact classes */ - struct shrinker *shrinker; + struct kmem_cache *handle_cachep; + struct kmem_cache *zspage_cachep; #ifdef CONFIG_ZSMALLOC_STAT struct dentry *stat_dentry; @@ -223,9 +220,14 @@ struct zs_pool { #ifdef CONFIG_COMPACTION struct work_struct free_work; #endif - /* protect page/zspage migration */ - rwlock_t migrate_lock; + const char *name; + + /* Compact classes */ + struct shrinker *shrinker; + /* Permit only once compaction at a time */ atomic_t compaction_in_progress; + /* compaction stats */ + struct zs_pool_stats stats; }; static void pool_write_unlock(struct zs_pool *pool) From patchwork Thu Jan 30 04:42:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13954301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EA9CC0218A for ; Thu, 30 Jan 2025 04:45:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58C902800BE; Wed, 29 Jan 2025 23:45:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 53C252800B9; Wed, 29 Jan 2025 23:45:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4028C2800BE; Wed, 29 Jan 2025 23:45:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 220792800B9 for ; Wed, 29 Jan 2025 23:45:24 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 93BC9A08BF for ; Thu, 30 Jan 2025 04:45:23 +0000 (UTC) X-FDA: 83062879326.30.633E251 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf05.hostedemail.com (Postfix) with ESMTP id A8DA0100010 for ; Thu, 30 Jan 2025 04:45:21 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=C7hTVV6L; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf05.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738212321; a=rsa-sha256; cv=none; b=gt0mLObyDCVGc3oMzTJmB/dU27YFdN8IaOdsoGPgQ3QvHIDQFRnrD8WsehRuhXlPxzRHfE iT9Xs8Q27xQcGZF1wuTf82wH956Cz3Ph7bCkqE8RgYzuvm1cEOvv38Ui4n2TJZJVpS3Qic xtu/pNJ7puRFYJG+dKx42RwgBKOHNw4= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=C7hTVV6L; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf05.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738212321; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9DZ2RsvFcvw71ALze2h8Qgc3kShnLK348TMZbsDChLU=; b=7fee2RWm/3Hf5G/ICDUuy0WtSMJO4E7nyejVlZxt/O6Km0oH0oEB6S43guKlry5AlUjV4T /EfEI2mzyCTB+ntltUS+hDPrk7SrNjceCp4YrAfqSjIuRyKcjxqevYFOFdkhoeV+8xjgju 77CRMoSDw6v3EgHSsp7cpQI9ZBUx+nc= Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2161eb94cceso3837305ad.2 for ; Wed, 29 Jan 2025 20:45:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212320; x=1738817120; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9DZ2RsvFcvw71ALze2h8Qgc3kShnLK348TMZbsDChLU=; b=C7hTVV6LtEreoKhBeX4UBQu3Y6aX1bvWl07zALgNJDrdVjh4I1DQaNl0BS2ByoTQXG m2fvAhHM2Azzm4sMK7zVm8O3bVrVVMS/0TfFBieNv6CwpPYSdoNejfLRjeH/5l4JSPYw gzaLq4MNl0JwF0cVWHZiAulXx20ccSXH5AaMc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212320; x=1738817120; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9DZ2RsvFcvw71ALze2h8Qgc3kShnLK348TMZbsDChLU=; b=xIWsd+HD21IOSLgyuwHmnCV6dMHtdZM12XjLaug+0i+W+MvFkCqwhOBiALX9WBqnnF K2deaTBzBSRiaC7DCjoaxfnAGWJj5NLiYm4khUp+h9yJqi1UwZ1E7A2HdFMH/jdSTUls AGGrb4u9Q0CmW5Bv2YLikg6BBz7SpbQvnj8Yp6McAIFjIHefR50k4wEluFtL2CgrrN8f pqMwv3mukPiOfu6YQMFO+GGPLwCDURS37q/32tN/vjOWVcIArmCGylRM1oVgYWJjzldh CHunlfXOwabp2h5qor2YAAwZ5V4G0rFvRhi6o7gVg41LtJb0o//YfxZeqYKhtBt8WUOc UCEQ== X-Forwarded-Encrypted: i=1; AJvYcCUiO8gQ0QBrMxWkmn50rvFSUmuTmsL3zca4DE6VuIdm7zBzWivmbIRf1t/Q/BWkdEVi0rGz7gDj0w==@kvack.org X-Gm-Message-State: AOJu0YzJs2fpw/efCWwKjfL5ZrYzimgqBEGAlbF26I5tB25YO1uJzbar tH+xjF+8AIWMU1Ab3GRYfTfoeJglV/zO+9eLPvGIYRpko/1/9zTPZ6NrC8/XOg0OnyqV5t+QmC4 = X-Gm-Gg: ASbGnctFkDa/ZnWmDqSfqSXWOlGtQFj0HKSMYhtqjiOAKZcqYL3XYwCbwXV0evQaApG 8DXFU8t07xrpNETYLTG79ekrzrPfrsXNBSKNJB25yYeVO/e8zRohQzi22fJIHRDgz4Bd+nq4u2w 3MTgB5ZXiQAEFkilKSBsL/ec+pVKwuf64R79+AXJZYkIWo4vNRkAdJ4aCj9i9X8aHoFI7Y8Mp7/ gBP7iJlWwq/cf2uSYiLkfVohoEpBlNawBCa1yiKoQn5Akkiz1SrSOo+at1rkZ8e/HG4TYBvfhDu ohfd4obKqUu/WQIn2Q== X-Google-Smtp-Source: AGHT+IFN7QhZkttButj+44a8KQus5JiMx2Oq0RGEUpSgZ97hnkDVL1qKxutrj1FjlwXhcYZwTuD5rg== X-Received: by 2002:a05:6a21:9004:b0:1e1:9f24:2e4c with SMTP id adf61e73a8af0-1ed7a5f0665mr8510856637.16.1738212320451; Wed, 29 Jan 2025 20:45:20 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-acebe385085sm321668a12.25.2025.01.29.20.45.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:20 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 3/7] zsmalloc: factor out size-class locking helpers Date: Thu, 30 Jan 2025 13:42:46 +0900 Message-ID: <20250130044455.2642465-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A8DA0100010 X-Stat-Signature: qcjyi98tkc737tn8wekocxcdc46poeco X-HE-Tag: 1738212321-636137 X-HE-Meta: U2FsdGVkX191ay6rFXOqISFfpjRzfILbD1K9gRfGsCS0galggIBe+UhhTqmMxmGAqE86zoqq2nOSQgWWccrzgv3YAkUj9qeLpujpFyoLASUQgMzHBgBi5IGxTss8ePEk7TeCBTmWgtMlmOWBUKthrAFBSOjHoPp0p4BWz97OmW6P8aDeaFDoBbALZtM5835Ul6THIoGpHrcPob3HC7oqwKYyKugwtrljt1Lln4c8daPgkiLkaQnWnbUjHMpxmi2f0P8fSrG5HtWnZNp80dJcvskG6ITGbLUSZ2ZbwO+Kubv7p9/7zWRL8oNLGuTeGFkvk0LxXWZ9mEnoJesoPLgetCG8BGzFmNWtINT9+kcvIyfQVLRSg6C+s9UmlT3ymWx+VHS4s1PcPbgojtqJiGCnL9lcnDcSRrQ40iTGaFFLx0FiGbdPtF6zp1Gl0zsRvvVf49qG86zkXdSqIVKdPXSBXdUtl2gD5vBHAWalmN0sqmmMfFwTd5Sz1gf20G4kc2FVSuVpZRmaCF+nK/auaziwfr5fAUIEe2xxiZNTSTNBCCMfH3Ognlnnw+6rEfYXLgV+9iZk96/tAWBnd/9IuhHLrPFQDnXyeJ+tmzZS1Isso2IKxwLaezEoFLpQl37tkc+u8vU7Uj/OgMK2jhXCei1bkzXzTNieMPXb1AQ+jAMEbn8mBE78OHWwH+wDYC0wXi+KqSdwDVK7othNKLDieAdKMfESF28WS/eWnzTpl3slF9n0G5aFt+l7JNTwfs4B5KmWoy+e72L9UO6UZJxqKwNYsQCIRY77Nfl2mpdLwNUjno9/9ph7CuKg5fzBBCSLAzxTrAk9vp7tsievUqkm/i9Qe/Sq7TUZH30LF5MU6A1xt4YXEjQnhDMybskZY7J4YgwIcHYkbPVqKoAD/PUX/z84+8Zn9nZGuEo3ERx/qU7sPRkN+rtaEkj06zf2GtzoQrBzg5MS6pgByc0rOyo1bmv YAEDyeXL kHAMtsLOxeHxWqXIUZLfzU6a7KYpjEPs/KT9zJVVpKmzFM+YtaZK7xlEVNUOGMayDA6AePrOpRfFvRQXiCAADpJ8YEXQ713V5zu5rc+DGIQBt708YymvssmrP5sQ0TGh1PE5EzBoeSJIWeFLaiVQrP5g7XCRVKg6qWxWyRwIoyAA5U7Xv2uvzK8PTzmVDOOupDjYqfpOq7DS3Jw0aY5F3uvCRZAl6oVDaAy+xuRbRpacFxz/EE5p/NsRWHDBaO85GBA2sK8+yLPPL7orwg+UJDEYyq/FSaUYqP6QwxbfFn2HJQ6hN4oS/cKHlvw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move open-coded size-class locking to dedicated helpers. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 2280ea17796b..9053777035af 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -255,6 +255,16 @@ static bool pool_lock_is_contended(struct zs_pool *pool) return rwlock_is_contended(&pool->migrate_lock); } +static void size_class_lock(struct size_class *class) +{ + spin_lock(&class->lock); +} + +static void size_class_unlock(struct size_class *class) +{ + spin_unlock(&class->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -615,8 +625,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) if (class->index != i) continue; - spin_lock(&class->lock); - + size_class_lock(class); seq_printf(s, " %5u %5u ", i, class->size); for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] += class_stat_read(class, fg); @@ -626,7 +635,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) obj_allocated = class_stat_read(class, ZS_OBJS_ALLOCATED); obj_used = class_stat_read(class, ZS_OBJS_INUSE); freeable = zs_can_compact(class); - spin_unlock(&class->lock); + size_class_unlock(class); objs_per_zspage = class->objs_per_zspage; pages_used = obj_allocated / objs_per_zspage * @@ -1401,7 +1410,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) class = pool->size_class[get_size_class_index(size)]; /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + size_class_lock(class); zspage = find_get_zspage(class); if (likely(zspage)) { obj_malloc(pool, zspage, handle); @@ -1412,7 +1421,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) goto out; } - spin_unlock(&class->lock); + size_class_unlock(class); zspage = alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1420,7 +1429,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } - spin_lock(&class->lock); + size_class_lock(class); obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1431,7 +1440,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); out: - spin_unlock(&class->lock); + size_class_unlock(class); return handle; } @@ -1485,7 +1494,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); pool_read_unlock(pool); class_stat_sub(class, ZS_OBJS_INUSE, 1); @@ -1495,7 +1504,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) if (fullness == ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1829,7 +1838,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, /* * the class lock protects zpage alloc/free in the zspage. */ - spin_lock(&class->lock); + size_class_lock(class); /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); @@ -1861,7 +1870,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * it's okay to release migration_lock. */ pool_write_unlock(pool); - spin_unlock(&class->lock); + size_class_unlock(class); migrate_write_unlock(zspage); zpdesc_get(newzpdesc); @@ -1905,10 +1914,10 @@ static void async_free_zspage(struct work_struct *work) if (class->index != i) continue; - spin_lock(&class->lock); + size_class_lock(class); list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); - spin_unlock(&class->lock); + size_class_unlock(class); } list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1916,10 +1925,10 @@ static void async_free_zspage(struct work_struct *work) lock_zspage(zspage); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); class_stat_sub(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); } }; @@ -1984,7 +1993,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * as well as zpage allocation/free */ pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); while (zs_can_compact(class)) { int fg; @@ -2014,11 +2023,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); dst_zspage = NULL; - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); cond_resched(); pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); } } @@ -2028,7 +2037,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (dst_zspage) putback_zspage(class, dst_zspage); - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); return pages_freed; From patchwork Thu Jan 30 04:42:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13954302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BEF2C0218A for ; Thu, 30 Jan 2025 04:45:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0DC62800BF; Wed, 29 Jan 2025 23:45:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB93D2800B9; Wed, 29 Jan 2025 23:45:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95B142800BF; Wed, 29 Jan 2025 23:45:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 75B982800B9 for ; Wed, 29 Jan 2025 23:45:29 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F2D8E460C6 for ; Thu, 30 Jan 2025 04:45:28 +0000 (UTC) X-FDA: 83062879536.14.5AA7D19 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf27.hostedemail.com (Postfix) with ESMTP id 150CA40004 for ; Thu, 30 Jan 2025 04:45:26 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=JF2xlHn+; spf=pass (imf27.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738212327; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7b48F13IaZHEp77kXliQ7Ulgv7dkxFEkUKlwPQJd7i8=; b=OD9FiNZ90YzXJAyiFM+/tbIKzb8ncVXThNhzp/lt69KIEp8L16QfylizujE4P7UeJCGzU3 8TwuwfTju6bm5qvAbGdgx8Y7HzIEAUICExHknRhKlW5v38YpUijs2yoAtRrphyXZlWdwQ8 lHKfAyIK9TB2WuFXZAti6fFMNWc0zB4= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=JF2xlHn+; spf=pass (imf27.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738212327; a=rsa-sha256; cv=none; b=k+sArxJwi7j+xrTQBm6laAeOU6IHdj8sc++KqAqEIlpC/MjxyDPNnmOh6AYVRqrIWuRqqM 6jRdfHp0Sh8S3ZdYFzJJxZ0gJEpDGVFTH9XVrqn37s+FDTylYtDH23IPy1u86h/FaD7HQV 8s9PbmG2uUtPsZV939JBMhuAK2WQ9zM= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2166022c5caso4764245ad.2 for ; Wed, 29 Jan 2025 20:45:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212326; x=1738817126; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7b48F13IaZHEp77kXliQ7Ulgv7dkxFEkUKlwPQJd7i8=; b=JF2xlHn+3K6ce3DMSnpuC+cmSMI4B+ZT9jiWi7Lw9N6G+CUexUml7FJjJXaILm5vxE N8STXdFVddtFz66FdhnOTH1EWLsegWBHn1yA+rH8g6IxxEwLmvuMFh8gZGYNUU1/vnx+ 5NgQg1egKp1SkK84/oAL6QfKl/LR81A1MUDL4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212326; x=1738817126; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7b48F13IaZHEp77kXliQ7Ulgv7dkxFEkUKlwPQJd7i8=; b=LbpDncnH/gu2rKFUIXz0clFdwjFUmSRmHNGNOs9HGBJ3ZdhvAcI1jh9O6YKmZgAE9s VB+hP5vsU4BUwyJg8uoWc9/UPmgKOFVpxH2arRxK+xkBgOkdDeMzquX3oTPoWCeBdYQI QUZjLDG/YbR8EasJv5CfzmFlWZsHwAMU/ciYW2T0fet5IpRa9bR/A0CKglHmgB9VG4kM 5JXNjwckk8EmwteQdcvm5zGf10XwE8X9p0MISYrjfXve9KRrTXx4SWk6myClKX2UjLxd DO3WZrlPnLBQAm70X3jqPGmZWbPBWRSwc5FoGaGGr3tjAuNF6IqAC9p6B+goXyKPZF+j bORQ== X-Forwarded-Encrypted: i=1; AJvYcCWsId4C0kjKi76F7C5vPW77VXnK/m3lCt6cbWyn24e8cn6OmBLT3msIq4DOuNqcfML/kfy9E8qQaA==@kvack.org X-Gm-Message-State: AOJu0YyRiWUhwXJHKoOvOi0hfO+Pt6X1ZwCTFDqNEIIzwxrRgmb/Pmcn bULKTGrrxarV8mR3APqjOG7aD/Qf/XmSbjB+DXWk+TMLE+kKRheoZAM2shtmSg== X-Gm-Gg: ASbGncuXGiAs47Z7Ms1z/kFFct3O0vvK2+8AbqiFwllE1Q/Y9e/ddmihQtZDU+YuIDe u4FloVaN9Gy9UFj0hbgDCB5SkOBcPOimKfN87q6nzEnnTjKTxpdEVxD3Rk/6jSoCfmkeyEakRRt bSNhUpUsLWhgaTQDh+9nuLYWEsS42I+K0Snja0nuMh5llkfqZ8PuneNUZKciCWyphOg/DylnhtV kT8jqmEgwQAq0ffFeunAUV13f6IfrE2MBaUt+nlCa2ZlIgoGFNNYZ/0eMAUGAoey5ekTherufCO hWhey69KHqxLLGazMw== X-Google-Smtp-Source: AGHT+IGash5oYuGdcs755/3JeUad/VkRFkG5UkbWIaV1ZyGTPZK/ra3hdD962IUxBHUrw6ORscpOlA== X-Received: by 2002:a17:903:594:b0:216:36ff:ba33 with SMTP id d9443c01a7336-21dd7c67e06mr67812015ad.26.1738212325806; Wed, 29 Jan 2025 20:45:25 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de31ee137sm4551505ad.23.2025.01.29.20.45.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:25 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 4/7] zsmalloc: make zspage lock preemptible Date: Thu, 30 Jan 2025 13:42:47 +0900 Message-ID: <20250130044455.2642465-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 150CA40004 X-Stat-Signature: 6tx4ip3dspka9pag1tzcxc3ek8ma1q7q X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738212326-781760 X-HE-Meta: U2FsdGVkX1+KP/ADrtwAEjjkQGPolumqcdWQI+NsQjnDsSAaCKjvN3yAmk3KoQvKKEy2PYKyjbuVvGU5FAehtw7WavIUVs/jdcl8Paaapm+ENt6zyem128aMM6MkNICBjjce9IQcVTHbwUDm/3nutnCecHOLw9meuj8EkysklWOLsqqjrjftLQ6yjW8lnt9vYLWwLoDb42WfNesmL1hUYB9+JMCPKB1/mTzWi/GcK1X7dHrYA3emh9gwLBQu28XXK+ms+8JD1HK2D5ESyDMZQkhKYI8Jd6bYo2Xa81vEuK3xUhZXUQWu7nUgjL9J657dKZsQ/xVIaRuU8Yzckk9xFfMZ+mUniWz/3GVLwvmDoo4HUx0yKRnK81KDG/BczCNGYramuJLiR/mH5mvX6VpILGT54okhAmEWk1i6tDGEoXDrAdD9IT+Vlpswe4z1evlZOOyueAi6dDpvQxR3NSvWq3bHTm2Q/zB+kFt7aN2O/AUt2PcE6pOPCY7zdQcSrArj6Q+l7WwGjoGvraq9eFSkXpwxrI3l8Bymi5vkXIaMEr6Cj4rbtSE+CJdsWJm0CjkA1yZVS/j4GgtV3KRQJc0CuH8zQ5v4bjb9EehfiB3S/m0325lBpME4d9s6aW/5sA3HRdPwj/m2aoKm2pNiYv83iaCXa65VBI1J0xOMheByUOk7+p7drZR3ck7KmGQ2Fiuys1pfP4hh+P2UNifv8Wap43dQGlwRn/KjsjyAZGS9u8ullSwXsSEaPS77cJ/uRPztTb8ecXmQL7uTbuSwVMHCeJUkgiP26c4HX2CTPayghGbwJquLPWD4zuOvNT4kgj6yJc1/iFg/X3+YyiVxmUM9tc7q4xpWz3p6lwAHRMtpk/MaR1XTKxyKnsbhGVq2yk+OE4wZ+uw7N2p30WsJ0Vrgqm3X7mYNq7vUfMqLIcd2nUlWbAhWWpahgV3GeLeOIeexKzlRZQR+3+xZVUwN48K LHTrBW8N HqTh6oa3QyUuBNKj9JKILbj1x9lBHDAIi+thDDWxQLl2a2bX/1IXJTZnTurFQgbqE74KkcaAMvQqSb6ZNJOPrltbU1aA8Y06DjcnpJx3RYe1pvN5eRTsRXQBpcBl0+OkBpmxJJFx4VTnhK0bje7oRweAlSLTwIORu35FjY/nogo+6QBChHf3FSIfY6Y2wDZsoR/DK+vUN6gYlpaUN72uE2m0Xw7TsdiNyHFlrg1bR6X5Fjw6OTY7UyhhcpTbOJB1vKXSDg+5T8g39zKai0pfPdfmI1fXxOJCV5LGaZQMvmmmTahqwD8hrp2eZUA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Switch over from rwlock_t to a atomic_t variable that takes negative value when the page is under migration, or positive values when the page is used by zsmalloc users (object map, etc.) Using a rwsem per-zspage is a little too memory heavy, a simple atomic_t should suffice. zspage lock is a leaf lock for zs_map_object(), where it's read-acquired. Since this lock now permits preemption extra care needs to be taken when it is write-acquired - all writers grab it in atomic context, so they cannot spin and wait for (potentially preempted) reader to unlock zspage. There are only two writers at this moment - migration and compaction. In both cases we use write-try-lock and bail out if zspage is read locked. Writers, on the other hand, never get preempted, so readers can spin waiting for the writer to unlock zspage. With this we can implement a preemptible object mapping. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 135 +++++++++++++++++++++++++++++++------------------- 1 file changed, 83 insertions(+), 52 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 9053777035af..d8cc8e2598cc 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -294,6 +294,9 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -306,7 +309,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + atomic_t lock; }; struct mapping_area { @@ -316,6 +319,59 @@ struct mapping_area { enum zs_mapmode vm_mm; /* mapping mode */ }; +static void zspage_lock_init(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); +} + +/* + * zspage lock permits preemption on the reader-side (there can be multiple + * readers). Writers (exclusive zspage ownership), on the other hand, are + * always run in atomic context and cannot spin waiting for a (potentially + * preempted) reader to unlock zspage. This, basically, means that writers + * can only call write-try-lock and must bail out if it didn't succeed. + * + * At the same time, writers cannot reschedule under zspage write-lock, + * so readers can spin waiting for the writer to unlock zspage. + */ +static void zspage_read_lock(struct zspage *zspage) +{ + atomic_t *lock = &zspage->lock; + int old = atomic_read(lock); + + do { + if (old == ZS_PAGE_WRLOCKED) { + cpu_relax(); + old = atomic_read(lock); + continue; + } + } while (!atomic_try_cmpxchg(lock, &old, old + 1)); +} + +static void zspage_read_unlock(struct zspage *zspage) +{ + atomic_dec(&zspage->lock); +} + +static bool zspage_try_write_lock(struct zspage *zspage) +{ + atomic_t *lock = &zspage->lock; + int old = ZS_PAGE_UNLOCKED; + + preempt_disable(); + if (atomic_try_cmpxchg(lock, &old, ZS_PAGE_WRLOCKED)) + return true; + + preempt_enable(); + return false; +} + +static void zspage_write_unlock(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); + preempt_enable(); +} + /* huge object: pages_per_zspage == 1 && maxobj_per_zspage == 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -327,12 +383,6 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } -static void migrate_lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); - #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); @@ -1028,7 +1078,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, return NULL; zspage->magic = ZSPAGE_MAGIC; - migrate_lock_init(zspage); + zspage_lock_init(zspage); for (i = 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1253,7 +1303,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); pool_read_unlock(pool); class = zspage_class(pool, zspage); @@ -1313,7 +1363,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } local_unlock(&zs_map_area.lock); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); @@ -1707,18 +1757,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc = get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1729,41 +1779,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc = zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ -static void migrate_lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->lock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage->lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION static const struct movable_operations zsmalloc_mops; @@ -1805,7 +1830,7 @@ static bool zs_page_isolate(struct page *page, isolate_mode_t mode) } static int zs_page_migrate(struct page *newpage, struct page *page, - enum migrate_mode mode) + enum migrate_mode mode) { struct zs_pool *pool; struct size_class *class; @@ -1821,15 +1846,12 @@ static int zs_page_migrate(struct page *newpage, struct page *page, VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); - /* We're committed, tell the world that this is a Zsmalloc page. */ - __zpdesc_set_zsmalloc(newzpdesc); - /* The page is locked, so this pointer must remain valid */ zspage = get_zspage(zpdesc); pool = zspage->pool; /* - * The pool migrate_lock protects the race between zpage migration + * The pool->migrate_lock protects the race between zpage migration * and zs_free. */ pool_write_lock(pool); @@ -1839,8 +1861,15 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * the class lock protects zpage alloc/free in the zspage. */ size_class_lock(class); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage write_lock protects zpage access via zs_map_object */ + if (!zspage_try_write_lock(zspage)) { + size_class_unlock(class); + pool_write_unlock(pool); + return -EINVAL; + } + + /* We're committed, tell the world that this is a Zsmalloc page. */ + __zpdesc_set_zsmalloc(newzpdesc); offset = get_first_obj_offset(zpdesc); s_addr = kmap_local_zpdesc(zpdesc); @@ -1871,7 +1900,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, */ pool_write_unlock(pool); size_class_unlock(class); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) != zpdesc_zone(zpdesc)) { @@ -2007,9 +2036,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (!src_zspage) break; - migrate_write_lock(src_zspage); + if (!zspage_try_write_lock(src_zspage)) + break; + migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); fg = putback_zspage(class, src_zspage); if (fg == ZS_INUSE_RATIO_0) { From patchwork Thu Jan 30 04:42:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13954303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16257C0218F for ; Thu, 30 Jan 2025 04:45:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 95BAD2800C0; Wed, 29 Jan 2025 23:45:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 90BC02800B9; Wed, 29 Jan 2025 23:45:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7ACB52800C0; Wed, 29 Jan 2025 23:45:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 593A52800B9 for ; Wed, 29 Jan 2025 23:45:35 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D0EF31C7796 for ; Thu, 30 Jan 2025 04:45:34 +0000 (UTC) X-FDA: 83062879788.04.BC66F13 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf17.hostedemail.com (Postfix) with ESMTP id E4C0440003 for ; Thu, 30 Jan 2025 04:45:32 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Hy+u1avx; spf=pass (imf17.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.173 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738212333; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tPaya4S3Kfqs3oMtYONdXDrgcp/vC/ArYdL0b17yO2E=; b=DtLbQKBSe/kNRDyBnKixnMyaQ4Qtl7shjXY+gEfemw43WFI24I/8PKfDIkGlUhsn+woa5z PL/8WID7KQQGJV2XUWNQgTPVqLy2b5lQPG8T7MAFR/P6bZQFevhO8x9cSJGps3m3iSeJjt Tux4ArzmManYFcM7WzXMuWesrsMjC2E= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Hy+u1avx; spf=pass (imf17.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.173 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738212333; a=rsa-sha256; cv=none; b=VxpoTjoyLCANh/Cziseejcn3Z2bH0Qfb2Rkd8/Ifyv3sWzCWEDmcmRUxpjLCKzKbPTz9h/ R0uZXPIseEpZ43iRNB4QVLOglSUAgpYb0k5c5rzYjB92Spjil6zl61XlkHm2fM2PiEYHyO gx91Xjm+Dge+h687EJmINIAQ17ezXrA= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2167141dfa1so5072665ad.1 for ; Wed, 29 Jan 2025 20:45:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212332; x=1738817132; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tPaya4S3Kfqs3oMtYONdXDrgcp/vC/ArYdL0b17yO2E=; b=Hy+u1avxGF1MS/Ls9YuyEKenCIr0TeOFUUDKEZGSWnM3I8NYCV5IDSVhPpXFALrPDW tSv6WLi3rslgMJZpPQnfb30qptoK8GkxcrLY9bXFZYvkB8PePwbt9zt5EN+B2Efm6x6H pbJHB06KrnvGj13GWcL4FFP5QYzTRg9kWFZns= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212332; x=1738817132; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tPaya4S3Kfqs3oMtYONdXDrgcp/vC/ArYdL0b17yO2E=; b=s17p6SlJsBa+Eq9rOyqcK9pTJ2YuzcL3C1Mjf3+CnQVLOIQgrwA/igx6HU2LCeBkX3 1LtQsuzqHjCQnOrhgolJAxH69qc8XBqL2v+w3sLZEn+bsewgvx5+n8QoP7ZMlH8Qgu8v 37zdU02J8+QVmuDqHWPc8e88lmDK586sLbAfutqRl5XDLPIMY7Vzl5nHD+A1+uFOp74S aXOf55SIejA58fggCqOZFcGgN5MJpnb2yV1WI6SClHHYZMss/QYUfJd1kljUX/Mx/6rR 5AJAO0UTWuqsyiLK+OQNb8+lyqVUK2k5UMaJ1sHnKrgnIXEe1BF4DDf2GE6C3gyxa/ow gPVA== X-Forwarded-Encrypted: i=1; AJvYcCUHOpMTKaOBuk0AAwFf+70C/MkkjNRs6jM2ZPsYDMTFzHNaz3CnpIpxKWuJ0KPBbcySMcmKySu3Bg==@kvack.org X-Gm-Message-State: AOJu0YwgIpf1tL02I53w7KMIAQ5QiOmBY3IgBHwRte/Sj6WSDFZpENIk 1H1J7+qDBl5GnK1AYgpyKggQ9Hu428XMSZoIWAO4NGvtFR+ith2jD5DX0fRvCw== X-Gm-Gg: ASbGncupAImwp5AnQESVKp0STKFciQ4NVDBb7R6RkmziopHgZ00sPcqmC7P/ZS9AjUH hkOANhny1QF5FQfgL+DlMTmMMU+ezK+RR7muR287cQc1w42uL0+86LHskILY/onvRsYIn/xI5pm Kq2RhS2cm/fEvDGxctLMwatEtVA6fFiTn0TtmWkGdWEed18Bz4Wkt85rketiXfIEiCikVNrEsd8 S+6E0H0RDwBdqQSpA/jW/Fqvsw8TIsZJ65UdjVXEZPV8J14UOdvNjzSMcNd10Dlw2F49X9uIm+u eimzWEuUn4N3S4eq+A== X-Google-Smtp-Source: AGHT+IFnBawetG6OISjgDXcbQ2wf8X+tTg83rOrIMUBmE/TOfvysIil5FAzSXvjYS33jjv+nXChEoA== X-Received: by 2002:a17:903:41c3:b0:216:1079:82bb with SMTP id d9443c01a7336-21de195cd1emr27201265ad.19.1738212331708; Wed, 29 Jan 2025 20:45:31 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de31ee557sm4533535ad.1.2025.01.29.20.45.29 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:31 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 5/7] zsmalloc: introduce new object mapping API Date: Thu, 30 Jan 2025 13:42:48 +0900 Message-ID: <20250130044455.2642465-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E4C0440003 X-Stat-Signature: szw3of6urugf6gipatixkdo653ayj1gg X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738212332-400844 X-HE-Meta: U2FsdGVkX19CSQEPLXe7iwx6skdjlo6LQMNRWZlChXCAOgZWHEgsvF2XGGq6S+hGpzltUsm16CRzgOj8wmwHcljXv7wCbMWF/p3MFPtyN3uJXTE8zeD2kEb3z/QUbGvMl6RS2ne3d96gt7JrIEtcZowMsvyaqV3f5hAf057R6LQerRGij2HIeCE6MHCLU2i24JyL8SVj+UWLfnYxd5RIIeqpT3/qsCRi/dIji14lUNHyxQBOiUaj+ho03Z96CjdOf6WGH5WnQgq+XzL9UveKtTDyfsEtpSp0Pj/uxEqRKqq5sIygkcvCDv/JYG0aM6H0qaQ6+YtbEdvZqTncZa4WIsKqhOUgBxYwXITmgjZgm2zH9p65RIht/tY0VNyDbwlK5gYeGBS81W3KeHlHFX/asl1RWnPbrDRZG4HEJhxdG/bfEIiuNx6IIhiDDP6csxxq5UlBELoSfOlmgpSm8lZC25NPV2M/k3TQrXYg7cunYTovOerFCl8bsYcfs/FFV/UErAv6xOOgEdwrL2T+FdA1Ma1QIPI7h+9hvMFw9xtEVAGPp9jnNxf3FX9WQaEiGnPtiCcBtEg27N6+UPqasL0B+Jj6WyLK31B+DRaB0WAIARDOeqmPOOL2nQHq5bxi1XsqHonpfxFcB2/GqKq8CFWFi7DWJFk7xk2RBq5WQiX9s2evsIJ6Fw+nEjGWtIhi2Clh/bQqV4Bg0Z9RAgx9GqV8QAH2VkJfVvnU3GZniXlrf2ygUZ67uD09QdILiOQBjmiU2lzCjrLcZsdv54gjYHRtG0xF8S6ADfni944dbcZpREb7GxAbdJFO3KyvRgW9Cwg+N0lOtw8BxajD1q3c3o1RUU9njBvMCpfwtD0nXAbM/o7Nmib/uLDZEGovUQ+6KKFaYj3QvxYSNGzGgpG8GDtIxm/2u5Z4vj37/iA31Hdvsrp50Pg1rkdYpbUpIoG5GaT1IDuehKi3m/0qSjLdMm1 0W7uSRwA cHC10PqNUzQLtrHYiLM2zmT7U+K0dDduKnesr9zEtLY4D+17B4YxXHctifb2ul26Sl+HDHUODr5DvUn6tXiy7ZMECUGfPHNHQ6uRPs7bfJ8sAwuMCJ+nW+E71lATFJDQxdJWRbJoKDnkUmbJsk40R+DjGx1qm5Gtaprcj2IChs6ZK+43U7ozSQf/DUMZZ/NJi1rkdAH0h3ATnOu1aGXkhCFvt4uxvVzc4YkgaGnlGdYqCTf2+ZOnVSVoCwlCZjWEWhYJNU9iZ0lNcUmBKILnpjDzomIcFMJQyCYzzf8kToz00qAmE4X7YIcFMd4kyktqXFqHs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 130 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 138 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..7d70983cf398 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size); void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index d8cc8e2598cc..67c934ed4be9 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1367,6 +1367,136 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr = kmap_local_zpdesc(zpdesc); + addr += off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] = PAGE_SIZE - off; + sizes[1] = class->size - sizes[0]; + addr = local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr += ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + handle_mem -= off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst = kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + /* this object spans two pages */ + size_t sizes[2]; + + off += ZS_HANDLE_SIZE; + + sizes[0] = PAGE_SIZE - off; + sizes[1] = mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. From patchwork Thu Jan 30 04:42:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13954304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F18AC0218F for ; Thu, 30 Jan 2025 04:45:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AA9C2800C1; Wed, 29 Jan 2025 23:45:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 95C0E2800B9; Wed, 29 Jan 2025 23:45:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 822BF2800C1; Wed, 29 Jan 2025 23:45:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 62D462800B9 for ; Wed, 29 Jan 2025 23:45:40 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 24E4B140820 for ; Thu, 30 Jan 2025 04:45:40 +0000 (UTC) X-FDA: 83062880040.01.2B7D767 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf10.hostedemail.com (Postfix) with ESMTP id 24E7AC0003 for ; Thu, 30 Jan 2025 04:45:37 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=CoBjNYNn; spf=pass (imf10.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738212338; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jm+b3R/xkgu1u5aIyHjEy9+3hXzV4kVBwpygpgrob3Q=; b=6xqKPithuDvcfchXX2HV+lT8GXNY6TmErXPBW8V3+VeaZlq2DroYZQINELagAN+LGvRL9r cIPe+iRXoLM9koj93FWhLEncj9VzQpLhAm5ukhYUriDkNGbnWvR78atuKrBYo+n33AF9In 2eEFdWYJ+GKD2zRrm+EhUG0FzN3YDzg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738212338; a=rsa-sha256; cv=none; b=sLUEfO3nPJD+7ufzHNgeruPkkaph3ssZPFVPKCf1UHLDY76HmHkbVquUJGJVu1ZUAj3yty LrDWXfJH1L3vqVoz4VktphAnk9f065HnOkoIXgLhi9Pg3h+fv0hK5h3J1mO8Oaewxm3VFG sYp7H76/8cQmUBpyu6nMUqkVj12NIJE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=CoBjNYNn; spf=pass (imf10.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-21669fd5c7cso5198815ad.3 for ; Wed, 29 Jan 2025 20:45:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212337; x=1738817137; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jm+b3R/xkgu1u5aIyHjEy9+3hXzV4kVBwpygpgrob3Q=; b=CoBjNYNnZNS3A3R9dgl+EaGKsKUK1t4WIEpQ2gb3mkRBegqMOJZSNrJHVrniBrF20i ZOEi5C3lWURYTBYA6rjmQo689PYKffW0BXgCteAvw8r748C4pe/W/nXhT0s0dFef9k5H IGykZOwcNWBxqp/Dq+dvZfwtvV+tVo2FB6ggE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212337; x=1738817137; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jm+b3R/xkgu1u5aIyHjEy9+3hXzV4kVBwpygpgrob3Q=; b=Vs/POhEslieYiWigJUpJKKBbp9E6gqkt/UMOV+r8EPptKczbk2L18QpOijyRkCsGY3 8GZbWNo81rikfb9Y44bxLlU8B2/sAznyJCK0p1Ca8w0IWcV8za3P6CEbOk26/KCGkiS0 ZQYUyecij/vjc0l4Wz1AXySdGz0ain9e4WqmSo7+nlQp8gfppaUEnsFsd/fI/68Fj4b8 rlFrQlCkdJ8UNl5pYvcMcqDpkXTc0KI67IO3XNnCsK5YDRy1F3PNhNMLslQeQhfLyne5 MyRoxp/kOxYNABATm+B8bMAlxOKIVbttGt0j0VFhSmDx5FV1FJaw5YgPXqckekG5Ojd4 tNCg== X-Forwarded-Encrypted: i=1; AJvYcCWcXV8RNWt2/RYLbjzMxMWkrc7dwm+wJrT/n/2VfbqImrhPEjOjXFnYx57/3c6sf/0EHX4wZA9tNA==@kvack.org X-Gm-Message-State: AOJu0Yxvo8y9O191uKpuvdjRpoGLtvvT3nXWs3wwL51SPERXk6BDQEHY fNP90+G6jJbujOsYovhgjV7rUeAmXIBcKZHbqJW+RUqNGEzPUuR2CgE2AVUmnw== X-Gm-Gg: ASbGnctO/qSdVCfHCiPPkB3kOG9fXyS9yvX6Rk5Gg36yF3WcddEmWp8lsfJQNWfEZvQ A/8rotOMm4WyG7nYuXItWd55MMJy15/hRLJMB1heOxSpfgyb4j63lPYtKIqIzvVp7Df4kt+ZbpM 6rVUrBYMAQ/FhjMVnH0VqR4W5cHhb1K3aZV2b/FC+atbTj01g4sJflre6ET/BtaGjmy+xJnCvds KOYf2M194cTBHdTAS5ujaUlpaI4a3EO6kd7rKOFmpuEhacsv+U6GqVNVq0MXe+UbTp+8LVBIeuY T2ZnavHpT/83pPWmhw== X-Google-Smtp-Source: AGHT+IHUyr8/2Fq8AovilKRJ28NBYk+8bvRw/3TGJlSrvnfXxKWiWthlwKdxoZwtf3TrFQyikvGQUQ== X-Received: by 2002:a17:903:594:b0:216:36ff:ba33 with SMTP id d9443c01a7336-21dd7c67e06mr67817495ad.26.1738212336948; Wed, 29 Jan 2025 20:45:36 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de33000e0sm4566895ad.167.2025.01.29.20.45.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:36 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 6/7] zram: switch to new zsmalloc object mapping API Date: Thu, 30 Jan 2025 13:42:49 +0900 Message-ID: <20250130044455.2642465-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 24E7AC0003 X-Stat-Signature: 87m415dogro5ifenca9o87excmjibr6s X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1738212337-400693 X-HE-Meta: U2FsdGVkX18nFme/2I2kzLU0kB9DYIEhLYv5vpyxa0fKsxoU2uXi9m/ESJWKHyJ2+D2nMLkWyFsTGWNo4V8nZf49DLViPilbligodx6xTeO+7pfXQXV4wIxEAqW/ZkYQPfm61UYKF1wXZm89K9buU6DSg2p9UVgFa3yvJQXeOLYemAvAA6Pfvgqzm19r1eaLSkpwwpJqg+u+GqMiNJ9/q7Tvctoc3jYxQL4EBuoYe0ZW8dsHAFzvq4Hfd9lfQyNAjiIGwOgJp0OxgQlUCvH0x9wG/a8CrSWZHPZ3QJIIyI38toRX4DLc1347g0+UQMYTeKr1+ado+5trOh0ZQZkp1+iqEqn0/Y6WNGdLsZNJkYW2nNS/AM/P4F2f7XMqchLIxhAiZFVHhVrLkyGvPUncgrXLuDRQvbAEU6W/pSUjyatifnhDMsrK9QBjJidP3dCqSyzIffHXGq4Q0WkHI6dBnjjcrpKMxAVzQOOrxeUAtCHcL92fFxeu5QFLSYwuPqR4b5BwS4TcOF+Go46axPG+ZP39YAkTUcvxWJyDlXnMi2yWZaKTBJ2BQnaA+Br16yvAAdvvv4CYN7zEq3XzjmXI3budmKYGPPr5ba7LazhX45v3LbLHfW6Iff/vP/if5SIVNCX4glgf8VuZY8yhGG9IBxMgAvYe98DLw1sLQKcWokET5vRqTUn1HKGLObncOJr0mjsnSG1CeB4KlXpjXMUzQYdAGUDCOpt/cE8wVwLQ4q0/WATza3fvQT1nbrgYQRLAIYEU/x3azlNXgwFu3uwo2i8RfWmRVkY5bkI16g7Y+hp/36b2gvrZY7an6/BP2rUueLMvM3RZkx29s74vh9OW39cGE4tHcv5gt3QKWrbCETpPPlrYYMVZAGnHaUOqSJCSHhjv0ESC0HC7QR9QK7m4LgshW44GgeovGa6HIuXHZ9ii7v/j9Mq0sb+Bx6N1/NuW6xxN1QdkBspeghj6UGl mtnkwv1u oQZpLBm+yk08aGRMKBvaDGOV4n6PAzhY4iJTE/tNt0LoJv1BdtM2Btt7OtYeBRmO0zNg/QTwnSwgNbZc5Pmdz1l7i/uPKK6kmt661NgtyK/e7Oabn2NVmxtnlwLiQBZhG5FPrH6zOIMGi8FxKe5TJ0DzmYa0TgQ7E2gAeHC7LKWzlHsjY6L4hu33cQpocCQk4htkVv6ve4jYmbRIXPKAWwbRycmh4N6rOt7Pza1ZhDKPEzuMzEHUgFXbbRy8bSyeMffyy0DY8vVkh/g3MwknQ74KI8EUpjNplmbQQUOS6sUcyhcU50AuwyiZj0w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use new read/write zsmalloc object API. For cases when RO mapped object spans two physical pages (requires temp buffer) compression streams now carry around one extra physical page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +++- drivers/block/zram/zcomp.h | 2 ++ drivers/block/zram/zram_drv.c | 28 ++++++++++------------------ 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index efd5919808d9..675f2a51ad5f 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -45,6 +45,7 @@ static const struct zcomp_ops *backends[] = { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *strm) { comp->ops->destroy_ctx(&strm->ctx); + vfree(strm->local_copy); vfree(strm->buffer); kfree(strm); } @@ -66,12 +67,13 @@ static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp) return NULL; } + strm->local_copy = vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra in case if * compressed data is larger than the original one. */ strm->buffer = vzalloc(2 * PAGE_SIZE); - if (!strm->buffer) { + if (!strm->buffer || !strm->local_copy) { zcomp_strm_free(comp, strm); return NULL; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 62330829db3f..9683d4aa822d 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct list_head entry; /* compression buffer */ void *buffer; + /* local copy of handle memory */ + void *local_copy; struct zcomp_ctx ctx; }; diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index ad3e8885b0d2..d73e8374e9cc 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1562,11 +1562,11 @@ static int read_incompressible_page(struct zram *zram, struct page *page, void *src, *dst; handle = zram_get_handle(zram, index); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src = zs_obj_read_begin(zram->mem_pool, handle, NULL); dst = kmap_local_page(page); copy_page(dst, src); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); return 0; } @@ -1584,11 +1584,11 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) prio = zram_get_priority(zram, index); zstrm = zcomp_stream_get(zram->comps[prio]); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src = zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); dst = kmap_local_page(page); ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); zcomp_stream_put(zram->comps[prio], zstrm); return ret; @@ -1684,7 +1684,7 @@ static int write_incompressible_page(struct zram *zram, struct page *page, u32 index) { unsigned long handle; - void *src, *dst; + void *src; /* * This function is called from preemptible context so we don't need @@ -1701,11 +1701,9 @@ static int write_incompressible_page(struct zram *zram, struct page *page, return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src = kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); @@ -1726,7 +1724,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) int ret = 0; unsigned long handle; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1769,11 +1767,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - zs_unmap_object(zram->mem_pool, handle); zram_slot_write_lock(zram, index); zram_set_handle(zram, index, handle); @@ -1882,7 +1877,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, unsigned int class_index_old; unsigned int class_index_new; u32 num_recomps = 0; - void *src, *dst; + void *src; int ret; handle_old = zram_get_handle(zram, index); @@ -2020,12 +2015,9 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, return 0; } - dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); zcomp_stream_put(zram->comps[prio], zstrm); - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); From patchwork Thu Jan 30 04:42:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13954305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10640C0218F for ; Thu, 30 Jan 2025 04:45:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AAA62800B9; Wed, 29 Jan 2025 23:45:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 959F328004D; Wed, 29 Jan 2025 23:45:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 823382800B9; Wed, 29 Jan 2025 23:45:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D85CD28004D for ; Wed, 29 Jan 2025 23:45:45 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8A417160870 for ; Thu, 30 Jan 2025 04:45:45 +0000 (UTC) X-FDA: 83062880250.08.4103548 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf11.hostedemail.com (Postfix) with ESMTP id A897440009 for ; Thu, 30 Jan 2025 04:45:43 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=AjawapCv; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf11.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.44 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738212343; a=rsa-sha256; cv=none; b=I9sCOLx3BZGj7cM0t6Rr1f2zpZW6+zPlJIdpcRCr5rOJT6bOprO8/IxygDntJqJnKb38hI ftYgP3sCYuCJEj3wLLsUSvtpV5eQ6nXg3Z3NGHp15AtRvQ4SZcaC/bYJK3W0Ev5kH933Np gGwIRLdEagpz5qchQ2/XsXFUCnTc4SM= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=AjawapCv; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf11.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.44 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738212343; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LjUByDm3ba8JrZJ2xgEVKiM3B6T3dmGtNNYJyOcrU3Y=; b=lfragN37FhpxsFTssgE90bLsCV9130Bue1dZqhygAHjH6JUOr9yKq5QB99NHKAsLvge+ev Od5L+iutb1+oh65cPhllsqonohkkvbyxF50tZvQIbFPvfgO4MWW6JZL9gkY+yflms3rUA5 4UD1waHzuh9QVE3b/Atcs+1cVSUX3dc= Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-2ef72924e53so562031a91.3 for ; Wed, 29 Jan 2025 20:45:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212342; x=1738817142; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LjUByDm3ba8JrZJ2xgEVKiM3B6T3dmGtNNYJyOcrU3Y=; b=AjawapCv6yHEtxhSZxATy0fhG4MwOxNQN+ai/wYN3aEyP/eriBQR2oqcSSVGSN/QQe 5QktMabqSvkCnJAI4gkcEbaSRQ8yEp57cRl25wp2OJZorc06jF9PmAJ233AdDOiVUc4V T/2vyW4IcVIBmH1s4F819qP/vOUnxJWsK7HDM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212342; x=1738817142; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LjUByDm3ba8JrZJ2xgEVKiM3B6T3dmGtNNYJyOcrU3Y=; b=Euz3vvkcOH4j9VIjfrt+MuNvRHdF67+6mbCqLnhptDXl7TihDUN1GvyHHLMJFcamIV 719z80sfYz21iSJx+fYds/J6Bsw4yOz/EB5DBrx4VKRGw71J3ANVA8gnBXX5iYHPptbj Mb3JUImP2i9KUq7EVN0aYbPDqNuYHeA5hcRwG94cvcOTEw4GWhvn2ozBnsMYSU/r0NvH BHsIMJy2af7GIws+SAB3vI5y/xny2DeojjjO5lCXiNQYhQdI7ZbkN6KerWsBcVUygdWR 09f5GrmM6PXVOEuI3Q2FnWlSuruYPk91i4vrahjFFgG+tBg9M+Za46o81VwLNy1ZzEA9 DPrg== X-Forwarded-Encrypted: i=1; AJvYcCVhJlwufiGTdziYsH1A5dWRl/E8LywAL7O5RAC3f1456jH1KBFhISxoTL0QyoU90EAXyIo6XH74oA==@kvack.org X-Gm-Message-State: AOJu0Ywu/SsNEV8tgccBglEI9Aonunkfln3u3J35W3KuM6AxJ5AsYZSE CslzOHaXnnsM7lfdtFfoeyN0nP7sspiSV8XwoJ7RWV/F2gEGNzxOi9dHmhBiEA== X-Gm-Gg: ASbGncsijFax9wFxt7Ym3dUwptHzDs+aBQvAbLaifn2WlrWS8j0pxp4S3g+83QC9mlP JqjSFkCyzRkzNbh6L2is8Oqy2hk2LiM3ApGZoVGsv2K4VHA60Cz1tzwyTKkASOm690MYIHUXMuw gadSWII7H9kh2e54Vmx2ZpyyqI1a1xPkXpED730fYrT7VqICJBn01W5bCufzwUogbEpbJthZ8++ 27PNLfmlL6Tpmt7/lM2A+StmqgdvlqqVoYy9I474L/DCv1KrVG3on/6nxp3lmXzMTwURXWxMXQ6 T0btWdEOiQA7FNkCxQ== X-Google-Smtp-Source: AGHT+IGkGP9NGStcdcyDxCKvmSx/umN39Bj6YAsUZh4NBQvKyyiQWE/6vXXCCD2x2gkOQQ39AR/qSQ== X-Received: by 2002:a17:90b:288a:b0:2ee:d35c:3996 with SMTP id 98e67ed59e1d1-2f83ac8371amr8132916a91.31.1738212342552; Wed, 29 Jan 2025 20:45:42 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83a194a0csm2205072a91.1.2025.01.29.20.45.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:42 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 7/7] zram: add might_sleep to zcomp API Date: Thu, 30 Jan 2025 13:42:50 +0900 Message-ID: <20250130044455.2642465-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: A897440009 X-Rspamd-Server: rspam10 X-Stat-Signature: 4padekadjxyj3aw198u1ct4eksw43oh4 X-HE-Tag: 1738212343-192758 X-HE-Meta: U2FsdGVkX1/4/RdgirZ8EQhnr+UeNAPG8IanzRkZ554PwIB0nPAEW79mfSvhgbC0d84231WI19hGgiKJeUrJu0R+3ykQIDi/vTpbf032uNmlamdzhJKXdteZ2KF30D6DF9R5te7HbfyPYf0j8yIeb7vzHQigdEACq8VeJJ4OiYd6qBLxEjqEMQXqU6CFweFLtN+y+5X1tOzBFN3XN59CXfixztlAiB0paLy2ncnOQSm547tt9q6eGdFo6zq+9BAoDKPxQlQ/yL9c68BeZ22+b11cROcgCYFdGLWeOfjbszV17wn3AR7v061b2rrdlWmo21tL0YVG7Vf22CwGXLK0oedZSrHHAcchfKCei6ipwcWvIobvrlLLJ69ta28pu/hdOvOXAnRTRZ3OPGeXXJwIQipCZXKYfaFGT0oVxJ4eSLOtHsrOWU33ArIbkEl86TpkmrbxmdEo6b3J9HczIK9LixgIuDAnA8H/ryCV30owTwVocfO+iz5Oz81IGEG/ORM+tDD8hPNBzPX+W+g3pvkv5DP+7bm3FN+1OtTEfQvGiD7lj3APX67OBJREI6OSrnmRDKVt0VfgipoGStIpPL3dsQl6vDDD0o5Hk83JyBhC3G0sMyrrKHnBLLx3RUlterWaw+ggkSOvBJSNWoSownkL5wytMNzEktIvTe7iHaPmLHsbpoMw6qc7c1UtoAO7Yy5PJL38z/IAYkBzmxiLkg0UJoIcNYdNZyvB6r+ebXJywMbyvmhslXWQ/GgojHsDYuaIHRotXPUhnA9VwJjcTpdRYkWSWTcUlWotqQELvgXFIsBZ+8qw6KaNX2yEphy3XdESb6CiSJTdMLKFkHpL9hsURt8n/V4HEhlejuxzYWem1vEpKRpTEYRVNSLNcEvbWRuo29c+Z+HiXogrV9fErJC+eEB/b4j95SsHSY+SiIkQP0UgrU1M8/ryGRA9NFK1CCfgu0SaaxXeLA5ZRkLS6yW JXOY3Ppi hbvvDNyVdBQZ2dPGm7pjw/TF1+olfTbivc6B44L09WVI/PwUCxB1bYLFSJDDIQpXpADW1jHbRvibGSxD0gcFFebzCSsVWYiYSR9JwC32XQjyQDAM2DFv9+ilv2kAPxbrMK/82Agu7/SCX8tP9zca43HirvL2Riv+IGcFmpqWBSC3yXAppuNigkbwcWmfAWcV7niO9qqaqNcgyLR3rf4AgmJapSPno1BDtjTD/SQilmSRgTA3wWjFK8CJ5N8lElB7EgstuvxVL3QFgjm0X6i6ULsDztVbvv+G11sIwkX7YTg9sXdnuaMh+1hEx19ZM2+6g5m0sb0mw09ujPGbY4dF/aNVPGk3p90W/Uf/v/s3glp1/thE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Explicitly state that zcomp compress/decompress must be called from non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 675f2a51ad5f..f4235735787b 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -185,6 +185,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, }; int ret; + might_sleep(); ret = comp->ops->compress(comp->params, &zstrm->ctx, &req); if (!ret) *dst_len = req.dst_len; @@ -201,6 +202,7 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, .dst_len = PAGE_SIZE, }; + might_sleep(); return comp->ops->decompress(comp->params, &zstrm->ctx, &req); }