From patchwork Tue Jan 30 01:36:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13536676 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55E18C47DB3 for ; Tue, 30 Jan 2024 01:42:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 011806B00A3; Mon, 29 Jan 2024 20:42:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ED8D26B00B3; Mon, 29 Jan 2024 20:42:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D51B76B00B4; Mon, 29 Jan 2024 20:42:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C29346B00A3 for ; Mon, 29 Jan 2024 20:42:28 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9BE42140651 for ; Tue, 30 Jan 2024 01:42:28 +0000 (UTC) X-FDA: 81734277576.28.27D3A77 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by imf20.hostedemail.com (Postfix) with ESMTP id D12DC1C000D for ; Tue, 30 Jan 2024 01:42:26 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=b77cHJX4; spf=pass (imf20.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706578946; a=rsa-sha256; cv=none; b=awUzE4fbXjjX9zIVlGUm59iR3cd334QAIa22x8+MeJi3a577M5E6BsK4Uod0YhRTtrKq/o zkhJSNgDNSbeECzS8BveUrEU0JuHf3ojWPWJXxU2R+lrrWLsUP8/6yPIzAGnReehTWxJEr R1DkT98T1tCRR/QUxy4pf4M9dCRMfz0= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=b77cHJX4; spf=pass (imf20.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706578946; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rkkvsf4pStxQTi2gKgo4CCnvmALR1LUjxBEOZSgiwUU=; b=qF/Pp7RCoe5Au/kHUMBiIkKohQg+zO2GtR9lMuR7agL+1Q1/iBhGMecrXPwuCN18Utm5bF xiX7DbLJsWFQdbCsqelvtaFkUcj+yaqEgoU28IwSJGehG1gQZ7dpyYGGcUYgKquH5Sj/VE JvWJK4UV/5B90+1jnCy003rA9LO4DMo= Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-68c4fb9e7e0so11622886d6.2 for ; Mon, 29 Jan 2024 17:42:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1706578946; x=1707183746; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rkkvsf4pStxQTi2gKgo4CCnvmALR1LUjxBEOZSgiwUU=; b=b77cHJX4WfEqaTl0B3i6uVN9mfwNnc4Wo+ERDV6THJofsASjqZD3tx6NdVBeQKLLe8 wRQoDosR+ZLOKRqIk8fbSSehRSSOKe+NwAav0qiYVmea5nJuMe9YREENqDl2Dlfa7R10 9XPYrJSSLPdMFqiNdFRqT8fz6GLAQrK3n8fU/LrFD6Od2nsAPJzGgdOQxnfthne2fRnt OJMGSTtOPnaaPKOh22g1DoKIH3WPbEgasAzZ2v77SCrBRo4fGl1Kqw7gyzgb4NNAN+FD Mb/z9HyQD5+Ru007ZZGfvQfbqBUFoOp0vPUkSGfXonCsHL3BPXfSMn5KSvrBfzwhX/L/ 3TfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706578946; x=1707183746; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rkkvsf4pStxQTi2gKgo4CCnvmALR1LUjxBEOZSgiwUU=; b=DF+Jt4gQNoiZymoPlNFYH/NmAf1D7AnGrEgH1kyx++DS3Cc1uSxjXLP26GktYaNysL qwxhFh0xcyXWnQgQj70m/EAdWTAPIn/dy3mId4aShveMqKTV0Udfpf9L54xeRpfGiDYv GiSTvIN/wQ/g9nLEXesJkupqcz2gvSLjKY35vv+WVQz0T2uTfKFmIyL16GA8mI/6hhTi hcxrDDbSuUS1mVsdShWO2mdl7rza6ApvdQ6YMLF+/01FoHRsHV2q/STylYyistE/XzPG geBhOTqR6lNggW3twlMXvOd5TE565JI5593uG8RE9ymqxQCvOz5mPnRowSmST5hbqFLS uFMA== X-Gm-Message-State: AOJu0Yy/O5IOilhFUc9p/IcTYmoFdjUcagfYuK16S3wn44bH2fn6bCl7 d4dPdqQFay+i1Lnq1gfoxVkDo6K9Kr31vlJTsaY01wNevEcC+T3dkufftPSKXa0= X-Google-Smtp-Source: AGHT+IEupJwBS6PzRmr4d1gT3dBmbqwm+T5GT7wGrB81djH8pkmwnFJL+V394X2JsA3CiQUdBjgjdA== X-Received: by 2002:ad4:5f89:0:b0:68c:5d15:25c1 with SMTP id jp9-20020ad45f89000000b0068c5d1525c1mr790999qvb.107.1706578946041; Mon, 29 Jan 2024 17:42:26 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id vz9-20020a05620a494900b007833c4dddfdsm3587691qkn.53.2024.01.29.17.42.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 17:42:25 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Nhat Pham , Yosry Ahmed , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 10/20] mm: zswap: function ordering: pool alloc & free Date: Mon, 29 Jan 2024 20:36:46 -0500 Message-ID: <20240130014208.565554-11-hannes@cmpxchg.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240130014208.565554-1-hannes@cmpxchg.org> References: <20240130014208.565554-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D12DC1C000D X-Stat-Signature: 39qme45xzqgny346wx4bcb11kap6455k X-Rspam-User: X-HE-Tag: 1706578946-226261 X-HE-Meta: U2FsdGVkX1+5IV5rRUixLz6r97zBJpM2dlPcrydPn6wgn5qmEw/d6Chge97wI6ZlcDCQ26HSprrP2b8IkRhj+737pJaI7vUOhXPBbFKxkOSpvWRq08vlE7E12TC1jrvOn52sd9Va6AaCJ0i4DAfzM7UHKVh+gqhGnYJdCqgN8txJUTbNg/X+15gMjZqMjtg1SXd+DTp7UKJsSbjxT1Y4FGVeYfQ3MLtIBMqLCmfgT1wGzzKjEI5W46R11cS7yFJRf68a9Iw1sMuq4BfWmsgWWsW6nbbzniUjadnAwdLDkQd3TBCvHJkXoYWduA3NjshCJ1VMX0byLuNngpx85bHHv0PFmkRkkht84d3rG1BfLwbGOiBc/fFCr+tStgC2GYl+3uPptMBApGdwh0IEzcg8w5kdDWvES0TQG2Fp/uHQb43RPXAj9v8GhMwcCGQVLc/rnJs87ExS5Xo3thH2TmYUzsowAPnheq+Lsiw1psMf3tyctnM8Sypg8Lxch6fDaDZBVCO1B6IzJsLXDvAJ++oVKgiWVq06+NQbw0uhTClHNy0i6ZEQDbHmMOoxfxHcKiQa9b/DLacw3JMuRkanxiJEt0968InbcOyvWJbRjd1lZdm/lrC4FvNdhwOC20t6ftCLUVkXPO8yCMz3/S3wqqHoGy9HMWjNPH9uEOfqpaRjrlXMHgl4gBM0+mkO8u5uhclT3GXUauE60FAJ5GecExMZgzH2Ec2C8BpaiHicDe7lLkIZRH8wv8b+GThvTO3BdsNf9CfEv48YnKTgPYAfzWL8bR3yujzmnEvIaN4lgZo0k2eOV5/qo8DPwUjcsRgqZj1UhiYvebn8ndBOoQHoaXCpcU7TALGIOC0UDo0WV+YgLvznCXNoQGYqy2madKh2JRwPB208gJaw6RTjRxAILvLsjQ2BOBYN4Lef1gr+p1jJQ9cEIpJ3kM8f7SIuhWdgYl7/YQpkreXFYfNfrPttde1 6uH3Ezma TbUbXouzDQtCfO/m4Zgv9hK1y2/nYr7fKCxS/+LmDZiXLNVfJTtDkgCiYLO4ufUQYAqtLOzY8lH9Fa/bY2o8IMHCkbykIsLcE9ftxoqVJsFp49HnPTPl2tyWArCSHWLGnCBmSLxz/tW8Lp8qTufRy5QK72g8T86mqVRVDAUbQTWSZzG5BHKSx//al1qtHk1gDZmp5+Q6mDe4+h1xU1gY+tYEKVJ8ajm77wy2HafZY7fgDzoxf1oKT2fo7jUp3THtOjpIyVNCA9tODFKq2o59KUYMcjPrw3AkWQrb292rR65mU/OOcotNhXvr09cY1W00mcL6YvWiORRAJROFa+pTkGtjF7xaWt8Re1NbyKi5NNLd2DstbNNSAuUhX1VYf7RDuwzkrh8iUr3u4aS2NX2dEjYSAXWicY61m93Ps/ve5dVapLBU/g7xJQfYsG3k1j/LX2cT4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The function ordering in zswap.c is a little chaotic, which requires jumping in unexpected directions when following related code. This is a series of patches that brings the file into the following order: - pool functions - lru functions - rbtree functions - zswap entry functions - compression/backend functions - writeback & shrinking functions - store, load, invalidate, swapon, swapoff - debugfs - init But it has to be split up such the moving still produces halfway readable diffs. In this patch, move pool allocation and freeing functions. Signed-off-by: Johannes Weiner Reviewed-by: Nhat Pham --- mm/zswap.c | 297 +++++++++++++++++++++++++++-------------------------- 1 file changed, 152 insertions(+), 145 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 082d076a758d..805d9a35f633 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -320,6 +320,158 @@ static void zswap_update_total_size(void) zswap_pool_total_size = total; } +/********************************* +* pool functions +**********************************/ + +static void zswap_alloc_shrinker(struct zswap_pool *pool); +static void shrink_worker(struct work_struct *w); + +static struct zswap_pool *zswap_pool_create(char *type, char *compressor) +{ + int i; + struct zswap_pool *pool; + char name[38]; /* 'zswap' + 32 char (max) num + \0 */ + gfp_t gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; + int ret; + + if (!zswap_has_pool) { + /* if either are unset, pool initialization failed, and we + * need both params to be set correctly before trying to + * create a pool. + */ + if (!strcmp(type, ZSWAP_PARAM_UNSET)) + return NULL; + if (!strcmp(compressor, ZSWAP_PARAM_UNSET)) + return NULL; + } + + pool = kzalloc(sizeof(*pool), GFP_KERNEL); + if (!pool) + return NULL; + + for (i = 0; i < ZSWAP_NR_ZPOOLS; i++) { + /* unique name for each pool specifically required by zsmalloc */ + snprintf(name, 38, "zswap%x", + atomic_inc_return(&zswap_pools_count)); + + pool->zpools[i] = zpool_create_pool(type, name, gfp); + if (!pool->zpools[i]) { + pr_err("%s zpool not available\n", type); + goto error; + } + } + pr_debug("using %s zpool\n", zpool_get_type(pool->zpools[0])); + + strscpy(pool->tfm_name, compressor, sizeof(pool->tfm_name)); + + pool->acomp_ctx = alloc_percpu(*pool->acomp_ctx); + if (!pool->acomp_ctx) { + pr_err("percpu alloc failed\n"); + goto error; + } + + ret = cpuhp_state_add_instance(CPUHP_MM_ZSWP_POOL_PREPARE, + &pool->node); + if (ret) + goto error; + + zswap_alloc_shrinker(pool); + if (!pool->shrinker) + goto error; + + pr_debug("using %s compressor\n", pool->tfm_name); + + /* being the current pool takes 1 ref; this func expects the + * caller to always add the new pool as the current pool + */ + kref_init(&pool->kref); + INIT_LIST_HEAD(&pool->list); + if (list_lru_init_memcg(&pool->list_lru, pool->shrinker)) + goto lru_fail; + shrinker_register(pool->shrinker); + INIT_WORK(&pool->shrink_work, shrink_worker); + atomic_set(&pool->nr_stored, 0); + + zswap_pool_debug("created", pool); + + return pool; + +lru_fail: + list_lru_destroy(&pool->list_lru); + shrinker_free(pool->shrinker); +error: + if (pool->acomp_ctx) + free_percpu(pool->acomp_ctx); + while (i--) + zpool_destroy_pool(pool->zpools[i]); + kfree(pool); + return NULL; +} + +static struct zswap_pool *__zswap_pool_create_fallback(void) +{ + bool has_comp, has_zpool; + + has_comp = crypto_has_acomp(zswap_compressor, 0, 0); + if (!has_comp && strcmp(zswap_compressor, + CONFIG_ZSWAP_COMPRESSOR_DEFAULT)) { + pr_err("compressor %s not available, using default %s\n", + zswap_compressor, CONFIG_ZSWAP_COMPRESSOR_DEFAULT); + param_free_charp(&zswap_compressor); + zswap_compressor = CONFIG_ZSWAP_COMPRESSOR_DEFAULT; + has_comp = crypto_has_acomp(zswap_compressor, 0, 0); + } + if (!has_comp) { + pr_err("default compressor %s not available\n", + zswap_compressor); + param_free_charp(&zswap_compressor); + zswap_compressor = ZSWAP_PARAM_UNSET; + } + + has_zpool = zpool_has_pool(zswap_zpool_type); + if (!has_zpool && strcmp(zswap_zpool_type, + CONFIG_ZSWAP_ZPOOL_DEFAULT)) { + pr_err("zpool %s not available, using default %s\n", + zswap_zpool_type, CONFIG_ZSWAP_ZPOOL_DEFAULT); + param_free_charp(&zswap_zpool_type); + zswap_zpool_type = CONFIG_ZSWAP_ZPOOL_DEFAULT; + has_zpool = zpool_has_pool(zswap_zpool_type); + } + if (!has_zpool) { + pr_err("default zpool %s not available\n", + zswap_zpool_type); + param_free_charp(&zswap_zpool_type); + zswap_zpool_type = ZSWAP_PARAM_UNSET; + } + + if (!has_comp || !has_zpool) + return NULL; + + return zswap_pool_create(zswap_zpool_type, zswap_compressor); +} + +static void zswap_pool_destroy(struct zswap_pool *pool) +{ + int i; + + zswap_pool_debug("destroying", pool); + + shrinker_free(pool->shrinker); + cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); + free_percpu(pool->acomp_ctx); + list_lru_destroy(&pool->list_lru); + + spin_lock(&zswap_pools_lock); + mem_cgroup_iter_break(NULL, pool->next_shrink); + pool->next_shrink = NULL; + spin_unlock(&zswap_pools_lock); + + for (i = 0; i < ZSWAP_NR_ZPOOLS; i++) + zpool_destroy_pool(pool->zpools[i]); + kfree(pool); +} + /* should be called under RCU */ #ifdef CONFIG_MEMCG static inline struct mem_cgroup *mem_cgroup_from_entry(struct zswap_entry *entry) @@ -970,151 +1122,6 @@ static void shrink_worker(struct work_struct *w) zswap_pool_put(pool); } -static struct zswap_pool *zswap_pool_create(char *type, char *compressor) -{ - int i; - struct zswap_pool *pool; - char name[38]; /* 'zswap' + 32 char (max) num + \0 */ - gfp_t gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; - int ret; - - if (!zswap_has_pool) { - /* if either are unset, pool initialization failed, and we - * need both params to be set correctly before trying to - * create a pool. - */ - if (!strcmp(type, ZSWAP_PARAM_UNSET)) - return NULL; - if (!strcmp(compressor, ZSWAP_PARAM_UNSET)) - return NULL; - } - - pool = kzalloc(sizeof(*pool), GFP_KERNEL); - if (!pool) - return NULL; - - for (i = 0; i < ZSWAP_NR_ZPOOLS; i++) { - /* unique name for each pool specifically required by zsmalloc */ - snprintf(name, 38, "zswap%x", - atomic_inc_return(&zswap_pools_count)); - - pool->zpools[i] = zpool_create_pool(type, name, gfp); - if (!pool->zpools[i]) { - pr_err("%s zpool not available\n", type); - goto error; - } - } - pr_debug("using %s zpool\n", zpool_get_type(pool->zpools[0])); - - strscpy(pool->tfm_name, compressor, sizeof(pool->tfm_name)); - - pool->acomp_ctx = alloc_percpu(*pool->acomp_ctx); - if (!pool->acomp_ctx) { - pr_err("percpu alloc failed\n"); - goto error; - } - - ret = cpuhp_state_add_instance(CPUHP_MM_ZSWP_POOL_PREPARE, - &pool->node); - if (ret) - goto error; - - zswap_alloc_shrinker(pool); - if (!pool->shrinker) - goto error; - - pr_debug("using %s compressor\n", pool->tfm_name); - - /* being the current pool takes 1 ref; this func expects the - * caller to always add the new pool as the current pool - */ - kref_init(&pool->kref); - INIT_LIST_HEAD(&pool->list); - if (list_lru_init_memcg(&pool->list_lru, pool->shrinker)) - goto lru_fail; - shrinker_register(pool->shrinker); - INIT_WORK(&pool->shrink_work, shrink_worker); - atomic_set(&pool->nr_stored, 0); - - zswap_pool_debug("created", pool); - - return pool; - -lru_fail: - list_lru_destroy(&pool->list_lru); - shrinker_free(pool->shrinker); -error: - if (pool->acomp_ctx) - free_percpu(pool->acomp_ctx); - while (i--) - zpool_destroy_pool(pool->zpools[i]); - kfree(pool); - return NULL; -} - -static struct zswap_pool *__zswap_pool_create_fallback(void) -{ - bool has_comp, has_zpool; - - has_comp = crypto_has_acomp(zswap_compressor, 0, 0); - if (!has_comp && strcmp(zswap_compressor, - CONFIG_ZSWAP_COMPRESSOR_DEFAULT)) { - pr_err("compressor %s not available, using default %s\n", - zswap_compressor, CONFIG_ZSWAP_COMPRESSOR_DEFAULT); - param_free_charp(&zswap_compressor); - zswap_compressor = CONFIG_ZSWAP_COMPRESSOR_DEFAULT; - has_comp = crypto_has_acomp(zswap_compressor, 0, 0); - } - if (!has_comp) { - pr_err("default compressor %s not available\n", - zswap_compressor); - param_free_charp(&zswap_compressor); - zswap_compressor = ZSWAP_PARAM_UNSET; - } - - has_zpool = zpool_has_pool(zswap_zpool_type); - if (!has_zpool && strcmp(zswap_zpool_type, - CONFIG_ZSWAP_ZPOOL_DEFAULT)) { - pr_err("zpool %s not available, using default %s\n", - zswap_zpool_type, CONFIG_ZSWAP_ZPOOL_DEFAULT); - param_free_charp(&zswap_zpool_type); - zswap_zpool_type = CONFIG_ZSWAP_ZPOOL_DEFAULT; - has_zpool = zpool_has_pool(zswap_zpool_type); - } - if (!has_zpool) { - pr_err("default zpool %s not available\n", - zswap_zpool_type); - param_free_charp(&zswap_zpool_type); - zswap_zpool_type = ZSWAP_PARAM_UNSET; - } - - if (!has_comp || !has_zpool) - return NULL; - - return zswap_pool_create(zswap_zpool_type, zswap_compressor); -} - -static void zswap_pool_destroy(struct zswap_pool *pool) -{ - int i; - - zswap_pool_debug("destroying", pool); - - shrinker_free(pool->shrinker); - cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); - free_percpu(pool->acomp_ctx); - list_lru_destroy(&pool->list_lru); - - spin_lock(&zswap_pools_lock); - mem_cgroup_iter_break(NULL, pool->next_shrink); - pool->next_shrink = NULL; - spin_unlock(&zswap_pools_lock); - - for (i = 0; i < ZSWAP_NR_ZPOOLS; i++) - zpool_destroy_pool(pool->zpools[i]); - kfree(pool); -} - static int __must_check zswap_pool_get(struct zswap_pool *pool) { if (!pool)