From patchwork Thu Dec 28 09:45:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13505764 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41662C47073 for ; Thu, 28 Dec 2023 09:46:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B31F66B00B2; Thu, 28 Dec 2023 04:46:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ABBC96B00B3; Thu, 28 Dec 2023 04:46:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 982766B00B4; Thu, 28 Dec 2023 04:46:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 83A3C6B00B2 for ; Thu, 28 Dec 2023 04:46:25 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 530521403D7 for ; Thu, 28 Dec 2023 09:46:25 +0000 (UTC) X-FDA: 81615746730.03.0EED690 Received: from out-186.mta0.migadu.com (out-186.mta0.migadu.com [91.218.175.186]) by imf09.hostedemail.com (Postfix) with ESMTP id 6CA96140015 for ; Thu, 28 Dec 2023 09:46:23 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf09.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.186 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703756783; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jx1R7SbuEotMSzAWjxU0CvOuBYFjtfh3gj1fldL0jmQ=; b=xWXhXVlXFbCIYPyttf4Kjcfl90VhxyyIZXV7gdtPO9pPMKQ931EarHo5DdvyWTa0AiNGYI uC3M67pTbH0WjP/DFdEoQhdmEMxdgNNc/dAfZa8UAXuhcmwrV1pyQq455ll8T/ewquLfEF vXujYVQTTde9R/T2nJ+/lLtPtKrN4PI= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf09.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.186 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703756783; a=rsa-sha256; cv=none; b=Eo220QwGMWPfp8tMnoWnCyYd1R3A0pkjM8Z2u3YQ/sgFRnyc0GNx/Lprf+/k/QyOPRe8Xv 3naMGEd/5OfZlfoR05bwacYl066EbIaAanAWXYQ/nqpScV+4wYCeinkjiJudkjxCUd3zBH zpbLHRmwTrcmsDp2/X6+4L0SRzIOF3E= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 28 Dec 2023 09:45:42 +0000 Subject: [PATCH v5 1/5] mm/zswap: reuse dstmem when decompress MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v5-1-9382162bbf05@bytedance.com> References: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> To: Barry Song <21cnbao@gmail.com>, Yosry Ahmed , Nhat Pham , Andrew Morton , Dan Streetman , Vitaly Wool , Johannes Weiner , Chris Li , Seth Jennings Cc: Yosry Ahmed , Nhat Pham , Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1703756775; l=4288; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=oTSr8h2HN5ZVvOErnBDPCcywNhpihCGwWVxYTWIN01A=; b=eZRaOUfTwA7ZmZqcFbY1sJcAk45CveTJ0kOs78M5taspw03yqX+HB6o5DFZ5Pe3HgwS+K9uVc H//cxrexvlCDM7yUYCt/hUgQCvkUWd1J3KZwst83q4EeCVmrfLBqRCg X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Stat-Signature: gc5f4o3w46rzmorxwoe8xnuckhxubdzm X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6CA96140015 X-Rspam: Yes X-HE-Tag: 1703756783-165320 X-HE-Meta: U2FsdGVkX19ZsNbR9eTDUqgAXz3ufHBSEmiEDbPtCln9K3LlJKB0ZwiK3u0arnBSwoAtdOLggZjyTMhVUyfHFLqKI34tM8aXsjYP7MCb2A6DHRfnqivYoAxmGnDBZSKEPXoKbCTIHU8kc45/oN8T4vvTS/KCqC4oUj3cVlqrtWHancO40fSC+m5+aHlrTFrtxWbLgNVpqx3SjqZHuOonFrWgqov58AODQCsA46MHbZsG6qd1LxwyrhA9ub8Blcig1Sc5J4WB5DHKp/CKy0rOieaLvLSWNhd2dHtn6s1dIsr7+n4pZk3X5YDwua9Nv6ZGm/BKmfUuNKxTzaSaGb6W5y5v78sJyVMQj/tbGLRy3mi7jMExopWatZkckpFiFVc4tQ1QqJ/91WquvrQdqvyDTNYaYKQTZBpRdDJDmB8KdKIq2RpoNwHtYR/IQ4+CasDwPiSTiDM6m5F4Ll87lKiHevLPunYiLS95KtP/LedfttLdV+GYR/sq5tN+UljuPI6q/5V0WRIr5Ra/NQ9KH6RWL9bgEQJY2Ifh2uZWN/pAR8ztW2x/vWKdSHqBksJlOZChTRBNRQSCVWFpW0I7bGMafj1o3mWpXMvGz84SXMw/gFzoKl4yAWWeuKIt580jT5BTQY8YyeURQx0+fCQBTGazmW1zsasbTK5fEOa1ZRx+Z4MATIbx7GrwQtgIcUoerRtYoXJhi+xTScTEInmlhOnvAu/pKuZYRSzxeB79ufFPEvjS6JqWjs5YpZrfN8/ikGUjGg5XEpNrcYWnrV3h9Z9w9VAC743sa6thBrAEctriV/O4H+IfK+P6hsqCC3BMzbeQ79Ntyn9XkqrKABNUwKKizbZ4YOqI9MDU4lqZa/yQ5HD/DH4IEgmtsSTIzxLROZ1bct5TgIUe1vRP8wTU05Ba0PyQGjr7JfhFA65sCbIoYuuvEFpbq7P5b10TAxP6yCQjE+9SyIpZOrf63/mODgm 35X1s9QB VUvy0/G2hnxY1qOPVokQihZRdtgJFs/RcHlPmQEC0I+k46HT2vE2kPDBpVZssIeUjpHoJvAUlU+Ne2RqACgqbp1VzNjnugNGjCj9k62K5KgD0M9gtzyvDXqDYbfmQfJQKxuc+sT0IdoTj/Daxqo8IaL0TuAMRvjDZ6Oqtvwb3nN2Ig0eeqmz3qhG+KblR8cTHJmbCfL76tgdgFGT0X7wpboroLAFZcYcdqo1axbwTz5PT4B7/E0nHKT0zk4q5K+heKuKb1hlOl3TzHz5gieQUoD2B9BRLYfvZXLBbJHOylceUpj4RlSLVri1X+qToMScdWiJ9ByEQlUyK4jTA+zcdQuSJZq/II1p7ggUjEd/07lC4qHcOl0vr6DcxBnZRx0jnADg74cTazFjUnGFQQL1BK8KKEpR0FyzZrj5tWAYQXpYacCs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the !zpool_can_sleep_mapped() case such as zsmalloc, we need to first copy the entry->handle memory to a temporary memory, which is allocated using kmalloc. Obviously we can reuse the per-compressor dstmem to avoid allocating every time, since it's percpu-compressor and protected in percpu mutex. Reviewed-by: Nhat Pham Reviewed-by: Yosry Ahmed Acked-by: Chris Li (Google) Signed-off-by: Chengming Zhou --- mm/zswap.c | 44 ++++++++++++-------------------------------- 1 file changed, 12 insertions(+), 32 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 7ee54a3d8281..41df2d97951b 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1417,19 +1417,13 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct crypto_acomp_ctx *acomp_ctx; struct zpool *pool = zswap_find_zpool(entry); bool page_was_allocated; - u8 *src, *tmp = NULL; + u8 *src; unsigned int dlen; int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, }; - if (!zpool_can_sleep_mapped(pool)) { - tmp = kmalloc(PAGE_SIZE, GFP_KERNEL); - if (!tmp) - return -ENOMEM; - } - /* try to allocate swap cache page */ mpol = get_task_policy(current); page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, @@ -1465,15 +1459,15 @@ static int zswap_writeback_entry(struct zswap_entry *entry, /* decompress */ acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); dlen = PAGE_SIZE; + mutex_lock(acomp_ctx->mutex); src = zpool_map_handle(pool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(pool)) { - memcpy(tmp, src, entry->length); - src = tmp; + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; zpool_unmap_handle(pool, entry->handle); } - mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); sg_set_page(&output, page, PAGE_SIZE, 0); @@ -1482,9 +1476,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, dlen = acomp_ctx->req->dlen; mutex_unlock(acomp_ctx->mutex); - if (!zpool_can_sleep_mapped(pool)) - kfree(tmp); - else + if (zpool_can_sleep_mapped(pool)) zpool_unmap_handle(pool, entry->handle); BUG_ON(ret); @@ -1508,9 +1500,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry, return ret; fail: - if (!zpool_can_sleep_mapped(pool)) - kfree(tmp); - /* * If we get here because the page is already in swapcache, a * load may be happening concurrently. It is safe and okay to @@ -1772,7 +1761,7 @@ bool zswap_load(struct folio *folio) struct zswap_entry *entry; struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; - u8 *src, *dst, *tmp; + u8 *src, *dst; struct zpool *zpool; unsigned int dlen; bool ret; @@ -1797,26 +1786,19 @@ bool zswap_load(struct folio *folio) } zpool = zswap_find_zpool(entry); - if (!zpool_can_sleep_mapped(zpool)) { - tmp = kmalloc(entry->length, GFP_KERNEL); - if (!tmp) { - ret = false; - goto freeentry; - } - } /* decompress */ dlen = PAGE_SIZE; - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(acomp_ctx->mutex); + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(zpool)) { - memcpy(tmp, src, entry->length); - src = tmp; + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; zpool_unmap_handle(zpool, entry->handle); } - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); sg_set_page(&output, page, PAGE_SIZE, 0); @@ -1827,15 +1809,13 @@ bool zswap_load(struct folio *folio) if (zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); - else - kfree(tmp); ret = true; stats: count_vm_event(ZSWPIN); if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); -freeentry: + spin_lock(&tree->lock); if (ret && zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); From patchwork Thu Dec 28 09:45:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13505765 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62D74C3DA6E for ; Thu, 28 Dec 2023 09:46:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB7F86B00B5; Thu, 28 Dec 2023 04:46:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C69278D0009; Thu, 28 Dec 2023 04:46:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABB376B00B7; Thu, 28 Dec 2023 04:46:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8829C6B00B5 for ; Thu, 28 Dec 2023 04:46:28 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 524D0C0819 for ; Thu, 28 Dec 2023 09:46:28 +0000 (UTC) X-FDA: 81615746856.16.D473D1D Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) by imf14.hostedemail.com (Postfix) with ESMTP id 5EEA3100008 for ; Thu, 28 Dec 2023 09:46:26 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf14.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703756786; a=rsa-sha256; cv=none; b=IizcvAx2matXrTUKaV9tqjGlxI308vgWHabxg2EIAXpc6rqxTuTAIolGJlu5Yfpmnjpns8 yEgBeZN7txsT0q6hOsQQRVJY9KmR6eWHPpRIqrpdT++YowYkHRlSItd+iIECjZbDdIXKnV l9MVDjDLXpvF1Ar4Tqmd4zqin68iIZw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf14.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703756786; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fA8EENSpBF/MCxUayb+Fm/Vp/azeMRMIqDljRrP6nzM=; b=Vh/6pvC2HtRYMGthz54C52MDN0DcYjCxn4S7FCZeUmSWhMqvaochg/Oeij+7dtUSxwwCWA Vl2vFkfmw1r6S/0E3JRWHAKVcJ3MJd8CwaoyjJF0eUqT0NRRY6MBUD0mEy0YlyRrvqBDZH zftEb8zhSUFa/5H0JMgw7pNXCk7bym8= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 28 Dec 2023 09:45:43 +0000 Subject: [PATCH v5 2/5] mm/zswap: refactor out __zswap_load() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v5-2-9382162bbf05@bytedance.com> References: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> To: Barry Song <21cnbao@gmail.com>, Yosry Ahmed , Nhat Pham , Andrew Morton , Dan Streetman , Vitaly Wool , Johannes Weiner , Chris Li , Seth Jennings Cc: Yosry Ahmed , Nhat Pham , Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1703756775; l=4752; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=lnynlxAzoO/I6Zt1K1UK1+SZANl9aJDpvDvBQOGa6zI=; b=Mw2uecc/fI59Mx5V+rK/pUM1En4Pm8RmX3vEwb1MyZ0/cgeVN1t7roGLvcoNdg1G08Pdf9aa4 JejgLpSnatdD186CpB+LEGo7WHv27VAVnQK65dalGBCWdTDn/ViMoHk X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 5EEA3100008 X-Stat-Signature: 38eeh7otj99o6xmb98qgzgsrrfxd4x4d X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspam: Yes X-HE-Tag: 1703756786-245743 X-HE-Meta: U2FsdGVkX18Xa6YkRQBNtlMiv3oBJVDVfallmRy623ls5lkCfflyHxKPXK8+dCkDeIavBv7f6xsSiIlEBnYoaSjIfrB9bz3EAaqqJh6qg5YRQ2y5WRmBfDaEKL/cfiC3A2Za60bm1oqWbFM4dFkA57CsC3F9Rh5soKXBnQw1eZWH7lbL+1jimGfjRr9+ajUGONO8Hf05024bOIPdquoii9vhiH8bG3ZhjlhXNZLg10iekYzGnwoSE7gdeDj5fqkoi1yvPjkuGR4qHngylAOLtPZ4LtC6ghALboZhT2/7nyZi+1K816YPBq8q+g3Fyvz81dyC1bb8L6duct1WEW/IVcZRSHLOOOaCD1bAS4vHdF6/5/87zfDFkSiJOpZY6yVP+Lsi9UIVLS4K9AARyjMlB0FUmel0JyIeSbHlUISO7Q4uw7CHIay8bdENDWd8Dry2XtpUNj9EA9a7YRdOoSum7UteidsJZmL3c76PISnlNuTj/GVQ8OYtTeTFgvNyA1+WNC5SU+Y5+4kJs1dG3TvIFECQibkmbM4d6Epz/COAx0LCbcTu5vQGiT+j1lQ+gBz4WQdxiDQd9BgL1XqAW6l6TFwIb6g1BBZHR/mU59grKyvRy2MRbOUN63U+1K6ETTqvWx+m8cGua8D8yM3Mvgr7gzKn/jNUf9Jn4EJPAA7CsMdUL7OfdABCXhCFQXhA+OHwqUpwddBm7NBkB/1DDtC/YoO0ZdfHlEZP81sUIiiR5HFxHVTwYbTHRqNyT6ya01LqgMI4bKjDXWgrNZTLrEqIvxumQzCvaheb4EikjVOrCkv6zztmew0QF6zT06b55qfiaL8mxykHk2Zr4HKgHMYwWOcnaELyM2h7jF5Kk0OMmY7sV0IJ2ack7074LSgoHUj5QrUfJfrCwxeRk5qLXHO8HSbZB96ziuXjbkfUIogXpCkA9/XpY4NBaCZ+y89AnbsVO+8opKJ20EXEwrRYxOI awrewJKC 3OmqxVq6GE4vTjQquM3K/vIZLeZp4gJ4/7eDEexY4aHX3yzPrxXHs7KvkkzT/DKaXh3o0Ju0cXIE1uCX7q9uwnuqNfMKu/2FS7W0MvRhiyWIGMr2RYjAxpDB0yMdNYMrDjJv2avorgdIbl+XBILknq3R6O1JKHuuYuvyw9D3/5Zp6pcR5hgcbX90q+LJXvWzO/1Dvg/8f7YPERylYi7ISFpCbm58PyZucUPjUgaOiFYCRAS7zk+BYfZiCZEKiohX46/zf6y61c4O+wHg2nKybc1Gs6szotXYQFcZjheCqPq0WgKd2tYd7oSkHr8BpFFa1mf8b9vI64FuxCeHU/rV+aK5O1LOmKz+MPgJ6jl3ajrWY2pU4RLH/8bHPzjJ6mJ0zYl3pRYAJOZkCZVfYn2rqK34d2EDCcHjh9lZ4eVBLiFSzTZacmXx6/ggWlfz7mvlwtsqheDIlnPz4TiRygj7/o6CLpPOGOi8Hc3/fdt54CX7rUC7ZqOhCS26vM441XBmrPH6aF3ZocQfwJVvWiMi95WUmlOLNfdqIcoVrXJm1h40MFkVMDtpXd1KC3odG4tk2GxzES3EPBJxeOdQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The zswap_load() and zswap_writeback_entry() have the same part that decompress the data from zswap_entry to page, so refactor out the common part as __zswap_load(entry, page). Reviewed-by: Nhat Pham Reviewed-by: Yosry Ahmed Acked-by: Chris Li (Google) Signed-off-by: Chengming Zhou --- mm/zswap.c | 92 ++++++++++++++++++++++---------------------------------------- 1 file changed, 32 insertions(+), 60 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 41df2d97951b..b25d7d03851d 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1392,6 +1392,35 @@ static int zswap_enabled_param_set(const char *val, return ret; } +static void __zswap_load(struct zswap_entry *entry, struct page *page) +{ + struct zpool *zpool = zswap_find_zpool(entry); + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; + u8 *src; + + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(acomp_ctx->mutex); + + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + if (!zpool_can_sleep_mapped(zpool)) { + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; + zpool_unmap_handle(zpool, entry->handle); + } + + sg_init_one(&input, src, entry->length); + sg_init_table(&output, 1); + sg_set_page(&output, page, PAGE_SIZE, 0); + acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); + BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); + BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); + mutex_unlock(acomp_ctx->mutex); + + if (zpool_can_sleep_mapped(zpool)) + zpool_unmap_handle(zpool, entry->handle); +} + /********************************* * writeback code **********************************/ @@ -1413,12 +1442,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, swp_entry_t swpentry = entry->swpentry; struct page *page; struct mempolicy *mpol; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; - struct zpool *pool = zswap_find_zpool(entry); bool page_was_allocated; - u8 *src; - unsigned int dlen; int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, @@ -1456,31 +1480,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, } spin_unlock(&tree->lock); - /* decompress */ - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - dlen = PAGE_SIZE; - mutex_lock(acomp_ctx->mutex); - - src = zpool_map_handle(pool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(pool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; - zpool_unmap_handle(pool, entry->handle); - } - - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); - ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; - mutex_unlock(acomp_ctx->mutex); - - if (zpool_can_sleep_mapped(pool)) - zpool_unmap_handle(pool, entry->handle); - - BUG_ON(ret); - BUG_ON(dlen != PAGE_SIZE); + __zswap_load(entry, page); /* page is up to date */ SetPageUptodate(page); @@ -1759,11 +1759,7 @@ bool zswap_load(struct folio *folio) struct page *page = &folio->page; struct zswap_tree *tree = zswap_trees[type]; struct zswap_entry *entry; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; - u8 *src, *dst; - struct zpool *zpool; - unsigned int dlen; + u8 *dst; bool ret; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1785,31 +1781,7 @@ bool zswap_load(struct folio *folio) goto stats; } - zpool = zswap_find_zpool(entry); - - /* decompress */ - dlen = PAGE_SIZE; - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); - - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; - zpool_unmap_handle(zpool, entry->handle); - } - - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); - if (crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)) - WARN_ON(1); - mutex_unlock(acomp_ctx->mutex); - - if (zpool_can_sleep_mapped(zpool)) - zpool_unmap_handle(zpool, entry->handle); - + __zswap_load(entry, page); ret = true; stats: count_vm_event(ZSWPIN); From patchwork Thu Dec 28 09:45:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13505766 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30605C3DA6E for ; Thu, 28 Dec 2023 09:46:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF4056B00B7; Thu, 28 Dec 2023 04:46:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BA4056B00B8; Thu, 28 Dec 2023 04:46:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABE736B00B9; Thu, 28 Dec 2023 04:46:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9CC1D6B00B7 for ; Thu, 28 Dec 2023 04:46:31 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6D6E940719 for ; Thu, 28 Dec 2023 09:46:31 +0000 (UTC) X-FDA: 81615746982.30.2A3A230 Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) by imf08.hostedemail.com (Postfix) with ESMTP id 859E8160011 for ; Thu, 28 Dec 2023 09:46:29 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703756789; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w25swSa3ioqiVHC69Omi02hh7LN7MiRLnWUcqit8XqE=; b=uVJbLRnycFxCIkU2kC1c0aMYWOEYH7R+mFtcKlfJ0KRiLAgB0WgMD6yCIvvsJHPZxJYUEI t/qTbOX6p9x1tdUW0CKu4UPQYGLEelAk1CRiwZ64nt++ZOXBhLTVxEwWWHwQsOQg2Xf9SQ UDuBLxk//7b9kPdussX0tcTgWVbHjcI= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703756789; a=rsa-sha256; cv=none; b=1yEHET9PlfK2bBmd7hsUhZxTCgZYV/ansTF8dY/WsVmu0SwccfqmYk3ynXiAIpb9IOKUrH e/cZevdav8+KUoif881FkN34HsbvgGyazm55TczIDlUVxeLA6qmDi2mip1z/Xr8LR+FeN4 ImxLKomOt+9wzBr95FRhSuwl3iSXcCE= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 28 Dec 2023 09:45:44 +0000 Subject: [PATCH v5 3/5] mm/zswap: cleanup zswap_load() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v5-3-9382162bbf05@bytedance.com> References: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> To: Barry Song <21cnbao@gmail.com>, Yosry Ahmed , Nhat Pham , Andrew Morton , Dan Streetman , Vitaly Wool , Johannes Weiner , Chris Li , Seth Jennings Cc: Yosry Ahmed , Nhat Pham , Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1703756775; l=1602; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=rJOYhYvLkD8XDGMZAga2jLzLZQr3NVhVcf7hB//cwrw=; b=obzgcjVPz/TBWcJwAld57WtoQimnl/ZjBFTNyWHoE9HJCR+kwFjlb7IU5MGOgMtiFZtkwTCk3 qPI+KW2XdgCDV21WlpxiCTcYbyZmw3o8+Ampqb53lIJjyKaDdsVv8Mc X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Stat-Signature: 56qyscufhwd9i184bqc4ai5chm4zaawf X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 859E8160011 X-Rspam: Yes X-HE-Tag: 1703756789-402604 X-HE-Meta: U2FsdGVkX1+4Da3HA4smiEoUa2AFI02MvsB3VKEiSsXAjb+Cy4CwjwGrqjeE8BSpmBRdvBb/FpIOw63MSW/C7+vYLk6apLCZW9ckLTVyJUMSvOV/2W6uz5wtP8Phx6s9all5/0Ve5gZYZfD1akxdAV5zanURLblpzBHiU343+ZHis1V7gzHIJrPtZGdicD0NVqzSzOamS9UJtHwfgWewua7U4wwkjwisv9e0BTx2Z8c+wJehQJltJtB0RnwehbA/z7o5SiiLR11N62yJJ7ecGsChamswzOvS2096l445g/XARtBqzAhSg6kR26eGNqfkhifUNa49SpKVb7+9h2QD1hFsvBCev5VPdJvG3Y9XQ6A5kGcigqV/11ePnaYxR3pnq0q9b5oHevoF6ktiraYDmX4rgyq4ianupIQwWIYTWzdLklPgvyEQd6e2Xb+tEESX1SwF+spE+PZ2EqG2S9kDu88LH+8NctxaCVwny/HkW0hn2qmue+z8HGud7NNwIuKR4yCdULan0G8JKz3yGlcJ36uqckkxU+r3U+RcGOxppZliuASJWPJNRhaIhWm3YY2oZUOQdFSEfvfG4Q2/VeRyxuJVq/4KPFQARoeAS+Q6JsMrk/235ixAT2BgDVgvVfA0OoBXTTzjV6KLyUq6mqC42G/qhUZ89aqjculrlUIRy5OOVtssRks+XbK1auGD37ye8fclVKHKvQ+uE5PJ24V2qoTC86YxsuZUlNXBDr5g9E2kHS44xJN3eSBy/XtUebccjSRVTiq6LFlAU7NoRRBLlHmhehyZBZzL+gTOEdGGIFITl5Yn90gs5ZXb9yFF9oY5zl5FsZt6l4cVahzuTtO+eqJ/+Pjl8nCOAaVe8hweANgyy8LrPPNIYA+Ol/RRycqYD/1kUohd+jy2v6cLgZO9Tu4qAigdwfbZRyn+vsMJ2zq+g//Y06nSZKSBIRRS+0014+b/sxcr7mePajoa6Oh dZ++1Z/e h6sA+rP0oxbFAmDOYINRr/XDQ+eppDpA0LkOogLzd+c8gSB9VtwP7SKOAUv8Oe91KLoKjJoUYqR8l059hWUyZCh/gCBxr9/frRASE/UZFk1bfb47VU7IrRO6OxFYF+fFHh/NAxFUDr+5cN/mn1hIoDag4g9v4TE206cnhsnwD6SwtllaTm7OjTJ/T69f6Fv/LaTWKofk9yrVcHadxAGn3zbUWc+nsJ1MU9qMc89vO/5CFaB0mO1E9D4A2VXag9PYC3ASkVXIn5swQzkGy+pB99MVRANbemxBsZrWZuizpzTeIQccLBTXZIUCFfIolCW+4XlO3rljZGRhQb4D/HUviGrVOBYPptQawoUun1ZTS0th+8b48ceGLFMshSXRplZYuASYIBPHbWtlvu/Zi1kvw1q9DUwcKhdFj9zy8cVB85ggWgTMJrlcSt1Bn26YLqKl30PNRf97OYaUweoA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After the common decompress part goes to __zswap_load(), we can cleanup the zswap_load() a little. Reviewed-by: Yosry Ahmed Acked-by: Chis Li (Google) Signed-off-by: Chengming Zhou --- mm/zswap.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index b25d7d03851d..618989463535 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1760,7 +1760,6 @@ bool zswap_load(struct folio *folio) struct zswap_tree *tree = zswap_trees[type]; struct zswap_entry *entry; u8 *dst; - bool ret; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1773,23 +1772,20 @@ bool zswap_load(struct folio *folio) } spin_unlock(&tree->lock); - if (!entry->length) { + if (entry->length) + __zswap_load(entry, page); + else { dst = kmap_local_page(page); zswap_fill_page(dst, entry->value); kunmap_local(dst); - ret = true; - goto stats; } - __zswap_load(entry, page); - ret = true; -stats: count_vm_event(ZSWPIN); if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); spin_lock(&tree->lock); - if (ret && zswap_exclusive_loads_enabled) { + if (zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); folio_mark_dirty(folio); } else if (entry->length) { @@ -1799,7 +1795,7 @@ bool zswap_load(struct folio *folio) zswap_entry_put(tree, entry); spin_unlock(&tree->lock); - return ret; + return true; } void zswap_invalidate(int type, pgoff_t offset) From patchwork Thu Dec 28 09:45:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13505767 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AD43C3DA6E for ; Thu, 28 Dec 2023 09:46:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B57CF8D0009; Thu, 28 Dec 2023 04:46:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B09B56B00BA; Thu, 28 Dec 2023 04:46:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F73A8D0009; Thu, 28 Dec 2023 04:46:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9075E6B00B9 for ; Thu, 28 Dec 2023 04:46:34 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6F865C0821 for ; Thu, 28 Dec 2023 09:46:34 +0000 (UTC) X-FDA: 81615747108.11.7B05BA7 Received: from out-172.mta0.migadu.com (out-172.mta0.migadu.com [91.218.175.172]) by imf02.hostedemail.com (Postfix) with ESMTP id 9310A80016 for ; Thu, 28 Dec 2023 09:46:32 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf02.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.172 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703756792; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PT9H08eJXH/H7Ml+YE+PXwYs8Lv/GEpxQ360CTv1z9M=; b=Lb48X7KX5FjQiNlPERt4PHBNqlJvhO+lpJYECApulDktVV8jDYgZfOElPVq8ty5Si7fSiT k4bPPg86bDsRVHVBUbPh2wMOBuekTwDUFutNxdxzUthbzu7hTGvHRkXwqpKDMsC5QGss4Y qCO7Xqwko+lAyfXkXzyorT8jRgB2tqY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf02.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.172 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703756792; a=rsa-sha256; cv=none; b=WiItJ4JSkPn1TVT6Oyi/gwrBFXBVE0QjLzV6gpXw6lwFAWXM3XI1k7CcbI0v5f1vsh6Esx iCd0xApmzK2Zu5ShFX2alF9zKi3n27qtymsFAbv6rKbs0+bRCMe/op8vY6x31zDkXYadD/ xC3m+YbXg8wY4JK6dbDhYnwIVP7dzc0= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 28 Dec 2023 09:45:45 +0000 Subject: [PATCH v5 4/5] mm/zswap: cleanup zswap_writeback_entry() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v5-4-9382162bbf05@bytedance.com> References: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> To: Barry Song <21cnbao@gmail.com>, Yosry Ahmed , Nhat Pham , Andrew Morton , Dan Streetman , Vitaly Wool , Johannes Weiner , Chris Li , Seth Jennings Cc: Yosry Ahmed , Nhat Pham , Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1703756775; l=2338; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=5hrzED6ruCvN+7YOlRuh0wDVEni4Jtsv4F8FioAoDqA=; b=QqMwHN+kvR++JhOibm6XsrYrAW9xa7b2cN1gUvUDvJ80DjHT4cwxyemA4TSJvXgr7FctXn0HV LoHrcUfJc1BBavTLmSy4+K67MYc0NCDltyXkviDKPDg+/7snuwJFxJP X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Queue-Id: 9310A80016 X-Rspamd-Server: rspam12 X-Stat-Signature: jw113eyq7mh7n9kt1g6hq7ic8extstsc X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1703756792-842694 X-HE-Meta: U2FsdGVkX1/GL8wVoYoKeQnj8t7Jj2JH5f2p9pA7C81o0e18VOPSDVgsBfCG1b7BMxfObqARbRm2p56d3+08DZv2Ddc6xTTMGbqmPDmm1AyVRmOHHyCQ5doV6fs5XyQAYrUPmQ7pQY9h6Qdd+58WVpSIqeeeh0pqo7shvOiszhv6p3L+rPSSPrU6O5NG1S1GAlh/HXvOJMGljsCvEJY5l58Kz5dws2m2nkWgfC+g+ZjHyR567lAExQwxJ44sL3igTTVuNUWjIP/R6ML1IcJ7tF0KWlg4C+ce8ZKHBVZKRKYY92/ZBg74kQC4+ziAy0QoIMTxmOILK8Me/FLlaXxlc3gHB9KTgRYomrsWzH0WKz8X/AqfbAX0AQzSIIE15vjbvjG5msltTANk4wUbRQq/P6FU0LoOJ/2FH9sVi1F9v3GDxpQomEEFjH4htYjLHnEeVCjWfXn3JRAuccXxJ+SQrKWTYY99ERRvW8gR/cFG9Hsh5LH/sBy9yvgfAeuKnUQ2oYkgjSKc4c4I9QGccGi6mROlf1P2ynn4ilQFi38UngU9dvGvk8OZnj8iPoOzLJCKtKZ1JRS0RM9NiSvZH4ZpBGtCbQAVlA2S+g1SGoWwaEKI89mvf8WAZz/yFwBEu2dpZScr+V2rBU0w7ngFanuP4qP67pqbUBla7h0ZCGVIYwgSPi1zhaQpAZ5vJxkSRO4IChGF6ddzxhOhEknpOS8JDqv91sd7PpoKbbOAt35VT9du6QVbJpClxvf3sKohWUBFp8jSz/XGt/b081dwdsSngE8kYkBDbzxtGMDYPlrqe5kgwLrhPX9nl9guZuMaiQY7v9MAXd7VMiV6ymcLNNw8aj+i1ztI5NPWr1gh5HSkjWxzwIaRVIoyAdQ9oBkYljGEhbJVgefqCgTqlLyca+LGc1YNhwv/udub7C7SlEmUDQM4m5nuZcmOUCwX7xY1dyoKpUDyr0r0eFhQd+72Jhn q4rM+zf3 jZ3e4CDM7V4IMbc3WVL1ccWgtAoXgl5YYW4MH+Ka1iR9d2M3WE1lm1qFWqZRQb7q3Hm1rOK6eWeq7zqC6Op4zNpOivIF+FmczmCMtiIJ2mHuoIwtDdlJ1Fd+fFQCtuuDRJgVr6dyJmkqTe9rJXkXb8RxcVSd3t1k/msWrUWXZy7vnxGFX5iAro+oi+PkjLJ03WZNvpwwktGur0ai1xcxM/iBNj1/0kysja4qE4zDsq1cwAZE3VYcy4Nt55VA3hwm6qrC2zili5evWu8NiY/PO7z8hWwrH3M7VUtq8Qx2wRt3h12Lz3aMxFpj0QGum+UQjS7k8r1gJcC+Z86yjdckPsaDWOK+hf88weCdZ10tUgMGJDYHsWzwxJ454hUDCMGds0hAxo14+0xqLP3EnKxtnv/6tASuYer/M3/teMfe4D0oYRI0VfYjTb8Shruh8Eeh2ovFrOV2gKVF1zfEk7fhSQmGNa8hXBOocf9Q5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Also after the common decompress part goes to __zswap_load(), we can cleanup the zswap_writeback_entry() a little. Reviewed-by: Yosry Ahmed Reviewed-by: Nhat Pham Acked-by: Chris Li (Google) Signed-off-by: Chengming Zhou --- mm/zswap.c | 29 ++++++++++------------------- 1 file changed, 10 insertions(+), 19 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 618989463535..e756b2af838b 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1443,7 +1443,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct page *page; struct mempolicy *mpol; bool page_was_allocated; - int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, }; @@ -1452,16 +1451,17 @@ static int zswap_writeback_entry(struct zswap_entry *entry, mpol = get_task_policy(current); page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, NO_INTERLEAVE_INDEX, &page_was_allocated, true); - if (!page) { - ret = -ENOMEM; - goto fail; - } + if (!page) + return -ENOMEM; - /* Found an existing page, we raced with load/swapin */ + /* + * Found an existing page, we raced with load/swapin. We generally + * writeback cold pages from zswap, and swapin means the page just + * became hot. Skip this page and let the caller find another one. + */ if (!page_was_allocated) { put_page(page); - ret = -EEXIST; - goto fail; + return -EEXIST; } /* @@ -1475,8 +1475,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, if (zswap_rb_search(&tree->rbroot, swp_offset(entry->swpentry)) != entry) { spin_unlock(&tree->lock); delete_from_swap_cache(page_folio(page)); - ret = -ENOMEM; - goto fail; + return -ENOMEM; } spin_unlock(&tree->lock); @@ -1497,15 +1496,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, __swap_writepage(page, &wbc); put_page(page); - return ret; - -fail: - /* - * If we get here because the page is already in swapcache, a - * load may be happening concurrently. It is safe and okay to - * not free the entry. It is also okay to return !0. - */ - return ret; + return 0; } static int zswap_is_page_same_filled(void *ptr, unsigned long *value) From patchwork Thu Dec 28 09:45:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13505768 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4DBBC3DA6E for ; Thu, 28 Dec 2023 09:46:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C3F96B00BA; Thu, 28 Dec 2023 04:46:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 64D058D0010; Thu, 28 Dec 2023 04:46:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 514FF8D000F; Thu, 28 Dec 2023 04:46:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 42FF36B00BA for ; Thu, 28 Dec 2023 04:46:38 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1D9BC40896 for ; Thu, 28 Dec 2023 09:46:38 +0000 (UTC) X-FDA: 81615747276.17.FB333CA Received: from out-183.mta0.migadu.com (out-183.mta0.migadu.com [91.218.175.183]) by imf19.hostedemail.com (Postfix) with ESMTP id 32F731A0011 for ; Thu, 28 Dec 2023 09:46:35 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf19.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.183 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703756796; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8nL6Q1WjhPPL8W6G4tIFkYM/l+TyZuse48lKBGP8lwc=; b=hpRXcwPtT609icMunVtzQZBE0hkoKiQah2+oZzV4xFnUufQl2oaRDP+5vf90tyP0A8ups8 gmx+bj6yt/Poia1sygZ/GjI0KCADSvU+tzjod5k+n1nWUoRqu60BRSFnrhDmYArU5G3dUO +Gc+6LNvpHI+yXBfG47rYRYz0Czoc/8= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf19.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.183 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703756796; a=rsa-sha256; cv=none; b=BDHdqKEDscqejFrN92ATc8W3HL4gOi04g03iP+78A8C+qLDmBOYcGkPRYkx09RIP/W4H3i Sutl9wYFwZV4sRpJM/QY3E9QThkU17RtMIsIXNVnXn4qieV/92z44AkZRTZnNICrKLPRaE bKC1pcyverzqmAFv6o3KUCvI9p2eAWc= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 28 Dec 2023 09:45:46 +0000 Subject: [PATCH v5 5/5] mm/zswap: change per-cpu mutex and buffer to per-acomp_ctx MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v5-5-9382162bbf05@bytedance.com> References: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> To: Barry Song <21cnbao@gmail.com>, Yosry Ahmed , Nhat Pham , Andrew Morton , Dan Streetman , Vitaly Wool , Johannes Weiner , Chris Li , Seth Jennings Cc: Yosry Ahmed , Nhat Pham , Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1703756775; l=7667; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=4WwCjcTM4cvOikkhRW0eIrRaRjoJsRX6DvdiNXR+aoM=; b=zjkyeZ8GJvLmdKBjLPG2tkPu01IHk9nTR5X4BWfcrNSIJNqRsEvvUkf2037UBEjkA7VN2xRdg U2WvnDymn6ZCVI0TllIFnqKc5NdiqSdYpssVP3etAh5tKo+MGJxgduq X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 32F731A0011 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: qb7e6e1dokiqyu863eekxwjsd4q1wksc X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1703756795-402674 X-HE-Meta: U2FsdGVkX1/JxKU1RoLlBGjMRu0yrKlV5QLnfnB0ut4EOeGlfAxDud4S2ImrUMCDXHmQ+dz+NyYuBbTY/fEpd/5B8C24GP58Tq237D5b5whzq+Uoy/B8OuIVVBnaarh/BWu0aQudka3ZlZyaj2XaNSbR7fe5T9rculYyGwKMI6Qe89gv6pX2gthaao2Jx6daT2kcEK5CCqWQ1hZZu/BLLMT+jnrIfSrc9fgyw08/P6BALu0SdE1cHLfewqwRCHIZ5JY3hA2PhMqTdMSRZ5EqjA6TfGzz00ta7Kro2VGVAse367euW9DOvQvlr+jPP8MtGvNQqRK6Qkg3MOZHGcdd8UWZAMIvDoEkbvvK5oVWiGJDBUSRpZCl5Aio9t7+unMcur+ETJCAPoeIL4ouo0PBYYSDPftr75MNqxmB44A/7GqE9XTXo5+XPqnRWcUovmNr4W/ZmammAuNuGk2pj+KHSgVHu25+cXTR+zLyfJ1FqXg3TO2mLIQrf3REpbp7L1hTZEIL1lUPYA9Arg2a0W8hIHNtdyBy5okAvpGFIs4tXT6toycnWSpcHOiuEQQnplPFCZsAlf5x1JrGv6AjmPTK3X9FpA8Q/KFWcWEXrcDA3i4pGQ6XOGUY/G4Z/wsp1XfU4oAfcAaD9NdO7N1GJmehPSCnpGFqS+qzDMGxn1sGgfYyhvN2oiG4Bq/h0i9lZ4hf4E1IQYHPOeviypYyMSL+k0W0Fj2LHWomZCjKnc5mP2aYdzjxM9m0j3Ha9kzyJGyOAS5ENxDGPJfzhAn7U2HZU25YLdMps7Q257rvXAEfAW4DTMvtbaGeOC9PQfKXmcZkJESXDSTYQ54x5si+8r69+4/NtfUerPcO0GK4Da54UtSdBvdNHPMOptntXiAYKyRP3hS1WykDDDyC4jQcGzKh5/zzmJ5lOx09NNAVTZA+fdR8z2Re0GVlVDixnfzcnzlgSlsuHyYiVvOroCcP/Wl 5wzjY9dc jdK6Mrx7pkr9RWBf2KMO2JZF2FRFe966CpM9eYGUa3mI43SkCywMV/Ee8p+tE+jxvPMpCJRrTdOxtSLnLmpD+I/U3ZCucq8HiucXqD1qH7kT4lQEs/Emj/HJP4O3DtXYP4+4XzqHV5WAOIaeXcDUmT/h7/FNcmsZVvwrTcIym+sL5TIr1pNM3X+bmlrbeG9RpyBa4bOKw7gWSHk/yXR/7rCxqpzlQ04EeMbd4wH8P/JHPZnIIGKo7nKpBlKcQkh218smXkY9rWR+yougudYbHY17oWpN8ldr6j0dos4GVNeskknf4vi2jVtc62ouqUdkpeHYZ3OBB0Yb2o/x5I9WZ7MVa9WQ8B62iFkrsfmb0tdW8ZU/yYX6fAFTBRfBKd+kSUpmWadEndNjrx03xu1+yK8RAjRB5+AyGjPmguJZZU7voEtnr8lzWdiOuu4B6/3WhyqSnrXlxSc5tpbM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: First of all, we need to rename acomp_ctx->dstmem field to buffer, since we are now using for purposes other than compression. Then we change per-cpu mutex and buffer to per-acomp_ctx, since them belong to the acomp_ctx and are necessary parts when used in the compress/decompress contexts. So we can remove the old per-cpu mutex and dstmem. Acked-by: Chris Li (Google) Reviewed-by: Nhat Pham Signed-off-by: Chengming Zhou --- include/linux/cpuhotplug.h | 1 - mm/zswap.c | 104 ++++++++++++++------------------------------- 2 files changed, 33 insertions(+), 72 deletions(-) diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index efc0c0b07efb..c3e06e21766a 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -124,7 +124,6 @@ enum cpuhp_state { CPUHP_ARM_BL_PREPARE, CPUHP_TRACE_RB_PREPARE, CPUHP_MM_ZS_PREPARE, - CPUHP_MM_ZSWP_MEM_PREPARE, CPUHP_MM_ZSWP_POOL_PREPARE, CPUHP_KVM_PPC_BOOK3S_PREPARE, CPUHP_ZCOMP_PREPARE, diff --git a/mm/zswap.c b/mm/zswap.c index e756b2af838b..7c18755c6e1c 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -166,8 +166,8 @@ struct crypto_acomp_ctx { struct crypto_acomp *acomp; struct acomp_req *req; struct crypto_wait wait; - u8 *dstmem; - struct mutex *mutex; + u8 *buffer; + struct mutex mutex; }; /* @@ -694,63 +694,26 @@ static void zswap_alloc_shrinker(struct zswap_pool *pool) /********************************* * per-cpu code **********************************/ -static DEFINE_PER_CPU(u8 *, zswap_dstmem); -/* - * If users dynamically change the zpool type and compressor at runtime, i.e. - * zswap is running, zswap can have more than one zpool on one cpu, but they - * are sharing dtsmem. So we need this mutex to be per-cpu. - */ -static DEFINE_PER_CPU(struct mutex *, zswap_mutex); - -static int zswap_dstmem_prepare(unsigned int cpu) -{ - struct mutex *mutex; - u8 *dst; - - dst = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); - if (!dst) - return -ENOMEM; - - mutex = kmalloc_node(sizeof(*mutex), GFP_KERNEL, cpu_to_node(cpu)); - if (!mutex) { - kfree(dst); - return -ENOMEM; - } - - mutex_init(mutex); - per_cpu(zswap_dstmem, cpu) = dst; - per_cpu(zswap_mutex, cpu) = mutex; - return 0; -} - -static int zswap_dstmem_dead(unsigned int cpu) -{ - struct mutex *mutex; - u8 *dst; - - mutex = per_cpu(zswap_mutex, cpu); - kfree(mutex); - per_cpu(zswap_mutex, cpu) = NULL; - - dst = per_cpu(zswap_dstmem, cpu); - kfree(dst); - per_cpu(zswap_dstmem, cpu) = NULL; - - return 0; -} - static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) { struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); struct crypto_acomp *acomp; struct acomp_req *req; + int ret; + + mutex_init(&acomp_ctx->mutex); + + acomp_ctx->buffer = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); + if (!acomp_ctx->buffer) + return -ENOMEM; acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu)); if (IS_ERR(acomp)) { pr_err("could not alloc crypto acomp %s : %ld\n", pool->tfm_name, PTR_ERR(acomp)); - return PTR_ERR(acomp); + ret = PTR_ERR(acomp); + goto acomp_fail; } acomp_ctx->acomp = acomp; @@ -758,8 +721,8 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) if (!req) { pr_err("could not alloc crypto acomp_request %s\n", pool->tfm_name); - crypto_free_acomp(acomp_ctx->acomp); - return -ENOMEM; + ret = -ENOMEM; + goto req_fail; } acomp_ctx->req = req; @@ -772,10 +735,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, crypto_req_done, &acomp_ctx->wait); - acomp_ctx->mutex = per_cpu(zswap_mutex, cpu); - acomp_ctx->dstmem = per_cpu(zswap_dstmem, cpu); - return 0; + +req_fail: + crypto_free_acomp(acomp_ctx->acomp); +acomp_fail: + kfree(acomp_ctx->buffer); + return ret; } static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) @@ -788,6 +754,7 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) acomp_request_free(acomp_ctx->req); if (!IS_ERR_OR_NULL(acomp_ctx->acomp)) crypto_free_acomp(acomp_ctx->acomp); + kfree(acomp_ctx->buffer); } return 0; @@ -1400,12 +1367,12 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) u8 *src; acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); + mutex_lock(&acomp_ctx->mutex); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; + memcpy(acomp_ctx->buffer, src, entry->length); + src = acomp_ctx->buffer; zpool_unmap_handle(zpool, entry->handle); } @@ -1415,7 +1382,7 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(&acomp_ctx->mutex); if (zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); @@ -1636,13 +1603,17 @@ bool zswap_store(struct folio *folio) /* compress */ acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); + mutex_lock(&acomp_ctx->mutex); - dst = acomp_ctx->dstmem; + dst = acomp_ctx->buffer; sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); - /* zswap_dstmem is of size (PAGE_SIZE * 2). Reflect same in sg_list */ + /* + * We need PAGE_SIZE * 2 here since there maybe over-compression case, + * and hardware-accelerators may won't check the dst buffer size, so + * giving the dst buffer with enough length to avoid buffer overflow. + */ sg_init_one(&output, dst, PAGE_SIZE * 2); acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); /* @@ -1682,7 +1653,7 @@ bool zswap_store(struct folio *folio) buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); memcpy(buf, dst, dlen); zpool_unmap_handle(zpool, handle); - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(&acomp_ctx->mutex); /* populate entry */ entry->swpentry = swp_entry(type, offset); @@ -1725,7 +1696,7 @@ bool zswap_store(struct folio *folio) return true; put_dstmem: - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(&acomp_ctx->mutex); put_pool: zswap_pool_put(entry->pool); freepage: @@ -1900,13 +1871,6 @@ static int zswap_setup(void) goto cache_fail; } - ret = cpuhp_setup_state(CPUHP_MM_ZSWP_MEM_PREPARE, "mm/zswap:prepare", - zswap_dstmem_prepare, zswap_dstmem_dead); - if (ret) { - pr_err("dstmem alloc failed\n"); - goto dstmem_fail; - } - ret = cpuhp_setup_state_multi(CPUHP_MM_ZSWP_POOL_PREPARE, "mm/zswap_pool:prepare", zswap_cpu_comp_prepare, @@ -1938,8 +1902,6 @@ static int zswap_setup(void) if (pool) zswap_pool_destroy(pool); hp_fail: - cpuhp_remove_state(CPUHP_MM_ZSWP_MEM_PREPARE); -dstmem_fail: kmem_cache_destroy(zswap_entry_cache); cache_fail: /* if built-in, we aren't unloaded on failure; don't allow use */