From patchwork Tue Dec 26 15:54:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13504905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5839C46CD4 for ; Tue, 26 Dec 2023 15:54:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 076116B0074; Tue, 26 Dec 2023 10:54:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 000416B0075; Tue, 26 Dec 2023 10:54:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE2086B007B; Tue, 26 Dec 2023 10:54:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D08126B0074 for ; Tue, 26 Dec 2023 10:54:52 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9FFB8C013A for ; Tue, 26 Dec 2023 15:54:52 +0000 (UTC) X-FDA: 81609417624.06.DDCDDD1 Received: from out-177.mta0.migadu.com (out-177.mta0.migadu.com [91.218.175.177]) by imf21.hostedemail.com (Postfix) with ESMTP id CB1FA1C0011 for ; Tue, 26 Dec 2023 15:54:50 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf21.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.177 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703606091; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7HUld81wKjr1TRbyjzq5xYIWkmaNoBa7by5QpFhFecU=; b=C8l0glQCflF6CeWEF7lTpYJxvN0zpZ1MI2MZwEbOEvu8Arq7ejeoDRF/pgwXiEj+ulFX6c mh9uqz++qolWAaa7xWEIkZ5QPBRoe2FsRUNz2bI7HUefmFGLop0QAQL588K1F9XeG2ALy9 eBhhsJUzwQJ8g1G+2dUDXF6KeqITzbY= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf21.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.177 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703606091; a=rsa-sha256; cv=none; b=fqqu1S2DZkpTqHpyG3al/sdwzpyriks45+A9viZX+eVXnaQpGv8N2HHe8lchZCltqFI8dT 19r7NWZnuR92u0adejFEQwNjorVoOvsUAXqU7DingFsGOlW7iYivyzWcH6SCD02v8EI7il +x744KJe9wxh6hlVAZi5hLfcG+gig4E= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Tue, 26 Dec 2023 15:54:08 +0000 Subject: [PATCH v4 1/6] mm/zswap: change dstmem size to one page MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v4-1-f228b059dd89@bytedance.com> References: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> To: Andrew Morton , Seth Jennings , Johannes Weiner , Vitaly Wool , Nhat Pham , Chris Li , Yosry Ahmed , Dan Streetman Cc: linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham , Yosry Ahmed , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1703606082; l=2061; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=AFQJqRTNy44iATFj1RKZvqHU27WPg0GQMTPIPLqlpaM=; b=vEekGORU58JK/Xo7doVi9/xQDqn1SRUlc3hL7E1KvYToyqnm/hNB5CBcjOCkuqUuNL67fzR90 Z+hf+n1SDUGCIkh8qx1cZk4a0ATT8In1GnwVY+KqUFhinM4gqqu4nFg X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Queue-Id: CB1FA1C0011 X-Rspamd-Server: rspam12 X-Stat-Signature: 47qqqr3o4qiitfquuwicf64oksabxwfk X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1703606090-465440 X-HE-Meta: U2FsdGVkX1+Ig7wi/4x6eVa515XQQonciFssn6GtKN4QEV5YO/fJd1m+36j9OlU3JGi2OmyVa0y7tD0Z0wfP/qz/LS3uFhrILiB0J4z/qixOj1z9cIOrQ2xlxG6q3ra2vyVESSg9sBZCKWPzO37rGtMHXewjKo1yf5l6f9qElyu2DAjN9RylKrBJgQwMRtZfLCKWzJRQSIqntERhvLbEejXVtvS+655GwscR2CGJxMIPF0zGxXTl/tWONX1pDcwH0FhPjF6OqMo53IezRwIm98gGsSkIQAnHBisg/xjo7j06wCSOY5VOxqUtMpDq9P+Gt1BiSgIcnyzPkEfg85cKfpSlEMv3HE08BQHB+SC3tDAAJIOyjlNKQ5UEITmHsmxjPa9ETGSkGY5DlGqllEMFyxcXf+wxbZHmU3IH33QeiuwxR8UkAAQ/ishqh+/2EeWTitlu7BwWnc0Sys2Hs+3j9z8+VVOs4JluF9CqGH/+eQ1t5T5u+aV8HOmliN4+0J9ZzK3MtcWlagFxNAP1hJ9RCWdk7OtXY8s9XtmkiroO3EI1xio662cXs8YrfRC+3kjqBOrPIk7WWrMti0o7jo9cp9ykf6WQUenT/TEci/An12XB1k9fq2C/BFgMgV0jQKkJh7m4CygROtlmoWwdqQyM1vWXY8Kh4sFOzAigpoPXlqcCsHpOcCEopZadEARKmV64mWzZEqXO8gQv/TWBbdw0lo5SEO/FzrahK4wgmuo4gIVlGRImvqcdneU9XyW2SDXDVWylU+ZvRMGv828H8fSLcrd+Bj4lBByWr51zgPBwyfPmZrsGpMuB2+NmjwEwdA1HaeyVXa4yGEpYzHDEktlHqPQ7ZJp65zDXJ9NVkOSOmlPByXQDXYqHpgeuYCrXr21qo07fXM1SF+IwplAFWIixG0OSWT9rkYrIIWNNr2y/6wcw8Jfv957XhreTc3zYuXOZTtiBo1H3wHgm8Q3TDXg Gw6gf2Na Ww7t0RzEtR/VoYgFv0yAQHRyUzdGH+WvGU95r9e406aepOS0IxfLefEXIFV7F4i4kDNkgdOc9GKLdSocPVQFfS7L+7izxb80iYDHMiYRUPyQAa1tEvMCOMiVfyw1wJ9S4wzEOE8f51fla66CSEx9FXFhSRzW6m/ouJmLEPtp4C4jiD30nnVTBjG/8JIUYG70eYqIlsVxZM2fUdUREPW4zFMRt+v9VdZgKU5OunUgnqbOb4kqbloltNJtFpg6+fn1jCXXpKQ0hJJU2VFu4f6MscHvpE3BrF6JDIRGGEHgxlRgQG/hsIOE3amF6zM58WaMJsBnJMjRNNykYWABPb3S9eav2Od7GpDtlseTJD+m1pT5uhpYpyIAtfwn+G0LlWmuJyj+7zx7L7Zd+7u45/SzZpfQabWlBkTct90tHuuqUV2qCdF3m1Qz2wfwhF1KkxHVW2l5tIgt8sDhnHulP5nVFZPJPjsd8ryXuCNGQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Change the dstmem size from 2 * PAGE_SIZE to only one page since we only need at most one page when compress, and the "dlen" is also PAGE_SIZE in acomp_request_set_params(). If the output size > PAGE_SIZE we don't wanna store the output in zswap anyway. So change it to one page, and delete the stale comment. There is no any history about the reason why we needed 2 pages, it has been 2 * PAGE_SIZE since the time zswap was first merged. According to Yosry and Nhat, one potential reason is that we used to store a zswap header containing the swap entry in the compressed page for writeback purposes, but we don't do that anymore. This patch works good in kernel build testing even when the input data doesn't compress at all (i.e. dlen == PAGE_SIZE), which we can see from the bpftrace tool: bpftrace -e 'k:zpool_malloc {@[(uint32)arg1==4096]=count()}' @[1]: 2 @[0]: 12011430 Reviewed-by: Yosry Ahmed Reviewed-by: Nhat Pham Acked-by: Chris Li (Google) Signed-off-by: Chengming Zhou --- mm/zswap.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 7ee54a3d8281..976f278aa507 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -707,7 +707,7 @@ static int zswap_dstmem_prepare(unsigned int cpu) struct mutex *mutex; u8 *dst; - dst = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); + dst = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); if (!dst) return -ENOMEM; @@ -1662,8 +1662,7 @@ bool zswap_store(struct folio *folio) sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); - /* zswap_dstmem is of size (PAGE_SIZE * 2). Reflect same in sg_list */ - sg_init_one(&output, dst, PAGE_SIZE * 2); + sg_init_one(&output, dst, PAGE_SIZE); acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); /* * it maybe looks a little bit silly that we send an asynchronous request, From patchwork Tue Dec 26 15:54:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13504906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84E64C46CD3 for ; Tue, 26 Dec 2023 15:54:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AF896B007D; Tue, 26 Dec 2023 10:54:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 139976B007E; Tue, 26 Dec 2023 10:54:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7E876B0080; Tue, 26 Dec 2023 10:54:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CBD146B007D for ; Tue, 26 Dec 2023 10:54:55 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9DDA3160505 for ; Tue, 26 Dec 2023 15:54:55 +0000 (UTC) X-FDA: 81609417750.11.BBD4D00 Received: from out-172.mta0.migadu.com (out-172.mta0.migadu.com [91.218.175.172]) by imf04.hostedemail.com (Postfix) with ESMTP id CA8E740008 for ; Tue, 26 Dec 2023 15:54:53 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf04.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.172 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703606094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8wWojJj68vjmM4kc0JFut5a3qFzZiRt4b0vz/ZVD3J4=; b=TQfycmwiT+iD/hpMA21jGJGBX/vmXuR9orvsOVy/qa38HuAYj/tLl7jHrbxw9ECMImbuo4 PbiN6DHYgDK3xoOerv3L8Gk9lKUSmph//K8z2IHICHjykRyod+Rr8MCvZlo5OzE/T/YvMR FogDFwGfZkQcnXC3zaqMVHap4mmS4UE= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf04.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.172 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703606094; a=rsa-sha256; cv=none; b=sEd/XtG7GQp0qEjNek7qyfOY2ENGPoVweoalSVU1c5C/l+SVhMipkkC+qGLu9kwwdXueXR ED1wIMQuJGjWYEAUxVsTC1B0CNic7cJ4iWWqEkOKbhUB+5PdFE9Ha+1dWQSzDgmpim+JlM bd/eQgDe7s6laywYqLjFN+ReWwhErCw= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Tue, 26 Dec 2023 15:54:09 +0000 Subject: [PATCH v4 2/6] mm/zswap: reuse dstmem when decompress MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v4-2-f228b059dd89@bytedance.com> References: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> To: Andrew Morton , Seth Jennings , Johannes Weiner , Vitaly Wool , Nhat Pham , Chris Li , Yosry Ahmed , Dan Streetman Cc: linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham , Yosry Ahmed , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1703606082; l=4288; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=ee6CjzVfu3cdsKJFW0DjU4Yqsb6snUyFV2P+WjqVZ2k=; b=/M2fa/388zR47dnBIiQiB32KQcYRHMBqBcSykhWTPfpxspJa5FXo9+SaEpDo1xt/3FIcNllNz cLnkXIGD4JGCQBc8dxPU/SL9D1/IOJXC80mVvuuJ/DxyQRPRAi4uCfR X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Queue-Id: CA8E740008 X-Rspamd-Server: rspam12 X-Stat-Signature: fndijk1u9sidfwe3yf59fgibcmjf83re X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1703606093-569179 X-HE-Meta: U2FsdGVkX1/+8KGkKKztwHN0lT37d90/TQQNN4VDRNElS6ZteK6Q4xkAoj0ubVyUJ/UHZZnM7A5NKXA8YjgXOT0ePd5wJ3yL6XxNKbcvIqPrd+i+gmyBecByVEdkn1E+uGNd5Ca75whHEVIpi+4bLP2ITRwOqT04nhJmkVS+56iX026qUecXTbz8cmNOcZBB+1AQkjveaRY2Zh8pKXv3vvNalaBh7WJhGJDw/0hGKhiCAOeDAxiKR2RrWCyWv5kLciyOQGmcxDVSFuJ0usF+czZq8WMQtndJdJI3JCNCPJNibu1nan9+kO70rHOO+1e6P40qoOLeoWCMnJ8DCn33Y20/F0LPsom3aIPsX/abT8pLiq3gOPivGorAhQAT7ttsfMtpY6VAfpE+qqItqeMQQ7+peikKnkdq0Lny80E1a5Mfz3n1BPA4EElnLfl/w96MYILCiN/OeaOSGZGzpthKJv5zUun9iudyVGakIFNy/GpUIu++DPnVunytR/1GpkWF/g+b420leKuPvR//WWQneYlXDKY7P+QUMrXOF3gz7nVRUAbFpiYi5tcnbbuINW2fqsTPh7MHWLAokukVUAGQLe/kxzhkDwh29bq4athcZpOzz7WHDuLThW598G+avZJ0GtK9h2L5+dcSeMhokPfv4EjfRiZ7Lr3a5lQnlOsGzvUg9RMRwkiaD6kyCS7oIv9M9SsiUK6SVvXU6Ljo/hmxxJQPUJjDJnWUhYexS41q06JUizFxJkLpyBjW/ioPhv6oECDBAnxVdPaqpVi56V7UxpdDNy9ACOVJkDUOfIKDOEIn5301cTmPvPSoMq/0DwdXFYbc1d5P80kLdQiH6qIkvMMs9Gix2w/dyYFgBRBpR74ib3eGhP4OwFb0G7WpGMNlgN9ti6wCx00kZyyIyZaAPbqMSg/rG0dADrGl7BN0JXGnTGlxcZTDFlMYB+kxywUh82Z0z+bHeLqnT84BRE1 2uvwmKMn O/1HIyFpoJSX+AssqTBdSb2H6TKffa97fN7s1u6F7dVGkLMs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the !zpool_can_sleep_mapped() case such as zsmalloc, we need to first copy the entry->handle memory to a temporary memory, which is allocated using kmalloc. Obviously we can reuse the per-compressor dstmem to avoid allocating every time, since it's percpu-compressor and protected in percpu mutex. Reviewed-by: Nhat Pham Reviewed-by: Yosry Ahmed Acked-by: Chris Li (Google) Signed-off-by: Chengming Zhou --- mm/zswap.c | 44 ++++++++++++-------------------------------- 1 file changed, 12 insertions(+), 32 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 976f278aa507..6b872744e962 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1417,19 +1417,13 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct crypto_acomp_ctx *acomp_ctx; struct zpool *pool = zswap_find_zpool(entry); bool page_was_allocated; - u8 *src, *tmp = NULL; + u8 *src; unsigned int dlen; int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, }; - if (!zpool_can_sleep_mapped(pool)) { - tmp = kmalloc(PAGE_SIZE, GFP_KERNEL); - if (!tmp) - return -ENOMEM; - } - /* try to allocate swap cache page */ mpol = get_task_policy(current); page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, @@ -1465,15 +1459,15 @@ static int zswap_writeback_entry(struct zswap_entry *entry, /* decompress */ acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); dlen = PAGE_SIZE; + mutex_lock(acomp_ctx->mutex); src = zpool_map_handle(pool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(pool)) { - memcpy(tmp, src, entry->length); - src = tmp; + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; zpool_unmap_handle(pool, entry->handle); } - mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); sg_set_page(&output, page, PAGE_SIZE, 0); @@ -1482,9 +1476,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, dlen = acomp_ctx->req->dlen; mutex_unlock(acomp_ctx->mutex); - if (!zpool_can_sleep_mapped(pool)) - kfree(tmp); - else + if (zpool_can_sleep_mapped(pool)) zpool_unmap_handle(pool, entry->handle); BUG_ON(ret); @@ -1508,9 +1500,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry, return ret; fail: - if (!zpool_can_sleep_mapped(pool)) - kfree(tmp); - /* * If we get here because the page is already in swapcache, a * load may be happening concurrently. It is safe and okay to @@ -1771,7 +1760,7 @@ bool zswap_load(struct folio *folio) struct zswap_entry *entry; struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; - u8 *src, *dst, *tmp; + u8 *src, *dst; struct zpool *zpool; unsigned int dlen; bool ret; @@ -1796,26 +1785,19 @@ bool zswap_load(struct folio *folio) } zpool = zswap_find_zpool(entry); - if (!zpool_can_sleep_mapped(zpool)) { - tmp = kmalloc(entry->length, GFP_KERNEL); - if (!tmp) { - ret = false; - goto freeentry; - } - } /* decompress */ dlen = PAGE_SIZE; - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(acomp_ctx->mutex); + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(zpool)) { - memcpy(tmp, src, entry->length); - src = tmp; + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; zpool_unmap_handle(zpool, entry->handle); } - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); sg_set_page(&output, page, PAGE_SIZE, 0); @@ -1826,15 +1808,13 @@ bool zswap_load(struct folio *folio) if (zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); - else - kfree(tmp); ret = true; stats: count_vm_event(ZSWPIN); if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); -freeentry: + spin_lock(&tree->lock); if (ret && zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); From patchwork Tue Dec 26 15:54:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13504907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60E2EC4706F for ; Tue, 26 Dec 2023 15:54:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0297E6B0080; Tue, 26 Dec 2023 10:54:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E7E906B0081; Tue, 26 Dec 2023 10:54:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6DB36B0082; Tue, 26 Dec 2023 10:54:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C3E626B0080 for ; Tue, 26 Dec 2023 10:54:58 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9BD47160505 for ; Tue, 26 Dec 2023 15:54:58 +0000 (UTC) X-FDA: 81609417876.18.B0004E5 Received: from out-183.mta0.migadu.com (out-183.mta0.migadu.com [91.218.175.183]) by imf17.hostedemail.com (Postfix) with ESMTP id CC33040007 for ; Tue, 26 Dec 2023 15:54:56 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.183 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703606097; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l+86cxEbcHQglJmKP3D0/ptU/ooniCJpeGNPcIFgpbo=; b=7JP1r0L0oC0dVTNvBzdiVW9Faoc8kBoYjNOdjkln6uYc2AJkRtz3/pPlLEVSUaMikcdFRu rl4OAOKSlb9FwYnGAaa3FdokYVBMYQYjFzETaWZtVLm7nJyrhXd5voJiWkDLH5FclASZSs 9nxwebGV87bGlf4GC+APHroY3BwZj3E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703606097; a=rsa-sha256; cv=none; b=1ZL22DKFUxCBjRGbB20o/IPHpvOKcubVamO+SsfEqfGOMeSwCeuKqz6wIiikBqKJ/DYobk Ul8bNatKJ8/K58o/TPt6jQkFNACZo6ZJN2Iu1xc3hCzXNfztezLbgeyap05PiMN9KIPsSb YzTo0jfRA6sTL69p6XQXDxG0lEp+FJY= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.183 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Tue, 26 Dec 2023 15:54:10 +0000 Subject: [PATCH v4 3/6] mm/zswap: refactor out __zswap_load() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v4-3-f228b059dd89@bytedance.com> References: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> To: Andrew Morton , Seth Jennings , Johannes Weiner , Vitaly Wool , Nhat Pham , Chris Li , Yosry Ahmed , Dan Streetman Cc: linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham , Yosry Ahmed , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1703606082; l=4752; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=rqaFBRN5T+K3BGykGqxbunVDkqGsdVLjPtenmvHXNns=; b=2whbnk++qzMgDfpxZEj6RN2GBW2XP2w/exd8KHJg/QUs3bQXx+0f1E7kPbE6TtAa+IGnjLVIP 4/8liyuX+0xCEjYzNGSZpQ+axcBpJmSX+TQmeVlzNI4BpBQ/XmcYob0 X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspamd-Queue-Id: CC33040007 X-Rspamd-Server: rspam11 X-Stat-Signature: 96afxjrthw15p476h3gyts88f3zzdfqp X-Rspam: Yes X-HE-Tag: 1703606096-485105 X-HE-Meta: U2FsdGVkX197lingr3zv90O9WzkbD0W5MCxlyZcBWJXluWYe64jQh0hySfguyVY6qP9wvRn+glN52KTczZDar9FKU5ahbTle5qzHUKeMdW7jVPLyHj7MpaZ+lObFIY4btVwCzRZpive6IJyER/h51XWFrZXphIEIcjbtyteBh1f7sXZQ+kUY8o/+Hg2oW8I+Nf/8WduK5HR+iSBh+FFTeUMvdEXg7UBpMuWqYTPGN+Oil1Yllf8ZoDLT/MxVmD6iOwYsGY0PM6DRFiq3otujumq15VBdCVZM2Pq8BiOO6PqyjEyVNdB35Hj+Jff1CVJ5Xl88IHBHeI5V1dwi3HDdK8Awbu4Cs/sqGG55iRtKBGAONBPkmMQnKElLzcEFmwOza93H8o1EzwYDpkcOEdD7/rVjaL2s5kigo2UmqJcNe3sGXyIBNLa864EL76kIbLRLZncg4/4Qu+h30UcG9Qdgzwww4SCRu9dJElBwU4ttb1Ehxs5YDMOJ4Gx7S9lF9Rxz/5MWc5NlbFq1tAz30ozpstxk6hb761pXCS8J0l7HEaTl96w2QJ7c/S9SFeUBb5+sSXuIsnHxsmjeGOaklgMqx8Go5fxRReCqJcL1LU7Ml7Uk+Z7jdeK0qLapz7n0aXdnadlI0m+LZc3cHwhtDgJXnll7Y70lpiN/YKxZa4uMwnzLt0RHF38/MgT8PXRVeEz4HdC/RWYBXyKkRMVAYBqDDuZzT+0/qiE8OV7vKY4E2zEr5mi2ZVUfT8EGiJzk7VIFQnYZih/X6LswtegJ9OQgA5MnqMhOEYBJ+/+USNG13MEBOQ0OeyETtrZESYlrYFJpzstEB3Ei+zSHVQZnuhV8nU2jfCQ5qLre9HCJV8ZsqccSYFfjg6PJ7pz0WffBHTSCWSRZGq7aIPZufgmSrJY0GBSfFBKw6d0U9+MxOguq2vBsoQah2Qt1gtPR5KjE+S+JVClO6S1qV9kmlqMHO/T n1zymiOZ ZGGKbFU0TL6kSikpLwQsWiwmhlMVLZ6YHhTaH1Q0T8iELGLr2hm9thJt4B7VxVC+jeoU8WNktcWfMK/Fv0Xju9hv36wDZt+aoj2Y+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The zswap_load() and zswap_writeback_entry() have the same part that decompress the data from zswap_entry to page, so refactor out the common part as __zswap_load(entry, page). Reviewed-by: Nhat Pham Reviewed-by: Yosry Ahmed Acked-by: Chris Li (Google) Signed-off-by: Chengming Zhou --- mm/zswap.c | 92 ++++++++++++++++++++++---------------------------------------- 1 file changed, 32 insertions(+), 60 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 6b872744e962..3433bd6b3cef 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1392,6 +1392,35 @@ static int zswap_enabled_param_set(const char *val, return ret; } +static void __zswap_load(struct zswap_entry *entry, struct page *page) +{ + struct zpool *zpool = zswap_find_zpool(entry); + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; + u8 *src; + + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(acomp_ctx->mutex); + + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + if (!zpool_can_sleep_mapped(zpool)) { + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; + zpool_unmap_handle(zpool, entry->handle); + } + + sg_init_one(&input, src, entry->length); + sg_init_table(&output, 1); + sg_set_page(&output, page, PAGE_SIZE, 0); + acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); + BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); + BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); + mutex_unlock(acomp_ctx->mutex); + + if (zpool_can_sleep_mapped(zpool)) + zpool_unmap_handle(zpool, entry->handle); +} + /********************************* * writeback code **********************************/ @@ -1413,12 +1442,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, swp_entry_t swpentry = entry->swpentry; struct page *page; struct mempolicy *mpol; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; - struct zpool *pool = zswap_find_zpool(entry); bool page_was_allocated; - u8 *src; - unsigned int dlen; int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, @@ -1456,31 +1480,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, } spin_unlock(&tree->lock); - /* decompress */ - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - dlen = PAGE_SIZE; - mutex_lock(acomp_ctx->mutex); - - src = zpool_map_handle(pool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(pool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; - zpool_unmap_handle(pool, entry->handle); - } - - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); - ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; - mutex_unlock(acomp_ctx->mutex); - - if (zpool_can_sleep_mapped(pool)) - zpool_unmap_handle(pool, entry->handle); - - BUG_ON(ret); - BUG_ON(dlen != PAGE_SIZE); + __zswap_load(entry, page); /* page is up to date */ SetPageUptodate(page); @@ -1758,11 +1758,7 @@ bool zswap_load(struct folio *folio) struct page *page = &folio->page; struct zswap_tree *tree = zswap_trees[type]; struct zswap_entry *entry; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; - u8 *src, *dst; - struct zpool *zpool; - unsigned int dlen; + u8 *dst; bool ret; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1784,31 +1780,7 @@ bool zswap_load(struct folio *folio) goto stats; } - zpool = zswap_find_zpool(entry); - - /* decompress */ - dlen = PAGE_SIZE; - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); - - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; - zpool_unmap_handle(zpool, entry->handle); - } - - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); - if (crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)) - WARN_ON(1); - mutex_unlock(acomp_ctx->mutex); - - if (zpool_can_sleep_mapped(zpool)) - zpool_unmap_handle(zpool, entry->handle); - + __zswap_load(entry, page); ret = true; stats: count_vm_event(ZSWPIN); From patchwork Tue Dec 26 15:54:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13504908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A162C46CD3 for ; Tue, 26 Dec 2023 15:55:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0010B6B0082; Tue, 26 Dec 2023 10:55:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ECD826B0083; Tue, 26 Dec 2023 10:55:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D45966B0085; Tue, 26 Dec 2023 10:55:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C339F6B0082 for ; Tue, 26 Dec 2023 10:55:01 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9A112120382 for ; Tue, 26 Dec 2023 15:55:01 +0000 (UTC) X-FDA: 81609418002.24.569BCEE Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) by imf18.hostedemail.com (Postfix) with ESMTP id B184E1C002A for ; Tue, 26 Dec 2023 15:54:59 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703606099; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xn7ksme6twQw82nMBXJ4dYxxzHaCP+EO0zfhYUSni6c=; b=28qjjyrq8hDQGIlo1M/ybBeLpHpyc8aGzL6BfcEOKtGR8KlTLqg2q1A2qU0uJIfWiQdjdF IEHN2aAh2FJF/rbotXchhL9eOz4MeT630pNh7qJAsb4UHMI8PuZFv203mRS5Zb0COoJqpB /ndfvt1DLU6R8+tBf+ChTzCpMqPKin0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703606099; a=rsa-sha256; cv=none; b=LE7780CAFJQsPti7YJIwxN4fQdtaYWX2GdPiSaZqF95//33zuYxN9yeqON6LSmww3RekCa yclbFQx0KJrvvgpo32V+T2Pk0KtXdx8W/eiZo7xQyrA/GnheCs1dEAe3JMAuJ0mreCwjgU lOyRBuoyf+v3EDCKHwg3qqubtO3cIZI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Tue, 26 Dec 2023 15:54:11 +0000 Subject: [PATCH v4 4/6] mm/zswap: cleanup zswap_load() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v4-4-f228b059dd89@bytedance.com> References: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> To: Andrew Morton , Seth Jennings , Johannes Weiner , Vitaly Wool , Nhat Pham , Chris Li , Yosry Ahmed , Dan Streetman Cc: linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham , Yosry Ahmed , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1703606082; l=1518; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=rkes+J7eJM7t0YHK81EpBcYXvy/jNev7hTPxYOeqbUg=; b=EMECwijc3Cop46ntV6xtp5hN60obHUUAxQdRv1B1TZNaLQmm/7W7OGjUfEwdkOlvVPeLksp90 8QboHPBWsnnD2aRQe8a9oroWGYNCKyszGFtZFKrkZZcaWZnrG3gi0aT X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspamd-Queue-Id: B184E1C002A X-Rspamd-Server: rspam11 X-Stat-Signature: ahz1tw1zs7io69bidhcwa4dqs6jcwkey X-Rspam: Yes X-HE-Tag: 1703606099-276389 X-HE-Meta: U2FsdGVkX1/DAcC+cE1l5qt4vbQQ84dWFE+N+iOsF8/D6JVCl7KkAllDJIcVv/hTEwGfH7BRn6MXSGF4HQ6crJoJ0nYFZmZtCseE7LLj7D6dYnOCEqvpwVaV2OEtoVnHMIio0p7OadN+soonnhSxlM4vHG+tLrMWQja53LweU/rxeYbo3ctS+lyKV7d3nXsQ5KhSW9h7GfUqz84XCgW3OOAEd9P+0Y4FntJbOviVx1/9QohGmqGD+PcBFdva3OKovZyica1k+j+9ntoTC5CQvprQt15OgURJt17NyBk58M/DdbpulzrGeYrWkIUa4MnEUr4B4LVDbW4R0+6ZkR3VNIqR0DmcBmmfYeqOQBr+keltoOjK2H11lyZnPX5v0EOHmTB3hmqox0GRy3i08V4LxqCjGDcxSF42j9FSu3PYvWhABuRTwW6YozwI1QWH0A7Ilw+5pgVlLM+w/MZKpuXYExOjkPXYYyCXCsWB3nCGENWHGqPP/DQ2QUyI+4Dv3xPMAGtUhdt73+FiWhQzEoAUoyhRpl2iBMSVE0XhU1uocAAS+vJMILrjR3L/xLEYS+MLCwwe7XzINOasZ+2acUEVanbTf2OeOLTlZJ/y0EWB4xZ+hgLl5bgfCWhHIFSapUAr+EtY7aZuQsuVRfKYt5CXjK/Q74DDD9a5OF3NEgOzbAuwCXF6yICYHHMceuxa/fS8gC2cZKK2WssDMxLHTXCxOjyJJr+bkLVbiYNQsocP2YkvNJMvY032qX+e73suSJBEDEI/KlePas339Mju/ZMnN3odnDLKNT74plTXKhc8VlgbKA4mKikGyuyN89N2UphYj11ohAvgQGs88t1UeisYnRSC2v4T+u2/+5EfACGQmBo7QgX2YVG3YudFDEmiUR3uARw5cEzQECWfkUyMWUEkGGi0TIA25k1RJs73iZwkb6Uzn308DUaohlv4LLh+c1rhLN8tUHyNE2lJVPecEt1 RmQazZD3 SvYwJEWfVQV5F965sV18VigDXX6KDgceQofN6399lje3+bHK/8JPUdM9Vy6hohH+PQXmqjiY+PxF6xKb/zUbYp0x1jZPtG2fodBVOAkeS77WllW3GCBlZOfsi2aMNDbHFiIqid5GtXZNRv16kFePuxPNhuqGIKtuJAjRa6TDpif+EBsE77LUn40OQ5bkuKOSvKOMxzGs/HfEgZ72bsEWtXz9lAS792YTnRUnfqimNAML7n43al8+NJpfrQ0T1eBQTcCN6Zxgc3NSmwhA2tetfuNt8qMofFNMCeTohgy73Q/Lwy3hYb9TmBfIwh3WJDqNUUg4EkJWnBCPdq4qEk/KKr5+b1Mhrk0710THw55uo8lQn7jqxeVE1eL9m83KV+j5kFTtElMCzAW5BBesahMMU8ZeABGXCvah6u7cT4DO4ktG34OhQfcKcUQP/PxwLFy9V+Ti57I8Rb99OkgTkKLH5hxpxNR4iU3y5lq7RE6FucoYwVxK9yE8H3RrjUA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After the common decompress part goes to __zswap_load(), we can cleanup the zswap_load() a little. Reviewed-by: Yosry Ahmed Acked-by: Chis Li (Google) Signed-off-by: Chengming Zhou --- mm/zswap.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 3433bd6b3cef..86886276cb81 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1759,7 +1759,6 @@ bool zswap_load(struct folio *folio) struct zswap_tree *tree = zswap_trees[type]; struct zswap_entry *entry; u8 *dst; - bool ret; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1776,19 +1775,16 @@ bool zswap_load(struct folio *folio) dst = kmap_local_page(page); zswap_fill_page(dst, entry->value); kunmap_local(dst); - ret = true; - goto stats; + } else { + __zswap_load(entry, page); } - __zswap_load(entry, page); - ret = true; -stats: count_vm_event(ZSWPIN); if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); spin_lock(&tree->lock); - if (ret && zswap_exclusive_loads_enabled) { + if (zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); folio_mark_dirty(folio); } else if (entry->length) { @@ -1798,7 +1794,7 @@ bool zswap_load(struct folio *folio) zswap_entry_put(tree, entry); spin_unlock(&tree->lock); - return ret; + return true; } void zswap_invalidate(int type, pgoff_t offset) From patchwork Tue Dec 26 15:54:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13504909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D5CCC46CD3 for ; Tue, 26 Dec 2023 15:55:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 001386B0085; Tue, 26 Dec 2023 10:55:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EF3636B0087; Tue, 26 Dec 2023 10:55:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6D7D6B0088; Tue, 26 Dec 2023 10:55:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C46956B0085 for ; Tue, 26 Dec 2023 10:55:04 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A07B91A05C1 for ; Tue, 26 Dec 2023 15:55:04 +0000 (UTC) X-FDA: 81609418128.03.4755F37 Received: from out-186.mta0.migadu.com (out-186.mta0.migadu.com [91.218.175.186]) by imf02.hostedemail.com (Postfix) with ESMTP id C772380006 for ; Tue, 26 Dec 2023 15:55:02 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf02.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.186 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703606103; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fj6IXPbKxUdaaVKYr+IKMxSX4wMJz6sg9yd9pI3CwMI=; b=YLfYTbMuhPeCEMYVeiXJa/UQlh4HsMvH5PYQkJXfsSyqOFGlziTa4B2yLYZgqCIBN3b+QD /upBuH+xxgQmyg5lM2fKyPH9Qd7KRo2XtZXsKnvJACjQxS5YqB+8DkrMVt+Qf6smNLzWjo OkY4u+o8EOPMoaUFtXwqrCNle2AQ54o= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf02.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.186 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703606103; a=rsa-sha256; cv=none; b=17Aq2JSrdsllYBz7kiHVhvfCFllATo8VqpOu8s2E5RIGHe/6Eeqxfe3Of2HEDp6sqB/GeF buf3hFJZ7DpZjb85VPJRrj0WeUC9nJpcNOkpgoFs1jS9DIo4U9VUm0gWmqwdZW01gsNNFP FNYCWzb3BqD24S4xoEs4Pbe9pt9f1iI= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Tue, 26 Dec 2023 15:54:12 +0000 Subject: [PATCH v4 5/6] mm/zswap: cleanup zswap_writeback_entry() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v4-5-f228b059dd89@bytedance.com> References: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> To: Andrew Morton , Seth Jennings , Johannes Weiner , Vitaly Wool , Nhat Pham , Chris Li , Yosry Ahmed , Dan Streetman Cc: linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham , Yosry Ahmed , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1703606082; l=2338; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=tAV69Q+4us6Py1b2R2r8p5EYfYNSenhTMFhuOp7vNNQ=; b=afViGCtigOHtj8SqyE1/3tYO10W9zGnBEqlNj/WHy0NkwORwH+k6pwR/NJpbIHP3TQeqORo2h ERJwMflduKtC/okQR5pR1LwHwhvGLXlCdgX/gX53ZckJzqrZB76ELZ2 X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: C772380006 X-Rspamd-Server: rspam02 X-Rspam-User: X-Stat-Signature: rfdz3ck48s36n4r5x4kzr9xw9qwtrtzt X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1703606102-886066 X-HE-Meta: U2FsdGVkX19LrUl9PViLXl4KTFLu84txoDbxjUniBkAsfTNW4N/eB0p02tdROG2FH2KkDXs0IC8Npjb4WYCatusLIHiFHSiur5qntXVGy/+ass0n843nUud6ESkLlpjGtVaI4RDY0vDkvvG/EI6nQ7gSWStN8y+6LeTDFlOn1CRvknaA+MXTAuMy8d3T9BYOelgBzc7tjR/PNC0BTkM1ga5ssQBKwZsQG6faBTnox+/JiQMDnaX70HDTjKrEkbQR3ZMkuGJCi8IajfHM6qa0RBt8Ed85rqZXEWX1sO/PzMvp7AePBGE4W9LwIT2oZdxUa743um0nAqw2aESQ0JN0rg1TIMStjHqIdj9otwST/z1iHvRa/L/gLrvckyvYGRRv+h3K4anmLvddSUPEkN5NUMYdKMrwQIB8u3+ok6gRxjh/3nNA4qpXlRHXnWoW6c55fBY1gAx8+EOgyEh5FxmLQxtxKLZezxU0XdMmuJFK5ApBS0SNATqk7AjvCqM/Dvlq9ntTMVpqeblsq14zDsMfHk69En8eolCiNHkrJCffddELwOY890Jp5A8EPJkNRGJEoclzQwlcMlIJlPxJA/H9oGK3SKGve4WsCdu7HgYB6hT5P18FA7f/3jC+x4Wm8sv3u9TshmcKOsbSGWm5k7CydwJE8sTgBE4Ooc3j1GUGATbt/egWRU4WC47Q0T3dRDbTBEwR412j+Op41ZVzr8jFQI7m/hcf1LOTlElgCDeg041EYqqKg7xAPffOYrk6Oshj3obSZwym2zAo2GpoxDi333eJ7Tk9U7uO+1vGatHGCc3jgrBhFNLXheDa0Ur8lq/OLSrStbSapnFuN9HL+aihfMBXtMTDYXkSUsx5bJJZd0sdBmbtKI0eS0MkRDEuPy24okE85NwTDdhuuH7ECqZ0GGz3tTwWnYsI6tVRHHHp2K44SvNcls1HJ3kQRueMBrrX0Ye9ts09jK9cH9/b+bg hMw2y+Mh 02GqeMhiGCgnMhrErYRvHg91TOrSbu9yDemez28SPATxIlBkPnZOWLVgImev89FLJR7FBNbmWtrzMbyHXvOSG4J2wncDBTlkd4xTS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Also after the common decompress part goes to __zswap_load(), we can cleanup the zswap_writeback_entry() a little. Reviewed-by: Yosry Ahmed Reviewed-by: Nhat Pham Acked-by: Chris Li (Google) Signed-off-by: Chengming Zhou --- mm/zswap.c | 29 ++++++++++------------------- 1 file changed, 10 insertions(+), 19 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 86886276cb81..40ee9f109f98 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1443,7 +1443,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct page *page; struct mempolicy *mpol; bool page_was_allocated; - int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, }; @@ -1452,16 +1451,17 @@ static int zswap_writeback_entry(struct zswap_entry *entry, mpol = get_task_policy(current); page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, NO_INTERLEAVE_INDEX, &page_was_allocated, true); - if (!page) { - ret = -ENOMEM; - goto fail; - } + if (!page) + return -ENOMEM; - /* Found an existing page, we raced with load/swapin */ + /* + * Found an existing page, we raced with load/swapin. We generally + * writeback cold pages from zswap, and swapin means the page just + * became hot. Skip this page and let the caller find another one. + */ if (!page_was_allocated) { put_page(page); - ret = -EEXIST; - goto fail; + return -EEXIST; } /* @@ -1475,8 +1475,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, if (zswap_rb_search(&tree->rbroot, swp_offset(entry->swpentry)) != entry) { spin_unlock(&tree->lock); delete_from_swap_cache(page_folio(page)); - ret = -ENOMEM; - goto fail; + return -ENOMEM; } spin_unlock(&tree->lock); @@ -1497,15 +1496,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, __swap_writepage(page, &wbc); put_page(page); - return ret; - -fail: - /* - * If we get here because the page is already in swapcache, a - * load may be happening concurrently. It is safe and okay to - * not free the entry. It is also okay to return !0. - */ - return ret; + return 0; } static int zswap_is_page_same_filled(void *ptr, unsigned long *value) From patchwork Tue Dec 26 15:54:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13504910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85D19C46CD3 for ; Tue, 26 Dec 2023 15:55:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 029526B0088; Tue, 26 Dec 2023 10:55:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ECBBE6B0089; Tue, 26 Dec 2023 10:55:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6E286B008A; Tue, 26 Dec 2023 10:55:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C72256B0088 for ; Tue, 26 Dec 2023 10:55:07 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9B47E8058B for ; Tue, 26 Dec 2023 15:55:07 +0000 (UTC) X-FDA: 81609418254.04.A6DE066 Received: from out-175.mta0.migadu.com (out-175.mta0.migadu.com [91.218.175.175]) by imf13.hostedemail.com (Postfix) with ESMTP id BB77F20011 for ; Tue, 26 Dec 2023 15:55:05 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.175 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703606105; a=rsa-sha256; cv=none; b=dZlr0QKSOooap3Ga6IEij5PdJeMi52Jd+ZIXDaoNGBbQt6X9uiYRk+3dn7QNTTAtpOGwM4 7ETRZSYM/kxS5MJtD3YoGnf8XLM9/YeMCoiwAmpE3iFehoiBtnWVh8KIAnjHgA0K1kdaLw 2f4OeENjZBafQQg6qjQGZ0qIrWltED4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.175 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703606105; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MAN+vf5ULkcLBKJ3hxoGKmhb7oVG85/z/AAApGulvX8=; b=L47CzDxMXaVDw8x6J4Y7eVE4ro/0tgZL2xx9SRueiZ5v2no/uFt0T4Tcuru4gM/nxDUmJI bN5yhR4F+n/toszHpFKnPL75ue+zOsMHwxlb+/sUvaohcY8tfhVLojzTIDc0ZQswOix6Sp KAU0jWIbPi3I5EcYZPsj9ubb6s3EqtA= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Tue, 26 Dec 2023 15:54:13 +0000 Subject: [PATCH v4 6/6] mm/zswap: change per-cpu mutex and buffer to per-acomp_ctx MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v4-6-f228b059dd89@bytedance.com> References: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v4-0-f228b059dd89@bytedance.com> To: Andrew Morton , Seth Jennings , Johannes Weiner , Vitaly Wool , Nhat Pham , Chris Li , Yosry Ahmed , Dan Streetman Cc: linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham , Yosry Ahmed , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1703606082; l=7176; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=eozKHlUpbkn5zdclwtiNUF72Ti9iB66PKm56EsjFhvY=; b=ZHdimCEH7YHQg7J/zZG0lpD8VDQR5PJsQQJTk4mFBZYPUirIGkxxEFhs+8NzqRUaI5EO7+GWo uZGuipzU28YBtj82Nq4HzIbIEAvGbfNtgvhtYTNlPP6vi7n5vBhjtn8 X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: BB77F20011 X-Stat-Signature: wcw48wtupamzbt3hhbmp5rrkfwagcpx5 X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1703606105-593492 X-HE-Meta: U2FsdGVkX19zkY/6BEJJkVFPc55r9oVvHgNgtDDpMNDaKLOrKhFvLOFzcJx4FXfuekdBQdw/6EM/dUeqjU917eDkpQizE9slHJDq19CVVbkmdamR1cF4lnFOfCM3lStRZfQBuDeN2qlapqcVUfWAXWCLh40I/lkV3+ZhhUbe08DN0NV85JADDsjYdajQeCFPffAUNhLWeLOijwe2kO+OXdIw77GLIyozfcVzJLYB8lCP2gRzsPoqvkpvG7DD7xpAKp6l0uzGocLV/ksij3vZC6Cmx7QoNChcKM73YDABSA1c1fcLW4IXVd+ERSrVuEwQxtp/TdlGYLz0hySu188i7bSUNpnbBDzdZK9jjTTFwMoHqdfFG61PL3R8rWWcfNjisI840fg0VxU2FUNWqKcazrQpDJ6ZL13PkX8aRBA0quLZ2AGw/A7vVbL25u7nT1jr7L+vxrmIDrslcD13XAawutnzWvItod3ofcgl7V5+SHEDqrMZPCzoH8a0857+2/9KkVSqfroVuAhh39bUBUG0ucMeL3acapW3p5KnH9/y1kbRGF0k0E2BCauftL+mAsWMxhtuxA0vPCXQaFdZdraZ78ETDnlJKE9P3TV958pNHTpC+sCPhsZZY2O3sfqS1ab2e350IEaubtC4u3/6HISYQkx/Dp59xRMbZs9cVUKi1it6oMq2lRbwIuJf6p15kZ0tm553D+lQAbR3cKk/BcD0pfRgMGHeOLyK7iuNUWrg96Gu8HfAKel7w9ARmefV8FNU/dAc53C6RVqi47eH8H57qnJ9JF4PBM19o+Gk146zwHX3HhujUBeiKpqUxz+4JoSMLgGNc+P3DkuGGCQkgE4nIpx8YNx4dIW0iIHFyZckkjErVdrJo+nC9oGetysna55+dYltRahidbltNtBCFTLCPK1f2QE4WlfE/Lx8LePx4VmB+MsKU8c5q6G9NBGbvagYEVO3+U4eYAZke4fsxi9 eRbrfzdw LVKX1KSdUPh2Oei6EuWNuEzOGQxxylaXtkWEfa84DAGaQCLhhD5Y66iCyuB2/IzlTTNB0xBuAQ9HhRO8MQ7s1r0kd3l7ezqCYRAveB/AoZ3FfWg0iwNW6wCbXUJKv9h1ScgDZACb+GTOdXw87hIIjz3X3NcrcvyGPcyxjQrQWv1+dgVnR/HODuS8P/1bgwoS5VNxdxDjiOzNi5iqSfP71/VUxvoorsZO6PgO+0sAtQ1DjIPa0iNKf4qYUC7Ly0AAYF+igKgKFaLtAAKTk5vwApNQl1+0C6+6Jy4CFJmjl33XD2f8uKpC+EM/HIa50iDkZfZnT0V1SkpgCq96jzE2PEAgLgwnVhrGsYC/iaAcYy3lYut5WtJFaEE1EcEEjSb39skaC6aaYmN1scrpkDEbdjOfqmvfp4FEpLg2OoMzC3ZvCbUw4u02NeesBKPR7Lq+45Bj2xNH1U8/PUwXuC8uOs4VnjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: First of all, we need to rename acomp_ctx->dstmem field to buffer, since we are now using for purposes other than compression. Then we change per-cpu mutex and buffer to per-acomp_ctx, since them belong to the acomp_ctx and are necessary parts when used in the compress/decompress contexts. So we can remove the old per-cpu mutex and dstmem. Acked-by: Chris Li (Google) Signed-off-by: Chengming Zhou Reviewed-by: Nhat Pham --- include/linux/cpuhotplug.h | 1 - mm/zswap.c | 98 +++++++++++++--------------------------------- 2 files changed, 28 insertions(+), 71 deletions(-) diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index efc0c0b07efb..c3e06e21766a 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -124,7 +124,6 @@ enum cpuhp_state { CPUHP_ARM_BL_PREPARE, CPUHP_TRACE_RB_PREPARE, CPUHP_MM_ZS_PREPARE, - CPUHP_MM_ZSWP_MEM_PREPARE, CPUHP_MM_ZSWP_POOL_PREPARE, CPUHP_KVM_PPC_BOOK3S_PREPARE, CPUHP_ZCOMP_PREPARE, diff --git a/mm/zswap.c b/mm/zswap.c index 40ee9f109f98..8014509736ad 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -166,8 +166,8 @@ struct crypto_acomp_ctx { struct crypto_acomp *acomp; struct acomp_req *req; struct crypto_wait wait; - u8 *dstmem; - struct mutex *mutex; + u8 *buffer; + struct mutex mutex; }; /* @@ -694,63 +694,26 @@ static void zswap_alloc_shrinker(struct zswap_pool *pool) /********************************* * per-cpu code **********************************/ -static DEFINE_PER_CPU(u8 *, zswap_dstmem); -/* - * If users dynamically change the zpool type and compressor at runtime, i.e. - * zswap is running, zswap can have more than one zpool on one cpu, but they - * are sharing dtsmem. So we need this mutex to be per-cpu. - */ -static DEFINE_PER_CPU(struct mutex *, zswap_mutex); - -static int zswap_dstmem_prepare(unsigned int cpu) -{ - struct mutex *mutex; - u8 *dst; - - dst = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); - if (!dst) - return -ENOMEM; - - mutex = kmalloc_node(sizeof(*mutex), GFP_KERNEL, cpu_to_node(cpu)); - if (!mutex) { - kfree(dst); - return -ENOMEM; - } - - mutex_init(mutex); - per_cpu(zswap_dstmem, cpu) = dst; - per_cpu(zswap_mutex, cpu) = mutex; - return 0; -} - -static int zswap_dstmem_dead(unsigned int cpu) -{ - struct mutex *mutex; - u8 *dst; - - mutex = per_cpu(zswap_mutex, cpu); - kfree(mutex); - per_cpu(zswap_mutex, cpu) = NULL; - - dst = per_cpu(zswap_dstmem, cpu); - kfree(dst); - per_cpu(zswap_dstmem, cpu) = NULL; - - return 0; -} - static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) { struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); struct crypto_acomp *acomp; struct acomp_req *req; + int ret; + + mutex_init(&acomp_ctx->mutex); + + acomp_ctx->buffer = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); + if (!acomp_ctx->buffer) + return -ENOMEM; acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu)); if (IS_ERR(acomp)) { pr_err("could not alloc crypto acomp %s : %ld\n", pool->tfm_name, PTR_ERR(acomp)); - return PTR_ERR(acomp); + ret = PTR_ERR(acomp); + goto acomp_fail; } acomp_ctx->acomp = acomp; @@ -758,8 +721,8 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) if (!req) { pr_err("could not alloc crypto acomp_request %s\n", pool->tfm_name); - crypto_free_acomp(acomp_ctx->acomp); - return -ENOMEM; + ret = -ENOMEM; + goto req_fail; } acomp_ctx->req = req; @@ -772,10 +735,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, crypto_req_done, &acomp_ctx->wait); - acomp_ctx->mutex = per_cpu(zswap_mutex, cpu); - acomp_ctx->dstmem = per_cpu(zswap_dstmem, cpu); - return 0; + +req_fail: + crypto_free_acomp(acomp_ctx->acomp); +acomp_fail: + kfree(acomp_ctx->buffer); + return ret; } static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) @@ -788,6 +754,7 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) acomp_request_free(acomp_ctx->req); if (!IS_ERR_OR_NULL(acomp_ctx->acomp)) crypto_free_acomp(acomp_ctx->acomp); + kfree(acomp_ctx->buffer); } return 0; @@ -1400,12 +1367,12 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) u8 *src; acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); + mutex_lock(&acomp_ctx->mutex); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; + memcpy(acomp_ctx->buffer, src, entry->length); + src = acomp_ctx->buffer; zpool_unmap_handle(zpool, entry->handle); } @@ -1415,7 +1382,7 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(&acomp_ctx->mutex); if (zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); @@ -1636,9 +1603,9 @@ bool zswap_store(struct folio *folio) /* compress */ acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); + mutex_lock(&acomp_ctx->mutex); - dst = acomp_ctx->dstmem; + dst = acomp_ctx->buffer; sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); @@ -1681,7 +1648,7 @@ bool zswap_store(struct folio *folio) buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); memcpy(buf, dst, dlen); zpool_unmap_handle(zpool, handle); - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(&acomp_ctx->mutex); /* populate entry */ entry->swpentry = swp_entry(type, offset); @@ -1724,7 +1691,7 @@ bool zswap_store(struct folio *folio) return true; put_dstmem: - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(&acomp_ctx->mutex); put_pool: zswap_pool_put(entry->pool); freepage: @@ -1899,13 +1866,6 @@ static int zswap_setup(void) goto cache_fail; } - ret = cpuhp_setup_state(CPUHP_MM_ZSWP_MEM_PREPARE, "mm/zswap:prepare", - zswap_dstmem_prepare, zswap_dstmem_dead); - if (ret) { - pr_err("dstmem alloc failed\n"); - goto dstmem_fail; - } - ret = cpuhp_setup_state_multi(CPUHP_MM_ZSWP_POOL_PREPARE, "mm/zswap_pool:prepare", zswap_cpu_comp_prepare, @@ -1937,8 +1897,6 @@ static int zswap_setup(void) if (pool) zswap_pool_destroy(pool); hp_fail: - cpuhp_remove_state(CPUHP_MM_ZSWP_MEM_PREPARE); -dstmem_fail: kmem_cache_destroy(zswap_entry_cache); cache_fail: /* if built-in, we aren't unloaded on failure; don't allow use */