From patchwork Mon Dec 18 08:22:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0812FC46CD2 for ; Mon, 18 Dec 2023 08:22:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FB6E8D0006; Mon, 18 Dec 2023 03:22:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 585638D0001; Mon, 18 Dec 2023 03:22:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D6D68D0006; Mon, 18 Dec 2023 03:22:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2BCC58D0001 for ; Mon, 18 Dec 2023 03:22:35 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F382AC07F1 for ; Mon, 18 Dec 2023 08:22:34 +0000 (UTC) X-FDA: 81579247428.20.25E9EBF Received: from out-182.mta1.migadu.com (out-182.mta1.migadu.com [95.215.58.182]) by imf14.hostedemail.com (Postfix) with ESMTP id 03084100010 for ; Mon, 18 Dec 2023 08:22:32 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702887753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UFtS65bQK5h5w9cm85RbKzaORRgbOq0Q5FLRevBhnWM=; b=mqJOJAoE2p6GymrQKWTSEq+6gsVwP2MZp4E/KH0qZcZ4hhuczllLckKEYvTj8hI04geYxH n44gI2j3cpBV4gdPUl90Ec4IIefkZWN9JFyy12dTU8Nr/VXbNfYSEgd9JCm9rW6Gzv1c+t ejAnFlxJb+3/cRqn5gvbSY4lX5Fowec= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702887753; a=rsa-sha256; cv=none; b=qun/c9ntVot4aDOJl6MfUCIjpGTkCGqi/fZtoCCPRN1hYTeAvRHlW9SriRi0VRKTKYvcJH uPS74TpVaDsOLQB9Sh5jpJG/6kR/2/hf1C2izgWpe4ykusIXq83JUgWWtSSspQDxmTL0z3 6JB/Z8wqiv6qqb6fIlZAdpG1N8QYTzQ= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 08:22:00 +0000 Subject: [PATCH v2 1/6] mm/zswap: change dstmem size to one page MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v2-1-daa5d9ae41a7@bytedance.com> References: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> To: Seth Jennings , Dan Streetman , Chris Li , Nhat Pham , Vitaly Wool , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-kernel@vger.kernel.org, Nhat Pham , linux-mm@kvack.org, Yosry Ahmed , Chengming Zhou , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1702887745; l=2012; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=knbPWkl1sBJOnq5pAv9yoWZfZi+/5lFRxFAsoTFiAeI=; b=wLIslRILYEE+m1faC45+NiuKmvx2YOrqYhYxcwRKSBnB/cOfAH9ePkRU4ZggKvwoWRd8ApqOE G59vRbwGIL6Cj27KS5MZI/t6FgZ7RE6FdEjYEhi/qQPb2ilNgqbCYBz X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Stat-Signature: jhyirgt1ojxo5gt4ddif8qaj76k88pbd X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 03084100010 X-Rspam-User: X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1702887752-101800 X-HE-Meta: U2FsdGVkX1/RQbGTkYbuE3XAM9QZbHfsTZ5XRaDe8jXAFp0cOSGeDIOS05ZRtVDT3Rmd/bm6MbHXN0yz1QNZfALXf4/EFE/X2bh5EuGhfK6+Jdd54jN7QWJalG0aId8HRHlkLVosf5RFLhkLWjl/YkzvpYiLWHa15jKQJ8jw0t4ht4MHoSy/PyGc/dJPkGAnsdM1/cxTPKnnS1Gg/ZKV0XJ/uBfqnnARbmRxFIBhYms0Yg/oI5tGJYGy/KB6u9S+whxc4QCA8ZIREOGds4k8T8AMEqU0Uo4qz5HcYf2h3CzSOdUFgHqUcTIB6hV0qPlJNRH7J0qFfRCKQvqM3iITebMvbHwpwX7lfsVBDmQLb9zLoPbux4rHdobJve34Z62Dr3ZFliCoejd8dUcZmG9V4sqP/tV7rLGrwdM9tF2ciiA9ZqxgE4bUvIaOEYnG+ru/NnTTtewLMFdD4y2VZfhcQTWLz10cXFakx2wAerQFT+ipAUM/lbevvkyiZgjOX5HweZvlTuywJRIqtm9awlaNqtEfpNoBk8dlS14h0/5aFI3xQ5gctZPzP9fRxaRcdOcIgA1Unbnk/EmVYqMT+lwT+I/Ckq9XSKsGVduIWig4fUf4BxpyWujLX93NAw6WJlH4lS7QUwWDmWe82YUwNpkdRyW1R2l/z6MuDqSC7guDnLCQLlcuKAR2D8lEP1tWCLGXghLxVzw3gjDI8NB4peY0hlR2UkUDfco5lwrPvtc7Cn0GZBMzxlrw8byCPT6S+OR5AbXey+uvGll8LZ3M6AbaDDyhWnjwLUgp96P4flmZ49JNHAey2naihXbmfrZkCmohKjLdSEUicClm5Q2uvE2d6aXbZFAWgZE7d7jrAkO2isC/uFvypbkfSY9y9Rt3Mv24VFBVIcVCXElaF/qZG3CxdkectG+CnzJ5ei7ZjB3H/88Vwm2JMl2iAvwwpkztAu+380OtreT74FjVpfitRdy RjDY4hmM lhkpr32Mm1QxWVetW+LFmUxYhYWjL1ZbxCP6T4504rxAZgM6jmzv3yxtCVY0BlLl9ClDAsylvZvhrqvQ+Xi1RTpS0h8DAC1TuOz20tW6N683z3T/zn4T2ffakqVwJaeHWY2DYTGA1oxf4aJk6K7Ie2LaWlOywJ163busqUn1Gd1DpFstMOuuOPPEj8M05B5H91YbwUz0W1BOCsVxBjRv38dLJn7uVpfwSWgoWMPURB8ufHVGfXtDbjbniCz8yhRdpztVWuKGGCBzs1mSmtrY48hPiSH+ui1U5U2fSX1kNkHQfDKixzOmy9pDJyCKmbrXpN4D6iG/f9Q+fXpvARELBF9bHsq5sqeBRaZmNhu3uWVsRYyfeQq9VZm1LIuaYa1PHH1XBxO4NOLNhGA6nAhL58JvqiA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Change the dstmem size from 2 * PAGE_SIZE to only one page since we only need at most one page when compress, and the "dlen" is also PAGE_SIZE in acomp_request_set_params(). If the output size > PAGE_SIZE we don't wanna store the output in zswap anyway. So change it to one page, and delete the stale comment. There is no any history about the reason why we needed 2 pages, it has been 2 * PAGE_SIZE since the time zswap was first merged. According to Yosry and Nhat, one potential reason is that we used to store a zswap header containing the swap entry in the compressed page for writeback purposes, but we don't do that anymore. This patch works good in kernel build testing even when the input data doesn't compress at all (i.e. dlen == PAGE_SIZE), which we can see from the bpftrace tool: bpftrace -e 'k:zpool_malloc {@[(uint32)arg1==4096]=count()}' @[1]: 2 @[0]: 12011430 Reviewed-by: Yosry Ahmed Reviewed-by: Nhat Pham Signed-off-by: Chengming Zhou Acked-by: Chris Li (Google) --- mm/zswap.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 7ee54a3d8281..976f278aa507 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -707,7 +707,7 @@ static int zswap_dstmem_prepare(unsigned int cpu) struct mutex *mutex; u8 *dst; - dst = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); + dst = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); if (!dst) return -ENOMEM; @@ -1662,8 +1662,7 @@ bool zswap_store(struct folio *folio) sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); - /* zswap_dstmem is of size (PAGE_SIZE * 2). Reflect same in sg_list */ - sg_init_one(&output, dst, PAGE_SIZE * 2); + sg_init_one(&output, dst, PAGE_SIZE); acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); /* * it maybe looks a little bit silly that we send an asynchronous request, From patchwork Mon Dec 18 08:22:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9609DC35274 for ; Mon, 18 Dec 2023 08:22:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED4708D0007; Mon, 18 Dec 2023 03:22:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E84F08D0001; Mon, 18 Dec 2023 03:22:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C631D8D0007; Mon, 18 Dec 2023 03:22:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AFB248D0001 for ; Mon, 18 Dec 2023 03:22:37 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7D18B16021D for ; Mon, 18 Dec 2023 08:22:37 +0000 (UTC) X-FDA: 81579247554.03.347A504 Received: from out-173.mta1.migadu.com (out-173.mta1.migadu.com [95.215.58.173]) by imf18.hostedemail.com (Postfix) with ESMTP id 8E59B1C0005 for ; Mon, 18 Dec 2023 08:22:35 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.173 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702887755; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=T3GWg0eQ/f74Kkc/EMROSUotrx6hZIGFelAG2ou5unE=; b=odkBYsSNmpP6q8XenOupyDPdVHlJHLkc14DJm8lH7huWpUGynpNVgykphITlnNJu3G8/NN iGZRz6CPFoXT+wqV0wcUuXtU2qcg8LgBs/dhGG5jYIAnpwapP3hk1rKxTYalPDQ4B/CnBA YaALptIiYjmNFe4LukAkT9pbS0o3jYU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702887755; a=rsa-sha256; cv=none; b=dnjAaBi6OaDn3qwBVfTBRNwVGTTFtlOtHIhZLQsC9IFqXNkXT3EF0fgeBBfs3lbPTcuN93 VHvqbw5yjcxyxspfasVK9o3P9sn12WjRTXwvpCg3DCX1SzMj4rQKee32RzclKVvAVrrTvq xZ/xSgmfU571cEr2dSxHio8nkhzhcFI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.173 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 08:22:01 +0000 Subject: [PATCH v2 2/6] mm/zswap: reuse dstmem when decompress MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v2-2-daa5d9ae41a7@bytedance.com> References: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> To: Seth Jennings , Dan Streetman , Chris Li , Nhat Pham , Vitaly Wool , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-kernel@vger.kernel.org, Nhat Pham , linux-mm@kvack.org, Yosry Ahmed , Chengming Zhou , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1702887745; l=4229; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=aB+tFXtYFtFovOCKjFmKhfN20WzOhqogX+a8O/o262s=; b=Kd7hGtuBRqJ6usbhtM5mZ4Sj4K7wMC6fwtx+Dp2KUGxlw6etTViNm+SeBpEiBp80WtiJS41o8 B1a8dk6TqidDIJodwMtroZsMduENCYRUKjYDOAzxK6wHZnm5C2hLLQ7 X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspamd-Queue-Id: 8E59B1C0005 X-Rspamd-Server: rspam11 X-Stat-Signature: 86yhz8apxzz4wxbirbsi61rehos8s5o9 X-Rspam: Yes X-HE-Tag: 1702887755-5409 X-HE-Meta: U2FsdGVkX1+JKm235B5P5bqpxBMZTNGROTJ6LsEdk5Qyeq+WxwT1Neu+gcT2cUw6N0lz6Q1HR8jSzDRblxXkyT2FQR7o9f9T1hZDGWV2Jc8HyZKyOZ1aozy034RcMhY2JwMysBdYRVHDAu2pffVB1Mb6yoJ7/i21C4x6wVsbNsXv/Avxt2qtBbScqlEZkheEaHhj3D53JPV/ZarxLoqObMLdqmrn0w24o8iRDdbDn6Xzud/rpBeCR74EOmOk6ZF/TEUTe45MJKmmFqWibvEGS5/f+W3Bn0EkNW54WpHtD26wEXcWKUDK9cwSIe2JjBi+coEAS7L05bhcgeUT847BijgtASlJXxGEyJ6Pspp9mXtiwAYiM11PQ8DnE3nfqVHSCRDNTma70gf91B6tq8mq4xmeRSB1+Warq4V3GaNEbM2bSSq83u1redU7zM8wsM6bGH3pOziTogzyICJGfndTwf2D5qITOJlQHy37brcaCH3VwTT8rStYVK+5pTrb0Y2jgBcgW52SA6mBlZt6CFPPXqtE7wHMWgQXJE1/aYXefFzHXpCbtTbwn+cpHQPQJhw+Kh+WeVCvHBhGOfLCAcaxNmhAHFqhaCi0gApNIMMzYW/6H6QdEYSc24bjTnPRU+zyMBGjsj99Kt9qjnjUgUVGi+88HJr9alLoTv35vzA3Bt7OP5ok+CILvwcNB/Ptx2exok2NizKNKC5YIhPIFzg2N7x0DODBfgF8YhWbYlmW0y556jNrv7z23ID6qK4abO08nm8mbymmOkuYlw7kwrOwMihDwwj1TSQtYYdQVRHaYWFn2EIDRdazf9mIEs8g/oD1Rem+ZAe4Ej976+nGgmKn6MZ1XNY1M0KEiKuXfZOgul+y23UgzqfyBWTc3fEpbK2lhUACcCHtAQaPKp4kAv2jHOsNBwwqUD7qnesB5IfbRLvZgGIuoq+QG9ArRcTxiqQbMt22fVhj5ZFrzj3/9G6 bOsJ9uAU NKLfJrGMyNemvuFmYT78oXE5h8y5fsChBL3sh1cuiEjavnVpbJqNR+D2jFc1epLlR8K/gf4VCOaXAB6nqc+IBViWFatyCHA45/R8Nq+Rw6KzgU4KD9nH8jsS0tnxOXk2ZpUCUhh+BbWUSl8Vw6nT/K5VKdCwdJszQ3jKTeE7hpcLgyOJ3kAoMnmrXYz4L3ZgsuKe9wKO3ZTXDJKEoHyCzC65zU4vf2wuhuu+78oEp+OhRYAEgTDQiZYnUytp1W1GTHoUObyHumL5tAMdtOYo8McDlWeJ5kUR8RqzpqDUaz1IwGd7uSy/FY3KVwDJC3m9ZGe/JtLSHsPtZWUTHzz9Cbt4zwqZAUJ5MuaDkT6xf1iTbek5Pd8BGCoSvmJ1zgMz9nkRqqwlXZxvgKZTiHKsxMQ4VmkB+efHfxJcnTdkrJCS3ZsZFtzQQnSgRPO2+ELVLky9o X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the !zpool_can_sleep_mapped() case such as zsmalloc, we need to first copy the entry->handle memory to a temporary memory, which is allocated using kmalloc. Obviously we can reuse the per-compressor dstmem to avoid allocating every time, since it's percpu-compressor and protected in percpu mutex. Reviewed-by: Nhat Pham Acked-by: Chris Li Signed-off-by: Chengming Zhou Reviewed-by: Yosry Ahmed --- mm/zswap.c | 44 ++++++++++++-------------------------------- 1 file changed, 12 insertions(+), 32 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 976f278aa507..6b872744e962 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1417,19 +1417,13 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct crypto_acomp_ctx *acomp_ctx; struct zpool *pool = zswap_find_zpool(entry); bool page_was_allocated; - u8 *src, *tmp = NULL; + u8 *src; unsigned int dlen; int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, }; - if (!zpool_can_sleep_mapped(pool)) { - tmp = kmalloc(PAGE_SIZE, GFP_KERNEL); - if (!tmp) - return -ENOMEM; - } - /* try to allocate swap cache page */ mpol = get_task_policy(current); page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, @@ -1465,15 +1459,15 @@ static int zswap_writeback_entry(struct zswap_entry *entry, /* decompress */ acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); dlen = PAGE_SIZE; + mutex_lock(acomp_ctx->mutex); src = zpool_map_handle(pool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(pool)) { - memcpy(tmp, src, entry->length); - src = tmp; + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; zpool_unmap_handle(pool, entry->handle); } - mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); sg_set_page(&output, page, PAGE_SIZE, 0); @@ -1482,9 +1476,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, dlen = acomp_ctx->req->dlen; mutex_unlock(acomp_ctx->mutex); - if (!zpool_can_sleep_mapped(pool)) - kfree(tmp); - else + if (zpool_can_sleep_mapped(pool)) zpool_unmap_handle(pool, entry->handle); BUG_ON(ret); @@ -1508,9 +1500,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry, return ret; fail: - if (!zpool_can_sleep_mapped(pool)) - kfree(tmp); - /* * If we get here because the page is already in swapcache, a * load may be happening concurrently. It is safe and okay to @@ -1771,7 +1760,7 @@ bool zswap_load(struct folio *folio) struct zswap_entry *entry; struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; - u8 *src, *dst, *tmp; + u8 *src, *dst; struct zpool *zpool; unsigned int dlen; bool ret; @@ -1796,26 +1785,19 @@ bool zswap_load(struct folio *folio) } zpool = zswap_find_zpool(entry); - if (!zpool_can_sleep_mapped(zpool)) { - tmp = kmalloc(entry->length, GFP_KERNEL); - if (!tmp) { - ret = false; - goto freeentry; - } - } /* decompress */ dlen = PAGE_SIZE; - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(acomp_ctx->mutex); + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(zpool)) { - memcpy(tmp, src, entry->length); - src = tmp; + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; zpool_unmap_handle(zpool, entry->handle); } - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); sg_set_page(&output, page, PAGE_SIZE, 0); @@ -1826,15 +1808,13 @@ bool zswap_load(struct folio *folio) if (zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); - else - kfree(tmp); ret = true; stats: count_vm_event(ZSWPIN); if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); -freeentry: + spin_lock(&tree->lock); if (ret && zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); From patchwork Mon Dec 18 08:22:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496352 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68A21C35274 for ; Mon, 18 Dec 2023 08:22:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADDB78D0008; Mon, 18 Dec 2023 03:22:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A65E38D0001; Mon, 18 Dec 2023 03:22:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 906158D0008; Mon, 18 Dec 2023 03:22:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 781E98D0001 for ; Mon, 18 Dec 2023 03:22:40 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 46B31802DB for ; Mon, 18 Dec 2023 08:22:40 +0000 (UTC) X-FDA: 81579247680.18.664E6AA Received: from out-174.mta1.migadu.com (out-174.mta1.migadu.com [95.215.58.174]) by imf07.hostedemail.com (Postfix) with ESMTP id 402FE40028 for ; Mon, 18 Dec 2023 08:22:38 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf07.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.174 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702887758; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5XHGau3pXpBIRdgWSOxIy56lBlKtyLT1nxaNBp7TWqM=; b=PXnKmp9hIrs7vWNbgiF/NNHDBUyiGKw3qC+OgCp3sfVDD6boOScaJwYjD4ckGIFcwH/Hoy KSa36j19cWfzQFRAFFNE/EYgJUa9Z1tcBWUX3Qz9sDfs+anPQEUuZY5vEp/qI7aqG0HLtW C+JPJhYfOvaNbyjTI+oeiJBUJiAHqpg= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf07.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.174 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702887758; a=rsa-sha256; cv=none; b=PG0aaLFwbMqGc3+2Wx6khmZ/e5MEXkBXZGTljXpOCN9+2ZxRqq45g9n25JZQlh+YZ0cUUp cQ9CsANsfXql4sNVIV/ujNYbN1kGL0gVmddVUwq207R+4bPjvxfbpGSPXUNxsGnSyTIliO +vylH4yA4Py6A+Lg1Ty4+Ttf7pF4+BU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 08:22:02 +0000 Subject: [PATCH v2 3/6] mm/zswap: refactor out __zswap_load() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v2-3-daa5d9ae41a7@bytedance.com> References: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> To: Seth Jennings , Dan Streetman , Chris Li , Nhat Pham , Vitaly Wool , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-kernel@vger.kernel.org, Nhat Pham , linux-mm@kvack.org, Yosry Ahmed , Chengming Zhou , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1702887745; l=4703; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=B99wC6AH9+R5m6Djw0+W/AJA7ag/bUbsLCKvYrrlo8Q=; b=i4FV3NiKGGQh3MrHNtbjPVINB+qPumTYyPJyDJWpxDhI64MoLtB2CBM8elGupDabajgL3lEDR 6iddEsOkX1UA09nUGr1s1pdovfIltzGUWgVzu3xtbR6x9alb5Q0sCF3 X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspamd-Queue-Id: 402FE40028 X-Rspamd-Server: rspam05 X-Stat-Signature: 91pnkep67zbd8th5tzqajt1xb43fmfdy X-Rspam: Yes X-HE-Tag: 1702887758-927002 X-HE-Meta: U2FsdGVkX18y2A/MCBtgOu7AgXtQfBBCFOks4YXh/QKnApR83bWjqvzMtCbmUPM4cHDtN4J+iG01aGViBeEw3P4/sVm9CTPUxwt6rVkAhpbfVi5EPd1zTC2dTsFOlBT4INq4vPA668B75D9GCNEBqYeQEnz1df/4ob5XsS0tc8+hsNHRBxCq3kSu4ArohctCnLys+gE4XFlWu8hd1AvrY6sO34dzc2IuBe/YTG2wyDmlhbRnnSV/8Wp1BbIMplWi5pk8c9yECR7fnFpTGRGZxl9wZ4NQs2YEk7AtzxBt/wNIDDqI1vOMEOlGsq0yoGgXHMKg/9FQuAUQDZb2an4nd/mAPe8LxIBYFjajTFa47oT254kNTkfSJ3Se2hAiZrRbxJV3qiO9mYFu/cks1x7OqfWxiliBX7DLeonK3GmWbdj+CILGchhIApKg8fzFqXs4URbMs8yywAVgG/sCO3ZxXHNmQNRHdeq1XdP4fVJVur2tFQ6n/NfeL3f8hT6+ZBlPks5dr4y6d8fIODcoL7iFOAfzKse1w1sCr4TPw/6uegMg98giBe5UrLvj+/A0uy2Hd0Yn3NtvZ/m/HTxdB+jU4LkerkBUTBBPUjOZEndbhFJDY34DAuObZXrAHR2Ak5aHZeZrXrMcWo42lgVX5B2s4KxVPYbuTOGg0ZJM16lFmvnkRutdMHyAxQraN/6H+3iQF/1wX5YXQlXMBMp9QhYteWqYWEcNSD7XCix8yBcNMYEkJVShKK50bQ1Rh963go8IY1pmRYZyjVaN5TZiZErwhXj15vYYS+8zAfN2UXv/MhbUE5CHJMkRaBlDTe0fvybai1YgIgebH3G6kOu1/F96pm4GokYCWyKSY3wrxN/FiPjQAtUSuU45CK9ZzSSgTRcntM61Irf7PTb1VJMmL2kzqDdeSD1f2C7arVY976KLwGvpvJc5sDnLaoShdIxBsYiFt5hIuHIc2HDUmT5gsfx MbtIioWX J8bQqiGEg3OUg7vtblB1CzNo/cbCH3ujjS1bbqyz7IfH7JZy5Bmhv2GTGoEBuk7gRTTp8rrhtXEhK+ZAihOTofRfwkxcF59mTEaDOTUW4PqYZkrPMMUNNNMczoz+zR+ehGXuL54VtiNu2YiertsIdb5WZOHPulqaDBmZ0Dj0OtsdVvjbDldFw0gJx4Dm5kvAptPDsh1yeL3r2KdPoRrBMROIrab6E5RsU6AtSEOxMP+tJSAiznqh0Ri9owcpH1vXeKdysBsEDEWtGo//yDz3bR77ICnER05/BBSTyx9wUfFzKF4bz/xukVoESHKBrjwyaRt3RSXotjAKcqB/DxthnR5D9DGbgLOT0MgmjSPqzTWiwlS6MFeGCrBV8pDyzLg+Sc+ieky077Q08x2nub+gN/8p/mOEeAgjAr3HP82Rs4qkeqMOIOv72xEmoauLwrBg9s2hsZ5X2DphS0/E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The zswap_load() and zswap_writeback_entry() have the same part that decompress the data from zswap_entry to page, so refactor out the common part as __zswap_load(entry, page). Reviewed-by: Nhat Pham Reviewed-by: Yosry Ahmed Signed-off-by: Chengming Zhou --- mm/zswap.c | 92 ++++++++++++++++++++++---------------------------------------- 1 file changed, 32 insertions(+), 60 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 6b872744e962..3433bd6b3cef 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1392,6 +1392,35 @@ static int zswap_enabled_param_set(const char *val, return ret; } +static void __zswap_load(struct zswap_entry *entry, struct page *page) +{ + struct zpool *zpool = zswap_find_zpool(entry); + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; + u8 *src; + + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(acomp_ctx->mutex); + + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + if (!zpool_can_sleep_mapped(zpool)) { + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; + zpool_unmap_handle(zpool, entry->handle); + } + + sg_init_one(&input, src, entry->length); + sg_init_table(&output, 1); + sg_set_page(&output, page, PAGE_SIZE, 0); + acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); + BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); + BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); + mutex_unlock(acomp_ctx->mutex); + + if (zpool_can_sleep_mapped(zpool)) + zpool_unmap_handle(zpool, entry->handle); +} + /********************************* * writeback code **********************************/ @@ -1413,12 +1442,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, swp_entry_t swpentry = entry->swpentry; struct page *page; struct mempolicy *mpol; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; - struct zpool *pool = zswap_find_zpool(entry); bool page_was_allocated; - u8 *src; - unsigned int dlen; int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, @@ -1456,31 +1480,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, } spin_unlock(&tree->lock); - /* decompress */ - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - dlen = PAGE_SIZE; - mutex_lock(acomp_ctx->mutex); - - src = zpool_map_handle(pool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(pool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; - zpool_unmap_handle(pool, entry->handle); - } - - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); - ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; - mutex_unlock(acomp_ctx->mutex); - - if (zpool_can_sleep_mapped(pool)) - zpool_unmap_handle(pool, entry->handle); - - BUG_ON(ret); - BUG_ON(dlen != PAGE_SIZE); + __zswap_load(entry, page); /* page is up to date */ SetPageUptodate(page); @@ -1758,11 +1758,7 @@ bool zswap_load(struct folio *folio) struct page *page = &folio->page; struct zswap_tree *tree = zswap_trees[type]; struct zswap_entry *entry; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; - u8 *src, *dst; - struct zpool *zpool; - unsigned int dlen; + u8 *dst; bool ret; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1784,31 +1780,7 @@ bool zswap_load(struct folio *folio) goto stats; } - zpool = zswap_find_zpool(entry); - - /* decompress */ - dlen = PAGE_SIZE; - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); - - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; - zpool_unmap_handle(zpool, entry->handle); - } - - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); - if (crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)) - WARN_ON(1); - mutex_unlock(acomp_ctx->mutex); - - if (zpool_can_sleep_mapped(zpool)) - zpool_unmap_handle(zpool, entry->handle); - + __zswap_load(entry, page); ret = true; stats: count_vm_event(ZSWPIN); From patchwork Mon Dec 18 08:22:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5DBCC46CD2 for ; Mon, 18 Dec 2023 08:22:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FAE28D0009; Mon, 18 Dec 2023 03:22:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AA688D0001; Mon, 18 Dec 2023 03:22:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 425268D0009; Mon, 18 Dec 2023 03:22:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2CE958D0001 for ; Mon, 18 Dec 2023 03:22:43 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F1F651608AB for ; Mon, 18 Dec 2023 08:22:42 +0000 (UTC) X-FDA: 81579247764.22.8DA2044 Received: from out-180.mta1.migadu.com (out-180.mta1.migadu.com [95.215.58.180]) by imf08.hostedemail.com (Postfix) with ESMTP id C488C160017 for ; Mon, 18 Dec 2023 08:22:40 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.180 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702887761; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zJqTjnZTPjIHItkHechsbAK+vOUzSf9pl11pwS87u1M=; b=Vs28ueZqt1ZICZJUXXeC0nXSp7PThPuXyUXCb3u9xfRVzVKDScm0jTY6AzQFJS+1LYy5DD XCqxVR+/4iTJ+OZgNiUS2NKPyu0i6CtVWYjeGF603Tv4U5B4sVEBMam3b6qQ0QFVofF769 Xz0p54uIGPYFnyGvmWR60TKva1E69DM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.180 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702887761; a=rsa-sha256; cv=none; b=QMLeO4uhmNgN0I+3GAol0X3af0C9eXxsigLqe2wHG8CISXGMKPg7a55WU6YTi6L9GbDUoo n6sx5eQRkZEzBpo3WW/ZLBsOWedXtcikjJ/SqSodN9+dRkX7e3zVizqE2YPDfVl4uItIGK WKSyrPG8C5T2v3gkJSPR42xdVbJ2+8s= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 08:22:03 +0000 Subject: [PATCH v2 4/6] mm/zswap: cleanup zswap_load() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v2-4-daa5d9ae41a7@bytedance.com> References: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> To: Seth Jennings , Dan Streetman , Chris Li , Nhat Pham , Vitaly Wool , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-kernel@vger.kernel.org, Nhat Pham , linux-mm@kvack.org, Yosry Ahmed , Chengming Zhou , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1702887745; l=1470; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=1eOM2WlTMMQMFmZcsE6sU450y2r77yOSYBMqqAB7HFE=; b=YOV157KveESD/iXPiDgVC9WaHMxuNJOt5qyf4ENQkrp7X4z9UFu4fkqe1LUaOnVs8aXOn3lks OXH9Dwct4c+At1moBhyAct1e80egxjcQg0Enm77kPeVULGpQ2btQ6ci X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C488C160017 X-Stat-Signature: j6xc87ejhfpsngnp8ytdbmyog6ip4rkz X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspam: Yes X-HE-Tag: 1702887760-111152 X-HE-Meta: U2FsdGVkX1/K5n+C3JjYej18XyFJfbTpzqpYx7nyf+86rj8bmcw8nrkTYRKWbhe6E9WBgppcWZjIBceOqO+B7CVqyCbNriro1mQkODpNrk8FXEBrCbSFODkl4wWg6lrFNHErqlCx/FaD9nVxNRVGPrx8/widFWarkImMnxDvbTGdHIxM5JanNCKgllF+LIa556ViO2JQxanDiDqNct0jTXyWtZBvJbJbc+XdYp9K0MwwmZ7TSnee9W8oEj90qkDeNmYf6ojwjKMAg+md4fm54dmmoFvGLDRKehTrzkg4g9AIvUdi/hOV1DUJA3mKXlOq3nR0rPoypFGnx/r4B6oRUtXmedDTiax/0ZCVS0b2JEtPee9prnYfkwOVLRjV/flStKL7rb7L0p1zF4bXyiaIcKs5ZNt5trw2HMkmq39XAbld+TfI7MRId/L68/FTBouDSiqVunRpqCThTlkfxYq8kUPWzsTeZYRBqxCkvUqo5xDH6mcggrlkv84epzgTFx9RQJ5/c/OP7xAbbITWdxtVdUJD+mAGqXm6XN1oJR8goOfLV67QwH+kkF8jggbhQix8aAqYIs/GErofwEnRPW6RTmKiS4pn1T8wPMiqcRV1GDI5oIXzXe1aS0+v/Yw0pLq4OG9fnFB4LugiuWl57jcNgDsMxRZxcmzXC+clgahKxgeyR40CfEiC0O27Ye4j4WYLAhdMx7i9kYY7W1CcxryYisAHNkEVvUIPIm4oIwHOfb0WhHHqj3loFP1vqShuXygpBQSa+DEXUIUGGw03gFAqZtdNxIO3Ai1sHlTCTeS75ykvhWsZkOFH/35XTZveJPtoPeYZiF9cxXT8YMjijbGEizzzJW4fUH/o+5fCQpKdJZqxoxOleC6iRDH+hf6t/cLp9J7cP60TpNrd3cPjuHfzaPRUwSc8FRSkcornILviWZceknZHNpHAy7iRrngY9CHATmyPUKVVs5gW/z9mKgV BYgaIvGs 9Ed9n8AmIdkNzZDnmBxGrv9DIVgDhgzBNotKooKpqsjxUfdZdYrIkvF+ipbM79N/a6b5cVqnUbC1QWw7umNBvulEvgUedDxvxG7ty+7cWkC95TEAOtUUIQJwovindWd4CdHd5deABYUH+76ylVI7dQ7gziuCvKeF/+06fr/1gartcQw42wdmNthstz+fB3o9Z59Ka/v74NHbkY2QpONDl5I1rDWHJv+WskgEthArrM+pkuyXvDBLFcKKcBVEaSyhpGZZrULJOqEgD41T4hu6/kvdPprVpbplUCTuLtbngozTSslhdFVTUcVBADdowTnEu2LK3i8+slb6kjZ4COIoLa6GnShTRm8oTEILUDATo9lUX/YLiNdvmZx4mkta3dUTlZfPKwsJXdBkNhmI731z6ZOMfRTsPg9FMku4GCZJ4RXCkEX7FauhY2I6F+g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After the common decompress part goes to __zswap_load(), we can cleanup the zswap_load() a little. Reviewed-by: Yosry Ahmed Signed-off-by: Chengming Zhou --- mm/zswap.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 3433bd6b3cef..86886276cb81 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1759,7 +1759,6 @@ bool zswap_load(struct folio *folio) struct zswap_tree *tree = zswap_trees[type]; struct zswap_entry *entry; u8 *dst; - bool ret; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1776,19 +1775,16 @@ bool zswap_load(struct folio *folio) dst = kmap_local_page(page); zswap_fill_page(dst, entry->value); kunmap_local(dst); - ret = true; - goto stats; + } else { + __zswap_load(entry, page); } - __zswap_load(entry, page); - ret = true; -stats: count_vm_event(ZSWPIN); if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); spin_lock(&tree->lock); - if (ret && zswap_exclusive_loads_enabled) { + if (zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); folio_mark_dirty(folio); } else if (entry->length) { @@ -1798,7 +1794,7 @@ bool zswap_load(struct folio *folio) zswap_entry_put(tree, entry); spin_unlock(&tree->lock); - return ret; + return true; } void zswap_invalidate(int type, pgoff_t offset) From patchwork Mon Dec 18 08:22:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A69CC35274 for ; Mon, 18 Dec 2023 08:22:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A40018D000A; Mon, 18 Dec 2023 03:22:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C8A68D0001; Mon, 18 Dec 2023 03:22:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81B8E8D000A; Mon, 18 Dec 2023 03:22:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6F2148D0001 for ; Mon, 18 Dec 2023 03:22:45 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 516621608A2 for ; Mon, 18 Dec 2023 08:22:45 +0000 (UTC) X-FDA: 81579247890.09.7660A69 Received: from out-177.mta1.migadu.com (out-177.mta1.migadu.com [95.215.58.177]) by imf19.hostedemail.com (Postfix) with ESMTP id 674531A0010 for ; Mon, 18 Dec 2023 08:22:43 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.177 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702887763; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7HR/MLv2aBLIXlJNr8b9Lqs7UJ1j9JsFWT1nUw7L6eQ=; b=N8fc81Ep4XYdiKSxHTmTu5s6pOBORCL5pbUZnN0hIR6gDtOto7PoLCCNiJ3teX0f9NRo6n odgbulG2ryl31ChhTrJqUxcgoM9oMRhUYvQGq959pQ7pGEvchKDw24CMIybW0ahkLjGZfW NfZGs57NkKskdMfXXkcbDODpOHEIUes= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702887763; a=rsa-sha256; cv=none; b=KwkgDbxpCDrrwLFTWinbcltVOmsP8N3z2gzAcPB0OxGdOH/xl0sUqhCbogavF6npmC86cu UQa5kxIOkUc4ja43yTPLoRWYhlCSscrYSZDQCahGXkItXn3UgzLsy7i70mFYQU0mwHb2sN eTks8xPrfJZh18g7r/fm5BD+R4YBmNk= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.177 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 08:22:04 +0000 Subject: [PATCH v2 5/6] mm/zswap: cleanup zswap_writeback_entry() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v2-5-daa5d9ae41a7@bytedance.com> References: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> To: Seth Jennings , Dan Streetman , Chris Li , Nhat Pham , Vitaly Wool , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-kernel@vger.kernel.org, Nhat Pham , linux-mm@kvack.org, Yosry Ahmed , Chengming Zhou , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1702887745; l=2215; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=u9D8B3ksO9G67avNjANTAO3qS+/SASwDD1QW2eXuPmA=; b=Y/k0K3dE80CEovxET6orobtOyqyRJJHYc5i0dttC7jiXtrRjSziXVF7ngHADT55LxnZB0hdN9 VU8d58ziWvID/Hp36IATF7GQgb3Z0riqtM9YUjDT6aMJe1wiTZkqkT7 X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspamd-Queue-Id: 674531A0010 X-Rspamd-Server: rspam11 X-Stat-Signature: gdxkcoenamegj1ssnqc4akfjtrhe7gks X-Rspam: Yes X-HE-Tag: 1702887763-49730 X-HE-Meta: U2FsdGVkX1+kQ236GltFO2OOBEnckGVhq8mX30rww86rwcUYw3dvP6TPt9UonQFfD6KkNpBNY8wPQcOrkRtY9N/RHdwMBuFhAjZd71X4aG8MwjZa/PJVjsmEHRsB+wZoSXaMok3BJfhms1I8LFwLn2Yav4cS2c6rzNkWJ3HRiVo9/25UOsyADiu9a6l7OIUE+jmA5J5F3kEJPLIEOn2l1m8yFZXFGoKe+O9dZo5IyHL0GI2jEdMDrZ4ss8rFHC+3V10emmIhpoA3G0kKKuMhQJxEsaB4IiInnZ1Sxw5C5/6f7KYb1RUYAaty0eAO2wuUr8O/gN5hcbf1bUA38VmbrUpmo0zIHlIqDcgDjRuSbbp/9JAKlXbWu3Fck5u/zwkxgZ0lHPtVWpK1//SjE1ID5XGWqRmBTNMJuN1Tb3sPEn0RChig3lNrtD0DSw83n5huksnzbASUJoSbMqtv0M6M14NcAqwmZCTeF6EWWnRCrhLC+SCuYZ5lZNn6lO64fm942csO3UfgqaRDCVwAr2mhNij01HzwreLJdO1jQINCU7igDxPb2e0Q7jCiGvMCEmRG7TfjWVKcLijYO0pwy1Cp3Of3CBh/o2/qoiTkmwnHkTd2EtInoA1l5anfja+K0wio7U5n5CuP6rqYBiw7D/0tEpaZ21m9sOLGK+CevmHCItK/B5KXOfk5jL6S1C51wD0M5nXfh9b2GECej8omJsrPV0qZ8b0fZG3XeNcWNjt5/hcnn42DPRh0skgg8IpvjapWcPmXssT8U+XdEch8L8kUya8Ar/+YqI1qTksj/46BAja/3Og2V7eHdDpVaU0AkaL4wTroc+cDURUFrWzKvoexD03zuHqFylCny7jA6klhAvGBiED05COc3ubCik6X8R+BHKGWtnJuq3WKxizOPjwWaXwhG4nADx2IgzuWYFoJyG5FBfbTOOd3VREZi6QwOHv/zgizKssTrLn/FnWZaUs EdI4NlgG 4H79FNq8f/T/f65YYyBUVy/K6tQhQkd/OuGkOgHUaRWhey/sDjo8JziiMgW2Kprbt+U8WkuoI82wR3NNetjltfSZqAhyriUcF99sYWze/58J93cZNHfq37yCjO+d+n3wI1x5jxtdhkThhJjgzasLVl9zLUy9fvRc9LtUazrexRMe19lWShkumCsdyW5bzpqLoLAnRjXYXrot/K9JZqN0e8wBvGReACHmw1+5+2ULAZidOWSSMCTjQGtm+tBTTKdCef+NFUhdi/u/YePCHUDrXcuHKSppG7a1TgW/7pts59fKiJVaKf4Hol9QOu57NA4KdaOnhqIgawnum3EoFObfBScX8Jk3GFboaasa8HA76mN3tKPBgCnmIDVA51Ormx5IbM5aF1IqN48F723IFZ/9tvCqXrHGwbhsd5rS77lAeAOuwPUTbH8mQ25J+JkFRAbMjZEbNTJc8KgcJ9gc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Also after the common decompress part goes to __zswap_load(), we can cleanup the zswap_writeback_entry() a little. Reviewed-by: Yosry Ahmed Reviewed-by: Nhat Pham Signed-off-by: Chengming Zhou --- mm/zswap.c | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 86886276cb81..2c349fd88904 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1443,7 +1443,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct page *page; struct mempolicy *mpol; bool page_was_allocated; - int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, }; @@ -1453,15 +1452,18 @@ static int zswap_writeback_entry(struct zswap_entry *entry, page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, NO_INTERLEAVE_INDEX, &page_was_allocated, true); if (!page) { - ret = -ENOMEM; - goto fail; + /* + * If we get here because the page is already in swapcache, a + * load may be happening concurrently. It is safe and okay to + * not free the entry. It is also okay to return !0. + */ + return -ENOMEM; } /* Found an existing page, we raced with load/swapin */ if (!page_was_allocated) { put_page(page); - ret = -EEXIST; - goto fail; + return -EEXIST; } /* @@ -1475,8 +1477,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, if (zswap_rb_search(&tree->rbroot, swp_offset(entry->swpentry)) != entry) { spin_unlock(&tree->lock); delete_from_swap_cache(page_folio(page)); - ret = -ENOMEM; - goto fail; + return -ENOMEM; } spin_unlock(&tree->lock); @@ -1497,15 +1498,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, __swap_writepage(page, &wbc); put_page(page); - return ret; - -fail: - /* - * If we get here because the page is already in swapcache, a - * load may be happening concurrently. It is safe and okay to - * not free the entry. It is also okay to return !0. - */ - return ret; + return 0; } static int zswap_is_page_same_filled(void *ptr, unsigned long *value) From patchwork Mon Dec 18 08:22:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09783C35274 for ; Mon, 18 Dec 2023 08:22:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 931088D000B; Mon, 18 Dec 2023 03:22:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B95A8D0001; Mon, 18 Dec 2023 03:22:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 732898D000B; Mon, 18 Dec 2023 03:22:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4F54D8D0001 for ; Mon, 18 Dec 2023 03:22:48 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 21F9A14016F for ; Mon, 18 Dec 2023 08:22:48 +0000 (UTC) X-FDA: 81579248016.20.A838C55 Received: from out-171.mta1.migadu.com (out-171.mta1.migadu.com [95.215.58.171]) by imf28.hostedemail.com (Postfix) with ESMTP id 157A5C001A for ; Mon, 18 Dec 2023 08:22:45 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf28.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702887766; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=581OMV4XATgN37VOge48RZVp6yt/xtrZwVFcAWt1qoE=; b=3ZCD5aX+y01RITdhzaTM5WENXhBini8AQxIj/gNi6tRY1NxpDByvZV8kBpgt2H91tyXgOc S97C02Wk0ZC7kTM11YmzvH9qTnBDcw5RKxsQFYaAsbeK43hyO5okdMV2Z63u1ZPKc/2DCI oSZ9Pzyax6aY94MZ8ZB8aFS6PPzfIqU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf28.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702887766; a=rsa-sha256; cv=none; b=Tl4cK+o2F40w3Q5sxuxLviuF5mNcMIpVT4BIp4TIqtmvZOKpzPxPpiPJPl3Kb/hOcvg0XB ZjxyZhJGvphYbnXh/AjUIXenxCm5biALuynCvGtFxj8nj4yQgvUIqGVjB0UP2IRgEYeY+a 1VIMPUMI3TBTqwUAFz4dC2Xy1hteXng= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 08:22:05 +0000 Subject: [PATCH v2 6/6] mm/zswap: directly use percpu mutex and buffer in load/store MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v2-6-daa5d9ae41a7@bytedance.com> References: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v2-0-daa5d9ae41a7@bytedance.com> To: Seth Jennings , Dan Streetman , Chris Li , Nhat Pham , Vitaly Wool , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-kernel@vger.kernel.org, Nhat Pham , linux-mm@kvack.org, Yosry Ahmed , Chengming Zhou , Chris Li X-Developer-Signature: v=1; a=ed25519-sha256; t=1702887745; l=10608; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=k6WkjFerLmMPPnB4BHXU0oLODY4+tgFUEVqHpaqy/GE=; b=L3QlA6B3Z8gMjlH/tKC9DFpE0gONp9++QxZWqj4jttxtA28i2k06FoJnhzPXl2dTnUCRe0O2I /yfAOqL4yTTBVAWzMzyY/vp/6g77noItOcvUhDa+h91UaW5hpum/Bxo X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Stat-Signature: 4n1rgybxi6ipsfahuqcfbbba5guz78g6 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 157A5C001A X-Rspam: Yes X-HE-Tag: 1702887765-638061 X-HE-Meta: U2FsdGVkX1+4NoNTxzAcNfTcD1Srt3TLsH6NWJxLUY6CAilHGjDhLYuQMa+6vPlVcw+EJP6o3Hh7T5gMfSHQAuGKci18pk/BM1+jGdbxClLqdCU5c7pZs3t69PMLoHg+yw5LHeFB312rLaEDBdbqi1IGoeIrZKkNzIiF49AD9lvPD7oYlOMMUgSACFudsahGS0ziRZt45zw57oeROMAeEpnsPg40LfwTiguCbSyNRWPexTqTksrfECqCjtIVRH1fFgItOwf2t0Ac1YCkizRvlp2R8f1Sucx95B/D9XdIiyzdBlaJwxqCD0L8Y2DTUEeiVaA1FkPJpod68xFTMKcrc7Xk945Y/kgk0xktPS8g/g4PeBT0lS4jTLuq9ObYsq2u5Y/6HubeIO3DEPol2AypxPeZiRmT94KqLRcM/ve27cnsHTR9xUTDRUEOKn8ZwI2/9rgtsP2L7AU9Lf1VNNejtv/6WDTjcVVdJQ29nPQvLwZOsSpj5kHs+N8UNeeCkyigwEm6/ZTKTwiaNz7ESgkGp3XO/PYWDlldT3WfW4o44hozNS/xcx+C/n1PrvKY70L1PYuvTAL5PMX33/ByHly40ixO3mNqbrSkrvT8Oc5vFQgdUZGLDqELYU3LiwlbN2o74kuBkBKAOBjtQSiMwEfF7jh6/kNh2wfYZkc993veuxnuYw+3XEbgEUI5XK26jq4nM5BC4q/9TfKUdtMRr/BmnJmZmUFhWb4VzAGuJHR9w3lcubbHOsj2V972UlOviZU2/1VPH0axHcw0pNgEjn0bo5lfHu65bDirmk2q6xiFCq0eaRlnWJHCG5cEyiAStECo7nMfYTDr/6DDBG/Eym0QcEraUeqzrrMPbG0j8qRXc+9TLo0x/o/EpL8AkIaun1vI109BynhZmBlpAFwq20Z5s0ijJGx45XKJvrnvZ8KTRDfbuYJYTZoMS86q0pRxF4U4rWeCH03ZO+BnfCnLKUT fWrLXxhF Adkomrpj4JpVXHYuofwM1bdN2Non5TQqH+a6agRDIqXYHk0Lc6kICMh90HFwyAdp0FE+V8qaPXCUCMBqsLMLzDwL9BmE23OzM5o5hSb3eI+XfflSUb7HRIlxTdI2fe4dJKLh5Ps2xGk8hqMLEGhAF/a1LCLcmqzR25krASyV/MJN+/f6ccIlnggs5H79a8WjZHEREN40pNTsKGVRDRbCipvRO4ZJwmVqw2jVC0amB7wuD2F2z6/uYgBtsGpQaDoMpmbBBNY7882gRUkaEKfMwT0wAjuv7JGGh6l8Rn+iXouhZan+SPugjjqqbamigODl/M1xlrM0eYvAP4djF4ofznwsdf+fmmcEz5yI1+9kc6iFdqROseyj+U3chnZmiS4yrGzWi X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since the introduce of reusing the dstmem in the load path, it seems confusing that we are now using acomp_ctx->dstmem and acomp_ctx->mutex now for purposes other than what the naming suggests. Yosry suggested removing these two fields from acomp_ctx, and directly using zswap_dstmem and zswap_mutex in both the load and store paths, rename them, and add proper comments above their definitions that they are for generic percpu buffering on the load and store paths. So this patch remove dstmem and mutex from acomp_ctx, and rename the zswap_dstmem to zswap_buffer, using the percpu mutex and buffer on the load and store paths. And refactor out __zswap_store() to only include the compress & store, since I found zswap_store() is too long. Suggested-by: Yosry Ahmed Signed-off-by: Chengming Zhou --- mm/zswap.c | 193 +++++++++++++++++++++++++++++++++---------------------------- 1 file changed, 104 insertions(+), 89 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 2c349fd88904..b7449294ec3a 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -166,8 +166,6 @@ struct crypto_acomp_ctx { struct crypto_acomp *acomp; struct acomp_req *req; struct crypto_wait wait; - u8 *dstmem; - struct mutex *mutex; }; /* @@ -694,7 +692,7 @@ static void zswap_alloc_shrinker(struct zswap_pool *pool) /********************************* * per-cpu code **********************************/ -static DEFINE_PER_CPU(u8 *, zswap_dstmem); +static DEFINE_PER_CPU(u8 *, zswap_buffer); /* * If users dynamically change the zpool type and compressor at runtime, i.e. * zswap is running, zswap can have more than one zpool on one cpu, but they @@ -702,39 +700,39 @@ static DEFINE_PER_CPU(u8 *, zswap_dstmem); */ static DEFINE_PER_CPU(struct mutex *, zswap_mutex); -static int zswap_dstmem_prepare(unsigned int cpu) +static int zswap_buffer_prepare(unsigned int cpu) { struct mutex *mutex; - u8 *dst; + u8 *buf; - dst = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); - if (!dst) + buf = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); + if (!buf) return -ENOMEM; mutex = kmalloc_node(sizeof(*mutex), GFP_KERNEL, cpu_to_node(cpu)); if (!mutex) { - kfree(dst); + kfree(buf); return -ENOMEM; } mutex_init(mutex); - per_cpu(zswap_dstmem, cpu) = dst; + per_cpu(zswap_buffer, cpu) = buf; per_cpu(zswap_mutex, cpu) = mutex; return 0; } -static int zswap_dstmem_dead(unsigned int cpu) +static int zswap_buffer_dead(unsigned int cpu) { struct mutex *mutex; - u8 *dst; + u8 *buf; mutex = per_cpu(zswap_mutex, cpu); kfree(mutex); per_cpu(zswap_mutex, cpu) = NULL; - dst = per_cpu(zswap_dstmem, cpu); - kfree(dst); - per_cpu(zswap_dstmem, cpu) = NULL; + buf = per_cpu(zswap_buffer, cpu); + kfree(buf); + per_cpu(zswap_buffer, cpu) = NULL; return 0; } @@ -772,9 +770,6 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, crypto_req_done, &acomp_ctx->wait); - acomp_ctx->mutex = per_cpu(zswap_mutex, cpu); - acomp_ctx->dstmem = per_cpu(zswap_dstmem, cpu); - return 0; } @@ -1392,20 +1387,98 @@ static int zswap_enabled_param_set(const char *val, return ret; } +static int __zswap_store(struct zswap_entry *entry, struct page *page) +{ + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; + struct zpool *zpool; + unsigned long handle; + unsigned int dlen; + u8 *buf, *dst; + gfp_t gfp; + int ret; + int cpu; + struct mutex *mutex; + + cpu = raw_smp_processor_id(); + mutex = per_cpu(zswap_mutex, cpu); + mutex_lock(mutex); + + acomp_ctx = per_cpu_ptr(entry->pool->acomp_ctx, cpu); + buf = per_cpu(zswap_buffer, cpu); + + sg_init_table(&input, 1); + sg_set_page(&input, page, PAGE_SIZE, 0); + sg_init_one(&output, buf, PAGE_SIZE); + acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, PAGE_SIZE); + /* + * it maybe looks a little bit silly that we send an asynchronous request, + * then wait for its completion synchronously. This makes the process look + * synchronous in fact. + * Theoretically, acomp supports users send multiple acomp requests in one + * acomp instance, then get those requests done simultaneously. but in this + * case, zswap actually does store and load page by page, there is no + * existing method to send the second page before the first page is done + * in one thread doing zwap. + * but in different threads running on different cpu, we have different + * acomp instance, so multiple threads can do (de)compression in parallel. + */ + ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait); + dlen = acomp_ctx->req->dlen; + + if (ret) { + zswap_reject_compress_fail++; + goto unlock; + } + + /* store */ + zpool = zswap_find_zpool(entry); + gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; + if (zpool_malloc_support_movable(zpool)) + gfp |= __GFP_HIGHMEM | __GFP_MOVABLE; + ret = zpool_malloc(zpool, dlen, gfp, &handle); + if (ret == -ENOSPC) { + zswap_reject_compress_poor++; + goto unlock; + } + if (ret) { + zswap_reject_alloc_fail++; + goto unlock; + } + dst = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); + memcpy(dst, buf, dlen); + zpool_unmap_handle(zpool, handle); + mutex_unlock(mutex); + + entry->handle = handle; + entry->length = dlen; + return 0; + +unlock: + mutex_unlock(mutex); + return ret; +} + static void __zswap_load(struct zswap_entry *entry, struct page *page) { struct zpool *zpool = zswap_find_zpool(entry); struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; - u8 *src; + u8 *src, *buf; + int cpu; + struct mutex *mutex; + + cpu = raw_smp_processor_id(); + mutex = per_cpu(zswap_mutex, cpu); + mutex_lock(mutex); - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); + acomp_ctx = per_cpu_ptr(entry->pool->acomp_ctx, cpu); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; + buf = per_cpu(zswap_buffer, cpu); + memcpy(buf, src, entry->length); + src = buf; zpool_unmap_handle(zpool, entry->handle); } @@ -1415,7 +1488,7 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(mutex); if (zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); @@ -1539,18 +1612,11 @@ bool zswap_store(struct folio *folio) struct page *page = &folio->page; struct zswap_tree *tree = zswap_trees[type]; struct zswap_entry *entry, *dupentry; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg = NULL; struct zswap_pool *pool; - struct zpool *zpool; - unsigned int dlen = PAGE_SIZE; - unsigned long handle, value; - char *buf; - u8 *src, *dst; - gfp_t gfp; - int ret; + u8 *src; + unsigned long value; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1635,60 +1701,11 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } - /* compress */ - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - - mutex_lock(acomp_ctx->mutex); - - dst = acomp_ctx->dstmem; - sg_init_table(&input, 1); - sg_set_page(&input, page, PAGE_SIZE, 0); - - sg_init_one(&output, dst, PAGE_SIZE); - acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); - /* - * it maybe looks a little bit silly that we send an asynchronous request, - * then wait for its completion synchronously. This makes the process look - * synchronous in fact. - * Theoretically, acomp supports users send multiple acomp requests in one - * acomp instance, then get those requests done simultaneously. but in this - * case, zswap actually does store and load page by page, there is no - * existing method to send the second page before the first page is done - * in one thread doing zwap. - * but in different threads running on different cpu, we have different - * acomp instance, so multiple threads can do (de)compression in parallel. - */ - ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; - - if (ret) { - zswap_reject_compress_fail++; - goto put_dstmem; - } - - /* store */ - zpool = zswap_find_zpool(entry); - gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; - if (zpool_malloc_support_movable(zpool)) - gfp |= __GFP_HIGHMEM | __GFP_MOVABLE; - ret = zpool_malloc(zpool, dlen, gfp, &handle); - if (ret == -ENOSPC) { - zswap_reject_compress_poor++; - goto put_dstmem; - } - if (ret) { - zswap_reject_alloc_fail++; - goto put_dstmem; - } - buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); - memcpy(buf, dst, dlen); - zpool_unmap_handle(zpool, handle); - mutex_unlock(acomp_ctx->mutex); + if (__zswap_store(entry, page)) + goto put_pool; /* populate entry */ entry->swpentry = swp_entry(type, offset); - entry->handle = handle; - entry->length = dlen; insert_entry: entry->objcg = objcg; @@ -1725,8 +1742,6 @@ bool zswap_store(struct folio *folio) return true; -put_dstmem: - mutex_unlock(acomp_ctx->mutex); put_pool: zswap_pool_put(entry->pool); freepage: @@ -1902,10 +1917,10 @@ static int zswap_setup(void) } ret = cpuhp_setup_state(CPUHP_MM_ZSWP_MEM_PREPARE, "mm/zswap:prepare", - zswap_dstmem_prepare, zswap_dstmem_dead); + zswap_buffer_prepare, zswap_buffer_dead); if (ret) { - pr_err("dstmem alloc failed\n"); - goto dstmem_fail; + pr_err("buffer alloc failed\n"); + goto buffer_fail; } ret = cpuhp_setup_state_multi(CPUHP_MM_ZSWP_POOL_PREPARE, @@ -1940,7 +1955,7 @@ static int zswap_setup(void) zswap_pool_destroy(pool); hp_fail: cpuhp_remove_state(CPUHP_MM_ZSWP_MEM_PREPARE); -dstmem_fail: +buffer_fail: kmem_cache_destroy(zswap_entry_cache); cache_fail: /* if built-in, we aren't unloaded on failure; don't allow use */