From patchwork Tue Jan 30 01:36:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13536673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37D29C47DB3 for ; Tue, 30 Jan 2024 01:42:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4CEBA6B00AE; Mon, 29 Jan 2024 20:42:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4515B6B00B0; Mon, 29 Jan 2024 20:42:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A3DE6B00B1; Mon, 29 Jan 2024 20:42:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 167C56B00AE for ; Mon, 29 Jan 2024 20:42:25 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id EF62C802DE for ; Tue, 30 Jan 2024 01:42:24 +0000 (UTC) X-FDA: 81734277408.23.E1AFC6A Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf08.hostedemail.com (Postfix) with ESMTP id 37E1E160002 for ; Tue, 30 Jan 2024 01:42:23 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=in4cnkkw; spf=pass (imf08.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.180 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706578943; a=rsa-sha256; cv=none; b=cI+ZfYvQnnpTJWwsGiMg+3Gg9Exbv0qcXgUO2r8B4S38UiUQnVY3wK/tue9Ag7OHyMJKx6 c6GJaRGcHomTIoauGc3HIoI4j12tcND5OP56XUJOsFFjz1zhhaWooJvykR6Q9vI075njcS RaKwnswC30P70voges60Pixd6nvL/iI= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=in4cnkkw; spf=pass (imf08.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.180 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706578943; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mpwOGEgV3dvwZoaILLJtRsR55d6E5hoMpUKMvxq3d70=; b=jFsAqW2xfRe5TiuXcm32fKDuqphCliUQ3oWGW203NGk0te3Q6mnUxpEOrjYXm2kEiAO3QM vD62lgG36ZF2W8zkTCtEmyrM/QJoaSqqpiQFQg8K7FUc4EASwUI4zLddc21hCUA4ORxprm IPOs/3Nm/Qt2Woy7eMQ/nqowTgls/6w= Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-783da26489aso280792685a.3 for ; Mon, 29 Jan 2024 17:42:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1706578942; x=1707183742; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mpwOGEgV3dvwZoaILLJtRsR55d6E5hoMpUKMvxq3d70=; b=in4cnkkwLNOR5rM549JsNbueXvdxSGPdERc4QAfaBzUqjM98P0A4GF/vCq06vLg6yA IXyk92SBRK8JQm+pHQRnBKMKF9fOGXXQGtGmTZjACm0t7lR73wgWLpsdy/UkmFQ4hzV0 dnIS5mDhZw6HK/FJUUx6tNKWZ+N10sK5lOnB5fDmYDpAGAhPZyMdup91gHztacS3JjUR jrOtV2/pzUpZsLh4cmHv1SvxL2P2PNt4yvUW/YtrGPvEH7QIYDus/zsCoSofCboGocwA pglNa9cxDNmYLMfU18YEzR575TUWjLfd8BiUfcCkKzYrsqTgtTT1b4di7vgWla32ch0K 3c5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706578942; x=1707183742; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mpwOGEgV3dvwZoaILLJtRsR55d6E5hoMpUKMvxq3d70=; b=g2dXfg3eOlxo3QBZMSXjOI4uljaeE6+clL5ERWVVV717cA/uk7f7Pm0lXWnL1mSDA/ 6goIDcM0SBUaiujOZ+yLr35r+QRyG8eDvQcdyOIq50JHQyCg2TiUBgBrTEu2P8M/h8Lg ntdB/WjhkE5sGyT8q5ew4DgLsLG0PqGV47pt7EXzqDse4vJvm5u31jQlIwP3n3iIAxXv fUrvqztJYi8+wP85UE7EfhL4gyNCOi5X0xyGHDCbBDaGFtpTxIYPjzgpdN4QNnLJiIus hMLwZvK7jhgmqkTgW9bFY2mkPcmidnvwbHRxCAYbLwsGHpygsNOVyYQ7JQxZq6HmrVIl FMgQ== X-Gm-Message-State: AOJu0Yz6FMCobsGnkD4J6BtGmLSIQ/cKEwZ/EeV+NxmkUyX3hUCLUq4g WNgpWEUJh0Ea4UIZxbhkIpKp7NVp1unIZTCOPzBX66m15jfvrw6W8+jpZIO1HxU= X-Google-Smtp-Source: AGHT+IGAaMHNgHtWkFaPNS95tGZ/hwePQLFIY+O4EOkM3Bj1xV6augPRQXjZ/4cnW0lftD5scOiZPw== X-Received: by 2002:a05:620a:b11:b0:783:d096:d761 with SMTP id t17-20020a05620a0b1100b00783d096d761mr5637186qkg.31.1706578942391; Mon, 29 Jan 2024 17:42:22 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id po30-20020a05620a385e00b00783e3b030e7sm2669196qkn.33.2024.01.29.17.42.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 17:42:21 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Nhat Pham , Yosry Ahmed , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 07/20] mm: zswap: break out zwap_compress() Date: Mon, 29 Jan 2024 20:36:43 -0500 Message-ID: <20240130014208.565554-8-hannes@cmpxchg.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240130014208.565554-1-hannes@cmpxchg.org> References: <20240130014208.565554-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 37E1E160002 X-Stat-Signature: 36xmrphegmqtfxzded6dmjtej3iirei5 X-Rspam-User: X-HE-Tag: 1706578943-92456 X-HE-Meta: U2FsdGVkX1/8UHMogSHtlH1aEHfkFVc33yOljSCxI86ip0Zzpjvolt6o5hifMb84ybdVlPA5bz6wubyuKAtUMAx9UZKh+7tccMswdlPL9AOSmbcmn7sR51TooOJrPgWjWpokxbgUpzlHAvjwc/ypvBvqnCNEIg3bTC5+nq2pTxOjlR/jmnOmdjR56jx4mK2NDGcC4GU2uuuDy3R02aCtbZrafHAjqhTKg287iGaozhFgHN+OwDIHWv2bsycDiNwZ3kOXYhnoKneMdJnE7LCYxbzl0zrU09XboJhQjx/8m3SjfiPt+jLamgS4d3FBBnZAigWaoBN6nBY+hCXl3cuLkyqq6I/MSoOsIKHEEr17xer9d+Rm0mY8FxnJpiY1Vw+fDN2Gd2zttdEi62qIYH1s6XgppcJdR7IGu+Q2meCJcBfRP2pWV+7rzW4h+EVTzZ9xaVrUFbOpJBnm3S/zaD3hxinZsG7w20oMOIXZONMp9bnRcxZihIIT9v0xa3ymO8cHJl2b3sQNYImh1bL0NKGy3iQh+laB99qfv2j/p59+KVRELJfy1toLpI4pIEl64fQOXqxYIYdD6GAQl69OzkTvtyGRcFImxUCMpjDBH9faAza53aWJXnoKF6LUcfwok4dwwSlNf5I1LBsPw2EZOWDf86NL7oTx4s+80+CjyyZfWmo7XVDbXLl/goIa5cXE5Ws/KBJIUFqUNqeBQdi4udKsfMHrVtCbDCAkrMwI34WHX7vDgd7qYvK5ZPfO5zgfGfHL4gy2yDH0UUSMaO8lyGrH4FnNpwn7JtkWc/6Ho3XGWAFZ20GZIliwoghjqFFzksMQG2TwIU1JUGLLwMJNBK3kSZeZOI0g8OZdwSNv2wYf2L4nmM7cHVfRbHrea3/UfeD/uGpnIlrKBgL4roB5zlNVbIbMuEi5/XjYSjc1qzJ4rSlSdk8zyMn9VaoAaxZOhHTyOkWFAUvt8U2Q3zKhnrD /RvL3Pht l0XSIbFSuJBA7MYfcziKQy0bOP+C15+T5anPm5/e/4fML8iQQgG4AyjzgNVbKVmlDaA/XQ60YeTQr98dPuqMMoB4prEhO90S1O8Cc3tyOjdUuPtT4DKhYc5+DRyod+AEbMzdSEG+VAhksFQUA6IUaLZoBTZ6FDOPrukSWhJ/LjTvjgQMj/LhUHZ280SXEamCwJKeKWPvGNGOVtLTJv14WHwIsEokDxMQhayFeB5QgNlD/jTOYJKv1s3KG9QqXENNEkFraKSilXiqIVnD0oylap00o/ZHF9XNATuZnWIUUcFUwTfdFKVmNFrh3alqGG2UPsyJ5B1/y6ODZytDbbxqFH8Zq7htS3RcXFvFk3qXPt0TZWngOeZQ36zBiIF/qWAMVHK87E6+Z41r82Ia8ZooK65EHSmn2eopxCg6qDCwLnUE7oHaT9llKPYUOExR5Y3FT6ppU49mRnV8YbUpw6Loofu0t9Inp5VY84VMGUecsBZ2juPc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: zswap_store() is long and mixes work at the zswap layer with work at the backend and compression layer. Move compression & backend work to zswap_compress(), mirroring zswap_decompress(). Signed-off-by: Johannes Weiner Reviewed-by: Chengming Zhou Acked-by: Yosry Ahmed Reviewed-by: Nhat Pham --- mm/zswap.c | 145 ++++++++++++++++++++++++++++------------------------- 1 file changed, 77 insertions(+), 68 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index bdc9f82fe4b9..f9b9494156ba 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1316,6 +1316,79 @@ static int zswap_enabled_param_set(const char *val, return ret; } +static bool zswap_compress(struct folio *folio, struct zswap_entry *entry) +{ + struct crypto_acomp_ctx *acomp_ctx; + struct scatterlist input, output; + unsigned int dlen = PAGE_SIZE; + unsigned long handle; + struct zpool *zpool; + char *buf; + gfp_t gfp; + int ret; + u8 *dst; + + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + + mutex_lock(&acomp_ctx->mutex); + + dst = acomp_ctx->buffer; + sg_init_table(&input, 1); + sg_set_page(&input, &folio->page, PAGE_SIZE, 0); + + /* + * We need PAGE_SIZE * 2 here since there maybe over-compression case, + * and hardware-accelerators may won't check the dst buffer size, so + * giving the dst buffer with enough length to avoid buffer overflow. + */ + sg_init_one(&output, dst, PAGE_SIZE * 2); + acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); + + /* + * it maybe looks a little bit silly that we send an asynchronous request, + * then wait for its completion synchronously. This makes the process look + * synchronous in fact. + * Theoretically, acomp supports users send multiple acomp requests in one + * acomp instance, then get those requests done simultaneously. but in this + * case, zswap actually does store and load page by page, there is no + * existing method to send the second page before the first page is done + * in one thread doing zwap. + * but in different threads running on different cpu, we have different + * acomp instance, so multiple threads can do (de)compression in parallel. + */ + ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait); + dlen = acomp_ctx->req->dlen; + if (ret) { + zswap_reject_compress_fail++; + goto unlock; + } + + zpool = zswap_find_zpool(entry); + gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; + if (zpool_malloc_support_movable(zpool)) + gfp |= __GFP_HIGHMEM | __GFP_MOVABLE; + ret = zpool_malloc(zpool, dlen, gfp, &handle); + if (ret == -ENOSPC) { + zswap_reject_compress_poor++; + goto unlock; + } + if (ret) { + zswap_reject_alloc_fail++; + goto unlock; + } + + buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); + memcpy(buf, dst, dlen); + zpool_unmap_handle(zpool, handle); + + entry->handle = handle; + entry->length = dlen; + +unlock: + mutex_unlock(&acomp_ctx->mutex); + return ret == 0; +} + static void zswap_decompress(struct zswap_entry *entry, struct page *page) { struct zpool *zpool = zswap_find_zpool(entry); @@ -1472,18 +1545,11 @@ bool zswap_store(struct folio *folio) struct page *page = &folio->page; struct zswap_tree *tree = swap_zswap_tree(swp); struct zswap_entry *entry, *dupentry; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg = NULL; struct zswap_pool *pool; - struct zpool *zpool; - unsigned int dlen = PAGE_SIZE; - unsigned long handle, value; - char *buf; - u8 *src, *dst; - gfp_t gfp; - int ret; + unsigned long value; + u8 *src; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1568,65 +1634,10 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } - /* compress */ - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - - mutex_lock(&acomp_ctx->mutex); - - dst = acomp_ctx->buffer; - sg_init_table(&input, 1); - sg_set_page(&input, &folio->page, PAGE_SIZE, 0); + if (!zswap_compress(folio, entry)) + goto put_pool; - /* - * We need PAGE_SIZE * 2 here since there maybe over-compression case, - * and hardware-accelerators may won't check the dst buffer size, so - * giving the dst buffer with enough length to avoid buffer overflow. - */ - sg_init_one(&output, dst, PAGE_SIZE * 2); - acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); - /* - * it maybe looks a little bit silly that we send an asynchronous request, - * then wait for its completion synchronously. This makes the process look - * synchronous in fact. - * Theoretically, acomp supports users send multiple acomp requests in one - * acomp instance, then get those requests done simultaneously. but in this - * case, zswap actually does store and load page by page, there is no - * existing method to send the second page before the first page is done - * in one thread doing zwap. - * but in different threads running on different cpu, we have different - * acomp instance, so multiple threads can do (de)compression in parallel. - */ - ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; - - if (ret) { - zswap_reject_compress_fail++; - goto put_dstmem; - } - - /* store */ - zpool = zswap_find_zpool(entry); - gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; - if (zpool_malloc_support_movable(zpool)) - gfp |= __GFP_HIGHMEM | __GFP_MOVABLE; - ret = zpool_malloc(zpool, dlen, gfp, &handle); - if (ret == -ENOSPC) { - zswap_reject_compress_poor++; - goto put_dstmem; - } - if (ret) { - zswap_reject_alloc_fail++; - goto put_dstmem; - } - buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); - memcpy(buf, dst, dlen); - zpool_unmap_handle(zpool, handle); - mutex_unlock(&acomp_ctx->mutex); - - /* populate entry */ entry->swpentry = swp; - entry->handle = handle; - entry->length = dlen; insert_entry: entry->objcg = objcg; @@ -1663,8 +1674,6 @@ bool zswap_store(struct folio *folio) return true; -put_dstmem: - mutex_unlock(&acomp_ctx->mutex); put_pool: zswap_pool_put(entry->pool); freepage: