From patchwork Sat Dec 21 06:31:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchana P Sridhar X-Patchwork-Id: 13917687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90E69E7718B for ; Sat, 21 Dec 2024 06:32:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 20C6D6B009C; Sat, 21 Dec 2024 01:31:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 194706B009D; Sat, 21 Dec 2024 01:31:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00EEA6B009E; Sat, 21 Dec 2024 01:31:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D47366B009C for ; Sat, 21 Dec 2024 01:31:32 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8A17314065A for ; Sat, 21 Dec 2024 06:31:32 +0000 (UTC) X-FDA: 82917994152.04.AB679B2 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) by imf24.hostedemail.com (Postfix) with ESMTP id DB47D18000A for ; Sat, 21 Dec 2024 06:31:25 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=YKOdtHZR; spf=pass (imf24.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.20 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734762663; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ty/U/e5wepM88PXPQeh90lRr6csMKg67G2D7fTLt4jI=; b=4eMhzEHyLX0NgXzOWR2vpUXBHK9Izn8NVCdESm4X1e3rF2cyabjNUPy1AhkD++RKJzi1mm RUPVhPrWSp177KcZCwmbjEeSRbnOAOuN0DeMaVwmFIHdgctX95M1SPYy1oMcRNN+6R1t3I j2ULHtNFeUTeVj7BIC/nR7Rk9ETs4Gg= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=YKOdtHZR; spf=pass (imf24.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.20 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734762663; a=rsa-sha256; cv=none; b=Li/XMcUNt8CKzLci0z1uA5ewYTkX614dtQHViBji2Ot7uc5hGlkghH14TekdOZRoMD7f90 tIA+osULw1ecxUfZAFqh3PkxeJVPYLiYlmvow6CJDLgRN77VssaGiOnwHRh65sWyRMQh2F JqxfRbtwTMPoIfW166HvodPXTX4nicY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1734762690; x=1766298690; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OMiXITVf1zg7CemO/j63XZqQRFVNfoKOofrwr8FdVdM=; b=YKOdtHZRhsbW0msLWY6OzlVf3hSiDtLTBKR7X6dMdQe0l0JYbulxsFs7 E1SrS/S48xy+leEPPKaG+F6BUzezt5Z+y2blAff6gg/nzdlRGbrKrVqWo mdQkXvMWPtRukXAE0fr8JcrLvITW/DIFTofV3SmZb858PjrdwOnMi6xDP pVMD2eySylcUU1fCSmtKtj8dVuIaRffLY5W0FXmptADs9NJFvufm/1zEY +JVw+WzSvoOfa4f5Bw60bC5SIDWarRMMKbSKFjG1T45jzAO5Vt1/jQ0WF abb/pwdH3t258Ast83LoeDGKbooDbmmyQB+DLkpNO/EI0JWExlZq/bnmf g==; X-CSE-ConnectionGUID: e09AlO8PT1KTOP0lEa2A5Q== X-CSE-MsgGUID: waF1gXZ8RQeM/fFu6D4JrA== X-IronPort-AV: E=McAfee;i="6700,10204,11292"; a="35021746" X-IronPort-AV: E=Sophos;i="6.12,253,1728975600"; d="scan'208";a="35021746" Received: from orviesa007.jf.intel.com ([10.64.159.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Dec 2024 22:31:21 -0800 X-CSE-ConnectionGUID: 0Yaxj/qcRrSccQi1sf6c2A== X-CSE-MsgGUID: 1hw2SbHQRge/3biOVZBL3w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="99184613" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa007.jf.intel.com with ESMTP; 20 Dec 2024 22:31:21 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v5 12/12] mm: zswap: Compress batching with Intel IAA in zswap_store() of large folios. Date: Fri, 20 Dec 2024 22:31:19 -0800 Message-Id: <20241221063119.29140-13-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241221063119.29140-1-kanchana.p.sridhar@intel.com> References: <20241221063119.29140-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: DB47D18000A X-Rspamd-Server: rspam12 X-Stat-Signature: k57ryusshi96yozoowwqddb9pazjch64 X-Rspam-User: X-HE-Tag: 1734762685-930670 X-HE-Meta: U2FsdGVkX1/w7u9ZzmjyQpe77qgvXPq61SMR2SLPPIeLQ5jGB8N+XhgAyRgWXUJrPvo+Fwcu0aJT2tljPDyJk4fBeanrWnKAmArtX0CBvlHQYXcO3xdoXpidmsYqbi/HXEv9NB1EvmZetufB84qwR5ndnQuOtCw1qWV9tFNYfyuvqpxyh9MvlLqq3NG0wuA89nJEbOKooPRWn91ue7xttajy/NF3+cPDrzDZmtv7PTCQXmgFe2t8GMLcmFjTwHspLsLpGWgzoEgLhj5UuAnzqv0CW6DkRQkqXsqNkbuyPIu2RhQrj8hTAK/UdmwhRtF8/QtoRzF1f+0UcPbORETam5WcjnM13spjOdgKzrc8B1hlRjH4ZMF5r+l2oW7peqAPVO942eGotej7gwjMQXgms44ZAN1IWZ9TSf1+amZoY1/w9BusrkloOJMO7kXXgq50XbvVYgDTlDerzkAf9VPrhMaMIf1k7t+3O8e03B54UUG3qdWCoO/41QIKUbvR2ilIZ5LygHdXvo2Ji7sEBfzmj/xihegr0TpVd+ZbnU8Svuz3FRYW3vdPfRVVRpWE+B1KxuVc04X5wktBF/0rGupvkRzBGd5I3g75vqyAlYa4otQYnhsI4QubYysvAGfXh+WZNEXYjelMdOBL9uLefM2Y8TKfzynfv56YSiiIYI0qeP2hT+WSc5GahUGSnLVLddPr10tp2KUbfLt8wuQD2LfvAiE5gWxCbem/tEVzuPjwx6y0gqYy1MPSlPX9vt1gONb9u35K7dWAkyCssQAQAe1iee9IgP1wI64+d0Nuzs8cvFiic00xOYDld66yH0W29u9kqrRoEEbu9m5MJmuyDQSyAZGv80VG4yf5aPj7C0D9TRaoqThg44ShJoSkRgovSRIKyf+MIj5jQ+T03aA8P8LdW3EcehaNH72kTfO/UCIXIiOgNQNmCSLuQb0mXAq9MfeD/n6LiJ9FaM5GpA0nN6q F+KbB4+k 3YLmy+BWUvIE9GmtL7b76Tx2Ebs9X+IUUMs7bzdQq5A6jmjPOXdyUd373Y+7BUe1fA/1MYKVmobZIvSbIttmI/4wCCi3os0PVurapRtGBJ4XhOIOX05bZP/uT2EZEQkGN5IggNx6GRTOtj06f+tlp1olVyZ1ndbCuGLVgnIyxB5X4sWA0QsDR1UaROA+7lzO4Rcc4MetDB2Cu+0vtiVG6r3JPSh1t1obk0cpC86ry+qZkEUs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: zswap_compress_folio() is modified to detect if the pool's acomp_ctx has more than one "nr_reqs", which will be the case if the cpu onlining code has allocated batching resources in the acomp_ctx based on the queries to acomp_has_async_batching() and crypto_acomp_batch_size(). If multiple "nr_reqs" are available in the acomp_ctx, it means compress batching can be used with a batch-size of "acomp_ctx->nr_reqs". If compress batching can be used with the given zswap pool, zswap_compress_folio() will invoke the newly added zswap_batch_compress() procedure to compress and store the folio in batches of "acomp_ctx->nr_reqs" pages. The batch size is effectively "acomp_ctx->nr_reqs". zswap_batch_compress() calls crypto_acomp_batch_compress() to compress each batch of (up to) "acomp_ctx->nr_reqs" pages. The iaa_crypto driver will compress each batch of pages in parallel in the Intel IAA hardware with 'async' mode and request chaining. Hence, zswap_batch_compress() does the same computes for a batch, as zswap_compress() does for a page; and returns true if the batch was successfully compressed/stored, and false otherwise. If the pool does not support compress batching, zswap_compress_folio() calls zswap_compress() for each individual page in the folio, as before. Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 105 insertions(+), 4 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 1be0f1807bfc..f336fafe24c4 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1467,17 +1467,118 @@ static void shrink_worker(struct work_struct *w) * main API **********************************/ +static bool zswap_batch_compress(struct folio *folio, + long index, + unsigned int batch_size, + struct zswap_entry *entries[], + struct zswap_pool *pool, + struct crypto_acomp_ctx *acomp_ctx) +{ + int comp_errors[ZSWAP_MAX_BATCH_SIZE] = { 0 }; + unsigned int dlens[ZSWAP_MAX_BATCH_SIZE]; + struct page *pages[ZSWAP_MAX_BATCH_SIZE]; + unsigned int i, nr_batch_pages; + bool ret = true; + + nr_batch_pages = min((unsigned int)(folio_nr_pages(folio) - index), batch_size); + + for (i = 0; i < nr_batch_pages; ++i) { + pages[i] = folio_page(folio, index + i); + dlens[i] = PAGE_SIZE; + } + + mutex_lock(&acomp_ctx->mutex); + + /* + * Batch compress @nr_batch_pages. If IAA is the compressor, the + * hardware will compress @nr_batch_pages in parallel. + */ + ret = crypto_acomp_batch_compress( + acomp_ctx->reqs, + &acomp_ctx->wait, + pages, + acomp_ctx->buffers, + dlens, + comp_errors, + nr_batch_pages); + + if (ret) { + /* + * All batch pages were successfully compressed. + * Store the pages in zpool. + */ + struct zpool *zpool = pool->zpool; + gfp_t gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; + + if (zpool_malloc_support_movable(zpool)) + gfp |= __GFP_HIGHMEM | __GFP_MOVABLE; + + for (i = 0; i < nr_batch_pages; ++i) { + unsigned long handle; + char *buf; + int err; + + err = zpool_malloc(zpool, dlens[i], gfp, &handle); + + if (err) { + if (err == -ENOSPC) + zswap_reject_compress_poor++; + else + zswap_reject_alloc_fail++; + + ret = false; + break; + } + + buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); + memcpy(buf, acomp_ctx->buffers[i], dlens[i]); + zpool_unmap_handle(zpool, handle); + + entries[i]->handle = handle; + entries[i]->length = dlens[i]; + } + } else { + /* Some batch pages had compression errors. */ + for (i = 0; i < nr_batch_pages; ++i) { + if (comp_errors[i]) { + if (comp_errors[i] == -ENOSPC) + zswap_reject_compress_poor++; + else + zswap_reject_compress_fail++; + } + } + } + + mutex_unlock(&acomp_ctx->mutex); + + return ret; +} + static bool zswap_compress_folio(struct folio *folio, struct zswap_entry *entries[], struct zswap_pool *pool) { long index, nr_pages = folio_nr_pages(folio); + struct crypto_acomp_ctx *acomp_ctx; + unsigned int batch_size; - for (index = 0; index < nr_pages; ++index) { - struct page *page = folio_page(folio, index); + acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); + batch_size = acomp_ctx->nr_reqs; - if (!zswap_compress(page, entries[index], pool)) - return false; + if ((batch_size > 1) && (nr_pages > 1)) { + for (index = 0; index < nr_pages; index += batch_size) { + + if (!zswap_batch_compress(folio, index, batch_size, + &entries[index], pool, acomp_ctx)) + return false; + } + } else { + for (index = 0; index < nr_pages; ++index) { + struct page *page = folio_page(folio, index); + + if (!zswap_compress(page, entries[index], pool)) + return false; + } } return true;