From patchwork Fri Oct 18 06:48:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchana P Sridhar X-Patchwork-Id: 13841261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7258D3C550 for ; Fri, 18 Oct 2024 06:48:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30AFA6B00AF; Fri, 18 Oct 2024 02:48:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BA516B00B0; Fri, 18 Oct 2024 02:48:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 098C16B00B2; Fri, 18 Oct 2024 02:48:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DC8616B00AF for ; Fri, 18 Oct 2024 02:48:13 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 05D74C01D7 for ; Fri, 18 Oct 2024 06:48:01 +0000 (UTC) X-FDA: 82685792994.17.00F47F4 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf09.hostedemail.com (Postfix) with ESMTP id B17CE14000B for ; Fri, 18 Oct 2024 06:48:03 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=bADo1Nlp; spf=pass (imf09.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729233897; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S9nsf42tROqW3D8twnvZYfm32QHvk8fwp780g5mQo+E=; b=q854Xl/kzHf6eULmcIvxomEmQO1EKa6xmUW3B+QCSREB9kc3X8gOqy+QaDtkR38zAEsN4Q cUgn8ngF0u+hgSLfNFFgng1+LcgbJV3BKRrbaE1U4Le1NhSUENDm1jAJ4w5ET8EbFLd5A3 1FtZ/4vIJhmgGsIPmOBy0iDRn6jyk8Q= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=bADo1Nlp; spf=pass (imf09.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729233897; a=rsa-sha256; cv=none; b=g/I5ZaBBnMF4VkPFrv1GCtoNxuX6qvsI8fExO2LrlM6jB591vJxfBFBApWvI6aAkKGdw1+ Vh32h08seoU6w0Jy0jzVSD9xvTI4r5ViuyCn/IS85OkyVM/4g9v4BqvRoWeWTUUFt/In0Z WKnDpeP5iPrmS/mT+haI7xbJn68auqs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1729234091; x=1760770091; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UPuvHsF/khjGuvZn4W/MjDOW3ucu5XL70qjTmnPaytQ=; b=bADo1Nlp1rd4HaFw98AJen6xbi6XE7N2lr2uUo1oUzEW+y75DvbHzNSb MrZe8n44jqaPidmGAez2fylKEEXuQrKzb0pIGfAhzeBBPAv2GWRRI2d+D +FGpRXvZD2NU6s9poLdaNiP3J6lz/GPHNrUjx+FApm4rBWVIbVteOJwqW 83l5RG4ycfgFmQmPryfSCCAfp758OyOQ7bg3rtLIq/pY+OOHvXI98swnR exn2pyHcnEQJV2I71lGMQ+aScpmVt+gP2izH2l5Mn9pshSZx/NbklD95e 1Gp0qbdoFfacks1Te2mk6xoVTPsRCRP/S0YnRN9QYrbkX96MCAza2QJgR A==; X-CSE-ConnectionGUID: HTV7zKLNRBaHYZcxRa7Xmw== X-CSE-MsgGUID: GeRE3O5pS/WCAP9URQtzYQ== X-IronPort-AV: E=McAfee;i="6700,10204,11228"; a="28963328" X-IronPort-AV: E=Sophos;i="6.11,212,1725346800"; d="scan'208";a="28963328" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2024 23:48:07 -0700 X-CSE-ConnectionGUID: NWgiSYFkS0SuqFRm5HR5lA== X-CSE-MsgGUID: UXdDj26UTrKRjCKhs7vctg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,212,1725346800"; d="scan'208";a="82744504" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by fmviesa003.fm.intel.com with ESMTP; 17 Oct 2024 23:48:06 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, hughd@google.com, willy@infradead.org, bfoster@redhat.com, dchinner@redhat.com, chrisl@kernel.org, david@redhat.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [RFC PATCH v1 2/7] mm: swap: Add IAA batch decompression API swap_crypto_acomp_decompress_batch(). Date: Thu, 17 Oct 2024 23:48:00 -0700 Message-Id: <20241018064805.336490-3-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241018064805.336490-1-kanchana.p.sridhar@intel.com> References: <20241018064805.336490-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B17CE14000B X-Stat-Signature: pe3fptzr13pr333k1tsjkzht631euaqz X-Rspam-User: X-HE-Tag: 1729234083-606090 X-HE-Meta: U2FsdGVkX18StdyqIDybPknpJloWehineDTEYa9Yf5KgzcoauHF9UPSC+8/Sf3YibKd9grv/hwTbIUXg3G19WT69C/Ec9/DJas2KfqKuLr+HCyCNEeiWTKex2ZeM0hVkD+aSyjfE+y8SWy3ExB6u+t5mhm1I+IA+JVfaV/prjiutJvoguerpfeJnmifU2ZheH8XRH44iP8Kb9L5A7G3L6yA0jRbeOrCCXw7W8AXYeShIM42XezA40CU8yiHEMq4pb/Dl9slIw+86PS2crJzU6BGnSwnlIwVoH2zCdB4p3bC/GNxiGRVExAw4S6ViW/k0hy8q8sLjtfM2pdLNkzXDcrBrYA4MUiyRil6xAOJBFw5/JW3ZUKeQdtGPDnR3zB/mPVQivK7qmqducMQlwuo7zKWQNh+Y4klS5TOcfOi0JtWXrppkCYqbuq1kYaEHqGZonD4LTdvYo8JhNJvjPdpEU1E6xM/dDrV397nI9Ah6FdXb4HnFU9pQJnDUWH/UhEl680nYXyKtnqYkUrBwNRA+Ho7uneT1hhHWfqbq/6nFgl32Bx/0uAyc95wwj6/at9t1xP/uTckuu51RA49y9UoENtLXY/+JkPcACRfqHlyAWzvRdhLlROf5Wd7uazTayzpWvievFBqZv3AdOH/BnWo9dJYyxLc1NkAFs4IHYjRpjIqMMULiW3Bd8+J7FwayPVTa2bmw8DB7n/13ipQ7qcrlf1+gtFvlhSTOFTcaU+PJ8FLo0Sq5qYqagXNZvG/QXi0aMseRAp8ALuWXNILjfQUVRdOsAFdEfihu7GBqIV5w5f1RJWJowY0WS3YZbMpAawKuSY63ty81YX2ZkN6cnPzr751ZQkzOpnCthdreCK+JBJKNKVk0kDZIrZvok/iSG+1OGBpwGmVEY6kxKqXHSwf+EmCX+WIRFxZDdjFrSdccFj0bgStCCrXAB59Wl7ZCBluPX8rv1NLEXOfzVm8ebhS /TmjXTrv K4mPfy+uOdEaTWVPvFL30J8n9nZinihk7r0xltlir+OCrIRLEARR+thUBbOVY5AR0olwq4vrmWtx24ELy4/insJ+oRf9XSqJMcvxH/jRmcknVkkD4f76tI+VgDTdscIMzWy7s8my4WK6Y8KhkDFg3FLiDb5vt7EUXr3YaIcOPiRlRr7I12lfz/ZT04MgCwyld0v5csmdZpg2AESaNUq14IJyqjtHvFMp/XDmX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Added a new API swap_crypto_acomp_decompress_batch() that does batch decompression. A system that has Intel IAA can avail of this API to submit a batch of decompress jobs for parallel decompression in the hardware, to improve performance. On a system without IAA, this API will process each decompress job sequentially. The purpose of this API is to be invocable from any swap module that needs to decompress multiple 4K folios, or a batch of pages in the general case. For instance, zswap would decompress up to (1UL << SWAP_RA_ORDER_CEILING) folios in batches of SWAP_CRYPTO_SUB_BATCH_SIZE (i.e. 8 if the system has IAA) pages prefetched by swapin_readahead(), which would improve readahead performance. Towards this eventual goal, the swap_crypto_acomp_decompress_batch() interface is implemented in swap_state.c and exported via mm/swap.h. It would be preferable for swap_crypto_acomp_decompress_batch() to be exported via include/linux/swap.h so that modules outside mm (for e.g. zram) can potentially use the API for batch decompressions with IAA, since the swapin_readahead() batching interface is common to all swap modules. I would appreciate RFC comments on this. Signed-off-by: Kanchana P Sridhar --- mm/swap.h | 42 +++++++++++++++++-- mm/swap_state.c | 109 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+), 4 deletions(-) diff --git a/mm/swap.h b/mm/swap.h index 08c04954304f..0bb386b5fdee 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -10,11 +10,12 @@ struct mempolicy; #include /* - * For IAA compression batching: - * Maximum number of IAA acomp compress requests that will be processed - * in a sub-batch. + * For IAA compression/decompression batching: + * Maximum number of IAA acomp compress/decompress requests that will be + * processed in a sub-batch. */ -#if defined(CONFIG_ZSWAP_STORE_BATCHING_ENABLED) +#if defined(CONFIG_ZSWAP_STORE_BATCHING_ENABLED) || \ + defined(CONFIG_ZSWAP_LOAD_BATCHING_ENABLED) #define SWAP_CRYPTO_SUB_BATCH_SIZE 8UL #else #define SWAP_CRYPTO_SUB_BATCH_SIZE 1UL @@ -60,6 +61,29 @@ void swap_crypto_acomp_compress_batch( int nr_pages, struct crypto_acomp_ctx *acomp_ctx); +/** + * This API provides IAA decompress batching functionality for use by swap + * modules. + * The acomp_ctx mutex should be locked/unlocked before/after calling this + * procedure. + * + * @srcs: The src buffers to be decompressed. + * @pages: The pages to store the buffers decompressed by IAA. + * @slens: src buffers' compressed lengths. + * @errors: Will contain a 0 if the page was successfully decompressed, or a + * non-0 error value to be processed by the calling function. + * @nr_pages: The number of pages, up to SWAP_CRYPTO_SUB_BATCH_SIZE, + * to be decompressed. + * @acomp_ctx: The acomp context for iaa_crypto/other compressor. + */ +void swap_crypto_acomp_decompress_batch( + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages, + struct crypto_acomp_ctx *acomp_ctx); + /* linux/mm/vmscan.c, linux/mm/page_io.c, linux/mm/zswap.c */ /* For batching of compressions in reclaim path. */ struct swap_in_memory_cache_cb { @@ -204,6 +228,16 @@ static inline void swap_write_in_memory_cache_unplug( { } +static inline void swap_crypto_acomp_decompress_batch( + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages, + struct crypto_acomp_ctx *acomp_ctx) +{ +} + static inline void swap_read_folio(struct folio *folio, struct swap_iocb **plug) { } diff --git a/mm/swap_state.c b/mm/swap_state.c index 117c3caa5679..3cebbff40804 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -855,6 +855,115 @@ void swap_crypto_acomp_compress_batch( } EXPORT_SYMBOL_GPL(swap_crypto_acomp_compress_batch); +/** + * This API provides IAA decompress batching functionality for use by swap + * modules. + * The acomp_ctx mutex should be locked/unlocked before/after calling this + * procedure. + * + * @srcs: The src buffers to be decompressed. + * @pages: The pages to store the buffers decompressed by IAA. + * @slens: src buffers' compressed lengths. + * @errors: Will contain a 0 if the page was successfully decompressed, or a + * non-0 error value to be processed by the calling function. + * @nr_pages: The number of pages, up to SWAP_CRYPTO_SUB_BATCH_SIZE, + * to be decompressed. + * @acomp_ctx: The acomp context for iaa_crypto/other compressor. + */ +void swap_crypto_acomp_decompress_batch( + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages, + struct crypto_acomp_ctx *acomp_ctx) +{ + struct scatterlist inputs[SWAP_CRYPTO_SUB_BATCH_SIZE]; + struct scatterlist outputs[SWAP_CRYPTO_SUB_BATCH_SIZE]; + unsigned int dlens[SWAP_CRYPTO_SUB_BATCH_SIZE]; + bool decompressions_done = false; + int i, j; + + BUG_ON(nr_pages > SWAP_CRYPTO_SUB_BATCH_SIZE); + + /* + * Prepare and submit acomp_reqs to IAA. + * IAA will process these decompress jobs in parallel in async mode. + * If the compressor does not support a poll() method, or if IAA is + * used in sync mode, the jobs will be processed sequentially using + * acomp_ctx->req[0] and acomp_ctx->wait. + */ + for (i = 0; i < nr_pages; ++i) { + j = acomp_ctx->acomp->poll ? i : 0; + + dlens[i] = PAGE_SIZE; + sg_init_one(&inputs[i], srcs[i], slens[i]); + sg_init_table(&outputs[i], 1); + sg_set_page(&outputs[i], pages[i], PAGE_SIZE, 0); + acomp_request_set_params(acomp_ctx->req[j], &inputs[i], + &outputs[i], slens[i], dlens[i]); + /* + * If the crypto_acomp provides an asynchronous poll() + * interface, submit the request to the driver now, and poll for + * a completion status later, after all descriptors have been + * submitted. If the crypto_acomp does not provide a poll() + * interface, submit the request and wait for it to complete, + * i.e., synchronously, before moving on to the next request. + */ + if (acomp_ctx->acomp->poll) { + errors[i] = crypto_acomp_decompress(acomp_ctx->req[j]); + + if (errors[i] != -EINPROGRESS) + errors[i] = -EINVAL; + else + errors[i] = -EAGAIN; + } else { + errors[i] = crypto_wait_req( + crypto_acomp_decompress(acomp_ctx->req[j]), + &acomp_ctx->wait); + if (!errors[i]) { + dlens[i] = acomp_ctx->req[j]->dlen; + BUG_ON(dlens[i] != PAGE_SIZE); + } + } + } + + /* + * If not doing async decompressions, the batch has been processed at + * this point and we can return. + */ + if (!acomp_ctx->acomp->poll) + return; + + /* + * Poll for and process IAA decompress job completions + * in out-of-order manner. + */ + while (!decompressions_done) { + decompressions_done = true; + + for (i = 0; i < nr_pages; ++i) { + /* + * Skip, if the decompression has already completed + * successfully or with an error. + */ + if (errors[i] != -EAGAIN) + continue; + + errors[i] = crypto_acomp_poll(acomp_ctx->req[i]); + + if (errors[i]) { + if (errors[i] == -EAGAIN) + decompressions_done = false; + } else { + dlens[i] = acomp_ctx->req[i]->dlen; + BUG_ON(dlens[i] != PAGE_SIZE); + } + } + } +} +EXPORT_SYMBOL_GPL(swap_crypto_acomp_decompress_batch); + #endif /* CONFIG_SWAP */ static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start,