From patchwork Sat Nov 23 07:01:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchana P Sridhar X-Patchwork-Id: 13883779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1ACA1E6ADE8 for ; Sat, 23 Nov 2024 07:01:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 465076B0088; Sat, 23 Nov 2024 02:01:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4160E6B0089; Sat, 23 Nov 2024 02:01:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28F9B6B008A; Sat, 23 Nov 2024 02:01:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0874C6B0088 for ; Sat, 23 Nov 2024 02:01:35 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BCBE41A1672 for ; Sat, 23 Nov 2024 07:01:34 +0000 (UTC) X-FDA: 82816464108.14.20AF86C Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.21]) by imf07.hostedemail.com (Postfix) with ESMTP id 1044F40007 for ; Sat, 23 Nov 2024 07:01:31 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="gnHF/Eqn"; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732345292; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uPB6RvclSJszS//sA27sTj58FbZmlxn8frZlRUCg9Xs=; b=049Fn8lJWoCu0OWilfEEll1XA/ymbGBMXUSwrN0HHGnMGCl8M1n6DjIvqpYu5NdUBVCD0g HwZu1GQP8kHy3jH0uy48FTm26ARclcQ55J0+K7rfpsbxEhTNaEZIF9LNTBvFfUI+8NDxeV VEOMpm7l2C08Le5BK2MwEO9F9rN5lec= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732345292; a=rsa-sha256; cv=none; b=dGs0IaPYIcctl7JVI2T9kTR67yX574Zux2MALDdpCyoJphTjBbjXJw+jDfvb22eIq1lLS+ MXQ/R6foxkmMnP97tE9OYgZvLKPGywi01S0GvXLokSpj08UKrvj8m1y8ITzSAknEHWUe8M c05mv509/o6E+pBTl6lYxrt3t7Cav1s= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="gnHF/Eqn"; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732345293; x=1763881293; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=I6blFzOYKhBMlEnizcaLhUUqkIJ1cKN/tKISn2YqJLA=; b=gnHF/EqnZPx5w5Q+0ViBiC7bfc3w6LUG6i1LDOCt5WMV+r10yJ3ipJ6p kLcKdi79XHIRpiDecqlV4sGO5p4i/6nXe+GPnLRH/c0qurWn4t8sqSNit x0r+fwBHlKPpFw/4iN1/SoEe/TDSwuxlnJ//D+iqLaCOEfsdlo4SPbxZa C0s6AR/fKJArirmipJxRHhhtX01Zf1nhAs+PLPK/9NlxqHq17kFmJHnv0 jwyNXjlyO+7fp1qveUTaRfDjw3RrhIRvaCyqtreeULY33R/13jadGjqjl v8bJ1c/WvpTOAqlEsXv6niZPuFc9XgVroIRGX9owBuRIyyV3zhmpbnk7U Q==; X-CSE-ConnectionGUID: TX9rD7MAQMCL1BuM/JekYA== X-CSE-MsgGUID: +9oUX1kCSJ6pjDpm8c4PMw== X-IronPort-AV: E=McAfee;i="6700,10204,11264"; a="32435486" X-IronPort-AV: E=Sophos;i="6.12,178,1728975600"; d="scan'208";a="32435486" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa113.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Nov 2024 23:01:29 -0800 X-CSE-ConnectionGUID: +9cFtZNpRvepDy39vtdjBA== X-CSE-MsgGUID: Ykq8GXYtQf6rbBjDUFo5yg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,178,1728975600"; d="scan'208";a="91573553" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 22 Nov 2024 23:01:28 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v4 03/10] crypto: iaa - Implement batch_compress(), batch_decompress() API in iaa_crypto. Date: Fri, 22 Nov 2024 23:01:20 -0800 Message-Id: <20241123070127.332773-4-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241123070127.332773-1-kanchana.p.sridhar@intel.com> References: <20241123070127.332773-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: t4sdnkj6amy4h8e3x8kafrj6778ew4yd X-Rspam-User: X-Rspamd-Queue-Id: 1044F40007 X-Rspamd-Server: rspam02 X-HE-Tag: 1732345291-222237 X-HE-Meta: U2FsdGVkX19JhJaS0bpb+0YXN7aXA4l6TDgW6pXnH/mc+2LbKujwNtccT8edSw1qSfqtlIqubtKXFf2if7Rioq19+XI6H9QymiSde+SAIUKemDiL7M8A0l776w1jT1kkhZyybOBGiQMDvGkVZDG6Zxg5MmojFSJGGN748IxCole2hCg3b7Tz1XHrUvwkrYLOQPkpBdHr50WOjSS+UJ50f2CkZ2vPntNeaziAw/hjvd3RfU5z8Fx+BbQVJF5ku7uEKbDDZB1KhvTGmZJsYY+Umum2jwPWXa8aVImCxVp+AvKkWYoQkHk5wqBd6Hyi/iGsjzAR00lswlB4+YqLvxGwZub1S3PbJmB9ydc7GnAd+ATRGLyMf4ZenvVo0i72TEg3JBPqDjEnj9TcXT/dLUkffPY069LIo1EE5YDNbhma4P+TUbRQucQ7HuBq4kzv6COj4XFrergD9avmL/NzquwVUp8y+9txl04uRr28R5690Vrc2KPMNZKUHNETKkIO6Y4YB9RpZ5B2X2SgYw35R81urRLKW8sA5x29ckjbFIrlEkOuguGfnzwwS0DzZOg58PEjQHgGj04tkBi/f/wuWU5ryHHlHcQRHqXyGeLF1UBRYu00MmFOvNw7hN8tyaHX6xQ9siiK0bapLAu8IQsdeD2uUH1TmBhkGJpD+mOoV5hkJxOLo7sjV97xjGVp0wtPksd+WEKNdLZGBoGr1wryArTfkXcrH1n4lTn1ZAaDTEFv1a8vGdcXP35LPO4PlIk8jvbIKSIuNsGDUwcwpEyCKu3bKNwLMXyQVs7PfsXFjJq7ic5duwZKf4fHaekHRQGrrP1QzYUIDprg8jAUu+GoFFfPPtg94gxdFiAJoUBXZ55iJJNdkHJ8p+7uvieBq7nzqeWUrSfEdw9rpxP8eoYsC5yqJMXKMg19nfO2myb1G2+QnLq+jdUvmAIxFQOl//gI90/DZXucsZVQ3delWsZJqii nSovgDmQ qjc6Hmre3fu29yTZ+SAbL2kDT9UHuUWdHyvQoJQ6L1icQky/S5sEToVQVweiEyxWFR0NfgtGQ2tYdm4yNKisgKUpoMvNj93p30yXxdQqkrIGoOMLJGaJlvBnsQnWShvzUkt70dHI3W4VEX09O/0wYnwdtN1ey9yJRBHd96yl7QdBYZhGPf0MWVWY4k7whGD2KSxboew4G6IO8GHGo7p6HFXhQUu9ijpITOJOtBtXJdbYRalunQOqifXzafhw4QvBmN4ogZAZwmPLGxAZy5UY88/UmeeXKPTYrZYu9DTHjq1guoF86cMkQXY0nZA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch provides iaa_crypto driver implementations for the newly added crypto_acomp batch_compress() and batch_decompress() interfaces. This allows swap modules such as zswap/zram to invoke batch parallel compression/decompression of pages on systems with Intel IAA, by invoking these API, respectively: crypto_acomp_batch_compress(...); crypto_acomp_batch_decompress(...); This enables zswap_batch_store() compress batching code to be developed in a manner similar to the current single-page synchronous calls to: crypto_acomp_compress(...); crypto_acomp_decompress(...); thereby, facilitating encapsulated and modular hand-off between the kernel zswap/zram code and the crypto_acomp layer. Suggested-by: Yosry Ahmed Suggested-by: Herbert Xu Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 337 +++++++++++++++++++++ 1 file changed, 337 insertions(+) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 2edaecd42cc6..cbf147a3c3cb 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1797,6 +1797,341 @@ static void compression_ctx_init(struct iaa_compression_ctx *ctx) ctx->use_irq = use_irq; } +static int iaa_comp_poll(struct acomp_req *req) +{ + struct idxd_desc *idxd_desc; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + struct idxd_wq *wq; + bool compress_op; + int ret; + + idxd_desc = req->base.data; + if (!idxd_desc) + return -EAGAIN; + + compress_op = (idxd_desc->iax_hw->opcode == IAX_OPCODE_COMPRESS); + wq = idxd_desc->wq; + iaa_wq = idxd_wq_get_private(wq); + idxd = iaa_wq->iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + ret = check_completion(dev, idxd_desc->iax_completion, true, true); + if (ret == -EAGAIN) + return ret; + if (ret) + goto out; + + req->dlen = idxd_desc->iax_completion->output_size; + + /* Update stats */ + if (compress_op) { + update_total_comp_bytes_out(req->dlen); + update_wq_comp_bytes(wq, req->dlen); + } else { + update_total_decomp_bytes_in(req->slen); + update_wq_decomp_bytes(wq, req->slen); + } + + if (iaa_verify_compress && (idxd_desc->iax_hw->opcode == IAX_OPCODE_COMPRESS)) { + struct crypto_tfm *tfm = req->base.tfm; + dma_addr_t src_addr, dst_addr; + u32 compression_crc; + + compression_crc = idxd_desc->iax_completion->crc; + + dma_sync_sg_for_device(dev, req->dst, 1, DMA_FROM_DEVICE); + dma_sync_sg_for_device(dev, req->src, 1, DMA_TO_DEVICE); + + src_addr = sg_dma_address(req->src); + dst_addr = sg_dma_address(req->dst); + + ret = iaa_compress_verify(tfm, req, wq, src_addr, req->slen, + dst_addr, &req->dlen, compression_crc); + } +out: + /* caller doesn't call crypto_wait_req, so no acomp_request_complete() */ + + dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); + dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); + + idxd_free_desc(idxd_desc->wq, idxd_desc); + + dev_dbg(dev, "%s: returning ret=%d\n", __func__, ret); + + return ret; +} + +static void iaa_set_req_poll( + struct acomp_req *reqs[], + int nr_reqs, + bool set_flag) +{ + int i; + + for (i = 0; i < nr_reqs; ++i) { + set_flag ? (reqs[i]->flags |= CRYPTO_ACOMP_REQ_POLL) : + (reqs[i]->flags &= ~CRYPTO_ACOMP_REQ_POLL); + } +} + +/** + * This API provides IAA compress batching functionality for use by swap + * modules. + * + * @reqs: @nr_pages asynchronous compress requests. + * @wait: crypto_wait for synchronous acomp batch compress. If NULL, the + * completions will be processed asynchronously. + * @pages: Pages to be compressed by IAA in parallel. + * @dsts: Pre-allocated destination buffers to store results of IAA + * compression. Each element of @dsts must be of size "PAGE_SIZE * 2". + * @dlens: Will contain the compressed lengths. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_pages: The number of pages, up to CRYPTO_BATCH_SIZE, + * to be compressed. + */ +static void iaa_comp_acompress_batch( + struct acomp_req *reqs[], + struct crypto_wait *wait, + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages) +{ + struct scatterlist inputs[CRYPTO_BATCH_SIZE]; + struct scatterlist outputs[CRYPTO_BATCH_SIZE]; + bool compressions_done = false; + bool poll = (async_mode && !use_irq); + int i; + + BUG_ON(nr_pages > CRYPTO_BATCH_SIZE); + BUG_ON(!poll && !wait); + + if (poll) + iaa_set_req_poll(reqs, nr_pages, true); + else + iaa_set_req_poll(reqs, nr_pages, false); + + /* + * Prepare and submit acomp_reqs to IAA. IAA will process these + * compress jobs in parallel if async-poll mode is enabled. + * If IAA is used in sync mode, the jobs will be processed sequentially + * using "wait". + */ + for (i = 0; i < nr_pages; ++i) { + sg_init_table(&inputs[i], 1); + sg_set_page(&inputs[i], pages[i], PAGE_SIZE, 0); + + /* + * Each dst buffer should be of size (PAGE_SIZE * 2). + * Reflect same in sg_list. + */ + sg_init_one(&outputs[i], dsts[i], PAGE_SIZE * 2); + acomp_request_set_params(reqs[i], &inputs[i], + &outputs[i], PAGE_SIZE, dlens[i]); + + /* + * If poll is in effect, submit the request now, and poll for + * a completion status later, after all descriptors have been + * submitted. If polling is not enabled, submit the request + * and wait for it to complete, i.e., synchronously, before + * moving on to the next request. + */ + if (poll) { + errors[i] = iaa_comp_acompress(reqs[i]); + + if (errors[i] != -EINPROGRESS) + errors[i] = -EINVAL; + else + errors[i] = -EAGAIN; + } else { + acomp_request_set_callback(reqs[i], + CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, wait); + errors[i] = crypto_wait_req(iaa_comp_acompress(reqs[i]), + wait); + if (!errors[i]) + dlens[i] = reqs[i]->dlen; + } + } + + /* + * If not doing async compressions, the batch has been processed at + * this point and we can return. + */ + if (!poll) + goto reset_reqs_wait; + + /* + * Poll for and process IAA compress job completions + * in out-of-order manner. + */ + while (!compressions_done) { + compressions_done = true; + + for (i = 0; i < nr_pages; ++i) { + /* + * Skip, if the compression has already completed + * successfully or with an error. + */ + if (errors[i] != -EAGAIN) + continue; + + errors[i] = iaa_comp_poll(reqs[i]); + + if (errors[i]) { + if (errors[i] == -EAGAIN) + compressions_done = false; + } else { + dlens[i] = reqs[i]->dlen; + } + } + } + +reset_reqs_wait: + /* + * For the same 'reqs[]' and 'wait' to be usable by + * iaa_comp_acompress()/iaa_comp_deacompress(): + * Clear the CRYPTO_ACOMP_REQ_POLL bit on the acomp_reqs. + * Reset the crypto_wait "wait" callback to reqs[0]. + */ + iaa_set_req_poll(reqs, nr_pages, false); + acomp_request_set_callback(reqs[0], + CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, wait); +} + +/** + * This API provides IAA decompress batching functionality for use by swap + * modules. + * + * @reqs: @nr_pages asynchronous decompress requests. + * @wait: crypto_wait for synchronous acomp batch decompress. If NULL, the + * driver must provide a way to process completions asynchronously. + * @srcs: The src buffers to be decompressed by IAA in parallel. + * @pages: The pages to store the decompressed buffers. + * @slens: Compressed lengths of @srcs. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_pages: The number of pages, up to CRYPTO_BATCH_SIZE, + * to be decompressed. + */ +static void iaa_comp_adecompress_batch( + struct acomp_req *reqs[], + struct crypto_wait *wait, + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages) +{ + struct scatterlist inputs[CRYPTO_BATCH_SIZE]; + struct scatterlist outputs[CRYPTO_BATCH_SIZE]; + unsigned int dlens[CRYPTO_BATCH_SIZE]; + bool decompressions_done = false; + bool poll = (async_mode && !use_irq); + int i; + + BUG_ON(nr_pages > CRYPTO_BATCH_SIZE); + BUG_ON(!poll && !wait); + + if (poll) + iaa_set_req_poll(reqs, nr_pages, true); + else + iaa_set_req_poll(reqs, nr_pages, false); + + /* + * Prepare and submit acomp_reqs to IAA. IAA will process these + * decompress jobs in parallel if async-poll mode is enabled. + * If IAA is used in sync mode, the jobs will be processed sequentially + * using "wait". + */ + for (i = 0; i < nr_pages; ++i) { + dlens[i] = PAGE_SIZE; + sg_init_one(&inputs[i], srcs[i], slens[i]); + sg_init_table(&outputs[i], 1); + sg_set_page(&outputs[i], pages[i], PAGE_SIZE, 0); + acomp_request_set_params(reqs[i], &inputs[i], + &outputs[i], slens[i], dlens[i]); + /* + * If poll is in effect, submit the request now, and poll for + * a completion status later, after all descriptors have been + * submitted. If polling is not enabled, submit the request + * and wait for it to complete, i.e., synchronously, before + * moving on to the next request. + */ + if (poll) { + errors[i] = iaa_comp_adecompress(reqs[i]); + + if (errors[i] != -EINPROGRESS) + errors[i] = -EINVAL; + else + errors[i] = -EAGAIN; + } else { + acomp_request_set_callback(reqs[i], + CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, wait); + errors[i] = crypto_wait_req(iaa_comp_adecompress(reqs[i]), + wait); + if (!errors[i]) { + dlens[i] = reqs[i]->dlen; + BUG_ON(dlens[i] != PAGE_SIZE); + } + } + } + + /* + * If not doing async decompressions, the batch has been processed at + * this point and we can return. + */ + if (!poll) + goto reset_reqs_wait; + + /* + * Poll for and process IAA decompress job completions + * in out-of-order manner. + */ + while (!decompressions_done) { + decompressions_done = true; + + for (i = 0; i < nr_pages; ++i) { + /* + * Skip, if the decompression has already completed + * successfully or with an error. + */ + if (errors[i] != -EAGAIN) + continue; + + errors[i] = iaa_comp_poll(reqs[i]); + + if (errors[i]) { + if (errors[i] == -EAGAIN) + decompressions_done = false; + } else { + dlens[i] = reqs[i]->dlen; + BUG_ON(dlens[i] != PAGE_SIZE); + } + } + } + +reset_reqs_wait: + /* + * For the same 'reqs[]' and 'wait' to be usable by + * iaa_comp_acompress()/iaa_comp_deacompress(): + * Clear the CRYPTO_ACOMP_REQ_POLL bit on the acomp_reqs. + * Reset the crypto_wait "wait" callback to reqs[0]. + */ + iaa_set_req_poll(reqs, nr_pages, false); + acomp_request_set_callback(reqs[0], + CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, wait); +} + static int iaa_comp_init_fixed(struct crypto_acomp *acomp_tfm) { struct crypto_tfm *tfm = crypto_acomp_tfm(acomp_tfm); @@ -1822,6 +2157,8 @@ static struct acomp_alg iaa_acomp_fixed_deflate = { .compress = iaa_comp_acompress, .decompress = iaa_comp_adecompress, .dst_free = dst_free, + .batch_compress = iaa_comp_acompress_batch, + .batch_decompress = iaa_comp_adecompress_batch, .base = { .cra_name = "deflate", .cra_driver_name = "deflate-iaa",