From patchwork Sun Nov 3 03:20:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860286 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62E1CE67819 for ; Sun, 3 Nov 2024 03:21:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FFFC6B007B; Sat, 2 Nov 2024 23:21:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D5BF6B0085; Sat, 2 Nov 2024 23:21:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89EAC6B0088; Sat, 2 Nov 2024 23:21:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 587806B007B for ; Sat, 2 Nov 2024 23:21:18 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E9BBC1A09AD for ; Sun, 3 Nov 2024 03:21:17 +0000 (UTC) X-FDA: 82743332070.10.D17BC08 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf18.hostedemail.com (Postfix) with ESMTP id 721431C0002 for ; Sun, 3 Nov 2024 03:21:02 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=W2DdAxRZ; spf=pass (imf18.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730604029; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eAOM1k5eBgatCXxcKi5hWs7e/Nc/Gb2n04WWWLD5dIE=; b=QfEvu0OuKqI0fc2f2jgyFmIHULCGOFJnQexHlhHArm96wf1pbXV6OCXvUs1WQhdFJ85a3O xVHJiRSbdyZMXCLWJw/hFptqdZHlDfhIWIVb2+Qh4UeMz6/odASJ+TriFbmiJ82JOrMhcx 7Tjpm4LtSPoqGIXrPw7xpg81Wip60+U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730604029; a=rsa-sha256; cv=none; b=jKR6b0318a17itEGDq6jF0JAD1DinJuCJ0XJtc7orLVdEfwpN1VDq8JMMDHrnx9ci1IPVd GKOxx6d0Jl99o6GiGPTE5GKP3Iyb2bbfkFZZomXl8eRTPOSjsXsMlV4V8YQ86a5DrXor8R /7HQWEZ7WpMFCwjzv3My/PsZ7c0crjE= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=W2DdAxRZ; spf=pass (imf18.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604075; x=1762140075; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8jyV13JKHn0vIl+wBpuhM8EZ+NN4RHEftxJMCaIna00=; b=W2DdAxRZWCYB2LtrLAqHou9u3uhJgSVSzesKcDWLgUPG9kwpATda2ZJH YAgSlmvPNoMdi/IUVJU7Sd8JEFTlqivqVAOf90nnQRnvDkb2RDYErcOxV jpKNL3HpFqwxXJD39yuI2tAy9snNmUbkErvfCvu9l7fW/TrWZiQPid0Oj Fe8+zY9J+hVOu6mhOkuMQ/Eo4qXqaU+92iqrwFVJDT3RW/RXLnQ3pVGoX j0O0Ir7DogCISBNCrErK9X/xCrcfAlMJP9zEk82hV8yB2dpc3oCNX6Y7G fiEfWri7RqCakxGq2+nxsuUr2Ehx08OanV2ef+S8gfShlnbfITnQhzW6X Q==; X-CSE-ConnectionGUID: V5DP5g7mRK2Y5L3exyL9zg== X-CSE-MsgGUID: O4rKYjBWSsGOLmdsdZusPg== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548220" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548220" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:12 -0700 X-CSE-ConnectionGUID: 3SSZ6/PoSPeJNuzxqB1iwA== X-CSE-MsgGUID: xv6vpDvIR3Ssdv3gkPqx7g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451868" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:13 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 01/13] crypto: acomp - Define two new interfaces for compress/decompress batching. Date: Sat, 2 Nov 2024 20:20:59 -0700 Message-Id: <20241103032111.333282-2-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: mtcwaptynkgg71efr149nqh1iwqcpcwx X-Rspam-User: X-Rspamd-Queue-Id: 721431C0002 X-Rspamd-Server: rspam02 X-HE-Tag: 1730604062-874492 X-HE-Meta: U2FsdGVkX189wBsX0hSpSWGc9KSmFVi9Bwo2rjQDpJMid5Kg8qiEGsO/VsH0D3+BSjYBDiaktLLFJhAspPhodx3LlhdseNPCgabI9DKzrtepPywfoHTxQhmGaW29OIXLZFzbOvCI0xtfq1e2uZs74rhnPziqiCDemysGIbdvZugbbhdnTKE5REPNKzCHxFpZyf35uImF6hRxgmX5wSK/2+Ds+0fkDWwOIkpe1s6S8S6SNS7hrFiN5SJxRhaDhpj32iCq3iatOu0o2Q+Y1HSV/+mFZUsdmm0Z7O14sRTw/jIU47JCG27VBomfMlbacKlxuxdRnyJVWsmc3UYsiAhxdl38x1wTHVUlveA8EuPpr/N96rRbc68DbT7zentLeSqr5nkF3JQXii0VYTrkFO9C3FB3ADo+Akz6ZA8wgOI5AiPwFZ3DG3oDTU+oUAAdLKD5SEkupIQnoZHbwTxNt8yFc7IH0kbIjkfK3uNeH8UMOR1WGfxRHeeftPosIg4lAi1sdTdovMziUu/rhGTipZvGcK9hggDAxhkBqr/A6TfgLlBY6xptW5ER0Uk+uLeD4XIOFa556ijIHKyFSnCiW5UuH7/HrcBThpprDbGnZSNvPhSMR5wbXfVcp+VWdiZf2T9UTdU8CNAjaCg2HV02Cej/+XlBXXsyL9NP520BUP+ENOXXdOsttFmJStdWCBAw4LTYDkogFuQr+fGjs8L2QImC9bwM+3We754xkAV25QcxsHJGgnyTvnmCKZIhZ319TkMraowhL1WlAkiAHm7yRyp/vJVVcOOuuGEBNgtRZTdAId+7Ia4Ec+8TsCcDZ6+hZpuI8qVP0xI/TV41DADVypbZS1XFsvtP+3D7889GIGCV0oO2bOjxOepsROKWl8HT/kvhy1stFmnbTwVw55D4eIqTb3TsY2N3bQjCvDtB/9AiiCJO21SU7Maf1wrQnV8W3xnVg58SeHZmahH3cx0iuFj 4+G3TNES n5FzSfLSkKa92mod3sPdY9dDKKh+PkBpSqzU3QWmrigCNp69y0Z3VjQSPWt/xMkQGpDKETNNWvJzD0MsoV1VHB/pmc2eGODcNKkkjBb6gNjd4M8mT1fS2C6Ff+GxoZ53pE6aivnPYvqEFjVToUEw198lUf5z3rFfxZqBE0CJmL1itwFrh78JMkIezUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This commit adds batch_compress() and batch_decompress() interfaces to: struct acomp_alg struct crypto_acomp This allows the iaa_crypto Intel IAA driver to register implementation for batch_compress() and batch_decompress() API, that can subsequently be invoked from the kernel zswap/zram swap modules to compress/decompress up to CRYPTO_BATCH_SIZE (i.e. 8) pages in parallel in the IAA hardware accelerator to improve swapout/swapin performance. Signed-off-by: Kanchana P Sridhar --- crypto/acompress.c | 2 + include/crypto/acompress.h | 82 +++++++++++++++++++++++++++++ include/crypto/internal/acompress.h | 16 ++++++ 3 files changed, 100 insertions(+) diff --git a/crypto/acompress.c b/crypto/acompress.c index 6fdf0ff9f3c0..a506db499a37 100644 --- a/crypto/acompress.c +++ b/crypto/acompress.c @@ -71,6 +71,8 @@ static int crypto_acomp_init_tfm(struct crypto_tfm *tfm) acomp->compress = alg->compress; acomp->decompress = alg->decompress; + acomp->batch_compress = alg->batch_compress; + acomp->batch_decompress = alg->batch_decompress; acomp->dst_free = alg->dst_free; acomp->reqsize = alg->reqsize; diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h index 54937b615239..ab0d9987bde1 100644 --- a/include/crypto/acompress.h +++ b/include/crypto/acompress.h @@ -37,12 +37,20 @@ struct acomp_req { void *__ctx[] CRYPTO_MINALIGN_ATTR; }; +/* + * The max compress/decompress batch size, for crypto algorithms + * that support batch_compress and batch_decompress API. + */ +#define CRYPTO_BATCH_SIZE 8UL + /** * struct crypto_acomp - user-instantiated objects which encapsulate * algorithms and core processing logic * * @compress: Function performs a compress operation * @decompress: Function performs a de-compress operation + * @batch_compress: Function performs a batch compress operation + * @batch_decompress: Function performs a batch decompress operation * @dst_free: Frees destination buffer if allocated inside the * algorithm * @reqsize: Context size for (de)compression requests @@ -51,6 +59,20 @@ struct acomp_req { struct crypto_acomp { int (*compress)(struct acomp_req *req); int (*decompress)(struct acomp_req *req); + void (*batch_compress)(struct acomp_req *reqs[], + struct crypto_wait *wait, + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages); + void (*batch_decompress)(struct acomp_req *reqs[], + struct crypto_wait *wait, + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages); void (*dst_free)(struct scatterlist *dst); unsigned int reqsize; struct crypto_tfm base; @@ -265,4 +287,64 @@ static inline int crypto_acomp_decompress(struct acomp_req *req) return crypto_acomp_reqtfm(req)->decompress(req); } +/** + * crypto_acomp_batch_compress() -- compress a batch of requests + * + * Function invokes the batch compress operation + * + * @reqs: @nr_pages asynchronous compress requests. + * @wait: crypto_wait for synchronous acomp batch compress. If NULL, the + * driver must provide a way to process completions asynchronously. + * @pages: Pages to be compressed. + * @dsts: Pre-allocated destination buffers to store results of compression. + * @dlens: Will contain the compressed lengths. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_pages: The number of pages, up to CRYPTO_BATCH_SIZE, + * to be compressed. + */ +static inline void crypto_acomp_batch_compress(struct acomp_req *reqs[], + struct crypto_wait *wait, + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages) +{ + struct crypto_acomp *tfm = crypto_acomp_reqtfm(reqs[0]); + + return tfm->batch_compress(reqs, wait, pages, dsts, + dlens, errors, nr_pages); +} + +/** + * crypto_acomp_batch_decompress() -- decompress a batch of requests + * + * Function invokes the batch decompress operation + * + * @reqs: @nr_pages asynchronous decompress requests. + * @wait: crypto_wait for synchronous acomp batch decompress. If NULL, the + * driver must provide a way to process completions asynchronously. + * @srcs: The src buffers to be decompressed. + * @pages: The pages to store the decompressed buffers. + * @slens: Compressed lengths of @srcs. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_pages: The number of pages, up to CRYPTO_BATCH_SIZE, + * to be decompressed. + */ +static inline void crypto_acomp_batch_decompress(struct acomp_req *reqs[], + struct crypto_wait *wait, + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages) +{ + struct crypto_acomp *tfm = crypto_acomp_reqtfm(reqs[0]); + + return tfm->batch_decompress(reqs, wait, srcs, pages, + slens, errors, nr_pages); +} + #endif diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h index 8831edaafc05..acfe2d9d5a83 100644 --- a/include/crypto/internal/acompress.h +++ b/include/crypto/internal/acompress.h @@ -17,6 +17,8 @@ * * @compress: Function performs a compress operation * @decompress: Function performs a de-compress operation + * @batch_compress: Function performs a batch compress operation + * @batch_decompress: Function performs a batch decompress operation * @dst_free: Frees destination buffer if allocated inside the algorithm * @init: Initialize the cryptographic transformation object. * This function is used to initialize the cryptographic @@ -37,6 +39,20 @@ struct acomp_alg { int (*compress)(struct acomp_req *req); int (*decompress)(struct acomp_req *req); + void (*batch_compress)(struct acomp_req *reqs[], + struct crypto_wait *wait, + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages); + void (*batch_decompress)(struct acomp_req *reqs[], + struct crypto_wait *wait, + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages); void (*dst_free)(struct scatterlist *dst); int (*init)(struct crypto_acomp *tfm); void (*exit)(struct crypto_acomp *tfm); From patchwork Sun Nov 3 03:21:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A2DBE6780A for ; Sun, 3 Nov 2024 03:21:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 322566B0085; Sat, 2 Nov 2024 23:21:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F8D16B0088; Sat, 2 Nov 2024 23:21:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 199C16B0089; Sat, 2 Nov 2024 23:21:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EE4136B0085 for ; Sat, 2 Nov 2024 23:21:18 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A224B1C797C for ; Sun, 3 Nov 2024 03:21:18 +0000 (UTC) X-FDA: 82743331902.25.6C55F9E Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf12.hostedemail.com (Postfix) with ESMTP id AE52740005 for ; Sun, 3 Nov 2024 03:21:02 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=j7cJVV08; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf12.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730603955; a=rsa-sha256; cv=none; b=O8cswYPskrIG0KM/Gjce8nmpcmSaEB6OJYJa4lt/L6ftc4vL4ia19gyn+PZphGwMt6INHv jnbYZyjX78v+D9Gm2bak9MWsvT0v7MIux+00xrVQyBr77rWjrCW10zHXzSDxKg23RWGU87 fzP1JCaKTx/ZDLcLv86lwXOqVsIgn7o= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=j7cJVV08; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf12.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730603955; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I/oxNup6yHKTkAJUW7pkB0pJRmgS83/ZcHxjD5s3KGY=; b=1gKfoDF55DrSTN2ZuWSOudDkE2T3y15YG1/VFZw180Y5MuYL8TKanh8ByHf2x4N9D92Bx6 Z2x1mPBFUnZg9bJ4yOnE8ydH1OsqYTxTlb6eZSsTiOmtmE9c+72AI8BUJtxHcxRqYBbXgM LwZcjpA0u77re4wtNqJdAi1j/N6DIyg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604076; x=1762140076; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LsH6bLbkT8Oq6ZmvZizx30VKkaEZwigMJxQ/s5KiGZI=; b=j7cJVV08gXhZ6Bjuuoxoj2PvRGkHyuouLJNCjzbnU3e8pMnhNL86eimC GdC8/fhT1h5YTdgXT2JWa4Gq8L9qER2+G2cFGgWRYBM4qbrnPNwNVyWbf 2f+WzicOnmHEBZ4hLQjXGvMBiQGLIeIqgXFLpSNWsZu4cVJQVNnf0dHY5 1/kHU4Fr6QOBNMo1l45gLELsVbhuhmnuCFQOR/gqZIoNoklnJGfyfZLNF ec+doEOMceJH4D9D5RcrYgZiWkWz+MRjGkUxONuc9MOh6u8CFTLR3C23p Db85CkC+EO1DhdLnCLGF2RWZ+ScUJgjFg2xAVeCiZ6/aF0ewZpz1n+Rmm w==; X-CSE-ConnectionGUID: iJZhHQ3cSpuiGeMr6gbVwA== X-CSE-MsgGUID: UbwNezJoQImwvFjbA3P6tg== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548232" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548232" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:12 -0700 X-CSE-ConnectionGUID: OModDCsfTFqmbXDkZFRLLQ== X-CSE-MsgGUID: 3DAXP24xSo+OwZV1sOY6/Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451871" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:13 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 02/13] crypto: iaa - Add an acomp_req flag CRYPTO_ACOMP_REQ_POLL to enable async mode. Date: Sat, 2 Nov 2024 20:21:00 -0700 Message-Id: <20241103032111.333282-3-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AE52740005 X-Stat-Signature: cgryp3e3ifuscq6kk1by9qzt64i18why X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1730604062-72077 X-HE-Meta: U2FsdGVkX18n/FR/wCaoZ1WciXl1/pwrrXjr8KL307c29t2sHDAI9U2t5RozSMbKToqhxOi0DksnjfOZFYTbWBYUWj4gLKexERxNcw2GZSh8UwEl9BG0CDWKEmxnfMwwuBMWlDfiRYbq3kOSDxS8SwA8f5yofxLb4ulR0TeOGQbem1J7qTfNo/TWwJ0pk+hn4+IqDHbL0Cm+1+2uALzn+FNzE9zPocLOIa6t9incxDdafmjRprvN34cT8VE4Rqd05/auAQB6swlrg3oTj8nrCkR1BiWPhlqFT9SEBFGujIi+DLvc8QFn9DVubqjF8rtmX205FS4EY1QymgHMY4l33+drJuMmTB04f9d6Lu3dnNKSnnEoIKTiWbVjewsIhmzZVFLcDKHR4SzqKcX2KjpGf9EzIM4BTYE1SDDMvt7sANTSur6tprFexyaWhAyCZutFbo3PFd9HR3P+bEi4K/BqixIYfrb6kpgCHS2WEL/vo/zpPqM6/+1k/QCG+TIwlTQItvm0Z6wnAes6Qxf6NYTSwXM+sz3JfIl2bnqIIZKotUcDMSulwlObdUSoSw7iAKTwBo0Jx/7k5oAny2TpBUMQ0mo7PP5ow3XHobd3g/AQDTCb/y3+KmQcOajnbK3+3ifo+d5O3ZLNh6mnVHqNfmDNGanq2sEQ6iQkk9bZeUqpdV4P3gBhUMdcrwnMD71QFEP273UPdybGxMNKQAsCIku737UYw6jh2AAyhhCFQU2MS8X7084co8LdymITLAUqYZs3XhOIC6Ls3ngeSwXt0J6AUvF4UZKnGR5qgMTf07WubGeuaFClT5LgLn0xNP3kJ19cpTaM1bprvME8on4VXr/Ac39TvQBxlfgP1csyRK2lhR8CwmDKvsd9u1xIUdFk4Qze4cVXZ1Bg+IN2rLOwTEvlWq4GUs5/bojt7U7PRgg19EoNTYtzTJC+LHOzdw2cpDF3LqY7jNpRjWohzxYngvo EObjgMgg WFPiNPzBOJ/oo8owHb82i54eJb2q5P1qc4p3ptA7G0VB3RTM4XehEAvoyYZsD6kn+udN5lE9ws+5G6ykfs9A4ZrH4SxOhvvV/nQWkEPWwPxAJEJwtgRqAeifqDchA9c4v2SCY/GUDauKpXNS5f7xiuNKGYtMrt7kM4UcZcW0HKT1jftkax0LzQSJjNm3RMogs3Y1eaQVqxih6D4yDCbW2Z8NeLhBKwlp1hI8eND+HUutbJVdbEzIW6nU4/a+8Z8aCYzLD59DBFbggJQnTkpUIYYFS7NBaDapRtPmm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If the iaa_crypto driver has async_mode set to true, and use_irq set to false, it can still be forced to use synchronous mode by turning off the CRYPTO_ACOMP_REQ_POLL flag in req->flags. All three of the following need to be true for a request to be processed in fully async poll mode: 1) async_mode should be "true" 2) use_irq should be "false" 3) req->flags & CRYPTO_ACOMP_REQ_POLL should be "true" Suggested-by: Herbert Xu Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 11 ++++++++++- include/crypto/acompress.h | 5 +++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 237f87000070..2edaecd42cc6 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1510,6 +1510,10 @@ static int iaa_comp_acompress(struct acomp_req *req) return -EINVAL; } + /* If the caller has requested no polling, disable async. */ + if (!(req->flags & CRYPTO_ACOMP_REQ_POLL)) + disable_async = true; + cpu = get_cpu(); wq = wq_table_next_wq(cpu); put_cpu(); @@ -1702,6 +1706,7 @@ static int iaa_comp_adecompress(struct acomp_req *req) { struct crypto_tfm *tfm = req->base.tfm; dma_addr_t src_addr, dst_addr; + bool disable_async = false; int nr_sgs, cpu, ret = 0; struct iaa_wq *iaa_wq; struct device *dev; @@ -1717,6 +1722,10 @@ static int iaa_comp_adecompress(struct acomp_req *req) return -EINVAL; } + /* If the caller has requested no polling, disable async. */ + if (!(req->flags & CRYPTO_ACOMP_REQ_POLL)) + disable_async = true; + if (!req->dst) return iaa_comp_adecompress_alloc_dest(req); @@ -1765,7 +1774,7 @@ static int iaa_comp_adecompress(struct acomp_req *req) req->dst, req->dlen, sg_dma_len(req->dst)); ret = iaa_decompress(tfm, req, wq, src_addr, req->slen, - dst_addr, &req->dlen, false); + dst_addr, &req->dlen, disable_async); if (ret == -EINPROGRESS) return ret; diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h index ab0d9987bde1..5973f5f67954 100644 --- a/include/crypto/acompress.h +++ b/include/crypto/acompress.h @@ -14,6 +14,11 @@ #include #define CRYPTO_ACOMP_ALLOC_OUTPUT 0x00000001 +/* + * If set, the driver must have a way to submit the req, then + * poll its completion status for success/error. + */ +#define CRYPTO_ACOMP_REQ_POLL 0x00000002 #define CRYPTO_ACOMP_DST_MAX 131072 /** From patchwork Sun Nov 3 03:21:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860288 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 128C4E6780A for ; Sun, 3 Nov 2024 03:21:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B3A5D6B0088; Sat, 2 Nov 2024 23:21:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A9C026B0089; Sat, 2 Nov 2024 23:21:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 914D56B008A; Sat, 2 Nov 2024 23:21:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 669F06B0088 for ; Sat, 2 Nov 2024 23:21:20 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DF94E1C797C for ; Sun, 3 Nov 2024 03:21:19 +0000 (UTC) X-FDA: 82743332196.24.3B4A49B Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf17.hostedemail.com (Postfix) with ESMTP id 7126740003 for ; Sun, 3 Nov 2024 03:20:54 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=huLGYf6v; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf17.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730603995; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mixV1IFnrI70cdoC3pE58mO5Bj3BpF3rnTx+GdEXUh8=; b=gNnnR9uwfWwbN7wPPtGEGZcN0QsL+g99+GIpnE3CW6ydf/cP8APRcYk4ed2MSf70eQRqQa UvMsKhHA4P/c4tCih8D6GfHEbpHjMFPRNBprEI9sYbz3OCBrAtNrXNimvAMlW59nKC21lt xVcBvUQm/8+KNTVg5AWJJ/hYzxZ7+Go= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=huLGYf6v; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf17.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730603995; a=rsa-sha256; cv=none; b=mS/JGAMtaGZ8mgCqtzLqPVnOLUfqHbFjaunu+CAKJwjxDWolWc3cNeuWcuWJjKCowG5Mk8 Sz/QMTbBIlAmjpTk3v5+qMedyeevQGkgR0tSTI8oBCtUVgEnW3f7pw3cA/u93C/cTRiXWu 1Qdc+TuOkfVaWaSMlZZqLElhAIoigm0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604077; x=1762140077; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=A/Xb7b4ZlfpZ9fjf2/i7i8CtYcuuNebRbHMn7yrPUpE=; b=huLGYf6v/YPDKRbrcPjO65wsecKPcUEs0D8+VL/dduqOYlfIEMhqtGgS HsXdHV7Yk32JI5dtcLW5+KKc9XTHwbGYxoRqrQY4liCxNiBeBDFOlzxOO tK5rt0sLd2mu+wp9hT4yio93r21AfRvaIavDD5MhGAELIjuwNz+TH4OmE jg534CNwIGzJWEmLvRQwY3VEdP+Z4x98qt1LEy8315Iw/BRcbFfLd0QfS 1XXw7l0VUV4xFs2gUTcztiHVQHmqeHygSl8sgOrXkZrIuUz3OMDmSQhpV Dk1XYRX1t1KHqaLH2LhqnjWuEd4/YoTkrWqH91MxLAYSRW14bhBVFCZPz A==; X-CSE-ConnectionGUID: 1rDjvlZKTkWAQ+f3bLP2Jg== X-CSE-MsgGUID: wuATlI+eRRK8F58HQNHvOA== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548239" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548239" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:13 -0700 X-CSE-ConnectionGUID: LgPwuFS0TdG3yCMIj+pSwA== X-CSE-MsgGUID: 7nNDVED+S1a6Z4X/Dbo88Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451875" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:13 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 03/13] crypto: iaa - Implement compress/decompress batching API in iaa_crypto. Date: Sat, 2 Nov 2024 20:21:01 -0700 Message-Id: <20241103032111.333282-4-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 7126740003 X-Stat-Signature: czp9xr1gf96o9cjoaaofyhne8szsggcr X-HE-Tag: 1730604054-77353 X-HE-Meta: U2FsdGVkX182KioCy+gctMnrDygFS55DPSV2Z+AUO13oDPnxqsGiLwMYM29jWcvNhJgCmmDMI5Vfd2tCnp+deEHot/SMJWa/hq9QcHdaFCp+qMn3OPLG71rXg5A62PZFazyhS4cKatWG5hmLAmgud6ubomS+JkvLOB/KAeHRmuq07fTkWdFs77vG/DUjX9Ybl2VmwTopg82287R3VE1iQh5LtcoN3oo1nPwHce/DwzFiSGjyHw1g8LYiexNUAYMEUA/BW3n4dBAxoRPC3DcMjf0z/P2AGZiHVbyk7637RI+8QlB29b5F7+X/0db3gYNmGoACWYu1mVZ1vQFLq8EZoaiHYlVCzksjLlDQyEMSmKXXgOqrRWcVXlCNrvpiuxXSi70g9ZOaxivSWpj1Obde+GhizZKmpX8WsHN+3FmCdHr6iVw/PjS3t7nZ+j18wTSHl2LOGmAN6QgEcUu+vAFtLv/Rh0a4TYedPNXQBy7tT54wey6zJMuqUAY62mH4IluISiimisAhPNEp2zEy6ifMg1ZZfw9+0abGja3DB4SCoWlIM6dskwwHtYyusra3tgRglRD3asuoUgqehgRa2xF0rUwCxa6S4BO0Q0T5q1cm1APXC2OAodEkGoFSm5+AWUTQ8uHtIfNSvMA90Lk3uvMtJ0/VJFYQw9Q26UF7Xi4mMfMgDQcc8z1YJhUMHVqfIO+Kt1KMS7MaebaxH1bWwOO4LqyVeDysx7Yi/HOOIHNYIepOLq7Mmp4NEZkF8LrfSixuaELZ/ePxhy0n6M9yve2qRdlkUXS1WWA+K3Lvj0IZ+dMrbXICu9XNAZ1aV4Aw1MUek0K91fPeB8WLLN9wi+9lr6AZZIZ3sQfM9/vu2DVY1ozowhY2l3AuVJwrxVASkInMoz3d25Cb0m4AvRLQ9pPe2TuEw5V0ym7SzJf9MPqhn2/vkp21jzh2pJCa+BgFf0OFO3D63SlK0LZbUFlnGNB FlwS8TTp lLbrGfGa9wXsGFiTIoZSh3JHhIEWuwUWAuIvpIKlG4CFX1ZAYCAAy2c38ig/R2OXKVLuUlfN62fbTArD5GNAonJAw3N+e/NMvxU0U+tM3AwKOMU0O+IoDsjV2TFxD6Lr+CX/9A6gFYxXqsKYBT5unfWhzhs0IdMK3f3UOmXTrbhL6iktff6dFCy12g7WhLYBe+WMzLSOh2pNgLCOFtDyTjSB6mg5dmWHy9vBtR4mE67MyQSpk9RclmDvjMjrvKBDPzO8DXNbgsCva6I7MEqbh2tuFIN/ml0bCHsEQlvwuunTYRGBcbFFi9ktl4A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch provides iaa_crypto driver implementations for the newly added crypto_acomp batch_compress() and batch_decompress() interfaces. This allows swap modules such as zswap/zram to invoke batch parallel compression/decompression of pages on systems with Intel IAA, by invoking these API, respectively: crypto_acomp_batch_compress(...); crypto_acomp_batch_decompress(...); This enables zswap_store() compress batching code to be developed in a manner similar to the current single-page synchronous calls to: crypto_acomp_compress(...); crypto_acomp_decompress(...); thereby, facilitating encapsulated and modular hand-off between the kernel zswap/zram code and the crypto_acomp layer. Suggested-by: Yosry Ahmed Suggested-by: Herbert Xu Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 313 +++++++++++++++++++++ 1 file changed, 313 insertions(+) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 2edaecd42cc6..3ac3a37fd2e6 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1797,6 +1797,317 @@ static void compression_ctx_init(struct iaa_compression_ctx *ctx) ctx->use_irq = use_irq; } +static int iaa_comp_poll(struct acomp_req *req) +{ + struct idxd_desc *idxd_desc; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + struct idxd_wq *wq; + bool compress_op; + int ret; + + idxd_desc = req->base.data; + if (!idxd_desc) + return -EAGAIN; + + compress_op = (idxd_desc->iax_hw->opcode == IAX_OPCODE_COMPRESS); + wq = idxd_desc->wq; + iaa_wq = idxd_wq_get_private(wq); + idxd = iaa_wq->iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + ret = check_completion(dev, idxd_desc->iax_completion, true, true); + if (ret == -EAGAIN) + return ret; + if (ret) + goto out; + + req->dlen = idxd_desc->iax_completion->output_size; + + /* Update stats */ + if (compress_op) { + update_total_comp_bytes_out(req->dlen); + update_wq_comp_bytes(wq, req->dlen); + } else { + update_total_decomp_bytes_in(req->slen); + update_wq_decomp_bytes(wq, req->slen); + } + + if (iaa_verify_compress && (idxd_desc->iax_hw->opcode == IAX_OPCODE_COMPRESS)) { + struct crypto_tfm *tfm = req->base.tfm; + dma_addr_t src_addr, dst_addr; + u32 compression_crc; + + compression_crc = idxd_desc->iax_completion->crc; + + dma_sync_sg_for_device(dev, req->dst, 1, DMA_FROM_DEVICE); + dma_sync_sg_for_device(dev, req->src, 1, DMA_TO_DEVICE); + + src_addr = sg_dma_address(req->src); + dst_addr = sg_dma_address(req->dst); + + ret = iaa_compress_verify(tfm, req, wq, src_addr, req->slen, + dst_addr, &req->dlen, compression_crc); + } +out: + /* caller doesn't call crypto_wait_req, so no acomp_request_complete() */ + + dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); + dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); + + idxd_free_desc(idxd_desc->wq, idxd_desc); + + dev_dbg(dev, "%s: returning ret=%d\n", __func__, ret); + + return ret; +} + +static void iaa_set_req_poll( + struct acomp_req *reqs[], + int nr_reqs, + bool set_flag) +{ + int i; + + for (i = 0; i < nr_reqs; ++i) { + set_flag ? (reqs[i]->flags |= CRYPTO_ACOMP_REQ_POLL) : + (reqs[i]->flags &= ~CRYPTO_ACOMP_REQ_POLL); + } +} + +/** + * This API provides IAA compress batching functionality for use by swap + * modules. + * + * @reqs: @nr_pages asynchronous compress requests. + * @wait: crypto_wait for synchronous acomp batch compress. If NULL, the + * completions will be processed asynchronously. + * @pages: Pages to be compressed by IAA in parallel. + * @dsts: Pre-allocated destination buffers to store results of IAA + * compression. Each element of @dsts must be of size "PAGE_SIZE * 2". + * @dlens: Will contain the compressed lengths. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_pages: The number of pages, up to CRYPTO_BATCH_SIZE, + * to be compressed. + */ +static void iaa_comp_acompress_batch( + struct acomp_req *reqs[], + struct crypto_wait *wait, + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages) +{ + struct scatterlist inputs[CRYPTO_BATCH_SIZE]; + struct scatterlist outputs[CRYPTO_BATCH_SIZE]; + bool compressions_done = false; + bool poll = (async_mode && !use_irq); + int i; + + BUG_ON(nr_pages > CRYPTO_BATCH_SIZE); + BUG_ON(!poll && !wait); + + if (poll) + iaa_set_req_poll(reqs, nr_pages, true); + else + iaa_set_req_poll(reqs, nr_pages, false); + + /* + * Prepare and submit acomp_reqs to IAA. IAA will process these + * compress jobs in parallel if async-poll mode is enabled. + * If IAA is used in sync mode, the jobs will be processed sequentially + * using "wait". + */ + for (i = 0; i < nr_pages; ++i) { + sg_init_table(&inputs[i], 1); + sg_set_page(&inputs[i], pages[i], PAGE_SIZE, 0); + + /* + * Each dst buffer should be of size (PAGE_SIZE * 2). + * Reflect same in sg_list. + */ + sg_init_one(&outputs[i], dsts[i], PAGE_SIZE * 2); + acomp_request_set_params(reqs[i], &inputs[i], + &outputs[i], PAGE_SIZE, dlens[i]); + + /* + * If poll is in effect, submit the request now, and poll for + * a completion status later, after all descriptors have been + * submitted. If polling is not enabled, submit the request + * and wait for it to complete, i.e., synchronously, before + * moving on to the next request. + */ + if (poll) { + errors[i] = iaa_comp_acompress(reqs[i]); + + if (errors[i] != -EINPROGRESS) + errors[i] = -EINVAL; + else + errors[i] = -EAGAIN; + } else { + acomp_request_set_callback(reqs[i], + CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, wait); + errors[i] = crypto_wait_req(iaa_comp_acompress(reqs[i]), + wait); + if (!errors[i]) + dlens[i] = reqs[i]->dlen; + } + } + + /* + * If not doing async compressions, the batch has been processed at + * this point and we can return. + */ + if (!poll) + return; + + /* + * Poll for and process IAA compress job completions + * in out-of-order manner. + */ + while (!compressions_done) { + compressions_done = true; + + for (i = 0; i < nr_pages; ++i) { + /* + * Skip, if the compression has already completed + * successfully or with an error. + */ + if (errors[i] != -EAGAIN) + continue; + + errors[i] = iaa_comp_poll(reqs[i]); + + if (errors[i]) { + if (errors[i] == -EAGAIN) + compressions_done = false; + } else { + dlens[i] = reqs[i]->dlen; + } + } + } +} + +/** + * This API provides IAA decompress batching functionality for use by swap + * modules. + * + * @reqs: @nr_pages asynchronous decompress requests. + * @wait: crypto_wait for synchronous acomp batch decompress. If NULL, the + * driver must provide a way to process completions asynchronously. + * @srcs: The src buffers to be decompressed by IAA in parallel. + * @pages: The pages to store the decompressed buffers. + * @slens: Compressed lengths of @srcs. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_pages: The number of pages, up to CRYPTO_BATCH_SIZE, + * to be decompressed. + */ +static void iaa_comp_adecompress_batch( + struct acomp_req *reqs[], + struct crypto_wait *wait, + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages) +{ + struct scatterlist inputs[CRYPTO_BATCH_SIZE]; + struct scatterlist outputs[CRYPTO_BATCH_SIZE]; + unsigned int dlens[CRYPTO_BATCH_SIZE]; + bool decompressions_done = false; + bool poll = (async_mode && !use_irq); + int i; + + BUG_ON(nr_pages > CRYPTO_BATCH_SIZE); + BUG_ON(!poll && !wait); + + if (poll) + iaa_set_req_poll(reqs, nr_pages, true); + else + iaa_set_req_poll(reqs, nr_pages, false); + + /* + * Prepare and submit acomp_reqs to IAA. IAA will process these + * decompress jobs in parallel if async-poll mode is enabled. + * If IAA is used in sync mode, the jobs will be processed sequentially + * using "wait". + */ + for (i = 0; i < nr_pages; ++i) { + dlens[i] = PAGE_SIZE; + sg_init_one(&inputs[i], srcs[i], slens[i]); + sg_init_table(&outputs[i], 1); + sg_set_page(&outputs[i], pages[i], PAGE_SIZE, 0); + acomp_request_set_params(reqs[i], &inputs[i], + &outputs[i], slens[i], dlens[i]); + /* + * If poll is in effect, submit the request now, and poll for + * a completion status later, after all descriptors have been + * submitted. If polling is not enabled, submit the request + * and wait for it to complete, i.e., synchronously, before + * moving on to the next request. + */ + if (poll) { + errors[i] = iaa_comp_adecompress(reqs[i]); + + if (errors[i] != -EINPROGRESS) + errors[i] = -EINVAL; + else + errors[i] = -EAGAIN; + } else { + acomp_request_set_callback(reqs[i], + CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, wait); + errors[i] = crypto_wait_req(iaa_comp_adecompress(reqs[i]), + wait); + if (!errors[i]) { + dlens[i] = reqs[i]->dlen; + BUG_ON(dlens[i] != PAGE_SIZE); + } + } + } + + /* + * If not doing async decompressions, the batch has been processed at + * this point and we can return. + */ + if (!poll) + return; + + /* + * Poll for and process IAA decompress job completions + * in out-of-order manner. + */ + while (!decompressions_done) { + decompressions_done = true; + + for (i = 0; i < nr_pages; ++i) { + /* + * Skip, if the decompression has already completed + * successfully or with an error. + */ + if (errors[i] != -EAGAIN) + continue; + + errors[i] = iaa_comp_poll(reqs[i]); + + if (errors[i]) { + if (errors[i] == -EAGAIN) + decompressions_done = false; + } else { + dlens[i] = reqs[i]->dlen; + BUG_ON(dlens[i] != PAGE_SIZE); + } + } + } +} + static int iaa_comp_init_fixed(struct crypto_acomp *acomp_tfm) { struct crypto_tfm *tfm = crypto_acomp_tfm(acomp_tfm); @@ -1822,6 +2133,8 @@ static struct acomp_alg iaa_acomp_fixed_deflate = { .compress = iaa_comp_acompress, .decompress = iaa_comp_adecompress, .dst_free = dst_free, + .batch_compress = iaa_comp_acompress_batch, + .batch_decompress = iaa_comp_adecompress_batch, .base = { .cra_name = "deflate", .cra_driver_name = "deflate-iaa", From patchwork Sun Nov 3 03:21:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 653FDE67819 for ; Sun, 3 Nov 2024 03:21:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 19C476B0089; Sat, 2 Nov 2024 23:21:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 127F66B008A; Sat, 2 Nov 2024 23:21:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBA726B008C; Sat, 2 Nov 2024 23:21:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AED186B008A for ; Sat, 2 Nov 2024 23:21:20 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 66EEA1C79FB for ; Sun, 3 Nov 2024 03:21:20 +0000 (UTC) X-FDA: 82743331608.03.AAC25A9 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf18.hostedemail.com (Postfix) with ESMTP id D14571C0002 for ; Sun, 3 Nov 2024 03:21:04 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QYvf3Vge; spf=pass (imf18.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730604031; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S7KSdyx1Yt75nyZU3ul+IeNaI1n+U4GJzibDvgrqEXQ=; b=H8ilLgAb3yTfdtiqQhm1VwDp7s1lp+Gbvseqz60vaIoTVQEVUey3fDz1rGv+tXzjHXWpbp BTSgqITFf+dGU2H8VCFE/4zOTU3d5ZMDxyvjD0nsSICES7xHxaiGapYCcBnxy5Y1fRpb11 U871hJKBkL9TKRjR0mSFPO+gRjOFZ20= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730604031; a=rsa-sha256; cv=none; b=3zmmqMhKiXcIVkghyxiJox5rFNRHGAVQiGv3HJufROf2tlJL4KI6MOLj4s7cJTTdThvkt3 jlFphMPTagornOvSCd30Nl5r52lfpK/wczbnLkDmLqXbyP29H6TIlz7IA2KSo+qsxAOS56 1PtmjfjGrMHaFhyCQq4ksyTLmKo1m/U= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QYvf3Vge; spf=pass (imf18.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604078; x=1762140078; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pX2jvauw5wJ2+nUrjP9yXpyaB7Rg+Ni/oUTJYJHTKmc=; b=QYvf3Vge82nCZKXslwen8rG0sJWlFIKkCGf+upm+UQqCMUpmlL+IGY2K payYVYnPAHC6XcYsEGfo7lmcHAaDVyDCwt6FDhhmazv8ZcgEpKm27ZDjb NKzWxrcawlh3Ldua+CuefTGD+7QiddNNZHxktgN1CEJJvV2Fsz8hdnPlZ eAGU6y14YmTsoSn3wzu41kOsziINggarKx0OqAehDcjLF2Krm+AcTWtOV EHaOlhWUzcnD/HS5xfGMs3qKc9rl0075FNbwXrFpa5mOTsKs/PhCfFhXv oZLCXgDct+wCU4vKA/w8WDJpoNbgsQyK5v41UsR53f84zcve3iMFH3nzG w==; X-CSE-ConnectionGUID: 66RJAFBTT3Gv/KH+Aui0nw== X-CSE-MsgGUID: oBILVIcCR6C47Y06wUwb5w== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548249" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548249" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:13 -0700 X-CSE-ConnectionGUID: wD/bMYhARqyYIrYFEHpnGg== X-CSE-MsgGUID: wRKdRZrCQCS1ApOrudAP+g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451878" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:13 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 04/13] crypto: iaa - Make async mode the default. Date: Sat, 2 Nov 2024 20:21:02 -0700 Message-Id: <20241103032111.333282-5-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: 6asw36sjyxjwy1bjwjwt8zc8x1u18ncf X-Rspam-User: X-Rspamd-Queue-Id: D14571C0002 X-Rspamd-Server: rspam02 X-HE-Tag: 1730604064-720315 X-HE-Meta: U2FsdGVkX19oyx+/aEfA7Tor+Z2OqExYpwwG23kh71TOjMHRzI4ZliDpqUhxiva3Gj+8cWjymU//Bk/xeReCQ93w0x4FsRA45TOhpMZwNgWJ6sAI2f13bpvVhLHvFE9QZT3PSjdfuueK+UxmYXD6cM5p/0rKm5rvWR5OloMx2WbpQL3ajCd4OZs8qH2qGng3eYcXJ5aINa71UgY+nse/AmOVuAC/vudGxg4cvZ0BIDjBFOkkpUyNZnOlGD01pSCbciruhj/5hj1aDka988VPbtBNEHVENbc+ZWvoq4Xc+96Xo0L3oRu5lMdNfWNJ5ZcpAkQSakLyifGQyd6Qo5DcEayrMFhC/J4UeztA9nwdvR9X0qw4FF3TBlex6NzOCdiZhKsWIiLrgXbTnyV/bJYDBpx7OTL0TCiwVOUfrehVfrCFqxoLetdEeRHqA3S4OBT6cJLJH7hFarfxSNyVqm6izs+JiRCZkNalVovpIIjmATxJooqbOrhqMMbIVZLPfQj4FHWQYZ32CfQZpkI9+EvjoCDWaRhUcFpzzrepsTgmoYSksXsSI9/yWror2nGZMH9h48JpkbimYZd0uan8Ic7W5nd2EC3fTCeEWrbZ/XPaJ0upayrPyznPkX9v6mTVYtFpWvounycCP5/zL73lJYI4Yx039noKTcckLEdQypUydmY2DwpF9vFS2QPD7NYFJ0Qf3AUnsBWoaXmrGnjL5WH5SxT3TPSXQt0Z+OOPnz3gxUWWrHIAFUpTTv50dS3c5EAMXPfELCqOl096OPuAdEAdQa4/6ZV95TY6Na8IWkkFsZ13bNfiJ96j3igEdXB/USb1jfJQpj9vRhTX1EdcDH2D6K0MP4rXrNpteE3RdLXUPeu8zlJDnZDDbVhpadoNKv5wpfGGOr+iRFrREWS8H5dXtl6WRYWSvHRcLTJSQlMrzLqDy946Hk6ZerX8dBJ3Go/t1p2SQ6798sCEzwC9hcM 9fn6xAq2 v7wFnvT9dO5Ou3TCIAqs/fKE2BwTqwnYBEoQwkRjx90wC/vzr+kjHvZwmj6n0d/P4kKQE66t4tY/2R4NJ541QMZYugseuJunaxvETBCG50w+iz9UWVssJXaURIYcLDqLBMzMglUXoeoZducljZ4AFp2sP9y3nTio7uIXSSumnWxehGp9cc8K8bjnRZw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch makes it easier for IAA hardware acceleration in the iaa_crypto driver to be loaded by default in the most efficient/recommended "async" mode for parallel compressions/decompressions, namely, asynchronous submission of descriptors, followed by polling for job completions. Earlier, the "sync" mode used to be the default. This way, anyone that wants to use IAA can do so after building the kernel, and without having to go through these steps to use async poll: 1) disable all the IAA device/wq bindings that happen at boot time 2) rmmod iaa_crypto 3) modprobe iaa_crypto 4) echo async > /sys/bus/dsa/drivers/crypto/sync_mode 5) re-run initialization of the IAA devices and wqs Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 3ac3a37fd2e6..13f9d22811ff 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -153,7 +153,7 @@ static DRIVER_ATTR_RW(verify_compress); */ /* Use async mode */ -static bool async_mode; +static bool async_mode = true; /* Use interrupts */ static bool use_irq; From patchwork Sun Nov 3 03:21:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860290 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8BD7E6781A for ; Sun, 3 Nov 2024 03:21:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A279F6B008A; Sat, 2 Nov 2024 23:21:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D7196B008C; Sat, 2 Nov 2024 23:21:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7DC656B0093; Sat, 2 Nov 2024 23:21:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 53D2E6B008A for ; Sat, 2 Nov 2024 23:21:21 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 12C6080D61 for ; Sun, 3 Nov 2024 03:21:21 +0000 (UTC) X-FDA: 82743332196.10.23CA1C8 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf12.hostedemail.com (Postfix) with ESMTP id 15C7740005 for ; Sun, 3 Nov 2024 03:21:04 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Zg7K0sgS; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf12.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730603958; a=rsa-sha256; cv=none; b=XPHLMOG1+Q2oVPtZ506C5uqm+MVleD5M7opofAoq2cUmb/ilxNUN6gwr7kp+4881HKZ8VN yuiDT6kzTP5yZMLstDNDA2xwY/Jrgms+NVcMtWD6bD0c3ICa34cZKZoelcPf5kOtSLrf3w GRepkNYJMvJfnVtKKpOkJ2eFZ1hcYEA= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Zg7K0sgS; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf12.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730603958; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oauYIVWgF52IbGjNwKTzwuD93EHH4qPYrLAmXIwHNFc=; b=eDswFrjJwSccN/K2k3y7+3o1mxWmoVGOO2UDtIzzfxsfzDd0ygbZaYWHrgC7YboK91MuoG /WbSKgZTUJbUSTkAgfDGopa7E5u3TJsuLQBE8sosbB7Jy4nD1RLzayOSWRgh1rLVghXh5i vN9PVGjcVz+BSeX6+5quC2J513YKkhk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604079; x=1762140079; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yFcfU2b3t6Krq275SHD6OXxnKnpJ7Jmrcy/87R0bztw=; b=Zg7K0sgSdKCbKpxgATQwaYuG4iLhlcueIDn6G1UuUVw2vlnoIHnuw1OE zKaKMbwfn2wJFIimomJ4YP+IaBHXpYKAgVPBjaQm1wplSVgAti5DKabFx tjLpd1ZLvaMQqUE5Nk1jQDZ4XlzqVOy4PdE6ca4ftQaadKU54NaBh+aGP VVlJmCPUm7KKsW1AvrWYwosMRSd7L2FYnUXVThdDW8CMGh8JUFYba1FX4 2fkxheJju7uG6B4313V+dx7qQTToMTYYZUZlNW2adCkbnZws2wTqeSqi9 0SKS4Jt7R9GxzWqepLYrQNZOlN9G/yKqvx2kgi7gRl1x0dyMeJbPWwX70 A==; X-CSE-ConnectionGUID: 0sgpA5XST1C6vLuWSE6BZw== X-CSE-MsgGUID: 909VbPLyTMqFdY0/S63zfw== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548260" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548260" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:13 -0700 X-CSE-ConnectionGUID: IZm7M30DRUaPx/sFE11RKA== X-CSE-MsgGUID: y0vGMHT2TVG++pkTY3TWAg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451881" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:13 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 05/13] crypto: iaa - Disable iaa_verify_compress by default. Date: Sat, 2 Nov 2024 20:21:03 -0700 Message-Id: <20241103032111.333282-6-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 15C7740005 X-Stat-Signature: p8dcdgdfazbfkk1aj9ud85u46sfr5w4k X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1730604064-566536 X-HE-Meta: U2FsdGVkX19ui8RXKlnSdz2qmINPJDPFgmLvzOfIbP4LMIhkM2YBWGd5cQgiUciQ0Tk1N1L1JNWmSj6hXPuTaidQFmXPi8kUhdIn5sK5JOipRg/jQqIE/olfL2Lmxc2WR0HchU2wMSTJk9bl6BgSJE6ScpyL59ddcUNDQX6Hnyfv5hTMAF37kEZFJNiiGEuTgLkXN4F4q2pg4eeb3EJfPqLw/gTGn7G8V6LOOfiFoOdtg/4mcVJw7EaFZcDzM8ZXuSrQFqY/oc5R6Ljqk+aq06sqjbp0ZD7wBA8mJoGRl4FNcWOkQ58I2xM4DHbGpqHCQEoOj7uLpvV/HyV5FYOP0HruJO7tH7yrn5/KW762Dpwyj5ThzHJOfxNsv4xatwtbEf2bnyyf3RT3o+42yRZSflkFTzCup4hYYzqV79NAsFVDRaNKmzCbmUItPWKMksPO5bfej27MDYu9som/h6jP3yv7H+glt4h2MABhfPicvv4VDQYbZIPv28AkpbVH04Wj5FNl44CD7RbCKnXe5odDtugmiDi1RaHProzIKOOhi6a1Q2PgbEceEbsBLvEJii2QRek1ZKPOpzlFeFte2YHGukNv0S+2YYyjhgv05NiFvBo6ZTPQUHTYS1X39FHkdG+Du53ZtgHugydq+2sCwSiT5ecxnUT6HCgMw4vbCXXf86SmvJ+6+8LB0ZGUTSCY3Rts9286vOyCbpSogoWs2XEAgIs0TG7fvP2ViS7gDTpU/lLb3R9y4eChWUEgychfw/d3gYh9osXSGMwaSEvO/4U2Iyu33RRpiSL64vqWqXwnSCT4UEc4jO93t7uh7YrjLQvAQD61BsQdq0c88R/3vBLP4l+FBleUYeyrRKACzGkgm7yj+8PHoCuP2yci1Q552pQYfHAoHLI7M/VFnCmwO9C+NOY6c5piOMqS2S2k+D9XwB1L9jo29xhApDB4ORbovcHZcJaBKzFcmb2JUeVpEL8 IPBRIOA4 O3N+Vyx//OoTeLzGTY1Vpt5aJd56TLtTkU4245cjXSlo2Ipc0rtiRRTrbGJTo83AjohB4tuO6pXJ1AoVhEMqFZ7nIqwfqFKSW3aQ+edp7OY5E6e855nB2GNhcHH3/vzqXMGW7lg4nzTfsVDEVRd2pw+xgtQP3W8jqNnjP2Vh96vRC4U5JvioOMQJUqQ1D0NDsvvYDyA/B8bHHeWgS6zsu3qY+Demtf94HAudLBRkJeMB73e4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch makes it easier for IAA hardware acceleration in the iaa_crypto driver to be loaded by default with "iaa_verify_compress" disabled, to facilitate performance comparisons with software compressors (which also do not run compress verification by default). Earlier, iaa_crypto compress verification used to be enabled by default. With this patch, if users want to enable compress verification, they can do so with these steps: 1) disable all the IAA device/wq bindings that happen at boot time 2) rmmod iaa_crypto 3) modprobe iaa_crypto 4) echo 1 > /sys/bus/dsa/drivers/crypto/verify_compress 5) re-run initialization of the IAA devices and wqs Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 13f9d22811ff..c4b143dd1ddd 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -94,7 +94,7 @@ static bool iaa_crypto_enabled; static bool iaa_crypto_registered; /* Verify results of IAA compress or not */ -static bool iaa_verify_compress = true; +static bool iaa_verify_compress = false; static ssize_t verify_compress_show(struct device_driver *driver, char *buf) { From patchwork Sun Nov 3 03:21:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860292 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF9FEE6781A for ; Sun, 3 Nov 2024 03:21:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C8896B008C; Sat, 2 Nov 2024 23:21:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E28986B0096; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5ED66B0096; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 71A126B0093 for ; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DBC62120B96 for ; Sun, 3 Nov 2024 03:21:22 +0000 (UTC) X-FDA: 82743331860.08.FB20A45 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf17.hostedemail.com (Postfix) with ESMTP id 6795B4000E for ; Sun, 3 Nov 2024 03:20:57 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=iFuJIJ5F; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf17.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730603998; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wN2LmZ5a5qfTWVX1cLgoIE5geGN37SlJY4gc3Svdtbo=; b=rkz+xIWZxRe/PI7RsQMmQ0pQG+cup4nOeuaxwSUyUTXKy6bDkaqMb5jnH1IssUqUZNZT4e ZwMHzGvt9UYYfZBA5VssObEX6/Bl+NQVG9+MKNpTryCx5OrUu02LdmTV6myv2/JJfN3LIs VSKmMkKfYiCYRUuVvopHItht0AtlG2I= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=iFuJIJ5F; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf17.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730603998; a=rsa-sha256; cv=none; b=ZyZ70q169bF6xYGzMwvcE4E2OduA5wrjC3L+SApO9b9rdkEjqbUKDNzVcOqKT2IOdedtLc eW+idM9oQqVh8gHMYFAzyOFSWlRfB7cNujx3fEBehaYOs+fLBJjmve7QmfrRRvHnq8MQMb TkIcbX9BWrt6uAyI5ZrBVQXRw8yhWcc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604080; x=1762140080; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dsGJj+DqE0i7pg9VVqHrElSTuA3RtibV8w9INDbEh+Q=; b=iFuJIJ5FiFXaf0MG4Qd0NZFBwLh8fV7JICB3nKpzXADRAk/bH/zdIfO9 gUt2BRU2OY92w2NGeNHgs5lUonWSC/QQ8jrv2b7DZ1Dh98tLGDpUe7OjH Rsp+P7Aj+nfP5XOZvjJrMoi6cO3bOM8YnG55No5eqrhAg+cyxth7lPntx d7CTV/GFsi52YZz6xvLUPz8FHlyThcffYlug58cZpOFmUoL6rqUilC49L qzds90jO2FNY40NgimSkCokWFcRn1GFXT4l6jXt5cE5sDLXPbeDvgI1pb VBMrJ1t/TLpvakY04MXmAIXbmopYlWijFoqcchCTwjwJuEkum8Xgdgfne g==; X-CSE-ConnectionGUID: Xn3xlgvkQeKZIHto/pnRrA== X-CSE-MsgGUID: xSfQEQDAQ/mr1GWggaNLsg== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548273" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548273" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:13 -0700 X-CSE-ConnectionGUID: r4p/5N5kTQ6zqMjLh5lsKg== X-CSE-MsgGUID: MBv2V3OuTLWdRXqpDMTzfg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451885" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:14 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 06/13] crypto: iaa - Change cpu-to-iaa mappings to evenly balance cores to IAAs. Date: Sat, 2 Nov 2024 20:21:04 -0700 Message-Id: <20241103032111.333282-7-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 6795B4000E X-Stat-Signature: 44pngxi847sqgneytr8w8pq84skcj881 X-HE-Tag: 1730604057-868542 X-HE-Meta: U2FsdGVkX1+Ewv/Zc+/pNI/BfxgSBD1NoT8gJAxGDIVe5/FAGvGEHatmvN8mYwmMbPI/c/QIDOhRX8WiE2wpdNpiSN2uM67xM2tbbVLBmB1g7Oxyu+BjDdOGxHp8iCsE7k3JgJhx7LdvM+X4EIECrMsP0fB7DBmbZe8oCSjDMuQOfbIeJlCdW33V3Vh4QcI32T8Xui0sz0e9TUHGi7aYR+e4sJgSUK5EWanurn9pgvy4PqxKbq3fV49xIODRhbXHN9643cmJuZQG0V123/Upgh/JBnC6+f48CMgV1sUw815GTm4Ik9jckqOb9gBQ2a8IiXpK8GiADGAnZPRs47k4MZYffLxSeDoVGEZV+CvmzO8qlBi4zOsXgriogCt1n0z6NRgOkPCsAR8vh3l/p5gqgaaO+VIWCbPA5APme5bnMmSkvtQJtPx/mg0MgJj2Y/LU7wv2hPJQPC7V9POFhLEptFT042jWQrq+MWgmuoXMwMA5YOUkMyrRneUV0ai/nMyz8YzyrnVS4Yu5vcIeNUxpXjoCOZ2daw4RhtOsNRnK49CZs5ggtCwrwiRMr/J0dYw0jj11FGpm/5xG3wETh84ZWCq9soiiWQplVtmQaZAdIydTNmpKxeeWIIqjZTwRE8CrvKt5S6eEBguZkfJQ2G8WUUblOLUwnmQ851t6wMhDAH0uN7ta/l4h4ehJwlSqrJeEl3SDoOzIkVIasX9ttCrxG8h9zW18trRKnsdTbCDNw1HGtpmyQDzBcGda44/is99WORNP6FIxRk5FZN1gLkeu+xGpIcsN2BsnpL0xb//OmnMq67VFeZ+kXMHx6vb+uL2hw/yo1S5Ktj5RXZEWQM1BrQuQyBkSBpc+exyNqx7DAcBe1CuD2izNYhkv1AjvFu/pnx94iEhrpxqZ5tAzkev8RXbyOz6ld0QBvlVS2ROTkuzPbs/Y37FNOYzmOuPYJnVrLZjNGs57plEtLkG16M/ UbWdhjJc 6QsSsBIykIi9jaKl5iycB6p1bx/q3cbiW5tXnIVzj3lF6cRkVdsDsTrlroA4Hh2oZKrfJ9lHR0iMPX9f55Npvk80TjlK2s2WGLTvBNsCy1y4IAjSM/BVjNei6f3mrObKwr31RFbhvLPEl6vN5he2BuWjHIQXuqCcej75u1mo6mdM/GKGhauPp6xsCsQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This change distributes the cpus more evenly among the IAAs in each socket. Old algorithm to assign cpus to IAA: ------------------------------------ If "nr_cpus" = nr_logical_cpus (includes hyper-threading), the current algorithm determines "nr_cpus_per_node" = nr_cpus / nr_nodes. Hence, on a 2-socket Sapphire Rapids server where each socket has 56 cores and 4 IAA devices, nr_cpus_per_node = 112. Further, cpus_per_iaa = (nr_nodes * nr_cpus_per_node) / nr_iaa Hence, cpus_per_iaa = 224/8 = 28. The iaa_crypto driver then assigns 28 "logical" node cpus per IAA device on that node, that results in this cpu-to-iaa mapping: lscpu|grep NUMA NUMA node(s): 2 NUMA node0 CPU(s): 0-55,112-167 NUMA node1 CPU(s): 56-111,168-223 NUMA node 0: cpu 0-27 28-55 112-139 140-167 iaa iax1 iax3 iax5 iax7 NUMA node 1: cpu 56-83 84-111 168-195 196-223 iaa iax9 iax11 iax13 iax15 This appears non-optimal for a few reasons: 1) The 2 logical threads on a core will get assigned to different IAA devices. For e.g.: cpu 0: iax1 cpu 112: iax5 2) One of the logical threads on a core is assigned to an IAA that is not closest to that core. For e.g. cpu 112. 3) If numactl is used to start processes sequentially on the logical cores, some of the IAA devices on the socket could be over-subscribed, while some could be under-utilized. This patch introduces a scheme to more evenly balance the logical cores to IAA devices on a socket. New algorithm to assign cpus to IAA: ------------------------------------ We introduce a function "cpu_to_iaa()" that takes a logical cpu and returns the IAA device closest to it. If "nr_cpus" = nr_logical_cpus (includes hyper-threading), the new algorithm determines "nr_cpus_per_node" = topology_num_cores_per_package(). Hence, on a 2-socket Sapphire Rapids server where each socket has 56 cores and 4 IAA devices, nr_cpus_per_node = 56. Further, cpus_per_iaa = (nr_nodes * nr_cpus_per_node) / nr_iaa Hence, cpus_per_iaa = 112/8 = 14. The iaa_crypto driver then assigns 14 "logical" node cpus per IAA device on that node, that results in this cpu-to-iaa mapping: NUMA node 0: cpu 0-13,112-125 14-27,126-139 28-41,140-153 42-55,154-167 iaa iax1 iax3 iax5 iax7 NUMA node 1: cpu 56-69,168-181 70-83,182-195 84-97,196-209 98-111,210-223 iaa iax9 iax11 iax13 iax15 This resolves the 3 issues with non-optimality of cpu-to-iaa mappings pointed out earlier with the existing approach. Originally-by: Tom Zanussi Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 84 ++++++++++++++-------- 1 file changed, 54 insertions(+), 30 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index c4b143dd1ddd..a12a8f9caa84 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -55,6 +55,46 @@ static struct idxd_wq *wq_table_next_wq(int cpu) return entry->wqs[entry->cur_wq]; } +/* + * Given a cpu, find the closest IAA instance. The idea is to try to + * choose the most appropriate IAA instance for a caller and spread + * available workqueues around to clients. + */ +static inline int cpu_to_iaa(int cpu) +{ + int node, n_cpus = 0, test_cpu, iaa = 0; + int nr_iaa_per_node; + const struct cpumask *node_cpus; + + if (!nr_nodes) + return 0; + + nr_iaa_per_node = nr_iaa / nr_nodes; + if (!nr_iaa_per_node) + return 0; + + for_each_online_node(node) { + node_cpus = cpumask_of_node(node); + if (!cpumask_test_cpu(cpu, node_cpus)) + continue; + + for_each_cpu(test_cpu, node_cpus) { + if ((n_cpus % nr_cpus_per_node) == 0) + iaa = node * nr_iaa_per_node; + + if (test_cpu == cpu) + return iaa; + + n_cpus++; + + if ((n_cpus % cpus_per_iaa) == 0) + iaa++; + } + } + + return -1; +} + static void wq_table_add(int cpu, struct idxd_wq *wq) { struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); @@ -895,8 +935,7 @@ static int wq_table_add_wqs(int iaa, int cpu) */ static void rebalance_wq_table(void) { - const struct cpumask *node_cpus; - int node, cpu, iaa = -1; + int cpu, iaa; if (nr_iaa == 0) return; @@ -906,37 +945,22 @@ static void rebalance_wq_table(void) clear_wq_table(); - if (nr_iaa == 1) { - for (cpu = 0; cpu < nr_cpus; cpu++) { - if (WARN_ON(wq_table_add_wqs(0, cpu))) { - pr_debug("could not add any wqs for iaa 0 to cpu %d!\n", cpu); - return; - } - } - - return; - } - - for_each_node_with_cpus(node) { - node_cpus = cpumask_of_node(node); - - for (cpu = 0; cpu < cpumask_weight(node_cpus); cpu++) { - int node_cpu = cpumask_nth(cpu, node_cpus); - - if (WARN_ON(node_cpu >= nr_cpu_ids)) { - pr_debug("node_cpu %d doesn't exist!\n", node_cpu); - return; - } + for (cpu = 0; cpu < nr_cpus; cpu++) { + iaa = cpu_to_iaa(cpu); + pr_debug("rebalance: cpu=%d iaa=%d\n", cpu, iaa); - if ((cpu % cpus_per_iaa) == 0) - iaa++; + if (WARN_ON(iaa == -1)) { + pr_debug("rebalance (cpu_to_iaa(%d)) failed!\n", cpu); + return; + } - if (WARN_ON(wq_table_add_wqs(iaa, node_cpu))) { - pr_debug("could not add any wqs for iaa %d to cpu %d!\n", iaa, cpu); - return; - } + if (WARN_ON(wq_table_add_wqs(iaa, cpu))) { + pr_debug("could not add any wqs for iaa %d to cpu %d!\n", iaa, cpu); + return; } } + + pr_debug("Finished rebalance local wqs."); } static inline int check_completion(struct device *dev, @@ -2332,7 +2356,7 @@ static int __init iaa_crypto_init_module(void) pr_err("IAA couldn't find any nodes with cpus\n"); return -ENODEV; } - nr_cpus_per_node = nr_cpus / nr_nodes; + nr_cpus_per_node = topology_num_cores_per_package(); if (crypto_has_comp("deflate-generic", 0, 0)) deflate_generic_tfm = crypto_alloc_comp("deflate-generic", 0, 0); From patchwork Sun Nov 3 03:21:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7BBBE67819 for ; Sun, 3 Nov 2024 03:21:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA01F6B0099; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C269B6B008C; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4F356B0098; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 66AB66B008C for ; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 19B71C0CE5 for ; Sun, 3 Nov 2024 03:21:23 +0000 (UTC) X-FDA: 82743331986.27.84A56AF Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf18.hostedemail.com (Postfix) with ESMTP id 660101C0002 for ; Sun, 3 Nov 2024 03:21:07 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="SX/Jm9EC"; spf=pass (imf18.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730604034; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/rB+AkisounQykFuvz+8msoIYVtOJaxKQVdKG+ZGBnM=; b=3RI84R3xH0plYZO0GBrPjWYTsQ2izFeoJMS1uht+SXJCItK/bIpvLq4114cHz4o2bzPv2j uvhHme0PKmhmiBCHDKHjPSDd7YRJi4OoJv5h4PnZWnHMNHNeHbLmxU4dCF9mA7WmZm8+on poJJdavcL4zhbzW74UXb5vkr11dnn1g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730604034; a=rsa-sha256; cv=none; b=KQZiCYoehA3Vh7zgq+A2ag2p19WB1qTCCYX9lxZr03nLLgkI6E5mCx14gwS7KgET1G46vd eQ2j1M09NV0QmvcIrdtfQpfePVObBmQvtd3R394d98+f4MUA8iLrh+Wxx2JY+X4GfWIaO2 ZlhbbDm2uG0EEItsVnCHBoZFEyGrv1Y= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="SX/Jm9EC"; spf=pass (imf18.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604081; x=1762140081; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xaKYi5UD1EXb+8QE17eRVa6qtYmQdKerYV4Ficq5ajk=; b=SX/Jm9ECCnyFCEgTTJWXaqu5Okuh6M7JEd9j/Raj0QZBlS7MnlU1TfME u2/pa0FgIQoYdY0jrka5QLMlBm35MPA3Lb3BlHCOWhRWeZk9vsLyu5zRz oV8sYUlroutg4YhN4F9NlN/cOv1SkQyzRvKB8Fenvut4oxghozJeAdTrn vJiuGKuTy4zTzuJgrjFeTWp0u5Fx4ZQkE2ng9zpQ93QUrEFViSuLBttiB GIY7dib82WoCY7pzMFcv27voOHhFpJpHxElVHbYZEaMSifyCAzPrIlp3K US9yhAezP6m9myRyXce67CFicJzc/gXAd/5BCHtm7ldEsKOQNubW4hnVA A==; X-CSE-ConnectionGUID: faoLMOk5Qiyf9pIzOwowmA== X-CSE-MsgGUID: ASqjZowfSJ2lRrI7tYVwAw== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548286" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548286" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:13 -0700 X-CSE-ConnectionGUID: /cmGs6QsS5u6HH45KiqcOA== X-CSE-MsgGUID: awS7bHeuS1Sdla/9uryrCg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451888" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:14 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 07/13] crypto: iaa - Distribute compress jobs to all IAA devices on a NUMA node. Date: Sat, 2 Nov 2024 20:21:05 -0700 Message-Id: <20241103032111.333282-8-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: eb3buhjj5bofrk6qtf5pxeadi174p93i X-Rspam-User: X-Rspamd-Queue-Id: 660101C0002 X-Rspamd-Server: rspam02 X-HE-Tag: 1730604067-821661 X-HE-Meta: U2FsdGVkX18p8A1HBGGHvkiQq04arAYm/p3uI1MlEopEZeUdB3/g3iaprzIOFZDDafKoAlX3rRB32GeUUJESnRwO5cpZ2QkjOW8R1Ap8WNjoTqlWozW5qmnJew+dSfM84/yjRj0Fd/Zld2hrGGdh+rgaQ3ILkcYK6W3Ii3Wm6xMOSKabRueqUBIvy4Q0IujDWLZmSFSGjXvhH/lPIRvRYqkM2brTJtEDdcQz9LQVmc08Ufhw9lUmDoT/Re6zdjuoTpAoheGcNtaTeT7kJx7A/GVl287IRqUA1KgResGL85B4d+ec6Y87Jhkv550QKUMjOh5x4jkM4+nkFLCrj5tdZFRXdtV4bdiBBuEY8RZPFtFflhiA2I91dVRHlGj4ks0nnNz4+bBpxfSXCtEAy2HpBWBNYkzltLjzWeVYktrk3hzilehT6v8cwwnKKKH/Di+sC9KtfV8Ifu4k+fO9wmsLrNLLg8pi8Q8h0jhUgEhFM9T69ryYMcKmYpR55wKnI6Qr2j5d9kJbiYjwlWHx+X/u44V06QHql7+Q7wYdsYYhGFAGuJQMzAmTgqRLVYPo8N3j5agpSf8krEq8jKW6UZJU6PindwiGy7meuuO35jn6Dbi+ryaK+FS9L0Ahxnr5sMHtJsgGcY7catmpW/1YbwwQD5Sfb09n6V3qXvsXykVhnl7+KHvQQNhbBJCGdNsrTvJdHBxrdu6u7Xah+wjX7N8XI6VIJXaVzMtkaoAJcg+On1mrbsELmRXprcbgi5VZtl99wtv25lWSInmTvwgnLTVCpcspbJM1BX7XywZbyiWiNNn74N5nHahKrjV1iDJqUgh9NayZYORX207Lwc9Fe4qwi0Y9ZR7NFVP5v+MtGM3/s56IxJ83GFrem4/osX3jyXXSEwlyHokm9eMXETalWuP/GF8RXD3sGNtVlvSIG/9o/AHb9ds2fyR7dIl8L+2mFyNjmxYbIsVB9FlpF44C4Xf wd7k3UnD 0ExI6xPL4hCa702Ez4q9PJvGoY3AIJzrjFKGGaNU37HSzijuvrDBPL36iF7XBfravrdY2hUkIwk1fOY03KOz2aCuOUDhT4Z+C4hxu4pifAPqXS3lWIwClJJCHADz8P2wdqkuMeHvERJOotoQofVCNmFsynNJgmrX7IpmaONKY+cwgHmBlX0A6OvJXaQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This change enables processes running on any logical core on a NUMA node to use all the IAA devices enabled on that NUMA node for compress jobs. In other words, compressions originating from any process in a node will be distributed in round-robin manner to the available IAA devices on the same socket. The main premise behind this change is to make sure that no compress engines on any IAA device are left un-utilized/under-utilized. In other words, the compress engines on all IAA devices are considered a global resource for that socket. This allows the use of all IAA devices present in a given NUMA node for (batched) compressions originating from zswap/zram, from all cores on this node. A new per-cpu "global_wq_table" implements this in the iaa_crypto driver. We can think of the global WQ per IAA as a WQ to which all cores on that socket can submit compress jobs. To avail of this feature, the user must configure 2 WQs per IAA in order to enable distribution of compress jobs to multiple IAA devices. Each IAA will have 2 WQs: wq.0 (local WQ): Used for decompress jobs from cores mapped by the cpu_to_iaa() "even balancing of logical cores to IAA devices" algorithm. wq.1 (global WQ): Used for compress jobs from *all* logical cores on that socket. The iaa_crypto driver will place all global WQs from all same-socket IAA devices in the global_wq_table per cpu on that socket. When the driver receives a compress job, it will lookup the "next" global WQ in the cpu's global_wq_table to submit the descriptor. The starting wq in the global_wq_table for each cpu is the global wq associated with the IAA nearest to it, so that we stagger the starting global wq for each process. This results in very uniform usage of all IAAs for compress jobs. Two new driver module parameters are added for this feature: g_wqs_per_iaa (default 1): /sys/bus/dsa/drivers/crypto/g_wqs_per_iaa This represents the number of global WQs that can be configured per IAA device. The default is 1, and is the recommended setting to enable the use of this feature once the user configures 2 WQs per IAA using higher level scripts as described in Documentation/driver-api/crypto/iaa/iaa-crypto.rst. g_consec_descs_per_gwq (default 1): /sys/bus/dsa/drivers/crypto/g_consec_descs_per_gwq This represents the number of consecutive compress jobs that will be submitted to the same global WQ (i.e. to the same IAA device) from a given core, before moving to the next global WQ. The default is 1, which is also the recommended setting to avail of this feature. The decompress jobs from any core will be sent to the "local" IAA, namely the one that the driver assigns with the cpu_to_iaa() mapping algorithm that evenly balances the assignment of logical cores to IAA devices on a NUMA node. On a 2-socket Sapphire Rapids server where each socket has 56 cores and 4 IAA devices, this is how the compress/decompress jobs will be mapped when the user configures 2 WQs per IAA device (which implies wq.1 will be added to the global WQ table for each logical core on that NUMA node): lscpu|grep NUMA NUMA node(s): 2 NUMA node0 CPU(s): 0-55,112-167 NUMA node1 CPU(s): 56-111,168-223 Compress jobs: -------------- NUMA node 0: All cpus (0-55,112-167) can send compress jobs to all IAA devices on the socket (iax1/iax3/iax5/iax7) in round-robin manner: iaa iax1 iax3 iax5 iax7 NUMA node 1: All cpus (56-111,168-223) can send compress jobs to all IAA devices on the socket (iax9/iax11/iax13/iax15) in round-robin manner: iaa iax9 iax11 iax13 iax15 Decompress jobs: ---------------- NUMA node 0: cpu 0-13,112-125 14-27,126-139 28-41,140-153 42-55,154-167 iaa iax1 iax3 iax5 iax7 NUMA node 1: cpu 56-69,168-181 70-83,182-195 84-97,196-209 98-111,210-223 iaa iax9 iax11 iax13 iax15 Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 305 ++++++++++++++++++++- 1 file changed, 290 insertions(+), 15 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index a12a8f9caa84..ca0a71b8f31d 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -29,14 +29,23 @@ static unsigned int nr_iaa; static unsigned int nr_cpus; static unsigned int nr_nodes; static unsigned int nr_cpus_per_node; - /* Number of physical cpus sharing each iaa instance */ static unsigned int cpus_per_iaa; static struct crypto_comp *deflate_generic_tfm; /* Per-cpu lookup table for balanced wqs */ -static struct wq_table_entry __percpu *wq_table; +static struct wq_table_entry __percpu *wq_table = NULL; + +/* Per-cpu lookup table for global wqs shared by all cpus. */ +static struct wq_table_entry __percpu *global_wq_table = NULL; + +/* + * Per-cpu counter of consecutive descriptors allocated to + * the same wq in the global_wq_table, so that we know + * when to switch to the next wq in the global_wq_table. + */ +static int __percpu *num_consec_descs_per_wq = NULL; static struct idxd_wq *wq_table_next_wq(int cpu) { @@ -104,26 +113,68 @@ static void wq_table_add(int cpu, struct idxd_wq *wq) entry->wqs[entry->n_wqs++] = wq; - pr_debug("%s: added iaa wq %d.%d to idx %d of cpu %d\n", __func__, - entry->wqs[entry->n_wqs - 1]->idxd->id, - entry->wqs[entry->n_wqs - 1]->id, entry->n_wqs - 1, cpu); + pr_debug("%s: added iaa local wq %d.%d to idx %d of cpu %d\n", __func__, + entry->wqs[entry->n_wqs - 1]->idxd->id, + entry->wqs[entry->n_wqs - 1]->id, entry->n_wqs - 1, cpu); +} + +static void global_wq_table_add(int cpu, struct idxd_wq *wq) +{ + struct wq_table_entry *entry = per_cpu_ptr(global_wq_table, cpu); + + if (WARN_ON(entry->n_wqs == entry->max_wqs)) + return; + + entry->wqs[entry->n_wqs++] = wq; + + pr_debug("%s: added iaa global wq %d.%d to idx %d of cpu %d\n", __func__, + entry->wqs[entry->n_wqs - 1]->idxd->id, + entry->wqs[entry->n_wqs - 1]->id, entry->n_wqs - 1, cpu); +} + +static void global_wq_table_set_start_wq(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(global_wq_table, cpu); + int start_wq = (entry->n_wqs / nr_iaa) * cpu_to_iaa(cpu); + + if ((start_wq >= 0) && (start_wq < entry->n_wqs)) + entry->cur_wq = start_wq; } static void wq_table_free_entry(int cpu) { struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); - kfree(entry->wqs); - memset(entry, 0, sizeof(*entry)); + if (entry) { + kfree(entry->wqs); + memset(entry, 0, sizeof(*entry)); + } + + entry = per_cpu_ptr(global_wq_table, cpu); + + if (entry) { + kfree(entry->wqs); + memset(entry, 0, sizeof(*entry)); + } } static void wq_table_clear_entry(int cpu) { struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); - entry->n_wqs = 0; - entry->cur_wq = 0; - memset(entry->wqs, 0, entry->max_wqs * sizeof(struct idxd_wq *)); + if (entry) { + entry->n_wqs = 0; + entry->cur_wq = 0; + memset(entry->wqs, 0, entry->max_wqs * sizeof(struct idxd_wq *)); + } + + entry = per_cpu_ptr(global_wq_table, cpu); + + if (entry) { + entry->n_wqs = 0; + entry->cur_wq = 0; + memset(entry->wqs, 0, entry->max_wqs * sizeof(struct idxd_wq *)); + } } LIST_HEAD(iaa_devices); @@ -163,6 +214,70 @@ static ssize_t verify_compress_store(struct device_driver *driver, } static DRIVER_ATTR_RW(verify_compress); +/* Number of global wqs per iaa*/ +static int g_wqs_per_iaa = 1; + +static ssize_t g_wqs_per_iaa_show(struct device_driver *driver, char *buf) +{ + return sprintf(buf, "%d\n", g_wqs_per_iaa); +} + +static ssize_t g_wqs_per_iaa_store(struct device_driver *driver, + const char *buf, size_t count) +{ + int ret = -EBUSY; + + mutex_lock(&iaa_devices_lock); + + if (iaa_crypto_enabled) + goto out; + + ret = kstrtoint(buf, 10, &g_wqs_per_iaa); + if (ret) + goto out; + + ret = count; +out: + mutex_unlock(&iaa_devices_lock); + + return ret; +} +static DRIVER_ATTR_RW(g_wqs_per_iaa); + +/* + * Number of consecutive descriptors to allocate from a + * given global wq before switching to the next wq in + * the global_wq_table. + */ +static int g_consec_descs_per_gwq = 1; + +static ssize_t g_consec_descs_per_gwq_show(struct device_driver *driver, char *buf) +{ + return sprintf(buf, "%d\n", g_consec_descs_per_gwq); +} + +static ssize_t g_consec_descs_per_gwq_store(struct device_driver *driver, + const char *buf, size_t count) +{ + int ret = -EBUSY; + + mutex_lock(&iaa_devices_lock); + + if (iaa_crypto_enabled) + goto out; + + ret = kstrtoint(buf, 10, &g_consec_descs_per_gwq); + if (ret) + goto out; + + ret = count; +out: + mutex_unlock(&iaa_devices_lock); + + return ret; +} +static DRIVER_ATTR_RW(g_consec_descs_per_gwq); + /* * The iaa crypto driver supports three 'sync' methods determining how * compressions and decompressions are performed: @@ -751,7 +866,20 @@ static void free_wq_table(void) for (cpu = 0; cpu < nr_cpus; cpu++) wq_table_free_entry(cpu); - free_percpu(wq_table); + if (wq_table) { + free_percpu(wq_table); + wq_table = NULL; + } + + if (global_wq_table) { + free_percpu(global_wq_table); + global_wq_table = NULL; + } + + if (num_consec_descs_per_wq) { + free_percpu(num_consec_descs_per_wq); + num_consec_descs_per_wq = NULL; + } pr_debug("freed wq table\n"); } @@ -774,6 +902,38 @@ static int alloc_wq_table(int max_wqs) } entry->max_wqs = max_wqs; + entry->n_wqs = 0; + entry->cur_wq = 0; + } + + global_wq_table = alloc_percpu(struct wq_table_entry); + if (!global_wq_table) { + free_wq_table(); + return -ENOMEM; + } + + for (cpu = 0; cpu < nr_cpus; cpu++) { + entry = per_cpu_ptr(global_wq_table, cpu); + entry->wqs = kzalloc(GFP_KERNEL, max_wqs * sizeof(struct wq *)); + if (!entry->wqs) { + free_wq_table(); + return -ENOMEM; + } + + entry->max_wqs = max_wqs; + entry->n_wqs = 0; + entry->cur_wq = 0; + } + + num_consec_descs_per_wq = alloc_percpu(int); + if (!num_consec_descs_per_wq) { + free_wq_table(); + return -ENOMEM; + } + + for (cpu = 0; cpu < nr_cpus; cpu++) { + int *num_consec_descs = per_cpu_ptr(num_consec_descs_per_wq, cpu); + *num_consec_descs = 0; } pr_debug("initialized wq table\n"); @@ -912,9 +1072,14 @@ static int wq_table_add_wqs(int iaa, int cpu) } list_for_each_entry(iaa_wq, &found_device->wqs, list) { - wq_table_add(cpu, iaa_wq->wq); + + if (((found_device->n_wq - g_wqs_per_iaa) < 1) || + (n_wqs_added < (found_device->n_wq - g_wqs_per_iaa))) { + wq_table_add(cpu, iaa_wq->wq); + } + pr_debug("rebalance: added wq for cpu=%d: iaa wq %d.%d\n", - cpu, iaa_wq->wq->idxd->id, iaa_wq->wq->id); + cpu, iaa_wq->wq->idxd->id, iaa_wq->wq->id); n_wqs_added++; } @@ -927,6 +1092,63 @@ static int wq_table_add_wqs(int iaa, int cpu) return ret; } +static int global_wq_table_add_wqs(void) +{ + struct iaa_device *iaa_device; + int ret = 0, n_wqs_added; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + int cpu, node, node_of_cpu = -1; + + for (cpu = 0; cpu < nr_cpus; cpu++) { + +#ifdef CONFIG_NUMA + node_of_cpu = -1; + for_each_online_node(node) { + const struct cpumask *node_cpus; + node_cpus = cpumask_of_node(node); + if (!cpumask_test_cpu(cpu, node_cpus)) + continue; + node_of_cpu = node; + break; + } +#endif + list_for_each_entry(iaa_device, &iaa_devices, list) { + idxd = iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + +#ifdef CONFIG_NUMA + if (dev && (node_of_cpu != dev->numa_node)) + continue; +#endif + + if (iaa_device->n_wq <= g_wqs_per_iaa) + continue; + + n_wqs_added = 0; + + list_for_each_entry(iaa_wq, &iaa_device->wqs, list) { + + if (n_wqs_added < (iaa_device->n_wq - g_wqs_per_iaa)) { + n_wqs_added++; + } + else { + global_wq_table_add(cpu, iaa_wq->wq); + pr_debug("rebalance: added global wq for cpu=%d: iaa wq %d.%d\n", + cpu, iaa_wq->wq->idxd->id, iaa_wq->wq->id); + } + } + } + + global_wq_table_set_start_wq(cpu); + } + + return ret; +} + /* * Rebalance the wq table so that given a cpu, it's easy to find the * closest IAA instance. The idea is to try to choose the most @@ -961,6 +1183,7 @@ static void rebalance_wq_table(void) } pr_debug("Finished rebalance local wqs."); + global_wq_table_add_wqs(); } static inline int check_completion(struct device *dev, @@ -1509,6 +1732,27 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, goto out; } +/* + * Caller should make sure to call only if the + * per_cpu_ptr "global_wq_table" is non-NULL + * and has at least one wq configured. + */ +static struct idxd_wq *global_wq_table_next_wq(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(global_wq_table, cpu); + int *num_consec_descs = per_cpu_ptr(num_consec_descs_per_wq, cpu); + + if ((*num_consec_descs) == g_consec_descs_per_gwq) { + if (++entry->cur_wq >= entry->n_wqs) + entry->cur_wq = 0; + *num_consec_descs = 0; + } + + ++(*num_consec_descs); + + return entry->wqs[entry->cur_wq]; +} + static int iaa_comp_acompress(struct acomp_req *req) { struct iaa_compression_ctx *compression_ctx; @@ -1521,6 +1765,7 @@ static int iaa_comp_acompress(struct acomp_req *req) struct idxd_wq *wq; struct device *dev; int order = -1; + struct wq_table_entry *entry; compression_ctx = crypto_tfm_ctx(tfm); @@ -1539,8 +1784,15 @@ static int iaa_comp_acompress(struct acomp_req *req) disable_async = true; cpu = get_cpu(); - wq = wq_table_next_wq(cpu); + entry = per_cpu_ptr(global_wq_table, cpu); + + if (!entry || entry->n_wqs == 0) { + wq = wq_table_next_wq(cpu); + } else { + wq = global_wq_table_next_wq(cpu); + } put_cpu(); + if (!wq) { pr_debug("no wq configured for cpu=%d\n", cpu); return -ENODEV; @@ -2393,13 +2645,32 @@ static int __init iaa_crypto_init_module(void) goto err_sync_attr_create; } + ret = driver_create_file(&iaa_crypto_driver.drv, + &driver_attr_g_wqs_per_iaa); + if (ret) { + pr_debug("IAA g_wqs_per_iaa attr creation failed\n"); + goto err_g_wqs_per_iaa_attr_create; + } + + ret = driver_create_file(&iaa_crypto_driver.drv, + &driver_attr_g_consec_descs_per_gwq); + if (ret) { + pr_debug("IAA g_consec_descs_per_gwq attr creation failed\n"); + goto err_g_consec_descs_per_gwq_attr_create; + } + if (iaa_crypto_debugfs_init()) pr_warn("debugfs init failed, stats not available\n"); pr_debug("initialized\n"); out: return ret; - +err_g_consec_descs_per_gwq_attr_create: + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_g_wqs_per_iaa); +err_g_wqs_per_iaa_attr_create: + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_sync_mode); err_sync_attr_create: driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_verify_compress); @@ -2423,6 +2694,10 @@ static void __exit iaa_crypto_cleanup_module(void) &driver_attr_sync_mode); driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_verify_compress); + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_g_wqs_per_iaa); + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_g_consec_descs_per_gwq); idxd_driver_unregister(&iaa_crypto_driver); iaa_aecs_cleanup_fixed(); crypto_free_comp(deflate_generic_tfm); From patchwork Sun Nov 3 03:21:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48ED7E6780A for ; Sun, 3 Nov 2024 03:21:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D13F6B0093; Sat, 2 Nov 2024 23:21:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 047B56B0095; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C96846B0093; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8632B6B0095 for ; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 47247812EF for ; Sun, 3 Nov 2024 03:21:23 +0000 (UTC) X-FDA: 82743332280.21.61B7B92 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf12.hostedemail.com (Postfix) with ESMTP id 608CD40005 for ; Sun, 3 Nov 2024 03:21:07 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VAKRaRP+; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf12.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730603960; a=rsa-sha256; cv=none; b=XQW/qkisoxp7ECgPZfBlV/wUIOoESi93WNWjm7kKbf35yqFcWTe55vMqZ2rDCYa/ngsnkM Y4kgJ9PE0dfZ7X0ivszLnonaf6v5fyfrNA9h5d7bZgsAj8Js4x+AM1CxGlqzv9ZYUTjIGB xzy3EEut2kwDotgksKPaVgVnIBaUZgY= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VAKRaRP+; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf12.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730603960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Qxz8u9Shn/SiPqBxAwwlyMbQX8CsM2ppl302cd3vziA=; b=7WOI7SlJc6i9goQmgUBelSIeUTFnlRSmLh9WYWp1b1Cs1lNT+PPOz5e042BoaodbjemYgK 7Gw6b9xeq3BwAT1N41L0048/90hAjqMMsZdWnMk/qiKJONddiKnpbirEu6HDrp3psaswzK qwbdbOh6iMahZUP3AL73VJ+sP9eJAog= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604081; x=1762140081; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=e3w3jsrwSRn0skEK+1OKRkuNEeuFjmM/VVAG1YViexY=; b=VAKRaRP+8Ee/4avyfGGba45eHC14TsWm4gBWRvwiD79KcANRXcr7nNrv NeYJQGmjIPk04+GnXnaIexg2BkItBk6Pb7oLToBRiMMvhlrFbV5Jf7e2L YHADuAiISrlmg5MQohOdjTxyCxIvPOjjq4EzW76OzvzJ7ivHExlUTDlrU COehzVp7C2WKv8sVHKqBIRRAtnulCejsiyHMnzuNAKogNuzryevqmiwwT HUuaLs+8HnrnRgwcyVPjUg7XYt1U3NPT63ibpkHYLfMyL+UPpavrBDdBn qCG5A0xcHu77FvTNjOn3Z3ZC7iFpRcuVfkdMg8j5XV7VipyCrEvn5LfDl g==; X-CSE-ConnectionGUID: LK5ZjHL8Q3qIjgkjy0GvEg== X-CSE-MsgGUID: KZTgbbjCQu+y/g7ltd95eA== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548298" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548298" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:13 -0700 X-CSE-ConnectionGUID: PVoUksXZR4mAcYTV3a/VVg== X-CSE-MsgGUID: azt2Ht8kSBu5utEhSx2oiA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451891" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:14 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 08/13] mm: zswap: acomp_ctx mutex lock/unlock optimizations. Date: Sat, 2 Nov 2024 20:21:06 -0700 Message-Id: <20241103032111.333282-9-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 608CD40005 X-Stat-Signature: 5do7p5w647r9x8b34zpouipcwf3qpxet X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1730604067-366890 X-HE-Meta: U2FsdGVkX1+lOV9AP8h1au7wIPp/dpGBLJlp8ID4+PFjkhUzxFuG9JwybQ76opLi+OJNmlwosdxe1UKupDcdHfFu+9kLw5L5B0oxZlOfoCHoUzd8hS9S/F8k/qTfQb19Oqni4W2WfDGhviMttIubt6InFjYWd8g9GN63oTxjUxSTazdh8Tlaa6oPD0S4gFFsDs5RBIgyJh0PFC8s00QolfoeFZgTvFcp9CP0HfZUzbfF6D8vuROXl9wdXz7yqcKF5zgMBT9WQfG2VY8/OJ1lpjnDWFO3YozefimNxj8ErykXYgIRWtW1dYptN15h1adF0yTHyOVjUpjP6/WIGIYeOlaggCxDqSXdKFvgl6l+V3BNKmbCKUT+5pvjkLha4k/sM+4TW1ObSUY0Sx5vAs47BTOgRzdPL6qBrYwPq8P5LwmCV9gF+jzJOLXeLFLRdnzG3qYCxq11JVB+WoFe8C/iEkazyuS+6Um00bXTAEgzVFMm7ZB2rkyeRRdNF0EwMi3uMb5mzdf6dbIafXICiBqRcYUEr5a+x+BrwC+ZDPPs1BiOHv7MEJpoHAdr7udBufqvOBaYMmhoNuWMZw0sMSgPUpejSA/cRDTW6hfh/fdJd3dv34vI5f2/eU6dzE/JSIm7y6Ts3dEjdFP4Li1KmF2FJClShODUyRA2uGH9y3i8g7Gt2WDTSPVG6KCyjpzYnvNvWc6wlTEOFAVn7//5lHa2cOGj+QmzYrpJC7sBgSkz/hnCYrW2suMhrmcjMKWoHTDYjbi+E8RnO6HcAKwF+EGstzAYIACsRVvQ6buxx+z7IBENiUebv5nhD97lA7G0GZ4w8N9HMQDsN+J6HOFEADgCTgcqSOKRLwN1oRhsDAYxSyFjcRHzCVw80G9hehR4D3vkt4dKJQrw1YMOr8nJkOH77r2/lbUiu0rO8F+do2d+K4T3RBeoyqwvrQXPz0cQMXEHmkSrfbbG9249xjpRKkM yXcBI3/5 Hikar3s20izycg8kVzazJ99G1sx5ETmrT37ckZyORxWWQW2M1oN6CNsGyLRui6+aEqbij8cQ6P5nMOae8dxDEzQkxtAaSJiN51UVLWPaYiH5ODtXTKdckWT3JLMp5Rq4Ad73e/FDlQB0BO8FvTqBPC/KB8XFeKriYIUCLE85xziJuF5MnvCNJfHAy6S2hXBV0cb2lU0Va63PeGkrGY60s/nTgX1lozyAz6F/h3HALDoaqssdgL/zTJ80+qk5f6d5FabiyOnX/AfMMpG0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch implements two changes with respect to the acomp_ctx mutex lock: 1) The mutex lock is not acquired/released in zswap_compress(). Instead, zswap_store() acquires the mutex lock once before compressing each page in a large folio, and releases the lock once all pages in the folio have been compressed. This should reduce some compute cycles in case of large folio stores. 2) In zswap_decompress(), the mutex lock is released after the conditional zpool_unmap_handle() based on "src != acomp_ctx->buffer" rather than before. This ensures that the value of "src" obtained earlier does not change. If the mutex lock is released before the comparison of "src" it is possible that another call to reclaim by the same process could obtain the mutex lock and over-write the value of "src". Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index f6316b66fb23..3e899fa61445 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -880,6 +880,9 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) return 0; } +/* + * The acomp_ctx->mutex must be locked/unlocked in the calling procedure. + */ static bool zswap_compress(struct page *page, struct zswap_entry *entry, struct zswap_pool *pool) { @@ -895,8 +898,6 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); - mutex_lock(&acomp_ctx->mutex); - dst = acomp_ctx->buffer; sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); @@ -949,7 +950,6 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, else if (alloc_ret) zswap_reject_alloc_fail++; - mutex_unlock(&acomp_ctx->mutex); return comp_ret == 0 && alloc_ret == 0; } @@ -986,10 +986,16 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); - mutex_unlock(&acomp_ctx->mutex); if (src != acomp_ctx->buffer) zpool_unmap_handle(zpool, entry->handle); + + /* + * It is safer to unlock the mutex after the check for + * "src != acomp_ctx->buffer" so that the value of "src" + * does not change. + */ + mutex_unlock(&acomp_ctx->mutex); } /********************************* @@ -1487,6 +1493,7 @@ bool zswap_store(struct folio *folio) { long nr_pages = folio_nr_pages(folio); swp_entry_t swp = folio->swap; + struct crypto_acomp_ctx *acomp_ctx; struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg = NULL; struct zswap_pool *pool; @@ -1526,6 +1533,9 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } + acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); + mutex_lock(&acomp_ctx->mutex); + for (index = 0; index < nr_pages; ++index) { struct page *page = folio_page(folio, index); ssize_t bytes; @@ -1547,6 +1557,7 @@ bool zswap_store(struct folio *folio) ret = true; put_pool: + mutex_unlock(&acomp_ctx->mutex); zswap_pool_put(pool); put_objcg: obj_cgroup_put(objcg); From patchwork Sun Nov 3 03:21:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860294 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2C21E6780A for ; Sun, 3 Nov 2024 03:21:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC9846B0096; Sat, 2 Nov 2024 23:21:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C52C96B009B; Sat, 2 Nov 2024 23:21:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A265E6B0098; Sat, 2 Nov 2024 23:21:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 79BA26B0095 for ; Sat, 2 Nov 2024 23:21:25 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3A6F881421 for ; Sun, 3 Nov 2024 03:21:25 +0000 (UTC) X-FDA: 82743333036.14.4FF2F6A Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf17.hostedemail.com (Postfix) with ESMTP id B2B564000E for ; Sun, 3 Nov 2024 03:20:59 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=W46yiSwj; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf17.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730604001; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uYIX/zTFwXg0bJkJ5zykysmMvate0NYv60i9B0H2JFE=; b=QQ3tinYPzQYhRnpqsnnWKTPvpUtAII304+BiN3FI70IpIKBGrkcsCzspGBmspLD8JTLqwd HStO4zVM2ymcwqCM6qgReU6zhz7lOXRQcDdAdOBskybE0Uh+nRw9R2GQCWlz979OaF4773 zNNU2v7JktyZeVHnK1zSb+3obc4jX5M= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=W46yiSwj; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf17.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730604001; a=rsa-sha256; cv=none; b=qnPlRUUTAamXIoRgbgYxpIGBS4zq6yQBqVD3TbuhgEDvVVOWadNq44b5TmH3xlM05hSCqN Wq9ZD71HZxrb5tRjcRwHCeiFC9AYU8aHqKptZ57DXy20Fm26goCcboTPaDnCxV3/Z+VTKg wKKnQfhBe3Qg7E6j7m+th42M5GzAvQE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604083; x=1762140083; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TyxYuGZ5om7/o45C+jhhBFW2E7GGaioFiNoenmMyD78=; b=W46yiSwjjmVfkYf4hcC2A4grl024bvGv2TBpSz6pLr+biJtJMTdp4czC rZQIHTc9cXDfFIh1JJB+ygJ9Rae/PGAvIE3Zx3IViChn0NVpmm0hWXyam C5fqaUzlMEm8cV90goAA9cjcJMr3WpUhJ31FS3x2/OHRkOKJbCXeXA4dt EGrj3O4aeetWdthO8YzWVC/RpRPQBc+P6Q0hF2xeQ2J/UzP7C5/+Kmu1k LMV7U6lZSR65AHBIzcQ+R36uLDbiaVce5z8QOM+Ge5N9QP3/q3xdPgFPs YphCQ/gqJ85if0JMedLy3Jkualb+hV8dX+2cTcnXSdlqtUprfPXp78UWt g==; X-CSE-ConnectionGUID: /kN/9EwcRGCGMg/Fuy/slg== X-CSE-MsgGUID: qZIyF2hhSo2nCK5Jargnkg== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548310" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548310" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:13 -0700 X-CSE-ConnectionGUID: UNQfB8L7Tt2fD3TR9VFxkw== X-CSE-MsgGUID: MNRAQAguTgm9nxQ6pxS+7A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451894" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:14 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 09/13] mm: zswap: Modify struct crypto_acomp_ctx to be configurable in nr of acomp_reqs. Date: Sat, 2 Nov 2024 20:21:07 -0700 Message-Id: <20241103032111.333282-10-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: B2B564000E X-Stat-Signature: jgwsfhh98rfdf8y9gkucnnp1xxhkp3fw X-HE-Tag: 1730604059-982735 X-HE-Meta: U2FsdGVkX1/Qy8cVnGa7WhryIUAemL6I0+dkQaXQ7wyKT0G/lKMlIHGmLRVUuBC657WOaVgNKgQk19gLa/s4FT533F2vVuIMDnpmS5Gn1zA6Y2oHajYCaZec40tPeO3gjov4F0oSsp6TqCTd1pYY7kvZXZM/Dxm6d26SHQxHhsfOY/BnKUOhh9wukHGDiJjhXmEsknrxqW/4daNqHALcOVdnBQohX8IkdPQrQ3D9hZ8+4VfV5GMqQIUx5RE3tCKVlF3BafdkWV2m9bV6tMMXB48cdMBKnUzs5gIQj4vbL5SegPAOnmwNi/a6YkQw/ArpJ0YOfI4pp8qA7UFTGGusFXF2cj0dr85GK7BgBTMToaa80pe0FmiNMuRBgUtNAOm+23odPZm8+W6mHLsRWAF154ZLb9pE7USOghQEjVUpZHOeePs5kfEMou56MFSa+SAXmpHgIn+F2c+IKPEbLW0rmsaC9sBNnshcnXxSotAnp/irbaKlLdj9y685w1Xv3a40Oi9tcyBBR9ACqdkojS1rAdWH2Q/qxvP8Vw1uLOOqj1G2NEAj2Qi9bD7/NnJDdI4EUQE1KjWg3VYRnr2ryLAF/Okmb9VfeT0DA/NDYZKD0YpgGXL/YzfnNTgeKde29djk9SMiTxhd+nUtRqK2jc1QupQI6Jsxr5fYHAZCsjwYhw2+edeiGM8WgBkWqCPm0cOeeTyPu/Z9n+UM5R5Z73uH9113qBlFd8xWZHWrRGYLViASB0vT0YU9Ce/JWKjddfwiwHSd7hUMOQkZ10Sz+WaB+Vhp/Rl2DChylrJyCqi7bqd5B4i18qHf5ziWKBYsXiwf5JD3SvxdcCyDbiGQGNovm593lec7PWM4lzULkUOMrRoD5I/0xIGUQ/pymO+4sDdBc3GukkeOF6bEaLxIgIS6STDTfPX/LOOo3D6VxqSdTtSHngIp+4MpDUyn4n4TuJ0HhW5InAxUPrbprAPL+Sm 2p5Cc8xg H0NDPkWoXP4V/tY8WyDIQLMBAARzE4OIM/9yTBCPHmIYIDfD9JzAazHelSoZlCYiRikxBZ0/qI74fB5ma+MUYd89kAnSEzXL1T84yqvecjGfRYP3Hzlvu14IujroaZBEMoRHmf3B463M+FYfQsxyNfnEJmbiD4vAPQbQ2WytkKpN+h2e9AKpJzMtUDrpcdnmw+3B6jSwEeT569T4UIDtd1QOu6UvOY3RWzhUy8XKy8oY64eI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Modified the definition of "struct crypto_acomp_ctx" to represent a configurable number of acomp_reqs and the required number of buffers. Accordingly, refactored the code that allocates/deallocates the acomp_ctx resources, so that it can be called to create a regular acomp_ctx with exactly one acomp_req/buffer, for use in the the existing non-batching zswap_store(), as well as to create a separate "batching acomp_ctx" with multiple acomp_reqs/buffers for IAA compress batching. Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 145 +++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 102 insertions(+), 43 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 3e899fa61445..f062c6dfcad4 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -143,9 +143,10 @@ bool zswap_never_enabled(void) struct crypto_acomp_ctx { struct crypto_acomp *acomp; - struct acomp_req *req; + struct acomp_req **reqs; + u8 **buffers; + unsigned int nr_reqs; struct crypto_wait wait; - u8 *buffer; struct mutex mutex; bool is_sleepable; }; @@ -241,6 +242,11 @@ static inline struct xarray *swap_zswap_tree(swp_entry_t swp) pr_debug("%s pool %s/%s\n", msg, (p)->tfm_name, \ zpool_get_type((p)->zpool)) +static int zswap_create_acomp_ctx(unsigned int cpu, + struct crypto_acomp_ctx *acomp_ctx, + char *tfm_name, + unsigned int nr_reqs); + /********************************* * pool functions **********************************/ @@ -813,69 +819,122 @@ static void zswap_entry_free(struct zswap_entry *entry) /********************************* * compressed storage functions **********************************/ -static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) +static int zswap_create_acomp_ctx(unsigned int cpu, + struct crypto_acomp_ctx *acomp_ctx, + char *tfm_name, + unsigned int nr_reqs) { - struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); - struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); struct crypto_acomp *acomp; - struct acomp_req *req; - int ret; + int ret = -ENOMEM; + int i, j; + acomp_ctx->nr_reqs = 0; mutex_init(&acomp_ctx->mutex); - acomp_ctx->buffer = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); - if (!acomp_ctx->buffer) - return -ENOMEM; - - acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu)); + acomp = crypto_alloc_acomp_node(tfm_name, 0, 0, cpu_to_node(cpu)); if (IS_ERR(acomp)) { pr_err("could not alloc crypto acomp %s : %ld\n", - pool->tfm_name, PTR_ERR(acomp)); - ret = PTR_ERR(acomp); - goto acomp_fail; + tfm_name, PTR_ERR(acomp)); + return PTR_ERR(acomp); } + acomp_ctx->acomp = acomp; acomp_ctx->is_sleepable = acomp_is_async(acomp); - req = acomp_request_alloc(acomp_ctx->acomp); - if (!req) { - pr_err("could not alloc crypto acomp_request %s\n", - pool->tfm_name); - ret = -ENOMEM; - goto req_fail; + acomp_ctx->buffers = kmalloc_node(nr_reqs * sizeof(u8 *), + GFP_KERNEL, cpu_to_node(cpu)); + for (i = 0; i < nr_reqs; ++i) { + acomp_ctx->buffers[i] = kmalloc_node(PAGE_SIZE * 2, + GFP_KERNEL, cpu_to_node(cpu)); + if (!acomp_ctx->buffers[i]) { + for (j = 0; j < i; ++j) + kfree(acomp_ctx->buffers[j]); + kfree(acomp_ctx->buffers); + ret = -ENOMEM; + goto buf_fail; + } } - acomp_ctx->req = req; + acomp_ctx->reqs = kmalloc_node(nr_reqs * sizeof(struct acomp_req *), + GFP_KERNEL, cpu_to_node(cpu)); + for (i = 0; i < nr_reqs; ++i) { + acomp_ctx->reqs[i] = acomp_request_alloc(acomp_ctx->acomp); + if (!acomp_ctx->reqs[i]) { + pr_err("could not alloc crypto acomp_request reqs[%d] %s\n", + i, tfm_name); + for (j = 0; j < i; ++j) + acomp_request_free(acomp_ctx->reqs[j]); + kfree(acomp_ctx->reqs); + ret = -ENOMEM; + goto req_fail; + } + } + + /* + * The crypto_wait is used only in fully synchronous, i.e., with scomp + * or non-poll mode of acomp, hence there is only one "wait" per + * acomp_ctx, with callback set to reqs[0], under the assumption that + * there is at least 1 request per acomp_ctx. + */ crypto_init_wait(&acomp_ctx->wait); /* * if the backend of acomp is async zip, crypto_req_done() will wakeup * crypto_wait_req(); if the backend of acomp is scomp, the callback * won't be called, crypto_wait_req() will return without blocking. */ - acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, + acomp_request_set_callback(acomp_ctx->reqs[0], CRYPTO_TFM_REQ_MAY_BACKLOG, crypto_req_done, &acomp_ctx->wait); + acomp_ctx->nr_reqs = nr_reqs; return 0; req_fail: + for (i = 0; i < nr_reqs; ++i) + kfree(acomp_ctx->buffers[i]); + kfree(acomp_ctx->buffers); +buf_fail: crypto_free_acomp(acomp_ctx->acomp); -acomp_fail: - kfree(acomp_ctx->buffer); return ret; } -static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) +static void zswap_delete_acomp_ctx(struct crypto_acomp_ctx *acomp_ctx) { - struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); - struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); - if (!IS_ERR_OR_NULL(acomp_ctx)) { - if (!IS_ERR_OR_NULL(acomp_ctx->req)) - acomp_request_free(acomp_ctx->req); + int i; + + for (i = 0; i < acomp_ctx->nr_reqs; ++i) + if (!IS_ERR_OR_NULL(acomp_ctx->reqs[i])) + acomp_request_free(acomp_ctx->reqs[i]); + kfree(acomp_ctx->reqs); + + for (i = 0; i < acomp_ctx->nr_reqs; ++i) + kfree(acomp_ctx->buffers[i]); + kfree(acomp_ctx->buffers); + if (!IS_ERR_OR_NULL(acomp_ctx->acomp)) crypto_free_acomp(acomp_ctx->acomp); - kfree(acomp_ctx->buffer); + + acomp_ctx->nr_reqs = 0; + acomp_ctx = NULL; } +} + +static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) +{ + struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); + struct crypto_acomp_ctx *acomp_ctx; + + acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); + return zswap_create_acomp_ctx(cpu, acomp_ctx, pool->tfm_name, 1); +} + +static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) +{ + struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); + struct crypto_acomp_ctx *acomp_ctx; + + acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); + zswap_delete_acomp_ctx(acomp_ctx); return 0; } @@ -898,7 +957,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); - dst = acomp_ctx->buffer; + dst = acomp_ctx->buffers[0]; sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); @@ -908,7 +967,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, * giving the dst buffer with enough length to avoid buffer overflow. */ sg_init_one(&output, dst, PAGE_SIZE * 2); - acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); + acomp_request_set_params(acomp_ctx->reqs[0], &input, &output, PAGE_SIZE, dlen); /* * it maybe looks a little bit silly that we send an asynchronous request, @@ -922,8 +981,8 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, * but in different threads running on different cpu, we have different * acomp instance, so multiple threads can do (de)compression in parallel. */ - comp_ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; + comp_ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->reqs[0]), &acomp_ctx->wait); + dlen = acomp_ctx->reqs[0]->dlen; if (comp_ret) goto unlock; @@ -975,24 +1034,24 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) */ if ((acomp_ctx->is_sleepable && !zpool_can_sleep_mapped(zpool)) || !virt_addr_valid(src)) { - memcpy(acomp_ctx->buffer, src, entry->length); - src = acomp_ctx->buffer; + memcpy(acomp_ctx->buffers[0], src, entry->length); + src = acomp_ctx->buffers[0]; zpool_unmap_handle(zpool, entry->handle); } sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); sg_set_folio(&output, folio, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); - BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); - BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); + acomp_request_set_params(acomp_ctx->reqs[0], &input, &output, entry->length, PAGE_SIZE); + BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->reqs[0]), &acomp_ctx->wait)); + BUG_ON(acomp_ctx->reqs[0]->dlen != PAGE_SIZE); - if (src != acomp_ctx->buffer) + if (src != acomp_ctx->buffers[0]) zpool_unmap_handle(zpool, entry->handle); /* * It is safer to unlock the mutex after the check for - * "src != acomp_ctx->buffer" so that the value of "src" + * "src != acomp_ctx->buffers[0]" so that the value of "src" * does not change. */ mutex_unlock(&acomp_ctx->mutex); From patchwork Sun Nov 3 03:21:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10133E6780A for ; Sun, 3 Nov 2024 03:21:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2126B6B0098; Sat, 2 Nov 2024 23:21:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A2046B009B; Sat, 2 Nov 2024 23:21:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F067D6B009C; Sat, 2 Nov 2024 23:21:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id AE5BD6B0095 for ; Sat, 2 Nov 2024 23:21:25 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 739AB80C4A for ; Sun, 3 Nov 2024 03:21:25 +0000 (UTC) X-FDA: 82743331692.09.1826F4A Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf18.hostedemail.com (Postfix) with ESMTP id EE3BB1C0002 for ; Sun, 3 Nov 2024 03:21:09 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QmOm2ZFk; spf=pass (imf18.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730604036; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rKUHjlpACeorWIyRjTReOgDpl9w8z8tHSSWJcOBHH8k=; b=WD7W3PAgQ6lIw0FTRFARSKoxdLlMU8ThPb8gTu5ZH40WBTaCaysXQNAkjBesCItK94+834 Bn7JpOGkVWZ0Mf5PBf9hfeOIzgCJLSzne1HwLD98JAjz3pZaBMhMtDFPG0wjZFstnQCNJA fjDqeuRdbCrM6yTcIR+6jaGi/Y9xFno= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730604036; a=rsa-sha256; cv=none; b=PZz8FaQyruBl13x9YKf8zKbH9Ad5aS0qUARYV5B10eWTdw8SfscI6fTgMNGsBEY28JEj2I tLBS7So1hJaQ653gWJwIuGwSwlBpmPqdz0wHQEZVwz5tq2CzxAPfY2wPAzobEAGAaWu8Q5 3T+iUnEJ174s65DjMxEOpMb71zKfoBQ= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QmOm2ZFk; spf=pass (imf18.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604083; x=1762140083; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kNwqiK541Yzu6mxMdWRHQdBvyKOFfv8Kurpx7eomG3Q=; b=QmOm2ZFkOHgBctSV2XaXNywDQF+sbR7RUYGTAYfReanGt7mZJUFDWm/G sssWRKJmdnDf95zSAl0y+UBmvEnH2/l7XyNJDf57otml+rF9kf1ynwvFB Cn9MQAPTvYO37CLm6V/k0AUbT76xlmyKGCGqP2N9x0zY7mZSIFDxTkYi8 nkhPs8ILlSha/I66CTQfQXHjeduC1KfHfWE1pmnQvftB4J7cxUPs0H/hP 0NrcQSyAmbQAPMHF9ptQxvfJU+/7+iNvb7Pa/8gMGBXQt3U1a1T4K+cut dKSRyOqGUAF1x4E4xZ35TDfURGCI2Rl6jsYNgNzFph0gKK/Rs00Kj7kPh Q==; X-CSE-ConnectionGUID: 4z63fgDUSIOCG4HFrm6rKw== X-CSE-MsgGUID: pwtQKZupTw2zqHXGpHc+fQ== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548339" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548339" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:14 -0700 X-CSE-ConnectionGUID: 9xQQE4tEQ9OqArTginJneA== X-CSE-MsgGUID: AHlf4kvqRIWS++X0cLkzHA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451899" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:14 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 10/13] mm: zswap: Add a per-cpu "acomp_batch_ctx" to struct zswap_pool. Date: Sat, 2 Nov 2024 20:21:08 -0700 Message-Id: <20241103032111.333282-11-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: ta3ptpw1pg45qyim7cexisdc5jg5hja7 X-Rspam-User: X-Rspamd-Queue-Id: EE3BB1C0002 X-Rspamd-Server: rspam02 X-HE-Tag: 1730604069-351312 X-HE-Meta: U2FsdGVkX18ZNUorjzFDmDXYw2hUNh4ItR/SFhjnXPX7mkPks0uqksU2DmjA7pDXei+gx9Owbj31yZ6qIOJVKxb/CEIFlTMnVrEh+FbwInWtwCemLp/S95uFUfeWMKn2vBYFW3r34pEbARFSA9dq+wTaXBDYiXGAuxHQ3v/zYNbdCqaH826jlJ0wmYAt3SJScx2XFTfsnvq1QwJlH9j8M+16jxpJ+GKpb+wQL7SOoBOA2q+jgdqerD28OMQZbSE7q9aDrH55Wfbfihll4VGp9zfWP7se3nGSvwmfg65LZKFzBbxRB8Ee94VGhwapIWAJnbATi31j62bPl0XSggQcVkGVyntLx7JzniYMTyxEE70XpKV3uISanWdFS4tfLmUb/yXC1lld270+VCal0AZHRPZgtHVOpKXbZVMC6HTrpQQPAyFQV4uDTH+7Q/8+pZ2eZtJtHAGsyfwvP7yIRnSjCbRSGQ22Lqy6IbBB1qO8npSfGYgBSnvbzQLa8g22fjguuoUdHKR8uV7RgI1G+YX1ZMfvgxO31QDpGF7XL1r0ZiOk/KoBdTTZFNNWA20Opy5GpqXOfBlFvTLDS6iwfAnxRBnHvA610zZ3WXUuY/COHZhl66iw7FvTi5SHIR9kZvF5YB/xddkp/NUGNX7NQplwT07p3qOj0Ao+jtqAWUqYYxeueLt/nBi2S1nSLMq0hUu1hYzXdNO9CPEE/Baxm5ucvAkyzBQDgS5lE9jFJbo5qmwGuTVF43FUi+liCtwl+2BSH5tHFCSZ8zOMEcyB34lyNqcaaASEInTpLEeC1OyN3HokRL6NDZZ47eN5EAh8gauXWLQxPJ1Th0Vk4lq6MlvKZLNoFNe5rUuUp28my1IgVvPOD5JGJYCsdmFLGnnpE4yEOhxEhIlLL1srQEkAS66aAU/N5m+mQnsIS+mcmZJX3/GOoIAZD9lwuhvp/1Hgub+PJclq7zLOiD9gU0kuMGe uQrOsW2J sFVKdZohKCGkXS5f/v1yqZ+bxED+9IN7T5d8jL66zNBY9Hc60wc+8Ipa6pPwHDNrB1zNYgyEkbyXWghmSm+o+0tl266wwpW+OTG1uKkkHvh2NhBVt0pOOM0gHXgHRMN+0r50BvUXtAEjDKG760HwhHpr4F+Lm7O7UZkBdzFP5LACCSjeVGtJssZz5gCbgbX/KPTa+6rIg2SaspVm/tzNr2KOCWp4g0Qowe8Zk93VhFYYtlacIPxqc7XE8PoCPO0k571Y9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch adds a separate per-cpu batching acomp context "acomp_batch_ctx" to the zswap_pool. The per-cpu acomp_batch_ctx pointer is allocated at pool creation time, but no per-cpu resources are allocated for it. The idea is to not incur the memory footprint cost of multiple acomp_reqs and buffers in the existing "acomp_ctx" for cases where compress batching is not possible; for instance, with software compressor algorithms, on systems without IAA, on systems with IAA that want to run the existing non-batching implementation of zswap_store() of large folios. By creating a separate acomp_batch_ctx, we have the ability to allocate additional memory per-cpu only if the zswap compressor supports batching, and if the user wants to enable the use of compress batching in zswap_store() to improve swapout performance of large folios. Suggested-by: Yosry Ahmed Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/mm/zswap.c b/mm/zswap.c index f062c6dfcad4..3ad81ec0f262 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -160,6 +160,7 @@ struct crypto_acomp_ctx { struct zswap_pool { struct zpool *zpool; struct crypto_acomp_ctx __percpu *acomp_ctx; + struct crypto_acomp_ctx __percpu *acomp_batch_ctx; struct percpu_ref ref; struct list_head list; struct work_struct release_work; @@ -287,10 +288,14 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) pool->acomp_ctx = alloc_percpu(*pool->acomp_ctx); if (!pool->acomp_ctx) { - pr_err("percpu alloc failed\n"); + pr_err("percpu acomp_ctx alloc failed\n"); goto error; } + pool->acomp_batch_ctx = alloc_percpu(*pool->acomp_batch_ctx); + if (!pool->acomp_batch_ctx) + pr_err("percpu acomp_batch_ctx alloc failed\n"); + ret = cpuhp_state_add_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); if (ret) @@ -312,6 +317,8 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) ref_fail: cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); error: + if (pool->acomp_batch_ctx) + free_percpu(pool->acomp_batch_ctx); if (pool->acomp_ctx) free_percpu(pool->acomp_ctx); if (pool->zpool) @@ -368,6 +375,8 @@ static void zswap_pool_destroy(struct zswap_pool *pool) cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); free_percpu(pool->acomp_ctx); + if (pool->acomp_batch_ctx) + free_percpu(pool->acomp_batch_ctx); zpool_destroy_pool(pool->zpool); kfree(pool); @@ -924,6 +933,11 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); struct crypto_acomp_ctx *acomp_ctx; + if (pool->acomp_batch_ctx) { + acomp_ctx = per_cpu_ptr(pool->acomp_batch_ctx, cpu); + acomp_ctx->nr_reqs = 0; + } + acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); return zswap_create_acomp_ctx(cpu, acomp_ctx, pool->tfm_name, 1); } @@ -933,6 +947,12 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); struct crypto_acomp_ctx *acomp_ctx; + if (pool->acomp_batch_ctx) { + acomp_ctx = per_cpu_ptr(pool->acomp_batch_ctx, cpu); + if (!IS_ERR_OR_NULL(acomp_ctx) && (acomp_ctx->nr_reqs > 0)) + zswap_delete_acomp_ctx(acomp_ctx); + } + acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); zswap_delete_acomp_ctx(acomp_ctx); From patchwork Sun Nov 3 03:21:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860296 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 112C7E6780A for ; Sun, 3 Nov 2024 03:21:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 680686B0095; Sat, 2 Nov 2024 23:21:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 60D7F6B009B; Sat, 2 Nov 2024 23:21:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3EED06B009D; Sat, 2 Nov 2024 23:21:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 046AA6B0095 for ; Sat, 2 Nov 2024 23:21:25 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A3542141961 for ; Sun, 3 Nov 2024 03:21:25 +0000 (UTC) X-FDA: 82743331818.01.9E0D2B1 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf12.hostedemail.com (Postfix) with ESMTP id AE8704000D for ; Sun, 3 Nov 2024 03:21:09 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=BFLR4Onp; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf12.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730603962; a=rsa-sha256; cv=none; b=iRVfHnIB2BKsNuRAHHanBfakm0BDNiurC1feidfk4/FZH6K2BpOY9Shu5TWsB9yGCkGnLR DViVsbeUtpBVD8GZuxkY2VZQ8UxzDAhLat//3TrKUaLJccYz1E0LqkY2rVx5Fbj5qIpHZd ZQfEYiB7Rq7DGuV50CzsuMWiSy3rkhM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=BFLR4Onp; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf12.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730603962; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nikya7qF4DmUxGnRrNHV5jgQ9mjh0pUcnSNk84HjAn0=; b=12XoADW2wNMmqboNjFaj3M71m+uAusx6tQkK7EaiS9MG160NUOnU2t7MkhncdnYEc0mO/1 BAKshDiN48rMDizLYv6TkNNmyuowzdWiLXw2q5wsIVJc5HZgcCSwHrDNvw++YECjjmAO2b bUQEh0hEOfxNBJ9v0WaQkybpQAPLeSU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604083; x=1762140083; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Mv1j1WkxVMtwiYbBTrEKO+5plJ/chufvUAiSaGvBrMw=; b=BFLR4OnpLP7k8H44Mnbutu69OujLwC/2Dohyw4VIC0i2uwGvnbcC8mPt 3EEBrSTOE6KvlsuQgSQZiU1Et1hfGeXtM1YynXAGcvQmk/L0w3hqKAXZv lX8EspgAcCJB0kUZb3tTfepjbPzwtiO/3KZodLIhQTZzAejWZbBEHOzKj 04J1nxtXItA2RFm/0DKE6mG806bNWdOhbHbjizHweENkWEFr2KeLmY9I3 69Oq0NDkjrhlTMtAMx9jHa3osfR5CyqSFv/PrRSRJsltMjUlB4dIaYyIo WmkJFlGL01T1aEAhTZWV/XKu6pMZJrleRMYUG5mfpb05tp1tywAtx3CzK Q==; X-CSE-ConnectionGUID: 5B/oebOvQISst28oH1EHxg== X-CSE-MsgGUID: 7qTQr5vtSFes1elvkIYmiw== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548347" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548347" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:14 -0700 X-CSE-ConnectionGUID: /wV0MMjJTpOp2HLAinTXkg== X-CSE-MsgGUID: 5gu66x9ZSmOJhWTIjpLa9g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451903" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:14 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 11/13] mm: zswap: Allocate acomp_batch_ctx resources for a given zswap_pool. Date: Sat, 2 Nov 2024 20:21:09 -0700 Message-Id: <20241103032111.333282-12-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AE8704000D X-Stat-Signature: sdqn8g1ufrziso6x6snnftg5o9g3839b X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1730604069-160506 X-HE-Meta: U2FsdGVkX1/cUJFFSWSi0ndLOv5VmykOA33xGNkyGaKYY+B1boaHzt1RmWgiVCETqwbjc/P/2nhiYxkDPwMSdEdSBWrDsyEH57BdWRKfgfDuVQiZ07De42jT/qPyufWLVa3h8jNLk3o2kfaxeafn2FWvQITmImNSwjKaTovnIwXLsDzVAJKti1CZ+e7TbhLOMxH3ZXYna+BDDS96gvs4/gEjhfbtk7bIQYqqR1C99gZiOqhM8ltkCzHYj4PthLlJmubvH3B7Q1FIWEf2A8CBol9xEgqs9RgUa7WXfjkdYYfUlzAhTlGub1G9gAm2wOY1r0rw9Bg+uo+ijxwfJVsndIb5CHe/p31OcTm3lmn5U+EUTmnMqZVknWcbn0VIpnCFtMIIgZjBC88eh1tDgexFe4sySr2yi0IEmQSwkuiQ/JXyshcn8rJlGN+cxWK4Ycr5p5imI46VoqBpfIiaUO4r1OgqBfW8oD4nYDHkC2H1VfxQgLO50jVjGD3X2LjtTOr7Nfg+y9R2e0QPavcJtN2mp+SrPs/b2SQlPonp5NwdeSjKuVja9Ty+tIsiQ+aNWCSSxUe+zzGAqMo4bGHl2DyBWNUh8gdQDTMxHT+UgQ3Nz50IyoZ25CvIShzhg1nNPGrH/8UrXg4OFbDKlBRkLdpZ+Y+9I0/3QbpKjq0/h40GBE/Mv4EY+lKALSdHz7HeqJloXf2UUs66uFRX8WMUEmbd4CpezLKZo/e8TuognErfFvU5kHpcdxNTzJmkKse92RUGG9E3RALZI9N52jX7KFbud5wnSDDMKtualZNBZFiJ5KIr3coRSfow2rNG19UaKR6iu5H8vjUVbiYbWDmgVOVNlM0BkI6xbl8UwHfNjE4tUU90g6WwxUZtJ5MQfPdMerj65sTX6NX1OI7vmwZQK+SdleaBCzhc/kHoLigSNHX9mzOGGHJWKeEn7B++CO0FplINRO5TyRfs1lIGxD5ahRh mcwi2Day rWWo4Rp7iMbNv4fNM8VSWUv+Kd+0v7PUlHhyT9RXY5qfZOUNeQUrirjGLlboTJ03/zIFrGKwP8l4tZb2znukdEhZPcZVK+KaeQKkYXkBPIucpmvR2hfwFuVpLVtjWtx0mb7/wjsbj2GuXyuI0RysVqWbIZt2jTBv9rlg05cM+xj+6fjxggumZ94mdhEQ2W1N2VTy00KTJMh7CRZgYHJnrsC8DKubTDWEbjjjtz/bzDs2/LVhJQjwDDEKZegUyTsorU/+2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If the zswap_pool is associated with an acomp_alg/crypto_acomp that has registered batch_compress() and batch_decompress() API, we can allocate the necessary batching resources for the pool's acomp_batch_ctx. This patch makes the above determination on incurring the per-cpu memory footprint cost for batching, and if so, goes ahead and allocates SWAP_CRYPTO_BATCH_SIZE (i.e. 8) acomp_reqs/buffers for the pool->acomp_batch_ctx on that specific cpu. It also "remembers" the pool's batching readiness as a result of the above, through a new enum batch_comp_status can_batch_comp; member added to struct zswap_pool, for fast retrieval during zswap_store(). This allows us a way to only incur the memory footprint cost of the pool->acomp_batch_ctx resources for a given cpu on which zswap_store() needs to process a large folio. Suggested-by: Yosry Ahmed Suggested-by: Ying Huang Signed-off-by: Kanchana P Sridhar --- include/linux/zswap.h | 7 +++++++ mm/zswap.c | 49 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index d961ead91bf1..9ad27ab3d222 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -7,6 +7,13 @@ struct lruvec; +/* + * For IAA compression batching: + * Maximum number of IAA acomp compress requests that will be processed + * in a batch: in parallel, if iaa_crypto async/no irq mode is enabled + * (the default); else sequentially, if iaa_crypto sync mode is in effect. + */ +#define SWAP_CRYPTO_BATCH_SIZE 8UL extern atomic_long_t zswap_stored_pages; #ifdef CONFIG_ZSWAP diff --git a/mm/zswap.c b/mm/zswap.c index 3ad81ec0f262..edd5910168c2 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -151,6 +151,12 @@ struct crypto_acomp_ctx { bool is_sleepable; }; +enum batch_comp_status { + UNINIT_BATCH_COMP = -1, + CANNOT_BATCH_COMP = 0, + BATCH_COMP_ENABLED = 1, +}; + /* * The lock ordering is zswap_tree.lock -> zswap_pool.lru_lock. * The only case where lru_lock is not acquired while holding tree.lock is @@ -159,6 +165,7 @@ struct crypto_acomp_ctx { */ struct zswap_pool { struct zpool *zpool; + enum batch_comp_status can_batch_comp; struct crypto_acomp_ctx __percpu *acomp_ctx; struct crypto_acomp_ctx __percpu *acomp_batch_ctx; struct percpu_ref ref; @@ -310,6 +317,7 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) goto ref_fail; INIT_LIST_HEAD(&pool->list); + pool->can_batch_comp = UNINIT_BATCH_COMP; zswap_pool_debug("created", pool); return pool; @@ -695,6 +703,36 @@ static int zswap_enabled_param_set(const char *val, return ret; } +/* Called only if sysctl vm.compress-batching is set to "1". */ +static __always_inline bool zswap_pool_can_batch(struct zswap_pool *pool) +{ + if (pool->can_batch_comp == BATCH_COMP_ENABLED) + return true; + + if (pool->can_batch_comp == CANNOT_BATCH_COMP) + return false; + + if ((pool->can_batch_comp == UNINIT_BATCH_COMP) && pool->acomp_batch_ctx) { + struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr( + pool->acomp_batch_ctx, + raw_smp_processor_id()); + if (!IS_ERR_OR_NULL(acomp_ctx)) { + if ((acomp_ctx->nr_reqs == SWAP_CRYPTO_BATCH_SIZE) || + (!acomp_ctx->nr_reqs && + !zswap_create_acomp_ctx(raw_smp_processor_id(), + acomp_ctx, + pool->tfm_name, + SWAP_CRYPTO_BATCH_SIZE))) { + pool->can_batch_comp = BATCH_COMP_ENABLED; + return true; + } + } + } + + pool->can_batch_comp = CANNOT_BATCH_COMP; + return false; +} + /********************************* * lru functions **********************************/ @@ -850,6 +888,17 @@ static int zswap_create_acomp_ctx(unsigned int cpu, acomp_ctx->acomp = acomp; acomp_ctx->is_sleepable = acomp_is_async(acomp); + /* + * Cannot create a batching ctx without the crypto acomp alg supporting + * batch_compress and batch_decompress API. + */ + if ((nr_reqs > 1) && (!acomp->batch_compress || !acomp->batch_decompress)) { + WARN_ONCE(1, "Cannot alloc acomp_ctx with %d reqs since crypto acomp %s\nhas not registered batch_compress() and/or batch_decompress()\n", + nr_reqs, tfm_name); + ret = -ENODEV; + goto buf_fail; + } + acomp_ctx->buffers = kmalloc_node(nr_reqs * sizeof(u8 *), GFP_KERNEL, cpu_to_node(cpu)); for (i = 0; i < nr_reqs; ++i) { From patchwork Sun Nov 3 03:21:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBA6DE6781A for ; Sun, 3 Nov 2024 03:21:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 436B66B009D; Sat, 2 Nov 2024 23:21:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BE076B009F; Sat, 2 Nov 2024 23:21:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 210ED6B009E; Sat, 2 Nov 2024 23:21:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id ECCF56B009C for ; Sat, 2 Nov 2024 23:21:27 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7864780D61 for ; Sun, 3 Nov 2024 03:21:27 +0000 (UTC) X-FDA: 82743332490.10.2F44D9A Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf17.hostedemail.com (Postfix) with ESMTP id 184694000E for ; Sun, 3 Nov 2024 03:21:01 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=SLtMYW5I; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf17.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730604003; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s6LLDw37i5cu+HINgWP4vPH/p7itmhc/p8xNggw8EKo=; b=e/9+vfxOSzPU4Nc7LpfDkjKdPnux4fTtHLMb1EFuB6qR1QA9C8RBHa0dixpb0THS0beVUv oTEbxO+/QpiLQ81w4+665Ow30ZUFSBHQ18sLweWHDOa1ZjCq4AEvKguAwy1Wum4ZZutkK9 Ydi2yhJC1tRFOsVw0NlY4/qnT/jYbYo= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=SLtMYW5I; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf17.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730604003; a=rsa-sha256; cv=none; b=wuaMculDIMrv5YtdOg+JV1TMxBWd5dSPCQnVzon41Obx+C8w4onlOFD9xG8lm/n20Pl7WS +bQBdvE2aAeI+9ZwaTnY6iknY4ojMNWtFcq7Iaoad++KKJ8/1GrgYtKOg6cQDXe4QmFqyR UatE/HHs6Kf6skBPLql4v+JnRe7vosA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604085; x=1762140085; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LRgON0J+B8CkVFIt5Q3iyXtNXhdwK4RCB9NPlp2lvok=; b=SLtMYW5IYRIa3npfuAmejcxeoKmD9w+hgErdMm3YmYm4lC1r4J0aiEk5 zHd1Y+kTUEPbLicK5kj/CCmatX43scQl4zbCitaaXMFMNzYxbcu2Gk1ve DP7E3yLtKGXGtJC/rkgvkr/chEuYxr1gAEmX/eqVGCiQctWArwf5t6zB0 y8txWY8r9i98ZXYI10B4+v1b/t1vGQDDevNMgXblVuBCO1Ihg1gThjmA1 L+sXNIJsAzH9swqK4q3KWO52UM7BygiFqu4J5HvHL1PixcDn4spTfuvX+ cIZMNuqMs9DH/Ah+FiBz11XABJbh5JBXDXRg0F5FdKNi+s3FNXPXj2JlA A==; X-CSE-ConnectionGUID: B1n2JjDXSkWTnxaiptkTLA== X-CSE-MsgGUID: Y/J37debTu67SWzDWPxvGA== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548357" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548357" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:14 -0700 X-CSE-ConnectionGUID: Vry+ngz1T2CnvIoMfXY38w== X-CSE-MsgGUID: NSoSHThITDqKrhfED8nBYQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451906" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:15 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 12/13] mm: Add sysctl vm.compress-batching switch for compress batching during swapout. Date: Sat, 2 Nov 2024 20:21:10 -0700 Message-Id: <20241103032111.333282-13-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 184694000E X-Stat-Signature: i5qdpyjjb9u3wph8try6xxtb5ub4k4ux X-HE-Tag: 1730604061-804772 X-HE-Meta: U2FsdGVkX18/u5+vbpJLA6l5c93DHhlNb3ReaysIw+hT26W8aDnLbAPO41S84deYfdzcWZ4Gb112TcfSWqBeS9RlRtN+64Q/iOmGyvJbkCLG2yMBnZowAg4HE2at+2SiDZ7Lih7kpgBSm6epIdBKMw95Ic93ZfWOe/w6crcLKCHNAj+sT8AcUM+ljCT9eJnGuWBBRI72NRASFEdySoiUqDaGJsyEkpG8B+805Uhz7df5PzYo1JFJMQ0nmQOsTUTXlVyt+eqcqznATcwHIb6/wN8WeZDL7cH/OMg05jCsgyjJU/ayexdVDd/JIVe0wqvz6QS2QCZ3hqEPm9/ob5XDzJB/olPbVDldsjN3qiNFzLXb+4wG0XQvnmJpnJBNlaz/zvTI/B64ujQrPq5WrDVmQ0zvIbdlolxF8P3gU6sB8ZjmPA+36KS1KYkeVVp9wy/RxjcR397zJl/WSR6OFYtxK79hzwmZGAUMVPkHDjHiZ2VTrtlg5iwcvu1knNW+uhD54F5UO9maJecVoVWzNxSp9k1R9G5OaYtfdbLhWdrz9SPQDiC6nGx5A2YaxMz2Tdo7bk/v7j2ZdqonMy6u5Y18+bthf3Tn/D19laJYrYATAXuKKcVUEdGOKxXvgKIF0jJAZrAJJ0/HKpgV4j/SWY3MpzZo1egSAi7IpcAMPrqb0GIPOOpm5rUZaSUj6Ya1apXfj+rChuEz1kTUzGPSJhpcBeOy4inKG+wwbpD/WIdr5CBf8kV9hTwd+mIJiEU+bJ7yDQxS5uv21x9o8KZu5Na0bKw2cOarT9Eb8jLYrTfBmois1HUieU7bBLVCeVSldBzTtvC6rtZfaCHweuARgRsMb8UNov8tgB+26kk4SaNcp+p8c3hwwdEdrzCcmx9O3GFs/riKGdVkPWaemzloKyXWB/CnlI9snYTidJ8/XYXWrEK3MPMzG51gcUt9k8CHeyU0L+CVQKncUw0rc5gfUbY hj664uNN IZWMh9iT5VnonEp7h5Dmg42D3qllkauzu02IuRWSGVLrfGJ823DlPLQpK3KvgR79N1Siq5/r6cCHtywmZBMU/nq/GDP5MWvaGDkAyLkG3+G+PKE2IBXlSXL77i67P6mDzA2EixO6A5BJYI3Dv5TK+UwD9MLlQOA9P7LCuhGkxsmvybJ+jn4pd8xCl6A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The sysctl vm.compress-batching parameter is 0 by default. If the platform has Intel IAA, the user can run experiments with IAA compress batching of large folios in zswap_store() as follows: sysctl vm.compress-batching=1 echo deflate-iaa > /sys/module/zswap/parameters/compressor This is expected to significantly improve zswap_store() latency of swapping out large folios due to parallel compression of 8 pages in the large folio at a time, in hardware. Setting vm.compress-batching to "1" takes effect only if the zswap compression algorithm's crypto_acomp registers implementations for the batch_compress() and batch_decompress() API. In other words, compress batching works only with the iaa_crypto driver, that does register these new batching API. It is a no-op for compressors that do not register the batching API. The sysctl vm.compress-batching acts as a switch because it takes effect upon future zswap_store() calls on any given core. If the switch is "1", large folios will use parallel batched compression of the folio's pages. If the switch is "0", zswap_store() will use sequential compression for storing every page in a large folio. Signed-off-by: Kanchana P Sridhar --- include/linux/mm.h | 2 ++ kernel/sysctl.c | 9 +++++++++ mm/swap.c | 6 ++++++ 3 files changed, 17 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 610653a8b0fe..c94ba5c36169 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -82,8 +82,10 @@ extern const int page_cluster_max; #ifdef CONFIG_SYSCTL extern int sysctl_legacy_va_layout; +extern unsigned int compress_batching; #else #define sysctl_legacy_va_layout 0 +#define compress_batching 0 #endif #ifdef CONFIG_HAVE_ARCH_MMAP_RND_BITS diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 79e6cb1d5c48..e298857595b4 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -2064,6 +2064,15 @@ static struct ctl_table vm_table[] = { .extra1 = SYSCTL_ZERO, .extra2 = (void *)&page_cluster_max, }, + { + .procname = "compress-batching", + .data = &compress_batching, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_douintvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, { .procname = "dirtytime_expire_seconds", .data = &dirtytime_expire_interval, diff --git a/mm/swap.c b/mm/swap.c index 638a3f001676..bc4c9079769e 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -47,6 +47,9 @@ int page_cluster; const int page_cluster_max = 31; +/* Enable/disable compress batching during swapout. */ +unsigned int compress_batching; + struct cpu_fbatches { /* * The following folio batches are grouped together because they are protected @@ -1074,4 +1077,7 @@ void __init swap_setup(void) * Right now other parts of the system means that we * _really_ don't want to cluster much more */ + + /* Disable compress batching during swapout by default. */ + compress_batching = 0; } From patchwork Sun Nov 3 03:21:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860298 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D091EE6780A for ; Sun, 3 Nov 2024 03:21:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A9EC06B009C; Sat, 2 Nov 2024 23:21:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A4CFA6B009E; Sat, 2 Nov 2024 23:21:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82E5E6B009F; Sat, 2 Nov 2024 23:21:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2D19F6B009C for ; Sat, 2 Nov 2024 23:21:28 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E3EB21A1653 for ; Sun, 3 Nov 2024 03:21:27 +0000 (UTC) X-FDA: 82743332532.23.1B9FBE1 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf18.hostedemail.com (Postfix) with ESMTP id 4B9361C0002 for ; Sun, 3 Nov 2024 03:21:12 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ZJUS44Hs; spf=pass (imf18.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730604039; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ndGJE0Wra+88ceSAOyTkcif11AziwaWvEs2KEV1flF4=; b=8nsTotdIKpXVpod4vQF+JDzd0j4YHRD6x9TKnBBCyQbFJFvf8Jsj/wZVKskAjGdvxzcyFN MTgSnfthR6PHQdvbjkbHN4eJksndwOLPpqUlyg9amdjUM1GY+47lG3JxV70DAzB3VWeDz8 jyCiRtBSrGnHTAzgiMQWKJbtER85t6Y= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730604039; a=rsa-sha256; cv=none; b=rUJ2dhLqDaN6ZS4KPfnno/E0Wk8UCckdFYYZ1pTx19u8JRBOm9vilm39LavQuP6/ppq6Oe IcrlU5AsH+Zyit8YI4iS1bdDlh/MnrE78/iBi0tXrmGbJcyODrXagPfamyFHrzI7vsF3Ys WjF00irFkmPLGGGzEfxRWTVnMOuWjfY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ZJUS44Hs; spf=pass (imf18.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604085; x=1762140085; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=WH3qapobYALthM4qNT8jALTfm8zDSXb8/W5HKpaYoW4=; b=ZJUS44Hs9SvJN8ZMdyWQQ6YiRPx4dLkIPAKR7D+8sjuQ68oesKJblP9S AOWvm8Jvi9bJii5LC/gghEKC92r6anNhyXsznpd/pb9C0NItmRzbTWbWu 74m9H5eq9YVBKtPy6cF/XzxL4vVWI5nyDx/MwSYY4Ztr80qRr+Jm9WpiZ kMGSUv7FZSrbMzTH0l7GyLok6qp9Dtw9wQMR6E+lssJxjtvqU8hQQ8jPc 9HDXK3Y3iDSxd718SQfLgt9UQLzGwWgDsXmuuWFJ7/OJYe58I3+2ALSWW QmcyD9/JF/wRmjPpGQqZ86ZCAvcV9XZ9wTrfhzIBEQhF8JfDiWXky2qlh g==; X-CSE-ConnectionGUID: PIj6HAblQNuPfd9GVT/ojQ== X-CSE-MsgGUID: IxZeMhkOT2+scYmSODXExg== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548366" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548366" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:14 -0700 X-CSE-ConnectionGUID: pe/zP34EQWi3HkuSI9WiUw== X-CSE-MsgGUID: h2Q8Nf1USqqojzyYd9UVXw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451909" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:15 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 13/13] mm: zswap: Compress batching with Intel IAA in zswap_store() of large folios. Date: Sat, 2 Nov 2024 20:21:11 -0700 Message-Id: <20241103032111.333282-14-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: qaa3xdmfzrshq8hbowua1hepm1c7oa8b X-Rspam-User: X-Rspamd-Queue-Id: 4B9361C0002 X-Rspamd-Server: rspam02 X-HE-Tag: 1730604072-184414 X-HE-Meta: U2FsdGVkX18U4HtKIDxs4RIpiOIQ9Z7N9djOclolSwcCh85axNH5XL91taKEp9TCBEy4rnyTBwfhNWDVEtDD2hIq7cYiIFO9LnApmlPC+7dwHFyr8oi4RxZfW0SVnNFFXGiOepHYjVAhvIOL/JA+k/Zp5wSYJANSJnjXSUL229FsRNJ10hJhssOgHuBU7RpmHRtz+eWWE4IUz1LoXx1wBwaKj5PyJTQzYo3t+hsEmMkaQHCK9ijZqRbbJnGzlPnp8RGEd8MSETw8rFzCieT4hyfpXD134v33XrFZAmKtkOc2bb8cd+LGCRY964vViOI0rPpauVXWvBLIEnd0mADpawbtwSY/2y4NqCXZOR4iqbHCD5gU/n7HpigFK/oV0DEn3WqQ5q/l/yXF9cM/XcgZki0CD/9a8zbTq8qZCXLnAUunMgU+OE14jB2iZ2E9aVcnWKnpod+PdnoRqYzIgre9ExkHFhP8vdHSKjnuI7XoiUc/eQFXZ3pPUDl364V1IvXww7GtFbEkLLWqhF6Nvt3SPJwBiEcEfyJTr5mI+dWIjmTs+WzVrrt+8tx/iy5yHJiMlCnxa9PUQB8AA04CW6fqRJj+VtsUbOWWjhKzFDXZfhZfVAUMnM3Ot/1qe1hH/+B0O3CjoRZc2+GBy+4emlHeXVFhgZiQcu0CaSJvCWaDH/JM4/kT1KgBEspDwtaYR2sPRYHIISm+WQkqnqMEgV6jQ6EKHBHKF0gFtKgGWlf5KRhR4y0sYXLh+5mlrrWqwyv8Yq+s/c4D7QjHuI2woe/tP0LNTuvZGQR4rFYZnTPZV56n2fEeO/g9IL90rcNpRhGhgtc3ubqREYLjPtNbowdFTu4kx5hK2J51HXpiOZgC5QdSXtGvajng9x81mJlBd7NK1CTmYWElif9ampDhCDs7U/wpi4X6/mb8gU7J+vRXXeDCjsq65hlue9/FrW6ToUy+n5fQCUI7s/9RXgoM2Iu Ry6E0pnN y5NUoIeXKCWUOxc0O4SykGCXYoCRrBVVP7ebSNBEqp0wE79oE8cI0iBAZNET8UkhGLotTP4O+eemFmq/k/uO5fD5LbtScC5Co97lndJjC3HRTait3JKPNGAvvxjneNEA/RNG/NZaTkqjID1VRYl7oBIzy8jlZrzzlNEg3OaLLS0iDnzlBfkwuYdm636wZho27EdLG0aG6Uqttb4hTqYG5Ms47+JI9w/XgNho8R89K7u67NYs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If the system has Intel IAA, and if sysctl vm.compress-batching is set to "1", zswap_store() will call crypto_acomp_batch_compress() to compress up to SWAP_CRYPTO_BATCH_SIZE (i.e. 8) pages in large folios in parallel using the multiple compress engines available in IAA hardware. On platforms with multiple IAA devices per socket, compress jobs from all cores in a socket will be distributed among all IAA devices on the socket by the iaa_crypto driver. With deflate-iaa configured as the zswap compressor, and sysctl vm.compress-batching is enabled, the first time zswap_store() has to swapout a large folio on any given cpu, it will allocate the pool->acomp_batch_ctx resources on that cpu, and set pool->can_batch_comp to BATCH_COMP_ENABLED. It will then proceed to call the main __zswap_store_batch_core() compress batching function. Subsequent calls to zswap_store() on the same cpu will go ahead and use the acomp_batch_ctx by checking the pool->can_batch_comp status. Hence, we allocate the per-cpu pool->acomp_batch_ctx resources only on an as-needed basis, to reduce memory footprint cost. The cost is not incurred on cores that never get to swapout a large folio. This patch introduces the main __zswap_store_batch_core() function for compress batching. This interface represents the extensible compress batching architecture that can potentially be called with a batch of any-order folios from shrink_folio_list(). In other words, although zswap_store() calls __zswap_store_batch_core() with exactly one large folio in this patch, we can reuse this interface to reclaim a batch of folios, to significantly improve the reclaim path efficiency due to IAA's parallel compression capability. The newly added functions that implement batched stores follow the general structure of zswap_store() of a large folio. Some amount of restructuring and optimization is done to minimize failure points for a batch, fail early and maximize the zswap store pipeline occupancy with SWAP_CRYPTO_BATCH_SIZE pages, potentially from multiple folios. This is intended to maximize reclaim throughput with the IAA hardware parallel compressions. Signed-off-by: Kanchana P Sridhar --- include/linux/zswap.h | 84 ++++++ mm/zswap.c | 625 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 709 insertions(+) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index 9ad27ab3d222..b868f269ffa7 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -31,6 +31,88 @@ struct zswap_lruvec_state { atomic_long_t nr_disk_swapins; }; +/* + * struct zswap_store_sub_batch_page: + * + * This represents one "zswap batching element", namely, the + * attributes associated with a page in a large folio that will + * be compressed and stored in zswap. The term "batch" is reserved + * for a conceptual "batch" of folios that can be sent to + * zswap_store() by reclaim. The term "sub-batch" is used to describe + * a collection of "zswap batching elements", i.e., an array of + * "struct zswap_store_sub_batch_page *". + * + * The zswap compress sub-batch size is specified by + * SWAP_CRYPTO_SUB_BATCH_SIZE, currently set as 8UL if the + * platform has Intel IAA. This means zswap can store a large folio + * by creating sub-batches of up to 8 pages and compressing this + * batch using IAA to parallelize the 8 compress jobs in hardware. + * For e.g., a 64KB folio can be compressed as 2 sub-batches of + * 8 pages each. This can significantly improve the zswap_store() + * performance for large folios. + * + * Although the page itself is represented directly, the structure + * adds a "u8 batch_idx" to represent an index for the folio in a + * conceptual "batch of folios" that can be passed to zswap_store(). + * Conceptually, this allows for up to 256 folios that can be passed + * to zswap_store(). If this conceptual number of folios sent to + * zswap_store() exceeds 256, the "batch_idx" needs to become u16. + */ +struct zswap_store_sub_batch_page { + u8 batch_idx; + swp_entry_t swpentry; + struct obj_cgroup *objcg; + struct zswap_entry *entry; + int error; /* folio error status. */ +}; + +/* + * struct zswap_store_pipeline_state: + * + * This stores state during IAA compress batching of (conceptually, a batch of) + * folios. The term pipelining in this context, refers to breaking down + * the batch of folios being reclaimed into sub-batches of + * SWAP_CRYPTO_SUB_BATCH_SIZE pages, batch compressing and storing the + * sub-batch. This concept could be further evolved to use overlap of CPU + * computes with IAA computes. For instance, we could stage the post-compress + * computes for sub-batch "N-1" to happen in parallel with IAA batch + * compression of sub-batch "N". + * + * We begin by developing the concept of compress batching. Pipelining with + * overlap can be future work. + * + * @errors: The errors status for the batch of reclaim folios passed in from + * a higher mm layer such as swap_writepage(). + * @pool: A valid zswap_pool. + * @acomp_ctx: The per-cpu pointer to the crypto_acomp_ctx for the @pool. + * @sub_batch: This is an array that represents the sub-batch of up to + * SWAP_CRYPTO_SUB_BATCH_SIZE pages that are being stored + * in zswap. + * @comp_dsts: The destination buffers for crypto_acomp_compress() for each + * page being compressed. + * @comp_dlens: The destination buffers' lengths from crypto_acomp_compress() + * obtained after crypto_acomp_poll() returns completion status, + * for each page being compressed. + * @comp_errors: Compression errors for each page being compressed. + * @nr_comp_pages: Total number of pages in @sub_batch. + * + * Note: + * The max sub-batch size is SWAP_CRYPTO_SUB_BATCH_SIZE, currently 8UL. + * Hence, if SWAP_CRYPTO_SUB_BATCH_SIZE exceeds 256, some of the + * u8 members (except @comp_dsts) need to become u16. + */ +struct zswap_store_pipeline_state { + int *errors; + struct zswap_pool *pool; + struct crypto_acomp_ctx *acomp_ctx; + struct zswap_store_sub_batch_page *sub_batch; + struct page **comp_pages; + u8 **comp_dsts; + unsigned int *comp_dlens; + int *comp_errors; + u8 nr_comp_pages; +}; + unsigned long zswap_total_pages(void); bool zswap_store(struct folio *folio); bool zswap_load(struct folio *folio); @@ -45,6 +127,8 @@ bool zswap_never_enabled(void); #else struct zswap_lruvec_state {}; +struct zswap_store_sub_batch_page {}; +struct zswap_store_pipeline_state {}; static inline bool zswap_store(struct folio *folio) { diff --git a/mm/zswap.c b/mm/zswap.c index edd5910168c2..40c5d921739f 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -255,6 +255,12 @@ static int zswap_create_acomp_ctx(unsigned int cpu, char *tfm_name, unsigned int nr_reqs); +static bool __zswap_store_batch_core( + int node_id, + struct folio **folios, + int *errors, + unsigned int nr_folios); + /********************************* * pool functions **********************************/ @@ -1617,6 +1623,12 @@ static ssize_t zswap_store_page(struct page *page, return -EINVAL; } +/* + * Modified to use the IAA compress batching framework implemented in + * __zswap_store_batch_core() if sysctl vm.compress-batching is 1. + * The batching code is intended to significantly improve folio store + * performance over the sequential code. + */ bool zswap_store(struct folio *folio) { long nr_pages = folio_nr_pages(folio); @@ -1629,6 +1641,38 @@ bool zswap_store(struct folio *folio) bool ret = false; long index; + /* + * Improve large folio zswap_store() latency with IAA compress batching, + * if this is enabled by setting sysctl vm.compress-batching to "1". + * If enabled, the large folio's pages are compressed in parallel in + * batches of SWAP_CRYPTO_BATCH_SIZE pages. If disabled, every page in + * the large folio is compressed sequentially. + */ + if (folio_test_large(folio) && READ_ONCE(compress_batching)) { + pool = zswap_pool_current_get(); + if (!pool) { + pr_err("Cannot setup acomp_batch_ctx for compress batching: no current pool found\n"); + goto sequential_store; + } + + if (zswap_pool_can_batch(pool)) { + int error = -1; + bool store_batch = __zswap_store_batch_core( + folio_nid(folio), + &folio, &error, 1); + + if (store_batch) { + zswap_pool_put(pool); + if (!error) + ret = true; + return ret; + } + } + zswap_pool_put(pool); + } + +sequential_store: + VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1715,6 +1759,587 @@ bool zswap_store(struct folio *folio) return ret; } +/* + * Note: If SWAP_CRYPTO_BATCH_SIZE exceeds 256, change the + * u8 stack variables in the next several functions, to u16. + */ + +/* + * Propagate the "sbp" error condition to other batch elements belonging to + * the same folio as "sbp". + */ +static __always_inline void zswap_store_propagate_errors( + struct zswap_store_pipeline_state *zst, + u8 error_batch_idx) +{ + u8 i; + + if (zst->errors[error_batch_idx]) + return; + + for (i = 0; i < zst->nr_comp_pages; ++i) { + struct zswap_store_sub_batch_page *sbp = &zst->sub_batch[i]; + + if (sbp->batch_idx == error_batch_idx) { + if (!sbp->error) { + if (!IS_ERR_VALUE(sbp->entry->handle)) + zpool_free(zst->pool->zpool, sbp->entry->handle); + + if (sbp->entry) { + zswap_entry_cache_free(sbp->entry); + sbp->entry = NULL; + } + sbp->error = -EINVAL; + } + } + } + + /* + * Set zswap status for the folio to "error" + * for use in swap_writepage. + */ + zst->errors[error_batch_idx] = -EINVAL; +} + +static __always_inline void zswap_process_comp_errors( + struct zswap_store_pipeline_state *zst) +{ + u8 i; + + for (i = 0; i < zst->nr_comp_pages; ++i) { + struct zswap_store_sub_batch_page *sbp = &zst->sub_batch[i]; + + if (zst->comp_errors[i]) { + if (zst->comp_errors[i] == -ENOSPC) + zswap_reject_compress_poor++; + else + zswap_reject_compress_fail++; + + if (!sbp->error) + zswap_store_propagate_errors(zst, + sbp->batch_idx); + } + } +} + +static void zswap_compress_batch(struct zswap_store_pipeline_state *zst) +{ + /* + * Compress up to SWAP_CRYPTO_BATCH_SIZE pages. + * It is important to note that the zswap pool's per-cpu "acomp_batch_ctx" + * resources are allocated only if the crypto_acomp has registered both, + * crypto_acomp_batch_compress() and crypto_acomp_batch_decompress() API. + * The iaa_crypto driver registers implementations for both these API. + * Hence, if IAA is the zswap compressor and sysctl vm.compress-batching + * is set to "1", the call to crypto_acomp_batch_compress() will + * compresses the pages in parallel, leading to significant performance + * improvements as compared to software compressors. + */ + crypto_acomp_batch_compress( + zst->acomp_ctx->reqs, + &zst->acomp_ctx->wait, + zst->comp_pages, + zst->comp_dsts, + zst->comp_dlens, + zst->comp_errors, + zst->nr_comp_pages); + + /* + * Scan the sub-batch for any compression errors, + * and invalidate pages with errors, along with other + * pages belonging to the same folio as the error pages. + */ + zswap_process_comp_errors(zst); +} + +static void zswap_zpool_store_sub_batch( + struct zswap_store_pipeline_state *zst) +{ + u8 i; + + for (i = 0; i < zst->nr_comp_pages; ++i) { + struct zswap_store_sub_batch_page *sbp = &zst->sub_batch[i]; + struct zpool *zpool; + unsigned long handle; + char *buf; + gfp_t gfp; + int err; + + /* Skip pages that had compress errors. */ + if (sbp->error) + continue; + + zpool = zst->pool->zpool; + gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; + if (zpool_malloc_support_movable(zpool)) + gfp |= __GFP_HIGHMEM | __GFP_MOVABLE; + err = zpool_malloc(zpool, zst->comp_dlens[i], gfp, &handle); + + if (err) { + if (err == -ENOSPC) + zswap_reject_compress_poor++; + else + zswap_reject_alloc_fail++; + + /* + * An error should be propagated to other pages of the + * same folio in the sub-batch, and zpool resources for + * those pages (in sub-batch order prior to this zpool + * error) should be de-allocated. + */ + zswap_store_propagate_errors(zst, sbp->batch_idx); + continue; + } + + buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); + memcpy(buf, zst->comp_dsts[i], zst->comp_dlens[i]); + zpool_unmap_handle(zpool, handle); + + sbp->entry->handle = handle; + sbp->entry->length = zst->comp_dlens[i]; + } +} + +/* + * Returns true if the entry was successfully + * stored in the xarray, and false otherwise. + */ +static bool zswap_store_entry(swp_entry_t page_swpentry, + struct zswap_entry *entry) +{ + struct zswap_entry *old = xa_store(swap_zswap_tree(page_swpentry), + swp_offset(page_swpentry), + entry, GFP_KERNEL); + if (xa_is_err(old)) { + int err = xa_err(old); + + WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); + zswap_reject_alloc_fail++; + return false; + } + + /* + * We may have had an existing entry that became stale when + * the folio was redirtied and now the new version is being + * swapped out. Get rid of the old. + */ + if (old) + zswap_entry_free(old); + + return true; +} + +static void zswap_batch_compress_post_proc( + struct zswap_store_pipeline_state *zst) +{ + int nr_objcg_pages = 0, nr_pages = 0; + struct obj_cgroup *objcg = NULL; + size_t compressed_bytes = 0; + u8 i; + + zswap_zpool_store_sub_batch(zst); + + for (i = 0; i < zst->nr_comp_pages; ++i) { + struct zswap_store_sub_batch_page *sbp = &zst->sub_batch[i]; + + if (sbp->error) + continue; + + if (!zswap_store_entry(sbp->swpentry, sbp->entry)) { + zswap_store_propagate_errors(zst, sbp->batch_idx); + continue; + } + + /* + * The entry is successfully compressed and stored in the tree, + * there is no further possibility of failure. Grab refs to the + * pool and objcg. These refs will be dropped by + * zswap_entry_free() when the entry is removed from the tree. + */ + zswap_pool_get(zst->pool); + if (sbp->objcg) + obj_cgroup_get(sbp->objcg); + + /* + * We finish initializing the entry while it's already in xarray. + * This is safe because: + * + * 1. Concurrent stores and invalidations are excluded by folio + * lock. + * + * 2. Writeback is excluded by the entry not being on the LRU yet. + * The publishing order matters to prevent writeback from seeing + * an incoherent entry. + */ + sbp->entry->pool = zst->pool; + sbp->entry->swpentry = sbp->swpentry; + sbp->entry->objcg = sbp->objcg; + sbp->entry->referenced = true; + if (sbp->entry->length) { + INIT_LIST_HEAD(&sbp->entry->lru); + zswap_lru_add(&zswap_list_lru, sbp->entry); + } + + if (!objcg && sbp->objcg) { + objcg = sbp->objcg; + } else if (objcg && sbp->objcg && (objcg != sbp->objcg)) { + obj_cgroup_charge_zswap(objcg, compressed_bytes); + count_objcg_events(objcg, ZSWPOUT, nr_objcg_pages); + compressed_bytes = 0; + nr_objcg_pages = 0; + objcg = sbp->objcg; + } + + if (sbp->objcg) { + compressed_bytes += sbp->entry->length; + ++nr_objcg_pages; + } + + ++nr_pages; + } /* for sub-batch pages. */ + + if (objcg) { + obj_cgroup_charge_zswap(objcg, compressed_bytes); + count_objcg_events(objcg, ZSWPOUT, nr_objcg_pages); + } + + atomic_long_add(nr_pages, &zswap_stored_pages); + count_vm_events(ZSWPOUT, nr_pages); +} + +static void zswap_store_sub_batch(struct zswap_store_pipeline_state *zst) +{ + u8 i; + + for (i = 0; i < zst->nr_comp_pages; ++i) { + zst->comp_dsts[i] = zst->acomp_ctx->buffers[i]; + zst->comp_dlens[i] = PAGE_SIZE; + } /* for sub-batch pages. */ + + /* + * Batch compress sub-batch "N". If IAA is the compressor, the + * hardware will compress multiple pages in parallel. + */ + zswap_compress_batch(zst); + + zswap_batch_compress_post_proc(zst); +} + +static void zswap_add_folio_pages_to_sb( + struct zswap_store_pipeline_state *zst, + struct folio* folio, + u8 batch_idx, + struct obj_cgroup *objcg, + struct zswap_entry *entries[], + long start_idx, + u8 add_nr_pages) +{ + long index; + + for (index = start_idx; index < (start_idx + add_nr_pages); ++index) { + u8 i = zst->nr_comp_pages; + struct zswap_store_sub_batch_page *sbp = &zst->sub_batch[i]; + struct page *page = folio_page(folio, index); + zst->comp_pages[i] = page; + sbp->swpentry = page_swap_entry(page); + sbp->batch_idx = batch_idx; + sbp->objcg = objcg; + sbp->entry = entries[index - start_idx]; + sbp->error = 0; + ++zst->nr_comp_pages; + } +} + +static __always_inline void zswap_store_reset_sub_batch( + struct zswap_store_pipeline_state *zst) +{ + zst->nr_comp_pages = 0; +} + +/* Allocate entries for the next sub-batch. */ +static int zswap_alloc_entries(u8 nr_entries, + struct zswap_entry *entries[], + int node_id) +{ + u8 i; + + for (i = 0; i < nr_entries; ++i) { + entries[i] = zswap_entry_cache_alloc(GFP_KERNEL, node_id); + if (!entries[i]) { + u8 j; + + zswap_reject_kmemcache_fail++; + for (j = 0; j < i; ++j) + zswap_entry_cache_free(entries[j]); + return -EINVAL; + } + + entries[i]->handle = (unsigned long)ERR_PTR(-EINVAL); + } + + return 0; +} + +/* + * If the zswap store fails or zswap is disabled, we must invalidate + * the possibly stale entries which were previously stored at the + * offsets corresponding to each page of the folio. Otherwise, + * writeback could overwrite the new data in the swapfile. + */ +static void zswap_delete_stored_entries(struct folio *folio) +{ + swp_entry_t swp = folio->swap; + unsigned type = swp_type(swp); + pgoff_t offset = swp_offset(swp); + struct zswap_entry *entry; + struct xarray *tree; + long index; + + for (index = 0; index < folio_nr_pages(folio); ++index) { + tree = swap_zswap_tree(swp_entry(type, offset + index)); + entry = xa_erase(tree, offset + index); + if (entry) + zswap_entry_free(entry); + } +} + +static void zswap_store_process_folio_errors( + struct folio **folios, + int *errors, + unsigned int nr_folios) +{ + u8 batch_idx; + + for (batch_idx = 0; batch_idx < nr_folios; ++batch_idx) + if (errors[batch_idx]) + zswap_delete_stored_entries(folios[batch_idx]); +} + +/* + * Store a (batch of) any-order large folio(s) in zswap. Each folio will be + * broken into sub-batches of SWAP_CRYPTO_BATCH_SIZE pages, the + * sub-batch will be compressed by IAA in parallel, and stored in zpool/xarray. + * + * This the main procedure for batching of folios, and batching within + * large folios. + * + * This procedure should only be called if zswap supports batching of stores. + * Otherwise, the sequential implementation for storing folios as in the + * current zswap_store() should be used. + * + * The signature of this procedure is meant to allow the calling function, + * (for instance, swap_writepage()) to pass an array @folios + * (the "reclaim batch") of @nr_folios folios to be stored in zswap. + * All folios in the batch must have the same swap type and folio_nid @node_id + * (simplifying assumptions only to manage code complexity). + * + * @errors and @folios have @nr_folios number of entries, with one-one + * correspondence (@errors[i] represents the error status of @folios[i], + * for i in @nr_folios). + * The calling function (for instance, swap_writepage()) should initialize + * @errors[i] to a non-0 value. + * If zswap successfully stores @folios[i], it will set @errors[i] to 0. + * If there is an error in zswap, it will set @errors[i] to -EINVAL. + */ +static bool __zswap_store_batch_core( + int node_id, + struct folio **folios, + int *errors, + unsigned int nr_folios) +{ + struct zswap_store_sub_batch_page sub_batch[SWAP_CRYPTO_BATCH_SIZE]; + struct page *comp_pages[SWAP_CRYPTO_BATCH_SIZE]; + u8 *comp_dsts[SWAP_CRYPTO_BATCH_SIZE] = { NULL }; + unsigned int comp_dlens[SWAP_CRYPTO_BATCH_SIZE]; + int comp_errors[SWAP_CRYPTO_BATCH_SIZE]; + struct crypto_acomp_ctx *acomp_ctx, *acomp_batch_ctx; + struct zswap_pool *pool; + /* + * For now, lets say a max of 256 large folios can be reclaimed + * at a time, as a batch. If this exceeds 256, change this to u16. + */ + u8 batch_idx; + + /* Initialize the compress batching pipeline state. */ + struct zswap_store_pipeline_state zst = { + .errors = errors, + .pool = NULL, + .acomp_ctx = NULL, + .sub_batch = sub_batch, + .comp_pages = comp_pages, + .comp_dsts = comp_dsts, + .comp_dlens = comp_dlens, + .comp_errors = comp_errors, + .nr_comp_pages = 0, + }; + + pool = zswap_pool_current_get(); + if (!pool) { + if (zswap_check_limits()) + queue_work(shrink_wq, &zswap_shrink_work); + goto check_old; + } + + /* + * Caller should make sure that __zswap_store_batch_core() is + * invoked only if sysctl vm.compress-batching is set to "1". + * + * Verify if we are still on the same cpu for which batching + * resources in acomp_batch_ctx were allocated in zswap_store(). + * If not, return to zswap_store() for sequential store of the folio. + */ + acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); + mutex_lock(&acomp_ctx->mutex); + + acomp_batch_ctx = raw_cpu_ptr(pool->acomp_batch_ctx); + if (!acomp_batch_ctx || !acomp_batch_ctx->nr_reqs) { + mutex_unlock(&acomp_ctx->mutex); + zswap_pool_put(pool); + return false; + } + + mutex_lock(&acomp_batch_ctx->mutex); + mutex_unlock(&acomp_ctx->mutex); + + zst.pool = pool; + zst.acomp_ctx = acomp_batch_ctx; + + /* + * Iterate over the folios passed in. Construct sub-batches of up to + * SWAP_CRYPTO_BATCH_SIZE pages, if necessary, by iterating through + * multiple folios from the input "folios". Process each sub-batch + * with IAA batch compression. Detect errors from batch compression + * and set the impacted folio's error status (this happens in + * zswap_store_process_errors()). + */ + for (batch_idx = 0; batch_idx < nr_folios; ++batch_idx) { + struct folio *folio = folios[batch_idx]; + BUG_ON(!folio); + long folio_start_idx, nr_pages = folio_nr_pages(folio); + struct zswap_entry *entries[SWAP_CRYPTO_BATCH_SIZE]; + struct obj_cgroup *objcg = NULL; + struct mem_cgroup *memcg = NULL; + + VM_WARN_ON_ONCE(!folio_test_locked(folio)); + VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); + + /* + * If zswap is disabled, we must invalidate the possibly stale entry + * which was previously stored at this offset. Otherwise, writeback + * could overwrite the new data in the swapfile. + */ + if (!zswap_enabled) + continue; + + /* Check cgroup limits */ + objcg = get_obj_cgroup_from_folio(folio); + if (objcg && !obj_cgroup_may_zswap(objcg)) { + memcg = get_mem_cgroup_from_objcg(objcg); + if (shrink_memcg(memcg)) { + mem_cgroup_put(memcg); + goto put_objcg; + } + mem_cgroup_put(memcg); + } + + if (zswap_check_limits()) + goto put_objcg; + + if (objcg) { + memcg = get_mem_cgroup_from_objcg(objcg); + if (memcg_list_lru_alloc(memcg, &zswap_list_lru, GFP_KERNEL)) { + mem_cgroup_put(memcg); + goto put_objcg; + } + mem_cgroup_put(memcg); + } + + /* + * By default, set zswap status to "success" for use in + * swap_writepage() when this returns. In case of errors, + * a negative error number will over-write this when + * zswap_store_process_errors() is called. + */ + errors[batch_idx] = 0; + + folio_start_idx = 0; + + while (nr_pages > 0) { + u8 add_nr_pages; + + /* + * If we have accumulated SWAP_CRYPTO_BATCH_SIZE + * pages, process the sub-batch: it could contain pages + * from multiple folios. + */ + if (zst.nr_comp_pages == SWAP_CRYPTO_BATCH_SIZE) { + zswap_store_sub_batch(&zst); + zswap_store_reset_sub_batch(&zst); + /* + * Stop processing this folio if it had + * compress errors. + */ + if (errors[batch_idx]) + goto put_objcg; + } + + add_nr_pages = min3(( + (long)SWAP_CRYPTO_BATCH_SIZE - + (long)zst.nr_comp_pages), + nr_pages, + (long)SWAP_CRYPTO_BATCH_SIZE); + + /* + * Allocate zswap_entries for this sub-batch. If we + * get errors while doing so, we can flag an error + * for the folio, call the shrinker and move on. + */ + if (zswap_alloc_entries(add_nr_pages, + entries, node_id)) { + zswap_store_reset_sub_batch(&zst); + errors[batch_idx] = -EINVAL; + goto put_objcg; + } + + zswap_add_folio_pages_to_sb( + &zst, + folio, + batch_idx, + objcg, + entries, + folio_start_idx, + add_nr_pages); + + nr_pages -= add_nr_pages; + folio_start_idx += add_nr_pages; + } /* this folio has pages to be compressed. */ + + obj_cgroup_put(objcg); + continue; + +put_objcg: + obj_cgroup_put(objcg); + if (zswap_pool_reached_full) + queue_work(shrink_wq, &zswap_shrink_work); + } /* for batch folios */ + + if (!zswap_enabled) + goto check_old; + + /* + * Process last sub-batch: it could contain pages from + * multiple folios. + */ + if (zst.nr_comp_pages) + zswap_store_sub_batch(&zst); + + mutex_unlock(&acomp_batch_ctx->mutex); + zswap_pool_put(pool); +check_old: + zswap_store_process_folio_errors(folios, errors, nr_folios); + return true; +} + bool zswap_load(struct folio *folio) { swp_entry_t swp = folio->swap;