From patchwork Thu Feb 6 07:20:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39257C02198 for ; Thu, 6 Feb 2025 07:21:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9572B6B0083; Thu, 6 Feb 2025 02:21:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F450280001; Thu, 6 Feb 2025 02:21:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62E906B0085; Thu, 6 Feb 2025 02:21:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 31C846B0083 for ; Thu, 6 Feb 2025 02:21:08 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B8B0C140F0A for ; Thu, 6 Feb 2025 07:21:07 +0000 (UTC) X-FDA: 83088673374.29.9AA4626 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf08.hostedemail.com (Postfix) with ESMTP id 82FBB160007 for ; Thu, 6 Feb 2025 07:21:05 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VORHs8Cu; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826465; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VTuCZ9d86Afx2KF5Lu7O1ER4q5XHyye+n0Sin0NYIIM=; b=h/M+6PmGXeKFIi8o5wk9UwaF+ucziUMANVfK0YclSs4ID7iY8FvmhsIMTC3Y/7DHxR0xfY bEC1sf8NFQIWUhlE2kaFqWdCBvsgLlytk68Swu8g84n9C6iVhoQxBCVxgkPP/PCAjfHPVy 6+BXUt2BkihpNqTQqQnY0TRPz9nmfUM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VORHs8Cu; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826465; a=rsa-sha256; cv=none; b=zOkMK5+VrHskrN2qI8ZYGh65Dk3eTOtxxnlzTTscwRsqX/RABoFSR/7mE87Z2pwHpm1ZfF uVEcvEv3Gr53ZMMGavGLCVO3PjA+IOc6vO6I5Fo3tdClzsxAk+bJ3/0tWUsEd55BUojxSP i4lsiYG+yxh9lfVFupl1U21HhpUKCyU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826466; x=1770362466; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Dg6cX4FN0ukGigef7HcQnBK2IX626trFMZTmcHymvI0=; b=VORHs8CupvGolrVgmjFrV9+u+6dmZDTXu0rpgjQ3r1FvB4Aynv3pLAep n7FRoacTfkATEmGbP2KVUxyk52FXpvc52Y35l5K7OQ6cJNK/9Ah2n5Ikm P2I51QrgVrnW/UWpcBg+KB8EYLHFjYmL+VOfAWqImQcu92ET1biQLilAD YGGHTxvGSDpAm3mHpN/Zo5Ed+YDZ1rknQLnNS4BCfGHA7C47K3aVTQDwL oFlNu6JX/oNUTr2xUtPR4fySiF9MwHmjhuhPIcr/SVBoH/rf/vf4gcmZA dL+XAnhHNZ9lzADhBvwgq3E6wtCLy/bWl+rJDv7mDH/ujZvHnzjd/7iag g==; X-CSE-ConnectionGUID: 8Om28koXQsWO5YL3Jaofxw== X-CSE-MsgGUID: zAtJmhZsQ4+Uvgd4+qyhPw== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962569" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962569" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:03 -0800 X-CSE-ConnectionGUID: 2/lOqTRoShKTaM2C9vEFFg== X-CSE-MsgGUID: o312Qv5QS82zTbitC+ehBQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022592" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:03 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 01/16] crypto: acomp - Add synchronous/asynchronous acomp request chaining. Date: Wed, 5 Feb 2025 23:20:47 -0800 Message-Id: <20250206072102.29045-2-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 82FBB160007 X-Stat-Signature: zz8zgiuzjudr6cz4uri4steb4d3oy4kc X-Rspam-User: X-HE-Tag: 1738826465-281905 X-HE-Meta: U2FsdGVkX18cZdy73S9d7Wi+s4zKf5/m7wNIsZaMQY0C5Yfie5LWxTJCuzJViPyBpI2CaaE2K+7Gl0OrL3kwx16E8II5PpyW+xfRAgtpF1PQEg3ggxc3csLEfGelHOxtVhiqkeULe97rzB2mPsIN8hH8g/VfE5Ddq0p8E2ySIAih/fW3rUnnKFbJMVDL+rc+FVES0mQdQR9ARQpj4VxjQ6Yi93tOsp3LaBJ73VSyNZaAlnvf0vcXL201DAnuGclrX6F5u48gDSv4F2/cCRuGJ3yGtmOppnFLD0zGy47jzXhCaXOq343VEAzeuCeyBrkAh5Sb3C6f1tjrq7jTGp5GhlkDn+jIGGS3TCQsesuPFcIi8B299YLuJqV+wsBblC91M6k8aL9cJAPqPmWr72sShPT3rSk3IfDbZ0SfsO2RdGxarIc5YwbFGitpmumuUDOE7Sdp7kjHf/dQjJkxJ+IpRSk6tLtrhdhab1DMFEJzhNb69CTPBiHl0sI0L1/CPcuphVg8wT2UOhxqurrM8Cx3D0sIt3c68C2tC9cN81tpYmxM5ucPC+wN5I5c1B/DTcVKQDfj/A/JWhuufk8iKRPtsfoAU6m57F/qlbGCfhu/fJcEkLCSmuhiwbVXUScgaB+OMudyoDkArjH8N5d4qLVrYT5a19Xku8jgeSG/nCViSwVXE4FPLlokCQl1+DPFFEPjd8xuF/nxTd9VFMquCRHlBc+o/ejL2seEmiL7D8C1w8L79OeNFkVorEE2w6ieVnsiVQhkvr5YrZX8YGb4t893HxDcLag6+dZbaxyoCTttqt+fs6zPM2i5OqftU+qsOmqENnhpszG+Pry5mSZFXrZLku2en3iIcpmWDO3sIo18slSXosqXML+niv4g6P5VpBM7pWbw7hl68gbF1ym/iF6fZzddAqurerSf+h187lqxK1Og2ePw4OFTuSXAiKkuPEvZzAVEPq/cXr021YHjZyv G7yPHX3r 5VCaYr/PCE9Ru4HszM8FwA5mXO4M9p4oEvWzmSN9puwitavTmilC1wpy9TAl0M6M6XrePJrkzS9uawSlf2+BUpj6weZD1vVDcU35137CjGLLJushL1VzlPssx2r38smH1jm6z+08SccEmCzmp+pUFW/AWxM5AEsVMD7gLYJcxSB/wAmC0ag0hBX0av7GRpvvQL9D5kaoIgSbY/OyETw14ZdyqEc6lUYaQG79ZQ2/fiyGhiaAZRCjzFU6P/tKpZnFMPoVp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch is based on Herbert Xu's request chaining for ahash ("[PATCH 2/6] crypto: hash - Add request chaining API") [1]. The generic framework for request chaining that's provided in the ahash implementation has been used as reference to develop a similar synchronous request chaining framework for crypto_acomp. Furthermore, this commit develops an asynchronous request chaining framework and API that iaa_crypto can use for request chaining with parallelism, in order to fully benefit from Intel IAA's multiple compress/decompress engines in hardware. This allows us to gain significant latency improvements with IAA batching as compared to synchronous request chaining. Usage of acomp request chaining API: ==================================== Any crypto_acomp compressor can avail of request chaining as follows: Step 1: Create request chain: Request 0 (the first req in the chain): void acomp_reqchain_init(struct acomp_req *req, u32 flags, crypto_completion_t compl, void *data); Subsequent requests: void acomp_request_chain(struct acomp_req *req, struct acomp_req *head); Step 2: Process the request chain using the specified compress/decompress "op": 2.a) Synchronous: the chain of requests is processed in series: int acomp_do_req_chain(struct acomp_req *req, int (*op)(struct acomp_req *req)); 2.b) Asynchronous: the chain of requests is processed in parallel using a submit-poll paradigm: int acomp_do_async_req_chain(struct acomp_req *req, int (*op_submit)(struct acomp_req *req), int (*op_poll)(struct acomp_req *req)); Request chaining will be used in subsequent patches to implement compress/decompress batching in the iaa_crypto driver for the two supported IAA driver sync_modes: sync_mode = 'sync' will use (2.a), sync_mode = 'async' will use (2.b). These files are directly re-used from [1] which is not yet merged: include/crypto/algapi.h include/linux/crypto.h Hence, I am adding Herbert as the co-developer of this acomp request chaining patch. [1]: https://lore.kernel.org/linux-crypto/677614fbdc70b31df2e26483c8d2cd1510c8af91.1730021644.git.herbert@gondor.apana.org.au/ Suggested-by: Herbert Xu Signed-off-by: Kanchana P Sridhar Co-developed-by: Herbert Xu Signed-off-by: --- crypto/acompress.c | 284 ++++++++++++++++++++++++++++ include/crypto/acompress.h | 41 ++++ include/crypto/algapi.h | 10 + include/crypto/internal/acompress.h | 10 + include/linux/crypto.h | 31 +++ 5 files changed, 376 insertions(+) diff --git a/crypto/acompress.c b/crypto/acompress.c index 6fdf0ff9f3c0..cb6444d09dd7 100644 --- a/crypto/acompress.c +++ b/crypto/acompress.c @@ -23,6 +23,19 @@ struct crypto_scomp; static const struct crypto_type crypto_acomp_type; +struct acomp_save_req_state { + struct list_head head; + struct acomp_req *req0; + struct acomp_req *cur; + int (*op)(struct acomp_req *req); + crypto_completion_t compl; + void *data; +}; + +static void acomp_reqchain_done(void *data, int err); +static int acomp_save_req(struct acomp_req *req, crypto_completion_t cplt); +static void acomp_restore_req(struct acomp_req *req); + static inline struct acomp_alg *__crypto_acomp_alg(struct crypto_alg *alg) { return container_of(alg, struct acomp_alg, calg.base); @@ -123,6 +136,277 @@ struct crypto_acomp *crypto_alloc_acomp_node(const char *alg_name, u32 type, } EXPORT_SYMBOL_GPL(crypto_alloc_acomp_node); +static int acomp_save_req(struct acomp_req *req, crypto_completion_t cplt) +{ + struct crypto_acomp *tfm = crypto_acomp_reqtfm(req); + struct acomp_save_req_state *state; + gfp_t gfp; + u32 flags; + + if (!acomp_is_async(tfm)) + return 0; + + flags = acomp_request_flags(req); + gfp = (flags & CRYPTO_TFM_REQ_MAY_SLEEP) ? GFP_KERNEL : GFP_ATOMIC; + state = kmalloc(sizeof(*state), gfp); + if (!state) + return -ENOMEM; + + state->compl = req->base.complete; + state->data = req->base.data; + state->req0 = req; + + req->base.complete = cplt; + req->base.data = state; + + return 0; +} + +static void acomp_restore_req(struct acomp_req *req) +{ + struct crypto_acomp *tfm = crypto_acomp_reqtfm(req); + struct acomp_save_req_state *state; + + if (!acomp_is_async(tfm)) + return; + + state = req->base.data; + + req->base.complete = state->compl; + req->base.data = state->data; + kfree(state); +} + +static int acomp_reqchain_finish(struct acomp_save_req_state *state, + int err, u32 mask) +{ + struct acomp_req *req0 = state->req0; + struct acomp_req *req = state->cur; + struct acomp_req *n; + + req->base.err = err; + + if (req == req0) + INIT_LIST_HEAD(&req->base.list); + else + list_add_tail(&req->base.list, &req0->base.list); + + list_for_each_entry_safe(req, n, &state->head, base.list) { + list_del_init(&req->base.list); + + req->base.flags &= mask; + req->base.complete = acomp_reqchain_done; + req->base.data = state; + state->cur = req; + err = state->op(req); + + if (err == -EINPROGRESS) { + if (!list_empty(&state->head)) + err = -EBUSY; + goto out; + } + + if (err == -EBUSY) + goto out; + + req->base.err = err; + list_add_tail(&req->base.list, &req0->base.list); + } + + acomp_restore_req(req0); + +out: + return err; +} + +static void acomp_reqchain_done(void *data, int err) +{ + struct acomp_save_req_state *state = data; + crypto_completion_t compl = state->compl; + + data = state->data; + + if (err == -EINPROGRESS) { + if (!list_empty(&state->head)) + return; + goto notify; + } + + err = acomp_reqchain_finish(state, err, CRYPTO_TFM_REQ_MAY_BACKLOG); + if (err == -EBUSY) + return; + +notify: + compl(data, err); +} + +int acomp_do_req_chain(struct acomp_req *req, + int (*op)(struct acomp_req *req)) +{ + struct crypto_acomp *tfm = crypto_acomp_reqtfm(req); + struct acomp_save_req_state *state; + struct acomp_save_req_state state0; + int err = 0; + + if (!acomp_request_chained(req) || list_empty(&req->base.list) || + !crypto_acomp_req_chain(tfm)) + return op(req); + + state = &state0; + + if (acomp_is_async(tfm)) { + err = acomp_save_req(req, acomp_reqchain_done); + if (err) { + struct acomp_req *r2; + + req->base.err = err; + list_for_each_entry(r2, &req->base.list, base.list) + r2->base.err = err; + + return err; + } + + state = req->base.data; + } + + state->op = op; + state->cur = req; + INIT_LIST_HEAD(&state->head); + list_splice(&req->base.list, &state->head); + + err = op(req); + if (err == -EBUSY || err == -EINPROGRESS) + return -EBUSY; + + return acomp_reqchain_finish(state, err, ~0); +} +EXPORT_SYMBOL_GPL(acomp_do_req_chain); + +static void acomp_async_reqchain_done(struct acomp_req *req0, + struct list_head *state, + int (*op_poll)(struct acomp_req *req)) +{ + struct acomp_req *req, *n; + bool req0_done = false; + int err; + + while (!list_empty(state)) { + + if (!req0_done) { + err = op_poll(req0); + if (!(err == -EAGAIN || err == -EINPROGRESS || err == -EBUSY)) { + req0->base.err = err; + req0_done = true; + } + } + + list_for_each_entry_safe(req, n, state, base.list) { + err = op_poll(req); + + if (err == -EAGAIN || err == -EINPROGRESS || err == -EBUSY) + continue; + + req->base.err = err; + list_del_init(&req->base.list); + list_add_tail(&req->base.list, &req0->base.list); + } + } + + while (!req0_done) { + err = op_poll(req0); + if (!(err == -EAGAIN || err == -EINPROGRESS || err == -EBUSY)) { + req0->base.err = err; + break; + } + } +} + +static int acomp_async_reqchain_finish(struct acomp_req *req0, + struct list_head *state, + int (*op_submit)(struct acomp_req *req), + int (*op_poll)(struct acomp_req *req)) +{ + struct acomp_req *req, *n; + int err = 0; + + INIT_LIST_HEAD(&req0->base.list); + + list_for_each_entry_safe(req, n, state, base.list) { + BUG_ON(req == req0); + + err = op_submit(req); + + if (!(err == -EINPROGRESS || err == -EBUSY)) { + req->base.err = err; + list_del_init(&req->base.list); + list_add_tail(&req->base.list, &req0->base.list); + } + } + + acomp_async_reqchain_done(req0, state, op_poll); + + return req0->base.err; +} + +int acomp_do_async_req_chain(struct acomp_req *req, + int (*op_submit)(struct acomp_req *req), + int (*op_poll)(struct acomp_req *req)) +{ + struct crypto_acomp *tfm = crypto_acomp_reqtfm(req); + struct list_head state; + struct acomp_req *r2; + int err = 0; + void *req0_data = req->base.data; + + if (!acomp_request_chained(req) || list_empty(&req->base.list) || + !acomp_is_async(tfm) || !crypto_acomp_req_chain(tfm)) { + + err = op_submit(req); + + if (err == -EINPROGRESS || err == -EBUSY) { + bool req0_done = false; + + while (!req0_done) { + err = op_poll(req); + if (!(err == -EAGAIN || err == -EINPROGRESS || err == -EBUSY)) { + req->base.err = err; + break; + } + } + } else { + req->base.err = err; + } + + req->base.data = req0_data; + if (acomp_is_async(tfm)) + req->base.complete(req->base.data, req->base.err); + + return err; + } + + err = op_submit(req); + req->base.err = err; + + if (err && !(err == -EINPROGRESS || err == -EBUSY)) + goto err_prop; + + INIT_LIST_HEAD(&state); + list_splice(&req->base.list, &state); + + err = acomp_async_reqchain_finish(req, &state, op_submit, op_poll); + req->base.data = req0_data; + req->base.complete(req->base.data, req->base.err); + + return err; + +err_prop: + list_for_each_entry(r2, &req->base.list, base.list) + r2->base.err = err; + + return err; +} +EXPORT_SYMBOL_GPL(acomp_do_async_req_chain); + struct acomp_req *acomp_request_alloc(struct crypto_acomp *acomp) { struct crypto_tfm *tfm = crypto_acomp_tfm(acomp); diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h index 54937b615239..eadc24514056 100644 --- a/include/crypto/acompress.h +++ b/include/crypto/acompress.h @@ -206,6 +206,7 @@ static inline void acomp_request_set_callback(struct acomp_req *req, req->base.data = data; req->base.flags &= CRYPTO_ACOMP_ALLOC_OUTPUT; req->base.flags |= flgs & ~CRYPTO_ACOMP_ALLOC_OUTPUT; + req->base.flags &= ~CRYPTO_TFM_REQ_CHAIN; } /** @@ -237,6 +238,46 @@ static inline void acomp_request_set_params(struct acomp_req *req, req->flags |= CRYPTO_ACOMP_ALLOC_OUTPUT; } +static inline u32 acomp_request_flags(struct acomp_req *req) +{ + return req->base.flags; +} + +static inline void acomp_reqchain_init(struct acomp_req *req, + u32 flags, crypto_completion_t compl, + void *data) +{ + acomp_request_set_callback(req, flags, compl, data); + crypto_reqchain_init(&req->base); +} + +static inline void acomp_reqchain_clear(struct acomp_req *req, void *data) +{ + struct crypto_wait *wait = (struct crypto_wait *)data; + reinit_completion(&wait->completion); + crypto_reqchain_clear(&req->base); + acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, data); +} + +static inline void acomp_request_chain(struct acomp_req *req, + struct acomp_req *head) +{ + crypto_request_chain(&req->base, &head->base); +} + +int acomp_do_req_chain(struct acomp_req *req, + int (*op)(struct acomp_req *req)); + +int acomp_do_async_req_chain(struct acomp_req *req, + int (*op_submit)(struct acomp_req *req), + int (*op_poll)(struct acomp_req *req)); + +static inline int acomp_request_err(struct acomp_req *req) +{ + return req->base.err; +} + /** * crypto_acomp_compress() -- Invoke asynchronous compress operation * diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h index 156de41ca760..c5df380c7d08 100644 --- a/include/crypto/algapi.h +++ b/include/crypto/algapi.h @@ -271,4 +271,14 @@ static inline u32 crypto_tfm_alg_type(struct crypto_tfm *tfm) return tfm->__crt_alg->cra_flags & CRYPTO_ALG_TYPE_MASK; } +static inline bool crypto_request_chained(struct crypto_async_request *req) +{ + return req->flags & CRYPTO_TFM_REQ_CHAIN; +} + +static inline bool crypto_tfm_req_chain(struct crypto_tfm *tfm) +{ + return tfm->__crt_alg->cra_flags & CRYPTO_ALG_REQ_CHAIN; +} + #endif /* _CRYPTO_ALGAPI_H */ diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h index 8831edaafc05..53b4ef59b48c 100644 --- a/include/crypto/internal/acompress.h +++ b/include/crypto/internal/acompress.h @@ -84,6 +84,16 @@ static inline void __acomp_request_free(struct acomp_req *req) kfree_sensitive(req); } +static inline bool acomp_request_chained(struct acomp_req *req) +{ + return crypto_request_chained(&req->base); +} + +static inline bool crypto_acomp_req_chain(struct crypto_acomp *tfm) +{ + return crypto_tfm_req_chain(&tfm->base); +} + /** * crypto_register_acomp() -- Register asynchronous compression algorithm * diff --git a/include/linux/crypto.h b/include/linux/crypto.h index b164da5e129e..7c27d557586c 100644 --- a/include/linux/crypto.h +++ b/include/linux/crypto.h @@ -13,6 +13,8 @@ #define _LINUX_CRYPTO_H #include +#include +#include #include #include #include @@ -124,6 +126,9 @@ */ #define CRYPTO_ALG_FIPS_INTERNAL 0x00020000 +/* Set if the algorithm supports request chains. */ +#define CRYPTO_ALG_REQ_CHAIN 0x00040000 + /* * Transform masks and values (for crt_flags). */ @@ -133,6 +138,7 @@ #define CRYPTO_TFM_REQ_FORBID_WEAK_KEYS 0x00000100 #define CRYPTO_TFM_REQ_MAY_SLEEP 0x00000200 #define CRYPTO_TFM_REQ_MAY_BACKLOG 0x00000400 +#define CRYPTO_TFM_REQ_CHAIN 0x00000800 /* * Miscellaneous stuff. @@ -174,6 +180,7 @@ struct crypto_async_request { struct crypto_tfm *tfm; u32 flags; + int err; }; /** @@ -540,5 +547,29 @@ int crypto_comp_decompress(struct crypto_comp *tfm, const u8 *src, unsigned int slen, u8 *dst, unsigned int *dlen); +static inline void crypto_reqchain_init(struct crypto_async_request *req) +{ + req->err = -EINPROGRESS; + req->flags |= CRYPTO_TFM_REQ_CHAIN; + INIT_LIST_HEAD(&req->list); +} + +static inline void crypto_reqchain_clear(struct crypto_async_request *req) +{ + req->flags &= ~CRYPTO_TFM_REQ_CHAIN; +} + +static inline void crypto_request_chain(struct crypto_async_request *req, + struct crypto_async_request *head) +{ + req->err = -EINPROGRESS; + list_add_tail(&req->list, &head->list); +} + +static inline bool crypto_tfm_is_async(struct crypto_tfm *tfm) +{ + return tfm->__crt_alg->cra_flags & CRYPTO_ALG_ASYNC; +} + #endif /* _LINUX_CRYPTO_H */ From patchwork Thu Feb 6 07:20:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962452 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9FA7C0219B for ; Thu, 6 Feb 2025 07:21:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E1E56B0085; Thu, 6 Feb 2025 02:21:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7417A6B0088; Thu, 6 Feb 2025 02:21:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5930F6B0089; Thu, 6 Feb 2025 02:21:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 36A316B0085 for ; Thu, 6 Feb 2025 02:21:09 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D483E1C88C2 for ; Thu, 6 Feb 2025 07:21:08 +0000 (UTC) X-FDA: 83088673416.22.F1C796D Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf07.hostedemail.com (Postfix) with ESMTP id A627640002 for ; Thu, 6 Feb 2025 07:21:06 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Y7TCIauq; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826466; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C3rPPGLrxyrK3EPMkpoxCITeN6Dqq/YdbbcRfprMcTU=; b=aLgE/nFkaedRF/F6WpJ2Y5kx9ZacyInrVBMBeSaUW7sD1yX65o9IwmakbtqCKEXIXYzmCt vWYyIFy2TBVHNRPFuqqhaUdBulZ278/tQP6CX484gTHC257BK0WgBri4+4w73hhsQi3Hoq F3/GHznfv7vC/MRXwwMJEpAnzZaVENE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826466; a=rsa-sha256; cv=none; b=URYBDhQ28opJUNr3W4qL6jhmRLB0jxuTFHb+A3j73ON6ICLZfjWqHInfFWd0l9RCIGMwkS IZSl9Yg0YfURpXYJLulHcYS0nJUPBcsZDLvoMdgBywz1Da0u1g0X0kCziMLjPAO1cFvC6+ qRjYw+2IiHZWZ5Shq6TGGE4b/6vOizU= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Y7TCIauq; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826467; x=1770362467; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+lL68B5VDme0ssy20KVs9HZUu1uuo/887sCYSoPmBdw=; b=Y7TCIauqpeuiVF846Pd9MIbY2v7OoXVko7iipEGoIaRc8VuU76gaULML KGvVT2OgORu1Jgn04WWZ3lbSV+zYgVd7kz1/KWRbad6OMx3Arcp7K7AWU 1j8QQiP3sd87PYBOrLWeMjkYz15W8ixYovsgYWONJ0prsU/mGQdrMRleN 0nrQl3dp/yHe0dMqUOwlcHy74nTQxmPvmg0WYkuBHnj4jcLfaGxwovrvg 1HDZXbDHRQHWrmSr7nbQN7NzzeMxsMfCwoyg0G4KKeNx4+t/Z/nzlkKM9 rH2rOoih9WRlGbTFwzPuIzQ384GMTZVguZyiGYDcupMB//jOGEFtUBrdr A==; X-CSE-ConnectionGUID: X7NHG4znROCg59lhkFxaLw== X-CSE-MsgGUID: t7CEVwqWTAWKistu0mWvwg== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962581" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962581" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:03 -0800 X-CSE-ConnectionGUID: MlGTuHLKSGqLU3rmNEC52Q== X-CSE-MsgGUID: xD8N+IO/Rm2SXpljUYj/EA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022596" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:03 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 02/16] crypto: acomp - Define new interfaces for compress/decompress batching. Date: Wed, 5 Feb 2025 23:20:48 -0800 Message-Id: <20250206072102.29045-3-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: 94utk6gne9xr6yr8ms7wp73y8p73db1p X-Rspam-User: X-Rspamd-Queue-Id: A627640002 X-Rspamd-Server: rspam03 X-HE-Tag: 1738826466-413195 X-HE-Meta: U2FsdGVkX18PGvuhLpR3qJQJf2uTyjlfkLwtLN1itlJErHC4dO3RsU61XXntrTmXMb98fDjsffE2xPBG3HGiyPiBGmS9VjEXYoNjyFoOdxEhhu26s3r+QYbLHMpXEH9vd0E7aYrHZEiqhcH3hNOQIx7McHFOWlfRoJCCE9CNp1Vfz9a9UgB15PxYqI9OB2R32LCYZ/CzDjk0fm8F/xPxbg+R8VYIyFsl3XbGfr4X2gqYlkYr5Lue7nQcp4oQjftcGHlVvGX6GXwxyrXGuHUZVNBkDQCplJ8IVsNllqjHHo3EflUgZ+F3nQxcAYWVzQgz8KorWCzwOqamw9V4/gXlPlxSyYA8ZJi96glU690c5e9XxMA2CDa5PDihngXSVlFeRLP90X987X2aHa+fZwiWkH4u+a46jXYOdxLAnLDxBa+maT6j7G0T+wwmVMNnGHFGPPkvsOlze/vNslHELBrj7Gxx/sitxVXbmnikEAtqTvods1EYtQv2i4/59hk1XkFJtQ0yXCRI/wTEoNvkTDAt6irXN1QirpK9uuWxT/SanYYJV8cb+CrZsd9Mo2ocKjm0y7UUyFnC81E4Wevdkv1BoSJ/ECXFKWN7w8iroTEYyjoAKR5zxjdIU+h54oJTC8TueTMKcf2AeG15CwvZLaUbBkpQ4DKcWCXf59DC3HxEboQ2lAWuC6WbmC6XDVxLQ/Qi2QtACx/Tz7GhDRULIjLWF/vefe4tPjGF67no0gmfF3hGxcwEp6pJ8qHrXhhTxRYX9ANVW4MgSOq5uNRk5WDSn8WkzC9JMdym9+1tdlCp3RYElZZKDwe+9+T2y+Mb7DC8shSQSE+vPxzd3Akkjm5YhX9HJYtMVlRnf0fj9LQnobhgq0VoadJU+j1C/01wS510v53J5z8FKXE8sxZ8gXzY21rXi0akL26p/uXWZ0O5yX+hcafPznqUehmABtTs5ERS4294iC69S+7rSuAM3Y5 aL4I1+c5 SWfIuwhyepFk4wHUZ0+NsixXpG95npI4hETQ4fIV3ZZLLLfF0UobY6QbiQNaiRguQp2d9bMCeXsUbzOjS5wM3QaIzqhbGCHL6Pn1nLYQoIiTobK8MX12WnESJ16FWuSlkkEKt/6yPYVLkcdHKH4ykQfrIIuD4O1z4H6TG6Q+wD2deqaH2RTu00e0+1g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This commit adds get_batch_size(), batch_compress() and batch_decompress() interfaces to: struct acomp_alg struct crypto_acomp A crypto_acomp compression algorithm that supports batching of compressions and decompressions must provide implementations for these API. A new helper function acomp_has_async_batching() can be invoked to query if a crypto_acomp has registered these batching interfaces. A higher level module like zswap can call acomp_has_async_batching() to detect if the compressor supports batching, and if so, it can call the new crypto_acomp_batch_size() to detect the maximum batch-size supported by the compressor. Based on this, zswap can use the minimum of any zswap-specific upper limits for batch-size and the compressor's max batch-size, to allocate batching resources. This allows the iaa_crypto Intel IAA driver to register implementations for the get_batch_size(), batch_compress() and batch_decompress() acomp_alg interfaces, that can subsequently be invoked from the kernel zswap/zram modules to compress/decompress pages in parallel in the IAA hardware accelerator to improve swapout/swapin performance through these newly added corresponding crypto_acomp API: crypto_acomp_batch_size() crypto_acomp_batch_compress() crypto_acomp_batch_decompress() Signed-off-by: Kanchana P Sridhar --- crypto/acompress.c | 3 + include/crypto/acompress.h | 111 ++++++++++++++++++++++++++++ include/crypto/internal/acompress.h | 19 +++++ 3 files changed, 133 insertions(+) diff --git a/crypto/acompress.c b/crypto/acompress.c index cb6444d09dd7..165559a8b9bd 100644 --- a/crypto/acompress.c +++ b/crypto/acompress.c @@ -84,6 +84,9 @@ static int crypto_acomp_init_tfm(struct crypto_tfm *tfm) acomp->compress = alg->compress; acomp->decompress = alg->decompress; + acomp->get_batch_size = alg->get_batch_size; + acomp->batch_compress = alg->batch_compress; + acomp->batch_decompress = alg->batch_decompress; acomp->dst_free = alg->dst_free; acomp->reqsize = alg->reqsize; diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h index eadc24514056..8451ade70fd8 100644 --- a/include/crypto/acompress.h +++ b/include/crypto/acompress.h @@ -43,6 +43,10 @@ struct acomp_req { * * @compress: Function performs a compress operation * @decompress: Function performs a de-compress operation + * @get_batch_size: Maximum batch-size for batching compress/decompress + * operations. + * @batch_compress: Function performs a batch compress operation + * @batch_decompress: Function performs a batch decompress operation * @dst_free: Frees destination buffer if allocated inside the * algorithm * @reqsize: Context size for (de)compression requests @@ -51,6 +55,21 @@ struct acomp_req { struct crypto_acomp { int (*compress)(struct acomp_req *req); int (*decompress)(struct acomp_req *req); + unsigned int (*get_batch_size)(void); + bool (*batch_compress)(struct acomp_req *reqs[], + struct crypto_wait *wait, + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages); + bool (*batch_decompress)(struct acomp_req *reqs[], + struct crypto_wait *wait, + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages); void (*dst_free)(struct scatterlist *dst); unsigned int reqsize; struct crypto_tfm base; @@ -142,6 +161,13 @@ static inline bool acomp_is_async(struct crypto_acomp *tfm) CRYPTO_ALG_ASYNC; } +static inline bool acomp_has_async_batching(struct crypto_acomp *tfm) +{ + return (acomp_is_async(tfm) && + (crypto_comp_alg_common(tfm)->base.cra_flags & CRYPTO_ALG_TYPE_ACOMPRESS) && + tfm->get_batch_size && tfm->batch_compress && tfm->batch_decompress); +} + static inline struct crypto_acomp *crypto_acomp_reqtfm(struct acomp_req *req) { return __crypto_acomp_tfm(req->base.tfm); @@ -306,4 +332,89 @@ static inline int crypto_acomp_decompress(struct acomp_req *req) return crypto_acomp_reqtfm(req)->decompress(req); } +/** + * crypto_acomp_batch_size() -- Get the algorithm's batch size + * + * Function returns the algorithm's batch size for batching operations + * + * @tfm: ACOMPRESS tfm handle allocated with crypto_alloc_acomp() + * + * Return: crypto_acomp's batch size. + */ +static inline unsigned int crypto_acomp_batch_size(struct crypto_acomp *tfm) +{ + if (acomp_has_async_batching(tfm)) + return tfm->get_batch_size(); + + return 1; +} + +/** + * crypto_acomp_batch_compress() -- Invoke asynchronous compress of + * a batch of requests + * + * Function invokes the asynchronous batch compress operation + * + * @reqs: @nr_pages asynchronous compress requests. + * @wait: crypto_wait for acomp batch compress with synchronous/asynchronous + * request chaining. If NULL, the driver must provide a way to process + * request completions asynchronously. + * @pages: Pages to be compressed. + * @dsts: Pre-allocated destination buffers to store results of compression. + * @dlens: Will contain the compressed lengths. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_pages: The number of pages to be compressed. + * + * Returns true if all compress requests complete successfully, + * false otherwise. + */ +static inline bool crypto_acomp_batch_compress(struct acomp_req *reqs[], + struct crypto_wait *wait, + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages) +{ + struct crypto_acomp *tfm = crypto_acomp_reqtfm(reqs[0]); + + return tfm->batch_compress(reqs, wait, pages, dsts, + dlens, errors, nr_pages); +} + +/** + * crypto_acomp_batch_decompress() -- Invoke asynchronous decompress of + * a batch of requests + * + * Function invokes the asynchronous batch decompress operation + * + * @reqs: @nr_pages asynchronous decompress requests. + * @wait: crypto_wait for acomp batch decompress with synchronous/asynchronous + * request chaining. If NULL, the driver must provide a way to process + * request completions asynchronously. + * @srcs: The src buffers to be decompressed. + * @pages: The pages to store the decompressed buffers. + * @slens: Compressed lengths of @srcs. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_pages: The number of pages to be decompressed. + * + * Returns true if all decompress requests complete successfully, + * false otherwise. + */ +static inline bool crypto_acomp_batch_decompress(struct acomp_req *reqs[], + struct crypto_wait *wait, + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages) +{ + struct crypto_acomp *tfm = crypto_acomp_reqtfm(reqs[0]); + + return tfm->batch_decompress(reqs, wait, srcs, pages, + slens, errors, nr_pages); +} + #endif diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h index 53b4ef59b48c..df0e192801ff 100644 --- a/include/crypto/internal/acompress.h +++ b/include/crypto/internal/acompress.h @@ -17,6 +17,10 @@ * * @compress: Function performs a compress operation * @decompress: Function performs a de-compress operation + * @get_batch_size: Maximum batch-size for batching compress/decompress + * operations. + * @batch_compress: Function performs a batch compress operation + * @batch_decompress: Function performs a batch decompress operation * @dst_free: Frees destination buffer if allocated inside the algorithm * @init: Initialize the cryptographic transformation object. * This function is used to initialize the cryptographic @@ -37,6 +41,21 @@ struct acomp_alg { int (*compress)(struct acomp_req *req); int (*decompress)(struct acomp_req *req); + unsigned int (*get_batch_size)(void); + bool (*batch_compress)(struct acomp_req *reqs[], + struct crypto_wait *wait, + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages); + bool (*batch_decompress)(struct acomp_req *reqs[], + struct crypto_wait *wait, + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages); void (*dst_free)(struct scatterlist *dst); int (*init)(struct crypto_acomp *tfm); void (*exit)(struct crypto_acomp *tfm); From patchwork Thu Feb 6 07:20:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89089C02196 for ; Thu, 6 Feb 2025 07:21:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65CA26B0088; Thu, 6 Feb 2025 02:21:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 608EF280001; Thu, 6 Feb 2025 02:21:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AD056B008A; Thu, 6 Feb 2025 02:21:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 215926B0088 for ; Thu, 6 Feb 2025 02:21:10 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E1828160F5D for ; Thu, 6 Feb 2025 07:21:09 +0000 (UTC) X-FDA: 83088673458.04.4B9CEE9 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf26.hostedemail.com (Postfix) with ESMTP id CB766140002 for ; Thu, 6 Feb 2025 07:21:07 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Hti2Atbt; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826468; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YD+ICK81+HgzK5rOlYxzVxOHxfGIIKv/2BMPBaqlY3U=; b=hGPJGwV4DJtczsPCfGn9Zx/DCQsKOd3Wik0uwEdOkgmtYdeAOjTkSp0lMpHHQN9PVwLQ9C QgOqX0YB7QCi+S2TPDKwNMcEFcaSsGD8dB5GyH/stIkLIWqlbNkjhxyYo7mygivkGBd21c nT5eK3JQPYJ9g2LPIALR/WvgTjjtkVQ= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Hti2Atbt; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826468; a=rsa-sha256; cv=none; b=wCZWQtZg2m2UAYbkGRZC8hGo8ASNiLFBIkEWGtcvO8DKe4utfMp/j27Zwn4iZcSQ5O0qkQ 7Knp7d9k2MsMNLJSCZVeSOBCYF7ZH8HDzLReto18AKGNprWsj5nydtSPFK2NfOQaBt4Lts b72YO27AvuuZjtIbE4mxhzMUr0Tktl4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826468; x=1770362468; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=aa0CRR/xEnOe7ecidFKr7n4dRE0EgIea9NyU6tuNilQ=; b=Hti2Atbt/RGtd56WyLWh2WKiJuNILxjScoWYMHo7Bvw8geiC70QVErez 6niHRv04f/M2IU/rfsB4cvxQpXyY+pgKf8C9ECFcShvVjUO0fwX5Jz+yE rOHpNGSj1Mfhqjd9MhghVFJ6JCXYQRrDAEoetl7zJP04TlICz0QN6fAFA SylZ2G9gOL560BP0efBRujRMMeb5rTKEo9RAY5GWGK40Nf9uxvF7JxD3W s9coSB1gzdN7/7WMYQkwh8dCxG0hHdajIhg0w7tqKC5nIGwG/tbIgJYwY cxupWeESprM8s8WhWaDr44RXkSrRhv7Z6nWTQ5Momski3LT0nEZLv5fef w==; X-CSE-ConnectionGUID: MG45gd7TR9uG4xR9dm5Y+A== X-CSE-MsgGUID: jbQUXx5sTzWK4oJkZ1RiTw== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962589" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962589" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:03 -0800 X-CSE-ConnectionGUID: 0qv2Q5XmQcmCbrrd3qHiSA== X-CSE-MsgGUID: ZBp6cQVISTO02ezo4hUqhA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022600" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:04 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 03/16] crypto: iaa - Add an acomp_req flag CRYPTO_ACOMP_REQ_POLL to enable async mode. Date: Wed, 5 Feb 2025 23:20:49 -0800 Message-Id: <20250206072102.29045-4-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CB766140002 X-Stat-Signature: ury8jahb58xqb88fadt8pqzx5qur37tu X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738826467-386456 X-HE-Meta: U2FsdGVkX1/dcQhBqfEQ8jTtMXt6VdPtNk489HzBYwZnShC/Xls2cQfnp8fWPdXh0VIiONeI7FSYVlZCh4KgWcHjs1zQURJESbXSPLrKGbRNps1BDk+USzkQgcpgQqUFbh5Vs1yPrtb5QuMsrYMPm1Jbeo3RwB8gbvmg0+iBkm0DxDu/v/i864U+r1Tf0pgP6IkpuiueXa9bmEMDpHF9VLPNg6PfvQEKmJjO0HHbFTWXwZJ8bj87FYC0dYcZx45buieqqik1LWALrIhwUvvwtDJRQ5fg4FiVrr8YD1emwhDHvO0qQffqEm85klHPyCH2giomYyQYfKDYIKqq3NAY6unSb66J2bF2f9nxc40qxZz5+WuhWSz0T1uCTic/zJdMXh4dzZj0ybY18LpztESRGMePZ9l09ORyCALU+21B4TQDfocgyCD8x3wgucbTCXx30zGWKiL18BB8zA7oYD/ONB60TqEefUMR28tO+8qhhgWEjgjP0gGWx/A+SBzR1PWlWRU7LUoBhL+49e1q5nL6Cc6YKIAo5nKIsR2+TGojMZH+pq9RGmjSDanBvq0xezjY8CflqD1zh6Tf1oBEw2UhHX8vflNJ+P22C8oq3kRXzbDilmwgBWi07D+qneJnCne/eWsyPkdX/y0tPQCqWULwlRIL0Tfkq0lapwawrM283VkzAYIq+l2+A8oTEARSCLytwLdBdLHEl8zUukAXMfsT9fSxj7ZaFhWS4tGdSedJvy6hKP1yx05hcreZwXQ85KxHse3jaJGwNxqfX3mXpChFgKPr+6VlfuvDrPvS6QnMYCd/cVxEAKwxILXvfhTW4b3A3ZTyBTGAB1rNDyskM3OxP9y874YjPVXff/3s92wQA+HFtVWTFCIrGID4lhob7/TADq8gnnbRxfNjCmX3zvi+cbxvV3B+9k9Ak2aGaov5J3ZFAQGlCUjSCLhx/C0NYCGZg6y3xiV9jMyeDrTOBP8 b4s1nO5p DwBoTsXfWMdx8GXmgGZTObY9CaBSPZnM92FhbellGq1fxvn5c4UXdVDFAzxIodRAVosHF81zcRYmDcz/4iEiyY0j50bpfdlvsYZpOiev0DycMHosgZVdZy6+rwsCNpqLo295RVS4WyKs+bj0AvDRYqaDNU/PlNY4bdDybr4pO8h0bhF0GTbxHaVED7vPk3sVygoaqAxKrscIhZQNElJFhUZJoy0eOQlxVyXeP5uWDL7Mvh6G1ghTegdc5v4daCjQn6t+vBRnA7O1sv5ZX0hHJpTvEDZY86LVAknXF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If the iaa_crypto driver has async_mode set to true, and use_irq set to false, it can still be forced to use synchronous mode by turning off the CRYPTO_ACOMP_REQ_POLL flag in req->flags. In other words, all three of the following need to be true for a request to be processed in fully async poll mode: 1) async_mode should be "true" 2) use_irq should be "false" 3) req->flags & CRYPTO_ACOMP_REQ_POLL should be "true" Suggested-by: Herbert Xu Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 11 ++++++++++- include/crypto/acompress.h | 5 +++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index c3776b0de51d..d7983ab3c34a 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1520,6 +1520,10 @@ static int iaa_comp_acompress(struct acomp_req *req) return -EINVAL; } + /* If the caller has requested no polling, disable async. */ + if (!(req->flags & CRYPTO_ACOMP_REQ_POLL)) + disable_async = true; + cpu = get_cpu(); wq = wq_table_next_wq(cpu); put_cpu(); @@ -1712,6 +1716,7 @@ static int iaa_comp_adecompress(struct acomp_req *req) { struct crypto_tfm *tfm = req->base.tfm; dma_addr_t src_addr, dst_addr; + bool disable_async = false; int nr_sgs, cpu, ret = 0; struct iaa_wq *iaa_wq; struct device *dev; @@ -1727,6 +1732,10 @@ static int iaa_comp_adecompress(struct acomp_req *req) return -EINVAL; } + /* If the caller has requested no polling, disable async. */ + if (!(req->flags & CRYPTO_ACOMP_REQ_POLL)) + disable_async = true; + if (!req->dst) return iaa_comp_adecompress_alloc_dest(req); @@ -1775,7 +1784,7 @@ static int iaa_comp_adecompress(struct acomp_req *req) req->dst, req->dlen, sg_dma_len(req->dst)); ret = iaa_decompress(tfm, req, wq, src_addr, req->slen, - dst_addr, &req->dlen, false); + dst_addr, &req->dlen, disable_async); if (ret == -EINPROGRESS) return ret; diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h index 8451ade70fd8..e090538e8406 100644 --- a/include/crypto/acompress.h +++ b/include/crypto/acompress.h @@ -14,6 +14,11 @@ #include #define CRYPTO_ACOMP_ALLOC_OUTPUT 0x00000001 +/* + * If set, the driver must have a way to submit the req, then + * poll its completion status for success/error. + */ +#define CRYPTO_ACOMP_REQ_POLL 0x00000002 #define CRYPTO_ACOMP_DST_MAX 131072 /** From patchwork Thu Feb 6 07:20:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A15AFC02196 for ; Thu, 6 Feb 2025 07:21:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DFB9B280002; Thu, 6 Feb 2025 02:21:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DAB08280001; Thu, 6 Feb 2025 02:21:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAE01280002; Thu, 6 Feb 2025 02:21:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8DC34280001 for ; Thu, 6 Feb 2025 02:21:10 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2A6471C88B8 for ; Thu, 6 Feb 2025 07:21:10 +0000 (UTC) X-FDA: 83088673500.17.EE5E300 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf08.hostedemail.com (Postfix) with ESMTP id E0EA4160002 for ; Thu, 6 Feb 2025 07:21:07 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=JWwT7QjX; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826468; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iOGT2aMkbNboYoSDjzDOlfOYvJJwxoHrAG/tTpyrOUU=; b=gnFrIb/A+6Av95M43QDaSctRWwuGyoYXglDGKH4v4gaIeCvsJ9va+CbpS1S7SnZzFAY8Pd VskbAwqX4Ny3XYHq4rFVs8lMmTsWZA1zxXRhn2zoxoxZXNz3cxxNmmKFjM5Hl3tQtg+4Hd g8LKOy9yYRFWVmcTcZjs6slvviB7ubM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=JWwT7QjX; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826468; a=rsa-sha256; cv=none; b=qPjvddQiYoSnbcWsh8a3Sbs0yg+z50Ie6KqPD4jYPL4JbCVTN9psFMixo1mfnSq+01+kHi 2JEV0KQmbcGxUl2dkiWHP63FFRdqiBaB1mI2w8T33KpuTL0QrIYDZdswP/b+JQI2ab/xx5 /hbIalNnHERuH9+Z8LqBOfW4OXDpEnA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826468; x=1770362468; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dcY8wHVUfujZgmvz5KO0cSP8J3+B0GuhP4os2QGOLPY=; b=JWwT7QjX3dm5mA4g1B8hti2buTPRJ0dbcH0HVKuIzSsRI9CCwGK+YGIs PYMQgyjpLIw+F81you/bJNOw81lkN4ggYm8PJqkKugqmBs3u1JNprPQ9C 2W8K5w8LNDenlPEoSqWfO05x0C7msInChgWv2Q7cwCv6D1oxcb/aqoqDp N89RQx+bxu2WLjicMjGwVjWJe9Y4wx/+ehPE3gzCotusz7dJXp98Bkknc ghKTBGpxlXMpOPMWaqnL4RQvIcqglkZ9k7nf4PCwlPSlwsZdhFzMxj3i9 lErVS5ZjT9b3X6I2GcHg1ebivsoGyRtHXGCpuBqRupKcTAXAT4RMy9Eze A==; X-CSE-ConnectionGUID: 0zAv/5HVRtOQSaTfdSbSUA== X-CSE-MsgGUID: 0TcADrHhS7+LeEx6nDJEBg== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962596" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962596" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:04 -0800 X-CSE-ConnectionGUID: dQlJdautS06LZgxZHf+H6w== X-CSE-MsgGUID: VYsq7wa+T4mvVCLRyjun2g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022604" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:04 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 04/16] crypto: iaa - Implement batch_compress(), batch_decompress() API in iaa_crypto. Date: Wed, 5 Feb 2025 23:20:50 -0800 Message-Id: <20250206072102.29045-5-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: E0EA4160002 X-Stat-Signature: iq8jz8bck8m1jieb16e464y4inbir5wk X-Rspam-User: X-HE-Tag: 1738826467-377585 X-HE-Meta: U2FsdGVkX1+OqxdyI1duC4GcjPRIdrKl3UnmJOXApPJgV0hlS22183l78IBpokH3MLRp8LBuAZydtIoRl+sblzqBRuMksBloBrq+GiwSXkwEUtEoAEk6DEjNmsUxy+MabKd7z/GMhVqne5Crp5KS3/6BWIUmMREi7hCXOczg8vJTKG6miIy3xjvdOcMjoIM4XMbmN6Q/+L4g+W6F/yxxt36TtEqnPEAggq0EKsNyP3DRjDxNHh4fSkSVKi4e1EEsAdRZ3tsbTRWmyaBcRN6EyecyHH4GM/DcsucrSYbLvRZ0ffCFYLBEOpCl6Jwx8exqLAWfNxyy+0NQgVHyK5A/uUSctiRP9tWt91Lxe2/32leLwUTyqwZsqzMhazpmEuulikLrquFQp8YrxRG4bujUgc5MRTJiJWKZEhSOoBxJVwOwrKXFgr92Zm0FGIR4OYfFyfIC7rFzMy7EUytHf8dOBDyr4AqJNZZ205v2VAaxaQIl6kAPfeQcyGhTANH+KvryXpFMlCLHoh63PNOA8CScCpBlKcKdSwBpjdwG0Sv1w4AKYRN4mD/Qs4Qvhy3Lsmxx+Q9egd0Fw/cWnuTmMqeZ2xfa1iV+S4dDExTbU2OsshHo4QcJS4WMajETDgDvaIlu9QuVjVtZFvZM2H4NXRM+r7URtNzeAmseWpXMXswTfhlvlS7cPYCsixFCjzX/Vn3/Rv3xZW6n3NjKwRcPl5meE4g0Px/4Ck24IfaDBWbvZdQx/RQXn68ci188o1bPNNuMPzA6631PAaT1ZY2KktYf9OLK70snuf0Tooh1UeWOUCC30nxMuzTW0wngqqtWhfv33Hqx/6NMtgyGLTivaTYDG/pgo5ib+NiXNsozij2MeTgAKgEa2PqywxpWwfxqqv63EYsvS8sVG0cGHkAmr4Ly9+YBaNwF59xuGStSgapku1wE3vLZ56949rDLFTLzFqhA9YLjaGWU9hgRtDjHDAd On47YV4K qLm62/sb60exNYhbEYnSpRWBr+pbBd3ptdisZYEnMpajBEPmxWX94pd7mchJmUuIe6VNCxginfQQh7FbEDCS2lK85ct/KnxKHP9s5PeMZueHVPMIJMg8YyvHE/9bCYTqJxP8vRvrstGKCvF37kLr9yicq4kjMe+hjHCDYOSZcqtnyeHVnALAsM5CZaW5dVEjvCAHWwFVMZ2ZQTE2rUFsOhUqw5wFr4a8zcdMjNST2FqzPJqkFnjS4YwyXEzXk7RCb1KvuiWehHmWjJgG0FeiMGQtCbB6soy2GNlr4OQFFA/8S34d7NlSOyHqfmg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch provides iaa_crypto driver implementations for the newly added crypto_acomp batch_compress() and batch_decompress() interfaces using acomp request chaining. iaa_crypto also implements the new crypto_acomp get_batch_size() interface that returns an iaa_driver specific constant, IAA_CRYPTO_MAX_BATCH_SIZE (set to 8U currently). This allows swap modules such as zswap/zram to allocate required batching resources and then invoke fully asynchronous batch parallel compression/decompression of pages on systems with Intel IAA, by invoking these API, respectively: crypto_acomp_batch_size(...); crypto_acomp_batch_compress(...); crypto_acomp_batch_decompress(...); This enables zswap compress batching code to be developed in a manner similar to the current single-page synchronous calls to: crypto_acomp_compress(...); crypto_acomp_decompress(...); thereby, facilitating encapsulated and modular hand-off between the kernel zswap/zram code and the crypto_acomp layer. Since iaa_crypto supports the use of acomp request chaining, this patch also adds CRYPTO_ALG_REQ_CHAIN to the iaa_acomp_fixed_deflate algorithm's cra_flags. Suggested-by: Yosry Ahmed Suggested-by: Herbert Xu Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto.h | 9 + drivers/crypto/intel/iaa/iaa_crypto_main.c | 395 ++++++++++++++++++++- 2 files changed, 403 insertions(+), 1 deletion(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto.h b/drivers/crypto/intel/iaa/iaa_crypto.h index 56985e395263..b3b67c44ec8a 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto.h +++ b/drivers/crypto/intel/iaa/iaa_crypto.h @@ -39,6 +39,15 @@ IAA_DECOMP_CHECK_FOR_EOB | \ IAA_DECOMP_STOP_ON_EOB) +/* + * The maximum compress/decompress batch size for IAA's implementation of + * the crypto_acomp batch_compress() and batch_decompress() interfaces. + * The IAA compression algorithms should provide the crypto_acomp + * get_batch_size() interface through a function that returns this + * constant. + */ +#define IAA_CRYPTO_MAX_BATCH_SIZE 8U + /* Representation of IAA workqueue */ struct iaa_wq { struct list_head list; diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index d7983ab3c34a..61134a7ad1da 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1807,6 +1807,396 @@ static void compression_ctx_init(struct iaa_compression_ctx *ctx) ctx->use_irq = use_irq; } +static int iaa_comp_poll(struct acomp_req *req) +{ + struct idxd_desc *idxd_desc; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + struct idxd_wq *wq; + bool compress_op; + int ret; + + idxd_desc = req->base.data; + if (!idxd_desc) + return -EAGAIN; + + compress_op = (idxd_desc->iax_hw->opcode == IAX_OPCODE_COMPRESS); + wq = idxd_desc->wq; + iaa_wq = idxd_wq_get_private(wq); + idxd = iaa_wq->iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + ret = check_completion(dev, idxd_desc->iax_completion, true, true); + if (ret == -EAGAIN) + return ret; + if (ret) + goto out; + + req->dlen = idxd_desc->iax_completion->output_size; + + /* Update stats */ + if (compress_op) { + update_total_comp_bytes_out(req->dlen); + update_wq_comp_bytes(wq, req->dlen); + } else { + update_total_decomp_bytes_in(req->slen); + update_wq_decomp_bytes(wq, req->slen); + } + + if (iaa_verify_compress && (idxd_desc->iax_hw->opcode == IAX_OPCODE_COMPRESS)) { + struct crypto_tfm *tfm = req->base.tfm; + dma_addr_t src_addr, dst_addr; + u32 compression_crc; + + compression_crc = idxd_desc->iax_completion->crc; + + dma_sync_sg_for_device(dev, req->dst, 1, DMA_FROM_DEVICE); + dma_sync_sg_for_device(dev, req->src, 1, DMA_TO_DEVICE); + + src_addr = sg_dma_address(req->src); + dst_addr = sg_dma_address(req->dst); + + ret = iaa_compress_verify(tfm, req, wq, src_addr, req->slen, + dst_addr, &req->dlen, compression_crc); + } +out: + /* caller doesn't call crypto_wait_req, so no acomp_request_complete() */ + + dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); + dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); + + idxd_free_desc(idxd_desc->wq, idxd_desc); + + dev_dbg(dev, "%s: returning ret=%d\n", __func__, ret); + + return ret; +} + +static unsigned int iaa_comp_get_batch_size(void) +{ + return IAA_CRYPTO_MAX_BATCH_SIZE; +} + +static void iaa_set_req_poll( + struct acomp_req *reqs[], + int nr_reqs, + bool set_flag) +{ + int i; + + for (i = 0; i < nr_reqs; ++i) { + set_flag ? (reqs[i]->flags |= CRYPTO_ACOMP_REQ_POLL) : + (reqs[i]->flags &= ~CRYPTO_ACOMP_REQ_POLL); + } +} + +/** + * This API provides IAA compress batching functionality for use by swap + * modules. + * + * @reqs: @nr_pages asynchronous compress requests. + * @wait: crypto_wait for acomp batch compress implemented using request + * chaining. Required if async_mode is "false". If async_mode is "true", + * and @wait is NULL, the completions will be processed using + * asynchronous polling of the requests' completion statuses. + * @pages: Pages to be compressed by IAA. + * @dsts: Pre-allocated destination buffers to store results of IAA + * compression. Each element of @dsts must be of size "PAGE_SIZE * 2". + * @dlens: Will contain the compressed lengths. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_pages: The number of pages, up to IAA_CRYPTO_MAX_BATCH_SIZE, + * to be compressed. + * + * Returns true if all compress requests complete successfully, + * false otherwise. + */ +static bool iaa_comp_acompress_batch( + struct acomp_req *reqs[], + struct crypto_wait *wait, + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages) +{ + struct scatterlist inputs[IAA_CRYPTO_MAX_BATCH_SIZE]; + struct scatterlist outputs[IAA_CRYPTO_MAX_BATCH_SIZE]; + bool compressions_done = false; + bool async = (async_mode && !use_irq); + bool async_poll = (async && !wait); + int i, err = 0; + + BUG_ON(nr_pages > IAA_CRYPTO_MAX_BATCH_SIZE); + BUG_ON(!async && !wait); + + if (async) + iaa_set_req_poll(reqs, nr_pages, true); + else + iaa_set_req_poll(reqs, nr_pages, false); + + /* + * Prepare and submit acomp_reqs to IAA. IAA will process these + * compress jobs in parallel if async_mode is true. + */ + for (i = 0; i < nr_pages; ++i) { + sg_init_table(&inputs[i], 1); + sg_set_page(&inputs[i], pages[i], PAGE_SIZE, 0); + + /* + * Each dst buffer should be of size (PAGE_SIZE * 2). + * Reflect same in sg_list. + */ + sg_init_one(&outputs[i], dsts[i], PAGE_SIZE * 2); + acomp_request_set_params(reqs[i], &inputs[i], + &outputs[i], PAGE_SIZE, dlens[i]); + + /* + * As long as the API is called with a valid "wait", chain the + * requests for synchronous/asynchronous compress ops. + * If async_mode is in effect, but the API is called with a + * NULL "wait", submit the requests first, and poll for + * their completion status later, after all descriptors have + * been submitted. + */ + if (!async_poll) { + /* acomp request chaining. */ + if (i) + acomp_request_chain(reqs[i], reqs[0]); + else + acomp_reqchain_init(reqs[0], 0, crypto_req_done, + wait); + } else { + errors[i] = iaa_comp_acompress(reqs[i]); + + if (errors[i] != -EINPROGRESS) { + errors[i] = -EINVAL; + err = -EINVAL; + } else { + errors[i] = -EAGAIN; + } + } + } + + if (!async_poll) { + if (async) + /* Process the request chain in parallel. */ + err = crypto_wait_req(acomp_do_async_req_chain(reqs[0], + iaa_comp_acompress, iaa_comp_poll), + wait); + else + /* Process the request chain in series. */ + err = crypto_wait_req(acomp_do_req_chain(reqs[0], + iaa_comp_acompress), wait); + + for (i = 0; i < nr_pages; ++i) { + errors[i] = acomp_request_err(reqs[i]); + if (errors[i]) { + err = -EINVAL; + pr_debug("Request chaining req %d compress error %d\n", i, errors[i]); + } else { + dlens[i] = reqs[i]->dlen; + } + } + + goto reset_reqs; + } + + /* + * Asynchronously poll for and process IAA compress job completions. + */ + while (!compressions_done) { + compressions_done = true; + + for (i = 0; i < nr_pages; ++i) { + /* + * Skip, if the compression has already completed + * successfully or with an error. + */ + if (errors[i] != -EAGAIN) + continue; + + errors[i] = iaa_comp_poll(reqs[i]); + + if (errors[i]) { + if (errors[i] == -EAGAIN) + compressions_done = false; + else + err = -EINVAL; + } else { + dlens[i] = reqs[i]->dlen; + } + } + } + +reset_reqs: + /* + * For the same 'reqs[]' to be usable by + * iaa_comp_acompress()/iaa_comp_deacompress(), + * clear the CRYPTO_ACOMP_REQ_POLL bit on all acomp_reqs, and the + * CRYPTO_TFM_REQ_CHAIN bit on the reqs[0]. + */ + iaa_set_req_poll(reqs, nr_pages, false); + if (!async_poll) + acomp_reqchain_clear(reqs[0], wait); + + return !err; +} + +/** + * This API provides IAA decompress batching functionality for use by swap + * modules. + * + * @reqs: @nr_pages asynchronous decompress requests. + * @wait: crypto_wait for acomp batch decompress implemented using request + * chaining. Required if async_mode is "false". If async_mode is "true", + * and @wait is NULL, the completions will be processed using + * asynchronous polling of the requests' completion statuses. + * @srcs: The src buffers to be decompressed by IAA. + * @pages: The pages to store the decompressed buffers. + * @slens: Compressed lengths of @srcs. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_pages: The number of pages, up to IAA_CRYPTO_MAX_BATCH_SIZE, + * to be decompressed. + * + * Returns true if all decompress requests complete successfully, + * false otherwise. + */ +static bool iaa_comp_adecompress_batch( + struct acomp_req *reqs[], + struct crypto_wait *wait, + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + int errors[], + int nr_pages) +{ + struct scatterlist inputs[IAA_CRYPTO_MAX_BATCH_SIZE]; + struct scatterlist outputs[IAA_CRYPTO_MAX_BATCH_SIZE]; + unsigned int dlens[IAA_CRYPTO_MAX_BATCH_SIZE]; + bool decompressions_done = false; + bool async = (async_mode && !use_irq); + bool async_poll = (async && !wait); + int i, err = 0; + + BUG_ON(nr_pages > IAA_CRYPTO_MAX_BATCH_SIZE); + BUG_ON(!async && !wait); + + if (async) + iaa_set_req_poll(reqs, nr_pages, true); + else + iaa_set_req_poll(reqs, nr_pages, false); + + /* + * Prepare and submit acomp_reqs to IAA. IAA will process these + * decompress jobs in parallel if async_mode is true. + */ + for (i = 0; i < nr_pages; ++i) { + dlens[i] = PAGE_SIZE; + sg_init_one(&inputs[i], srcs[i], slens[i]); + sg_init_table(&outputs[i], 1); + sg_set_page(&outputs[i], pages[i], PAGE_SIZE, 0); + acomp_request_set_params(reqs[i], &inputs[i], + &outputs[i], slens[i], dlens[i]); + + /* + * As long as the API is called with a valid "wait", chain the + * requests for synchronous/asynchronous decompress ops. + * If async_mode is in effect, but the API is called with a + * NULL "wait", submit the requests first, and poll for + * their completion status later, after all descriptors have + * been submitted. + */ + if (!async_poll) { + /* acomp request chaining. */ + if (i) + acomp_request_chain(reqs[i], reqs[0]); + else + acomp_reqchain_init(reqs[0], 0, crypto_req_done, + wait); + } else { + errors[i] = iaa_comp_adecompress(reqs[i]); + + if (errors[i] != -EINPROGRESS) { + errors[i] = -EINVAL; + err = -EINVAL; + } else { + errors[i] = -EAGAIN; + } + } + } + + if (!async_poll) { + if (async) + /* Process the request chain in parallel. */ + err = crypto_wait_req(acomp_do_async_req_chain(reqs[0], + iaa_comp_adecompress, iaa_comp_poll), + wait); + else + /* Process the request chain in series. */ + err = crypto_wait_req(acomp_do_req_chain(reqs[0], + iaa_comp_adecompress), wait); + + for (i = 0; i < nr_pages; ++i) { + errors[i] = acomp_request_err(reqs[i]); + if (errors[i]) { + err = -EINVAL; + pr_debug("Request chaining req %d decompress error %d\n", i, errors[i]); + } else { + dlens[i] = reqs[i]->dlen; + BUG_ON(dlens[i] != PAGE_SIZE); + } + } + + goto reset_reqs; + } + + /* + * Asynchronously poll for and process IAA decompress job completions. + */ + while (!decompressions_done) { + decompressions_done = true; + + for (i = 0; i < nr_pages; ++i) { + /* + * Skip, if the decompression has already completed + * successfully or with an error. + */ + if (errors[i] != -EAGAIN) + continue; + + errors[i] = iaa_comp_poll(reqs[i]); + + if (errors[i]) { + if (errors[i] == -EAGAIN) + decompressions_done = false; + else + err = -EINVAL; + } else { + dlens[i] = reqs[i]->dlen; + BUG_ON(dlens[i] != PAGE_SIZE); + } + } + } + +reset_reqs: + /* + * For the same 'reqs[]' to be usable by + * iaa_comp_acompress()/iaa_comp_deacompress(), + * clear the CRYPTO_ACOMP_REQ_POLL bit on all acomp_reqs, and the + * CRYPTO_TFM_REQ_CHAIN bit on the reqs[0]. + */ + iaa_set_req_poll(reqs, nr_pages, false); + if (!async_poll) + acomp_reqchain_clear(reqs[0], wait); + + return !err; +} + static int iaa_comp_init_fixed(struct crypto_acomp *acomp_tfm) { struct crypto_tfm *tfm = crypto_acomp_tfm(acomp_tfm); @@ -1832,10 +2222,13 @@ static struct acomp_alg iaa_acomp_fixed_deflate = { .compress = iaa_comp_acompress, .decompress = iaa_comp_adecompress, .dst_free = dst_free, + .get_batch_size = iaa_comp_get_batch_size, + .batch_compress = iaa_comp_acompress_batch, + .batch_decompress = iaa_comp_adecompress_batch, .base = { .cra_name = "deflate", .cra_driver_name = "deflate-iaa", - .cra_flags = CRYPTO_ALG_ASYNC, + .cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_REQ_CHAIN, .cra_ctxsize = sizeof(struct iaa_compression_ctx), .cra_module = THIS_MODULE, .cra_priority = IAA_ALG_PRIORITY, From patchwork Thu Feb 6 07:20:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF9FAC02198 for ; Thu, 6 Feb 2025 07:21:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2642280003; Thu, 6 Feb 2025 02:21:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DB857280001; Thu, 6 Feb 2025 02:21:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2376280003; Thu, 6 Feb 2025 02:21:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9C0E8280001 for ; Thu, 6 Feb 2025 02:21:11 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3930880F90 for ; Thu, 6 Feb 2025 07:21:11 +0000 (UTC) X-FDA: 83088673542.15.67A9BD8 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf07.hostedemail.com (Postfix) with ESMTP id 10BDD40002 for ; Thu, 6 Feb 2025 07:21:08 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=EzP5EKWK; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826469; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m/dou/H2UkY9yXWnZnPSMkcDuvFpOgh8cWn8KY5n2nY=; b=AOULJk8QPK/jjGfSGv2WHxBR0ikDjHpI23N80S8GtkYNVJzNNWEcLSf2BSiR1yk+bn8ALj jzPUBS5DKnY+FbG2p7VnOvKu2kAxbkT6gUxqhoWVS6e8lLex4cdB1uKo2stRJ0mcW8EcXh /k5i0lCWmu571mFz5IkhByLG/HxmsLQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826469; a=rsa-sha256; cv=none; b=O/VJT3W1rEe7TvxHUt8pXCPuLCKi5vBTRcOexS9vJ/CGbzETXevJXoLjnz2bHifj6iP27D YXqIh/6SgViOgtDDe5FMcaYQ5UbfCnjrLeJ6kVibEDB9v9Yx3k+9ggQIrKS1sfDUuDUJJz bLPobBEAkkg2oMtf8TgXUO74ACa2zv8= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=EzP5EKWK; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826469; x=1770362469; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7tFXFDivua82eFBjtOF61mRcWq3imTHakYE6ae6fo/I=; b=EzP5EKWKs5dPsHY+KARfWgtyL3cm3wixPdWhMsmwLQ5uE22QzBq4uLKL jkCrWUoFsvXLIJsYdOc4KYusjxo/MxJaxcngtfFO7Bh3MZDU5ZmR2VHKy RlXYQ8nEbQiVdJXooqb6P56tfXUR6T7oDmLQo1rwjaOlNsvMo1WxM+C0w GFyESnQfb7OzQzgHWusnn1C+UbVbgHykPYhiFh3aD6FRXxDGX+CcUN7q1 QGP63ddd0svfpbsMEbrpU2wkOjGLrj6K7r5UJ1x+dQJwFNMoHkv0j9gCO fYAwOeiI/plY5ivX4x/2yPoFIxwSLdaxfLz+Ipy+XH7/OOWarEcJ4qnGJ A==; X-CSE-ConnectionGUID: 92bKSGpFR/KWKGGW61kGoA== X-CSE-MsgGUID: q+/oEi1SRl67H2u9vA8UQw== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962606" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962606" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:04 -0800 X-CSE-ConnectionGUID: 6CGkxBTESVGUYwLAu6XXGg== X-CSE-MsgGUID: fgn9TkDzTSCQXltQ0Mf7rw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022607" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:04 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 05/16] crypto: iaa - Enable async mode and make it the default. Date: Wed, 5 Feb 2025 23:20:51 -0800 Message-Id: <20250206072102.29045-6-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: 5xyehnx8xe7nprtsqf9onxfb7eyzrs95 X-Rspam-User: X-Rspamd-Queue-Id: 10BDD40002 X-Rspamd-Server: rspam03 X-HE-Tag: 1738826468-100528 X-HE-Meta: U2FsdGVkX1+AzOGnPji3jZfwW/tnSjRQ7ZrwEFFA1+Pq6RMQlFwI+sevMRMlKB2W3wVbf/Rg0rFoI1h9bXnMxcGiptvNLSysWzX1Tae3tYjitH4jxKkL+oILJonX+1coF97hQrJGFvhk3w+qv8p3GjlQCpGNFCXuv9KA7rV4xuW5DNUGwWQXvyiViVPKZNNjadcIofpnI9a+VmDRR122zwhInwnWpm5bkRKxBzcG2QNXRNQDd8eGQRQjiIATJYA+tgansx6vb5hhZcEt83Z6BsjFxbjTQ5vxaCBE4bki/TcwnFMOk4nec+PFeSvTqdZvHj5+MditPwicJOLaR2/FlnhlRmJCqnoeehtnLfQiwUZIp10NrUogcRpyS5BaWxZlJ6lH4xX35KWUlAhb83WtOsxnnAw4l4bd6svaIWiTCKRS25RS5Fbr3Zxzzn0KwQJt4W161CMzxUeM6rnTyz8nLD2AJ2JQC8sw3JyBDnGS6BE8igX4CgHEWBIlq40APhK7cSQV6wuFvdf1pkpITgXVhC0O3oa9781cZK9QIEO0otiE2MFkdWBGn+/8IFly0SrcSDfBa+Cc2PODU5yEagdCRavCMy31yLl5ixHxwZSgqW1BT0ww2eh9sLNyd5OeNKw/Swcqlt4zkD0qhZvSvbBuReaaEzBnFJZt8ywCkkbBldKdg8Yy67T49pcFfZ/P3hO/6elyhiRg+tS0LVeYxhfsrbzxKvRYrQelyku5N4RGDmLap4jjiPPVLVtDb3B74nZuh9BCFt3rAT1sj3XNvxs7n7p2ajsk/ljwB3+/B7q8GJdqCYmpFTGFQJR0eWRI6uQf473gS/Ae1XbXBJ3y2SgxD+oeZXI28et8juC+OLsO/UzOeLHHfmTZ54NUec/pw9mGP2MTfp1AVJsPcBix5O1NvL12jigernVEfpSbXyUMdt17Gv8mBlnvZcar18wNGLPXPBQ82aIh2PKR9aqP0iu JAz0LsGg jWX48r7SXT2aEGBkZtnapU8AgCw2T84SE++dqOz8uwV3vCvtdhmKBi0y4p1NE9I7/Gz7x298/PXHxv/L/XwGqYIsKd05tPgh0ZMwMvnMe38FLn84glQaGD8SrK2O1mE4rv9MH4Culd2bI97Cbf8IyU94C7Lo7D0HU33vd5H+G19ABuJ5M2z4fOfitNg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch enables the 'async' sync_mode in the driver. Further, it sets the default sync_mode to 'async', which makes it easier for IAA hardware acceleration in the iaa_crypto driver to be loaded by default in the most efficient/recommended 'async' mode for parallel compressions/decompressions, namely, asynchronous submission of descriptors, followed by polling for job completions with or without request chaining. Earlier, the "sync" mode used to be the default. This way, anyone who wants to use IAA for zswap/zram can do so after building the kernel, and without having to go through these steps to use async mode: 1) disable all the IAA device/wq bindings that happen at boot time 2) rmmod iaa_crypto 3) modprobe iaa_crypto 4) echo async > /sys/bus/dsa/drivers/crypto/sync_mode 5) re-run initialization of the IAA devices and wqs Signed-off-by: Kanchana P Sridhar --- Documentation/driver-api/crypto/iaa/iaa-crypto.rst | 11 ++--------- drivers/crypto/intel/iaa/iaa_crypto_main.c | 4 ++-- 2 files changed, 4 insertions(+), 11 deletions(-) diff --git a/Documentation/driver-api/crypto/iaa/iaa-crypto.rst b/Documentation/driver-api/crypto/iaa/iaa-crypto.rst index 8e50b900d51c..782da5230fcd 100644 --- a/Documentation/driver-api/crypto/iaa/iaa-crypto.rst +++ b/Documentation/driver-api/crypto/iaa/iaa-crypto.rst @@ -272,7 +272,7 @@ The available attributes are: echo async_irq > /sys/bus/dsa/drivers/crypto/sync_mode Async mode without interrupts (caller must poll) can be enabled by - writing 'async' to it (please see Caveat):: + writing 'async' to it:: echo async > /sys/bus/dsa/drivers/crypto/sync_mode @@ -281,14 +281,7 @@ The available attributes are: echo sync > /sys/bus/dsa/drivers/crypto/sync_mode - The default mode is 'sync'. - - Caveat: since the only mechanism that iaa_crypto currently implements - for async polling without interrupts is via the 'sync' mode as - described earlier, writing 'async' to - '/sys/bus/dsa/drivers/crypto/sync_mode' will internally enable the - 'sync' mode. This is to ensure correct iaa_crypto behavior until true - async polling without interrupts is enabled in iaa_crypto. + The default mode is 'async'. .. _iaa_default_config: diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 61134a7ad1da..6d49f82165fe 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -153,7 +153,7 @@ static DRIVER_ATTR_RW(verify_compress); */ /* Use async mode */ -static bool async_mode; +static bool async_mode = true; /* Use interrupts */ static bool use_irq; @@ -173,7 +173,7 @@ static int set_iaa_sync_mode(const char *name) async_mode = false; use_irq = false; } else if (sysfs_streq(name, "async")) { - async_mode = false; + async_mode = true; use_irq = false; } else if (sysfs_streq(name, "async_irq")) { async_mode = true; From patchwork Thu Feb 6 07:20:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962456 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88F80C02198 for ; Thu, 6 Feb 2025 07:21:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE50D280005; Thu, 6 Feb 2025 02:21:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E9535280004; Thu, 6 Feb 2025 02:21:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE701280005; Thu, 6 Feb 2025 02:21:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9E474280001 for ; Thu, 6 Feb 2025 02:21:12 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5395780F8E for ; Thu, 6 Feb 2025 07:21:12 +0000 (UTC) X-FDA: 83088673584.27.D8AAF40 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf26.hostedemail.com (Postfix) with ESMTP id 0B4B6140002 for ; Thu, 6 Feb 2025 07:21:09 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=HzeNa8SM; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826470; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M4SVokmn+Bow46EUfXu/KTDCIdzMHroC23ILD26n6g4=; b=degcpXYwB0wIgf5g9YMXWb5fzWKJx840z8rJzJZcwPRXQWN2ZZ9yWQu/E04CJH1zHdLCH4 WybOUzvB/otMvBgXK1x6wAeQMdL6SDpUZPmbWxVfrkrOa8jIbiYJHGPlxBjQjO0CP094DD m0SLN4BHD9swkbVmqh3DHQL8QZuxUIM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=HzeNa8SM; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826470; a=rsa-sha256; cv=none; b=t36HkN4Ydz+DYSOZveTcRmMvtjkw79DUUDdxGGjMoT7EvFXzmPDuRReTL7MtpqKu0aiNsh B9Z9+iqzwibIpm9M8dbnBLghOQ6qzYyIof0XYi1s7rxgCFGt0bfSJYoHmlYyLGrH7v9DF9 NYmm8XI+R/GzCLJoLMVa+dagNWWAGuY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826470; x=1770362470; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3AC0t67W9YXnQaqtfbv5O8FRwsYURPHnfKQ9TbAZwNM=; b=HzeNa8SM/FomhHnKz0EIwo8w4WBxxJKULzocV69QsSTXhp8tcyu4HUEr +QOPuyoz/cGJADOJIt8d+KUCMHWDhJbITAK0D2R6R/K5pLywDyQfCe2l2 rYtXGfUz9DCL763TSFUAw624Ap0K2H8x+nWOA12WKakNZvdHOwpszSHsT T2rtkZq8B7ANXsAqth+2onE+v7Uttv+njLltE932Fhyb0PTRdhc5DDi1M CxZd7KyCzj9SNK5vNiGFxFuoQPjWO9LwraIj5LTkPEOhnqIVAdNEL2VaF jrdQ3NoQxVWqO2Vfuz+DL+dNFObqsSXyuq9sfjvSmMSO1XP3vJdV+ZeDx A==; X-CSE-ConnectionGUID: rY/NeFE4QPOS9iFbYgPnPw== X-CSE-MsgGUID: M8w2UB4yRkOImDQ0scBMIA== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962617" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962617" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:04 -0800 X-CSE-ConnectionGUID: BdejDC0cSAOKneED2yE52A== X-CSE-MsgGUID: pucbyX2lQJO2lO5OQM3S4g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022610" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:04 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 06/16] crypto: iaa - Disable iaa_verify_compress by default. Date: Wed, 5 Feb 2025 23:20:52 -0800 Message-Id: <20250206072102.29045-7-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0B4B6140002 X-Stat-Signature: r5g8fh9f78qbqeydbbmdmatbatsn4s5b X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738826469-695353 X-HE-Meta: U2FsdGVkX19BxfjWdWFkmf7WEjotzlVys6xuipYpdPj7SFqOixEvoj1d5FFum6do+cxmKOCb1bbUf6jjVidAJf1toXnt3hi9Y63SjcvmzKIAUaMF0X7ugfcrP/iGFUOLJ+tTBdP2GSShjXo/VFZ4KhSVtwZyhI7LODxXPhbgY8mt4ajzz5M268lv7IS1MANEduQevUJympgkI1B3TU3rSbX7ZysX0wDvwy4btPilME380EIJhrgQWboP1fRzyE6PnoWkHC/jKvyqYXqZsk8uhDoAv4IG33UcFiD1hu3SJrSHjY0AuTu3VHX7ysqJHUr63pAplVFFV3Jb8TVSK+eno43gQvZLLBnMNBip+Lr7B+xvggBhK+wvWGU9d6JjY/vHuYmva6l9A/AAFq5iLClVwkw/BxpcbR9MKVoVShfL8p0TbsiAfeMLquyy3ceNPWPwmuxbCNbUVI/scan7395738W0/VuBLI1DM2NAJvWUVMMxLuk2ezkxKHpyytXMMboEfPp9m9jV00GuGU9Y7h//kG8ocopEhYQ9OCg/gLkp9TijxDi/g1Tc8jJC6HcHqkavLHMWDNnQ8/FsDHYIM0rcJ74iN1VepdZvCRmb8YzhmkmOzAI3+K38KBVP1DL5b7S+HMkn5jtOZ/LAoa6TYUA59Vcfr3Zy10Dm7pMN+if9ClvanTokBg6f2STeZU7OOYy2WcGN8wtLAjafNQKfWJpOomE43CEVjxIF0sn4foFMFIMgiPZHkQOegw1ZoXBfgA4LIAAlwSwX8CUvtHvwGKXeRFGZQGah06PvoqxRp2TVT4yzkPBd1DPbxfg4QvPkOkgQ9A+sNoUpelH7OCmBl0KglSikEdCBrpiFOHSr3GtNgntlM+K6ox8DNntkjELW1pTOBPYlYfmjS71ShWPjieSS3homVwzT08B6cQ4JnX5F5Za8LpHM0PiS5SITGlP2//yogmnRDwRLNY0md9H81PH nQaLUnPQ 7zUUVdGBaSA9xLsqoXgjneu9WtNwgS/5dSkGyCS6M+UHv+iFc7sFmjvIR6Al7prGkujBtfAHDXz/aIKeX9KwqeBmNmwqKzmtLqlVMwVX7FcerdXAyiHW3fIt6k5ip+cIkLjkZGSnP6QgUZCyHb11+JA6x/V+myDLyKQGqwujf/H930hgMvEMCigNilegq1OObSSNxvPYyzslYK8D6qA+VMMgb179ZDRvQ6oS5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch makes it easier for IAA hardware acceleration in the iaa_crypto driver to be loaded by default with "iaa_verify_compress" disabled, to facilitate performance comparisons with software compressors (which also do not run compress verification by default). Earlier, iaa_crypto compress verification used to be enabled by default. With this patch, if users want to enable compress verification, they can do so with these steps: 1) disable all the IAA device/wq bindings that happen at boot time 2) rmmod iaa_crypto 3) modprobe iaa_crypto 4) echo 1 > /sys/bus/dsa/drivers/crypto/verify_compress 5) re-run initialization of the IAA devices and wqs Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 6d49f82165fe..f4807a828034 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -94,7 +94,7 @@ static bool iaa_crypto_enabled; static bool iaa_crypto_registered; /* Verify results of IAA compress or not */ -static bool iaa_verify_compress = true; +static bool iaa_verify_compress = false; static ssize_t verify_compress_show(struct device_driver *driver, char *buf) { From patchwork Thu Feb 6 07:20:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 814FFC02198 for ; Thu, 6 Feb 2025 07:21:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F37E280001; Thu, 6 Feb 2025 02:21:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 19EB9280006; Thu, 6 Feb 2025 02:21:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8962280001; Thu, 6 Feb 2025 02:21:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A2949280004 for ; Thu, 6 Feb 2025 02:21:12 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6830F4A4F2 for ; Thu, 6 Feb 2025 07:21:12 +0000 (UTC) X-FDA: 83088673584.21.083AD89 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf08.hostedemail.com (Postfix) with ESMTP id 3F574160002 for ; Thu, 6 Feb 2025 07:21:10 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=iiuBDfar; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826470; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cgWz04yA1KiIPVpx+NJ8BR5a3VjX94jOsrOv6iISinw=; b=5fsraTQAfbeaytlwo82PllpcYxNIXDypW9yqPGqMa0ydc2FFinjE5rwMjxaFLJehO2VN4R gejYx2bKPIDlt9hpoB9kmB24tZTB2YlUOzKGeBCX4A6P0Mp2b1vxlm83q94CP13LKhyE4f A6PcOjnxICEWlA0XY5vrwJrvkRa8RX8= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=iiuBDfar; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826470; a=rsa-sha256; cv=none; b=grueDvQ0OPZnnPIueAE8YSFhf6CvLUjhlZ0a9KVwgyAEcxbBbA4XHecZYudNnokjQnD2Ee 4VuryoF5U57VGSqDp5j62DzZW1u0AENcTxxSfZk5ss6TpSv49fYvYCa9ewWVk/5T1S0UuR aYB5ycEXtMJ++644be8lcVbisnptofE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826470; x=1770362470; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yBCr+FraBHYq64d9eiZRbYqzgC3sqMOsBxQBn5IGL/8=; b=iiuBDfarFcdOu5NhBWqah+M+P1NTfTgc77tRh0dESn0jL/V6Pou38e/O nkH0BMQxEKo+dEjquk7Edlvbf8cSRsGSqXTlepp5Yg0IgxOx5Ab0TjubG 78+4rS7gIiikm3teuIGVW7Za6t0a2mWiEJfz4C53WmmlcotBCo2dW7Zx7 t8fcs6rxIUY7e+d0GEPBZy9R3+F3t8Mpwr0bey3RznMcIsV950XmxH1ku 1YiA8gSBlsZCj9rskimCeOgYtcF6M7w0+V7xeuXjLdrKv1hCfmL93LVFH /9JLHjsoxv+e07QDmxDn10jZbFjf/g/jxxXJi0xa5c1F/q1GiaK6LkOzz w==; X-CSE-ConnectionGUID: Sg6P7h1ZT4qS58ZHzMoSVQ== X-CSE-MsgGUID: YDnGSYCYQo6hSTxNrUMnvA== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962639" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962639" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:04 -0800 X-CSE-ConnectionGUID: LtK4e2iJTwuCI8wnN5o80Q== X-CSE-MsgGUID: bhKYVzw+RqiEF8300phynA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022615" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:04 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 07/16] crypto: iaa - Re-organize the iaa_crypto driver code. Date: Wed, 5 Feb 2025 23:20:53 -0800 Message-Id: <20250206072102.29045-8-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3F574160002 X-Stat-Signature: 54hsfsqx1koxnubfh3xpm8e6u5nwi6qt X-Rspam-User: X-HE-Tag: 1738826470-105524 X-HE-Meta: U2FsdGVkX19mBHI15eKJhGqfY+rq1r0nH57RRKEO0a1JyLJDaIyVyExpYYDQNcmlXmtSJ3MVgk76NF2qilelpv939kJbqIJi7vXDlHVRd7VC33BYDuzbfnDYkMvHlSG5nfs3NETBHlu3KhFgiXY9/UeplA//bLKisPSEtCF883d0NELd8+Gzdw+BF5toMDUDRu3gIBuuh4XFUnAJKkSRXs95fTn7Gy11hM4JHdFcNwtiodwYf1l8uZbwn8DcFpSt0bkuFtNE9Y7EVkHTKolS0Ybi6GN82cBnqEl9YrVeJGBWjaTXGDcgloz2Ao/GraHXCM6bN7wWKM/f+SCaQI7ed8sppjg1eWyWQ/rw8ySk7HQQsCU4W+/Ou8RWSCb+fNeQKFt6bpQgb99xiisxFELBqL4psfOrc36KQq9X2g5X5X+ktfKIIqQrMabwLH22n3lxVvh1oEqw0uIGPkkRyrG0wcnaqrkUSDbNZuKLgtw0PdYMNXGwLwvQgfN/QYHETPLov8HdK1ECB3dzX4NuMuTgWMHGXsE/g0kJBzb60m9oWhkC/ybQE2ImbS1sngP1QB9UMaxnc3jvMdxOpNC4DioM3O5Vd1oJ7BhGlcSNG4snLZ2slzqxLkWFzS4GCVc69pBbyeWzl1Hyb0Tp5inU9eIYTl3o3+0zdiRD2vZAfwk02UgoqtyUKsY2YAAphWPMqnYo0UqbEUFup5I3EKYFU7CBOkHTMYIgU7kAVZnxL7rj5kgcV9yskqg3SQ3joiiEgn3ExzkwhF0v1PNdBdrXUL0NOTmYTPK++7GoFHtStSa97R+iiFDtod51+zW//ksBmVGkY02RrjlqMINE1CXdT4J4o0esadzEuk5ul0QxJv2zTFCx2y7yP/etjDHdjES8YMmoBD9vFjVyfLs0b2oZNY9wfxs3VkaZvhnxwJ1JHBNZUDOJQ/gIDPlEH/75Y+ZiDc9nQmMunzQzO1vgNqlpPD2 sFPKI/8K RRGLz8GXOhorlUUmtpFgRWOdbT9VeZ7YLICVRGZPgFUn4rjE77eH2pGoloscysjKewNbDgMH+VRBTJN7MOrzkD/CK0XgPMPxMASsXfSvNRRTjKAntpn8WqyCZ9ttHCglnwMn/aCz1TNPn7QvnVUHKE4eFeLFmRXVFvymG4W67l70HfLVsIPnArVH3A62dnPLLYO7Rvty9HF7/uCRg9croSqLbV/zNBfwLdMI916eQSq2KMBU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch merely reorganizes the code in iaa_crypto_main.c, so that the functions are consolidated into logically related sub-sections of code. This is expected to make the code more maintainable and for it to be easier to replace functional layers and/or add new features. Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 540 +++++++++++---------- 1 file changed, 275 insertions(+), 265 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index f4807a828034..2c5b7ce041d6 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -24,6 +24,9 @@ #define IAA_ALG_PRIORITY 300 +/************************************** + * Driver internal global variables. + **************************************/ /* number of iaa instances probed */ static unsigned int nr_iaa; static unsigned int nr_cpus; @@ -36,55 +39,46 @@ static unsigned int cpus_per_iaa; static struct crypto_comp *deflate_generic_tfm; /* Per-cpu lookup table for balanced wqs */ -static struct wq_table_entry __percpu *wq_table; +static struct wq_table_entry __percpu *wq_table = NULL; -static struct idxd_wq *wq_table_next_wq(int cpu) -{ - struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); - - if (++entry->cur_wq >= entry->n_wqs) - entry->cur_wq = 0; - - if (!entry->wqs[entry->cur_wq]) - return NULL; - - pr_debug("%s: returning wq at idx %d (iaa wq %d.%d) from cpu %d\n", __func__, - entry->cur_wq, entry->wqs[entry->cur_wq]->idxd->id, - entry->wqs[entry->cur_wq]->id, cpu); - - return entry->wqs[entry->cur_wq]; -} - -static void wq_table_add(int cpu, struct idxd_wq *wq) -{ - struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); - - if (WARN_ON(entry->n_wqs == entry->max_wqs)) - return; - - entry->wqs[entry->n_wqs++] = wq; - - pr_debug("%s: added iaa wq %d.%d to idx %d of cpu %d\n", __func__, - entry->wqs[entry->n_wqs - 1]->idxd->id, - entry->wqs[entry->n_wqs - 1]->id, entry->n_wqs - 1, cpu); -} - -static void wq_table_free_entry(int cpu) -{ - struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); +/* Verify results of IAA compress or not */ +static bool iaa_verify_compress = false; - kfree(entry->wqs); - memset(entry, 0, sizeof(*entry)); -} +/* + * The iaa crypto driver supports three 'sync' methods determining how + * compressions and decompressions are performed: + * + * - sync: the compression or decompression completes before + * returning. This is the mode used by the async crypto + * interface when the sync mode is set to 'sync' and by + * the sync crypto interface regardless of setting. + * + * - async: the compression or decompression is submitted and returns + * immediately. Completion interrupts are not used so + * the caller is responsible for polling the descriptor + * for completion. This mode is applicable to only the + * async crypto interface and is ignored for anything + * else. + * + * - async_irq: the compression or decompression is submitted and + * returns immediately. Completion interrupts are + * enabled so the caller can wait for the completion and + * yield to other threads. When the compression or + * decompression completes, the completion is signaled + * and the caller awakened. This mode is applicable to + * only the async crypto interface and is ignored for + * anything else. + * + * These modes can be set using the iaa_crypto sync_mode driver + * attribute. + */ -static void wq_table_clear_entry(int cpu) -{ - struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); +/* Use async mode */ +static bool async_mode = true; +/* Use interrupts */ +static bool use_irq; - entry->n_wqs = 0; - entry->cur_wq = 0; - memset(entry->wqs, 0, entry->max_wqs * sizeof(struct idxd_wq *)); -} +static struct iaa_compression_mode *iaa_compression_modes[IAA_COMP_MODES_MAX]; LIST_HEAD(iaa_devices); DEFINE_MUTEX(iaa_devices_lock); @@ -93,9 +87,9 @@ DEFINE_MUTEX(iaa_devices_lock); static bool iaa_crypto_enabled; static bool iaa_crypto_registered; -/* Verify results of IAA compress or not */ -static bool iaa_verify_compress = false; - +/************************************************** + * Driver attributes along with get/set functions. + **************************************************/ static ssize_t verify_compress_show(struct device_driver *driver, char *buf) { return sprintf(buf, "%d\n", iaa_verify_compress); @@ -123,40 +117,6 @@ static ssize_t verify_compress_store(struct device_driver *driver, } static DRIVER_ATTR_RW(verify_compress); -/* - * The iaa crypto driver supports three 'sync' methods determining how - * compressions and decompressions are performed: - * - * - sync: the compression or decompression completes before - * returning. This is the mode used by the async crypto - * interface when the sync mode is set to 'sync' and by - * the sync crypto interface regardless of setting. - * - * - async: the compression or decompression is submitted and returns - * immediately. Completion interrupts are not used so - * the caller is responsible for polling the descriptor - * for completion. This mode is applicable to only the - * async crypto interface and is ignored for anything - * else. - * - * - async_irq: the compression or decompression is submitted and - * returns immediately. Completion interrupts are - * enabled so the caller can wait for the completion and - * yield to other threads. When the compression or - * decompression completes, the completion is signaled - * and the caller awakened. This mode is applicable to - * only the async crypto interface and is ignored for - * anything else. - * - * These modes can be set using the iaa_crypto sync_mode driver - * attribute. - */ - -/* Use async mode */ -static bool async_mode = true; -/* Use interrupts */ -static bool use_irq; - /** * set_iaa_sync_mode - Set IAA sync mode * @name: The name of the sync mode @@ -219,8 +179,9 @@ static ssize_t sync_mode_store(struct device_driver *driver, } static DRIVER_ATTR_RW(sync_mode); -static struct iaa_compression_mode *iaa_compression_modes[IAA_COMP_MODES_MAX]; - +/**************************** + * Driver compression modes. + ****************************/ static int find_empty_iaa_compression_mode(void) { int i = -EINVAL; @@ -411,11 +372,6 @@ static void free_device_compression_mode(struct iaa_device *iaa_device, IDXD_OP_FLAG_WR_SRC2_AECS_COMP | \ IDXD_OP_FLAG_AECS_RW_TGLS) -static int check_completion(struct device *dev, - struct iax_completion_record *comp, - bool compress, - bool only_once); - static int init_device_compression_mode(struct iaa_device *iaa_device, struct iaa_compression_mode *mode, int idx, struct idxd_wq *wq) @@ -502,6 +458,10 @@ static void remove_device_compression_modes(struct iaa_device *iaa_device) } } +/*********************************************************** + * Functions for use in crypto probe and remove interfaces: + * allocate/init/query/deallocate devices/wqs. + ***********************************************************/ static struct iaa_device *iaa_device_alloc(void) { struct iaa_device *iaa_device; @@ -614,16 +574,6 @@ static void del_iaa_wq(struct iaa_device *iaa_device, struct idxd_wq *wq) } } -static void clear_wq_table(void) -{ - int cpu; - - for (cpu = 0; cpu < nr_cpus; cpu++) - wq_table_clear_entry(cpu); - - pr_debug("cleared wq table\n"); -} - static void free_iaa_device(struct iaa_device *iaa_device) { if (!iaa_device) @@ -704,43 +654,6 @@ static int iaa_wq_put(struct idxd_wq *wq) return ret; } -static void free_wq_table(void) -{ - int cpu; - - for (cpu = 0; cpu < nr_cpus; cpu++) - wq_table_free_entry(cpu); - - free_percpu(wq_table); - - pr_debug("freed wq table\n"); -} - -static int alloc_wq_table(int max_wqs) -{ - struct wq_table_entry *entry; - int cpu; - - wq_table = alloc_percpu(struct wq_table_entry); - if (!wq_table) - return -ENOMEM; - - for (cpu = 0; cpu < nr_cpus; cpu++) { - entry = per_cpu_ptr(wq_table, cpu); - entry->wqs = kcalloc(max_wqs, sizeof(struct wq *), GFP_KERNEL); - if (!entry->wqs) { - free_wq_table(); - return -ENOMEM; - } - - entry->max_wqs = max_wqs; - } - - pr_debug("initialized wq table\n"); - - return 0; -} - static int save_iaa_wq(struct idxd_wq *wq) { struct iaa_device *iaa_device, *found = NULL; @@ -829,6 +742,87 @@ static void remove_iaa_wq(struct idxd_wq *wq) cpus_per_iaa = 1; } +/*************************************************************** + * Mapping IAA devices and wqs to cores with per-cpu wq_tables. + ***************************************************************/ +static void wq_table_free_entry(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); + + kfree(entry->wqs); + memset(entry, 0, sizeof(*entry)); +} + +static void wq_table_clear_entry(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); + + entry->n_wqs = 0; + entry->cur_wq = 0; + memset(entry->wqs, 0, entry->max_wqs * sizeof(struct idxd_wq *)); +} + +static void clear_wq_table(void) +{ + int cpu; + + for (cpu = 0; cpu < nr_cpus; cpu++) + wq_table_clear_entry(cpu); + + pr_debug("cleared wq table\n"); +} + +static void free_wq_table(void) +{ + int cpu; + + for (cpu = 0; cpu < nr_cpus; cpu++) + wq_table_free_entry(cpu); + + free_percpu(wq_table); + + pr_debug("freed wq table\n"); +} + +static int alloc_wq_table(int max_wqs) +{ + struct wq_table_entry *entry; + int cpu; + + wq_table = alloc_percpu(struct wq_table_entry); + if (!wq_table) + return -ENOMEM; + + for (cpu = 0; cpu < nr_cpus; cpu++) { + entry = per_cpu_ptr(wq_table, cpu); + entry->wqs = kcalloc(max_wqs, sizeof(struct wq *), GFP_KERNEL); + if (!entry->wqs) { + free_wq_table(); + return -ENOMEM; + } + + entry->max_wqs = max_wqs; + } + + pr_debug("initialized wq table\n"); + + return 0; +} + +static void wq_table_add(int cpu, struct idxd_wq *wq) +{ + struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); + + if (WARN_ON(entry->n_wqs == entry->max_wqs)) + return; + + entry->wqs[entry->n_wqs++] = wq; + + pr_debug("%s: added iaa wq %d.%d to idx %d of cpu %d\n", __func__, + entry->wqs[entry->n_wqs - 1]->idxd->id, + entry->wqs[entry->n_wqs - 1]->id, entry->n_wqs - 1, cpu); +} + static int wq_table_add_wqs(int iaa, int cpu) { struct iaa_device *iaa_device, *found_device = NULL; @@ -939,6 +933,29 @@ static void rebalance_wq_table(void) } } +/*************************************************************** + * Assign work-queues for driver ops using per-cpu wq_tables. + ***************************************************************/ +static struct idxd_wq *wq_table_next_wq(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); + + if (++entry->cur_wq >= entry->n_wqs) + entry->cur_wq = 0; + + if (!entry->wqs[entry->cur_wq]) + return NULL; + + pr_debug("%s: returning wq at idx %d (iaa wq %d.%d) from cpu %d\n", __func__, + entry->cur_wq, entry->wqs[entry->cur_wq]->idxd->id, + entry->wqs[entry->cur_wq]->id, cpu); + + return entry->wqs[entry->cur_wq]; +} + +/************************************************* + * Core iaa_crypto compress/decompress functions. + *************************************************/ static inline int check_completion(struct device *dev, struct iax_completion_record *comp, bool compress, @@ -1020,13 +1037,130 @@ static int deflate_generic_decompress(struct acomp_req *req) static int iaa_remap_for_verify(struct device *dev, struct iaa_wq *iaa_wq, struct acomp_req *req, - dma_addr_t *src_addr, dma_addr_t *dst_addr); + dma_addr_t *src_addr, dma_addr_t *dst_addr) +{ + int ret = 0; + int nr_sgs; + + dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); + dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); + + nr_sgs = dma_map_sg(dev, req->src, sg_nents(req->src), DMA_FROM_DEVICE); + if (nr_sgs <= 0 || nr_sgs > 1) { + dev_dbg(dev, "verify: couldn't map src sg for iaa device %d," + " wq %d: ret=%d\n", iaa_wq->iaa_device->idxd->id, + iaa_wq->wq->id, ret); + ret = -EIO; + goto out; + } + *src_addr = sg_dma_address(req->src); + dev_dbg(dev, "verify: dma_map_sg, src_addr %llx, nr_sgs %d, req->src %p," + " req->slen %d, sg_dma_len(sg) %d\n", *src_addr, nr_sgs, + req->src, req->slen, sg_dma_len(req->src)); + + nr_sgs = dma_map_sg(dev, req->dst, sg_nents(req->dst), DMA_TO_DEVICE); + if (nr_sgs <= 0 || nr_sgs > 1) { + dev_dbg(dev, "verify: couldn't map dst sg for iaa device %d," + " wq %d: ret=%d\n", iaa_wq->iaa_device->idxd->id, + iaa_wq->wq->id, ret); + ret = -EIO; + dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_FROM_DEVICE); + goto out; + } + *dst_addr = sg_dma_address(req->dst); + dev_dbg(dev, "verify: dma_map_sg, dst_addr %llx, nr_sgs %d, req->dst %p," + " req->dlen %d, sg_dma_len(sg) %d\n", *dst_addr, nr_sgs, + req->dst, req->dlen, sg_dma_len(req->dst)); +out: + return ret; +} static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, struct idxd_wq *wq, dma_addr_t src_addr, unsigned int slen, dma_addr_t dst_addr, unsigned int *dlen, - u32 compression_crc); + u32 compression_crc) +{ + struct iaa_device_compression_mode *active_compression_mode; + struct iaa_compression_ctx *ctx = crypto_tfm_ctx(tfm); + struct iaa_device *iaa_device; + struct idxd_desc *idxd_desc; + struct iax_hw_desc *desc; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + int ret = 0; + + iaa_wq = idxd_wq_get_private(wq); + iaa_device = iaa_wq->iaa_device; + idxd = iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + active_compression_mode = get_iaa_device_compression_mode(iaa_device, ctx->mode); + + idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); + if (IS_ERR(idxd_desc)) { + dev_dbg(dev, "idxd descriptor allocation failed\n"); + dev_dbg(dev, "iaa compress failed: ret=%ld\n", + PTR_ERR(idxd_desc)); + return PTR_ERR(idxd_desc); + } + desc = idxd_desc->iax_hw; + + /* Verify (optional) - decompress and check crc, suppress dest write */ + + desc->flags = IDXD_OP_FLAG_CRAV | IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CC; + desc->opcode = IAX_OPCODE_DECOMPRESS; + desc->decompr_flags = IAA_DECOMP_FLAGS | IAA_DECOMP_SUPPRESS_OUTPUT; + desc->priv = 0; + + desc->src1_addr = (u64)dst_addr; + desc->src1_size = *dlen; + desc->dst_addr = (u64)src_addr; + desc->max_dst_size = slen; + desc->completion_addr = idxd_desc->compl_dma; + + dev_dbg(dev, "(verify) compression mode %s," + " desc->src1_addr %llx, desc->src1_size %d," + " desc->dst_addr %llx, desc->max_dst_size %d," + " desc->src2_addr %llx, desc->src2_size %d\n", + active_compression_mode->name, + desc->src1_addr, desc->src1_size, desc->dst_addr, + desc->max_dst_size, desc->src2_addr, desc->src2_size); + + ret = idxd_submit_desc(wq, idxd_desc); + if (ret) { + dev_dbg(dev, "submit_desc (verify) failed ret=%d\n", ret); + goto err; + } + + ret = check_completion(dev, idxd_desc->iax_completion, false, false); + if (ret) { + dev_dbg(dev, "(verify) check_completion failed ret=%d\n", ret); + goto err; + } + + if (compression_crc != idxd_desc->iax_completion->crc) { + ret = -EINVAL; + dev_dbg(dev, "(verify) iaa comp/decomp crc mismatch:" + " comp=0x%x, decomp=0x%x\n", compression_crc, + idxd_desc->iax_completion->crc); + print_hex_dump(KERN_INFO, "cmp-rec: ", DUMP_PREFIX_OFFSET, + 8, 1, idxd_desc->iax_completion, 64, 0); + goto err; + } + + idxd_free_desc(wq, idxd_desc); +out: + return ret; +err: + idxd_free_desc(wq, idxd_desc); + dev_dbg(dev, "iaa compress failed: ret=%d\n", ret); + + goto out; +} static void iaa_desc_complete(struct idxd_desc *idxd_desc, enum idxd_complete_type comp_type, @@ -1245,133 +1379,6 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, goto out; } -static int iaa_remap_for_verify(struct device *dev, struct iaa_wq *iaa_wq, - struct acomp_req *req, - dma_addr_t *src_addr, dma_addr_t *dst_addr) -{ - int ret = 0; - int nr_sgs; - - dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); - dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); - - nr_sgs = dma_map_sg(dev, req->src, sg_nents(req->src), DMA_FROM_DEVICE); - if (nr_sgs <= 0 || nr_sgs > 1) { - dev_dbg(dev, "verify: couldn't map src sg for iaa device %d," - " wq %d: ret=%d\n", iaa_wq->iaa_device->idxd->id, - iaa_wq->wq->id, ret); - ret = -EIO; - goto out; - } - *src_addr = sg_dma_address(req->src); - dev_dbg(dev, "verify: dma_map_sg, src_addr %llx, nr_sgs %d, req->src %p," - " req->slen %d, sg_dma_len(sg) %d\n", *src_addr, nr_sgs, - req->src, req->slen, sg_dma_len(req->src)); - - nr_sgs = dma_map_sg(dev, req->dst, sg_nents(req->dst), DMA_TO_DEVICE); - if (nr_sgs <= 0 || nr_sgs > 1) { - dev_dbg(dev, "verify: couldn't map dst sg for iaa device %d," - " wq %d: ret=%d\n", iaa_wq->iaa_device->idxd->id, - iaa_wq->wq->id, ret); - ret = -EIO; - dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_FROM_DEVICE); - goto out; - } - *dst_addr = sg_dma_address(req->dst); - dev_dbg(dev, "verify: dma_map_sg, dst_addr %llx, nr_sgs %d, req->dst %p," - " req->dlen %d, sg_dma_len(sg) %d\n", *dst_addr, nr_sgs, - req->dst, req->dlen, sg_dma_len(req->dst)); -out: - return ret; -} - -static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, - struct idxd_wq *wq, - dma_addr_t src_addr, unsigned int slen, - dma_addr_t dst_addr, unsigned int *dlen, - u32 compression_crc) -{ - struct iaa_device_compression_mode *active_compression_mode; - struct iaa_compression_ctx *ctx = crypto_tfm_ctx(tfm); - struct iaa_device *iaa_device; - struct idxd_desc *idxd_desc; - struct iax_hw_desc *desc; - struct idxd_device *idxd; - struct iaa_wq *iaa_wq; - struct pci_dev *pdev; - struct device *dev; - int ret = 0; - - iaa_wq = idxd_wq_get_private(wq); - iaa_device = iaa_wq->iaa_device; - idxd = iaa_device->idxd; - pdev = idxd->pdev; - dev = &pdev->dev; - - active_compression_mode = get_iaa_device_compression_mode(iaa_device, ctx->mode); - - idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); - if (IS_ERR(idxd_desc)) { - dev_dbg(dev, "idxd descriptor allocation failed\n"); - dev_dbg(dev, "iaa compress failed: ret=%ld\n", - PTR_ERR(idxd_desc)); - return PTR_ERR(idxd_desc); - } - desc = idxd_desc->iax_hw; - - /* Verify (optional) - decompress and check crc, suppress dest write */ - - desc->flags = IDXD_OP_FLAG_CRAV | IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CC; - desc->opcode = IAX_OPCODE_DECOMPRESS; - desc->decompr_flags = IAA_DECOMP_FLAGS | IAA_DECOMP_SUPPRESS_OUTPUT; - desc->priv = 0; - - desc->src1_addr = (u64)dst_addr; - desc->src1_size = *dlen; - desc->dst_addr = (u64)src_addr; - desc->max_dst_size = slen; - desc->completion_addr = idxd_desc->compl_dma; - - dev_dbg(dev, "(verify) compression mode %s," - " desc->src1_addr %llx, desc->src1_size %d," - " desc->dst_addr %llx, desc->max_dst_size %d," - " desc->src2_addr %llx, desc->src2_size %d\n", - active_compression_mode->name, - desc->src1_addr, desc->src1_size, desc->dst_addr, - desc->max_dst_size, desc->src2_addr, desc->src2_size); - - ret = idxd_submit_desc(wq, idxd_desc); - if (ret) { - dev_dbg(dev, "submit_desc (verify) failed ret=%d\n", ret); - goto err; - } - - ret = check_completion(dev, idxd_desc->iax_completion, false, false); - if (ret) { - dev_dbg(dev, "(verify) check_completion failed ret=%d\n", ret); - goto err; - } - - if (compression_crc != idxd_desc->iax_completion->crc) { - ret = -EINVAL; - dev_dbg(dev, "(verify) iaa comp/decomp crc mismatch:" - " comp=0x%x, decomp=0x%x\n", compression_crc, - idxd_desc->iax_completion->crc); - print_hex_dump(KERN_INFO, "cmp-rec: ", DUMP_PREFIX_OFFSET, - 8, 1, idxd_desc->iax_completion, 64, 0); - goto err; - } - - idxd_free_desc(wq, idxd_desc); -out: - return ret; -err: - idxd_free_desc(wq, idxd_desc); - dev_dbg(dev, "iaa compress failed: ret=%d\n", ret); - - goto out; -} - static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, struct idxd_wq *wq, dma_addr_t src_addr, unsigned int slen, @@ -2197,6 +2204,9 @@ static bool iaa_comp_adecompress_batch( return !err; } +/********************************************* + * Interfaces to crypto_alg and crypto_acomp. + *********************************************/ static int iaa_comp_init_fixed(struct crypto_acomp *acomp_tfm) { struct crypto_tfm *tfm = crypto_acomp_tfm(acomp_tfm); From patchwork Thu Feb 6 07:20:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962458 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 141E9C02196 for ; Thu, 6 Feb 2025 07:21:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26464280006; Thu, 6 Feb 2025 02:21:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F83A280004; Thu, 6 Feb 2025 02:21:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E94A5280006; Thu, 6 Feb 2025 02:21:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C4F43280004 for ; Thu, 6 Feb 2025 02:21:13 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 82C4D48DB0 for ; Thu, 6 Feb 2025 07:21:13 +0000 (UTC) X-FDA: 83088673626.17.AB64DDB Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf07.hostedemail.com (Postfix) with ESMTP id 4DFD640002 for ; Thu, 6 Feb 2025 07:21:11 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jhx+iBbx; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826471; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RWDQtD8JBolOXGg3gP10iWxectYlSPs4LO8ucBz7Q68=; b=IMdLnsLctaH8eEmxExeFnhzSVXWUsrf+LaRx7Tnrw6GhOaXnTyZAzfGusYibk9cwI2/s3n 4kGHqp31W23wfo37tJ4/+TF5Ri1c3AJaU+w6w488hIIER2J2DH0xuMUKPlYGVsXC1J1Kju RchbBh1Z8R4gfRPSmhgG0NbVRTgIHAU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826471; a=rsa-sha256; cv=none; b=sHEAnQq/iP6Qg+Pzag2id8mkwntxU/wUg6yD5lGbsqe/e1jtR8/zedrgtMsxBgICP7iOeP y7iqG8AJjwlz6QRHQgRKGkRhx2H89JqfqRWe5fS4F+H8NCMVqN/vBCckmaU39vSzw2LNLr jL6+GfoNI9mRFMERc+2nwZPzch/Cf9s= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jhx+iBbx; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826471; x=1770362471; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XgmFrlxIUAmI4N9Gxiy77mwDBx+ourmvlIMPZ8/24cI=; b=jhx+iBbxfB4rmYwcbxouekGuR5JkiNXJ1rSbspK7XuUbrEkKTBGkM0aC BEeHzZpAhdPFSwd0NLKudC35WF2z8FdX0bo6MRv1BVW/j+x8TRzWhRMsc x19hp72v2DD0BqLlPdNsoyM99bawaTse3fhzQByxmFXnFZZzyL13jbLo0 h3ZUVqMvLkW0LM7pmR8La6/+Th6bbC8XElrZqVA69sVqyp84mHSOYhy7M w5kwzRrfZo1zkuVy8XxfH70202M6FHP+6WiPl3u79nvMC3GMjTCf1YiuP vNysPI6IZyvqOgbIptPImtm4Rsrjc0pxcsaQphWx0h0k4CTcDPcJThiwt g==; X-CSE-ConnectionGUID: NlZxVF26RjeM1fVCQdoa9A== X-CSE-MsgGUID: pve5nPfOSt+k7K+91I7yEg== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962653" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962653" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:04 -0800 X-CSE-ConnectionGUID: E+mypAIGSsyfqbUYbDaiTA== X-CSE-MsgGUID: xF8J7SyoQEKA/Tfb3Eb2Ng== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022619" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:04 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 08/16] crypto: iaa - Map IAA devices/wqs to cores based on packages instead of NUMA. Date: Wed, 5 Feb 2025 23:20:54 -0800 Message-Id: <20250206072102.29045-9-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: ym6wubbtuxj6ryayafd6xsw7c9y56spc X-Rspam-User: X-Rspamd-Queue-Id: 4DFD640002 X-Rspamd-Server: rspam03 X-HE-Tag: 1738826471-255930 X-HE-Meta: U2FsdGVkX1+Jv5hAV4sSu5rK2Z2ns9IDlbt50xQB+L4r4WtoRUejvRbtnZKwxKduES4bgRWwbiabJK8NoRtyTqetQEg9+31P/u85EOprccR8DG9+Z759/CQAHU2eGvCaM0HOO/uIZPZmq+Av0ckqAv9VFuRjvbbieXGqp8ComjVV/sAzdiD5atGZBJwUo59s/+DEgL69DiNH7gI/3V4vMp+LyczalSMVSuoqCIiAOaOamlNcLjv3l7jNzzXOs3BTEPw0lQYNlr2Qg9dV0C5KSXLCTQIbkAtNuCnPaTtf3rN0MPCSLs9UIl+16YLso6UBac97+VWYiOPn9600zqJp0XB+3Jcmv8+j0/XPKaDe08M7R5R3qnpLWAUwhT5jn3N5LYYs+3Ec7hijGa9nMl+5MLSL7sfaEcRLOcFI6bYIq5IqGF1d0SQi4qeQBOXJx/o/Y3wNYyZNn3dr9ERClAPDo3nS3yl4ANZoA/toCUsdI8QF3TaSPOdQtqyCzGXhRMZGw9LgXLjVEONwIWe6O6uoHp07wzTI3sSRnfYnlvx+3JweDNOIo+B6L3XhO82VNcMBwhhrprQYd5cctrOw1CeYzljMDOr4yvwbEClCRlrgHuT1zE6RjMWIC3xa63iJmFQnJUCFYz//b2g6aImUWDul2adlEP3xo5kJ8vq8nk5zCfNtTs7BOAljwJWbEwV0pp2QOnyaP4zVFiWOQJNHyIdaQGLsRhP468QYXGZY2mR+x4smSym2P4o1n5qxKmqKL9QIY6FsKzqP7aoneMcV7Mk+lYcKMgTai6ylaRqLVvO3sXxNXEWIY/bsIfcYrKrAve5JeCKeAH3G7OtWAu+Z3xRmSEJbTucnIXeR6UE2sleYI5eRT2ZzbctJYB/Qm/FLhnOGk9J8eqyxM+OqvgcCsb9vv4fVy7iOIyztMnrsBoFZ6DouB3Kj3wpAfMyG9azvW/29KkA1MUNgiXm/3Dc7932 THLbfD6v hrkQFnF0FRnkMzkNpASP2l4Y4+QltdaWkXMrQ0J/oNQaGbIAflbraWMVuRNLVRJFgVtEXmjX0ZTMmukFIKoD0LLpcmezYmgfLioUnK/HPEBwc2KXExVwFk5X+Rcd03/Bqz/CaKFtfECsQgoZsTbIwB1JwVdp9OFiDH7OIsIYfko+usfzSAzCyzaH5MQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch modifies the algorithm for mapping available IAA devices and wqs to cores, as they are being discovered, based on packages instead of NUMA nodes. This leads to a more realistic mapping of IAA devices as compression/decompression resources for a package, rather than for a NUMA node. This also resolves problems that were observed during internal validation on Intel platforms with many more NUMA nodes than packages: for such cases, the earlier NUMA based allocation caused some IAAs to be over-subscribed and some to not be utilized at all. As a result of this change from NUMA to packages, some of the core functions used by the iaa_crypto driver's "probe" and "remove" API have been re-written. The new infrastructure maintains a static/global mapping of "local wqs" per IAA device, in the "struct iaa_device" itself. The earlier implementation would allocate memory per-cpu for this data, which never changes once the IAA devices/wqs have been initialized. Two main outcomes from this new iaa_crypto driver infrastructure are: 1) Resolves "task blocked for more than x seconds" errors observed during internal validation on Intel systems with the earlier NUMA node based mappings, which was root-caused to the non-optimal IAA-to-core mappings described earlier. 2) Results in a NUM_THREADS factor reduction in memory footprint cost of initializing IAA devices/wqs, due to eliminating the per-cpu copies of each IAA device's wqs. On a 384 cores Intel Granite Rapids server with 8 IAA devices, this saves 140MiB. Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto.h | 17 +- drivers/crypto/intel/iaa/iaa_crypto_main.c | 276 ++++++++++++--------- 2 files changed, 171 insertions(+), 122 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto.h b/drivers/crypto/intel/iaa/iaa_crypto.h index b3b67c44ec8a..74d25e62df12 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto.h +++ b/drivers/crypto/intel/iaa/iaa_crypto.h @@ -55,6 +55,7 @@ struct iaa_wq { struct idxd_wq *wq; int ref; bool remove; + bool mapped; struct iaa_device *iaa_device; @@ -72,6 +73,13 @@ struct iaa_device_compression_mode { dma_addr_t aecs_comp_table_dma_addr; }; +struct wq_table_entry { + struct idxd_wq **wqs; + int max_wqs; + int n_wqs; + int cur_wq; +}; + /* Representation of IAA device with wqs, populated by probe */ struct iaa_device { struct list_head list; @@ -82,19 +90,14 @@ struct iaa_device { int n_wq; struct list_head wqs; + struct wq_table_entry *iaa_local_wqs; + atomic64_t comp_calls; atomic64_t comp_bytes; atomic64_t decomp_calls; atomic64_t decomp_bytes; }; -struct wq_table_entry { - struct idxd_wq **wqs; - int max_wqs; - int n_wqs; - int cur_wq; -}; - #define IAA_AECS_ALIGN 32 /* diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 2c5b7ce041d6..418f78454875 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -30,8 +30,9 @@ /* number of iaa instances probed */ static unsigned int nr_iaa; static unsigned int nr_cpus; -static unsigned int nr_nodes; -static unsigned int nr_cpus_per_node; +static unsigned int nr_packages; +static unsigned int nr_cpus_per_package; +static unsigned int nr_iaa_per_package; /* Number of physical cpus sharing each iaa instance */ static unsigned int cpus_per_iaa; @@ -462,17 +463,46 @@ static void remove_device_compression_modes(struct iaa_device *iaa_device) * Functions for use in crypto probe and remove interfaces: * allocate/init/query/deallocate devices/wqs. ***********************************************************/ -static struct iaa_device *iaa_device_alloc(void) +static struct iaa_device *iaa_device_alloc(struct idxd_device *idxd) { + struct wq_table_entry *local; struct iaa_device *iaa_device; iaa_device = kzalloc(sizeof(*iaa_device), GFP_KERNEL); if (!iaa_device) - return NULL; + goto err; + + iaa_device->idxd = idxd; + + /* IAA device's local wqs. */ + iaa_device->iaa_local_wqs = kzalloc(sizeof(struct wq_table_entry), GFP_KERNEL); + if (!iaa_device->iaa_local_wqs) + goto err; + + local = iaa_device->iaa_local_wqs; + + local->wqs = kzalloc(iaa_device->idxd->max_wqs * sizeof(struct wq *), GFP_KERNEL); + if (!local->wqs) + goto err; + + local->max_wqs = iaa_device->idxd->max_wqs; + local->n_wqs = 0; INIT_LIST_HEAD(&iaa_device->wqs); return iaa_device; + +err: + if (iaa_device) { + if (iaa_device->iaa_local_wqs) { + if (iaa_device->iaa_local_wqs->wqs) + kfree(iaa_device->iaa_local_wqs->wqs); + kfree(iaa_device->iaa_local_wqs); + } + kfree(iaa_device); + } + + return NULL; } static bool iaa_has_wq(struct iaa_device *iaa_device, struct idxd_wq *wq) @@ -491,12 +521,10 @@ static struct iaa_device *add_iaa_device(struct idxd_device *idxd) { struct iaa_device *iaa_device; - iaa_device = iaa_device_alloc(); + iaa_device = iaa_device_alloc(idxd); if (!iaa_device) return NULL; - iaa_device->idxd = idxd; - list_add_tail(&iaa_device->list, &iaa_devices); nr_iaa++; @@ -537,6 +565,7 @@ static int add_iaa_wq(struct iaa_device *iaa_device, struct idxd_wq *wq, iaa_wq->wq = wq; iaa_wq->iaa_device = iaa_device; idxd_wq_set_private(wq, iaa_wq); + iaa_wq->mapped = false; list_add_tail(&iaa_wq->list, &iaa_device->wqs); @@ -580,6 +609,13 @@ static void free_iaa_device(struct iaa_device *iaa_device) return; remove_device_compression_modes(iaa_device); + + if (iaa_device->iaa_local_wqs) { + if (iaa_device->iaa_local_wqs->wqs) + kfree(iaa_device->iaa_local_wqs->wqs); + kfree(iaa_device->iaa_local_wqs); + } + kfree(iaa_device); } @@ -716,9 +752,14 @@ static int save_iaa_wq(struct idxd_wq *wq) if (WARN_ON(nr_iaa == 0)) return -EINVAL; - cpus_per_iaa = (nr_nodes * nr_cpus_per_node) / nr_iaa; + cpus_per_iaa = (nr_packages * nr_cpus_per_package) / nr_iaa; if (!cpus_per_iaa) cpus_per_iaa = 1; + + nr_iaa_per_package = nr_iaa / nr_packages; + if (!nr_iaa_per_package) + nr_iaa_per_package = 1; + out: return 0; } @@ -735,53 +776,45 @@ static void remove_iaa_wq(struct idxd_wq *wq) } if (nr_iaa) { - cpus_per_iaa = (nr_nodes * nr_cpus_per_node) / nr_iaa; + cpus_per_iaa = (nr_packages * nr_cpus_per_package) / nr_iaa; if (!cpus_per_iaa) cpus_per_iaa = 1; - } else + + nr_iaa_per_package = nr_iaa / nr_packages; + if (!nr_iaa_per_package) + nr_iaa_per_package = 1; + } else { cpus_per_iaa = 1; + nr_iaa_per_package = 1; + } } /*************************************************************** * Mapping IAA devices and wqs to cores with per-cpu wq_tables. ***************************************************************/ -static void wq_table_free_entry(int cpu) -{ - struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); - - kfree(entry->wqs); - memset(entry, 0, sizeof(*entry)); -} - -static void wq_table_clear_entry(int cpu) -{ - struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); - - entry->n_wqs = 0; - entry->cur_wq = 0; - memset(entry->wqs, 0, entry->max_wqs * sizeof(struct idxd_wq *)); -} - -static void clear_wq_table(void) +/* + * Given a cpu, find the closest IAA instance. The idea is to try to + * choose the most appropriate IAA instance for a caller and spread + * available workqueues around to clients. + */ +static inline int cpu_to_iaa(int cpu) { - int cpu; - - for (cpu = 0; cpu < nr_cpus; cpu++) - wq_table_clear_entry(cpu); + int package_id, base_iaa, iaa = 0; - pr_debug("cleared wq table\n"); -} + if (!nr_packages || !nr_iaa_per_package) + return 0; -static void free_wq_table(void) -{ - int cpu; + package_id = topology_logical_package_id(cpu); + base_iaa = package_id * nr_iaa_per_package; + iaa = base_iaa + ((cpu % nr_cpus_per_package) / cpus_per_iaa); - for (cpu = 0; cpu < nr_cpus; cpu++) - wq_table_free_entry(cpu); + pr_debug("cpu = %d, package_id = %d, base_iaa = %d, iaa = %d", + cpu, package_id, base_iaa, iaa); - free_percpu(wq_table); + if (iaa >= 0 && iaa < nr_iaa) + return iaa; - pr_debug("freed wq table\n"); + return (nr_iaa - 1); } static int alloc_wq_table(int max_wqs) @@ -795,13 +828,11 @@ static int alloc_wq_table(int max_wqs) for (cpu = 0; cpu < nr_cpus; cpu++) { entry = per_cpu_ptr(wq_table, cpu); - entry->wqs = kcalloc(max_wqs, sizeof(struct wq *), GFP_KERNEL); - if (!entry->wqs) { - free_wq_table(); - return -ENOMEM; - } + entry->wqs = NULL; entry->max_wqs = max_wqs; + entry->n_wqs = 0; + entry->cur_wq = 0; } pr_debug("initialized wq table\n"); @@ -809,33 +840,27 @@ static int alloc_wq_table(int max_wqs) return 0; } -static void wq_table_add(int cpu, struct idxd_wq *wq) +static void wq_table_add(int cpu, struct wq_table_entry *iaa_local_wqs) { struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); - if (WARN_ON(entry->n_wqs == entry->max_wqs)) - return; - - entry->wqs[entry->n_wqs++] = wq; + entry->wqs = iaa_local_wqs->wqs; + entry->max_wqs = iaa_local_wqs->max_wqs; + entry->n_wqs = iaa_local_wqs->n_wqs; + entry->cur_wq = 0; - pr_debug("%s: added iaa wq %d.%d to idx %d of cpu %d\n", __func__, + pr_debug("%s: cpu %d: added %d iaa local wqs up to wq %d.%d\n", __func__, + cpu, entry->n_wqs, entry->wqs[entry->n_wqs - 1]->idxd->id, - entry->wqs[entry->n_wqs - 1]->id, entry->n_wqs - 1, cpu); + entry->wqs[entry->n_wqs - 1]->id); } static int wq_table_add_wqs(int iaa, int cpu) { struct iaa_device *iaa_device, *found_device = NULL; - int ret = 0, cur_iaa = 0, n_wqs_added = 0; - struct idxd_device *idxd; - struct iaa_wq *iaa_wq; - struct pci_dev *pdev; - struct device *dev; + int ret = 0, cur_iaa = 0; list_for_each_entry(iaa_device, &iaa_devices, list) { - idxd = iaa_device->idxd; - pdev = idxd->pdev; - dev = &pdev->dev; if (cur_iaa != iaa) { cur_iaa++; @@ -843,7 +868,8 @@ static int wq_table_add_wqs(int iaa, int cpu) } found_device = iaa_device; - dev_dbg(dev, "getting wq from iaa_device %d, cur_iaa %d\n", + dev_dbg(&found_device->idxd->pdev->dev, + "getting wq from iaa_device %d, cur_iaa %d\n", found_device->idxd->id, cur_iaa); break; } @@ -858,29 +884,58 @@ static int wq_table_add_wqs(int iaa, int cpu) } cur_iaa = 0; - idxd = found_device->idxd; - pdev = idxd->pdev; - dev = &pdev->dev; - dev_dbg(dev, "getting wq from only iaa_device %d, cur_iaa %d\n", + dev_dbg(&found_device->idxd->pdev->dev, + "getting wq from only iaa_device %d, cur_iaa %d\n", found_device->idxd->id, cur_iaa); } - list_for_each_entry(iaa_wq, &found_device->wqs, list) { - wq_table_add(cpu, iaa_wq->wq); - pr_debug("rebalance: added wq for cpu=%d: iaa wq %d.%d\n", - cpu, iaa_wq->wq->idxd->id, iaa_wq->wq->id); - n_wqs_added++; + wq_table_add(cpu, found_device->iaa_local_wqs); + +out: + return ret; +} + +static int map_iaa_device_wqs(struct iaa_device *iaa_device) +{ + struct wq_table_entry *local; + int ret = 0, n_wqs_added = 0; + struct iaa_wq *iaa_wq; + + local = iaa_device->iaa_local_wqs; + + list_for_each_entry(iaa_wq, &iaa_device->wqs, list) { + if (iaa_wq->mapped && ++n_wqs_added) + continue; + + pr_debug("iaa_device %px: processing wq %d.%d\n", iaa_device, iaa_device->idxd->id, iaa_wq->wq->id); + + if (WARN_ON(local->n_wqs == local->max_wqs)) + break; + + local->wqs[local->n_wqs++] = iaa_wq->wq; + pr_debug("iaa_device %px: added local wq %d.%d\n", iaa_device, iaa_device->idxd->id, iaa_wq->wq->id); + + iaa_wq->mapped = true; + ++n_wqs_added; } - if (!n_wqs_added) { - pr_debug("couldn't find any iaa wqs!\n"); + if (!n_wqs_added && !iaa_device->n_wq) { + pr_debug("iaa_device %d: couldn't find any iaa wqs!\n", iaa_device->idxd->id); ret = -EINVAL; - goto out; } -out: + return ret; } +static void map_iaa_devices(void) +{ + struct iaa_device *iaa_device; + + list_for_each_entry(iaa_device, &iaa_devices, list) { + BUG_ON(map_iaa_device_wqs(iaa_device)); + } +} + /* * Rebalance the wq table so that given a cpu, it's easy to find the * closest IAA instance. The idea is to try to choose the most @@ -889,48 +944,42 @@ static int wq_table_add_wqs(int iaa, int cpu) */ static void rebalance_wq_table(void) { - const struct cpumask *node_cpus; - int node, cpu, iaa = -1; + int cpu, iaa; if (nr_iaa == 0) return; - pr_debug("rebalance: nr_nodes=%d, nr_cpus %d, nr_iaa %d, cpus_per_iaa %d\n", - nr_nodes, nr_cpus, nr_iaa, cpus_per_iaa); + map_iaa_devices(); - clear_wq_table(); + pr_debug("rebalance: nr_packages=%d, nr_cpus %d, nr_iaa %d, cpus_per_iaa %d\n", + nr_packages, nr_cpus, nr_iaa, cpus_per_iaa); - if (nr_iaa == 1) { - for (cpu = 0; cpu < nr_cpus; cpu++) { - if (WARN_ON(wq_table_add_wqs(0, cpu))) { - pr_debug("could not add any wqs for iaa 0 to cpu %d!\n", cpu); - return; - } + for (cpu = 0; cpu < nr_cpus; cpu++) { + iaa = cpu_to_iaa(cpu); + pr_debug("rebalance: cpu=%d iaa=%d\n", cpu, iaa); + + if (WARN_ON(iaa == -1)) { + pr_debug("rebalance (cpu_to_iaa(%d)) failed!\n", cpu); + return; } - return; + if (WARN_ON(wq_table_add_wqs(iaa, cpu))) { + pr_debug("could not add any wqs for iaa %d to cpu %d!\n", iaa, cpu); + return; + } } - for_each_node_with_cpus(node) { - node_cpus = cpumask_of_node(node); - - for (cpu = 0; cpu < cpumask_weight(node_cpus); cpu++) { - int node_cpu = cpumask_nth(cpu, node_cpus); - - if (WARN_ON(node_cpu >= nr_cpu_ids)) { - pr_debug("node_cpu %d doesn't exist!\n", node_cpu); - return; - } - - if ((cpu % cpus_per_iaa) == 0) - iaa++; + pr_debug("Finished rebalance local wqs."); +} - if (WARN_ON(wq_table_add_wqs(iaa, node_cpu))) { - pr_debug("could not add any wqs for iaa %d to cpu %d!\n", iaa, cpu); - return; - } - } +static void free_wq_tables(void) +{ + if (wq_table) { + free_percpu(wq_table); + wq_table = NULL; } + + pr_debug("freed local wq table\n"); } /*************************************************************** @@ -2347,7 +2396,7 @@ static int iaa_crypto_probe(struct idxd_dev *idxd_dev) free_iaa_wq(idxd_wq_get_private(wq)); err_save: if (first_wq) - free_wq_table(); + free_wq_tables(); err_alloc: mutex_unlock(&iaa_devices_lock); idxd_drv_disable_wq(wq); @@ -2397,7 +2446,9 @@ static void iaa_crypto_remove(struct idxd_dev *idxd_dev) if (nr_iaa == 0) { iaa_crypto_enabled = false; - free_wq_table(); + free_wq_tables(); + BUG_ON(!list_empty(&iaa_devices)); + INIT_LIST_HEAD(&iaa_devices); module_put(THIS_MODULE); pr_info("iaa_crypto now DISABLED\n"); @@ -2423,16 +2474,11 @@ static struct idxd_device_driver iaa_crypto_driver = { static int __init iaa_crypto_init_module(void) { int ret = 0; - int node; + INIT_LIST_HEAD(&iaa_devices); nr_cpus = num_possible_cpus(); - for_each_node_with_cpus(node) - nr_nodes++; - if (!nr_nodes) { - pr_err("IAA couldn't find any nodes with cpus\n"); - return -ENODEV; - } - nr_cpus_per_node = nr_cpus / nr_nodes; + nr_cpus_per_package = topology_num_cores_per_package(); + nr_packages = topology_max_packages(); if (crypto_has_comp("deflate-generic", 0, 0)) deflate_generic_tfm = crypto_alloc_comp("deflate-generic", 0, 0); From patchwork Thu Feb 6 07:20:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962460 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4509FC02196 for ; Thu, 6 Feb 2025 07:21:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82D8B280007; Thu, 6 Feb 2025 02:21:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DD76280004; Thu, 6 Feb 2025 02:21:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45CEC280008; Thu, 6 Feb 2025 02:21:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0F53E280007 for ; Thu, 6 Feb 2025 02:21:15 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C2659A0F28 for ; Thu, 6 Feb 2025 07:21:14 +0000 (UTC) X-FDA: 83088673668.07.376F8E6 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf08.hostedemail.com (Postfix) with ESMTP id 8BBD9160002 for ; Thu, 6 Feb 2025 07:21:12 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=J2G0LJXO; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826472; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tjngmrlkW7UqZQ0sA0jUJWmmdEL6rxb2rHtg5uisnNA=; b=5PYRpkTGL1K+7qnJXuHdI/1Aj2jpXyvjA18EsgKAW/W2XscVO74iCrm32Rt4dXGvmILbjv 24JPtoJo+RZ89Mw+KmavlarIFDY+arQ2N/D5zUkBpS3N52jeknrrejQ9AnfG7Yu9o75YFt N+u3h3zBlkcRxi4YrODPes0FCRVxMc0= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=J2G0LJXO; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826472; a=rsa-sha256; cv=none; b=5RwUWr2IY3Eas99LW5AIeWBLbOcTQGtpdfTBbUXI++UH1jIvw/4jAIxLGCaf4ca94IlDWn wWm7LEZQydR9k1go3/Kfi9lxssKjm50wmgyE0vMuU298tEoLzHaFnXE5GwX1IOXVp+6yYm 0eGr4sapEsmO1/4FCMyvAAUV8/oU0dI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826473; x=1770362473; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IDbtEQQk6Os3XMIKUQM72g73nrBxFon7dOQsQv4YhrE=; b=J2G0LJXOwgciXWytMjmLTe55g9nXpvHSxCnvlLzJlPRgwuposis58Wwn 0RpbBs4NUTGXCvpDBKhRelj+ucePp6qIuIUaBgMTqjg4xnB9TNyGOifKO vNcmNWUkxg+1LBAlih1h/Pzz9DK63GSw5PKOiIJ0A7NO3/wXU0pnwdkRe qR4ScuJALDTQtG7T5liko80JmSG7VKZTJ02xwOyI4p08tU/ZMpp8pYDbp bKd95+x8Hj1VjDrDDquYwNHSwqrvKl6fU0kwU6RQzbg/HX8xumNgYaUT2 V0LdI2nj6fJtdCzN5c9clFC9sAVR5xK2Go0PlnPE+KzzeI4oiEQxSdlr3 Q==; X-CSE-ConnectionGUID: 0BYznckYRdSTD0LMTB5vlg== X-CSE-MsgGUID: gsJbzB4hSq29CUeK2BG+rQ== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962671" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962671" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:05 -0800 X-CSE-ConnectionGUID: fTaq8w1OTDOe5U0+CuIpFQ== X-CSE-MsgGUID: fwL/Ol1sQDq4u9YW0o2vEA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022624" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:05 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 09/16] crypto: iaa - Distribute compress jobs from all cores to all IAAs on a package. Date: Wed, 5 Feb 2025 23:20:55 -0800 Message-Id: <20250206072102.29045-10-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 8BBD9160002 X-Stat-Signature: xcuz7gj3i919jdqdjfj76j46yodijusp X-Rspam-User: X-HE-Tag: 1738826472-683459 X-HE-Meta: U2FsdGVkX1973jQCZr/LsEyQ7BnTEPpu39Eohm4iPrxpWURGyd8RiUwIuTBPSleyXwSpgI6K2N+G+DcBtbKbyahteAl0H7C+sTPrhTSZre1oPo9jgmh9nAqK01BJb5ZeN+guPIh55+CYX5utnyMCUiACKv/uQotNAuQ6VUfC9heigMLNCEQ6FhV2GsdxvCf8ym9IsnYoWxtuuZnPknR/Y2/SJ5loCRkmiRQLJt4B4vq00L+zIaZSR+piw02iYfOuuJy4uIwSCJIj/sKEHulxaFejuPWpEwByyDOStsBuPVYU3lhf3B7IkWwWNtF6g/zg4zG579smM9yZBbh186lIcL3VVjoxizelyrcoXkwvss2iYAUaBSRKX8oyuzfRuMffu2igisGJSWZsc4AIOmEZPO9lCFVIIi1au8TKu0WBVMpLgJH5gZ5nFnlp8ZVweEJ/ZCL/7QKq06mco7ZyJWjC3aThJ8tXgPhrCLLB1nDOwRQvwBDfyezL4SF2ulSamuamvvQwecsoxWGzJE4o98lht6n3uJMCvazRK9HoWBNrtD1SaLCUOA3ExQBu9El80fw+/jmjy7v3zTqQGyRZW08Ie/DP7JCl1/Q/ityP3ifBIHLH2BDId6xZED/hgHjuPh3wO4/EYvqwkM2WrJf0Y1of7xPzXaYn+7Dzvjls9IjFLzOYNxJ1GKaR8xGt3+igpo7kfoYwQPXnxgvoogQtueCVJxnaprthWeuv9tjyKZR7k4rYAU53jY3Hrf0qDRCawdHpzTJHooZh4M27BdWHZ3WdrqS60sYCtovl4v4ikSGzviZpSmTJ61SN9XDSyXKCH586/heT53h1j58HhThxIU1XevR8y1aUEcEHlD8NRvTgXrYi11YRFAPNNX/NysNEox7dItoBHGy9AO8Z1+bdJ0ia7/vBZra1YNdHigHOH5oeRwFm+rVUeZEVVH9oj5Vo7kLQv8dZW+88gKaX8BKJMZm BXsJBM6z jnN3W0E/Wjqt32xbg2jQ7KUZi5+GehmUTAyLWPbU9uaTpLFvgEUa0CJknMfC72xS/aVqPTWMG2lBttQqguY4hPRRUZybiCbbtABwaIKRyG49pT0khKJxw1m/fFGS3dcokH5pSnDjLZSFt2H/4U4bBqnn/GkscgH3jteGCWiBOMqwP2asblWuacP288v1vyF5qa/TOaS7Fx+umuxszDfKO7lor/Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This change enables processes running on any logical core on a package to use all the IAA devices enabled on that package for compress jobs. In other words, compressions originating from any process in a package will be distributed in round-robin manner to the available IAA devices on the same package. This is not the default behavior, and is recommended only for highly contended scenarios, when there is significant swapout/swapin activity. The commit log describes how to enable this feature through driver parameters, but the key thing to note is that this requires configuring 2 work-queues per IAA device (each with 64 entries), with 1 WQ used solely for decompress jobs, and the other WQ used solely for compress jobs. Hence the above recommendation. The main premise behind this change is to make sure that no compress engines on any IAA device are un-utilized/under-utilized/over-utilized. In other words, the compress engines on all IAA devices are considered a global resource for that package, thus maximizing compression throughput. This allows the use of all IAA devices present in a given package for (batched) compressions originating from zswap/zram, from all cores on this package. A new per-cpu "global_wq_table" implements this in the iaa_crypto driver. We can think of the global WQ per IAA as a WQ to which all cores on that package can submit compress jobs. To avail of this feature, the user must configure 2 WQs per IAA in order to enable distribution of compress jobs to multiple IAA devices. Each IAA will have 2 WQs: wq.0 (local WQ): Used for decompress jobs from cores mapped by the cpu_to_iaa() "even balancing of logical cores to IAA devices" algorithm. wq.1 (global WQ): Used for compress jobs from *all* logical cores on that package. The iaa_crypto driver will place all global WQs from all same-package IAA devices in the global_wq_table per cpu on that package. When the driver receives a compress job, it will lookup the "next" global WQ in the cpu's global_wq_table to submit the descriptor. The starting wq in the global_wq_table for each cpu is the global wq associated with the IAA nearest to it, so that we stagger the starting global wq for each process. This results in very uniform usage of all IAAs for compress jobs. Two new driver module parameters are added for this feature: g_wqs_per_iaa (default 0): /sys/bus/dsa/drivers/crypto/g_wqs_per_iaa This represents the number of global WQs that can be configured per IAA device. The recommended setting is 1 to enable the use of this feature once the user configures 2 WQs per IAA using higher level scripts as described in Documentation/driver-api/crypto/iaa/iaa-crypto.rst. g_consec_descs_per_gwq (default 1): /sys/bus/dsa/drivers/crypto/g_consec_descs_per_gwq This represents the number of consecutive compress jobs that will be submitted to the same global WQ (i.e. to the same IAA device) from a given core, before moving to the next global WQ. The default is 1, which is also the recommended setting to avail of this feature. The decompress jobs from any core will be sent to the "local" IAA, namely the one that the driver assigns with the cpu_to_iaa() mapping algorithm that evenly balances the assignment of logical cores to IAA devices on a package. On a 2-package Sapphire Rapids server where each package has 56 cores and 4 IAA devices, this is how the compress/decompress jobs will be mapped when the user configures 2 WQs per IAA device (which implies wq.1 will be added to the global WQ table for each logical core on that package): package(s): 2 package0 CPU(s): 0-55,112-167 package1 CPU(s): 56-111,168-223 Compress jobs: -------------- package 0: iaa_crypto will send compress jobs from all cpus (0-55,112-167) to all IAA devices on the package (iax1/iax3/iax5/iax7) in round-robin manner: iaa: iax1 iax3 iax5 iax7 package 1: iaa_crypto will send compress jobs from all cpus (56-111,168-223) to all IAA devices on the package (iax9/iax11/iax13/iax15) in round-robin manner: iaa: iax9 iax11 iax13 iax15 Decompress jobs: ---------------- package 0: cpu 0-13,112-125 14-27,126-139 28-41,140-153 42-55,154-167 iaa: iax1 iax3 iax5 iax7 package 1: cpu 56-69,168-181 70-83,182-195 84-97,196-209 98-111,210-223 iaa: iax9 iax11 iax13 iax15 Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto.h | 1 + drivers/crypto/intel/iaa/iaa_crypto_main.c | 385 ++++++++++++++++++++- 2 files changed, 378 insertions(+), 8 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto.h b/drivers/crypto/intel/iaa/iaa_crypto.h index 74d25e62df12..c46c70ecf355 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto.h +++ b/drivers/crypto/intel/iaa/iaa_crypto.h @@ -91,6 +91,7 @@ struct iaa_device { struct list_head wqs; struct wq_table_entry *iaa_local_wqs; + struct wq_table_entry *iaa_global_wqs; atomic64_t comp_calls; atomic64_t comp_bytes; diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 418f78454875..4ca9028d6050 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -42,6 +42,18 @@ static struct crypto_comp *deflate_generic_tfm; /* Per-cpu lookup table for balanced wqs */ static struct wq_table_entry __percpu *wq_table = NULL; +static struct wq_table_entry **pkg_global_wq_tables = NULL; + +/* Per-cpu lookup table for global wqs shared by all cpus. */ +static struct wq_table_entry __percpu *global_wq_table = NULL; + +/* + * Per-cpu counter of consecutive descriptors allocated to + * the same wq in the global_wq_table, so that we know + * when to switch to the next wq in the global_wq_table. + */ +static int __percpu *num_consec_descs_per_wq = NULL; + /* Verify results of IAA compress or not */ static bool iaa_verify_compress = false; @@ -79,6 +91,16 @@ static bool async_mode = true; /* Use interrupts */ static bool use_irq; +/* Number of global wqs per iaa*/ +static int g_wqs_per_iaa = 0; + +/* + * Number of consecutive descriptors to allocate from a + * given global wq before switching to the next wq in + * the global_wq_table. + */ +static int g_consec_descs_per_gwq = 1; + static struct iaa_compression_mode *iaa_compression_modes[IAA_COMP_MODES_MAX]; LIST_HEAD(iaa_devices); @@ -180,6 +202,60 @@ static ssize_t sync_mode_store(struct device_driver *driver, } static DRIVER_ATTR_RW(sync_mode); +static ssize_t g_wqs_per_iaa_show(struct device_driver *driver, char *buf) +{ + return sprintf(buf, "%d\n", g_wqs_per_iaa); +} + +static ssize_t g_wqs_per_iaa_store(struct device_driver *driver, + const char *buf, size_t count) +{ + int ret = -EBUSY; + + mutex_lock(&iaa_devices_lock); + + if (iaa_crypto_enabled) + goto out; + + ret = kstrtoint(buf, 10, &g_wqs_per_iaa); + if (ret) + goto out; + + ret = count; +out: + mutex_unlock(&iaa_devices_lock); + + return ret; +} +static DRIVER_ATTR_RW(g_wqs_per_iaa); + +static ssize_t g_consec_descs_per_gwq_show(struct device_driver *driver, char *buf) +{ + return sprintf(buf, "%d\n", g_consec_descs_per_gwq); +} + +static ssize_t g_consec_descs_per_gwq_store(struct device_driver *driver, + const char *buf, size_t count) +{ + int ret = -EBUSY; + + mutex_lock(&iaa_devices_lock); + + if (iaa_crypto_enabled) + goto out; + + ret = kstrtoint(buf, 10, &g_consec_descs_per_gwq); + if (ret) + goto out; + + ret = count; +out: + mutex_unlock(&iaa_devices_lock); + + return ret; +} +static DRIVER_ATTR_RW(g_consec_descs_per_gwq); + /**************************** * Driver compression modes. ****************************/ @@ -465,7 +541,7 @@ static void remove_device_compression_modes(struct iaa_device *iaa_device) ***********************************************************/ static struct iaa_device *iaa_device_alloc(struct idxd_device *idxd) { - struct wq_table_entry *local; + struct wq_table_entry *local, *global; struct iaa_device *iaa_device; iaa_device = kzalloc(sizeof(*iaa_device), GFP_KERNEL); @@ -488,6 +564,20 @@ static struct iaa_device *iaa_device_alloc(struct idxd_device *idxd) local->max_wqs = iaa_device->idxd->max_wqs; local->n_wqs = 0; + /* IAA device's global wqs. */ + iaa_device->iaa_global_wqs = kzalloc(sizeof(struct wq_table_entry), GFP_KERNEL); + if (!iaa_device->iaa_global_wqs) + goto err; + + global = iaa_device->iaa_global_wqs; + + global->wqs = kzalloc(iaa_device->idxd->max_wqs * sizeof(struct wq *), GFP_KERNEL); + if (!global->wqs) + goto err; + + global->max_wqs = iaa_device->idxd->max_wqs; + global->n_wqs = 0; + INIT_LIST_HEAD(&iaa_device->wqs); return iaa_device; @@ -499,6 +589,8 @@ static struct iaa_device *iaa_device_alloc(struct idxd_device *idxd) kfree(iaa_device->iaa_local_wqs->wqs); kfree(iaa_device->iaa_local_wqs); } + if (iaa_device->iaa_global_wqs) + kfree(iaa_device->iaa_global_wqs); kfree(iaa_device); } @@ -616,6 +708,12 @@ static void free_iaa_device(struct iaa_device *iaa_device) kfree(iaa_device->iaa_local_wqs); } + if (iaa_device->iaa_global_wqs) { + if (iaa_device->iaa_global_wqs->wqs) + kfree(iaa_device->iaa_global_wqs->wqs); + kfree(iaa_device->iaa_global_wqs); + } + kfree(iaa_device); } @@ -817,6 +915,58 @@ static inline int cpu_to_iaa(int cpu) return (nr_iaa - 1); } +static void free_global_wq_table(void) +{ + if (global_wq_table) { + free_percpu(global_wq_table); + global_wq_table = NULL; + } + + if (num_consec_descs_per_wq) { + free_percpu(num_consec_descs_per_wq); + num_consec_descs_per_wq = NULL; + } + + pr_debug("freed global wq table\n"); +} + +static int pkg_global_wq_tables_alloc(void) +{ + int i, j; + + pkg_global_wq_tables = kzalloc(nr_packages * sizeof(*pkg_global_wq_tables), GFP_KERNEL); + if (!pkg_global_wq_tables) + return -ENOMEM; + + for (i = 0; i < nr_packages; ++i) { + pkg_global_wq_tables[i] = kzalloc(sizeof(struct wq_table_entry), GFP_KERNEL); + + if (!pkg_global_wq_tables[i]) { + for (j = 0; j < i; ++j) + kfree(pkg_global_wq_tables[j]); + kfree(pkg_global_wq_tables); + pkg_global_wq_tables = NULL; + return -ENOMEM; + } + pkg_global_wq_tables[i]->wqs = NULL; + } + + return 0; +} + +static void pkg_global_wq_tables_dealloc(void) +{ + int i; + + for (i = 0; i < nr_packages; ++i) { + if (pkg_global_wq_tables[i]->wqs) + kfree(pkg_global_wq_tables[i]->wqs); + kfree(pkg_global_wq_tables[i]); + } + kfree(pkg_global_wq_tables); + pkg_global_wq_tables = NULL; +} + static int alloc_wq_table(int max_wqs) { struct wq_table_entry *entry; @@ -835,6 +985,35 @@ static int alloc_wq_table(int max_wqs) entry->cur_wq = 0; } + global_wq_table = alloc_percpu(struct wq_table_entry); + if (!global_wq_table) + return 0; + + for (cpu = 0; cpu < nr_cpus; cpu++) { + entry = per_cpu_ptr(global_wq_table, cpu); + + entry->wqs = NULL; + entry->max_wqs = max_wqs; + entry->n_wqs = 0; + entry->cur_wq = 0; + } + + num_consec_descs_per_wq = alloc_percpu(int); + if (!num_consec_descs_per_wq) { + free_global_wq_table(); + return 0; + } + + for (cpu = 0; cpu < nr_cpus; cpu++) { + int *num_consec_descs = per_cpu_ptr(num_consec_descs_per_wq, cpu); + *num_consec_descs = 0; + } + + if (pkg_global_wq_tables_alloc()) { + free_global_wq_table(); + return 0; + } + pr_debug("initialized wq table\n"); return 0; @@ -895,13 +1074,120 @@ static int wq_table_add_wqs(int iaa, int cpu) return ret; } +static void pkg_global_wq_tables_reinit(void) +{ + int i, cur_iaa = 0, pkg = 0, nr_pkg_wqs = 0; + struct iaa_device *iaa_device; + struct wq_table_entry *global; + + if (!pkg_global_wq_tables) + return; + + /* Reallocate per-package wqs. */ + list_for_each_entry(iaa_device, &iaa_devices, list) { + global = iaa_device->iaa_global_wqs; + nr_pkg_wqs += global->n_wqs; + + if (++cur_iaa == nr_iaa_per_package) { + nr_pkg_wqs = nr_pkg_wqs ? max_t(int, iaa_device->idxd->max_wqs, nr_pkg_wqs) : 0; + + if (pkg_global_wq_tables[pkg]->wqs) { + kfree(pkg_global_wq_tables[pkg]->wqs); + pkg_global_wq_tables[pkg]->wqs = NULL; + } + + if (nr_pkg_wqs) + pkg_global_wq_tables[pkg]->wqs = kzalloc(nr_pkg_wqs * + sizeof(struct wq *), + GFP_KERNEL); + + pkg_global_wq_tables[pkg]->n_wqs = 0; + pkg_global_wq_tables[pkg]->cur_wq = 0; + pkg_global_wq_tables[pkg]->max_wqs = nr_pkg_wqs; + + if (++pkg == nr_packages) + break; + cur_iaa = 0; + nr_pkg_wqs = 0; + } + } + + pkg = 0; + cur_iaa = 0; + + /* Re-initialize per-package wqs. */ + list_for_each_entry(iaa_device, &iaa_devices, list) { + global = iaa_device->iaa_global_wqs; + + if (pkg_global_wq_tables[pkg]->wqs) + for (i = 0; i < global->n_wqs; ++i) + pkg_global_wq_tables[pkg]->wqs[pkg_global_wq_tables[pkg]->n_wqs++] = global->wqs[i]; + + pr_debug("pkg_global_wq_tables[%d] has %d wqs", pkg, pkg_global_wq_tables[pkg]->n_wqs); + + if (++cur_iaa == nr_iaa_per_package) { + if (++pkg == nr_packages) + break; + cur_iaa = 0; + } + } +} + +static void global_wq_table_add(int cpu, struct wq_table_entry *pkg_global_wq_table) +{ + struct wq_table_entry *entry = per_cpu_ptr(global_wq_table, cpu); + + /* This could be NULL. */ + entry->wqs = pkg_global_wq_table->wqs; + entry->max_wqs = pkg_global_wq_table->max_wqs; + entry->n_wqs = pkg_global_wq_table->n_wqs; + entry->cur_wq = 0; + + if (entry->wqs) + pr_debug("%s: cpu %d: added %d iaa global wqs up to wq %d.%d\n", __func__, + cpu, entry->n_wqs, + entry->wqs[entry->n_wqs - 1]->idxd->id, + entry->wqs[entry->n_wqs - 1]->id); +} + +static void global_wq_table_set_start_wq(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(global_wq_table, cpu); + int start_wq = g_wqs_per_iaa * (cpu_to_iaa(cpu) % nr_iaa_per_package); + + if ((start_wq >= 0) && (start_wq < entry->n_wqs)) + entry->cur_wq = start_wq; +} + +static void global_wq_table_add_wqs(void) +{ + int cpu; + + if (!pkg_global_wq_tables) + return; + + for (cpu = 0; cpu < nr_cpus; cpu += nr_cpus_per_package) { + /* cpu's on the same package get the same global_wq_table. */ + int package_id = topology_logical_package_id(cpu); + int pkg_cpu; + + for (pkg_cpu = cpu; pkg_cpu < cpu + nr_cpus_per_package; ++pkg_cpu) { + if (pkg_global_wq_tables[package_id]->n_wqs > 0) { + global_wq_table_add(pkg_cpu, pkg_global_wq_tables[package_id]); + global_wq_table_set_start_wq(pkg_cpu); + } + } + } +} + static int map_iaa_device_wqs(struct iaa_device *iaa_device) { - struct wq_table_entry *local; + struct wq_table_entry *local, *global; int ret = 0, n_wqs_added = 0; struct iaa_wq *iaa_wq; local = iaa_device->iaa_local_wqs; + global = iaa_device->iaa_global_wqs; list_for_each_entry(iaa_wq, &iaa_device->wqs, list) { if (iaa_wq->mapped && ++n_wqs_added) @@ -909,11 +1195,18 @@ static int map_iaa_device_wqs(struct iaa_device *iaa_device) pr_debug("iaa_device %px: processing wq %d.%d\n", iaa_device, iaa_device->idxd->id, iaa_wq->wq->id); - if (WARN_ON(local->n_wqs == local->max_wqs)) - break; + if ((!n_wqs_added || ((n_wqs_added + g_wqs_per_iaa) < iaa_device->n_wq)) && + (local->n_wqs < local->max_wqs)) { + + local->wqs[local->n_wqs++] = iaa_wq->wq; + pr_debug("iaa_device %px: added local wq %d.%d\n", iaa_device, iaa_device->idxd->id, iaa_wq->wq->id); + } else { + if (WARN_ON(global->n_wqs == global->max_wqs)) + break; - local->wqs[local->n_wqs++] = iaa_wq->wq; - pr_debug("iaa_device %px: added local wq %d.%d\n", iaa_device, iaa_device->idxd->id, iaa_wq->wq->id); + global->wqs[global->n_wqs++] = iaa_wq->wq; + pr_debug("iaa_device %px: added global wq %d.%d\n", iaa_device, iaa_device->idxd->id, iaa_wq->wq->id); + } iaa_wq->mapped = true; ++n_wqs_added; @@ -969,6 +1262,10 @@ static void rebalance_wq_table(void) } } + if (iaa_crypto_enabled && pkg_global_wq_tables) { + pkg_global_wq_tables_reinit(); + global_wq_table_add_wqs(); + } pr_debug("Finished rebalance local wqs."); } @@ -979,7 +1276,17 @@ static void free_wq_tables(void) wq_table = NULL; } - pr_debug("freed local wq table\n"); + if (global_wq_table) { + free_percpu(global_wq_table); + global_wq_table = NULL; + } + + if (num_consec_descs_per_wq) { + free_percpu(num_consec_descs_per_wq); + num_consec_descs_per_wq = NULL; + } + + pr_debug("freed wq tables\n"); } /*************************************************************** @@ -1002,6 +1309,35 @@ static struct idxd_wq *wq_table_next_wq(int cpu) return entry->wqs[entry->cur_wq]; } +/* + * Caller should make sure to call only if the + * per_cpu_ptr "global_wq_table" is non-NULL + * and has at least one wq configured. + */ +static struct idxd_wq *global_wq_table_next_wq(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(global_wq_table, cpu); + int *num_consec_descs = per_cpu_ptr(num_consec_descs_per_wq, cpu); + + /* + * Fall-back to local IAA's wq if there were no global wqs configured + * for any IAA device, or if there were problems in setting up global + * wqs for this cpu's package. + */ + if (!entry->wqs) + return wq_table_next_wq(cpu); + + if ((*num_consec_descs) == g_consec_descs_per_gwq) { + if (++entry->cur_wq >= entry->n_wqs) + entry->cur_wq = 0; + *num_consec_descs = 0; + } + + ++(*num_consec_descs); + + return entry->wqs[entry->cur_wq]; +} + /************************************************* * Core iaa_crypto compress/decompress functions. *************************************************/ @@ -1563,6 +1899,7 @@ static int iaa_comp_acompress(struct acomp_req *req) struct idxd_wq *wq; struct device *dev; int order = -1; + struct wq_table_entry *entry; compression_ctx = crypto_tfm_ctx(tfm); @@ -1581,8 +1918,15 @@ static int iaa_comp_acompress(struct acomp_req *req) disable_async = true; cpu = get_cpu(); - wq = wq_table_next_wq(cpu); + entry = per_cpu_ptr(global_wq_table, cpu); + + if (!entry || !entry->wqs || entry->n_wqs == 0) { + wq = wq_table_next_wq(cpu); + } else { + wq = global_wq_table_next_wq(cpu); + } put_cpu(); + if (!wq) { pr_debug("no wq configured for cpu=%d\n", cpu); return -ENODEV; @@ -2446,6 +2790,7 @@ static void iaa_crypto_remove(struct idxd_dev *idxd_dev) if (nr_iaa == 0) { iaa_crypto_enabled = false; + pkg_global_wq_tables_dealloc(); free_wq_tables(); BUG_ON(!list_empty(&iaa_devices)); INIT_LIST_HEAD(&iaa_devices); @@ -2515,6 +2860,20 @@ static int __init iaa_crypto_init_module(void) goto err_sync_attr_create; } + ret = driver_create_file(&iaa_crypto_driver.drv, + &driver_attr_g_wqs_per_iaa); + if (ret) { + pr_debug("IAA g_wqs_per_iaa attr creation failed\n"); + goto err_g_wqs_per_iaa_attr_create; + } + + ret = driver_create_file(&iaa_crypto_driver.drv, + &driver_attr_g_consec_descs_per_gwq); + if (ret) { + pr_debug("IAA g_consec_descs_per_gwq attr creation failed\n"); + goto err_g_consec_descs_per_gwq_attr_create; + } + if (iaa_crypto_debugfs_init()) pr_warn("debugfs init failed, stats not available\n"); @@ -2522,6 +2881,12 @@ static int __init iaa_crypto_init_module(void) out: return ret; +err_g_consec_descs_per_gwq_attr_create: + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_g_wqs_per_iaa); +err_g_wqs_per_iaa_attr_create: + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_sync_mode); err_sync_attr_create: driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_verify_compress); @@ -2545,6 +2910,10 @@ static void __exit iaa_crypto_cleanup_module(void) &driver_attr_sync_mode); driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_verify_compress); + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_g_wqs_per_iaa); + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_g_consec_descs_per_gwq); idxd_driver_unregister(&iaa_crypto_driver); iaa_aecs_cleanup_fixed(); crypto_free_comp(deflate_generic_tfm); From patchwork Thu Feb 6 07:20:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C457CC0219B for ; Thu, 6 Feb 2025 07:21:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 460E2280009; Thu, 6 Feb 2025 02:21:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 40F01280004; Thu, 6 Feb 2025 02:21:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1EB06280008; Thu, 6 Feb 2025 02:21:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E7B51280004 for ; Thu, 6 Feb 2025 02:21:14 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A187A140F11 for ; Thu, 6 Feb 2025 07:21:14 +0000 (UTC) X-FDA: 83088673668.16.14ED312 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf26.hostedemail.com (Postfix) with ESMTP id 68E43140002 for ; Thu, 6 Feb 2025 07:21:12 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=S+GpFWbf; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826472; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gnSmxZqV9wqCUiLPVpgANRl6E1t7e66SUFDb7Q2bhFk=; b=ENbUatr/MufLA/TMA3X/uPyJV0NyTf+QVawaiDd1vL9IV9TjtUBHrsVA9PACqFwVQz5CT+ FsbmS6W6jVGe/D91T/FBUhgd0UqhnyZOljcp1vpC0qLf4qELj/fDfEYSYUCux+ZjBFAhWs lphs2nkNPDL8e2gr/FKge6JdiPQKJcA= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=S+GpFWbf; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826472; a=rsa-sha256; cv=none; b=fCjiLX4pfLqkYskaSuXV9fee/PeBbyFoE/o1RIYCGBZumONoYBZafjU3x6vVXhuSQRd9wQ bPGFkAOK74otfRDOXx18KVblNnO98xhQ4Ftk/ETZo/gjWPgoZsVtCyMCvdgtruCJ2vjvsX tw7rwPf51iwv3VBCZqjMmJ9K5En2bxY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826472; x=1770362472; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=N5FH+kdEATUlMgFfKUdUQJnWoZvD4Wvii2jQxOoHG0g=; b=S+GpFWbf69GTJ+n+c+8YZuByUEAr1ImTpLxlkKN1HAPEtSybqX7T7tBL VeWWA/IxYaskeWfzGxMHaCWZzIeQHq7WYhltLq2E+EhWyhp14YcdvChZH 9dCbmTJOM6lkhIXiDbPxH0AIuvjAO0VU7viMcNp9wsrkw/EaVRPnDFVD6 sP1UWBuLtaXbtOfFm04pEVg6/f1sde+hM3BoHpHk0qaYaRCBuneXY77Ah 3Dl2Lz26BwGzpyNbtLWbh+/fgTUTxUypYRpqgkk8Au0alfkCnYLSTKBHt E6BaqxKWUwjMvT9Zc91Fl8PkgJoFr/uNthEfjyuk1/ddDX554fXylN/0w w==; X-CSE-ConnectionGUID: 3gIEiB2BR4awWRPWIdTmog== X-CSE-MsgGUID: 3YiZQn8rTGiojQ5HhgHHWw== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962677" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962677" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:05 -0800 X-CSE-ConnectionGUID: 6++sgLdmQ82mgLCGkPkmNA== X-CSE-MsgGUID: TSTCr4N2SO6ManNSMSYRPQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022628" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:05 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 10/16] crypto: iaa - Descriptor allocation timeouts with mitigations in iaa_crypto. Date: Wed, 5 Feb 2025 23:20:56 -0800 Message-Id: <20250206072102.29045-11-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 68E43140002 X-Stat-Signature: xgzh1dezrmox4mesrczyc4fdsdpsw4b9 X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738826472-364127 X-HE-Meta: U2FsdGVkX1/sgo2MoGIWdE0N1eEvMptLMvKRsnTZzuEM7SVmtIVVNIfaHWSKVUz+HQWuOuim39ISG5AuEnE24GpB9vDUgCoFfOYAiWYORPGoXqtHABmTDDSxQUCVz69Z0TiR2fR1xVE5GIrTfLWx72dkGQC+J8aj0zggHSN10KaHCM0AzixheRHDDKSac445nAwavbUWGybW8sOqlHToHHpEE1JQ2ykMO/bhsRkF0JXLi5n0YgJzVlBeqlQOJUPuKDE/2WvAUIN/YQWP/vbvYWkbaRfuN9t56Pv3aFSvVoRSXFEYgkKzSWcrkfvXzuWXsmtMK1fKDfGPoMOLrdNxz1spQa07tNNBzxhWqda4VstLHqPsYvvGJx3vHOjUeJj5JB1aew03f9YwnDnG6Sjvs9E7sxtsMo4hSvW6x4/g7/GCoC0CzrYjv6UiahdTZ+rurooWX5q0P6EjC2+/7mPqHs9QsZK+s1/BV3DdjiIMJ37Ngcn4zPwERCmoRb4BsXjR9zecWxVMmk/fiu9WVNW1BRTSYLmS3jKhsOlcC4HR6NJrpB10+JterAep1+zbv2UuUhFoLwWUG/LUNbin+bndm9W6hdFlP91PH7vKJuHaWyEfuLtcNU6BZkmGihEWXoHjNK81bf42AI8IwFzu9/Bascz1mkhBxiBuHdputsqq0bzAJyqophOZMyBzoEMqsSPu+Otr7O3gge7R3VTRlat3uQ6BaGx8IFyi780cnQ2KwRO/RnMZ0duLpoUEbqQnVAGhZ4LZKWBYwwMKJ3f4CWznSVta1nZvp3Gqh0YpN7YpmWYSRMEGpWkfQoHfbYp1uPl63RB8REVc6HSNbdwzkBkAejvndjD622qOHI3GQOuTx0Sa8As1+hBy8SGHtmyn7Pu4DkM9G8A7v+iIZt0fGAwFZzYJcOzmAelixx3/Z8N7Q5l6bCc3OJHUfaeSJj9zXKUAVeCBSyVwfZ/M73vt/FR w2qcmRqr 1REUsgYUgWU9X8Wbg5vOce3zcyvNh+KirnnaJbAFefxKMer2aTGIjNVmN+sBlj5YxImcF0gGoqJD0hnZGqaeoBjGGUT/NI+hnLc2tU6JdovKRiySgyW9EGBI7cmZ8HrkvZ4cY1jH0h7Fn5dC/6azKdr1JY0B0C6685nksCgN7EaX8gxF7ST0tg837Y4qgMNjc6/3wAVObYgpbrYinuYgzQMqA8s2GDxfC1s5I X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch modifies the descriptor allocation from blocking to non-blocking with bounded retries or "timeouts". This is necessary to prevent task blocked errors with higher compress/decompress batch-sizes in high contention scenarios, for instance, when the platform has only 1 IAA device enabled. With 1 IAA device enabled per package on a dual-package SPR with 56 cores/package, there are 112 logical cores mapped to this single IAA device. In this scenario, the task blocked errors can occur because idxd_alloc_desc() is called with IDXD_OP_BLOCK. Any process that is able to obtain IAA_CRYPTO_MAX_BATCH_SIZE (8UL) descriptors, will cause contention for allocating descriptors for all other processes. Under IDXD_OP_BLOCK, this can cause compress/decompress jobs to stall in stress test scenarios (e.g. zswap_store() of 2M folios). In order to make the iaa_crypto driver be more fail-safe, this commit implements the following: 1) Change compress/decompress descriptor allocations to be non-blocking with retries ("timeouts"). 2) Return compress error to zswap if descriptor allocation with timeouts fails during compress ops. zswap_store() will return an error and the folio gets stored in the backing swap device. 3) Fallback to software decompress if descriptor allocation with timeouts fails during decompress ops. 4) Bug fixes for freeing the descriptor consistently in all error cases. With these fixes, there are no task blocked errors seen under stress testing conditions, and no performance degradation observed. Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto.h | 3 + drivers/crypto/intel/iaa/iaa_crypto_main.c | 81 +++++++++++++--------- 2 files changed, 51 insertions(+), 33 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto.h b/drivers/crypto/intel/iaa/iaa_crypto.h index c46c70ecf355..b175e9065025 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto.h +++ b/drivers/crypto/intel/iaa/iaa_crypto.h @@ -21,6 +21,9 @@ #define IAA_COMPLETION_TIMEOUT 1000000 +#define IAA_ALLOC_DESC_COMP_TIMEOUT 1000 +#define IAA_ALLOC_DESC_DECOMP_TIMEOUT 500 + #define IAA_ANALYTICS_ERROR 0x0a #define IAA_ERROR_DECOMP_BUF_OVERFLOW 0x0b #define IAA_ERROR_COMP_BUF_OVERFLOW 0x19 diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 4ca9028d6050..5292d8f7ebd6 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1406,6 +1406,7 @@ static int deflate_generic_decompress(struct acomp_req *req) void *src, *dst; int ret; + req->dlen = PAGE_SIZE; src = kmap_local_page(sg_page(req->src)) + req->src->offset; dst = kmap_local_page(sg_page(req->dst)) + req->dst->offset; @@ -1469,7 +1470,8 @@ static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, struct iaa_device_compression_mode *active_compression_mode; struct iaa_compression_ctx *ctx = crypto_tfm_ctx(tfm); struct iaa_device *iaa_device; - struct idxd_desc *idxd_desc; + struct idxd_desc *idxd_desc = ERR_PTR(-EAGAIN); + int alloc_desc_retries = 0; struct iax_hw_desc *desc; struct idxd_device *idxd; struct iaa_wq *iaa_wq; @@ -1485,7 +1487,11 @@ static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, active_compression_mode = get_iaa_device_compression_mode(iaa_device, ctx->mode); - idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); + while ((idxd_desc == ERR_PTR(-EAGAIN)) && (alloc_desc_retries++ < IAA_ALLOC_DESC_DECOMP_TIMEOUT)) { + idxd_desc = idxd_alloc_desc(wq, IDXD_OP_NONBLOCK); + cpu_relax(); + } + if (IS_ERR(idxd_desc)) { dev_dbg(dev, "idxd descriptor allocation failed\n"); dev_dbg(dev, "iaa compress failed: ret=%ld\n", @@ -1661,7 +1667,8 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, struct iaa_device_compression_mode *active_compression_mode; struct iaa_compression_ctx *ctx = crypto_tfm_ctx(tfm); struct iaa_device *iaa_device; - struct idxd_desc *idxd_desc; + struct idxd_desc *idxd_desc = ERR_PTR(-EAGAIN); + int alloc_desc_retries = 0; struct iax_hw_desc *desc; struct idxd_device *idxd; struct iaa_wq *iaa_wq; @@ -1677,7 +1684,11 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, active_compression_mode = get_iaa_device_compression_mode(iaa_device, ctx->mode); - idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); + while ((idxd_desc == ERR_PTR(-EAGAIN)) && (alloc_desc_retries++ < IAA_ALLOC_DESC_COMP_TIMEOUT)) { + idxd_desc = idxd_alloc_desc(wq, IDXD_OP_NONBLOCK); + cpu_relax(); + } + if (IS_ERR(idxd_desc)) { dev_dbg(dev, "idxd descriptor allocation failed\n"); dev_dbg(dev, "iaa compress failed: ret=%ld\n", PTR_ERR(idxd_desc)); @@ -1753,15 +1764,10 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, *compression_crc = idxd_desc->iax_completion->crc; - if (!ctx->async_mode || disable_async) - idxd_free_desc(wq, idxd_desc); -out: - return ret; err: idxd_free_desc(wq, idxd_desc); - dev_dbg(dev, "iaa compress failed: ret=%d\n", ret); - - goto out; +out: + return ret; } static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, @@ -1773,7 +1779,8 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, struct iaa_device_compression_mode *active_compression_mode; struct iaa_compression_ctx *ctx = crypto_tfm_ctx(tfm); struct iaa_device *iaa_device; - struct idxd_desc *idxd_desc; + struct idxd_desc *idxd_desc = ERR_PTR(-EAGAIN); + int alloc_desc_retries = 0; struct iax_hw_desc *desc; struct idxd_device *idxd; struct iaa_wq *iaa_wq; @@ -1789,12 +1796,18 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, active_compression_mode = get_iaa_device_compression_mode(iaa_device, ctx->mode); - idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); + while ((idxd_desc == ERR_PTR(-EAGAIN)) && (alloc_desc_retries++ < IAA_ALLOC_DESC_DECOMP_TIMEOUT)) { + idxd_desc = idxd_alloc_desc(wq, IDXD_OP_NONBLOCK); + cpu_relax(); + } + if (IS_ERR(idxd_desc)) { dev_dbg(dev, "idxd descriptor allocation failed\n"); dev_dbg(dev, "iaa decompress failed: ret=%ld\n", PTR_ERR(idxd_desc)); - return PTR_ERR(idxd_desc); + ret = PTR_ERR(idxd_desc); + idxd_desc = NULL; + goto fallback_software_decomp; } desc = idxd_desc->iax_hw; @@ -1837,7 +1850,7 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, ret = idxd_submit_desc(wq, idxd_desc); if (ret) { dev_dbg(dev, "submit_desc failed ret=%d\n", ret); - goto err; + goto fallback_software_decomp; } /* Update stats */ @@ -1851,19 +1864,20 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, } ret = check_completion(dev, idxd_desc->iax_completion, false, false); + +fallback_software_decomp: if (ret) { - dev_dbg(dev, "%s: check_completion failed ret=%d\n", __func__, ret); - if (idxd_desc->iax_completion->status == IAA_ANALYTICS_ERROR) { + dev_dbg(dev, "%s: desc allocation/submission/check_completion failed ret=%d\n", __func__, ret); + if (idxd_desc && idxd_desc->iax_completion->status == IAA_ANALYTICS_ERROR) { pr_warn("%s: falling back to deflate-generic decompress, " "analytics error code %x\n", __func__, idxd_desc->iax_completion->error_code); - ret = deflate_generic_decompress(req); - if (ret) { - dev_dbg(dev, "%s: deflate-generic failed ret=%d\n", - __func__, ret); - goto err; - } - } else { + } + + ret = deflate_generic_decompress(req); + + if (ret) { + pr_err("%s: iaa decompress failed: fallback to deflate-generic software decompress error ret=%d\n", __func__, ret); goto err; } } else { @@ -1872,19 +1886,15 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, *dlen = req->dlen; - if (!ctx->async_mode || disable_async) - idxd_free_desc(wq, idxd_desc); - /* Update stats */ update_total_decomp_bytes_in(slen); update_wq_decomp_bytes(wq, slen); + +err: + if (idxd_desc) + idxd_free_desc(wq, idxd_desc); out: return ret; -err: - idxd_free_desc(wq, idxd_desc); - dev_dbg(dev, "iaa decompress failed: ret=%d\n", ret); - - goto out; } static int iaa_comp_acompress(struct acomp_req *req) @@ -2375,6 +2385,7 @@ static bool iaa_comp_acompress_batch( if (errors[i] != -EINPROGRESS) { errors[i] = -EINVAL; err = -EINVAL; + pr_debug("%s desc was not submitted\n", __func__); } else { errors[i] = -EAGAIN; } @@ -2521,9 +2532,13 @@ static bool iaa_comp_adecompress_batch( } else { errors[i] = iaa_comp_adecompress(reqs[i]); + /* + * If it failed desc allocation/submission, errors[i] can + * be 0 or error value from software decompress. + */ if (errors[i] != -EINPROGRESS) { - errors[i] = -EINVAL; err = -EINVAL; + pr_debug("%s desc was not submitted\n", __func__); } else { errors[i] = -EAGAIN; } From patchwork Thu Feb 6 07:20:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5261C02196 for ; Thu, 6 Feb 2025 07:21:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5E6B3280008; Thu, 6 Feb 2025 02:21:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F9B6280004; Thu, 6 Feb 2025 02:21:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 34A02280008; Thu, 6 Feb 2025 02:21:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1389D280004 for ; Thu, 6 Feb 2025 02:21:16 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C95341A0F3F for ; Thu, 6 Feb 2025 07:21:15 +0000 (UTC) X-FDA: 83088673710.08.BFF9FBB Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf07.hostedemail.com (Postfix) with ESMTP id ABC4B40002 for ; Thu, 6 Feb 2025 07:21:13 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="PQcU5Be/"; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826473; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rREwwk16coHUWq++YKmtv6+iXThCUXtHhy5m11W7Y0U=; b=SmkWSTl3RV9x+lQoDLvNpp8PJBCip9/dgDO/f/qlcdZenqpbOolgfCnWSfpZc4K/PrjuMH +AHLetG6hNjw+ZNA3TBgJTbFdNyYUz+X1FoJ/1jU1cGk12g0sfLtofK0DqfoRyXoElD2ZR iq73PAl+e+VKssxgi7b/E1T1F9DYjJU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826473; a=rsa-sha256; cv=none; b=j77/8HtDLMA/HTyvEkOrnyCjNm91jKgYlGiEJc8hOEjMUOuxsYW0oqotnlkZnkf5CCO7ij I+ZE2dLviAXSqgTjsn8FL6kShEdHVIksLUB1SC6gzqGY49ToMTZIg7Fd7VAzjyZ20jMcr8 oD48cbQIVGbIovAHaZqi8zZNm+17wdk= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="PQcU5Be/"; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826474; x=1770362474; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6c1d9813y0GRE+6bR9SnHZofzGJx7eVHq/LRDvJCxss=; b=PQcU5Be/ksWH0EG3uY89LyXb+S6sSnvF+LPVypNjjYKu7tEV+JCrbe0W NvjC7w7WTb27M+7byjsB6W+xXmc+l/wK040g7whlPJDb4+06aY2dE6p4z L6ONtOTf8jr5TzTIoBLf6z1YT2l0xxDnLiWd1oXvzo1a5JY0qJfaetlMd gvh9TgSjKQn7OdsFIiGH3SaHIk5bm6uEj4ACRMf4qVAJ9tEWahzqEXHvQ dxxfvNK/EYvgIm3SLoElep4q8HvaX6IiJhDy1n68ePABbGgfoP7mk11N3 2uHRvtKa9ke+Q2C2G0KboRGwZvFMw5vE3xrO8XljqCxgoWQUDXxb81RwT w==; X-CSE-ConnectionGUID: GyzbHGXYRn6xbhhcNG2krw== X-CSE-MsgGUID: YjXZiTdZQ3+9NKq6GdK3Qg== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962690" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962690" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:05 -0800 X-CSE-ConnectionGUID: SKYLn851TW6vkdfKb0Rdkw== X-CSE-MsgGUID: 4cehqHtlRLeQcDOBefapEQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022632" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:05 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 11/16] crypto: iaa - Fix for "deflate_generic_tfm" global being accessed without locks. Date: Wed, 5 Feb 2025 23:20:57 -0800 Message-Id: <20250206072102.29045-12-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: 6h3qjnnsoaz57by1qpbsr8ypajoefqup X-Rspam-User: X-Rspamd-Queue-Id: ABC4B40002 X-Rspamd-Server: rspam03 X-HE-Tag: 1738826473-729684 X-HE-Meta: U2FsdGVkX189rnfbPY/Q6vE67FoJtV632dhDeEWK4pKFdFxjgydw6YanGOskAcDef2KSTCGcpNc6UMWfNvKEw8jir8/UDjHlZ6G5BI32SlbhYm3LPyce2iIDHg3Lb5wJllOhrIbkwIM171MBwP9wfTPg1+8136b7+ojKlQ08HAOXcusP5v+YZ/3oVmirrmC8YIKDNZ51Q1E0nr9ws/24PIMJQEICnpYyPno2p14IxrQHww+TjUc5QD3WPrGC9W8t2i9a53jxYFOwAvB/waDdI4hBmuFCs0/RP4h7PXou9BxBsYIES0lXShRkY8AU3RYeWpkG/xN1QK9vI85D0PRlbzTEFNYi24fk4chcLYAPO0PFJJoi8nANxSIhkohMsqk9cn+LP2qSh0orPkzaisnTNkmyi+ywjQcKb7NhPaatxnnCc229GCTHvmCDH5r4+WtPt5o1mmvOjA5T7gWshl5LOnSh2gnHsS0ESGOIyjbrtLqDWlK9AKYMAn3CN2rk9ruohD6RzRRlXLmLTRfx1NWBBdMEZUY8YyzeE/AUwWT4W80BPu7LHBSDw9MmlcKOhIW/47D503/+H1b9okopTVJEhzGsZyEZxheuYKxdS65S1+JyO+E0+O3HZbqCo5A3+Cy9apVksZj/wiI3oYQMwVpLajFyXZfIALZTsuprb+gxH8CyaJZ7um8MxMgwFWNybyBGiol1A5LBkvbpgapp6gs1he79dfmjAXzM6vqF6UkAbSyEFIS0QDfTcWaN25F7RixDY71vm8Qk6zQRiozOd4W1YeqFu7Yp41r4Oh13BM1grpLB+IYv5xCtjitA+BwF7ewDJRwf/R44UqPgiyZ2EDFafR9tCIQTSMdaJgqGgD9cvG2cWxelGIAkNnYLPNHmri/lva1hycxasgza1ukIL28wbSnR20SEMfG4kCJgVwiHeYZZmt3caHbmJ7FQPb7SCNehmE2A249xmPY8VGgdnEo het/OVrT TXVodeLBAo92jtgMABhY1AD9GgEUq0UzQXIeVxu49fQ93m5dFXr8BdL7JwWYBIzKqBd12zaxTqpbARMA9cbYBn5gMG0h0qV55GgApebY88JL9L3/C0ruz+iJNvF9en9xxsd8qwkuNj0pNcYvwES1+jx7XmafbkC1dZlfjPCa5q1TzQYUYSJ6qIE+OPxaprBEk72PYVYfmihH9zRIOFX5JknH+oiSSf87vT31pl8vcNSKKDtA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The mainline implementation of "deflate_generic_decompress" has a bug in the usage of this global variable: static struct crypto_comp *deflate_generic_tfm; The "deflate_generic_tfm" is allocated at module init time, and freed during module cleanup. Any calls to software decompress, for instance, if descriptor allocation fails or job submission fails, will trigger this bug in the deflate_generic_decompress() procedure. The problem is the unprotected access of "deflate_generic_tfm" in this procedure. While stress testing workloads under high memory pressure, with 1 IAA device and "deflate-iaa" as the compressor, the descriptor allocation times out and the software fallback route is taken. With multiple processes calling: ret = crypto_comp_decompress(deflate_generic_tfm, src, req->slen, dst, &req->dlen); we end up with data corruption, that results in req->dlen being larger than PAGE_SIZE. zswap_decompress() subsequently raises a kernel bug. This bug can manifest under high contention and memory pressure situations with high likelihood. This has been resolved by adding a mutex, which is locked before accessing "deflate_generic_tfm" and unlocked after the crypto_comp call is done. Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 5292d8f7ebd6..6796c783dd16 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -105,6 +105,7 @@ static struct iaa_compression_mode *iaa_compression_modes[IAA_COMP_MODES_MAX]; LIST_HEAD(iaa_devices); DEFINE_MUTEX(iaa_devices_lock); +DEFINE_MUTEX(deflate_generic_tfm_lock); /* If enabled, IAA hw crypto algos are registered, unavailable otherwise */ static bool iaa_crypto_enabled; @@ -1407,6 +1408,9 @@ static int deflate_generic_decompress(struct acomp_req *req) int ret; req->dlen = PAGE_SIZE; + + mutex_lock(&deflate_generic_tfm_lock); + src = kmap_local_page(sg_page(req->src)) + req->src->offset; dst = kmap_local_page(sg_page(req->dst)) + req->dst->offset; @@ -1416,6 +1420,8 @@ static int deflate_generic_decompress(struct acomp_req *req) kunmap_local(src); kunmap_local(dst); + mutex_unlock(&deflate_generic_tfm_lock); + update_total_sw_decomp_calls(); return ret; From patchwork Thu Feb 6 07:20:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC5A1C02196 for ; Thu, 6 Feb 2025 07:21:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98EAC28000C; Thu, 6 Feb 2025 02:21:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 917DE28000B; Thu, 6 Feb 2025 02:21:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71A4328000C; Thu, 6 Feb 2025 02:21:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4B98F280004 for ; Thu, 6 Feb 2025 02:21:18 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0132BC0F4A for ; Thu, 6 Feb 2025 07:21:17 +0000 (UTC) X-FDA: 83088673836.15.3C85AB6 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf26.hostedemail.com (Postfix) with ESMTP id BE5DB140002 for ; Thu, 6 Feb 2025 07:21:14 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jjuVXBHK; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826475; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wJeRHi8PdIL+/6sqlqyoxEfcWSS7xsFU9sSDV6xUOP0=; b=4RWHR+0w2G5+IonDsQFHWR2XskJHqBrglcpsNQ0vt+m0vBlx34ZAioH1w1KmmIbLl/o/Tu U+EcC+ZEhS6NBe66NOWPTw8cG1iO6J3mou5FqQthHcwjA37y5bOjCkzonwyjqMJ5TiczRm 9+7S/H0wYi4NNcS/AC76KVp7QHmO2HM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jjuVXBHK; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826475; a=rsa-sha256; cv=none; b=IjBsrZd04brn2vD4wxnp8Glueh90TCbbm81uwWyL5Fx+tUvCJ+YGpIWzYr9F+EFG2LUAX2 aytlei7eaxtQaoySJS1O7Y9Rb3wSZpridbZJcPqRcIiVJuNMm45LHtYXew23+A5xMqDIAK odZ7Iqn4B4WKSa6arQ4d9EJR/7lC0Fo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826475; x=1770362475; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jrbewW+SO0IZ8ImSpKIkSqygAyFsg0B5SijCQLy64nI=; b=jjuVXBHKYX37py3ChwTYcodTfVKdKsQxH/fRmeayhYy3O43SmVfsckRs ZJlxuNJjGbUZg1mcNP1in20motupZG/cWKrsyh2Za60lBRRSBLELfXMcH rmJirWypjvUg3JB3NsfQhxrXknvrobize0nA6rpUaMOjPBeOz1CQVPqFA 0XcIPw6MueVzAq/s6C5uKKHew74GghIXnc6Nn7aWLbdCxa8hdpCCqnUdj iHcQbOtGltVtKKQyX6kOHDCLeWXspuuCh2HQ/iBuY0C6T2UunCva8gb6A UxEm1xtuInj0T0zNKXKkRiULmJOyzkhKXKd6T/Im0aeobjY2d2p2JylYI w==; X-CSE-ConnectionGUID: Ss2H1e9VRLOUeXaINDFZnQ== X-CSE-MsgGUID: EiEtlBgqSX6Q4iJrRWxn1g== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962705" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962705" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:05 -0800 X-CSE-ConnectionGUID: fxqLBazpQmmqNiuGMIhKtg== X-CSE-MsgGUID: veEW1kXWQMSAIAuvPJ7kCA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022635" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:05 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 12/16] mm: zswap: Allocate pool batching resources if the compressor supports batching. Date: Wed, 5 Feb 2025 23:20:58 -0800 Message-Id: <20250206072102.29045-13-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BE5DB140002 X-Stat-Signature: rocmjicd1zk7rukdkxj7z6mauat5c6da X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738826474-78659 X-HE-Meta: U2FsdGVkX1+GZm2jz2VeVA23nsnVnDfRXKh91C5s9yDXPngBVBIg3kF3R1GRs2STo56E0bFfs50Seoezw1SGGUEGQ+x04yLR1hijXtHYlG/pFvYPcrzf5y4ztR+J+u7f9HbXVxgHHr4pl3KupNBFpwDHfcpGj5Z3yOSWU1VyBSLy9cg0Xb8CbPfRgqcAPiHhHVCosIfpsTLocN1mgb4RM9ySfj7X1zRPa4HFCrb+12jigtXCReNAhZBba5IlOJW1hbe+0MCpYx/f+VH8HYvGe5KCpCTx/tjK8SuCo7UU5RY74GnJ/u1Ofvs1wUroBAJ2q54cLN7zgZ24puKCdutGtwxL4VX0JgL2tnnHzsRUaBiymIDkJepDRhN9kVLZsr3PsXYhIfANG97LpnHACpsRnIEEDNE03Q4nQfjQ4kuH4YXcIoJEUTxZWRmevRTl4hcrkO8G1RxvgJbydct9lkhFjTvn9Ak7p/PNwGNOL2YTi5Nsc2CfIoyWFrALQCdJQ0S4/4zy5rBKrkqINOdu23OUEnmUArXgm7Ys+CTSLQDYwD2a3FdSVIS0zLFR7BeuiRcD6eFbiIXVctrtXevAny/BWb3R52P6Xmaf8mTObBvSY0cfn/o1NQPXN5dZSTH4Gzmr3o/PYEm4+ACDLnjmY9bUGUzay0D4GDma0nC8G1ATa+Peeayqc67Q7E/JEtpwq4A6tJqFsFtXvEABwJe8Cnywzc/unBTwn2wF0wC4GdEic6Wl1CEaEflf9I1IwfUXYE3G2x+WeHO08fVO1e1q8o+6wEXQPXsdJHacL2X+hIRG7p8HQtKbs3TMw9SHutHGkeHfeZVcds5y1Qryr8rAjg4higqvOsD5/3EPYjs6KxoSOQVD4NwOgkYsPsf9xvOgAA1KZh05k2ZYSzSWmpcYp9TG8uYTH1d/OiY1Je1PubzWk1xNORHFLLOaCW5j0P/pAgz1kPGsn4bhKnargcudsFZ eC9klOZz GjGFv1PC0k7O8lpxQ+IZa1jtRdQbFfT5H6B3ifT+5yuaZJrM6yw9V938gTIKxRChofpWLPLCWlAGn+SRf6RLvuIZ7Yxa3GityjZzHGnWPFKZRvB8rLlwfdquvFnoIXv/QSoG4S3w5W19RTG9+WWvNv1CfWG00WGJMAqOidZq3RW5Vvgw9RSFDhQUPnMBvf13KJ/Vh13kLowCtGR/1ltLjsvC/yw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch adds support for the per-CPU acomp_ctx to track multiple compression/decompression requests. The zswap_cpu_comp_prepare() cpu onlining code will check if the compressor supports batching. If so, it will allocate the necessary batching resources. However, zswap does not use more than one request yet. Follow-up patches will actually utilize the multiple acomp_ctx requests/buffers for batch compression/decompression of multiple pages. The newly added ZSWAP_MAX_BATCH_SIZE limits the amount of extra memory used for batching. There is no extra memory usage for compressors that do not support batching. Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 132 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 98 insertions(+), 34 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index a2baceed3bf9..dc7d1ff04b22 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -78,6 +78,16 @@ static bool zswap_pool_reached_full; #define ZSWAP_PARAM_UNSET "" +/* + * For compression batching of large folios: + * Maximum number of acomp compress requests that will be processed + * in a batch, iff the zswap compressor supports batching. + * This limit exists because we preallocate enough requests and buffers + * in the per-cpu acomp_ctx accordingly. Hence, a higher limit means higher + * memory usage. + */ +#define ZSWAP_MAX_BATCH_SIZE 8U + static int zswap_setup(void); /* Enable/disable zswap */ @@ -143,9 +153,10 @@ bool zswap_never_enabled(void) struct crypto_acomp_ctx { struct crypto_acomp *acomp; - struct acomp_req *req; + struct acomp_req **reqs; + u8 **buffers; + unsigned int nr_reqs; struct crypto_wait wait; - u8 *buffer; struct mutex mutex; bool is_sleepable; }; @@ -821,15 +832,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); struct crypto_acomp *acomp = NULL; - struct acomp_req *req = NULL; - u8 *buffer = NULL; - int ret; + unsigned int nr_reqs = 1; + int ret = -ENOMEM; + int i; - buffer = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); - if (!buffer) { - ret = -ENOMEM; - goto fail; - } + acomp_ctx->buffers = NULL; + acomp_ctx->reqs = NULL; + acomp_ctx->nr_reqs = 0; acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu)); if (IS_ERR(acomp)) { @@ -839,12 +848,30 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) goto fail; } - req = acomp_request_alloc(acomp); - if (!req) { - pr_err("could not alloc crypto acomp_request %s\n", - pool->tfm_name); - ret = -ENOMEM; + if (acomp_has_async_batching(acomp)) + nr_reqs = min(ZSWAP_MAX_BATCH_SIZE, crypto_acomp_batch_size(acomp)); + + acomp_ctx->buffers = kcalloc_node(nr_reqs, sizeof(u8 *), GFP_KERNEL, cpu_to_node(cpu)); + if (!acomp_ctx->buffers) + goto fail; + + for (i = 0; i < nr_reqs; ++i) { + acomp_ctx->buffers[i] = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); + if (!acomp_ctx->buffers[i]) + goto fail; + } + + acomp_ctx->reqs = kcalloc_node(nr_reqs, sizeof(struct acomp_req *), GFP_KERNEL, cpu_to_node(cpu)); + if (!acomp_ctx->reqs) goto fail; + + for (i = 0; i < nr_reqs; ++i) { + acomp_ctx->reqs[i] = acomp_request_alloc(acomp); + if (!acomp_ctx->reqs[i]) { + pr_err("could not alloc crypto acomp_request reqs[%d] %s\n", + i, pool->tfm_name); + goto fail; + } } /* @@ -853,6 +880,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) * again resulting in a deadlock. */ mutex_lock(&acomp_ctx->mutex); + + /* + * The crypto_wait is used only in fully synchronous, i.e., with scomp + * or non-poll mode of acomp, hence there is only one "wait" per + * acomp_ctx, with callback set to reqs[0], under the assumption that + * there is at least 1 request per acomp_ctx. + */ crypto_init_wait(&acomp_ctx->wait); /* @@ -860,20 +894,33 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) * crypto_wait_req(); if the backend of acomp is scomp, the callback * won't be called, crypto_wait_req() will return without blocking. */ - acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, + acomp_request_set_callback(acomp_ctx->reqs[0], CRYPTO_TFM_REQ_MAY_BACKLOG, crypto_req_done, &acomp_ctx->wait); - acomp_ctx->buffer = buffer; + acomp_ctx->nr_reqs = nr_reqs; acomp_ctx->acomp = acomp; acomp_ctx->is_sleepable = acomp_is_async(acomp); - acomp_ctx->req = req; mutex_unlock(&acomp_ctx->mutex); return 0; fail: + if (acomp_ctx->buffers) { + for (i = 0; i < nr_reqs; ++i) + kfree(acomp_ctx->buffers[i]); + kfree(acomp_ctx->buffers); + acomp_ctx->buffers = NULL; + } + + if (acomp_ctx->reqs) { + for (i = 0; i < nr_reqs; ++i) + if (!IS_ERR_OR_NULL(acomp_ctx->reqs[i])) + acomp_request_free(acomp_ctx->reqs[i]); + kfree(acomp_ctx->reqs); + acomp_ctx->reqs = NULL; + } + if (acomp) crypto_free_acomp(acomp); - kfree(buffer); return ret; } @@ -883,14 +930,31 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); mutex_lock(&acomp_ctx->mutex); + if (!IS_ERR_OR_NULL(acomp_ctx)) { - if (!IS_ERR_OR_NULL(acomp_ctx->req)) - acomp_request_free(acomp_ctx->req); - acomp_ctx->req = NULL; + int i; + + if (acomp_ctx->reqs) { + for (i = 0; i < acomp_ctx->nr_reqs; ++i) + if (!IS_ERR_OR_NULL(acomp_ctx->reqs[i])) + acomp_request_free(acomp_ctx->reqs[i]); + kfree(acomp_ctx->reqs); + acomp_ctx->reqs = NULL; + } + + if (acomp_ctx->buffers) { + for (i = 0; i < acomp_ctx->nr_reqs; ++i) + kfree(acomp_ctx->buffers[i]); + kfree(acomp_ctx->buffers); + acomp_ctx->buffers = NULL; + } + if (!IS_ERR_OR_NULL(acomp_ctx->acomp)) crypto_free_acomp(acomp_ctx->acomp); - kfree(acomp_ctx->buffer); + + acomp_ctx->nr_reqs = 0; } + mutex_unlock(&acomp_ctx->mutex); return 0; @@ -903,7 +967,7 @@ static struct crypto_acomp_ctx *acomp_ctx_get_cpu_lock(struct zswap_pool *pool) for (;;) { acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); mutex_lock(&acomp_ctx->mutex); - if (likely(acomp_ctx->req)) + if (likely(acomp_ctx->reqs)) return acomp_ctx; /* * It is possible that we were migrated to a different CPU after @@ -935,7 +999,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, u8 *dst; acomp_ctx = acomp_ctx_get_cpu_lock(pool); - dst = acomp_ctx->buffer; + dst = acomp_ctx->buffers[0]; sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); @@ -945,7 +1009,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, * giving the dst buffer with enough length to avoid buffer overflow. */ sg_init_one(&output, dst, PAGE_SIZE * 2); - acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); + acomp_request_set_params(acomp_ctx->reqs[0], &input, &output, PAGE_SIZE, dlen); /* * it maybe looks a little bit silly that we send an asynchronous request, @@ -959,8 +1023,8 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, * but in different threads running on different cpu, we have different * acomp instance, so multiple threads can do (de)compression in parallel. */ - comp_ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; + comp_ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->reqs[0]), &acomp_ctx->wait); + dlen = acomp_ctx->reqs[0]->dlen; if (comp_ret) goto unlock; @@ -1011,19 +1075,19 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) */ if ((acomp_ctx->is_sleepable && !zpool_can_sleep_mapped(zpool)) || !virt_addr_valid(src)) { - memcpy(acomp_ctx->buffer, src, entry->length); - src = acomp_ctx->buffer; + memcpy(acomp_ctx->buffers[0], src, entry->length); + src = acomp_ctx->buffers[0]; zpool_unmap_handle(zpool, entry->handle); } sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); sg_set_folio(&output, folio, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); - BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); - BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); + acomp_request_set_params(acomp_ctx->reqs[0], &input, &output, entry->length, PAGE_SIZE); + BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->reqs[0]), &acomp_ctx->wait)); + BUG_ON(acomp_ctx->reqs[0]->dlen != PAGE_SIZE); - if (src != acomp_ctx->buffer) + if (src != acomp_ctx->buffers[0]) zpool_unmap_handle(zpool, entry->handle); acomp_ctx_put_unlock(acomp_ctx); } From patchwork Thu Feb 6 07:20:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962462 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF8ACC02196 for ; Thu, 6 Feb 2025 07:21:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71D0D28000A; Thu, 6 Feb 2025 02:21:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CE18280004; Thu, 6 Feb 2025 02:21:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51FA028000A; Thu, 6 Feb 2025 02:21:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2D55F280004 for ; Thu, 6 Feb 2025 02:21:17 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DE5B51C88C1 for ; Thu, 6 Feb 2025 07:21:16 +0000 (UTC) X-FDA: 83088673752.16.D7BAC29 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf08.hostedemail.com (Postfix) with ESMTP id D0562160002 for ; Thu, 6 Feb 2025 07:21:14 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=k9UjCEjT; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826475; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kJtkvR6syFxv+hOQ8eJ4pefrzelQlHkLW8z1Vmr+FFg=; b=BIJ0Xjvrm1IYTpqqBTvnvf8RUThEfgIjsPulmAGeYE/+z6C3aSqP2pZNQUTaDMND2NLe+1 DPPvqDraykc0cyX48R4EDEM/NPrzlYrMCwskEWe7Lk/eZ7NlVnVcmlGpNzmWLJfqSxxLop a9/u6drAi5/rYkzgq4Z7HJihEOk5sQM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=k9UjCEjT; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826475; a=rsa-sha256; cv=none; b=NW51PvIht0UvhZ0yqVIFRWp+zzCWjc0Oaq4Lk/y+6DpTIkY+lRmbBtT3Mh2Q+QgTe6wz99 6CyY3aqFbzvJtT8EkwmyoWq1w6bNe73BtkuVy0QD6Ntfbp2BDa1jE/fiN1OSWNNCpuaBq/ pk3e40jr2nzECg7iN8pBZ4K8ZkHiWHc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826475; x=1770362475; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MDL8+i/ID50qONhYxICYmIy+BnPTX+KVFghJNhMgSNU=; b=k9UjCEjT0DDDGUCgi96yIqzrdqao8RiySDqwjMkuoIBuZsElD11nxiNv PQ5Krxu3//z5MjcmmnQmxY393/XzSBCtDgq4HDUC9hjU9zC5HSZHOhUa2 zXr6gs8DyNgaL2OlIYhZP9L5I1bLSNGBQNSulmqFYZSfKbAyKX7UaK0h2 3Y/qZ1c3pq82Kko9ImFP+qxDrSwOpcqEnEgrf15cGKGDaf/ZzHE20mXOP ucZZpKMqgL2nbJYX3P879FzkS5s5anwb4Bi9OkCaQcZrI0XRZpxruOv8U 22hrfV+sVicIm98Efzav51bVlGyxkOZY3ZYb3GkThWT7Vv1abE5Btkbjb g==; X-CSE-ConnectionGUID: cazEGVrOT/axDSFKnyrlqA== X-CSE-MsgGUID: FVpNsXVyQR6I/BfvwxyPPQ== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962716" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962716" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:05 -0800 X-CSE-ConnectionGUID: AgAlP4i4QmilcDGyzbBDYg== X-CSE-MsgGUID: w4+5RT7XT0WkI456g626+A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022642" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:05 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 13/16] mm: zswap: Restructure & simplify zswap_store() to make it amenable for batching. Date: Wed, 5 Feb 2025 23:20:59 -0800 Message-Id: <20250206072102.29045-14-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D0562160002 X-Stat-Signature: 8usenn89dxderzpurobtqqskqnxrf1af X-Rspam-User: X-HE-Tag: 1738826474-998044 X-HE-Meta: U2FsdGVkX19kA8XNM6iO4B1kyJzD1AGdfOfU3n+ooGvzGWWnZYJ/m0sv+yFqj2SlKh2AaeWvn+o2cmWLvK9FLEdx89PCn9wPYT8gasKTNz2qtqMcaiTAYJ03V406fjoNvDFrb2NqFmvKXEPCXmSp2vpHAHdPg5GhZeR/tnKmVhm0JZIZAP0Cx1jr6v6VeWFdA7/OVkJb159Esrih4ksJxPi0Swx4t/sEP25ySnQyuQgeeWsWPfPySibVmHR3QQfGLEDGBSYBNQCxFz/sEjuA1tTUvvr2iuWJRB9+uXkzYvZR5Qsa+Z2cYY0NEoY7S+HgSv01E9P6kN/9lNt1P5Vs4fd0mqtllfOi4dwMWiP11yP7WVxNem80EUlqWAHinWqQaj8cSANLQSPVxuDfGKc3xgIJn84n0l6jOoUJPvandCjwJKTYnX0BHB7JjVxxcc3CuYigugn0Po6J7V+P/6kzJuBbl2G57/J3nHjsU5BtBwNNjg175o10W4HBoP4+aKewqVbarIo94BMLwxi5lKDVbsZehPAxw0ih/+KNJKcFhjkvYAGhBzF3iaS29V+j7RFn6sunQ7l7UcQRLHj5cWjtiFnhtXl/R+aJ69kMMY1xy7/dKPXCL8YtmzlH6LgJyjE7P45Mkd3RDaxQXY4ArjoV6PRzws+ZIQdYX57X5f7ZUNeHc49EhrEJIkq/rTtY2S4zWA8bxjFJ0TRJU0uJcY1ronPtl6ENe8TBtxjNKzJD6pG7q1u7h6sIHB3aDkmnIX5nNbizsd/HRPhQVdL3b4nF5aNhiR3FKLieD+1RofZf1S/FNCx8Nmxx0XxvPH6eQXoopAWTIJAxqBryFBawpGwoQimwEqOpy/2CdEjWd6XGWz2maEDqjyw3MchTCPqTxJCgII9Tmv9wn95tYzlTGcSvIidCQporaxsViNZXJzfDCd3iCX6JGaDpy98yHSd3EtjTFbs9h7gJ1LeUwOM09bW a2eLyVj5 DX3wAPQ+L1Axw9TtjgLW998sdLx8uQXByjMSaen6C2zC3rl5CuKfuloFLhZgzm1uwCK37FWgNqKZIFcmBHpaCEOvuDOHInXHYMWuoWlzDaLKxBf4WmsHc1lFArFZLvvAoED01rWnEQWpktzeeeWlmseeZqpLvzHrVg7UTFUk1Tr61MSDH0fxxWHYsblf1Xqj1edA/BzEy9TQ9+gkN5Upm1qMC3/xm6D1dONJEZAyw7vZPZMw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch introduces zswap_store_folio() that implements all the computes done earlier in zswap_store_page() for a single-page, for all the pages in a folio. This allows us to move the loop over the folio's pages from zswap_store() to zswap_store_folio(). zswap_store_folio() starts by allocating all zswap entries required to store the folio. Next, it calls zswap_compress() for all pages in the folio. Finally, it adds the entries to the xarray and LRU, charges zswap memory and increments zswap stats. The error handling and cleanup required for all failure scenarios that can occur while storing a folio in zswap are now consolidated to a "store_folio_failed" label in zswap_store_folio(). These changes facilitate developing support for compress batching in zswap_store_folio(). Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 164 ++++++++++++++++++++++++++++++++--------------------- 1 file changed, 98 insertions(+), 66 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index dc7d1ff04b22..af682bf0f690 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1509,81 +1509,116 @@ static void shrink_worker(struct work_struct *w) * main API **********************************/ -static bool zswap_store_page(struct page *page, - struct obj_cgroup *objcg, - struct zswap_pool *pool) +/* + * Store all pages in a folio. + * + * The error handling from all failure points is consolidated to the + * "store_folio_failed" label, based on the initialization of the zswap entries' + * handles to ERR_PTR(-EINVAL) at allocation time, and the fact that the + * entry's handle is subsequently modified only upon a successful zpool_malloc() + * after the page is compressed. + */ +static bool zswap_store_folio(struct folio *folio, + struct obj_cgroup *objcg, + struct zswap_pool *pool) { - swp_entry_t page_swpentry = page_swap_entry(page); - struct zswap_entry *entry, *old; + long index, from_index = 0, nr_pages = folio_nr_pages(folio); + struct zswap_entry **entries = NULL; + int node_id = folio_nid(folio); - /* allocate entry */ - entry = zswap_entry_cache_alloc(GFP_KERNEL, page_to_nid(page)); - if (!entry) { - zswap_reject_kmemcache_fail++; + entries = kmalloc(nr_pages * sizeof(*entries), GFP_KERNEL); + if (!entries) return false; - } - if (!zswap_compress(page, entry, pool)) - goto compress_failed; + for (index = 0; index < nr_pages; ++index) { + entries[index] = zswap_entry_cache_alloc(GFP_KERNEL, node_id); - old = xa_store(swap_zswap_tree(page_swpentry), - swp_offset(page_swpentry), - entry, GFP_KERNEL); - if (xa_is_err(old)) { - int err = xa_err(old); + if (!entries[index]) { + zswap_reject_kmemcache_fail++; + nr_pages = index; + goto store_folio_failed; + } - WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); - zswap_reject_alloc_fail++; - goto store_failed; + entries[index]->handle = (unsigned long)ERR_PTR(-EINVAL); } - /* - * We may have had an existing entry that became stale when - * the folio was redirtied and now the new version is being - * swapped out. Get rid of the old. - */ - if (old) - zswap_entry_free(old); + for (index = 0; index < nr_pages; ++index) { + struct page *page = folio_page(folio, index); - /* - * The entry is successfully compressed and stored in the tree, there is - * no further possibility of failure. Grab refs to the pool and objcg, - * charge zswap memory, and increment zswap_stored_pages. - * The opposite actions will be performed by zswap_entry_free() - * when the entry is removed from the tree. - */ - zswap_pool_get(pool); - if (objcg) { - obj_cgroup_get(objcg); - obj_cgroup_charge_zswap(objcg, entry->length); + if (!zswap_compress(page, entries[index], pool)) + goto store_folio_failed; } - atomic_long_inc(&zswap_stored_pages); - /* - * We finish initializing the entry while it's already in xarray. - * This is safe because: - * - * 1. Concurrent stores and invalidations are excluded by folio lock. - * - * 2. Writeback is excluded by the entry not being on the LRU yet. - * The publishing order matters to prevent writeback from seeing - * an incoherent entry. - */ - entry->pool = pool; - entry->swpentry = page_swpentry; - entry->objcg = objcg; - entry->referenced = true; - if (entry->length) { - INIT_LIST_HEAD(&entry->lru); - zswap_lru_add(&zswap_list_lru, entry); + for (index = 0; index < nr_pages; ++index) { + swp_entry_t page_swpentry = page_swap_entry(folio_page(folio, index)); + struct zswap_entry *old, *entry = entries[index]; + + old = xa_store(swap_zswap_tree(page_swpentry), + swp_offset(page_swpentry), + entry, GFP_KERNEL); + if (xa_is_err(old)) { + int err = xa_err(old); + + WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); + zswap_reject_alloc_fail++; + from_index = index; + goto store_folio_failed; + } + + /* + * We may have had an existing entry that became stale when + * the folio was redirtied and now the new version is being + * swapped out. Get rid of the old. + */ + if (old) + zswap_entry_free(old); + + /* + * The entry is successfully compressed and stored in the tree, there is + * no further possibility of failure. Grab refs to the pool and objcg, + * charge zswap memory, and increment zswap_stored_pages. + * The opposite actions will be performed by zswap_entry_free() + * when the entry is removed from the tree. + */ + zswap_pool_get(pool); + if (objcg) { + obj_cgroup_get(objcg); + obj_cgroup_charge_zswap(objcg, entry->length); + } + atomic_long_inc(&zswap_stored_pages); + + /* + * We finish initializing the entry while it's already in xarray. + * This is safe because: + * + * 1. Concurrent stores and invalidations are excluded by folio lock. + * + * 2. Writeback is excluded by the entry not being on the LRU yet. + * The publishing order matters to prevent writeback from seeing + * an incoherent entry. + */ + entry->pool = pool; + entry->swpentry = page_swpentry; + entry->objcg = objcg; + entry->referenced = true; + if (entry->length) { + INIT_LIST_HEAD(&entry->lru); + zswap_lru_add(&zswap_list_lru, entry); + } } + kfree(entries); return true; -store_failed: - zpool_free(pool->zpool, entry->handle); -compress_failed: - zswap_entry_cache_free(entry); +store_folio_failed: + for (index = from_index; index < nr_pages; ++index) { + if (!IS_ERR_VALUE(entries[index]->handle)) + zpool_free(pool->zpool, entries[index]->handle); + + zswap_entry_cache_free(entries[index]); + } + + kfree(entries); return false; } @@ -1595,7 +1630,6 @@ bool zswap_store(struct folio *folio) struct mem_cgroup *memcg = NULL; struct zswap_pool *pool; bool ret = false; - long index; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1629,12 +1663,9 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } - for (index = 0; index < nr_pages; ++index) { - struct page *page = folio_page(folio, index); + if (!zswap_store_folio(folio, objcg, pool)) + goto put_pool; - if (!zswap_store_page(page, objcg, pool)) - goto put_pool; - } if (objcg) count_objcg_events(objcg, ZSWPOUT, nr_pages); @@ -1661,6 +1692,7 @@ bool zswap_store(struct folio *folio) pgoff_t offset = swp_offset(swp); struct zswap_entry *entry; struct xarray *tree; + long index; for (index = 0; index < nr_pages; ++index) { tree = swap_zswap_tree(swp_entry(type, offset + index)); From patchwork Thu Feb 6 07:21:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962464 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FC13C0219B for ; Thu, 6 Feb 2025 07:21:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C79C1280004; Thu, 6 Feb 2025 02:21:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B3A2C28000D; Thu, 6 Feb 2025 02:21:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 877AB280004; Thu, 6 Feb 2025 02:21:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4E57428000B for ; Thu, 6 Feb 2025 02:21:18 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0DD4AB1CB3 for ; Thu, 6 Feb 2025 07:21:18 +0000 (UTC) X-FDA: 83088673836.19.89F22AE Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf07.hostedemail.com (Postfix) with ESMTP id DFECD40002 for ; Thu, 6 Feb 2025 07:21:15 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=EiTLpnNK; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826476; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C2fTyg6uTdE2d1L0g/1iCUCcd19/Vjnv1DkKko59n3k=; b=q0Oh7Q21ffuxXiPjLkneVZPLKijTGCdN96UL/f68jNeyUcsJ+GpUuVnW2k1YUj4ox5mMrd LYhpp8nOldjgmVmhHfNjCa20sclh5GTAtl95e+vlhHiGo/y3X0gpRCost6kHpcJh65Buc3 ke0LkyhwXyq+aDuLWgAznZIJZsLBjjA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826476; a=rsa-sha256; cv=none; b=PmAnoq7w/QzBg2OfttdJmzf5NtLAqsxbU8Ux2DOTuazx83MO1ZvfG/4owlM2/2Rrch8ApO W4BFEZBDY708h3FwzytMWSgWaZw4MTKrr239ArEbis/nnnc6Bya9rwlPs2Q8ycul62vecO S9CU/taLB+9j2U57l084c5hxlh0gTJI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=EiTLpnNK; spf=pass (imf07.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826476; x=1770362476; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=iqZv8rJXdWI52EmscIRZweGSd8xTWCD0Rr/EovMHLuY=; b=EiTLpnNKxlQMfp3PegIjWW6IcQyI6pCR3EZ1NgFo9bOu6A074d2ZwMqU DZedXZgQ1bvVU8C0Y07YS1AoP/NHWJ80YhCkF8rmsIiA7ZyqRYqfmEHaC 0i4fILL5l+id8iHlOJcyFNzri/UwOzedAxKltp2RjG17WWqODKje1N0fk cQLkQmVG8lg98UuI/rSO4RxWVMDFsj9wJcTwmUjiOC46x0N6UFfUWRsnT wxZzkBJvjEMuhsu0GTYZc1gY3dulIPR32hY58QTBDjV+3V9j0f1sVjhwU nWcY0cLU5OBlzFmgmUBTSGGBPTGCAXg+Z0F6q34PP3coXrWPyNpUBa286 Q==; X-CSE-ConnectionGUID: MaoiOuphTimg347Wi/a34g== X-CSE-MsgGUID: uwT6zM37SN2Oiya9spT3Ug== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962728" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962728" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:06 -0800 X-CSE-ConnectionGUID: ILym3snHR9yvBXAi4N76GQ== X-CSE-MsgGUID: LTcU3nkoQeadkuMSgJpNTQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022646" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:06 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 14/16] mm: zswap: Introduce zswap_compress_folio() to compress all pages in a folio. Date: Wed, 5 Feb 2025 23:21:00 -0800 Message-Id: <20250206072102.29045-15-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: gmp18m9jpoc96up61gmnj6y3xz6z8o43 X-Rspam-User: X-Rspamd-Queue-Id: DFECD40002 X-Rspamd-Server: rspam03 X-HE-Tag: 1738826475-638331 X-HE-Meta: U2FsdGVkX18bPt1dYySNiJo7alWGlpvhU+jOoAnyXDDw0aqfdAoMbpUCLRbQ0C8HTmH4hoxiAS3kUf9yn0w6aMZzUyTvj3Kqe5aXn/M8LegtTwIGslt7iPWg1aYyXD1GMAfMQ7M1fv+b5ABfPxSbRQUDChOZ2lk6F3xkYNRQnjQhgt0zHPJRheNF1dZ1lHbsqon6z+DNgWUi5rFmpj49bdQEocPTlklvU7aYiiKA27Lke1tKNpRVzwr+ieenmZdEc3xt7BzS6/4E73bFS3GGcsrai0ng1+HpH7nVyZ+g/0wDl2TyeXX7bygA/0/Vh228XgLPtjaBkNr9eEESsMaoEgqFA/MI5sjEOlNk/3i+I900HTh0T7nJ5VGY1cqr6EpvRVmJWHIT/2Bmt1uLzg049Y/XxA7xiD/ZyuoT8E7i9QWUmfEWBB34ABeW9ez78e/GNiSZCy1TyoDMXpojvvQ2l1fQ6gh3LqY2TULzsT2mDpmMMPg+EdDLMBjQUZ40DyMUN/b/JW0jhj3GsvGTvuuHPHJ9hyCTz8BNN/C2LuzKF0CetI0BfpVzfkUEBavqIr9T3i3h59vOP0aFCiCmFDBvtf1Yog7lOm8tDO/hUzO63PLITbkkiuHFDYkUd+WHeSatPR7zs5pxe/s2axf4bPY+v4ttjfuKFLCtY2em0O3o785uSzJW5wdYT04w0uo07pBW8J4dX2NtQwuCC5Mb13m4mZMuiLO91baNZubeckvbWudue7QdickWImXzem+qruI3Vut402j4VYabGpjQHgex6nRdaQ8we5Yf/+sFdPvhcMkuvh8jlLhA/i+BP+brbAKuxVieDqaE6b3/KeU13ZbCVidCO4RdqSd+mkBrCEivQVpNtNY+TR6tC/xsrLuJVt1J1VozroBc7AY4rFXuh3x4HkZCAklsAMwsyfSm58AeIfEmxmYsuuMpb7nzjaUJLEMH/aSs2dw62XJS5S9UJUW ZFHnr/B3 HALIrEmhyRqNZLyhjQjIFOg+o7V8oRcxlbo6WHoC3zijugJhgTdBA7VaXLcxFcJ7uhhldBSClyWqH21ay4HMqoxnzscFm8sPe6CKnVi+OzscXqEKGJsekN3/gUn8Z+dc0WSX/7eYI27vX4CleeBvPsO3rojhebSauE0S7Y5P20V3MFXB9OWliaatuFRB8VZElIpso77DDO/YgLRoGnnggXNf/otYxmcLFyA7E7V0A9TtODGg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch encapsulates away the computes for compressing the pages in a folio and storing the compressed memory in zpool, into a distinct zswap_compress_folio() procedure. For now, zswap_compress_folio() simply calls zswap_compress() for each page in the folio it is called with. This facilitates adding compress batching in subsequent patches. Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index af682bf0f690..6563d12e907b 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1509,6 +1509,22 @@ static void shrink_worker(struct work_struct *w) * main API **********************************/ +static bool zswap_compress_folio(struct folio *folio, + struct zswap_entry *entries[], + struct zswap_pool *pool) +{ + long index, nr_pages = folio_nr_pages(folio); + + for (index = 0; index < nr_pages; ++index) { + struct page *page = folio_page(folio, index); + + if (!zswap_compress(page, entries[index], pool)) + return false; + } + + return true; +} + /* * Store all pages in a folio. * @@ -1542,12 +1558,8 @@ static bool zswap_store_folio(struct folio *folio, entries[index]->handle = (unsigned long)ERR_PTR(-EINVAL); } - for (index = 0; index < nr_pages; ++index) { - struct page *page = folio_page(folio, index); - - if (!zswap_compress(page, entries[index], pool)) - goto store_folio_failed; - } + if (!zswap_compress_folio(folio, entries, pool)) + goto store_folio_failed; for (index = 0; index < nr_pages; ++index) { swp_entry_t page_swpentry = page_swap_entry(folio_page(folio, index)); From patchwork Thu Feb 6 07:21:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A07FC02198 for ; Thu, 6 Feb 2025 07:21:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B3C1428000D; Thu, 6 Feb 2025 02:21:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AED1228000B; Thu, 6 Feb 2025 02:21:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F24B28000D; Thu, 6 Feb 2025 02:21:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 67A8428000B for ; Thu, 6 Feb 2025 02:21:19 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 240C8140EB4 for ; Thu, 6 Feb 2025 07:21:19 +0000 (UTC) X-FDA: 83088673878.30.91D483E Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf08.hostedemail.com (Postfix) with ESMTP id 0FE36160002 for ; Thu, 6 Feb 2025 07:21:16 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="SMp/7Pap"; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826477; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IDldilDkUG1d2Q+acSDeuZ0ZeLEQFWCVTmN2am9GxbM=; b=Ut0jhmHC/eLH0WUpxpuUdN0ZWD8+2exeI7XQVYH53ekhPGoYIcZYc+CCpvXbinUIdjNkkh tgYxryjZ/ztgK2oWLSzJO4y/KrGeOb00Mt1WT9gps+sqmqHubnRRfTc79Qt0gHBBpBpl2T UowAMeN6D2Psd9wIRVaMAuox7mKEUdw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="SMp/7Pap"; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826477; a=rsa-sha256; cv=none; b=oxlhbtxdDdnTUNkdDX7SDQYZ0kboujg5XWQj+piBg7nNCMVcaFAJQ18ZO6vV2O7J4mQMJr M8hJDjp2pDb5XIIRnNn/R9k3GW1gIGqZpv+wKjbekfRFawuL72zlED5AbQi7toP0RMEYSs Pkm7XuMF26Hj7HtWaZ1eJRD6Ttn3vas= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826477; x=1770362477; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hGMXf+St2ZchUBtwIHiThEldg2uqLPjq0XggSNACRI4=; b=SMp/7PapntHdA4oEXZdkwjsWNwLynY5jHflxPfS6NcYh71Dec1MG7A1c 51UQmsguVn+85aIw0iybDutEDDJwYG/+qpuLqmwMsth/jBtl/qgkANeaP sxrmPBawO/K+lmTgDQSVTE20/nZCINCvhUr4fPEBNvjYE4TAduwNstTQG eO7jZe1gESCqnXP5tUagIDJf9T9OjcDYFsSSDzN3Mo2vacZ8rHqn3PjAJ ncGJUBTQaidt/MbnRviapb34nNGccxe/hRTLB269K0JMiXvevThCr90z4 bYyi/gVAB1me5bquLMHmjIEOqfVO3Dd8/Ad5P+WE+37KgyGcuB9ZpM0dt g==; X-CSE-ConnectionGUID: aHrGr6M5S3OrnBE4NlVijQ== X-CSE-MsgGUID: 2FeaU/9US+KSrQBhvDpnQA== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962741" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962741" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:06 -0800 X-CSE-ConnectionGUID: T5sTzomPTre7jUusWIkprA== X-CSE-MsgGUID: 3PROnXzOSNuq6V4KrEWVwQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022649" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:06 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 15/16] mm: zswap: Compress batching with Intel IAA in zswap_store() of large folios. Date: Wed, 5 Feb 2025 23:21:01 -0800 Message-Id: <20250206072102.29045-16-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 0FE36160002 X-Stat-Signature: doxo3g3c9izt639ajqsafpzbhz8rpykz X-Rspam-User: X-HE-Tag: 1738826476-740714 X-HE-Meta: U2FsdGVkX19ngMxCYXlbh7HLLS6hTf3YPwCenadOusz6ZjJC0aKsuZ7nKyOi6MnL0Ow+u3K/CHdI6jak0IMWvwHUhWO/3383rL4+iz7uJ6XbPJsUOWZzBy7/am8sSWH6EXWh8lxdooizzahRwytVDY7kDpsla+WePFYJTrhQeGMq1kBdGbBO/cIBzZQy8+BuHcANLcxDQWf5zBrrM5WU6Ivw35a//x9eSi5pd7kv7kbWvqDjv4GX3bhY6VThwUxrYYU+Sv3pqn8cQ0tDWoif4UV0ZhbW1TwBy1q1V6SXpU56zrnSx369rEfnIbjnuD98yFPuadic9IBcmz7orGg6Dc9js9Wt/FWDu7TKVOW4nVUx0lC7MKMUqH2NhZga0hU48O7IDQHPrVYanA/DtIEz5B1ypqm9kjMSRFrvyeWE3Ri4AL+DBdmqpTrwrKtoQI0PjiDEOxB0uFzS3r2+zPErUg+tKmJK6g/BHW01MeHTkJZptjvNjxiyVIbNAoCuhjp0TL3uB3gsvd587J+I5UhSWT8K798atnb7nNYRccnPvJDglOMdMzeXyHnvJ+9DFVfkviMfisqi4mVtmwohDzUO8XHOGFj3MPbipPBDrq5VhVu+QB/fKix8ROLQ+ul9yNd6FkinOfZ4muTwAIGCH+jYHXQYC3vIfE78rZQLSdcjN7mWS2XCfqSXNz+1pa/1IeylhNut9/m2lBgLBIvlVCV34uUzbFv33sl7ydM+ZCkjldo2pqXF6HZH2rLUzYq7ACQN5brD6JE3PQigzsDCojkeRg9eV8Zmks9edqbfkAKQpQAxCMDPHLKcn7Ne8XsJ5qQOXGT6rrms37HeiGVzFFlzg1ANRROiYgNKxMrp3mrxhIwebq5YisQ0Ph01wJ4cyxE0jCqEbLC2/6wq5tzMUZONeNd4P9CWAR/o3heqyt8l18Zorm84Jav8FMdRSqAgC278YrHlEdPNvHqpCvhSbpp sTlCnH8u jll7OYA5XMap4z5Sf7bSMQa3GnWe4Jn0ALzpQR1GnE9Io9UvnBeEcL3dqDuODcG7EGVPGLCMOk5X6CP2AOPt1S1dzdaEw2jxWkeGqaGSMwsARUKR1a4PuIsSHkCoJ8HLfSz6emsAUdGsMG00rhHUa81T2ZWm7R4zlVVw2QAYJ0fjx/p5Nh3ZXYnl5YYdFa5yGl3eM3EvW7gH8FTB+dEIQTlCcxRBNf6aAYWFa X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: zswap_compress_folio() is modified to detect if the pool's acomp_ctx has more than one "nr_reqs", which will be the case if the cpu onlining code has allocated multiple batching resources in the acomp_ctx. If so, it means compress batching can be used with a batch-size of "acomp_ctx->nr_reqs". If compress batching can be used, zswap_compress_folio() will invoke the newly added zswap_batch_compress() procedure to compress and store the folio in batches of "acomp_ctx->nr_reqs" pages. With Intel IAA, the iaa_crypto driver will compress each batch of pages in parallel in hardware. Hence, zswap_batch_compress() does the same computes for a batch, as zswap_compress() does for a page; and returns true if the batch was successfully compressed/stored, and false otherwise. If the pool does not support compress batching, or the folio has only one page, zswap_compress_folio() calls zswap_compress() for each individual page in the folio, as before. Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 122 +++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 113 insertions(+), 9 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 6563d12e907b..f1cba77eda62 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -985,10 +985,11 @@ static void acomp_ctx_put_unlock(struct crypto_acomp_ctx *acomp_ctx) mutex_unlock(&acomp_ctx->mutex); } +/* The per-cpu @acomp_ctx mutex should be locked/unlocked in the caller. */ static bool zswap_compress(struct page *page, struct zswap_entry *entry, - struct zswap_pool *pool) + struct zswap_pool *pool, + struct crypto_acomp_ctx *acomp_ctx) { - struct crypto_acomp_ctx *acomp_ctx; struct scatterlist input, output; int comp_ret = 0, alloc_ret = 0; unsigned int dlen = PAGE_SIZE; @@ -998,7 +999,6 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, gfp_t gfp; u8 *dst; - acomp_ctx = acomp_ctx_get_cpu_lock(pool); dst = acomp_ctx->buffers[0]; sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); @@ -1051,7 +1051,6 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, else if (alloc_ret) zswap_reject_alloc_fail++; - acomp_ctx_put_unlock(acomp_ctx); return comp_ret == 0 && alloc_ret == 0; } @@ -1509,20 +1508,125 @@ static void shrink_worker(struct work_struct *w) * main API **********************************/ +/* The per-cpu @acomp_ctx mutex should be locked/unlocked in the caller. */ +static bool zswap_batch_compress(struct folio *folio, + long index, + unsigned int batch_size, + struct zswap_entry *entries[], + struct zswap_pool *pool, + struct crypto_acomp_ctx *acomp_ctx) +{ + int comp_errors[ZSWAP_MAX_BATCH_SIZE] = { 0 }; + unsigned int dlens[ZSWAP_MAX_BATCH_SIZE]; + struct page *pages[ZSWAP_MAX_BATCH_SIZE]; + unsigned int i, nr_batch_pages; + bool ret = true; + + nr_batch_pages = min((unsigned int)(folio_nr_pages(folio) - index), batch_size); + + for (i = 0; i < nr_batch_pages; ++i) { + pages[i] = folio_page(folio, index + i); + dlens[i] = PAGE_SIZE; + } + + /* + * Batch compress @nr_batch_pages. If IAA is the compressor, the + * hardware will compress @nr_batch_pages in parallel. + */ + ret = crypto_acomp_batch_compress( + acomp_ctx->reqs, + NULL, + pages, + acomp_ctx->buffers, + dlens, + comp_errors, + nr_batch_pages); + + if (ret) { + /* + * All batch pages were successfully compressed. + * Store the pages in zpool. + */ + struct zpool *zpool = pool->zpool; + gfp_t gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; + + if (zpool_malloc_support_movable(zpool)) + gfp |= __GFP_HIGHMEM | __GFP_MOVABLE; + + for (i = 0; i < nr_batch_pages; ++i) { + unsigned long handle; + char *buf; + int err; + + err = zpool_malloc(zpool, dlens[i], gfp, &handle); + + if (err) { + if (err == -ENOSPC) + zswap_reject_compress_poor++; + else + zswap_reject_alloc_fail++; + + ret = false; + break; + } + + buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); + memcpy(buf, acomp_ctx->buffers[i], dlens[i]); + zpool_unmap_handle(zpool, handle); + + entries[i]->handle = handle; + entries[i]->length = dlens[i]; + } + } else { + /* Some batch pages had compression errors. */ + for (i = 0; i < nr_batch_pages; ++i) { + if (comp_errors[i]) { + if (comp_errors[i] == -ENOSPC) + zswap_reject_compress_poor++; + else + zswap_reject_compress_fail++; + } + } + } + + return ret; +} + static bool zswap_compress_folio(struct folio *folio, struct zswap_entry *entries[], struct zswap_pool *pool) { long index, nr_pages = folio_nr_pages(folio); + struct crypto_acomp_ctx *acomp_ctx; + unsigned int batch_size; + bool ret = true; - for (index = 0; index < nr_pages; ++index) { - struct page *page = folio_page(folio, index); + acomp_ctx = acomp_ctx_get_cpu_lock(pool); + batch_size = acomp_ctx->nr_reqs; + + if ((batch_size > 1) && (nr_pages > 1)) { + for (index = 0; index < nr_pages; index += batch_size) { + + if (!zswap_batch_compress(folio, index, batch_size, + &entries[index], pool, acomp_ctx)) { + ret = false; + goto unlock_acomp_ctx; + } + } + } else { + for (index = 0; index < nr_pages; ++index) { + struct page *page = folio_page(folio, index); - if (!zswap_compress(page, entries[index], pool)) - return false; + if (!zswap_compress(page, entries[index], pool, acomp_ctx)) { + ret = false; + goto unlock_acomp_ctx; + } + } } - return true; +unlock_acomp_ctx: + acomp_ctx_put_unlock(acomp_ctx); + return ret; } /* From patchwork Thu Feb 6 07:21:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962466 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90722C02198 for ; Thu, 6 Feb 2025 07:21:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3002E28000B; Thu, 6 Feb 2025 02:21:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 289C428000E; Thu, 6 Feb 2025 02:21:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE49328000B; Thu, 6 Feb 2025 02:21:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B9BC228000E for ; Thu, 6 Feb 2025 02:21:19 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 838AA120FA1 for ; Thu, 6 Feb 2025 07:21:19 +0000 (UTC) X-FDA: 83088673878.20.3D01662 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf26.hostedemail.com (Postfix) with ESMTP id 23C9F140004 for ; Thu, 6 Feb 2025 07:21:16 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=O5eYAZik; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826477; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8EU1cEulSoUeRMsQ5j6EX05ZxWv12/PHivX9qxqjyb0=; b=dHZUVqV5labTW8B9aCb24RPP7I7R+gIIB8bq1IoHiVjccab6MhEj4wmArgAGRe1G8+b5P7 Y/dXzXJoSLyftEgYiUB3tKZ8WRfoEk5mlnh2k5zXvssosc0ua6r5w5GZpVlyAy1o0dAobg M3MDckXvAUJod5n93wZhqoaF4CeS2FM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=O5eYAZik; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826477; a=rsa-sha256; cv=none; b=0SLoaVtWvrV9rbSlCDkvtHNyT/FAhQqNMJfpjwg74VBwLMq+VF8/UajtCw6OGfRL+a59Bo 3pc0ACr58b/Y0dphMgWdV51wgnAbke8t0tkxol2yj7f81RbX44eYfecNbM/nmIv307bBIc jElpknM66W2QmpEpT2iW/7ygAH1ZC6g= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826477; x=1770362477; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2J9gtcIwHxH1MpbEJssRtTtbQQgA3xu9INBQ54csBJY=; b=O5eYAZikCExUuQIZ7OeVyni78s6TnfqRkbYcrSgtddlNE87g6pfskQUa vLLfxgj/CZPBmnwhKP96S160d/DJ+3KHlWBBjHn8MFhJ/D3AvV+4okfHs 96aFEAtvHBBlT6/uDeBDIRc1Ww6m+2nyz3L06uJ36FUBYvB+HiQH+yoX5 WamXgtHTDHpQj2Tcoq5XdA1n1/HEeuaphoT3+pyUMcGXduPVdCYO98VA1 4Oc6wcsTbo4RHB7xmmLDOP1fkvi0+CHonjtlB5zUynIK3R2HNbee7O4yS 5ixWbIpyZuMoFwM5vAo9ko7SBt9cw4GcmE7zjsyH76iybLXuVEitMGGEu g==; X-CSE-ConnectionGUID: T30kZslXQPmH9/3wxEo0OA== X-CSE-MsgGUID: tj5GZ2EpSjOb20CvEup9nw== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962755" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962755" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:06 -0800 X-CSE-ConnectionGUID: IlfEp/8QS12Kl4i9twxvoA== X-CSE-MsgGUID: bYZGwRz4Rc256Qiw4cxcsg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022652" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:06 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 16/16] mm: zswap: Fix for zstd performance regression with 2M folios. Date: Wed, 5 Feb 2025 23:21:02 -0800 Message-Id: <20250206072102.29045-17-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 23C9F140004 X-Stat-Signature: bwtnrp5hyczb9bzqfndww5er4o8k46xy X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738826476-549044 X-HE-Meta: U2FsdGVkX18ezTl1TS0yPIJXAPve6h8s3AUXdTJAmKyZQCzeea1eqTiKgv1AkhXUyoFWQF8ZXt6+VbklYlLLm4h+f9rnyRVwAEBcgOXTqq3z58P0n1Pa54pp2eEJpg6NL9ky6lYFHaCF95/qeNV81HNtDdWOGRneWfN5vFcCXy5MiWIEZ31/H5dQABVb8t6dZ7NnJt9v8Cze/FT9ZCCo2fd2NqiZeAGFNeXkddLtVD8ps0QJ00bVtWwXohDVkwnfNY9aSnn7GlbCOaqz2+5xpgsgwnbpuG0ushkBI8fFGWqXrD6TSfhd+maeK7U4Rn5lE8DvyH3wR2sbdBUSi18K4fWA8HUrr4zZejgGE8irR2nguKr/Xn3eW+Zp8He2EswxGtlkBNVMqxedV4Gj6UmQ3XWGCHF+xx/QhsTUoX4Tvewb4ZPyLrsUdMmR35qUcdJKMgnWFk9OyGKQoi23VHudZiGGozqBGVZnEo7L2TgFNoeiHynFI/97LRSZLWNjDL4ZyL4MBTpqUKacY4lWZbOevX/BkAXdWtmRJzB4xJzAf+7GOHSkHVDSM+1smD5PPkY8Tcd6TpRgKvcWbkOjVt8+EsoF9HAr8+pHSe8k4AkLARKXvrXc6bFx6rRfbeLmHz+WGrSlwDBvMxrj063iosy7DBAHOIzfGz9hhFTfIRZmo8MMt2V5ZCXIBTKr8CwJXzuWSLKi6UVX7TqVFnJwO/00uht4cmsdqf4iIUGItO0jBnTGdr3ywinM0cQ4a8j007J5Fox8r5JlrmDaUG5SSVzfqt5Dw+ROElHOFWFioDrqLteMW0dQvwwBStDwDtkInl+SyONJnzUlLz47oICOoN3MH+5wX7UKkE778nY/5iK2HuyGaHmX1f8yLmKcnxhnOmdR0BsHz62phTDUQcjPQ9mRqm0lHhSbvFuZanM2TMfHDN9hQMp9ZUEBXRgwEkunrWvUK6jux1sZfouta9kkbFa 78KlXCmj 5P4zOe0IwEB3NSY8Dh8i3iGVytSfAqMKn82AQ4R1wufR31eZABSyCm94wkoODrHqkSlMX0jJbNhBraK/f29zBtpX7l3PxcQWWRBezXet0USvpYnKImR4ajk1cK98MceGIOuUm6+m/hTWEojgVndd8RemHeC+0taLl+t0LeWUgxfQq1ruQmOYEKqys0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With the previous patch that enables support for batch compressions in zswap_compress_folio(), a 6.2% throughput regression was seen with zstd and 2M folios, using vm-scalability/usemem. For compressors that don't support batching, this was root-caused to the following zswap_store_folio() structure: Batched stores: --------------- - Allocate all entries, - Compress all entries, - Store all entries in xarray/LRU. Hence, the above structure is maintained only for batched stores, and the following structure is implemented for sequential stores of large folio pages, that fixes the zstd regression, while preserving common code paths for batched and sequential stores of a folio: Sequential stores: ------------------ For each page in folio: - allocate an entry, - compress the page, - store the entry in xarray/LRU. This is submitted as a separate patch only for code review purposes. I will squash this with the previous commit in subsequent versions of this patch-series. Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 193 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 111 insertions(+), 82 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index f1cba77eda62..7bfc720a6201 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1592,40 +1592,32 @@ static bool zswap_batch_compress(struct folio *folio, return ret; } -static bool zswap_compress_folio(struct folio *folio, - struct zswap_entry *entries[], - struct zswap_pool *pool) +static __always_inline bool zswap_compress_folio(struct folio *folio, + struct zswap_entry *entries[], + long from_index, + struct zswap_pool *pool, + unsigned int batch_size, + struct crypto_acomp_ctx *acomp_ctx) { - long index, nr_pages = folio_nr_pages(folio); - struct crypto_acomp_ctx *acomp_ctx; - unsigned int batch_size; + long index = from_index, nr_pages = folio_nr_pages(folio); bool ret = true; - acomp_ctx = acomp_ctx_get_cpu_lock(pool); - batch_size = acomp_ctx->nr_reqs; - if ((batch_size > 1) && (nr_pages > 1)) { - for (index = 0; index < nr_pages; index += batch_size) { + for (; index < nr_pages; index += batch_size) { if (!zswap_batch_compress(folio, index, batch_size, &entries[index], pool, acomp_ctx)) { ret = false; - goto unlock_acomp_ctx; + break; } } } else { - for (index = 0; index < nr_pages; ++index) { - struct page *page = folio_page(folio, index); + struct page *page = folio_page(folio, index); - if (!zswap_compress(page, entries[index], pool, acomp_ctx)) { - ret = false; - goto unlock_acomp_ctx; - } - } + if (!zswap_compress(page, entries[index], pool, acomp_ctx)) + ret = false; } -unlock_acomp_ctx: - acomp_ctx_put_unlock(acomp_ctx); return ret; } @@ -1637,92 +1629,128 @@ static bool zswap_compress_folio(struct folio *folio, * handles to ERR_PTR(-EINVAL) at allocation time, and the fact that the * entry's handle is subsequently modified only upon a successful zpool_malloc() * after the page is compressed. + * + * For compressors that don't support batching, the following structure + * showed a performance regression with zstd/2M folios: + * + * Batched stores: + * --------------- + * - Allocate all entries, + * - Compress all entries, + * - Store all entries in xarray/LRU. + * + * Hence, the above structure is maintained only for batched stores, and the + * following structure is implemented for sequential stores of large folio pages, + * that fixes the regression, while preserving common code paths for batched + * and sequential stores of a folio: + * + * Sequential stores: + * ------------------ + * For each page in folio: + * - allocate an entry, + * - compress the page, + * - store the entry in xarray/LRU. */ static bool zswap_store_folio(struct folio *folio, struct obj_cgroup *objcg, struct zswap_pool *pool) { - long index, from_index = 0, nr_pages = folio_nr_pages(folio); + long index = 0, from_index = 0, nr_pages, nr_folio_pages = folio_nr_pages(folio); struct zswap_entry **entries = NULL; + struct crypto_acomp_ctx *acomp_ctx; int node_id = folio_nid(folio); + unsigned int batch_size; - entries = kmalloc(nr_pages * sizeof(*entries), GFP_KERNEL); + entries = kmalloc(nr_folio_pages * sizeof(*entries), GFP_KERNEL); if (!entries) return false; - for (index = 0; index < nr_pages; ++index) { - entries[index] = zswap_entry_cache_alloc(GFP_KERNEL, node_id); + acomp_ctx = acomp_ctx_get_cpu_lock(pool); + batch_size = acomp_ctx->nr_reqs; - if (!entries[index]) { - zswap_reject_kmemcache_fail++; - nr_pages = index; - goto store_folio_failed; + nr_pages = (batch_size > 1) ? nr_folio_pages : 1; + + while (1) { + for (index = from_index; index < nr_pages; ++index) { + entries[index] = zswap_entry_cache_alloc(GFP_KERNEL, node_id); + + if (!entries[index]) { + zswap_reject_kmemcache_fail++; + nr_pages = index; + goto store_folio_failed; + } + + entries[index]->handle = (unsigned long)ERR_PTR(-EINVAL); } - entries[index]->handle = (unsigned long)ERR_PTR(-EINVAL); - } + if (!zswap_compress_folio(folio, entries, from_index, pool, batch_size, acomp_ctx)) + goto store_folio_failed; - if (!zswap_compress_folio(folio, entries, pool)) - goto store_folio_failed; + for (index = from_index; index < nr_pages; ++index) { + swp_entry_t page_swpentry = page_swap_entry(folio_page(folio, index)); + struct zswap_entry *old, *entry = entries[index]; - for (index = 0; index < nr_pages; ++index) { - swp_entry_t page_swpentry = page_swap_entry(folio_page(folio, index)); - struct zswap_entry *old, *entry = entries[index]; + old = xa_store(swap_zswap_tree(page_swpentry), + swp_offset(page_swpentry), + entry, GFP_KERNEL); + if (xa_is_err(old)) { + int err = xa_err(old); - old = xa_store(swap_zswap_tree(page_swpentry), - swp_offset(page_swpentry), - entry, GFP_KERNEL); - if (xa_is_err(old)) { - int err = xa_err(old); + WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); + zswap_reject_alloc_fail++; + from_index = index; + goto store_folio_failed; + } - WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); - zswap_reject_alloc_fail++; - from_index = index; - goto store_folio_failed; - } + /* + * We may have had an existing entry that became stale when + * the folio was redirtied and now the new version is being + * swapped out. Get rid of the old. + */ + if (old) + zswap_entry_free(old); - /* - * We may have had an existing entry that became stale when - * the folio was redirtied and now the new version is being - * swapped out. Get rid of the old. - */ - if (old) - zswap_entry_free(old); + /* + * The entry is successfully compressed and stored in the tree, there is + * no further possibility of failure. Grab refs to the pool and objcg, + * charge zswap memory, and increment zswap_stored_pages. + * The opposite actions will be performed by zswap_entry_free() + * when the entry is removed from the tree. + */ + zswap_pool_get(pool); + if (objcg) { + obj_cgroup_get(objcg); + obj_cgroup_charge_zswap(objcg, entry->length); + } + atomic_long_inc(&zswap_stored_pages); - /* - * The entry is successfully compressed and stored in the tree, there is - * no further possibility of failure. Grab refs to the pool and objcg, - * charge zswap memory, and increment zswap_stored_pages. - * The opposite actions will be performed by zswap_entry_free() - * when the entry is removed from the tree. - */ - zswap_pool_get(pool); - if (objcg) { - obj_cgroup_get(objcg); - obj_cgroup_charge_zswap(objcg, entry->length); + /* + * We finish initializing the entry while it's already in xarray. + * This is safe because: + * + * 1. Concurrent stores and invalidations are excluded by folio lock. + * + * 2. Writeback is excluded by the entry not being on the LRU yet. + * The publishing order matters to prevent writeback from seeing + * an incoherent entry. + */ + entry->pool = pool; + entry->swpentry = page_swpentry; + entry->objcg = objcg; + entry->referenced = true; + if (entry->length) { + INIT_LIST_HEAD(&entry->lru); + zswap_lru_add(&zswap_list_lru, entry); + } } - atomic_long_inc(&zswap_stored_pages); - /* - * We finish initializing the entry while it's already in xarray. - * This is safe because: - * - * 1. Concurrent stores and invalidations are excluded by folio lock. - * - * 2. Writeback is excluded by the entry not being on the LRU yet. - * The publishing order matters to prevent writeback from seeing - * an incoherent entry. - */ - entry->pool = pool; - entry->swpentry = page_swpentry; - entry->objcg = objcg; - entry->referenced = true; - if (entry->length) { - INIT_LIST_HEAD(&entry->lru); - zswap_lru_add(&zswap_list_lru, entry); - } + from_index = nr_pages++; + + if (nr_pages > nr_folio_pages) + break; } + acomp_ctx_put_unlock(acomp_ctx); kfree(entries); return true; @@ -1734,6 +1762,7 @@ static bool zswap_store_folio(struct folio *folio, zswap_entry_cache_free(entries[index]); } + acomp_ctx_put_unlock(acomp_ctx); kfree(entries); return false; }