From patchwork Thu Mar 3 18:00:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cabiddu, Giovanni" X-Patchwork-Id: 12767865 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BE18C433EF for ; Thu, 3 Mar 2022 18:13:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232134AbiCCSOU (ORCPT ); Thu, 3 Mar 2022 13:14:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230013AbiCCSOU (ORCPT ); Thu, 3 Mar 2022 13:14:20 -0500 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52CE7EBBB4 for ; Thu, 3 Mar 2022 10:13:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646331214; x=1677867214; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/aSIoxxrcUjK5q+GvpyOjEsZQzL2+bA6hBE05WY83wQ=; b=PAPUoWMvcJNvATtD1/KKcF/jTJoCsoMMC8QF4yxjZpwrfuI9e7RaQ1P3 uqbyhJsNuy5x3SL5NK2JwArP+IwssI7aYHUpF4MjkZKQz4E7rjkG63DRI 4vOwomTG3UT5tztLGjyLcZe10dr2us6a3GhJmzhi0qiB9Ovogho2V7utP EoyJzf31buBTzfk/LBb+70IDmRL5dlBzb5Ez5MCQcqz1tvfzhCCa2TG/v l6aFSky+wXEAp4zV7XjmwAh4io/cLqv/KqSjKf4SFuYDCApJpDlDitLZ0 TdXSrjkPvEajetMrYLyPAxzKCK5GI8A6CT2PGLxZb5icITFjep6ef6jjy Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10275"; a="278447742" X-IronPort-AV: E=Sophos;i="5.90,151,1643702400"; d="scan'208";a="278447742" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Mar 2022 10:00:52 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,151,1643702400"; d="scan'208";a="640279657" Received: from silpixa00400314.ir.intel.com (HELO silpixa00400314.ger.corp.intel.com) ([10.237.222.76]) by fmsmga002.fm.intel.com with ESMTP; 03 Mar 2022 10:00:50 -0800 From: Giovanni Cabiddu To: herbert@gondor.apana.org.au Cc: linux-crypto@vger.kernel.org, qat-linux@intel.com, Kyle Sanderson , Vlad Dronov , Giovanni Cabiddu , Mikulas Patocka , Marco Chiappero , Wojciech Ziemba Subject: [RFC 1/3] crypto: qat - use pre-allocated buffers in datapath Date: Thu, 3 Mar 2022 18:00:34 +0000 Message-Id: <20220303180036.13475-2-giovanni.cabiddu@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220303180036.13475-1-giovanni.cabiddu@intel.com> References: <20220303180036.13475-1-giovanni.cabiddu@intel.com> MIME-Version: 1.0 Organization: Intel Research and Development Ireland Ltd - Co. Reg. #308263 - Collinstown Industrial Park, Leixlip, County Kildare - Ireland Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org In order to do DMAs, the QAT device requires that scatterlists are mapped and translated in a format that the firmware can understand. This is defined as the composition of an sgl descriptor header, the struct qat_alg_buf_list, plus a variable number of flat buffer descriptors, the struct qat_alg_buf. The allocation and mapping of these data structures is done each time a request is received from the skcipher and aead APIs. In an OOM situation, this behaviour might lead to a dead-lock as allocations might fail. Based on the conversation in [1], increase the size of the aead and skcipher request contexts to include an SGL descriptor that can handle maximum 4 flat buffers. If requests exceed 4 entries buffers, memory is allocated dynamically. [1] https://lore.kernel.org/linux-crypto/20200722072932.GA27544@gondor.apana.org.au/ Fixes: d370cec32194 ("crypto: qat - Intel(R) QAT crypto interface") Reported-by: Mikulas Patocka Signed-off-by: Giovanni Cabiddu Reviewed-by: Marco Chiappero Reviewed-by: Wojciech Ziemba --- drivers/crypto/qat/qat_common/qat_algs.c | 62 ++++++++++++---------- drivers/crypto/qat/qat_common/qat_crypto.h | 24 +++++++++ 2 files changed, 59 insertions(+), 27 deletions(-) diff --git a/drivers/crypto/qat/qat_common/qat_algs.c b/drivers/crypto/qat/qat_common/qat_algs.c index f998ed58457c..862184aec3d4 100644 --- a/drivers/crypto/qat/qat_common/qat_algs.c +++ b/drivers/crypto/qat/qat_common/qat_algs.c @@ -46,19 +46,6 @@ static DEFINE_MUTEX(algs_lock); static unsigned int active_devs; -struct qat_alg_buf { - u32 len; - u32 resrvd; - u64 addr; -} __packed; - -struct qat_alg_buf_list { - u64 resrvd; - u32 num_bufs; - u32 num_mapped_bufs; - struct qat_alg_buf bufers[]; -} __packed __aligned(64); - /* Common content descriptor */ struct qat_alg_cd { union { @@ -693,7 +680,10 @@ static void qat_alg_free_bufl(struct qat_crypto_instance *inst, bl->bufers[i].len, DMA_BIDIRECTIONAL); dma_unmap_single(dev, blp, sz, DMA_TO_DEVICE); - kfree(bl); + + if (!qat_req->sgl_src_valid) + kfree(bl); + if (blp != blpout) { /* If out of place operation dma unmap only data */ int bufless = blout->num_bufs - blout->num_mapped_bufs; @@ -704,7 +694,9 @@ static void qat_alg_free_bufl(struct qat_crypto_instance *inst, DMA_BIDIRECTIONAL); } dma_unmap_single(dev, blpout, sz_out, DMA_TO_DEVICE); - kfree(blout); + + if (!qat_req->sgl_dst_valid) + kfree(blout); } } @@ -721,15 +713,23 @@ static int qat_alg_sgl_to_bufl(struct qat_crypto_instance *inst, dma_addr_t blp = DMA_MAPPING_ERROR; dma_addr_t bloutp = DMA_MAPPING_ERROR; struct scatterlist *sg; - size_t sz_out, sz = struct_size(bufl, bufers, n + 1); + size_t sz_out, sz = struct_size(bufl, bufers, n); + int node = dev_to_node(&GET_DEV(inst->accel_dev)); if (unlikely(!n)) return -EINVAL; - bufl = kzalloc_node(sz, GFP_ATOMIC, - dev_to_node(&GET_DEV(inst->accel_dev))); - if (unlikely(!bufl)) - return -ENOMEM; + qat_req->sgl_src_valid = false; + qat_req->sgl_dst_valid = false; + + if (n > QAT_MAX_BUFF_DESC) { + bufl = kzalloc_node(sz, GFP_ATOMIC, node); + if (unlikely(!bufl)) + return -ENOMEM; + } else { + bufl = &qat_req->sgl_src.sgl_hdr; + qat_req->sgl_src_valid = true; + } for_each_sg(sgl, sg, n, i) bufl->bufers[i].addr = DMA_MAPPING_ERROR; @@ -760,12 +760,17 @@ static int qat_alg_sgl_to_bufl(struct qat_crypto_instance *inst, struct qat_alg_buf *bufers; n = sg_nents(sglout); - sz_out = struct_size(buflout, bufers, n + 1); + sz_out = struct_size(buflout, bufers, n); sg_nctr = 0; - buflout = kzalloc_node(sz_out, GFP_ATOMIC, - dev_to_node(&GET_DEV(inst->accel_dev))); - if (unlikely(!buflout)) - goto err_in; + + if (n > QAT_MAX_BUFF_DESC) { + buflout = kzalloc_node(sz_out, GFP_ATOMIC, node); + if (unlikely(!buflout)) + goto err_in; + } else { + buflout = &qat_req->sgl_dst.sgl_hdr; + qat_req->sgl_dst_valid = true; + } bufers = buflout->bufers; for_each_sg(sglout, sg, n, i) @@ -810,7 +815,9 @@ static int qat_alg_sgl_to_bufl(struct qat_crypto_instance *inst, dma_unmap_single(dev, buflout->bufers[i].addr, buflout->bufers[i].len, DMA_BIDIRECTIONAL); - kfree(buflout); + + if (!qat_req->sgl_dst_valid) + kfree(buflout); err_in: if (!dma_mapping_error(dev, blp)) @@ -823,7 +830,8 @@ static int qat_alg_sgl_to_bufl(struct qat_crypto_instance *inst, bufl->bufers[i].len, DMA_BIDIRECTIONAL); - kfree(bufl); + if (!qat_req->sgl_src_valid) + kfree(bufl); dev_err(dev, "Failed to map buf for dma\n"); return -ENOMEM; diff --git a/drivers/crypto/qat/qat_common/qat_crypto.h b/drivers/crypto/qat/qat_common/qat_crypto.h index b6a4c95ae003..8f2aa4804ed0 100644 --- a/drivers/crypto/qat/qat_common/qat_crypto.h +++ b/drivers/crypto/qat/qat_common/qat_crypto.h @@ -21,6 +21,26 @@ struct qat_crypto_instance { atomic_t refctr; }; +#define QAT_MAX_BUFF_DESC 4 + +struct qat_alg_buf { + u32 len; + u32 resrvd; + u64 addr; +} __packed; + +struct qat_alg_buf_list { + u64 resrvd; + u32 num_bufs; + u32 num_mapped_bufs; + struct qat_alg_buf bufers[]; +} __packed __aligned(64); + +struct qat_alg_fixed_buf_list { + struct qat_alg_buf_list sgl_hdr; + struct qat_alg_buf descriptors[QAT_MAX_BUFF_DESC]; +} __packed __aligned(64); + struct qat_crypto_request_buffs { struct qat_alg_buf_list *bl; dma_addr_t blp; @@ -53,6 +73,10 @@ struct qat_crypto_request { u8 iv[AES_BLOCK_SIZE]; }; bool encryption; + bool sgl_src_valid; + bool sgl_dst_valid; + struct qat_alg_fixed_buf_list sgl_src; + struct qat_alg_fixed_buf_list sgl_dst; }; static inline bool adf_hw_dev_has_crypto(struct adf_accel_dev *accel_dev) From patchwork Thu Mar 3 18:00:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cabiddu, Giovanni" X-Patchwork-Id: 12767867 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2264BC433EF for ; Thu, 3 Mar 2022 18:13:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230013AbiCCSOZ (ORCPT ); Thu, 3 Mar 2022 13:14:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232161AbiCCSOX (ORCPT ); Thu, 3 Mar 2022 13:14:23 -0500 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 917EDED97F for ; Thu, 3 Mar 2022 10:13:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646331216; x=1677867216; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=475KWS36204xKiy0tqQJ+UmV+QHP6FZza11GytI92l8=; b=cC5RqPaSlgDoe+cRmsKis5bsjRhVZU/MewBkY6AePifmE4IqfkLQDqLe tmch4b0yuTweSimlw1Ku3cHElAinXZvGZR1xbqzgclJBx8/x7VbgL6SM/ lpPBp8McfMg6yJqRVmSE3XbJ1YRCCcSNyuni7DEUdf4pgZeS5PJKC1gS7 HTjZC4L7JOmNaTUQ34TPbiLTMxHfzsaNFTs1O8RmGy+AY/DuybG+4tRiD HRGnVSodkG1jXT2ZLgQ+HsviWbCM5KmB3KhNt0YyHuQteLN4Wf40d7GLu 9GGs44/B9DrX4rbydx7RDDWepRlS3scos2nOtDDqEz8UMGCSIwB4LpmPW Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10275"; a="278447770" X-IronPort-AV: E=Sophos;i="5.90,151,1643702400"; d="scan'208";a="278447770" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Mar 2022 10:00:55 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,151,1643702400"; d="scan'208";a="640279706" Received: from silpixa00400314.ir.intel.com (HELO silpixa00400314.ger.corp.intel.com) ([10.237.222.76]) by fmsmga002.fm.intel.com with ESMTP; 03 Mar 2022 10:00:52 -0800 From: Giovanni Cabiddu To: herbert@gondor.apana.org.au Cc: linux-crypto@vger.kernel.org, qat-linux@intel.com, Kyle Sanderson , Vlad Dronov , Vishnu Das Ramachandran , Mikulas Patocka , Giovanni Cabiddu , Marco Chiappero , Wojciech Ziemba Subject: [RFC 2/3] crypto: qat - add backlog mechanism Date: Thu, 3 Mar 2022 18:00:35 +0000 Message-Id: <20220303180036.13475-3-giovanni.cabiddu@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220303180036.13475-1-giovanni.cabiddu@intel.com> References: <20220303180036.13475-1-giovanni.cabiddu@intel.com> MIME-Version: 1.0 Organization: Intel Research and Development Ireland Ltd - Co. Reg. #308263 - Collinstown Industrial Park, Leixlip, County Kildare - Ireland Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Vishnu Das Ramachandran The implementations of aead and skcipher in the QAT driver are not properly supporting requests with the CRYPTO_TFM_REQ_MAY_BACKLOG flag set. If the HW queue is full, the driver returns -EBUSY but does not enqueue the request. This can result in applications like dm-crypt waiting indefinitely for a completion of a request that was never submitted to the hardware. Fix this by adding a software backlog queue: if the ring buffer is more than eighty five percent full, then the request is enqueued to a backlog list, a worker thread is scheduled to resubmit it at later time and the error code -EBUSY is returned up to the caller. The request for which -EBUSY is returned is then marked as -EINPROGRESS in response callback, when response for that request is received. Fixes: d370cec32194 ("crypto: qat - Intel(R) QAT crypto interface") Reported-by: Mikulas Patocka Reported-by: Kyle Sanderson Signed-off-by: Vishnu Das Ramachandran Co-developed-by: Giovanni Cabiddu Signed-off-by: Giovanni Cabiddu Reviewed-by: Marco Chiappero Reviewed-by: Wojciech Ziemba --- drivers/crypto/qat/qat_common/adf_transport.c | 12 + drivers/crypto/qat/qat_common/adf_transport.h | 1 + .../qat/qat_common/adf_transport_internal.h | 1 + drivers/crypto/qat/qat_common/qat_algs.c | 230 ++++++++++++++---- drivers/crypto/qat/qat_common/qat_crypto.h | 2 + 5 files changed, 204 insertions(+), 42 deletions(-) diff --git a/drivers/crypto/qat/qat_common/adf_transport.c b/drivers/crypto/qat/qat_common/adf_transport.c index 8ba28409fb74..bc81a91b681a 100644 --- a/drivers/crypto/qat/qat_common/adf_transport.c +++ b/drivers/crypto/qat/qat_common/adf_transport.c @@ -8,6 +8,9 @@ #include "adf_cfg.h" #include "adf_common_drv.h" +#define ADF_MAX_RING_THRESHOLD 85 +#define ADF_CFG_PERCENT_TO_NUM(tot, percent) (((tot) * (percent)) / 100) + static inline u32 adf_modulo(u32 data, u32 shift) { u32 div = data >> shift; @@ -77,6 +80,11 @@ static void adf_disable_ring_irq(struct adf_etr_bank_data *bank, u32 ring) bank->irq_mask); } +bool adf_ring_nearly_full(struct adf_etr_ring_data *ring) +{ + return atomic_read(ring->inflights) > ring->threshold; +} + int adf_send_message(struct adf_etr_ring_data *ring, u32 *msg) { struct adf_hw_csr_ops *csr_ops = GET_CSR_OPS(ring->bank->accel_dev); @@ -217,6 +225,7 @@ int adf_create_ring(struct adf_accel_dev *accel_dev, const char *section, struct adf_etr_bank_data *bank; struct adf_etr_ring_data *ring; char val[ADF_CFG_MAX_VAL_LEN_IN_BYTES]; + int max_inflights; u32 ring_num; int ret; @@ -261,6 +270,9 @@ int adf_create_ring(struct adf_accel_dev *accel_dev, const char *section, ring->callback = callback; ring->msg_size = ADF_BYTES_TO_MSG_SIZE(msg_size); ring->ring_size = adf_verify_ring_size(msg_size, num_msgs); + max_inflights = ADF_MAX_INFLIGHTS(ring->ring_size, ring->msg_size); + ring->threshold = ADF_CFG_PERCENT_TO_NUM(max_inflights, + ADF_MAX_RING_THRESHOLD); ring->head = 0; ring->tail = 0; atomic_set(ring->inflights, 0); diff --git a/drivers/crypto/qat/qat_common/adf_transport.h b/drivers/crypto/qat/qat_common/adf_transport.h index 2c95f1697c76..e6ef6f9b7691 100644 --- a/drivers/crypto/qat/qat_common/adf_transport.h +++ b/drivers/crypto/qat/qat_common/adf_transport.h @@ -14,6 +14,7 @@ int adf_create_ring(struct adf_accel_dev *accel_dev, const char *section, const char *ring_name, adf_callback_fn callback, int poll_mode, struct adf_etr_ring_data **ring_ptr); +bool adf_ring_nearly_full(struct adf_etr_ring_data *ring); int adf_send_message(struct adf_etr_ring_data *ring, u32 *msg); void adf_remove_ring(struct adf_etr_ring_data *ring); #endif diff --git a/drivers/crypto/qat/qat_common/adf_transport_internal.h b/drivers/crypto/qat/qat_common/adf_transport_internal.h index 501bcf0f1809..8b2c92ba7ca1 100644 --- a/drivers/crypto/qat/qat_common/adf_transport_internal.h +++ b/drivers/crypto/qat/qat_common/adf_transport_internal.h @@ -22,6 +22,7 @@ struct adf_etr_ring_data { spinlock_t lock; /* protects ring data struct */ u16 head; u16 tail; + u32 threshold; u8 ring_number; u8 ring_size; u8 msg_size; diff --git a/drivers/crypto/qat/qat_common/qat_algs.c b/drivers/crypto/qat/qat_common/qat_algs.c index 862184aec3d4..765377cebcca 100644 --- a/drivers/crypto/qat/qat_common/qat_algs.c +++ b/drivers/crypto/qat/qat_common/qat_algs.c @@ -43,6 +43,8 @@ (GET_HW_DATA(accel_dev)->accel_capabilities_mask & \ ICP_ACCEL_CAPABILITIES_AES_V2) +#define ADF_MAX_RETRIES 10 + static DEFINE_MUTEX(algs_lock); static unsigned int active_devs; @@ -60,6 +62,13 @@ struct qat_alg_cd { }; } __aligned(64); +struct qat_alg_crypto_backlog { + struct workqueue_struct *wq; + struct list_head backlog_list; + spinlock_t list_lock; /* protects backlog list */ + struct work_struct work; +}; + struct qat_alg_aead_ctx { struct qat_alg_cd *enc_cd; struct qat_alg_cd *dec_cd; @@ -77,6 +86,7 @@ struct qat_alg_aead_ctx { }; char ipad[SHA512_BLOCK_SIZE]; /* sufficient for SHA-1/SHA-256 as well */ char opad[SHA512_BLOCK_SIZE]; + struct qat_alg_crypto_backlog bk_log; }; struct qat_alg_skcipher_ctx { @@ -89,6 +99,7 @@ struct qat_alg_skcipher_ctx { struct qat_crypto_instance *inst; struct crypto_skcipher *ftfm; struct crypto_cipher *tweak; + struct qat_alg_crypto_backlog bk_log; bool fallback; int mode; }; @@ -849,6 +860,8 @@ static void qat_aead_alg_callback(struct icp_qat_fw_la_resp *qat_resp, qat_alg_free_bufl(inst, qat_req); if (unlikely(qat_res != ICP_QAT_FW_COMN_STATUS_FLAG_OK)) res = -EBADMSG; + if (qat_req->enqueued) + areq->base.complete(&areq->base, -EINPROGRESS); areq->base.complete(&areq->base, res); } @@ -907,6 +920,130 @@ static void qat_alg_update_iv(struct qat_crypto_request *qat_req) } } +static void qat_alg_send_backlogged_req(struct qat_alg_crypto_backlog *bk_log, + struct adf_etr_ring_data *sym_tx, + work_func_t worker_func) +{ + struct qat_crypto_request *qat_req = NULL; + struct list_head *request = NULL; + + spin_lock_bh(&bk_log->list_lock); + while (!list_empty(&bk_log->backlog_list)) { + request = bk_log->backlog_list.next; + qat_req = list_entry(request, struct qat_crypto_request, list); + + if (adf_send_message(sym_tx, (u32 *)&qat_req->req)) { + /* If adf_send_message() fails, trigger worker */ + INIT_WORK(&bk_log->work, worker_func); + queue_work(bk_log->wq, &bk_log->work); + break; + } + list_del(request); + } + spin_unlock_bh(&bk_log->list_lock); +} + +static void qat_alg_aead_swq_worker(struct work_struct *work) +{ + struct qat_alg_crypto_backlog *bk_log = + container_of(work, struct qat_alg_crypto_backlog, work); + struct qat_alg_aead_ctx *aead_ctx = + container_of(bk_log, struct qat_alg_aead_ctx, bk_log); + + qat_alg_send_backlogged_req(bk_log, + aead_ctx->inst->sym_tx, + qat_alg_aead_swq_worker); +} + +static void qat_alg_skciph_swq_worker(struct work_struct *work) +{ + struct qat_alg_crypto_backlog *bk_log = + container_of(work, struct qat_alg_crypto_backlog, work); + struct qat_alg_skcipher_ctx *skcipher_ctx = + container_of(bk_log, struct qat_alg_skcipher_ctx, bk_log); + + qat_alg_send_backlogged_req(bk_log, + skcipher_ctx->inst->sym_tx, + qat_alg_skciph_swq_worker); +} + +static int qat_alg_backlog_req(struct qat_crypto_request *qat_req, + struct qat_alg_crypto_backlog *bk_log, + work_func_t worker_func) +{ + /* For any backlogged request, enqueued should be true. */ + qat_req->enqueued = true; + + /* Add request to backlog list */ + spin_lock_bh(&bk_log->list_lock); + if (list_empty(&bk_log->backlog_list)) { + /* Add work to queue, only if it is the first element in list */ + INIT_WORK(&bk_log->work, worker_func); + queue_work(bk_log->wq, &bk_log->work); + } + list_add_tail(&qat_req->list, &bk_log->backlog_list); + spin_unlock_bh(&bk_log->list_lock); + + return -EBUSY; +} + +static int qat_alg_send_may_backlog_req(struct qat_crypto_request *qat_req, + struct qat_alg_crypto_backlog *bk_log, + struct adf_etr_ring_data *sym_tx, + work_func_t worker_func) +{ + /* If any request is already backlogged, then add to backlog list */ + if (!list_empty(&bk_log->backlog_list)) + return qat_alg_backlog_req(qat_req, bk_log, worker_func); + + qat_req->enqueued = adf_ring_nearly_full(sym_tx); + + /* + * If ring utilization is more than ADF_MAX_RING_THRESHOLD percent, + * then add to backlog list. + */ + if (qat_req->enqueued) + return qat_alg_backlog_req(qat_req, bk_log, worker_func); + + /* If adding request to HW ring fails, then add to backlog list */ + if (adf_send_message(sym_tx, (u32 *)&qat_req->req)) + return qat_alg_backlog_req(qat_req, bk_log, worker_func); + + return -EINPROGRESS; +} + +static int qat_alg_send_message(struct qat_crypto_request *qat_req, + struct qat_crypto_instance *inst) +{ + int ret = 0, ctr = 0; + + /* For requests without backlog support, enqueued should be false. */ + qat_req->enqueued = false; + + do { + ret = adf_send_message(inst->sym_tx, (u32 *)&qat_req->req); + } while (ret == -EAGAIN && ctr++ < ADF_MAX_RETRIES); + if (ret == -EAGAIN) { + qat_alg_free_bufl(inst, qat_req); + return -EAGAIN; + } + + return -EINPROGRESS; +} + +static int qat_alg_skcipher_send_message(struct qat_crypto_request *qat_req) +{ + struct qat_alg_skcipher_ctx *ctx = qat_req->skcipher_ctx; + + if (qat_req->skcipher_req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG) { + return qat_alg_send_may_backlog_req(qat_req, &ctx->bk_log, + ctx->inst->sym_tx, + qat_alg_skciph_swq_worker); + } + + return qat_alg_send_message(qat_req, ctx->inst); +} + static void qat_skcipher_alg_callback(struct icp_qat_fw_la_resp *qat_resp, struct qat_crypto_request *qat_req) { @@ -925,6 +1062,8 @@ static void qat_skcipher_alg_callback(struct icp_qat_fw_la_resp *qat_resp, memcpy(sreq->iv, qat_req->iv, AES_BLOCK_SIZE); + if (qat_req->enqueued) + sreq->base.complete(&sreq->base, -EINPROGRESS); sreq->base.complete(&sreq->base, res); } @@ -937,6 +1076,19 @@ void qat_alg_callback(void *resp) qat_req->cb(qat_resp, qat_req); } +static int qat_alg_aead_send_message(struct qat_crypto_request *qat_req) +{ + struct qat_alg_aead_ctx *ctx = qat_req->aead_ctx; + + if (qat_req->aead_req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG) { + return qat_alg_send_may_backlog_req(qat_req, &ctx->bk_log, + ctx->inst->sym_tx, + qat_alg_aead_swq_worker); + } + + return qat_alg_send_message(qat_req, ctx->inst); +} + static int qat_alg_aead_dec(struct aead_request *areq) { struct crypto_aead *aead_tfm = crypto_aead_reqtfm(areq); @@ -947,7 +1099,7 @@ static int qat_alg_aead_dec(struct aead_request *areq) struct icp_qat_fw_la_auth_req_params *auth_param; struct icp_qat_fw_la_bulk_req *msg; int digst_size = crypto_aead_authsize(aead_tfm); - int ret, ctr = 0; + int ret; u32 cipher_len; cipher_len = areq->cryptlen - digst_size; @@ -973,15 +1125,8 @@ static int qat_alg_aead_dec(struct aead_request *areq) auth_param = (void *)((u8 *)cipher_param + sizeof(*cipher_param)); auth_param->auth_off = 0; auth_param->auth_len = areq->assoclen + cipher_param->cipher_length; - do { - ret = adf_send_message(ctx->inst->sym_tx, (u32 *)msg); - } while (ret == -EAGAIN && ctr++ < 10); - if (ret == -EAGAIN) { - qat_alg_free_bufl(ctx->inst, qat_req); - return -EBUSY; - } - return -EINPROGRESS; + return qat_alg_aead_send_message(qat_req); } static int qat_alg_aead_enc(struct aead_request *areq) @@ -994,7 +1139,7 @@ static int qat_alg_aead_enc(struct aead_request *areq) struct icp_qat_fw_la_auth_req_params *auth_param; struct icp_qat_fw_la_bulk_req *msg; u8 *iv = areq->iv; - int ret, ctr = 0; + int ret; if (areq->cryptlen % AES_BLOCK_SIZE != 0) return -EINVAL; @@ -1021,15 +1166,7 @@ static int qat_alg_aead_enc(struct aead_request *areq) auth_param->auth_off = 0; auth_param->auth_len = areq->assoclen + areq->cryptlen; - do { - ret = adf_send_message(ctx->inst->sym_tx, (u32 *)msg); - } while (ret == -EAGAIN && ctr++ < 10); - - if (ret == -EAGAIN) { - qat_alg_free_bufl(ctx->inst, qat_req); - return -EBUSY; - } - return -EINPROGRESS; + return qat_alg_aead_send_message(qat_req); } static int qat_alg_skcipher_rekey(struct qat_alg_skcipher_ctx *ctx, @@ -1182,7 +1319,7 @@ static int qat_alg_skcipher_encrypt(struct skcipher_request *req) struct qat_crypto_request *qat_req = skcipher_request_ctx(req); struct icp_qat_fw_la_cipher_req_params *cipher_param; struct icp_qat_fw_la_bulk_req *msg; - int ret, ctr = 0; + int ret; if (req->cryptlen == 0) return 0; @@ -1206,15 +1343,7 @@ static int qat_alg_skcipher_encrypt(struct skcipher_request *req) qat_alg_set_req_iv(qat_req); - do { - ret = adf_send_message(ctx->inst->sym_tx, (u32 *)msg); - } while (ret == -EAGAIN && ctr++ < 10); - - if (ret == -EAGAIN) { - qat_alg_free_bufl(ctx->inst, qat_req); - return -EBUSY; - } - return -EINPROGRESS; + return qat_alg_skcipher_send_message(qat_req); } static int qat_alg_skcipher_blk_encrypt(struct skcipher_request *req) @@ -1251,7 +1380,7 @@ static int qat_alg_skcipher_decrypt(struct skcipher_request *req) struct qat_crypto_request *qat_req = skcipher_request_ctx(req); struct icp_qat_fw_la_cipher_req_params *cipher_param; struct icp_qat_fw_la_bulk_req *msg; - int ret, ctr = 0; + int ret; if (req->cryptlen == 0) return 0; @@ -1276,15 +1405,7 @@ static int qat_alg_skcipher_decrypt(struct skcipher_request *req) qat_alg_set_req_iv(qat_req); qat_alg_update_iv(qat_req); - do { - ret = adf_send_message(ctx->inst->sym_tx, (u32 *)msg); - } while (ret == -EAGAIN && ctr++ < 10); - - if (ret == -EAGAIN) { - qat_alg_free_bufl(ctx->inst, qat_req); - return -EBUSY; - } - return -EINPROGRESS; + return qat_alg_skcipher_send_message(qat_req); } static int qat_alg_skcipher_blk_decrypt(struct skcipher_request *req) @@ -1313,6 +1434,19 @@ static int qat_alg_skcipher_xts_decrypt(struct skcipher_request *req) return qat_alg_skcipher_decrypt(req); } +static int qat_alg_wq_init(struct qat_alg_crypto_backlog *bk_log, char *wq_name) +{ + bk_log->wq = alloc_ordered_workqueue(wq_name, WQ_MEM_RECLAIM); + if (!bk_log->wq) { + pr_err("work queue allocation failure"); + return -EFAULT; + } + INIT_LIST_HEAD(&bk_log->backlog_list); + spin_lock_init(&bk_log->list_lock); + + return 0; +} + static int qat_alg_aead_init(struct crypto_aead *tfm, enum icp_qat_hw_auth_algo hash, const char *hash_name) @@ -1324,7 +1458,8 @@ static int qat_alg_aead_init(struct crypto_aead *tfm, return PTR_ERR(ctx->hash_tfm); ctx->qat_hash_alg = hash; crypto_aead_set_reqsize(tfm, sizeof(struct qat_crypto_request)); - return 0; + + return qat_alg_wq_init(&ctx->bk_log, "qat_alg_aead_send_wq"); } static int qat_alg_aead_sha1_init(struct crypto_aead *tfm) @@ -1364,13 +1499,20 @@ static void qat_alg_aead_exit(struct crypto_aead *tfm) dma_free_coherent(dev, sizeof(struct qat_alg_cd), ctx->dec_cd, ctx->dec_cd_paddr); } + if (ctx->bk_log.wq) + destroy_workqueue(ctx->bk_log.wq); + ctx->bk_log.wq = NULL; + qat_crypto_put_instance(inst); } static int qat_alg_skcipher_init_tfm(struct crypto_skcipher *tfm) { + struct qat_alg_skcipher_ctx *ctx = crypto_skcipher_ctx(tfm); + crypto_skcipher_set_reqsize(tfm, sizeof(struct qat_crypto_request)); - return 0; + + return qat_alg_wq_init(&ctx->bk_log, "qat_alg_skciph_send_wq"); } static int qat_alg_skcipher_init_xts_tfm(struct crypto_skcipher *tfm) @@ -1394,7 +1536,7 @@ static int qat_alg_skcipher_init_xts_tfm(struct crypto_skcipher *tfm) crypto_skcipher_reqsize(ctx->ftfm)); crypto_skcipher_set_reqsize(tfm, reqsize); - return 0; + return qat_alg_wq_init(&ctx->bk_log, "qat_alg_skciph_send_wq"); } static void qat_alg_skcipher_exit_tfm(struct crypto_skcipher *tfm) @@ -1421,6 +1563,10 @@ static void qat_alg_skcipher_exit_tfm(struct crypto_skcipher *tfm) sizeof(struct icp_qat_hw_cipher_algo_blk), ctx->dec_cd, ctx->dec_cd_paddr); } + if (ctx->bk_log.wq) + destroy_workqueue(ctx->bk_log.wq); + ctx->bk_log.wq = NULL; + qat_crypto_put_instance(inst); } diff --git a/drivers/crypto/qat/qat_common/qat_crypto.h b/drivers/crypto/qat/qat_common/qat_crypto.h index 8f2aa4804ed0..136516190a50 100644 --- a/drivers/crypto/qat/qat_common/qat_crypto.h +++ b/drivers/crypto/qat/qat_common/qat_crypto.h @@ -53,6 +53,7 @@ struct qat_crypto_request_buffs { struct qat_crypto_request; struct qat_crypto_request { + struct list_head list; struct icp_qat_fw_la_bulk_req req; union { struct qat_alg_aead_ctx *aead_ctx; @@ -65,6 +66,7 @@ struct qat_crypto_request { struct qat_crypto_request_buffs buf; void (*cb)(struct icp_qat_fw_la_resp *resp, struct qat_crypto_request *req); + bool enqueued; union { struct { __be64 iv_hi; From patchwork Thu Mar 3 18:00:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cabiddu, Giovanni" X-Patchwork-Id: 12767866 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FD07C433F5 for ; Thu, 3 Mar 2022 18:13:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232161AbiCCSO0 (ORCPT ); Thu, 3 Mar 2022 13:14:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233816AbiCCSOZ (ORCPT ); Thu, 3 Mar 2022 13:14:25 -0500 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 904ECEF7AF for ; Thu, 3 Mar 2022 10:13:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646331219; x=1677867219; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YvFtguEdVtYSvc/8BUWq1lfFeWQ9inpeeCp0rT5BQ+M=; b=UGIIrBE8QdnGO3150RnXXCGYOqi/jim7bsIsf0uHxwLbqeDVrGssx61F fvKj2wcWQA8zQsQmyolR7tlP30KfKkAph7LDRVEWJ5qcCf8vL+NTBd8BJ NxXWoSo9ZVL1KS+wd8xvFXF1NxEz8SG12JVTl4lX7SrA4G0YEMW5cNkoa CYNIxvUvVgjtPehebmvRiXAAI1q+GK9iJ/ywGmSvhc69DF6m9thMVxCfP d48B3VF029BjdaSXmj/pZApnoAhWduYFClBYAzUm4XqUV858838SaRPAw KTRx8BMujCoaK/Q+7FkhpHy+m4DIJgU9KsqadOaG6sFu1DaIbkWnsYxp7 Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10275"; a="278447794" X-IronPort-AV: E=Sophos;i="5.90,151,1643702400"; d="scan'208";a="278447794" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Mar 2022 10:00:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,151,1643702400"; d="scan'208";a="640279734" Received: from silpixa00400314.ir.intel.com (HELO silpixa00400314.ger.corp.intel.com) ([10.237.222.76]) by fmsmga002.fm.intel.com with ESMTP; 03 Mar 2022 10:00:55 -0800 From: Giovanni Cabiddu To: herbert@gondor.apana.org.au Cc: linux-crypto@vger.kernel.org, qat-linux@intel.com, Kyle Sanderson , Vlad Dronov , Giovanni Cabiddu , Marco Chiappero , Wojciech Ziemba Subject: [RFC 3/3] crypto: qat - remove CRYPTO_ALG_ALLOCATES_MEMORY flag Date: Thu, 3 Mar 2022 18:00:36 +0000 Message-Id: <20220303180036.13475-4-giovanni.cabiddu@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220303180036.13475-1-giovanni.cabiddu@intel.com> References: <20220303180036.13475-1-giovanni.cabiddu@intel.com> MIME-Version: 1.0 Organization: Intel Research and Development Ireland Ltd - Co. Reg. #308263 - Collinstown Industrial Park, Leixlip, County Kildare - Ireland Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Remove the CRYPTO_ALG_ALLOCATES_MEMORY flag from the aead and skcipher alg structures. This allows the QAT driver to be used by dm-crypt after it has been changed to (1) use pre-allocated buffers in the datapath and (2) to handle skcipher and aead requests with the CRYPTO_TFM_REQ_MAY_BACKLOG flag set. Signed-off-by: Giovanni Cabiddu Reviewed-by: Marco Chiappero Reviewed-by: Wojciech Ziemba --- drivers/crypto/qat/qat_common/qat_algs.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/drivers/crypto/qat/qat_common/qat_algs.c b/drivers/crypto/qat/qat_common/qat_algs.c index 765377cebcca..38ccb20c507e 100644 --- a/drivers/crypto/qat/qat_common/qat_algs.c +++ b/drivers/crypto/qat/qat_common/qat_algs.c @@ -1588,7 +1588,7 @@ static struct aead_alg qat_aeads[] = { { .cra_name = "authenc(hmac(sha1),cbc(aes))", .cra_driver_name = "qat_aes_cbc_hmac_sha1", .cra_priority = 4001, - .cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY, + .cra_flags = CRYPTO_ALG_ASYNC, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct qat_alg_aead_ctx), .cra_module = THIS_MODULE, @@ -1605,7 +1605,7 @@ static struct aead_alg qat_aeads[] = { { .cra_name = "authenc(hmac(sha256),cbc(aes))", .cra_driver_name = "qat_aes_cbc_hmac_sha256", .cra_priority = 4001, - .cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY, + .cra_flags = CRYPTO_ALG_ASYNC, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct qat_alg_aead_ctx), .cra_module = THIS_MODULE, @@ -1622,7 +1622,7 @@ static struct aead_alg qat_aeads[] = { { .cra_name = "authenc(hmac(sha512),cbc(aes))", .cra_driver_name = "qat_aes_cbc_hmac_sha512", .cra_priority = 4001, - .cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY, + .cra_flags = CRYPTO_ALG_ASYNC, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct qat_alg_aead_ctx), .cra_module = THIS_MODULE, @@ -1640,7 +1640,7 @@ static struct skcipher_alg qat_skciphers[] = { { .base.cra_name = "cbc(aes)", .base.cra_driver_name = "qat_aes_cbc", .base.cra_priority = 4001, - .base.cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY, + .base.cra_flags = CRYPTO_ALG_ASYNC, .base.cra_blocksize = AES_BLOCK_SIZE, .base.cra_ctxsize = sizeof(struct qat_alg_skcipher_ctx), .base.cra_alignmask = 0, @@ -1658,7 +1658,7 @@ static struct skcipher_alg qat_skciphers[] = { { .base.cra_name = "ctr(aes)", .base.cra_driver_name = "qat_aes_ctr", .base.cra_priority = 4001, - .base.cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY, + .base.cra_flags = CRYPTO_ALG_ASYNC, .base.cra_blocksize = 1, .base.cra_ctxsize = sizeof(struct qat_alg_skcipher_ctx), .base.cra_alignmask = 0, @@ -1676,8 +1676,7 @@ static struct skcipher_alg qat_skciphers[] = { { .base.cra_name = "xts(aes)", .base.cra_driver_name = "qat_aes_xts", .base.cra_priority = 4001, - .base.cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK | - CRYPTO_ALG_ALLOCATES_MEMORY, + .base.cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK, .base.cra_blocksize = AES_BLOCK_SIZE, .base.cra_ctxsize = sizeof(struct qat_alg_skcipher_ctx), .base.cra_alignmask = 0,