From patchwork Wed Aug 24 12:42:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Kashyap Desai X-Patchwork-Id: 12953481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89636C32796 for ; Wed, 24 Aug 2022 12:43:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235001AbiHXMn4 (ORCPT ); Wed, 24 Aug 2022 08:43:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234401AbiHXMn4 (ORCPT ); Wed, 24 Aug 2022 08:43:56 -0400 Received: from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com [IPv6:2607:f8b0:4864:20::1043]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F17D0696C6 for ; Wed, 24 Aug 2022 05:43:54 -0700 (PDT) Received: by mail-pj1-x1043.google.com with SMTP id f21so16924662pjt.2 for ; Wed, 24 Aug 2022 05:43:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:message-id:date:subject:cc:to:from:from:to:cc; bh=RY5fJt5pBkeV13hstwSAh2HxJAvcMB4XLOhwdqrBZto=; b=JHiJrXymHXnTaVtLDq5QKQ9pIV0420KMrMpjwvpJ7s4vz/N7K3qCrXU4FxKG6AOGGh W0AXTmP4o+giflP3sEvQS6EJ8GGReXT6wdLSuTVd4DQtnht0GsoipTZVo6fCL8RMyl2p n7/HIy4mbIc7GhTtZZVIO0owcE2H5FV25eRNQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc; bh=RY5fJt5pBkeV13hstwSAh2HxJAvcMB4XLOhwdqrBZto=; b=p3lnc2fQs7BHz6k0yZnZhbHOAw8ieiU4KnTk5wqitthCp3bYS1Jm9kdr5zc0aCb5Uv ozjpNcrP9g9T17Ub0rf5ga+2AzQEtgPvKq5Wkz1CovOy0SFWQtdm1Mg1wXrCeRs1fAdW uvdIHnRYh5v7+Df55eQSgJgH62tm4+olU/rikeSxZ88O0Kix41++vO9t8e9chqj6zb7D xLnfXK/vpegm1DEJjBN6N76x+lOrGltp5UTZSiXE3AlGfR7ZbHFDKy/dFJBAQtrEiCfw PUfLz/o1qpmRAzYuXxpZP74fA3FKEQx+AwaCLEqwQIkDivgj1bJJYlB05vH2AfnLoU1/ 8jSA== X-Gm-Message-State: ACgBeo2nA1oEXPSuyILjBo8Nmk5VPC3C8D+N5ht1z3xZ68Cd9JVn15BF xTOzLI6WEutUUrk1gU210JjKq9FKrTfM6+VE6Wgvs3QpzV2iKYctcsGdo6d1swygFjctVCJOzNb X+vgHfZZ6NGtcacXO7gCx7IA6MXpZJ9dPt5iLOZwWUiWX05mdLPPIXQeHQkJTmseezKhwZk68I0 iW8XcfImxo X-Google-Smtp-Source: AA6agR5BNQDnou7KQLNn0IxyIyqdRVFsnWVuQeikKDVeJFSiZqQnUT8Vbk0uJJMpBBGSNdXg0DzwmA== X-Received: by 2002:a17:903:1ce:b0:16f:145c:a842 with SMTP id e14-20020a17090301ce00b0016f145ca842mr28312933plh.83.1661345034031; Wed, 24 Aug 2022 05:43:54 -0700 (PDT) Received: from amd_smc.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id j2-20020a63fc02000000b0041a615381d5sm11010769pgi.4.2022.08.24.05.43.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Aug 2022 05:43:52 -0700 (PDT) From: Kashyap Desai To: linux-rdma@vger.kernel.org Cc: selvin.xavier@broadcom.com, andrew.gospodarek@broadcom.com, Kashyap Desai , "Jason Gunthorpe" , "Max Gurtovoy" , "Sagi Grimberg" Subject: [PATCH RFC] IB/iser: add task reference counter for tx commands Date: Wed, 24 Aug 2022 18:12:36 +0530 Message-Id: <20220824124236.812395-1-kashyap.desai@broadcom.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org In [0], Max Gurtovoy already explained a future plan to add reference count. Below example has Initiator Task Tag = 0x1e. We are tracking the same ITT.         [177617.255969]  session508: iscsi_prep_scsi_cmd_pdu iscsi prep [write cid 0 sc 000000009d6ff976 cdb 0x2a itt 0x1e len 8192 cmdsn 90 win 64]         [177617.255982] iser: iscsi_iser_task_xmit: cmd [itt 1e total 8192 imm 8192 unsol_data 0 (1) >>  [177617.255985] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x1e]         [177617.255992] iser: iser_reg_dma: Single DMA entry: lkey=0xffffffff, rkey=0x0, addr=0x9c5de000, length=0x2000         [177617.255995] iser: iser_prepare_write_cmd: Cmd itt:30, WRITE, adding imm.data sz: 8192         [177617.256002] iser: iser_send_command: from iser_send_command iser_task ffff9e39dad96898 tx_count 1 (2) >>  [177617.256016] iser: iser_cmd_comp: from iser_cmd_comp iser_task ffff9e39dad96898 tx_count 0 (3) >>  [177617.256045] iser: iser_task_rsp: op 0x21 itt 0x1e dlen 0         [177617.256049]  session508: __iscsi_complete_pdu [op 0x21 cid 0 itt 0x1e len 0]         [177617.256052]  session508: iscsi_scsi_cmd_rsp cmd rsp done [sc 000000009d6ff976 res 0 itt 0x1e]         [177617.256055]  session508: iscsi_complete_task complete task itt 0x1e state 3 sc 000000009d6ff976 (4) >>  [177617.256057]  session508: iscsi_free_task freeing task itt 0x1e state 1 sc 000000009d6ff976 (5) >>  [177617.256067] bnxt_en 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0x41137800 flags=0x0000]         [177617.256068] bnxt_en 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0xc1595098 flags=0x0000] (1) Is when iser stack has posted TX DESC type = ISCSI_TX_SCSI_COMMAND having two SGL.     sgl[0] is contain iscsi_header and sgl[1] is scsi data (of length 8K)     of scsi command cdb = WRITE_10 (2) In a good case, TX completion is received,     but in some cases TX completion is not yet received to the iser stack. (3) Is when iser stack gets completion of RX DESC of ITT = 0x1e.     This is actual scsi completion of TX_DESC from target. (4) Is when the iser stack destroys all TX DESC resources associated with ITT = 0x1e. (5) Is when HCA is still accessing WQE entry of ITT = 0x1e (possible in retransmission path).     After (4), sgl[0] and sgl[1] dma addresses are unmapped and not allowed to be used by HCA. Get the iscsi_task reference count if TX DESC is successfully posted. Put the iscsi_task reference count once TX_DESC is completed. This method will make sure iscsi_free_task will be called only after TX and RX DESC received for the task->itt. This patch depends upon [0] as it required signaled completion. [0] https://patchwork.kernel.org/project/linux-rdma/patch/20211215135721.3662-5-mgurtovoy@nvidia.com/ Cc: "Jason Gunthorpe" Cc: "Max Gurtovoy" Cc: "Sagi Grimberg" Signed-off-by: Kashyap Desai --- drivers/infiniband/ulp/iser/iscsi_iser.c | 5 ++++- drivers/infiniband/ulp/iser/iser_initiator.c | 6 ++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c index 620ae5b2d80d..f1704fef9dc8 100644 --- a/drivers/infiniband/ulp/iser/iscsi_iser.c +++ b/drivers/infiniband/ulp/iser/iscsi_iser.c @@ -339,9 +339,12 @@ static int iscsi_iser_task_xmit(struct iscsi_task *task) /* Send the cmd PDU */ if (!iser_task->command_sent) { + iscsi_get_task(task); error = iser_send_command(conn, task); - if (error) + if (error) { + iscsi_put_task(task); goto iscsi_iser_task_xmit_exit; + } iser_task->command_sent = 1; } diff --git a/drivers/infiniband/ulp/iser/iser_initiator.c b/drivers/infiniband/ulp/iser/iser_initiator.c index 7b83f48f60c5..1f0b1601b3b3 100644 --- a/drivers/infiniband/ulp/iser/iser_initiator.c +++ b/drivers/infiniband/ulp/iser/iser_initiator.c @@ -669,6 +669,12 @@ void iser_task_rsp(struct ib_cq *cq, struct ib_wc *wc) void iser_cmd_comp(struct ib_cq *cq, struct ib_wc *wc) { + struct iser_tx_desc *desc = iser_tx(wc->wr_cqe); + struct iscsi_task *task; + task = (void *)desc - sizeof(struct iscsi_task); + + iscsi_put_task(task); + if (unlikely(wc->status != IB_WC_SUCCESS)) iser_err_comp(wc, "command"); }