From patchwork Tue Mar 4 00:01:03 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nicholas A. Bellinger" X-Patchwork-Id: 3758031 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 58C7D9F1EE for ; Tue, 4 Mar 2014 00:23:33 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 60FFD203B0 for ; Tue, 4 Mar 2014 00:23:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 68A1920396 for ; Tue, 4 Mar 2014 00:23:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755896AbaCDAX3 (ORCPT ); Mon, 3 Mar 2014 19:23:29 -0500 Received: from mail-oa0-f47.google.com ([209.85.219.47]:60121 "EHLO mail-oa0-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755867AbaCDAXQ (ORCPT ); Mon, 3 Mar 2014 19:23:16 -0500 Received: by mail-oa0-f47.google.com with SMTP id i11so1523728oag.20 for ; Mon, 03 Mar 2014 16:23:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=t3b6rC9qS56NPMsjXSJNZIwzbgkYuj+3jmLBWjMtbmE=; b=LQBGaQXpi5jlU9bopheyRzkIz2mdIEwpybk7eWp6Zd3fL0mfQqJpSkJhpD5NUpZI7B HvD04HcwFFQdXgdK8vI7BKqPo+pHawShxrDgPRsEJC3NqoYocBWmYmKMrmf8Hf04nJli Wb2pb6hXCQv0COzpw/rfqwrfW5u05FpQlS/2lVXlV2qgZc/fTKt83nqolj3VZCEMaeok 9UC/hShzLO7EhR/oeegdldzTzmV+khw4vIL1CV+eIvaifkspgFAiRLQaNmH7b2k+BUk3 OjQxmqWAMx0NlFPVLb47LYqOQOvdV7r6ryhSoahRaBjNYVh3zN+WryM1rhtjnhx6gkbG bLyQ== X-Gm-Message-State: ALoCoQkFsbxD0YBW/2Ysbg+FBwhBvbkmcS8kRKrxQLR99uDaWnNFp+YYao3NeS/hToORZtimYnKm X-Received: by 10.60.136.170 with SMTP id qb10mr109920oeb.34.1393892596274; Mon, 03 Mar 2014 16:23:16 -0800 (PST) Received: from localhost.localdomain (mail.linux-iscsi.org. [67.23.28.174]) by mx.google.com with ESMTPSA id lh7sm85755278oeb.6.2014.03.03.16.23.15 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 03 Mar 2014 16:23:15 -0800 (PST) From: "Nicholas A. Bellinger" To: target-devel Cc: linux-rdma , linux-scsi , Or Gerlitz , Sagi Grimberg , Nicholas Bellinger Subject: [PATCH 4/6] iser-target: Fix post_send_buf_count for RDMA READ/WRITE Date: Tue, 4 Mar 2014 00:01:03 +0000 Message-Id: <1393891265-22910-5-git-send-email-nab@daterainc.com> X-Mailer: git-send-email 1.7.2.5 In-Reply-To: <1393891265-22910-1-git-send-email-nab@daterainc.com> References: <1393891265-22910-1-git-send-email-nab@daterainc.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Nicholas Bellinger This patch fixes the incorrect setting of ->post_send_buf_count related to RDMA WRITEs + READs where isert_rdma_rw->send_wr_num was not being taken into account. This includes incrementing ->post_send_buf_count within isert_put_datain() + isert_get_dataout(), decrementing within __isert_send_completion() + isert_response_completion(), and clearing wr->send_wr_num within isert_completion_rdma_read() This is necessary because even though IB_SEND_SIGNALED is not set for RDMA WRITEs + READs, during a QP failure event the work requests will be returned with exception status from the TX completion queue. Cc: Sagi Grimberg Cc: Or Gerlitz Signed-off-by: Nicholas Bellinger --- drivers/infiniband/ulp/isert/ib_isert.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 8283f5a..c9d488f 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -1536,6 +1536,7 @@ isert_completion_rdma_read(struct iser_tx_desc *tx_desc, iscsit_stop_dataout_timer(cmd); device->unreg_rdma_mem(isert_cmd, isert_conn); cmd->write_data_done = wr->cur_rdma_length; + wr->send_wr_num = 0; pr_debug("Cmd: %p RDMA_READ comp calling execute_cmd\n", isert_cmd); spin_lock_bh(&cmd->istate_lock); @@ -1600,6 +1601,7 @@ isert_response_completion(struct iser_tx_desc *tx_desc, struct ib_device *ib_dev) { struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd; + struct isert_rdma_wr *wr = &isert_cmd->rdma_wr; if (cmd->i_state == ISTATE_SEND_TASKMGTRSP || cmd->i_state == ISTATE_SEND_LOGOUTRSP || @@ -1611,7 +1613,7 @@ isert_response_completion(struct iser_tx_desc *tx_desc, queue_work(isert_comp_wq, &isert_cmd->comp_work); return; } - atomic_dec(&isert_conn->post_send_buf_count); + atomic_sub(wr->send_wr_num + 1, &isert_conn->post_send_buf_count); cmd->i_state = ISTATE_SENT_STATUS; isert_completion_put(tx_desc, isert_cmd, ib_dev); @@ -1649,7 +1651,7 @@ __isert_send_completion(struct iser_tx_desc *tx_desc, case ISER_IB_RDMA_READ: pr_debug("isert_send_completion: Got ISER_IB_RDMA_READ:\n"); - atomic_dec(&isert_conn->post_send_buf_count); + atomic_sub(wr->send_wr_num, &isert_conn->post_send_buf_count); isert_completion_rdma_read(tx_desc, isert_cmd); break; default: @@ -2375,12 +2377,12 @@ isert_put_datain(struct iscsi_conn *conn, struct iscsi_cmd *cmd) isert_init_send_wr(isert_conn, isert_cmd, &isert_cmd->tx_desc.send_wr, true); - atomic_inc(&isert_conn->post_send_buf_count); + atomic_add(wr->send_wr_num + 1, &isert_conn->post_send_buf_count); rc = ib_post_send(isert_conn->conn_qp, wr->send_wr, &wr_failed); if (rc) { pr_warn("ib_post_send() failed for IB_WR_RDMA_WRITE\n"); - atomic_dec(&isert_conn->post_send_buf_count); + atomic_sub(wr->send_wr_num + 1, &isert_conn->post_send_buf_count); } pr_debug("Cmd: %p posted RDMA_WRITE + Response for iSER Data READ\n", isert_cmd); @@ -2408,12 +2410,12 @@ isert_get_dataout(struct iscsi_conn *conn, struct iscsi_cmd *cmd, bool recovery) return rc; } - atomic_inc(&isert_conn->post_send_buf_count); + atomic_add(wr->send_wr_num, &isert_conn->post_send_buf_count); rc = ib_post_send(isert_conn->conn_qp, wr->send_wr, &wr_failed); if (rc) { pr_warn("ib_post_send() failed for IB_WR_RDMA_READ\n"); - atomic_dec(&isert_conn->post_send_buf_count); + atomic_sub(wr->send_wr_num, &isert_conn->post_send_buf_count); } pr_debug("Cmd: %p posted RDMA_READ memory for ISER Data WRITE\n", isert_cmd);