From patchwork Fri Feb 10 23:56:05 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 9567615 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1B32F602B6 for ; Fri, 10 Feb 2017 23:56:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 073B8285D2 for ; Fri, 10 Feb 2017 23:56:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F052A285EA; Fri, 10 Feb 2017 23:56:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6DE49285D2 for ; Fri, 10 Feb 2017 23:56:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751523AbdBJX4W (ORCPT ); Fri, 10 Feb 2017 18:56:22 -0500 Received: from esa1.hgst.iphmx.com ([68.232.141.245]:7613 "EHLO esa1.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750874AbdBJX4T (ORCPT ); Fri, 10 Feb 2017 18:56:19 -0500 X-IronPort-AV: E=Sophos;i="5.33,348,1477929600"; d="scan'208";a="83251593" Received: from unknown (HELO milsmgep14.sandisk.com) ([63.163.107.225]) by ob1.hgst.iphmx.com with ESMTP; 11 Feb 2017 07:56:18 +0800 Received: from MILHUBIP04.sdcorp.global.sandisk.com (Unknown_Domain [10.201.67.162]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by (Symantec Messaging Gateway) with SMTP id 1D.D5.18895.2235E985; Fri, 10 Feb 2017 15:56:18 -0800 (PST) Received: from milsmgip11.sandisk.com (10.177.9.6) by MILHUBIP04.sdcorp.global.sandisk.com (10.177.9.97) with Microsoft SMTP Server id 14.3.319.2; Fri, 10 Feb 2017 15:56:15 -0800 X-AuditID: 0ac94371-0fadb980000049cf-3a-589e53225b7b Received: from exp-402881.sdcorp.global.sandisk.com ( [10.177.8.100]) by (Symantec Messaging Gateway) with SMTP id 7D.A4.11415.E135E985; Fri, 10 Feb 2017 15:56:14 -0800 (PST) From: Bart Van Assche To: Doug Ledford CC: , Bart Van Assche , Israel Rukshin , Max Gurtovoy , Laurence Oberman , Steve Feeley , Subject: [PATCH 2/8] IB/srp: Fix race conditions related to task management Date: Fri, 10 Feb 2017 15:56:05 -0800 Message-ID: <20170210235611.3243-3-bart.vanassche@sandisk.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170210235611.3243-1-bart.vanassche@sandisk.com> References: <20170210235611.3243-1-bart.vanassche@sandisk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrLLMWRmVeSWpSXmKPExsXCddJ5ka5S8LwIg+cdQhYvz39gtbixeC2L xbNDvSwW5z+8YrI4/egAu8WCjY8YHdg8nk0/zOTxft9VNo/Pm+QCmKO4bFJSczLLUov07RK4 Mt62zGcs6NSrOPm0h6mB8adqFyMnh4SAicTySwvZuhi5OIQEljJJ3HhzmBnC2cYoceLXUUaY qieb50NVbWaU+L7vOCtIgk3ASOLb+5ksILaIgJrEpleL2EGKmAU6mCT+nz0KViQs4C3x9U8z mM0ioCrx6flOJhCbV8BOYu7SwywQG+QldrVdBKvhFLCX+L5sGpgtBFTz5usnVpChEgLTWCVW TjjIAtEsKHFy5hMwm1lAQuLgixfMEA3qEieXzGeawCg0C0nZLCRlCxiZVjGK5WbmFOempxYY mugVJ+alZBZn6yXn525ihAR64Q7G17e9DzEKcDAq8fByHJ0bIcSaWFZcmXuIUYKDWUmEN8l1 XoQQb0piZVVqUX58UWlOavEhRmkOFiVx3izZqRFCAumJJanZqakFqUUwWSYOTqkGRsP60FUb us/L/zJZJN2yqf/yxQIH79Z/Pomcqm+/vr1m0DH7PevNytL5fD9+TXzUwOZgl7U3qL7taPRV 3rcrzglO9r+3tvSPbEBHxJUXhUtsllqbSWqv+Lrlt/xiidNLL32ScLy5UPx7DJN8lPe9u/uu bDzs8pr1Rd7/kGsZEd2q/nOr9xTZnlZiKc5INNRiLipOBAB8I6E1cAIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrIJMWRmVeSWpSXmKPExsXCtZEjRVcueF6EwbNGK4uDP9sYLV6e/8Bq cWPxWhaLZ4d6WSzOf3jFZHH60QF2iwUbHzFafH35gtmBw+PZ9MNMHu/3XWXzmLbmPJPH501y ASxRXDYpqTmZZalF+nYJXBlvW+YzFnTqVZx82sPUwPhTtYuRk0NCwETiyeb5bF2MXBxCAhsZ JZp2bmMDSbAJGEl8ez+TBcQWEVCT2PRqETtIEbNAF5PE+ba/jCAJYQFvia9/mllBbBYBVYlP z3cygdi8AnYS3yecY4LYIC+xq+0iWA2ngL3E92XTwGwhoJo3Xz+xTmDkXsDIsIpRLDczpzg3 PbPA0FCvODEvJbM4Wy85P3cTIzhQOCN3MD6daH6IkYmDU6qBsXfflnvh2Zz+IkUckwoEi7++ dpq2LyJ/k/Wy0xXW79OKHzQpe0x8v+bvylDeA/aZ6zNO+L09rH3PcPf0VSa+6Qd1fCXPBKdX 5cxb9XsGwwpB9r2bcz54HBJP+L9K98s9/2XVZcb2Wu8OhS5+qL8z9lHq0eONlb037E29ZF9k cfE1TTOZI7WhQImlOCPRUIu5qDgRAC1dHaDEAQAA MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Avoid that srp_process_rsp() overwrites the status information in ch if the SRP target response timed out and processing of another task management function has already started. Avoid that issuing multiple task management functions concurrently triggers list corruption. This patch prevents that the following stack trace appears in the system log: WARNING: CPU: 8 PID: 9269 at lib/list_debug.c:52 __list_del_entry_valid+0xbc/0xc0 list_del corruption. prev->next should be ffffc90004bb7b00, but was ffff8804052ecc68 CPU: 8 PID: 9269 Comm: sg_reset Tainted: G W 4.10.0-rc7-dbg+ #3 Call Trace: dump_stack+0x68/0x93 __warn+0xc6/0xe0 warn_slowpath_fmt+0x4a/0x50 __list_del_entry_valid+0xbc/0xc0 wait_for_completion_timeout+0x12e/0x170 srp_send_tsk_mgmt+0x1ef/0x2d0 [ib_srp] srp_reset_device+0x5b/0x110 [ib_srp] scsi_ioctl_reset+0x1c7/0x290 scsi_ioctl+0x12a/0x420 sd_ioctl+0x9d/0x100 blkdev_ioctl+0x51e/0x9f0 block_ioctl+0x38/0x40 do_vfs_ioctl+0x8f/0x700 SyS_ioctl+0x3c/0x70 entry_SYSCALL_64_fastpath+0x18/0xad Signed-off-by: Bart Van Assche Cc: Israel Rukshin Cc: Max Gurtovoy Cc: Laurence Oberman Cc: Steve Feeley Cc: --- drivers/infiniband/ulp/srp/ib_srp.c | 45 ++++++++++++++++++++++++------------- drivers/infiniband/ulp/srp/ib_srp.h | 1 + 2 files changed, 30 insertions(+), 16 deletions(-) diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index 4068d34f5427..511eb4b2e6e0 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -1889,12 +1889,17 @@ static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp) if (unlikely(rsp->tag & SRP_TAG_TSK_MGMT)) { spin_lock_irqsave(&ch->lock, flags); ch->req_lim += be32_to_cpu(rsp->req_lim_delta); + if (rsp->tag == ch->tsk_mgmt_tag) { + ch->tsk_mgmt_status = -1; + if (be32_to_cpu(rsp->resp_data_len) >= 4) + ch->tsk_mgmt_status = rsp->data[3]; + complete(&ch->tsk_mgmt_done); + } else { + shost_printk(KERN_ERR, target->scsi_host, + "Received tsk mgmt response too late for tag %#llx\n", + rsp->tag); + } spin_unlock_irqrestore(&ch->lock, flags); - - ch->tsk_mgmt_status = -1; - if (be32_to_cpu(rsp->resp_data_len) >= 4) - ch->tsk_mgmt_status = rsp->data[3]; - complete(&ch->tsk_mgmt_done); } else { scmnd = scsi_host_find_tag(target->scsi_host, rsp->tag); if (scmnd) { @@ -2538,19 +2543,18 @@ srp_change_queue_depth(struct scsi_device *sdev, int qdepth) } static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun, - u8 func) + u8 func, u8 *status) { struct srp_target_port *target = ch->target; struct srp_rport *rport = target->rport; struct ib_device *dev = target->srp_host->srp_dev->dev; struct srp_iu *iu; struct srp_tsk_mgmt *tsk_mgmt; + int res; if (!ch->connected || target->qp_in_error) return -1; - init_completion(&ch->tsk_mgmt_done); - /* * Lock the rport mutex to avoid that srp_create_ch_ib() is * invoked while a task management function is being sent. @@ -2573,10 +2577,16 @@ static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun, tsk_mgmt->opcode = SRP_TSK_MGMT; int_to_scsilun(lun, &tsk_mgmt->lun); - tsk_mgmt->tag = req_tag | SRP_TAG_TSK_MGMT; tsk_mgmt->tsk_mgmt_func = func; tsk_mgmt->task_tag = req_tag; + spin_lock_irq(&ch->lock); + ch->tsk_mgmt_tag = (ch->tsk_mgmt_tag + 1) | SRP_TAG_TSK_MGMT; + tsk_mgmt->tag = ch->tsk_mgmt_tag; + spin_unlock_irq(&ch->lock); + + init_completion(&ch->tsk_mgmt_done); + ib_dma_sync_single_for_device(dev, iu->dma, sizeof *tsk_mgmt, DMA_TO_DEVICE); if (srp_post_send(ch, iu, sizeof(*tsk_mgmt))) { @@ -2585,13 +2595,15 @@ static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun, return -1; } + res = wait_for_completion_timeout(&ch->tsk_mgmt_done, + msecs_to_jiffies(SRP_ABORT_TIMEOUT_MS)); + if (res > 0 && status) + *status = ch->tsk_mgmt_status; mutex_unlock(&rport->mutex); - if (!wait_for_completion_timeout(&ch->tsk_mgmt_done, - msecs_to_jiffies(SRP_ABORT_TIMEOUT_MS))) - return -1; + WARN_ON_ONCE(res < 0); - return 0; + return res > 0 ? 0 : -1; } static int srp_abort(struct scsi_cmnd *scmnd) @@ -2617,7 +2629,7 @@ static int srp_abort(struct scsi_cmnd *scmnd) shost_printk(KERN_ERR, target->scsi_host, "Sending SRP abort for tag %#x\n", tag); if (srp_send_tsk_mgmt(ch, tag, scmnd->device->lun, - SRP_TSK_ABORT_TASK) == 0) + SRP_TSK_ABORT_TASK, NULL) == 0) ret = SUCCESS; else if (target->rport->state == SRP_RPORT_LOST) ret = FAST_IO_FAIL; @@ -2635,14 +2647,15 @@ static int srp_reset_device(struct scsi_cmnd *scmnd) struct srp_target_port *target = host_to_target(scmnd->device->host); struct srp_rdma_ch *ch; int i; + u8 status; shost_printk(KERN_ERR, target->scsi_host, "SRP reset_device called\n"); ch = &target->ch[0]; if (srp_send_tsk_mgmt(ch, SRP_TAG_NO_REQ, scmnd->device->lun, - SRP_TSK_LUN_RESET)) + SRP_TSK_LUN_RESET, &status)) return FAILED; - if (ch->tsk_mgmt_status) + if (status) return FAILED; for (i = 0; i < target->ch_count; i++) { diff --git a/drivers/infiniband/ulp/srp/ib_srp.h b/drivers/infiniband/ulp/srp/ib_srp.h index 21c69695f9d4..32ed40db3ca2 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.h +++ b/drivers/infiniband/ulp/srp/ib_srp.h @@ -163,6 +163,7 @@ struct srp_rdma_ch { int max_ti_iu_len; int comp_vector; + u64 tsk_mgmt_tag; struct completion tsk_mgmt_done; u8 tsk_mgmt_status; bool connected;