From patchwork Tue Feb 14 18:56:31 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 9572519 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id CBB40601E7 for ; Tue, 14 Feb 2017 18:56:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A0BC127DCD for ; Tue, 14 Feb 2017 18:56:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 935032845E; Tue, 14 Feb 2017 18:56:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DF7BF27DCD for ; Tue, 14 Feb 2017 18:56:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752869AbdBNS4q (ORCPT ); Tue, 14 Feb 2017 13:56:46 -0500 Received: from esa1.hgst.iphmx.com ([68.232.141.245]:26512 "EHLO esa1.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752860AbdBNS4p (ORCPT ); Tue, 14 Feb 2017 13:56:45 -0500 X-IronPort-AV: E=Sophos;i="5.33,348,1477929600"; d="scan'208";a="86579469" Received: from unknown (HELO milsmgep14.sandisk.com) ([63.163.107.225]) by ob1.hgst.iphmx.com with ESMTP; 15 Feb 2017 02:56:43 +0800 Received: from MILHUBIP04.sdcorp.global.sandisk.com (Unknown_Domain [10.201.67.162]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by (Symantec Messaging Gateway) with SMTP id 08.9B.18895.BE253A85; Tue, 14 Feb 2017 10:56:43 -0800 (PST) Received: from milsmgip12.sandisk.com (10.177.8.100) by MILHUBIP04.sdcorp.global.sandisk.com (10.177.9.97) with Microsoft SMTP Server id 14.3.319.2; Tue, 14 Feb 2017 10:56:40 -0800 X-AuditID: 0ac94371-0fadb980000049cf-8a-58a352eb7400 Received: from exp-402881.sdcorp.global.sandisk.com ( [10.177.9.6]) by (Symantec Messaging Gateway) with SMTP id 51.D6.18148.8E253A85; Tue, 14 Feb 2017 10:56:40 -0800 (PST) From: Bart Van Assche To: Doug Ledford CC: , Bart Van Assche , Israel Rukshin , Max Gurtovoy , Laurence Oberman , Steve Feeley , Subject: [PATCH v2 3/8] IB/srp: Fix race conditions related to task management Date: Tue, 14 Feb 2017 10:56:31 -0800 Message-ID: <20170214185636.29250-4-bart.vanassche@sandisk.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170214185636.29250-1-bart.vanassche@sandisk.com> References: <20170214185636.29250-1-bart.vanassche@sandisk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrNLMWRmVeSWpSXmKPExsXCddJ5ke7roMURBkumG1m8PP+B1eLG4rUs Fs8O9bJYnP/wisni9KMD7BYLNj5idGDzeDb9MJPH+31X2Tw+b5ILYI7isklJzcksSy3St0vg yljc08pasFCvYt3kZawNjP1qXYycHBICJhLXbyxi62Lk4hASWMok0bzvNxOEs4NR4siiWaxd jBxgVR9fSIE0CAlsZJR4tSkFxGYTMJL49n4mC4gtIqAmsenVInaQXmaBDiaJ/2ePsoIkhAX8 JFYsWc0OYrMIqEpsO9UNZvMK2EvcPt/ACnGFvMSutotgNqeAg8TMbwtZIZbZS5ztnwM2VEJg GqvE1dM/oJoFJU7OfAK2mVlAQuLgixfMEA3qEieXzGeawCg0C0nZLCRlCxiZVjGK5WbmFOem pxYYmugVJ+alZBZn6yXn525ihAR54Q7G17e9DzEKcDAq8fCekF0cIcSaWFZcmXuIUYKDWUmE V8MBKMSbklhZlVqUH19UmpNafIhRmoNFSZw3S3ZqhJBAemJJanZqakFqEUyWiYNTqoHxWE5t 0RvJxzZfKw8dfDv921PT23832ir1q/cIrgpOU2jv33RWTy02P1vohtKE7dMXTpvku5J7e/n7 u545mxlFf8xfFfnp3engp47fy3nfLH8/yctnyfWp8z9eULrqUKTnd1NFgbvyudQly1dx/O2z z5fk3ZZ8fTTx4UzJPayxvt9X2Lx5oHpqkhJLcUaioRZzUXEiACtt3nRuAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrAJMWRmVeSWpSXmKPExsXCtZGTTfdF0OIIg4PzOS0O/mxjtHh5/gOr xY3Fa1ksnh3qZbE4/+EVk8XpRwfYLRZsfMRo8fXlC2YHDo9n0w8zebzfd5XNY9qa80wenzfJ BbBEcdmkpOZklqUW6dslcGUs7mllLVioV7Fu8jLWBsZ+tS5GDg4JAROJjy+kuhi5OIQE1jNK TGxbytjFyMnBJmAk8e39TBYQW0RATWLTq0XsIEXMAl1MEufb/oIVCQv4SaxYspodxGYRUJXY dqobzOYVsJd48O0aE4gtISAvsavtIiuIzSngIDHz20IwWwio5mz/HPYJjNwLGBlWMYrlZuYU 56ZnFhga6RUn5qVkFmfrJefnbmKEhEnUDsbrE80PMTJxcEo1MAptvLaTZZ7xV87XbpHy+Yre +ld0i6MmfJfb/yKspOpyzr1H8qlMWhb3thyTW+zVdXq+6bWYLwUdqoeS3Q0r7k1dt2xpa0Zm h87q98Vy2fJuk46eZ15j7SOsVWGjkPIuLV+RdZIsV4DK5vrLawyfTzh5YN7JlW/fTuh8dtUy zNV291WDnWfjbJVYijMSDbWYi4oTASTEqZjDAQAA MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Avoid that srp_process_rsp() overwrites the status information in ch if the SRP target response timed out and processing of another task management function has already started. Avoid that issuing multiple task management functions concurrently triggers list corruption. This patch prevents that the following stack trace appears in the system log: WARNING: CPU: 8 PID: 9269 at lib/list_debug.c:52 __list_del_entry_valid+0xbc/0xc0 list_del corruption. prev->next should be ffffc90004bb7b00, but was ffff8804052ecc68 CPU: 8 PID: 9269 Comm: sg_reset Tainted: G W 4.10.0-rc7-dbg+ #3 Call Trace: dump_stack+0x68/0x93 __warn+0xc6/0xe0 warn_slowpath_fmt+0x4a/0x50 __list_del_entry_valid+0xbc/0xc0 wait_for_completion_timeout+0x12e/0x170 srp_send_tsk_mgmt+0x1ef/0x2d0 [ib_srp] srp_reset_device+0x5b/0x110 [ib_srp] scsi_ioctl_reset+0x1c7/0x290 scsi_ioctl+0x12a/0x420 sd_ioctl+0x9d/0x100 blkdev_ioctl+0x51e/0x9f0 block_ioctl+0x38/0x40 do_vfs_ioctl+0x8f/0x700 SyS_ioctl+0x3c/0x70 entry_SYSCALL_64_fastpath+0x18/0xad Signed-off-by: Bart Van Assche Cc: Israel Rukshin Cc: Max Gurtovoy Cc: Laurence Oberman Cc: Steve Feeley Cc: --- drivers/infiniband/ulp/srp/ib_srp.c | 45 ++++++++++++++++++++++++------------- drivers/infiniband/ulp/srp/ib_srp.h | 1 + 2 files changed, 30 insertions(+), 16 deletions(-) diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index 12027e3a4dec..8d6ce03da1e8 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -1884,12 +1884,17 @@ static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp) if (unlikely(rsp->tag & SRP_TAG_TSK_MGMT)) { spin_lock_irqsave(&ch->lock, flags); ch->req_lim += be32_to_cpu(rsp->req_lim_delta); + if (rsp->tag == ch->tsk_mgmt_tag) { + ch->tsk_mgmt_status = -1; + if (be32_to_cpu(rsp->resp_data_len) >= 4) + ch->tsk_mgmt_status = rsp->data[3]; + complete(&ch->tsk_mgmt_done); + } else { + shost_printk(KERN_ERR, target->scsi_host, + "Received tsk mgmt response too late for tag %#llx\n", + rsp->tag); + } spin_unlock_irqrestore(&ch->lock, flags); - - ch->tsk_mgmt_status = -1; - if (be32_to_cpu(rsp->resp_data_len) >= 4) - ch->tsk_mgmt_status = rsp->data[3]; - complete(&ch->tsk_mgmt_done); } else { scmnd = scsi_host_find_tag(target->scsi_host, rsp->tag); if (scmnd && scmnd->host_scribble) { @@ -2528,19 +2533,18 @@ srp_change_queue_depth(struct scsi_device *sdev, int qdepth) } static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun, - u8 func) + u8 func, u8 *status) { struct srp_target_port *target = ch->target; struct srp_rport *rport = target->rport; struct ib_device *dev = target->srp_host->srp_dev->dev; struct srp_iu *iu; struct srp_tsk_mgmt *tsk_mgmt; + int res; if (!ch->connected || target->qp_in_error) return -1; - init_completion(&ch->tsk_mgmt_done); - /* * Lock the rport mutex to avoid that srp_create_ch_ib() is * invoked while a task management function is being sent. @@ -2563,10 +2567,16 @@ static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun, tsk_mgmt->opcode = SRP_TSK_MGMT; int_to_scsilun(lun, &tsk_mgmt->lun); - tsk_mgmt->tag = req_tag | SRP_TAG_TSK_MGMT; tsk_mgmt->tsk_mgmt_func = func; tsk_mgmt->task_tag = req_tag; + spin_lock_irq(&ch->lock); + ch->tsk_mgmt_tag = (ch->tsk_mgmt_tag + 1) | SRP_TAG_TSK_MGMT; + tsk_mgmt->tag = ch->tsk_mgmt_tag; + spin_unlock_irq(&ch->lock); + + init_completion(&ch->tsk_mgmt_done); + ib_dma_sync_single_for_device(dev, iu->dma, sizeof *tsk_mgmt, DMA_TO_DEVICE); if (srp_post_send(ch, iu, sizeof(*tsk_mgmt))) { @@ -2575,13 +2585,15 @@ static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun, return -1; } + res = wait_for_completion_timeout(&ch->tsk_mgmt_done, + msecs_to_jiffies(SRP_ABORT_TIMEOUT_MS)); + if (res > 0 && status) + *status = ch->tsk_mgmt_status; mutex_unlock(&rport->mutex); - if (!wait_for_completion_timeout(&ch->tsk_mgmt_done, - msecs_to_jiffies(SRP_ABORT_TIMEOUT_MS))) - return -1; + WARN_ON_ONCE(res < 0); - return 0; + return res > 0 ? 0 : -1; } static int srp_abort(struct scsi_cmnd *scmnd) @@ -2607,7 +2619,7 @@ static int srp_abort(struct scsi_cmnd *scmnd) shost_printk(KERN_ERR, target->scsi_host, "Sending SRP abort for tag %#x\n", tag); if (srp_send_tsk_mgmt(ch, tag, scmnd->device->lun, - SRP_TSK_ABORT_TASK) == 0) + SRP_TSK_ABORT_TASK, NULL) == 0) ret = SUCCESS; else if (target->rport->state == SRP_RPORT_LOST) ret = FAST_IO_FAIL; @@ -2625,14 +2637,15 @@ static int srp_reset_device(struct scsi_cmnd *scmnd) struct srp_target_port *target = host_to_target(scmnd->device->host); struct srp_rdma_ch *ch; int i; + u8 status; shost_printk(KERN_ERR, target->scsi_host, "SRP reset_device called\n"); ch = &target->ch[0]; if (srp_send_tsk_mgmt(ch, SRP_TAG_NO_REQ, scmnd->device->lun, - SRP_TSK_LUN_RESET)) + SRP_TSK_LUN_RESET, &status)) return FAILED; - if (ch->tsk_mgmt_status) + if (status) return FAILED; for (i = 0; i < target->ch_count; i++) { diff --git a/drivers/infiniband/ulp/srp/ib_srp.h b/drivers/infiniband/ulp/srp/ib_srp.h index 21c69695f9d4..32ed40db3ca2 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.h +++ b/drivers/infiniband/ulp/srp/ib_srp.h @@ -163,6 +163,7 @@ struct srp_rdma_ch { int max_ti_iu_len; int comp_vector; + u64 tsk_mgmt_tag; struct completion tsk_mgmt_done; u8 tsk_mgmt_status; bool connected;