From patchwork Fri Jun 22 21:52:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 10483113 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B063560388 for ; Fri, 22 Jun 2018 21:53:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9D9622903B for ; Fri, 22 Jun 2018 21:53:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 923E229042; Fri, 22 Jun 2018 21:53:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 68F6D2903B for ; Fri, 22 Jun 2018 21:53:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933568AbeFVVxO (ORCPT ); Fri, 22 Jun 2018 17:53:14 -0400 Received: from esa2.hgst.iphmx.com ([68.232.143.124]:64357 "EHLO esa2.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754546AbeFVVxL (ORCPT ); Fri, 22 Jun 2018 17:53:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1529704417; x=1561240417; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Vbey8qYUkwkyc6nqK74haI1Z5AOgBuqLOEyDvrFtAxk=; b=TNGIIDn/pQuTDrFNdn9B6PLexp4RpkGVeENwHW/wrmA14lPRnZcjnf9u YqZdw9VpPgB9S1P72Supw1TXATYklV6B4kTNHtDwIb6HsykIrHbI/YNc4 xijMJ/pEZFWLIyrzKX+xtO4Of+jy/FC6Rwle0GmjL7IjguCMdpbNf/EHF TSjrXOclkqaY6ieHYtiXO8NMNLeca3UzJY+S+FUIeG7Ik2e4Sc/vr00VO dnX8VlpWkbizMFxAsTiFSGJ6DfmHBO+u2L54/uBeFsVC5dP6vLW0H0ohW 9bn1ffsyKa2un1+4YS5qz/tjw7+4tJIO099XhddKrpXC+pEwm8QYHyd1Q g==; X-IronPort-AV: E=Sophos;i="5.51,259,1526313600"; d="scan'208";a="178350996" Received: from uls-op-cesaip01.wdc.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 23 Jun 2018 05:53:34 +0800 Received: from uls-op-cesaip01.wdc.com ([10.248.3.36]) by uls-op-cesaep01.wdc.com with ESMTP; 22 Jun 2018 14:42:56 -0700 Received: from thinkpad-bart.sdcorp.global.sandisk.com ([10.111.67.248]) by uls-op-cesaip01.wdc.com with ESMTP; 22 Jun 2018 14:53:10 -0700 From: Bart Van Assche To: "Martin K . Petersen" Cc: Christoph Hellwig , target-devel@vger.kernel.org, Bart Van Assche , Hannes Reinecke , Sagi Grimberg Subject: [PATCH 06/20] target: Make the session shutdown code also wait for commands that are being aborted Date: Fri, 22 Jun 2018 14:52:53 -0700 Message-Id: <20180622215307.8758-7-bart.vanassche@wdc.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180622215307.8758-1-bart.vanassche@wdc.com> References: <20180622215307.8758-1-bart.vanassche@wdc.com> Sender: target-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Target drivers must call target_sess_cmd_list_set_waiting() and target_wait_for_sess_cmds() before freeing a session. Since freeing a session is only safe after all commands that are associated with a session have finished, make target_wait_for_sess_cmds() also wait for commands that are being aborted. Instead of setting a flag in each pending command from target_sess_cmd_list_set_waiting() and waiting in target_wait_for_sess_cmds() on a per-command completion, only set a per-session flag in the former function and wait on a per-session completion in the latter function. This change is safe because once a SCSI initiator system has submitted a command a target system is always allowed to execute it to completion. See also commit 0f4a943168f3 ("target: Fix remote-port TMR ABORT + se_cmd fabric stop"). This patch is based on the following two patches: * Bart Van Assche, target: Simplify session shutdown code, February 19, 2015 (https://github.com/bvanassche/linux/commit/8df5463d7d7619f2f1b70cfe5172eaef0aa52815). * Christoph Hellwig, target: Rework session shutdown code, December 7, 2015 (http://thread.gmane.org/gmane.linux.scsi.target.devel/10695). Signed-off-by: Bart Van Assche Reviewed-by: Mike Christie Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Sagi Grimberg --- drivers/target/target_core_tmr.c | 5 +- drivers/target/target_core_transport.c | 77 +++++++------------------- include/target/target_core_base.h | 3 +- 3 files changed, 24 insertions(+), 61 deletions(-) diff --git a/drivers/target/target_core_tmr.c b/drivers/target/target_core_tmr.c index 9c7bc1ca341a..da8125dd3a4c 100644 --- a/drivers/target/target_core_tmr.c +++ b/drivers/target/target_core_tmr.c @@ -142,7 +142,7 @@ static bool __target_check_io_state(struct se_cmd *se_cmd, return false; } } - if (sess->sess_tearing_down || se_cmd->cmd_wait_set) { + if (sess->sess_tearing_down) { pr_debug("Attempted to abort io tag: %llu already shutdown," " skipping\n", se_cmd->tag); spin_unlock(&se_cmd->t_state_lock); @@ -187,7 +187,6 @@ void core_tmr_abort_task( if (!__target_check_io_state(se_cmd, se_sess, 0)) continue; - list_del_init(&se_cmd->se_cmd_list); spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags); cancel_work_sync(&se_cmd->work); @@ -259,7 +258,7 @@ static void core_tmr_drain_tmr_list( spin_unlock(&sess->sess_cmd_lock); continue; } - if (sess->sess_tearing_down || cmd->cmd_wait_set) { + if (sess->sess_tearing_down) { spin_unlock(&cmd->t_state_lock); spin_unlock(&sess->sess_cmd_lock); continue; diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c index 595b2f12ca6a..087dddfc5c35 100644 --- a/drivers/target/target_core_transport.c +++ b/drivers/target/target_core_transport.c @@ -235,8 +235,8 @@ void transport_init_session(struct se_session *se_sess) INIT_LIST_HEAD(&se_sess->sess_list); INIT_LIST_HEAD(&se_sess->sess_acl_list); INIT_LIST_HEAD(&se_sess->sess_cmd_list); - INIT_LIST_HEAD(&se_sess->sess_wait_list); spin_lock_init(&se_sess->sess_cmd_lock); + init_waitqueue_head(&se_sess->cmd_list_wq); } EXPORT_SYMBOL(transport_init_session); @@ -2728,13 +2728,15 @@ static void target_release_cmd_kref(struct kref *kref) if (se_sess) { spin_lock_irqsave(&se_sess->sess_cmd_lock, flags); list_del_init(&se_cmd->se_cmd_list); + if (list_empty(&se_sess->sess_cmd_list)) + wake_up(&se_sess->cmd_list_wq); spin_lock(&se_cmd->t_state_lock); fabric_stop = (se_cmd->transport_state & CMD_T_FABRIC_STOP) && (se_cmd->transport_state & CMD_T_ABORTED); spin_unlock(&se_cmd->t_state_lock); - if (se_cmd->cmd_wait_set || fabric_stop) { + if (fabric_stop) { spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags); target_free_cmd_mem(se_cmd); complete(&se_cmd->cmd_wait_comp); @@ -2863,78 +2865,41 @@ void target_show_cmd(const char *pfx, struct se_cmd *cmd) EXPORT_SYMBOL(target_show_cmd); /** - * target_sess_cmd_list_set_waiting - Flag all commands in - * sess_cmd_list to complete cmd_wait_comp. Set - * sess_tearing_down so no more commands are queued. + * target_sess_cmd_list_set_waiting - Set sess_tearing_down so no new commands are queued. * @se_sess: session to flag */ void target_sess_cmd_list_set_waiting(struct se_session *se_sess) { - struct se_cmd *se_cmd, *tmp_cmd; unsigned long flags; - int rc; spin_lock_irqsave(&se_sess->sess_cmd_lock, flags); - if (se_sess->sess_tearing_down) { - spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags); - return; - } se_sess->sess_tearing_down = 1; - list_splice_init(&se_sess->sess_cmd_list, &se_sess->sess_wait_list); - - list_for_each_entry_safe(se_cmd, tmp_cmd, - &se_sess->sess_wait_list, se_cmd_list) { - rc = kref_get_unless_zero(&se_cmd->cmd_kref); - if (rc) { - se_cmd->cmd_wait_set = 1; - spin_lock(&se_cmd->t_state_lock); - se_cmd->transport_state |= CMD_T_FABRIC_STOP; - spin_unlock(&se_cmd->t_state_lock); - } else - list_del_init(&se_cmd->se_cmd_list); - } - spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags); } EXPORT_SYMBOL(target_sess_cmd_list_set_waiting); /** - * target_wait_for_sess_cmds - Wait for outstanding descriptors + * target_wait_for_sess_cmds - Wait for outstanding commands * @se_sess: session to wait for active I/O */ void target_wait_for_sess_cmds(struct se_session *se_sess) { - struct se_cmd *se_cmd, *tmp_cmd; - unsigned long flags; - bool tas; - - list_for_each_entry_safe(se_cmd, tmp_cmd, - &se_sess->sess_wait_list, se_cmd_list) { - pr_debug("Waiting for se_cmd: %p t_state: %d, fabric state:" - " %d\n", se_cmd, se_cmd->t_state, - se_cmd->se_tfo->get_cmd_state(se_cmd)); - - spin_lock_irqsave(&se_cmd->t_state_lock, flags); - tas = (se_cmd->transport_state & CMD_T_TAS); - spin_unlock_irqrestore(&se_cmd->t_state_lock, flags); - - if (!target_put_sess_cmd(se_cmd)) { - if (tas) - target_put_sess_cmd(se_cmd); - } - - wait_for_completion(&se_cmd->cmd_wait_comp); - pr_debug("After cmd_wait_comp: se_cmd: %p t_state: %d" - " fabric state: %d\n", se_cmd, se_cmd->t_state, - se_cmd->se_tfo->get_cmd_state(se_cmd)); - - se_cmd->se_tfo->release_cmd(se_cmd); - } - - spin_lock_irqsave(&se_sess->sess_cmd_lock, flags); - WARN_ON(!list_empty(&se_sess->sess_cmd_list)); - spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags); + struct se_cmd *cmd; + int ret; + WARN_ON_ONCE(!se_sess->sess_tearing_down); + + spin_lock_irq(&se_sess->sess_cmd_lock); + do { + ret = wait_event_interruptible_lock_irq_timeout( + se_sess->cmd_list_wq, + list_empty(&se_sess->sess_cmd_list), + se_sess->sess_cmd_lock, 180 * HZ); + list_for_each_entry(cmd, &se_sess->sess_cmd_list, se_cmd_list) + target_show_cmd("session shutdown: still waiting for ", + cmd); + } while (ret <= 0); + spin_unlock_irq(&se_sess->sess_cmd_lock); } EXPORT_SYMBOL(target_wait_for_sess_cmds); diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h index 448f291125c2..906c1fc485e4 100644 --- a/include/target/target_core_base.h +++ b/include/target/target_core_base.h @@ -443,7 +443,6 @@ struct se_cmd { u8 scsi_asc; u8 scsi_ascq; u16 scsi_sense_length; - unsigned cmd_wait_set:1; unsigned unknown_data_length:1; bool state_active:1; u64 tag; /* SAM command identifier aka task tag */ @@ -606,8 +605,8 @@ struct se_session { struct list_head sess_list; struct list_head sess_acl_list; struct list_head sess_cmd_list; - struct list_head sess_wait_list; spinlock_t sess_cmd_lock; + wait_queue_head_t cmd_list_wq; void *sess_cmd_map; struct sbitmap_queue sess_tag_pool; };