From patchwork Wed Feb 8 22:24:50 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 9563659 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id F3D026020F for ; Wed, 8 Feb 2017 22:28:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C6E3C2852B for ; Wed, 8 Feb 2017 22:28:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BBDFE2852C; Wed, 8 Feb 2017 22:28:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E8DF2284F2 for ; Wed, 8 Feb 2017 22:28:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752008AbdBHW2J (ORCPT ); Wed, 8 Feb 2017 17:28:09 -0500 Received: from esa2.hgst.iphmx.com ([68.232.143.124]:63830 "EHLO esa2.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751433AbdBHW2E (ORCPT ); Wed, 8 Feb 2017 17:28:04 -0500 X-IronPort-AV: E=Sophos;i="5.35,348,1483977600"; d="scan'208";a="76618212" Received: from unknown (HELO milsmgep14.sandisk.com) ([63.163.107.225]) by ob1.hgst.iphmx.com with ESMTP; 09 Feb 2017 06:28:23 +0800 Received: from MILHUBIP04.sdcorp.global.sandisk.com (Unknown_Domain [10.201.67.162]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by (Symantec Messaging Gateway) with SMTP id 87.72.18895.CCA9B985; Wed, 8 Feb 2017 14:25:17 -0800 (PST) Received: from milsmgip12.sandisk.com (10.177.9.6) by MILHUBIP04.sdcorp.global.sandisk.com (10.177.9.97) with Microsoft SMTP Server id 14.3.319.2; Wed, 8 Feb 2017 14:25:11 -0800 X-AuditID: 0ac94371-0fadb980000049cf-00-589b9acc2f7d Received: from exp-402881.sdcorp.global.sandisk.com ( [10.177.9.6]) by (Symantec Messaging Gateway) with SMTP id 64.42.18148.7CA9B985; Wed, 8 Feb 2017 14:25:11 -0800 (PST) From: Bart Van Assche To: Bart Van Assche CC: , Christoph Hellwig , Sagi Grimberg , Andy Grover Subject: [PATCH v4 20/37] target: Simplify session shutdown code Date: Wed, 8 Feb 2017 14:24:50 -0800 Message-ID: <20170208222507.25715-21-bart.vanassche@sandisk.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170208222507.25715-1-bart.vanassche@sandisk.com> References: <20170208222507.25715-1-bart.vanassche@sandisk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrGLMWRmVeSWpSXmKPExsXCddJ5ke7ZWbMjDF5sY7PYP+sZk8XK1UeZ LG60PWCzaF36lsmBxWP3zQY2j2fTDzN5vN93lc3j8ya5AJYoLpuU1JzMstQifbsEroyvJ6QK ZrtXLHpymrmBcaVNFyMnh4SAicTkVUvYuhi5OIQEljJJNPdNYIRwtjJKPOo9xQZTtevQKWaI xEZGife3NzGBJNgEjCS+vZ/JAmKLCBhI/O69ANbNLNDFKPFm2X6wImEBB4nfG3+wdjFycLAI qEjsbUkFCfMChft6H0EtkJfY1XaRFcTmBIpPePmMEcQWErCXeHXgC9hMCYEmVon156awQTQL Spyc+QRsMbOAhMTBFy+YIRrUJU4umc80gVFoFpKyWUjKFjAyrWIUy83MKc5NTy0wNNErTsxL ySzO1kvOz93ECAntwh2Mr297H2IU4GBU4uG9YD07Qog1say4MvcQowQHs5IIb8o0oBBvSmJl VWpRfnxRaU5q8SFGaQ4WJXHeLNmpEUIC6YklqdmpqQWpRTBZJg5OqQbGS6q7Jtnz3eiOaLvR esWKo/B4AccBh/X/txwXFQ9P62f7oJLD0NFfLvTOuPqIafXkpvuhrNfr8v5U6m/YGW18aO2E YOX4jviS8pu3PTy3Peva9nKP+wSVWh9Wx2dsEc0NGWlynpETPZfr+b/d/P7JCnapk6W33AMu iD5vz4+/t3rpotXNlk+VWIozEg21mIuKEwFCc3APaQIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrCJMWRmVeSWpSXmKPExsXCtZGTTff4rNkRBj+P6Vnsn/WMyeLgzzZG i5WrjzJZ3Gh7wGbRuvQtkwOrx+6bDWwez6YfZvJ4v+8qm8e0NeeZPD5vkgtgjeKySUnNySxL LdK3S+DK+HpCqmC2e8WiJ6eZGxhX2nQxcnJICJhI7Dp0irmLkYtDSGA9o0RDewcbSIJNwEji 2/uZLCC2iICBxO/eC4wgNrNAD6PE2T0aILawgIPE740/WEFsFgFVidvnW4F6OTh4geI9b+og 5stL7Gq7CFbCCRSe8PIZ2BghAXuJVwe+ME5g5F7AyLCKUSw3M6c4Nz2zwNBIrzgxLyWzOFsv OT93EyMkHKJ2MF6faH6IkYmDU6qB8Uih7uZvfo++JnvZ2Ux+JXDzZaeot6LSD5apydlGVwq8 F7detBFdtUeZrXLWinKD/Il1k1m1G2XXL9/itWKxvdQ+/gR3ab/YXNbnb1aUf1qZqJ6SclNC T2DjqWdFyzbzaGvPi52gcOSm+AxVhq7qTTVJ859Ml9BufvjReJLX51vXHVtyFBfPVmIpzkg0 1GIuKk4EAN7zE1q3AQAA MIME-Version: 1.0 Sender: target-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Target drivers must call target_sess_cmd_list_set_waiting() and target_wait_for_sess_cmds() before freeing a session. Instead of setting a flag in each pending command from the former function and waiting in the latter function on a per-command completion, only set a per-session flag in the former function and wait on a per-session completion in the latter function. Note: this patch changes the behavior from aborting outstanding commands back to waiting for command completion. This makes the session shutdown code simpler. This change is safe because once a SCSI initiator system has submitted a command a target system is always allowed to execute it to completion. See also commit 0f4a943168f3 ("target: Fix remote-port TMR ABORT + se_cmd fabric stop"). This patch is based on the following two patches: * Bart Van Assche, target: Simplify session shutdown code, February 19, 2015 (https://github.com/bvanassche/linux/commit/8df5463d7d7619f2f1b70cfe5172eaef0aa52815). * Christoph Hellwig, target: Rework session shutdown code, December 7, 2015 (http://thread.gmane.org/gmane.linux.scsi.target.devel/10695). Signed-off-by: Bart Van Assche Reviewed-by: Hannes Reinecke Cc: Christoph Hellwig Cc: Sagi Grimberg Cc: Andy Grover --- drivers/target/target_core_tmr.c | 1 - drivers/target/target_core_transport.c | 103 ++++++++++----------------------- include/target/target_core_base.h | 4 +- 3 files changed, 31 insertions(+), 77 deletions(-) diff --git a/drivers/target/target_core_tmr.c b/drivers/target/target_core_tmr.c index 29d15eb5ebd5..398526704001 100644 --- a/drivers/target/target_core_tmr.c +++ b/drivers/target/target_core_tmr.c @@ -174,7 +174,6 @@ void core_tmr_abort_task( } se_cmd->send_abort_response = false; - list_del_init(&se_cmd->se_cmd_list); spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags); while (!wait_for_completion_timeout(&se_cmd->complete, 180 * HZ)) diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c index aac1fc1b2a7c..1ea63b261641 100644 --- a/drivers/target/target_core_transport.c +++ b/drivers/target/target_core_transport.c @@ -237,8 +237,8 @@ struct se_session *transport_init_session(enum target_prot_op sup_prot_ops) INIT_LIST_HEAD(&se_sess->sess_list); INIT_LIST_HEAD(&se_sess->sess_acl_list); INIT_LIST_HEAD(&se_sess->sess_cmd_list); - INIT_LIST_HEAD(&se_sess->sess_wait_list); spin_lock_init(&se_sess->sess_cmd_lock); + init_waitqueue_head(&se_sess->cmd_list_wq); se_sess->sup_prot_ops = sup_prot_ops; se_sess->tmf_wq = alloc_workqueue("tmf-%p", WQ_UNBOUND, 1, se_sess); @@ -1231,7 +1231,6 @@ void transport_init_se_cmd( INIT_LIST_HEAD(&cmd->se_cmd_list); INIT_LIST_HEAD(&cmd->state_list); init_completion(&cmd->t_transport_stop_comp); - init_completion(&cmd->cmd_wait_comp); init_completion(&cmd->complete); spin_lock_init(&cmd->t_state_lock); kref_init(&cmd->cmd_kref); @@ -2532,9 +2531,14 @@ int transport_generic_free_cmd(struct se_cmd *cmd, int wait_for_tasks) if (cmd->se_lun) transport_lun_remove_cmd(cmd); } - if (aborted) { + /* + * Since the iSCSI and iSER targets driver assume that a SCSI command + * can be freed once transport_generic_free_cmd() returns, wait here + * for cmd->complete. + */ + if (cmd->transport_state & CMD_T_ABORTED) { pr_debug("Detected CMD_T_ABORTED for ITT: %llu\n", cmd->tag); - wait_for_completion(&cmd->cmd_wait_comp); + wait_for_completion(&cmd->complete); } return transport_put_cmd(cmd); } @@ -2594,26 +2598,16 @@ static void target_release_cmd_kref(struct kref *kref) struct se_cmd *se_cmd = container_of(kref, struct se_cmd, cmd_kref); struct se_session *se_sess = se_cmd->se_sess; unsigned long flags; - bool fabric_stop; WARN_ON_ONCE(atomic_read(&se_cmd->tgt_ref) != 0); if (se_sess) { spin_lock_irqsave(&se_sess->sess_cmd_lock, flags); - - spin_lock(&se_cmd->t_state_lock); - fabric_stop = (se_cmd->transport_state & CMD_T_FABRIC_STOP) && - (se_cmd->transport_state & CMD_T_ABORTED); - spin_unlock(&se_cmd->t_state_lock); - - if (se_cmd->cmd_wait_set || fabric_stop) { - list_del_init(&se_cmd->se_cmd_list); - spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags); - target_free_cmd_mem(se_cmd); - complete(&se_cmd->cmd_wait_comp); - return; + if (likely(!list_empty(&se_cmd->se_cmd_list))) { + list_del(&se_cmd->se_cmd_list); + if (list_empty(&se_sess->sess_cmd_list)) + wake_up(&se_sess->cmd_list_wq); } - list_del_init(&se_cmd->se_cmd_list); spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags); } @@ -2685,78 +2679,41 @@ void target_show_cmd(const char *ctxt, struct se_cmd *cmd) atomic_read(&cmd->cmd_kref.refcount)); } -/* target_sess_cmd_list_set_waiting - Flag all commands in - * sess_cmd_list to complete cmd_wait_comp. Set - * sess_tearing_down so no more commands are queued. +/** + * target_sess_cmd_list_set_waiting - Prevent new commands to be queued * @se_sess: session to flag */ void target_sess_cmd_list_set_waiting(struct se_session *se_sess) { - struct se_cmd *se_cmd, *tmp_cmd; unsigned long flags; - int rc; spin_lock_irqsave(&se_sess->sess_cmd_lock, flags); - if (se_sess->sess_tearing_down) { - spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags); - return; - } se_sess->sess_tearing_down = 1; - list_splice_init(&se_sess->sess_cmd_list, &se_sess->sess_wait_list); - - list_for_each_entry_safe(se_cmd, tmp_cmd, - &se_sess->sess_wait_list, se_cmd_list) { - rc = kref_get_unless_zero(&se_cmd->cmd_kref); - if (rc) { - se_cmd->cmd_wait_set = 1; - spin_lock(&se_cmd->t_state_lock); - se_cmd->transport_state |= CMD_T_FABRIC_STOP; - spin_unlock(&se_cmd->t_state_lock); - } else - list_del_init(&se_cmd->se_cmd_list); - } - spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags); } EXPORT_SYMBOL(target_sess_cmd_list_set_waiting); -/* target_wait_for_sess_cmds - Wait for outstanding descriptors +/** + * target_wait_for_sess_cmds - Wait for outstanding commands * @se_sess: session to wait for active I/O */ void target_wait_for_sess_cmds(struct se_session *se_sess) { - struct se_cmd *se_cmd, *tmp_cmd; - unsigned long flags; - bool tas; - - list_for_each_entry_safe(se_cmd, tmp_cmd, - &se_sess->sess_wait_list, se_cmd_list) { - pr_debug("Waiting for se_cmd: %p t_state: %d, fabric state:" - " %d\n", se_cmd, se_cmd->t_state, - se_cmd->se_tfo->get_cmd_state(se_cmd)); - - spin_lock_irqsave(&se_cmd->t_state_lock, flags); - tas = (se_cmd->transport_state & CMD_T_TAS); - spin_unlock_irqrestore(&se_cmd->t_state_lock, flags); - - if (!__target_put_sess_cmd(se_cmd)) { - if (tas) - target_put_sess_cmd(se_cmd); - } - - wait_for_completion(&se_cmd->cmd_wait_comp); - pr_debug("After cmd_wait_comp: se_cmd: %p t_state: %d" - " fabric state: %d\n", se_cmd, se_cmd->t_state, - se_cmd->se_tfo->get_cmd_state(se_cmd)); - - WARN_ON_ONCE(!se_cmd->se_sess); - target_put_sess_cmd(se_cmd); - } - - spin_lock_irqsave(&se_sess->sess_cmd_lock, flags); - WARN_ON(!list_empty(&se_sess->sess_cmd_list)); - spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags); + struct se_cmd *cmd; + int ret; + WARN_ON_ONCE(!se_sess->sess_tearing_down); + + spin_lock_irq(&se_sess->sess_cmd_lock); + do { + ret = wait_event_interruptible_lock_irq_timeout( + se_sess->cmd_list_wq, + list_empty(&se_sess->sess_cmd_list), + se_sess->sess_cmd_lock, 180 * HZ); + list_for_each_entry(cmd, &se_sess->sess_cmd_list, se_cmd_list) + target_show_cmd(__func__, cmd); + } while (ret <= 0); + spin_unlock_irq(&se_sess->sess_cmd_lock); } EXPORT_SYMBOL(target_wait_for_sess_cmds); diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h index e5e0fd852ff5..187caa258253 100644 --- a/include/target/target_core_base.h +++ b/include/target/target_core_base.h @@ -441,7 +441,6 @@ struct se_cmd { u8 scsi_asc; u8 scsi_ascq; u16 scsi_sense_length; - unsigned cmd_wait_set:1; unsigned unknown_data_length:1; bool state_active:1; bool send_abort_response:1; @@ -474,7 +473,6 @@ struct se_cmd { struct se_session *se_sess; struct se_tmr_req *se_tmr_req; struct list_head se_cmd_list; - struct completion cmd_wait_comp; const struct target_core_fabric_ops *se_tfo; sense_reason_t (*execute_cmd)(struct se_cmd *); sense_reason_t (*transport_complete_callback)(struct se_cmd *, bool, int *); @@ -606,8 +604,8 @@ struct se_session { struct list_head sess_list; struct list_head sess_acl_list; struct list_head sess_cmd_list; - struct list_head sess_wait_list; spinlock_t sess_cmd_lock; + wait_queue_head_t cmd_list_wq; void *sess_cmd_map; struct percpu_ida sess_tag_pool; struct workqueue_struct *tmf_wq;