From patchwork Sun Mar 19 01:56:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 13180180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84963C74A5B for ; Sun, 19 Mar 2023 01:56:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229759AbjCSB4f (ORCPT ); Sat, 18 Mar 2023 21:56:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229488AbjCSB4c (ORCPT ); Sat, 18 Mar 2023 21:56:32 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02A0F23106; Sat, 18 Mar 2023 18:56:30 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32IMhPLj004217; Sun, 19 Mar 2023 01:56:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=Bwhq5XR2raTVEtpcdJxY9++wOuI7JTgDaLHhrcEaak0=; b=ZW32ux4fZClLhMXvxBk7bZKeZV6vd828VrNS64Iqtosa0ICyRbzH/XIRzSTfos5awidf q72yAnrwgPIsWWAnqChGcLfZKbTDLDmn2+dg1Hex2MUhfCGBMlxNj73TV70xsHChiv3z uz8uDE532fTuQhvOcwVS37z8/5QuxyvB6J0U83ZgjN5Yl/e/RzS8OYkNRpSXTSGpXdns Qx+R7Zo6G85GZuxly5ekQzev5DI4EZOsvbUA9vEt2AcPCrwjgyqvK1y3NkkOUfe/Kg77 DY9PGIxzHwefQZer26EC7+U0Rnu64JtwTo12r6AwthtMF4TcybMPxA5T+SE9WUfo3p+S Yg== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3pd5uu911k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:28 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 32INUPOc030373; Sun, 19 Mar 2023 01:56:26 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pd3raaubv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:26 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 32J1uO5G014254; Sun, 19 Mar 2023 01:56:26 GMT Received: from mnchrist-mac.us.oracle.com (dhcp-10-154-118-173.vpn.oracle.com [10.154.118.173]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3pd3raauaw-2; Sun, 19 Mar 2023 01:56:26 +0000 From: Mike Christie To: mlombard@redhat.com, martin.petersen@oracle.com, d.bogdanov@yadro.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Cc: Mike Christie Subject: [PATCH v5 1/9] scsi: target: Move sess cmd counter to new struct Date: Sat, 18 Mar 2023 20:56:12 -0500 Message-Id: <20230319015620.96006-2-michael.christie@oracle.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230319015620.96006-1-michael.christie@oracle.com> References: <20230319015620.96006-1-michael.christie@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-18_16,2023-03-16_02,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 adultscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303190014 X-Proofpoint-GUID: UlKn8CNWI_zPN_uYXxjkIA48Wi_VrNo- X-Proofpoint-ORIG-GUID: UlKn8CNWI_zPN_uYXxjkIA48Wi_VrNo- Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org iSCSI needs to wait on outstanding commands like how srp and the FC/fcoe drivers do. It can't use target_stop_session because for MCS support we can't stop the entire session during recovery because if other connections are ok then we want to be able to continue to execute IO on them. This patch moves the per session cmd counters to a new struct, so iSCSI can allocate it per connection. The xcopy code can also just not allocate it in the future since it doesn't need to track commands. Signed-off-by: Mike Christie Reviewed-by: Maurizio Lombardi --- drivers/target/target_core_tpg.c | 2 +- drivers/target/target_core_transport.c | 135 ++++++++++++++++------- include/target/iscsi/iscsi_target_core.h | 1 + include/target/target_core_base.h | 13 ++- 4 files changed, 107 insertions(+), 44 deletions(-) diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c index 736847c933e5..8ebccdbd94f0 100644 --- a/drivers/target/target_core_tpg.c +++ b/drivers/target/target_core_tpg.c @@ -328,7 +328,7 @@ static void target_shutdown_sessions(struct se_node_acl *acl) restart: spin_lock_irqsave(&acl->nacl_sess_lock, flags); list_for_each_entry(sess, &acl->acl_sess_list, sess_acl_list) { - if (atomic_read(&sess->stopped)) + if (sess->cmd_cnt && atomic_read(&sess->cmd_cnt->stopped)) continue; list_del_init(&sess->sess_acl_list); diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c index 5926316252eb..3d6034f00dcd 100644 --- a/drivers/target/target_core_transport.c +++ b/drivers/target/target_core_transport.c @@ -220,11 +220,49 @@ void transport_subsystem_check_init(void) sub_api_initialized = 1; } -static void target_release_sess_cmd_refcnt(struct percpu_ref *ref) +static void target_release_cmd_refcnt(struct percpu_ref *ref) { - struct se_session *sess = container_of(ref, typeof(*sess), cmd_count); + struct target_cmd_counter *cmd_cnt = container_of(ref, + typeof(*cmd_cnt), + refcnt); + wake_up(&cmd_cnt->refcnt_wq); +} + +static struct target_cmd_counter *target_alloc_cmd_counter(void) +{ + struct target_cmd_counter *cmd_cnt; + int rc; + + cmd_cnt = kzalloc(sizeof(*cmd_cnt), GFP_KERNEL); + if (!cmd_cnt) + return NULL; - wake_up(&sess->cmd_count_wq); + init_completion(&cmd_cnt->stop_done); + init_waitqueue_head(&cmd_cnt->refcnt_wq); + atomic_set(&cmd_cnt->stopped, 0); + + rc = percpu_ref_init(&cmd_cnt->refcnt, target_release_cmd_refcnt, 0, + GFP_KERNEL); + if (rc) + goto free_cmd_cnt; + + return cmd_cnt; + +free_cmd_cnt: + kfree(cmd_cnt); + return NULL; +} + +static void target_free_cmd_counter(struct target_cmd_counter *cmd_cnt) +{ + /* + * Drivers like loop do not call target_stop_session during session + * shutdown so we have to drop the ref taken at init time here. + */ + if (!atomic_read(&cmd_cnt->stopped)) + percpu_ref_put(&cmd_cnt->refcnt); + + percpu_ref_exit(&cmd_cnt->refcnt); } /** @@ -238,25 +276,17 @@ int transport_init_session(struct se_session *se_sess) INIT_LIST_HEAD(&se_sess->sess_list); INIT_LIST_HEAD(&se_sess->sess_acl_list); spin_lock_init(&se_sess->sess_cmd_lock); - init_waitqueue_head(&se_sess->cmd_count_wq); - init_completion(&se_sess->stop_done); - atomic_set(&se_sess->stopped, 0); - return percpu_ref_init(&se_sess->cmd_count, - target_release_sess_cmd_refcnt, 0, GFP_KERNEL); + se_sess->cmd_cnt = target_alloc_cmd_counter(); + if (!se_sess->cmd_cnt) + return -ENOMEM; + + return 0; } EXPORT_SYMBOL(transport_init_session); void transport_uninit_session(struct se_session *se_sess) { - /* - * Drivers like iscsi and loop do not call target_stop_session - * during session shutdown so we have to drop the ref taken at init - * time here. - */ - if (!atomic_read(&se_sess->stopped)) - percpu_ref_put(&se_sess->cmd_count); - - percpu_ref_exit(&se_sess->cmd_count); + target_free_cmd_counter(se_sess->cmd_cnt); } /** @@ -2970,9 +3000,16 @@ int target_get_sess_cmd(struct se_cmd *se_cmd, bool ack_kref) se_cmd->se_cmd_flags |= SCF_ACK_KREF; } - if (!percpu_ref_tryget_live(&se_sess->cmd_count)) - ret = -ESHUTDOWN; - + /* + * Users like xcopy do not use counters since they never do a stop + * and wait. + */ + if (se_sess->cmd_cnt) { + if (!percpu_ref_tryget_live(&se_sess->cmd_cnt->refcnt)) + ret = -ESHUTDOWN; + else + se_cmd->cmd_cnt = se_sess->cmd_cnt; + } if (ret && ack_kref) target_put_sess_cmd(se_cmd); @@ -2993,7 +3030,7 @@ static void target_free_cmd_mem(struct se_cmd *cmd) static void target_release_cmd_kref(struct kref *kref) { struct se_cmd *se_cmd = container_of(kref, struct se_cmd, cmd_kref); - struct se_session *se_sess = se_cmd->se_sess; + struct target_cmd_counter *cmd_cnt = se_cmd->cmd_cnt; struct completion *free_compl = se_cmd->free_compl; struct completion *abrt_compl = se_cmd->abrt_compl; @@ -3004,7 +3041,8 @@ static void target_release_cmd_kref(struct kref *kref) if (abrt_compl) complete(abrt_compl); - percpu_ref_put(&se_sess->cmd_count); + if (cmd_cnt) + percpu_ref_put(&cmd_cnt->refcnt); } /** @@ -3123,46 +3161,65 @@ void target_show_cmd(const char *pfx, struct se_cmd *cmd) } EXPORT_SYMBOL(target_show_cmd); -static void target_stop_session_confirm(struct percpu_ref *ref) +static void target_stop_cmd_counter_confirm(struct percpu_ref *ref) +{ + struct target_cmd_counter *cmd_cnt = container_of(ref, + struct target_cmd_counter, + refcnt); + complete_all(&cmd_cnt->stop_done); +} + +/** + * target_stop_cmd_counter - Stop new IO from being added to the counter. + * @cmd_cnt: counter to stop + */ +static void target_stop_cmd_counter(struct target_cmd_counter *cmd_cnt) { - struct se_session *se_sess = container_of(ref, struct se_session, - cmd_count); - complete_all(&se_sess->stop_done); + pr_debug("Stopping command counter.\n"); + if (!atomic_cmpxchg(&cmd_cnt->stopped, 0, 1)) + percpu_ref_kill_and_confirm(&cmd_cnt->refcnt, + target_stop_cmd_counter_confirm); } /** * target_stop_session - Stop new IO from being queued on the session. - * @se_sess: session to stop + * @se_sess: session to stop */ void target_stop_session(struct se_session *se_sess) { - pr_debug("Stopping session queue.\n"); - if (atomic_cmpxchg(&se_sess->stopped, 0, 1) == 0) - percpu_ref_kill_and_confirm(&se_sess->cmd_count, - target_stop_session_confirm); + target_stop_cmd_counter(se_sess->cmd_cnt); } EXPORT_SYMBOL(target_stop_session); /** - * target_wait_for_sess_cmds - Wait for outstanding commands - * @se_sess: session to wait for active I/O + * target_wait_for_cmds - Wait for outstanding cmds. + * @cmd_cnt: counter to wait for active I/O for. */ -void target_wait_for_sess_cmds(struct se_session *se_sess) +static void target_wait_for_cmds(struct target_cmd_counter *cmd_cnt) { int ret; - WARN_ON_ONCE(!atomic_read(&se_sess->stopped)); + WARN_ON_ONCE(!atomic_read(&cmd_cnt->stopped)); do { pr_debug("Waiting for running cmds to complete.\n"); - ret = wait_event_timeout(se_sess->cmd_count_wq, - percpu_ref_is_zero(&se_sess->cmd_count), - 180 * HZ); + ret = wait_event_timeout(cmd_cnt->refcnt_wq, + percpu_ref_is_zero(&cmd_cnt->refcnt), + 180 * HZ); } while (ret <= 0); - wait_for_completion(&se_sess->stop_done); + wait_for_completion(&cmd_cnt->stop_done); pr_debug("Waiting for cmds done.\n"); } + +/** + * target_wait_for_sess_cmds - Wait for outstanding commands + * @se_sess: session to wait for active I/O + */ +void target_wait_for_sess_cmds(struct se_session *se_sess) +{ + target_wait_for_cmds(se_sess->cmd_cnt); +} EXPORT_SYMBOL(target_wait_for_sess_cmds); /* diff --git a/include/target/iscsi/iscsi_target_core.h b/include/target/iscsi/iscsi_target_core.h index 94d06ddfd80a..229118156a1f 100644 --- a/include/target/iscsi/iscsi_target_core.h +++ b/include/target/iscsi/iscsi_target_core.h @@ -600,6 +600,7 @@ struct iscsit_conn { struct iscsi_tpg_np *tpg_np; /* Pointer to parent session */ struct iscsit_session *sess; + struct target_cmd_counter *cmd_cnt; int bitmap_id; int rx_thread_active; struct task_struct *rx_thread; diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h index 12c9ba16217e..bd299790e99c 100644 --- a/include/target/target_core_base.h +++ b/include/target/target_core_base.h @@ -494,6 +494,7 @@ struct se_cmd { struct se_lun *se_lun; /* Only used for internal passthrough and legacy TCM fabric modules */ struct se_session *se_sess; + struct target_cmd_counter *cmd_cnt; struct se_tmr_req *se_tmr_req; struct llist_node se_cmd_list; struct completion *free_compl; @@ -619,22 +620,26 @@ static inline struct se_node_acl *fabric_stat_to_nacl(struct config_item *item) acl_fabric_stat_group); } -struct se_session { +struct target_cmd_counter { + struct percpu_ref refcnt; + wait_queue_head_t refcnt_wq; + struct completion stop_done; atomic_t stopped; +}; + +struct se_session { u64 sess_bin_isid; enum target_prot_op sup_prot_ops; enum target_prot_type sess_prot_type; struct se_node_acl *se_node_acl; struct se_portal_group *se_tpg; void *fabric_sess_ptr; - struct percpu_ref cmd_count; struct list_head sess_list; struct list_head sess_acl_list; spinlock_t sess_cmd_lock; - wait_queue_head_t cmd_count_wq; - struct completion stop_done; void *sess_cmd_map; struct sbitmap_queue sess_tag_pool; + struct target_cmd_counter *cmd_cnt; }; struct se_device; From patchwork Sun Mar 19 01:56:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 13180181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B285C7618A for ; Sun, 19 Mar 2023 01:56:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229776AbjCSB4g (ORCPT ); Sat, 18 Mar 2023 21:56:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229514AbjCSB4d (ORCPT ); Sat, 18 Mar 2023 21:56:33 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D26423108; Sat, 18 Mar 2023 18:56:31 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32J1tcQV019999; Sun, 19 Mar 2023 01:56:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=kB9mEZNwxh58CQvvF48mDe3zuRREcqGmFrb6cvb3Jtk=; b=Rinzp3Vn77sG0rm94VxzRFoUhTcwOWc/o37Mg2dOGHru6Aaj8TvkHzEDks+jElGEpPkc S/K7aTFmS3rWW2KothakgZ/hJsdZm1YfcsCekiXYlML4Y5Cd7Au6H9FR4lcTH703Q8si G5xrbWb2iuKF17qrDjSJgUu/fjZt7sarIul0pDl2irLRW3oTo/R9OyTCEKkTYZ6Rk3l9 h+jWg267I3Jv3HdWlTbFu8KPCU8i6MgWalWPt5xjg0U3lzhXXiJoCk9TPh9LNx2viTd5 L3dg3DFEqrYxLNuBkErQFcD4WBRD1bvlrcN5G+SgThDvCVwzPG98blhMKymjfkdAk3aY 0w== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3pd5bc93gc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:28 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 32INcDQQ030397; Sun, 19 Mar 2023 01:56:28 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pd3raauc5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:28 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 32J1uO5I014254; Sun, 19 Mar 2023 01:56:27 GMT Received: from mnchrist-mac.us.oracle.com (dhcp-10-154-118-173.vpn.oracle.com [10.154.118.173]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3pd3raauaw-3; Sun, 19 Mar 2023 01:56:27 +0000 From: Mike Christie To: mlombard@redhat.com, martin.petersen@oracle.com, d.bogdanov@yadro.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Cc: Mike Christie Subject: [PATCH v5 2/9] scsi: target: Move cmd counter allocation Date: Sat, 18 Mar 2023 20:56:13 -0500 Message-Id: <20230319015620.96006-3-michael.christie@oracle.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230319015620.96006-1-michael.christie@oracle.com> References: <20230319015620.96006-1-michael.christie@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-18_16,2023-03-16_02,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 adultscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303190014 X-Proofpoint-ORIG-GUID: qMVBVoRy4ikw-Yez3XTtjNkPjjBsJKKB X-Proofpoint-GUID: qMVBVoRy4ikw-Yez3XTtjNkPjjBsJKKB Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org iSCSI needs to allocate its cmd counter per connection for MCS support where we need to stop and wait on commands running on a connection instead of per session. This moves the cmd counter allocation to target_setup_session which is used by drivers that need the stop+wait behavior per session. xcopy doesn't need stop+wait at all, so we will be ok moving the cmd counter allocation outside of transport_init_session. Signed-off-by: Mike Christie --- drivers/target/iscsi/iscsi_target_login.c | 10 +++++ drivers/target/target_core_internal.h | 1 - drivers/target/target_core_transport.c | 55 +++++++++++------------ drivers/target/target_core_xcopy.c | 15 +------ include/target/target_core_fabric.h | 4 +- 5 files changed, 42 insertions(+), 43 deletions(-) diff --git a/drivers/target/iscsi/iscsi_target_login.c b/drivers/target/iscsi/iscsi_target_login.c index 27e448c2d066..8ab6c0107d89 100644 --- a/drivers/target/iscsi/iscsi_target_login.c +++ b/drivers/target/iscsi/iscsi_target_login.c @@ -324,8 +324,18 @@ static int iscsi_login_zero_tsih_s1( goto free_ops; } + /* + * This is temp for iser. It will be moved to per conn in later + * patches for iscsi. + */ + sess->se_sess->cmd_cnt = target_alloc_cmd_counter(); + if (!sess->se_sess->cmd_cnt) + goto free_se_sess; + return 0; +free_se_sess: + transport_free_session(sess->se_sess); free_ops: kfree(sess->sess_ops); free_id: diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h index 38a6d08f75b3..85e35cf582e5 100644 --- a/drivers/target/target_core_internal.h +++ b/drivers/target/target_core_internal.h @@ -138,7 +138,6 @@ int init_se_kmem_caches(void); void release_se_kmem_caches(void); u32 scsi_get_new_index(scsi_index_t); void transport_subsystem_check_init(void); -void transport_uninit_session(struct se_session *); unsigned char *transport_dump_cmd_direction(struct se_cmd *); void transport_dump_dev_state(struct se_device *, char *, int *); void transport_dump_dev_info(struct se_device *, struct se_lun *, diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c index 3d6034f00dcd..60647a49a1d3 100644 --- a/drivers/target/target_core_transport.c +++ b/drivers/target/target_core_transport.c @@ -228,7 +228,7 @@ static void target_release_cmd_refcnt(struct percpu_ref *ref) wake_up(&cmd_cnt->refcnt_wq); } -static struct target_cmd_counter *target_alloc_cmd_counter(void) +struct target_cmd_counter *target_alloc_cmd_counter(void) { struct target_cmd_counter *cmd_cnt; int rc; @@ -252,6 +252,7 @@ static struct target_cmd_counter *target_alloc_cmd_counter(void) kfree(cmd_cnt); return NULL; } +EXPORT_SYMBOL_GPL(target_alloc_cmd_counter); static void target_free_cmd_counter(struct target_cmd_counter *cmd_cnt) { @@ -271,24 +272,14 @@ static void target_free_cmd_counter(struct target_cmd_counter *cmd_cnt) * * The caller must have zero-initialized @se_sess before calling this function. */ -int transport_init_session(struct se_session *se_sess) +void transport_init_session(struct se_session *se_sess) { INIT_LIST_HEAD(&se_sess->sess_list); INIT_LIST_HEAD(&se_sess->sess_acl_list); spin_lock_init(&se_sess->sess_cmd_lock); - se_sess->cmd_cnt = target_alloc_cmd_counter(); - if (!se_sess->cmd_cnt) - return -ENOMEM; - - return 0; } EXPORT_SYMBOL(transport_init_session); -void transport_uninit_session(struct se_session *se_sess) -{ - target_free_cmd_counter(se_sess->cmd_cnt); -} - /** * transport_alloc_session - allocate a session object and initialize it * @sup_prot_ops: bitmask that defines which T10-PI modes are supported. @@ -296,7 +287,6 @@ void transport_uninit_session(struct se_session *se_sess) struct se_session *transport_alloc_session(enum target_prot_op sup_prot_ops) { struct se_session *se_sess; - int ret; se_sess = kmem_cache_zalloc(se_sess_cache, GFP_KERNEL); if (!se_sess) { @@ -304,11 +294,7 @@ struct se_session *transport_alloc_session(enum target_prot_op sup_prot_ops) " se_sess_cache\n"); return ERR_PTR(-ENOMEM); } - ret = transport_init_session(se_sess); - if (ret < 0) { - kmem_cache_free(se_sess_cache, se_sess); - return ERR_PTR(ret); - } + transport_init_session(se_sess); se_sess->sup_prot_ops = sup_prot_ops; return se_sess; @@ -474,8 +460,13 @@ target_setup_session(struct se_portal_group *tpg, int (*callback)(struct se_portal_group *, struct se_session *, void *)) { + struct target_cmd_counter *cmd_cnt; struct se_session *sess; + int rc; + cmd_cnt = target_alloc_cmd_counter(); + if (!cmd_cnt) + return ERR_PTR(-ENOMEM); /* * If the fabric driver is using percpu-ida based pre allocation * of I/O descriptor tags, go ahead and perform that setup now.. @@ -485,29 +476,36 @@ target_setup_session(struct se_portal_group *tpg, else sess = transport_alloc_session(prot_op); - if (IS_ERR(sess)) - return sess; + if (IS_ERR(sess)) { + rc = PTR_ERR(sess); + goto free_cnt; + } + sess->cmd_cnt = cmd_cnt; sess->se_node_acl = core_tpg_check_initiator_node_acl(tpg, (unsigned char *)initiatorname); if (!sess->se_node_acl) { - transport_free_session(sess); - return ERR_PTR(-EACCES); + rc = -EACCES; + goto free_sess; } /* * Go ahead and perform any remaining fabric setup that is * required before transport_register_session(). */ if (callback != NULL) { - int rc = callback(tpg, sess, private); - if (rc) { - transport_free_session(sess); - return ERR_PTR(rc); - } + rc = callback(tpg, sess, private); + if (rc) + goto free_sess; } transport_register_session(tpg, sess->se_node_acl, sess, private); return sess; + +free_sess: + transport_free_session(sess); +free_cnt: + target_free_cmd_counter(cmd_cnt); + return ERR_PTR(rc); } EXPORT_SYMBOL(target_setup_session); @@ -632,7 +630,8 @@ void transport_free_session(struct se_session *se_sess) sbitmap_queue_free(&se_sess->sess_tag_pool); kvfree(se_sess->sess_cmd_map); } - transport_uninit_session(se_sess); + if (se_sess->cmd_cnt) + target_free_cmd_counter(se_sess->cmd_cnt); kmem_cache_free(se_sess_cache, se_sess); } EXPORT_SYMBOL(transport_free_session); diff --git a/drivers/target/target_core_xcopy.c b/drivers/target/target_core_xcopy.c index 49eaee022ef1..49a83500c8b7 100644 --- a/drivers/target/target_core_xcopy.c +++ b/drivers/target/target_core_xcopy.c @@ -461,8 +461,6 @@ static const struct target_core_fabric_ops xcopy_pt_tfo = { int target_xcopy_setup_pt(void) { - int ret; - xcopy_wq = alloc_workqueue("xcopy_wq", WQ_MEM_RECLAIM, 0); if (!xcopy_wq) { pr_err("Unable to allocate xcopy_wq\n"); @@ -479,9 +477,7 @@ int target_xcopy_setup_pt(void) INIT_LIST_HEAD(&xcopy_pt_nacl.acl_list); INIT_LIST_HEAD(&xcopy_pt_nacl.acl_sess_list); memset(&xcopy_pt_sess, 0, sizeof(struct se_session)); - ret = transport_init_session(&xcopy_pt_sess); - if (ret < 0) - goto destroy_wq; + transport_init_session(&xcopy_pt_sess); xcopy_pt_nacl.se_tpg = &xcopy_pt_tpg; xcopy_pt_nacl.nacl_sess = &xcopy_pt_sess; @@ -490,19 +486,12 @@ int target_xcopy_setup_pt(void) xcopy_pt_sess.se_node_acl = &xcopy_pt_nacl; return 0; - -destroy_wq: - destroy_workqueue(xcopy_wq); - xcopy_wq = NULL; - return ret; } void target_xcopy_release_pt(void) { - if (xcopy_wq) { + if (xcopy_wq) destroy_workqueue(xcopy_wq); - transport_uninit_session(&xcopy_pt_sess); - } } /* diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h index 38f0662476d1..65527174b8bc 100644 --- a/include/target/target_core_fabric.h +++ b/include/target/target_core_fabric.h @@ -133,7 +133,9 @@ struct se_session *target_setup_session(struct se_portal_group *, struct se_session *, void *)); void target_remove_session(struct se_session *); -int transport_init_session(struct se_session *se_sess); +struct target_cmd_counter *target_alloc_cmd_counter(void); + +void transport_init_session(struct se_session *se_sess); struct se_session *transport_alloc_session(enum target_prot_op); int transport_alloc_session_tags(struct se_session *, unsigned int, unsigned int); From patchwork Sun Mar 19 01:56:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 13180182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6E0FC761A6 for ; Sun, 19 Mar 2023 01:56:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229514AbjCSB4h (ORCPT ); Sat, 18 Mar 2023 21:56:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229724AbjCSB4e (ORCPT ); Sat, 18 Mar 2023 21:56:34 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 911A623114; Sat, 18 Mar 2023 18:56:32 -0700 (PDT) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32INTQEA029524; Sun, 19 Mar 2023 01:56:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=keuCnTYWfNUQYorkCDlFE6ZRa5IfkVBfsJkrKx+DgJI=; b=iMrNQTQNvuLtWMpTnXl66jbZIFE2UjXjq+2FbxL1CABVHnmqejVYZrKWkrYpXXrsA8vX wF1etYzgx2tocxPl+Fn/YndSaUa4LONvwSTKOVsOB3KwGLNXt311PRtg1HHUzOuwbxd3 P5rPCci5HVfrOwdz3PLD799tS3H9d8DEzGTiVhAZTGRpOtaojeCMf5hscitJ+Z4QDoQu JLPWO2ukeLotYLkn5sP65vAGqycdpO12mI75SbsH4kHFrhdSZWjwiZpVkxkHXUgCqKq1 AbtlFjQ/CsBzDhqnNO88Vx2aVYxUC74VspZMofZfoLfn06SSTPB3gz40lqAWbR91lekz nA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3pd3qdh3kt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:30 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 32IN2RhV030974; Sun, 19 Mar 2023 01:56:29 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pd3raaucn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:29 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 32J1uO5K014254; Sun, 19 Mar 2023 01:56:29 GMT Received: from mnchrist-mac.us.oracle.com (dhcp-10-154-118-173.vpn.oracle.com [10.154.118.173]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3pd3raauaw-4; Sun, 19 Mar 2023 01:56:29 +0000 From: Mike Christie To: mlombard@redhat.com, martin.petersen@oracle.com, d.bogdanov@yadro.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Cc: Mike Christie Subject: [PATCH v5 3/9] scsi: target: Pass in cmd counter to use during cmd setup Date: Sat, 18 Mar 2023 20:56:14 -0500 Message-Id: <20230319015620.96006-4-michael.christie@oracle.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230319015620.96006-1-michael.christie@oracle.com> References: <20230319015620.96006-1-michael.christie@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-18_16,2023-03-16_02,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 adultscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303190014 X-Proofpoint-GUID: qHFboD-Sh_-jcvzsP6kaTg67TmwANe-q X-Proofpoint-ORIG-GUID: qHFboD-Sh_-jcvzsP6kaTg67TmwANe-q Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org This allows target_get_sess_cmd users to pass in the cmd counter they want to use. Right now we pass in the session's cmd counter but in the next patch iSCSI will switch from per session to per conn so this patch will be needed for that conversion. Signed-off-by: Mike Christie --- drivers/target/iscsi/iscsi_target.c | 10 +++++---- drivers/target/target_core_transport.c | 28 ++++++++++++-------------- drivers/target/target_core_xcopy.c | 8 ++++---- drivers/usb/gadget/function/f_tcm.c | 4 ++-- include/target/target_core_fabric.h | 8 +++++--- 5 files changed, 30 insertions(+), 28 deletions(-) diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c index baf4da7bb3b4..87927a36f90d 100644 --- a/drivers/target/iscsi/iscsi_target.c +++ b/drivers/target/iscsi/iscsi_target.c @@ -1190,9 +1190,10 @@ int iscsit_setup_scsi_cmd(struct iscsit_conn *conn, struct iscsit_cmd *cmd, * Initialize struct se_cmd descriptor from target_core_mod infrastructure */ __target_init_cmd(&cmd->se_cmd, &iscsi_ops, - conn->sess->se_sess, be32_to_cpu(hdr->data_length), - cmd->data_direction, sam_task_attr, - cmd->sense_buffer + 2, scsilun_to_int(&hdr->lun)); + conn->sess->se_sess, be32_to_cpu(hdr->data_length), + cmd->data_direction, sam_task_attr, + cmd->sense_buffer + 2, scsilun_to_int(&hdr->lun), + conn->sess->se_sess->cmd_cnt); pr_debug("Got SCSI Command, ITT: 0x%08x, CmdSN: 0x%08x," " ExpXferLen: %u, Length: %u, CID: %hu\n", hdr->itt, @@ -2055,7 +2056,8 @@ iscsit_handle_task_mgt_cmd(struct iscsit_conn *conn, struct iscsit_cmd *cmd, __target_init_cmd(&cmd->se_cmd, &iscsi_ops, conn->sess->se_sess, 0, DMA_NONE, TCM_SIMPLE_TAG, cmd->sense_buffer + 2, - scsilun_to_int(&hdr->lun)); + scsilun_to_int(&hdr->lun), + conn->sess->se_sess->cmd_cnt); target_get_sess_cmd(&cmd->se_cmd, true); diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c index 60647a49a1d3..c395606ab1a9 100644 --- a/drivers/target/target_core_transport.c +++ b/drivers/target/target_core_transport.c @@ -1441,14 +1441,12 @@ target_cmd_size_check(struct se_cmd *cmd, unsigned int size) * * Preserves the value of @cmd->tag. */ -void __target_init_cmd( - struct se_cmd *cmd, - const struct target_core_fabric_ops *tfo, - struct se_session *se_sess, - u32 data_length, - int data_direction, - int task_attr, - unsigned char *sense_buffer, u64 unpacked_lun) +void __target_init_cmd(struct se_cmd *cmd, + const struct target_core_fabric_ops *tfo, + struct se_session *se_sess, u32 data_length, + int data_direction, int task_attr, + unsigned char *sense_buffer, u64 unpacked_lun, + struct target_cmd_counter *cmd_cnt) { INIT_LIST_HEAD(&cmd->se_delayed_node); INIT_LIST_HEAD(&cmd->se_qf_node); @@ -1468,6 +1466,7 @@ void __target_init_cmd( cmd->sam_task_attr = task_attr; cmd->sense_buffer = sense_buffer; cmd->orig_fe_lun = unpacked_lun; + cmd->cmd_cnt = cmd_cnt; if (!(cmd->se_cmd_flags & SCF_USE_CPUID)) cmd->cpuid = raw_smp_processor_id(); @@ -1687,7 +1686,8 @@ int target_init_cmd(struct se_cmd *se_cmd, struct se_session *se_sess, * target_core_fabric_ops->queue_status() callback */ __target_init_cmd(se_cmd, se_tpg->se_tpg_tfo, se_sess, data_length, - data_dir, task_attr, sense, unpacked_lun); + data_dir, task_attr, sense, unpacked_lun, + se_sess->cmd_cnt); /* * Obtain struct se_cmd->cmd_kref reference. A second kref_get here is @@ -1982,7 +1982,8 @@ int target_submit_tmr(struct se_cmd *se_cmd, struct se_session *se_sess, BUG_ON(!se_tpg); __target_init_cmd(se_cmd, se_tpg->se_tpg_tfo, se_sess, - 0, DMA_NONE, TCM_SIMPLE_TAG, sense, unpacked_lun); + 0, DMA_NONE, TCM_SIMPLE_TAG, sense, unpacked_lun, + se_sess->cmd_cnt); /* * FIXME: Currently expect caller to handle se_cmd->se_tmr_req * allocation failure. @@ -2986,7 +2987,6 @@ EXPORT_SYMBOL(transport_generic_free_cmd); */ int target_get_sess_cmd(struct se_cmd *se_cmd, bool ack_kref) { - struct se_session *se_sess = se_cmd->se_sess; int ret = 0; /* @@ -3003,11 +3003,9 @@ int target_get_sess_cmd(struct se_cmd *se_cmd, bool ack_kref) * Users like xcopy do not use counters since they never do a stop * and wait. */ - if (se_sess->cmd_cnt) { - if (!percpu_ref_tryget_live(&se_sess->cmd_cnt->refcnt)) + if (se_cmd->cmd_cnt) { + if (!percpu_ref_tryget_live(&se_cmd->cmd_cnt->refcnt)) ret = -ESHUTDOWN; - else - se_cmd->cmd_cnt = se_sess->cmd_cnt; } if (ret && ack_kref) target_put_sess_cmd(se_cmd); diff --git a/drivers/target/target_core_xcopy.c b/drivers/target/target_core_xcopy.c index 49a83500c8b7..91ed015b588c 100644 --- a/drivers/target/target_core_xcopy.c +++ b/drivers/target/target_core_xcopy.c @@ -591,8 +591,8 @@ static int target_xcopy_read_source( (unsigned long long)src_lba, transfer_length_block, src_bytes); __target_init_cmd(se_cmd, &xcopy_pt_tfo, &xcopy_pt_sess, src_bytes, - DMA_FROM_DEVICE, 0, &xpt_cmd.sense_buffer[0], 0); - + DMA_FROM_DEVICE, 0, &xpt_cmd.sense_buffer[0], 0, + NULL); rc = target_xcopy_setup_pt_cmd(&xpt_cmd, xop, src_dev, &cdb[0], remote_port); if (rc < 0) { @@ -636,8 +636,8 @@ static int target_xcopy_write_destination( (unsigned long long)dst_lba, transfer_length_block, dst_bytes); __target_init_cmd(se_cmd, &xcopy_pt_tfo, &xcopy_pt_sess, dst_bytes, - DMA_TO_DEVICE, 0, &xpt_cmd.sense_buffer[0], 0); - + DMA_TO_DEVICE, 0, &xpt_cmd.sense_buffer[0], 0, + NULL); rc = target_xcopy_setup_pt_cmd(&xpt_cmd, xop, dst_dev, &cdb[0], remote_port); if (rc < 0) { diff --git a/drivers/usb/gadget/function/f_tcm.c b/drivers/usb/gadget/function/f_tcm.c index 658e2e21fdd0..c21acebe8aae 100644 --- a/drivers/usb/gadget/function/f_tcm.c +++ b/drivers/usb/gadget/function/f_tcm.c @@ -1054,7 +1054,7 @@ static void usbg_cmd_work(struct work_struct *work) tv_nexus->tvn_se_sess->se_tpg->se_tpg_tfo, tv_nexus->tvn_se_sess, cmd->data_len, DMA_NONE, cmd->prio_attr, cmd->sense_iu.sense, - cmd->unpacked_lun); + cmd->unpacked_lun, NULL); goto out; } @@ -1183,7 +1183,7 @@ static void bot_cmd_work(struct work_struct *work) tv_nexus->tvn_se_sess->se_tpg->se_tpg_tfo, tv_nexus->tvn_se_sess, cmd->data_len, DMA_NONE, cmd->prio_attr, cmd->sense_iu.sense, - cmd->unpacked_lun); + cmd->unpacked_lun, NULL); goto out; } diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h index 65527174b8bc..d507e7885f17 100644 --- a/include/target/target_core_fabric.h +++ b/include/target/target_core_fabric.h @@ -151,9 +151,11 @@ void transport_deregister_session_configfs(struct se_session *); void transport_deregister_session(struct se_session *); -void __target_init_cmd(struct se_cmd *, - const struct target_core_fabric_ops *, - struct se_session *, u32, int, int, unsigned char *, u64); +void __target_init_cmd(struct se_cmd *cmd, + const struct target_core_fabric_ops *tfo, + struct se_session *sess, u32 data_length, int data_direction, + int task_attr, unsigned char *sense_buffer, u64 unpacked_lun, + struct target_cmd_counter *cmd_cnt); int target_init_cmd(struct se_cmd *se_cmd, struct se_session *se_sess, unsigned char *sense, u64 unpacked_lun, u32 data_length, int task_attr, int data_dir, int flags); From patchwork Sun Mar 19 01:56:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 13180183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43610C77B60 for ; Sun, 19 Mar 2023 01:56:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229808AbjCSB4i (ORCPT ); Sat, 18 Mar 2023 21:56:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58620 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229488AbjCSB4g (ORCPT ); Sat, 18 Mar 2023 21:56:36 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 279F52312C; Sat, 18 Mar 2023 18:56:35 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32J1uX5b013888; Sun, 19 Mar 2023 01:56:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=j4euk1gcVE9ECbI1zFtyWPtER0ZZeTvaCA+w2iILEP0=; b=UxXzElrjCTZA262OsO4MNr94IGTLwYdGrdamgfhIZY18o5LD793VjwX7fTftTlMU0wde hu6ucxSsAScueXDQopIgbWlOqVdsu6+SZgAfSiW6g9dwpjJmTHqbP9/kEXGTJON7RxW1 QzewxNM5x2lZ12b1AhcTzCMmNRBZeH94d8/58RXZJdwiFyQk5/lmQhJHOO/U8OJLJ8vK w4GQkfUC1mXeK9J2WnGAntF6mcYyG4i221S9a3FcfsLaVBtzOka7fSuZnCoZZ2asFqU6 CDgeCQLiDGfsTatTJ9aBWYMeOfEGsI1D7WL/fTJqGjhjZko40YwdMS2i+IfQ6rBUXsqo Wg== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3pd3wg95u3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:32 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 32INXwcT031168; Sun, 19 Mar 2023 01:56:31 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pd3raaud5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:31 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 32J1uO5M014254; Sun, 19 Mar 2023 01:56:30 GMT Received: from mnchrist-mac.us.oracle.com (dhcp-10-154-118-173.vpn.oracle.com [10.154.118.173]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3pd3raauaw-5; Sun, 19 Mar 2023 01:56:30 +0000 From: Mike Christie To: mlombard@redhat.com, martin.petersen@oracle.com, d.bogdanov@yadro.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Cc: Mike Christie Subject: [PATCH v5 4/9] scsi: target: iscsit/isert: Alloc per conn cmd counter Date: Sat, 18 Mar 2023 20:56:15 -0500 Message-Id: <20230319015620.96006-5-michael.christie@oracle.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230319015620.96006-1-michael.christie@oracle.com> References: <20230319015620.96006-1-michael.christie@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-18_16,2023-03-16_02,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 adultscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303190014 X-Proofpoint-ORIG-GUID: Ek13pIDQZEiRP0J__KzlUHX5tkaVRLAG X-Proofpoint-GUID: Ek13pIDQZEiRP0J__KzlUHX5tkaVRLAG Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org This has iscsit allocate a per conn cmd counter and converts iscsit/isert to use it instead of the per session one. Signed-off-by: Mike Christie --- drivers/infiniband/ulp/isert/ib_isert.c | 4 ++-- drivers/target/iscsi/iscsi_target.c | 4 ++-- drivers/target/iscsi/iscsi_target_login.c | 17 +++++++---------- drivers/target/target_core_transport.c | 9 ++++++--- include/target/target_core_fabric.h | 3 +++ 5 files changed, 20 insertions(+), 17 deletions(-) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 75404885cf98..f290cd49698e 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -2506,8 +2506,8 @@ isert_wait4cmds(struct iscsit_conn *conn) isert_info("iscsit_conn %p\n", conn); if (conn->sess) { - target_stop_session(conn->sess->se_sess); - target_wait_for_sess_cmds(conn->sess->se_sess); + target_stop_cmd_counter(conn->cmd_cnt); + target_wait_for_cmds(conn->cmd_cnt); } } diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c index 87927a36f90d..11115c207844 100644 --- a/drivers/target/iscsi/iscsi_target.c +++ b/drivers/target/iscsi/iscsi_target.c @@ -1193,7 +1193,7 @@ int iscsit_setup_scsi_cmd(struct iscsit_conn *conn, struct iscsit_cmd *cmd, conn->sess->se_sess, be32_to_cpu(hdr->data_length), cmd->data_direction, sam_task_attr, cmd->sense_buffer + 2, scsilun_to_int(&hdr->lun), - conn->sess->se_sess->cmd_cnt); + conn->cmd_cnt); pr_debug("Got SCSI Command, ITT: 0x%08x, CmdSN: 0x%08x," " ExpXferLen: %u, Length: %u, CID: %hu\n", hdr->itt, @@ -2057,7 +2057,7 @@ iscsit_handle_task_mgt_cmd(struct iscsit_conn *conn, struct iscsit_cmd *cmd, conn->sess->se_sess, 0, DMA_NONE, TCM_SIMPLE_TAG, cmd->sense_buffer + 2, scsilun_to_int(&hdr->lun), - conn->sess->se_sess->cmd_cnt); + conn->cmd_cnt); target_get_sess_cmd(&cmd->se_cmd, true); diff --git a/drivers/target/iscsi/iscsi_target_login.c b/drivers/target/iscsi/iscsi_target_login.c index 8ab6c0107d89..274bdd7845ca 100644 --- a/drivers/target/iscsi/iscsi_target_login.c +++ b/drivers/target/iscsi/iscsi_target_login.c @@ -324,18 +324,8 @@ static int iscsi_login_zero_tsih_s1( goto free_ops; } - /* - * This is temp for iser. It will be moved to per conn in later - * patches for iscsi. - */ - sess->se_sess->cmd_cnt = target_alloc_cmd_counter(); - if (!sess->se_sess->cmd_cnt) - goto free_se_sess; - return 0; -free_se_sess: - transport_free_session(sess->se_sess); free_ops: kfree(sess->sess_ops); free_id: @@ -1157,8 +1147,14 @@ static struct iscsit_conn *iscsit_alloc_conn(struct iscsi_np *np) goto free_conn_cpumask; } + conn->cmd_cnt = target_alloc_cmd_counter(); + if (!conn->cmd_cnt) + goto free_conn_allowed_cpumask; + return conn; +free_conn_allowed_cpumask: + free_cpumask_var(conn->allowed_cpumask); free_conn_cpumask: free_cpumask_var(conn->conn_cpumask); free_conn_ops: @@ -1172,6 +1168,7 @@ static struct iscsit_conn *iscsit_alloc_conn(struct iscsi_np *np) void iscsit_free_conn(struct iscsit_conn *conn) { + target_free_cmd_counter(conn->cmd_cnt); free_cpumask_var(conn->allowed_cpumask); free_cpumask_var(conn->conn_cpumask); kfree(conn->conn_ops); diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c index c395606ab1a9..86adff2a86ed 100644 --- a/drivers/target/target_core_transport.c +++ b/drivers/target/target_core_transport.c @@ -254,7 +254,7 @@ struct target_cmd_counter *target_alloc_cmd_counter(void) } EXPORT_SYMBOL_GPL(target_alloc_cmd_counter); -static void target_free_cmd_counter(struct target_cmd_counter *cmd_cnt) +void target_free_cmd_counter(struct target_cmd_counter *cmd_cnt) { /* * Drivers like loop do not call target_stop_session during session @@ -265,6 +265,7 @@ static void target_free_cmd_counter(struct target_cmd_counter *cmd_cnt) percpu_ref_exit(&cmd_cnt->refcnt); } +EXPORT_SYMBOL_GPL(target_free_cmd_counter); /** * transport_init_session - initialize a session object @@ -3170,13 +3171,14 @@ static void target_stop_cmd_counter_confirm(struct percpu_ref *ref) * target_stop_cmd_counter - Stop new IO from being added to the counter. * @cmd_cnt: counter to stop */ -static void target_stop_cmd_counter(struct target_cmd_counter *cmd_cnt) +void target_stop_cmd_counter(struct target_cmd_counter *cmd_cnt) { pr_debug("Stopping command counter.\n"); if (!atomic_cmpxchg(&cmd_cnt->stopped, 0, 1)) percpu_ref_kill_and_confirm(&cmd_cnt->refcnt, target_stop_cmd_counter_confirm); } +EXPORT_SYMBOL_GPL(target_stop_cmd_counter); /** * target_stop_session - Stop new IO from being queued on the session. @@ -3192,7 +3194,7 @@ EXPORT_SYMBOL(target_stop_session); * target_wait_for_cmds - Wait for outstanding cmds. * @cmd_cnt: counter to wait for active I/O for. */ -static void target_wait_for_cmds(struct target_cmd_counter *cmd_cnt) +void target_wait_for_cmds(struct target_cmd_counter *cmd_cnt) { int ret; @@ -3208,6 +3210,7 @@ static void target_wait_for_cmds(struct target_cmd_counter *cmd_cnt) wait_for_completion(&cmd_cnt->stop_done); pr_debug("Waiting for cmds done.\n"); } +EXPORT_SYMBOL_GPL(target_wait_for_cmds); /** * target_wait_for_sess_cmds - Wait for outstanding commands diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h index d507e7885f17..b188b1e90e1e 100644 --- a/include/target/target_core_fabric.h +++ b/include/target/target_core_fabric.h @@ -133,7 +133,10 @@ struct se_session *target_setup_session(struct se_portal_group *, struct se_session *, void *)); void target_remove_session(struct se_session *); +void target_stop_cmd_counter(struct target_cmd_counter *cmd_cnt); +void target_wait_for_cmds(struct target_cmd_counter *cmd_cnt); struct target_cmd_counter *target_alloc_cmd_counter(void); +void target_free_cmd_counter(struct target_cmd_counter *cmd_cnt); void transport_init_session(struct se_session *se_sess); struct se_session *transport_alloc_session(enum target_prot_op); From patchwork Sun Mar 19 01:56:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 13180184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35633C7618A for ; Sun, 19 Mar 2023 01:56:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229850AbjCSB4q (ORCPT ); Sat, 18 Mar 2023 21:56:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58636 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229795AbjCSB4h (ORCPT ); Sat, 18 Mar 2023 21:56:37 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E54223114; Sat, 18 Mar 2023 18:56:36 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32IMNonB030348; Sun, 19 Mar 2023 01:56:34 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=1Jwyqw/NnrJyiljQssCnAl9/TwXkefH7z/8hrT6NHLc=; b=ST2cxYc0tFGTzqL6JQAtlNXTabebtUr5O8oOiD6WiwjuMv2f/EbN/FVIYzeOXupuHG3a BgUg41Wd23aSCxllYerRM4JcKdzZJqYOj1yaK+t+MylQ0MRR0LEGHe5Z2LLFCuyulWL8 Fx7YKAutzz8+STEevJA7wy8BkddHtGLNRzqUWRNlqoO4EKTdwEgaXIVGSurdyRRLBaMd xc+bqaX67n1spcwnM6cksG9oxCowB/1rqCeUX8gGf2PDZltWhj27ZgNqIDZHS3YeZgxb uAXN99YIC52qAB1uH6k3r8oSlK2YzZSir0q5L8TBIXOehkN9IvtO9k9+poWyw06lfxYQ qQ== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3pd5uu911n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:34 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 32INoVYR030404; Sun, 19 Mar 2023 01:56:32 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pd3raaudq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:32 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 32J1uO5O014254; Sun, 19 Mar 2023 01:56:32 GMT Received: from mnchrist-mac.us.oracle.com (dhcp-10-154-118-173.vpn.oracle.com [10.154.118.173]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3pd3raauaw-6; Sun, 19 Mar 2023 01:56:32 +0000 From: Mike Christie To: mlombard@redhat.com, martin.petersen@oracle.com, d.bogdanov@yadro.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Cc: Mike Christie Subject: [PATCH v5 5/9] scsi: target: iscsit: stop/wait on cmds during conn close Date: Sat, 18 Mar 2023 20:56:16 -0500 Message-Id: <20230319015620.96006-6-michael.christie@oracle.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230319015620.96006-1-michael.christie@oracle.com> References: <20230319015620.96006-1-michael.christie@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-18_16,2023-03-16_02,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 adultscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303190014 X-Proofpoint-GUID: 27_k6FiNiAuzLHz-AUuDNWdaUAF4f5R- X-Proofpoint-ORIG-GUID: 27_k6FiNiAuzLHz-AUuDNWdaUAF4f5R- Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org This fixes a bug added in: commit f36199355c64 ("scsi: target: iscsi: Fix cmd abort fabric stop race") If we have multiple sessions to the same se_device we can hit a race where a LUN_RESET on one session cleans up the se_cmds from under another session which is being closed. This results in the closing session freeing its conn/session structs while they are still in use. The bug is: 1. Session1 has IO se_cmd1. 2. Session2 can also have se_cmds for IO and optionally TMRs for ABORTS but then gets a LUN_RESET. 3. The LUN_RESET on session2 sees the se_cmds on session1 and during the drain stages marks them all with CMD_T_ABORTED. 4. session1 is now closed so iscsit_release_commands_from_conn only sees se_cmds with the CMD_T_ABORTED bit set and returns immediately even though we have outstanding commands. 5. session1's connection and session are freed. 6. The backend request for se_cmd1 completes and it accesses the freed connection/session. This hooks the iscsit layer into the cmd counter code, so we can wait for all outstanding se_cmds before freeing the connection. Fixes: f36199355c64 ("scsi: target: iscsi: Fix cmd abort fabric stop race") Signed-off-by: Mike Christie Reviewed-by: Maurizio Lombardi --- drivers/target/iscsi/iscsi_target.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c index 11115c207844..83b007141229 100644 --- a/drivers/target/iscsi/iscsi_target.c +++ b/drivers/target/iscsi/iscsi_target.c @@ -4245,6 +4245,16 @@ static void iscsit_release_commands_from_conn(struct iscsit_conn *conn) iscsit_free_cmd(cmd, true); } + + /* + * Wait on commands that were cleaned up via the aborted_task path. + * LLDs that implement iscsit_wait_conn will already have waited for + * commands. + */ + if (!conn->conn_transport->iscsit_wait_conn) { + target_stop_cmd_counter(conn->cmd_cnt); + target_wait_for_cmds(conn->cmd_cnt); + } } static void iscsit_stop_timers_for_cmds( From patchwork Sun Mar 19 01:56:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 13180188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CC0BC76196 for ; Sun, 19 Mar 2023 01:57:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229944AbjCSB5A (ORCPT ); Sat, 18 Mar 2023 21:57:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229841AbjCSB4x (ORCPT ); Sat, 18 Mar 2023 21:56:53 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D93A23136; Sat, 18 Mar 2023 18:56:47 -0700 (PDT) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32J18Q7v002689; Sun, 19 Mar 2023 01:56:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=ISUq+MN63A7B6PSmlAC3g/PRg+2OJeKDRhGTqTC8+qo=; b=SUIC0WfHgZUmX0zggyzCEmrZqA2N/zIee3Xdjat7WcQkkvMlE8GGdedMScewtfc8I3bH fLwvaYS1SJ/J04B4n2/BN2TSa1ow8s8wStkvbi7LCoU1JJMAzvM2PwF9TZ/1IO8hAc9Q yWOyEoAuC4Tm2de851SVssuZkF72UdQ3ljfLEMZMqpQf9dOryWfgf0Gf5juPSg1/iAZP UjWLfrObyc9oiPcUuBkIe7WQh9UGShxop5nbJE2y25gbgG2neHCLRsYFOPlV1olsAjSb Ad9ufNJFwjNucucdcnrNPop2Imth+Ko1TS52TcAfZObPCm35qUa7OD101PHugcG4MPMt kw== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3pd56as27b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:36 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 32IN3uhx031062; Sun, 19 Mar 2023 01:56:34 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pd3raaued-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:34 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 32J1uO5Q014254; Sun, 19 Mar 2023 01:56:34 GMT Received: from mnchrist-mac.us.oracle.com (dhcp-10-154-118-173.vpn.oracle.com [10.154.118.173]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3pd3raauaw-7; Sun, 19 Mar 2023 01:56:33 +0000 From: Mike Christie To: mlombard@redhat.com, martin.petersen@oracle.com, d.bogdanov@yadro.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Cc: Forza , Mike Christie Subject: [PATCH v5 6/9] scsi: target: iscsit: free cmds before session free Date: Sat, 18 Mar 2023 20:56:17 -0500 Message-Id: <20230319015620.96006-7-michael.christie@oracle.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230319015620.96006-1-michael.christie@oracle.com> References: <20230319015620.96006-1-michael.christie@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-18_16,2023-03-16_02,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 adultscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303190014 X-Proofpoint-ORIG-GUID: vMX9Aygi-PADMV5VgBL3LdyR0cd3ppbB X-Proofpoint-GUID: vMX9Aygi-PADMV5VgBL3LdyR0cd3ppbB Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org From: Dmitry Bogdanov Commands from recovery entries are freed after its session has been closed. That leads to use-after-free at command free or NPE with such call trace: Time2Retain timer expired for SID: 1, cleaning up iSCSI session. BUG: kernel NULL pointer dereference, address: 0000000000000140 RIP: 0010:sbitmap_queue_clear+0x3a/0xa0 Call Trace: target_release_cmd_kref+0xd1/0x1f0 [target_core_mod] transport_generic_free_cmd+0xd1/0x180 [target_core_mod] iscsit_free_cmd+0x53/0xd0 [iscsi_target_mod] iscsit_free_connection_recovery_entries+0x29d/0x320 [iscsi_target_mod] iscsit_close_session+0x13a/0x140 [iscsi_target_mod] iscsit_check_post_dataout+0x440/0x440 [iscsi_target_mod] call_timer_fn+0x24/0x140 Move cleanup of recovery enrties to before session freeing. Reported-by: Forza Signed-off-by: Dmitry Bogdanov Signed-off-by: Mike Christie Reviewed-by: Maurizio Lombardi --- drivers/target/iscsi/iscsi_target.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c index 83b007141229..26634dbea9b8 100644 --- a/drivers/target/iscsi/iscsi_target.c +++ b/drivers/target/iscsi/iscsi_target.c @@ -4529,6 +4529,9 @@ int iscsit_close_session(struct iscsit_session *sess, bool can_sleep) iscsit_stop_time2retain_timer(sess); spin_unlock_bh(&se_tpg->session_lock); + if (sess->sess_ops->ErrorRecoveryLevel == 2) + iscsit_free_connection_recovery_entries(sess); + /* * transport_deregister_session_configfs() will clear the * struct se_node_acl->nacl_sess pointer now as a iscsi_np process context @@ -4552,9 +4555,6 @@ int iscsit_close_session(struct iscsit_session *sess, bool can_sleep) transport_deregister_session(sess->se_sess); - if (sess->sess_ops->ErrorRecoveryLevel == 2) - iscsit_free_connection_recovery_entries(sess); - iscsit_free_all_ooo_cmdsns(sess); spin_lock_bh(&se_tpg->session_lock); From patchwork Sun Mar 19 01:56:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 13180185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1006CC7619A for ; Sun, 19 Mar 2023 01:56:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229838AbjCSB4r (ORCPT ); Sat, 18 Mar 2023 21:56:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58694 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229488AbjCSB4k (ORCPT ); Sat, 18 Mar 2023 21:56:40 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF92523121; Sat, 18 Mar 2023 18:56:38 -0700 (PDT) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32J1lsdX014925; Sun, 19 Mar 2023 01:56:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=mFf2nVt2qaz7X8zKMdYCakppiUj1kCW6IhOw1vBQ90o=; b=qLH85COZEVtOCesDfugViiCe/ztnLMf+USijnxEV7cnZJS4cMJUrQGH1fxpjvgk2yJog 1tqWMWOUKHpOFS7It51b1owVkUuC4jNf7qPck90S46jitTbEeL88s62KIiQAhPTObYo9 TmdHWNfcdaQOnU7Vllac3i2OrENPta3TpCEWzUSVk7fx+4mAQZkRzGn0ud/UI5TqPDCb c94/zjyah7g9fiHcqIyH2gpW+k0d9ZBOxyRIm+oosIEkt3Z70ppcVAYBKAhJXTNQIQ18 PEyb8ft/jeTBhS3i0n5Bbiyf4rEglEu2p5z9jT0nd7wpQA1bwK5NhYTJYxrp3cpVYNOS +Q== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3pd433h5hs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:36 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 32INoVYT030404; Sun, 19 Mar 2023 01:56:36 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pd3raaues-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:36 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 32J1uO5S014254; Sun, 19 Mar 2023 01:56:35 GMT Received: from mnchrist-mac.us.oracle.com (dhcp-10-154-118-173.vpn.oracle.com [10.154.118.173]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3pd3raauaw-8; Sun, 19 Mar 2023 01:56:35 +0000 From: Mike Christie To: mlombard@redhat.com, martin.petersen@oracle.com, d.bogdanov@yadro.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Cc: Mike Christie Subject: [PATCH v5 7/9] scsi: target: Fix multiple LUN_RESET handling Date: Sat, 18 Mar 2023 20:56:18 -0500 Message-Id: <20230319015620.96006-8-michael.christie@oracle.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230319015620.96006-1-michael.christie@oracle.com> References: <20230319015620.96006-1-michael.christie@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-18_16,2023-03-16_02,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 adultscore=0 mlxlogscore=972 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303190014 X-Proofpoint-ORIG-GUID: lF4YNaBa7vAo0gu_OsGFhR7J-VHaScg1 X-Proofpoint-GUID: lF4YNaBa7vAo0gu_OsGFhR7J-VHaScg1 Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org This fixes a bug where an initiator thinks a LUN_RESET has cleaned up running commands when it hasn't. The bug was added in: commit 51ec502a3266 ("target: Delete tmr from list before processing") The problem occurs when: 1. We have N IO cmds running in the target layer spread over 2 sessions. 2. The initiator sends a LUN_RESET for each session. 3. session1's LUN_RESET loops over all the running commands from both sessions and moves them to its local drain_task_list. 4. session2's LUN_RESET does not see the LUN_RESET from session1 because the commit above has it remove itself. session2 also does not see any commands since the other reset moved them off the state lists. 5. sessions2's LUN_RESET will then complete with a successful response. 6. sessions2's inititor believes the running commands on its session are now cleaned up due to the successful response and cleans up the running commands from its side. It then restarts them. 7. The commands do eventually complete on the backend and the target starts to return aborted task statuses for them. The initiator will either throw a invalid ITT error or might accidentally lookup a new task if the ITT has been reallocated already. This fixes the bug by reverting the patch, and serializes the execution of LUN_RESETs and Preempt and Aborts. The patch also prevents us from waiting on LUN_RESETs in core_tmr_drain_tmr_list, because it turns out the original patch fixed a bug that was not mentioned. For LUN_RESET1 core_tmr_drain_tmr_list can see a second LUN_RESET and wait on it. Then the second reset will run core_tmr_drain_tmr_list and see the first reset and wait on it resulting in a deadlock. Fixes: 51ec502a3266 ("target: Delete tmr from list before processing") Signed-off-by: Mike Christie --- drivers/target/target_core_device.c | 1 + drivers/target/target_core_tmr.c | 26 +++++++++++++++++++++++--- include/target/target_core_base.h | 1 + 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c index f6e58410ec3f..aeb03136773d 100644 --- a/drivers/target/target_core_device.c +++ b/drivers/target/target_core_device.c @@ -782,6 +782,7 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name) spin_lock_init(&dev->t10_alua.lba_map_lock); INIT_WORK(&dev->delayed_cmd_work, target_do_delayed_work); + mutex_init(&dev->lun_reset_mutex); dev->t10_wwn.t10_dev = dev; /* diff --git a/drivers/target/target_core_tmr.c b/drivers/target/target_core_tmr.c index 2b95b4550a63..4718db628222 100644 --- a/drivers/target/target_core_tmr.c +++ b/drivers/target/target_core_tmr.c @@ -188,14 +188,23 @@ static void core_tmr_drain_tmr_list( * LUN_RESET tmr.. */ spin_lock_irqsave(&dev->se_tmr_lock, flags); - if (tmr) - list_del_init(&tmr->tmr_list); list_for_each_entry_safe(tmr_p, tmr_pp, &dev->dev_tmr_list, tmr_list) { + if (tmr_p == tmr) + continue; + cmd = tmr_p->task_cmd; if (!cmd) { pr_err("Unable to locate struct se_cmd for TMR\n"); continue; } + + /* + * We only execute one LUN_RESET at a time so we can't wait + * on them below. + */ + if (tmr_p->function == TMR_LUN_RESET) + continue; + /* * If this function was called with a valid pr_res_key * parameter (eg: for PROUT PREEMPT_AND_ABORT service action @@ -379,14 +388,25 @@ int core_tmr_lun_reset( tmr_nacl->initiatorname); } } + + + /* + * We only allow one reset or preempt and abort to execute at a time + * to prevent one call from claiming all the cmds causing a second + * call from returning while cmds it should have waited on are still + * running. + */ + mutex_lock(&dev->lun_reset_mutex); + pr_debug("LUN_RESET: %s starting for [%s], tas: %d\n", (preempt_and_abort_list) ? "Preempt" : "TMR", dev->transport->name, tas); - core_tmr_drain_tmr_list(dev, tmr, preempt_and_abort_list); core_tmr_drain_state_list(dev, prout_cmd, tmr_sess, tas, preempt_and_abort_list); + mutex_unlock(&dev->lun_reset_mutex); + /* * Clear any legacy SPC-2 reservation when called during * LOGICAL UNIT RESET diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h index bd299790e99c..8cc42ad65c92 100644 --- a/include/target/target_core_base.h +++ b/include/target/target_core_base.h @@ -872,6 +872,7 @@ struct se_device { struct rcu_head rcu_head; int queue_cnt; struct se_device_queue *queues; + struct mutex lun_reset_mutex; }; struct target_opcode_descriptor { From patchwork Sun Mar 19 01:56:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 13180186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC222C76196 for ; Sun, 19 Mar 2023 01:56:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229621AbjCSB4r (ORCPT ); Sat, 18 Mar 2023 21:56:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229828AbjCSB4l (ORCPT ); Sat, 18 Mar 2023 21:56:41 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 507A723126; Sat, 18 Mar 2023 18:56:40 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32J1e1Xq011262; Sun, 19 Mar 2023 01:56:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=Vs5ClgrThKkKwGwkg1TTeJEje8rkPKXZD3vQrZsgCXY=; b=glfQt73iNyFS6meWCiznim7wtBQGNcPA5Gaey6vjq0jRp53HR+fI/fgOVNeF8tF4rSnr uPkg1t5TNkTOeYcNYQbmogSTFr3swfzlpW/YYBw0yJWnuy25Oujb84ibd/03u+hPcvaM ShPs3E3glkx9vRkkuMS1DsRtJValiK10m1pHa/JxyIRKtmG/DdadTpjgqm/qex1WQWKA PKdnLm8UgtMaJOGjXvUNfjrJSkRbGkVMWTeK1R5VBYkCKmWpHLxQgog6HN10UD227vvC mZ6a5+9/PWWnBdoxthmj1BESaONgqqad7Wb+FuNT71I4ZoBv62waQvP0PsVGx0JCidUM GQ== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3pd5bc93gd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:38 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 32ILsD8W031026; Sun, 19 Mar 2023 01:56:37 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pd3raauf1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:37 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 32J1uO5U014254; Sun, 19 Mar 2023 01:56:37 GMT Received: from mnchrist-mac.us.oracle.com (dhcp-10-154-118-173.vpn.oracle.com [10.154.118.173]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3pd3raauaw-9; Sun, 19 Mar 2023 01:56:37 +0000 From: Mike Christie To: mlombard@redhat.com, martin.petersen@oracle.com, d.bogdanov@yadro.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Cc: Mike Christie Subject: [PATCH v5 8/9] scsi: target: iscsit: Fix TAS handling during conn cleanup Date: Sat, 18 Mar 2023 20:56:19 -0500 Message-Id: <20230319015620.96006-9-michael.christie@oracle.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230319015620.96006-1-michael.christie@oracle.com> References: <20230319015620.96006-1-michael.christie@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-18_16,2023-03-16_02,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 adultscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303190014 X-Proofpoint-ORIG-GUID: svuAyi0C9-eBRWVVNMKA_r3K-o4NM2aU X-Proofpoint-GUID: svuAyi0C9-eBRWVVNMKA_r3K-o4NM2aU Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org This fixes a bug added in: commit f36199355c64 ("scsi: target: iscsi: Fix cmd abort fabric stop race") If CMD_T_TAS is set on the se_cmd we must call iscsit_free_cmd to do the last put on the cmd and free it, because the connection is down and we will not up sending the response and doing the put from the normal IO path. This patch adds a check for CMD_T_TAS in iscsit_release_commands_from_conn so we now detect this case and run iscsit_free_cmd. Fixes: f36199355c64 ("scsi: target: iscsi: Fix cmd abort fabric stop race") Signed-off-by: Mike Christie --- drivers/target/iscsi/iscsi_target.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c index 26634dbea9b8..07e196b44b91 100644 --- a/drivers/target/iscsi/iscsi_target.c +++ b/drivers/target/iscsi/iscsi_target.c @@ -4220,9 +4220,12 @@ static void iscsit_release_commands_from_conn(struct iscsit_conn *conn) list_for_each_entry_safe(cmd, cmd_tmp, &tmp_list, i_conn_node) { struct se_cmd *se_cmd = &cmd->se_cmd; - if (se_cmd->se_tfo != NULL) { - spin_lock_irq(&se_cmd->t_state_lock); - if (se_cmd->transport_state & CMD_T_ABORTED) { + if (!se_cmd->se_tfo) + continue; + + spin_lock_irq(&se_cmd->t_state_lock); + if (se_cmd->transport_state & CMD_T_ABORTED) { + if (!(se_cmd->transport_state & CMD_T_TAS)) /* * LIO's abort path owns the cleanup for this, * so put it back on the list and let @@ -4230,11 +4233,10 @@ static void iscsit_release_commands_from_conn(struct iscsit_conn *conn) */ list_move_tail(&cmd->i_conn_node, &conn->conn_cmd_list); - } else { - se_cmd->transport_state |= CMD_T_FABRIC_STOP; - } - spin_unlock_irq(&se_cmd->t_state_lock); + } else { + se_cmd->transport_state |= CMD_T_FABRIC_STOP; } + spin_unlock_irq(&se_cmd->t_state_lock); } spin_unlock_bh(&conn->cmd_lock); From patchwork Sun Mar 19 01:56:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 13180187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 593A3C761AF for ; Sun, 19 Mar 2023 01:56:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229878AbjCSB4s (ORCPT ); Sat, 18 Mar 2023 21:56:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229841AbjCSB4p (ORCPT ); Sat, 18 Mar 2023 21:56:45 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2FE3F23136; Sat, 18 Mar 2023 18:56:43 -0700 (PDT) Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32IN53he031401; Sun, 19 Mar 2023 01:56:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=bQj9D5/Blk0oa9b/s1jwxU4dcx2AJDPRw4owWknMFGw=; b=hFH+gBe1rxJMfMXionoBcj9xl8y7wt1L1+O3ACFrT5SPXyKmHweFTcRr6vfGpMwf1P7y IXRtsYN1V6kvBDlFD0WrI5U0++XWs/iAtD3OTIFNGGOfurTqZthCJWgVdwcAAFKf3zUY DBmW9X53p/CShr20wylW3Yy2oUFISPUFNixcjqr5MVEpo6atOJEg4WR+6ZMJvyR6jWRE rN/DtgQV6qI3y2kpGw7zKyoQgMHsalaoxP2hPkcpo4as6+yVJ20LGWAuF5MQ7hw9/IjF 0NlWFiE7DloCJR3YsGN+yO1XWxQ21pXa8N3qv+YtB07uBfWuhduSq4zC60EltO5aJ1bv NA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3pd47th5ss-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:40 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 32INUPOg030373; Sun, 19 Mar 2023 01:56:39 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pd3raaufh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Mar 2023 01:56:39 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 32J1uO5W014254; Sun, 19 Mar 2023 01:56:39 GMT Received: from mnchrist-mac.us.oracle.com (dhcp-10-154-118-173.vpn.oracle.com [10.154.118.173]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3pd3raauaw-10; Sun, 19 Mar 2023 01:56:38 +0000 From: Mike Christie To: mlombard@redhat.com, martin.petersen@oracle.com, d.bogdanov@yadro.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Cc: Mike Christie Subject: [PATCH v5 9/9] scsi: target: iscsi: handle abort for WRITE_PENDING cmds Date: Sat, 18 Mar 2023 20:56:20 -0500 Message-Id: <20230319015620.96006-10-michael.christie@oracle.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230319015620.96006-1-michael.christie@oracle.com> References: <20230319015620.96006-1-michael.christie@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-18_16,2023-03-16_02,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 adultscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303190014 X-Proofpoint-GUID: cwH0ZqNDDnPr3b2jz5__Ru1DQ73vfTsv X-Proofpoint-ORIG-GUID: cwH0ZqNDDnPr3b2jz5__Ru1DQ73vfTsv Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org From: Dmitry Bogdanov Sometimes an initiator does not send data for WRITE commands and tries to abort it. The abort hangs waiting for frontend driver completion. iSCSI driver waits for for data and that timeout eventually initiates connection reinstatment. The connection closing releases the commands in the connection, but those aborted commands still did not handle the abort and did not decrease a command ref counter. Because of that the connection reinstatement hangs indefinitely and prevents re-login for that initiator. This patch adds a handling in TCM of the abort for the WRITE_PENDING commands at connection closing moment to make it possible to release them. Signed-off-by: Dmitry Bogdanov [Rebase and expand comment] Signed-off-by: Mike Christie --- drivers/target/iscsi/iscsi_target.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c index 07e196b44b91..834cce50f9b0 100644 --- a/drivers/target/iscsi/iscsi_target.c +++ b/drivers/target/iscsi/iscsi_target.c @@ -26,6 +26,7 @@ #include #include +#include #include #include "iscsi_target_parameters.h" #include "iscsi_target_seq_pdu_list.h" @@ -4236,6 +4237,16 @@ static void iscsit_release_commands_from_conn(struct iscsit_conn *conn) } else { se_cmd->transport_state |= CMD_T_FABRIC_STOP; } + + if (cmd->se_cmd.t_state == TRANSPORT_WRITE_PENDING) { + /* + * We never submitted the cmd to LIO core, so we have + * to tell LIO to perform the completion process. + */ + spin_unlock_irq(&se_cmd->t_state_lock); + target_complete_cmd(&cmd->se_cmd, SAM_STAT_TASK_ABORTED); + continue; + } spin_unlock_irq(&se_cmd->t_state_lock); } spin_unlock_bh(&conn->cmd_lock);