From patchwork Tue Mar 5 10:58:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saurav Kashyap X-Patchwork-Id: 10839305 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 922551390 for ; Tue, 5 Mar 2019 11:00:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7E30E2B442 for ; Tue, 5 Mar 2019 11:00:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 725392BB30; Tue, 5 Mar 2019 11:00:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D1482A9B5 for ; Tue, 5 Mar 2019 11:00:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727492AbfCELAa (ORCPT ); Tue, 5 Mar 2019 06:00:30 -0500 Received: from mail-eopbgr740052.outbound.protection.outlook.com ([40.107.74.52]:5185 "EHLO NAM01-BN3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727401AbfCELA3 (ORCPT ); Tue, 5 Mar 2019 06:00:29 -0500 Received: from DM6PR07CA0071.namprd07.prod.outlook.com (20.177.221.48) by BYAPR07MB4376.namprd07.prod.outlook.com (52.135.224.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1665.19; Tue, 5 Mar 2019 11:00:25 +0000 Received: from BY2NAM05FT020.eop-nam05.prod.protection.outlook.com (2a01:111:f400:7e52::208) by DM6PR07CA0071.outlook.office365.com (2603:10b6:5:74::48) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.1665.16 via Frontend Transport; Tue, 5 Mar 2019 11:00:25 +0000 Authentication-Results: spf=fail (sender IP is 199.233.58.38) smtp.mailfrom=marvell.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=fail action=none header.from=marvell.com; Received-SPF: Fail (protection.outlook.com: domain of marvell.com does not designate 199.233.58.38 as permitted sender) receiver=protection.outlook.com; client-ip=199.233.58.38; helo=CAEXCH02.caveonetworks.com; Received: from CAEXCH02.caveonetworks.com (199.233.58.38) by BY2NAM05FT020.mail.protection.outlook.com (10.152.100.157) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.20.1686.5 via Frontend Transport; Tue, 5 Mar 2019 11:00:25 +0000 Received: from dut1171.mv.qlogic.com (10.112.88.18) by CAEXCH02.caveonetworks.com (10.67.98.110) with Microsoft SMTP Server (TLS) id 14.2.347.0; Tue, 5 Mar 2019 03:00:11 -0800 Received: from dut1171.mv.qlogic.com (localhost [127.0.0.1]) by dut1171.mv.qlogic.com (8.14.7/8.14.7) with ESMTP id x25B0B0c013424; Tue, 5 Mar 2019 03:00:11 -0800 Received: (from root@localhost) by dut1171.mv.qlogic.com (8.14.7/8.14.7/Submit) id x25B0Bt2013423; Tue, 5 Mar 2019 03:00:11 -0800 From: Saurav Kashyap To: CC: , Subject: [PATCH 23/26] qedf: Cleanup rrq_work after QEDF_CMD_OUTSTANDING is cleared. Date: Tue, 5 Mar 2019 02:58:58 -0800 Message-ID: <20190305105901.13185-24-skashyap@marvell.com> X-Mailer: git-send-email 2.12.0 In-Reply-To: <20190305105901.13185-1-skashyap@marvell.com> References: <20190305105901.13185-1-skashyap@marvell.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Matching-Connectors: 131962572256742070;(abac79dc-c90b-41ba-8033-08d666125e47);(abac79dc-c90b-41ba-8033-08d666125e47) X-Forefront-Antispam-Report: CIP:199.233.58.38;IPV:CAL;CTRY:US;EFV:NLI;SFV:NSPM;SFS:(10009020)(346002)(376002)(396003)(136003)(39860400002)(2980300002)(1109001)(1110001)(339900001)(189003)(199004)(305945005)(11346002)(446003)(53936002)(126002)(48376002)(2616005)(47776003)(36756003)(81156014)(336012)(81166006)(8676002)(50466002)(2906002)(356004)(6666004)(1076003)(476003)(5660300002)(6862004)(26826003)(87636003)(498600001)(85426001)(69596002)(86362001)(16586007)(80596001)(486006)(2351001)(4326008)(26005)(14444005)(8936002)(51416003)(50226002)(68736007)(105606002)(36906005)(42186006)(76176011)(106466001)(316002)(97736004)(54906003);DIR:OUT;SFP:1101;SCL:1;SRVR:BYAPR07MB4376;H:CAEXCH02.caveonetworks.com;FPR:;SPF:Fail;LANG:en;PTR:InfoDomainNonexistent;MX:1;A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2c3acca2-97be-4fff-3cc3-08d6a159c579 X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(5600127)(711020)(4605104)(2017052603328);SRVR:BYAPR07MB4376; X-MS-TrafficTypeDiagnostic: BYAPR07MB4376: X-Microsoft-Antispam-PRVS: X-Forefront-PRVS: 0967749BC1 X-Microsoft-Exchange-Diagnostics: 1;BYAPR07MB4376;23:xk3alJJUhX44g4oNi6rE/NY9dlm9++KlwiLVf1aPW5/Jdn1X9Mi3OWSobMbEezIuOk9Jk2xZwNgjW3oykqscmFWXNvcA/aRKucu8aQL3bRN7/A4JOb9bHJG8YmAe7ILaCLJwT59ls+rk28SDepNgcJVU+1FVaCK7wXArghHJD1PanxEDsRJfiUyfsRbc19IJPzZTnV04nGmkSkPBGmOqZp/c/QLTtn1dRSxZc9EBqzZnruB4EGrmVrpIexvp0lor5dVeOVSpySdEzwxPD91fYAtqFpIgLxk4P4u+Qb3IqBW9xhMMAtcFLkXayqF3g+fjC7/P5UOCTDtITBWtrignJl2jACanOQ+xo5hiBcH41b0YUhIg24fXNUAMjjQTg/VxaJmtiVb6sqsJx1FW+BvyuhKvALosr1dp2rmq4K/sa1pcyRIsEp08gBUO4jXeMSqIlnC1KLDMuCn6nOtMluDRtBDISVR+XVeB4qOH9DC0C3ogiFro3r9YgXF2scQTAxW0XM5w8H5mUScQyDTb2dGjeDWTGSKKLq56qv4M+ngBAVrUq3AyZyA0ZEoXI02pTT/lUlIA1zDiulf9S7MpuEtPjb8YX097keoWStGLMu9mNdD+2qNBYNg5f13bD1lzobB4P16e29zIyCNNOVT9Qs38VESyT+WQcQ02c0oZkOjqSNw+zSFqIPBWtYw37vvPREPksHXugBDdOimggzvKu6OM32EGXCVkxNXSX82VNQw+bCUqHK9/81my5W16JKGT9z2IS2cwIzuCyvYx5qyd/Q1lDvqg7u03VfmdzioQxv4wWXxrcxHnD6MolJKkPkDp6W47jVlNrca23sQjotgLaGBb79Y69uF64ZGy2T/aQnKUpWmg6PDoqrePmMbeHE6HfeGSNOrXH2g/WnBaJGwFWBG52tswbJxAcxGZeAM2SCjPCI5JgvKZ4Wrq4c1nuDjjsLwiyF7eXyqrUFiEO8fEn2B0wAGWAGrOKQcxF3M3iVYOmB1d9zFuf597+mHIlNkI7e1RVXTLgF/51qwkilJDIvj6Ay6MjVX2z2vRZzVajGmZwSGpzP6AWJ5kbntffbgRtMANOaw0xjCZdhn3BPocVmmjmXD0ZyUyhNCEIpC0om2haO0U6gpSRcBKVPcq0dW1E6OtvLRcDVAaGx79Z1uCtgO8mpAzOy3YkalDWUZolOHJWp0KYe2Y2FGA1ru563euIV6TinAF5f2cRcpxpR0W77Nap4Z4KaTXpRG+ONvBVMrYRadjxNr+FcwIMP9DUy13FDvj X-Microsoft-Antispam-Message-Info: jueSy9WXw1Pd4APnlv1BmZudSfKl3mHLxBaM+Yf5tAd5tIzILQ9ne11/acMHROdixfKDz2/3JUzuwLUEWGqsTUfhOqpU8xADkaeape3UH+mkikneKzS2i2drsap04f2sfDdtHS95QgbpVrUFv3XRiNbHoIn8AJlchxpnpEukTtIGBr7g/2chD523SuI2KqNkFffMZBYqZDtsGlg3XZCJsolp2oV4N4lLmy+qMhohuRCh2OKMj9YhM+139LAIbGabk9ebBI/NPtPeS/zjDzzG018G80FaPK1ySsHlTiN80Lnv9OCfzBH9738udbhB+b2f/j1h5m2e9wUcvlVZPcCriBO2pY0qEQ0vKfYjop3Rf6FCR676qoK1lqJKBQ9BlCdq0CzpHi8D1XHEvKQjuKzixFP30aZdgL/pw6sTw5+P6So= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2019 11:00:25.4210 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2c3acca2-97be-4fff-3cc3-08d6a159c579 X-MS-Exchange-CrossTenant-Id: 5afe0b00-7697-4969-b663-5eab37d5f47e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5afe0b00-7697-4969-b663-5eab37d5f47e;Ip=[199.233.58.38];Helo=[CAEXCH02.caveonetworks.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR07MB4376 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Shyam Sundar Here is the relevant logs for the problem we are solving: qedf_flush_active_ios:1707]:3: Flush active i/o's num=0x17 fcport=0xffff948168fbcc80 port_id=0x550200 scsi_id=0. qedf_flush_active_ios:1708]:3: Locking flush mutex. qedf_flush_active_ios:1758]:3: Not outstanding, xid=0xaaf, cmd_type=3 refcount=1. qedf_flush_active_ios:1896]:3: Flushed 0x16 I/Os, active=0x1. qedf_flush_active_ios:1901]:3: Flushed 0x16 I/Os, active=0x1 cnt=60. qedf_send_rrq:295]:3: Sending RRQ orig io = ffffb48b8f7d7158, orig_xid = 0xaaf qedf_initiate_els:37]:3: Sending ELS qedf_initiate_els:68]:3: initiate_els els_req = 0xffffb48b8f6d3098 cb_arg = ffff948fd5e4de80 xid = 4c6 qedf_init_mp_req:2172]:3: Entered. qedf_init_mp_task:727]:3: Initializing MP task for cmd_type=4 qedf_initiate_els:134]:3: Ringing doorbell for ELS req qedf_flush_active_ios:1901]:3: Flushed 0x16 I/Os, active=0x2 cnt=20. qedf_cmd_timeout:96]:3: ELS timeout, xid=0x4c6. qedf_rrq_compl:186]:3: Entered. qedf_rrq_compl:204]:3: rrq_compl: orig io = ffffb48b8f7d7158, orig xid = 0xaaf, rrq_xid = 0x4c6, refcount=1 qedf_flush_active_ios:1935]:3: Unlocking flush mutex. qedf_upload_connection:1579]:3: Uploading connection port_id=550200. We found an ABTS command for which CMD_OUTSTANDING was cleared (line 3). For this command, delayed send_rrq was queued, but would take 10 secs to execute. Adding capability to detect that (based on io_req->state that is being introduced), and attempt to cancel rrq_work. If we succeed, we drop the reference and free the io_req. If we cannot, then the els will get sent out and we will wait for 10 secs for it to complete. Signed-off-by: Shyam Sundar Signed-off-by: Saurav Kashyap --- drivers/scsi/qedf/qedf.h | 17 +++++++++++++++ drivers/scsi/qedf/qedf_io.c | 53 +++++++++++++++++++++++++++++++-------------- 2 files changed, 54 insertions(+), 16 deletions(-) diff --git a/drivers/scsi/qedf/qedf.h b/drivers/scsi/qedf/qedf.h index 1e0d142..f647817 100644 --- a/drivers/scsi/qedf/qedf.h +++ b/drivers/scsi/qedf/qedf.h @@ -119,9 +119,26 @@ struct qedf_ioreq { u8 io_req_flags; uint8_t tm_flags; struct qedf_rport *fcport; +#define QEDF_CMD_ST_INACTIVE 0 +#define QEDFC_CMD_ST_IO_ACTIVE 1 +#define QEDFC_CMD_ST_ABORT_ACTIVE 2 +#define QEDFC_CMD_ST_ABORT_ACTIVE_EH 3 +#define QEDFC_CMD_ST_CLEANUP_ACTIVE 4 +#define QEDFC_CMD_ST_CLEANUP_ACTIVE_EH 5 +#define QEDFC_CMD_ST_RRQ_ACTIVE 6 +#define QEDFC_CMD_ST_RRQ_WAIT 7 +#define QEDFC_CMD_ST_OXID_RETIRE_WAIT 8 +#define QEDFC_CMD_ST_TMF_ACTIVE 9 +#define QEDFC_CMD_ST_DRAIN_ACTIVE 10 +#define QEDFC_CMD_ST_CLEANED 11 +#define QEDFC_CMD_ST_ELS_ACTIVE 12 + atomic_t state; unsigned long flags; enum qedf_ioreq_event event; size_t data_xfer_len; + /* ID: 001: Alloc cmd (qedf_alloc_cmd) */ + /* ID: 002: Initiate ABTS (qedf_initiate_abts) */ + /* ID: 003: For RRQ (qedf_process_abts_compl) */ struct kref refcount; struct qedf_cmd_mgr *cmd_mgr; struct io_bdt *bd_tbl; diff --git a/drivers/scsi/qedf/qedf_io.c b/drivers/scsi/qedf/qedf_io.c index d8d55bc..d98f5f4 100644 --- a/drivers/scsi/qedf/qedf_io.c +++ b/drivers/scsi/qedf/qedf_io.c @@ -170,6 +170,7 @@ static void qedf_handle_rrq(struct work_struct *work) struct qedf_ioreq *io_req = container_of(work, struct qedf_ioreq, rrq_work.work); + atomic_set(&io_req->state, QEDFC_CMD_ST_RRQ_ACTIVE); qedf_send_rrq(io_req); } @@ -374,7 +375,8 @@ struct qedf_ioreq *qedf_alloc_cmd(struct qedf_rport *fcport, u8 cmd_type) io_req->lun = -1; /* Hold the io_req against deletion */ - kref_init(&io_req->refcount); + kref_init(&io_req->refcount); /* ID: 001 */ + atomic_set(&io_req->state, QEDFC_CMD_ST_IO_ACTIVE); /* Bind io_bdt for this io_req */ /* Have a static link between io_req and io_bdt_pool */ @@ -447,6 +449,7 @@ void qedf_release_cmd(struct kref *ref) atomic_inc(&cmd_mgr->free_list_cnt); atomic_dec(&fcport->num_active_ios); + atomic_set(&io_req->state, QEDF_CMD_ST_INACTIVE); if (atomic_read(&fcport->num_active_ios) < 0) QEDF_WARN(&(fcport->qedf->dbg_ctx), "active_ios < 0.\n"); @@ -1627,6 +1630,21 @@ void qedf_flush_active_ios(struct qedf_rport *fcport, int lun) QEDF_INFO(&qedf->dbg_ctx, QEDF_LOG_IO, "Not outstanding, xid=0x%x, cmd_type=%d refcount=%d.\n", io_req->xid, io_req->cmd_type, refcount); + /* If RRQ work has been queue, try to cancel it and + * free the io_req + */ + if (atomic_read(&io_req->state) == + QEDFC_CMD_ST_RRQ_WAIT) { + if (cancel_delayed_work_sync + (&io_req->rrq_work)) { + QEDF_INFO(&qedf->dbg_ctx, QEDF_LOG_IO, + "Putting reference for pending RRQ work xid=0x%x.\n", + io_req->xid); + /* ID: 003 */ + kref_put(&io_req->refcount, + qedf_release_cmd); + } + } continue; } @@ -1650,6 +1668,7 @@ void qedf_flush_active_ios(struct qedf_rport *fcport, int lun) } if (io_req->cmd_type == QEDF_ABTS) { + /* ID: 004 */ rc = kref_get_unless_zero(&io_req->refcount); if (!rc) { QEDF_ERR(&(qedf->dbg_ctx), @@ -1665,24 +1684,25 @@ void qedf_flush_active_ios(struct qedf_rport *fcport, int lun) if (cancel_delayed_work_sync(&io_req->rrq_work)) { QEDF_INFO(&qedf->dbg_ctx, QEDF_LOG_IO, - "Putting reference for pending RRQ work xid=0x%x.\n", + "Putting ref for cancelled RRQ work xid=0x%x.\n", io_req->xid); kref_put(&io_req->refcount, qedf_release_cmd); } - /* Cancel any timeout work */ - cancel_delayed_work_sync(&io_req->timeout_work); - - if (!test_bit(QEDF_CMD_IN_ABORT, &io_req->flags)) - goto free_cmd; - - qedf_initiate_cleanup(io_req, true); + if (cancel_delayed_work_sync(&io_req->timeout_work)) { + QEDF_INFO(&qedf->dbg_ctx, QEDF_LOG_IO, + "Putting ref for cancelled tmo work xid=0x%x.\n", + io_req->xid); + qedf_initiate_cleanup(io_req, true); + /* Notify eh_abort handler that ABTS is + * complete + */ + complete(&io_req->abts_done); + clear_bit(QEDF_CMD_IN_ABORT, &io_req->flags); + /* ID: 002 */ + kref_put(&io_req->refcount, qedf_release_cmd); + } flush_cnt++; - - /* Notify eh_abort handler that ABTS is complete */ - kref_put(&io_req->refcount, qedf_release_cmd); - complete(&io_req->abts_done); - goto free_cmd; } @@ -1722,7 +1742,7 @@ void qedf_flush_active_ios(struct qedf_rport *fcport, int lun) qedf_initiate_cleanup(io_req, true); free_cmd: - kref_put(&io_req->refcount, qedf_release_cmd); + kref_put(&io_req->refcount, qedf_release_cmd); /* ID: 004 */ } wait_cnt = 60; @@ -1929,7 +1949,7 @@ void qedf_process_abts_compl(struct qedf_ctx *qedf, struct fcoe_cqe *cqe, QEDF_INFO(&(qedf->dbg_ctx), QEDF_LOG_SCSI_TM, "ABTS response - ACC Send RRQ after R_A_TOV\n"); io_req->event = QEDF_IOREQ_EV_ABORT_SUCCESS; - rc = kref_get_unless_zero(&io_req->refcount); + rc = kref_get_unless_zero(&io_req->refcount); /* ID: 003 */ if (!rc) { QEDF_INFO(&qedf->dbg_ctx, QEDF_LOG_SCSI_TM, "kref is already zero so ABTS was already completed or flushed xid=0x%x.\n", @@ -1942,6 +1962,7 @@ void qedf_process_abts_compl(struct qedf_ctx *qedf, struct fcoe_cqe *cqe, */ queue_delayed_work(qedf->dpc_wq, &io_req->rrq_work, msecs_to_jiffies(qedf->lport->r_a_tov)); + atomic_set(&io_req->state, QEDFC_CMD_ST_RRQ_WAIT); break; /* For error cases let the cleanup return the command */ case FC_RCTL_BA_RJT: