From patchwork Tue Aug 13 16:39:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 11092655 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9921A112C for ; Tue, 13 Aug 2019 16:39:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8997F285D8 for ; Tue, 13 Aug 2019 16:39:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7D94A285DB; Tue, 13 Aug 2019 16:39:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 26265285D8 for ; Tue, 13 Aug 2019 16:39:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726705AbfHMQj6 (ORCPT ); Tue, 13 Aug 2019 12:39:58 -0400 Received: from mx1.redhat.com ([209.132.183.28]:16986 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727923AbfHMQj6 (ORCPT ); Tue, 13 Aug 2019 12:39:58 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5E667314D646; Tue, 13 Aug 2019 16:39:55 +0000 (UTC) Received: from rh2.redhat.com (ovpn-122-147.rdu2.redhat.com [10.10.122.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id AF3334EE74; Tue, 13 Aug 2019 16:39:54 +0000 (UTC) From: Mike Christie To: axboe@kernel.dk, josef@toxicpanda.com, linux-block@vger.kernel.org Cc: Mike Christie Subject: [PATCH 1/4] nbd: add set cmd timeout helper Date: Tue, 13 Aug 2019 11:39:49 -0500 Message-Id: <20190813163952.23486-2-mchristi@redhat.com> In-Reply-To: <20190813163952.23486-1-mchristi@redhat.com> References: <20190813163952.23486-1-mchristi@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.49]); Tue, 13 Aug 2019 16:39:58 +0000 (UTC) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add a helper to set the cmd timeout. It does not really do a lot now, but will be more useful in the next patches. Reviewed-by: Josef Bacik Signed-off-by: Mike Christie --- drivers/block/nbd.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index e21d2ded732b..69d0e5260e1d 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -1246,6 +1246,12 @@ static bool nbd_is_valid_blksize(unsigned long blksize) return true; } +static void nbd_set_cmd_timeout(struct nbd_device *nbd, u64 timeout) +{ + nbd->tag_set.timeout = timeout * HZ; + blk_queue_rq_timeout(nbd->disk->queue, timeout * HZ); +} + /* Must be called with config_lock held */ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd, unsigned int cmd, unsigned long arg) @@ -1276,10 +1282,8 @@ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd, nbd_size_set(nbd, config->blksize, arg); return 0; case NBD_SET_TIMEOUT: - if (arg) { - nbd->tag_set.timeout = arg * HZ; - blk_queue_rq_timeout(nbd->disk->queue, arg * HZ); - } + if (arg) + nbd_set_cmd_timeout(nbd, arg); return 0; case NBD_SET_FLAGS: @@ -1799,11 +1803,9 @@ static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info) if (ret) goto out; - if (info->attrs[NBD_ATTR_TIMEOUT]) { - u64 timeout = nla_get_u64(info->attrs[NBD_ATTR_TIMEOUT]); - nbd->tag_set.timeout = timeout * HZ; - blk_queue_rq_timeout(nbd->disk->queue, timeout * HZ); - } + if (info->attrs[NBD_ATTR_TIMEOUT]) + nbd_set_cmd_timeout(nbd, + nla_get_u64(info->attrs[NBD_ATTR_TIMEOUT])); if (info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]) { config->dead_conn_timeout = nla_get_u64(info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]); @@ -1971,11 +1973,9 @@ static int nbd_genl_reconfigure(struct sk_buff *skb, struct genl_info *info) if (ret) goto out; - if (info->attrs[NBD_ATTR_TIMEOUT]) { - u64 timeout = nla_get_u64(info->attrs[NBD_ATTR_TIMEOUT]); - nbd->tag_set.timeout = timeout * HZ; - blk_queue_rq_timeout(nbd->disk->queue, timeout * HZ); - } + if (info->attrs[NBD_ATTR_TIMEOUT]) + nbd_set_cmd_timeout(nbd, + nla_get_u64(info->attrs[NBD_ATTR_TIMEOUT])); if (info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]) { config->dead_conn_timeout = nla_get_u64(info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]); From patchwork Tue Aug 13 16:39:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 11092661 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E2DF21510 for ; Tue, 13 Aug 2019 16:40:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D46D22845C for ; Tue, 13 Aug 2019 16:40:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C8D27285DB; Tue, 13 Aug 2019 16:40:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A9457285D8 for ; Tue, 13 Aug 2019 16:39:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728172AbfHMQj7 (ORCPT ); Tue, 13 Aug 2019 12:39:59 -0400 Received: from mx1.redhat.com ([209.132.183.28]:31630 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727806AbfHMQj6 (ORCPT ); Tue, 13 Aug 2019 12:39:58 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 44188348BCA; Tue, 13 Aug 2019 16:39:56 +0000 (UTC) Received: from rh2.redhat.com (ovpn-122-147.rdu2.redhat.com [10.10.122.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 89E3A38E06; Tue, 13 Aug 2019 16:39:55 +0000 (UTC) From: Mike Christie To: axboe@kernel.dk, josef@toxicpanda.com, linux-block@vger.kernel.org Cc: Mike Christie Subject: [PATCH 2/4] nbd: add function to convert blk req op to nbd cmd Date: Tue, 13 Aug 2019 11:39:50 -0500 Message-Id: <20190813163952.23486-3-mchristi@redhat.com> In-Reply-To: <20190813163952.23486-1-mchristi@redhat.com> References: <20190813163952.23486-1-mchristi@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Tue, 13 Aug 2019 16:39:58 +0000 (UTC) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This adds a helper function to convert a block req op to a nbd cmd type. It will be used in the last patch to log the type in the timeout handler. Reviewed-by: Josef Bacik Signed-off-by: Mike Christie --- drivers/block/nbd.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index 69d0e5260e1d..c6ff8f922fd7 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -344,6 +344,22 @@ static void sock_shutdown(struct nbd_device *nbd) dev_warn(disk_to_dev(nbd->disk), "shutting down sockets\n"); } +static u32 req_to_nbd_cmd_type(struct request *req) +{ + switch (req_op(req)) { + case REQ_OP_DISCARD: + return NBD_CMD_TRIM; + case REQ_OP_FLUSH: + return NBD_CMD_FLUSH; + case REQ_OP_WRITE: + return NBD_CMD_WRITE; + case REQ_OP_READ: + return NBD_CMD_READ; + default: + return U32_MAX; + } +} + static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req, bool reserved) { @@ -480,22 +496,9 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd, int index) iov_iter_kvec(&from, WRITE, &iov, 1, sizeof(request)); - switch (req_op(req)) { - case REQ_OP_DISCARD: - type = NBD_CMD_TRIM; - break; - case REQ_OP_FLUSH: - type = NBD_CMD_FLUSH; - break; - case REQ_OP_WRITE: - type = NBD_CMD_WRITE; - break; - case REQ_OP_READ: - type = NBD_CMD_READ; - break; - default: + type = req_to_nbd_cmd_type(req); + if (type == U32_MAX) return -EIO; - } if (rq_data_dir(req) == WRITE && (config->flags & NBD_FLAG_READ_ONLY)) { From patchwork Tue Aug 13 16:39:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 11092659 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 93CCE112C for ; Tue, 13 Aug 2019 16:40:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 845A62845C for ; Tue, 13 Aug 2019 16:40:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 78AA928600; Tue, 13 Aug 2019 16:40:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2970A2845C for ; Tue, 13 Aug 2019 16:40:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727806AbfHMQj7 (ORCPT ); Tue, 13 Aug 2019 12:39:59 -0400 Received: from mx1.redhat.com ([209.132.183.28]:60987 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727923AbfHMQj6 (ORCPT ); Tue, 13 Aug 2019 12:39:58 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1C18230D0FDD; Tue, 13 Aug 2019 16:39:57 +0000 (UTC) Received: from rh2.redhat.com (ovpn-122-147.rdu2.redhat.com [10.10.122.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6E0B5349D3; Tue, 13 Aug 2019 16:39:56 +0000 (UTC) From: Mike Christie To: axboe@kernel.dk, josef@toxicpanda.com, linux-block@vger.kernel.org Cc: Mike Christie Subject: [PATCH 3/4] nbd: add missing config put Date: Tue, 13 Aug 2019 11:39:51 -0500 Message-Id: <20190813163952.23486-4-mchristi@redhat.com> In-Reply-To: <20190813163952.23486-1-mchristi@redhat.com> References: <20190813163952.23486-1-mchristi@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.40]); Tue, 13 Aug 2019 16:39:58 +0000 (UTC) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Fix bug added with the patch: commit 8f3ea35929a0806ad1397db99a89ffee0140822a Author: Josef Bacik Date: Mon Jul 16 12:11:35 2018 -0400 nbd: handle unexpected replies better where if the timeout handler runs when the completion path is and we fail to grab the mutex in the timeout handler we will leave a config reference and cannot free the config later. Reviewed-by: Josef Bacik Signed-off-by: Mike Christie --- drivers/block/nbd.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index c6ff8f922fd7..ebc98cf76365 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -373,8 +373,10 @@ static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req, } config = nbd->config; - if (!mutex_trylock(&cmd->lock)) + if (!mutex_trylock(&cmd->lock)) { + nbd_config_put(nbd); return BLK_EH_RESET_TIMER; + } if (config->num_connections > 1) { dev_err_ratelimited(nbd_to_dev(nbd), From patchwork Tue Aug 13 16:39:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 11092657 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 21451184E for ; Tue, 13 Aug 2019 16:40:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0CB962845C for ; Tue, 13 Aug 2019 16:40:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0109828600; Tue, 13 Aug 2019 16:39:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 893F12845C for ; Tue, 13 Aug 2019 16:39:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728183AbfHMQj7 (ORCPT ); Tue, 13 Aug 2019 12:39:59 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41986 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728172AbfHMQj6 (ORCPT ); Tue, 13 Aug 2019 12:39:58 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id F10F718A484; Tue, 13 Aug 2019 16:39:57 +0000 (UTC) Received: from rh2.redhat.com (ovpn-122-147.rdu2.redhat.com [10.10.122.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 491A142422; Tue, 13 Aug 2019 16:39:57 +0000 (UTC) From: Mike Christie To: axboe@kernel.dk, josef@toxicpanda.com, linux-block@vger.kernel.org Cc: Mike Christie Subject: [PATCH 4/4] nbd: fix zero cmd timeout handling v2 Date: Tue, 13 Aug 2019 11:39:52 -0500 Message-Id: <20190813163952.23486-5-mchristi@redhat.com> In-Reply-To: <20190813163952.23486-1-mchristi@redhat.com> References: <20190813163952.23486-1-mchristi@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Tue, 13 Aug 2019 16:39:58 +0000 (UTC) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This fixes a regression added in 4.9 with commit: commit 0eadf37afc2500e1162c9040ec26a705b9af8d47 Author: Josef Bacik Date: Thu Sep 8 12:33:40 2016 -0700 nbd: allow block mq to deal with timeouts where before the patch userspace would set the timeout to 0 to disable it. With the above patch, a zero timeout tells the block layer to use the default value of 30 seconds. For setups where commands can take a long time or experience transient issues like network disruptions this then results in IO errors being sent to the application. To fix this, the patch still uses the common block layer timeout framework, but if zero is set, nbd just logs a message and then resets the timer when it expires. Reviewed-by: Josef Bacik Signed-off-by: Mike Christie --- V2 - We used to allow apps to reset the timeout to 0. The first version of the patch only allowed it to be initialized to 0, so this adds the reset feature back. drivers/block/nbd.c | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index ebc98cf76365..98c618e5732c 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -121,6 +121,7 @@ struct nbd_cmd { struct mutex lock; int index; int cookie; + int retries; blk_status_t status; unsigned long flags; u32 cmd_cookie; @@ -407,10 +408,25 @@ static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req, nbd_config_put(nbd); return BLK_EH_DONE; } - } else { - dev_err_ratelimited(nbd_to_dev(nbd), - "Connection timed out\n"); } + + if (!nbd->tag_set.timeout) { + /* + * Userspace sets timeout=0 to disable socket disconnection, + * so just warn and reset the timer. + */ + cmd->retries++; + dev_info(nbd_to_dev(nbd), "Possible stuck request %p: control (%s@%llu,%uB). Runtime %u seconds\n", + req, nbdcmd_to_ascii(req_to_nbd_cmd_type(req)), + (unsigned long long)blk_rq_pos(req) << 9, + blk_rq_bytes(req), (req->timeout / HZ) * cmd->retries); + + mutex_unlock(&cmd->lock); + nbd_config_put(nbd); + return BLK_EH_RESET_TIMER; + } + + dev_err_ratelimited(nbd_to_dev(nbd), "Connection timed out\n"); set_bit(NBD_TIMEDOUT, &config->runtime_flags); cmd->status = BLK_STS_IOERR; mutex_unlock(&cmd->lock); @@ -531,6 +547,7 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd, int index) } cmd->index = index; cmd->cookie = nsock->cookie; + cmd->retries = 0; request.type = htonl(type | nbd_cmd_flags); if (type != NBD_CMD_FLUSH) { request.from = cpu_to_be64((u64)blk_rq_pos(req) << 9); @@ -1254,7 +1271,8 @@ static bool nbd_is_valid_blksize(unsigned long blksize) static void nbd_set_cmd_timeout(struct nbd_device *nbd, u64 timeout) { nbd->tag_set.timeout = timeout * HZ; - blk_queue_rq_timeout(nbd->disk->queue, timeout * HZ); + if (timeout) + blk_queue_rq_timeout(nbd->disk->queue, timeout * HZ); } /* Must be called with config_lock held */ @@ -1287,8 +1305,7 @@ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd, nbd_size_set(nbd, config->blksize, arg); return 0; case NBD_SET_TIMEOUT: - if (arg) - nbd_set_cmd_timeout(nbd, arg); + nbd_set_cmd_timeout(nbd, arg); return 0; case NBD_SET_FLAGS: