From patchwork Tue Nov 13 09:56:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "jianchao.wang" X-Patchwork-Id: 10680161 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6E47513B5 for ; Tue, 13 Nov 2018 09:58:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E0442A28D for ; Tue, 13 Nov 2018 09:58:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 523052A2C6; Tue, 13 Nov 2018 09:58:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB9312A28D for ; Tue, 13 Nov 2018 09:58:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731762AbeKMTy4 (ORCPT ); Tue, 13 Nov 2018 14:54:56 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:46224 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731528AbeKMTyz (ORCPT ); Tue, 13 Nov 2018 14:54:55 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD9sElA017838; Tue, 13 Nov 2018 09:57:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=KjtFBzl48kWHUaiGs45dIZaVO/sEyPqGFzaPjx5bZbs=; b=WoPLa08dFh70M75vP5o8Rk+bvqut0yF0Nr5sgjjfvRU+LyckU1JUWhTJjbjAfZkpR9ck kyJ2bf5w5s6u6ZjmS9tYO5/bLUGfs9v0dByABneAci4Z5YDixdUKX8GjCAZszGMShTER FWC/zHPeHxwCSEfi2zKy8YaYOlCK/EgjyhRFF5PR5staPCf37JPGrfQ3Qg3pFLYz33Ac 0uzmyVKbYTLBfZOub/2zsw/qFt3H53hD9ikWnq8IF3NVqJ0BMraX5qMwLlynyRYmCLUM NZNmwmKNpLmawy8IOtYSrflNifv+B3/Ehzajxda7CNThKYSNfh589J0H6KMo3zNN1cb5 0w== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2130.oracle.com with ESMTP id 2nnwg19fxf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 09:57:32 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD9vVtr023568 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 09:57:32 GMT Received: from abhmp0020.oracle.com (abhmp0020.oracle.com [141.146.116.26]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD9vV1G032358; Tue, 13 Nov 2018 09:57:31 GMT Received: from will-ThinkCentre-M93p.cn.oracle.com (/10.182.70.234) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 13 Nov 2018 01:57:31 -0800 From: Jianchao Wang To: axboe@kernel.dk Cc: ming.lei@redhat.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V6 1/5] blk-mq: refactor the code of issue request directly Date: Tue, 13 Nov 2018 17:56:52 +0800 Message-Id: <1542103016-21037-2-git-send-email-jianchao.w.wang@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1542103016-21037-1-git-send-email-jianchao.w.wang@oracle.com> References: <1542103016-21037-1-git-send-email-jianchao.w.wang@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130093 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Merge blk_mq_try_issue_directly and __blk_mq_try_issue_directly into one interface to unify the interfaces to issue requests directly. The merged interface takes over the requests totally, it could insert, end or do nothing based on the return value of .queue_rq and 'bypass' parameter. Then caller needn't any other handling any more. Signed-off-by: Jianchao Wang --- block/blk-mq.c | 93 ++++++++++++++++++++++++++++------------------------------ 1 file changed, 45 insertions(+), 48 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 411be60..14b4d06 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1766,78 +1766,75 @@ static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx, return ret; } -static blk_status_t __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, +static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, struct request *rq, blk_qc_t *cookie, - bool bypass_insert) + bool bypass) { struct request_queue *q = rq->q; bool run_queue = true; + blk_status_t ret = BLK_STS_RESOURCE; + int srcu_idx; + hctx_lock(hctx, &srcu_idx); /* - * RCU or SRCU read lock is needed before checking quiesced flag. + * hctx_lock is needed before checking quiesced flag. * - * When queue is stopped or quiesced, ignore 'bypass_insert' from - * blk_mq_request_issue_directly(), and return BLK_STS_OK to caller, - * and avoid driver to try to dispatch again. + * When queue is stopped or quiesced, ignore 'bypass', insert + * and return BLK_STS_OK to caller, and avoid driver to try to + * dispatch again. */ - if (blk_mq_hctx_stopped(hctx) || blk_queue_quiesced(q)) { + if (unlikely(blk_mq_hctx_stopped(hctx) || blk_queue_quiesced(q))) { run_queue = false; - bypass_insert = false; - goto insert; + bypass = false; + goto out_unlock; } - if (q->elevator && !bypass_insert) - goto insert; + /* + * Bypass the potential scheduler on the bottom device. + */ + if (unlikely(q->elevator && !bypass)) + goto out_unlock; - if (!blk_mq_get_dispatch_budget(hctx)) - goto insert; + if (unlikely(!blk_mq_get_dispatch_budget(hctx))) + goto out_unlock; - if (!blk_mq_get_driver_tag(rq)) { + if (unlikely(!blk_mq_get_driver_tag(rq))) { blk_mq_put_dispatch_budget(hctx); - goto insert; + goto out_unlock; } - return __blk_mq_issue_directly(hctx, rq, cookie); -insert: - if (bypass_insert) - return BLK_STS_RESOURCE; - - blk_mq_sched_insert_request(rq, false, run_queue, false); - return BLK_STS_OK; -} - -static void blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, - struct request *rq, blk_qc_t *cookie) -{ - blk_status_t ret; - int srcu_idx; - - might_sleep_if(hctx->flags & BLK_MQ_F_BLOCKING); + ret = __blk_mq_issue_directly(hctx, rq, cookie); - hctx_lock(hctx, &srcu_idx); +out_unlock: + hctx_unlock(hctx, srcu_idx); - ret = __blk_mq_try_issue_directly(hctx, rq, cookie, false); - if (ret == BLK_STS_RESOURCE || ret == BLK_STS_DEV_RESOURCE) - blk_mq_sched_insert_request(rq, false, true, false); - else if (ret != BLK_STS_OK) - blk_mq_end_request(rq, ret); + switch (ret) { + case BLK_STS_OK: + break; + case BLK_STS_DEV_RESOURCE: + case BLK_STS_RESOURCE: + if (!bypass) { + blk_mq_sched_insert_request(rq, false, run_queue, false); + ret = BLK_STS_OK; + } + break; + default: + if (!bypass) { + blk_mq_end_request(rq, ret); + ret = BLK_STS_OK; + } + break; + } - hctx_unlock(hctx, srcu_idx); + return ret; } blk_status_t blk_mq_request_issue_directly(struct request *rq) { - blk_status_t ret; - int srcu_idx; blk_qc_t unused_cookie; - struct blk_mq_hw_ctx *hctx = rq->mq_hctx; - hctx_lock(hctx, &srcu_idx); - ret = __blk_mq_try_issue_directly(hctx, rq, &unused_cookie, true); - hctx_unlock(hctx, srcu_idx); - - return ret; + return blk_mq_try_issue_directly(rq->mq_hctx, rq, &unused_cookie, true); } void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, @@ -1958,13 +1955,13 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) if (same_queue_rq) { data.hctx = same_queue_rq->mq_hctx; blk_mq_try_issue_directly(data.hctx, same_queue_rq, - &cookie); + &cookie, false); } } else if ((q->nr_hw_queues > 1 && is_sync) || (!q->elevator && !data.hctx->dispatch_busy)) { blk_mq_put_ctx(data.ctx); blk_mq_bio_to_request(rq, bio); - blk_mq_try_issue_directly(data.hctx, rq, &cookie); + blk_mq_try_issue_directly(data.hctx, rq, &cookie, false); } else { blk_mq_put_ctx(data.ctx); blk_mq_bio_to_request(rq, bio); From patchwork Tue Nov 13 09:56:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "jianchao.wang" X-Patchwork-Id: 10680159 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7BAAF13B5 for ; Tue, 13 Nov 2018 09:58:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C2972A1FD for ; Tue, 13 Nov 2018 09:58:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5EAC22A227; Tue, 13 Nov 2018 09:58:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0ED3E2A1FD for ; Tue, 13 Nov 2018 09:58:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731854AbeKMTy7 (ORCPT ); Tue, 13 Nov 2018 14:54:59 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:56316 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731528AbeKMTy6 (ORCPT ); Tue, 13 Nov 2018 14:54:58 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD9sUDO172208; Tue, 13 Nov 2018 09:57:35 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=2kpgH3uGPuhpX7ndqfQ+P4DE4uncydkuu2AhXgfQn8w=; b=EOqiiiBqchg7v0yT6Waf3p3LztVtnefASHI4wdXtgy55dAm/WHIYWzUSCLHlD/Bnvpbu gNPqljCqForCAf7PdcAd48ydRUnCSI0eIaKmxdaFgAfFwcvISWytFu3BAxYpX3v42Q0m p+hcympNGJzRI0qJWve8xqr7mxFHip/3VPZELN8QcmSdhfyqFhXCj/q1zwTRNtpLPZpA WmeZR+M/z3N6zRgXz0qYGMLYBZMYPdfj21uqbEHpCa/86vJEhv73dJe1Z2ZaKz/QIrGx 6vrZHCGTdHvKJp10KMDjxML6ZZNdZhORTw5kEkZ4EWa3FS9auNdF+utmc5BkrdZYGfcz fw== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2nnwc0hgxj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 09:57:35 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD9vXMr007252 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 09:57:34 GMT Received: from abhmp0020.oracle.com (abhmp0020.oracle.com [141.146.116.26]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD9vXTi020167; Tue, 13 Nov 2018 09:57:33 GMT Received: from will-ThinkCentre-M93p.cn.oracle.com (/10.182.70.234) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 13 Nov 2018 01:57:33 -0800 From: Jianchao Wang To: axboe@kernel.dk Cc: ming.lei@redhat.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V6 2/5] blk-mq: fix issue directly case when q is stopped or quiesced Date: Tue, 13 Nov 2018 17:56:53 +0800 Message-Id: <1542103016-21037-3-git-send-email-jianchao.w.wang@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1542103016-21037-1-git-send-email-jianchao.w.wang@oracle.com> References: <1542103016-21037-1-git-send-email-jianchao.w.wang@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130093 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When try to issue request directly, if the queue is stopped or quiesced, 'bypass' will be ignored and return BLK_STS_OK to caller to avoid it dispatch request again. Then the request will be inserted with blk_mq_sched_insert_request. This is not correct for dm-rq case where we should avoid to pass through the underlying path's io scheduler. To fix it, use blk_mq_request_bypass_insert to insert the request to hctx->dispatch when we cannot pass through io scheduler but have to insert. Signed-off-by: Jianchao Wang --- block/blk-mq.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 14b4d06..11c52bb 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1772,7 +1772,7 @@ static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, bool bypass) { struct request_queue *q = rq->q; - bool run_queue = true; + bool run_queue = true, force = false; blk_status_t ret = BLK_STS_RESOURCE; int srcu_idx; @@ -1786,7 +1786,7 @@ static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, */ if (unlikely(blk_mq_hctx_stopped(hctx) || blk_queue_quiesced(q))) { run_queue = false; - bypass = false; + force = true; goto out_unlock; } @@ -1817,6 +1817,9 @@ static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, if (!bypass) { blk_mq_sched_insert_request(rq, false, run_queue, false); ret = BLK_STS_OK; + } else if (force) { + blk_mq_request_bypass_insert(rq, run_queue); + ret = BLK_STS_OK; } break; default: From patchwork Tue Nov 13 09:56:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "jianchao.wang" X-Patchwork-Id: 10680157 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 17B7214BA for ; Tue, 13 Nov 2018 09:57:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0796A2A209 for ; Tue, 13 Nov 2018 09:57:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F016D2A227; Tue, 13 Nov 2018 09:57:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9EFB02A209 for ; Tue, 13 Nov 2018 09:57:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731528AbeKMTy7 (ORCPT ); Tue, 13 Nov 2018 14:54:59 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:46264 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731853AbeKMTy7 (ORCPT ); Tue, 13 Nov 2018 14:54:59 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD9sBUH017578; Tue, 13 Nov 2018 09:57:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=cH1NUSTPaeNFjjdGymczsoRVZoo+NNx1HDR8ByXFEBQ=; b=NOlvZ3P+k7qbnHJiRXrv7jB1gYZf+/dHgsumRDDtn3R+RF6gNpfyMkBDpgBtB3hNZGsW rJk6qNN/t1TtU7hkgYca37m6Yc4WYjLtsFBVrGomlLvdrqf0a9wzZakhFYTSqZdzhYEB oQKXLnMtbWiyTdqpJ32/LsFLacT17NI0D3Sg4hXfYLX/dTSLP7To704YAxLPD+K2/H9r J4+I6bZAcg/goC3PTtpH6iIGerUYWYnK/ZYgUU2fLa+5ukTH0OvxWVl2p4U9XyyXkJE2 klVXXPGZHOd+l1OKVezH8DcN/IwwZgTOygEgIAaIqZSntiK02CNKaQE7yxvn/bxu2X96 vQ== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2nnwg19fxr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 09:57:36 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD9vZ8h013900 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 09:57:35 GMT Received: from abhmp0020.oracle.com (abhmp0020.oracle.com [141.146.116.26]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD9vZBG025573; Tue, 13 Nov 2018 09:57:35 GMT Received: from will-ThinkCentre-M93p.cn.oracle.com (/10.182.70.234) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 13 Nov 2018 01:57:34 -0800 From: Jianchao Wang To: axboe@kernel.dk Cc: ming.lei@redhat.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V6 3/5] blk-mq: ensure hctx to be ran on mapped cpu when issue directly Date: Tue, 13 Nov 2018 17:56:54 +0800 Message-Id: <1542103016-21037-4-git-send-email-jianchao.w.wang@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1542103016-21037-1-git-send-email-jianchao.w.wang@oracle.com> References: <1542103016-21037-1-git-send-email-jianchao.w.wang@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=971 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130093 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When issue request directly and the task is migrated out of the original cpu where it allocates request, hctx could be ran on the cpu where it is not mapped. To fix this, - insert the request forcibly if BLK_MQ_F_BLOCKING is set. - check whether the current is mapped to the hctx, if not, insert forcibly. - invoke __blk_mq_issue_directly under preemption disabled. Signed-off-by: Jianchao Wang --- block/blk-mq.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 11c52bb..58f15cc 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1776,6 +1776,17 @@ static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, blk_status_t ret = BLK_STS_RESOURCE; int srcu_idx; + if (hctx->flags & BLK_MQ_F_BLOCKING) { + force = true; + goto out; + } + + if (unlikely(!cpumask_test_cpu(get_cpu(), hctx->cpumask))) { + put_cpu(); + force = true; + goto out; + } + hctx_lock(hctx, &srcu_idx); /* * hctx_lock is needed before checking quiesced flag. @@ -1808,7 +1819,8 @@ static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, out_unlock: hctx_unlock(hctx, srcu_idx); - + put_cpu(); +out: switch (ret) { case BLK_STS_OK: break; From patchwork Tue Nov 13 09:56:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "jianchao.wang" X-Patchwork-Id: 10680155 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6ADF514BA for ; Tue, 13 Nov 2018 09:57:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5153B2A2CD for ; Tue, 13 Nov 2018 09:57:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 456232A31D; Tue, 13 Nov 2018 09:57:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E407C2A2C7 for ; Tue, 13 Nov 2018 09:57:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731946AbeKMTzA (ORCPT ); Tue, 13 Nov 2018 14:55:00 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:56350 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731901AbeKMTzA (ORCPT ); Tue, 13 Nov 2018 14:55:00 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD9sEp4172092; Tue, 13 Nov 2018 09:57:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=8r8OQMhYzZUlvXn+dSgCh4VfEI2yZFjtELYK27Lgzl8=; b=0PXmTeAfclotStpUykgTPmEGYvGc1ZTvUvnJI143HpNEov2wmKqFnKAhIW65sVForO9X Vl59PFWt2vX3yTWv3dl7HwSOdGtBRchaHzPK8dt4fU9isA+q/mZs5+N3wRwdbLDNmxt7 QTbo/pw3rPn0AJQbQjJM6FosBDClBO7biuXJlv9vytjtDEIPqBOgJ95dG5OBn0Anv7ps waLGSTtGny595qJJ/5ZikesXRV6dnZ6McgbjP/kIjRQIgm6mAFGd+foG/tlIbhbbDOvC w++Dk0oIVy8QORXEG169D9k7S6H5VdXV6UFip6LRrwPZjsK72pznvpKXdH4WOkUzeEYG qA== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2120.oracle.com with ESMTP id 2nnwc0hgxs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 09:57:37 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD9vbVk023778 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 09:57:37 GMT Received: from abhmp0020.oracle.com (abhmp0020.oracle.com [141.146.116.26]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD9va8n025588; Tue, 13 Nov 2018 09:57:37 GMT Received: from will-ThinkCentre-M93p.cn.oracle.com (/10.182.70.234) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 13 Nov 2018 01:57:36 -0800 From: Jianchao Wang To: axboe@kernel.dk Cc: ming.lei@redhat.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V6 4/5] blk-mq: issue directly with bypass 'false' in blk_mq_sched_insert_requests Date: Tue, 13 Nov 2018 17:56:55 +0800 Message-Id: <1542103016-21037-5-git-send-email-jianchao.w.wang@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1542103016-21037-1-git-send-email-jianchao.w.wang@oracle.com> References: <1542103016-21037-1-git-send-email-jianchao.w.wang@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130093 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP It is not necessary to issue request directly with bypass 'true' in blk_mq_sched_insert_requests and handle the non-issued requests itself. Just set bypass to 'false' and let blk_mq_try_issue_directly handle them totally. Signed-off-by: Jianchao Wang --- block/blk-mq-sched.c | 8 +++----- block/blk-mq.c | 13 +++---------- 2 files changed, 6 insertions(+), 15 deletions(-) diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 66fda19..9af57c8 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -410,12 +410,10 @@ void blk_mq_sched_insert_requests(struct blk_mq_hw_ctx *hctx, * busy in case of 'none' scheduler, and this way may save * us one extra enqueue & dequeue to sw queue. */ - if (!hctx->dispatch_busy && !e && !run_queue_async) { + if (!hctx->dispatch_busy && !e && !run_queue_async) blk_mq_try_issue_list_directly(hctx, list); - if (list_empty(list)) - return; - } - blk_mq_insert_requests(hctx, ctx, list); + else + blk_mq_insert_requests(hctx, ctx, list); } blk_mq_run_hw_queue(hctx, run_queue_async); diff --git a/block/blk-mq.c b/block/blk-mq.c index 58f15cc..f41a815 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1855,21 +1855,14 @@ blk_status_t blk_mq_request_issue_directly(struct request *rq) void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, struct list_head *list) { + blk_qc_t unused_cookie; + while (!list_empty(list)) { - blk_status_t ret; struct request *rq = list_first_entry(list, struct request, queuelist); list_del_init(&rq->queuelist); - ret = blk_mq_request_issue_directly(rq); - if (ret != BLK_STS_OK) { - if (ret == BLK_STS_RESOURCE || - ret == BLK_STS_DEV_RESOURCE) { - list_add(&rq->queuelist, list); - break; - } - blk_mq_end_request(rq, ret); - } + blk_mq_try_issue_directly(hctx, rq, &unused_cookie, false); } } From patchwork Tue Nov 13 09:56:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "jianchao.wang" X-Patchwork-Id: 10680153 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9527613B5 for ; Tue, 13 Nov 2018 09:57:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 851E52A209 for ; Tue, 13 Nov 2018 09:57:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 799F42A346; Tue, 13 Nov 2018 09:57:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1D7042A2CD for ; Tue, 13 Nov 2018 09:57:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731568AbeKMTzD (ORCPT ); Tue, 13 Nov 2018 14:55:03 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:33056 "EHLO aserp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731901AbeKMTzC (ORCPT ); Tue, 13 Nov 2018 14:55:02 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD9sNVQ180668; Tue, 13 Nov 2018 09:57:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=z0dYalhlLnoIMtFuMo7kFrizuS8Yivowb+uHkoImADo=; b=0Lrb7ycMEO9K568YWW/GycajySoKMioLQsgplaK7Ppr2VQ50Zg14YGnOGZJiEVsi3aw7 BoW1Zd2RQLi/CekwyVtQkO2AWaykdGz71/deX3AP0Q5c3zi5zjoNtiCdUBtdpD0zICnu 9covUfBY/PR2qjXNqtJ0t4t96tYfz8yMr2lOoduI/yVv/5VRGn3ChWaX9aXcYwO05k9Q ET9/jpfh3DWqmCESNK0cz5hTZmwIbkANWPdhkrdLZnPwg64ksZ46rfjI7k35KJKl+gHL JJpOq1ZcYXf5L2kYBFlgaExqQNT2EIZz9g6T7LNFs11jGEjM9gWGJ14GF2QshStYtI+1 tQ== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2120.oracle.com with ESMTP id 2nnw6ehjhe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 09:57:39 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD9vdHH014166 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 09:57:39 GMT Received: from abhmp0020.oracle.com (abhmp0020.oracle.com [141.146.116.26]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD9vcBa025593; Tue, 13 Nov 2018 09:57:38 GMT Received: from will-ThinkCentre-M93p.cn.oracle.com (/10.182.70.234) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 13 Nov 2018 01:57:38 -0800 From: Jianchao Wang To: axboe@kernel.dk Cc: ming.lei@redhat.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V6 5/5] blk-mq: replace and kill blk_mq_request_issue_directly Date: Tue, 13 Nov 2018 17:56:56 +0800 Message-Id: <1542103016-21037-6-git-send-email-jianchao.w.wang@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1542103016-21037-1-git-send-email-jianchao.w.wang@oracle.com> References: <1542103016-21037-1-git-send-email-jianchao.w.wang@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130093 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Replace blk_mq_request_issue_directly with blk_mq_try_issue_directly in blk_insert_cloned_request and kill it as nobody uses it any more. Signed-off-by: Jianchao Wang --- block/blk-core.c | 4 +++- block/blk-mq.c | 9 +-------- block/blk-mq.h | 7 ++++--- 3 files changed, 8 insertions(+), 12 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index fdc0ad2..e4eedc7 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1421,6 +1421,8 @@ static int blk_cloned_rq_check_limits(struct request_queue *q, */ blk_status_t blk_insert_cloned_request(struct request_queue *q, struct request *rq) { + blk_qc_t unused_cookie; + if (blk_cloned_rq_check_limits(q, rq)) return BLK_STS_IOERR; @@ -1436,7 +1438,7 @@ blk_status_t blk_insert_cloned_request(struct request_queue *q, struct request * * bypass a potential scheduler on the bottom device for * insert. */ - return blk_mq_request_issue_directly(rq); + return blk_mq_try_issue_directly(rq->mq_hctx, rq, &unused_cookie, true); } EXPORT_SYMBOL_GPL(blk_insert_cloned_request); diff --git a/block/blk-mq.c b/block/blk-mq.c index f41a815..b5316c78 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1766,7 +1766,7 @@ static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx, return ret; } -static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, +blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, struct request *rq, blk_qc_t *cookie, bool bypass) @@ -1845,13 +1845,6 @@ static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, return ret; } -blk_status_t blk_mq_request_issue_directly(struct request *rq) -{ - blk_qc_t unused_cookie; - - return blk_mq_try_issue_directly(rq->mq_hctx, rq, &unused_cookie, true); -} - void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, struct list_head *list) { diff --git a/block/blk-mq.h b/block/blk-mq.h index facb6e9..f18c27c 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -61,9 +61,10 @@ void __blk_mq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, void blk_mq_request_bypass_insert(struct request *rq, bool run_queue); void blk_mq_insert_requests(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx, struct list_head *list); - -/* Used by blk_insert_cloned_request() to issue request directly */ -blk_status_t blk_mq_request_issue_directly(struct request *rq); +blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, + struct request *rq, + blk_qc_t *cookie, + bool bypass); void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, struct list_head *list);