From patchwork Sat May 2 23:54:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 11524169 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 053BA81 for ; Sat, 2 May 2020 23:55:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E155421775 for ; Sat, 2 May 2020 23:55:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GXHueRMB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726483AbgEBXz0 (ORCPT ); Sat, 2 May 2020 19:55:26 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:53055 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725798AbgEBXz0 (ORCPT ); Sat, 2 May 2020 19:55:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588463725; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=++1YgR4HmPwW+Gr2I22tX2N4dKnmaM4easkky93J5gc=; b=GXHueRMBO75oLsSDwqLuzSCcWrETYbLMNEyNYxwibr2L0UhecnnUt5d2rQ46nkIrxMYtoT eVC70LrJpeSa3Z1NPBpoB6pW9NFYH0+zj87tfkENfUrI+Es1MwoI62NAfZk5VxNDF5JUWL af1F7o5S/CN6F9sycF1XAfEPhFUYQ1I= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-164-ZFECDHLrPV6HGDATGr5bQw-1; Sat, 02 May 2020 19:55:18 -0400 X-MC-Unique: ZFECDHLrPV6HGDATGr5bQw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B761D1005510; Sat, 2 May 2020 23:55:16 +0000 (UTC) Received: from localhost (ovpn-8-17.pek2.redhat.com [10.72.8.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7B9D160BE1; Sat, 2 May 2020 23:55:07 +0000 (UTC) From: Ming Lei To: Jens Axboe Cc: linux-block@vger.kernel.org, Ming Lei , John Garry , Bart Van Assche , Hannes Reinecke , Christoph Hellwig , Thomas Gleixner , Mike Snitzer , dm-devel@redhat.com, Hannes Reinecke , "Martin K . Petersen" Subject: [PATCH V9 01/11] block: clone nr_integrity_segments and write_hint in blk_rq_prep_clone Date: Sun, 3 May 2020 07:54:44 +0800 Message-Id: <20200502235454.1118520-2-ming.lei@redhat.com> In-Reply-To: <20200502235454.1118520-1-ming.lei@redhat.com> References: <20200502235454.1118520-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org So far blk_rq_prep_clone() is only used for setup one underlying cloned request from dm-rq request. block intetrity can be enabled for both dm-rq and the underlying queues, so it is reasonable to clone rq's nr_integrity_segments. Also write_hint is from bio, it should have been cloned too. So clone nr_integrity_segments and write_hint in blk_rq_prep_clone. Cc: John Garry Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Thomas Gleixner Cc: Mike Snitzer Cc: dm-devel@redhat.com Reviewed-by: Hannes Reinecke Reviewed-by: Christoph Hellwig Reviewed-by: Martin K. Petersen Signed-off-by: Ming Lei --- block/blk-core.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/block/blk-core.c b/block/blk-core.c index 7f11560bfddb..1fe73051fec3 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1630,8 +1630,12 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src, rq->rq_flags |= RQF_SPECIAL_PAYLOAD; rq->special_vec = rq_src->special_vec; } +#ifdef CONFIG_BLK_DEV_INTEGRITY + rq->nr_integrity_segments = rq_src->nr_integrity_segments; +#endif rq->nr_phys_segments = rq_src->nr_phys_segments; rq->ioprio = rq_src->ioprio; + rq->write_hint = rq_src->write_hint; return 0; From patchwork Sat May 2 23:54:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 11524171 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B44CF14B4 for ; Sat, 2 May 2020 23:55:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9C53D2137B for ; Sat, 2 May 2020 23:55:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bpbsxdWk" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726493AbgEBXzd (ORCPT ); Sat, 2 May 2020 19:55:33 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:46399 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725798AbgEBXzd (ORCPT ); Sat, 2 May 2020 19:55:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588463731; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/fiNh3tZPJTEEjHU8so2xEDBf/sHFznv5Ifn+GeJ+6o=; b=bpbsxdWkB2jQhQiSYEysbC3ZGnkBa4ZreQUweMWbXnf5VnfQMcrobjOMAquXNQ0vQvM4Me MxosYpPURT5tt2O3FzqHcTnu+rk39LhlkdReHIw7vCdGJ20sAmRef4yIbHRGvNvJzWwRPm vwrxtC2cD5BGZEhtmwA4A78YhB++/Jw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-92-fi8XbbMrOayEmxPXHXAjgw-1; Sat, 02 May 2020 19:55:29 -0400 X-MC-Unique: fi8XbbMrOayEmxPXHXAjgw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5347080058A; Sat, 2 May 2020 23:55:28 +0000 (UTC) Received: from localhost (ovpn-8-17.pek2.redhat.com [10.72.8.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 774D25D9CD; Sat, 2 May 2020 23:55:19 +0000 (UTC) From: Ming Lei To: Jens Axboe Cc: linux-block@vger.kernel.org, Ming Lei , John Garry , Bart Van Assche , Hannes Reinecke , Christoph Hellwig , Thomas Gleixner , Mike Snitzer , dm-devel@redhat.com, Hannes Reinecke , "Martin K . Petersen" Subject: [PATCH V9 02/11] block: add helper for copying request Date: Sun, 3 May 2020 07:54:45 +0800 Message-Id: <20200502235454.1118520-3-ming.lei@redhat.com> In-Reply-To: <20200502235454.1118520-1-ming.lei@redhat.com> References: <20200502235454.1118520-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add one new helper of blk_rq_copy_request() to copy request, and the helper will be used in this patch for re-submitting request, so make it as a block layer internal helper. Cc: John Garry Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Thomas Gleixner Cc: Mike Snitzer Cc: dm-devel@redhat.com Reviewed-by: Hannes Reinecke Reviewed-by: Christoph Hellwig Reviewed-by: Martin K. Petersen Signed-off-by: Ming Lei --- block/blk-core.c | 31 ++++++++++++++++++------------- block/blk.h | 2 ++ 2 files changed, 20 insertions(+), 13 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 1fe73051fec3..ec50d7e6be21 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1581,6 +1581,23 @@ void blk_rq_unprep_clone(struct request *rq) } EXPORT_SYMBOL_GPL(blk_rq_unprep_clone); +void blk_rq_copy_request(struct request *rq, struct request *rq_src) +{ + /* Copy attributes of the original request to the clone request. */ + rq->__sector = blk_rq_pos(rq_src); + rq->__data_len = blk_rq_bytes(rq_src); + if (rq_src->rq_flags & RQF_SPECIAL_PAYLOAD) { + rq->rq_flags |= RQF_SPECIAL_PAYLOAD; + rq->special_vec = rq_src->special_vec; + } +#ifdef CONFIG_BLK_DEV_INTEGRITY + rq->nr_integrity_segments = rq_src->nr_integrity_segments; +#endif + rq->nr_phys_segments = rq_src->nr_phys_segments; + rq->ioprio = rq_src->ioprio; + rq->write_hint = rq_src->write_hint; +} + /** * blk_rq_prep_clone - Helper function to setup clone request * @rq: the request to be setup @@ -1623,19 +1640,7 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src, rq->bio = rq->biotail = bio; } - /* Copy attributes of the original request to the clone request. */ - rq->__sector = blk_rq_pos(rq_src); - rq->__data_len = blk_rq_bytes(rq_src); - if (rq_src->rq_flags & RQF_SPECIAL_PAYLOAD) { - rq->rq_flags |= RQF_SPECIAL_PAYLOAD; - rq->special_vec = rq_src->special_vec; - } -#ifdef CONFIG_BLK_DEV_INTEGRITY - rq->nr_integrity_segments = rq_src->nr_integrity_segments; -#endif - rq->nr_phys_segments = rq_src->nr_phys_segments; - rq->ioprio = rq_src->ioprio; - rq->write_hint = rq_src->write_hint; + blk_rq_copy_request(rq, rq_src); return 0; diff --git a/block/blk.h b/block/blk.h index 73bd3b1c6938..73dd86dc8f47 100644 --- a/block/blk.h +++ b/block/blk.h @@ -120,6 +120,8 @@ static inline void blk_rq_bio_prep(struct request *rq, struct bio *bio, rq->rq_disk = bio->bi_disk; } +void blk_rq_copy_request(struct request *rq, struct request *rq_src); + #ifdef CONFIG_BLK_DEV_INTEGRITY void blk_flush_integrity(void); bool __bio_integrity_endio(struct bio *); From patchwork Sat May 2 23:54:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 11524173 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2032714B4 for ; Sat, 2 May 2020 23:55:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0926C2137B for ; Sat, 2 May 2020 23:55:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ONnVrHjZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726544AbgEBXzl (ORCPT ); Sat, 2 May 2020 19:55:41 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:23330 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725798AbgEBXzl (ORCPT ); Sat, 2 May 2020 19:55:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588463740; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V6NEHSs6Kznu7F3ENuHAG2a0qaMyrMHyvbHE8GdQgTQ=; b=ONnVrHjZzMjHP4TnDBqRpEpuvc4TmKwR9uyF/aFsofN4rql4jW+y93+I5I/bYafV4AHwjQ WdzvgC/gFoM7dyIYfAn9qeXNhFzldfisNIVs4O3szKBkFeGtbyfUOEtkm//KWQXyruG1bM 3Ysnk4oZXPFJDjo5sJGucgtHHM17A2k= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-395-gQsh4aGoP0yO9MFTBeqgYw-1; Sat, 02 May 2020 19:55:36 -0400 X-MC-Unique: gQsh4aGoP0yO9MFTBeqgYw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A1F6C180082C; Sat, 2 May 2020 23:55:34 +0000 (UTC) Received: from localhost (ovpn-8-17.pek2.redhat.com [10.72.8.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 436876B8D7; Sat, 2 May 2020 23:55:30 +0000 (UTC) From: Ming Lei To: Jens Axboe Cc: linux-block@vger.kernel.org, Ming Lei , Bart Van Assche , Hannes Reinecke , Christoph Hellwig , Thomas Gleixner , John Garry , Hannes Reinecke , "Martin K . Petersen" Subject: [PATCH V9 03/11] blk-mq: mark blk_mq_get_driver_tag as static Date: Sun, 3 May 2020 07:54:46 +0800 Message-Id: <20200502235454.1118520-4-ming.lei@redhat.com> In-Reply-To: <20200502235454.1118520-1-ming.lei@redhat.com> References: <20200502235454.1118520-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Now all callers of blk_mq_get_driver_tag are in blk-mq.c, so mark it as static. Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Thomas Gleixner Cc: John Garry Reviewed-by: Christoph Hellwig Reviewed-by: Hannes Reinecke Reviewed-by: Martin K. Petersen Signed-off-by: Ming Lei --- block/blk-mq.c | 2 +- block/blk-mq.h | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index bcc3a2397d4a..de4e6654258d 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1015,7 +1015,7 @@ static inline unsigned int queued_to_index(unsigned int queued) return min(BLK_MQ_MAX_DISPATCH_ORDER - 1, ilog2(queued) + 1); } -bool blk_mq_get_driver_tag(struct request *rq) +static bool blk_mq_get_driver_tag(struct request *rq) { struct blk_mq_alloc_data data = { .q = rq->q, diff --git a/block/blk-mq.h b/block/blk-mq.h index 10bfdfb494fa..e7d1da4b1f73 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -44,7 +44,6 @@ bool blk_mq_dispatch_rq_list(struct request_queue *, struct list_head *, bool); void blk_mq_add_to_requeue_list(struct request *rq, bool at_head, bool kick_requeue_list); void blk_mq_flush_busy_ctxs(struct blk_mq_hw_ctx *hctx, struct list_head *list); -bool blk_mq_get_driver_tag(struct request *rq); struct request *blk_mq_dequeue_from_ctx(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *start); From patchwork Sat May 2 23:54:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 11524175 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9ACFA81 for ; Sat, 2 May 2020 23:55:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7C5DD20757 for ; Sat, 2 May 2020 23:55:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RQXyQ4Vk" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726561AbgEBXzq (ORCPT ); Sat, 2 May 2020 19:55:46 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:38574 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725798AbgEBXzp (ORCPT ); Sat, 2 May 2020 19:55:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588463743; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g/7trEk+LWF+d2n+fqBGCOoZW/cakUkrdcii0umzqI8=; b=RQXyQ4Vko7PYA6IMEcJiv918qwFwVEtolvxZnE9bdmc9+74OXYO3nrgNJuO744ZuLySPhI JL993xK2Mwt1tAn9ZZc3uEg6ry3uZpeRP296WpetfAZuKK0HE8sQTYGAs9T1BozUQ1hdkq hviXTtg2XT9ezq5/SXCNbabhxwL9T68= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-341-cdQ3BLJLOB-pZLKUr73CBw-1; Sat, 02 May 2020 19:55:39 -0400 X-MC-Unique: cdQ3BLJLOB-pZLKUr73CBw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 389D587308E; Sat, 2 May 2020 23:55:38 +0000 (UTC) Received: from localhost (ovpn-8-17.pek2.redhat.com [10.72.8.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5D4D76B8D7; Sat, 2 May 2020 23:55:37 +0000 (UTC) From: Ming Lei To: Jens Axboe Cc: linux-block@vger.kernel.org, Ming Lei , Bart Van Assche , Hannes Reinecke , Christoph Hellwig , Thomas Gleixner , John Garry , Hannes Reinecke Subject: [PATCH V9 04/11] blk-mq: assign rq->tag in blk_mq_get_driver_tag Date: Sun, 3 May 2020 07:54:47 +0800 Message-Id: <20200502235454.1118520-5-ming.lei@redhat.com> In-Reply-To: <20200502235454.1118520-1-ming.lei@redhat.com> References: <20200502235454.1118520-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Especially for none elevator, rq->tag is assigned after the request is allocated, so there isn't any way to figure out if one request is in being dispatched. Also the code path wrt. driver tag becomes a bit difference between none and io scheduler. When one hctx becomes inactive, we have to prevent any request from being dispatched to LLD. And get driver tag provides one perfect chance to do that. Meantime we can drain any such requests by checking if rq->tag is assigned. So only assign rq->tag until blk_mq_get_driver_tag() is called. This way also simplifies code of dealing with driver tag a lot. Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Thomas Gleixner Cc: John Garry Reviewed-by: Hannes Reinecke Reviewed-by: Christoph Hellwig Signed-off-by: Ming Lei --- block/blk-flush.c | 18 ++---------- block/blk-mq.c | 75 ++++++++++++++++++++++++----------------------- block/blk-mq.h | 21 +++++++------ block/blk.h | 5 ---- 4 files changed, 51 insertions(+), 68 deletions(-) diff --git a/block/blk-flush.c b/block/blk-flush.c index c7f396e3d5e2..977edf95d711 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -236,13 +236,8 @@ static void flush_end_io(struct request *flush_rq, blk_status_t error) error = fq->rq_status; hctx = flush_rq->mq_hctx; - if (!q->elevator) { - blk_mq_tag_set_rq(hctx, flush_rq->tag, fq->orig_rq); - flush_rq->tag = -1; - } else { - blk_mq_put_driver_tag(flush_rq); - flush_rq->internal_tag = -1; - } + flush_rq->internal_tag = -1; + blk_mq_put_driver_tag(flush_rq); running = &fq->flush_queue[fq->flush_running_idx]; BUG_ON(fq->flush_pending_idx == fq->flush_running_idx); @@ -317,14 +312,7 @@ static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq, flush_rq->mq_ctx = first_rq->mq_ctx; flush_rq->mq_hctx = first_rq->mq_hctx; - if (!q->elevator) { - fq->orig_rq = first_rq; - flush_rq->tag = first_rq->tag; - blk_mq_tag_set_rq(flush_rq->mq_hctx, first_rq->tag, flush_rq); - } else { - flush_rq->internal_tag = first_rq->internal_tag; - } - + flush_rq->internal_tag = first_rq->internal_tag; flush_rq->cmd_flags = REQ_OP_FLUSH | REQ_PREFLUSH; flush_rq->cmd_flags |= (flags & REQ_DRV) | (flags & REQ_FAILFAST_MASK); flush_rq->rq_flags |= RQF_FLUSH_SEQ; diff --git a/block/blk-mq.c b/block/blk-mq.c index de4e6654258d..6569d802e0d8 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -276,18 +276,8 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data, struct request *rq = tags->static_rqs[tag]; req_flags_t rq_flags = 0; - if (data->flags & BLK_MQ_REQ_INTERNAL) { - rq->tag = -1; - rq->internal_tag = tag; - } else { - if (data->hctx->flags & BLK_MQ_F_TAG_SHARED) { - rq_flags = RQF_MQ_INFLIGHT; - atomic_inc(&data->hctx->nr_active); - } - rq->tag = tag; - rq->internal_tag = -1; - data->hctx->tags->rqs[rq->tag] = rq; - } + rq->internal_tag = tag; + rq->tag = -1; /* csd/requeue_work/fifo_time is initialized before use */ rq->q = data->q; @@ -471,14 +461,18 @@ static void __blk_mq_free_request(struct request *rq) struct request_queue *q = rq->q; struct blk_mq_ctx *ctx = rq->mq_ctx; struct blk_mq_hw_ctx *hctx = rq->mq_hctx; - const int sched_tag = rq->internal_tag; blk_pm_mark_last_busy(rq); rq->mq_hctx = NULL; - if (rq->tag != -1) - blk_mq_put_tag(hctx->tags, ctx, rq->tag); - if (sched_tag != -1) - blk_mq_put_tag(hctx->sched_tags, ctx, sched_tag); + + if (hctx->sched_tags) { + if (rq->tag >= 0) + blk_mq_put_tag(hctx->tags, ctx, rq->tag); + blk_mq_put_tag(hctx->sched_tags, ctx, rq->internal_tag); + } else { + blk_mq_put_tag(hctx->tags, ctx, rq->internal_tag); + } + blk_mq_sched_restart(hctx); blk_queue_exit(q); } @@ -526,7 +520,7 @@ inline void __blk_mq_end_request(struct request *rq, blk_status_t error) blk_stat_add(rq, now); } - if (rq->internal_tag != -1) + if (rq->q->elevator && rq->internal_tag != -1) blk_mq_sched_completed_request(rq, now); blk_account_io_done(rq, now); @@ -1015,33 +1009,40 @@ static inline unsigned int queued_to_index(unsigned int queued) return min(BLK_MQ_MAX_DISPATCH_ORDER - 1, ilog2(queued) + 1); } -static bool blk_mq_get_driver_tag(struct request *rq) +static bool __blk_mq_get_driver_tag(struct request *rq) { struct blk_mq_alloc_data data = { - .q = rq->q, - .hctx = rq->mq_hctx, - .flags = BLK_MQ_REQ_NOWAIT, - .cmd_flags = rq->cmd_flags, + .q = rq->q, + .hctx = rq->mq_hctx, + .flags = BLK_MQ_REQ_NOWAIT, + .cmd_flags = rq->cmd_flags, }; - bool shared; - if (rq->tag != -1) - return true; + if (data.hctx->sched_tags) { + if (blk_mq_tag_is_reserved(data.hctx->sched_tags, + rq->internal_tag)) + data.flags |= BLK_MQ_REQ_RESERVED; + rq->tag = blk_mq_get_tag(&data); + } else { + rq->tag = rq->internal_tag; + } - if (blk_mq_tag_is_reserved(data.hctx->sched_tags, rq->internal_tag)) - data.flags |= BLK_MQ_REQ_RESERVED; + if (rq->tag == -1) + return false; - shared = blk_mq_tag_busy(data.hctx); - rq->tag = blk_mq_get_tag(&data); - if (rq->tag >= 0) { - if (shared) { - rq->rq_flags |= RQF_MQ_INFLIGHT; - atomic_inc(&data.hctx->nr_active); - } - data.hctx->tags->rqs[rq->tag] = rq; + if (blk_mq_tag_busy(data.hctx)) { + rq->rq_flags |= RQF_MQ_INFLIGHT; + atomic_inc(&data.hctx->nr_active); } + data.hctx->tags->rqs[rq->tag] = rq; + return true; +} - return rq->tag != -1; +static bool blk_mq_get_driver_tag(struct request *rq) +{ + if (rq->tag != -1) + return true; + return __blk_mq_get_driver_tag(rq); } static int blk_mq_dispatch_wake(wait_queue_entry_t *wait, unsigned mode, diff --git a/block/blk-mq.h b/block/blk-mq.h index e7d1da4b1f73..d0c72d7d07c8 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -196,26 +196,25 @@ static inline bool blk_mq_get_dispatch_budget(struct blk_mq_hw_ctx *hctx) return true; } -static inline void __blk_mq_put_driver_tag(struct blk_mq_hw_ctx *hctx, - struct request *rq) +static inline void blk_mq_put_driver_tag(struct request *rq) { - blk_mq_put_tag(hctx->tags, rq->mq_ctx, rq->tag); + struct blk_mq_hw_ctx *hctx = rq->mq_hctx; + int tag = rq->tag; + + if (tag < 0) + return; + rq->tag = -1; + if (hctx->sched_tags) + blk_mq_put_tag(hctx->tags, rq->mq_ctx, tag); + if (rq->rq_flags & RQF_MQ_INFLIGHT) { rq->rq_flags &= ~RQF_MQ_INFLIGHT; atomic_dec(&hctx->nr_active); } } -static inline void blk_mq_put_driver_tag(struct request *rq) -{ - if (rq->tag == -1 || rq->internal_tag == -1) - return; - - __blk_mq_put_driver_tag(rq->mq_hctx, rq); -} - static inline void blk_mq_clear_mq_map(struct blk_mq_queue_map *qmap) { int cpu; diff --git a/block/blk.h b/block/blk.h index 73dd86dc8f47..591cc07e40f9 100644 --- a/block/blk.h +++ b/block/blk.h @@ -26,11 +26,6 @@ struct blk_flush_queue { struct list_head flush_data_in_flight; struct request *flush_rq; - /* - * flush_rq shares tag with this rq, both can't be active - * at the same time - */ - struct request *orig_rq; struct lock_class_key key; spinlock_t mq_flush_lock; }; From patchwork Sat May 2 23:54:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 11524181 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED6F281 for ; Sat, 2 May 2020 23:55:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D56C720721 for ; Sat, 2 May 2020 23:55:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="FYOj96CY" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726556AbgEBXzw (ORCPT ); Sat, 2 May 2020 19:55:52 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:42315 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725798AbgEBXzw (ORCPT ); Sat, 2 May 2020 19:55:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588463750; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pwsxvx9XGKVjjWm3npf1QZjbGRHrFSBwUvaLtvz+20c=; b=FYOj96CY9cuLW8p1lqvjqu7wnmvxSbYed4nut2HuDyRqTICDotx3CO0CzcA3mdfS+K90sH rpoFZimphDGxS1PY7r91CEQBxPaMt8KcHwFdX09Igygv+94HQQdMEGOiYIsW8/v60KgDy7 rH8HB9RhwCL16a0Gk8Evc2l0QwTpZkk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-25-uO_84yfWN-qPeFCA6dbR1Q-1; Sat, 02 May 2020 19:55:44 -0400 X-MC-Unique: uO_84yfWN-qPeFCA6dbR1Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CCA4D800D24; Sat, 2 May 2020 23:55:42 +0000 (UTC) Received: from localhost (ovpn-8-17.pek2.redhat.com [10.72.8.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id DE1D76B8D7; Sat, 2 May 2020 23:55:41 +0000 (UTC) From: Ming Lei To: Jens Axboe Cc: linux-block@vger.kernel.org, Ming Lei , John Garry , Bart Van Assche , Hannes Reinecke , Christoph Hellwig , Thomas Gleixner Subject: [PATCH V9 05/11] blk-mq: support rq filter callback when iterating rqs Date: Sun, 3 May 2020 07:54:48 +0800 Message-Id: <20200502235454.1118520-6-ming.lei@redhat.com> In-Reply-To: <20200502235454.1118520-1-ming.lei@redhat.com> References: <20200502235454.1118520-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Now request is thought as in-flight only when its state is updated as MQ_RQ_IN_FLIGHT, which is done by driver via blk_mq_start_request(). Actually from blk-mq's view, one rq can be thought as in-flight after its tag is >= 0. Passing one rq filter callback so that we can iterating requests very flexible. Implement blk_mq_all_tag_busy_iter() which accepts a 'busy_fn' argument to filter over which commands to iterate, and make the existing blk_mq_tag_busy_iter() a wrapper for the new function. Cc: John Garry Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Thomas Gleixner Reviewed-by: Christoph Hellwig Signed-off-by: Ming Lei Reviewed-by: Hannes Reinecke --- block/blk-mq-tag.c | 39 +++++++++++++++++++++++++++------------ block/blk-mq-tag.h | 4 ++++ 2 files changed, 31 insertions(+), 12 deletions(-) diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index 586c9d6e904a..2e43b827c96d 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -255,6 +255,7 @@ static void bt_for_each(struct blk_mq_hw_ctx *hctx, struct sbitmap_queue *bt, struct bt_tags_iter_data { struct blk_mq_tags *tags; busy_tag_iter_fn *fn; + busy_rq_iter_fn *busy_rq_fn; void *data; bool reserved; }; @@ -274,7 +275,7 @@ static bool bt_tags_iter(struct sbitmap *bitmap, unsigned int bitnr, void *data) * test and set the bit before assining ->rqs[]. */ rq = tags->rqs[bitnr]; - if (rq && blk_mq_request_started(rq)) + if (rq && iter_data->busy_rq_fn(rq, iter_data->data, reserved)) return iter_data->fn(rq, iter_data->data, reserved); return true; @@ -294,11 +295,13 @@ static bool bt_tags_iter(struct sbitmap *bitmap, unsigned int bitnr, void *data) * bitmap_tags member of struct blk_mq_tags. */ static void bt_tags_for_each(struct blk_mq_tags *tags, struct sbitmap_queue *bt, - busy_tag_iter_fn *fn, void *data, bool reserved) + busy_tag_iter_fn *fn, busy_rq_iter_fn *busy_rq_fn, + void *data, bool reserved) { struct bt_tags_iter_data iter_data = { .tags = tags, .fn = fn, + .busy_rq_fn = busy_rq_fn, .data = data, .reserved = reserved, }; @@ -310,19 +313,30 @@ static void bt_tags_for_each(struct blk_mq_tags *tags, struct sbitmap_queue *bt, /** * blk_mq_all_tag_busy_iter - iterate over all started requests in a tag map * @tags: Tag map to iterate over. - * @fn: Pointer to the function that will be called for each started - * request. @fn will be called as follows: @fn(rq, @priv, - * reserved) where rq is a pointer to a request. 'reserved' - * indicates whether or not @rq is a reserved request. Return - * true to continue iterating tags, false to stop. + * @fn: Pointer to the function that will be called for each request + * when .busy_rq_fn(rq) returns true. @fn will be called as + * follows: @fn(rq, @priv, reserved) where rq is a pointer to a + * request. 'reserved' indicates whether or not @rq is a reserved + * request. Return true to continue iterating tags, false to stop. + * @busy_rq_fn: Pointer to the function that will be called for each request, + * @busy_rq_fn's type is same with @fn. Only when @busy_rq_fn(rq, + * @priv, reserved) returns true, @fn will be called on this rq. * @priv: Will be passed as second argument to @fn. */ -static void blk_mq_all_tag_busy_iter(struct blk_mq_tags *tags, - busy_tag_iter_fn *fn, void *priv) +void blk_mq_all_tag_busy_iter(struct blk_mq_tags *tags, + busy_tag_iter_fn *fn, busy_rq_iter_fn *busy_rq_fn, + void *priv) { if (tags->nr_reserved_tags) - bt_tags_for_each(tags, &tags->breserved_tags, fn, priv, true); - bt_tags_for_each(tags, &tags->bitmap_tags, fn, priv, false); + bt_tags_for_each(tags, &tags->breserved_tags, fn, busy_rq_fn, + priv, true); + bt_tags_for_each(tags, &tags->bitmap_tags, fn, busy_rq_fn, priv, false); +} + +static bool blk_mq_default_busy_rq(struct request *rq, void *data, + bool reserved) +{ + return blk_mq_request_started(rq); } /** @@ -342,7 +356,8 @@ void blk_mq_tagset_busy_iter(struct blk_mq_tag_set *tagset, for (i = 0; i < tagset->nr_hw_queues; i++) { if (tagset->tags && tagset->tags[i]) - blk_mq_all_tag_busy_iter(tagset->tags[i], fn, priv); + blk_mq_all_tag_busy_iter(tagset->tags[i], fn, + blk_mq_default_busy_rq, priv); } } EXPORT_SYMBOL(blk_mq_tagset_busy_iter); diff --git a/block/blk-mq-tag.h b/block/blk-mq-tag.h index 2b8321efb682..fdf095d513e5 100644 --- a/block/blk-mq-tag.h +++ b/block/blk-mq-tag.h @@ -21,6 +21,7 @@ struct blk_mq_tags { struct list_head page_list; }; +typedef bool (busy_rq_iter_fn)(struct request *, void *, bool); extern struct blk_mq_tags *blk_mq_init_tags(unsigned int nr_tags, unsigned int reserved_tags, int node, int alloc_policy); extern void blk_mq_free_tags(struct blk_mq_tags *tags); @@ -34,6 +35,9 @@ extern int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx, extern void blk_mq_tag_wakeup_all(struct blk_mq_tags *tags, bool); void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_iter_fn *fn, void *priv); +void blk_mq_all_tag_busy_iter(struct blk_mq_tags *tags, + busy_tag_iter_fn *fn, busy_rq_iter_fn *busy_rq_fn, + void *priv); static inline struct sbq_wait_state *bt_wait_ptr(struct sbitmap_queue *bt, struct blk_mq_hw_ctx *hctx) From patchwork Sat May 2 23:54:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 11524183 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D924D81 for ; Sat, 2 May 2020 23:55:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BC98120757 for ; Sat, 2 May 2020 23:55:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Y23fFyub" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726570AbgEBXzz (ORCPT ); Sat, 2 May 2020 19:55:55 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:28871 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725798AbgEBXzz (ORCPT ); Sat, 2 May 2020 19:55:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588463753; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4cYmu8u7Su/YfxI55lR2XgTo3nEgPIwvTivtINlM90A=; b=Y23fFyubpw4a2l7oMEAY1mLXNpdflMGQfJZUcNGXkzdPh8vt1yVlAT09Zx5fZXyX+3GawI cylVJ5KZdDs6WvckiIo4HhF55/80hdzur0NmHq0SwNY+S0U4gFt564CiGOtnhM2vAZt0vu 4OTsyVj/B4ULRTX6Vg6soamY8rpc3FA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-359-4OzhKVm1OW2QX_uy5z9-1g-1; Sat, 02 May 2020 19:55:47 -0400 X-MC-Unique: 4OzhKVm1OW2QX_uy5z9-1g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 65AEB87308F; Sat, 2 May 2020 23:55:46 +0000 (UTC) Received: from localhost (ovpn-8-17.pek2.redhat.com [10.72.8.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8610260612; Sat, 2 May 2020 23:55:45 +0000 (UTC) From: Ming Lei To: Jens Axboe Cc: linux-block@vger.kernel.org, Ming Lei , John Garry , Bart Van Assche , Hannes Reinecke , Christoph Hellwig , Thomas Gleixner Subject: [PATCH V9 06/11] blk-mq: prepare for draining IO when hctx's all CPUs are offline Date: Sun, 3 May 2020 07:54:49 +0800 Message-Id: <20200502235454.1118520-7-ming.lei@redhat.com> In-Reply-To: <20200502235454.1118520-1-ming.lei@redhat.com> References: <20200502235454.1118520-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Most of blk-mq drivers depend on managed IRQ's auto-affinity to setup up queue mapping. Thomas mentioned the following point[1]: " That was the constraint of managed interrupts from the very beginning: The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again. " However, current blk-mq implementation doesn't quiesce hw queue before the last CPU in the hctx is shutdown. Even worse, CPUHP_BLK_MQ_DEAD is one cpuhp state handled after the CPU is down, so there isn't any chance to quiesce hctx for blk-mq wrt. CPU hotplug. Add new cpuhp state of CPUHP_AP_BLK_MQ_ONLINE for blk-mq to stop queues and wait for completion of in-flight requests. We will stop hw queue and wait for completion of in-flight requests when one hctx is becoming dead in the following patch. This way may cause dead-lock for some stacking blk-mq drivers, such as dm-rq and loop. Add blk-mq flag of BLK_MQ_F_NO_MANAGED_IRQ and mark it for dm-rq and loop, so we needn't to wait for completion of in-flight requests from dm-rq & loop, then the potential dead-lock can be avoided. [1] https://lore.kernel.org/linux-block/alpine.DEB.2.21.1904051331270.1802@nanos.tec.linutronix.de/ Cc: John Garry Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Thomas Gleixner Signed-off-by: Ming Lei --- block/blk-mq-debugfs.c | 1 + block/blk-mq.c | 19 +++++++++++++++++++ drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 3 +++ include/linux/cpuhotplug.h | 1 + 6 files changed, 26 insertions(+), 2 deletions(-) diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 96b7a35c898a..ddec58743e88 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -239,6 +239,7 @@ static const char *const hctx_flag_name[] = { HCTX_FLAG_NAME(TAG_SHARED), HCTX_FLAG_NAME(BLOCKING), HCTX_FLAG_NAME(NO_SCHED), + HCTX_FLAG_NAME(NO_MANAGED_IRQ), }; #undef HCTX_FLAG_NAME diff --git a/block/blk-mq.c b/block/blk-mq.c index 6569d802e0d8..54c107be7a47 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2273,6 +2273,16 @@ int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, return -ENOMEM; } +static int blk_mq_hctx_notify_online(unsigned int cpu, struct hlist_node *node) +{ + return 0; +} + +static int blk_mq_hctx_notify_offline(unsigned int cpu, struct hlist_node *node) +{ + return 0; +} + /* * 'cpu' is going away. splice any existing rq_list entries from this * software queue to the hw queue dispatch list, and ensure that it @@ -2309,6 +2319,9 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node) static void blk_mq_remove_cpuhp(struct blk_mq_hw_ctx *hctx) { + if (!(hctx->flags & BLK_MQ_F_NO_MANAGED_IRQ)) + cpuhp_state_remove_instance_nocalls(CPUHP_AP_BLK_MQ_ONLINE, + &hctx->cpuhp_online); cpuhp_state_remove_instance_nocalls(CPUHP_BLK_MQ_DEAD, &hctx->cpuhp_dead); } @@ -2368,6 +2381,9 @@ static int blk_mq_init_hctx(struct request_queue *q, { hctx->queue_num = hctx_idx; + if (!(hctx->flags & BLK_MQ_F_NO_MANAGED_IRQ)) + cpuhp_state_add_instance_nocalls(CPUHP_AP_BLK_MQ_ONLINE, + &hctx->cpuhp_online); cpuhp_state_add_instance_nocalls(CPUHP_BLK_MQ_DEAD, &hctx->cpuhp_dead); hctx->tags = set->tags[hctx_idx]; @@ -3621,6 +3637,9 @@ static int __init blk_mq_init(void) { cpuhp_setup_state_multi(CPUHP_BLK_MQ_DEAD, "block/mq:dead", NULL, blk_mq_hctx_notify_dead); + cpuhp_setup_state_multi(CPUHP_AP_BLK_MQ_ONLINE, "block/mq:online", + blk_mq_hctx_notify_online, + blk_mq_hctx_notify_offline); return 0; } subsys_initcall(blk_mq_init); diff --git a/drivers/block/loop.c b/drivers/block/loop.c index da693e6a834e..784f2e038b55 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -2037,7 +2037,7 @@ static int loop_add(struct loop_device **l, int i) lo->tag_set.queue_depth = 128; lo->tag_set.numa_node = NUMA_NO_NODE; lo->tag_set.cmd_size = sizeof(struct loop_cmd); - lo->tag_set.flags = BLK_MQ_F_SHOULD_MERGE; + lo->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_NO_MANAGED_IRQ; lo->tag_set.driver_data = lo; err = blk_mq_alloc_tag_set(&lo->tag_set); diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c index 3f8577e2c13b..5f1ff70ac029 100644 --- a/drivers/md/dm-rq.c +++ b/drivers/md/dm-rq.c @@ -547,7 +547,7 @@ int dm_mq_init_request_queue(struct mapped_device *md, struct dm_table *t) md->tag_set->ops = &dm_mq_ops; md->tag_set->queue_depth = dm_get_blk_mq_queue_depth(); md->tag_set->numa_node = md->numa_node_id; - md->tag_set->flags = BLK_MQ_F_SHOULD_MERGE; + md->tag_set->flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_NO_MANAGED_IRQ; md->tag_set->nr_hw_queues = dm_get_blk_mq_nr_hw_queues(); md->tag_set->driver_data = md; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index d7307795439a..3763207d88eb 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -140,6 +140,8 @@ struct blk_mq_hw_ctx { */ atomic_t nr_active; + /** @cpuhp_online: List to store request if CPU is going to die */ + struct hlist_node cpuhp_online; /** @cpuhp_dead: List to store request if some CPU die. */ struct hlist_node cpuhp_dead; /** @kobj: Kernel object for sysfs. */ @@ -391,6 +393,7 @@ struct blk_mq_ops { enum { BLK_MQ_F_SHOULD_MERGE = 1 << 0, BLK_MQ_F_TAG_SHARED = 1 << 1, + BLK_MQ_F_NO_MANAGED_IRQ = 1 << 2, BLK_MQ_F_BLOCKING = 1 << 5, BLK_MQ_F_NO_SCHED = 1 << 6, BLK_MQ_F_ALLOC_POLICY_START_BIT = 8, diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 77d70b633531..24b3a77810b6 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -152,6 +152,7 @@ enum cpuhp_state { CPUHP_AP_SMPBOOT_THREADS, CPUHP_AP_X86_VDSO_VMA_ONLINE, CPUHP_AP_IRQ_AFFINITY_ONLINE, + CPUHP_AP_BLK_MQ_ONLINE, CPUHP_AP_ARM_MVEBU_SYNC_CLOCKS, CPUHP_AP_X86_INTEL_EPB_ONLINE, CPUHP_AP_PERF_ONLINE, From patchwork Sat May 2 23:54:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 11524185 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B2F7514B4 for ; Sat, 2 May 2020 23:55:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 95C3E2137B for ; Sat, 2 May 2020 23:55:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fUojJOB8" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726579AbgEBXz5 (ORCPT ); Sat, 2 May 2020 19:55:57 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:20227 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725798AbgEBXz4 (ORCPT ); Sat, 2 May 2020 19:55:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588463755; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jmiZH921k9UZn6ArAL1L3ESjAJcRXKac7zyRsLtPvWg=; b=fUojJOB8gDtHxVcI6PipthHLVvPDQTc+rCZxMGdaZvicdlmQbxfqkAZz6rz/gn/OFceWJ7 fOSW396l5794Mq57Q1kABlTQxmMkuLInnuFOIyrvJw3LUhb65obCCQUEvG9gQFCTeEb4bL E8M1FYaNjOp7BG1TP0D8wV1/IejBlQ0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-158-d9mbVnlpPZuIu0wfVk5yDw-1; Sat, 02 May 2020 19:55:53 -0400 X-MC-Unique: d9mbVnlpPZuIu0wfVk5yDw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5D6A287308F; Sat, 2 May 2020 23:55:52 +0000 (UTC) Received: from localhost (ovpn-8-17.pek2.redhat.com [10.72.8.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF46460C05; Sat, 2 May 2020 23:55:48 +0000 (UTC) From: Ming Lei To: Jens Axboe Cc: linux-block@vger.kernel.org, Ming Lei , John Garry , Bart Van Assche , Hannes Reinecke , Christoph Hellwig , Thomas Gleixner Subject: [PATCH V9 07/11] blk-mq: stop to handle IO and drain IO before hctx becomes inactive Date: Sun, 3 May 2020 07:54:50 +0800 Message-Id: <20200502235454.1118520-8-ming.lei@redhat.com> In-Reply-To: <20200502235454.1118520-1-ming.lei@redhat.com> References: <20200502235454.1118520-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Before one CPU becomes offline, check if it is the last online CPU of hctx. If yes, mark this hctx as inactive, meantime wait for completion of all in-flight IOs originated from this hctx. Meantime check if this hctx has become inactive in blk_mq_get_driver_tag(), if yes, release the allocated tag. This way guarantees that there isn't any inflight IO before shutdowning the managed IRQ line when all CPUs of this IRQ line is offline. Cc: John Garry Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Thomas Gleixner Signed-off-by: Ming Lei --- block/blk-mq-debugfs.c | 1 + block/blk-mq.c | 124 +++++++++++++++++++++++++++++++++++++---- include/linux/blk-mq.h | 3 + 3 files changed, 117 insertions(+), 11 deletions(-) diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index ddec58743e88..dc66cb689d2f 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -213,6 +213,7 @@ static const char *const hctx_state_name[] = { HCTX_STATE_NAME(STOPPED), HCTX_STATE_NAME(TAG_ACTIVE), HCTX_STATE_NAME(SCHED_RESTART), + HCTX_STATE_NAME(INACTIVE), }; #undef HCTX_STATE_NAME diff --git a/block/blk-mq.c b/block/blk-mq.c index 54c107be7a47..4a2250ac4fbb 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1038,11 +1038,36 @@ static bool __blk_mq_get_driver_tag(struct request *rq) return true; } -static bool blk_mq_get_driver_tag(struct request *rq) +static bool blk_mq_get_driver_tag(struct request *rq, bool direct_issue) { if (rq->tag != -1) return true; - return __blk_mq_get_driver_tag(rq); + + if (!__blk_mq_get_driver_tag(rq)) + return false; + /* + * In case that direct issue IO process is migrated to other CPU + * which may not belong to this hctx, add one memory barrier so we + * can order driver tag assignment and checking BLK_MQ_S_INACTIVE. + * Otherwise, barrier() is enough given both setting BLK_MQ_S_INACTIVE + * and driver tag assignment are run on the same CPU because + * BLK_MQ_S_INACTIVE is only set after the last CPU of this hctx is + * becoming offline. + * + * Process migration might happen after the check on current processor + * id, smp_mb() is implied by processor migration, so no need to worry + * about it. + */ + if (unlikely(direct_issue && rq->mq_ctx->cpu != raw_smp_processor_id())) + smp_mb(); + else + barrier(); + + if (unlikely(test_bit(BLK_MQ_S_INACTIVE, &rq->mq_hctx->state))) { + blk_mq_put_driver_tag(rq); + return false; + } + return true; } static int blk_mq_dispatch_wake(wait_queue_entry_t *wait, unsigned mode, @@ -1091,7 +1116,7 @@ static bool blk_mq_mark_tag_wait(struct blk_mq_hw_ctx *hctx, * Don't clear RESTART here, someone else could have set it. * At most this will cost an extra queue run. */ - return blk_mq_get_driver_tag(rq); + return blk_mq_get_driver_tag(rq, false); } wait = &hctx->dispatch_wait; @@ -1117,7 +1142,7 @@ static bool blk_mq_mark_tag_wait(struct blk_mq_hw_ctx *hctx, * allocation failure and adding the hardware queue to the wait * queue. */ - ret = blk_mq_get_driver_tag(rq); + ret = blk_mq_get_driver_tag(rq, false); if (!ret) { spin_unlock(&hctx->dispatch_wait_lock); spin_unlock_irq(&wq->lock); @@ -1218,7 +1243,7 @@ bool blk_mq_dispatch_rq_list(struct request_queue *q, struct list_head *list, break; } - if (!blk_mq_get_driver_tag(rq)) { + if (!blk_mq_get_driver_tag(rq, false)) { /* * The initial allocation attempt failed, so we need to * rerun the hardware queue when a tag is freed. The @@ -1250,7 +1275,7 @@ bool blk_mq_dispatch_rq_list(struct request_queue *q, struct list_head *list, bd.last = true; else { nxt = list_first_entry(list, struct request, queuelist); - bd.last = !blk_mq_get_driver_tag(nxt); + bd.last = !blk_mq_get_driver_tag(nxt, false); } ret = q->mq_ops->queue_rq(hctx, &bd); @@ -1864,7 +1889,7 @@ static blk_status_t __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, if (!blk_mq_get_dispatch_budget(hctx)) goto insert; - if (!blk_mq_get_driver_tag(rq)) { + if (!blk_mq_get_driver_tag(rq, true)) { blk_mq_put_dispatch_budget(hctx); goto insert; } @@ -2273,13 +2298,87 @@ int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, return -ENOMEM; } -static int blk_mq_hctx_notify_online(unsigned int cpu, struct hlist_node *node) +struct count_inflight_data { + unsigned count; + struct blk_mq_hw_ctx *hctx; +}; + +static bool blk_mq_count_inflight_rq(struct request *rq, void *data, + bool reserved) { - return 0; + struct count_inflight_data *count_data = data; + + /* + * Can't check rq's state because it is updated to MQ_RQ_IN_FLIGHT + * in blk_mq_start_request(), at that time we can't prevent this rq + * from being issued. + * + * So check if driver tag is assigned, if yes, count this rq as + * inflight. + */ + if (rq->tag >= 0 && rq->mq_hctx == count_data->hctx) + count_data->count++; + + return true; +} + +static bool blk_mq_inflight_rq(struct request *rq, void *data, + bool reserved) +{ + return rq->tag >= 0; +} + +static unsigned blk_mq_tags_inflight_rqs(struct blk_mq_hw_ctx *hctx) +{ + struct count_inflight_data count_data = { + .hctx = hctx, + }; + + blk_mq_all_tag_busy_iter(hctx->tags, blk_mq_count_inflight_rq, + blk_mq_inflight_rq, &count_data); + return count_data.count; +} + +static inline bool blk_mq_last_cpu_in_hctx(unsigned int cpu, + struct blk_mq_hw_ctx *hctx) +{ + if (cpumask_next_and(-1, hctx->cpumask, cpu_online_mask) != cpu) + return false; + if (cpumask_next_and(cpu, hctx->cpumask, cpu_online_mask) < nr_cpu_ids) + return false; + return true; } static int blk_mq_hctx_notify_offline(unsigned int cpu, struct hlist_node *node) { + struct blk_mq_hw_ctx *hctx = hlist_entry_safe(node, + struct blk_mq_hw_ctx, cpuhp_online); + + if (!cpumask_test_cpu(cpu, hctx->cpumask)) + return 0; + + if (!blk_mq_last_cpu_in_hctx(cpu, hctx)) + return 0; + + /* + * Order setting BLK_MQ_S_INACTIVE versus checking rq->tag and rqs[tag], + * in blk_mq_tags_inflight_rqs. It pairs with the smp_mb() in + * blk_mq_get_driver_tag. + */ + set_bit(BLK_MQ_S_INACTIVE, &hctx->state); + smp_mb__after_atomic(); + while (blk_mq_tags_inflight_rqs(hctx)) + msleep(5); + return 0; +} + +static int blk_mq_hctx_notify_online(unsigned int cpu, struct hlist_node *node) +{ + struct blk_mq_hw_ctx *hctx = hlist_entry_safe(node, + struct blk_mq_hw_ctx, cpuhp_online); + + if (cpumask_test_cpu(cpu, hctx->cpumask)) + clear_bit(BLK_MQ_S_INACTIVE, &hctx->state); return 0; } @@ -2290,12 +2389,15 @@ static int blk_mq_hctx_notify_offline(unsigned int cpu, struct hlist_node *node) */ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node) { - struct blk_mq_hw_ctx *hctx; + struct blk_mq_hw_ctx *hctx = hlist_entry_safe(node, + struct blk_mq_hw_ctx, cpuhp_dead); struct blk_mq_ctx *ctx; LIST_HEAD(tmp); enum hctx_type type; - hctx = hlist_entry_safe(node, struct blk_mq_hw_ctx, cpuhp_dead); + if (!cpumask_test_cpu(cpu, hctx->cpumask)) + return 0; + ctx = __blk_mq_get_ctx(hctx->queue, cpu); type = hctx->type; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 3763207d88eb..77bf861d72ec 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -403,6 +403,9 @@ enum { BLK_MQ_S_TAG_ACTIVE = 1, BLK_MQ_S_SCHED_RESTART = 2, + /* hw queue is inactive after all its CPUs become offline */ + BLK_MQ_S_INACTIVE = 3, + BLK_MQ_MAX_DEPTH = 10240, BLK_MQ_CPU_WORK_BATCH = 8, From patchwork Sat May 2 23:54:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 11524189 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5F8A381 for ; Sat, 2 May 2020 23:56:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 46A7C20721 for ; Sat, 2 May 2020 23:56:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="aAf3WWNd" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726587AbgEBX4S (ORCPT ); Sat, 2 May 2020 19:56:18 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:35819 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725798AbgEBX4S (ORCPT ); Sat, 2 May 2020 19:56:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588463776; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fPQPfCG9JPbAFPbDPYgzhBfBf5xwjmeYNQuoIFyRJNk=; b=aAf3WWNdJhgbFLTEw7XU1UV4KtfIFOw6a7FlLEifTmmGJuKjQlq1PZdqhjvkgqa2sD4T6j oaLpSa8gNGfuzYkUebp5mV1l5kA2M3MmYXriWFRQMFR+NokfdCWqY6ENUsyVaz3WkN/LEr /Hen1tTizNbQ5HSBPq9JYet7JE2G8nA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-190-2loDGNlqMo--yeMp3VqWrw-1; Sat, 02 May 2020 19:56:01 -0400 X-MC-Unique: 2loDGNlqMo--yeMp3VqWrw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 07B3F80058A; Sat, 2 May 2020 23:56:00 +0000 (UTC) Received: from localhost (ovpn-8-17.pek2.redhat.com [10.72.8.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2408A61526; Sat, 2 May 2020 23:55:54 +0000 (UTC) From: Ming Lei To: Jens Axboe Cc: linux-block@vger.kernel.org, Ming Lei , John Garry , Bart Van Assche , Hannes Reinecke , Christoph Hellwig , Thomas Gleixner Subject: [PATCH V9 08/11] block: add blk_end_flush_machinery Date: Sun, 3 May 2020 07:54:51 +0800 Message-Id: <20200502235454.1118520-9-ming.lei@redhat.com> In-Reply-To: <20200502235454.1118520-1-ming.lei@redhat.com> References: <20200502235454.1118520-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Flush requests aren't same with normal FS request: 1) one dedicated per-hctx flush rq is pre-allocated for sending flush request 2) flush request si issued to hardware via one machinary so that flush merge can be applied We can't simply re-submit flush rqs via blk_steal_bios(), so add blk_end_flush_machinery to collect flush requests which needs to be resubmitted: - if one flush command without DATA is enough, send one flush, complete this kind of requests - otherwise, add the request into a list and let caller re-submit it. Cc: John Garry Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Thomas Gleixner Reviewed-by: Christoph Hellwig Signed-off-by: Ming Lei --- block/blk-flush.c | 123 +++++++++++++++++++++++++++++++++++++++++++--- block/blk.h | 4 ++ 2 files changed, 120 insertions(+), 7 deletions(-) diff --git a/block/blk-flush.c b/block/blk-flush.c index 977edf95d711..745d878697ed 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -170,10 +170,11 @@ static void blk_flush_complete_seq(struct request *rq, unsigned int cmd_flags; BUG_ON(rq->flush.seq & seq); - rq->flush.seq |= seq; + if (!error) + rq->flush.seq |= seq; cmd_flags = rq->cmd_flags; - if (likely(!error)) + if (likely(!error && !fq->flush_queue_terminating)) seq = blk_flush_cur_seq(rq); else seq = REQ_FSEQ_DONE; @@ -200,9 +201,15 @@ static void blk_flush_complete_seq(struct request *rq, * normal completion and end it. */ BUG_ON(!list_empty(&rq->queuelist)); - list_del_init(&rq->flush.list); - blk_flush_restore_request(rq); - blk_mq_end_request(rq, error); + + /* Terminating code will end the request from flush queue */ + if (likely(!fq->flush_queue_terminating)) { + list_del_init(&rq->flush.list); + blk_flush_restore_request(rq); + blk_mq_end_request(rq, error); + } else { + list_move_tail(&rq->flush.list, pending); + } break; default: @@ -279,7 +286,8 @@ static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq, struct request *flush_rq = fq->flush_rq; /* C1 described at the top of this file */ - if (fq->flush_pending_idx != fq->flush_running_idx || list_empty(pending)) + if (fq->flush_pending_idx != fq->flush_running_idx || + list_empty(pending) || fq->flush_queue_terminating) return; /* C2 and C3 @@ -331,7 +339,7 @@ static void mq_flush_data_end_io(struct request *rq, blk_status_t error) struct blk_flush_queue *fq = blk_get_flush_queue(q, ctx); if (q->elevator) { - WARN_ON(rq->tag < 0); + WARN_ON(rq->tag < 0 && !fq->flush_queue_terminating); blk_mq_put_driver_tag(rq); } @@ -503,3 +511,104 @@ void blk_free_flush_queue(struct blk_flush_queue *fq) kfree(fq->flush_rq); kfree(fq); } + +static void __blk_end_queued_flush(struct blk_flush_queue *fq, + unsigned int queue_idx, struct list_head *resubmit_list, + struct list_head *flush_list) +{ + struct list_head *queue = &fq->flush_queue[queue_idx]; + struct request *rq, *nxt; + + list_for_each_entry_safe(rq, nxt, queue, flush.list) { + unsigned int seq = blk_flush_cur_seq(rq); + + list_del_init(&rq->flush.list); + blk_flush_restore_request(rq); + if (!blk_rq_sectors(rq) || seq == REQ_FSEQ_POSTFLUSH ) + list_add_tail(&rq->queuelist, flush_list); + else + list_add_tail(&rq->queuelist, resubmit_list); + } +} + +static void blk_end_queued_flush(struct blk_flush_queue *fq, + struct list_head *resubmit_list, struct list_head *flush_list) +{ + unsigned long flags; + + spin_lock_irqsave(&fq->mq_flush_lock, flags); + __blk_end_queued_flush(fq, 0, resubmit_list, flush_list); + __blk_end_queued_flush(fq, 1, resubmit_list, flush_list); + spin_unlock_irqrestore(&fq->mq_flush_lock, flags); +} + +/* complete requests which just requires one flush command */ +static void blk_complete_flush_requests(struct blk_flush_queue *fq, + struct list_head *flush_list) +{ + struct block_device *bdev; + struct request *rq; + int error = -ENXIO; + + if (list_empty(flush_list)) + return; + + rq = list_first_entry(flush_list, struct request, queuelist); + + /* Send flush via one active hctx so we can move on */ + bdev = bdget_disk(rq->rq_disk, 0); + if (bdev) { + error = blkdev_issue_flush(bdev, GFP_KERNEL, NULL); + bdput(bdev); + } + + while (!list_empty(flush_list)) { + rq = list_first_entry(flush_list, struct request, queuelist); + list_del_init(&rq->queuelist); + blk_mq_end_request(rq, error); + } +} + +/* + * Called when this hctx is inactive and all CPUs of this hctx is dead, + * otherwise don't reuse this function. + * + * Terminate this hw queue's flush machinery, and try to complete flush + * IO requests if possible, such as any flush IO without data, or flush + * data IO in POSTFLUSH stage. Otherwise, add the flush IOs into @list + * and let caller to re-submit them. + */ +void blk_end_flush_machinery(struct blk_mq_hw_ctx *hctx, + struct list_head *in, struct list_head *out) +{ + LIST_HEAD(resubmit_list); + LIST_HEAD(flush_list); + struct blk_flush_queue *fq = hctx->fq; + struct request *rq, *nxt; + unsigned long flags; + + spin_lock_irqsave(&fq->mq_flush_lock, flags); + fq->flush_queue_terminating = 1; + spin_unlock_irqrestore(&fq->mq_flush_lock, flags); + + /* End inflight flush requests */ + list_for_each_entry_safe(rq, nxt, in, queuelist) { + WARN_ON(!(rq->rq_flags & RQF_FLUSH_SEQ)); + list_del_init(&rq->queuelist); + rq->end_io(rq, BLK_STS_AGAIN); + } + + /* End queued requests */ + blk_end_queued_flush(fq, &resubmit_list, &flush_list); + + /* Send flush and complete requests which just need one flush req */ + blk_complete_flush_requests(fq, &flush_list); + + spin_lock_irqsave(&fq->mq_flush_lock, flags); + /* reset flush queue so that it is ready to work next time */ + fq->flush_pending_idx = fq->flush_running_idx = 0; + fq->flush_queue_terminating = 0; + spin_unlock_irqrestore(&fq->mq_flush_lock, flags); + + list_splice_init(&resubmit_list, out); +} diff --git a/block/blk.h b/block/blk.h index 591cc07e40f9..133fb0b99759 100644 --- a/block/blk.h +++ b/block/blk.h @@ -20,6 +20,7 @@ struct blk_flush_queue { unsigned int flush_queue_delayed:1; unsigned int flush_pending_idx:1; unsigned int flush_running_idx:1; + unsigned int flush_queue_terminating:1; blk_status_t rq_status; unsigned long flush_pending_since; struct list_head flush_queue[2]; @@ -454,4 +455,7 @@ int __bio_add_pc_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, bool *same_page); +void blk_end_flush_machinery(struct blk_mq_hw_ctx *hctx, + struct list_head *in, struct list_head *out); + #endif /* BLK_INTERNAL_H */ From patchwork Sat May 2 23:54:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 11524187 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 367E314B4 for ; Sat, 2 May 2020 23:56:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1BE9E20721 for ; Sat, 2 May 2020 23:56:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="AJKkMKv+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726580AbgEBX4L (ORCPT ); Sat, 2 May 2020 19:56:11 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:29386 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725798AbgEBX4L (ORCPT ); Sat, 2 May 2020 19:56:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588463770; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ALPAdvtUh5zZQ0CekDAR2jJsyxPgrFmti13KYPrz2fU=; b=AJKkMKv+e1mySYT0T9bc16TMN3Jo6z8A/hEUW+yZfnkeugL6Ty+/s46Oz6E+wqP9FXK4HU 8hcLpYftU8kmLxoSlxAv7+hcB05FbquYK1yaFFrNGu7+qeOHics775DUre0qjErR7mlCFJ 6RBZWTQFSzX48jN+UdMkcvPzfZuOZso= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-4-OP0JVzyqMZC5S6kqtPqvXg-1; Sat, 02 May 2020 19:56:08 -0400 X-MC-Unique: OP0JVzyqMZC5S6kqtPqvXg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E251345F; Sat, 2 May 2020 23:56:06 +0000 (UTC) Received: from localhost (ovpn-8-17.pek2.redhat.com [10.72.8.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id DC4E26B8D7; Sat, 2 May 2020 23:56:02 +0000 (UTC) From: Ming Lei To: Jens Axboe Cc: linux-block@vger.kernel.org, Ming Lei , John Garry , Bart Van Assche , Hannes Reinecke , Christoph Hellwig , Thomas Gleixner Subject: [PATCH V9 09/11] blk-mq: add blk_mq_hctx_handle_dead_cpu for handling cpu dead Date: Sun, 3 May 2020 07:54:52 +0800 Message-Id: <20200502235454.1118520-10-ming.lei@redhat.com> In-Reply-To: <20200502235454.1118520-1-ming.lei@redhat.com> References: <20200502235454.1118520-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add helper of blk_mq_hctx_handle_dead_cpu for handling cpu dead, and prepare for handling inactive hctx. No functional change. Cc: John Garry Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Thomas Gleixner Reviewed-by: Hannes Reinecke Signed-off-by: Ming Lei --- block/blk-mq.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 4a2250ac4fbb..73e1a1d4c1c5 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2382,22 +2382,13 @@ static int blk_mq_hctx_notify_online(unsigned int cpu, struct hlist_node *node) return 0; } -/* - * 'cpu' is going away. splice any existing rq_list entries from this - * software queue to the hw queue dispatch list, and ensure that it - * gets run. - */ -static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node) +static void blk_mq_hctx_handle_dead_cpu(struct blk_mq_hw_ctx *hctx, + unsigned int cpu) { - struct blk_mq_hw_ctx *hctx = hlist_entry_safe(node, - struct blk_mq_hw_ctx, cpuhp_dead); struct blk_mq_ctx *ctx; LIST_HEAD(tmp); enum hctx_type type; - if (!cpumask_test_cpu(cpu, hctx->cpumask)) - return 0; - ctx = __blk_mq_get_ctx(hctx->queue, cpu); type = hctx->type; @@ -2409,13 +2400,27 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node) spin_unlock(&ctx->lock); if (list_empty(&tmp)) - return 0; + return; spin_lock(&hctx->lock); list_splice_tail_init(&tmp, &hctx->dispatch); spin_unlock(&hctx->lock); blk_mq_run_hw_queue(hctx, true); +} + +/* + * 'cpu' is going away. splice any existing rq_list entries from this + * software queue to the hw queue dispatch list, and ensure that it + * gets run. + */ +static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node) +{ + struct blk_mq_hw_ctx *hctx = hlist_entry_safe(node, + struct blk_mq_hw_ctx, cpuhp_dead); + + if (cpumask_test_cpu(cpu, hctx->cpumask)) + blk_mq_hctx_handle_dead_cpu(hctx, cpu); return 0; } From patchwork Sat May 2 23:54:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 11524193 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7704814B4 for ; Sat, 2 May 2020 23:56:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5EEA220721 for ; Sat, 2 May 2020 23:56:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="U7uUf1Xy" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725798AbgEBX4W (ORCPT ); Sat, 2 May 2020 19:56:22 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:23427 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726599AbgEBX4W (ORCPT ); Sat, 2 May 2020 19:56:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588463780; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7yiS/4+UH0HFtyS/j7BW3iy+VtDUz4EgaInqbF4vfD4=; b=U7uUf1XyIbc51z+KlRrikClIIUDHfE6MDZyZuWneQonfv9U5tLu3Tlvh/mJ8p/FqHWZBKI HAOYeCSy6OSYRB79DPYkx9Fo6JIj5sYFYyucmtOyKogFvzLjScKeuHXE7aTt6ydAj6EGpE UabqMKbs4xyDRZGe0gU1eifhavw/8zw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-500-NbscUYTcNOeX_GESwzlycQ-1; Sat, 02 May 2020 19:56:14 -0400 X-MC-Unique: NbscUYTcNOeX_GESwzlycQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1B42C1005510; Sat, 2 May 2020 23:56:13 +0000 (UTC) Received: from localhost (ovpn-8-17.pek2.redhat.com [10.72.8.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id BA32D5D9CD; Sat, 2 May 2020 23:56:09 +0000 (UTC) From: Ming Lei To: Jens Axboe Cc: linux-block@vger.kernel.org, Ming Lei , John Garry , Bart Van Assche , Hannes Reinecke , Christoph Hellwig , Thomas Gleixner Subject: [PATCH V9 10/11] blk-mq: re-submit IO in case that hctx is inactive Date: Sun, 3 May 2020 07:54:53 +0800 Message-Id: <20200502235454.1118520-11-ming.lei@redhat.com> In-Reply-To: <20200502235454.1118520-1-ming.lei@redhat.com> References: <20200502235454.1118520-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org When all CPUs in one hctx are offline and this hctx becomes inactive, we shouldn't run this hw queue for completing request any more. So allocate request from one live hctx, and clone & resubmit the request, either it is from sw queue or scheduler queue. Cc: John Garry Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Thomas Gleixner Signed-off-by: Ming Lei --- block/blk-mq.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 100 insertions(+), 5 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 73e1a1d4c1c5..3b777d9fd4ee 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2382,6 +2382,98 @@ static int blk_mq_hctx_notify_online(unsigned int cpu, struct hlist_node *node) return 0; } +static void blk_mq_resubmit_end_rq(struct request *rq, blk_status_t error) +{ + struct request *orig_rq = rq->end_io_data; + + blk_mq_cleanup_rq(orig_rq); + blk_mq_end_request(orig_rq, error); + + blk_put_request(rq); +} + +static void blk_mq_resubmit_rq(struct request *rq) +{ + struct request *nrq; + unsigned int flags = 0; + struct blk_mq_hw_ctx *hctx = rq->mq_hctx; + struct blk_mq_tags *tags = rq->q->elevator ? hctx->sched_tags : + hctx->tags; + bool reserved = blk_mq_tag_is_reserved(tags, rq->internal_tag); + + if (rq->rq_flags & RQF_PREEMPT) + flags |= BLK_MQ_REQ_PREEMPT; + if (reserved) + flags |= BLK_MQ_REQ_RESERVED; + + /* avoid allocation failure by clearing NOWAIT */ + nrq = blk_get_request(rq->q, rq->cmd_flags & ~REQ_NOWAIT, flags); + if (!nrq) + return; + + blk_rq_copy_request(nrq, rq); + + nrq->timeout = rq->timeout; + nrq->rq_disk = rq->rq_disk; + nrq->part = rq->part; + + memcpy(blk_mq_rq_to_pdu(nrq), blk_mq_rq_to_pdu(rq), + rq->q->tag_set->cmd_size); + + nrq->end_io = blk_mq_resubmit_end_rq; + nrq->end_io_data = rq; + nrq->bio = rq->bio; + nrq->biotail = rq->biotail; + + if (blk_insert_cloned_request(nrq->q, nrq) != BLK_STS_OK) + blk_mq_request_bypass_insert(nrq, false, true); +} + +static void blk_mq_hctx_deactivate(struct blk_mq_hw_ctx *hctx) +{ + LIST_HEAD(sched); + LIST_HEAD(re_submit); + LIST_HEAD(flush_in); + LIST_HEAD(flush_out); + struct request *rq, *nxt; + struct elevator_queue *e = hctx->queue->elevator; + + if (!e) { + blk_mq_flush_busy_ctxs(hctx, &re_submit); + } else { + while ((rq = e->type->ops.dispatch_request(hctx))) { + if (rq->mq_hctx != hctx) + list_add(&rq->queuelist, &sched); + else + list_add(&rq->queuelist, &re_submit); + } + } + while (!list_empty(&sched)) { + rq = list_first_entry(&sched, struct request, queuelist); + list_del_init(&rq->queuelist); + blk_mq_sched_insert_request(rq, true, true, true); + } + + /* requests in dispatch list have to be re-submitted too */ + spin_lock(&hctx->lock); + list_splice_tail_init(&hctx->dispatch, &re_submit); + spin_unlock(&hctx->lock); + + /* blk_end_flush_machinery will cover flush request */ + list_for_each_entry_safe(rq, nxt, &re_submit, queuelist) { + if (rq->rq_flags & RQF_FLUSH_SEQ) + list_move(&rq->queuelist, &flush_in); + } + blk_end_flush_machinery(hctx, &flush_in, &flush_out); + list_splice_tail(&flush_out, &re_submit); + + while (!list_empty(&re_submit)) { + rq = list_first_entry(&re_submit, struct request, queuelist); + list_del_init(&rq->queuelist); + blk_mq_resubmit_rq(rq); + } +} + static void blk_mq_hctx_handle_dead_cpu(struct blk_mq_hw_ctx *hctx, unsigned int cpu) { @@ -2410,17 +2502,20 @@ static void blk_mq_hctx_handle_dead_cpu(struct blk_mq_hw_ctx *hctx, } /* - * 'cpu' is going away. splice any existing rq_list entries from this - * software queue to the hw queue dispatch list, and ensure that it - * gets run. + * @cpu has gone away. If this hctx is inactive, we can't dispatch request + * to the hctx any more, so clone and re-submit requests from this hctx */ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node) { struct blk_mq_hw_ctx *hctx = hlist_entry_safe(node, struct blk_mq_hw_ctx, cpuhp_dead); - if (cpumask_test_cpu(cpu, hctx->cpumask)) - blk_mq_hctx_handle_dead_cpu(hctx, cpu); + if (cpumask_test_cpu(cpu, hctx->cpumask)) { + if (test_bit(BLK_MQ_S_INACTIVE, &hctx->state)) + blk_mq_hctx_deactivate(hctx); + else + blk_mq_hctx_handle_dead_cpu(hctx, cpu); + } return 0; } From patchwork Sat May 2 23:54:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 11524191 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5FD4014B4 for ; Sat, 2 May 2020 23:56:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 48E4D2137B for ; Sat, 2 May 2020 23:56:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Uu/7IO1w" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726608AbgEBX4V (ORCPT ); Sat, 2 May 2020 19:56:21 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:21723 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725798AbgEBX4V (ORCPT ); Sat, 2 May 2020 19:56:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588463780; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z0E6FjJfym2wxaJpRjfmaRhSzHJS6rkCL0z9quTLXr8=; b=Uu/7IO1w1/iREB3kH7EOb9g4ifwdQ/d857XFqBxGnE7qUf3gMazpGctYIklODsubazfbga 727WYYCn9dzcAp2b0oXd4oqQIQR1wNwD7i0nOVYTPCXs/eyy7Lhu8xLvXzv+YhL5KKJUQC GTnU0xfEGq9ZsjCchJKu84tah+5TgHA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-103-PtN0QPOxPUaHQGX8xSkxSw-1; Sat, 02 May 2020 19:56:18 -0400 X-MC-Unique: PtN0QPOxPUaHQGX8xSkxSw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8462B1005510; Sat, 2 May 2020 23:56:16 +0000 (UTC) Received: from localhost (ovpn-8-17.pek2.redhat.com [10.72.8.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id AB6A260612; Sat, 2 May 2020 23:56:15 +0000 (UTC) From: Ming Lei To: Jens Axboe Cc: linux-block@vger.kernel.org, Ming Lei , John Garry , Bart Van Assche , Hannes Reinecke , Christoph Hellwig , Thomas Gleixner , Hannes Reinecke Subject: [PATCH V9 11/11] block: deactivate hctx when the hctx is actually inactive Date: Sun, 3 May 2020 07:54:54 +0800 Message-Id: <20200502235454.1118520-12-ming.lei@redhat.com> In-Reply-To: <20200502235454.1118520-1-ming.lei@redhat.com> References: <20200502235454.1118520-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Run queue on dead CPU still may be triggered in some corner case, such as one request is requeued after CPU hotplug is handled. So handle this corner case during run queue. Cc: John Garry Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Thomas Gleixner Reviewed-by: Hannes Reinecke Reviewed-by: Christoph Hellwig Signed-off-by: Ming Lei --- block/blk-mq.c | 30 ++++++++++-------------------- 1 file changed, 10 insertions(+), 20 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 3b777d9fd4ee..12dee4ecd5cc 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -43,6 +43,8 @@ static void blk_mq_poll_stats_start(struct request_queue *q); static void blk_mq_poll_stats_fn(struct blk_stat_callback *cb); +static void blk_mq_hctx_deactivate(struct blk_mq_hw_ctx *hctx); + static int blk_mq_poll_stats_bkt(const struct request *rq) { int ddir, sectors, bucket; @@ -1373,28 +1375,16 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx) int srcu_idx; /* - * We should be running this queue from one of the CPUs that - * are mapped to it. - * - * There are at least two related races now between setting - * hctx->next_cpu from blk_mq_hctx_next_cpu() and running - * __blk_mq_run_hw_queue(): - * - * - hctx->next_cpu is found offline in blk_mq_hctx_next_cpu(), - * but later it becomes online, then this warning is harmless - * at all - * - * - hctx->next_cpu is found online in blk_mq_hctx_next_cpu(), - * but later it becomes offline, then the warning can't be - * triggered, and we depend on blk-mq timeout handler to - * handle dispatched requests to this hctx + * BLK_MQ_S_INACTIVE may not deal with some requeue corner case: + * one request is requeued after cpu unplug is handled, so check + * if the hctx is actually inactive. If yes, deactive it and + * re-submit all requests in the queue. */ if (!cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask) && - cpu_online(hctx->next_cpu)) { - printk(KERN_WARNING "run queue from wrong CPU %d, hctx %s\n", - raw_smp_processor_id(), - cpumask_empty(hctx->cpumask) ? "inactive": "active"); - dump_stack(); + cpumask_next_and(-1, hctx->cpumask, cpu_online_mask) >= + nr_cpu_ids) { + blk_mq_hctx_deactivate(hctx); + return; } /*