From patchwork Fri Dec 4 19:13:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 11952175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20D17C4361A for ; Fri, 4 Dec 2020 19:14:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B62D222C9F for ; Fri, 4 Dec 2020 19:14:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728729AbgLDTOr (ORCPT ); Fri, 4 Dec 2020 14:14:47 -0500 Received: from Galois.linutronix.de ([193.142.43.55]:49360 "EHLO galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727199AbgLDTOr (ORCPT ); Fri, 4 Dec 2020 14:14:47 -0500 From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1607109244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QcuZUB5nFMpQRoWQVOhcAERBptrSeWMm+81ZrjusF/E=; b=Ix8PWWjzu/kGiLUn7Z9GiXFsohekv5JA3QpvEW4QGaL4UjPKoroTztHHgYAfyAo6+a7lxc 5aiRhSAcvCrN1ezE/JBvXAY15rILQlQp5QDg83PeZJhSRkxrSmdA5NPsmZs9rO90qoqpGa GPOAn8rf6SJkxYMxKECUVgGvqKGdgJBmJxAQHTZMI0n3d4x51Om07KmD43MrWaQwx3HoQ7 zAH914iD/kh+yL8ptsD6qv7Ulso77bnfpB5ZCdwp+yMF+lmPrGEcOOCqqxvXM6Wqp6yQhU FvfB2Ytl2cfcQO2RmL2rHkELeVn77lMQj5xRiv7E1Kegi3/yrLvyQ9EAX/bfzA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1607109244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QcuZUB5nFMpQRoWQVOhcAERBptrSeWMm+81ZrjusF/E=; b=zQl3ZB/WCJPKhJ8qQqUf9rDGHaV5GzEHVsZ9d63umJ4hTcKzoJRieBQjo+qq/tt85V5/eX WdaSm8URbHmPvcCg== To: linux-block@vger.kernel.org Cc: Jens Axboe , Thomas Gleixner , Peter Zijlstra , Daniel Wagner , Mike Galbraith , Christoph Hellwig , Sagi Grimberg , Sebastian Andrzej Siewior Subject: [PATCH 1/3] blk-mq: Don't complete on a remote CPU in force threaded mode Date: Fri, 4 Dec 2020 20:13:54 +0100 Message-Id: <20201204191356.2516405-2-bigeasy@linutronix.de> In-Reply-To: <20201204191356.2516405-1-bigeasy@linutronix.de> References: <20201204191356.2516405-1-bigeasy@linutronix.de> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org With force threaded interrupts enabled, raising softirq from an SMP function call will always result in waking the ksoftirqd thread. This is not optimal given that the thread runs at SCHED_OTHER priority. Completing the request in hard IRQ-context on PREEMPT_RT (which enforces the force threaded mode) is bad because the completion handler may acquire sleeping locks which violate the locking context. Disable request completing on a remote CPU in force threaded mode. Signed-off-by: Sebastian Andrzej Siewior Reviewed-by: Christoph Hellwig --- block/blk-mq.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/block/blk-mq.c b/block/blk-mq.c index b9c2efab5db78..7091bb097c63f 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -650,6 +650,14 @@ static inline bool blk_mq_complete_need_ipi(struct request *rq) if (!IS_ENABLED(CONFIG_SMP) || !test_bit(QUEUE_FLAG_SAME_COMP, &rq->q->queue_flags)) return false; + /* + * With force threaded interrupts enabled, raising softirq from an SMP + * function call will always result in waking the ksoftirqd thread. + * This is probably worse than completing the request on a different + * cache domain. + */ + if (force_irqthreads) + return false; /* same CPU or cache domain? Complete locally */ if (cpu == rq->mq_ctx->cpu || From patchwork Fri Dec 4 19:13:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 11952177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E501C19425 for ; Fri, 4 Dec 2020 19:14:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2FA8922C9F for ; Fri, 4 Dec 2020 19:14:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727866AbgLDTOw (ORCPT ); Fri, 4 Dec 2020 14:14:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387493AbgLDTOr (ORCPT ); Fri, 4 Dec 2020 14:14:47 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02657C061A4F for ; Fri, 4 Dec 2020 11:14:07 -0800 (PST) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1607109245; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NAlQ2JBtbq6g5nCibdn4i+qWOJybyEQ2RwkSZYQMVn8=; b=MfJlRSf7K6D1LAI0UrTRHI88q5wWkChUSCvijMdhyP2wUnGRWu5LDwKEj2pcg7DcwPniYX gh7bcMbHXKYKDXYE5CeqROmzICiMh5MtOGcOKeKY/CZAO7PiYIsiPKL514+3RKt2YA37Pt CqLnQ2d1h0/IzmlF68TjWZDrDmo8isik+swbSMgOEzEoKEVOc98cW0erNikTzSXBw54pX8 mFDjwbeabx/RT3NKbEGWrh4IHub2e879xoVmpqZ9ORA7TGQO5LAOZrSvciVYZr8//yq8f+ U8Sa3dQjfkqagPdfgZzTekdlDRWUbGLS3B6KFsitiH8+sRhykYQKPYsbXQU4dA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1607109245; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NAlQ2JBtbq6g5nCibdn4i+qWOJybyEQ2RwkSZYQMVn8=; b=d4fWv6+PYGuXMvLlhtRCKvjNKxwnCe4AZgTZZVSwECTm7dY6VbUKQB0i+YvbK2Ap0d4dOf w2JviSYhwSL5fqCw== To: linux-block@vger.kernel.org Cc: Jens Axboe , Thomas Gleixner , Peter Zijlstra , Daniel Wagner , Mike Galbraith , Christoph Hellwig , Sagi Grimberg , Sebastian Andrzej Siewior Subject: [PATCH 2/3] blk-mq: Always complete remote completions requests in softirq Date: Fri, 4 Dec 2020 20:13:55 +0100 Message-Id: <20201204191356.2516405-3-bigeasy@linutronix.de> In-Reply-To: <20201204191356.2516405-1-bigeasy@linutronix.de> References: <20201204191356.2516405-1-bigeasy@linutronix.de> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Controllers with multiple queues have their IRQ-handelers pinned to a CPU. The core shouldn't need to complete the request on a remote CPU. Remove this case and always raise the softirq to complete the request. Signed-off-by: Sebastian Andrzej Siewior Reviewed-by: Daniel Wagner --- block/blk-mq.c | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 7091bb097c63f..3c0e94913d874 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -628,19 +628,7 @@ static void __blk_mq_complete_request_remote(void *data) { struct request *rq = data; - /* - * For most of single queue controllers, there is only one irq vector - * for handling I/O completion, and the only irq's affinity is set - * to all possible CPUs. On most of ARCHs, this affinity means the irq - * is handled on one specific CPU. - * - * So complete I/O requests in softirq context in case of single queue - * devices to avoid degrading I/O performance due to irqsoff latency. - */ - if (rq->q->nr_hw_queues == 1) - blk_mq_trigger_softirq(rq); - else - rq->q->mq_ops->complete(rq); + blk_mq_trigger_softirq(rq); } static inline bool blk_mq_complete_need_ipi(struct request *rq) From patchwork Fri Dec 4 19:13:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 11952179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54F3EC4167B for ; Fri, 4 Dec 2020 19:14:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 019E822CAE for ; Fri, 4 Dec 2020 19:14:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727199AbgLDTOw (ORCPT ); Fri, 4 Dec 2020 14:14:52 -0500 Received: from Galois.linutronix.de ([193.142.43.55]:49392 "EHLO galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728079AbgLDTOs (ORCPT ); Fri, 4 Dec 2020 14:14:48 -0500 From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1607109245; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sBZVA+aUErTEHExNncbFL8kYuUQMpfZGtQo4vtsCy2c=; b=HBCiux5IzSNRxBSR/n3Sinu760jNxDTg9mB1HEgzPCIjKRvrmGEDX0jL7cYQCdprY2/Tr1 QvWh8CBYIc+l3p3Dp4Q5uqUcLZNf2DJQAbvOl7vU9zYOHg9XaUYiwV3Ki7B2eNB2ebYtM0 sn3SJdU2cbGgvkVCUWA2M89qb6ClD/Iw5GS15E83oqkkHg3VsFQYWt6tN0mM5UjlpwGgoW Aaw9+wpDWryHfPnn0zfVlPWpjXAF+qE1uwyF53t0gbymcHxpkiDzJXwgvgQ5T1LXnrr6gb JgE/m/iRUAHbHi/RGqLqyqjEvrmsTAhN9o0H0hpYqxfGqLtRY8+F5i/M7X1t7A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1607109245; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sBZVA+aUErTEHExNncbFL8kYuUQMpfZGtQo4vtsCy2c=; b=GOwMbCjVDKMonuC3piKobx2KCG9xwLAf9Q78PU/WxZvz5+x0fdtW70gzNBvgm3FJANwqW5 tZdgo64j0qUIvMDQ== To: linux-block@vger.kernel.org Cc: Jens Axboe , Thomas Gleixner , Peter Zijlstra , Daniel Wagner , Mike Galbraith , Christoph Hellwig , Sagi Grimberg , Sebastian Andrzej Siewior Subject: [PATCH 3/3] blk-mq: Use llist_head for blk_cpu_done Date: Fri, 4 Dec 2020 20:13:56 +0100 Message-Id: <20201204191356.2516405-4-bigeasy@linutronix.de> In-Reply-To: <20201204191356.2516405-1-bigeasy@linutronix.de> References: <20201204191356.2516405-1-bigeasy@linutronix.de> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org With llist_head it is possible to avoid the locking (the irq-off region) when items are added. This makes it possible to add items on a remote CPU. llist_add() returns true if the list was previously empty. This can be used to invoke the SMP function call / raise sofirq only if the first item was added (otherwise it is already pending). This simplifies the code a little and reduces the IRQ-off regions. With this change it possible to reduce the SMP-function call a simple __raise_softirq_irqoff(). blk_mq_complete_request_remote() needs a preempt-disable section if the request needs to complete on the local CPU. Some callers (USB-storage) invoke this preemptible context and the request needs to be enqueued on the same CPU as the softirq is raised. Signed-off-by: Sebastian Andrzej Siewior --- block/blk-mq.c | 77 ++++++++++++++---------------------------- include/linux/blkdev.h | 2 +- 2 files changed, 27 insertions(+), 52 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 3c0e94913d874..b5138327952a4 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -41,7 +41,7 @@ #include "blk-mq-sched.h" #include "blk-rq-qos.h" -static DEFINE_PER_CPU(struct list_head, blk_cpu_done); +static DEFINE_PER_CPU(struct llist_head, blk_cpu_done); static void blk_mq_poll_stats_start(struct request_queue *q); static void blk_mq_poll_stats_fn(struct blk_stat_callback *cb); @@ -567,68 +567,32 @@ void blk_mq_end_request(struct request *rq, blk_status_t error) } EXPORT_SYMBOL(blk_mq_end_request); -/* - * Softirq action handler - move entries to local list and loop over them - * while passing them to the queue registered handler. - */ -static __latent_entropy void blk_done_softirq(struct softirq_action *h) +static void blk_complete_reqs(struct llist_head *cpu_list) { - struct list_head *cpu_list, local_list; + struct llist_node *entry; + struct request *rq, *rq_next; - local_irq_disable(); - cpu_list = this_cpu_ptr(&blk_cpu_done); - list_replace_init(cpu_list, &local_list); - local_irq_enable(); + entry = llist_del_all(cpu_list); + entry = llist_reverse_order(entry); - while (!list_empty(&local_list)) { - struct request *rq; - - rq = list_entry(local_list.next, struct request, ipi_list); - list_del_init(&rq->ipi_list); + llist_for_each_entry_safe(rq, rq_next, entry, ipi_list) rq->q->mq_ops->complete(rq); - } } -static void blk_mq_trigger_softirq(struct request *rq) +static __latent_entropy void blk_done_softirq(struct softirq_action *h) { - struct list_head *list; - unsigned long flags; - - local_irq_save(flags); - list = this_cpu_ptr(&blk_cpu_done); - list_add_tail(&rq->ipi_list, list); - - /* - * If the list only contains our just added request, signal a raise of - * the softirq. If there are already entries there, someone already - * raised the irq but it hasn't run yet. - */ - if (list->next == &rq->ipi_list) - raise_softirq_irqoff(BLOCK_SOFTIRQ); - local_irq_restore(flags); + blk_complete_reqs(this_cpu_ptr(&blk_cpu_done)); } static int blk_softirq_cpu_dead(unsigned int cpu) { - /* - * If a CPU goes away, splice its entries to the current CPU - * and trigger a run of the softirq - */ - local_irq_disable(); - list_splice_init(&per_cpu(blk_cpu_done, cpu), - this_cpu_ptr(&blk_cpu_done)); - raise_softirq_irqoff(BLOCK_SOFTIRQ); - local_irq_enable(); - + blk_complete_reqs(&per_cpu(blk_cpu_done, cpu)); return 0; } - static void __blk_mq_complete_request_remote(void *data) { - struct request *rq = data; - - blk_mq_trigger_softirq(rq); + __raise_softirq_irqoff(BLOCK_SOFTIRQ); } static inline bool blk_mq_complete_need_ipi(struct request *rq) @@ -659,6 +623,7 @@ static inline bool blk_mq_complete_need_ipi(struct request *rq) bool blk_mq_complete_request_remote(struct request *rq) { + struct llist_head *cpu_list; WRITE_ONCE(rq->state, MQ_RQ_COMPLETE); /* @@ -669,12 +634,22 @@ bool blk_mq_complete_request_remote(struct request *rq) return false; if (blk_mq_complete_need_ipi(rq)) { - INIT_CSD(&rq->csd, __blk_mq_complete_request_remote, rq); - smp_call_function_single_async(rq->mq_ctx->cpu, &rq->csd); + unsigned int cpu; + + cpu = rq->mq_ctx->cpu; + cpu_list = &per_cpu(blk_cpu_done, cpu); + if (llist_add(&rq->ipi_list, cpu_list)) { + INIT_CSD(&rq->csd, __blk_mq_complete_request_remote, rq); + smp_call_function_single_async(cpu, &rq->csd); + } } else { if (rq->q->nr_hw_queues > 1) return false; - blk_mq_trigger_softirq(rq); + preempt_disable(); + cpu_list = this_cpu_ptr(&blk_cpu_done); + if (llist_add(&rq->ipi_list, cpu_list)) + raise_softirq(BLOCK_SOFTIRQ); + preempt_enable(); } return true; @@ -3905,7 +3880,7 @@ static int __init blk_mq_init(void) int i; for_each_possible_cpu(i) - INIT_LIST_HEAD(&per_cpu(blk_cpu_done, i)); + init_llist_head(&per_cpu(blk_cpu_done, i)); open_softirq(BLOCK_SOFTIRQ, blk_done_softirq); cpuhp_setup_state_nocalls(CPUHP_BLOCK_SOFTIRQ_DEAD, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 17cedf0dc83db..7b05390766eec 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -156,7 +156,7 @@ struct request { */ union { struct hlist_node hash; /* merge hash */ - struct list_head ipi_list; + struct llist_node ipi_list; }; /*