From patchwork Sun May 15 13:12:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12850030 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F11EC433EF for ; Sun, 15 May 2022 13:12:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236835AbiEONMp (ORCPT ); Sun, 15 May 2022 09:12:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236816AbiEONMp (ORCPT ); Sun, 15 May 2022 09:12:45 -0400 Received: from pv50p00im-ztdg10022001.me.com (pv50p00im-ztdg10022001.me.com [17.58.6.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C09D2DF86 for ; Sun, 15 May 2022 06:12:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1652620364; bh=kxtyT4v3y7eBOPoFyfuhFFHTGOnQN8ILhgOcq6JGNUQ=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=eE24tBBDWSlJ5e7i/FU/ox4di9IXhw2YSH0yFXbqiRmxGnJQl96wfNWN4/ntNAJ3N iRCK0vMI8ODUAlqRFixahuU3AiuTzgBQFj+55+nJvkAfMOEm+mtjoiHO15B6f23A1p 5xamA2mpIErqUe6YFSZtG6c8yUJVFxFIYMpX83WnqkNXA9eFK1xkFxTJO9i1kE6XlC xvQjvjV9/kWrS2x5Xpu/4BVCVXvcocLR7qSC7gTZIpOv55yyzVurG26XGMG+YEVwT8 +iIlX3znumBIA/m6sb8dUaUwL1SPxzGD4zfHg48Szr/u/WBuf3aGEfqJl5fXTVvDrT EVpEpgq2q9naQ== Received: from localhost.localdomain (pv50p00im-dlb-asmtp-mailmevip.me.com [17.56.9.10]) by pv50p00im-ztdg10022001.me.com (Postfix) with ESMTPSA id 382B83E1F20; Sun, 15 May 2022 13:12:40 +0000 (UTC) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 01/11] io-wq: add a worker flag for individual exit Date: Sun, 15 May 2022 21:12:20 +0800 Message-Id: <20220515131230.155267-2-haoxu.linux@icloud.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220515131230.155267-1-haoxu.linux@icloud.com> References: <20220515131230.155267-1-haoxu.linux@icloud.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-15_07:2022-05-13,2022-05-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=544 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2009150000 definitions=main-2205150069 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu From: Hao Xu Add a worker flag to control exit of an individual worker, this is needed for fixed worker in the next patches but also as a generic functionality. Signed-off-by: Hao Xu --- fs/io-wq.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/fs/io-wq.c b/fs/io-wq.c index 824623bcf1a5..0c26805ca6de 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -26,6 +26,7 @@ enum { IO_WORKER_F_RUNNING = 2, /* account as running */ IO_WORKER_F_FREE = 4, /* worker on free list */ IO_WORKER_F_BOUND = 8, /* is doing bounded work */ + IO_WORKER_F_EXIT = 32, /* worker is exiting */ }; enum { @@ -639,8 +640,12 @@ static int io_wqe_worker(void *data) while (!test_bit(IO_WQ_BIT_EXIT, &wq->state)) { long ret; + if (worker->flags & IO_WORKER_F_EXIT) + break; + set_current_state(TASK_INTERRUPTIBLE); - while (io_acct_run_queue(acct)) + while (!(worker->flags & IO_WORKER_F_EXIT) && + io_acct_run_queue(acct)) io_worker_handle_work(worker); raw_spin_lock(&wqe->lock); @@ -656,6 +661,10 @@ static int io_wqe_worker(void *data) raw_spin_unlock(&wqe->lock); if (io_flush_signals()) continue; + if (worker->flags & IO_WORKER_F_EXIT) { + __set_current_state(TASK_RUNNING); + break; + } ret = schedule_timeout(WORKER_IDLE_TIMEOUT); if (signal_pending(current)) { struct ksignal ksig; From patchwork Sun May 15 13:12:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12850031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5B61C433F5 for ; Sun, 15 May 2022 13:12:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236816AbiEONMv (ORCPT ); Sun, 15 May 2022 09:12:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230260AbiEONMu (ORCPT ); Sun, 15 May 2022 09:12:50 -0400 Received: from pv50p00im-ztdg10022001.me.com (pv50p00im-ztdg10022001.me.com [17.58.6.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 379D113F4D for ; Sun, 15 May 2022 06:12:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1652620368; bh=uJ0pqWWpnuc7GJRogOjjyoIXjcnFfu3jefX0WElOP9s=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=T78zuiDWrecsG4+3X45ts7Q7/qgdqgA5pYoLCQTiZ3Z6NspafRXfJ5VKGrlxF30GN t+F5X5MEi5/Sn1muKjNrrrqxFz3XTg6uVFYP8xvz5dVT7iRGf6XPa1CH7A59Pz+IrQ csou1pKTrTzw+0cJRVoBanT6HRFALVs9VzW+wTdZfGjQ/f7UViZt2iRDZNq4PxzJ4/ WJxiK1BEmZZJWpMiUlILNNWw3PqP1v4GonpjvZe3DA1d1R97oeus+72AD9GY/RTtrH xgyy2wqyB0a4u6eRP5NNvHWYPKc2Bk8LvqdSz/G1mVJQsddVlr0j8AWx6KG/GDlETv ryPxXX3FbCeZQ== Received: from localhost.localdomain (pv50p00im-dlb-asmtp-mailmevip.me.com [17.56.9.10]) by pv50p00im-ztdg10022001.me.com (Postfix) with ESMTPSA id 35FE03E1F13; Sun, 15 May 2022 13:12:44 +0000 (UTC) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 02/11] io-wq: change argument of create_io_worker() for convienence Date: Sun, 15 May 2022 21:12:21 +0800 Message-Id: <20220515131230.155267-3-haoxu.linux@icloud.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220515131230.155267-1-haoxu.linux@icloud.com> References: <20220515131230.155267-1-haoxu.linux@icloud.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-15_07:2022-05-13,2022-05-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=800 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2009150000 definitions=main-2205150069 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu From: Hao Xu Change index to acct itself for create_io_worker() for convienence in the next patches. Signed-off-by: Hao Xu --- fs/io-wq.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/fs/io-wq.c b/fs/io-wq.c index 0c26805ca6de..35ce622f77ba 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -139,7 +139,8 @@ struct io_cb_cancel_data { bool cancel_all; }; -static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index); +static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, + struct io_wqe_acct *acct); static void io_wqe_dec_running(struct io_worker *worker); static bool io_acct_cancel_pending_work(struct io_wqe *wqe, struct io_wqe_acct *acct, @@ -306,7 +307,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) raw_spin_unlock(&wqe->lock); atomic_inc(&acct->nr_running); atomic_inc(&wqe->wq->worker_refs); - return create_io_worker(wqe->wq, wqe, acct->index); + return create_io_worker(wqe->wq, wqe, acct); } static void io_wqe_inc_running(struct io_worker *worker) @@ -335,7 +336,7 @@ static void create_worker_cb(struct callback_head *cb) } raw_spin_unlock(&wqe->lock); if (do_create) { - create_io_worker(wq, wqe, worker->create_index); + create_io_worker(wq, wqe, acct); } else { atomic_dec(&acct->nr_running); io_worker_ref_put(wq); @@ -812,9 +813,10 @@ static void io_workqueue_create(struct work_struct *work) kfree(worker); } -static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index) +static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, + struct io_wqe_acct *acct) { - struct io_wqe_acct *acct = &wqe->acct[index]; + int index = acct->index; struct io_worker *worker; struct task_struct *tsk; From patchwork Sun May 15 13:12:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12850032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88034C433EF for ; Sun, 15 May 2022 13:12:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230260AbiEONMz (ORCPT ); Sun, 15 May 2022 09:12:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236841AbiEONMz (ORCPT ); Sun, 15 May 2022 09:12:55 -0400 Received: from pv50p00im-ztdg10022001.me.com (pv50p00im-ztdg10022001.me.com [17.58.6.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77D2CDF67 for ; Sun, 15 May 2022 06:12:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1652620372; bh=C1bI0TWHN23Y0viwQoofcVG0b8qJVUVJWBIOvXCIB4c=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=ODVxDXfaYp7clxkLSZXOjw7tVh6GOCrC303b2bPGzfFqpL5jzSY2Gs/g7ZOjdinSV Rt/OPGhbeHiLDX5WpWQ0kf3e1xTF1SGPmokpd4sX0njkA3Rh/AuhC7CLMmmitkB1q3 BsIQpx/om8dt9VaF6zjpzjkCamDmXeXvTO/BVUWj+4+BKEqgm8bHTktyxRNzp7rGfS BlsUrUoUARY7s1wiqKRKKssa45mpLcKpXvxQAiulKWmuVS1JYNwWPifuYfhbxH4XPR xvNvwtb86u+Ae804ccI8UtAYzj3ZjHhwc/ifX/PGhfjyM/71kN3JPBGQ5bLPuIWat7 d7kDo1V2cx09A== Received: from localhost.localdomain (pv50p00im-dlb-asmtp-mailmevip.me.com [17.56.9.10]) by pv50p00im-ztdg10022001.me.com (Postfix) with ESMTPSA id 1FF553E1F11; Sun, 15 May 2022 13:12:48 +0000 (UTC) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 03/11] io-wq: add infra data structure for fixed workers Date: Sun, 15 May 2022 21:12:22 +0800 Message-Id: <20220515131230.155267-4-haoxu.linux@icloud.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220515131230.155267-1-haoxu.linux@icloud.com> References: <20220515131230.155267-1-haoxu.linux@icloud.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-15_07:2022-05-13,2022-05-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=807 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2009150000 definitions=main-2205150069 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu From: Hao Xu Add data sttructure and basic initialization for fixed worker. Signed-off-by: Hao Xu --- fs/io-wq.c | 105 +++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 94 insertions(+), 11 deletions(-) diff --git a/fs/io-wq.c b/fs/io-wq.c index 35ce622f77ba..73fbe62617b7 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -26,6 +26,7 @@ enum { IO_WORKER_F_RUNNING = 2, /* account as running */ IO_WORKER_F_FREE = 4, /* worker on free list */ IO_WORKER_F_BOUND = 8, /* is doing bounded work */ + IO_WORKER_F_FIXED = 16, /* is a fixed worker */ IO_WORKER_F_EXIT = 32, /* worker is exiting */ }; @@ -37,6 +38,68 @@ enum { IO_ACCT_STALLED_BIT = 0, /* stalled on hash */ }; +struct io_wqe_acct { +/* + * union { + * // (1) for wqe->acct (normal worker) + * struct { + * unsigned nr_workers; + * unsigned max_workers; + * struct io_wq_work_list work_list; + * }; + * // (2) for wqe->fixed_acct (fixed worker) + * struct { + * unsigned nr_workers; + * unsigned max_workers; + * unsigned nr_fixed; + * unsigned max_works; + * struct io_worker **fixed_workers; + * }; + * // (3) for fixed worker's private acct + * struct { + * unsigned nr_works; + * unsigned max_works; + * struct io_wq_work_list work_list; + * }; + * }; + */ + union { + struct { + unsigned nr_workers; + unsigned max_workers; + }; + struct { + unsigned nr_works; + }; + }; + unsigned max_works; + union { + struct io_wq_work_list work_list; + struct { + unsigned nr_fixed; + struct io_worker **fixed_workers; + }; + }; + + /* + * nr_running is not meaningful for fixed worker + * but still keep the same logic for it for the + * convinence for now. So do nr_workers and + * max_workers. + */ + atomic_t nr_running; + /* + * For 1), it protects the work_list, the other two member nr_workers + * and max_workers are protected by wqe->lock. + * For 2), it protects nr_fixed, max_works, fixed_workers + * For 3), it protects nr_works, max_works and work_list. + */ + raw_spinlock_t lock; + int index; + unsigned long flags; + bool fixed_worker_registered; +}; + /* * One for each thread in a wqe pool */ @@ -62,6 +125,8 @@ struct io_worker { struct rcu_head rcu; struct work_struct work; }; + int index; + struct io_wqe_acct acct; }; #if BITS_PER_LONG == 64 @@ -72,16 +137,6 @@ struct io_worker { #define IO_WQ_NR_HASH_BUCKETS (1u << IO_WQ_HASH_ORDER) -struct io_wqe_acct { - unsigned nr_workers; - unsigned max_workers; - int index; - atomic_t nr_running; - raw_spinlock_t lock; - struct io_wq_work_list work_list; - unsigned long flags; -}; - enum { IO_WQ_ACCT_BOUND, IO_WQ_ACCT_UNBOUND, @@ -94,6 +149,7 @@ enum { struct io_wqe { raw_spinlock_t lock; struct io_wqe_acct acct[IO_WQ_ACCT_NR]; + struct io_wqe_acct fixed_acct[IO_WQ_ACCT_NR]; int node; @@ -1205,6 +1261,31 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data) atomic_set(&acct->nr_running, 0); INIT_WQ_LIST(&acct->work_list); raw_spin_lock_init(&acct->lock); + + acct = &wqe->fixed_acct[i]; + acct->index = i; + INIT_WQ_LIST(&acct->work_list); + raw_spin_lock_init(&acct->lock); + /* + * nr_running for a fixed worker is meaningless + * for now, init it to 1 to wround around the + * io_wqe_dec_running logic + */ + atomic_set(&acct->nr_running, 1); + /* + * max_workers for a fixed worker is meaningless + * for now, init it so since number of fixed workers + * should be controlled by users. + */ + acct->max_workers = task_rlimit(current, RLIMIT_NPROC); + raw_spin_lock_init(&acct->lock); + /* + * For fixed worker, not necessary + * but do it explicitly for clearity + */ + acct->nr_fixed = 0; + acct->max_works = 0; + acct->fixed_workers = NULL; } wqe->wq = wq; raw_spin_lock_init(&wqe->lock); @@ -1287,7 +1368,7 @@ static void io_wq_exit_workers(struct io_wq *wq) static void io_wq_destroy(struct io_wq *wq) { - int node; + int i, node; cpuhp_state_remove_instance_nocalls(io_wq_online, &wq->cpuhp_node); @@ -1299,6 +1380,8 @@ static void io_wq_destroy(struct io_wq *wq) }; io_wqe_cancel_pending_work(wqe, &match); free_cpumask_var(wqe->cpu_mask); + for (i = 0; i < IO_WQ_ACCT_NR; i++) + kfree(wqe->fixed_acct[i].fixed_workers); kfree(wqe); } io_wq_put_hash(wq->hash); From patchwork Sun May 15 13:12:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12850033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EE76C433FE for ; Sun, 15 May 2022 13:12:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236850AbiEONM5 (ORCPT ); Sun, 15 May 2022 09:12:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236841AbiEONM4 (ORCPT ); Sun, 15 May 2022 09:12:56 -0400 Received: from pv50p00im-ztdg10022001.me.com (pv50p00im-ztdg10022001.me.com [17.58.6.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C61D5DF67 for ; Sun, 15 May 2022 06:12:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1652620375; bh=x6IB+rJOUQrmoPA9dhyLFYmiqltZCCEup0t7ONB9B3s=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=lzGtwEWRIssY6BO/oSP98HD0PCObvNv0CnG7a1VPz4fn8VwkMksC7XIGVgDvkXe0O DHbVN0cwJzXkBo7vFuzqiFFWTZSKqiGiTWnIeW8Q2xgHOGWK6/37wLXnRzTt5lzpHw 73INBcpXec5s98DZdJjrP+eaB8amqJ/mk8eyOT3KUUoxEgDyrfVRVSt5QwDMIpOeZI 12OT+xlbD3adHSEdO+fjIDNq2TS6BW9FgVkyS642HK5fs8Fl++zMwrwOZQ0JAQqcbT nqoOIjgplSzFBmhRnRgZzGlrqIlX3T33ycgyZ8kRyvAzjX7ZO7mG6tabooo61ulg+h reg0tglHffPGQ== Received: from localhost.localdomain (pv50p00im-dlb-asmtp-mailmevip.me.com [17.56.9.10]) by pv50p00im-ztdg10022001.me.com (Postfix) with ESMTPSA id F35253E1F13; Sun, 15 May 2022 13:12:52 +0000 (UTC) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 04/11] io-wq: tweak io_get_acct() Date: Sun, 15 May 2022 21:12:23 +0800 Message-Id: <20220515131230.155267-5-haoxu.linux@icloud.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220515131230.155267-1-haoxu.linux@icloud.com> References: <20220515131230.155267-1-haoxu.linux@icloud.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-15_07:2022-05-13,2022-05-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=787 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2009150000 definitions=main-2205150069 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu From: Hao Xu Add an argument for io_get_acct() to indicate fixed or normal worker Signed-off-by: Hao Xu --- fs/io-wq.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/fs/io-wq.c b/fs/io-wq.c index 73fbe62617b7..8db3132dc3a1 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -215,20 +215,24 @@ static void io_worker_release(struct io_worker *worker) complete(&worker->ref_done); } -static inline struct io_wqe_acct *io_get_acct(struct io_wqe *wqe, bool bound) +static inline struct io_wqe_acct *io_get_acct(struct io_wqe *wqe, bool bound, + bool fixed) { - return &wqe->acct[bound ? IO_WQ_ACCT_BOUND : IO_WQ_ACCT_UNBOUND]; + unsigned index = bound ? IO_WQ_ACCT_BOUND : IO_WQ_ACCT_UNBOUND; + + return fixed ? &wqe->fixed_acct[index] : &wqe->acct[index]; } static inline struct io_wqe_acct *io_work_get_acct(struct io_wqe *wqe, struct io_wq_work *work) { - return io_get_acct(wqe, !(work->flags & IO_WQ_WORK_UNBOUND)); + return io_get_acct(wqe, !(work->flags & IO_WQ_WORK_UNBOUND), false); } static inline struct io_wqe_acct *io_wqe_get_acct(struct io_worker *worker) { - return io_get_acct(worker->wqe, worker->flags & IO_WORKER_F_BOUND); + return io_get_acct(worker->wqe, worker->flags & IO_WORKER_F_BOUND, + worker->flags & IO_WORKER_F_FIXED); } static void io_worker_ref_put(struct io_wq *wq) @@ -1131,7 +1135,7 @@ static void io_wqe_cancel_pending_work(struct io_wqe *wqe, int i; retry: for (i = 0; i < IO_WQ_ACCT_NR; i++) { - struct io_wqe_acct *acct = io_get_acct(wqe, i == 0); + struct io_wqe_acct *acct = io_get_acct(wqe, i == 0, false); if (io_acct_cancel_pending_work(wqe, acct, match)) { if (match->cancel_all) From patchwork Sun May 15 13:12:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12850034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBD9FC433F5 for ; Sun, 15 May 2022 13:13:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236856AbiEONNB (ORCPT ); Sun, 15 May 2022 09:13:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236841AbiEONNA (ORCPT ); Sun, 15 May 2022 09:13:00 -0400 Received: from pv50p00im-ztdg10022001.me.com (pv50p00im-ztdg10022001.me.com [17.58.6.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C49C13F4C for ; Sun, 15 May 2022 06:12:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1652620378; bh=LbSTszaxwSmnz3KA0tGezW5v1AbO2VnrRtfC3UGgb0Y=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=ITEFtRA3ytQh84H9rk/z4ehAhtOGfvRwwKa4GhcGL/21gRoHtP0cUvtYp++NHXeot WZLGZ5UzlDtCGf/JZ9JSx9TDrQBAiPE5R8azB4aogtrQ7AxHDBoKhK69YCl1GLEwgi 80QqFtnavhehVORlDUCXtOChXtlBbWmJLwiBIlvvkbzmuxMN2gfcZhjHmbVYRdXbb/ 7izuBOkwVSQdpeXxakx0my/0ZI762zv2QJpPSlzlG24UwaiJoteJi7t5Yx+ekaLDJ2 b9HbjRlQCD8Wfd2tspzBPoIZsMC5mO373Rbv1d+UZFzq624iYjwxmVAUE/iUvwX1TK rFD9BYI2pycsA== Received: from localhost.localdomain (pv50p00im-dlb-asmtp-mailmevip.me.com [17.56.9.10]) by pv50p00im-ztdg10022001.me.com (Postfix) with ESMTPSA id 1A1653E1F17; Sun, 15 May 2022 13:12:55 +0000 (UTC) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 05/11] io-wq: fixed worker initialization Date: Sun, 15 May 2022 21:12:24 +0800 Message-Id: <20220515131230.155267-6-haoxu.linux@icloud.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220515131230.155267-1-haoxu.linux@icloud.com> References: <20220515131230.155267-1-haoxu.linux@icloud.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-15_07:2022-05-13,2022-05-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=821 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2009150000 definitions=main-2205150069 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu From: Hao Xu Implementation of the fixed worker initialization. Signed-off-by: Hao Xu --- fs/io-wq.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/fs/io-wq.c b/fs/io-wq.c index 8db3132dc3a1..329b3ff01545 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -781,6 +781,26 @@ void io_wq_worker_sleeping(struct task_struct *tsk) io_wqe_dec_running(worker); } +static void io_init_new_fixed_worker(struct io_wqe *wqe, + struct io_worker *worker) +{ + struct io_wqe_acct *acct = io_wqe_get_acct(worker); + struct io_wqe_acct *iw_acct = &worker->acct; + unsigned index = acct->index; + unsigned *nr_fixed; + + raw_spin_lock(&acct->lock); + nr_fixed = &acct->nr_fixed; + acct->fixed_workers[*nr_fixed] = worker; + worker->index = (*nr_fixed)++; + iw_acct->nr_works = 0; + iw_acct->max_works = acct->max_works; + iw_acct->index = index; + INIT_WQ_LIST(&iw_acct->work_list); + raw_spin_lock_init(&iw_acct->lock); + raw_spin_unlock(&acct->lock); +} + static void io_init_new_worker(struct io_wqe *wqe, struct io_worker *worker, struct task_struct *tsk) { @@ -794,6 +814,8 @@ static void io_init_new_worker(struct io_wqe *wqe, struct io_worker *worker, list_add_tail_rcu(&worker->all_list, &wqe->all_list); worker->flags |= IO_WORKER_F_FREE; raw_spin_unlock(&wqe->lock); + if (worker->flags & IO_WORKER_F_FIXED) + io_init_new_fixed_worker(wqe, worker); wake_up_new_task(tsk); } @@ -900,6 +922,8 @@ static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, if (index == IO_WQ_ACCT_BOUND) worker->flags |= IO_WORKER_F_BOUND; + if (&wqe->fixed_acct[index] == acct) + worker->flags |= IO_WORKER_F_FIXED; tsk = create_io_thread(io_wqe_worker, worker, wqe->node); if (!IS_ERR(tsk)) { From patchwork Sun May 15 13:12:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12850035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2A3BC433F5 for ; Sun, 15 May 2022 13:13:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236847AbiEONNE (ORCPT ); Sun, 15 May 2022 09:13:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236841AbiEONND (ORCPT ); Sun, 15 May 2022 09:13:03 -0400 Received: from pv50p00im-ztdg10022001.me.com (pv50p00im-ztdg10022001.me.com [17.58.6.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF9BE13F55 for ; Sun, 15 May 2022 06:13:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1652620382; bh=ONT5hvc4ak0dbvjGAMFJw+Jx/e6Ar6ZjDNAyJ2HnnT8=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=AK5IZKf2E23JO2CA1MeiopmI8brlBTIlQe2QPXRRS0OrV7jdYoIFQDh1gtBlc1ybc m5XbGKaxA48KnfBCQow680SeuqO0RMw1yRYf1vVqDc5MLUlamaK7OaQh8LKemuBBHe 7Pcw83v9eP8jutkE6yx3qEJrxWgFpMfYH9ODHjpQFI6DzhNioCFDeQnSabyduGkuFn Iz1UjIF/5oEAU0MEXb/dbTkjDIaf+Bkxl2dx8pS8uYZGfs7Un3Pxi87eobw5dsslMQ IsnbFCfB6OvIPZrnnKbFTh+IKymUfw/f3GvyEecyJr9TApyrwW20Wl44QMHiEwqJxf ZrvFU1JOkskuA== Received: from localhost.localdomain (pv50p00im-dlb-asmtp-mailmevip.me.com [17.56.9.10]) by pv50p00im-ztdg10022001.me.com (Postfix) with ESMTPSA id 8DE143E1F16; Sun, 15 May 2022 13:12:59 +0000 (UTC) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 06/11] io-wq: fixed worker exit Date: Sun, 15 May 2022 21:12:25 +0800 Message-Id: <20220515131230.155267-7-haoxu.linux@icloud.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220515131230.155267-1-haoxu.linux@icloud.com> References: <20220515131230.155267-1-haoxu.linux@icloud.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-15_07:2022-05-13,2022-05-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=983 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2009150000 definitions=main-2205150069 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu From: Hao Xu Implement the fixed worker exit Signed-off-by: Hao Xu --- fs/io-wq.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/fs/io-wq.c b/fs/io-wq.c index 329b3ff01545..c57920ad90a0 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -266,6 +266,29 @@ static bool io_task_worker_match(struct callback_head *cb, void *data) return worker == data; } +static void io_fixed_worker_exit(struct io_worker *worker) +{ + int *nr_fixed; + int index = worker->acct.index; + struct io_wqe *wqe = worker->wqe; + struct io_wqe_acct *acct = io_get_acct(wqe, index == 0, true); + struct io_worker **fixed_workers; + + raw_spin_lock(&acct->lock); + fixed_workers = acct->fixed_workers; + if (!fixed_workers || worker->index == -1) { + raw_spin_unlock(&acct->lock); + return; + } + nr_fixed = &acct->nr_fixed; + /* reuse variable index to represent fixed worker index in its array */ + index = worker->index; + fixed_workers[index] = fixed_workers[*nr_fixed - 1]; + (*nr_fixed)--; + fixed_workers[index]->index = index; + raw_spin_unlock(&acct->lock); +} + static void io_worker_exit(struct io_worker *worker) { struct io_wqe *wqe = worker->wqe; @@ -689,6 +712,7 @@ static int io_wqe_worker(void *data) struct io_wqe *wqe = worker->wqe; struct io_wq *wq = wqe->wq; bool last_timeout = false; + bool fixed = worker->flags & IO_WORKER_F_FIXED; char buf[TASK_COMM_LEN]; worker->flags |= (IO_WORKER_F_UP | IO_WORKER_F_RUNNING); @@ -739,6 +763,8 @@ static int io_wqe_worker(void *data) if (test_bit(IO_WQ_BIT_EXIT, &wq->state)) io_worker_handle_work(worker); + if (fixed) + io_fixed_worker_exit(worker); audit_free(current); io_worker_exit(worker); From patchwork Sun May 15 13:12:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12850036 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17359C433F5 for ; Sun, 15 May 2022 13:13:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231373AbiEONNK (ORCPT ); Sun, 15 May 2022 09:13:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236858AbiEONNK (ORCPT ); Sun, 15 May 2022 09:13:10 -0400 Received: from pv50p00im-ztdg10022001.me.com (pv50p00im-ztdg10022001.me.com [17.58.6.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE66D13F4D for ; Sun, 15 May 2022 06:13:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1652620388; bh=y6vkx1JrYS6Yj1jtrm0AnFQTMC0tyUNU8zGu3cGL3CY=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=vD3mgn3gnA1yErRobWJVBIOgi3kSaHZ/RmYMmPCSGV4SEO7Ix3QhCPaExVAQiEa35 NeHCsqzqNW3FeKz7lyX1rIhpYFbkb5mZB69Z2KxLbgZwseviIhP45MhqXqBga2u2Z9 h++K6SjNSiW6DY63w7h6kVmW8qxGETyjjVv30n1wHxf3Mds1SMrba5Jl8t9wl2rEER lz1z3Zk31shEwV+3Y7Naj9hKmpeXlWXqAYgPzRqpKI124f6kGyvbw/VyC/jFp95ig9 Xz5ft1Ac+zOGsH/94T896aS4r3G93f61KVY9Yztjp/1PCm6rQaXA9FmZp31GvkGTCx dMnnYHxDQoj6g== Received: from localhost.localdomain (pv50p00im-dlb-asmtp-mailmevip.me.com [17.56.9.10]) by pv50p00im-ztdg10022001.me.com (Postfix) with ESMTPSA id DC7B43E1D60; Sun, 15 May 2022 13:13:02 +0000 (UTC) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 07/11] io-wq: implement fixed worker logic Date: Sun, 15 May 2022 21:12:26 +0800 Message-Id: <20220515131230.155267-8-haoxu.linux@icloud.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220515131230.155267-1-haoxu.linux@icloud.com> References: <20220515131230.155267-1-haoxu.linux@icloud.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-15_07:2022-05-13,2022-05-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2009150000 definitions=main-2205150069 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu From: Hao Xu The current implementation of io-wq has big spinlock contension. The main reason is the single work list model. All producers(who insert works) and consumers(io-workers) have to grap wqe->lock to move ahead. Set max_worker to 3 or 4, do a fio read test, we can see 40%~50% lock contension. Introduce fixed io-workers which sticks there to handle works and have their own work list. previous: producer0 ---insert---> work_list ---get---> io-worker0,1,2 now: ---> private work_list0 --get--> fixed-worker0 / producer0 --insert----> private work_list1 --get--> fixed-worker1 | \ | ---> private work_list2 --get--> fixed-worker2 | |---insert---> public work_list --get--> (normal)io-worker Since each fixed-worker has a private work list, the contension will be limited to a smaller range(the private work list). Logic of fixed-worker: first handle private works then public ones. Logic of normal io-worker: only handle public works. Logic of producer: 1) randomly pick a private work list and check if it is full, insert the work if it's not 2) insert the work to the public work list if 1) fails. The get logic of a private list: fixed-worker grab all the works in its private work list(like what tctx_task_work() does) rather than one by one.(this code is in the next patches as a optimization) To achieve this, we need to add an io_wqe_acct for each fixed-worker struct, and though this we can leverage the old code as much as possible, which makes the new design clean and compatible. Good things of this feature: 1) bound and unbound work lists now have different spinlocks. 2) much smaller contension between work producers and consumers. 3) fixed workers are friendly for users to control: binding cpus, reset priority etc. Wrote a nop test program to test it, 3 fixed-workers VS 3 normal workers. normal workers: ./run_nop_wqe.sh nop_wqe_normal 200000 100 3 1-3 time spent: 10464397 usecs IOPS: 1911242 time spent: 9610976 usecs IOPS: 2080954 time spent: 9807361 usecs IOPS: 2039284 fixed workers: ./run_nop_wqe.sh nop_wqe_fixed 200000 100 3 1-3 time spent: 17314274 usecs IOPS: 1155116 time spent: 17016942 usecs IOPS: 1175299 time spent: 17908684 usecs IOPS: 1116776 About 2x improvement. From perf result, almost no acct->lock contension. Test program: https://github.com/HowHsu/liburing/tree/fixed_worker Signed-off-by: Hao Xu --- fs/io-wq.c | 148 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 122 insertions(+), 26 deletions(-) diff --git a/fs/io-wq.c b/fs/io-wq.c index c57920ad90a0..8e152c076dd5 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -73,6 +73,7 @@ struct io_wqe_acct { }; }; unsigned max_works; + unsigned work_seq; union { struct io_wq_work_list work_list; struct { @@ -631,9 +632,9 @@ static void io_assign_current_work(struct io_worker *worker, static void io_wqe_enqueue(struct io_wqe *wqe, struct io_wq_work *work); -static void io_worker_handle_work(struct io_worker *worker) +static void io_worker_handle_work(struct io_worker *worker, + struct io_wqe_acct *acct) { - struct io_wqe_acct *acct = io_wqe_get_acct(worker); struct io_wqe *wqe = worker->wqe; struct io_wq *wq = wqe->wq; bool do_kill = test_bit(IO_WQ_BIT_EXIT, &wq->state); @@ -705,19 +706,31 @@ static void io_worker_handle_work(struct io_worker *worker) } while (1); } +static inline void io_worker_handle_private_work(struct io_worker *worker) +{ + io_worker_handle_work(worker, &worker->acct); +} + +static inline void io_worker_handle_public_work(struct io_worker *worker) +{ + io_worker_handle_work(worker, io_wqe_get_acct(worker)); +} + static int io_wqe_worker(void *data) { struct io_worker *worker = data; - struct io_wqe_acct *acct = io_wqe_get_acct(worker); struct io_wqe *wqe = worker->wqe; struct io_wq *wq = wqe->wq; - bool last_timeout = false; + struct io_wqe_acct *acct = + io_get_acct(wqe, worker->flags & IO_WORKER_F_BOUND, false); bool fixed = worker->flags & IO_WORKER_F_FIXED; + bool last_timeout = false; char buf[TASK_COMM_LEN]; worker->flags |= (IO_WORKER_F_UP | IO_WORKER_F_RUNNING); - snprintf(buf, sizeof(buf), "iou-wrk-%d", wq->task->pid); + snprintf(buf, sizeof(buf), fixed ? "iou-fix-%d" : "iou-wrk-%d", + wq->task->pid); set_task_comm(current, buf); audit_alloc_kernel(current); @@ -729,13 +742,24 @@ static int io_wqe_worker(void *data) break; set_current_state(TASK_INTERRUPTIBLE); - while (!(worker->flags & IO_WORKER_F_EXIT) && - io_acct_run_queue(acct)) - io_worker_handle_work(worker); - + if (fixed) { + while (io_acct_run_queue(&worker->acct)) + io_worker_handle_private_work(worker); + if (io_acct_run_queue(acct)) + io_worker_handle_public_work(worker); + } else { + while (io_acct_run_queue(acct)) + io_worker_handle_public_work(worker); + } raw_spin_lock(&wqe->lock); - /* timed out, exit unless we're the last worker */ - if (last_timeout && acct->nr_workers > 1) { + /* timed out, a worker will exit only if: + * - not a fixed worker + * - not the last non-fixed worker + * + * the second condition is due to we need at least one worker to + * handle the public work list. + */ + if (last_timeout && !fixed && acct->nr_workers > 1) { acct->nr_workers--; raw_spin_unlock(&wqe->lock); __set_current_state(TASK_RUNNING); @@ -761,10 +785,18 @@ static int io_wqe_worker(void *data) last_timeout = !ret; } - if (test_bit(IO_WQ_BIT_EXIT, &wq->state)) - io_worker_handle_work(worker); - if (fixed) + if (test_bit(IO_WQ_BIT_EXIT, &wq->state) && !fixed) + io_worker_handle_public_work(worker); + if (fixed) { io_fixed_worker_exit(worker); + /* + * Check and handle private work list again + * to avoid race with private work insertion + * TODO: an alternative way is to deliver + * works to the public work list + */ + io_worker_handle_private_work(worker); + } audit_free(current); io_worker_exit(worker); @@ -1008,9 +1040,9 @@ static void io_run_cancel(struct io_wq_work *work, struct io_wqe *wqe) } while (work); } -static void io_wqe_insert_work(struct io_wqe *wqe, struct io_wq_work *work) +static void io_wqe_insert_work(struct io_wqe *wqe, struct io_wq_work *work, + struct io_wqe_acct *acct) { - struct io_wqe_acct *acct = io_work_get_acct(wqe, work); unsigned int hash; struct io_wq_work *tail; @@ -1029,6 +1061,45 @@ static void io_wqe_insert_work(struct io_wqe *wqe, struct io_wq_work *work) wq_list_add_after(&work->list, &tail->list, &acct->work_list); } +static bool io_wqe_insert_private_work(struct io_wqe *wqe, + struct io_wq_work *work, + struct io_wqe_acct *acct) +{ + unsigned int nr_fixed; + struct io_worker *fixed_worker; + struct io_wqe_acct *iw_acct; + unsigned int fixed_worker_index; + + raw_spin_lock(&acct->lock); + nr_fixed = acct->nr_fixed; + if (!nr_fixed) { + raw_spin_unlock(&acct->lock); + return false; + } + + fixed_worker_index = (acct->work_seq++) % nr_fixed; + fixed_worker = acct->fixed_workers[fixed_worker_index]; + if (!fixed_worker || fixed_worker->flags & IO_WORKER_F_EXIT) { + raw_spin_unlock(&acct->lock); + return false; + } + iw_acct = &fixed_worker->acct; + + raw_spin_lock(&iw_acct->lock); + if (iw_acct->nr_works < iw_acct->max_works) { + io_wqe_insert_work(wqe, work, iw_acct); + iw_acct->nr_works++; + raw_spin_unlock(&iw_acct->lock); + wake_up_process(fixed_worker->task); + raw_spin_unlock(&acct->lock); + return true; + } + raw_spin_unlock(&iw_acct->lock); + raw_spin_unlock(&acct->lock); + + return false; +} + static bool io_wq_work_match_item(struct io_wq_work *work, void *data) { return work == data; @@ -1037,6 +1108,7 @@ static bool io_wq_work_match_item(struct io_wq_work *work, void *data) static void io_wqe_enqueue(struct io_wqe *wqe, struct io_wq_work *work) { struct io_wqe_acct *acct = io_work_get_acct(wqe, work); + struct io_wqe_acct *fixed_acct; struct io_cb_cancel_data match; unsigned work_flags = work->flags; bool do_create; @@ -1051,8 +1123,14 @@ static void io_wqe_enqueue(struct io_wqe *wqe, struct io_wq_work *work) return; } + fixed_acct = io_get_acct(wqe, !acct->index, true); + if (fixed_acct->fixed_worker_registered && !io_wq_is_hashed(work)) { + if (io_wqe_insert_private_work(wqe, work, fixed_acct)) + return; + } + raw_spin_lock(&acct->lock); - io_wqe_insert_work(wqe, work); + io_wqe_insert_work(wqe, work, acct); clear_bit(IO_ACCT_STALLED_BIT, &acct->flags); raw_spin_unlock(&acct->lock); @@ -1138,9 +1216,9 @@ static bool io_wq_worker_cancel(struct io_worker *worker, void *data) static inline void io_wqe_remove_pending(struct io_wqe *wqe, struct io_wq_work *work, - struct io_wq_work_node *prev) + struct io_wq_work_node *prev, + struct io_wqe_acct *acct) { - struct io_wqe_acct *acct = io_work_get_acct(wqe, work); unsigned int hash = io_get_work_hash(work); struct io_wq_work *prev_work = NULL; @@ -1167,7 +1245,7 @@ static bool io_acct_cancel_pending_work(struct io_wqe *wqe, work = container_of(node, struct io_wq_work, list); if (!match->fn(work, match->data)) continue; - io_wqe_remove_pending(wqe, work, prev); + io_wqe_remove_pending(wqe, work, prev, acct); raw_spin_unlock(&acct->lock); io_run_cancel(work, wqe); match->nr_pending++; @@ -1182,17 +1260,35 @@ static bool io_acct_cancel_pending_work(struct io_wqe *wqe, static void io_wqe_cancel_pending_work(struct io_wqe *wqe, struct io_cb_cancel_data *match) { - int i; -retry: - for (i = 0; i < IO_WQ_ACCT_NR; i++) { - struct io_wqe_acct *acct = io_get_acct(wqe, i == 0, false); + int i, j; + struct io_wqe_acct *acct, *iw_acct; +retry_public: + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + acct = io_get_acct(wqe, i == 0, false); if (io_acct_cancel_pending_work(wqe, acct, match)) { if (match->cancel_all) - goto retry; - break; + goto retry_public; + return; } } + +retry_private: + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + acct = io_get_acct(wqe, i == 0, true); + raw_spin_lock(&acct->lock); + for (j = 0; j < acct->nr_fixed; j++) { + iw_acct = &acct->fixed_workers[j]->acct; + if (io_acct_cancel_pending_work(wqe, iw_acct, match)) { + if (match->cancel_all) { + raw_spin_unlock(&acct->lock); + goto retry_private; + } + break; + } + } + raw_spin_unlock(&acct->lock); + } } static void io_wqe_cancel_running_work(struct io_wqe *wqe, From patchwork Sun May 15 13:12:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12850037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7758EC433EF for ; Sun, 15 May 2022 13:13:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236754AbiEONNN (ORCPT ); Sun, 15 May 2022 09:13:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39406 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236841AbiEONNN (ORCPT ); Sun, 15 May 2022 09:13:13 -0400 Received: from pv50p00im-ztdg10022001.me.com (pv50p00im-ztdg10022001.me.com [17.58.6.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71B6613F55 for ; Sun, 15 May 2022 06:13:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1652620392; bh=YeTgbXY1hFXBn3cF4GCvZFOS6oanqEyDJGOjwqDwC7M=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=RQL9Q0+kTzhu6NE/mCOB7/LHK/iZKbBq17w/DUsqnmrdhHrd11O8G7/eqA3ZtYi3g 4M1Svb51om5W7wfkspQ1CX8+Tu2Gq8QbXDL2VLfz8j3O2vsSuchUv5beta/QcvuEoI 179aXnveVHYAvbrTPEqvrQSMhjmQqqa0xm/68VqQGDZDgfU5VSTrMs75ifX2yuL8lI JnCY6B6B86CdGNmvrIWf/6HVHPa8IXFRAgwuxBdqU+b310jPDwaRLMhOPBHjwcVtnI or0EkOqKduv8MS5EmiK9zb8PK0YvvTo124WyXs/L3afV2NS9XhiWasZ+elsObNnWI8 V6p/DBBPjrg+g== Received: from localhost.localdomain (pv50p00im-dlb-asmtp-mailmevip.me.com [17.56.9.10]) by pv50p00im-ztdg10022001.me.com (Postfix) with ESMTPSA id 7C3DB3E1D26; Sun, 15 May 2022 13:13:09 +0000 (UTC) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 08/11] io-wq: batch the handling of fixed worker private works Date: Sun, 15 May 2022 21:12:27 +0800 Message-Id: <20220515131230.155267-9-haoxu.linux@icloud.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220515131230.155267-1-haoxu.linux@icloud.com> References: <20220515131230.155267-1-haoxu.linux@icloud.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-15_07:2022-05-13,2022-05-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=684 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2009150000 definitions=main-2205150069 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu From: Hao Xu Reduce acct->lock contension by batching the handling of private work list for fixed_workers. Signed-off-by: Hao Xu --- fs/io-wq.c | 42 +++++++++++++++++++++++++++++++++--------- fs/io-wq.h | 5 +++++ 2 files changed, 38 insertions(+), 9 deletions(-) diff --git a/fs/io-wq.c b/fs/io-wq.c index 8e152c076dd5..7c13cc01e5e5 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -547,8 +547,23 @@ static bool io_wait_on_hash(struct io_wqe *wqe, unsigned int hash) return ret; } +static inline void conditional_acct_lock(struct io_wqe_acct *acct, + bool needs_lock) +{ + if (needs_lock) + raw_spin_lock(&acct->lock); +} + +static inline void conditional_acct_unlock(struct io_wqe_acct *acct, + bool needs_lock) +{ + if (needs_lock) + raw_spin_unlock(&acct->lock); +} + static struct io_wq_work *io_get_next_work(struct io_wqe_acct *acct, - struct io_worker *worker) + struct io_worker *worker, + bool needs_lock) __must_hold(acct->lock) { struct io_wq_work_node *node, *prev; @@ -556,6 +571,7 @@ static struct io_wq_work *io_get_next_work(struct io_wqe_acct *acct, unsigned int stall_hash = -1U; struct io_wqe *wqe = worker->wqe; + conditional_acct_lock(acct, needs_lock); wq_list_for_each(node, prev, &acct->work_list) { unsigned int hash; @@ -564,6 +580,7 @@ static struct io_wq_work *io_get_next_work(struct io_wqe_acct *acct, /* not hashed, can run anytime */ if (!io_wq_is_hashed(work)) { wq_list_del(&acct->work_list, node, prev); + conditional_acct_unlock(acct, needs_lock); return work; } @@ -575,6 +592,7 @@ static struct io_wq_work *io_get_next_work(struct io_wqe_acct *acct, if (!test_and_set_bit(hash, &wqe->wq->hash->map)) { wqe->hash_tail[hash] = NULL; wq_list_cut(&acct->work_list, &tail->list, prev); + conditional_acct_unlock(acct, needs_lock); return work; } if (stall_hash == -1U) @@ -591,15 +609,16 @@ static struct io_wq_work *io_get_next_work(struct io_wqe_acct *acct, * work being added and clearing the stalled bit. */ set_bit(IO_ACCT_STALLED_BIT, &acct->flags); - raw_spin_unlock(&acct->lock); + conditional_acct_unlock(acct, needs_lock); unstalled = io_wait_on_hash(wqe, stall_hash); - raw_spin_lock(&acct->lock); + conditional_acct_lock(acct, needs_lock); if (unstalled) { clear_bit(IO_ACCT_STALLED_BIT, &acct->flags); if (wq_has_sleeper(&wqe->wq->hash->wait)) wake_up(&wqe->wq->hash->wait); } } + conditional_acct_unlock(acct, needs_lock); return NULL; } @@ -633,7 +652,7 @@ static void io_assign_current_work(struct io_worker *worker, static void io_wqe_enqueue(struct io_wqe *wqe, struct io_wq_work *work); static void io_worker_handle_work(struct io_worker *worker, - struct io_wqe_acct *acct) + struct io_wqe_acct *acct, bool needs_lock) { struct io_wqe *wqe = worker->wqe; struct io_wq *wq = wqe->wq; @@ -649,9 +668,7 @@ static void io_worker_handle_work(struct io_worker *worker, * can't make progress, any work completion or insertion will * clear the stalled flag. */ - raw_spin_lock(&acct->lock); - work = io_get_next_work(acct, worker); - raw_spin_unlock(&acct->lock); + work = io_get_next_work(acct, worker, needs_lock); if (work) { __io_worker_busy(wqe, worker); @@ -708,12 +725,19 @@ static void io_worker_handle_work(struct io_worker *worker, static inline void io_worker_handle_private_work(struct io_worker *worker) { - io_worker_handle_work(worker, &worker->acct); + struct io_wqe_acct acct; + + raw_spin_lock(&worker->acct.lock); + acct = worker->acct; + wq_list_clean(&worker->acct.work_list); + worker->acct.nr_works = 0; + raw_spin_unlock(&worker->acct.lock); + io_worker_handle_work(worker, &acct, false); } static inline void io_worker_handle_public_work(struct io_worker *worker) { - io_worker_handle_work(worker, io_wqe_get_acct(worker)); + io_worker_handle_work(worker, io_wqe_get_acct(worker), true); } static int io_wqe_worker(void *data) diff --git a/fs/io-wq.h b/fs/io-wq.h index ba6eee76d028..ef3ce577e6b7 100644 --- a/fs/io-wq.h +++ b/fs/io-wq.h @@ -40,6 +40,11 @@ struct io_wq_work_list { (list)->first = NULL; \ } while (0) +static inline void wq_list_clean(struct io_wq_work_list *list) +{ + list->first = list->last = NULL; +} + static inline void wq_list_add_after(struct io_wq_work_node *node, struct io_wq_work_node *pos, struct io_wq_work_list *list) From patchwork Sun May 15 13:12:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12850038 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05F64C433F5 for ; Sun, 15 May 2022 13:13:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236841AbiEONNR (ORCPT ); Sun, 15 May 2022 09:13:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236859AbiEONNQ (ORCPT ); Sun, 15 May 2022 09:13:16 -0400 Received: from pv50p00im-ztdg10022001.me.com (pv50p00im-ztdg10022001.me.com [17.58.6.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7630313F57 for ; Sun, 15 May 2022 06:13:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1652620395; bh=LE5V/iIh+SDCPYldNcWVy0tTktUXqXrbGWnxxq/DV5Q=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=fXCyFh9e0H7zUa54cfzhiDHLmMPTCJJB1cwIG0P74WgXxwBsrFSause48czmPXVUG w1FVSo1tpcO5oHWs+aq5jTh9wdwO7sESNf1ylIsMysBVc8pzZ0qGv9tipK3oqKUwZt NlQjRMXUlHhcW3CVUoXo5GZp2r60xnIIipOajX7vnNjRRCMcfpgqKIxJjo63W/Q3+X ZYAsrLmXLvPcSUriO0p9ORLXvFb72Kheqla782KSYVIRtshJ+JkoCOFqavaMfBAZzP E0VGrqd4cWDY2z6Yj8wwuaFXK9CGSFBHF8SFZExluUVqSvh+TVX5+PPPoUmXhofeHs sI4YrMdS1BHxA== Received: from localhost.localdomain (pv50p00im-dlb-asmtp-mailmevip.me.com [17.56.9.10]) by pv50p00im-ztdg10022001.me.com (Postfix) with ESMTPSA id A6D1C3E1D48; Sun, 15 May 2022 13:13:12 +0000 (UTC) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 09/11] io_uring: add register fixed worker interface Date: Sun, 15 May 2022 21:12:28 +0800 Message-Id: <20220515131230.155267-10-haoxu.linux@icloud.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220515131230.155267-1-haoxu.linux@icloud.com> References: <20220515131230.155267-1-haoxu.linux@icloud.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-15_07:2022-05-13,2022-05-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=726 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2009150000 definitions=main-2205150069 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu From: Hao Xu Add an io_uring_register() interface to register fixed workers and indicate its work capacity. The argument is an array of two elements each is struct { __s32 nr_workers; __s32 max_works; } (nr_workers, max_works) meaning nr_workers or max_works < -1 invalid nr_workers or max_works == -1 get the old value back nr_workers or max_works >= 0 get the old value and set to the new value Signed-off-by: Hao Xu --- fs/io-wq.c | 101 ++++++++++++++++++++++++++++++++++ fs/io-wq.h | 3 + fs/io_uring.c | 71 ++++++++++++++++++++++++ include/uapi/linux/io_uring.h | 11 ++++ 4 files changed, 186 insertions(+) diff --git a/fs/io-wq.c b/fs/io-wq.c index 7c13cc01e5e5..66d3c741613f 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -1678,6 +1678,107 @@ int io_wq_max_workers(struct io_wq *wq, int *new_count) return 0; } +/* + * Set max number of fixed workers and the capacity of private work list, + * returns old value. If new_count is -1, then just return the old value. + */ +int io_wq_fixed_workers(struct io_wq *wq, + struct io_uring_fixed_worker_arg *new_count) +{ + struct io_uring_fixed_worker_arg prev[IO_WQ_ACCT_NR]; + bool first_node = true; + int i, node; + bool readonly[2] = { + (new_count[0].nr_workers == -1 && new_count[0].max_works == -1), + (new_count[1].nr_workers == -1 && new_count[1].max_works == -1), + }; + + BUILD_BUG_ON((int) IO_WQ_ACCT_BOUND != (int) IO_WQ_BOUND); + BUILD_BUG_ON((int) IO_WQ_ACCT_UNBOUND != (int) IO_WQ_UNBOUND); + BUILD_BUG_ON((int) IO_WQ_ACCT_NR != 2); + + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + if (new_count[i].nr_workers > task_rlimit(current, RLIMIT_NPROC)) + new_count[i].nr_workers = + task_rlimit(current, RLIMIT_NPROC); + } + + rcu_read_lock(); + for_each_node(node) { + int j; + struct io_wqe *wqe = wq->wqes[node]; + + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + struct io_wqe_acct *acct = &wqe->fixed_acct[i]; + int *nr_fixed, *max_works; + struct io_worker **fixed_workers; + int nr = new_count[i].nr_workers; + + raw_spin_lock(&acct->lock); + nr_fixed = &acct->nr_fixed; + max_works = &acct->max_works; + fixed_workers = acct->fixed_workers; + if (first_node) { + prev[i].nr_workers = *nr_fixed; + prev[i].max_works = *max_works; + } + if (readonly[i]) { + raw_spin_unlock(&acct->lock); + continue; + } + if (*nr_fixed == nr || nr == -1) { + *max_works = new_count[i].max_works; + raw_spin_unlock(&acct->lock); + continue; + } + for (j = 0; j < *nr_fixed; j++) { + struct io_worker *worker = fixed_workers[j]; + + if (!worker) + continue; + worker->flags |= IO_WORKER_F_EXIT; + /* + * Mark index to -1 to avoid false deletion + * in io_fixed_worker_exit() + */ + worker->index = -1; + /* + * Once a worker is in fixed_workers array + * it is definitely there before we release + * the acct->lock below. That's why we don't + * need to increment the worker->ref here. + */ + wake_up_process(worker->task); + } + kfree(fixed_workers); + acct->fixed_workers = NULL; + *nr_fixed = 0; + *max_works = new_count[i].max_works; + acct->fixed_workers = kzalloc_node( + sizeof(*fixed_workers) * nr, + GFP_KERNEL, wqe->node); + if (!acct->fixed_workers) { + raw_spin_unlock(&acct->lock); + return -ENOMEM; + } + raw_spin_unlock(&acct->lock); + for (j = 0; j < nr; j++) + io_wqe_create_worker(wqe, acct); + + acct->fixed_worker_registered = !!nr; + } + first_node = false; + } + rcu_read_unlock(); + + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + new_count[i].nr_workers = prev[i].nr_workers; + new_count[i].max_works = prev[i].max_works; + } + + return 0; +} + static __init int io_wq_init(void) { int ret; diff --git a/fs/io-wq.h b/fs/io-wq.h index ef3ce577e6b7..bf90488b0283 100644 --- a/fs/io-wq.h +++ b/fs/io-wq.h @@ -2,6 +2,7 @@ #define INTERNAL_IO_WQ_H #include +#include struct io_wq; @@ -202,6 +203,8 @@ void io_wq_hash_work(struct io_wq_work *work, void *val); int io_wq_cpu_affinity(struct io_wq *wq, cpumask_var_t mask); int io_wq_max_workers(struct io_wq *wq, int *new_count); +int io_wq_fixed_workers(struct io_wq *wq, + struct io_uring_fixed_worker_arg *new_count); static inline bool io_wq_is_hashed(struct io_wq_work *work) { diff --git a/fs/io_uring.c b/fs/io_uring.c index 3c39f5413c1b..b223dbd44891 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -11974,6 +11974,71 @@ static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, return ret; } +static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx, + void __user *arg) + __must_hold(&ctx->uring_lock) +{ + struct io_uring_task *tctx = NULL; + struct io_sq_data *sqd = NULL; + struct io_uring_fixed_worker_arg new_count[2]; + int i, ret; + + if (copy_from_user(new_count, arg, sizeof(new_count))) + return -EFAULT; + for (i = 0; i < ARRAY_SIZE(new_count); i++) { + int nr_workers = new_count[i].nr_workers; + int max_works = new_count[i].max_works; + + if (nr_workers < -1 || max_works < -1) + return -EINVAL; + } + + if (ctx->flags & IORING_SETUP_SQPOLL) { + sqd = ctx->sq_data; + if (sqd) { + /* + * Observe the correct sqd->lock -> ctx->uring_lock + * ordering. Fine to drop uring_lock here, we hold + * a ref to the ctx. + */ + refcount_inc(&sqd->refs); + mutex_unlock(&ctx->uring_lock); + mutex_lock(&sqd->lock); + mutex_lock(&ctx->uring_lock); + if (sqd->thread) + tctx = sqd->thread->io_uring; + } + } else { + tctx = current->io_uring; + } + + if (tctx && tctx->io_wq) { + ret = io_wq_fixed_workers(tctx->io_wq, new_count); + if (ret) + goto err; + } else { + memset(new_count, -1, sizeof(new_count)); + } + + if (sqd) { + mutex_unlock(&sqd->lock); + io_put_sq_data(sqd); + } + + if (copy_to_user(arg, new_count, sizeof(new_count))) + return -EFAULT; + + /* that's it for SQPOLL, only the SQPOLL task creates requests */ + if (sqd) + return 0; + +err: + if (sqd) { + mutex_unlock(&sqd->lock); + io_put_sq_data(sqd); + } + return ret; +} static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, void __user *arg, unsigned nr_args) __releases(ctx->uring_lock) @@ -12105,6 +12170,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, case IORING_UNREGISTER_RING_FDS: ret = io_ringfd_unregister(ctx, arg, nr_args); break; + case IORING_REGISTER_IOWQ_FIXED_WORKERS: + ret = -EINVAL; + if (!arg || nr_args != 2) + break; + ret = io_register_iowq_fixed_workers(ctx, arg); + break; default: ret = -EINVAL; break; diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 15f821af9242..6fc649259142 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -384,6 +384,12 @@ enum { IORING_REGISTER_RING_FDS = 20, IORING_UNREGISTER_RING_FDS = 21, + /* set number of fixed workers and number + * of works in a private work list which + * belongs to a fixed worker + */ + IORING_REGISTER_IOWQ_FIXED_WORKERS = 22, + /* this goes last */ IORING_REGISTER_LAST }; @@ -487,4 +493,9 @@ struct io_uring_getevents_arg { __u64 ts; }; +struct io_uring_fixed_worker_arg { + __s32 nr_workers; + __s32 max_works; +}; + #endif From patchwork Sun May 15 13:12:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12850039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 026C6C433F5 for ; Sun, 15 May 2022 13:13:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236859AbiEONNU (ORCPT ); Sun, 15 May 2022 09:13:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236858AbiEONNU (ORCPT ); Sun, 15 May 2022 09:13:20 -0400 Received: from pv50p00im-ztdg10022001.me.com (pv50p00im-ztdg10022001.me.com [17.58.6.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E603E13F5C for ; Sun, 15 May 2022 06:13:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1652620398; bh=rJGAPD/Ui8BYsy9qFK5mTIpUdBqNqsK4aBpwJj9KTRI=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=zl4ZQ8qLKs38wIQP/UnOluQa4Et8jGq6/NWQjUtiax8RI5I0atFbewJr5gUnrOZxk 49Kit4gGc33Jo88AvWY3LVXVYw50yDQXfQXRUzZ2jy1g3sOpBjaYWbYFPN8hLnqDTa Lpwmhl38dioNWdVnFrqB/62ZePzkwje87Du0jxftB2qv+lr9MuhWAq0fWOHC/2E5ns nZ6ZkFgHocPVqvZgMVJMRsPZZkJVsmF7j4KhLHeSLPUKPBLmDS3NeW7vNZsXVt81DA RzMhcWXZW3hUIkFq5w1VIGC3mQjBfjxAE7xtP7YRhFsb7mB56HC4OhN+0tNDJA9EvS jBIyN+j++xWiw== Received: from localhost.localdomain (pv50p00im-dlb-asmtp-mailmevip.me.com [17.56.9.10]) by pv50p00im-ztdg10022001.me.com (Postfix) with ESMTPSA id CD2E33E1D0E; Sun, 15 May 2022 13:13:15 +0000 (UTC) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 10/11] io-wq: add an work list for fixed worker Date: Sun, 15 May 2022 21:12:29 +0800 Message-Id: <20220515131230.155267-11-haoxu.linux@icloud.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220515131230.155267-1-haoxu.linux@icloud.com> References: <20220515131230.155267-1-haoxu.linux@icloud.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-15_07:2022-05-13,2022-05-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=861 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2009150000 definitions=main-2205150069 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu From: Hao Xu Previously when a fixed worker handles its private works, it get all of them from worker->acct.work_list to a temporary acct->work_list. This prevents work cancellation since the cancellation process cannot find works from this temporary work list. Thus add a new acct so to address it. Signed-off-by: Hao Xu --- fs/io-wq.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/fs/io-wq.c b/fs/io-wq.c index 66d3c741613f..c6e4179a9961 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -128,6 +128,7 @@ struct io_worker { }; int index; struct io_wqe_acct acct; + struct io_wqe_acct exec_acct; }; #if BITS_PER_LONG == 64 @@ -725,14 +726,17 @@ static void io_worker_handle_work(struct io_worker *worker, static inline void io_worker_handle_private_work(struct io_worker *worker) { - struct io_wqe_acct acct; + struct io_wqe_acct *acct = &worker->acct; + struct io_wqe_acct *exec_acct = &worker->exec_acct; - raw_spin_lock(&worker->acct.lock); - acct = worker->acct; - wq_list_clean(&worker->acct.work_list); - worker->acct.nr_works = 0; - raw_spin_unlock(&worker->acct.lock); - io_worker_handle_work(worker, &acct, false); + raw_spin_lock(&acct->lock); + exec_acct->nr_works = acct->nr_works; + exec_acct->max_works = acct->max_works; + exec_acct->work_list = acct->work_list; + wq_list_clean(&acct->work_list); + acct->nr_works = 0; + raw_spin_unlock(&acct->lock); + io_worker_handle_work(worker, exec_acct, false); } static inline void io_worker_handle_public_work(struct io_worker *worker) @@ -868,6 +872,7 @@ static void io_init_new_fixed_worker(struct io_wqe *wqe, { struct io_wqe_acct *acct = io_wqe_get_acct(worker); struct io_wqe_acct *iw_acct = &worker->acct; + struct io_wqe_acct *exec_acct = &worker->exec_acct; unsigned index = acct->index; unsigned *nr_fixed; @@ -880,6 +885,7 @@ static void io_init_new_fixed_worker(struct io_wqe *wqe, iw_acct->index = index; INIT_WQ_LIST(&iw_acct->work_list); raw_spin_lock_init(&iw_acct->lock); + raw_spin_lock_init(&exec_acct->lock); raw_spin_unlock(&acct->lock); } From patchwork Sun May 15 13:12:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12850040 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6736AC433EF for ; Sun, 15 May 2022 13:13:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236858AbiEONN0 (ORCPT ); Sun, 15 May 2022 09:13:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234539AbiEONNZ (ORCPT ); Sun, 15 May 2022 09:13:25 -0400 Received: from pv50p00im-ztdg10022001.me.com (pv50p00im-ztdg10022001.me.com [17.58.6.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 311D0DF71 for ; Sun, 15 May 2022 06:13:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1652620403; bh=osERllyiWWDuMwoMXBhRuq5EfjuyDVz5xKPNUP5znhU=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=IpYdqZTU+JbOVqdTIxGV7idv6RIeyqBhiql29FVbHQ+SpaZ+B3YmP0MioBTomnY9H lBhuh3aFD/4fSwzrGI7rhTOo9Rx4had70DpG3rsL6jRRJNZHIHOJaXK0uznGu4bmmU WfptpmlY2qHuWuEwuBYkiPJvmRtDY0Iv6xmHbitzV3C6sS0ywM5E7ecz+LHAbn7Pf6 0A14aY6x8uc1jtCn2Ux6tXmFpkpl3Pzo9gBfgzGAbzuYBo1tdQWb6WFE2d4R3yQDyU HQ4yzChrWSF+0ed+l63xDyLxw9W2wyx4sqL2MjtxstcCDoMZpt3tdqVFTcFNnZUIWE xgC2HOSc9e7Mg== Received: from localhost.localdomain (pv50p00im-dlb-asmtp-mailmevip.me.com [17.56.9.10]) by pv50p00im-ztdg10022001.me.com (Postfix) with ESMTPSA id 6AE6C3E1CCA; Sun, 15 May 2022 13:13:18 +0000 (UTC) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 11/11] io_uring: cancel works in exec work list for fixed worker Date: Sun, 15 May 2022 21:12:30 +0800 Message-Id: <20220515131230.155267-12-haoxu.linux@icloud.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220515131230.155267-1-haoxu.linux@icloud.com> References: <20220515131230.155267-1-haoxu.linux@icloud.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-15_07:2022-05-13,2022-05-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2009150000 definitions=main-2205150069 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu From: Hao Xu When users want to cancel a request, look into the exec work list of fixed worker as well. It's not sane to ignore it. Signed-off-by: Hao Xu --- fs/io-wq.c | 46 ++++++++++++++++++++++++++++++---------------- 1 file changed, 30 insertions(+), 16 deletions(-) diff --git a/fs/io-wq.c b/fs/io-wq.c index c6e4179a9961..5345a5f57e4f 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -1287,32 +1287,26 @@ static bool io_acct_cancel_pending_work(struct io_wqe *wqe, return false; } -static void io_wqe_cancel_pending_work(struct io_wqe *wqe, - struct io_cb_cancel_data *match) -{ +static void io_wqe_cancel_pending_work_fixed(struct io_wqe *wqe, + struct io_cb_cancel_data *match, + bool exec) { int i, j; struct io_wqe_acct *acct, *iw_acct; -retry_public: - for (i = 0; i < IO_WQ_ACCT_NR; i++) { - acct = io_get_acct(wqe, i == 0, false); - if (io_acct_cancel_pending_work(wqe, acct, match)) { - if (match->cancel_all) - goto retry_public; - return; - } - } - -retry_private: +retry: for (i = 0; i < IO_WQ_ACCT_NR; i++) { acct = io_get_acct(wqe, i == 0, true); raw_spin_lock(&acct->lock); for (j = 0; j < acct->nr_fixed; j++) { - iw_acct = &acct->fixed_workers[j]->acct; + if (exec) + iw_acct = &acct->fixed_workers[j]->acct; + else + iw_acct = &acct->fixed_workers[j]->exec_acct; + if (io_acct_cancel_pending_work(wqe, iw_acct, match)) { if (match->cancel_all) { raw_spin_unlock(&acct->lock); - goto retry_private; + goto retry; } break; } @@ -1321,6 +1315,26 @@ static void io_wqe_cancel_pending_work(struct io_wqe *wqe, } } +static void io_wqe_cancel_pending_work(struct io_wqe *wqe, + struct io_cb_cancel_data *match) +{ + int i; + struct io_wqe_acct *acct; + +retry: + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + acct = io_get_acct(wqe, i == 0, false); + if (io_acct_cancel_pending_work(wqe, acct, match)) { + if (match->cancel_all) + goto retry; + return; + } + } + + io_wqe_cancel_pending_work_fixed(wqe, match, false); + io_wqe_cancel_pending_work_fixed(wqe, match, true); +} + static void io_wqe_cancel_running_work(struct io_wqe *wqe, struct io_cb_cancel_data *match) {