From patchwork Fri Aug 19 15:27:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53648C32771 for ; Fri, 19 Aug 2022 15:28:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349022AbiHSP2R (ORCPT ); Fri, 19 Aug 2022 11:28:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349566AbiHSP2N (ORCPT ); Fri, 19 Aug 2022 11:28:13 -0400 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB972296 for ; Fri, 19 Aug 2022 08:28:10 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922889; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NEspg2Hen2izljK6C4Uf0o2lN1O6kLVZv/GIs0QRq74=; b=erbTCQv/naU1vuHzIcKw0EgYh/QcZhJIXUXd61h43TnznGdKIlTkX4mOq/kifS0fBpgC2v Uso5WSWmOve6yHI2lu9QAn7MZILJOqE5/pK8d3o/dGXf2SSzBWa38GJZL+uqfXwVqRaDRO CME1ZwZf+8S3NUhc43GWAWYEvzP9hZE= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 01/19] io_uring: change return value of create_io_worker() and io_wqe_create_worker() Date: Fri, 19 Aug 2022 23:27:20 +0800 Message-Id: <20220819152738.1111255-2-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Change return value of create_io_worker() and io_wqe_create_worker() so that it tells the detail error code. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index c6536d4b2da0..f631acbd50df 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -140,7 +140,7 @@ struct io_cb_cancel_data { bool cancel_all; }; -static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index); +static int create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index); static void io_wqe_dec_running(struct io_worker *worker); static bool io_acct_cancel_pending_work(struct io_wqe *wqe, struct io_wqe_acct *acct, @@ -289,7 +289,7 @@ static bool io_wqe_activate_free_worker(struct io_wqe *wqe, * We need a worker. If we find a free one, we're good. If not, and we're * below the max number of workers, create one. */ -static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) +static int io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) { /* * Most likely an attempt to queue unbounded work on an io_wq that @@ -301,7 +301,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) raw_spin_lock(&wqe->lock); if (acct->nr_workers >= acct->max_workers) { raw_spin_unlock(&wqe->lock); - return true; + return 0; } acct->nr_workers++; raw_spin_unlock(&wqe->lock); @@ -790,7 +790,7 @@ static void io_workqueue_create(struct work_struct *work) kfree(worker); } -static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index) +static int create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index) { struct io_wqe_acct *acct = &wqe->acct[index]; struct io_worker *worker; @@ -806,7 +806,7 @@ static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index) acct->nr_workers--; raw_spin_unlock(&wqe->lock); io_worker_ref_put(wq); - return false; + return -ENOMEM; } refcount_set(&worker->ref, 1); @@ -828,7 +828,7 @@ static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index) schedule_work(&worker->work); } - return true; + return 0; } /* @@ -933,7 +933,7 @@ static void io_wqe_enqueue(struct io_wqe *wqe, struct io_wq_work *work) !atomic_read(&acct->nr_running))) { bool did_create; - did_create = io_wqe_create_worker(wqe, acct); + did_create = !io_wqe_create_worker(wqe, acct); if (likely(did_create)) return; From patchwork Fri Aug 19 15:27:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 949D6C32772 for ; Fri, 19 Aug 2022 15:28:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349328AbiHSP2T (ORCPT ); Fri, 19 Aug 2022 11:28:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349649AbiHSP2S (ORCPT ); Fri, 19 Aug 2022 11:28:18 -0400 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A47215737 for ; Fri, 19 Aug 2022 08:28:17 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922895; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KahsA1qAdE1eUEiiRPuPwsPCrW+nKU4ePjzOFyhkWsU=; b=a7bgh/Zt3bmm4SjKncYc75nGnAqMowgGrcEAIBhcTi5aNGovrOmFnkNSSTIAXQdsP4RYcr Djdg9C0FmBeAJQqda/aIqkWu+7rYPSU+O01mNyNmYcHwPuvF4b3J/hNvjTTOEDnCui4EpA MF/fnYJXVlCxGrCQ4UA48xG/hFoMYG8= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 02/19] io_uring: add IORING_SETUP_URINGLET Date: Fri, 19 Aug 2022 23:27:21 +0800 Message-Id: <20220819152738.1111255-3-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add a new setup flag to turn on/off uringlet mode. Signed-off-by: Hao Xu --- include/uapi/linux/io_uring.h | 4 ++++ io_uring/io_uring.c | 4 +++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 1463cfecb56b..68507c23b079 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -152,6 +152,10 @@ enum { * Only one task is allowed to submit requests */ #define IORING_SETUP_SINGLE_ISSUER (1U << 12) +/* + * uringlet mode + */ +#define IORING_SETUP_URINGLET (1U << 13) enum io_uring_op { IORING_OP_NOP, diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ebfdb2212ec2..5e4f5b1684dd 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3226,6 +3226,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, struct file *file; int ret; + if (p->flags & IORING_SETUP_URINGLET) + return -EINVAL; if (!entries) return -EINVAL; if (entries > IORING_MAX_ENTRIES) { @@ -3400,7 +3402,7 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params) IORING_SETUP_R_DISABLED | IORING_SETUP_SUBMIT_ALL | IORING_SETUP_COOP_TASKRUN | IORING_SETUP_TASKRUN_FLAG | IORING_SETUP_SQE128 | IORING_SETUP_CQE32 | - IORING_SETUP_SINGLE_ISSUER)) + IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_URINGLET)) return -EINVAL; return io_uring_create(entries, &p, params); From patchwork Fri Aug 19 15:27:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7242C32771 for ; Fri, 19 Aug 2022 15:28:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349602AbiHSP23 (ORCPT ); Fri, 19 Aug 2022 11:28:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46388 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349752AbiHSP2Z (ORCPT ); Fri, 19 Aug 2022 11:28:25 -0400 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40B279FE6 for ; Fri, 19 Aug 2022 08:28:21 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WD7SoHYymjSTTeE3AJHxzP2Cwryq4jimi6cQjT+wM8Q=; b=KidFq+2U4P5iyOSXqi4Icv4Wj6t3RweY4GvFnyeE8yewJuRaDaNVS2eXeGCeU52k9ygE0h Tdbw0bFJHdC+oJaDOuNOUhYF6ByLYm/Nxbgepz5oPT8pyIglzNNTCcGdeiVIUeTkwV/P60 JG1I0vOU1gOTbur+MGQpMnSkEVQqICU= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 03/19] io_uring: make worker pool per ctx for uringlet mode Date: Fri, 19 Aug 2022 23:27:22 +0800 Message-Id: <20220819152738.1111255-4-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu For uringlet mode, make worker pool per ctx. This is much easier for implementation. We can make it better later if it's necessary. In uringlet mode, we need to find the specific ctx in a worker. Add a member private for this. We set wq->task to NULL for uringlet as a mark that this is a uringler io-wq. Signed-off-by: Hao Xu --- include/linux/io_uring_types.h | 1 + io_uring/io-wq.c | 11 ++++++++++- io_uring/io-wq.h | 4 ++++ io_uring/io_uring.c | 9 +++++++++ io_uring/tctx.c | 8 +++++++- 5 files changed, 31 insertions(+), 2 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 677a25d44d7f..c8093e733a35 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -356,6 +356,7 @@ struct io_ring_ctx { unsigned sq_thread_idle; /* protected by ->completion_lock */ unsigned evfd_last_cq_tail; + struct io_wq *let; }; enum { diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index f631acbd50df..aaa58cbacf60 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -127,6 +127,8 @@ struct io_wq { struct task_struct *task; + void *private; + struct io_wqe *wqes[]; }; @@ -392,6 +394,11 @@ static bool io_queue_worker_create(struct io_worker *worker, return false; } +static inline bool io_wq_is_uringlet(struct io_wq *wq) +{ + return wq->private; +} + static void io_wqe_dec_running(struct io_worker *worker) { struct io_wqe_acct *acct = io_wqe_get_acct(worker); @@ -1153,6 +1160,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data) wq->hash = data->hash; wq->free_work = data->free_work; wq->do_work = data->do_work; + wq->private = data->private; ret = -ENOMEM; for_each_node(node) { @@ -1188,7 +1196,8 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data) INIT_LIST_HEAD(&wqe->all_list); } - wq->task = get_task_struct(data->task); + if (data->task) + wq->task = get_task_struct(data->task); atomic_set(&wq->worker_refs, 1); init_completion(&wq->worker_done); return wq; diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index 31228426d192..b9f5ce4493e0 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -41,6 +41,7 @@ struct io_wq_data { struct task_struct *task; io_wq_work_fn *do_work; free_work_fn *free_work; + void *private; }; struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data); @@ -80,4 +81,7 @@ static inline bool io_wq_current_is_worker(void) return in_task() && (current->flags & PF_IO_WORKER) && current->worker_private; } + +extern struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx, + struct task_struct *task); #endif diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 5e4f5b1684dd..cb011a04653b 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3318,6 +3318,15 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, ret = io_sq_offload_create(ctx, p); if (ret) goto err; + + if (ctx->flags & IORING_SETUP_URINGLET) { + ctx->let = io_init_wq_offload(ctx, current); + if (IS_ERR(ctx->let)) { + ret = PTR_ERR(ctx->let); + goto err; + } + } + /* always set a rsrc node */ ret = io_rsrc_node_switch_start(ctx); if (ret) diff --git a/io_uring/tctx.c b/io_uring/tctx.c index 7f97d97fef0a..09c91cd7b5bf 100644 --- a/io_uring/tctx.c +++ b/io_uring/tctx.c @@ -12,7 +12,7 @@ #include "io_uring.h" #include "tctx.h" -static struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx, +struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx, struct task_struct *task) { struct io_wq_hash *hash; @@ -34,9 +34,15 @@ static struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx, mutex_unlock(&ctx->uring_lock); data.hash = hash; + /* for uringlet, wq->task is the iouring instance creator */ data.task = task; data.free_work = io_wq_free_work; data.do_work = io_wq_submit_work; + /* distinguish normal iowq and uringlet by wq->private for now */ + if (ctx->flags & IORING_SETUP_URINGLET) + data.private = ctx; + else + data.private = NULL; /* Do QD, or 4 * CPUS, whatever is smallest */ concurrency = min(ctx->sq_entries, 4 * num_online_cpus()); From patchwork Fri Aug 19 15:27:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6AADC3F6B0 for ; Fri, 19 Aug 2022 15:28:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349649AbiHSP2b (ORCPT ); Fri, 19 Aug 2022 11:28:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46516 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349763AbiHSP22 (ORCPT ); Fri, 19 Aug 2022 11:28:28 -0400 Received: from out1.migadu.com (out1.migadu.com [91.121.223.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5944110F for ; Fri, 19 Aug 2022 08:28:25 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922903; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EuuOlSXmNNGkOG1ruyJmE/aFr3PdiEC8n0WEJ/S84wY=; b=WrPXOhI8DWARoyhQGvspy2y2r1UaiBJsIws1Yp70x5Mar/8dgOY2/n2IYeBZDibN5+8xaj 9uJCKWc4xzG4EYHXzLfHHLKgC7F61t8fJFQdPHEAxKpKgeGT538BNT9QTL3agEmpAlBXDW kWpV+mswMLwyZ0DdS/ul6TrPDQniSgA= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 04/19] io-wq: split io_wqe_worker() to io_wqe_worker_normal() and io_wqe_worker_let() Date: Fri, 19 Aug 2022 23:27:23 +0800 Message-Id: <20220819152738.1111255-5-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu io_wqe_worker_normal() is the normal io worker, and io_wqe_worker_let() is the handler for uringlet mode. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 82 ++++++++++++++++++++++++++++++++++++++++----- io_uring/io-wq.h | 8 ++++- io_uring/io_uring.c | 8 +++-- io_uring/io_uring.h | 2 +- 4 files changed, 87 insertions(+), 13 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index aaa58cbacf60..b533db18d7c0 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -20,6 +20,7 @@ #include "io-wq.h" #include "slist.h" #include "io_uring.h" +#include "tctx.h" #define WORKER_IDLE_TIMEOUT (5 * HZ) @@ -617,19 +618,12 @@ static void io_worker_handle_work(struct io_worker *worker) } while (1); } -static int io_wqe_worker(void *data) +static void io_wqe_worker_normal(struct io_worker *worker) { - struct io_worker *worker = data; struct io_wqe_acct *acct = io_wqe_get_acct(worker); struct io_wqe *wqe = worker->wqe; struct io_wq *wq = wqe->wq; bool last_timeout = false; - char buf[TASK_COMM_LEN]; - - worker->flags |= (IO_WORKER_F_UP | IO_WORKER_F_RUNNING); - - snprintf(buf, sizeof(buf), "iou-wrk-%d", wq->task->pid); - set_task_comm(current, buf); while (!test_bit(IO_WQ_BIT_EXIT, &wq->state)) { long ret; @@ -664,6 +658,78 @@ static int io_wqe_worker(void *data) if (test_bit(IO_WQ_BIT_EXIT, &wq->state)) io_worker_handle_work(worker); +} + +#define IO_URINGLET_EMPTY_LIMIT 100000 +#define URINGLET_WORKER_IDLE_TIMEOUT 1 + +static void io_wqe_worker_let(struct io_worker *worker) +{ + struct io_wqe *wqe = worker->wqe; + struct io_wq *wq = wqe->wq; + + /* TODO this one breaks encapsulation */ + if (unlikely(io_uring_add_tctx_node(wq->private))) + goto out; + + while (!test_bit(IO_WQ_BIT_EXIT, &wq->state)) { + unsigned int empty_count = 0; + + __io_worker_busy(wqe, worker); + set_current_state(TASK_INTERRUPTIBLE); + + do { + enum io_uringlet_state submit_state; + + submit_state = wq->do_work(wq->private); + if (submit_state == IO_URINGLET_SCHEDULED) { + empty_count = 0; + break; + } else if (submit_state == IO_URINGLET_EMPTY) { + if (++empty_count > IO_URINGLET_EMPTY_LIMIT) + break; + } else { + empty_count = 0; + } + cond_resched(); + } while (1); + + raw_spin_lock(&wqe->lock); + __io_worker_idle(wqe, worker); + raw_spin_unlock(&wqe->lock); + schedule_timeout(URINGLET_WORKER_IDLE_TIMEOUT); + if (signal_pending(current)) { + struct ksignal ksig; + + if (!get_signal(&ksig)) + continue; + break; + } + } + + __set_current_state(TASK_RUNNING); +out: + wq->free_work(NULL); +} + +static int io_wqe_worker(void *data) +{ + struct io_worker *worker = data; + struct io_wqe *wqe = worker->wqe; + struct io_wq *wq = wqe->wq; + bool uringlet = io_wq_is_uringlet(wq); + char buf[TASK_COMM_LEN]; + + worker->flags |= (IO_WORKER_F_UP | IO_WORKER_F_RUNNING); + + snprintf(buf, sizeof(buf), uringlet ? "iou-let-%d" : "iou-wrk-%d", + wq->task->pid); + set_task_comm(current, buf); + + if (uringlet) + io_wqe_worker_let(worker); + else + io_wqe_worker_normal(worker); io_worker_exit(worker); return 0; diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index b9f5ce4493e0..b862b04e49ce 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -21,8 +21,14 @@ enum io_wq_cancel { IO_WQ_CANCEL_NOTFOUND, /* work not found */ }; +enum io_uringlet_state { + IO_URINGLET_INLINE, + IO_URINGLET_EMPTY, + IO_URINGLET_SCHEDULED, +}; + typedef struct io_wq_work *(free_work_fn)(struct io_wq_work *); -typedef void (io_wq_work_fn)(struct io_wq_work *); +typedef int (io_wq_work_fn)(struct io_wq_work *); struct io_wq_hash { refcount_t refs; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cb011a04653b..b57e9059a388 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1612,7 +1612,7 @@ struct io_wq_work *io_wq_free_work(struct io_wq_work *work) return req ? &req->work : NULL; } -void io_wq_submit_work(struct io_wq_work *work) +int io_wq_submit_work(struct io_wq_work *work) { struct io_kiocb *req = container_of(work, struct io_kiocb, work); const struct io_op_def *def = &io_op_defs[req->opcode]; @@ -1632,7 +1632,7 @@ void io_wq_submit_work(struct io_wq_work *work) if (work->flags & IO_WQ_WORK_CANCEL) { fail: io_req_task_queue_fail(req, err); - return; + return 0; } if (!io_assign_file(req, issue_flags)) { err = -EBADF; @@ -1666,7 +1666,7 @@ void io_wq_submit_work(struct io_wq_work *work) } if (io_arm_poll_handler(req, issue_flags) == IO_APOLL_OK) - return; + return 0; /* aborted or ready, in either case retry blocking */ needs_poll = false; issue_flags &= ~IO_URING_F_NONBLOCK; @@ -1675,6 +1675,8 @@ void io_wq_submit_work(struct io_wq_work *work) /* avoid locking problems by failing it from a clean context */ if (ret < 0) io_req_task_queue_fail(req, ret); + + return 0; } inline struct file *io_file_get_fixed(struct io_kiocb *req, int fd, diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 2f73f83af960..b20d2506a60f 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -69,7 +69,7 @@ void io_free_batch_list(struct io_ring_ctx *ctx, struct io_wq_work_node *node); int io_req_prep_async(struct io_kiocb *req); struct io_wq_work *io_wq_free_work(struct io_wq_work *work); -void io_wq_submit_work(struct io_wq_work *work); +int io_wq_submit_work(struct io_wq_work *work); void io_free_req(struct io_kiocb *req); void io_queue_next(struct io_kiocb *req); From patchwork Fri Aug 19 15:27:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB3CDC32773 for ; Fri, 19 Aug 2022 15:28:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349664AbiHSP2f (ORCPT ); Fri, 19 Aug 2022 11:28:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46766 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349743AbiHSP2d (ORCPT ); Fri, 19 Aug 2022 11:28:33 -0400 Received: from out1.migadu.com (out1.migadu.com [91.121.223.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 519519FD4 for ; Fri, 19 Aug 2022 08:28:31 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922909; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J4jWtznh3gADBZLpivIklxKj5BL1zAoiqN9NDNbiHuc=; b=nkDAfz4bKPPyJyz9VMoa1g/EzFqOQNCwly79NPqZhysSXnO97+f2p/zs7CuQ3fmPDONo7j b/9yHwlwt+4hMSadTebdElHXWdU+/NbAtpyFQif1zoyOnqvSOo0rhWIcCBbt5jhEC1qwl3 vEPsQSJO70QfpKwv9ojS/NgntHQTcAY= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 05/19] io_uring: add io_uringler_offload() for uringlet mode Date: Fri, 19 Aug 2022 23:27:24 +0800 Message-Id: <20220819152738.1111255-6-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu In uringlet mode, a io_uring_enter call shouldn't do the sqe submission work, but just offload it to io-workers. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 18 ++++++++++++++++++ io_uring/io-wq.h | 1 + io_uring/io_uring.c | 21 ++++++++++++++------- 3 files changed, 33 insertions(+), 7 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index b533db18d7c0..212ea16cbb5e 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -771,6 +771,24 @@ void io_wq_worker_sleeping(struct task_struct *tsk) io_wqe_dec_running(worker); } +int io_uringlet_offload(struct io_wq *wq) +{ + struct io_wqe *wqe = wq->wqes[numa_node_id()]; + struct io_wqe_acct *acct = io_get_acct(wqe, true); + bool waken; + + raw_spin_lock(&wqe->lock); + rcu_read_lock(); + waken = io_wqe_activate_free_worker(wqe, acct); + rcu_read_unlock(); + raw_spin_unlock(&wqe->lock); + + if (waken) + return 0; + + return io_wqe_create_worker(wqe, acct); +} + static void io_init_new_worker(struct io_wqe *wqe, struct io_worker *worker, struct task_struct *tsk) { diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index b862b04e49ce..66d2aeb17951 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -90,4 +90,5 @@ static inline bool io_wq_current_is_worker(void) extern struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx, struct task_struct *task); +extern int io_uringlet_offload(struct io_wq *wq); #endif diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index b57e9059a388..554041705e96 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3051,15 +3051,22 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, if (unlikely(ret)) goto out; - mutex_lock(&ctx->uring_lock); - ret = io_submit_sqes(ctx, to_submit); - if (ret != to_submit) { + if (!(ctx->flags & IORING_SETUP_URINGLET)) { + mutex_lock(&ctx->uring_lock); + ret = io_submit_sqes(ctx, to_submit); + if (ret != to_submit) { + mutex_unlock(&ctx->uring_lock); + goto out; + } + if ((flags & IORING_ENTER_GETEVENTS) && ctx->syscall_iopoll) + goto iopoll_locked; mutex_unlock(&ctx->uring_lock); - goto out; + } else { + ret = io_uringlet_offload(ctx->let); + if (ret) + goto out; + ret = to_submit; } - if ((flags & IORING_ENTER_GETEVENTS) && ctx->syscall_iopoll) - goto iopoll_locked; - mutex_unlock(&ctx->uring_lock); } if (flags & IORING_ENTER_GETEVENTS) { int ret2; From patchwork Fri Aug 19 15:27:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948872 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF5DFC32771 for ; Fri, 19 Aug 2022 15:28:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349573AbiHSP2o (ORCPT ); Fri, 19 Aug 2022 11:28:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348832AbiHSP2n (ORCPT ); Fri, 19 Aug 2022 11:28:43 -0400 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5C2E3E756 for ; Fri, 19 Aug 2022 08:28:41 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922920; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qc6R5KcTrXw6lUJn6MhZDB6ZDV0+9RmkY24ZBkoG9Ws=; b=SaO/NpW6iGeFYm26UWmjGNq7mOthbmkTrjVSgKAyxKbuvMe59b3jI7c2UCXlnEdj7iSzxe tL/xMpj+G2R1D0dFd0bmK0q3MocJezz/QR3EFrGRnWFihRPJZdWiy+evfxjQ2MGS6Ofu/p touFIG57BZuuPzIR/TBASSpFCKkMUiY= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 06/19] io-wq: change the io-worker scheduling logic Date: Fri, 19 Aug 2022 23:27:25 +0800 Message-Id: <20220819152738.1111255-7-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu We do io-worker creation when a io-worker gets sleeping and some condition is met. For uringlet mode, we need to do the scheduling too. A uringlet worker gets sleeping because of blocking in some place below io_uring layer in the kernel stack. So we should wake up or create a new uringlet worker in this situation. Meanwhile, setting up a flag to let the sqe submitter know it had been scheduled out. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 212ea16cbb5e..5f54af7579a4 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -404,14 +404,28 @@ static void io_wqe_dec_running(struct io_worker *worker) { struct io_wqe_acct *acct = io_wqe_get_acct(worker); struct io_wqe *wqe = worker->wqe; + struct io_wq *wq = wqe->wq; + bool zero_refs; if (!(worker->flags & IO_WORKER_F_UP)) return; - if (!atomic_dec_and_test(&acct->nr_running)) - return; - if (!io_acct_run_queue(acct)) - return; + zero_refs = atomic_dec_and_test(&acct->nr_running); + + if (io_wq_is_uringlet(wq)) { + bool activated; + + raw_spin_lock(&wqe->lock); + rcu_read_lock(); + activated = io_wqe_activate_free_worker(wqe, acct); + rcu_read_unlock(); + raw_spin_unlock(&wqe->lock); + if (activated) + return; + } else { + if (!zero_refs || !io_acct_run_queue(acct)) + return; + } atomic_inc(&acct->nr_running); atomic_inc(&wqe->wq->worker_refs); From patchwork Fri Aug 19 15:27:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35341C32771 for ; Fri, 19 Aug 2022 15:29:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349286AbiHSP3n (ORCPT ); Fri, 19 Aug 2022 11:29:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348832AbiHSP3m (ORCPT ); Fri, 19 Aug 2022 11:29:42 -0400 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D476C889F for ; Fri, 19 Aug 2022 08:29:41 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922980; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IGSA/9LIwmVlni7oicB/oQoyBnCtVKkHk456vzjz0Tk=; b=pZoi7C7KMgLrSno9WX2Wk1TlQ1KcSn5k0B+fH1pvWQgTsvgibCi2P6mgJR3UEgmEV+edVY RKIKO9zQnogsSLmChu0b8HmVBlFkYDaAQPTs3My0ta1C+b8YTsBaRBb6KlijjfVvE7DeFD Tw6xSf9CN23wTUMLtpjmc0AT7PPshwE= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 07/19] io-wq: move worker state flags to io-wq.h Date: Fri, 19 Aug 2022 23:27:26 +0800 Message-Id: <20220819152738.1111255-8-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Move worker state flags to io-wq.h so that we can levarage them later. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 7 ------- io_uring/io-wq.h | 8 ++++++++ 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 5f54af7579a4..55f1063f24c7 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -24,13 +24,6 @@ #define WORKER_IDLE_TIMEOUT (5 * HZ) -enum { - IO_WORKER_F_UP = 1, /* up and active */ - IO_WORKER_F_RUNNING = 2, /* account as running */ - IO_WORKER_F_FREE = 4, /* worker on free list */ - IO_WORKER_F_BOUND = 8, /* is doing bounded work */ -}; - enum { IO_WQ_BIT_EXIT = 0, /* wq exiting */ }; diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index 66d2aeb17951..504a8a8e3fd8 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -27,6 +27,14 @@ enum io_uringlet_state { IO_URINGLET_SCHEDULED, }; +enum { + IO_WORKER_F_UP = 1, /* up and active */ + IO_WORKER_F_RUNNING = 2, /* account as running */ + IO_WORKER_F_FREE = 4, /* worker on free list */ + IO_WORKER_F_BOUND = 8, /* is doing bounded work */ + IO_WORKER_F_SCHEDULED = 16, /* worker had been scheduled out before */ +}; + typedef struct io_wq_work *(free_work_fn)(struct io_wq_work *); typedef int (io_wq_work_fn)(struct io_wq_work *); From patchwork Fri Aug 19 15:27:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948874 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D1EAC32771 for ; Fri, 19 Aug 2022 15:29:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348832AbiHSP3r (ORCPT ); Fri, 19 Aug 2022 11:29:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349600AbiHSP3q (ORCPT ); Fri, 19 Aug 2022 11:29:46 -0400 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 944FBE68E5 for ; Fri, 19 Aug 2022 08:29:45 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ODhPmDjom+4oOLEwqrIFYRuyeK5mh/1I2zNwWSRUWlo=; b=jv4n//T+TBPum3/iOGn4amOoPSYoUC/eTxsJQAPtM/ZXxI/VrSE5BCzdyJ/NXssZxJpe4j ePnPHLq9kwKOs5lIL0i30sitWWA3dtbjP8cz1GOaJQrz7KI9OnP4R+054oEYnuS5+e+rK7 Bw4qqyTPvzO8KD//PsBFAtSASln9VS8= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 08/19] io-wq: add IO_WORKER_F_SUBMIT and its friends Date: Fri, 19 Aug 2022 23:27:27 +0800 Message-Id: <20220819152738.1111255-9-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add IO_WORKER_F_SUBMIT to indicate that a uringlet worker is submitting sqes and thus we should do some scheduling when it blocks. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 20 ++++++++++++++++++++ io_uring/io-wq.h | 1 + 2 files changed, 21 insertions(+) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 55f1063f24c7..7e58bb5857ee 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -393,6 +393,21 @@ static inline bool io_wq_is_uringlet(struct io_wq *wq) return wq->private; } +static inline void io_worker_set_submit(struct io_worker *worker) +{ + worker->flags |= IO_WORKER_F_SUBMIT; +} + +static inline void io_worker_clean_submit(struct io_worker *worker) +{ + worker->flags &= ~IO_WORKER_F_SUBMIT; +} + +static inline bool io_worker_test_submit(struct io_worker *worker) +{ + return worker->flags & IO_WORKER_F_SUBMIT; +} + static void io_wqe_dec_running(struct io_worker *worker) { struct io_wqe_acct *acct = io_wqe_get_acct(worker); @@ -408,6 +423,9 @@ static void io_wqe_dec_running(struct io_worker *worker) if (io_wq_is_uringlet(wq)) { bool activated; + if (!io_worker_test_submit(worker)) + return; + raw_spin_lock(&wqe->lock); rcu_read_lock(); activated = io_wqe_activate_free_worker(wqe, acct); @@ -688,7 +706,9 @@ static void io_wqe_worker_let(struct io_worker *worker) do { enum io_uringlet_state submit_state; + io_worker_set_submit(worker); submit_state = wq->do_work(wq->private); + io_worker_clean_submit(worker); if (submit_state == IO_URINGLET_SCHEDULED) { empty_count = 0; break; diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index 504a8a8e3fd8..1485e9009784 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -33,6 +33,7 @@ enum { IO_WORKER_F_FREE = 4, /* worker on free list */ IO_WORKER_F_BOUND = 8, /* is doing bounded work */ IO_WORKER_F_SCHEDULED = 16, /* worker had been scheduled out before */ + IO_WORKER_F_SUBMIT = 32, /* uringlet worker is submitting sqes */ }; typedef struct io_wq_work *(free_work_fn)(struct io_wq_work *); From patchwork Fri Aug 19 15:27:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DF40C32771 for ; Fri, 19 Aug 2022 15:29:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348966AbiHSP3z (ORCPT ); Fri, 19 Aug 2022 11:29:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49290 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349079AbiHSP3w (ORCPT ); Fri, 19 Aug 2022 11:29:52 -0400 Received: from out1.migadu.com (out1.migadu.com [91.121.223.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C98DAE68C2 for ; Fri, 19 Aug 2022 08:29:49 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922987; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Dn1SuKfvdTW1N/o/HS4A6lbLDjo5jUVIvfIV9+6/E5U=; b=oisBl32dUVt/PQBLOx/C9wsskJ8NNn6BtP3FpmDOfi9CJB3cPkmDkdntiYu8ZhMOcDqGwm nsvogO1w43ZbKpt9bIK37kSjW47eo/1BMScqLkSNsXpkU8ti2Tysm/PRnW2kZINc56Z1Bu kMlDZ5QGjrthC2rWYQqQ/LTSqAiYIK0= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 09/19] io-wq: add IO_WORKER_F_SCHEDULED and its friends Date: Fri, 19 Aug 2022 23:27:28 +0800 Message-Id: <20220819152738.1111255-10-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Signed-off-by: Hao Xu --- io_uring/io-wq.c | 29 ++--------------------------- io_uring/io-wq.h | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 27 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 7e58bb5857ee..fe4faff79cf8 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -32,33 +32,6 @@ enum { IO_ACCT_STALLED_BIT = 0, /* stalled on hash */ }; -/* - * One for each thread in a wqe pool - */ -struct io_worker { - refcount_t ref; - unsigned flags; - struct hlist_nulls_node nulls_node; - struct list_head all_list; - struct task_struct *task; - struct io_wqe *wqe; - - struct io_wq_work *cur_work; - struct io_wq_work *next_work; - raw_spinlock_t lock; - - struct completion ref_done; - - unsigned long create_state; - struct callback_head create_work; - int create_index; - - union { - struct rcu_head rcu; - struct work_struct work; - }; -}; - #if BITS_PER_LONG == 64 #define IO_WQ_HASH_ORDER 6 #else @@ -426,6 +399,7 @@ static void io_wqe_dec_running(struct io_worker *worker) if (!io_worker_test_submit(worker)) return; + io_worker_set_scheduled(worker); raw_spin_lock(&wqe->lock); rcu_read_lock(); activated = io_wqe_activate_free_worker(wqe, acct); @@ -706,6 +680,7 @@ static void io_wqe_worker_let(struct io_worker *worker) do { enum io_uringlet_state submit_state; + io_worker_clean_scheduled(worker); io_worker_set_submit(worker); submit_state = wq->do_work(wq->private); io_worker_clean_submit(worker); diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index 1485e9009784..81146dba2ae6 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -3,9 +3,37 @@ #include #include +#include struct io_wq; +/* + * One for each thread in a wqe pool + */ +struct io_worker { + refcount_t ref; + unsigned flags; + struct hlist_nulls_node nulls_node; + struct list_head all_list; + struct task_struct *task; + struct io_wqe *wqe; + + struct io_wq_work *cur_work; + struct io_wq_work *next_work; + raw_spinlock_t lock; + + struct completion ref_done; + + unsigned long create_state; + struct callback_head create_work; + int create_index; + + union { + struct rcu_head rcu; + struct work_struct work; + }; +}; + enum { IO_WQ_WORK_CANCEL = 1, IO_WQ_WORK_HASHED = 2, @@ -97,6 +125,21 @@ static inline bool io_wq_current_is_worker(void) current->worker_private; } +static inline void io_worker_set_scheduled(struct io_worker *worker) +{ + worker->flags |= IO_WORKER_F_SCHEDULED; +} + +static inline void io_worker_clean_scheduled(struct io_worker *worker) +{ + worker->flags &= ~IO_WORKER_F_SCHEDULED; +} + +static inline bool io_worker_test_scheduled(struct io_worker *worker) +{ + return worker->flags & IO_WORKER_F_SCHEDULED; +} + extern struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx, struct task_struct *task); extern int io_uringlet_offload(struct io_wq *wq); From patchwork Fri Aug 19 15:27:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53368C32772 for ; Fri, 19 Aug 2022 15:29:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349079AbiHSP3z (ORCPT ); Fri, 19 Aug 2022 11:29:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349600AbiHSP3y (ORCPT ); Fri, 19 Aug 2022 11:29:54 -0400 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 446BCE68EB for ; Fri, 19 Aug 2022 08:29:53 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922991; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=17APm0cBylBacEAcxH6uZOaQSgVlLXSxL+VAr75MkP4=; b=Zvid4+WBBkzBlnKJPVqj5XIwqlbhDzGQmrsJmLuKX8YGvi5xk9rMilCElOftKG8QOyZeWI ChZo8KYxy04BbOc8Umhx7cAHPy6MOTsgX9pTUopNmNVKkSgPPCx7yMblCJQ76jN/TzUedf v1f/NvXrmxp+KhYax9dLbGBbwMFSbMM= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 10/19] io_uring: add io_submit_sqes_let() Date: Fri, 19 Aug 2022 23:27:29 +0800 Message-Id: <20220819152738.1111255-11-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add io_submit_sqes_let() for submitting sqes in uringlet mode, and update logic in schedule time and the io-wq init time. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 1 + io_uring/io_uring.c | 55 +++++++++++++++++++++++++++++++++++++++++++++ io_uring/io_uring.h | 2 ++ io_uring/tctx.c | 8 ++++--- 4 files changed, 63 insertions(+), 3 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index fe4faff79cf8..00a1cdefb787 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -399,6 +399,7 @@ static void io_wqe_dec_running(struct io_worker *worker) if (!io_worker_test_submit(worker)) return; + io_uringlet_end(wq->private); io_worker_set_scheduled(worker); raw_spin_lock(&wqe->lock); rcu_read_lock(); diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 554041705e96..a5fb6fa02ded 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2054,6 +2054,12 @@ static void io_commit_sqring(struct io_ring_ctx *ctx) smp_store_release(&rings->sq.head, ctx->cached_sq_head); } +void io_uringlet_end(struct io_ring_ctx *ctx) +{ + io_submit_state_end(ctx); + io_commit_sqring(ctx); +} + /* * Fetch an sqe, if one is available. Note this returns a pointer to memory * that is mapped by userspace. This means that care needs to be taken to @@ -2141,6 +2147,55 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) io_commit_sqring(ctx); return ret; } +int io_submit_sqes_let(struct io_wq_work *work) +{ + struct io_ring_ctx *ctx = (struct io_ring_ctx *)work; + unsigned int entries; + bool scheduled = false; + void *worker = current->worker_private; + + entries = io_sqring_entries(ctx); + if (!entries) + return IO_URINGLET_EMPTY; + + io_get_task_refs(entries); + io_submit_state_start(&ctx->submit_state, entries); + do { + const struct io_uring_sqe *sqe; + struct io_kiocb *req; + + if (unlikely(!io_alloc_req_refill(ctx))) + break; + req = io_alloc_req(ctx); + sqe = io_get_sqe(ctx); + if (unlikely(!sqe)) { + io_req_add_to_cache(req, ctx); + break; + } + + if (unlikely(io_submit_sqe(ctx, req, sqe))) + break; + /* TODO this one breaks encapsulation */ + scheduled = io_worker_test_scheduled(worker); + if (unlikely(scheduled)) { + entries--; + break; + } + } while (--entries); + + /* TODO do this at the schedule time too */ + if (unlikely(entries)) + current->io_uring->cached_refs += entries; + + /* Commit SQ ring head once we've consumed and submitted all SQEs */ + + if (scheduled) + return IO_URINGLET_SCHEDULED; + + io_uringlet_end(ctx); + return IO_URINGLET_INLINE; +} + struct io_wait_queue { struct wait_queue_entry wq; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index b20d2506a60f..b95d92619607 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -64,9 +64,11 @@ int io_uring_alloc_task_context(struct task_struct *task, int io_poll_issue(struct io_kiocb *req, bool *locked); int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr); +int io_submit_sqes_let(struct io_wq_work *work); int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin); void io_free_batch_list(struct io_ring_ctx *ctx, struct io_wq_work_node *node); int io_req_prep_async(struct io_kiocb *req); +void io_uringlet_end(struct io_ring_ctx *ctx); struct io_wq_work *io_wq_free_work(struct io_wq_work *work); int io_wq_submit_work(struct io_wq_work *work); diff --git a/io_uring/tctx.c b/io_uring/tctx.c index 09c91cd7b5bf..0c15fb8b9a2e 100644 --- a/io_uring/tctx.c +++ b/io_uring/tctx.c @@ -37,12 +37,14 @@ struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx, /* for uringlet, wq->task is the iouring instance creator */ data.task = task; data.free_work = io_wq_free_work; - data.do_work = io_wq_submit_work; /* distinguish normal iowq and uringlet by wq->private for now */ - if (ctx->flags & IORING_SETUP_URINGLET) + if (ctx->flags & IORING_SETUP_URINGLET) { data.private = ctx; - else + data.do_work = io_submit_sqes_let; + } else { data.private = NULL; + data.do_work = io_wq_submit_work; + } /* Do QD, or 4 * CPUS, whatever is smallest */ concurrency = min(ctx->sq_entries, 4 * num_online_cpus()); From patchwork Fri Aug 19 15:27:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CBD1C32771 for ; Fri, 19 Aug 2022 15:29:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349552AbiHSP35 (ORCPT ); Fri, 19 Aug 2022 11:29:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349346AbiHSP34 (ORCPT ); Fri, 19 Aug 2022 11:29:56 -0400 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E4487101C4 for ; Fri, 19 Aug 2022 08:29:55 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922994; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eKSA674mYIi3+xSnO7yHzmLX7iobcHsHIt3i/73o/ag=; b=eaJpmRfhG3B1evVrE9Z/WdR9RlSEp/9JZ+kubzqH/4a9ChujrpqZEHoq/bqSop5rhO+0hE PV4AtAJckUuqiqQe7c9DGm8Xr02ItEyW7HVjVxgxnBI7kzxWSE1fJbIyP7wFvFfveq9CjR sAMq08e5Y9lExrJMxpK9vtUU6mYT13I= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 11/19] io_uring: don't allocate io-wq for a worker in uringlet mode Date: Fri, 19 Aug 2022 23:27:30 +0800 Message-Id: <20220819152738.1111255-12-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu A uringlet worker doesn't need any io-wq pool. Signed-off-by: Hao Xu --- io_uring/tctx.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/io_uring/tctx.c b/io_uring/tctx.c index 0c15fb8b9a2e..b04d361bcf34 100644 --- a/io_uring/tctx.c +++ b/io_uring/tctx.c @@ -81,12 +81,17 @@ __cold int io_uring_alloc_task_context(struct task_struct *task, return ret; } - tctx->io_wq = io_init_wq_offload(ctx, task); - if (IS_ERR(tctx->io_wq)) { - ret = PTR_ERR(tctx->io_wq); - percpu_counter_destroy(&tctx->inflight); - kfree(tctx); - return ret; + /* + * don't allocate io-wq in uringlet mode + */ + if (!(ctx->flags & IORING_SETUP_URINGLET)) { + tctx->io_wq = io_init_wq_offload(ctx, task); + if (IS_ERR(tctx->io_wq)) { + ret = PTR_ERR(tctx->io_wq); + percpu_counter_destroy(&tctx->inflight); + kfree(tctx); + return ret; + } } xa_init(&tctx->xa); From patchwork Fri Aug 19 15:27:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 401D0C32771 for ; Fri, 19 Aug 2022 15:30:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349457AbiHSPaB (ORCPT ); Fri, 19 Aug 2022 11:30:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349346AbiHSPaB (ORCPT ); Fri, 19 Aug 2022 11:30:01 -0400 Received: from out1.migadu.com (out1.migadu.com [91.121.223.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E490E190F for ; Fri, 19 Aug 2022 08:30:00 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660922998; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+YsntrCxlmtQqRERzu/3Vp4m/8M9+TB2udMIBj6/Xpk=; b=R2VLhr8kTQoRMeW3knzwqBu5KNW+rO/K2pzok6EeVvjQMtNyzkpFmzF7lzw+9wJjCe6OEb K9et+WVp7mbeJN6LnCjxGvCD5Fj8nNEmrMfjvenJvANGqnlRciKCtHBRK8wXBH+yWPCq2x ZxezT4kXDsdB2MMbjqgdhMg7gGbZMgI= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 12/19] io_uring: add uringlet worker cancellation function Date: Fri, 19 Aug 2022 23:27:31 +0800 Message-Id: <20220819152738.1111255-13-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu uringlet worker submits sqes, so we need to do some cancellation work before it exits. Signed-off-by: Hao Xu --- io_uring/io_uring.c | 6 ++++++ io_uring/io_uring.h | 1 + io_uring/tctx.c | 2 ++ 3 files changed, 9 insertions(+) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a5fb6fa02ded..67d02dc16ea5 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2922,6 +2922,12 @@ void __io_uring_cancel(bool cancel_all) io_uring_cancel_generic(cancel_all, NULL); } +struct io_wq_work *io_uringlet_cancel(struct io_wq_work *work) +{ + __io_uring_cancel(true); + return NULL; +} + static void *io_uring_validate_mmap_request(struct file *file, loff_t pgoff, size_t sz) { diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index b95d92619607..011d0beb33bf 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -71,6 +71,7 @@ int io_req_prep_async(struct io_kiocb *req); void io_uringlet_end(struct io_ring_ctx *ctx); struct io_wq_work *io_wq_free_work(struct io_wq_work *work); +struct io_wq_work *io_uringlet_cancel(struct io_wq_work *work); int io_wq_submit_work(struct io_wq_work *work); void io_free_req(struct io_kiocb *req); diff --git a/io_uring/tctx.c b/io_uring/tctx.c index b04d361bcf34..e10b20725066 100644 --- a/io_uring/tctx.c +++ b/io_uring/tctx.c @@ -41,9 +41,11 @@ struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx, if (ctx->flags & IORING_SETUP_URINGLET) { data.private = ctx; data.do_work = io_submit_sqes_let; + data.free_work = io_uringlet_cancel; } else { data.private = NULL; data.do_work = io_wq_submit_work; + data.free_work = io_wq_free_work; } /* Do QD, or 4 * CPUS, whatever is smallest */ From patchwork Fri Aug 19 15:27:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4738C32771 for ; Fri, 19 Aug 2022 15:30:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349539AbiHSPaG (ORCPT ); Fri, 19 Aug 2022 11:30:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349346AbiHSPaF (ORCPT ); Fri, 19 Aug 2022 11:30:05 -0400 Received: from out1.migadu.com (out1.migadu.com [91.121.223.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 018BFE7242 for ; Fri, 19 Aug 2022 08:30:03 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660923002; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TwWnIxxzn5kzktussrHgbIB7YahPQii5RZvqgpXAMjU=; b=SCMs7unDIJz4PS8Ca5xPNV7udynbeu+Gsgc/HL9EQnHL4+bYRer7q9f1Ngw3aSIfDjMl3X FSxD4iqbQ1DXmVjUMe+g78EeCezTOLMzr0Gyz82FIJfBY/4DL+kmReZlOBOEaXlbpJ2Ddh ckNSGODznb1JgIs9dH8KmjZBLQQo/Cw= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 13/19] io-wq: add wq->owner for uringlet mode Date: Fri, 19 Aug 2022 23:27:32 +0800 Message-Id: <20220819152738.1111255-14-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu In uringlet mode, we allow exact one worker to submit sqes at the same time. nr_running is not a good choice to aim that. Add an member wq->owner and its lock to achieve that, this avoids race condition between workers. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 00a1cdefb787..9fcaeea7a478 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -96,6 +96,9 @@ struct io_wq { void *private; + raw_spinlock_t lock; + struct io_worker *owner; + struct io_wqe *wqes[]; }; @@ -381,6 +384,8 @@ static inline bool io_worker_test_submit(struct io_worker *worker) return worker->flags & IO_WORKER_F_SUBMIT; } +#define IO_WQ_OWNER_TRANSMIT ((struct io_worker *)-1) + static void io_wqe_dec_running(struct io_worker *worker) { struct io_wqe_acct *acct = io_wqe_get_acct(worker); @@ -401,6 +406,10 @@ static void io_wqe_dec_running(struct io_worker *worker) io_uringlet_end(wq->private); io_worker_set_scheduled(worker); + raw_spin_lock(&wq->lock); + wq->owner = IO_WQ_OWNER_TRANSMIT; + raw_spin_unlock(&wq->lock); + raw_spin_lock(&wqe->lock); rcu_read_lock(); activated = io_wqe_activate_free_worker(wqe, acct); @@ -674,6 +683,17 @@ static void io_wqe_worker_let(struct io_worker *worker) while (!test_bit(IO_WQ_BIT_EXIT, &wq->state)) { unsigned int empty_count = 0; + struct io_worker *owner; + + raw_spin_lock(&wq->lock); + owner = wq->owner; + if (owner && owner != IO_WQ_OWNER_TRANSMIT && owner != worker) { + raw_spin_unlock(&wq->lock); + set_current_state(TASK_INTERRUPTIBLE); + goto sleep; + } + wq->owner = worker; + raw_spin_unlock(&wq->lock); __io_worker_busy(wqe, worker); set_current_state(TASK_INTERRUPTIBLE); @@ -697,6 +717,7 @@ static void io_wqe_worker_let(struct io_worker *worker) cond_resched(); } while (1); +sleep: raw_spin_lock(&wqe->lock); __io_worker_idle(wqe, worker); raw_spin_unlock(&wqe->lock); @@ -780,6 +801,14 @@ int io_uringlet_offload(struct io_wq *wq) struct io_wqe_acct *acct = io_get_acct(wqe, true); bool waken; + raw_spin_lock(&wq->lock); + if (wq->owner) { + raw_spin_unlock(&wq->lock); + return 0; + } + wq->owner = IO_WQ_OWNER_TRANSMIT; + raw_spin_unlock(&wq->lock); + raw_spin_lock(&wqe->lock); rcu_read_lock(); waken = io_wqe_activate_free_worker(wqe, acct); @@ -1248,6 +1277,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data) wq->free_work = data->free_work; wq->do_work = data->do_work; wq->private = data->private; + raw_spin_lock_init(&wq->lock); ret = -ENOMEM; for_each_node(node) { From patchwork Fri Aug 19 15:27:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948880 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35446C32771 for ; Fri, 19 Aug 2022 15:30:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349346AbiHSPaK (ORCPT ); Fri, 19 Aug 2022 11:30:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349193AbiHSPaJ (ORCPT ); Fri, 19 Aug 2022 11:30:09 -0400 Received: from out1.migadu.com (out1.migadu.com [91.121.223.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9DF91A1BF for ; Fri, 19 Aug 2022 08:30:08 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660923007; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tnOpeS2ajJ798g1WuvV3vR2mYcvGc8pJKjoqStH2WY0=; b=oNLfQZptiqvgxXNCEHgQxBtAXxHOEBgYrynISCWFW4/J4oJlyW1mmghGA2V8tfJ2TBQqhs rsdGGMWIYj/ss0DKPXi6tEHPGJwB2sumb788oXLMhBtzcMMHd5QKk2xS3CXgAolr/Dbwgf 5S2uRCugwHgPKTZe/t55oZxoJgGVifE= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 14/19] io_uring: modify issue_flags for uringlet mode Date: Fri, 19 Aug 2022 23:27:33 +0800 Message-Id: <20220819152738.1111255-15-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu We don't need IO_URING_F_NONBLOCK in uringlet mode. Signed-off-by: Hao Xu --- io_uring/io_uring.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 67d02dc16ea5..0c14b90b8b47 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1751,8 +1751,12 @@ static inline void io_queue_sqe(struct io_kiocb *req) __must_hold(&req->ctx->uring_lock) { int ret; + unsigned int issue_flags = IO_URING_F_COMPLETE_DEFER; - ret = io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_COMPLETE_DEFER); + if (!(req->ctx->flags & IORING_SETUP_URINGLET)) + issue_flags |= IO_URING_F_NONBLOCK; + + ret = io_issue_sqe(req, issue_flags); /* * We async punt it if the file wasn't marked NOWAIT, or if the file From patchwork Fri Aug 19 15:27:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A566C32771 for ; Fri, 19 Aug 2022 15:30:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349599AbiHSPaO (ORCPT ); Fri, 19 Aug 2022 11:30:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349507AbiHSPaN (ORCPT ); Fri, 19 Aug 2022 11:30:13 -0400 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71702E725B for ; Fri, 19 Aug 2022 08:30:12 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660923010; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lfVJrGZWAV/CxAj15pj4d9hqcvYN2cySKDv1KGRNM08=; b=htF3glim5whu2JDXAWgyvooctoBS8y7YJkmTV4m4JVqD+qM94fSMKvwuua8YPgpiCc1y8Z uVo5olDsyn5S5G2uTUPzoj7aDERj89NbkBxlELzTQBKPX+xZsTE1S34771tFLC7Kb3b7My peayBE07ucLFqI6PkcVAVaT4seAXYXQ= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 15/19] io_uring: don't use inline completion cache if scheduled Date: Fri, 19 Aug 2022 23:27:34 +0800 Message-Id: <20220819152738.1111255-16-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu In uringlet mode, if a worker has been scheduled out during sqe submission, we cannot use inline completion for that sqe. Signed-off-by: Hao Xu --- io_uring/io_uring.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0c14b90b8b47..a109dcb48702 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1582,7 +1582,14 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) revert_creds(creds); if (ret == IOU_OK) { - if (issue_flags & IO_URING_F_COMPLETE_DEFER) + bool uringlet = req->ctx->flags & IORING_SETUP_URINGLET; + bool scheduled = false; + + if (uringlet) + scheduled = + io_worker_test_scheduled(current->worker_private); + + if ((issue_flags & IO_URING_F_COMPLETE_DEFER) && !scheduled) io_req_complete_defer(req); else io_req_complete_post(req); From patchwork Fri Aug 19 15:27:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCC43C32771 for ; Fri, 19 Aug 2022 15:30:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349507AbiHSPaR (ORCPT ); Fri, 19 Aug 2022 11:30:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349577AbiHSPaQ (ORCPT ); Fri, 19 Aug 2022 11:30:16 -0400 Received: from out1.migadu.com (out1.migadu.com [91.121.223.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77321FF8EF for ; Fri, 19 Aug 2022 08:30:15 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660923014; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k6K5sCnfZDewEDUqgPTjWysfUpc7lfJuLBqyMQeq0xI=; b=mTEhrkVYNfzHafM2P6xMwZYXzSMq+dosZb8AoZptt7L684ovYLVQxwpNFDr8DgXa/xnWlw j6kEi7WuM9MwFgjX9HOlm8FWsAEE/llFm5RVnpiwFLD0l+KCZnSk8+jImxnGBxFjfVJk2L kRb7z7z3qXvFiDLDmnSgNViD9nYygT4= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 16/19] io_uring: release ctx->let when a ring exits Date: Fri, 19 Aug 2022 23:27:35 +0800 Message-Id: <20220819152738.1111255-17-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Release the uringlet worker pool when a ring exits, and reclaim the resource. Signed-off-by: Hao Xu --- io_uring/io_uring.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a109dcb48702..bbe8948f4771 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2699,6 +2699,11 @@ static __cold void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx) unsigned long index; struct creds *creds; + if (ctx->flags & IORING_SETUP_URINGLET) { + io_wq_exit_start(ctx->let); + io_wq_put_and_exit(ctx->let); + } + mutex_lock(&ctx->uring_lock); percpu_ref_kill(&ctx->refs); if (ctx->rings) From patchwork Fri Aug 19 15:27:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6180C32771 for ; Fri, 19 Aug 2022 15:30:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349577AbiHSPaV (ORCPT ); Fri, 19 Aug 2022 11:30:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349501AbiHSPaU (ORCPT ); Fri, 19 Aug 2022 11:30:20 -0400 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9773100960 for ; Fri, 19 Aug 2022 08:30:18 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660923017; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BaOSqMP3kdrzOOmzv/EVzRkvK1ACxkPU8HvJac4cKRs=; b=KS0m+wFy1HFNHLpuwsKKezC8pkR4Kk02akrucJdscxWwLtaMbVJhdeEdCVBFpJnUlULJiN 4MFyK/faxpXQxfn4H96DMq0YCi82Lgl+Hw6mRAbyITge1hnl5eNPis10TvtfLe4ZNnrriG qrPfMBXbb+xxdCifrI/Cpwoj7k0zoxk= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 17/19] io_uring: disable task plug for now Date: Fri, 19 Aug 2022 23:27:36 +0800 Message-Id: <20220819152738.1111255-18-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu This is a temporary commit, the task plug causes hung and the reason is unclear for now. So disable it in uringlet mode for now. Signed-off-by: Hao Xu --- io_uring/io_uring.c | 1 + 1 file changed, 1 insertion(+) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index bbe8948f4771..a48e34f63845 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2171,6 +2171,7 @@ int io_submit_sqes_let(struct io_wq_work *work) io_get_task_refs(entries); io_submit_state_start(&ctx->submit_state, entries); + ctx->submit_state->need_plug = false; do { const struct io_uring_sqe *sqe; struct io_kiocb *req; From patchwork Fri Aug 19 15:27:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948884 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AB37C32771 for ; Fri, 19 Aug 2022 15:30:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349367AbiHSPan (ORCPT ); Fri, 19 Aug 2022 11:30:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348983AbiHSPaY (ORCPT ); Fri, 19 Aug 2022 11:30:24 -0400 Received: from out1.migadu.com (out1.migadu.com [91.121.223.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 825B9AE74 for ; Fri, 19 Aug 2022 08:30:23 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660923022; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mg1OAUvD7o1eNEDbi7UJD/CkSdfbPxvgSM86lxHSws4=; b=Iejoh7L8c3jU7K4Pbvefsfs6fSmWUYNB571Y9TvyTHDTQef+iu4jSqPMaYCJ1CHcnfpTpR 2qCPpJ4dnLXgofJaBBoQjJ76pjT/GeE7z2/iYilwVLUcdMpdfp0EoShiJLZd5ZgZm5aVr9 AkXWQcxuwMINjl61wdtkcN+Q7Lt7ULQ= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 18/19] io-wq: only do io_uringlet_end() at the first schedule time Date: Fri, 19 Aug 2022 23:27:37 +0800 Message-Id: <20220819152738.1111255-19-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu A request may block multiple times during its life cycle. We should only do io_uringlet_end() at the first time since this function may modify ctx->submit_state info and for the non-first time, the task already lost the control of submitting sqes. Allowing it to do so will damage the submission state. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 14 +++++++++++--- io_uring/io_uring.c | 2 +- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 9fcaeea7a478..f845b7daced8 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -400,16 +400,24 @@ static void io_wqe_dec_running(struct io_worker *worker) if (io_wq_is_uringlet(wq)) { bool activated; + bool first_block; if (!io_worker_test_submit(worker)) return; - io_uringlet_end(wq->private); - io_worker_set_scheduled(worker); raw_spin_lock(&wq->lock); - wq->owner = IO_WQ_OWNER_TRANSMIT; + first_block = (wq->owner == worker ? true : false); raw_spin_unlock(&wq->lock); + io_worker_set_scheduled(worker); + + if (first_block) { + io_uringlet_end(wq->private); + raw_spin_lock(&wq->lock); + wq->owner = IO_WQ_OWNER_TRANSMIT; + raw_spin_unlock(&wq->lock); + } + raw_spin_lock(&wqe->lock); rcu_read_lock(); activated = io_wqe_activate_free_worker(wqe, acct); diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a48e34f63845..7ebc83b3a33f 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2171,7 +2171,7 @@ int io_submit_sqes_let(struct io_wq_work *work) io_get_task_refs(entries); io_submit_state_start(&ctx->submit_state, entries); - ctx->submit_state->need_plug = false; + ctx->submit_state.need_plug = false; do { const struct io_uring_sqe *sqe; struct io_kiocb *req; From patchwork Fri Aug 19 15:27:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12948885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02E71C32773 for ; Fri, 19 Aug 2022 15:30:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348983AbiHSPan (ORCPT ); Fri, 19 Aug 2022 11:30:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349752AbiHSPaa (ORCPT ); Fri, 19 Aug 2022 11:30:30 -0400 Received: from out1.migadu.com (out1.migadu.com [91.121.223.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEF0610097A for ; Fri, 19 Aug 2022 08:30:28 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1660923027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=564pKdunX04HqFVimIx2VcnHHMvL0ZSIADakfvP38+s=; b=GHrLN/s0I/+FMnAb2CAuCkp/Cr+xTV35REnonJVn4sOBMnihguZ+Kk+U2O/eRVXJZwD+Ot 1QIyd0s3oWRYc/ZcJ8e8wiNYfQ7gi8U5Y4jDttrDroy3jwlgRl9UoCUItxKBDSRfu0tWyA +i9MN7cccNlTd6T95un4k0xdDRh4xtU= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Ingo Molnar , Wanpeng Li Subject: [PATCH 19/19] io_uring: wire up uringlet Date: Fri, 19 Aug 2022 23:27:38 +0800 Message-Id: <20220819152738.1111255-20-hao.xu@linux.dev> In-Reply-To: <20220819152738.1111255-1-hao.xu@linux.dev> References: <20220819152738.1111255-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Enable the uringlet mode. Signed-off-by: Hao Xu --- io_uring/io_uring.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 7ebc83b3a33f..72474b512063 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3313,8 +3313,6 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, struct file *file; int ret; - if (p->flags & IORING_SETUP_URINGLET) - return -EINVAL; if (!entries) return -EINVAL; if (entries > IORING_MAX_ENTRIES) {