From patchwork Mon Mar 18 22:00:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595944 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D5035A781; Mon, 18 Mar 2024 22:02:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799335; cv=none; b=GpFS26109JCSdxJmCZz67qlg4paip8kG0WJ032vHFOzr6YmHKf1Y8J44VTpmAFgHINVK1ulgewmdmeSD1kyBZTuOpY7+tyXeXlqrDYGNi3IXYLgMK15wyKh5ZPRy7ZUU54NFZQ69gp6b7xjfty67PxqP8B9Dh+x8MRGsHiTXfrk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799335; c=relaxed/simple; bh=D2/O6AEH2zUmAJbf6CgZefbxyFofmhE4vCawTBwn1Xg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=B/NRYhqESdToNKLHrF6EEMBxG9jO0hBoKXhdwcRxNuU2bEP9eMiq/f2aGEADmFhQrMdGdTFm54pXcuVWKc/BhW1IeVfJLxv/x+vH7KT+O464iAKuf52DKrcD5LFHYaAqQVebc+K4quzUpkd5Pez/hPVSzwZC1g8pH6ngNAmteTQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=haccvG1b; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="haccvG1b" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-34169f28460so1050417f8f.2; Mon, 18 Mar 2024 15:02:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799331; x=1711404131; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=X9YjALaP/hn1N1s+9XuXIoyFFiEVJsOMKt0K7ANflvs=; b=haccvG1b4rMuY4uMNyqfygBWFJGtbQnpAz1g+kBi5EkpE0vp9pWvBtCDaEqm6YfWxN k1duWhY0RhQLCjjuTfS+fXG3thKZZWyjia+7Gx/KbMUqj6n2zS7pzrgCH/NExLRPMp8V Ni1944w3UY0X9uBmBoXx4Cmx1r36Bk9SLRPJ6jiK2DCnK6TnB7tysQ58ugn3mFb2uBI1 faS4ORFfsr+AJTf6Vl4fAQIpC1YUo6CaSr7uVCU+vBQG8WKxaRSzDvQobZiQwkyyBe8Q aNN4318zkBK1AkGCAfhF8r5nisv8FP37WE99PK9evmosnxU+t/Z1sOjKAjdOwYL9bIFF JTUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799331; x=1711404131; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X9YjALaP/hn1N1s+9XuXIoyFFiEVJsOMKt0K7ANflvs=; b=oPzu8qdo+cutSpnMJY7WPcbcuQLtUTkp1s8e3fPGWOIVqF5Ay2LuJNe6cP+VV10G9/ mmweJ13p7MOdv+n0ltErXr+16VmDowmgeJydEISTSX1BIRJzjexGWouC8nheTTHqr6lX pg5Kh+1+zeYdKC4YrZ9js6v3XaoCumcUAeVP23WyoWPRX/AmtjXe5U1ckOHTLwZhOSyO nCLP6xsDko8BinzPeSN+GIvkuSDspxPjsQ+ZtotMLl6OjGjIXGchetfBDJoLaUiCybYK F0CgdEsyVeDSFjNXW8zlb4sshTgI2ZxoL1RkEaGIayYwG1RPuiBs+fVv9wS4GvD6HGeE tovg== X-Gm-Message-State: AOJu0YzRSpTebWlG8huQ29vG1yNo6Vi2qWm2j/o3mLlt5cIGervVl7pH 3wYWwDYmG/CJhStqRKE+C2Y/35An/oIYGgBK1Qo3yMnFXVZhKVdFJBcYNw0k X-Google-Smtp-Source: AGHT+IF6E9o7zGLit5II2+ujC29Av45Yd6zdndxagmMTsSpmB7gG1ceZwEHDk7JDl2w6NWCN8ZPu+Q== X-Received: by 2002:a05:6000:2a1:b0:33e:7750:781d with SMTP id l1-20020a05600002a100b0033e7750781dmr591137wry.56.1710799330795; Mon, 18 Mar 2024 15:02:10 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:10 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 01/13] io_uring/cmd: move io_uring_try_cancel_uring_cmd() Date: Mon, 18 Mar 2024 22:00:23 +0000 Message-ID: <43a3937af4933655f0fd9362c381802f804f43de.1710799188.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring_try_cancel_uring_cmd() is a part of the cmd handling so let's move it closer to all cmd bits into uring_cmd.c Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- io_uring/io_uring.c | 39 +-------------------------------------- io_uring/io_uring.h | 7 +++++++ io_uring/uring_cmd.c | 30 ++++++++++++++++++++++++++++++ io_uring/uring_cmd.h | 3 +++ 4 files changed, 41 insertions(+), 38 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 5d4b448fdc50..6ca7f2a9c296 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -95,6 +95,7 @@ #include "waitid.h" #include "futex.h" #include "napi.h" +#include "uring_cmd.h" #include "timeout.h" #include "poll.h" @@ -173,13 +174,6 @@ static struct ctl_table kernel_io_uring_disabled_table[] = { }; #endif -static inline void io_submit_flush_completions(struct io_ring_ctx *ctx) -{ - if (!wq_list_empty(&ctx->submit_state.compl_reqs) || - ctx->submit_state.cqes_count) - __io_submit_flush_completions(ctx); -} - static inline unsigned int __io_cqring_events(struct io_ring_ctx *ctx) { return ctx->cached_cq_tail - READ_ONCE(ctx->rings->cq.head); @@ -3237,37 +3231,6 @@ static __cold bool io_uring_try_cancel_iowq(struct io_ring_ctx *ctx) return ret; } -static bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx, - struct task_struct *task, bool cancel_all) -{ - struct hlist_node *tmp; - struct io_kiocb *req; - bool ret = false; - - lockdep_assert_held(&ctx->uring_lock); - - hlist_for_each_entry_safe(req, tmp, &ctx->cancelable_uring_cmd, - hash_node) { - struct io_uring_cmd *cmd = io_kiocb_to_cmd(req, - struct io_uring_cmd); - struct file *file = req->file; - - if (!cancel_all && req->task != task) - continue; - - if (cmd->flags & IORING_URING_CMD_CANCELABLE) { - /* ->sqe isn't available if no async data */ - if (!req_has_async_data(req)) - cmd->sqe = NULL; - file->f_op->uring_cmd(cmd, IO_URING_F_CANCEL); - ret = true; - } - } - io_submit_flush_completions(ctx); - - return ret; -} - static __cold bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, struct task_struct *task, bool cancel_all) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 6426ee382276..935d8d0747dc 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -154,6 +154,13 @@ static inline void io_req_task_work_add(struct io_kiocb *req) __io_req_task_work_add(req, 0); } +static inline void io_submit_flush_completions(struct io_ring_ctx *ctx) +{ + if (!wq_list_empty(&ctx->submit_state.compl_reqs) || + ctx->submit_state.cqes_count) + __io_submit_flush_completions(ctx); +} + #define io_for_each_link(pos, head) \ for (pos = (head); pos; pos = pos->link) diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 42f63adfa54a..1551848a9394 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -14,6 +14,36 @@ #include "rsrc.h" #include "uring_cmd.h" +bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx, + struct task_struct *task, bool cancel_all) +{ + struct hlist_node *tmp; + struct io_kiocb *req; + bool ret = false; + + lockdep_assert_held(&ctx->uring_lock); + + hlist_for_each_entry_safe(req, tmp, &ctx->cancelable_uring_cmd, + hash_node) { + struct io_uring_cmd *cmd = io_kiocb_to_cmd(req, + struct io_uring_cmd); + struct file *file = req->file; + + if (!cancel_all && req->task != task) + continue; + + if (cmd->flags & IORING_URING_CMD_CANCELABLE) { + /* ->sqe isn't available if no async data */ + if (!req_has_async_data(req)) + cmd->sqe = NULL; + file->f_op->uring_cmd(cmd, IO_URING_F_CANCEL); + ret = true; + } + } + io_submit_flush_completions(ctx); + return ret; +} + static void io_uring_cmd_del_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags) { diff --git a/io_uring/uring_cmd.h b/io_uring/uring_cmd.h index 8117684ec3ca..7356bf9aa655 100644 --- a/io_uring/uring_cmd.h +++ b/io_uring/uring_cmd.h @@ -3,3 +3,6 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags); int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_uring_cmd_prep_async(struct io_kiocb *req); + +bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx, + struct task_struct *task, bool cancel_all); \ No newline at end of file From patchwork Mon Mar 18 22:00:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595945 Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9060659B74; Mon, 18 Mar 2024 22:02:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799336; cv=none; b=HJ+qbwctAV7NuyX52GI20qBonpwy29/EsJxzS3e945yt18mcnTcZe0GG7wWR265Pxa9EmmbA2mTV5xhNJkSgi53Kk6FkY5XT7kJY4HJHtwNj3+yw3PTCPEJhxNAyU9rLHxzCNUqUtxdfEHQUoTK5qD3KjO0YzJmVHD4Yf8bRze8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799336; c=relaxed/simple; bh=H99ZcmrUACp8z/6xofR/+Bu+uFUG0paOzPEVB0ENvxk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rGAuWt0hu0RZeqSCELGnhPdl2lDanx8rb7Jykb3cmcLIp3K5PFGNyBpo4hgvUC0QS5adGcf4H4z0HJOPT2GXTc/oXYgv3lsrEa5maWMbOE02vnqA8NWLSNIHiZLwn3fDQsjG+ob3qi59Tn0GR6AuM5d9C+grJ4LXs+HVEzY8weg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Fl6MGgt6; arc=none smtp.client-ip=209.85.128.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Fl6MGgt6" Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-41413c99748so7940035e9.0; Mon, 18 Mar 2024 15:02:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799332; x=1711404132; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3Wbs7GQpve7jyA3Hi1WAUjWpcV8NDvkjvnKIw65sYB4=; b=Fl6MGgt6TaDR8Bc85z6zZJq4OtoTnbKkEJHdJqBIVVkbNjLXmpmWRKPpR5sL3qphds 3K2GoTx7CerlzkPTfOxew6T8+rtEvTpHV4E0baQko4FXwwnbiOtyKZWFXdxaoL/4BDfu OxatWHarNvplIU2Pn2WxU7DMP84ScyjIh3zMH6X7SMEKwx3i42zIG8fPhPT2sU1Ey3Xp fviPf9fO2JdeDQWgVugrTcU5lGDiSSnLlsjAg2pF20adYjrG4UVvBFGKdBoLcz8Fi25v yj5xorReaJ8GXgvTYFMrGS1i/tD/3oZ8ehdEo6S95cf38VWfelEBc7ZT1Pi254lCdiy2 l6Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799332; x=1711404132; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3Wbs7GQpve7jyA3Hi1WAUjWpcV8NDvkjvnKIw65sYB4=; b=CYX3eUihlSWYTeoH03yTDE1Wn5xcvse1cGe5iQ46tKjfuDmKf60deNbsNvRQ9scxDa X9eS/qBit+882iTsDA06SHis4oskosFw1mdmfD0donZIdE7lZGg2upMvyTPl6nUVKzWY pR3BPNIIgzVY7/NZZVOS4SOR+7dMkw+TQNaMM9ibgvXPm7u3m/tIwURcrlhRi2hXMRe/ nCzZfcN2A4XSJGMf9jKjMG/OfmRQyyIDYB9NHs1GK6rRkgePvOlklrRi0A8oRQ9Zc/Al TqEyJqsqTsjB9WH5YdPf5tm+SuG6/UIIPGRikwh69soJA6R1pZrM92Y/dU9nOUkuQvBv LdIA== X-Gm-Message-State: AOJu0YwlwUm9CfKJADQeF4XNhTKeNJ/LnyO8XRolbpoQDhtB9oP2I8+H f58evJgeuio8nkUL3+9eGqwprQb2URfJY3RdeHlECMNkaj6HySZPsIAMYNH1 X-Google-Smtp-Source: AGHT+IHF9p8DiUjEfE/MjPDImUl4SXeqMwTnef6SzO3Q1jmJZzGo8RCmaj6gfTi5CRi16JV+kFq9/g== X-Received: by 2002:a05:600c:444f:b0:414:835:6ed2 with SMTP id v15-20020a05600c444f00b0041408356ed2mr448819wmn.35.1710799332202; Mon, 18 Mar 2024 15:02:12 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:11 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 02/13] io_uring/cmd: kill one issue_flags to tw conversion Date: Mon, 18 Mar 2024 22:00:24 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring cmd converts struct io_tw_state to issue_flags and later back to io_tw_state, it's awfully ill-fated, not to mention that intermediate issue_flags state is not correct. Get rid of the last conversion, drag through tw everything that came with IO_URING_F_UNLOCKED, and replace io_req_complete_defer() with a direct call to io_req_complete_defer(), at least for the time being. Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- io_uring/uring_cmd.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 1551848a9394..7c1c58c5837e 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -130,11 +130,11 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, if (req->ctx->flags & IORING_SETUP_IOPOLL) { /* order with io_iopoll_req_issued() checking ->iopoll_complete */ smp_store_release(&req->iopoll_completed, 1); + } else if (!(issue_flags & IO_URING_F_UNLOCKED)) { + io_req_complete_defer(req); } else { - struct io_tw_state ts = { - .locked = !(issue_flags & IO_URING_F_UNLOCKED), - }; - io_req_task_complete(req, &ts); + req->io_task_work.func = io_req_task_complete; + io_req_task_work_add(req); } } EXPORT_SYMBOL_GPL(io_uring_cmd_done); From patchwork Mon Mar 18 22:00:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595946 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B9DE5A4C4; Mon, 18 Mar 2024 22:02:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799338; cv=none; b=Xkw2olPxe9AsyM6XBOpzSxv47SxNxKpDRqRbFVcJnMcwGHGdA5yEaDnf3EvGd6a64U+fjTqfjeMFduKdXx0o9i7lsJkEdDnw/RIEetbIzl8cxVgvcT5Jqw8z9QASOslcK7cm/dJNFNgJv2B2ZuVT7awLFQNNKNhmqVRkI8L86sc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799338; c=relaxed/simple; bh=9vY/DtUnP1rqLrlOMEZB7dtzAeVsa67WGy2r3MI5zp0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NG04r3npXmbesHwohNLBnHh3r1K7xtaU76RfLnoRSN7b81BMyXYVpi6OwBTC/RyrAuU2OwM5ofXN1bs+4k5WWTtJ6NfWwy87x2B/Doi9D86DGPqOqHzRICQKzIjNhZBSho6oS5nkhbIWRjKvUJ21eCkEjwU130okGYnZ14NA/eY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=NX1fgGfr; arc=none smtp.client-ip=209.85.221.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NX1fgGfr" Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-33ed6078884so1805406f8f.1; Mon, 18 Mar 2024 15:02:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799334; x=1711404134; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5Vz9qyCAXh48FsaL90phLZ5p6nLM8jmfgDzdMhV6mLY=; b=NX1fgGfrsFL/+llp3VE2ZAiYrJj2P6cZnjpvWoAJaO1x9uoi6Dg2o8lps74gBOPto6 5L0PIT9JcVcFZz1PsV2idliTDZusSWJ0DsaClhG4VV/WBtk9GpKMkZ7gzpffN6QV00bD Ql9cj+nACHxlS8c0JW7/diJqqWKMjvIc50AXBk5M/qqSZLgIwPWyn/kcX+bv5iGeNCUq 29aQc6mifhYUiIFIm8owAHViV2lvm5CW16X3MYdL1A1dkaFUQY8gCmnxBtXOlH8xMRQZ H+WfeuaQz5cmTBbLCo9OrtIRpjvAr8WG4YQRa/nE5FSvSxxUu4g8/gWEr2LILxFkUDWs XTmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799334; x=1711404134; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5Vz9qyCAXh48FsaL90phLZ5p6nLM8jmfgDzdMhV6mLY=; b=iHTMcAycSMqHDEy13jbhHcO7oRgVWcskHXb/+QbXUzc4FcwWcw6v9c5Y11bBsJXr62 TOPUxHCIGJP0vbXCBhwZkadJykV4hTKgqx+1ZSmETcGwSpjqykS65c7V8Wj1C1iXvoc3 a85YDiPs5e85Bfda/YH9ds/8Finly9kF6HhCWVFtdXu/eKUUO7k3dfn6GJujYo5mBctH cTJ7Je9pfaLe94tbsoZPu4MvonxIoRxE0RRjq9hgbBhrEo1lekYwWU/rjjNlvXM4hhA4 5ZT97H+20JVlM7e8pf/hjQhCCcSClIwQvW+STjBPZn/xXRIigXWWiK0ZW1O56N57jVfq 7mEg== X-Gm-Message-State: AOJu0YxQ86FUmbqja62VLZ9CMPxx6vvIdsJqtEczBGiA2ukqjvFpXvTT xV8jNBf0L7lFzYERzSawf3mkJ8CDGu2wOcUjZqDyysPf/vlnfBMHlU3mSGYu X-Google-Smtp-Source: AGHT+IG5jlNB2MPc7bXOq8Dnjr9Xh8n0k8tMahOd1m55YdFwq5AVjcgtPyOc6ckF8HkXT6a0o5GcUA== X-Received: by 2002:adf:cf01:0:b0:33e:6ef3:b68e with SMTP id o1-20020adfcf01000000b0033e6ef3b68emr622948wrj.34.1710799333795; Mon, 18 Mar 2024 15:02:13 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:12 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 03/13] io_uring/cmd: fix tw <-> issue_flags conversion Date: Mon, 18 Mar 2024 22:00:25 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 !IO_URING_F_UNLOCKED does not translate to availability of the deferred completion infra, IO_URING_F_COMPLETE_DEFER does, that what we should pass and look for to use io_req_complete_defer() and other variants. Luckily, it's not a real problem as two wrongs actually made it right, at least as far as io_uring_cmd_work() goes. Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- io_uring/uring_cmd.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 7c1c58c5837e..759f919b14a9 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -36,7 +36,8 @@ bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx, /* ->sqe isn't available if no async data */ if (!req_has_async_data(req)) cmd->sqe = NULL; - file->f_op->uring_cmd(cmd, IO_URING_F_CANCEL); + file->f_op->uring_cmd(cmd, IO_URING_F_CANCEL | + IO_URING_F_COMPLETE_DEFER); ret = true; } } @@ -86,7 +87,11 @@ EXPORT_SYMBOL_GPL(io_uring_cmd_mark_cancelable); static void io_uring_cmd_work(struct io_kiocb *req, struct io_tw_state *ts) { struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); - unsigned issue_flags = ts->locked ? 0 : IO_URING_F_UNLOCKED; + unsigned issue_flags = IO_URING_F_UNLOCKED; + + /* locked task_work executor checks the deffered list completion */ + if (ts->locked) + issue_flags = IO_URING_F_COMPLETE_DEFER; ioucmd->task_work_cb(ioucmd, issue_flags); } @@ -130,7 +135,9 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, if (req->ctx->flags & IORING_SETUP_IOPOLL) { /* order with io_iopoll_req_issued() checking ->iopoll_complete */ smp_store_release(&req->iopoll_completed, 1); - } else if (!(issue_flags & IO_URING_F_UNLOCKED)) { + } else if (issue_flags & IO_URING_F_COMPLETE_DEFER) { + if (WARN_ON_ONCE(issue_flags & IO_URING_F_UNLOCKED)) + return; io_req_complete_defer(req); } else { req->io_task_work.func = io_req_task_complete; From patchwork Mon Mar 18 22:00:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595947 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0648159B74; Mon, 18 Mar 2024 22:02:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799338; cv=none; b=uyKFUp8qxzWzfGuMaam+q3TerPyNcj0EI8eZzDigqL4qpKHOoQyCqAoJeNSJXM97CrvY+z87tlhz5n+uhlkD45wNt+T1ebhJ3PC4Q5ChkrsRAKFvi+wX3RDXX/ZvBV88Pu3jyvw58qNlE9S8qNk/1pqZpKfdu42a8eryJgu6CWM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799338; c=relaxed/simple; bh=JCp/LX1aRNcLI0yu32VofH3AASA/z92mcX8UJU7sh+E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T1j/6g+UlUmK6izyKrM3+L8omZ1FxDB1+ztX92rA11BhFajGIUjOoZKZ9+k4nS2O7K/m4scF91EVni3/2AEwb6SWj/CL87aSag0AhoHl8LWhuws94FXrLihUnIYOrhm0B4idHmtMeNtFlraWc5LDMI8wFDX5SE49wxh5w97ejXk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=HP7Qrd0Y; arc=none smtp.client-ip=209.85.221.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HP7Qrd0Y" Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-33ed7ef0ae8so1560014f8f.0; Mon, 18 Mar 2024 15:02:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799334; x=1711404134; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=okOU/yJL1S1r/fv/BiyDDnckwReFnZwpj+K4WdugD8s=; b=HP7Qrd0YQMy8HtQhKMiUQfZanZsIFpflA6YQG3DkDsZwPLGeSHIe+sdV4z6N/q0+8u OkF13gv5u1qkFulyYY3S2DTAbXJjzSRx3/Rlrn9eBfLx8aH/pfy1CLnyWzj0Y64c9rCW 0zP7tv1NoxATsfzpSX+An2PiwPXCatLBBMVssTkbEctTn8Jpc+jD3u2189447FYtt3vP KoBADp4Sv4DJG1WahOtAyQ3lDJnXpvcVSLKuIna7VMzFtxvsPnxEtpuQdMWzA/E99PwT tN2sZN+N4elZgE/XRAIOXRvU8RaAzW8AQ+OjWYITwopHTrXLcZSyFgge9qUiGzdV3/qO HEIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799334; x=1711404134; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=okOU/yJL1S1r/fv/BiyDDnckwReFnZwpj+K4WdugD8s=; b=IRo2ExtimE4/jPeLarGKBWGf4WQne193WJDbA80L8HSEGa8h446H703UjqUv4dZmJs qBZG7xxSIpRMxFqB5q1QxjJOIUzoQDcX1jDS2I7l83SC24oS1BXli0FwKJ7zZmokDgwp XgXAlt6HNlLDId+k7BB6OuXEFZoyeG/96+HuxM+9m85ssaYmck/lSSHc0Q5oATJPiucU SE6ucXn/ladplzTIiHsJJ5IaTYU0jf3x0NYDcIswK4DWDJZD1a1ZWfKiSrc9pbomNplR cOlxkjcNAcqrvXtkhfpMNZQ6ONUSTZEFnC90WxvQ9VFCv0vG2bfG/SXVUZdjlJ7hFpvY HS2Q== X-Gm-Message-State: AOJu0Yx94zfNcV65mrNd+JKInMcm6Eu/3awS2MpAkb6FryiMnxBGQSpu Bg+t+iU67PRxM6ulW6fT1JvwAt1GAHf+W7mfA+3aGOAklFoYSS2uno7mxoYV X-Google-Smtp-Source: AGHT+IEGBPiDPkvBVjkzhWjsnSXQB1PjPEfVyl0X+QJ0aEZFNWG9zv/gpGduHelfP66esEg+5Ja7tg== X-Received: by 2002:adf:ab14:0:b0:33e:d396:bc41 with SMTP id q20-20020adfab14000000b0033ed396bc41mr363729wrc.71.1710799334606; Mon, 18 Mar 2024 15:02:14 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:14 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 04/13] io_uring/cmd: introduce io_uring_cmd_complete Date: Mon, 18 Mar 2024 22:00:26 +0000 Message-ID: <82ff8a45f2c3eb5f3a04a33f0692e5e4a1320455.1710799188.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring_cmd_complete() does exactly what io_uring_cmd_done() does, that is completing the request, but doesn't ask for issue_flags argument. We have a couple of users hardcoding some random issue_flags values in drivers, which they absolutely should not do. This function will be used to get rid of them. Also, add comments warning users that they're only allowed to pass issue_flags that were given from io_uring. Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- include/linux/io_uring/cmd.h | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index e453a997c060..bf94ed4135d8 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -26,12 +26,25 @@ static inline const void *io_uring_sqe_cmd(const struct io_uring_sqe *sqe) #if defined(CONFIG_IO_URING) int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd); + +/* + * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd + * and the corresponding io_uring request. + * + * Note: the caller should never hard code @issue_flags and is only allowed + * to pass the mask provided by the core io_uring code. + */ void io_uring_cmd_done(struct io_uring_cmd *cmd, ssize_t ret, ssize_t res2, unsigned issue_flags); + void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, void (*task_work_cb)(struct io_uring_cmd *, unsigned), unsigned flags); +/* + * Note: the caller should never hard code @issue_flags and only use the + * mask provided by the core io_uring code. + */ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags); @@ -56,6 +69,21 @@ static inline void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, } #endif +/* + * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd + * and the corresponding io_uring request. Similar to io_uring_cmd_done() but + * doesn't need issue_flags. + * + * Note, must not be used with cancellable requests. + */ +static inline void io_uring_cmd_complete(struct io_uring_cmd *ioucmd, + ssize_t ret, ssize_t res2) +{ + if (WARN_ON_ONCE(ioucmd->flags & IORING_URING_CMD_CANCELABLE)) + return; + io_uring_cmd_done(ioucmd, ret, res2, IO_URING_F_UNLOCKED); +} + /* users must follow the IOU_F_TWQ_LAZY_WAKE semantics */ static inline void io_uring_cmd_do_in_task_lazy(struct io_uring_cmd *ioucmd, void (*task_work_cb)(struct io_uring_cmd *, unsigned)) From patchwork Mon Mar 18 22:00:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595948 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6403E5A4D8; Mon, 18 Mar 2024 22:02:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799340; cv=none; b=pJ2XhwNVNhw0QrS+AWUq+2AtBOmSfNZIAfmhsW+O2mcb6S7vXUi3hK5URkLQ3xXdYiO6sJT/JSUCDVJMoz3A/8UzEoKj5qekkWiOhFlnaYQy8kZiycqzpjnOR8koJrkfnsQXyoWjP3+U2DDY6ljP4WJEe7T36Xnhib/rtU14mNk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799340; c=relaxed/simple; bh=88PXd6LH82VU+BV9k/zBBXLt/9uURnfB+ehKJC4vzV8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jMli0+60fBQWbQAi/it9nJyKLUc+jBNLsKZV9YLM8+LHwxKGzxYzPKjgaQuAHZmBTHYL+Ob2Hf62p7qEPJMy/dSEYRaG9/gC8IA33J/2ZLOulkZxYIltNvu8rr5z5ErlA7eCkxBnMd60XogXvK3eLMta4RhcEwWKWxwwhCMs0YU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PnnVfuwq; arc=none smtp.client-ip=209.85.128.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PnnVfuwq" Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-41412411622so9432135e9.2; Mon, 18 Mar 2024 15:02:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799336; x=1711404136; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WIJUrcLYe0QNkJgTemkNItt6bDCqZ/mmH4VKVCsbA/4=; b=PnnVfuwq+i5xHQoM7zK2PAtWV9hp9DiCdh9m+Qsj7hBNn3jwOrUGlrNDdw509mMs0W 3FBlb+CV7Rqbv7/d1j7sEFKkRB9kDpTF4dQ9VQ7kGNeyV8XRSS/XZPXrZ21OWCRWkIuZ nlvlBlc6djUjhO00xsc/Wui8cxumlaLE+zbtrI5ahs9ToZwzRkHWbxibdfWg5xnTO5oo OMsDRYCDQ6WPGFdzawp/N3ztmpkQtMDIvQ2Vs9eENAfFvF+xXgutca5ff9PtJOmkx+LQ O5DAEk0o1POI7Dq5pba/OaXFRzsHW21KY9OUyG8HcWN7rnficv56krgWLlrCbL0xzXMj lOTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799336; x=1711404136; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WIJUrcLYe0QNkJgTemkNItt6bDCqZ/mmH4VKVCsbA/4=; b=U8iN21AhQJDVKcg6oSi/k4ZumdZ74nlmrKcHP3gmCIUqcqjbgUryA2zLIpTQ2GYfOj gk0G6ZHK9QXCP09qdTEYskW9jxkLCdBhxf2DltiwgdGabFDrzyT9FZZsssyIg66BtSFA Wf13to9PKU4/WaNNTUiIPbJzNZcwTUWJ+RunWFpBEGqQUs2yiVCFpFaP9XdziCaG66my g9g6+woZCTRVaYGmw9GV8C0p7FgbyWq7L2oLkPLhdYqEhcJlm0QcG2XT92ovmm/l4idQ CK0cnkd5jLHxZLlJyqLx0tZOZ9+NJZSQzkpKtfVh/axzE366VLyGMCY6XXSxgBrzzFr7 0EDQ== X-Gm-Message-State: AOJu0YxNLS7n5wGIQkGpLxaKf4KcbNF1/6D7z3D7YyaZBwV+PzoS6qIK qhdWA4DUaLtj+uV3N7S5o5NCDylrDjUlfWdLISJBLzU91aIpzWyJFNxFKNid X-Google-Smtp-Source: AGHT+IHxLr4Z0Qn3NF9Snv6GN+3xdScEJ4/3a0pDbUpS1XqaudYxjx/welOJr97lx6w1r2PxBgVR1A== X-Received: by 2002:a05:600c:3b0a:b0:414:cc0:e4d6 with SMTP id m10-20020a05600c3b0a00b004140cc0e4d6mr3599637wms.26.1710799336169; Mon, 18 Mar 2024 15:02:16 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:15 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 05/13] nvme/io_uring: don't hard code IO_URING_F_UNLOCKED Date: Mon, 18 Mar 2024 22:00:27 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 uring_cmd implementations should not try to guess issue_flags, use a freshly added helper io_uring_cmd_complete() instead. Reviewed-by: Kanchan Joshi Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- drivers/nvme/host/ioctl.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 3dfd5ae99ae0..1a7b5af42dbc 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -426,10 +426,13 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, * For iopoll, complete it directly. * Otherwise, move the completion to task work. */ - if (blk_rq_is_poll(req)) - nvme_uring_task_cb(ioucmd, IO_URING_F_UNLOCKED); - else + if (blk_rq_is_poll(req)) { + if (pdu->bio) + blk_rq_unmap_user(pdu->bio); + io_uring_cmd_complete(ioucmd, pdu->status, pdu->result); + } else { io_uring_cmd_do_in_task_lazy(ioucmd, nvme_uring_task_cb); + } return RQ_END_IO_FREE; } From patchwork Mon Mar 18 22:00:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595949 Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC0F859B55; Mon, 18 Mar 2024 22:02:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799341; cv=none; b=r7Lx1McRLGfEeswjNjMPhDOPuarioC27j5AspnwrqXeJ/Y0QZ1gj02qHmOD5dfNwzmgmeXiN4r199eT+fU9oiCuRd8uX0BtIih1luXu3BKmNTKltEaNRNCKv1z/gZGfl+bq9Ab84li2jIMJFIMq0RkVhJ96RW1x17T842AFphvQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799341; c=relaxed/simple; bh=CqEy9YlnIJxs2HVEo6RpmOFUMTKd1shNymbvV4OtMgQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pRbBiXJukQI84n7mwBXFvi+mFGHPJQ4enQMGT3KRly96i335cbXxuqn0yyPtootGQvWZvoo4qjb56OH/mRQISTXVCt4bCSzasxnQaxpTh39MHxdDNc1LST7LcNQW56kXiX6Ue0PBoP+61tKYHV/dYOuM19ayFV7MMomWvlrpfaI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VpJkYudD; arc=none smtp.client-ip=209.85.221.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VpJkYudD" Received: by mail-wr1-f54.google.com with SMTP id ffacd0b85a97d-34005b5927eso1508266f8f.1; Mon, 18 Mar 2024 15:02:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799338; x=1711404138; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xk51Ayd0oOL8B8bNip0Qvci6ISd/wlEH/T4TunQP5iw=; b=VpJkYudDvTP5T4S04wJUuX7wJL2y9Yw/tIN3ubAvSHW9DOXm+9Wxtt7jNm54ZlaoDV lpg05NTOUtLxa/4aKaQVS6GVKwNeuXnHuyLfh5u7OagtO/kDAqO44/wSuw5h+YAJC4SZ OgHv3emNtNp8uHURmuoxWnsA0+m32758e/1ticytzzVK99KFKruXKwx0ROex8CQroDTK If2+G1fnXEQ6mlT8hvyvKJj6lzqAuartCxGc+u93ZbnOcT1rhySC2hzKJs3S1xS7QXn/ vnUDRBudUWOZNe7m1m9vfMN6XcDZ5FtUFEDkcteWidpGHmoZTDg30ZbbdGrzq+qKt+Sj CL+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799338; x=1711404138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xk51Ayd0oOL8B8bNip0Qvci6ISd/wlEH/T4TunQP5iw=; b=Ejj1G3XKg0YBaDMQjPN/5cmxkQZcDHekaGCY0EntDCu9iQ/5x3TkE9ZBqq/kSTd+9D xfGDLyNZK/2g7j54eXGNv9qk1um+9X/tPCde7+zg3eD5ReZmK8KUoWqZTVAj+USK/VLT b4d0m93KZrRFL1WiyU8/v490bdXZFSGI76q318/D31duhVnku4Z1B0rwymp3LkICxv8m 51lQAFpeqn1dJ6FRmIjcAmzEXUJNjHAUYqwVDMZA+iX8tLp11s5SXupuIvAEQFJAQxjs x2ps/tgsyEZ7XAJeDUxWnxVK6J1PqDvYjcKFVzfghynA4TW5058Ucec1v1Y7XABbv+f8 DQVw== X-Gm-Message-State: AOJu0YwOyyiwpOTKBdsnrIznGf658XrISH3tTHhUUU6IcIqa1KSLrxVE v0q9QPA1cxWzjqWlpCpk7yFu/qI8mmcry5QgpR0D38MrJr8K7+SEwh8fqpvH X-Google-Smtp-Source: AGHT+IEIrbrIkE/08hBiwsXKc5D/fiNXGxss0DGlADUl8m6/aSQyZCFk6tFuRpKVT0PCVLMNBTXodg== X-Received: by 2002:adf:f7c9:0:b0:33e:c6a2:7f6 with SMTP id a9-20020adff7c9000000b0033ec6a207f6mr9152664wrq.11.1710799337585; Mon, 18 Mar 2024 15:02:17 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:16 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 06/13] io_uring/rw: avoid punting to io-wq directly Date: Mon, 18 Mar 2024 22:00:28 +0000 Message-ID: <413564e550fe23744a970e1783dfa566291b0e6f.1710799188.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 kiocb_done() should care to specifically redirecting requests to io-wq. Remove the hopping to tw to then queue an io-wq, return -EAGAIN and let the core code io_uring handle offloading. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 8 ++++---- io_uring/io_uring.h | 1 - io_uring/rw.c | 8 +------- 3 files changed, 5 insertions(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 6ca7f2a9c296..feff8f530c22 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -492,7 +492,7 @@ static void io_prep_async_link(struct io_kiocb *req) } } -void io_queue_iowq(struct io_kiocb *req, struct io_tw_state *ts_dont_use) +static void io_queue_iowq(struct io_kiocb *req) { struct io_kiocb *link = io_prep_linked_timeout(req); struct io_uring_task *tctx = req->task->io_uring; @@ -1499,7 +1499,7 @@ void io_req_task_submit(struct io_kiocb *req, struct io_tw_state *ts) if (unlikely(req->task->flags & PF_EXITING)) io_req_defer_failed(req, -EFAULT); else if (req->flags & REQ_F_FORCE_ASYNC) - io_queue_iowq(req, ts); + io_queue_iowq(req); else io_queue_sqe(req); } @@ -2082,7 +2082,7 @@ static void io_queue_async(struct io_kiocb *req, int ret) break; case IO_APOLL_ABORTED: io_kbuf_recycle(req, 0); - io_queue_iowq(req, NULL); + io_queue_iowq(req); break; case IO_APOLL_OK: break; @@ -2129,7 +2129,7 @@ static void io_queue_sqe_fallback(struct io_kiocb *req) if (unlikely(req->ctx->drain_active)) io_drain_req(req); else - io_queue_iowq(req, NULL); + io_queue_iowq(req); } } diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 935d8d0747dc..0861d49e83de 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -79,7 +79,6 @@ struct file *io_file_get_fixed(struct io_kiocb *req, int fd, void __io_req_task_work_add(struct io_kiocb *req, unsigned flags); bool io_alloc_async_data(struct io_kiocb *req); void io_req_task_queue(struct io_kiocb *req); -void io_queue_iowq(struct io_kiocb *req, struct io_tw_state *ts_dont_use); void io_req_task_complete(struct io_kiocb *req, struct io_tw_state *ts); void io_req_task_queue_fail(struct io_kiocb *req, int ret); void io_req_task_submit(struct io_kiocb *req, struct io_tw_state *ts); diff --git a/io_uring/rw.c b/io_uring/rw.c index 0585ebcc9773..576934dbf833 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -187,12 +187,6 @@ static inline loff_t *io_kiocb_update_pos(struct io_kiocb *req) return NULL; } -static void io_req_task_queue_reissue(struct io_kiocb *req) -{ - req->io_task_work.func = io_queue_iowq; - io_req_task_work_add(req); -} - #ifdef CONFIG_BLOCK static bool io_resubmit_prep(struct io_kiocb *req) { @@ -405,7 +399,7 @@ static int kiocb_done(struct io_kiocb *req, ssize_t ret, if (req->flags & REQ_F_REISSUE) { req->flags &= ~REQ_F_REISSUE; if (io_resubmit_prep(req)) - io_req_task_queue_reissue(req); + return -EAGAIN; else io_req_task_queue_fail(req, final_ret); } From patchwork Mon Mar 18 22:00:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595950 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 568AB5A4D8; Mon, 18 Mar 2024 22:02:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799343; cv=none; b=igM3/YbNNev4jRQ2RiTa0Kknc+3+wLMFFcD2ZB7XwKZ3FAo8Mv5kD4gjUbmjr9V2jVDMO2cAosMDlRutMsaUcSSdkQ9Zy+jmxJGMinzn2YHmSt/cF2yxZNiC0sUhi68dsK5MFg/DjnwL8diAMTDZ3LGmbkL3B9q+HusscN1Zat0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799343; c=relaxed/simple; bh=Ae97tVzbWkRAu0LFf9aZ9RBHFn37wBgTKbXqflC4/ns=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uJV2Bb8rcDZ0SLV/4kTY4qC7dTxj3AEyI8Hy9WYfdUSU9Z/8XnW5gQdD2F7xR3Nrj8rdf+7PGTA/+XUxCb2zkmkpal25VsI15K97aVtJnIbznV1T8S74HIUK5nbSSGx/S0vAt0supXRmoEarPS3onq0wDMgr7jKQouB5SPquqeg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Jq9pgAKe; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Jq9pgAKe" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-33ed4d8e9edso2026543f8f.2; Mon, 18 Mar 2024 15:02:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799339; x=1711404139; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TcROH2LXl8gKrYr6qUKYiPVrUt+epYbzK5tF/Y3MseE=; b=Jq9pgAKem6APyf9gZq2lsuYVZSEeiInBsyDv8XVP5vlVckoxiZzUorTu9/s4r6FOTB S+22suEQBTkSTMp5wSTTjYvZuJizSkyyBJCwGa3BcwPjs7vG/DpmSefaPErwQfllJ3Q/ 2X/RFQmhBPPn9FDW46gj8n34akigfLVhL4p4Pgv51vYdPjogl3ujtk9l4lkppW876Eg4 w6c0rKmaKsyqIfQnWXzKHwFLZJZoS9law9waqhZN+AEETYYrbP+rlzIV6nFRkgc55k4l GAX6FdQo/LF/Hn80YmCyFSijrOpRZw1gwBK1wlpitQcSIjThKz0XJ2+9OHOUzzhSRU++ 4Aog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799339; x=1711404139; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TcROH2LXl8gKrYr6qUKYiPVrUt+epYbzK5tF/Y3MseE=; b=EvP7oOnMdVd3Hc5HJaSj7IHvm2xYnZQWG4sWb+KwYAD3BbwZTOlUmnLRNyhqIyVhPP llKLexi9EhY+e3L3EZenZPYafiEG8tvWmsOPOQOB97QyDoQqOLSakNwDnWpov6sK1qvD pwmF/mZtOCocXLdV02qcxzdVrJNahGTXG0SQzJbZCMJ2odG9Qa0qedjv8w7cFS6YPGHH +hmo9gTSuWU88yx6FoUlpjbt9vFqDwtv7dN2E2PwykulmDXwWFNTU9btcaSiORS0AOsJ OMUiNG3NgZIHeAPeky/iE9Jl3FyjQIVWLmQq3OyPUrrTEM0bJUsF8/0ulnq4Um8ixp5v 4dMQ== X-Gm-Message-State: AOJu0YxyIaGByqBu1NshC6E7+Y28p0BZFAmq86Wdq9ei7+Ob3hwHczvD oFzk+WXJAjuOp5w/dpac1StrtF73mCf4DBfxNyduwYbvj29fDoCeYUh/kH1Z X-Google-Smtp-Source: AGHT+IGXgwHD5pHiwwmQT18GEQKxv6BNPQAqIYvsLZnQWeRdzzw7fpBEHTCqH4kr+zOOtQ/U8RWLxg== X-Received: by 2002:a5d:5051:0:b0:33e:aea8:6969 with SMTP id h17-20020a5d5051000000b0033eaea86969mr10272351wrt.27.1710799339086; Mon, 18 Mar 2024 15:02:19 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:17 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 07/13] io_uring: force tw ctx locking Date: Mon, 18 Mar 2024 22:00:29 +0000 Message-ID: <6ae858f2ef562e6ed9f13c60978c0d48926954ba.1710799188.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We can run normal task_work without locking the ctx, however we try to lock anyway and most handlers prefer or require it locked. It might have been interesting to multi-submitter ring with high contention completing async read/write requests via task_work, however that will still need to go through io_req_complete_post() and potentially take the lock for rsrc node putting or some other case. In other words, it's hard to care about it, so alawys force the locking. The case described would also because of various io_uring caches. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index feff8f530c22..66669cc9a675 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1185,8 +1185,9 @@ struct llist_node *io_handle_tw_list(struct llist_node *node, if (req->ctx != ctx) { ctx_flush_and_put(ctx, &ts); ctx = req->ctx; - /* if not contended, grab and improve batching */ - ts.locked = mutex_trylock(&ctx->uring_lock); + + ts.locked = true; + mutex_lock(&ctx->uring_lock); percpu_ref_get(&ctx->refs); } INDIRECT_CALL_2(req->io_task_work.func, @@ -1447,11 +1448,9 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, struct io_tw_state *ts, if (io_run_local_work_continue(ctx, ret, min_events)) goto again; - if (ts->locked) { - io_submit_flush_completions(ctx); - if (io_run_local_work_continue(ctx, ret, min_events)) - goto again; - } + io_submit_flush_completions(ctx); + if (io_run_local_work_continue(ctx, ret, min_events)) + goto again; trace_io_uring_local_work_run(ctx, ret, loops); return ret; @@ -1475,14 +1474,12 @@ static inline int io_run_local_work_locked(struct io_ring_ctx *ctx, static int io_run_local_work(struct io_ring_ctx *ctx, int min_events) { - struct io_tw_state ts = {}; + struct io_tw_state ts = { .locked = true }; int ret; - ts.locked = mutex_trylock(&ctx->uring_lock); + mutex_lock(&ctx->uring_lock); ret = __io_run_local_work(ctx, &ts, min_events); - if (ts.locked) - mutex_unlock(&ctx->uring_lock); - + mutex_unlock(&ctx->uring_lock); return ret; } From patchwork Mon Mar 18 22:00:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595951 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D24759B55; Mon, 18 Mar 2024 22:02:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799344; cv=none; b=Er2PFpGhB6UnnQ2UCDobqnR0tFHSmpZ7PlZaI8yPhn1rhWv85qusYp0/yQI9PHwbZ0ph00NzA6Q59glIfj7ddDiCsr0l5Ohd91rXnqm35ts/gP3ksynRr7sj1KNqpxBMbU+AzfZ1oy9y0jlSqX1xdCF8Zg9/MX7VjaVk5pFuz6M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799344; c=relaxed/simple; bh=t950zfbRnIL9KiwFOxrn/U9ZBQBztGN0eOm7I1WQAbU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WjAekl2+o6P2i49eyXWSEdhpzjsDo4YKlvoCqiayZtyC6m4CWcOKV/lgN3HfkjVbWU46v3Oh3SyunGcjQxiyNe8CYJ9mLY4sJPefaX3be9Oin42AMOkK+L3zoiuFA4nqXnCwBOkWtCc2Mh7eDwjQdYBCPT6FWRi3XQXLklvzOfg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kspzqL7x; arc=none smtp.client-ip=209.85.221.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kspzqL7x" Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-33ed7ba1a42so1636500f8f.2; Mon, 18 Mar 2024 15:02:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799340; x=1711404140; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dt1ylqgFYRaj19v2BXS6Ij5MenTjqtsRMOdM1nmLkmg=; b=kspzqL7x+aZp3DdVzPZz1DV/ORytf4ycqLz8x/uKmq2RbEtg3u2QObWESZySR3UT34 Q8tTowThyXubxtM61C2BqSdpOE8UcV3gv7U39iL7qTk5vb73cudA7TzkqZQ4bWkeOWrG IZN133kEZoTr9VSyp+1hr9sPxigpUYahtTYQVTR6HDFqvmdYLYgixapuD0IgpZvIZCgB uzEwtMtDEoLkADyJMgjQFsCv5HoKP67QyFE+Cn+CVCWiF8xRJtqjHui1otQ7YO2IR34W lNDMWZtrlHlStkLS4R9BrqCVbZ7Tre3kbYJKrJ5leTDG+fqdvdrvD6z1Mc/mu6ekjBSH VHRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799340; x=1711404140; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dt1ylqgFYRaj19v2BXS6Ij5MenTjqtsRMOdM1nmLkmg=; b=WOsWl3b/h+Y4VHMChppOQC6cpJ9MoIwlcq2Alfp+eBP13azxLDtS+euErgHXa1MjEq OReX7iGbl9ud39j5MemXaGxdlqbsGHfnTGRNODzH03PW7nnLGBgjQPSE03sxNh3Vz6/6 r+h6BTYTVzMsuV72L7Xjz8FkwULNJo2RxeX2o1zu7unUzYOluToOH+mI4J2O8QGv26t8 TS4Stvb++7NoeZRL29rRxovDVnx6EpzbPfwNgNhtqbREzGfWAvUe31onVgm6Iy+TzAhU py8zHpavZsLKda9DHk6jHgbXdgvfoGQYcKLXwRB7atBRHNywtBMS7yLDciR0jFz6Rjwx +ibg== X-Gm-Message-State: AOJu0YyvyIztaeuRfzwwg+jhtDIOJ8Y4MEljQZ8/d/2j/eKdzQ1eTswm 5mO7atOo0hfb7GbTenCZ+OxeVVk8TWVW5gfn3/qaccdhUf3Gwisz4ZkcLn/z X-Google-Smtp-Source: AGHT+IGhnnTYlMX3btOtOwcuDzrq2vyV0WQGUtpB2OLIpUiCYLHsRZQpLxAjHNUJjzQwNFiBfhWwiw== X-Received: by 2002:a5d:4a01:0:b0:33e:7979:edc6 with SMTP id m1-20020a5d4a01000000b0033e7979edc6mr470009wrq.15.1710799340487; Mon, 18 Mar 2024 15:02:20 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:19 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 08/13] io_uring: remove struct io_tw_state::locked Date: Mon, 18 Mar 2024 22:00:30 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 ctx is always locked for task_work now, so get rid of struct io_tw_state::locked. Note I'm stopping one step before removing io_tw_state altogether, which is not empty, because it still serves the purpose of indicating which function is a tw callback and forcing users not to invoke them carelessly out of a wrong context. The removal can always be done later. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 2 -- io_uring/io_uring.c | 31 ++++++++----------------------- io_uring/io_uring.h | 5 +---- io_uring/poll.c | 2 +- io_uring/rw.c | 6 ++---- io_uring/timeout.c | 8 ++------ io_uring/uring_cmd.c | 8 ++------ io_uring/waitid.c | 2 +- 8 files changed, 17 insertions(+), 47 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index e24893625085..5a2afbc93887 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -439,8 +439,6 @@ struct io_ring_ctx { }; struct io_tw_state { - /* ->uring_lock is taken, callbacks can use io_tw_lock to lock it */ - bool locked; }; enum { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 66669cc9a675..9e8afc006fc9 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -246,14 +246,12 @@ static __cold void io_fallback_req_func(struct work_struct *work) fallback_work.work); struct llist_node *node = llist_del_all(&ctx->fallback_llist); struct io_kiocb *req, *tmp; - struct io_tw_state ts = { .locked = true, }; + struct io_tw_state ts = {}; percpu_ref_get(&ctx->refs); mutex_lock(&ctx->uring_lock); llist_for_each_entry_safe(req, tmp, node, io_task_work.node) req->io_task_work.func(req, &ts); - if (WARN_ON_ONCE(!ts.locked)) - return; io_submit_flush_completions(ctx); mutex_unlock(&ctx->uring_lock); percpu_ref_put(&ctx->refs); @@ -1157,11 +1155,9 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, struct io_tw_state *ts) return; if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) atomic_andnot(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); - if (ts->locked) { - io_submit_flush_completions(ctx); - mutex_unlock(&ctx->uring_lock); - ts->locked = false; - } + + io_submit_flush_completions(ctx); + mutex_unlock(&ctx->uring_lock); percpu_ref_put(&ctx->refs); } @@ -1185,8 +1181,6 @@ struct llist_node *io_handle_tw_list(struct llist_node *node, if (req->ctx != ctx) { ctx_flush_and_put(ctx, &ts); ctx = req->ctx; - - ts.locked = true; mutex_lock(&ctx->uring_lock); percpu_ref_get(&ctx->refs); } @@ -1459,22 +1453,16 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, struct io_tw_state *ts, static inline int io_run_local_work_locked(struct io_ring_ctx *ctx, int min_events) { - struct io_tw_state ts = { .locked = true, }; - int ret; + struct io_tw_state ts = {}; if (llist_empty(&ctx->work_llist)) return 0; - - ret = __io_run_local_work(ctx, &ts, min_events); - /* shouldn't happen! */ - if (WARN_ON_ONCE(!ts.locked)) - mutex_lock(&ctx->uring_lock); - return ret; + return __io_run_local_work(ctx, &ts, min_events); } static int io_run_local_work(struct io_ring_ctx *ctx, int min_events) { - struct io_tw_state ts = { .locked = true }; + struct io_tw_state ts = {}; int ret; mutex_lock(&ctx->uring_lock); @@ -1702,10 +1690,7 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) void io_req_task_complete(struct io_kiocb *req, struct io_tw_state *ts) { - if (ts->locked) - io_req_complete_defer(req); - else - io_req_complete_post(req, IO_URING_F_UNLOCKED); + io_req_complete_defer(req); } /* diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 0861d49e83de..7f921daae9c3 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -351,10 +351,7 @@ static inline bool io_task_work_pending(struct io_ring_ctx *ctx) static inline void io_tw_lock(struct io_ring_ctx *ctx, struct io_tw_state *ts) { - if (!ts->locked) { - mutex_lock(&ctx->uring_lock); - ts->locked = true; - } + lockdep_assert_held(&ctx->uring_lock); } /* diff --git a/io_uring/poll.c b/io_uring/poll.c index 6db1dcb2c797..8901dd118e50 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -322,7 +322,7 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts) __poll_t mask = mangle_poll(req->cqe.res & req->apoll_events); - if (!io_fill_cqe_req_aux(req, ts->locked, mask, + if (!io_fill_cqe_req_aux(req, true, mask, IORING_CQE_F_MORE)) { io_req_set_res(req, mask, 0); return IOU_POLL_REMOVE_POLL_USE_RES; diff --git a/io_uring/rw.c b/io_uring/rw.c index 576934dbf833..c7f9246ff508 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -305,11 +305,9 @@ void io_req_rw_complete(struct io_kiocb *req, struct io_tw_state *ts) io_req_io_end(req); - if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) { - unsigned issue_flags = ts->locked ? 0 : IO_URING_F_UNLOCKED; + if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) + req->cqe.flags |= io_put_kbuf(req, 0); - req->cqe.flags |= io_put_kbuf(req, issue_flags); - } io_req_task_complete(req, ts); } diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 7fd7dbb211d6..0a48e6acd0b2 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -72,10 +72,7 @@ static void io_timeout_complete(struct io_kiocb *req, struct io_tw_state *ts) struct io_ring_ctx *ctx = req->ctx; if (!io_timeout_finish(timeout, data)) { - bool filled; - filled = io_fill_cqe_req_aux(req, ts->locked, -ETIME, - IORING_CQE_F_MORE); - if (filled) { + if (io_fill_cqe_req_aux(req, true, -ETIME, IORING_CQE_F_MORE)) { /* re-arm timer */ spin_lock_irq(&ctx->timeout_lock); list_add(&timeout->list, ctx->timeout_list.prev); @@ -301,7 +298,6 @@ int io_timeout_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) static void io_req_task_link_timeout(struct io_kiocb *req, struct io_tw_state *ts) { - unsigned issue_flags = ts->locked ? 0 : IO_URING_F_UNLOCKED; struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout); struct io_kiocb *prev = timeout->prev; int ret = -ENOENT; @@ -313,7 +309,7 @@ static void io_req_task_link_timeout(struct io_kiocb *req, struct io_tw_state *t .data = prev->cqe.user_data, }; - ret = io_try_cancel(req->task->io_uring, &cd, issue_flags); + ret = io_try_cancel(req->task->io_uring, &cd, 0); } io_req_set_res(req, ret ?: -ETIME, 0); io_req_task_complete(req, ts); diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 759f919b14a9..4614ce734fee 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -87,13 +87,9 @@ EXPORT_SYMBOL_GPL(io_uring_cmd_mark_cancelable); static void io_uring_cmd_work(struct io_kiocb *req, struct io_tw_state *ts) { struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); - unsigned issue_flags = IO_URING_F_UNLOCKED; - /* locked task_work executor checks the deffered list completion */ - if (ts->locked) - issue_flags = IO_URING_F_COMPLETE_DEFER; - - ioucmd->task_work_cb(ioucmd, issue_flags); + /* task_work executor checks the deffered list completion */ + ioucmd->task_work_cb(ioucmd, IO_URING_F_COMPLETE_DEFER); } void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, diff --git a/io_uring/waitid.c b/io_uring/waitid.c index 77d340666cb9..6362ec20abc0 100644 --- a/io_uring/waitid.c +++ b/io_uring/waitid.c @@ -118,7 +118,7 @@ static int io_waitid_finish(struct io_kiocb *req, int ret) static void io_waitid_complete(struct io_kiocb *req, int ret) { struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid); - struct io_tw_state ts = { .locked = true }; + struct io_tw_state ts = {}; /* anyone completing better be holding a reference */ WARN_ON_ONCE(!(atomic_read(&iw->refs) & IO_WAITID_REF_MASK)); From patchwork Mon Mar 18 22:00:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595952 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F8325A79D; Mon, 18 Mar 2024 22:02:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799345; cv=none; b=tkQ3iPzPIOvAY2wVH0MNW9mlwDMBpecgp7w18gwvHoo3hreFC3CL9govTrbM0IJDPtXEVuWsTHa9VfAB5RbPY5kMeuszJU9TUZeGh+1pVjtapfuQY4WrgQqsKagcMz+/0Mzc/iqaoNTOjtPGeXAsxvXxPEtoSHOtYVTPcL0o3i0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799345; c=relaxed/simple; bh=7VqekwisfeZVEBvPctdmASLNCv3mAxskHtDDVKJbF80=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=urvZjJR46Pzq5Gr0mA3f0rWdF+GxYJ4v3w9We5aQLWpLtb9UtAPE1PGPoqygn3PvuztIjB8UhtNfyLX0ZHTpQp9R/yaaTnAF3zp6y3RJ2oIuBrvGXkMFoMcxa00U9cIPSZlMBlkCzF/mtyARFyYUcgE/JAMQOCnb2UQBky1+qy4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=J1SgDkc8; arc=none smtp.client-ip=209.85.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="J1SgDkc8" Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-4141156f245so9533615e9.2; Mon, 18 Mar 2024 15:02:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799341; x=1711404141; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xgD3bsMP0S2gpC+HqO1lBHKa+Q6P0EutKXjfqR3KaNY=; b=J1SgDkc82kXlx2h5Eo5I/ECWZXUPqNNPzwhQPeT3K4Vezt41VC9E5KNQp2K6TETi9j FvpNcS1z9CeXIcLdlMq+/b8jGTzJEu9TYD6cYe+vADzlFZX/sPDBGdvtXRTtjudUEC36 uH+rdkLf+WlEB1NEX3PpFCcpCPBmSKVJdrXkSGtDnyvtZJr36F7UkxRfobfjHfNwfstN FS3eJt8MPneZ+h1x5WV4Wjjdq60miOl5opeXfA5fIfTDJqsc98y+vix0bxmDxw7x4Y/A ZsH2Vuij/3XMVhtGW5zJWijC03xonlo+oUNKC/tay4WSP+DkbYFAUoXLIbxeVdgs6Taa wb9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799341; x=1711404141; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xgD3bsMP0S2gpC+HqO1lBHKa+Q6P0EutKXjfqR3KaNY=; b=P+r+2Zc3hOKG54s0+C7IlQ3qK0JxTZtLptq7550Xwppt1uQ8Mp7pRuLpriz80zR09r 4mzxqPJUv6tlKCIOcIgjvKFLUW4JALODgHtj77Xp6FtgiJgbzkPmQWiK7ba5taV5gU3N q/ATwD+R363aPFNELHmnlx8lALuifFGxcRv9XdkEg5877025LD6CrT/C1t2MV+Qx/SV2 e0BmAYRDbOmzj7dj1n1f8FhWiBXf8hlrgjmtiUTglrMyDuV1AYXIN+NFH0Jsewi8Wxia fdbYPW7j963ltD8D4O9UibjOWagt+l202i2c0AaaFsmXtXddcZJyLKaC9kwQ7OtdiqWm VEnQ== X-Gm-Message-State: AOJu0YzpCk255uQoW9jPg16Bi48rV7Dzk8esAv4vOqf2XfOOKHHTLTSD W2Lru3Nh5ak3cMjUy9nQVHzCI0CaS6hr1wQtdeiCSMlLytR965m4OtHIevhz X-Google-Smtp-Source: AGHT+IGpVFNIVG/JdV3D5UBVImz15cictZpMO91uVMKgkFVvNoy6LrrsbNNbbrj9PXZ41KJgK/ZOMw== X-Received: by 2002:a5d:4d82:0:b0:33e:c7e3:9024 with SMTP id b2-20020a5d4d82000000b0033ec7e39024mr8761668wru.58.1710799341471; Mon, 18 Mar 2024 15:02:21 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:20 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 09/13] io_uring: refactor io_fill_cqe_req_aux Date: Mon, 18 Mar 2024 22:00:31 +0000 Message-ID: <93423d106c33116c7d06bf277f651aa68b427328.1710799188.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The restriction on multishot execution context disallowing io-wq is driven by rules of io_fill_cqe_req_aux(), it should only be called in the master task context, either from the syscall path or in task_work. Since task_work now always takes the ctx lock implying IO_URING_F_COMPLETE_DEFER, we can just assume that the function is always called with its defer argument set to true. Kill the argument. Also rename the function for more consistency as "fill" in CQE related functions was usually meant for raw interfaces only copying data into the CQ without any locking, waking the user and other accounting "post" functions take care of. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 16 +++------------- io_uring/io_uring.h | 2 +- io_uring/net.c | 6 ++---- io_uring/poll.c | 3 +-- io_uring/rw.c | 4 +--- io_uring/timeout.c | 2 +- 6 files changed, 9 insertions(+), 24 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9e8afc006fc9..9a4cc46582b2 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -907,40 +907,30 @@ static void __io_flush_post_cqes(struct io_ring_ctx *ctx) state->cqes_count = 0; } -static bool __io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, - bool allow_overflow) +bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) { bool filled; io_cq_lock(ctx); filled = io_fill_cqe_aux(ctx, user_data, res, cflags); - if (!filled && allow_overflow) + if (!filled) filled = io_cqring_event_overflow(ctx, user_data, res, cflags, 0, 0); io_cq_unlock_post(ctx); return filled; } -bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) -{ - return __io_post_aux_cqe(ctx, user_data, res, cflags, true); -} - /* * A helper for multishot requests posting additional CQEs. * Should only be used from a task_work including IO_URING_F_MULTISHOT. */ -bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags) +bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) { struct io_ring_ctx *ctx = req->ctx; u64 user_data = req->cqe.user_data; struct io_uring_cqe *cqe; lockdep_assert(!io_wq_current_is_worker()); - - if (!defer) - return __io_post_aux_cqe(ctx, user_data, res, cflags, false); - lockdep_assert_held(&ctx->uring_lock); if (ctx->submit_state.cqes_count == ARRAY_SIZE(ctx->completion_cqes)) { diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 7f921daae9c3..460290e1bdec 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -67,7 +67,7 @@ int io_run_task_work_sig(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); -bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags); +bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages); diff --git a/io_uring/net.c b/io_uring/net.c index 1e7665ff6ef7..ed798e185bbf 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -706,8 +706,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, * receive from this socket. */ if ((req->flags & REQ_F_APOLL_MULTISHOT) && !mshot_finished && - io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, - *ret, cflags | IORING_CQE_F_MORE)) { + io_req_post_cqe(req, *ret, cflags | IORING_CQE_F_MORE)) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); int mshot_retry_ret = IOU_ISSUE_SKIP_COMPLETE; @@ -1428,8 +1427,7 @@ int io_accept(struct io_kiocb *req, unsigned int issue_flags) if (ret < 0) return ret; - if (io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, - ret, IORING_CQE_F_MORE)) + if (io_req_post_cqe(req, ret, IORING_CQE_F_MORE)) goto retry; io_req_set_res(req, ret, 0); diff --git a/io_uring/poll.c b/io_uring/poll.c index 8901dd118e50..5d55bbf1de15 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -322,8 +322,7 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts) __poll_t mask = mangle_poll(req->cqe.res & req->apoll_events); - if (!io_fill_cqe_req_aux(req, true, mask, - IORING_CQE_F_MORE)) { + if (!io_req_post_cqe(req, mask, IORING_CQE_F_MORE)) { io_req_set_res(req, mask, 0); return IOU_POLL_REMOVE_POLL_USE_RES; } diff --git a/io_uring/rw.c b/io_uring/rw.c index c7f9246ff508..35216e8adc29 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -955,9 +955,7 @@ int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags) cflags = io_put_kbuf(req, issue_flags); rw->len = 0; /* similarly to above, reset len to 0 */ - if (io_fill_cqe_req_aux(req, - issue_flags & IO_URING_F_COMPLETE_DEFER, - ret, cflags | IORING_CQE_F_MORE)) { + if (io_req_post_cqe(req, ret, cflags | IORING_CQE_F_MORE)) { if (issue_flags & IO_URING_F_MULTISHOT) { /* * Force retry, as we might have more data to diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 0a48e6acd0b2..3458ca550b83 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -72,7 +72,7 @@ static void io_timeout_complete(struct io_kiocb *req, struct io_tw_state *ts) struct io_ring_ctx *ctx = req->ctx; if (!io_timeout_finish(timeout, data)) { - if (io_fill_cqe_req_aux(req, true, -ETIME, IORING_CQE_F_MORE)) { + if (io_req_post_cqe(req, -ETIME, IORING_CQE_F_MORE)) { /* re-arm timer */ spin_lock_irq(&ctx->timeout_lock); list_add(&timeout->list, ctx->timeout_list.prev); From patchwork Mon Mar 18 22:00:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595953 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 327D759B55; Mon, 18 Mar 2024 22:02:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799347; cv=none; b=RCL+H/zAZqyi1iIvn3U89q/V73LPJIloHbHGRPPNovdNCqg/KrN3XbV9LiAZv0dWRJ3btDG5M2NkmFV7kVyBGjwj2opUlmxQGyAachDGU7i4/PkGI8s+PFUwgJdItR0gQ+en8q+gRyawjLMDj/g+b5AnR5IcXDWa2DO1pDyAg6Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799347; c=relaxed/simple; bh=6DyMkqr75yfWeDIM3662oU67xNgT52unN6qI7wySI0I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sslbehndoyfQIjToI+GWBlaCjDeiYn20mLDHGQihPch1Z8f4KUSzi+ptjmlLuIv1LLqCR11JyinGaGzSU1JJrYIEGAJbj+/1xERCwgBVF4HkrOuLU7SzVcO0NjwgAaLO9Je7XZMi+JVFgvNeAdXPtjFbIa9mVmrkPVvKeHnhj20= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PWDpkAFZ; arc=none smtp.client-ip=209.85.128.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PWDpkAFZ" Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-414645c349dso172205e9.1; Mon, 18 Mar 2024 15:02:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799343; x=1711404143; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jSNiymRZGIzeU7c1VJWT/0qmrS71IrYaJn9rdaLg9SE=; b=PWDpkAFZkrV5+89jlUe2OO3wpbpVGJcS0zwJbbPh07msqQyxVym6tTauQGnxE4eor8 1ZQozeWv00aqSQhxahlB2EVoq2voICcM27xFyZhobHUCWu3bszZjAIG3AgeKp/nQXDMC kCsl7JZaDoIpbadGA1Dsq1C+xLeIw+y/mxz+0Bzy4o6qBYnlUCjte58AUbLh7T50JXz2 Jb21XZMbywZUkeaknAUX/PXqaLRcp7djmg9fS3jbu601awkUUAP/3leTUkzLVBxT6pDy 8RV8sW7txQXAOsrh42S/3b/QCO91K0CGOamL/GnlLvaVYdfgRGYmMNo23uWJL43RWVb8 JEQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799343; x=1711404143; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jSNiymRZGIzeU7c1VJWT/0qmrS71IrYaJn9rdaLg9SE=; b=KiWn9JRV0L1xlMOtT9TJky2dxKNyrW9OxSe6F8vTAUYB7s+YqypL6ibWgt1ka6kbpb KaEv2KvCEFbF6FVuj5qYrRaf6rSP5XR02IfZ8jndq57eIUnAJzF5+fc2VFlMV9VO8R+L hwduEIVMgb+LvJ/+DhPSSnD6p+GRcmZ8qL76c3ti1EiWY6VORnmzlrCHicgj+hNGDzlR 1iDvj7qb8hb6wnhHBsfmThlYh4PfhSCEh7qQdlvnVSD6ouHf66Vk2NRDRRFYsQx/ZI0j iPHG3AVDiS2WLN67LAyWpVn9+kb+MyVaQLgIVJEZ+N7nJcOsACXvFlckPtuVpTU20WBt NaGg== X-Gm-Message-State: AOJu0YxDbkA+8lOIo5g3UpYjxYB2gikQ0sQKCLHdpS5Dcw0CQAhtfp6h FsLlI8KSCwNul+OAlO2WioCb/4C/PUb+bX3TqNNgaor2yaWzQnJqaJoA4Gv2 X-Google-Smtp-Source: AGHT+IGzyT+9CzTPfMvaQYGC94NAByYBy4IWyhF8w6H1tBlBz/NHLbPzKMrEoZTJqtGZ9317QQNzwg== X-Received: by 2002:adf:c98f:0:b0:33e:48dd:2927 with SMTP id f15-20020adfc98f000000b0033e48dd2927mr811748wrh.18.1710799342833; Mon, 18 Mar 2024 15:02:22 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:21 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 10/13] io_uring: get rid of intermediate aux cqe caches Date: Mon, 18 Mar 2024 22:00:32 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_post_aux_cqe(), which is used for multishot requests, delays completions by putting CQEs into a temporary array for the purpose completion lock/flush batching. DEFER_TASKRUN doesn't need any locking, so for it we can put completions directly into the CQ and defer post completion handling with a flag. That leaves !DEFER_TASKRUN, which is not that interesting / hot for multishot requests, so have conditional locking with deferred flush for them. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 +- io_uring/io_uring.c | 62 +++++++--------------------------- io_uring/io_uring.h | 2 +- 3 files changed, 15 insertions(+), 52 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 5a2afbc93887..ea7e5488b3be 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -205,6 +205,7 @@ struct io_submit_state { bool plug_started; bool need_plug; + bool cq_flush; unsigned short submit_nr; unsigned int cqes_count; struct blk_plug plug; @@ -342,8 +343,6 @@ struct io_ring_ctx { unsigned cq_last_tm_flush; } ____cacheline_aligned_in_smp; - struct io_uring_cqe completion_cqes[16]; - spinlock_t completion_lock; /* IRQ completion list, under ->completion_lock */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9a4cc46582b2..f5e2b5bef10f 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -630,6 +630,12 @@ static inline void __io_cq_lock(struct io_ring_ctx *ctx) spin_lock(&ctx->completion_lock); } +static inline void __io_cq_unlock(struct io_ring_ctx *ctx) +{ + if (!ctx->lockless_cq) + spin_unlock(&ctx->completion_lock); +} + static inline void io_cq_lock(struct io_ring_ctx *ctx) __acquires(ctx->completion_lock) { @@ -882,31 +888,6 @@ static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, return false; } -static void __io_flush_post_cqes(struct io_ring_ctx *ctx) - __must_hold(&ctx->uring_lock) -{ - struct io_submit_state *state = &ctx->submit_state; - unsigned int i; - - lockdep_assert_held(&ctx->uring_lock); - for (i = 0; i < state->cqes_count; i++) { - struct io_uring_cqe *cqe = &ctx->completion_cqes[i]; - - if (!io_fill_cqe_aux(ctx, cqe->user_data, cqe->res, cqe->flags)) { - if (ctx->lockless_cq) { - spin_lock(&ctx->completion_lock); - io_cqring_event_overflow(ctx, cqe->user_data, - cqe->res, cqe->flags, 0, 0); - spin_unlock(&ctx->completion_lock); - } else { - io_cqring_event_overflow(ctx, cqe->user_data, - cqe->res, cqe->flags, 0, 0); - } - } - } - state->cqes_count = 0; -} - bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) { bool filled; @@ -927,31 +908,16 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) { struct io_ring_ctx *ctx = req->ctx; - u64 user_data = req->cqe.user_data; - struct io_uring_cqe *cqe; + bool posted; lockdep_assert(!io_wq_current_is_worker()); lockdep_assert_held(&ctx->uring_lock); - if (ctx->submit_state.cqes_count == ARRAY_SIZE(ctx->completion_cqes)) { - __io_cq_lock(ctx); - __io_flush_post_cqes(ctx); - /* no need to flush - flush is deferred */ - __io_cq_unlock_post(ctx); - } - - /* For defered completions this is not as strict as it is otherwise, - * however it's main job is to prevent unbounded posted completions, - * and in that it works just as well. - */ - if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq)) - return false; - - cqe = &ctx->completion_cqes[ctx->submit_state.cqes_count++]; - cqe->user_data = user_data; - cqe->res = res; - cqe->flags = cflags; - return true; + __io_cq_lock(ctx); + posted = io_fill_cqe_aux(ctx, req->cqe.user_data, res, cflags); + ctx->submit_state.cq_flush = true; + __io_cq_unlock_post(ctx); + return posted; } static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) @@ -1545,9 +1511,6 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) struct io_wq_work_node *node; __io_cq_lock(ctx); - /* must come first to preserve CQE ordering in failure cases */ - if (state->cqes_count) - __io_flush_post_cqes(ctx); __wq_list_for_each(node, &state->compl_reqs) { struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); @@ -1569,6 +1532,7 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) io_free_batch_list(ctx, state->compl_reqs.first); INIT_WQ_LIST(&state->compl_reqs); } + ctx->submit_state.cq_flush = false; } static unsigned io_cqring_events(struct io_ring_ctx *ctx) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 460290e1bdec..5119265a11c2 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -156,7 +156,7 @@ static inline void io_req_task_work_add(struct io_kiocb *req) static inline void io_submit_flush_completions(struct io_ring_ctx *ctx) { if (!wq_list_empty(&ctx->submit_state.compl_reqs) || - ctx->submit_state.cqes_count) + ctx->submit_state.cq_flush) __io_submit_flush_completions(ctx); } From patchwork Mon Mar 18 22:00:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595954 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E3A1C5A7BC; Mon, 18 Mar 2024 22:02:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799348; cv=none; b=g8F1O5UDgys7XM6jrUOEHYcG+7zbBz8f8/sJW5bVX05bPPycO445zXAyfY9N1L3TfCM2ouzvaZmO2Gwgw9tq2m8Dnk7CT/Hwt5O/huQG2dv6NJWWfxChbzo5x1jbGtsiQ75YflQadSRGqkJ9bo7QNq+H1wW2IyrfdTvxnwhdxaA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799348; c=relaxed/simple; bh=oqixOxrfdkDOvec1XoeBE2EM4nMZPOSKrMBoEnsSrdQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nXM54mq6XqXe5SAreygXKjUMB8SqVIn2TFg1nGALN8W0vw/1ya6Y52Vx7V1LHKHXTLF+M43vwqOEN4uk5SLe29CxRCAf3P2Edv0b2ow3XJ/NDtCnRAQZsdX74KBYrSZIWzfWXfp5D0taMMis75Oz2O27t/4WZAr1K1rGq2E6ml4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ciUfa2gT; arc=none smtp.client-ip=209.85.128.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ciUfa2gT" Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-41464711dc8so121745e9.1; Mon, 18 Mar 2024 15:02:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799344; x=1711404144; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A5jfmKec2lF2nW2r2UOqdUesI1xPrnav8F/FOUhv5is=; b=ciUfa2gTKfMmshEwuQUR2EgFbjXx8zFuL4tmaOasRZ9mevv5UoBIw2aZ22t5+VPpcv 08OzWgvLTQbAPSJOkHi5Y617tUMJhWkT9ualuPPGVAuehlcADUJyOnJwpX0jjND4sbMK Mq/rn8godJwcdcWxVl2w/O7diC45ZZuEEGfzU8wLl86GkuYq8BDB3lNMIpF3WgFizNNe yf48OdYALU+exadz5NLfsd19hIE1UPn97XGmhV6SIZO3NXFqZolhzIMPe9IRnvPg5Sa3 YWJHqHuiZRq929U8tPLYooN34EfIxMdXkjNVX9R70SaJyaxyNVuJSsUamdHR5UNZpAJJ kLxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799344; x=1711404144; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A5jfmKec2lF2nW2r2UOqdUesI1xPrnav8F/FOUhv5is=; b=nkI0c5VqagiGpY5TwwyXLdijlmp4C06hVBEIAIuCIvalY40hZjz88gn0W/mWjicw2E q2OmoUaYxoQL6SFphImsittgzn343FBh0aWAalLjDITlDkj7JGWuQmpeTt59M0+pm26/ wMoD/5R1XLydDxOoylYIGzcX7HEexj330Q7xBLlXH5nfMyA9vgSa1MaOJ3JwlLsMuM90 AQ/ei0Dhoemw650IetyZOfJNo2ICrX0iipabR7G+UoJcl/7myoMHW0AwZfEsCohY7pSj N62x0NF6mJTHruZbVenGiVItxTh5yxAVwShwsd4xEZq/zfyWgi+UP1bKQDJ9yF0uPYOy qeRg== X-Gm-Message-State: AOJu0YzdBhb9M2JkXZICNSw37Wb56bOXllcKFSTyPIE4AxfyNluKutS5 /sjbk2j/hy7ucWhK4W9GCt92mqJUD1qNWUPHWrE/sl2YKaDljO6ptmi0BgnW X-Google-Smtp-Source: AGHT+IEEEHA0oXQ+dQrx9Ww9y2adlnS8rJNy7I1ljbMdN0OV9Yx1K5r/hrjClKbDyMet04dt1eTHVg== X-Received: by 2002:adf:f406:0:b0:33e:3555:bb57 with SMTP id g6-20020adff406000000b0033e3555bb57mr538215wro.42.1710799344202; Mon, 18 Mar 2024 15:02:24 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:23 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 11/13] io_uring: remove current check from complete_post Date: Mon, 18 Mar 2024 22:00:33 +0000 Message-ID: <24ec27f27db0d8f58c974d8118dca1d345314ddc.1710799188.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 task_work execution is now always locked, and we shouldn't get into io_req_complete_post() from them. That means that complete_post() is always called out of the original task context and we don't even need to check current. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index f5e2b5bef10f..077c65757281 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -972,7 +972,7 @@ void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; - if (ctx->task_complete && ctx->submitter_task != current) { + if (ctx->task_complete) { req->io_task_work.func = io_req_task_complete; io_req_task_work_add(req); } else if (!(issue_flags & IO_URING_F_UNLOCKED) || From patchwork Mon Mar 18 22:00:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595955 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5946A5B1EC; Mon, 18 Mar 2024 22:02:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799348; cv=none; b=jNMEIzxbLK03ZPBH3I6vheVr85R05Rzyt85kfe01lqys+55nVtJWg2RStfQ6PdHuf/+/UEhuhSB9vVhHewQWEZ9fWLjXSNJsTJ6yyCaCGluL2cRdG7RMUgji7/qb0+kOYiIlmPy7AhOfn6ZAqwezgursnDkMflzm4nyVPCgIT1s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799348; c=relaxed/simple; bh=lR20SPFN7AEYF4hlpLAqRs9SBH6pj/xZec3GIigfjiw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Rt9QnSnB+4SE+a5hm2+9L5N9SL4tcAiB/LNBeazJ81nzuhg8ELRfs7AJyehwx4codoMBKwU9f96fyS/mXv/NlNFp2L9KxNlV1mseBpMC5jHTMLIq/BuKv8GIlRYlcofGvCtfo+am5SDC78l15k23gnQIjnDbOOxj8Xla4rMnkG0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YaHDtb9r; arc=none smtp.client-ip=209.85.221.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YaHDtb9r" Received: by mail-wr1-f52.google.com with SMTP id ffacd0b85a97d-33ed5b6bf59so1985719f8f.0; Mon, 18 Mar 2024 15:02:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799345; x=1711404145; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J2smzkL0ccPd6EW+XOKsfNYEGXriaM8JzkEqntm67yY=; b=YaHDtb9r0iAg0/QGRtFi8wPztOhvk+uXcCkWv5I43CdhyDGOusUeLvi2EH2/qP/b2f Duq98pz6wbwYHNZY7oaDDTX1Rht7lX98n8cvQVrKrJPvu4Vae8Fk+Iq2eMYdhmendKKq yVVIYKwFjrwg0SrKz8b+z9O0YyX5iHBIq2zGBGjb9sB73smrlPvMiOqW4+3PbdorVVm/ bxt1+lGPV6mbslUNv8cton0FBjPHXoIVEwsnk1ZOBsZ/8PTZVOhnHhuCeb00xdIBIumR 1K0o7+fnpEgqVg+Cw6aPPqUolNk3QEbU98hBb4KMWbQeWAD/A6Yeb0Q/tKSizgLdrwoT p4ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799345; x=1711404145; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J2smzkL0ccPd6EW+XOKsfNYEGXriaM8JzkEqntm67yY=; b=ptWlkhurhNlyMew2MnWOeja0/L8bXTJjrJnUxoVgdIVS+UooZK2FX4r/dfHTjAJBxh kRlec8fz/zJbTIfBogxZygLNqAdsxvMhaDA0l6FP45ZjhKEX2TbBkr0igBVLUkhTO8vR eXr7sqNY+MqoBMDO80e0FiqrgiUtDq5fn03q5oTH2oNdnsx9Q9WG3+qdsBRv1vLIWc/A 850OK2yaFZ9KOePU/YBPvM9jbq4hxoO13Pb4oLkHTspFNtRXz2A6RvNbafAdKYn7w3+/ 2YZuNWGBjuzUnRbhxc7MCet5n2S7jWobMfr/Gg6u8tjJOTq1uLSPTmJGUgHPRMLt2fdC hvYA== X-Gm-Message-State: AOJu0YyfO0c6GD/v4LLeXBmQnXRUYOYproLUV4s/HMR98lgkEKJKsHoY 6ikF7GBvqlsI905HFhKGBeVnLr3+4/XtrC77rdvJuu9zJSxLw+MryZ2bb0Of X-Google-Smtp-Source: AGHT+IHt1DQjEAjH4adUYqb6vaerxNfUMIA3bCl+8k9bzEYXFHUTDdCMfvko51IsAZ6QywtQSovnPw== X-Received: by 2002:adf:f3d2:0:b0:33d:b376:8a07 with SMTP id g18-20020adff3d2000000b0033db3768a07mr7879097wrp.8.1710799345172; Mon, 18 Mar 2024 15:02:25 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:24 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 12/13] io_uring: refactor io_req_complete_post() Date: Mon, 18 Mar 2024 22:00:34 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Make io_req_complete_post() to push all IORING_SETUP_IOPOLL requests to task_work, it's much cleaner and should normally happen. We couldn't do it before because there was a possibility of looping in complete_post() -> tw -> complete_post() -> ... Also, unexport the function and inline __io_req_complete_post(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 29 +++++++++++------------------ io_uring/io_uring.h | 1 - 2 files changed, 11 insertions(+), 19 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 077c65757281..b07ab65591bf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -920,11 +920,21 @@ bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) return posted; } -static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) +static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; struct io_rsrc_node *rsrc_node = NULL; + /* + * Handle special CQ sync cases via task_work. DEFER_TASKRUN requires + * the submitter task context, IOPOLL protects with uring_lock. + */ + if (ctx->task_complete || (ctx->flags & IORING_SETUP_IOPOLL)) { + req->io_task_work.func = io_req_task_complete; + io_req_task_work_add(req); + return; + } + io_cq_lock(ctx); if (!(req->flags & REQ_F_CQE_SKIP)) { if (!io_fill_cqe_req(ctx, req)) @@ -968,23 +978,6 @@ static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) } } -void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) -{ - struct io_ring_ctx *ctx = req->ctx; - - if (ctx->task_complete) { - req->io_task_work.func = io_req_task_complete; - io_req_task_work_add(req); - } else if (!(issue_flags & IO_URING_F_UNLOCKED) || - !(ctx->flags & IORING_SETUP_IOPOLL)) { - __io_req_complete_post(req, issue_flags); - } else { - mutex_lock(&ctx->uring_lock); - __io_req_complete_post(req, issue_flags & ~IO_URING_F_UNLOCKED); - mutex_unlock(&ctx->uring_lock); - } -} - void io_req_defer_failed(struct io_kiocb *req, s32 res) __must_hold(&ctx->uring_lock) { diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 5119265a11c2..f694e7e6fb25 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -65,7 +65,6 @@ bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow); void io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); -void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); From patchwork Mon Mar 18 22:00:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13595956 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB0AF5B1EE; Mon, 18 Mar 2024 22:02:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799350; cv=none; b=R1GbibO6plblFxowFpvmLa3nJYnyJyAkpcz6ZELmowuTFooNc8BsOOoo9xBiJB8a/6gvQpZ+Y/PMY/oR/pHBhkN3vtA7FO5A8ReWra6VQwrMERq9srLMcAAsInHm9HxProUPsUZJgtLsPO6rb3dvut8ky6VrF06+ssaSGfUSdpo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799350; c=relaxed/simple; bh=hfys4+yftUF/Eg+JX7tAPxBW3SY311Y0Eg1rWS0rdiU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XS1k5BbflT4xXoSOL8GeRGMIbNpSryMgqIh6OEUFc1l1dtiAxKGkphhOsa5SsyIPzYEJA05lO0Y3WQL4O8/vqoAwhfRWsccm5tANqN5SLGEJFTRqD/VCgsbZI6wBuIPC7Fmqivy2Em/SPZwd+aljJdVNasNe1c4hUBUNdnMPWlc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=COw/2wNr; arc=none smtp.client-ip=209.85.128.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="COw/2wNr" Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-4140aadfe2eso15002775e9.1; Mon, 18 Mar 2024 15:02:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799346; x=1711404146; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R9xszpFpC2h6c5zr4j/9+f30Fztzglf7ofzCjanzBI4=; b=COw/2wNr477/PPoU56s+BOaiP4+7UYUU/lPoYTv7VhV5MyzNMa6n5zBfWmMh95jNB3 I0hXaD8iC3/Zeo7Z4XGcPBhmg/YacdEZNYUIO1IRvd0fueoGQK2sP4LAQ8Thb4DCnj9K 90oT16ULPVMAn3+sm439LSpESK/T4eTaN7slyk5uxvyJFErAShDWUMEf/zs/Hu5PQ5r3 ANczJoeAhVBfjKFMJhRP1io31vngnS9PZpBSRlbWtyvXs3E2GapwnkUEqYF0/dZ2dglP 6aH5Huq+MBinXqCWO7G1ROWnb+QQ5H8F1DPH9yaF+oqFKwezLR99o6mNiTSKGfT/LfEK 8tRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799346; x=1711404146; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R9xszpFpC2h6c5zr4j/9+f30Fztzglf7ofzCjanzBI4=; b=wv+S/N6IWoBuveQpoK5BQ1SzCCDuC3nRIW5WUE5VvA0LouhsdKqDGby2sLI/zy/NRT Sh04TwpH+YFNqg8UoVbbpanDQvb7+uFTLpUFZr69GbF8I4bzrsoQ/yfNe16+RXhL7ttF oqDAkNGHst6q+F8vKgJAf3z+qSm0TdFKXxwJHl4vUr7RQnriybkJvhZPH10/QZtMmqzw bpPklVDZgnhJ+Kxq/FCJSC6sbZZ9NQ1YO+jmuLUsM3+jH/bMRUJnpfAGATX5SGKsQVYl G6yY/kINTd1eGsIlTpShP+8G1OUPYltFid8z4UoxTlPRlWu6Z90yRfSvkTAJlNV9W2QX UQbA== X-Gm-Message-State: AOJu0Yzy962stLkOZxTNaY4ANPUb2j3T/c3RXEhwaSkg7F411cprg07V STxg5tGX6MOLXZZJUcMFPoqFvpq9h+PH3qGUMIaqvIz/XhZtI6KoVgytgIEY X-Google-Smtp-Source: AGHT+IHyu+4co3oMLwzRvcdftxQwFCQ6jGwEcrCYjg4gnVrW7bX1akvxWlYEFdnVAn/FqcLeh5L23Q== X-Received: by 2002:a05:6000:2c4:b0:33e:c8a1:148b with SMTP id o4-20020a05600002c400b0033ec8a1148bmr12050383wry.42.1710799346617; Mon, 18 Mar 2024 15:02:26 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:25 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 13/13] io_uring: clean up io_lockdep_assert_cq_locked Date: Mon, 18 Mar 2024 22:00:35 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move CONFIG_PROVE_LOCKING checks inside of io_lockdep_assert_cq_locked() and kill the else branch. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.h | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index f694e7e6fb25..bae8c1e937c1 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -119,9 +119,9 @@ enum { void io_eventfd_ops(struct rcu_head *rcu); void io_activate_pollwq(struct io_ring_ctx *ctx); -#if defined(CONFIG_PROVE_LOCKING) static inline void io_lockdep_assert_cq_locked(struct io_ring_ctx *ctx) { +#if defined(CONFIG_PROVE_LOCKING) lockdep_assert(in_task()); if (ctx->flags & IORING_SETUP_IOPOLL) { @@ -140,12 +140,8 @@ static inline void io_lockdep_assert_cq_locked(struct io_ring_ctx *ctx) else lockdep_assert(current == ctx->submitter_task); } -} -#else -static inline void io_lockdep_assert_cq_locked(struct io_ring_ctx *ctx) -{ -} #endif +} static inline void io_req_task_work_add(struct io_kiocb *req) {