From patchwork Fri Mar 8 13:55:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13586853 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D39343FBB7 for ; Fri, 8 Mar 2024 13:58:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709906331; cv=none; b=BJRBWJBuRHDoBq/Euqq6kaatW7JffAMocjWGhGwSnO+hw5DolxLtgooU6rC9TeHXKZJOeHuQXXEPNJXQHVaVhpgwFlA3c3YZ7VbS+3XWDwIY8UC1zjNozsVuzAv0CLfB6bdfzPUeFBpbZ0/FvFmaB0vPTn3jsdsYRK1XFF3gKNg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709906331; c=relaxed/simple; bh=kLBOIJPjNWZ/WKku1IIFFux9O3I4hS7Otz3AeffrseM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H3WcmjvtEqIqL8hy5Vj5D1egtM60dhfnwiAIp7jQMp4skUkKd2WHoW9Hz3JLzZ3R1P9BSdw9gwqLnvdtfOh4z5vxp+mi07mK0ukF/cnLAeyQoSMec857xxwKE5pc9rvGgQFLq5MHMgegsgVcifz8FyUCP3/g3es6iu5g95JddDg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=B2R9P9I0; arc=none smtp.client-ip=209.85.218.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="B2R9P9I0" Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a26ed1e05c7so311654366b.2 for ; Fri, 08 Mar 2024 05:58:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709906328; x=1710511128; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NJZOeTUKvaZ9tzibdK6mZ+2xfJqhd2aZvSUwYP3TffE=; b=B2R9P9I0PO7xUxbXLJnBfGFvcVIbp9y1SCJKSb4STqTEVwAIof0cTErEHxRmQ0Ex70 yp4ZdLIGOxTqwQ3/IkIe/cuvVN30EqO9rC43pZ8vF3Nr/HBAPZjEEDf5DlV2fCtkM6KH YA/1XUdPvxxDe8Lu6ng5g9bQ/qQrxXuoGFqv97jGqJhbAGSnlEWkMXP/HbT1p15gbBzl Ke+0QbRpdc7WxttEaVG0gbeqIlGLKsqFdFM9Hc76PgIWA9iXhUbPh6PYqIv6HuWJt6IO VAxwBoNG53ukKM/BnrxmdqCLd/02cICXjCB/fccffBMQ3gK4ETmxNqDARx/u6J+vYZR0 QWIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709906328; x=1710511128; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NJZOeTUKvaZ9tzibdK6mZ+2xfJqhd2aZvSUwYP3TffE=; b=jGJV8hYDrJsxpBaPkBenT9GXnB0jrlPl2f4ZjifXG8oQhmqAQ/btgL63hlxRcgxjbM lIbr4UMONTTYROqULtjFDhyfklxpqheF+JK+MTNfOzr/2HvkgbnotJDzXbWlt2In0Asj gNlUoG9glD6gwBzgysGd1NGJgfgVuZkVLhA7r2cHjm8J9VmNijvD64Ipklv6XQGuTaHy jwmfykEKRcRlBwhFpBIkrX5r1gmayEbRbPb6krPDjDu2X+vyUY4ijd/HR7O85Xo2zTKK GyPLYI87dY24HuCvAt6MbP7z7EIKDwBWq9v0nRkR6Mv6bhPbpb65i/jK8o07Rxlr1qQa +EZw== X-Gm-Message-State: AOJu0Yz0yd9roQEyiFWmW+1l8xUZQZLL3bpo+RKSZVMw/hJNu0R7at8e erqKeod3UWUENx34KPCdsOMpWrt0Jrsvxw8LN3uxu1BtVwuFzDbjf3yn/l+zz1o= X-Google-Smtp-Source: AGHT+IE7WUfgQyBrzBqByBnm6+rLWWemHeajvmampNCKsn5B54w6l9RLPMdIygz/5sFdMLb4UkDaAQ== X-Received: by 2002:a17:906:458:b0:a45:a731:d021 with SMTP id e24-20020a170906045800b00a45a731d021mr6464183eja.33.1709906327803; Fri, 08 Mar 2024 05:58:47 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:d306]) by smtp.gmail.com with ESMTPSA id p16-20020a170906229000b00a442979e5e5sm9303189eja.220.2024.03.08.05.58.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Mar 2024 05:58:47 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 1/3] io_uring: fix mshot io-wq checks Date: Fri, 8 Mar 2024 13:55:56 +0000 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 When checking for concurrent CQE posting, we're not only interested in requests running from the poll handler but also strayed requests ended up in normal io-wq execution. We're disallowing multishots in general from io-wq, not only when they came in a certain way. Cc: stable@vger.kernel.org Fixes: 17add5cea2bba ("io_uring: force multishot CQEs into task context") Signed-off-by: Pavel Begunkov --- io_uring/net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/net.c b/io_uring/net.c index 52f0d3b735fd..d4ab4bdaf845 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -87,7 +87,7 @@ static inline bool io_check_multishot(struct io_kiocb *req, * generic paths but multipoll may decide to post extra cqes. */ return !(issue_flags & IO_URING_F_IOWQ) || - !(issue_flags & IO_URING_F_MULTISHOT) || + !(req->flags & REQ_F_APOLL_MULTISHOT) || !req->ctx->task_complete; } From patchwork Fri Mar 8 13:55:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13586854 Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 164AD40856 for ; Fri, 8 Mar 2024 13:58:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709906332; cv=none; b=VK6nxNcc7M5C3VtIFhn77mgFTFy417UIw3/tcfqUPTkYlzjx8pQMXGlTTbB0mnWL4nARUcaPJinSFMGt37gw3R42zuz1oErSWbhY2pVn5IB5BtvdhU4k6Z0tRAJxZEwQ8zmv1YYI5Og1iUw/JoL/5Hvhz1qkMo1e97b11cafP2A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709906332; c=relaxed/simple; bh=qS8MalAQoWV0g80GfTeHW22WyX+rrDzkzwvU/GsWdI4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PIVvmxpE9DYo43ENC1NWrRuw5UyxXo4uT8njwgXwL+yaJGii1t3YnDsm6JyuhIm5zvy/2KKHPvWE62qvGZqIUyz6CLP+XMlifwJZmD3B6JbJOXX6k7hhUDTirLHM/d4FDVaHJdse2fpzOrwshnfflKD5nZkGrheTw3hPNu76mUQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GNduEvSP; arc=none smtp.client-ip=209.85.208.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GNduEvSP" Received: by mail-ed1-f50.google.com with SMTP id 4fb4d7f45d1cf-567fbbd723cso1026381a12.3 for ; Fri, 08 Mar 2024 05:58:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709906329; x=1710511129; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pk//fNMLELl6slqa+qiyE5ltZQGonksP78x9CswhUW4=; b=GNduEvSPRO8DzKe8fXe6D51naH4qD+gnd5M8uUZSxX9+IZD5zSk4SqsmoTUJOyDNry Wz87FNE3N8oGq7Xr+uMPEKGBXadpEpAaMqK+UJSuyJOLxlJAvSWCZ494slAiui8GZV2p ZihbjUSqBSd73PNrauBGZpum7cbQ+Zie3sXvrOJZELrmtmG/Zj9uaXtD1QXN5cFjEbHG sW8pbzs8ssePA1F9tYG+W44h8hR96aMSvFU7wgEzfAjgJ4B2PngXK4mcEaho8Bgy0ct5 z5IMGrQW/jO1Tg9ZFZnYHRELiJ/nR3uZ5qXw2o9AY/XdhxY/AWDL4EXRslYu9Kym/mCC NWUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709906329; x=1710511129; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pk//fNMLELl6slqa+qiyE5ltZQGonksP78x9CswhUW4=; b=TQ9PG/qFx0C8XMJPfYRUB4IZPY5mas3OTBGArAuJUi2ob/xlL3HEouUd8xPH451vTI xlECIu4L2enjWKlAvwp+PGsTePLF8ANFo8ublzdfQWAHZ8QGPZqz0qgx8xbUBQPW7+GX DEP9ZdQq6f2caaD5ldRw7JB5/OUyGaFh5Nfi72dvMo7hkdTxzQb75L1wPRWLw5Y/5jbI O8uGlz+Bw3u299u/KXu0JT/0kXtA2kokUBiFKLqvPinI0miFMtTYt3I82T5W2xwdhnkX 1gGesMyV7z4ZhL5nYL8WEfWF2GMETQAZJOj9x2qIwDcXVlq4zUMLguokaU++Wrqmwho/ DoLg== X-Gm-Message-State: AOJu0Yxk1VUeZF6lN3ch9A3JCy0RIPfE4Vrb/6QwI8c9o1mp3LsCp1mf tBBYfMvUvAGtQqJ8W6Kow8uHkaYKzuKJ3CgCJFHD+u5ahDhS9DnNXgg8rsUWQhU= X-Google-Smtp-Source: AGHT+IE6Gkjq1czsvBw2dQ01+66LsVkndWdaM7rHeqtdSBrd1MJpECu6UV7GM92GCLe8uIlsW+rHHw== X-Received: by 2002:a17:906:2448:b0:a45:3308:560d with SMTP id a8-20020a170906244800b00a453308560dmr10325316ejb.71.1709906328832; Fri, 08 Mar 2024 05:58:48 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:d306]) by smtp.gmail.com with ESMTPSA id p16-20020a170906229000b00a442979e5e5sm9303189eja.220.2024.03.08.05.58.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Mar 2024 05:58:48 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 2/3] io_uring: refactor DEFER_TASKRUN multishot checks Date: Fri, 8 Mar 2024 13:55:57 +0000 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We disallow DEFER_TASKRUN multishots from running by io-wq, which is checked by individual opcodes in the issue path. We can consolidate all it in io_wq_submit_work() at the same time moving the checks out of the hot path. Suggested-by: Jens Axboe Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 20 ++++++++++++++++++++ io_uring/net.c | 21 --------------------- io_uring/rw.c | 2 -- 3 files changed, 20 insertions(+), 23 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cf2f514b7cc0..cf348c33f485 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -944,6 +944,8 @@ bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags) u64 user_data = req->cqe.user_data; struct io_uring_cqe *cqe; + lockdep_assert(!io_wq_current_is_worker()); + if (!defer) return __io_post_aux_cqe(ctx, user_data, res, cflags, false); @@ -1968,6 +1970,24 @@ void io_wq_submit_work(struct io_wq_work *work) goto fail; } + /* + * If DEFER_TASKRUN is set, it's only allowed to post CQEs from the + * submitter task context. Final request completions are handed to the + * right context, however this is not the case of auxiliary CQEs, + * which is the main mean of operation for multishot requests. + * Don't allow any multishot execution from io-wq. It's more restrictive + * than necessary and also cleaner. + */ + if (req->flags & REQ_F_APOLL_MULTISHOT) { + err = -EBADFD; + if (!io_file_can_poll(req)) + goto fail; + err = -ECANCELED; + if (io_arm_poll_handler(req, issue_flags) != IO_APOLL_OK) + goto fail; + return; + } + if (req->flags & REQ_F_FORCE_ASYNC) { bool opcode_poll = def->pollin || def->pollout; diff --git a/io_uring/net.c b/io_uring/net.c index d4ab4bdaf845..14d6bae60747 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -78,19 +78,6 @@ struct io_sr_msg { */ #define MULTISHOT_MAX_RETRY 32 -static inline bool io_check_multishot(struct io_kiocb *req, - unsigned int issue_flags) -{ - /* - * When ->locked_cq is set we only allow to post CQEs from the original - * task context. Usual request completions will be handled in other - * generic paths but multipoll may decide to post extra cqes. - */ - return !(issue_flags & IO_URING_F_IOWQ) || - !(req->flags & REQ_F_APOLL_MULTISHOT) || - !req->ctx->task_complete; -} - int io_shutdown_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_shutdown *shutdown = io_kiocb_to_cmd(req, struct io_shutdown); @@ -853,9 +840,6 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST)) return io_setup_async_msg(req, kmsg, issue_flags); - if (!io_check_multishot(req, issue_flags)) - return io_setup_async_msg(req, kmsg, issue_flags); - flags = sr->msg_flags; if (force_nonblock) flags |= MSG_DONTWAIT; @@ -951,9 +935,6 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST)) return -EAGAIN; - if (!io_check_multishot(req, issue_flags)) - return -EAGAIN; - sock = sock_from_file(req->file); if (unlikely(!sock)) return -ENOTSOCK; @@ -1403,8 +1384,6 @@ int io_accept(struct io_kiocb *req, unsigned int issue_flags) struct file *file; int ret, fd; - if (!io_check_multishot(req, issue_flags)) - return -EAGAIN; retry: if (!fixed) { fd = __get_unused_fd_flags(accept->flags, accept->nofile); diff --git a/io_uring/rw.c b/io_uring/rw.c index 7733449271f2..6f465b6b5dde 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -933,8 +933,6 @@ int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags) */ if (!io_file_can_poll(req)) return -EBADFD; - if (issue_flags & IO_URING_F_IOWQ) - return -EAGAIN; ret = __io_read(req, issue_flags); From patchwork Fri Mar 8 13:55:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13586855 Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81D9640861 for ; Fri, 8 Mar 2024 13:58:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709906333; cv=none; b=fPd6X1ZldVhju63uAR7MDNFNOBhfGah0NyRqqHZ5VA9p0sLOCVeZjgWYbVr6GdnJbLWfpHlYA1XaWZiyAsJZxql1dLimqZt6G43BtwODKKseP3B5uAWeZcuSi7/dlYdoJqFba85AFzRWLt2QlTkQpCMU9DA03bL4o9fkL4A2gjs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709906333; c=relaxed/simple; bh=khcwLyqqiVRcsZyypFi3ummptCOWvb6cl50HfX4VkXA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=c1jfZUSeXufNt4WYgMQBEX/u8GmY9X1VTctEm2fGgZYsmwLojDtMb0SKnmV+5STcyjV0I0TzYKwyaxFGfAB1N0djEsQM0i/lmYi/9LqlmnqjQXfULnw/0zJbaV3KV+++3kpu4I3RcsMf3CdOpmvS3BxGbsJ3KMDjDHrO4PYxhqk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=d7ddB5mf; arc=none smtp.client-ip=209.85.218.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="d7ddB5mf" Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-a44f2d894b7so293957066b.1 for ; Fri, 08 Mar 2024 05:58:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709906329; x=1710511129; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RwlLkpRFPLsQAbqtFkO6qlmsxqFQ5iPS8mUmYKIoc9k=; b=d7ddB5mfsqNb+FS14dErEK49mFbW4oZEgpAzYZ7oSFzwzh34D5UMcGgGubyzivvg42 uDnTn3RH5c2sedJFHP9CiMOfKEqREVhp3BDPoABa8bAhZWiuQ9lldHwdA1uiNhVlsIXy GOsvq1NTPgZ7VUKTEjXelOtyvf+dea0Cx8nuaJRhgDoD+YJD7ixPGkBHnl8RV6Myshcw XlxD4chGVpgjJL5mx+O0CKaSk7F8LNIfqzqPse5ddkmlI8vf2GOD37AvB/pK8O4XmptV Zgt6UBqnfG9+TRdzZRNJEGY7edJWEhbkDLr428eZuLaI3yg2dJsQ/El7OFADsEROYUz2 4HVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709906329; x=1710511129; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RwlLkpRFPLsQAbqtFkO6qlmsxqFQ5iPS8mUmYKIoc9k=; b=X4e5oOesKqilfk+SUd7bi7L8B3L3pZuhC4Zh8nDehoJNV9FraDK/1nm7ZfwCpOA+XV sj4ogjdmDWRAX8HnEuqPBMl6zOrVG2xMJtriwnhvYb5vbTs+kYQyoVhciQt9N86LGiSS b8cmxnYpJpUpBizoD13+en18EkpFkYdRXLguA2wdK8msoYtZDT/UzH0/rLwGzmSyR3rL Wc29wIXnxTWAslI+tA5YFmHLxvK8ZjvZlQBdfESrNmAqIa/aYM0hirBvmU/4bHI9BoY0 /x8n5KtIU2lHb74sjJJZSnQlKrVVjE3UbMw5c68DWY1SuDgw0S8kXjv315+lUfEC1tLX CBfA== X-Gm-Message-State: AOJu0YxUAK8bXAN8ALnSJ0a+JFY01HERJ8deANiNJi+Sx9UsW6zh066t UChw/T0jp5PVDFYHl1ILoIW+IdScRhVYSJ0kD5Nulk016iDlmA/d4BD1i5hAkx4= X-Google-Smtp-Source: AGHT+IH6Z+qu4DmHAVfMnFY5jvZP6EM0hu3R93QO7lpDTNOwQ1ngQq8t43DLJ+5TZs7ZnwuQhpHFYA== X-Received: by 2002:a17:906:b294:b0:a45:bf15:640b with SMTP id q20-20020a170906b29400b00a45bf15640bmr5071519ejz.70.1709906329247; Fri, 08 Mar 2024 05:58:49 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:d306]) by smtp.gmail.com with ESMTPSA id p16-20020a170906229000b00a442979e5e5sm9303189eja.220.2024.03.08.05.58.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Mar 2024 05:58:49 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 3/3] io_uring/net: dedup io_recv_finish req completion Date: Fri, 8 Mar 2024 13:55:58 +0000 Message-ID: <0e338dcb33c88de83809fda021cba9e7c9681620.1709905727.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 There are two block in io_recv_finish() completing the request, which we can combine and remove jumping. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 14d6bae60747..96808f429b7a 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -685,20 +685,12 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, if (msg->msg_inq && msg->msg_inq != -1) cflags |= IORING_CQE_F_SOCK_NONEMPTY; - if (!(req->flags & REQ_F_APOLL_MULTISHOT)) { - io_req_set_res(req, *ret, cflags); - *ret = IOU_OK; - return true; - } - - if (mshot_finished) - goto finish; - /* * Fill CQE for this receive and see if we should keep trying to * receive from this socket. */ - if (io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, + if ((req->flags & REQ_F_APOLL_MULTISHOT) && !mshot_finished && + io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, *ret, cflags | IORING_CQE_F_MORE)) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); int mshot_retry_ret = IOU_ISSUE_SKIP_COMPLETE; @@ -718,8 +710,8 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, *ret = -EAGAIN; return true; } - /* Otherwise stop multishot but use the current result. */ -finish: + + /* Finish the request / stop multishot. */ io_req_set_res(req, *ret, cflags); if (issue_flags & IO_URING_F_MULTISHOT)