From patchwork Thu Jun 23 09:34:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12891990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1869C433EF for ; Thu, 23 Jun 2022 09:35:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230144AbiFWJfJ (ORCPT ); Thu, 23 Jun 2022 05:35:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229899AbiFWJfI (ORCPT ); Thu, 23 Jun 2022 05:35:08 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D0E24925E for ; Thu, 23 Jun 2022 02:35:07 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id g18so1262584wrb.10 for ; Thu, 23 Jun 2022 02:35:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OpnL0/Edc6qwJvOc9mzgGZ0HMTdXfCTcomSoBlCvR7Q=; b=cLh03/UGM4nwBvgwSxTqLRtmwY/fAz9pHBV8NxUN7G6DTcPZmwJE6co9ZHiuSfpLFg ci1Re6DXOC8ujigAYaN0/V/dD7eJ6nwoiI1yvH/neqrsT61LL3MDEsTpPWNUMygn02CZ 8Pkuw6Kw7ZR+D/qdn6pZLbbXr/EqGzjkcMyf1Z0l7anSc6ClgWi1DSyb8TjruPo3Sm71 UUKVHTnNVc0KphSFWJCcXdMO61uzpO9CVYDp8Oblh7/Ilc1tbMhEFrX8dU0pcCQDZZa/ 1epmwNHHfRsgdVDwi2fEyAqEF9McCPl2AvnGSmALye16TFbbIv0VCwIvBQtfxr28VDzU 8KeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OpnL0/Edc6qwJvOc9mzgGZ0HMTdXfCTcomSoBlCvR7Q=; b=d32N+vHdgNto2XQBy8q3HliXjSZKbdkKGlsPtYEodl12GyzFphKn2BNmS1VXpJosZ/ 4UXLfTjSeKNr0RAVI1ctAmDEqVpyH4vQaGUQn6KkxzXUXyPODXPC5YtuzHxzkve9uLsO vlx054+tzO40ib8+g0S53Y8YfeAZ5Slr6r+59aTczCxUoAg4RQPLTLJlPjBxMuNcIiR1 gTadmJNAMm74uxuGTG5i7KjKG122zitDQHZAm9vLa+mD4KatICvmant+5QHO4p6PMQCW e7HxCz8I3agxt/y9opY0RBaA6eKMNvUX9M935VzPU2vNDmWm/NHYHXfer8B0jhfPbAtv fMlA== X-Gm-Message-State: AJIora9oH0ljJriKEcwyhLb2ii7dquFmvk2d6sCJs+UehEqP2GDyFXa2 4ZjOArN7DXae6QVg6BjUamgSP3lNaMp36eMn X-Google-Smtp-Source: AGRyM1sVV85kge9JSE+wHtTwfPT3X7SQH6Rnd5Ge7ntCCWYIPSccO9HXttWuEyI7cjTJupCEBoO7kg== X-Received: by 2002:a05:6000:1869:b0:21b:933c:7e2 with SMTP id d9-20020a056000186900b0021b933c07e2mr7314511wri.252.1655976905524; Thu, 23 Jun 2022 02:35:05 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id r21-20020a05600c35d500b003a02f957245sm2431202wmq.26.2022.06.23.02.35.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 02:35:05 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 1/6] io_uring: clean poll ->private flagging Date: Thu, 23 Jun 2022 10:34:30 +0100 Message-Id: <9a61240555c64ac0b7a9b0eb59a9efeb638a35a4.1655976119.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We store a req pointer in wqe->private but also take one bit to mark double poll entries. Replace macro helpers with inline functions for better type checking and also name the double flag. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index bd3110750cfa..210b174b155b 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -39,6 +39,22 @@ struct io_poll_table { #define IO_POLL_CANCEL_FLAG BIT(31) #define IO_POLL_REF_MASK GENMASK(30, 0) +#define IO_WQE_F_DOUBLE 1 + +static inline struct io_kiocb *wqe_to_req(struct wait_queue_entry *wqe) +{ + unsigned long priv = (unsigned long)wqe->private; + + return (struct io_kiocb *)(priv & ~IO_WQE_F_DOUBLE); +} + +static inline bool wqe_is_double(struct wait_queue_entry *wqe) +{ + unsigned long priv = (unsigned long)wqe->private; + + return priv & IO_WQE_F_DOUBLE; +} + /* * If refs part of ->poll_refs (see IO_POLL_REF_MASK) is 0, it's free. We can * bump it and acquire ownership. It's disallowed to modify requests while not @@ -306,8 +322,6 @@ static void io_poll_cancel_req(struct io_kiocb *req) io_poll_execute(req, 0, 0); } -#define wqe_to_req(wait) ((void *)((unsigned long) (wait)->private & ~1)) -#define wqe_is_double(wait) ((unsigned long) (wait)->private & 1) #define IO_ASYNC_POLL_COMMON (EPOLLONESHOT | EPOLLPRI) static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, @@ -392,7 +406,7 @@ static void __io_queue_proc(struct io_poll *poll, struct io_poll_table *pt, return; } /* mark as double wq entry */ - wqe_private |= 1; + wqe_private |= IO_WQE_F_DOUBLE; req->flags |= REQ_F_DOUBLE_POLL; io_init_poll_iocb(poll, first->events, first->wait.func); *poll_ptr = poll; From patchwork Thu Jun 23 09:34:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12891991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5171C43334 for ; Thu, 23 Jun 2022 09:35:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229899AbiFWJfJ (ORCPT ); Thu, 23 Jun 2022 05:35:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229490AbiFWJfI (ORCPT ); Thu, 23 Jun 2022 05:35:08 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 124D649264 for ; Thu, 23 Jun 2022 02:35:08 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id i10so23316698wrc.0 for ; Thu, 23 Jun 2022 02:35:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wCYu+CuA4ghhjalWpw26bcKN04Bihmyd2uysIWUIfg4=; b=B+CUgVRN3JJj2MplQmOEIjtICVX30As0R3Nbz0XzMW0WFSD/7BpNMZ4DvEcQrB2+OQ 04Bx42babkeu3tzJTd4xuYTRgnSasZeYB3I/0yldJf3wQfvy2mhzbsoxnUs5s6pzqqqg aCHMA8s313pe5e8btuypNtqCnVgKjCweRA+NydDDv1deU9bUxklmZKeE4NXrUVyB3j6A i9+FCABN/xs2Bmq+wQHUqCx30hkNuU9EFxheomYz9fZlXKJEi4VQADgjYldL/T5hPmxY krshmLLv2RjIHBoWcDeBF2W7XT9LSnMTqRXeE0CWwUIlC6KBb+hrR/FaTfwJXkGYqB8k /WCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wCYu+CuA4ghhjalWpw26bcKN04Bihmyd2uysIWUIfg4=; b=nT+zkus2JKBo6fFr9VF3Ad1C1f62MiRkx9L+cn68AAzhBAoj3K2sYbzcc0qk0jqSv6 b9OehJxEFkj3zoHShxAHX5Eif1sqvPJkUB6UC338IUFmaXoraYz8/TwpHRW2Bmx7qJI8 6fcGy4zLdQ46h6mCcchmwmUcEuvoOKUNB1UaJsQMlAq4p/UGn6hOl3Q8TJVvvc1OKgxK 7VDLWgmM/kYUrF7qgd9UVBO+YGUpkVAR9fH7nsrxq2y7Pf1Q3FT2sR7QKZcDF/x7fojI UpWYbNgI6tZeOhjta260yQWl6aNxjsIlnJK6qmsq3oYNuiNebqJ+ZqeyTXsyhkWtKGWj V1Ow== X-Gm-Message-State: AJIora8bSdMfsuKqa/sBShnXNNoU4C7b/2fivkSwtPqN45umLgLjxIYI O9qyPaL0ZXURXE9Av9pdIRrwutYBYtFvVAXK X-Google-Smtp-Source: AGRyM1vmjIuZNfxAH8/8DmivHenM6MbRmno2m/1058CZvHInInWNnKJoN1X4Z+dYWsxrrDhKACaGmA== X-Received: by 2002:adf:f411:0:b0:21b:8397:860e with SMTP id g17-20020adff411000000b0021b8397860emr7261681wro.50.1655976906402; Thu, 23 Jun 2022 02:35:06 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id r21-20020a05600c35d500b003a02f957245sm2431202wmq.26.2022.06.23.02.35.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 02:35:05 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 2/6] io_uring: remove events caching atavisms Date: Thu, 23 Jun 2022 10:34:31 +0100 Message-Id: <12efd4e15c6a90cf9e5b59807cfcb57852b51dc7.1655976119.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Remove events argument from *io_poll_execute(), it's not needed and not used. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 210b174b155b..7de8c52793cd 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -289,8 +289,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) io_req_complete_failed(req, ret); } -static void __io_poll_execute(struct io_kiocb *req, int mask, - __poll_t __maybe_unused events) +static void __io_poll_execute(struct io_kiocb *req, int mask) { io_req_set_res(req, mask, 0); /* @@ -308,18 +307,17 @@ static void __io_poll_execute(struct io_kiocb *req, int mask, io_req_task_work_add(req); } -static inline void io_poll_execute(struct io_kiocb *req, int res, - __poll_t events) +static inline void io_poll_execute(struct io_kiocb *req, int res) { if (io_poll_get_ownership(req)) - __io_poll_execute(req, res, events); + __io_poll_execute(req, res); } static void io_poll_cancel_req(struct io_kiocb *req) { io_poll_mark_cancelled(req); /* kick tw, which should complete the request */ - io_poll_execute(req, 0, 0); + io_poll_execute(req, 0); } #define IO_ASYNC_POLL_COMMON (EPOLLONESHOT | EPOLLPRI) @@ -334,7 +332,7 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, if (unlikely(mask & POLLFREE)) { io_poll_mark_cancelled(req); /* we have to kick tw in case it's not already */ - io_poll_execute(req, 0, poll->events); + io_poll_execute(req, 0); /* * If the waitqueue is being freed early but someone is already @@ -369,7 +367,7 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, else req->flags &= ~REQ_F_SINGLE_POLL; } - __io_poll_execute(req, mask, poll->events); + __io_poll_execute(req, mask); } return 1; } @@ -487,7 +485,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, req->apoll_events |= EPOLLONESHOT; ipt->error = 0; } - __io_poll_execute(req, mask, poll->events); + __io_poll_execute(req, mask); return 0; } @@ -497,7 +495,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, */ v = atomic_dec_return(&req->poll_refs); if (unlikely(v & IO_POLL_REF_MASK)) - __io_poll_execute(req, 0, poll->events); + __io_poll_execute(req, 0); return 0; } From patchwork Thu Jun 23 09:34:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12891992 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50A01C43334 for ; Thu, 23 Jun 2022 09:35:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230326AbiFWJfN (ORCPT ); Thu, 23 Jun 2022 05:35:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229490AbiFWJfM (ORCPT ); Thu, 23 Jun 2022 05:35:12 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13E2349257 for ; Thu, 23 Jun 2022 02:35:09 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id z9so10664326wmf.3 for ; Thu, 23 Jun 2022 02:35:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CHxW2CtP1hlZ1Q8f7DTNKmbOSEwU/odj12C32OaND+I=; b=VdwTizYSc6FNA4OvH/vIZXFCgjSaWyfANxSt9llOJXYt2AKrfWbkm2QysqYwU6B2wI 117Dc2/iQF8Cmm+LwheobyUYqc7S8FHChqJpPlniPlqN/rEMOSOi+HR3ZqxqD+Y6EI1x hvHI7mLOEmHIdcgbXZLnvLPIhT16qYLFYeP/UIJHl+coDk3Nu9Tv5tUSKYMOifxXiMQz DX7CD7yGBVPma9UDKzgqvD9xZacQ+ooDsdUwOtpzbuYk/5WZj2TasIZR/ZCR2z9S4PMu FQD9ZAlFQUpVpFuRZ2FzyXdvYZxGNAhJv3R1Rspg+QIthfmisf+usVPZN+ge42caCar3 Jdpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CHxW2CtP1hlZ1Q8f7DTNKmbOSEwU/odj12C32OaND+I=; b=JrxlOr9x1nczWLd0pxWF9iNeGEU4EvMYS7d/FLtlbqWqkzik4G1BO/35XA3NIUxpS8 1IRrKKfGPl+O6/tLFWtdurPa/b7lUKyJ6K7mmpIK5PqLMfr9sHfzPOPVXJhIiRMEr1V8 mFR593GaJTgzru371ULV6+4jxP9aXjUas31NQwkyFGKXLlaLBLCiWBm72ud77m4MYn10 +P74+Sg7q0ByjOuw/Htrke+6ntWKuZQ85iTzFkDYMvtqmXuVQmxraQIwftggbragJFRH niKP+H1HnB7JVqqQ5vPP27oB1n/lyLq5ED1Dk7JVIcBnHr8qAuG8raP1/b/frgLVxYt6 +bNA== X-Gm-Message-State: AJIora8FCIurtpc+PYMsg/kG5AOLbUqmzb87GFAq46ynWt9RaOCaB0lb z3LjFqd2n6UB+rz9oa0KYQggqVJulsuEQtDA X-Google-Smtp-Source: AGRyM1uMytgwG6hB8luwmYlYxjZk+L2w5gMf+smWva9Clx7A84qxLh/qUH0Nj/U0XvUt9v6/I6yrxw== X-Received: by 2002:a7b:c4d8:0:b0:39c:97ed:baa3 with SMTP id g24-20020a7bc4d8000000b0039c97edbaa3mr2981029wmk.58.1655976907377; Thu, 23 Jun 2022 02:35:07 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id r21-20020a05600c35d500b003a02f957245sm2431202wmq.26.2022.06.23.02.35.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 02:35:06 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 3/6] io_uring: add a helper for apoll alloc Date: Thu, 23 Jun 2022 10:34:32 +0100 Message-Id: <2f93282b47dd678e805dd0d7097f66968ced495c.1655976119.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Extract a helper function for apoll allocation, makes the code easier to read. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 7de8c52793cd..aef77f2a8a9a 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -508,10 +508,33 @@ static void io_async_queue_proc(struct file *file, struct wait_queue_head *head, __io_queue_proc(&apoll->poll, pt, head, &apoll->double_poll); } +static struct async_poll *io_req_alloc_apoll(struct io_kiocb *req, + unsigned issue_flags) +{ + struct io_ring_ctx *ctx = req->ctx; + struct async_poll *apoll; + + if (req->flags & REQ_F_POLLED) { + apoll = req->apoll; + kfree(apoll->double_poll); + } else if (!(issue_flags & IO_URING_F_UNLOCKED) && + !list_empty(&ctx->apoll_cache)) { + apoll = list_first_entry(&ctx->apoll_cache, struct async_poll, + poll.wait.entry); + list_del_init(&apoll->poll.wait.entry); + } else { + apoll = kmalloc(sizeof(*apoll), GFP_ATOMIC); + if (unlikely(!apoll)) + return NULL; + } + apoll->double_poll = NULL; + req->apoll = apoll; + return apoll; +} + int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) { const struct io_op_def *def = &io_op_defs[req->opcode]; - struct io_ring_ctx *ctx = req->ctx; struct async_poll *apoll; struct io_poll_table ipt; __poll_t mask = POLLPRI | POLLERR | EPOLLET; @@ -546,21 +569,10 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) } if (def->poll_exclusive) mask |= EPOLLEXCLUSIVE; - if (req->flags & REQ_F_POLLED) { - apoll = req->apoll; - kfree(apoll->double_poll); - } else if (!(issue_flags & IO_URING_F_UNLOCKED) && - !list_empty(&ctx->apoll_cache)) { - apoll = list_first_entry(&ctx->apoll_cache, struct async_poll, - poll.wait.entry); - list_del_init(&apoll->poll.wait.entry); - } else { - apoll = kmalloc(sizeof(*apoll), GFP_ATOMIC); - if (unlikely(!apoll)) - return IO_APOLL_ABORTED; - } - apoll->double_poll = NULL; - req->apoll = apoll; + + apoll = io_req_alloc_apoll(req, issue_flags); + if (!apoll) + return IO_APOLL_ABORTED; req->flags |= REQ_F_POLLED; ipt.pt._qproc = io_async_queue_proc; From patchwork Thu Jun 23 09:34:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12891993 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84DE2C433EF for ; Thu, 23 Jun 2022 09:35:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230266AbiFWJfO (ORCPT ); Thu, 23 Jun 2022 05:35:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230157AbiFWJfM (ORCPT ); Thu, 23 Jun 2022 05:35:12 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09F2349269 for ; Thu, 23 Jun 2022 02:35:10 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id s1so27065767wra.9 for ; Thu, 23 Jun 2022 02:35:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=K1E1JLCe4USh2kriw3egu2gKlFBp9BC0Bq/laD0bxNE=; b=WVAHGOzjjHvrRALc+qALDe2+cIWqAUbrJBbaT/cNn69W+IXMEb/TJEweuqYiOUhZ9N e2f9aOsM4gmP23SVr96jHF1/qwBOagKAGQowpBvBQ7hAYOPRfcmnurs7XLY5dyPYManF ecA5wco/uOaXq/CfvXgYvczPLyjiKmsyjyQ/vaXzMGBa5mRCH4w05FGTiqPNAq5m9EA8 Bzpenz9hzYmmXS53GDmsvnaEo6Zh/e/jGxDYt9aJOUJcv1oNcy5V/W6AfpqGcb247CT9 uqYPkK+AlObyI/K0oOAS5qbRm/kSHFNVc5MAmdsUMrlKsI4m6uuS4hRnPm9lr2IDF+Yg gJbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=K1E1JLCe4USh2kriw3egu2gKlFBp9BC0Bq/laD0bxNE=; b=73q2L1QtBYAD15H3WKIb9cCK652b4lZ5IGhMAR+GS1aIW4FKxASvq57rRwpdNNDSB9 6ELynoIA1qBXQ/1cW5mHfxXr1NuhAZy1RXmoSj81AYJxLSydH/dSx41tOxHu1VItbgdY lb0wt6uih2g78fV+GhJ4BMD7KbH4uM9pjrMY57cE/yKNWzAkVe8oWgjdCImPYhzGQMc+ SaU/Z8HAD73MIx7vIIQ1ynEcDIAYsbxGNadgS0vCyan9F8LX2t7m/93JxpYSycG4R+XQ 7VXeJNRDjKAMI4y70/HeUZb6plMGWsVZRoPbvvc/5fjNrnW3LCSy4m515gYPK1g2KMyp 8owg== X-Gm-Message-State: AJIora9+MZtaRIGr6tbJH4MlEh9juVRl+RPAvNk5LprtroZsHE5dBC6m pIIY9ZsearjHBwUj4eMau/FytDrN7DtHAPl4 X-Google-Smtp-Source: AGRyM1s7Nbr6t5dPeVf3PnRpyMbAOoO0XYfwFxTM/6ieDzhW1gkOvr/UMBKBx9zsqXSyz1G6E8tDkQ== X-Received: by 2002:a5d:45c1:0:b0:21b:883e:6116 with SMTP id b1-20020a5d45c1000000b0021b883e6116mr7121967wrs.346.1655976908310; Thu, 23 Jun 2022 02:35:08 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id r21-20020a05600c35d500b003a02f957245sm2431202wmq.26.2022.06.23.02.35.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 02:35:07 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 4/6] io_uring: change arm poll return values Date: Thu, 23 Jun 2022 10:34:33 +0100 Message-Id: <529e29e9f97f2e6e383ccd44234d8b576a83a921.1655976119.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The rules for __io_arm_poll_handler()'s result parsing are complicated, as the first step don't pass return a mask but pass back a positive return code and fill ipt->result_mask. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index aef77f2a8a9a..80113b036c88 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -34,6 +34,8 @@ struct io_poll_table { struct io_kiocb *req; int nr_entries; int error; + /* output value, set only if arm poll returns >0 */ + __poll_t result_mask; }; #define IO_POLL_CANCEL_FLAG BIT(31) @@ -462,8 +464,9 @@ static int __io_arm_poll_handler(struct io_kiocb *req, if (mask && ((poll->events & (EPOLLET|EPOLLONESHOT)) == (EPOLLET|EPOLLONESHOT))) { io_poll_remove_entries(req); + ipt->result_mask = mask; /* no one else has access to the req, forget about the ref */ - return mask; + return 1; } if (!mask && unlikely(ipt->error || !ipt->nr_entries)) { @@ -813,7 +816,7 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) ret = __io_arm_poll_handler(req, poll, &ipt, poll->events); if (ret) { - io_req_set_res(req, ret, 0); + io_req_set_res(req, ipt.result_mask, 0); return IOU_OK; } if (ipt.error) { From patchwork Thu Jun 23 09:34:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12891994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFAABCCA47C for ; Thu, 23 Jun 2022 09:35:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229490AbiFWJfO (ORCPT ); Thu, 23 Jun 2022 05:35:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230176AbiFWJfN (ORCPT ); Thu, 23 Jun 2022 05:35:13 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33A754926F for ; Thu, 23 Jun 2022 02:35:11 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id z9so10664403wmf.3 for ; Thu, 23 Jun 2022 02:35:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+Uv54uWG/MQ63FNuwpeJRmurx927Ooxj18ye9JB7U1Q=; b=bymVe3Lkq4wHfS7JIUNl2WTMmlV4756O4ixQCktLwgXk7glVvUyniCumn13ILngo/9 E8wxrvOi7e7OtDY/ezL5YAUS7k6VPQksVJcZ/Z8zVTinE8iF8OuFXf/hsaWpZ5TloDYr 4RrTrXu4ySDjXqdEdmdvZDf1BQ+MD9QN5gBThOngDzgIdw+N0Kb80LBtptkOmdeY8Y9y qjAMeCT1mEY9dZ9HGPD/J/i8Fd2l+JCp2khPEhXhtR7syRMJbkqsUMiNurZ4fKWL3p5Y f9xL9CoGVLeb9CGZPnhJMuvPC7s2fIVTgfl5zcK72tcmdjSDOXG7a+n5jKIqPpSIhKei ilPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+Uv54uWG/MQ63FNuwpeJRmurx927Ooxj18ye9JB7U1Q=; b=792cK0SHsvlYi3mNdfY/xT8rmgQFZolJFKZbHIncuefkvfs30z5x8hJO8VzPikp+Zk qfoOVI4obefYlnnu86CoJjtWiXqlCT7BUuP5RvrnuFZsSQvTyzsvxELgGrFZqRBnr+U2 fafbmWG94BMGbos6CWeLfi7zuCmtY3cekwNijMtkz69p1/WDkvM2ztR6BA7p+CTaSdFr T6DS4hlW0b6johUO9nVWFDA+DPHEYRdtgUvRof/nIUfzWMyPVqL0l8/34z7v+iwMSTeU u3UMNVCTNoO8xSGUtaT11egFbDumSBG2xSQ/h4kncSbSvVmDgxG5Nh8FDkZTYvxF5Tdg UUFQ== X-Gm-Message-State: AJIora9SxhuCt/RhBpPv5Yoi4fBjCewQDAXATNxCOZOzlOk/WX3YrFVr dH/+VR4Y9LSNDXGH8WH1BP9/EWTN7bAd8ota X-Google-Smtp-Source: AGRyM1vuGqaQreT5eM8wPF3w0dc+XwkWIapg1Bi1sILGqq60oDRKo+BH/z6gSm+DYZs8osRfLJGeiQ== X-Received: by 2002:a05:600c:500a:b0:39c:53b8:7414 with SMTP id n10-20020a05600c500a00b0039c53b87414mr3034927wmr.19.1655976909333; Thu, 23 Jun 2022 02:35:09 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id r21-20020a05600c35d500b003a02f957245sm2431202wmq.26.2022.06.23.02.35.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 02:35:08 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 5/6] io_uring: refactor poll arm error handling Date: Thu, 23 Jun 2022 10:34:34 +0100 Message-Id: <6dd4786bca9a3d1609f85865936349cac08ac8e0.1655976119.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org __io_arm_poll_handler() errors parsing is a horror, in case it failed it returns 0 and the caller is expected to look at ipt.error, which already led us to a number of problems before. When it returns a valid mask, leave it as it's not, i.e. return 1 and store the mask in ipt.result_mask. In case of a failure that can be handled inline return an error code (negative value), and return 0 if __io_arm_poll_handler() took ownership of the request and will complete it. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 44 +++++++++++++++++++++----------------------- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 80113b036c88..149205eae418 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -435,6 +435,12 @@ static void io_poll_queue_proc(struct file *file, struct wait_queue_head *head, (struct io_poll **) &pt->req->async_data); } +/* + * Returns 0 when it's handed over for polling. The caller owns the requests if + * it returns non-zero, but otherwise should not touch it. Negative values + * contain an error code. When the result is >0, the polling has completed + * inline and ipt.result_mask is set to the mask. + */ static int __io_arm_poll_handler(struct io_kiocb *req, struct io_poll *poll, struct io_poll_table *ipt, __poll_t mask) @@ -461,6 +467,17 @@ static int __io_arm_poll_handler(struct io_kiocb *req, atomic_set(&req->poll_refs, 1); mask = vfs_poll(req->file, &ipt->pt) & poll->events; + if (unlikely(ipt->error || !ipt->nr_entries)) { + io_poll_remove_entries(req); + + if (mask && !(poll->events & EPOLLET)) { + ipt->result_mask = mask; + return 1; + } else { + return ipt->error ?: -EINVAL; + } + } + if (mask && ((poll->events & (EPOLLET|EPOLLONESHOT)) == (EPOLLET|EPOLLONESHOT))) { io_poll_remove_entries(req); @@ -469,25 +486,12 @@ static int __io_arm_poll_handler(struct io_kiocb *req, return 1; } - if (!mask && unlikely(ipt->error || !ipt->nr_entries)) { - io_poll_remove_entries(req); - if (!ipt->error) - ipt->error = -EINVAL; - return 0; - } - if (req->flags & REQ_F_HASH_LOCKED) io_poll_req_insert_locked(req); else io_poll_req_insert(req); if (mask && (poll->events & EPOLLET)) { - /* can't multishot if failed, just queue the event we've got */ - if (unlikely(ipt->error || !ipt->nr_entries)) { - poll->events |= EPOLLONESHOT; - req->apoll_events |= EPOLLONESHOT; - ipt->error = 0; - } __io_poll_execute(req, mask); return 0; } @@ -582,9 +586,8 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) io_kbuf_recycle(req, issue_flags); ret = __io_arm_poll_handler(req, &apoll->poll, &ipt, mask); - if (ret || ipt.error) - return ret ? IO_APOLL_READY : IO_APOLL_ABORTED; - + if (ret) + return ret > 0 ? IO_APOLL_READY : IO_APOLL_ABORTED; trace_io_uring_poll_arm(req, mask, apoll->poll.events); return IO_APOLL_OK; } @@ -815,16 +818,11 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) req->flags &= ~REQ_F_HASH_LOCKED; ret = __io_arm_poll_handler(req, poll, &ipt, poll->events); - if (ret) { + if (ret > 0) { io_req_set_res(req, ipt.result_mask, 0); return IOU_OK; } - if (ipt.error) { - req_set_fail(req); - return ipt.error; - } - - return IOU_ISSUE_SKIP_COMPLETE; + return ret ?: IOU_ISSUE_SKIP_COMPLETE; } int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) From patchwork Thu Jun 23 09:34:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12891995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A6E5C433EF for ; Thu, 23 Jun 2022 09:35:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230176AbiFWJfP (ORCPT ); Thu, 23 Jun 2022 05:35:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230157AbiFWJfO (ORCPT ); Thu, 23 Jun 2022 05:35:14 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 646D749686 for ; Thu, 23 Jun 2022 02:35:12 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id r20so6658302wra.1 for ; Thu, 23 Jun 2022 02:35:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ObHf+Q+U332P8r4EPQq6M4FmfZMgUHMGiVM2B9IjNdQ=; b=RJVSSHVqtR5jcfQ7ZooaDyrGxj9SfmgW4dG391OdDq9pFRhHwWPwh9KYQuVNC811sI B4+hH63a6X8vS0QoKxFebpNHQ1CJYIY1DcHP9snTkoA2vYXrg2TgJP1rG/2WXHCKUDsw 4t5Lt3GonF0RuR8pDWc0tqc59HIa4b9zN+5kORGNNFBLTn0TJLcsudofHhUdql/61DpU JJ/nsuix6+FTkiJ6+NR1y4r7fa/R+p1yDanLyUyiqwAt8FWXp/JYkwfFMJAICI+cCWZM YzIlPFIwuvhdvreZJQLwJgrPAbknFZcEeG0qlR4ig7BhuP0MCziOpGEY/DsNn8556uDs 9WpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ObHf+Q+U332P8r4EPQq6M4FmfZMgUHMGiVM2B9IjNdQ=; b=uOPYXHVP4+Kzk30QNGX5+TFGoQOhhL0nsjPNnOtpffcX3yZMn1IsRPEU/lPTRW/LM8 NDVHS2TfOMKDLcqIvuGh9w1iaqZ+v9JJV1oWjo6iRRVqmSgoLzR/JGZyqMzqv7UDmD2v Y4Ro9iNpef1k4H0n+maSCKJwf+Kir5zdgPJQxkLd4DnquALPR1PwzAw1FktcEmvPMwNr mJ5u7tMpZY3unWRHLDjXROy15Cws97DYasEtntuqcEJaSoSV7pqynfpX97R2fWvRJY7B BtIaO3nyJChE+T+c6BUEQwfi2dIRXi/CBlOMLt9PLof5w9JU3ZACerpZPXK2eFjS1zQM Zxug== X-Gm-Message-State: AJIora/O8ajbHnR7QLrW8McLkXofvdrGUyQ169ZS45fXzExfpJtCB9cK ovg+mUEuQo2V1tLEq1FAwAUJcxjTLUuMPI/Y X-Google-Smtp-Source: AGRyM1uRIEZoYSfbHHVsWDavxSMJFfrwapFG5mxPDZK5YMz7WmiXBaYEcGh0yn06SXBQvUbvj1ccXQ== X-Received: by 2002:adf:d22b:0:b0:21b:90be:2dc8 with SMTP id k11-20020adfd22b000000b0021b90be2dc8mr7179336wrh.423.1655976910409; Thu, 23 Jun 2022 02:35:10 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id r21-20020a05600c35d500b003a02f957245sm2431202wmq.26.2022.06.23.02.35.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 02:35:10 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 6/6] io_uring: optimise submission side poll_refs Date: Thu, 23 Jun 2022 10:34:35 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The final poll_refs put in __io_arm_poll_handler() takes quite some cycles. When we're arming from the original task context task_work won't be run, so in this case we can assume that we won't race with task_works and so not take the initial ownership ref. One caveat is that after arming a poll we may race with it, so we have to add a bunch of io_poll_get_ownership() hidden inside of io_poll_can_finish_inline() whenever we want to complete arming inline. For the same reason we can't just set REQ_F_DOUBLE_POLL in __io_queue_proc() and so need to sync with the first poll entry by taking its wq head lock. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 88 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 67 insertions(+), 21 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 149205eae418..69b2f4bab3b2 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -34,6 +34,7 @@ struct io_poll_table { struct io_kiocb *req; int nr_entries; int error; + bool owning; /* output value, set only if arm poll returns >0 */ __poll_t result_mask; }; @@ -374,6 +375,27 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, return 1; } +static void io_poll_double_prepare(struct io_kiocb *req) +{ + struct wait_queue_head *head; + struct io_poll *poll = io_poll_get_single(req); + + /* head is RCU protected, see io_poll_remove_entries() comments */ + rcu_read_lock(); + head = smp_load_acquire(&poll->head); + if (head) { + /* + * poll arm may not hold ownership and so race with + * io_poll_wake() by modifying req->flags. There is only one + * poll entry queued, serialise with it by taking its head lock. + */ + spin_lock_irq(&head->lock); + req->flags |= REQ_F_DOUBLE_POLL; + spin_unlock_irq(&head->lock); + } + rcu_read_unlock(); +} + static void __io_queue_proc(struct io_poll *poll, struct io_poll_table *pt, struct wait_queue_head *head, struct io_poll **poll_ptr) @@ -405,16 +427,19 @@ static void __io_queue_proc(struct io_poll *poll, struct io_poll_table *pt, pt->error = -ENOMEM; return; } + + io_poll_double_prepare(req); /* mark as double wq entry */ wqe_private |= IO_WQE_F_DOUBLE; - req->flags |= REQ_F_DOUBLE_POLL; io_init_poll_iocb(poll, first->events, first->wait.func); *poll_ptr = poll; if (req->opcode == IORING_OP_POLL_ADD) req->flags |= REQ_F_ASYNC_DATA; + } else { + /* fine to modify, there is no poll queued to race with us */ + req->flags |= REQ_F_SINGLE_POLL; } - req->flags |= REQ_F_SINGLE_POLL; pt->nr_entries++; poll->head = head; poll->wait.private = (void *) wqe_private; @@ -435,6 +460,12 @@ static void io_poll_queue_proc(struct file *file, struct wait_queue_head *head, (struct io_poll **) &pt->req->async_data); } +static bool io_poll_can_finish_inline(struct io_kiocb *req, + struct io_poll_table *pt) +{ + return pt->owning || io_poll_get_ownership(req); +} + /* * Returns 0 when it's handed over for polling. The caller owns the requests if * it returns non-zero, but otherwise should not touch it. Negative values @@ -443,7 +474,8 @@ static void io_poll_queue_proc(struct file *file, struct wait_queue_head *head, */ static int __io_arm_poll_handler(struct io_kiocb *req, struct io_poll *poll, - struct io_poll_table *ipt, __poll_t mask) + struct io_poll_table *ipt, __poll_t mask, + unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; int v; @@ -452,34 +484,45 @@ static int __io_arm_poll_handler(struct io_kiocb *req, req->work.cancel_seq = atomic_read(&ctx->cancel_seq); io_init_poll_iocb(poll, mask, io_poll_wake); poll->file = req->file; - req->apoll_events = poll->events; ipt->pt._key = mask; ipt->req = req; ipt->error = 0; ipt->nr_entries = 0; - /* - * Take the ownership to delay any tw execution up until we're done - * with poll arming. see io_poll_get_ownership(). + * Polling is either completed here or via task_work, so if we're in the + * task context we're naturally serialised with tw by merit of running + * the same task. When it's io-wq, take the ownership to prevent tw + * from running. However, when we're in the task context, skip taking + * it as an optimisation. + * + * Note: even though the request won't be completed/freed, without + * ownership we still can race with io_poll_wake(). + * io_poll_can_finish_inline() tries to deal with that. */ - atomic_set(&req->poll_refs, 1); + ipt->owning = issue_flags & IO_URING_F_UNLOCKED; + + atomic_set(&req->poll_refs, (int)ipt->owning); mask = vfs_poll(req->file, &ipt->pt) & poll->events; if (unlikely(ipt->error || !ipt->nr_entries)) { io_poll_remove_entries(req); - if (mask && !(poll->events & EPOLLET)) { + if (!io_poll_can_finish_inline(req, ipt)) { + io_poll_mark_cancelled(req); + return 0; + } else if (mask && !(poll->events & EPOLLET)) { ipt->result_mask = mask; return 1; - } else { - return ipt->error ?: -EINVAL; } + return ipt->error ?: -EINVAL; } if (mask && ((poll->events & (EPOLLET|EPOLLONESHOT)) == (EPOLLET|EPOLLONESHOT))) { + if (!io_poll_can_finish_inline(req, ipt)) + return 0; io_poll_remove_entries(req); ipt->result_mask = mask; /* no one else has access to the req, forget about the ref */ @@ -491,18 +534,21 @@ static int __io_arm_poll_handler(struct io_kiocb *req, else io_poll_req_insert(req); - if (mask && (poll->events & EPOLLET)) { + if (mask && (poll->events & EPOLLET) && + io_poll_can_finish_inline(req, ipt)) { __io_poll_execute(req, mask); return 0; } - /* - * Release ownership. If someone tried to queue a tw while it was - * locked, kick it off for them. - */ - v = atomic_dec_return(&req->poll_refs); - if (unlikely(v & IO_POLL_REF_MASK)) - __io_poll_execute(req, 0); + if (ipt->owning) { + /* + * Release ownership. If someone tried to queue a tw while it was + * locked, kick it off for them. + */ + v = atomic_dec_return(&req->poll_refs); + if (unlikely(v & IO_POLL_REF_MASK)) + __io_poll_execute(req, 0); + } return 0; } @@ -585,7 +631,7 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) io_kbuf_recycle(req, issue_flags); - ret = __io_arm_poll_handler(req, &apoll->poll, &ipt, mask); + ret = __io_arm_poll_handler(req, &apoll->poll, &ipt, mask, issue_flags); if (ret) return ret > 0 ? IO_APOLL_READY : IO_APOLL_ABORTED; trace_io_uring_poll_arm(req, mask, apoll->poll.events); @@ -817,7 +863,7 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) else req->flags &= ~REQ_F_HASH_LOCKED; - ret = __io_arm_poll_handler(req, poll, &ipt, poll->events); + ret = __io_arm_poll_handler(req, poll, &ipt, poll->events, issue_flags); if (ret > 0) { io_req_set_res(req, ipt.result_mask, 0); return IOU_OK;