From patchwork Wed Nov 23 11:33:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13053514 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52D9CC46467 for ; Wed, 23 Nov 2022 11:35:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237271AbiKWLfL (ORCPT ); Wed, 23 Nov 2022 06:35:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237278AbiKWLfF (ORCPT ); Wed, 23 Nov 2022 06:35:05 -0500 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1B9010EA1A for ; Wed, 23 Nov 2022 03:35:03 -0800 (PST) Received: by mail-wm1-x332.google.com with SMTP id v124-20020a1cac82000000b003cf7a4ea2caso1200862wme.5 for ; Wed, 23 Nov 2022 03:35:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A0DW1JSauK3bY460xsnmlLPiYfNI+QTI3O3eNQDmUns=; b=YtnIqCvh/xgvy2X9x1j79mdcWmrmExXdVsfIXSyqnntQIF//PNQYWkV51zvtgmJb6W KgWaD3VvK55Qf5pRiephud5trgZiSVOvh/1ApBsOKHd/HVJNUepBSnUKrVsY+6+M40mc XB4+8OSY5IXv2mqqNOl98xxJWVqwgkCLyuTkc76yh6CdId934tCte8HqpL7JL05cHKVg fUJspkR3FoAeHwzZqrl/MlqHy6XjsvglQKY7SaPM3QM15FVmgYzs41LknLka+1bBum0o LvGAhdCITvWcTVAmlUjxpv6BgMhQowb/W5IIxFq3QtfGesQyOi8cJSZOVkrBXbg178Jw A4WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A0DW1JSauK3bY460xsnmlLPiYfNI+QTI3O3eNQDmUns=; b=nxw4dyWGDXwFJ6uum2wWqeLgxpvlHXnP9cDydbkD1jDDFoQEChXECJvMzd/8Pui/lm pmWicwheZaDOAtC2WS4bVj6tzoYOkHLwclM9gX+QhXYva13bLMhvOMZ+3xHtDqD28uHg A/u1Rj8b9qFaTqZjk5lNhpey3PHk4FMA1pTu+dnKceydrj90gt269Ebi79X+VKc71vHr yRVoSWfSr55iC+W8nSL63BaDGvIWawEZCFuRxefaPFEuhkBFwFuuYCojbh5+HAQ9P49X gubkUavU0yChU9DQoUKWUJHSxRJTuMLXlnq6Gx3cBAld44JJIdMaMOivapU7kVeac7KH /5dQ== X-Gm-Message-State: ANoB5pn29pHHiAcAPNRhD8UEJLU9DocmfH4u2tUo7W81f8ZO8846ODUA 98B6F0r1ngY2RoTVAv5RYT6h7W+q4lg= X-Google-Smtp-Source: AA0mqf7SgqYT6X2V+myj6s5fN/ozUC+QsWKUMSOVgFCHaOKBYnxkgs65F+21dXlUHtzMhQl8oBwXew== X-Received: by 2002:a05:600c:4fcd:b0:3cf:a11a:c775 with SMTP id o13-20020a05600c4fcd00b003cfa11ac775mr9466668wmq.153.1669203302272; Wed, 23 Nov 2022 03:35:02 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:e1b]) by smtp.gmail.com with ESMTPSA id g9-20020a05600c4ec900b003cfd58409desm2262064wmq.13.2022.11.23.03.35.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Nov 2022 03:35:01 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 1/7] io_uring: add completion locking for iopoll Date: Wed, 23 Nov 2022 11:33:36 +0000 Message-Id: <84d86b5c117feda075471c5c9e65208e0dccf5d0.1669203009.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There are pieces of code that may allow iopoll to race filling cqes, temporarily add spinlocking around posting events. Cc: stable@vger.kernel.org Signed-off-by: Pavel Begunkov --- io_uring/rw.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/io_uring/rw.c b/io_uring/rw.c index 1ce065709724..61c326831949 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -1049,6 +1049,7 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) else if (!pos) return 0; + spin_lock(&ctx->completion_lock); prev = start; wq_list_for_each_resume(pos, prev) { struct io_kiocb *req = container_of(pos, struct io_kiocb, comp_list); @@ -1063,11 +1064,11 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) req->cqe.flags = io_put_kbuf(req, 0); __io_fill_cqe_req(req->ctx, req); } - + io_commit_cqring(ctx); + spin_unlock(&ctx->completion_lock); if (unlikely(!nr_events)) return 0; - io_commit_cqring(ctx); io_cqring_ev_posted_iopoll(ctx); pos = start ? start->next : ctx->iopoll_list.first; wq_list_cut(&ctx->iopoll_list, prev, start); From patchwork Wed Nov 23 11:33:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13053515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E7FCC47089 for ; Wed, 23 Nov 2022 11:35:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237278AbiKWLfL (ORCPT ); Wed, 23 Nov 2022 06:35:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237281AbiKWLfF (ORCPT ); Wed, 23 Nov 2022 06:35:05 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E76D11DA22 for ; Wed, 23 Nov 2022 03:35:04 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id e11so16037127wru.8 for ; Wed, 23 Nov 2022 03:35:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2MIExCVbcFv8WjOmUdmSVgU/bQB8FiUSuBIUeZkm1cg=; b=Ib4e4whjwYqacZE24sQmkCI/pWXU05PF8y8hK0CMqL+G9bckIKNJfWSuUvubIKTsss XTOhRzeIBVvmNBBRGwtfbkL0AIcSkhU97gq4x9D2xUKmEb04o7rIaPymsApW5QweOSUE qvPEK4FmEVH2XaAhMEBX4R3crML6uv5+FMknj2pOqzhPakq/pw03IQlvUTqM2hUBJsjj w2Ln2mH8ZP0Cc/8SDeIKysHbKRRa/AjFcfR8l+dLisYPeDj5SoPTRZqAN7NihId6dVcP zxRlVYneom4Q/RX0q+e2hjKIu8qIVMZeeEICgu6NupikhQIU1V5n5ZAPJJ06qtjXF0SF vUMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2MIExCVbcFv8WjOmUdmSVgU/bQB8FiUSuBIUeZkm1cg=; b=1xczrT7youjLR+9LS2mr60XgJn8DbuSqqR9ueITgq71OIKtRxc7G9zKn4bUxE1YV0w oHM9UHQC6IiN3y2nlJc/8iS2+gnsuJ8TNWarkcuBN8QFJtVzKRrCqa2nywTOisXhFpxz MRBa3lcgdDmWKdKljHIDrM6OJtBCEHwh82ARaINcpFNLNalSSbc3k8kKpeidYGK2GtV8 rTaecXAp9Mvlirheoaft2dgVLr9bR6EBux+URJweZuU6xeC3uCapr5LcR+rL5U1RBtK7 22b1ON6oKPs48ruXtuFnvtvvs+abYFz4KL3NY2kdRBBosOQQzp9ny5mEaA4EW4ugjDkM iHJg== X-Gm-Message-State: ANoB5pk2WkpAdYs1C33u+xBwgDqM1SqyH8zZ+RPFVQHswAA+SGRvytNz R8aJlRb1kMT7NbmL6TvDgKmnvHkL9Vk= X-Google-Smtp-Source: AA0mqf4t85piBUIJ93/jKQye6OrD1XSJjcRiiMy7+3OsRm1SGhKlURpPsRiIP3rTYCONjdoZb7gcCg== X-Received: by 2002:a5d:6088:0:b0:22e:5149:441d with SMTP id w8-20020a5d6088000000b0022e5149441dmr16987066wrt.661.1669203302938; Wed, 23 Nov 2022 03:35:02 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:e1b]) by smtp.gmail.com with ESMTPSA id g9-20020a05600c4ec900b003cfd58409desm2262064wmq.13.2022.11.23.03.35.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Nov 2022 03:35:02 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 2/7] io_uring: hold locks for io_req_complete_failed Date: Wed, 23 Nov 2022 11:33:37 +0000 Message-Id: <70760344eadaecf2939287084b9d4ba5c05a6984.1669203009.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org A preparation patch, make sure we always hold uring_lock around io_req_complete_failed(). The only place deviating from the rule is io_cancel_defer_files(), queue a tw instead. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2087ada65284..a4e6866f24c8 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -861,9 +861,12 @@ inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags) } void io_req_complete_failed(struct io_kiocb *req, s32 res) + __must_hold(&ctx->uring_lock) { const struct io_op_def *def = &io_op_defs[req->opcode]; + lockdep_assert_held(&req->ctx->uring_lock); + req_set_fail(req); io_req_set_res(req, res, io_put_kbuf(req, IO_URING_F_UNLOCKED)); if (def->fail) @@ -1614,6 +1617,7 @@ static u32 io_get_sequence(struct io_kiocb *req) } static __cold void io_drain_req(struct io_kiocb *req) + __must_hold(&ctx->uring_lock) { struct io_ring_ctx *ctx = req->ctx; struct io_defer_entry *de; @@ -2847,7 +2851,7 @@ static __cold bool io_cancel_defer_files(struct io_ring_ctx *ctx, while (!list_empty(&list)) { de = list_first_entry(&list, struct io_defer_entry, list); list_del_init(&de->list); - io_req_complete_failed(de->req, -ECANCELED); + io_req_task_queue_fail(de->req, -ECANCELED); kfree(de); } return true; From patchwork Wed Nov 23 11:33:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13053517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B947C4708A for ; Wed, 23 Nov 2022 11:35:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236322AbiKWLfM (ORCPT ); Wed, 23 Nov 2022 06:35:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237290AbiKWLfG (ORCPT ); Wed, 23 Nov 2022 06:35:06 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 480C511DA29 for ; Wed, 23 Nov 2022 03:35:05 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id ja4-20020a05600c556400b003cf6e77f89cso2805451wmb.0 for ; Wed, 23 Nov 2022 03:35:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ltvh2Gi7o+LSCX0EJRNMZ1KcgcZ4TbP6TUTFtEMBCLE=; b=A6eoK7csYMx/DqzAwvDZaPBM5FYD2k2oeQYUxcc21bAWYITAOlQbB27u8XiEXxWXtR OO8GSeBIBzSisvfozV9LEnxzbqKvS+aS/mGe5OTM1bn3Z6UrgjPiwJN6fIr3taLnMFEI cG2sXC+Js5ILd3AK5sOHuLyUQv9ycENNl44l9otDiZCqLnl0cKimmmllb7cLsO2JnkHW k3cRV3qokKi3KxvNCKLKyCy0mwD14+6vhw5/d9GZMvcJopPQ7vB2i4IZjcMerFaQOKgG U+qcZLSY+JIsBYhAm4002kWaylkdsUzmOlR1sm1ySUMo9AVOhwkNzHjMYWXqrdIHgNUA FQCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ltvh2Gi7o+LSCX0EJRNMZ1KcgcZ4TbP6TUTFtEMBCLE=; b=N1I4WdJusSUMw/tbVnncwp2rYocjOu+v9UMIS1Ql7bi1xq3ZyyHf3uQDjPeHbVbciq K0ztBoy/TaerbKanywWh5rpbxtwHKcfYGyLXm97q1NcZrYr7xEXlO8v5vkFrarhcfE8G zMMCjnufMnrBYHtJaFA1XmGJaP5NHnkSHT8dUVKXiulP8Kt07mPEnHKkgJb8XWVMvOGx SiMpfh5OI4MFyVOe3Sb+By1ahNT8B6pBUTR6e4za69X46AhL/hg7sOxMvZfh0zv9J0Ce 8gazwOGGe9zRI7j8GjO+o5m7jwPrlPURktXSRWEOOUD7C9l1ycUT1gQEWohjmNIBild7 Fleg== X-Gm-Message-State: ANoB5pnDqdXUIVZ/U2DhWbELugWMe9HYnZ8R592+eFbxucKj2xX+x6P+ MP8gm8JBgORtUIo4aHfsvzTHmeZr+qs= X-Google-Smtp-Source: AA0mqf7FOgOuwZqamGIC9oQo9LMbnP/Rr7Gg2UR/f8uHbn0fevP01VdJkzLw0/2mxCnCH3+eNHmXYQ== X-Received: by 2002:a05:600c:3514:b0:3cf:a985:7692 with SMTP id h20-20020a05600c351400b003cfa9857692mr13238933wmq.104.1669203303657; Wed, 23 Nov 2022 03:35:03 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:e1b]) by smtp.gmail.com with ESMTPSA id g9-20020a05600c4ec900b003cfd58409desm2262064wmq.13.2022.11.23.03.35.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Nov 2022 03:35:03 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 3/7] io_uring: use io_req_task_complete() in timeout Date: Wed, 23 Nov 2022 11:33:38 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Use a more generic io_req_task_complete() in timeout completion task_work instead of io_req_complete_post(). Signed-off-by: Pavel Begunkov --- io_uring/timeout.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/timeout.c b/io_uring/timeout.c index e8a8c2099480..a819818df7b3 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -282,11 +282,11 @@ static void io_req_task_link_timeout(struct io_kiocb *req, bool *locked) ret = io_try_cancel(req->task->io_uring, &cd, issue_flags); } io_req_set_res(req, ret ?: -ETIME, 0); - io_req_complete_post(req); + io_req_task_complete(req, locked); io_put_req(prev); } else { io_req_set_res(req, -ETIME, 0); - io_req_complete_post(req); + io_req_task_complete(req, locked); } } From patchwork Wed Nov 23 11:33:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13053516 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06261C433FE for ; Wed, 23 Nov 2022 11:35:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237281AbiKWLfN (ORCPT ); Wed, 23 Nov 2022 06:35:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44358 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237310AbiKWLfH (ORCPT ); Wed, 23 Nov 2022 06:35:07 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0479F72D4 for ; Wed, 23 Nov 2022 03:35:05 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id r9-20020a1c4409000000b003d02dd48c45so882673wma.0 for ; Wed, 23 Nov 2022 03:35:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=alCckDgLta/2b15StUxHZvhqjHuwSHam7jn/kHiSi0c=; b=YPP3kOab9ZkazKcgMB3j0Ul7O0hIuyfpcjZ9lN8Geo5lB+uE+UF7/hcW9GpaxyiUhG S/BKTkd9k8bv2nSdnaipq5AejyB4d7swmaBpO8pAgKEp6qjaZ/rqUGKygFx+914XSbMS LEStRW20xX0ssEoquCWO2t0+Gd/IfpiHcn6ag8nW7J+M2KHKd+gVYflK+Yb7sIeroZs/ 3ZJs98nvIGcHD9kNEw8CPYsfkth6IFxvJt6a7P22bmZZVKgNRE/+2ZbzV/V9F2nih0LU 4gP2Mv0zu3eWsOkKtkMDLuC13pou5CE3jQUuCHPMaeRv65qpiz18WmOJI+iikQmLsfUG QkRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=alCckDgLta/2b15StUxHZvhqjHuwSHam7jn/kHiSi0c=; b=ZlSZRYyhZisTGvDcJbrbVAdn3PNLYZLfNdQ4ALhc3Sg4g4U86AEb5SSoruWfCeol5D gF+jc4hChIYhgADOrysnYzxeBt5A5L5NaaoZjsu9UHPHsxFXZlDZMQ8LyPngCTFgZW80 vnGgrVc//BtHMzlIub846vCv2fhZ4n9XyqVDrCimNaUc6ZQCO6erK0VWxjrax4plBst1 F4SZVoKeafYNRmUun4fnaNzhA6p0wTlwkgDfsm7leuTQ5TwycPKtb87BGCEZ0rYFoRBr O0mxADI5rexVVD55U0OKxVN9TPknzoidE7AzqS/tezZnbRrWYFB3Q/hsAbF+VkS9otLB Q5ow== X-Gm-Message-State: ANoB5plCrXosgALgdtwdZhGS6V2BbIlTE9TUdsMdpQHWAXBy7hV8NeD/ G+R3afvl1vDDkU2dw+vUSGQEOQN7cT8= X-Google-Smtp-Source: AA0mqf5WUhBP2fMp4aUJssc5aq8f4+50o265jcirc3PQgoaX8jpGOrcF5z8W/caKw8goj6V/yrYCBg== X-Received: by 2002:a05:600c:4d0a:b0:3c6:a7a1:eebd with SMTP id u10-20020a05600c4d0a00b003c6a7a1eebdmr19354734wmp.176.1669203304346; Wed, 23 Nov 2022 03:35:04 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:e1b]) by smtp.gmail.com with ESMTPSA id g9-20020a05600c4ec900b003cfd58409desm2262064wmq.13.2022.11.23.03.35.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Nov 2022 03:35:03 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 4/7] io_uring: remove io_req_tw_post_queue Date: Wed, 23 Nov 2022 11:33:39 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Remove io_req_tw_post() and io_req_tw_post_queue(), we can use io_req_task_complete() instead. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 12 ------------ io_uring/io_uring.h | 8 +++++++- io_uring/timeout.c | 6 +++--- 3 files changed, 10 insertions(+), 16 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a4e6866f24c8..81e7e51816fb 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1225,18 +1225,6 @@ int io_run_local_work(struct io_ring_ctx *ctx) return ret; } -static void io_req_tw_post(struct io_kiocb *req, bool *locked) -{ - io_req_complete_post(req); -} - -void io_req_tw_post_queue(struct io_kiocb *req, s32 res, u32 cflags) -{ - io_req_set_res(req, res, cflags); - req->io_task_work.func = io_req_tw_post; - io_req_task_work_add(req); -} - static void io_req_task_cancel(struct io_kiocb *req, bool *locked) { /* not needed for normal modes, but SQPOLL depends on it */ diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index af3f82bd4017..002b6cc842a5 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -50,7 +50,6 @@ static inline bool io_req_ffs_set(struct io_kiocb *req) void __io_req_task_work_add(struct io_kiocb *req, bool allow_local); bool io_is_uring_fops(struct file *file); bool io_alloc_async_data(struct io_kiocb *req); -void io_req_tw_post_queue(struct io_kiocb *req, s32 res, u32 cflags); void io_req_task_queue(struct io_kiocb *req); void io_queue_iowq(struct io_kiocb *req, bool *dont_use); void io_req_task_complete(struct io_kiocb *req, bool *locked); @@ -366,4 +365,11 @@ static inline bool io_allowed_run_tw(struct io_ring_ctx *ctx) ctx->submitter_task == current); } +static inline void io_req_queue_tw_complete(struct io_kiocb *req, s32 res) +{ + io_req_set_res(req, res, 0); + req->io_task_work.func = io_req_task_complete; + io_req_task_work_add(req); +} + #endif diff --git a/io_uring/timeout.c b/io_uring/timeout.c index a819818df7b3..5b4bc93fd6e0 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -63,7 +63,7 @@ static bool io_kill_timeout(struct io_kiocb *req, int status) atomic_set(&req->ctx->cq_timeouts, atomic_read(&req->ctx->cq_timeouts) + 1); list_del_init(&timeout->list); - io_req_tw_post_queue(req, status, 0); + io_req_queue_tw_complete(req, status); return true; } return false; @@ -159,7 +159,7 @@ void io_disarm_next(struct io_kiocb *req) req->flags &= ~REQ_F_ARM_LTIMEOUT; if (link && link->opcode == IORING_OP_LINK_TIMEOUT) { io_remove_next_linked(req); - io_req_tw_post_queue(link, -ECANCELED, 0); + io_req_queue_tw_complete(link, -ECANCELED); } } else if (req->flags & REQ_F_LINK_TIMEOUT) { struct io_ring_ctx *ctx = req->ctx; @@ -168,7 +168,7 @@ void io_disarm_next(struct io_kiocb *req) link = io_disarm_linked_timeout(req); spin_unlock_irq(&ctx->timeout_lock); if (link) - io_req_tw_post_queue(link, -ECANCELED, 0); + io_req_queue_tw_complete(link, -ECANCELED); } if (unlikely((req->flags & REQ_F_FAIL) && !(req->flags & REQ_F_HARDLINK))) From patchwork Wed Nov 23 11:33:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13053518 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E16FC3A59F for ; Wed, 23 Nov 2022 11:35:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237343AbiKWLfO (ORCPT ); Wed, 23 Nov 2022 06:35:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237328AbiKWLfH (ORCPT ); Wed, 23 Nov 2022 06:35:07 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA54511DA39 for ; Wed, 23 Nov 2022 03:35:06 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id v1so28901794wrt.11 for ; Wed, 23 Nov 2022 03:35:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y2QNLJ4mI8t02sLnWQNBMUUxoLK3U5B8jMH3lwt4n7E=; b=CEckmgPu6l+Jm0qGjxTpnktWEEcltPKgEoqQ+IOEcqbAnBg4EC/dWdrM/mo8KEzy5q Wc2vDAPFEdICRQmNMANIQodgQQYcSLnwJC3h82iJiuu28UIAjTEEQTDR6AUaND44565X 2CcuPsBkZnhQttr5q1o6dbAS8VU7cqkfy3YgVV6ohI9JrAXrOeLTbZDxYq+CWYFZvMwT lq1obhccybh15k8cDrY5RVtvzcpLEuCFYfT1R4edVnR6FSwn+jUHh+/1dFGOC3D2qfjW eacqNpgUjXipGyXevdJeMVSJkCVj03xM4KzBT2/YV/B5KOZd/5ucxQujswBwOL3oJjt4 FCsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Y2QNLJ4mI8t02sLnWQNBMUUxoLK3U5B8jMH3lwt4n7E=; b=b3VTN/GrujeiDySVztlUFrninFlvVCf+709c9oNtYg6+9VAiLkXrvf2V8KuLTveyUv 0kgF8l7Cc3mh4MuC4KirR/6szxL4dBj8jxmOBEWfCtPzceMhm03CYZzwvJfmjpDUkAox gXAf9z7+D/t6UicGsOvgBnlL1YRkoU5+3otzPbJKrU0CBnZrlifinBMs1Pe5LXxYCseJ p3XGLubBMD/Y1uFbfB8Wi7Xht9Ubbj3geZlfIyfq75+0ua8VGXRel6fCxENRqB8pTFsJ zWQbaRd49HPJXaUvW35xHk1myaf7+CPHLp3gZxLwXsaSfs5KHgZQa0HP3COluxYjLxVz G6sQ== X-Gm-Message-State: ANoB5pmPUEjdBMYrsZLezMDH2+xqdFalIAKPWtYzHoJHnW8Om492FlpL gYflH9AJ4t9A01u+gfnhKzuKHuE+4Nc= X-Google-Smtp-Source: AA0mqf53qiz+xgRHFGqvcAAChvpHXUktJOKvcCL8fbKt6N1CJ5Ecazwsfjt9iwGdOe0W9nSs7maR0w== X-Received: by 2002:adf:dc86:0:b0:236:754e:f8b4 with SMTP id r6-20020adfdc86000000b00236754ef8b4mr16801670wrj.478.1669203305213; Wed, 23 Nov 2022 03:35:05 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:e1b]) by smtp.gmail.com with ESMTPSA id g9-20020a05600c4ec900b003cfd58409desm2262064wmq.13.2022.11.23.03.35.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Nov 2022 03:35:04 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 5/7] io_uring: inline __io_req_complete_put() Date: Wed, 23 Nov 2022 11:33:40 +0000 Message-Id: <1923a4dfe80fa877f859a22ed3df2d5fc8ecf02b.1669203009.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Inline __io_req_complete_put() into io_req_complete_post(), there are no other users. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 81e7e51816fb..bd9b286eb031 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -813,15 +813,19 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, return filled; } -static void __io_req_complete_put(struct io_kiocb *req) +void io_req_complete_post(struct io_kiocb *req) { + struct io_ring_ctx *ctx = req->ctx; + + io_cq_lock(ctx); + if (!(req->flags & REQ_F_CQE_SKIP)) + __io_fill_cqe_req(ctx, req); + /* * If we're the last reference to this request, add to our locked * free_list cache. */ if (req_ref_put_and_test(req)) { - struct io_ring_ctx *ctx = req->ctx; - if (req->flags & IO_REQ_LINK_FLAGS) { if (req->flags & IO_DISARM_MASK) io_disarm_next(req); @@ -842,16 +846,6 @@ static void __io_req_complete_put(struct io_kiocb *req) wq_list_add_head(&req->comp_list, &ctx->locked_free_list); ctx->locked_free_nr++; } -} - -void io_req_complete_post(struct io_kiocb *req) -{ - struct io_ring_ctx *ctx = req->ctx; - - io_cq_lock(ctx); - if (!(req->flags & REQ_F_CQE_SKIP)) - __io_fill_cqe_req(ctx, req); - __io_req_complete_put(req); io_cq_unlock_post(ctx); } From patchwork Wed Nov 23 11:33:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13053519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 040E7C4332F for ; Wed, 23 Nov 2022 11:35:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237499AbiKWLfZ (ORCPT ); Wed, 23 Nov 2022 06:35:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236402AbiKWLfI (ORCPT ); Wed, 23 Nov 2022 06:35:08 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8DDCEE0A for ; Wed, 23 Nov 2022 03:35:07 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id i64-20020a1c3b43000000b003d016c21100so1204688wma.3 for ; Wed, 23 Nov 2022 03:35:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GPobMHqO/KcnKYrLjIDJGleMQe2iQ6KyZgXk4HC1KKQ=; b=n+7JFFMv8/qzP772rFSXFBfTy9KwQEAcaULRTs6pq+YJlNI9z1F9x7IYN1fdn2ltK7 RbPJ0NyyCQlMzEFS/kWAQjJXDq1qPJ7+awzSDdmPHmXHzBszOAyEFMox8a62qwqvaJvJ XgT12VH6sG8fD00i8Vj0EQ0E16q/VTc8k3rz8rXKy/zaqwNb2LmkEUDxYQW9b7O6fWuv zR7w2ZJtsLbHMd2+gwDlcYPZWbUw6jyIxXzetgaNipG5y4q8lOMVx53VEs9h96sdysIz QcMS/tZA/S4mR+WxW1CsfM6qpfdKm6qVc1iQpguAZZBjwVS17S7HWkAQWapMgznIl/la sLSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GPobMHqO/KcnKYrLjIDJGleMQe2iQ6KyZgXk4HC1KKQ=; b=sqsfvZfVedOFiK2M0IpLq6inXr7h/2dFQD6kycj5QVI0GGxxJvwh5v4PS7lPSE1NHK wxeXYjH1ij+3A8MYahdSECxKHOk2Vj9GR2xpmGa0GQevcOYI35y9cIyXf0zXJsYCEZHj FL1QTLtsQeT94lTMOyiLIu0/ZCjPD6IvKFc6xHCMaNcO23KlW2z84r3Z41ur1Xh5O7r7 KjRT085akYfEV9XQxi6Gl4LWVOpANOJY+DPEUAq2u0E047C2AUN19KYV+y0CQ47/uVE+ S2ObArReowso8Xwn68BhJKPH7UfqAee5DIJwcyFXQNdVRi5tpY4Be0QDfYSHPljFvMFo 9TKg== X-Gm-Message-State: ANoB5pkCKfpC+GPDNJ2+lwe/MeQMfL+Wqs78IZZ1FsHh7hGvQdKTtTB+ 0jxeWsV0yhQ5fSoW1TrJ3Gr7PqFaey4= X-Google-Smtp-Source: AA0mqf4MF1exsB3Jy8jCH9/34koXVT1hUDucE6YJAPO9bK/esodUA/2URdMkrXvCsM1V29Q577GgRw== X-Received: by 2002:a1c:ed04:0:b0:3cf:d08d:3eb2 with SMTP id l4-20020a1ced04000000b003cfd08d3eb2mr19986814wmh.129.1669203305882; Wed, 23 Nov 2022 03:35:05 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:e1b]) by smtp.gmail.com with ESMTPSA id g9-20020a05600c4ec900b003cfd58409desm2262064wmq.13.2022.11.23.03.35.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Nov 2022 03:35:05 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 6/7] io_uring: iopoll protect complete_post Date: Wed, 23 Nov 2022 11:33:41 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_req_complete_post() may be used by iopoll enabled rings, grab locks in this case. That requires to pass issue_flags to propagate the locking state. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 21 +++++++++++++++------ io_uring/io_uring.h | 10 ++++++++-- io_uring/kbuf.c | 4 ++-- io_uring/poll.c | 2 +- io_uring/uring_cmd.c | 2 +- 5 files changed, 27 insertions(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index bd9b286eb031..54a9966bbb47 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -813,7 +813,7 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, return filled; } -void io_req_complete_post(struct io_kiocb *req) +static void __io_req_complete_post(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; @@ -849,9 +849,18 @@ void io_req_complete_post(struct io_kiocb *req) io_cq_unlock_post(ctx); } -inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags) +void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { - io_req_complete_post(req); + if (!(issue_flags & IO_URING_F_UNLOCKED) || + !(req->ctx->flags & IORING_SETUP_IOPOLL)) { + __io_req_complete_post(req); + } else { + struct io_ring_ctx *ctx = req->ctx; + + mutex_lock(&ctx->uring_lock); + __io_req_complete_post(req); + mutex_unlock(&ctx->uring_lock); + } } void io_req_complete_failed(struct io_kiocb *req, s32 res) @@ -865,7 +874,7 @@ void io_req_complete_failed(struct io_kiocb *req, s32 res) io_req_set_res(req, res, io_put_kbuf(req, IO_URING_F_UNLOCKED)); if (def->fail) def->fail(req); - io_req_complete_post(req); + io_req_complete_post(req, 0); } /* @@ -1449,7 +1458,7 @@ void io_req_task_complete(struct io_kiocb *req, bool *locked) if (*locked) io_req_complete_defer(req); else - io_req_complete_post(req); + io_req_complete_post_tw(req, locked); } /* @@ -1717,7 +1726,7 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) if (issue_flags & IO_URING_F_COMPLETE_DEFER) io_req_complete_defer(req); else - io_req_complete_post(req); + io_req_complete_post(req, issue_flags); } else if (ret != IOU_ISSUE_SKIP_COMPLETE) return ret; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 002b6cc842a5..e908966f081a 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -30,12 +30,18 @@ int io_run_task_work_sig(struct io_ring_ctx *ctx); int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); void io_req_complete_failed(struct io_kiocb *req, s32 res); -void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); -void io_req_complete_post(struct io_kiocb *req); +void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); +static inline void io_req_complete_post_tw(struct io_kiocb *req, bool *locked) +{ + unsigned flags = *locked ? 0 : IO_URING_F_UNLOCKED; + + io_req_complete_post(req, flags); +} + struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages); struct file *io_file_get_normal(struct io_kiocb *req, int fd); diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 25cd724ade18..c33b53b7f435 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -311,7 +311,7 @@ int io_remove_buffers(struct io_kiocb *req, unsigned int issue_flags) /* complete before unlock, IOPOLL may need the lock */ io_req_set_res(req, ret, 0); - __io_req_complete(req, issue_flags); + io_req_complete_post(req, 0); io_ring_submit_unlock(ctx, issue_flags); return IOU_ISSUE_SKIP_COMPLETE; } @@ -460,7 +460,7 @@ int io_provide_buffers(struct io_kiocb *req, unsigned int issue_flags) req_set_fail(req); /* complete before unlock, IOPOLL may need the lock */ io_req_set_res(req, ret, 0); - __io_req_complete(req, issue_flags); + io_req_complete_post(req, 0); io_ring_submit_unlock(ctx, issue_flags); return IOU_ISSUE_SKIP_COMPLETE; } diff --git a/io_uring/poll.c b/io_uring/poll.c index 2830b7daf952..583fc0d745a6 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -298,7 +298,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) io_poll_tw_hash_eject(req, locked); if (ret == IOU_POLL_REMOVE_POLL_USE_RES) - io_req_complete_post(req); + io_req_complete_post_tw(req, locked); else if (ret == IOU_POLL_DONE) io_req_task_submit(req, locked); else diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index e50de0b6b9f8..446a189b78b0 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -56,7 +56,7 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2) /* order with io_iopoll_req_issued() checking ->iopoll_complete */ smp_store_release(&req->iopoll_completed, 1); else - __io_req_complete(req, 0); + io_req_complete_post(req, 0); } EXPORT_SYMBOL_GPL(io_uring_cmd_done); From patchwork Wed Nov 23 11:33:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13053520 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE50FC433FE for ; Wed, 23 Nov 2022 11:35:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237013AbiKWLf1 (ORCPT ); Wed, 23 Nov 2022 06:35:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236962AbiKWLfJ (ORCPT ); Wed, 23 Nov 2022 06:35:09 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A800FAEBA for ; Wed, 23 Nov 2022 03:35:08 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id b12so15145440wrn.2 for ; Wed, 23 Nov 2022 03:35:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SjP9WNNy11R90tHNf8lVvOLzHW5TTZzK32JlDgD+taA=; b=U9wjYC4y6qqGvEteKdOk7rZ4HJESVZiG/tl/aMX7MIQQAAb6bEN7YcRWB4BNZRAW6v soXZzDDmZ3nx81//5n+0KI6AkauCT7vNJjlip9KQ8dC86pyE0NNyY71M9x5cIubXmPS2 G45r4b+11S96NQ1UWC3B8HzGqqOtS8M68ZPCpX6JcvxUqIUpzmNd3gNXRwY0DXoGjWmL +idjErIHVDu0FVwKVb+FL2hUcXCdo0V+eOtPntIGuVwlV7hRUxBKD1KIF95rLRv5fRZG fK4Ss0PqANHNDn05UnigPsOM529jMDx5L5H+e84ZYN/LSbzslaqOr+48c2PK5m3koP6A Ftbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SjP9WNNy11R90tHNf8lVvOLzHW5TTZzK32JlDgD+taA=; b=uD5uo3q90ax5nZCIquI0e7wJnHJT3pNiO7jqyraDkaBH7QtKuF2MSerucnl2tZ6XbX 70GCUKT7toF+WpkTWIodTpnbi66dzY2PRXmjghtPPGLSOO/G2yZEWGShOvCTTDaw0qGd aFe0aKl2TalMERMXOvB9+ut6IeJXuYszaiTfzVGA8z1CbKFB94hjviMtMyD4X2VMysJG FXF66IRzZAgCCMepxQpCUcO03W17J9mkfE0RgjLztfDS1WKAmGSOR50pjVawRE5ZHUik Z/9aS9f997LzMXLsqqbmhHlm5f8mnaCo5JokD92KNmX7Ik3Owh+R4zBs27wOEYHD7Yo4 HzkQ== X-Gm-Message-State: ANoB5pmNcI2LBVxn+N6pEJTTfg7Ggd2qEvcny3EsaXd+ziEaBcIyd+Rd W0uDJJLyiy2tH38WQSP6gC9BXZnvoNY= X-Google-Smtp-Source: AA0mqf6iOCpvwT4MiPSKDWdV53apVtAdjZFBcpeV/QuwMLb2KI8HX2SylbqKry7P0fdHqHJVpKrD0w== X-Received: by 2002:adf:fe90:0:b0:232:eb2:6a33 with SMTP id l16-20020adffe90000000b002320eb26a33mr10299367wrr.325.1669203306608; Wed, 23 Nov 2022 03:35:06 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:e1b]) by smtp.gmail.com with ESMTPSA id g9-20020a05600c4ec900b003cfd58409desm2262064wmq.13.2022.11.23.03.35.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Nov 2022 03:35:06 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 7/7] io_uring: remove iopoll spinlock Date: Wed, 23 Nov 2022 11:33:42 +0000 Message-Id: <7e171c8b530656b14a671c59100ca260e46e7f2a.1669203009.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This reverts commit 85f0e5da546bb215a27103ac8c698b8f60309ee0 io_req_complete_post() should now behave well even in case of IOPOLL, we can remove completion_lock locking. Signed-off-by: Pavel Begunkov --- io_uring/rw.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/io_uring/rw.c b/io_uring/rw.c index 61c326831949..1ce065709724 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -1049,7 +1049,6 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) else if (!pos) return 0; - spin_lock(&ctx->completion_lock); prev = start; wq_list_for_each_resume(pos, prev) { struct io_kiocb *req = container_of(pos, struct io_kiocb, comp_list); @@ -1064,11 +1063,11 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) req->cqe.flags = io_put_kbuf(req, 0); __io_fill_cqe_req(req->ctx, req); } - io_commit_cqring(ctx); - spin_unlock(&ctx->completion_lock); + if (unlikely(!nr_events)) return 0; + io_commit_cqring(ctx); io_cqring_ev_posted_iopoll(ctx); pos = start ? start->next : ctx->iopoll_list.first; wq_list_cut(&ctx->iopoll_list, prev, start);