From patchwork Tue Jan 10 18:00:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13095442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E26AC46467 for ; Tue, 10 Jan 2023 18:03:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235331AbjAJSDr (ORCPT ); Tue, 10 Jan 2023 13:03:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238680AbjAJSDV (ORCPT ); Tue, 10 Jan 2023 13:03:21 -0500 Received: from mail-pg1-x530.google.com (mail-pg1-x530.google.com [IPv6:2607:f8b0:4864:20::530]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B3BB201C for ; Tue, 10 Jan 2023 10:01:01 -0800 (PST) Received: by mail-pg1-x530.google.com with SMTP id f3so8790456pgc.2 for ; Tue, 10 Jan 2023 10:01:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YBLTJKNJmXg7kXEDzNfqLolzkfF/YX6ZhQGxLLurmW4=; b=Xw4DaHPGOdoMkjAzraFF0T63CdFBrtAwbSLcaxNFl0r0tIUBueMO6GuaF4S3OewdK1 B2rEyhbBLJX8F6qY/kOQ1cbdVLdowfmUxV7XW0B9ZcLghMVQ+loSrUN1vIQYH+E8a0z6 qCpDPui4cFKBs7AtQt4TBAf6W+BWaQesOJJHvLwHbEsK7ccFpycgWHPUemNxVVyMibsk /LZNqPJ/X734D3rQMZVeHlvr/wYC2Mtog0Qqm0N0jFpRXfmBSqVxtZ7Hzf2b+qaTBnRk JoLFhfVxhrz1Os+vUrZBqcF9rPi4Ei3ISSN8xdSpugzm/sLG3ljta9Cr5Php8m5CNzj0 nhQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YBLTJKNJmXg7kXEDzNfqLolzkfF/YX6ZhQGxLLurmW4=; b=jXi5Nw5M8sm6A9LYxHuhOesok9GiOCXATzmugnG4RhWdMQlSxDOktaZ0bhiFnIwEsD 7wqfZiWPmhtrG1J56/AvXlSKt2Tl2GTe78uhlhavl3nVza3PKvg8ezNivARjQyWloUec rBDQj5KjttvggMQ/tir/Km2KSo1pzN1l9kX5Xk/3zO0hIkc903FIGf8NL1zSOX7nGKjM N9pFj8DmTLAXJ4qGcSMswBT107GKxcMsPxE9wJ8cxs6oOkN81j9xFj3NntUIceuvw2JQ zGXr3sYZrXBy/nJqnXrVZ3V4ABUvWC3+TYUYKjM9CjSz3DuJhOifL+0YwTITLXt2Jdvl Ubhg== X-Gm-Message-State: AFqh2kqGWIsFUdcXzcw+q3LwXAcmIy9tJv8fgVTphNPj6OP20b9Cv1R9 1hil/wI/vPEaIbtVVOUCebjMQ9WbUix6wW+z X-Google-Smtp-Source: AMrXdXubZ//VBneOk2DqToKpdNnG5H/zjDo7dfEUk8+bmP6Te8tZPRJCzqxXCNKBWgB0WxeTqtf6uw== X-Received: by 2002:a62:3845:0:b0:58b:453e:d12d with SMTP id f66-20020a623845000000b0058b453ed12dmr268438pfa.1.1673373660304; Tue, 10 Jan 2023 10:01:00 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id i24-20020a056a00225800b00583698ba91dsm8405877pfu.40.2023.01.10.10.00.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Jan 2023 10:00:59 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 1/2] io_uring/fdinfo: include locked hash table in fdinfo output Date: Tue, 10 Jan 2023 11:00:54 -0700 Message-Id: <20230110180055.204657-2-axboe@kernel.dk> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230110180055.204657-1-axboe@kernel.dk> References: <20230110180055.204657-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org A previous commit split the hash table for polled requests into two parts, but didn't get the fdinfo output updated. This means that it's less useful for debugging, as we may think a given request is not pending poll. Fix this up by dumping the locked hash table contents too. Fixes: 9ca9fb24d5fe ("io_uring: mutex locked poll hashing") Signed-off-by: Jens Axboe --- io_uring/fdinfo.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c index 2e04850a657b..882bd56b01ed 100644 --- a/io_uring/fdinfo.c +++ b/io_uring/fdinfo.c @@ -170,12 +170,11 @@ static __cold void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, xa_for_each(&ctx->personalities, index, cred) io_uring_show_cred(m, index, cred); } - if (has_lock) - mutex_unlock(&ctx->uring_lock); seq_puts(m, "PollList:\n"); for (i = 0; i < (1U << ctx->cancel_table.hash_bits); i++) { struct io_hash_bucket *hb = &ctx->cancel_table.hbs[i]; + struct io_hash_bucket *hbl = &ctx->cancel_table_locked.hbs[i]; struct io_kiocb *req; spin_lock(&hb->lock); @@ -183,8 +182,17 @@ static __cold void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, seq_printf(m, " op=%d, task_works=%d\n", req->opcode, task_work_pending(req->task)); spin_unlock(&hb->lock); + + if (!has_lock) + continue; + hlist_for_each_entry(req, &hbl->list, hash_node) + seq_printf(m, " op=%d, task_works=%d\n", req->opcode, + task_work_pending(req->task)); } + if (has_lock) + mutex_unlock(&ctx->uring_lock); + seq_puts(m, "CqOverflowList:\n"); spin_lock(&ctx->completion_lock); list_for_each_entry(ocqe, &ctx->cq_overflow_list, list) { From patchwork Tue Jan 10 18:00:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13095443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06EA2C54EBC for ; Tue, 10 Jan 2023 18:03:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231236AbjAJSDs (ORCPT ); Tue, 10 Jan 2023 13:03:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45478 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238177AbjAJSDV (ORCPT ); Tue, 10 Jan 2023 13:03:21 -0500 Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A8A99FD5 for ; Tue, 10 Jan 2023 10:01:02 -0800 (PST) Received: by mail-pg1-x531.google.com with SMTP id 7so8793962pga.1 for ; Tue, 10 Jan 2023 10:01:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yQjfdBAuI+lCVIhAS7ja+ercSmY/+2TyjJmiFgzuNZM=; b=ehO+pW7c9jcZ5X0YHTx2bs3lN8BdFLqIHPcguf/KQk/WuFg8QWIOjQma+kQAR7ImI8 PbZPlRCqAKEFMxxew+pVxfmpE7IlOReCc5w90lTBYp2O0baHYRWV6uEDzeMNuD2xLbzg ZkrkKxK8aOaSRUqkZ0eI577Hfo8R17hm612IhzR2XxW835I19vkblL7zHNDL+IIZAq4+ jB1NNlpuJ7QXeefjDPJtbSQ0yvrpcmogqLIJqMHhiwBnPaK5ykGROLHFHnBxeJ+DU7Cg QkUHqgdQ3/JuSaENOGDQKeb0fhF7/9UoYxPBWajcBKlAe7AAPgUpWf9FcEsCmqEUfXnv oshg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yQjfdBAuI+lCVIhAS7ja+ercSmY/+2TyjJmiFgzuNZM=; b=FQRrvhZzIPLYQNQE2dzH2OyxehU7z4pfCCmwb1qBMYKjOcs8LBVLMOdvFcxOHPEk/g LsrITGCCpx6PQFghqW/clQPho5TIGETAAIR90QEnNOtcJhRE+kYnMZ5dT3/kfJvSWQbY Lz6HYbS/JbQpXGJLCFkpGhlp9ecgvn0xdX9uV/xLYbcCi2p75NUNr7oPeDx4rkzYboq0 eZFs4U9fOTvDi47QmFfk459kF1Rh2S8rpcOr46oA/QyqA+0xye7e36S2gx8fSCNXIlBp QvFAihL2d5ZvRmL96F3htfzfn3J80xW2yRN7QzrnDtlqztmDJbOjYnu6qy0G1b4a/yYU T7Ag== X-Gm-Message-State: AFqh2kpnnmnCh70TfKLBqpmKHcP6/E/9+DUwqHZY72FNA3v4oOTYadEm 7GmWLpHEN7Pbn71qgjVFYh0yP81rME75r17X X-Google-Smtp-Source: AMrXdXsbK7c7PFA8/80nEZD8iBW5Lp052SkcqSmIAqs7oKHwM2X4HItK1ULxy9XA5m2d5r127W6RnA== X-Received: by 2002:a62:1695:0:b0:57f:f570:128d with SMTP id 143-20020a621695000000b0057ff570128dmr17615137pfw.1.1673373661385; Tue, 10 Jan 2023 10:01:01 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id i24-20020a056a00225800b00583698ba91dsm8405877pfu.40.2023.01.10.10.01.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Jan 2023 10:01:00 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 2/2] io_uring/poll: attempt request issue after racy poll wakeup Date: Tue, 10 Jan 2023 11:00:55 -0700 Message-Id: <20230110180055.204657-3-axboe@kernel.dk> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230110180055.204657-1-axboe@kernel.dk> References: <20230110180055.204657-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org If we have multiple requests waiting on the same target poll waitqueue, then it's quite possible to get a request triggered and get disappointed in not being able to make any progress with it. If we race in doing so, we'll potentially leave the poll request on the internal tables, but removed from the waitqueue. That means that any subsequent trigger of the poll waitqueue will not kick that request into action, causing an application to potentially wait for completion of a request that will never happen. Fix this by adding a new poll return state, IOU_POLL_REISSUE. Rather than have complicated logic for how to re-arm a given type of request, just punt it for a reissue. While in there, move the 'ret' variable to the only section where it gets used. This avoids confusion the scope of it. Fixes: 49f1c68e048f ("io_uring: optimise submission side poll_refs") Signed-off-by: Jens Axboe --- io_uring/poll.c | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index cf6a70bd54e0..0aca4ee6142b 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -223,21 +223,22 @@ enum { IOU_POLL_DONE = 0, IOU_POLL_NO_ACTION = 1, IOU_POLL_REMOVE_POLL_USE_RES = 2, + IOU_POLL_REISSUE = 3, }; /* * All poll tw should go through this. Checks for poll events, manages * references, does rewait, etc. * - * Returns a negative error on failure. IOU_POLL_NO_ACTION when no action require, - * which is either spurious wakeup or multishot CQE is served. - * IOU_POLL_DONE when it's done with the request, then the mask is stored in req->cqe.res. - * IOU_POLL_REMOVE_POLL_USE_RES indicates to remove multishot poll and that the result - * is stored in req->cqe. + * Returns a negative error on failure. IOU_POLL_NO_ACTION when no action + * require, which is either spurious wakeup or multishot CQE is served. + * IOU_POLL_DONE when it's done with the request, then the mask is stored in + * req->cqe.res. IOU_POLL_REMOVE_POLL_USE_RES indicates to remove multishot + * poll and that the result is stored in req->cqe. */ static int io_poll_check_events(struct io_kiocb *req, bool *locked) { - int v, ret; + int v; /* req->task == current here, checking PF_EXITING is safe */ if (unlikely(req->task->flags & PF_EXITING)) @@ -276,10 +277,15 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) if (!req->cqe.res) { struct poll_table_struct pt = { ._key = req->apoll_events }; req->cqe.res = vfs_poll(req->file, &pt) & req->apoll_events; + /* + * We got woken with a mask, but someone else got to + * it first. The above vfs_poll() doesn't add us back + * to the waitqueue, so if we get nothing back, we + * should be safe and attempt a reissue. + */ + if ((unlikely(!req->cqe.res))) + return IOU_POLL_REISSUE; } - - if ((unlikely(!req->cqe.res))) - continue; if (req->apoll_events & EPOLLONESHOT) return IOU_POLL_DONE; @@ -294,7 +300,7 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) return IOU_POLL_REMOVE_POLL_USE_RES; } } else { - ret = io_poll_issue(req, locked); + int ret = io_poll_issue(req, locked); if (ret == IOU_STOP_MULTISHOT) return IOU_POLL_REMOVE_POLL_USE_RES; if (ret < 0) @@ -330,6 +336,8 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) poll = io_kiocb_to_cmd(req, struct io_poll); req->cqe.res = mangle_poll(req->cqe.res & poll->events); + } else if (ret == IOU_POLL_REISSUE) { + io_req_task_submit(req, locked); } else if (ret != IOU_POLL_REMOVE_POLL_USE_RES) { req->cqe.res = ret; req_set_fail(req); @@ -342,7 +350,7 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) if (ret == IOU_POLL_REMOVE_POLL_USE_RES) io_req_task_complete(req, locked); - else if (ret == IOU_POLL_DONE) + else if (ret == IOU_POLL_DONE || ret == IOU_POLL_REISSUE) io_req_task_submit(req, locked); else io_req_defer_failed(req, ret);