From patchwork Thu Sep 7 12:50:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13376662 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95725EC8743 for ; Thu, 7 Sep 2023 16:59:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236287AbjIGQ7c (ORCPT ); Thu, 7 Sep 2023 12:59:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243634AbjIGQm5 (ORCPT ); Thu, 7 Sep 2023 12:42:57 -0400 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14CD61BC9 for ; Thu, 7 Sep 2023 09:41:55 -0700 (PDT) Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-52e5900cf77so1591674a12.2 for ; Thu, 07 Sep 2023 09:41:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694104859; x=1694709659; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MSrZuUeNMuob64GpbHLuDyuPGaYxErAJAjtjCoeRfL0=; b=Tsruertx/Cw2zGs0edaRFI3BiFpuMEx3+IZJRAq9k50ntsgn/hNKK9ZflNKnx85qSb Jb2MWYo8qZb0CArgpVHMo1oyUQmThSzqmtGJ7K0GIU78tzWzb9zmqlKiDsopJ3E4GBsp +9YjVF0XBUlzCMM2TkxDbeUFBWvNV8p/Qw5Rvrosrk1XVxyPbyeU5yFtCbNcf3x1zBJd vuxVwx81cIyJqJu56Btkw5NYD1aUI4Bn8iYVJEw9iuFCSzRGZi8SUL5ggfpKR8Uf2f+U 89Mh+xP4fJIKeyuCb1zcr1el66JCLeILowuyVs+VP083AHHAC2I8o2GzIM1hHv0NJeZ/ Z9tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694104859; x=1694709659; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MSrZuUeNMuob64GpbHLuDyuPGaYxErAJAjtjCoeRfL0=; b=MQgcy8xJchNPdt+uPddK8DqYH1YhqSjOTpCv+o6M3ptHQxBUol6hZHL3a5OO5Gk08D oJonWdyrQL0aWT2Hihm4002BCpy5Xy9lKb5qC8WfYHf8Q11GB3PRNrbcKMBMxOSYzJKr k3+KfcYb8/3ELCUE4HOVzE2t0CwszufKd+3WXA4grP+gbI569Ii0UMcrg85pKJyeus7I MRH97396mV0eI5DlK7bhiEP2AiqAFHKdSenxOOxuezSjnQbGAMXdXP2E9yhBH9/6Zxym ATWMpZ4QwDr/qhSslFx6SzHhYl1qKwsTXN2yTdasOdiw8jTFz9OVuriDRfD2bgRlYBKs UviA== X-Gm-Message-State: AOJu0YzkSR4rXMualKG5vlp1juJuBuNsjFG2m49DB8K5uaBKVn8fedLA DY8rqto7CIy3TLCVUsKNdp47F4PHcX8= X-Google-Smtp-Source: AGHT+IFZp/xoc6auc0uhRtQNkoA2V+g6WlYajwVPXk2id26mVku3eVwu09eUpbW4F33NLWw7DA531A== X-Received: by 2002:aa7:d402:0:b0:514:9ab4:3524 with SMTP id z2-20020aa7d402000000b005149ab43524mr4994167edq.7.1694091034410; Thu, 07 Sep 2023 05:50:34 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.16]) by smtp.gmail.com with ESMTPSA id p11-20020a056402074b00b005231e1780aasm9612279edy.91.2023.09.07.05.50.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Sep 2023 05:50:34 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 1/2] io_uring: break out of iowq iopoll on teardown Date: Thu, 7 Sep 2023 13:50:07 +0100 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io-wq will retry iopoll even when it failed with -EAGAIN. If that races with task exit, which sets TIF_NOTIFY_SIGNAL for all its workers, such workers might potentially infinitely spin retrying iopoll again and again and each time failing on some allocation / waiting / etc. Don't keep spinning if io-wq is dying. Fixes: 561fb04a6a225 ("io_uring: replace workqueue usage with io-wq") Cc: stable@vger.kernel.org Signed-off-by: Pavel Begunkov --- io_uring/io-wq.c | 10 ++++++++++ io_uring/io-wq.h | 1 + io_uring/io_uring.c | 2 ++ 3 files changed, 13 insertions(+) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 62f345587df5..1ecc8c748768 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -174,6 +174,16 @@ static void io_worker_ref_put(struct io_wq *wq) complete(&wq->worker_done); } +bool io_wq_worker_stopped(void) +{ + struct io_worker *worker = current->worker_private; + + if (WARN_ON_ONCE(!io_wq_current_is_worker())) + return true; + + return test_bit(IO_WQ_BIT_EXIT, &worker->wq->state); +} + static void io_worker_cancel_cb(struct io_worker *worker) { struct io_wq_acct *acct = io_wq_get_acct(worker); diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index 06d9ca90c577..2b2a6406dd8e 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -52,6 +52,7 @@ void io_wq_hash_work(struct io_wq_work *work, void *val); int io_wq_cpu_affinity(struct io_uring_task *tctx, cpumask_var_t mask); int io_wq_max_workers(struct io_wq *wq, int *new_count); +bool io_wq_worker_stopped(void); static inline bool io_wq_is_hashed(struct io_wq_work *work) { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 88599852af82..4674203c1cac 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1942,6 +1942,8 @@ void io_wq_submit_work(struct io_wq_work *work) if (!needs_poll) { if (!(req->ctx->flags & IORING_SETUP_IOPOLL)) break; + if (io_wq_worker_stopped()) + break; cond_resched(); continue; } From patchwork Thu Sep 7 12:50:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13376661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 367A0EC873E for ; Thu, 7 Sep 2023 16:59:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236814AbjIGQ7C (ORCPT ); Thu, 7 Sep 2023 12:59:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245073AbjIGQ7B (ORCPT ); Thu, 7 Sep 2023 12:59:01 -0400 Received: from mail-lj1-x231.google.com (mail-lj1-x231.google.com [IPv6:2a00:1450:4864:20::231]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 769E5E46 for ; Thu, 7 Sep 2023 09:58:27 -0700 (PDT) Received: by mail-lj1-x231.google.com with SMTP id 38308e7fff4ca-2bcc187e0b5so21141641fa.1 for ; Thu, 07 Sep 2023 09:58:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694105841; x=1694710641; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HkUs4bQ1E8KB48LLONDQtfwRnAGik6JPV6A26/jKNqY=; b=QgiC9/g+s61gWBOwiUegN3iLqZ9Hre7snJXt9Ei9DwA98FwizVrZod9NAnEqQhovQz FgsshU3g5imQUeCnvjqWKCBzM9CEmW3AYjpuBj27TdsKLFYyqGBDMj1mz4EZqFQttyNf f/KT3a1vRUkvELgq3hdGoCuqFyp+1LgYsujHw6n+ZkkPjY0Lrr1y8B06vRzP69KYHm4r SMHwf7D5z9A9K9xcLoTG0sVImMzUTdoFuvTsXEn3m8Q6YC7v75V3WWJDusBWY3Fg126d cv8HVJnoxOMY6Xb53NFk1vxwG7Wrmk97wmPtlXFZgoG+D7BhBCSJ3UhxV5jD9SGdVYjY 3qDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694105841; x=1694710641; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HkUs4bQ1E8KB48LLONDQtfwRnAGik6JPV6A26/jKNqY=; b=nbZToe6qEUAAxWF4asxFU1TZCn9GtZjzMgQ4kdAyECo/z/WoYLl7KwbDwB+pRryR4r Xhdo7lIjar8BJIxLhB5LfOZIkNAwPa5SqGhHwihrE5G/2z7RI4b9jZ8p3dkQQGN8F4dK IyVxjRW+TrZQPjpSicHNbPrSpwuGBblatD5x4DAIGxkmXARFDQI8siuH3nF30JtuPfpQ cVP1AagmmsUkh6I1TIoGFX0Vm+q61sZQ2XfKAFqqIBRakTI3JQNiY3edmxFbr76wFar1 ASPJK3I3iymste4kLbbcg4SIysqBtKgMlM/mZIxCvrHE0NVMpBVjncnEfXKjXp2T+zx3 dj5g== X-Gm-Message-State: AOJu0YxEX3k4Lsdy4Fr+0/6z9H2g7VU7J3n4UnNbYNm6lFCxY9CjvFaO 3fAMopyTjzxYd9Y3ywFTeoq9V/wTn54= X-Google-Smtp-Source: AGHT+IGmH7p2LBA2oJDPrCEkq7/BtszlGP65Sf1ghdvfgVDXBbHtkGRDy0w/m1A5DF5q3u45er8JxA== X-Received: by 2002:a19:650f:0:b0:4fd:d08c:fa3e with SMTP id z15-20020a19650f000000b004fdd08cfa3emr4512294lfb.42.1694091035467; Thu, 07 Sep 2023 05:50:35 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.16]) by smtp.gmail.com with ESMTPSA id p11-20020a056402074b00b005231e1780aasm9612279edy.91.2023.09.07.05.50.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Sep 2023 05:50:34 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 2/2] io_uring: fix unprotected iopoll overflow Date: Thu, 7 Sep 2023 13:50:08 +0100 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org [ 71.490669] WARNING: CPU: 3 PID: 17070 at io_uring/io_uring.c:769 io_cqring_event_overflow+0x47b/0x6b0 [ 71.498381] Call Trace: [ 71.498590] [ 71.501858] io_req_cqe_overflow+0x105/0x1e0 [ 71.502194] __io_submit_flush_completions+0x9f9/0x1090 [ 71.503537] io_submit_sqes+0xebd/0x1f00 [ 71.503879] __do_sys_io_uring_enter+0x8c5/0x2380 [ 71.507360] do_syscall_64+0x39/0x80 We decoupled CQ locking from ->task_complete but haven't fixed up places forcing locking for CQ overflows. Fixes: ec26c225f06f5 ("io_uring: merge iopoll and normal completion paths") Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4674203c1cac..6cce8948bddf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -883,7 +883,7 @@ static void __io_flush_post_cqes(struct io_ring_ctx *ctx) struct io_uring_cqe *cqe = &ctx->completion_cqes[i]; if (!io_fill_cqe_aux(ctx, cqe->user_data, cqe->res, cqe->flags)) { - if (ctx->task_complete) { + if (ctx->lockless_cq) { spin_lock(&ctx->completion_lock); io_cqring_event_overflow(ctx, cqe->user_data, cqe->res, cqe->flags, 0, 0); @@ -1541,7 +1541,7 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) if (!(req->flags & REQ_F_CQE_SKIP) && unlikely(!io_fill_cqe_req(ctx, req))) { - if (ctx->task_complete) { + if (ctx->lockless_cq) { spin_lock(&ctx->completion_lock); io_req_cqe_overflow(req); spin_unlock(&ctx->completion_lock);