From patchwork Wed Jan 17 00:57:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13521333 Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D559E10EB for ; Wed, 17 Jan 2024 00:59:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705453168; cv=none; b=FoQowb4UUg8tAVR3+r5VybkKE0OyJHcMtDDoZGNOcUXC37TtxMCm/U9npnBbcCgTcexCh/+HLRapgte3dUTR3bYcWCFXv0Bw/CwBgWJFNn/sZNEnUPswePiepOLB+oX//1RHno81ZFbF3wu8VuyPoFxg1IWLkUQFDGOLOrtH6+E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705453168; c=relaxed/simple; bh=YMznjEf2a8GlGttKETk1u68AD78RsrhIxfuBPyoIpow=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:Received:From: To:Cc:Subject:Date:Message-ID:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; b=PU1LRQwwMgX83L+v49LFwBr3DwUL2j1wXwnupZsHZtFm2CPu0Z9OX3czPH8vqhgorhK1nZap6fcqEw0iNQGCP5/f2pQ3QeyRoQDjNtWExHiH+sGg+qs2Nhyoh42cWE3Hz+iOxHI26ZGiA88qviJGf40SwlMieHRkx0JX7kOGw+A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GJW6ckN7; arc=none smtp.client-ip=209.85.218.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GJW6ckN7" Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-a26ed1e05c7so1255981566b.2 for ; Tue, 16 Jan 2024 16:59:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705453164; x=1706057964; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=12lNFekc1W9jCZeXaALJhzL0q/MnoIeGAfSUPTmqtYE=; b=GJW6ckN7TtgljZtthHhukCs8z1sq8D08T9FD58AAD2Bt/w6oEzxgyeaedx4n/9q7nM XNMrd2wark3dCRwCPE8OzyPPraHvipzVTvRAFQLdKwQ6yQmrzHoWMnUbxDcP23j35VnZ fH/JsgAugK6Gi1PKbXlgkvvLk6nIdg/e7K+Sp4BBD/V4khpPz04Rr6JMcYYh2EEfx/32 sH/efDPP4leLc+BIwYwKwVjYDtMOWm/QB37/iEl0GJAPdrkNTN9xMbJuqzRUz77OJxpZ JebeCsaukZbpYKUYiiLjNFEj+9DDR89TaFWI3guGLWRbJUCc1Mnh0GKCeVzCYGI9+L3X t1Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705453164; x=1706057964; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=12lNFekc1W9jCZeXaALJhzL0q/MnoIeGAfSUPTmqtYE=; b=UzXcCtIc4EuU70ISjK/xpDXrWk/hagELjqXVCQICd5TyDrqqFDiBbtDUoS6lbKkAbT OAETk96XmwPXsaEnsbY4DovMOb+g3QbcVJ9fGxg/wPpTg+0beE6NC6hVE9ivWJrhqDI3 O0+sNl8RDnGG3CIDYfPMpQiaX9H87M0CNdtFkig9nglMD/AVnzZQIsGZIW8+dviIDkDE uNvexLMu0u737SV1wf9rqySppnbSWlnCHwVT8nhsJ/UCTz8EfvR8lmYNUciYdrQAN+4W pwjD94wJPj63vWyEf6+nRvuvZrm6C17Fe43Gu5vZOiSJLjTjMYp075bMFuG68Ib2qUCw tqHQ== X-Gm-Message-State: AOJu0YxpQSu0LHHFrYIcfDTpJpEZrW4GpweeA04x6HC7/nJGF1Cg/bLY YMSCydqgYsJ7XNxsCdhyDgHWRsuO8LY= X-Google-Smtp-Source: AGHT+IENoGOkfZ9OQGpcilq7E43F0vj3/UPqCZf9p+Mm7ASHIerNTx9fB4AU0pb9VHMvh+Hgvbn/1A== X-Received: by 2002:a17:906:1411:b0:a23:4e3a:4356 with SMTP id p17-20020a170906141100b00a234e3a4356mr2271441ejc.182.1705453164528; Tue, 16 Jan 2024 16:59:24 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.96]) by smtp.gmail.com with ESMTPSA id t15-20020a17090605cf00b00a28aa4871c7sm7038982ejt.205.2024.01.16.16.59.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jan 2024 16:59:23 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 1/4] io_uring: adjust defer tw counting Date: Wed, 17 Jan 2024 00:57:26 +0000 Message-ID: <108b971e958deaf7048342930c341ba90f75d806.1705438669.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The UINT_MAX work item counting bias in io_req_local_work_add() in case of !IOU_F_TWQ_LAZY_WAKE works in a sense that we will not miss a wake up, however it's still eerie. In particular, if we add a lazy work item after a non-lazy one, we'll increment it and get nr_tw==0, and subsequent adds may try to unnecessarily wake up the task, which is though not so likely to happen in real workloads. Half the bias, it's still large enough to be larger than any valid ->cq_wait_nr, which is limited by IORING_MAX_CQ_ENTRIES, but further have a good enough of space before it overflows. Fixes: 8751d15426a31 ("io_uring: reduce scheduling due to tw") Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 50c9f04bc193..d40c767a6216 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1325,7 +1325,7 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) nr_tw = nr_tw_prev + 1; /* Large enough to fail the nr_wait comparison below */ if (!(flags & IOU_F_TWQ_LAZY_WAKE)) - nr_tw = -1U; + nr_tw = INT_MAX; req->nr_tw = nr_tw; req->io_task_work.node.next = first; From patchwork Wed Jan 17 00:57:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13521334 Received: from mail-lf1-f48.google.com (mail-lf1-f48.google.com [209.85.167.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E3F91364 for ; Wed, 17 Jan 2024 00:59:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705453170; cv=none; b=iLVSYrpc9iq/xmc8B3S3p5Og300TgbNgeXGFAngr1Fy8zvtUTmMmD1IY64BpDFdnOdnxNrBPaxJKQZL6rQLWT9ajFsql2zXULm/WCThDhx0C/PGiUTGqzzik/YtQ0ptxenPCM1hc7HAK//PccTBaR+hISj6HlezwiKWbN5rWM8A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705453170; c=relaxed/simple; bh=GZOtWT8RhrA8gP3IHGXdFKHEPDDMisYczZFB8QHXRbA=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:Received:From: To:Cc:Subject:Date:Message-ID:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; b=Q0te20ZgBkHOCbG21tu96dHxXGOlMLs8ntyGiHeCPUxXuYL0Yn0sfFVci/VLfieQzQ7WtUnMHNpUQ/vV4ZGuM9K0SCWUonbHPmzLlsoOJXGZmPbuAWbOQ6TJChNRww7kPpM3liCWUG3snYu3szBZRmHBkYLidym/dCJdl4tJJec= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Lf2j/YOK; arc=none smtp.client-ip=209.85.167.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Lf2j/YOK" Received: by mail-lf1-f48.google.com with SMTP id 2adb3069b0e04-50e7c6e3c63so10877537e87.3 for ; Tue, 16 Jan 2024 16:59:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705453166; x=1706057966; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QTWrBvavpdON3JOLMJEeULVf6rRDB6nv+5wcPW2bSRI=; b=Lf2j/YOKC7CfbjQs0/+cksM38oXSG2LsM3wyJy1Ctf+fGZp6UHVahFUpqP1oO5vDf8 GpXceG0r01J1RfFO8Ew1lbKpyTqJdjzHDVZoFXbGL/vuNQmm7cT0Qiic4X7vcyH7kcON qjrH2do8DOl5HLeSuMgEqgjhj+EG8gto0OlzLR4osnHE4bz0axiWaour4uNHFBhLtiJm 1BLRiSojNswzAbRJMqzIHJnHlqtTT+DiMB/bkcG4eDTTt/zD8wGqAf4i1VZp9oS9RSub mgTl3OwP8fTwjjgyZaC7usHpMWM4DfCtrSHpeCLwuBgXZM8iq350c0tnYay2LJDz7tBz X6cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705453166; x=1706057966; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QTWrBvavpdON3JOLMJEeULVf6rRDB6nv+5wcPW2bSRI=; b=Sf5DEY6Rbt2CgjyKj3m+ppvH2EReYy4d4eQXr3k+PpzjBXSO57E5D5PVfKQ/iLqTjY gcbYlDBSbllFSpO6FV0XSq1zBs265fG8BAyqN9YPSkCOhbmgcpPjm6HO1Jr0L7i9gHye dWjLCXy4XFdIJwrVyknEkQEY07OaDSZZ2sSTtnm8Q96Im4Gvp78Q8Aed4Zl7aceyVBT8 VVgkx7cn+8K6ezy1JUnj09UtdwEuoCjxHrDpYlaGhCE14Ing/xILRwf9t8ueZJDwRYdg n7nKVMFbTwVP9j7aSmyS9Te227ByE7f+dD5Oj05wAmDE/cAbdQvtmXGMu6q/kWwhiSic e7pQ== X-Gm-Message-State: AOJu0YyLwKGZWLlds1Ei1doBooLS/2PlyxF/bDbYdiMUyHp12eI4i1ZV wFfVGgMQ0AgsXJkvuBg89nZ4sxq5Bcg= X-Google-Smtp-Source: AGHT+IE8i20rvraovxuRj/c4/dDkPwEzPHW7RgfeCEoRwXafNYoXQSSvm1KVNxhvPbDrYV9PKbFgNQ== X-Received: by 2002:ac2:4a8c:0:b0:50e:7f67:b669 with SMTP id l12-20020ac24a8c000000b0050e7f67b669mr1704452lfp.65.1705453165677; Tue, 16 Jan 2024 16:59:25 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.96]) by smtp.gmail.com with ESMTPSA id t15-20020a17090605cf00b00a28aa4871c7sm7038982ejt.205.2024.01.16.16.59.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jan 2024 16:59:24 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 2/4] io_uring: clean up local tw add-wait sync Date: Wed, 17 Jan 2024 00:57:27 +0000 Message-ID: <3007f3c2d53c72b61de56919ef56b53158b8276f.1705438669.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Kill a smp_mb__after_atomic() right before wake_up, it's useless, and add a comment explaining implicit barriers from cmpxchg and synchronsation around ->cq_wait_nr with the waiter. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d40c767a6216..3ab7e6a46149 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1332,6 +1332,14 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) } while (!try_cmpxchg(&ctx->work_llist.first, &first, &req->io_task_work.node)); + /* + * cmpxchg implies a full barrier, which pairs with the barrier + * in set_current_state() on the io_cqring_wait() side. It's used + * to ensure that either we see updated ->cq_wait_nr, or waiters + * going to sleep will observe the work added to the list, which + * is similar to the wait/wawke task state sync. + */ + if (!first) { if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); @@ -1346,8 +1354,6 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) /* either not enough or the previous add has already woken it up */ if (nr_wait > nr_tw || nr_tw_prev >= nr_wait) return; - /* pairs with set_current_state() in io_cqring_wait() */ - smp_mb__after_atomic(); wake_up_state(ctx->submitter_task, TASK_INTERRUPTIBLE); } From patchwork Wed Jan 17 00:57:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13521335 Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34266138E for ; Wed, 17 Jan 2024 00:59:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705453170; cv=none; b=LE2TV/1IHr+x0hLRNSJtdZKhwrl0zp1wh+3ql1PcaPV+lMWJI2cg8m1MS/pEEUqHuAVd/DFRx7B7TjB2fsLME65moU75oo1wiRFcomnXG5XnjYZ6NVY1cNE6ya0EmfWZhfLyLpdAELxU6675yoLNYU4nEf4fj52FLgj0nL4Rgdw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705453170; c=relaxed/simple; bh=rRFNEh5/N51L2i8Mdq5SIEhRw7W3IA/LOnvqibLLf78=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:Received:From: To:Cc:Subject:Date:Message-ID:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; b=tG0yQUc0ioObmR/5PKs83J3PmWmuyfLuCib0/EhDjdvWFgQ1B1FdplXps02rLtfYtCjXlnoo4PAN9b/+MlSrGkxecAEn5wXgvlW7n9sw/O2j6FKb2sIjk7C3ZN/VbQlL4IiqqzQZQKSxVKc8Mqli3aHApyIfEED0y6ZYpU9TD34= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XwJo1JTL; arc=none smtp.client-ip=209.85.208.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XwJo1JTL" Received: by mail-ed1-f52.google.com with SMTP id 4fb4d7f45d1cf-55817a12ad8so8410138a12.2 for ; Tue, 16 Jan 2024 16:59:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705453167; x=1706057967; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0tg1lNXY4N8veawkRu72zeREgoxTdgotTNGnpnLmM60=; b=XwJo1JTLR9TPOVogdYoHWVPIJgFTB9bIhxA5nVoCskwOYpeSGwPfbA02POiCo+3a+V +m2LhvEJXGXYNLp4YqwuLAsv3EGTQc42xZnF89wHLDDKykB4++CdrMMYpS4wClFyDqdv HG9KlilMlxC9saMST8okXMkNooeEK7y6wTJNnRu2jlK+BMqVffaIaoHGhGQ9uWkRUrY5 ynKiSsPr3/Kc0QJJpV29JY4PXaCEaHZljYqXtA9fJbekE6ayd+3t1Z8wLQbCYTHzkea6 gjHgRED18RC6meUxwUQV3rKuiS0lqXMlTFmP/X+az4QySljLD7220z3kPIqECnClbYAX TNag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705453167; x=1706057967; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0tg1lNXY4N8veawkRu72zeREgoxTdgotTNGnpnLmM60=; b=D1GB6AtF98oAkO46s+5+MD53XwbwytgKtAg2qEftANOp9uRmdm3um057k1lPS1xg5F RnsoWD2vKQDVklxTNHalkhlifMkMzmcaiAJnMQN2cF3zSMGmQcvbXouYLjh7/NkanzvU Vro1s+vuTXk63hnkoxtLUhgji8WbxEmmvr3IrqEdv8Vb5DqZmJfRqukRhZGkOHZdXz9S klR0rk9nxiOTr/djlmL8t+ViCrPM3OduqXwpXKDSHsNGq3RtqgJ+y6KM1oznFctLQcnC +Dm5A9DDSk4l+h7WEbZi9ID4JVbO2dAd8xGSt2RojfensHphZZIvNTWCETAvJNHJFAZe Xshw== X-Gm-Message-State: AOJu0YyTLUlwCch/HBID7Ft592T7cnqcqPb4Yb1Ju1TP35B2nqbTJdF0 zNljbrzYxTRYXcE296qngMk7WqFo9Qw= X-Google-Smtp-Source: AGHT+IHhNYuofaQX09fGAVkWba+n7ihK8UYhm8pSbuqC90qp50ZtF2m4lbVeBVW3d1W7A73YlSmKPA== X-Received: by 2002:a17:906:38c2:b0:a28:b021:abe5 with SMTP id r2-20020a17090638c200b00a28b021abe5mr3903084ejd.126.1705453167123; Tue, 16 Jan 2024 16:59:27 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.96]) by smtp.gmail.com with ESMTPSA id t15-20020a17090605cf00b00a28aa4871c7sm7038982ejt.205.2024.01.16.16.59.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jan 2024 16:59:26 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 3/4] io_uring: clean *local_work_add var naming Date: Wed, 17 Jan 2024 00:57:28 +0000 Message-ID: <3b8be483b52f58a524185bb88694b8a268e7e85d.1705438669.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 if (!first) { ... } While it reads as do something if it's not the first entry, it does exactly the opposite because "first" here is a pointer to the first entry. Remove the confusion by renaming it into "head". Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3ab7e6a46149..3508198d17ba 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1304,16 +1304,16 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) { struct io_ring_ctx *ctx = req->ctx; unsigned nr_wait, nr_tw, nr_tw_prev; - struct llist_node *first; + struct llist_node *head; if (req->flags & (REQ_F_LINK | REQ_F_HARDLINK)) flags &= ~IOU_F_TWQ_LAZY_WAKE; - first = READ_ONCE(ctx->work_llist.first); + head = READ_ONCE(ctx->work_llist.first); do { nr_tw_prev = 0; - if (first) { - struct io_kiocb *first_req = container_of(first, + if (head) { + struct io_kiocb *first_req = container_of(head, struct io_kiocb, io_task_work.node); /* @@ -1328,8 +1328,8 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) nr_tw = INT_MAX; req->nr_tw = nr_tw; - req->io_task_work.node.next = first; - } while (!try_cmpxchg(&ctx->work_llist.first, &first, + req->io_task_work.node.next = head; + } while (!try_cmpxchg(&ctx->work_llist.first, &head, &req->io_task_work.node)); /* @@ -1340,7 +1340,7 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) * is similar to the wait/wawke task state sync. */ - if (!first) { + if (!head) { if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); if (ctx->has_evfd) From patchwork Wed Jan 17 00:57:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13521336 Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4CE0B15BF for ; Wed, 17 Jan 2024 00:59:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705453172; cv=none; b=iCyEmElVsOUjmmRphHQnCQpAO66I7FyxFt8HrrIctEKmrjQgSen6IDjYbdOEfTNzeDLyp9GO9CWzttosUKY1IYQl4fbNC7FTK5V0thY8FqyTXBpTQ/HlkSPsSx9Cd1e2G1nk1t0gzFxp+5xm8NtTOyZgcNH2qLWyMp0CDcCbm6w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705453172; c=relaxed/simple; bh=LOg4EdZ5UDksxVyNDbpKjQDGPdohXkB0ZpRjq/QqbAY=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:Received:From: To:Cc:Subject:Date:Message-ID:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; b=HU59aDN+Yj/cwxBU3mSKDSDybSZvVUaEKnAY3bpZfVyXkwN6hfm3KmSZokU/F2ncb2msb84uhULWKPfE6JeTdmYaDg++Bb2qxmr/382V7pWCzkVC8OBj04/xGYR+ovo3PKNI0qv+w5VCmz9jzoYnOPJbDTuUIoG+fllOFGSGiRQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WNNmVmyM; arc=none smtp.client-ip=209.85.167.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WNNmVmyM" Received: by mail-lf1-f42.google.com with SMTP id 2adb3069b0e04-50e7ddd999bso11623229e87.1 for ; Tue, 16 Jan 2024 16:59:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705453168; x=1706057968; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XF1DWHT1/nwHkfy7jjWqjsO6epemJ97JrIogpTWqeoE=; b=WNNmVmyM8G3mz72DWDl+Bt+JCjjY9h6pjC11YdQDXIj+attZqnNEquqSAcvc8a02f4 APvIjopoXQj0aHDeoB8wUpg0hcujwagVHFwy9RZZeOlMPFpxoZUFcbz1LnNKVFYG9Lj0 GS58zSF7/aSfUQIpSF2yJHiUUm8ecvWBUHr6bZvbB4RZvY8ZALeEs25bV9AWfarrvkVC 3YBO0hmfjmqxjbCIkvq3fMfQu2uNGOeSs3BmLdwdjm+SDChgVFbB55op+giQ/WW36Woc Cu02xo3EyFdrMSAuqnzK6JikEFvamPNa8RsG8Z9C7mAYJ1n8KtKgmpP+b/LeoI6AZ2jp 4d9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705453168; x=1706057968; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XF1DWHT1/nwHkfy7jjWqjsO6epemJ97JrIogpTWqeoE=; b=fYBjTRnQhLagLa1vanfzGsNxJ9oEQnEDL63CC4lj0t01On6Y9nQ6mKOa/ljd9iVx87 UbMyyemvzup8GRoqTGqqakGsh2TEHzer3IfskOzuaHrZo1h1xW5NtCL4+byGwmN1HuPy q6HtnSeWlLmHJbPbD5h0aaHeYXI0GC+TMnYSelFzO+ofxNzhl3tWhnlotWhEFCjjn/yZ +hwOsXjo5Vp/0X9529GdNAF5vzEGTHzkZQAG4DozkHoJ4qlJX1mZsaO8+8EL1YelHDLN +IUK68ef0S1dXZJuHWnxnFYv6XplVMfAbAophOe5fpxL/kyP3D5geyYI965DRuEGNdvl sldg== X-Gm-Message-State: AOJu0YzAmfY+ICyr5oIau5ayVY9EDqtZT2BZkeOx7iXAU6UfSUYq7n+H RQgePSZlfkug4S9tfBtJlHdoRsshO6Y= X-Google-Smtp-Source: AGHT+IGltLT8JsKtmUfkkFYai264UObsPgXjxlr8aWT9L1e2FI7czR/RxPHdJ5yshgY2m13A8J63KA== X-Received: by 2002:a05:6512:74:b0:50e:79a5:545a with SMTP id i20-20020a056512007400b0050e79a5545amr3467146lfo.23.1705453168096; Tue, 16 Jan 2024 16:59:28 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.96]) by smtp.gmail.com with ESMTPSA id t15-20020a17090605cf00b00a28aa4871c7sm7038982ejt.205.2024.01.16.16.59.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jan 2024 16:59:27 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 4/4] io_uring: combine cq_wait_nr checks Date: Wed, 17 Jan 2024 00:57:29 +0000 Message-ID: <38def30282654d980673976cd42fde9bab19b297.1705438669.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Instead of explicitly checking ->cq_wait_nr for whether there are waiting, which is currently represented by 0, we can store there a large value and the nr_tw will automatically filter out those cases. Add a named constant for that and for the wake up bias value. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3508198d17ba..b5fa3c7df1cf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -137,6 +137,14 @@ struct io_defer_entry { #define IO_DISARM_MASK (REQ_F_ARM_LTIMEOUT | REQ_F_LINK_TIMEOUT | REQ_F_FAIL) #define IO_REQ_LINK_FLAGS (REQ_F_LINK | REQ_F_HARDLINK) +/* + * No waiters. It's larger than any valid value of the tw counter + * so that tests against ->cq_wait_nr would fail and skip wake_up(). + */ +#define IO_CQ_WAKE_INIT (-1U) +/* Forced wake up if there is a waiter regardless of ->cq_wait_nr */ +#define IO_CQ_WAKE_FORCE (IO_CQ_WAKE_INIT >> 1) + static bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, struct task_struct *task, bool cancel_all); @@ -303,6 +311,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) goto err; ctx->flags = p->flags; + atomic_set(&ctx->cq_wait_nr, IO_CQ_WAKE_INIT); init_waitqueue_head(&ctx->sqo_sq_wait); INIT_LIST_HEAD(&ctx->sqd_list); INIT_LIST_HEAD(&ctx->cq_overflow_list); @@ -1306,6 +1315,13 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) unsigned nr_wait, nr_tw, nr_tw_prev; struct llist_node *head; + /* See comment above IO_CQ_WAKE_INIT */ + BUILD_BUG_ON(IO_CQ_WAKE_FORCE <= IORING_MAX_CQ_ENTRIES); + + /* + * We don't know how many reuqests is there in the link and whether + * they can even be queued lazily, fall back to non-lazy. + */ if (req->flags & (REQ_F_LINK | REQ_F_HARDLINK)) flags &= ~IOU_F_TWQ_LAZY_WAKE; @@ -1322,10 +1338,14 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) */ nr_tw_prev = READ_ONCE(first_req->nr_tw); } + + /* + * Theoretically, it can overflow, but that's fine as one of + * previous adds should've tried to wake the task. + */ nr_tw = nr_tw_prev + 1; - /* Large enough to fail the nr_wait comparison below */ if (!(flags & IOU_F_TWQ_LAZY_WAKE)) - nr_tw = INT_MAX; + nr_tw = IO_CQ_WAKE_FORCE; req->nr_tw = nr_tw; req->io_task_work.node.next = head; @@ -1348,11 +1368,11 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) } nr_wait = atomic_read(&ctx->cq_wait_nr); - /* no one is waiting */ - if (!nr_wait) + /* not enough or no one is waiting */ + if (nr_tw < nr_wait) return; - /* either not enough or the previous add has already woken it up */ - if (nr_wait > nr_tw || nr_tw_prev >= nr_wait) + /* the previous add has already woken it up */ + if (nr_tw_prev >= nr_wait) return; wake_up_state(ctx->submitter_task, TASK_INTERRUPTIBLE); } @@ -2620,7 +2640,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, ret = io_cqring_wait_schedule(ctx, &iowq); __set_current_state(TASK_RUNNING); - atomic_set(&ctx->cq_wait_nr, 0); + atomic_set(&ctx->cq_wait_nr, IO_CQ_WAKE_INIT); /* * Run task_work after scheduling and before io_should_wake().