From patchwork Mon Jan 9 14:46:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13093684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10CDBC5479D for ; Mon, 9 Jan 2023 14:51:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234869AbjAIOvX (ORCPT ); Mon, 9 Jan 2023 09:51:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231288AbjAIOu7 (ORCPT ); Mon, 9 Jan 2023 09:50:59 -0500 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5441A59FB5 for ; Mon, 9 Jan 2023 06:47:27 -0800 (PST) Received: by mail-ed1-x52a.google.com with SMTP id v10so11611383edi.8 for ; Mon, 09 Jan 2023 06:47:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sPPCDXuaEQHmOJtwcbG7jAg/56GzHF5CmEGxm6YsYGM=; b=j0N0OuSf1ka7JQt9QlgWNvb1hydBrbC3dD4PsL3Gnf1VO23NF9a2PaWU3IxSGV+f7t nokZB2tF9vqLMRi4RgX85Mo4PVQdUaGFuCMxcnqOUAYYM3DhGelvurruzDe+CO6tCbTr dVevPYpYGi72bKw5sbdZg407/ChhGxbiaEC3r7OPT1MkYA4H/ENlsTGlBAOf3+SEOcbb ZMOALfuyEQ2C5VOiI2KKn5x8W6UG3XlgJCkobY768te8IzrM4MyA2WIZVsz67Anmva+e BpKzySq0TvNFGv8QJN5xHIU6MoK8rQ8Dg8AEnJe4H0TFV7I4H57oiqDQcMUs8/dxLWP7 jQ5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sPPCDXuaEQHmOJtwcbG7jAg/56GzHF5CmEGxm6YsYGM=; b=oZ/kpXPO9dhDmMctuaJiSXGIzd1dxZjYQHu0FYcXMy5yhDlGVWYDsj2GH3wrkH7NAb /JXxPem+q/K6L18etv8xb4OVULPYSWMIrZPwqz+W19O4wpK6o09KK3zOsylGCPeftQnK if4A7/yaToWIfcTHmoBGAduHcC8pQzQ6hbnQbk5hBNUWs+d3MfBrWAKHvn4GSGR/3m01 4QEEXn+XcdH9zUkSXcQUw8QkzBRf1f7+6aRZW9mUGciTUbduC3C9/+Nug0TsvkcGVOrq T7bB05vI/2p35txm6+/tUiHjEFvHlya6QwbbXuN3SINlwbeHI45NXHaUzqJ5U1W5FA/5 xb9w== X-Gm-Message-State: AFqh2kr7tqgyTeRjIXfwm6P43ZsAw6Ck3S+ZsB3uEANKQgemey0HX2ay lbjotibK+1riXNi+oQxME0b1ulkCL9w= X-Google-Smtp-Source: AMrXdXuWXC0FrYVMsEcpvS3ydVPdFihUCgPOP/ZxH3n94ZIr8LmWbNtJEhd5Ml4k1+xafEsmQ6DD9g== X-Received: by 2002:a05:6402:e83:b0:45c:a5f2:ffea with SMTP id h3-20020a0564020e8300b0045ca5f2ffeamr77914448eda.7.1673275645536; Mon, 09 Jan 2023 06:47:25 -0800 (PST) Received: from 127.0.0.1localhost (188.29.102.7.threembb.co.uk. [188.29.102.7]) by smtp.gmail.com with ESMTPSA id x11-20020a170906b08b00b0084c62b7b7d8sm3839578ejy.187.2023.01.09.06.47.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 06:47:25 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v3 01/11] io_uring: move submitter_task out of cold cacheline Date: Mon, 9 Jan 2023 14:46:03 +0000 Message-Id: <415ca91dc5ad1dec612b892e489cda98e1069542.1673274244.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org ->submitter_task is used somewhat more frequent now than before, i.e. for local tw enqueue and run, let's move it from the end of ctx, which is full of cold data, to the first cacheline with mostly constants. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 128a67a40065..8dfb6c4a35d9 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -195,11 +195,7 @@ struct io_alloc_cache { struct io_ring_ctx { /* const or read-mostly hot data */ struct { - struct percpu_ref refs; - - struct io_rings *rings; unsigned int flags; - enum task_work_notify_mode notify_method; unsigned int compat: 1; unsigned int drain_next: 1; unsigned int restricted: 1; @@ -210,6 +206,11 @@ struct io_ring_ctx { unsigned int syscall_iopoll: 1; /* all CQEs should be posted only by the submitter task */ unsigned int task_complete: 1; + + enum task_work_notify_mode notify_method; + struct io_rings *rings; + struct task_struct *submitter_task; + struct percpu_ref refs; } ____cacheline_aligned_in_smp; /* submission data */ @@ -320,7 +321,6 @@ struct io_ring_ctx { /* Keep this last, we don't need it for the fast path */ struct io_restriction restrictions; - struct task_struct *submitter_task; /* slow path rsrc auxilary data, used by update/register */ struct io_rsrc_node *rsrc_backup_node; From patchwork Mon Jan 9 14:46:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13093685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2268C54EBE for ; Mon, 9 Jan 2023 14:51:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235094AbjAIOvY (ORCPT ); Mon, 9 Jan 2023 09:51:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235112AbjAIOvB (ORCPT ); Mon, 9 Jan 2023 09:51:01 -0500 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E39223E0F3 for ; Mon, 9 Jan 2023 06:47:27 -0800 (PST) Received: by mail-ej1-x62a.google.com with SMTP id u9so20765610ejo.0 for ; Mon, 09 Jan 2023 06:47:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g0YL0CbM08Co0Kl3ZAc24/DI1DzRJeHVzkmvnIupDg8=; b=Zu0i+qW4S7SD8ORjsK69ZLS6AM/JikIphqfsk0/IR9K1XNH07Ru23Eu/1D7c56P3Dw +J85y5h7+VbiIp4UanPfuzuXACBV0HM+hQrzZ+NhlZcscTg64Ae/Z6wziNjIdChRhM8P FpXnvunRlYxuHUT6UIyS1uOgxWsNRFLpwFlLl54aTITyJTGSEdy1w9czRhe4QraNyLeL z7Xy+2Ndl+Dj3ipktxxEUoJgWWfjE3LurLv/3BhlweaZJOB785qF6vLY8DaddTtBu6Yz cPsUNnSd0ThIqlSU8eMbSvJmPauXQSDggBWbG3n6Kczv62Ql533cW8DkXgUryAjZ/CA4 Jksg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g0YL0CbM08Co0Kl3ZAc24/DI1DzRJeHVzkmvnIupDg8=; b=W79VFsVqQe+iMtoqK5nDSgEvydNoMP3f1YfY3gnBvUN+QhBcj4lG32Zqno5MzgJY+s NUat6JtBA28dd2KAj0wwSRRWvRhR9EhRi018OaLHRztwmiPdaEIWeIN2bZ+PayttbnlA iXkyuZSP9Mwmu+WnGauRDPrkJWxOu1Howv8cvcLExFfBEYZ78kBM6vyfwWMamVB3+HVz dF7U+SC6SrWD2/IKRZQ60hBKgGCcHuHPvZhhAAvC52comphrz0mJqTUiotiQReYifzCz eqlao6biSSuQH8gHXu+ftJQCqBEACDEcPVBZW4LdetsbAiPdYTk4X1Luh5HrBt2c+FWJ CVaQ== X-Gm-Message-State: AFqh2kpz/jrL+bJMT1tFjJgWkH6rj2Dx1mWLtydQSI4WZbW9pSy8+kaP BEv/IxUNAkXRokAurctKDop8TOwEMaE= X-Google-Smtp-Source: AMrXdXsCYIf3HgiI2be2VbABBDbEsaNLdHX8u4VPdAlejXJZmO6/Oiro8tTQgmma6O9Ae0Lox7qhNw== X-Received: by 2002:a17:907:b026:b0:7c0:e7a8:bc41 with SMTP id fu38-20020a170907b02600b007c0e7a8bc41mr49948833ejc.74.1673275646313; Mon, 09 Jan 2023 06:47:26 -0800 (PST) Received: from 127.0.0.1localhost (188.29.102.7.threembb.co.uk. [188.29.102.7]) by smtp.gmail.com with ESMTPSA id x11-20020a170906b08b00b0084c62b7b7d8sm3839578ejy.187.2023.01.09.06.47.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 06:47:26 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v3 02/11] io_uring: refactor io_wake_function Date: Mon, 9 Jan 2023 14:46:04 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Remove a local variable ctx in io_wake_function(), we don't need it if io_should_wake() triggers it to wake up. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 89dd2827a977..157e6ef6da7c 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2441,15 +2441,13 @@ static inline bool io_should_wake(struct io_wait_queue *iowq) static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode, int wake_flags, void *key) { - struct io_wait_queue *iowq = container_of(curr, struct io_wait_queue, - wq); - struct io_ring_ctx *ctx = iowq->ctx; + struct io_wait_queue *iowq = container_of(curr, struct io_wait_queue, wq); /* * Cannot safely flush overflowed CQEs from here, ensure we wake up * the task, and the next invocation will do it. */ - if (io_should_wake(iowq) || io_has_work(ctx)) + if (io_should_wake(iowq) || io_has_work(iowq->ctx)) return autoremove_wake_function(curr, mode, wake_flags, key); return -1; } From patchwork Mon Jan 9 14:46:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13093686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D27F7C5479D for ; Mon, 9 Jan 2023 14:51:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233378AbjAIOvZ (ORCPT ); Mon, 9 Jan 2023 09:51:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235133AbjAIOvC (ORCPT ); Mon, 9 Jan 2023 09:51:02 -0500 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9EA25C1E1 for ; Mon, 9 Jan 2023 06:47:28 -0800 (PST) Received: by mail-ej1-x629.google.com with SMTP id gh17so20715133ejb.6 for ; Mon, 09 Jan 2023 06:47:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GXZ0R/vngPIHzg49UY7ejzbbt7EIlOcMiLJQ3nWY+40=; b=VjFnHpJtRI4Jxz1fORNuvOs0FeJs0FQ6dsB7TTYfLTn9xZ7QE8o6E36kzx8jqCFmim neWPhd5WcYLvLqpBKpx8MHBr/f/MDIytjqFtuFJdaCzYKT1Ox5Rn4LUJO7r39BawvN7+ 1cuKCVQZwitmTkEXTRbQ0+i/zv587Q03oxnJoCTjPexmGGDai4V7sQZDR5b62cnQwBw6 8fmCotkAnHm0/I2lM13/0BUz/9tC9scrv/QxDD8TnpWJoBx7p3HOiQYrXj5LFrqsbLdN 9H+edrvN/7nx08OoXDpL70OjBm5e1Gg2zSBSTAaFqjxXqasH7z9sDhP/jH2FtfQy0B4K Tjew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GXZ0R/vngPIHzg49UY7ejzbbt7EIlOcMiLJQ3nWY+40=; b=FwtVJJAcjmoQ31kBu53/mD5i156GXR1z7wXWANH2r/l/4Md5j0i1wbZqL61nEU54fM DR+PcA+a40Fj4ASS60UevpurQMUpM5z3JvZ50vjfCiRnMNlrZwanf4gAIHgsHktmIP4A 4aAbmSTPjNLWAwDJFbcGVK06bg9WbTgOo0f25iZjgFpKWdWnQr98dLTReBBMghB9LQ+B RljvUp/Vd9G27vztTfbohYsF1HPNIW1gNGSy8Z61KQXcY8nl3/r0LHhzPMeu/jevyvmq PIHeKV5b0+7rsuN+TTTelIgEusOOEA2xZZiJtl30ZlrDk6KOTewHLwbi3cicJAICz/2i reVQ== X-Gm-Message-State: AFqh2kpeRQFmfwUeB0Ldxifi7epmvrL0DKrpbVMcIUSYZGmPPKZbOfFC D9wpzALSgi0Tt+GghjCXDpuCIY7dIzc= X-Google-Smtp-Source: AMrXdXsvprBQVk2T97NG12cff8/Pa36XKnxVNIMH4YpYguBBSL2eN/ZA2GZUZCO9AhoDFyPG2he4pw== X-Received: by 2002:a17:907:9a85:b0:81b:fc79:be51 with SMTP id km5-20020a1709079a8500b0081bfc79be51mr58742182ejc.50.1673275647025; Mon, 09 Jan 2023 06:47:27 -0800 (PST) Received: from 127.0.0.1localhost (188.29.102.7.threembb.co.uk. [188.29.102.7]) by smtp.gmail.com with ESMTPSA id x11-20020a170906b08b00b0084c62b7b7d8sm3839578ejy.187.2023.01.09.06.47.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 06:47:26 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v3 03/11] io_uring: don't set TASK_RUNNING in local tw runner Date: Mon, 9 Jan 2023 14:46:05 +0000 Message-Id: <9d9422c429ef3f9457b4f4b8288bf4789564f33b.1673274244.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The CQ waiting loop sets TASK_RUNNING before trying to execute task_work, no need to repeat it in io_run_local_work(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 157e6ef6da7c..961ebc031992 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1339,11 +1339,9 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked) int io_run_local_work(struct io_ring_ctx *ctx) { - bool locked; + bool locked = mutex_trylock(&ctx->uring_lock); int ret; - __set_current_state(TASK_RUNNING); - locked = mutex_trylock(&ctx->uring_lock); ret = __io_run_local_work(ctx, &locked); if (locked) mutex_unlock(&ctx->uring_lock); @@ -2455,6 +2453,7 @@ static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode, int io_run_task_work_sig(struct io_ring_ctx *ctx) { if (!llist_empty(&ctx->work_llist)) { + __set_current_state(TASK_RUNNING); if (io_run_local_work(ctx) > 0) return 1; } From patchwork Mon Jan 9 14:46:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13093687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12840C5479D for ; Mon, 9 Jan 2023 14:51:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235330AbjAIOva (ORCPT ); Mon, 9 Jan 2023 09:51:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235096AbjAIOvK (ORCPT ); Mon, 9 Jan 2023 09:51:10 -0500 Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C59385C92A for ; Mon, 9 Jan 2023 06:47:29 -0800 (PST) Received: by mail-ej1-x62b.google.com with SMTP id fc4so20635828ejc.12 for ; Mon, 09 Jan 2023 06:47:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zgLLMema6cbOOELLEJ326faKwOgd8bvFvm55EOI/udo=; b=ftMci7ZJhkRe1Wwsv7v73fFcdU+Be3JQt9rDO07MXjb6YXQdsNz2wi9huFpM+1o8pl eHlUiCkPynXNA9URL4uWoDoEI5vh458DCAKdSJfjUVHi1CJmEyA7V8gAumMGk8pR6V7q H4eAZ0kCZpu03c/E2BKTRcOKuAat/OR07CSzG/5HP9QWcwTZF7lTTL0E5cFhecZEL1RB hmdrrOiNswRDhiYhak+H3L+baP4jEXuHkThtAWyODYevrREv5aRUgvS4mdKtY2Za+1YR kLXQv7KZ+XCPMtwvsENUeA9r/0gO48Aai3241kYefReNfBCSeNs9Y/DqhYiY+IFD8w71 zYdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zgLLMema6cbOOELLEJ326faKwOgd8bvFvm55EOI/udo=; b=wlZ55uKZsj567/xNMst4oMA1RXmUiWTpxroWQLhMdPj4zGBEEujalvZ7T2YXDbZW8k KTFlb/FPj769t1K8MSbVkUhvK0EyeNQzI7Jtdd7OAtwOyKpNwiAoXDA+5lTxnfE4mdba 8GUqAm4tpt510pnYQZA4R+l1UnkfZlbQF2apuXo/QyBhj6a51w/ihhVRFTwIB78y9NIN hnSd0N8eWJrQTbmy8AIagDPXHTL30qI9ek7E5Ql/0cuB+AC8KKAcQiR19kUXL4VFRi5V swm/3rDUaNRqiJHrlX6GvqWexmmBKHAzn0/G519lmG6rwI6grGXkuk74aXE+cZ0Cj8f6 Rc0g== X-Gm-Message-State: AFqh2kpZh33eP+y++yV+cqNsd3VHR0B4UeFa+O6LrvlOLZUe3r5CJ9LV BeJBJOuxfBS+TVWr5YrBtJGd2CLhT/8= X-Google-Smtp-Source: AMrXdXssEcQUiVDYiA4KY7REsHRUN5FRBAdpZ4iFY8lIpmEmsfzVvXODtFe7DRjsGTvB5pA27Vx5Ow== X-Received: by 2002:a17:907:9712:b0:7aa:491c:6cdf with SMTP id jg18-20020a170907971200b007aa491c6cdfmr69222698ejc.18.1673275648219; Mon, 09 Jan 2023 06:47:28 -0800 (PST) Received: from 127.0.0.1localhost (188.29.102.7.threembb.co.uk. [188.29.102.7]) by smtp.gmail.com with ESMTPSA id x11-20020a170906b08b00b0084c62b7b7d8sm3839578ejy.187.2023.01.09.06.47.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 06:47:27 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v3 04/11] io_uring: mark io_run_local_work static Date: Mon, 9 Jan 2023 14:46:06 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_run_local_work is enclosed in io_uring.c, we don't need to export it. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- io_uring/io_uring.h | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 961ebc031992..5b62386413d9 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1337,7 +1337,7 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked) } -int io_run_local_work(struct io_ring_ctx *ctx) +static int io_run_local_work(struct io_ring_ctx *ctx) { bool locked = mutex_trylock(&ctx->uring_lock); int ret; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 011932935c36..78896d1f7916 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -29,7 +29,6 @@ struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx, bool overflow); bool io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); -int io_run_local_work(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); From patchwork Mon Jan 9 14:46:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13093688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EAA7C54EBD for ; Mon, 9 Jan 2023 14:51:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235355AbjAIOvb (ORCPT ); Mon, 9 Jan 2023 09:51:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235326AbjAIOvL (ORCPT ); Mon, 9 Jan 2023 09:51:11 -0500 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 680541EAC1 for ; Mon, 9 Jan 2023 06:47:34 -0800 (PST) Received: by mail-ej1-x631.google.com with SMTP id ss4so13419058ejb.11 for ; Mon, 09 Jan 2023 06:47:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=044SK/jIYqht5aLvZ5M6vMX/a8GVtEexKNEoV10/0T8=; b=qHb284//Era/CPs1rkp6Dmy6XDEqx15zi0Kbil8IyOx6UCh3DGEZTCvvuFsBAJsOXz QrSqMCM5uzpezfQjpdagZQnj+o9yRzTaxjLnpHWY6Tu8tQ94ogChWhgo88pkie6jyCyy iyUBU+rTsUpvZIwI202wINtvRz0wSdrSiRn5z4P+QD9vMMNEbkIwLJIgsBcMFQ+SreVW duxgZ0yz9G9Ae3b/q108ffsaHepQf4WmO41V77qmiIjlaERCpvdmP5XOCyd/tl7FRAzR PGH7YwTzFzjIESLpSC4RRqzOmhCMbdzHFAFA5SGmmnV9fcCIx4/Qm59DkLO3OkHQZjvN n3qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=044SK/jIYqht5aLvZ5M6vMX/a8GVtEexKNEoV10/0T8=; b=vxZvu6duk+6gNxsTGfgpoWqHMnf7Fz/non0TgcbPHf8snotgddEjn5BVZ/7/+KVRxu f17TMTizHSzre58dia83ylhubZyOutoo0Htxt6kKW822firUhTsF/8PyaFAwUpiS9Jxh OOccyrpCR+w8e6gJDZJ8MBiMxoYx840bgKC+I2D0s4QnQn7z7Xu0SsU0WcTJpHPYkyx7 moCqqEUK5ZnGUwrnQaTY7wqvvaXF5BEOcTN98DbhH6S/4eDSE75v1EJiW7UuocffRD4B VowGYXhZSJ+x2GRX+qEkVrthp5GIMBhhXpOpoHKJ/bVBdifSglag2MnWokQzFzhc5kF7 16Dw== X-Gm-Message-State: AFqh2krQVN14VeijhO3NZVh5QzFpuV1pplqMashhvKfUyqmKaxX2fgnG ugLru0bNg2YJG+ynZ0HbOACR4DsIyqM= X-Google-Smtp-Source: AMrXdXtaE59CQwoXHXgd8lIUWw7RnFIsMQYwU7PQXqVJEn9BpfGODmoN1lfljKYZv24ds070k5i3hQ== X-Received: by 2002:a17:907:d50e:b0:81f:fc05:2ba0 with SMTP id wb14-20020a170907d50e00b0081ffc052ba0mr56263572ejc.2.1673275652857; Mon, 09 Jan 2023 06:47:32 -0800 (PST) Received: from 127.0.0.1localhost (188.29.102.7.threembb.co.uk. [188.29.102.7]) by smtp.gmail.com with ESMTPSA id x11-20020a170906b08b00b0084c62b7b7d8sm3839578ejy.187.2023.01.09.06.47.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 06:47:32 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v3 05/11] io_uring: move io_run_local_work_locked Date: Mon, 9 Jan 2023 14:46:07 +0000 Message-Id: <91757bcb33e5774e49fed6f2b6e058630608119b.1673274244.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_run_local_work_locked() is only used in io_uring.c, move it there. With that we can also make __io_run_local_work() static. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 18 +++++++++++++++++- io_uring/io_uring.h | 17 ----------------- 2 files changed, 17 insertions(+), 18 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 5b62386413d9..c251acf0964e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1296,7 +1296,7 @@ static void __cold io_move_task_work_from_local(struct io_ring_ctx *ctx) } } -int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked) +static int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked) { struct llist_node *node; struct llist_node fake; @@ -1337,6 +1337,22 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked) } +static inline int io_run_local_work_locked(struct io_ring_ctx *ctx) +{ + bool locked; + int ret; + + if (llist_empty(&ctx->work_llist)) + return 0; + + locked = true; + ret = __io_run_local_work(ctx, &locked); + /* shouldn't happen! */ + if (WARN_ON_ONCE(!locked)) + mutex_lock(&ctx->uring_lock); + return ret; +} + static int io_run_local_work(struct io_ring_ctx *ctx) { bool locked = mutex_trylock(&ctx->uring_lock); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 78896d1f7916..b5975e353aa1 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -28,7 +28,6 @@ enum { struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx, bool overflow); bool io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); -int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); void io_req_defer_failed(struct io_kiocb *req, s32 res); void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); @@ -283,22 +282,6 @@ static inline bool io_task_work_pending(struct io_ring_ctx *ctx) return task_work_pending(current) || !wq_list_empty(&ctx->work_llist); } -static inline int io_run_local_work_locked(struct io_ring_ctx *ctx) -{ - bool locked; - int ret; - - if (llist_empty(&ctx->work_llist)) - return 0; - - locked = true; - ret = __io_run_local_work(ctx, &locked); - /* shouldn't happen! */ - if (WARN_ON_ONCE(!locked)) - mutex_lock(&ctx->uring_lock); - return ret; -} - static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked) { if (!*locked) { From patchwork Mon Jan 9 14:46:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13093689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE382C54EBE for ; Mon, 9 Jan 2023 14:51:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234085AbjAIOvc (ORCPT ); Mon, 9 Jan 2023 09:51:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41432 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229469AbjAIOvM (ORCPT ); Mon, 9 Jan 2023 09:51:12 -0500 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2CCA03AB20 for ; Mon, 9 Jan 2023 06:47:36 -0800 (PST) Received: by mail-ej1-x629.google.com with SMTP id gh17so20716292ejb.6 for ; Mon, 09 Jan 2023 06:47:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Wr/mGWhWn9t52OwGgCmZLcp0xRNtvHfRRAMQOwaF/Jc=; b=KbpTwash6f6utPt49IN1XXS2mqVY2NR3N2vXGlV/2dkW3n8JaBhT2i8O9gMG8EuyEN +QDcIAJ49Oo7mzhG8Q58rAoaEnTSqpF0BSHJsUvynzMctJUiB9lmWRRiHJOAPgIUzCHo S6EMhqYkmfRsXFc8qZy4LwUkGbhWJ1JK8fFL9h/MMYqKZoN3Q8SEIhyqRhmYHFB1DQ08 dXxwKRPZ53gwBi9hq7cma06y+eekOeI8F4FxJM3lcYfDyFiZ1B6tv6GjazjrGUW5naTV ysl4koCLIMxXmVmjjlSlb4mclrfQ+qZnwwyoR+S1j+/y0D4gy8Iyxtn4YfNGwnLwCsXV uh8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Wr/mGWhWn9t52OwGgCmZLcp0xRNtvHfRRAMQOwaF/Jc=; b=ik5CYRqUb/7nMZlrDzQh78E/I+bwv4V9CxWSbvkxt2nfZvbv8pR8vFdpxfouCMGJzC 5Oqvu43fs/vfiT58pVgoq3l0TCMkIAsTEhRpDEAC4WF8qQmnLbKWqqWR0VReokuos8eF sEvwLijvtrKubHykwt32gpEBo1XZ6lqc4VWIm7fcL6y4FRzD8TO2p8tZxz1PHcWrMBPZ oHuYs3f0xYuEePCzN2/ePsLPZYBAeUmCaCCCNQwVfhrvm3Ohkr8Y7De8duSmtY2WNY/k wcf48TtB+PMZcwrBdz15vkWyvPeqlWtgJYh1qnxWRjUAu2SYbpLO5BDBYin/er/C2z/o 4BIg== X-Gm-Message-State: AFqh2kpX0g6RmC2vHL2PSuYwqHxkC6Pss+tIqMMmS3mj5UhrlFMLDRTP khc+Kb8Nhp19+wEty8iXFinLRYkxLAo= X-Google-Smtp-Source: AMrXdXtK/PZZSNK6YMBOvPi/G2ZD+BYjYfNLHoyALHrw4ewgwuLYDx4Uh82hNaSXpE9mdz1dU8S4MQ== X-Received: by 2002:a17:907:a407:b0:84c:7974:8a73 with SMTP id sg7-20020a170907a40700b0084c79748a73mr46696891ejc.57.1673275655641; Mon, 09 Jan 2023 06:47:35 -0800 (PST) Received: from 127.0.0.1localhost (188.29.102.7.threembb.co.uk. [188.29.102.7]) by smtp.gmail.com with ESMTPSA id x11-20020a170906b08b00b0084c62b7b7d8sm3839578ejy.187.2023.01.09.06.47.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 06:47:35 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v3 06/11] io_uring: separate wq for ring polling Date: Mon, 9 Jan 2023 14:46:08 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Don't use ->cq_wait for ring polling but add a separate wait queue for it. We need it for following patches. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 2 +- io_uring/io_uring.c | 3 ++- io_uring/io_uring.h | 9 +++++++++ 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 8dfb6c4a35d9..0d94ee191c15 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -319,7 +319,7 @@ struct io_ring_ctx { } ____cacheline_aligned_in_smp; /* Keep this last, we don't need it for the fast path */ - + struct wait_queue_head poll_wq; struct io_restriction restrictions; /* slow path rsrc auxilary data, used by update/register */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index c251acf0964e..8d19b0812f30 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -316,6 +316,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) xa_init_flags(&ctx->personalities, XA_FLAGS_ALLOC1); mutex_init(&ctx->uring_lock); init_waitqueue_head(&ctx->cq_wait); + init_waitqueue_head(&ctx->poll_wq); spin_lock_init(&ctx->completion_lock); spin_lock_init(&ctx->timeout_lock); INIT_WQ_LIST(&ctx->iopoll_list); @@ -2788,7 +2789,7 @@ static __poll_t io_uring_poll(struct file *file, poll_table *wait) struct io_ring_ctx *ctx = file->private_data; __poll_t mask = 0; - poll_wait(file, &ctx->cq_wait, wait); + poll_wait(file, &ctx->poll_wq, wait); /* * synchronizes with barrier from wq_has_sleeper call in * io_commit_cqring diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index b5975e353aa1..c75bbb94703c 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -220,9 +220,18 @@ static inline void io_commit_cqring(struct io_ring_ctx *ctx) smp_store_release(&ctx->rings->cq.tail, ctx->cached_cq_tail); } +static inline void io_poll_wq_wake(struct io_ring_ctx *ctx) +{ + if (waitqueue_active(&ctx->poll_wq)) + __wake_up(&ctx->poll_wq, TASK_NORMAL, 0, + poll_to_key(EPOLL_URING_WAKE | EPOLLIN)); +} + /* requires smb_mb() prior, see wq_has_sleeper() */ static inline void __io_cqring_wake(struct io_ring_ctx *ctx) { + io_poll_wq_wake(ctx); + /* * Trigger waitqueue handler on all waiters on our waitqueue. This * won't necessarily wake up all the tasks, io_should_wake() will make From patchwork Mon Jan 9 14:46:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13093690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08444C63797 for ; Mon, 9 Jan 2023 14:51:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234744AbjAIOvd (ORCPT ); Mon, 9 Jan 2023 09:51:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234538AbjAIOvN (ORCPT ); Mon, 9 Jan 2023 09:51:13 -0500 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44A373E0FF for ; Mon, 9 Jan 2023 06:47:38 -0800 (PST) Received: by mail-ej1-x62a.google.com with SMTP id vm8so20704451ejc.2 for ; Mon, 09 Jan 2023 06:47:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Q/BxbVSZtF4cpWDNpJ1f3L845XrrbImRE4XcImaP/qM=; b=BgUZ1XgXhG5L9fbtDuxIKvUGw1ad03CPH0DE0G78N4kNVykqol91rQ3gZDfs6E7AMd 9ahZCXfuB8fgQYQngdqy+wFEmPxuPMGug9KPu1Le4rCCS+Ac/puxkYXTm6LUdoqUZRpa eefDUartKNFqfnrC62TxokQwMf2LB6OlshOWGy9aCEP0FYIVqJMBUdJQRTHSidDB+f4/ 6XES/SiTvDut0f9/YWviLUZKyruOAH+jsjCzYYNPWsEyyrjfY2hazvFkKpgzCqkurOoo phHwIqbRtbJo9iTcVY9h8kNiTbL4/sadmcjLD3uZWVX1x6uVlIgwD2eJFHRaWJak9llr MhvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q/BxbVSZtF4cpWDNpJ1f3L845XrrbImRE4XcImaP/qM=; b=kXIZWq6f5uh5jfmZneGGeZKpyqO9HslqBs9gBmgSDh3QeFZq/DLDGZnvfcK3v9K20C 66zAgYo9zYUYMGmamlcUSy6n/rZaQfDLXUIjN4hcYMtSUZpgo4ZTWpIMEWNBB6VJ53wN 0sTNogpC8gab/96zHvLmut0Ue4IaV0N36JanNaaqRjWTOGXkUrzx1SOnTfJCLhbEIDvE UpgNJCMjPOmjBpvAoeBvCi2dKzgXhnbBFq/Nh8jQGqhDGGo+oX4xo2lUnurOuAhXrBV+ VQlzPA+ojqnLpTPFqkt9DpQLFBZgl87uBTTf14uAuv4xkgcbrfi5Pct2e4iUg/bZEGrR aoiA== X-Gm-Message-State: AFqh2krTo2yUzTfmw97Mb9giAGmiVIVJ9Y3QTCkFtH4zNZF32R/j8Smm JTWxAa2pp8DqaxdTsPSObwmemAUfe4o= X-Google-Smtp-Source: AMrXdXuO5aDjE0NgWUeObdawfsRs85oopZaCYewBOJaYnJ3UoLelTqe9jjpCoQtx1e7LY5E2Fq0tZg== X-Received: by 2002:a17:907:cb84:b0:836:e8a6:f6f5 with SMTP id un4-20020a170907cb8400b00836e8a6f6f5mr57333633ejc.66.1673275656547; Mon, 09 Jan 2023 06:47:36 -0800 (PST) Received: from 127.0.0.1localhost (188.29.102.7.threembb.co.uk. [188.29.102.7]) by smtp.gmail.com with ESMTPSA id x11-20020a170906b08b00b0084c62b7b7d8sm3839578ejy.187.2023.01.09.06.47.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 06:47:36 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v3 07/11] io_uring: add lazy poll_wq activation Date: Mon, 9 Jan 2023 14:46:09 +0000 Message-Id: <060785e8e9137a920b232c0c7f575b131af19cac.1673274244.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Even though io_poll_wq_wake()'s waitqueue_active reuses a barrier we do for another waitqueue, it's not going to be the case in the future and so we want to have a fast path for it when the ring has never been polled. Move poll_wq wake ups into __io_commit_cqring_flush() using a new flag called ->poll_activated. The idea behind the flag is to set it when the ring was polled for the first time. This requires additional sync to not miss events, which is done here by using task_work for ->task_complete rings, and by default enabling the flag for all other types of rings. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 2 ++ io_uring/io_uring.c | 60 +++++++++++++++++++++++++++++++++- io_uring/io_uring.h | 7 ++-- 3 files changed, 64 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 0d94ee191c15..7b5e90520278 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -206,6 +206,7 @@ struct io_ring_ctx { unsigned int syscall_iopoll: 1; /* all CQEs should be posted only by the submitter task */ unsigned int task_complete: 1; + unsigned int poll_activated: 1; enum task_work_notify_mode notify_method; struct io_rings *rings; @@ -357,6 +358,7 @@ struct io_ring_ctx { u32 iowq_limits[2]; bool iowq_limits_set; + struct callback_head poll_wq_task_work; struct list_head defer_list; unsigned sq_thread_idle; /* protected by ->completion_lock */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8d19b0812f30..a6ed022c1356 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -573,6 +573,8 @@ static void io_eventfd_flush_signal(struct io_ring_ctx *ctx) void __io_commit_cqring_flush(struct io_ring_ctx *ctx) { + if (ctx->poll_activated) + io_poll_wq_wake(ctx); if (ctx->off_timeout_used) io_flush_timeouts(ctx); if (ctx->drain_active) { @@ -2784,11 +2786,53 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) kfree(ctx); } +static __cold void io_activate_pollwq_cb(struct callback_head *cb) +{ + struct io_ring_ctx *ctx = container_of(cb, struct io_ring_ctx, + poll_wq_task_work); + + mutex_lock(&ctx->uring_lock); + ctx->poll_activated = true; + mutex_unlock(&ctx->uring_lock); + + /* + * Wake ups for some events between start of polling and activation + * might've been lost due to loose synchronisation. + */ + wake_up_all(&ctx->poll_wq); + percpu_ref_put(&ctx->refs); +} + +static __cold void io_activate_pollwq(struct io_ring_ctx *ctx) +{ + spin_lock(&ctx->completion_lock); + /* already activated or in progress */ + if (ctx->poll_activated || ctx->poll_wq_task_work.func) + goto out; + if (WARN_ON_ONCE(!ctx->task_complete)) + goto out; + if (!ctx->submitter_task) + goto out; + /* + * with ->submitter_task only the submitter task completes requests, we + * only need to sync with it, which is done by injecting a tw + */ + init_task_work(&ctx->poll_wq_task_work, io_activate_pollwq_cb); + percpu_ref_get(&ctx->refs); + if (task_work_add(ctx->submitter_task, &ctx->poll_wq_task_work, TWA_SIGNAL)) + percpu_ref_put(&ctx->refs); +out: + spin_unlock(&ctx->completion_lock); +} + static __poll_t io_uring_poll(struct file *file, poll_table *wait) { struct io_ring_ctx *ctx = file->private_data; __poll_t mask = 0; + if (unlikely(!ctx->poll_activated)) + io_activate_pollwq(ctx); + poll_wait(file, &ctx->poll_wq, wait); /* * synchronizes with barrier from wq_has_sleeper call in @@ -3595,6 +3639,13 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, !(ctx->flags & IORING_SETUP_SQPOLL)) ctx->task_complete = true; + /* + * lazy poll_wq activation relies on ->task_complete for synchronisation + * purposes, see io_activate_pollwq() + */ + if (!ctx->task_complete) + ctx->poll_activated = true; + /* * When SETUP_IOPOLL and SETUP_SQPOLL are both enabled, user * space applications don't need to do io completion events @@ -3888,8 +3939,15 @@ static int io_register_enable_rings(struct io_ring_ctx *ctx) if (!(ctx->flags & IORING_SETUP_R_DISABLED)) return -EBADFD; - if (ctx->flags & IORING_SETUP_SINGLE_ISSUER && !ctx->submitter_task) + if (ctx->flags & IORING_SETUP_SINGLE_ISSUER && !ctx->submitter_task) { ctx->submitter_task = get_task_struct(current); + /* + * Lazy activation attempts would fail if it was polled before + * submitter_task is set. + */ + if (wq_has_sleeper(&ctx->poll_wq)) + io_activate_pollwq(ctx); + } if (ctx->restrictions.registered) ctx->restricted = 1; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index c75bbb94703c..5113e0ddb01d 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -222,7 +222,7 @@ static inline void io_commit_cqring(struct io_ring_ctx *ctx) static inline void io_poll_wq_wake(struct io_ring_ctx *ctx) { - if (waitqueue_active(&ctx->poll_wq)) + if (wq_has_sleeper(&ctx->poll_wq)) __wake_up(&ctx->poll_wq, TASK_NORMAL, 0, poll_to_key(EPOLL_URING_WAKE | EPOLLIN)); } @@ -230,8 +230,6 @@ static inline void io_poll_wq_wake(struct io_ring_ctx *ctx) /* requires smb_mb() prior, see wq_has_sleeper() */ static inline void __io_cqring_wake(struct io_ring_ctx *ctx) { - io_poll_wq_wake(ctx); - /* * Trigger waitqueue handler on all waiters on our waitqueue. This * won't necessarily wake up all the tasks, io_should_wake() will make @@ -316,7 +314,8 @@ static inline void io_req_complete_defer(struct io_kiocb *req) static inline void io_commit_cqring_flush(struct io_ring_ctx *ctx) { - if (unlikely(ctx->off_timeout_used || ctx->drain_active || ctx->has_evfd)) + if (unlikely(ctx->off_timeout_used || ctx->drain_active || + ctx->has_evfd || ctx->poll_activated)) __io_commit_cqring_flush(ctx); } From patchwork Mon Jan 9 14:46:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13093691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15382C5479D for ; Mon, 9 Jan 2023 14:51:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234828AbjAIOve (ORCPT ); Mon, 9 Jan 2023 09:51:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229649AbjAIOvO (ORCPT ); Mon, 9 Jan 2023 09:51:14 -0500 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E53E53E841 for ; Mon, 9 Jan 2023 06:47:38 -0800 (PST) Received: by mail-ej1-x62d.google.com with SMTP id az20so1624758ejc.1 for ; Mon, 09 Jan 2023 06:47:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7g4vgDRhIhKHTbt0MChXU5vp/QdCButiCZ9Su7pb0b8=; b=CHbMHC0uXaxbNKNzqTBlD0wDepE3G6WeD0Z15shtB9BZt5mmj2MqtKaQ62HR/ukZ1E uX3ID5v9jrAw5U/LJwOQCb2R0cRJ99VRAQOK9ton95ti8Y88t6O93nCDlfYoNnUlMUvQ ptdpGrd3rZLouSoeDDYVom67/Wx3dvEPlXk+TnOoeG+P1TvOYCLce/maOmAyD5mvqw/x yEPddoM8KC6V3iDFVmNGSv75rRpYetFzMafJuBExUHyAIulIeVpgNQSlQVvSvniwIpQs uRtuqintDss0mH/pzNBlZ+9wOe6eHVuicvOXxVs6IzqXPQmlMiN2Muy8aGi5ghffeIjs UHhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7g4vgDRhIhKHTbt0MChXU5vp/QdCButiCZ9Su7pb0b8=; b=AwkN1UZZlDNGLEVsZ8FR5bjujbw7FB4tsSXyHSpk8jfNNmeiFADOt5LqJlYnQqX5rA 71LC0/JHaTfeyuTgyVwhkz2kpGxPhQWflAnVNspBytCHMfqMS2b3EVSJmQjSD8kW+LVB 4FzWztRzNo0hg4moK7XldNSvApDKV4Fqwlxrlcq9CWer1I9ZQNF0VZRj4D8O6OFh/ZGg z+plFX4IQEhWLXDaN3KpSCX3kYb95ZpoedaAP+AQ5LHiQ5d1Q9pc+uyciLZXhTR/a99C bc1H88Ebh5reVVjOmAAbZwHovvPqa09EBJVNXpr9MmjfNVzMzzR12TIRhuejskejiGy6 m2xQ== X-Gm-Message-State: AFqh2kojfE7AhMH3KQ/ngimM0Xe4/Vw83nnOc9o1m44z38QwO/0JObxN AFEF5qX9MQ8EIEE9KZLo8F90vxRmoVw= X-Google-Smtp-Source: AMrXdXtAE+7WmhaMEg6Wk1Y95LJBvOVkgrCR1+XhVRX0hqfmvrQUNOMwraoOrIezC6oRcaF6z+0ACw== X-Received: by 2002:a17:907:c717:b0:7c1:ad6:638a with SMTP id ty23-20020a170907c71700b007c10ad6638amr60330073ejc.17.1673275657331; Mon, 09 Jan 2023 06:47:37 -0800 (PST) Received: from 127.0.0.1localhost (188.29.102.7.threembb.co.uk. [188.29.102.7]) by smtp.gmail.com with ESMTPSA id x11-20020a170906b08b00b0084c62b7b7d8sm3839578ejy.187.2023.01.09.06.47.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 06:47:37 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v3 08/11] io_uring: wake up optimisations Date: Mon, 9 Jan 2023 14:46:10 +0000 Message-Id: <60ad9768ec74435a0ddaa6eec0ffa7729474f69f.1673274244.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Flush completions is done either from the submit syscall or by the task_work, both are in the context of the submitter task, and when it goes for a single threaded rings like implied by ->task_complete, there won't be any waiters on ->cq_wait but the master task. That means that there can be no tasks sleeping on cq_wait while we run __io_submit_flush_completions() and so waking up can be skipped. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a6ed022c1356..0dabd0f3271f 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -621,6 +621,25 @@ static inline void __io_cq_unlock_post(struct io_ring_ctx *ctx) io_cqring_wake(ctx); } +static inline void __io_cq_unlock_post_flush(struct io_ring_ctx *ctx) + __releases(ctx->completion_lock) +{ + io_commit_cqring(ctx); + __io_cq_unlock(ctx); + io_commit_cqring_flush(ctx); + + /* + * As ->task_complete implies that the ring is single tasked, cq_wait + * may only be waited on by the current in io_cqring_wait(), but since + * it will re-check the wakeup conditions once we return we can safely + * skip waking it up. + */ + if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN)) { + smp_mb(); + __io_cqring_wake(ctx); + } +} + void io_cq_unlock_post(struct io_ring_ctx *ctx) __releases(ctx->completion_lock) { @@ -1480,7 +1499,7 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) } } } - __io_cq_unlock_post(ctx); + __io_cq_unlock_post_flush(ctx); if (!wq_list_empty(&ctx->submit_state.compl_reqs)) { io_free_batch_list(ctx, state->compl_reqs.first); From patchwork Mon Jan 9 14:46:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13093692 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3781C54EBE for ; Mon, 9 Jan 2023 14:51:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230082AbjAIOvf (ORCPT ); Mon, 9 Jan 2023 09:51:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234615AbjAIOvP (ORCPT ); Mon, 9 Jan 2023 09:51:15 -0500 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0B753E84B for ; Mon, 9 Jan 2023 06:47:39 -0800 (PST) Received: by mail-ej1-x62d.google.com with SMTP id tz12so20677652ejc.9 for ; Mon, 09 Jan 2023 06:47:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kOPEVl8PMOwFRDDJfm+ptrMLsP3nfY1Nz8mblxqtLUE=; b=pCfDpHqPJWyRpPxWWsd5dK/6ivdLlC4j0DOVAMKqm/QaayY/IJRY0SvvHlZSsoZJf2 SzILCk+hSNN9wLDRLBgYabn49ah3JXvSp6uYQMLCMt7beBsXWQe7KHLvSCAfXMDkmlXA bf+03ICJqwn8uRGwHUEgDEPOLqXr5wLMiu8L/tTv/jsRkVtdiwr6+qrO/hxa8HzCwzrL yAmmUSikRjOyW4QNtJPvL+j79vhRWT2ZjgCPl4M83XCyP4GlrTFggxAlRuYSFgZ5ke2j 2yga7MHnTMI1MKdxCOIm4yDFmpwKyyaE0I1Fypxk0+g6aPPylpV92cCNik1wnUJg18lI eIWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kOPEVl8PMOwFRDDJfm+ptrMLsP3nfY1Nz8mblxqtLUE=; b=ZoaX7AIbeiI9eKKMznrSEJ6ZzJJfkwf8L9cMmILtHp+T/HtT9VkUT3gq2y+XFNMCyk 1h1HHTWglXHuAF2jsS6xNjTXktOhRQQYi2GbJvCvxxoYsqduUBDvHO3wgE2wamOfAQri CBnzyVNm7uKA2mEh+t53A+bBBHv63CErzPJKAmk0gIoz811B+srLhhkHRNiot0rAnx2C 61hPBmDxs51SeRLiW2vltywah3HYYYcRuO6VpPhnK1naq1aIPoFFDCzo4Vj1ywNxAcvu P94eCE3rCJS35rbKlPwna2TzfTMDttv914jHF3nCd3EWRLMbGzyJWgjbptj8VjmI0PPv YuYQ== X-Gm-Message-State: AFqh2kq0xIP6zcWSGSsturj0ootFM4RCP0kUrNpHPUm+23WWik/cVIx9 1PA1waUeYun5Rle7A+bgzhjDmlwZvnY= X-Google-Smtp-Source: AMrXdXtvUWJr+ZZPGbVIiCMLrfgP1bRsSxdE/B5+US1Re47M37nNtVUaZ8pjf/woKmc7rlxZki6SKg== X-Received: by 2002:a17:907:9d19:b0:7b9:f9d8:9554 with SMTP id kt25-20020a1709079d1900b007b9f9d89554mr3705810ejc.40.1673275658223; Mon, 09 Jan 2023 06:47:38 -0800 (PST) Received: from 127.0.0.1localhost (188.29.102.7.threembb.co.uk. [188.29.102.7]) by smtp.gmail.com with ESMTPSA id x11-20020a170906b08b00b0084c62b7b7d8sm3839578ejy.187.2023.01.09.06.47.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 06:47:37 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v3 09/11] io_uring: waitqueue-less cq waiting Date: Mon, 9 Jan 2023 14:46:11 +0000 Message-Id: <103d174d35d919d4cb0922d8a9c93a8f0c35f74a.1673274244.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org With DEFER_TASKRUN only ctx->submitter_task might be waiting for CQEs, we can use this to optimise io_cqring_wait(). Replace ->cq_wait waitqueue with waking the task directly. It works but misses an important optimisation covered by the following patch, so this patch without follow ups might hurt performance. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0dabd0f3271f..62d879b14873 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1263,7 +1263,7 @@ static void io_req_local_work_add(struct io_kiocb *req) percpu_ref_put(&ctx->refs); return; } - /* need it for the following io_cqring_wake() */ + /* needed for the following wake up */ smp_mb__after_atomic(); if (unlikely(atomic_read(&req->task->io_uring->in_idle))) { @@ -1274,10 +1274,9 @@ static void io_req_local_work_add(struct io_kiocb *req) if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); - if (ctx->has_evfd) io_eventfd_signal(ctx); - __io_cqring_wake(ctx); + wake_up_state(ctx->submitter_task, TASK_INTERRUPTIBLE); percpu_ref_put(&ctx->refs); } @@ -2578,12 +2577,17 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, do { unsigned long check_cq; - prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, - TASK_INTERRUPTIBLE); + if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { + set_current_state(TASK_INTERRUPTIBLE); + } else { + prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, + TASK_INTERRUPTIBLE); + } + ret = io_cqring_wait_schedule(ctx, &iowq); + __set_current_state(TASK_RUNNING); if (ret < 0) break; - __set_current_state(TASK_RUNNING); /* * Run task_work after scheduling and before io_should_wake(). * If we got woken because of task_work being processed, run it @@ -2611,7 +2615,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, cond_resched(); } while (1); - finish_wait(&ctx->cq_wait, &iowq.wq); + if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN)) + finish_wait(&ctx->cq_wait, &iowq.wq); restore_saved_sigmask_unless(ret == -EINTR); return READ_ONCE(rings->cq.head) == READ_ONCE(rings->cq.tail) ? ret : 0; From patchwork Mon Jan 9 14:46:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13093693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93AAFC63797 for ; Mon, 9 Jan 2023 14:51:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234548AbjAIOvg (ORCPT ); Mon, 9 Jan 2023 09:51:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234815AbjAIOvQ (ORCPT ); Mon, 9 Jan 2023 09:51:16 -0500 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDB76271A6 for ; Mon, 9 Jan 2023 06:47:42 -0800 (PST) Received: by mail-ej1-x62d.google.com with SMTP id tz12so20678164ejc.9 for ; Mon, 09 Jan 2023 06:47:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+S/GxxwY7MGRf0XItvsCVaZvfgzyrrliz/LimNKh8PM=; b=brWT38DZI6jp30z52TAmTZbMjUZM9WlUXb0LR18iHP/Xz9nmQggcZInJdZLdfA5pU1 /nIC0zQB9zpVXp++fyJFzdPmiY1IRfrloHy0zbTkjZENpQx4p/Drx4DPvrC83AfWv7nS 4VdxaR9HSf0tI5kbc3JcnbP8vs9FJPSQPgEI3vXhsWVfz0d0pJ0tqWqDLxd0BjBESwyg rWpZQwcIe4dnux62kRKHrQXC/mMsRhGQcUpYRYX1a0nlylQyieRuuwnpIogLjRLzxvZX tv4yINuPCLN69pMwVOZ6WzQmo04+cYgtqJaTfz4/uqrzjJSyGFoZMVBRkaDdL+yEs5sQ qJjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+S/GxxwY7MGRf0XItvsCVaZvfgzyrrliz/LimNKh8PM=; b=fpJIY+mPfjPVbcADPnElPS2JmiBZT3scjiFhM+kkqwvMvUVJR4EzEeceOl6ewCF67+ ACQ/M6J+mRYpcsOfZwgDrOiT9uzrG5bfo12351EQllQAbjrpQBm2rShN127YBpxrl7J9 NC6c+B1UPz5FI3Mwqicj44ldtUjUdplQ1xv26SDCQiE72JQoqD1s0pt5EbW4F7zcR576 au6fzioj3FbN6MXY0FNXdDpu3odWLwBxf3UDg2m8W+mQKtkvoASj0ilL1XK/pybxl4Lx ZLcaof0dLWfgOR8MMosXLiOb5LkppGHVMYHQJG3pqW/bll0eIUFPLQTCPv+Gawrr6Pop 5oPw== X-Gm-Message-State: AFqh2kqGRzuvgU3hUkyaRRg29cgBYrwJnp0bFKKXyAP7KrAyvOF1j4kj to6FB+b0wTUOjGmHUEh8IeysKFSihXg= X-Google-Smtp-Source: AMrXdXv5wcjMRIG307l7B27Mo1dP5AjPR9yVMlnSJv+8cj4dpHReUKYweZtF6OLkA1KDN4Cfj22LBA== X-Received: by 2002:a17:907:d388:b0:846:cdd9:d29 with SMTP id vh8-20020a170907d38800b00846cdd90d29mr51808678ejc.28.1673275662313; Mon, 09 Jan 2023 06:47:42 -0800 (PST) Received: from 127.0.0.1localhost (188.29.102.7.threembb.co.uk. [188.29.102.7]) by smtp.gmail.com with ESMTPSA id x11-20020a170906b08b00b0084c62b7b7d8sm3839578ejy.187.2023.01.09.06.47.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 06:47:42 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v3 10/11] io_uring: add io_req_local_work_add wake fast path Date: Mon, 9 Jan 2023 14:46:12 +0000 Message-Id: <717702d772825a6647e6c315b4690277ba84c3fc.1673274244.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Don't wake the master task after queueing a deferred tw unless it's currently waiting in io_cqring_wait. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 7b5e90520278..cc0cf0705b8f 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -295,6 +295,7 @@ struct io_ring_ctx { spinlock_t completion_lock; bool poll_multi_queue; + bool cq_waiting; /* * ->iopoll_list is protected by the ctx->uring_lock for diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 62d879b14873..9c95ceb1a9f2 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1276,7 +1276,9 @@ static void io_req_local_work_add(struct io_kiocb *req) atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); if (ctx->has_evfd) io_eventfd_signal(ctx); - wake_up_state(ctx->submitter_task, TASK_INTERRUPTIBLE); + + if (READ_ONCE(ctx->cq_waiting)) + wake_up_state(ctx->submitter_task, TASK_INTERRUPTIBLE); percpu_ref_put(&ctx->refs); } @@ -2578,6 +2580,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, unsigned long check_cq; if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { + WRITE_ONCE(ctx->cq_waiting, 1); set_current_state(TASK_INTERRUPTIBLE); } else { prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, @@ -2586,6 +2589,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, ret = io_cqring_wait_schedule(ctx, &iowq); __set_current_state(TASK_RUNNING); + WRITE_ONCE(ctx->cq_waiting, 0); + if (ret < 0) break; /* From patchwork Mon Jan 9 14:46:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13093694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30177C54EBD for ; Mon, 9 Jan 2023 14:51:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234815AbjAIOvh (ORCPT ); Mon, 9 Jan 2023 09:51:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41256 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235277AbjAIOvQ (ORCPT ); Mon, 9 Jan 2023 09:51:16 -0500 Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 237EE3E87D for ; Mon, 9 Jan 2023 06:47:45 -0800 (PST) Received: by mail-ej1-x633.google.com with SMTP id cf18so14363075ejb.5 for ; Mon, 09 Jan 2023 06:47:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ht36KmsVzo7KCc1QVvOj8IAAQ1yudsZVITKpWhMsVEg=; b=I4jZ/1WlbMajhdpno6XRqIe2dtgzKvnP0Ij7gjPyGDZT8NCA1ehrGKw9vF4xcO2JDz 0uVZ0vpVsW3tDrbMQQDeCJ3c0TE++RQnwIEPsC6tsztBePoKIJ8KnhNyBMRRXZfvy4f9 1f2t9fJmp6v1YzxOaM1VUY7wbgdXdltKC+GgZTxjRrHJdTYZTHaqswpnNMvtaI17mE92 KOBj/S4po13IQwhH5eUlfLg7nKTIQOPSLvnp+qHNe6MxAR7faM+eIyJqG8KZnh7LJsVr 5dTVWGLDYmltZq9W0WgfWuoJD6rxcYMXQyF+3jHJsLXNZGba/K0mJGd8XMt5BEs72V0D hU8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ht36KmsVzo7KCc1QVvOj8IAAQ1yudsZVITKpWhMsVEg=; b=hsnFCGpnQXTSqxDG9RXgi0EVLj/XoIBrSR9/Tvd+rd7mkfjGdC4aEw41yF/i3Eqmen txmXBWjeVhXoxVrvvjSXr1XGjdO79nD/p1Hb91QpqpBLcZ+0OowXFKj5HdN0b24BwGlJ jwmnzTMTGwvIpHIeQ8/eloZDbhUiytbgiutbR/NRjkStqmrRJjHklF5ic7qphyjgPTbi sMWkjtyLu2KLTunsuUwkw0OACRje7Wq5FGI8aDbm2D9qBsIFZvtuiz52Ns+dXdVpFgD9 +4DEvOhJPyWmCvVkc0MVrhOOtWZG54pIfy/E1fqJLIiheO9ciF9/ZrZ9FlFefOmGi39i Mq5Q== X-Gm-Message-State: AFqh2kohYvziOL4oJqI3L8oVH3/9IM4KoLH2WPdLMJoyxLvEY8955LDw 1EjeYOLFZU9ZcqKYqtVHKDG5kzcs7+s= X-Google-Smtp-Source: AMrXdXsWgPP1z1sU2mAFUHeLdG4D4iL5OwRMNOAGDwfwoTFizT4z8S+JuxpfCMp0pppmgzgI/vyNGQ== X-Received: by 2002:a17:907:b686:b0:7c1:7c3a:ffba with SMTP id vm6-20020a170907b68600b007c17c3affbamr61202938ejc.35.1673275663588; Mon, 09 Jan 2023 06:47:43 -0800 (PST) Received: from 127.0.0.1localhost (188.29.102.7.threembb.co.uk. [188.29.102.7]) by smtp.gmail.com with ESMTPSA id x11-20020a170906b08b00b0084c62b7b7d8sm3839578ejy.187.2023.01.09.06.47.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 06:47:43 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v3 11/11] io_uring: optimise deferred tw execution Date: Mon, 9 Jan 2023 14:46:13 +0000 Message-Id: <8839534891f0a2f1076e78554a31ea7e099f7de5.1673274244.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We needed fake nodes in __io_run_local_work() and to avoid unecessary wake ups while the task already running task_works, but we don't need them anymore since wake ups are protected by cq_waiting, which is always cleared by the time we're executing deferred task_work items. Note that because of loose sync around cq_waiting clearing io_req_local_work_add() may wake the task more than once, but that's fine and should be rare to not hurt perf. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9c95ceb1a9f2..1dd0fc0412c8 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1322,17 +1322,16 @@ static void __cold io_move_task_work_from_local(struct io_ring_ctx *ctx) static int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked) { struct llist_node *node; - struct llist_node fake; - struct llist_node *current_final = NULL; + unsigned int loops = 0; int ret = 0; - unsigned int loops = 1; if (WARN_ON_ONCE(ctx->submitter_task != current)) return -EEXIST; - - node = io_llist_xchg(&ctx->work_llist, &fake); + if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) + atomic_andnot(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); again: - while (node != current_final) { + node = io_llist_xchg(&ctx->work_llist, NULL); + while (node) { struct llist_node *next = node->next; struct io_kiocb *req = container_of(node, struct io_kiocb, io_task_work.node); @@ -1341,23 +1340,14 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked) ret++; node = next; } + loops++; - if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) - atomic_andnot(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); - - node = io_llist_cmpxchg(&ctx->work_llist, &fake, NULL); - if (node != &fake) { - loops++; - current_final = &fake; - node = io_llist_xchg(&ctx->work_llist, &fake); + if (!llist_empty(&ctx->work_llist)) goto again; - } - if (*locked) io_submit_flush_completions(ctx); trace_io_uring_local_work_run(ctx, ret, loops); return ret; - } static inline int io_run_local_work_locked(struct io_ring_ctx *ctx)