From patchwork Fri Jul 7 16:20:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andres Freund X-Patchwork-Id: 13305097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F42BEB64D9 for ; Fri, 7 Jul 2023 16:20:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232327AbjGGQUO (ORCPT ); Fri, 7 Jul 2023 12:20:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229642AbjGGQUN (ORCPT ); Fri, 7 Jul 2023 12:20:13 -0400 Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D21B1BEE; Fri, 7 Jul 2023 09:20:11 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id B4A845C0151; Fri, 7 Jul 2023 12:20:10 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Fri, 07 Jul 2023 12:20:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=anarazel.de; h= cc:cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:message-id:mime-version:reply-to:sender:subject :subject:to:to; s=fm2; t=1688746810; x=1688833210; bh=RM0mgP8xGV Q+Nh9DbRaHnprRjsG173RHiqAP1dmkn7c=; b=fzU62APvTW8+dgJYTmIYymrU/c LY+HivEwSTRkIC1LW8y8xnrkCKUtGQVsiZ0UUPwcQrqoJcpYgoBBoNziWePuvUPD 1qlMr+gHSSz921QIi+sjfN5F4kT6MT5YhfxP/uyjcPoutAHfYoGdyBtO9iOXz3bO fZ6Y+ihOnPZm0qiCpgYm/eAGKzCtYTQTgx0kyOuD+QJbpm1IyXAWpKxIsYFLXOdY l1+NsKwHMBNv3bWlLjpPsk/U3GWcWnhZFD4gzqXuhxMUjWh9AwdmkIaLBdRpRfgC mTGGCYVTRGSVyV+N/4HhqNijVR++uWeeFInYkQ4ihKox1xqWEpkLruCEum6Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:message-id:mime-version:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; t=1688746810; x=1688833210; bh=RM0mgP8xGVQ+N h9DbRaHnprRjsG173RHiqAP1dmkn7c=; b=ky5qLHzlOrBhd0QJ3x3ae+U2fCDEZ aWHzMKZYV+I39bYLCuZxu7dSH4JbWROejltB48WB0GmaVu+d/oRJHuMSp3/e8szu CJ+lqMxh8JpG9r7xTh/1+nH4tlwOC9Leq/x0gwWWR+hzFM8BxqQs6LnM7G4Adt1W gEbm87TREA3IjLtHjJ23hLoznXDkcB5tYP+8Jw55Jc5s8W2utj8KD3xq8z5zz5QV ftDU+D/PtMUEBMThl5xjE3H38CpUTXxHxabS60k5isXtTX0w/YcNQxTGCcskQtWf V1zJU/sxyzCZ+f7irAB7FTywjdiRvBX/7qi4e0/NdsdjL0SRoLT081Fng== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrvddugdellecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffoggfgsedtkeertdertddtnecuhfhrohhmpeetnhgurhgvshcu hfhrvghunhguuceorghnughrvghssegrnhgrrhgriigvlhdruggvqeenucggtffrrghtth gvrhhnpeekieeigeekleeggfeugfehudeigeejfeehvefhleetgeeujeffvdegueefgedt veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegrnh gurhgvshesrghnrghrrgiivghlrdguvg X-ME-Proxy: Feedback-ID: id4a34324:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 7 Jul 2023 12:20:10 -0400 (EDT) From: Andres Freund To: Jens Axboe , io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Pavel Begunkov Subject: [PATCH v1] io_uring: Use io_schedule* in cqring wait Date: Fri, 7 Jul 2023 09:20:07 -0700 Message-Id: <20230707162007.194068-1-andres@anarazel.de> X-Mailer: git-send-email 2.38.0 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org I observed poor performance of io_uring compared to synchronous IO. That turns out to be caused by deeper CPU idle states entered with io_uring, due to io_uring using plain schedule(), whereas synchronous IO uses io_schedule(). The losses due to this are substantial. On my cascade lake workstation, t/io_uring from the fio repository e.g. yields regressions between 20% and 40% with the following command: ./t/io_uring -r 5 -X0 -d 1 -s 1 -c 1 -p 0 -S$use_sync -R 0 /mnt/t2/fio/write.0.0 This is repeatable with different filesystems, using raw block devices and using different block devices. Use io_schedule_prepare() / io_schedule_finish() in io_cqring_wait_schedule() to address the difference. After that using io_uring is on par or surpassing synchronous IO (using registered files etc makes it reliably win, but arguably is a less fair comparison). There are other calls to schedule() in io_uring/, but none immediately jump out to be similarly situated, so I did not touch them. Similarly, it's possible that mutex_lock_io() should be used, but it's not clear if there are cases where that matters. Cc: Jens Axboe Cc: Pavel Begunkov Cc: io-uring@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Andres Freund --- io_uring/io_uring.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3bca7a79efda..4661a39de716 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2575,6 +2575,9 @@ int io_run_task_work_sig(struct io_ring_ctx *ctx) static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) { + int ret; + int token; + if (unlikely(READ_ONCE(ctx->check_cq))) return 1; if (unlikely(!llist_empty(&ctx->work_llist))) @@ -2585,11 +2588,20 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, return -EINTR; if (unlikely(io_should_wake(iowq))) return 0; + + /* + * Use io_schedule_prepare/finish, so cpufreq can take into account + * that the task is waiting for IO - turns out to be important for low + * QD IO. + */ + token = io_schedule_prepare(); + ret = 0; if (iowq->timeout == KTIME_MAX) schedule(); else if (!schedule_hrtimeout(&iowq->timeout, HRTIMER_MODE_ABS)) - return -ETIME; - return 0; + ret = -ETIME; + io_schedule_finish(token); + return ret; } /*