From patchwork Fri Jun 23 11:23:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13290372 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4CE1C001B0 for ; Fri, 23 Jun 2023 11:24:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229501AbjFWLYk (ORCPT ); Fri, 23 Jun 2023 07:24:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229902AbjFWLYj (ORCPT ); Fri, 23 Jun 2023 07:24:39 -0400 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7ED87199D for ; Fri, 23 Jun 2023 04:24:38 -0700 (PDT) Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-98cd280cf94so55249266b.1 for ; Fri, 23 Jun 2023 04:24:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687519477; x=1690111477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+lwiOcyRcQxHexdz5pJ0BkWCi2jupIP80iYWP8FCMjU=; b=NrLyE/dUTEwRthfs8DlmOtOSmviumQfA6Xn8hjSZabDJo6wtL/5FJFk+nbXIWWMHR0 +ja9ah/1kgKxvhZmOgRrd6EagukEigHXYEDISgKKE9nGl60ZMfbDjOgUF0GIq1Z99dOs 4FndB+63BdAP2Z7+Q9l3Oa9D5vHEPaHumr5YhG8/DXQ+haP75Y8a6bc2tePU8QV8+4yi kg1FDgq3s4va+4mU/RX/rgaeGH3yCzjLz9jSb8FqdFkGoloDS+3opdzsvDYYgAjlrIQM l3NHdlZ43FWESnP4gvj8ResuErVNqOtCghgkEuYBIa1OwVKFFtJIKYBaa7SUPYO/MkOF P52g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687519477; x=1690111477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+lwiOcyRcQxHexdz5pJ0BkWCi2jupIP80iYWP8FCMjU=; b=XTFPxDlHxmCdH0KkfrbTFASYak++GP8KtURDbUktFv0hoFwqfcKXpfx97KLOI8Y/0K 9GThDSH6A58PLND9x+tjlBuBJZuZIDUHnChGa7KZfu5eDKxPuVypaAYgR7Vdju5KQICR amOLEBLI6JAxfo3FGIXqW+baqM4F2EfudfDRAdwEIwuQyCcueUVZ+X+9cJE6RrT2FHau z667qoQCYAAIleb122sLGlKdbjWoeeeaIEZitX6rM5dTCoT4ZOhETDm+1V2HLkqkosv4 VkAcR2hT93+7c5wBz+AlW23InZcpBe8c0ZIFGq1chJ6oh3Xfr4+opC8+G7RdjG4xFyEN TPCw== X-Gm-Message-State: AC+VfDwkjUOssE/QwltJUVlGAV7pCyJGvkwosQDjwJWAH/VUslC2m6Dg QAJ9LZFbl9dhyMi7e9o6rOl8Q+Bqh/8= X-Google-Smtp-Source: ACHHUZ5HLrelac8iGSiCk/ZD7Dou63qrthmmXTkIu2ALG94l0puD1LBS+PDnp+liTNYjDexxV1Q28g== X-Received: by 2002:a17:907:a408:b0:98d:3ae:b683 with SMTP id sg8-20020a170907a40800b0098d03aeb683mr4469262ejc.19.1687519476729; Fri, 23 Jun 2023 04:24:36 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:7d95]) by smtp.gmail.com with ESMTPSA id h10-20020a1709067cca00b00969f44bbef3sm5959769ejp.11.2023.06.23.04.24.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 04:24:36 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 01/11] io_uring: open code io_put_req_find_next Date: Fri, 23 Jun 2023 12:23:21 +0100 Message-Id: <38b5c5e48e4adc8e6a0cd16fdd5c1531d7ff81a9.1687518903.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There is only one user of io_put_req_find_next() and it doesn't make much sense to have it. Open code the function. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 25 +++++++------------------ 1 file changed, 7 insertions(+), 18 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ae4cb3c4e730..b488a03ba009 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1586,22 +1586,6 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) } } -/* - * Drop reference to request, return next in chain (if there is one) if this - * was the last reference to this request. - */ -static inline struct io_kiocb *io_put_req_find_next(struct io_kiocb *req) -{ - struct io_kiocb *nxt = NULL; - - if (req_ref_put_and_test(req)) { - if (unlikely(req->flags & IO_REQ_LINK_FLAGS)) - nxt = io_req_find_next(req); - io_free_req(req); - } - return nxt; -} - static unsigned io_cqring_events(struct io_ring_ctx *ctx) { /* See comment at the top of this file */ @@ -1954,9 +1938,14 @@ int io_poll_issue(struct io_kiocb *req, struct io_tw_state *ts) struct io_wq_work *io_wq_free_work(struct io_wq_work *work) { struct io_kiocb *req = container_of(work, struct io_kiocb, work); + struct io_kiocb *nxt = NULL; - req = io_put_req_find_next(req); - return req ? &req->work : NULL; + if (req_ref_put_and_test(req)) { + if (req->flags & IO_REQ_LINK_FLAGS) + nxt = io_req_find_next(req); + io_free_req(req); + } + return nxt ? &nxt->work : NULL; } void io_wq_submit_work(struct io_wq_work *work) From patchwork Fri Jun 23 11:23:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13290374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6222AC001B0 for ; Fri, 23 Jun 2023 11:24:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231329AbjFWLYm (ORCPT ); Fri, 23 Jun 2023 07:24:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231214AbjFWLYk (ORCPT ); Fri, 23 Jun 2023 07:24:40 -0400 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 524432139 for ; Fri, 23 Jun 2023 04:24:39 -0700 (PDT) Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-51bde6a8c20so548112a12.0 for ; Fri, 23 Jun 2023 04:24:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687519477; x=1690111477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MucC/umbtesljG3Fj1Nw4ax1NtgiGvOG45XGet5fe6Q=; b=YdNyYymG4yxn2S+DQeZxhE3IvhNLdmRNsYB3+LDRheESKslSk+COgR2Ywt0dHMpIts qePUxGfpx5khIjZ4ugnVbGUHOPl9fN4VtDENQdEBoCA909tigRrd4n4Mu1sPLgFcQsgS 8zVKBKg61EB9ebo4xyeNlA1h7F9J8qC1mZO7SL0j88xEHRWESws0o3uee/x1OoiptKqf p1bHlVI+M6/DRUjN+u9ywNoyKdD40TIlnSsxUe4fN209xE0I6OyT2grXSLIpJjU/RHjU /GrEhdqGlTMybKWKKvUbgd9HYlKvlevPs9EqixDHM6FfEWNS04pd54xjjVU86ZZqqBBa +QTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687519477; x=1690111477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MucC/umbtesljG3Fj1Nw4ax1NtgiGvOG45XGet5fe6Q=; b=RBUTL4uSRxxUMBhB3A42f/4Mf/Jl0WtuisAOJX6Y85IwQk+SedQj9Ja2zDUbaGherQ 94bC61jNp/F8cPooRnbszxaPwASvwT2ZTyzR5js+v8HtNGYmPZz/0n6/dRV4l98FZbyn JEBYa4bEQZryE6EHc6813scl9RDtuxj7kqE4KhUmxnmgOE7cK/7h0x6ILraP7IntZvKt +Ri5isyeDR6bvgVkYChlW8c6ap9pyjOJsMDDAUyOdfrn5/fjPFrO19BykQIcCwx+PoyM 9pe94RLBWSM6IcvYKpXUIMoJVBlHTETQTc6BO6M6ecxWw1hc7ZLXgljwE/wSdmq2fp5u oXoA== X-Gm-Message-State: AC+VfDyBlQ9BcgJYeN69WwJ3I3Ickeg/+n3DkJQCc5WivDMd4EF9Nzb+ x8ZWnPyPoWE6lXGFw80fCQrvhnC2ZJE= X-Google-Smtp-Source: ACHHUZ6YiqUywxIAwlPPeglLki1SAIlPwE56wWdjuLiAdKz0CeVIIfsHo1Ih0gKPjt5gY75PKVcmrw== X-Received: by 2002:a17:907:7f0f:b0:989:1f66:e452 with SMTP id qf15-20020a1709077f0f00b009891f66e452mr9205346ejc.2.1687519477495; Fri, 23 Jun 2023 04:24:37 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:7d95]) by smtp.gmail.com with ESMTPSA id h10-20020a1709067cca00b00969f44bbef3sm5959769ejp.11.2023.06.23.04.24.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 04:24:36 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 02/11] io_uring: remove io_free_req_tw Date: Fri, 23 Jun 2023 12:23:22 +0100 Message-Id: <434a2be8f33d474ad888ce1c17fe5ea7bbcb2a55.1687518903.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Request completion is a very hot path in general, but there are 3 places that can be doing it: io_free_batch_list(), io_req_complete_post() and io_free_req_tw(). io_free_req_tw() is used rather marginally and we don't care about it. Killing it can help to clean up and optimise the left two, do that by replacing it with io_req_task_complete(). There are two things to consider: 1) io_free_req() is called when all refs are put, so we need to reinit references. The easiest way to do that is to clear REQ_F_REFCOUNT. 2) We also don't need a cqe from it, so silence it with REQ_F_CQE_SKIP. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 23 +++++------------------ 1 file changed, 5 insertions(+), 18 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index b488a03ba009..43805d2621f5 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1121,26 +1121,13 @@ static inline void io_dismantle_req(struct io_kiocb *req) io_put_file(req->file); } -static __cold void io_free_req_tw(struct io_kiocb *req, struct io_tw_state *ts) -{ - struct io_ring_ctx *ctx = req->ctx; - - if (req->rsrc_node) { - io_tw_lock(ctx, ts); - io_put_rsrc_node(ctx, req->rsrc_node); - } - io_dismantle_req(req); - io_put_task_remote(req->task, 1); - - spin_lock(&ctx->completion_lock); - wq_list_add_head(&req->comp_list, &ctx->locked_free_list); - ctx->locked_free_nr++; - spin_unlock(&ctx->completion_lock); -} - __cold void io_free_req(struct io_kiocb *req) { - req->io_task_work.func = io_free_req_tw; + /* refs were already put, restore them for io_req_task_complete() */ + req->flags &= ~REQ_F_REFCOUNT; + /* we only want to free it, don't post CQEs */ + req->flags |= REQ_F_CQE_SKIP; + req->io_task_work.func = io_req_task_complete; io_req_task_work_add(req); } From patchwork Fri Jun 23 11:23:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13290373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6F5EEB64D7 for ; Fri, 23 Jun 2023 11:24:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230444AbjFWLYm (ORCPT ); Fri, 23 Jun 2023 07:24:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231315AbjFWLYk (ORCPT ); Fri, 23 Jun 2023 07:24:40 -0400 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3A701739 for ; Fri, 23 Jun 2023 04:24:39 -0700 (PDT) Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-988b204ce5fso56014366b.3 for ; Fri, 23 Jun 2023 04:24:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687519478; x=1690111478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=afSlWixgBKf99bh3ReZ4Sy8Z5ib88ed8o5fCVVxdkfo=; b=ZRsAP9XdCMyPBM23tofQhp6aF8fpIA9tt/PESgKI5rcCQiDxFtY6nEPFAVK+V0jZRj scWDMz1vmeazmcXEEztfPl3Y5P9jlcgZZA6m6baBjQC1pCn8uq4hQvja/msCnsqa7bi6 zWyFFkqvRKXmeoF/T6VRL/XYGvR6Rytok5Av1BSMapeWOi5t1bUEPBII7bvR9asfnfZm tC47efPyDlDm9yyX6xHZc30RqTO8Lzzv2vfT5xkOes1TfqyQdBMesch4QHOJExYugxiM p8lMPBBFhmjsfLitYiMBBrdF6WXYqmcwtZXRV1aEU4jun+wuF9dw4ACifHrqGTCHNBm+ 8NfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687519478; x=1690111478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=afSlWixgBKf99bh3ReZ4Sy8Z5ib88ed8o5fCVVxdkfo=; b=fizSGAg6D34GNmNI5xI3B+1L4B/DveR4ke1s48jylyC5F7WVCzprCEl7zLgdyYBF5E A08nXAkIhan+sek0dK7rO6f8CeLzWZz0tymkpMdCPih1fBOFE1SeHnLDmp1MduMUxSyh UZI7Y9PgPAOF2ZO60I5YynU7lUJKAFNAhyY1pcl/49EIHYhVporr+OboDclGZp4t1wCG NY2Pl+u69H2BUZJVzOtDZzqkh/o3Cwi7OQlvEemoa+o2YdfHJybkq4SqKcNunm2+RKQH 6FyK6LC+X7fdDCKmBCUZfgJ3OGmWrTWWY3l4EXKFdQa2W9WnDsVA9CUMMtwLDKq2SBq/ /lFQ== X-Gm-Message-State: AC+VfDxyfVvY6AiJ1zJ2J4XK0Ea8YtBDqabXpoehY1xqD5qCgoPCwqpc jjHZWOm+nI/b40R7KNzrO7ItglwieJI= X-Google-Smtp-Source: ACHHUZ6VYUS5/BHqL2cUcDynLzGMfLrXdDEFf2WKHJxJxKfYDuIrU0Xh3Kf+2FOuhKB/iLvVTEC9yA== X-Received: by 2002:a17:907:928d:b0:953:8249:1834 with SMTP id bw13-20020a170907928d00b0095382491834mr18049318ejc.16.1687519477983; Fri, 23 Jun 2023 04:24:37 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:7d95]) by smtp.gmail.com with ESMTPSA id h10-20020a1709067cca00b00969f44bbef3sm5959769ejp.11.2023.06.23.04.24.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 04:24:37 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 03/11] io_uring: inline io_dismantle_req() Date: Fri, 23 Jun 2023 12:23:23 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_dismantle_req() is only used in __io_req_complete_post(), open code it there. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 43805d2621f5..50fe345bdced 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -146,7 +146,6 @@ static bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, struct task_struct *task, bool cancel_all); -static void io_dismantle_req(struct io_kiocb *req); static void io_clean_op(struct io_kiocb *req); static void io_queue_sqe(struct io_kiocb *req); static void io_move_task_work_from_local(struct io_ring_ctx *ctx); @@ -991,7 +990,11 @@ static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) } } io_put_kbuf_comp(req); - io_dismantle_req(req); + if (unlikely(req->flags & IO_REQ_CLEAN_FLAGS)) + io_clean_op(req); + if (!(req->flags & REQ_F_FIXED_FILE)) + io_put_file(req->file); + rsrc_node = req->rsrc_node; /* * Selected buffer deallocation in io_clean_op() assumes that @@ -1111,16 +1114,6 @@ __cold bool __io_alloc_req_refill(struct io_ring_ctx *ctx) return true; } -static inline void io_dismantle_req(struct io_kiocb *req) -{ - unsigned int flags = req->flags; - - if (unlikely(flags & IO_REQ_CLEAN_FLAGS)) - io_clean_op(req); - if (!(flags & REQ_F_FIXED_FILE)) - io_put_file(req->file); -} - __cold void io_free_req(struct io_kiocb *req) { /* refs were already put, restore them for io_req_task_complete() */ From patchwork Fri Jun 23 11:23:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13290375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78429C001B1 for ; Fri, 23 Jun 2023 11:24:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231358AbjFWLYn (ORCPT ); Fri, 23 Jun 2023 07:24:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229902AbjFWLYl (ORCPT ); Fri, 23 Jun 2023 07:24:41 -0400 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 42D38199D for ; Fri, 23 Jun 2023 04:24:40 -0700 (PDT) Received: by mail-ej1-x62a.google.com with SMTP id a640c23a62f3a-98cd280cf94so55251566b.1 for ; Fri, 23 Jun 2023 04:24:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687519478; x=1690111478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+Urqbe8d5meDgYTnTRdNWXyukQbW5jl8lzicwvsbbj8=; b=BlFLKrNfu9YuetzcJ6pYXL0/P5086YczEzPViXFSCda153kbwVni6ryb3dXOuwtCa0 4Ur3ptLboIv8OKxjQOKFyneYCfAgbcG7T5JixKr0lFTdZMJr0KVEsI9yhpnCRimDeF0R HCiIHp1vzrseJKhPhgsGg52HaEHS/7hudlJK1hkJFWURvw5dO6h1Z7vlmPEwXRpgoQHZ SA/14Vvbc1XMkFPBUCmEGOxxPT8YSW3VDcizQVRNDgysuqWKBspK2KSmgBJUGIpN2vpS o4u+Q9vbsIRbPG/KT/P8UTOUTqnVMy0Hcqdau2WPEWy92ha0nBJEbtclPsQFF/zezTLB BCDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687519478; x=1690111478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+Urqbe8d5meDgYTnTRdNWXyukQbW5jl8lzicwvsbbj8=; b=I8r1/rkmu/jEWgIJ6OZnOEW3MsP1xjewR8/WBqKobA7/K4KW8XQGat0QUvXyq3s8BY WveiOD82fgyce50HzAkXNWCFMfbiiKYMZIzbpJSLgynJdG6UbWfM58QtATpJocGz8CQ3 2Q42sdEvP4WU1SO3CkV5IYrINaIvCEWZQ9pz4Y4SIoQyFKUJ1o4UsTNj+Nx7DasltfMJ MXrCR3gbc9ii8RagIMlbOw3oj95cO/aZLmM8xPIkB1vswZJYvtML4fJHp6WOTCrukPHb qclI6Shr5iASF1qoHzQrsVR9ehpM9E8a+H3ptEtz7La3A9xzOJxve2A9ug6EfgOMHWnc nDjQ== X-Gm-Message-State: AC+VfDzlVNFfqWpVjSb3vtPwPIpx4q8FSK6EDAnqc8ZVmPzC12x7Z2H2 F79WGTOw/UBnNCwl+3sWxdFHkWQxPAc= X-Google-Smtp-Source: ACHHUZ4cVgt35XsgJNsGPKbc/lCn7nv+dHmx5nq27y4lmJ7XQrVwQT02pTpjp3bRFIDt/UeILkS10A== X-Received: by 2002:a17:906:64db:b0:98d:5ae2:f1c with SMTP id p27-20020a17090664db00b0098d5ae20f1cmr2174893ejn.34.1687519478377; Fri, 23 Jun 2023 04:24:38 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:7d95]) by smtp.gmail.com with ESMTPSA id h10-20020a1709067cca00b00969f44bbef3sm5959769ejp.11.2023.06.23.04.24.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 04:24:38 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 04/11] io_uring: move io_clean_op() Date: Fri, 23 Jun 2023 12:23:24 +0100 Message-Id: <1b7163b2ba7c3a8322d972c79c1b0a9301b3057e.1687518903.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Move io_clean_op() up in the source file and remove the forward declaration, as the function doesn't have tricky dependencies anymore. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 67 ++++++++++++++++++++++----------------------- 1 file changed, 33 insertions(+), 34 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 50fe345bdced..4d8613996644 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -146,7 +146,6 @@ static bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, struct task_struct *task, bool cancel_all); -static void io_clean_op(struct io_kiocb *req); static void io_queue_sqe(struct io_kiocb *req); static void io_move_task_work_from_local(struct io_ring_ctx *ctx); static void __io_submit_flush_completions(struct io_ring_ctx *ctx); @@ -367,6 +366,39 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq) return false; } +static void io_clean_op(struct io_kiocb *req) +{ + if (req->flags & REQ_F_BUFFER_SELECTED) { + spin_lock(&req->ctx->completion_lock); + io_put_kbuf_comp(req); + spin_unlock(&req->ctx->completion_lock); + } + + if (req->flags & REQ_F_NEED_CLEANUP) { + const struct io_cold_def *def = &io_cold_defs[req->opcode]; + + if (def->cleanup) + def->cleanup(req); + } + if ((req->flags & REQ_F_POLLED) && req->apoll) { + kfree(req->apoll->double_poll); + kfree(req->apoll); + req->apoll = NULL; + } + if (req->flags & REQ_F_INFLIGHT) { + struct io_uring_task *tctx = req->task->io_uring; + + atomic_dec(&tctx->inflight_tracked); + } + if (req->flags & REQ_F_CREDS) + put_cred(req->creds); + if (req->flags & REQ_F_ASYNC_DATA) { + kfree(req->async_data); + req->async_data = NULL; + } + req->flags &= ~IO_REQ_CLEAN_FLAGS; +} + static inline void io_req_track_inflight(struct io_kiocb *req) { if (!(req->flags & REQ_F_INFLIGHT)) { @@ -1823,39 +1855,6 @@ static __cold void io_drain_req(struct io_kiocb *req) spin_unlock(&ctx->completion_lock); } -static void io_clean_op(struct io_kiocb *req) -{ - if (req->flags & REQ_F_BUFFER_SELECTED) { - spin_lock(&req->ctx->completion_lock); - io_put_kbuf_comp(req); - spin_unlock(&req->ctx->completion_lock); - } - - if (req->flags & REQ_F_NEED_CLEANUP) { - const struct io_cold_def *def = &io_cold_defs[req->opcode]; - - if (def->cleanup) - def->cleanup(req); - } - if ((req->flags & REQ_F_POLLED) && req->apoll) { - kfree(req->apoll->double_poll); - kfree(req->apoll); - req->apoll = NULL; - } - if (req->flags & REQ_F_INFLIGHT) { - struct io_uring_task *tctx = req->task->io_uring; - - atomic_dec(&tctx->inflight_tracked); - } - if (req->flags & REQ_F_CREDS) - put_cred(req->creds); - if (req->flags & REQ_F_ASYNC_DATA) { - kfree(req->async_data); - req->async_data = NULL; - } - req->flags &= ~IO_REQ_CLEAN_FLAGS; -} - static bool io_assign_file(struct io_kiocb *req, const struct io_issue_def *def, unsigned int issue_flags) { From patchwork Fri Jun 23 11:23:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13290376 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89040EB64DD for ; Fri, 23 Jun 2023 11:24:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231451AbjFWLYn (ORCPT ); Fri, 23 Jun 2023 07:24:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231316AbjFWLYl (ORCPT ); Fri, 23 Jun 2023 07:24:41 -0400 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E28E213C for ; Fri, 23 Jun 2023 04:24:40 -0700 (PDT) Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-988f066f665so57183766b.2 for ; Fri, 23 Jun 2023 04:24:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687519479; x=1690111479; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gDXyyrGu+GtMDO6VObST1r4sSfZY+sVWPFHmiQYWm98=; b=I7U7AvCRmQxblQCJ4yrf2sUa694qZ6TN5jEmmsimZO9CJH8QEXb05SEhi10pPHH3LX cFe3woL/Rc14uO45Qz/DNTzrpRBo0akuqP4iNhWQsWTpj6Eg3ooyYjGRnMOnxoL6gob2 +t92hDo/q7RBNhUaYVJVCN4KWHZcNqztrQS1VZaS66f6M83sPvfIiznIlixmOR62ExP0 tzykbqkn8KEHk6C2mtAqWb5kvw/MKU1ntOB9xwfln5I8LMmldxDmwUdg+oZ6pFkvlzR1 Bg1tixAH9Z6s8qnnWRhPI73rNvb0x5PkijudbNUoHdhg23KoAGAmMWtmLvsnqwr2gfpM lTmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687519479; x=1690111479; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gDXyyrGu+GtMDO6VObST1r4sSfZY+sVWPFHmiQYWm98=; b=bRV6NEt0L3lNlHuUAmjTxdSqRWi+2djKkClMJeFZjKiEf1KmDsiXTnmJQIbjuSLXWH ZsUpKHLh4Nrx4H388IJw+ZTvK5HAr4Uu9DUVniOYHERXgwJyAmeUBpcnz0j+adIO1EHE 4ZVSnsHpZvuGbSQLoloFwa7IGOahAsM67OCNLVGRnkDii/HnppAShtfGKIiRZcR1tjzp CPs14TsWkbP9boZG4kkHi+uEusthQoAu6Ao3ItXI9BE5tADnb7IVfEJAJU/nvahY9b5m Nxu+ANyYYKYlwuBTnnZ9sJCzl4Xex7iCgPUgz9HdIbHdUNfG6EUOwTTDyxuD0IDrN8E3 CRFw== X-Gm-Message-State: AC+VfDx3tHslE00ZZLru3X+I5tBWbW9yaQf04iPfLhPZErgfVNw/OI5P IXcLWd8t7Obxx++0oigoF6wEdetqFE4= X-Google-Smtp-Source: ACHHUZ4pYldR2iKfOP0VA8zqMUvvrjEJ/zQ1CE25QOjFdMYQVyVusFrQxkZG/mpc0X0BwxLNIg9BaQ== X-Received: by 2002:a17:907:25cd:b0:989:3148:e9a with SMTP id ae13-20020a17090725cd00b0098931480e9amr8478796ejc.41.1687519478816; Fri, 23 Jun 2023 04:24:38 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:7d95]) by smtp.gmail.com with ESMTPSA id h10-20020a1709067cca00b00969f44bbef3sm5959769ejp.11.2023.06.23.04.24.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 04:24:38 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 05/11] io_uring: don't batch task put on reqs free Date: Fri, 23 Jun 2023 12:23:25 +0100 Message-Id: <4a7ef7dce845fe2bd35507bf389d6bd2d5c1edf0.1687518903.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We're trying to batch io_put_task() in io_free_batch_list(), but considering that the hot path is a simple inc, it's most cerainly and probably faster to just do io_put_task() instead of task tracking. We don't care about io_put_task_remote() as it's only for IOPOLL where polling/waiting is done by not the submitter task. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 32 ++++++++++---------------------- 1 file changed, 10 insertions(+), 22 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4d8613996644..3eec5c761d0a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -754,29 +754,29 @@ static void io_cqring_overflow_flush(struct io_ring_ctx *ctx) } /* can be called by any task */ -static void io_put_task_remote(struct task_struct *task, int nr) +static void io_put_task_remote(struct task_struct *task) { struct io_uring_task *tctx = task->io_uring; - percpu_counter_sub(&tctx->inflight, nr); + percpu_counter_sub(&tctx->inflight, 1); if (unlikely(atomic_read(&tctx->in_cancel))) wake_up(&tctx->wait); - put_task_struct_many(task, nr); + put_task_struct(task); } /* used by a task to put its own references */ -static void io_put_task_local(struct task_struct *task, int nr) +static void io_put_task_local(struct task_struct *task) { - task->io_uring->cached_refs += nr; + task->io_uring->cached_refs++; } /* must to be called somewhat shortly after putting a request */ -static inline void io_put_task(struct task_struct *task, int nr) +static inline void io_put_task(struct task_struct *task) { if (likely(task == current)) - io_put_task_local(task, nr); + io_put_task_local(task); else - io_put_task_remote(task, nr); + io_put_task_remote(task); } void io_task_refs_refill(struct io_uring_task *tctx) @@ -1033,7 +1033,7 @@ static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) * we don't hold ->completion_lock. Clean them here to avoid * deadlocks. */ - io_put_task_remote(req->task, 1); + io_put_task_remote(req->task); wq_list_add_head(&req->comp_list, &ctx->locked_free_list); ctx->locked_free_nr++; } @@ -1518,9 +1518,6 @@ void io_queue_next(struct io_kiocb *req) void io_free_batch_list(struct io_ring_ctx *ctx, struct io_wq_work_node *node) __must_hold(&ctx->uring_lock) { - struct task_struct *task = NULL; - int task_refs = 0; - do { struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); @@ -1550,19 +1547,10 @@ void io_free_batch_list(struct io_ring_ctx *ctx, struct io_wq_work_node *node) io_req_put_rsrc_locked(req, ctx); - if (req->task != task) { - if (task) - io_put_task(task, task_refs); - task = req->task; - task_refs = 0; - } - task_refs++; + io_put_task(req->task); node = req->comp_list.next; io_req_add_to_cache(req, ctx); } while (node); - - if (task) - io_put_task(task, task_refs); } static void __io_submit_flush_completions(struct io_ring_ctx *ctx) From patchwork Fri Jun 23 11:23:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13290379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5049C001DC for ; Fri, 23 Jun 2023 11:24:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231475AbjFWLYp (ORCPT ); Fri, 23 Jun 2023 07:24:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58558 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231315AbjFWLYm (ORCPT ); Fri, 23 Jun 2023 07:24:42 -0400 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 311FA2685 for ; Fri, 23 Jun 2023 04:24:41 -0700 (PDT) Received: by mail-ej1-x62a.google.com with SMTP id a640c23a62f3a-987c932883bso62415666b.0 for ; Fri, 23 Jun 2023 04:24:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687519479; x=1690111479; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s4MKuIvOtPa6LDaVdjrreVhyD6dBmluphGJKPl09Ii8=; b=inDJGp5e3DN9rltz8e6/bqRcSEXMESFBCL1Y2ct291MueGyvzDzDEKqbKPYb+bTU9j vfVRfqmbC6+1KOkzIZNfler3AOMIsZBLyYn0fD0L20OKP7jtrgp918E2miLcsJrr+O9E FKyXqi12MM/DZN0mX4ltin4I6po40zZY9wpDETdHPSOg7HA2IagYsl0E5kcWQwOGh/kr dYvdBcWtYEnbZCSyf1ChouVtqI2ZLC3OKJOoujiBfSflEIGrXW7REGocc34aiAXEWaAJ 5F5IhGO666NmIkDaUitW9lVX0gf+zfT+3G7UoTf+Zr1YEKhVYiqlfBe9yQGtxajGIX1s 6Ggw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687519479; x=1690111479; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s4MKuIvOtPa6LDaVdjrreVhyD6dBmluphGJKPl09Ii8=; b=hXva9bGhe+CaZcmz/BkATrO7gJhHs/R/mv3DfjYYeXnJp/R+Umn1WlTUXsvlI8QBO7 uzlctKf8OeT6LByIhbqpevI2FyGHCOiyZWAk2qZkukGiY/rjliUgxe0yhsbtmJ074XQg 24HebbzEXbqwYij+97/fnG1h6vf+0wUXJKuTlPlHW+0diAYF8c7/UN2SnZJjdi94Qguf XQTUjTuj9ak8XtX+jGSD2fh/fPggP+nzWmPeUqHkPYd1Z2ZNs0E9rkMAp3nZ6FW3eOTK lVk250GlOWEMKi0O2yD8MF9H1aX3WSB7DyEc8of8rxx1iLfozWmU7v0nnTV/FUijLXbO U+Cg== X-Gm-Message-State: AC+VfDyUS0kjkNgLPV20IbPusyi4mknuIT8Zwxi34z5dhaBqhcHuigJM MeGQLzdU+qQzoLBvx/A7XPWMx1koQec= X-Google-Smtp-Source: ACHHUZ5tTlVmSECfli21N3+80awF0bnPmAZVLwl0p9w6FmBxoIJR6IG2Ew6noi1eGXh4qU5YRDvg2g== X-Received: by 2002:a17:907:783:b0:982:c8d0:683f with SMTP id xd3-20020a170907078300b00982c8d0683fmr19276297ejb.18.1687519479489; Fri, 23 Jun 2023 04:24:39 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:7d95]) by smtp.gmail.com with ESMTPSA id h10-20020a1709067cca00b00969f44bbef3sm5959769ejp.11.2023.06.23.04.24.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 04:24:39 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 06/11] io_uring: remove IOU_F_TWQ_FORCE_NORMAL Date: Fri, 23 Jun 2023 12:23:26 +0100 Message-Id: <2e55571e8ff2927ae3cc12da606d204e2485525b.1687518903.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Extract a function for non-local task_work_add, and use it directly from io_move_task_work_from_local(). Now we don't use IOU_F_TWQ_FORCE_NORMAL and it can be killed. As a small positive side effect we don't grab task->io_uring in io_req_normal_work_add anymore, which is not needed for io_req_local_work_add(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 25 ++++++++++++++----------- io_uring/io_uring.h | 5 +---- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3eec5c761d0a..776d1aa73d26 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1317,7 +1317,7 @@ static __cold void io_fallback_tw(struct io_uring_task *tctx) } } -static void io_req_local_work_add(struct io_kiocb *req, unsigned flags) +static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) { struct io_ring_ctx *ctx = req->ctx; unsigned nr_wait, nr_tw, nr_tw_prev; @@ -1368,19 +1368,11 @@ static void io_req_local_work_add(struct io_kiocb *req, unsigned flags) wake_up_state(ctx->submitter_task, TASK_INTERRUPTIBLE); } -void __io_req_task_work_add(struct io_kiocb *req, unsigned flags) +static void io_req_normal_work_add(struct io_kiocb *req) { struct io_uring_task *tctx = req->task->io_uring; struct io_ring_ctx *ctx = req->ctx; - if (!(flags & IOU_F_TWQ_FORCE_NORMAL) && - (ctx->flags & IORING_SETUP_DEFER_TASKRUN)) { - rcu_read_lock(); - io_req_local_work_add(req, flags); - rcu_read_unlock(); - return; - } - /* task_work already pending, we're done */ if (!llist_add(&req->io_task_work.node, &tctx->task_list)) return; @@ -1394,6 +1386,17 @@ void __io_req_task_work_add(struct io_kiocb *req, unsigned flags) io_fallback_tw(tctx); } +void __io_req_task_work_add(struct io_kiocb *req, unsigned flags) +{ + if (req->ctx->flags & IORING_SETUP_DEFER_TASKRUN) { + rcu_read_lock(); + io_req_local_work_add(req, flags); + rcu_read_unlock(); + } else { + io_req_normal_work_add(req); + } +} + static void __cold io_move_task_work_from_local(struct io_ring_ctx *ctx) { struct llist_node *node; @@ -1404,7 +1407,7 @@ static void __cold io_move_task_work_from_local(struct io_ring_ctx *ctx) io_task_work.node); node = node->next; - __io_req_task_work_add(req, IOU_F_TWQ_FORCE_NORMAL); + io_req_normal_work_add(req); } } diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 9718897133db..20ba6df49b1f 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -16,9 +16,6 @@ #endif enum { - /* don't use deferred task_work */ - IOU_F_TWQ_FORCE_NORMAL = 1, - /* * A hint to not wake right away but delay until there are enough of * tw's queued to match the number of CQEs the task is waiting for. @@ -26,7 +23,7 @@ enum { * Must not be used wirh requests generating more than one CQE. * It's also ignored unless IORING_SETUP_DEFER_TASKRUN is set. */ - IOU_F_TWQ_LAZY_WAKE = 2, + IOU_F_TWQ_LAZY_WAKE = 1, }; enum { From patchwork Fri Jun 23 11:23:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13290378 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4AC9C001B0 for ; Fri, 23 Jun 2023 11:24:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231363AbjFWLYp (ORCPT ); Fri, 23 Jun 2023 07:24:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231330AbjFWLYm (ORCPT ); Fri, 23 Jun 2023 07:24:42 -0400 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC88E1739 for ; Fri, 23 Jun 2023 04:24:41 -0700 (PDT) Received: by mail-ej1-x630.google.com with SMTP id a640c23a62f3a-988689a5f44so44976966b.1 for ; Fri, 23 Jun 2023 04:24:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687519480; x=1690111480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=U8IzdFjveQ9hwmkRa48Ek1vMmW1XJWaCIZgZ0bHO5ik=; b=mx4U/yiUy6F4aZVnwnZVRTyHCR7UqNc0GwOEmKT/1cXhyZFNraMikAjl8gpLCoO1Ql I/amnD0LzDmQpO9Oh9Dz6Rck0gbgA5yb4RFeyObt8ciFnT/ZNBso6DRJt+sPSsWjSPvj 5PmqT9m5T6OIF38GiRCfsL1Ud4zjlwc0hgybX3i1oKR68Xk75F5KHpFKeGtOOnvVRypm 58qeUDyFfvu3TLheO3bXNoYo0DubqOVfBVy/xLpbBaIb3CPOHPxLCZARIW1Nn7MmrJgj QF2/vb3iqnsbwIdfxzbbq/oQTwAXwVQTQ1hC1589AUWpC36h1Ssmyq5QINNoLemqL3mb Fx4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687519480; x=1690111480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U8IzdFjveQ9hwmkRa48Ek1vMmW1XJWaCIZgZ0bHO5ik=; b=bCZtArToCEX8aQ1czOFE69IuDVDT+FeG3jl8mAaBhfXezE5Sj4VUqB/h95Rq9YRlZ1 sPMCDqXaMCYA6sJ58orzGH1AAHjAvfqFi4OaWVHsaeomdBKvIizvGtskiku97BBbJHiz i3brkz/BpmuF6XsSynJ2xa0G8vwpXAnGjy2SsPdXY3juBYYTYLvItqe0S3/xJT6Hvb43 0XYjfw+6drIYUTISM35tIHcj35nuLMG+Ji5FGvSRrCPjl5chRITyUuQ8tpB9FgADO8hz cyiGha02tg6on5aT1/IxQFC/BscrkCCofCknqDg6mXsSlpgOihbH2qXyeCIcqEN6nzxv bZWg== X-Gm-Message-State: AC+VfDwAa3TAunrY0ZPq1xmmVVUEeEkIwxnXpt9TAWWNHmRCL5H2Aesz vZqK3uVCS/nkJYravQev9zxLdEks2aI= X-Google-Smtp-Source: ACHHUZ4/uktIQEljZ46am/ae4mVKpKSl9L5/v6kSTX5kCcd0ELLeW3SE2bvxrvvTaKUZQE65+HOIMA== X-Received: by 2002:a17:907:3f99:b0:973:e349:43c9 with SMTP id hr25-20020a1709073f9900b00973e34943c9mr19203901ejc.77.1687519480072; Fri, 23 Jun 2023 04:24:40 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:7d95]) by smtp.gmail.com with ESMTPSA id h10-20020a1709067cca00b00969f44bbef3sm5959769ejp.11.2023.06.23.04.24.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 04:24:39 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 07/11] io_uring: kill io_cq_unlock() Date: Fri, 23 Jun 2023 12:23:27 +0100 Message-Id: <7dabb36856db2b562e78780480396c52c29b2bf4.1687518903.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We're abusing ->completion_lock helpers. io_cq_unlock() neither locking conditionally nor doing CQE flushing, which means that callers must have some side reason of taking the lock and should do it directly. Open code io_cq_unlock() into io_cqring_overflow_kill() and clean it up. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 776d1aa73d26..2f55abb676c0 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -644,12 +644,6 @@ static inline void io_cq_lock(struct io_ring_ctx *ctx) spin_lock(&ctx->completion_lock); } -static inline void io_cq_unlock(struct io_ring_ctx *ctx) - __releases(ctx->completion_lock) -{ - spin_unlock(&ctx->completion_lock); -} - /* keep it inlined for io_submit_flush_completions() */ static inline void __io_cq_unlock_post(struct io_ring_ctx *ctx) __releases(ctx->completion_lock) @@ -694,10 +688,10 @@ static void io_cqring_overflow_kill(struct io_ring_ctx *ctx) struct io_overflow_cqe *ocqe; LIST_HEAD(list); - io_cq_lock(ctx); + spin_lock(&ctx->completion_lock); list_splice_init(&ctx->cq_overflow_list, &list); clear_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq); - io_cq_unlock(ctx); + spin_unlock(&ctx->completion_lock); while (!list_empty(&list)) { ocqe = list_first_entry(&list, struct io_overflow_cqe, list); From patchwork Fri Jun 23 11:23:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13290377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D20BEEB64D7 for ; Fri, 23 Jun 2023 11:24:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229902AbjFWLYo (ORCPT ); Fri, 23 Jun 2023 07:24:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231363AbjFWLYn (ORCPT ); Fri, 23 Jun 2023 07:24:43 -0400 Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61F542139 for ; Fri, 23 Jun 2023 04:24:42 -0700 (PDT) Received: by mail-ej1-x635.google.com with SMTP id a640c23a62f3a-9891c73e0fbso106318066b.1 for ; Fri, 23 Jun 2023 04:24:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687519480; x=1690111480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iJYqXXZ2rSE8cC4yUUULEsr78gxRL+8pWiAzPGjpLC8=; b=imTIlEuXTuDeeA/PWid1DZ3+BqTN9h+FKoNQ628hWXeQvfY23n76NlLbBQSh+f23Jk Lo3uB9xB7MgYHQ5DgHLifHa0m/gxXrP5yusuPiCKOsIXIxYF5fzHwTUpkmpNyzBU9VJJ qwCNIoJkaDkV51HWj1Ozo+zTVorwgjlLvrPp0JvU9okShH3m0gKYFNuM+0JlNyP3UEvT WlM9487KPS28BKU6zl5NZcWh4bJY3lLKb3FzhCFTSjZWLgQwVmvz8h1SY9Pf3eUmNGjw 4sAh3eO8j1PjEMe72dVVuLaXwugfl+CyQSDhyOLZ+PC1qBuQLlnS1NLGpVUIYwRJUBXc FCiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687519480; x=1690111480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iJYqXXZ2rSE8cC4yUUULEsr78gxRL+8pWiAzPGjpLC8=; b=HRNe+OHdR51XAnz6Mv4lIg35vDM3CFxRw7mcJczGFeEECFkDU5E9PMg2C5r3rya81A 8DoWlv2FoyoH2qZyS6ovnIjtK+DjtbggStNxhWhXccEgohjqJhhqOvfEMlaaRDIG3Xp2 wNRjqce8GyhSSZb7DSN3HOLtAZOljmv9wGJjB0aBkwguPXfP5e3Z9pbIfSH3q8fTINct gtJHyJC5pcgPN1U5oINA+os3l4+EZ1lmalpoK3UIP2Phveh9409NvwI9j02Y1aCP+K/h w4Sl/v1Oh1h8kcjLkM/btSiNA9cMsd1TG4oUy74HrU06A17j2VqX5fBsL83w8HCcohai ttSA== X-Gm-Message-State: AC+VfDwohzg20hG6trKVj6uxF49Y3llUfLeTkP0WqAMmvlIMYDes1CDf Rjm0KltUwL+riFZS/WzBkBcc08pw3hQ= X-Google-Smtp-Source: ACHHUZ65A82y6FVBc/eIPZDK+rL1dJk+fB9AfDaofzMBjPFYcAE/i8kd4np0OA7vgLid0tnkb3f3zw== X-Received: by 2002:a17:907:7ba5:b0:982:4b35:c0b6 with SMTP id ne37-20020a1709077ba500b009824b35c0b6mr20617803ejc.1.1687519480637; Fri, 23 Jun 2023 04:24:40 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:7d95]) by smtp.gmail.com with ESMTPSA id h10-20020a1709067cca00b00969f44bbef3sm5959769ejp.11.2023.06.23.04.24.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 04:24:40 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 08/11] io_uring: fix acquire/release annotations Date: Fri, 23 Jun 2023 12:23:28 +0100 Message-Id: <2a098f9144c24cab622f8bf90b39f44da5d0401e.1687518903.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We do conditional locking, so __io_cq_lock() and friends not always actually grab/release the lock, so kill misleading annotations. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2f55abb676c0..8cb0f60d2885 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -626,7 +626,6 @@ void __io_commit_cqring_flush(struct io_ring_ctx *ctx) } static inline void __io_cq_lock(struct io_ring_ctx *ctx) - __acquires(ctx->completion_lock) { if (!ctx->task_complete) spin_lock(&ctx->completion_lock); @@ -646,7 +645,6 @@ static inline void io_cq_lock(struct io_ring_ctx *ctx) /* keep it inlined for io_submit_flush_completions() */ static inline void __io_cq_unlock_post(struct io_ring_ctx *ctx) - __releases(ctx->completion_lock) { io_commit_cqring(ctx); __io_cq_unlock(ctx); @@ -655,7 +653,6 @@ static inline void __io_cq_unlock_post(struct io_ring_ctx *ctx) } static void __io_cq_unlock_post_flush(struct io_ring_ctx *ctx) - __releases(ctx->completion_lock) { io_commit_cqring(ctx); From patchwork Fri Jun 23 11:23:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13290380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53E22EB64D7 for ; Fri, 23 Jun 2023 11:24:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231315AbjFWLYr (ORCPT ); Fri, 23 Jun 2023 07:24:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231544AbjFWLYo (ORCPT ); Fri, 23 Jun 2023 07:24:44 -0400 Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 455082685 for ; Fri, 23 Jun 2023 04:24:43 -0700 (PDT) Received: by mail-ej1-x62b.google.com with SMTP id a640c23a62f3a-988c495f35fso56965066b.1 for ; Fri, 23 Jun 2023 04:24:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687519481; x=1690111481; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V86l4zwMKuhgaLWzEpQqCEbB3yYQpidNZsksimiGaz4=; b=IQBPUqetn1ie4xNvdoLm+Hgwmh1xmbvsmmMdqEtIPF1TR0lGVaWWmz0D6wxHrdednt s6oqELQM78fb/Y+NpWFS3mqRwzsV4CP1zr7Pj4z8lnpaiduZLfWJ3ZUpjUjnwxQVj/81 5yRRkk1b4fRscmRGvRvwMgnFLmPyaKFTlLPCZdbZkqH05yZ1R7Rv45h3JrA3TELdFUc8 eXTfrXfFd4Oumq9rFbANYVNS0+JkNFBdQpDw7MxLhyQvp3jsickjsuyLIBL3oc4411rL I4K/628jB0J7U84NHFPTm9u3FXaC/fEBj6D1clgRz2tYqUezKgYpzzMSx1c4WH80jRD1 5oeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687519481; x=1690111481; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V86l4zwMKuhgaLWzEpQqCEbB3yYQpidNZsksimiGaz4=; b=fwkYlajSXutpVSHoVUKnkvooK+l2WX4wyaiW/AfHJBus/20h+hoJt9NevjyYBfJvsP 9+SOLgdyhYv9xF6R7uEeE5vYnB1f8VCNZK1VPz2tH3EsaUhDkSRPk6tFEu/zof9MdVcm dhZ9Hq+n0zf3FOxeg0bBrImVwT++eRM53FSskpIPmm+n3mQwz692M0ba2Aya/qI9j2p1 xdY2+MzzmKDydVnTKeJitXk45XgYRwREolRV1lnU6Uazz5Uhnelkm99xlVFzG7QAtaDV pL+/JAwReqy8W8cJiVKXDLsBv1/9o/clz53VHEo+atn7rsGf+4f3pKno9DwHdjsGRPb1 hKIQ== X-Gm-Message-State: AC+VfDwaF9teSHz4+dVdE2gzSvp540NDd1qGdWS1jxJvPaz/KFGiQ6Ap qrz54Kn2e6FA6+pHi7OOLdJJEbXGNTI= X-Google-Smtp-Source: ACHHUZ7VH2gN2mTMoxpaln2HubwWXLfStemXcce2s2mPJ2+x0sTvHjGiJwq+SKgoUthnLj2A4cDSWg== X-Received: by 2002:a17:907:2d28:b0:98d:81c7:f01c with SMTP id gs40-20020a1709072d2800b0098d81c7f01cmr1412191ejc.38.1687519481394; Fri, 23 Jun 2023 04:24:41 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:7d95]) by smtp.gmail.com with ESMTPSA id h10-20020a1709067cca00b00969f44bbef3sm5959769ejp.11.2023.06.23.04.24.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 04:24:41 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 09/11] io_uring: inline __io_cq_unlock Date: Fri, 23 Jun 2023 12:23:29 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org __io_cq_unlock is not very helpful, and users should be calling flush variants anyway. Open code the function. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8cb0f60d2885..39d83b631107 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -631,12 +631,6 @@ static inline void __io_cq_lock(struct io_ring_ctx *ctx) spin_lock(&ctx->completion_lock); } -static inline void __io_cq_unlock(struct io_ring_ctx *ctx) -{ - if (!ctx->task_complete) - spin_unlock(&ctx->completion_lock); -} - static inline void io_cq_lock(struct io_ring_ctx *ctx) __acquires(ctx->completion_lock) { @@ -647,7 +641,9 @@ static inline void io_cq_lock(struct io_ring_ctx *ctx) static inline void __io_cq_unlock_post(struct io_ring_ctx *ctx) { io_commit_cqring(ctx); - __io_cq_unlock(ctx); + if (!ctx->task_complete) + spin_unlock(&ctx->completion_lock); + io_commit_cqring_flush(ctx); io_cqring_wake(ctx); } @@ -664,7 +660,7 @@ static void __io_cq_unlock_post_flush(struct io_ring_ctx *ctx) */ io_commit_cqring_flush(ctx); } else { - __io_cq_unlock(ctx); + spin_unlock(&ctx->completion_lock); io_commit_cqring_flush(ctx); io_cqring_wake(ctx); } From patchwork Fri Jun 23 11:23:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13290381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC857C001B0 for ; Fri, 23 Jun 2023 11:24:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231383AbjFWLYs (ORCPT ); Fri, 23 Jun 2023 07:24:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231316AbjFWLYo (ORCPT ); Fri, 23 Jun 2023 07:24:44 -0400 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC4F01739 for ; Fri, 23 Jun 2023 04:24:43 -0700 (PDT) Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-9891c73e0fbso106321266b.1 for ; Fri, 23 Jun 2023 04:24:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687519482; x=1690111482; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W/0GCHAnwAQDlppeivwuLOoAX8CU5QX5fMWO37pepWE=; b=l2TZWufFQaUmTeC4g8BmLmBbR7dxJYsNxHDDr+l6IpOIsczxiB9T07j4cqadUVqYqr 4YxqcEyS1bUllVz8njPrDhQz5dgY7xJEIwOH3aTA6e04XSE74bzhdSnqR+Quk/MxH9jI MOSVyge3leQSyXKW2ZZGP+Gor62/0q91g3bVJfmm/dWgEiGVtV7c+a9nMGLQI3EM5M0u Lv1p6sI44eg/tverg/ctsUkeIB+KaeaeGynaH9PsAcsKM5bO0K+vxAcoHZ5jJoBMA4ly xw35gj/MqbbhHreq0rIXT8HdrWqDbhFSdFul7s6J+LHWydFa0hs432eCvGh97XLkTT0V RvYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687519482; x=1690111482; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W/0GCHAnwAQDlppeivwuLOoAX8CU5QX5fMWO37pepWE=; b=cTtDc6I36gwJP+hHjjfiy5HhhmTw43iKsFfruEeDXrILDKinc6P+JJdukG6Np1A6MB hbXmzHutj3/X2j9nuuYpznHYGaYwQrufTxEIzaRj/FkgDS7w67s2CGCLCy11uMdqVBZf uzc5ZRxurS81+xB64NGqDSWjtbA3blY9JHhDzQTHM+tEKE6VxokZ0TnHs14EZyMvwl32 CK7ZWja2CpsfN5qApfB3ihxbP8yuekvJmJA+c9DamontiiQ7GWzkqAQ/H+cjDGdZc9Ss GoJRmuCGqe1aF9odWIoihEW8IS4kjZvIFjyGcwG3xWx3Jj2Flab2vOwqP2Xgwjy3o7yT n8pg== X-Gm-Message-State: AC+VfDzgVlCpVw192zTObBmTg+PxKtMPjKQG5xmWeHHOgAs5Wd6WEp9r hTCmIKWir+GGdr/ghLSpsLDGeOdD/pI= X-Google-Smtp-Source: ACHHUZ4KV9HEjAWSBUk7IIJud9h5j9nkeODFGjA2HB9U3oFC1GIiNca7eVUq9qp0xkxx3ezUz4h7Vg== X-Received: by 2002:a17:907:3e16:b0:985:259f:6f50 with SMTP id hp22-20020a1709073e1600b00985259f6f50mr21242995ejc.34.1687519482154; Fri, 23 Jun 2023 04:24:42 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:7d95]) by smtp.gmail.com with ESMTPSA id h10-20020a1709067cca00b00969f44bbef3sm5959769ejp.11.2023.06.23.04.24.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 04:24:42 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 10/11] io_uring: make io_cq_unlock_post static Date: Fri, 23 Jun 2023 12:23:30 +0100 Message-Id: <3dc8127dda4514e1dd24bb32035faac887c5fa37.1687518903.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_cq_unlock_post() is exclusively used in io_uring/io_uring.c, mark it static and don't expose to other files. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- io_uring/io_uring.h | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 39d83b631107..70fffed83e95 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -666,7 +666,7 @@ static void __io_cq_unlock_post_flush(struct io_ring_ctx *ctx) } } -void io_cq_unlock_post(struct io_ring_ctx *ctx) +static void io_cq_unlock_post(struct io_ring_ctx *ctx) __releases(ctx->completion_lock) { io_commit_cqring(ctx); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 20ba6df49b1f..d3606d30cf6f 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -110,8 +110,6 @@ static inline void io_req_task_work_add(struct io_kiocb *req) #define io_for_each_link(pos, head) \ for (pos = (head); pos; pos = pos->link) -void io_cq_unlock_post(struct io_ring_ctx *ctx); - static inline struct io_uring_cqe *io_get_cqe_overflow(struct io_ring_ctx *ctx, bool overflow) { From patchwork Fri Jun 23 11:23:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13290382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9DC1EB64DD for ; Fri, 23 Jun 2023 11:24:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231521AbjFWLYu (ORCPT ); Fri, 23 Jun 2023 07:24:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231449AbjFWLYp (ORCPT ); Fri, 23 Jun 2023 07:24:45 -0400 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 99341213C for ; Fri, 23 Jun 2023 04:24:44 -0700 (PDT) Received: by mail-ej1-x62d.google.com with SMTP id a640c23a62f3a-988b75d8b28so56974866b.3 for ; Fri, 23 Jun 2023 04:24:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687519483; x=1690111483; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Z0z++lc82JOEdePiP0bM/cvCnEew7S7/z0AveFJp59E=; b=SMBR64+uLdVjkuc/X7vRJ8Plb4K6z5cEJiHFR7dVReyWzTPEfNORKowHYwibkGVSqR kaUfIAy+9N8Wb0YZx4SWMu2ALexH1Er70VX6DGeoJGsw5OXgawt02Sy3x91HjqqLH+TY e0/Ylx2sXCyoN/jbmY6dSDLdznVeNZ+KXyW/7qOSfD/OzZl/GwqTqehG5Q0aanI0j4pP SiqD8zWN00hhDs7lxQ8Fboa5ZB2FuF3y6JgpIk/8hMr4FHZ5IZsKpq5sLYIXbMD6l3AF xKG/VVg2t2fXTP8YG/+iqyomY5UcgC4uTthFP0i6KtElZ+lrwv4ImVQxClHwVUjoA2pw OkTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687519483; x=1690111483; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Z0z++lc82JOEdePiP0bM/cvCnEew7S7/z0AveFJp59E=; b=cPD9FUreZKjyv8DDHRZ1AU8RPdE91KA02BcyX1BIXkXnZWqz2dR0TxnpcEHbIBez3n L68G/snoKzxtvnVBhmKULeIfa913UXMnoppjh3MouU/jh2LUmiog03bl5qQpIf0NE4WV Z+nxepCqQ6zXdRBoBaz21Kr/BOxFXMbtJunEDfiLQsffbUdgHs3fIa0qfq9R+g2alBE2 58ILGyX9YeFLXEdy63VTmT/ptosTHSxo0ZW40pg5FT4fTNrPopR2WmRs+emjWOi1i9fS lVbhimjfyKG2/DqElc4kx92an9MLdFNi7Re+dxrnLaCkEIlGzt4Z2C81J9AF3PWl3s29 Fgmg== X-Gm-Message-State: AC+VfDyvZgymL8B8foTUSfKircweYuAbVHG5i43bYx4StYvtRGZA1iIe 0rydKvTJ6LWVRykdIbgwaLA6Zt5137o= X-Google-Smtp-Source: ACHHUZ7loyAMB6cZr+hgWaREpKy2atE9FS/EThHwHXM+Ir2+NSiMp6nqeXzu7lI8fSbF13o2MBx23A== X-Received: by 2002:a17:907:c0a:b0:96f:d345:d0f7 with SMTP id ga10-20020a1709070c0a00b0096fd345d0f7mr18583984ejc.62.1687519482729; Fri, 23 Jun 2023 04:24:42 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:7d95]) by smtp.gmail.com with ESMTPSA id h10-20020a1709067cca00b00969f44bbef3sm5959769ejp.11.2023.06.23.04.24.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 04:24:42 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 11/11] io_uring: merge conditional unlock flush helpers Date: Fri, 23 Jun 2023 12:23:31 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There is no reason not to use __io_cq_unlock_post_flush for intermediate aux CQE flushing, all ->task_complete should apply there, i.e. if set it should be the submitter task. Combine them, get rid of of __io_cq_unlock_post() and rename the left function. This place was also taking a couple percents of CPU according to profiles for max throughput net benchmarks due to multishot recv flooding it with completions. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 70fffed83e95..1b53a2ab0a27 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -637,18 +637,7 @@ static inline void io_cq_lock(struct io_ring_ctx *ctx) spin_lock(&ctx->completion_lock); } -/* keep it inlined for io_submit_flush_completions() */ static inline void __io_cq_unlock_post(struct io_ring_ctx *ctx) -{ - io_commit_cqring(ctx); - if (!ctx->task_complete) - spin_unlock(&ctx->completion_lock); - - io_commit_cqring_flush(ctx); - io_cqring_wake(ctx); -} - -static void __io_cq_unlock_post_flush(struct io_ring_ctx *ctx) { io_commit_cqring(ctx); @@ -1568,7 +1557,7 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) } } } - __io_cq_unlock_post_flush(ctx); + __io_cq_unlock_post(ctx); if (!wq_list_empty(&ctx->submit_state.compl_reqs)) { io_free_batch_list(ctx, state->compl_reqs.first);