From patchwork Fri Nov 4 10:59:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13031638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E230C43219 for ; Fri, 4 Nov 2022 11:03:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231869AbiKDLDP (ORCPT ); Fri, 4 Nov 2022 07:03:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231903AbiKDLCc (ORCPT ); Fri, 4 Nov 2022 07:02:32 -0400 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A44372CCBD for ; Fri, 4 Nov 2022 04:02:30 -0700 (PDT) Received: by mail-ej1-x62f.google.com with SMTP id ud5so12346157ejc.4 for ; Fri, 04 Nov 2022 04:02:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dKAY+S5nt8WWrRARXiiMhTGuuGow8SACAPMNZMPYgB4=; b=HRXwUPhRr8rnauN89ZLRf0p7CFtmtqiUNMMeQCC+A1JrWdBTbF5QEGTz7DGsOR+OCb JIqSYHAeC2nswwZ//SQ2alK1UWqNjm0Q62Iuc+PrCDoSt2H2y6i50ouqDyDqTPeJG8gI 6FS5f2TUOyFV+nEELZusZGc/QTIlGY9aT2vYOuZy2xYNm8Xu9vT8kGq2tKHqnowbVwXy Q1UuUbJZFtofbMQ9txJWko9IsSaaVdOAj1W/vWKsjZvCEherwgmY1hfwj7NLe6PfR6Oj CxvwjqM/UD2VkFRESaKTgKDejUvljw8MhUom89H3tvRudqbJI04x0LDdgjbi3ioraR7v Zytw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dKAY+S5nt8WWrRARXiiMhTGuuGow8SACAPMNZMPYgB4=; b=vtUkKcIz8y1pbg8Ht04qeazrwr7RUSOGdp6/AFn2nS3d/FgrHs0C07CJU8xw4WANgd wrhuMWe2d2jOBpleB0lY3YxU22xNWQm5eLr0VlpDon+bA7/oRkf3GxoD0SXgW/z1LVUt m1zPiNZOuuSeG5B/9CKk21+vhV72FvHzm1G0hOo/L3RcRL/3T4z4VKzLHG+ooVZTViqa qxUb0YCNih+ITg8TfmiR9pbY4xxMe4X1z8EuQgzbrt9P0BV2322T2JsEScnCbjycSIIY c55UhfTSJAivxlv4azzbVRQrG+ZSPC9N7RxngSgK6aYIFLRR6oR0JLeHnriYr95avj9f Ubxw== X-Gm-Message-State: ACrzQf1SSNFGH7fuC/giCCne63Hcs9BS18Z/DarIBr+Ym7B88eWreeiM OGQ6IGNPIdRo+7rWEJ5Zl6r5ogPhlJs= X-Google-Smtp-Source: AMsMyM6Ws71iRnbdvv3hUw6JuiXF7ckDo7OErRxJJUeBsMGrtXBQUsv8mLnYvOBwp0K6h7kNzY1rzQ== X-Received: by 2002:a17:906:d142:b0:7ad:eb7f:bc9a with SMTP id br2-20020a170906d14200b007adeb7fbc9amr19054993ejb.337.1667559748693; Fri, 04 Nov 2022 04:02:28 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:4173]) by smtp.gmail.com with ESMTPSA id u25-20020aa7db99000000b00458947539desm1757768edt.78.2022.11.04.04.02.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Nov 2022 04:02:28 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 1/7] io_uring: move kbuf put out of generic tw complete Date: Fri, 4 Nov 2022 10:59:40 +0000 Message-Id: <94374c7649aaefc3a17808dc4701f25ccd457e25.1667557923.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There are multiple users of io_req_task_complete() including zc notifications, but only read requests use selected buffers. As we already have an rw specific tw function, move io_put_kbuf() in there. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 6 ------ io_uring/rw.c | 6 ++++++ 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ac8c488e3077..db0dec120f09 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1471,12 +1471,6 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) void io_req_task_complete(struct io_kiocb *req, bool *locked) { - if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) { - unsigned issue_flags = *locked ? 0 : IO_URING_F_UNLOCKED; - - req->cqe.flags |= io_put_kbuf(req, issue_flags); - } - if (*locked) io_req_complete_defer(req); else diff --git a/io_uring/rw.c b/io_uring/rw.c index bb47cc4da713..1ce065709724 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -286,6 +286,12 @@ static inline int io_fixup_rw_res(struct io_kiocb *req, long res) static void io_req_rw_complete(struct io_kiocb *req, bool *locked) { io_req_io_end(req); + + if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) { + unsigned issue_flags = *locked ? 0 : IO_URING_F_UNLOCKED; + + req->cqe.flags |= io_put_kbuf(req, issue_flags); + } io_req_task_complete(req, locked); } From patchwork Fri Nov 4 10:59:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13031639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A34EC4332F for ; Fri, 4 Nov 2022 11:03:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231873AbiKDLDP (ORCPT ); Fri, 4 Nov 2022 07:03:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231906AbiKDLCd (ORCPT ); Fri, 4 Nov 2022 07:02:33 -0400 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE9222C124 for ; Fri, 4 Nov 2022 04:02:31 -0700 (PDT) Received: by mail-ed1-x52a.google.com with SMTP id a67so7016880edf.12 for ; Fri, 04 Nov 2022 04:02:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iTMFnPh1n90Hv/m+Li9XRv95Ppzz2u2qp0OsR4mKGF0=; b=jIQcxt0sQTe2ZHqDQT4pUMQ0X7a3QaTCDhtMHrFsrFyY/xJ3kIp453/TVtj8PBuves D9LrFhf+eGIqbc+6DjgXrRqyUMdLMeR0YCH2TZXKQ6hvITkn831sLx+nPKcozPiO5sDK c7Rah8P16FMJK36VBZC/VDLoyluxghtx5dHE4BXqbDDeKhM6lKLlzWvcbeFjj/ilvH9c BTdb6FpJI+Loz5KzXs4HoogDeWoHJIMU6hy32Rj40tIZCcvkln4p0gwxzHg0Ipf0HRrl 2J5iaCs/1zY1fv2fhSF5KcPaTZuOPYyoR4iOl5TaUwjAXse3Vs2A+NuS31rdk4b18BJF v4og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iTMFnPh1n90Hv/m+Li9XRv95Ppzz2u2qp0OsR4mKGF0=; b=Fa4zcP0yXGqrFXC3xRLKW94vQaodLWOCOreejl1ZLlPKdfEnb5i2Qoa6GmrXE0EbjF EK2oxOve8IbiDkzR7KYepGEkWqSzgmjbZf00x8Pjyk0mfRKhpgUUyvFMubnoToK+jUXE M41AZ3Ca7gRiq1cKQ2dtqFlxiE8AGSoU62QSDE6b6wyztcwtfe7r8d7DvzXe56r3Kp7p F6xMMIYlsnw4YpQ/MLZozK8it3pgcPmt2DMJWxyMmUIewwR0duwqYjrZvlArpvjRQawo 1OldZWMngLkROusQHzT6IAj4eWq53fr08O3U9YpuOFURKwa7BNq2JYpKASlU7crahnFQ qT4Q== X-Gm-Message-State: ACrzQf2yHSHDsr4QyJND9PR4dvIMC2CTAkcMSazanZ703eEXGLahyr8L Imib946rO4jpJTSf+dpUhhmDV5/fj5s= X-Google-Smtp-Source: AMsMyM6UjdQcswrjB8AYRRgEv+NGQxHB4xscMhVDEeX8fgV41zuX3pZx2Y6afxPUhVBdAS0qnUiutw== X-Received: by 2002:a05:6402:1771:b0:463:c94d:c7d9 with SMTP id da17-20020a056402177100b00463c94dc7d9mr16974998edb.135.1667559750011; Fri, 04 Nov 2022 04:02:30 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:4173]) by smtp.gmail.com with ESMTPSA id u25-20020aa7db99000000b00458947539desm1757768edt.78.2022.11.04.04.02.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Nov 2022 04:02:29 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 2/7] io_uring/net: remove extra notif rsrc setup Date: Fri, 4 Nov 2022 10:59:41 +0000 Message-Id: X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_send_zc_prep() sets up notification's rsrc_node when needed, don't unconditionally install it on notif alloc. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 1 - 1 file changed, 1 deletion(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 4bfef10161fa..59dafc42b8e0 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -60,7 +60,6 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) notif->task = current; io_get_task_refs(1); notif->rsrc_node = NULL; - io_req_set_rsrc_node(notif, ctx, 0); nd = io_notif_to_data(notif); nd->account_pages = 0; From patchwork Fri Nov 4 10:59:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13031643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C38BC4321E for ; Fri, 4 Nov 2022 11:03:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230157AbiKDLDT (ORCPT ); Fri, 4 Nov 2022 07:03:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229900AbiKDLCi (ORCPT ); Fri, 4 Nov 2022 07:02:38 -0400 Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76A562CDC1 for ; Fri, 4 Nov 2022 04:02:36 -0700 (PDT) Received: by mail-ej1-x635.google.com with SMTP id k2so12342182ejr.2 for ; Fri, 04 Nov 2022 04:02:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HrKFQ51HZlKuVt9GY4ZNgwEwhz5/8UDovxxA86AN5mc=; b=bU6JVE+EJbIOlsetxvV/1/i53fyeXnSn2uaE7yigQBdUE1g3bfxTY32cVom/xAL2Uq kfawbUBgoz3sFLBa150ccBZEy2sSXmXPW7t6gsQKLd3rEiTtwAt+wdLm0Ii1nfVxW5S4 YfJdTgakQTESKzaCl9McmEVGrqQuowXHO6okNdsVtLTlLVJV9XdyqbyzmAD7z11Mfu+N EhG98OpHmn4+dEMzsaHnKeQAMGvXkvkFZN/r/DEwVZ5blJ36L7FqCpu7N6jjUNYVO4/U 3/nYbK4crR2qjtqm1RjNLXUFLboQA5llxA01iHddkSCGzKnyhhYIsef6oo4THbSt4gLE SLNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HrKFQ51HZlKuVt9GY4ZNgwEwhz5/8UDovxxA86AN5mc=; b=UwJhU4gl2MBIU9eIzhLo+b2g0mGTBNM5z9LnEE4BALCtyvLxD8CwcKSawbdNKfHaNx YnrGZyvI3lhbshV8DtEqY6R/vFoeeaOtduhtEpjIYIhvDjSZUV22blQ5ozh9hyOOwxOp zEsfBRPIUWTFYXUTZuGo+dnMn4S35pQB5c9iKMfADFZuVwFJnens2db++jLE0HNvb0Dl kekzOy+F39M/yZxaTtmOzLBqt8jCa/c4eU89BM1zORGVkhn0BXugQI4tu3wV4kyPeRe/ X2BxF0/q1LtwViuDX/tYlhj7QmHD8Ing1+0ukH7Lbh2uELgMmw8rRD8+NwRHIqqgd/62 j59w== X-Gm-Message-State: ACrzQf2pKyU5YaF9H5Lpx+sNZ2ewCHNypcPyzbPeOEHUZx2p0gSWQaHV ZpwXNlB745+LFhamEZJLthzDBKuOxuQ= X-Google-Smtp-Source: AMsMyM5qUXix6wJGqGU8twKx6WZY+Ne/sPotEsNTyUizqy3mk5EaypB+bWssg8enjnEn/35g+XnM1g== X-Received: by 2002:a17:907:25c5:b0:783:f5df:900e with SMTP id ae5-20020a17090725c500b00783f5df900emr31926173ejc.491.1667559750918; Fri, 04 Nov 2022 04:02:30 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:4173]) by smtp.gmail.com with ESMTPSA id u25-20020aa7db99000000b00458947539desm1757768edt.78.2022.11.04.04.02.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Nov 2022 04:02:30 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 3/7] io_uring/net: preset notif tw handler Date: Fri, 4 Nov 2022 10:59:42 +0000 Message-Id: <7acdbea5e20eadd844513320cd454af14ba50f64.1667557923.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We're going to have multiple notification tw functions. In preparation for future changes default the tw callback in advance so later we can replace it with other versions. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 59dafc42b8e0..6afb58b94297 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -39,10 +39,8 @@ static void io_uring_tx_zerocopy_callback(struct sk_buff *skb, WRITE_ONCE(nd->zc_copied, true); } - if (refcount_dec_and_test(&uarg->refcnt)) { - notif->io_task_work.func = __io_notif_complete_tw; + if (refcount_dec_and_test(&uarg->refcnt)) io_req_task_work_add(notif); - } } struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) @@ -60,6 +58,7 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) notif->task = current; io_get_task_refs(1); notif->rsrc_node = NULL; + notif->io_task_work.func = __io_notif_complete_tw; nd = io_notif_to_data(notif); nd->account_pages = 0; @@ -76,8 +75,6 @@ void io_notif_flush(struct io_kiocb *notif) struct io_notif_data *nd = io_notif_to_data(notif); /* drop slot's master ref */ - if (refcount_dec_and_test(&nd->uarg.refcnt)) { - notif->io_task_work.func = __io_notif_complete_tw; + if (refcount_dec_and_test(&nd->uarg.refcnt)) io_req_task_work_add(notif); - } } From patchwork Fri Nov 4 10:59:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13031641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B55B3C43217 for ; Fri, 4 Nov 2022 11:03:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231336AbiKDLDR (ORCPT ); Fri, 4 Nov 2022 07:03:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39496 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231912AbiKDLCe (ORCPT ); Fri, 4 Nov 2022 07:02:34 -0400 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91A7E2C653 for ; Fri, 4 Nov 2022 04:02:33 -0700 (PDT) Received: by mail-ed1-x531.google.com with SMTP id l11so7060757edb.4 for ; Fri, 04 Nov 2022 04:02:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=X/FFWQ0jqit+tHnoCmQVZSk0lg7We50/psae2QKn1MA=; b=UJ3Fni1z+4x+OfRraIdR9tGY7UwVCsoM+k04yL6sewiC3Tdki3vQ/VEGCpI7KE//eT 07wKUWfkX8PgdCBbyhFjgTQIojEh7umj454vky2GN4JqLp98OT1+RmuIwp/ijA+YwQuR rES54sRShTC8P4hTQ08hA5m6EPkfMqqSxgF9CNmcWdo3CdoBLJc9obngT4lNchikWj4G PXETskHo80q7MD0KJo9ST3bpyYzL0jbJTW4xGq9QuTiglW9RsD/kJ5yqywOZqpTHMfHU 7mhNkAzf+YzSD10ed0qIftQKXVGm0DqOfrGxzPKgiyzXkKY1JHIUL06eLxUZ5WfNyY99 7i6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X/FFWQ0jqit+tHnoCmQVZSk0lg7We50/psae2QKn1MA=; b=0a5J6b+7X39u0158/nHgbw2h799otI5VX3AK+4z2ZfqhaVCrNeSkG04YN/aRb/iCzz RRpLetkOp8hagto4Pd394E3YhO/Y4CkXnBrmKdFtHf1Dj3jrnihSCdKs5FppmJRYEKlT TaLdkpZIdfqJRPgMrInB7nsVdyclKCk5AlKXfrXWcR14pk05UOlc9NvOVB90njYHt8JI j+bHBIoPEEXVL/PTXR9Nbwi2aSlCHzurpR4+kn9lKvhRSXZTnBPSbdxQjciafB9ERepz UUh0lJL2w8IzYefrvkFT1PoPjXmvxjRAXGFTijP+QwoeT6WpLG3Mw2Zm/kxECCer5B+v Be8g== X-Gm-Message-State: ACrzQf0y19ErpmW2F7Q0T12kKgrzQRGbLj5KkGS1IsvR+Xkr1f3Lre2d 54mqaP2L2ZJixq/fLhiXTdpFijRP+UI= X-Google-Smtp-Source: AMsMyM6FY+jYrDJ4KQwegUwgCsomVjKVoZeOwbpGiyOywa7yrKnBgSCB6TqKz03I0NysoKC+92Ud5w== X-Received: by 2002:aa7:c417:0:b0:463:3f0c:be12 with SMTP id j23-20020aa7c417000000b004633f0cbe12mr28232877edq.35.1667559751837; Fri, 04 Nov 2022 04:02:31 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:4173]) by smtp.gmail.com with ESMTPSA id u25-20020aa7db99000000b00458947539desm1757768edt.78.2022.11.04.04.02.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Nov 2022 04:02:31 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 4/7] io_uring/net: rename io_uring_tx_zerocopy_callback Date: Fri, 4 Nov 2022 10:59:43 +0000 Message-Id: <24d78325403ca6dcb1ec4bced1e33cacc9b832a5.1667557923.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Just a simple renaming patch, io_uring_tx_zerocopy_callback() is too bulky and doesn't follow usual naming style. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 6afb58b94297..5b0e7bb1198f 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -25,9 +25,8 @@ static void __io_notif_complete_tw(struct io_kiocb *notif, bool *locked) io_req_task_complete(notif, locked); } -static void io_uring_tx_zerocopy_callback(struct sk_buff *skb, - struct ubuf_info *uarg, - bool success) +static void io_tx_ubuf_callback(struct sk_buff *skb, struct ubuf_info *uarg, + bool success) { struct io_notif_data *nd = container_of(uarg, struct io_notif_data, uarg); struct io_kiocb *notif = cmd_to_io_kiocb(nd); @@ -63,7 +62,7 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) nd = io_notif_to_data(notif); nd->account_pages = 0; nd->uarg.flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; - nd->uarg.callback = io_uring_tx_zerocopy_callback; + nd->uarg.callback = io_tx_ubuf_callback; nd->zc_report = nd->zc_used = nd->zc_copied = false; refcount_set(&nd->uarg.refcnt, 1); return notif; From patchwork Fri Nov 4 10:59:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13031640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D934C4321E for ; Fri, 4 Nov 2022 11:03:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231876AbiKDLDR (ORCPT ); Fri, 4 Nov 2022 07:03:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231916AbiKDLCf (ORCPT ); Fri, 4 Nov 2022 07:02:35 -0400 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA3C62C65A for ; Fri, 4 Nov 2022 04:02:34 -0700 (PDT) Received: by mail-ed1-x534.google.com with SMTP id l11so7060824edb.4 for ; Fri, 04 Nov 2022 04:02:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s7kMLHKuOOjs0FBY3ISWz8diaVZj6jaS7ZiegJtMLHc=; b=TOSESZB9Goyu6yYbab0Nrrk6t0nTmhn7oVq1k7hEz+ljdfLT8jRVl9YUXxronVLtAW ucBtyGrVN4p1soWRH83IOhp2R27hsPDq9aYtbOGW/NIYOysIyJpNlqGEgNYWE8tLcTKy k3h1dfTsvjM4D4Z17jcaHaJQns3UDUXvhENVuL5w9ak7jpee82BJyKTWat/8wRt9WN17 u60o3PNWMB4STCGrqGIl0+CXA+r+XXhUw2s+uUKq6cZM9kI1PJ+iVdT79C9uM/Z0GlbM OggYNk4PLrv9D4vtKBvAjU3h1Gj8XNiST7gvQ6qUJXbIsFUev16tQEyVXK/IZDZXmAuT iOOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s7kMLHKuOOjs0FBY3ISWz8diaVZj6jaS7ZiegJtMLHc=; b=Jypu5K9oy6Z9EY9G5AgkSgkm86aHe0ZMwoMJ+XLiU0ePAJFwAWPmKthFx4CppI/6eB 2/vtU4zv2d7ibLrQb9ETQ6itVMzbQDrW1DCSQPMHpYxz63rAr945tYxkupJrE9svwWB8 ZC6U5LdOIyjqWrxTuvycNIElSyLQeVKV6slGlMrH56erFNfx+92WRYsr5ZMz9JL+03fo fTVsf197h9zu/IDb1K0gRDjN0x5neDAvXsEpPARtDXBxQBHA6fh95tdRcTmmoT6EJIoN CPbYlD5bZ65KGn8RSKh8qE4jMeCzYjRFp+COLi2Towgc7E+4IzYef+o98JIHdHjjtb3m A32A== X-Gm-Message-State: ACrzQf3QBNEgE537DqJxP5rx60aTguOIgB5HSdA6NB+UOY27Hqzw83ja wzcg0ZM+BSYGOv/RaMGYaQDNjXZ/2E4= X-Google-Smtp-Source: AMsMyM6EaUNSWJZFzxePXCVUQ6o6Rf8oU8RX3/CTYN9NGMphW3ioaVOriClcJ7xngVat62hOiZtBjA== X-Received: by 2002:a05:6402:5024:b0:440:e4ad:f7b6 with SMTP id p36-20020a056402502400b00440e4adf7b6mr7274313eda.358.1667559752828; Fri, 04 Nov 2022 04:02:32 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:4173]) by smtp.gmail.com with ESMTPSA id u25-20020aa7db99000000b00458947539desm1757768edt.78.2022.11.04.04.02.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Nov 2022 04:02:32 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 5/7] io_uring/net: inline io_notif_flush() Date: Fri, 4 Nov 2022 10:59:44 +0000 Message-Id: <332359e7bd124138dfe51340bbec829c9b265c18.1667557923.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_notif_flush() is pretty simple, we can inline it. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 10 ---------- io_uring/notif.h | 11 ++++++++++- 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 5b0e7bb1198f..c287adf24e66 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -67,13 +67,3 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) refcount_set(&nd->uarg.refcnt, 1); return notif; } - -void io_notif_flush(struct io_kiocb *notif) - __must_hold(&slot->notif->ctx->uring_lock) -{ - struct io_notif_data *nd = io_notif_to_data(notif); - - /* drop slot's master ref */ - if (refcount_dec_and_test(&nd->uarg.refcnt)) - io_req_task_work_add(notif); -} diff --git a/io_uring/notif.h b/io_uring/notif.h index 4ae696273c78..7f00176020d3 100644 --- a/io_uring/notif.h +++ b/io_uring/notif.h @@ -18,7 +18,6 @@ struct io_notif_data { bool zc_copied; }; -void io_notif_flush(struct io_kiocb *notif); struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx); static inline struct io_notif_data *io_notif_to_data(struct io_kiocb *notif) @@ -26,6 +25,16 @@ static inline struct io_notif_data *io_notif_to_data(struct io_kiocb *notif) return io_kiocb_to_cmd(notif, struct io_notif_data); } +static inline void io_notif_flush(struct io_kiocb *notif) + __must_hold(¬if->ctx->uring_lock) +{ + struct io_notif_data *nd = io_notif_to_data(notif); + + /* drop slot's master ref */ + if (refcount_dec_and_test(&nd->uarg.refcnt)) + io_req_task_work_add(notif); +} + static inline int io_notif_account_mem(struct io_kiocb *notif, unsigned len) { struct io_ring_ctx *ctx = notif->ctx; From patchwork Fri Nov 4 10:59:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13031642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A88C5C43219 for ; Fri, 4 Nov 2022 11:03:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229485AbiKDLDS (ORCPT ); Fri, 4 Nov 2022 07:03:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38950 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231918AbiKDLCg (ORCPT ); Fri, 4 Nov 2022 07:02:36 -0400 Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7799F2C65C for ; Fri, 4 Nov 2022 04:02:35 -0700 (PDT) Received: by mail-ej1-x62b.google.com with SMTP id bj12so12221902ejb.13 for ; Fri, 04 Nov 2022 04:02:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=E10mPC6PQtfwv3JsRzakhMmHgVlpBD6yxwSuPka/SgY=; b=WCm2gVxjlK91l8Loecf7bj7l1NHvUAtiHfMxLeferarKSRGNwJ15jZfGzDlQykyzDh qZdQotRIvK7uEGuZIXWWrwV93rKqi9wHMlg3wGgPllc1dgeX/IuGFknLwpggLmSZuKTR XtSv4yVw6FPXH42w3+sWhAMXdaGCFq6EcJQslfNHGQQzKN3ixNxU5HB4WaGFWhnbbSDz u1VDe3QKJdEhzj4Lrm9RtBS6Td7UPghB+ItAZHd0CDsP+e1gvP6Y05x+M6W9YbUXXFOv bFksJzgARrFnsRMaX8yiYLXi3pPRNRmnFzj68PnUeXyRmS29hS7wpx5lrLpWUBZsTQjI G4UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E10mPC6PQtfwv3JsRzakhMmHgVlpBD6yxwSuPka/SgY=; b=ZvmRmfoi1uoXEqWtrSDicy57V4mQgYmihdBFLur9wAslSAqC1SG8/wmzVM2WxvFYq/ LfSzmZfx8y6ONqm0tZOJvvFHSkCNEtjabKS54EA0TvzWKUOHM+uoN3/f7UsJQUCKmz8l bR7sdjtjea5xcuYjZb2vyTZPbWij7+4VGLkRYQdAZxpf0YfC1Wdcv3025fMq6tLTm28y 7U68dznrS756hQfVVInA3cF4s7FCGpvrCeGw57ZfU//JmPG9938gUXuC0cHWuoM50DPp qelICTXgXWyFchc/GQ2yi1CG8FYfHJm7tkpobj3BN4POmQk2xOjJlWsfFGoIKwWOO9Iz Nebw== X-Gm-Message-State: ACrzQf0QMYAt1lbMwREBgoW/pjFdiLyYXwTl/j8bj0aqOtgUbz3X4iK2 QN3Jvm89tqBQnrlSmrlD7DUt9w+KQNo= X-Google-Smtp-Source: AMsMyM4l2Sl2+vbkKweXRHbCSO+umHaBn2k1f0TcmdB+ntissPP4U0t6twimvquVryfKpoEKpDStbQ== X-Received: by 2002:a17:907:7f92:b0:78d:ed9c:d86f with SMTP id qk18-20020a1709077f9200b0078ded9cd86fmr33254367ejc.251.1667559753684; Fri, 04 Nov 2022 04:02:33 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:4173]) by smtp.gmail.com with ESMTPSA id u25-20020aa7db99000000b00458947539desm1757768edt.78.2022.11.04.04.02.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Nov 2022 04:02:33 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 6/7] io_uring: move zc reporting from the hot path Date: Fri, 4 Nov 2022 10:59:45 +0000 Message-Id: <40de4a6409042478e1f35adc4912e23226cb1b5c.1667557923.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add custom tw and notif callbacks on top of usual bits also handling zc reporting. That moves it from the hot path. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 22 ++++++++++++++-------- io_uring/notif.c | 31 +++++++++++++++++++++++++++---- io_uring/notif.h | 1 + 3 files changed, 42 insertions(+), 12 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 0a8cdc5ae7af..556a48bcdbe4 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -925,6 +925,9 @@ void io_send_zc_cleanup(struct io_kiocb *req) } } +#define IO_ZC_FLAGS_COMMON (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_FIXED_BUF) +#define IO_ZC_FLAGS_VALID (IO_ZC_FLAGS_COMMON | IORING_SEND_ZC_REPORT_USAGE) + int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); @@ -937,11 +940,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (req->flags & REQ_F_CQE_SKIP) return -EINVAL; - zc->flags = READ_ONCE(sqe->ioprio); - if (zc->flags & ~(IORING_RECVSEND_POLL_FIRST | - IORING_RECVSEND_FIXED_BUF | - IORING_SEND_ZC_REPORT_USAGE)) - return -EINVAL; notif = zc->notif = io_alloc_notif(ctx); if (!notif) return -ENOMEM; @@ -949,6 +947,17 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) notif->cqe.res = 0; notif->cqe.flags = IORING_CQE_F_NOTIF; req->flags |= REQ_F_NEED_CLEANUP; + + zc->flags = READ_ONCE(sqe->ioprio); + if (unlikely(zc->flags & ~IO_ZC_FLAGS_COMMON)) { + if (zc->flags & ~IO_ZC_FLAGS_VALID) + return -EINVAL; + if (zc->flags & IORING_SEND_ZC_REPORT_USAGE) { + io_notif_set_extended(notif); + io_notif_to_data(notif)->zc_report = true; + } + } + if (zc->flags & IORING_RECVSEND_FIXED_BUF) { unsigned idx = READ_ONCE(sqe->buf_index); @@ -958,9 +967,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) req->imu = READ_ONCE(ctx->user_bufs[idx]); io_req_set_rsrc_node(notif, ctx, 0); } - if (zc->flags & IORING_SEND_ZC_REPORT_USAGE) { - io_notif_to_data(notif)->zc_report = true; - } if (req->opcode == IORING_OP_SEND_ZC) { if (READ_ONCE(sqe->__pad3[0])) diff --git a/io_uring/notif.c b/io_uring/notif.c index c287adf24e66..9864bde3e2ef 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -18,11 +18,17 @@ static void __io_notif_complete_tw(struct io_kiocb *notif, bool *locked) __io_unaccount_mem(ctx->user, nd->account_pages); nd->account_pages = 0; } + io_req_task_complete(notif, locked); +} + +static void io_notif_complete_tw_ext(struct io_kiocb *notif, bool *locked) +{ + struct io_notif_data *nd = io_notif_to_data(notif); if (nd->zc_report && (nd->zc_copied || !nd->zc_used)) notif->cqe.res |= IORING_NOTIF_USAGE_ZC_COPIED; - io_req_task_complete(notif, locked); + __io_notif_complete_tw(notif, locked); } static void io_tx_ubuf_callback(struct sk_buff *skb, struct ubuf_info *uarg, @@ -31,15 +37,33 @@ static void io_tx_ubuf_callback(struct sk_buff *skb, struct ubuf_info *uarg, struct io_notif_data *nd = container_of(uarg, struct io_notif_data, uarg); struct io_kiocb *notif = cmd_to_io_kiocb(nd); + if (refcount_dec_and_test(&uarg->refcnt)) + io_req_task_work_add(notif); +} + +static void io_tx_ubuf_callback_ext(struct sk_buff *skb, struct ubuf_info *uarg, + bool success) +{ + struct io_notif_data *nd = container_of(uarg, struct io_notif_data, uarg); + if (nd->zc_report) { if (success && !nd->zc_used && skb) WRITE_ONCE(nd->zc_used, true); else if (!success && !nd->zc_copied) WRITE_ONCE(nd->zc_copied, true); } + io_tx_ubuf_callback(skb, uarg, success); +} - if (refcount_dec_and_test(&uarg->refcnt)) - io_req_task_work_add(notif); +void io_notif_set_extended(struct io_kiocb *notif) +{ + struct io_notif_data *nd = io_notif_to_data(notif); + + nd->zc_report = false; + nd->zc_used = false; + nd->zc_copied = false; + notif->io_task_work.func = io_notif_complete_tw_ext; + io_notif_to_data(notif)->uarg.callback = io_tx_ubuf_callback_ext; } struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) @@ -63,7 +87,6 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) nd->account_pages = 0; nd->uarg.flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; nd->uarg.callback = io_tx_ubuf_callback; - nd->zc_report = nd->zc_used = nd->zc_copied = false; refcount_set(&nd->uarg.refcnt, 1); return notif; } diff --git a/io_uring/notif.h b/io_uring/notif.h index 7f00176020d3..c88c800cd89d 100644 --- a/io_uring/notif.h +++ b/io_uring/notif.h @@ -19,6 +19,7 @@ struct io_notif_data { }; struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx); +void io_notif_set_extended(struct io_kiocb *notif); static inline struct io_notif_data *io_notif_to_data(struct io_kiocb *notif) { From patchwork Fri Nov 4 10:59:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13031644 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B42DC433FE for ; Fri, 4 Nov 2022 11:03:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229900AbiKDLDT (ORCPT ); Fri, 4 Nov 2022 07:03:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230297AbiKDLCi (ORCPT ); Fri, 4 Nov 2022 07:02:38 -0400 Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63EE82CDC0 for ; Fri, 4 Nov 2022 04:02:36 -0700 (PDT) Received: by mail-ej1-x635.google.com with SMTP id y14so12274175ejd.9 for ; Fri, 04 Nov 2022 04:02:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WolyIFn2U5osDC5mnfvXJt+vvPB7uFoWsvLJZrMnIx0=; b=S/XGRJ0B0wWarYLlkOOP83iTi+moCOdmuhNiFIb9t9fmGIg996r9KCycRLUx0PmL6q eJ9Pekt1e12Odhwz6aY1B45kiPURgvdTf9mAUXfFQ+D6ZV64XsPuUo4+UJxpCnWo2nFG l0fYZnPRwv7vdbvrsuG4BXdct136UpWdIVckTDdZfUWUusC87U+FvfNUJq7/LY/8P2jE +TtOi9UN0gtbq7CaHWLBTsq873Vcna5r7VYAJiruGw2pJ2IcykuyOn5Sy0fnlZ/LPhlS ib1fyGMlM66Z+e8pNdA3kEC71Ua0tOOof8YZ/bRQySTSOTyCjtfyMPy1r9pWRdTJ6a3M ey9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WolyIFn2U5osDC5mnfvXJt+vvPB7uFoWsvLJZrMnIx0=; b=gbwTI8jVzsIBnBV656niXWd/bja+32WMi6OSuqLWr2ZEb+S0gC5+8EuHaMTmYSsnnN cWQyn94+z8LsNBNAL9epnrsmSf6lALOTvqJl25KYZxIe6hHhbuvwfC5qNOSKYw+Fjwdx /hz2sye2TocQsYQly2LNDeovmg/Dww3q6kshDgDsbIPNql9owlCkMiY27zKyCnVK352t rmJlH9swua3nUETgirfH8QojjsGTmuJBL4JVYFs2W+3iVEiI6Y97lKd2AbtEqjhCQk1a OxCgfSshM1jjypo79S7ffC1b4mk/51dmPgkDTvIxZYM+nFZuUDI7RGDLJNMB6DDk8xgs onlw== X-Gm-Message-State: ACrzQf3J7JPxIjNLGs64naJkZdrVABBmHY1tTybGkWAdhNuFSSSgnvwv x9a1oqcnYyACVhglygybhLIp6p0uYCM= X-Google-Smtp-Source: AMsMyM7Kw+AgqGpJ2vdIZjx3rrsGnPpEorpZZAfgAHL5ji6LrzN0c4lQ1aJn2GTUoAckZc0Fs7zyCg== X-Received: by 2002:a17:906:d96f:b0:7ad:f0af:5c07 with SMTP id rp15-20020a170906d96f00b007adf0af5c07mr17390624ejb.572.1667559754650; Fri, 04 Nov 2022 04:02:34 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:4173]) by smtp.gmail.com with ESMTPSA id u25-20020aa7db99000000b00458947539desm1757768edt.78.2022.11.04.04.02.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Nov 2022 04:02:34 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 7/7] io_uring/net: move mm accounting to a slower path Date: Fri, 4 Nov 2022 10:59:46 +0000 Message-Id: <1062f270273ad11c1b7b45ec59a6a317533d5e64.1667557923.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We can also move mm accounting to the extended callbacks. It removes a few cycles from the hot path including skipping one function call and setting io_req_task_complete as a callback directly. For user backed I/O it shouldn't make any difference taking into considering atomic mm accounting and page pinning. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 3 +++ io_uring/notif.c | 31 +++++++++++++------------------ 2 files changed, 16 insertions(+), 18 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 556a48bcdbe4..b7192e3e4d2d 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1099,6 +1099,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) return ret; msg.sg_from_iter = io_sg_from_iter; } else { + io_notif_set_extended(zc->notif); ret = import_single_range(WRITE, zc->buf, zc->len, &iov, &msg.msg_iter); if (unlikely(ret)) @@ -1160,6 +1161,8 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) unsigned flags; int ret, min_ret = 0; + io_notif_set_extended(sr->notif); + sock = sock_from_file(req->file); if (unlikely(!sock)) return -ENOTSOCK; diff --git a/io_uring/notif.c b/io_uring/notif.c index 9864bde3e2ef..c4bb793ebf0e 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -9,11 +9,14 @@ #include "notif.h" #include "rsrc.h" -static void __io_notif_complete_tw(struct io_kiocb *notif, bool *locked) +static void io_notif_complete_tw_ext(struct io_kiocb *notif, bool *locked) { struct io_notif_data *nd = io_notif_to_data(notif); struct io_ring_ctx *ctx = notif->ctx; + if (nd->zc_report && (nd->zc_copied || !nd->zc_used)) + notif->cqe.res |= IORING_NOTIF_USAGE_ZC_COPIED; + if (nd->account_pages && ctx->user) { __io_unaccount_mem(ctx->user, nd->account_pages); nd->account_pages = 0; @@ -21,16 +24,6 @@ static void __io_notif_complete_tw(struct io_kiocb *notif, bool *locked) io_req_task_complete(notif, locked); } -static void io_notif_complete_tw_ext(struct io_kiocb *notif, bool *locked) -{ - struct io_notif_data *nd = io_notif_to_data(notif); - - if (nd->zc_report && (nd->zc_copied || !nd->zc_used)) - notif->cqe.res |= IORING_NOTIF_USAGE_ZC_COPIED; - - __io_notif_complete_tw(notif, locked); -} - static void io_tx_ubuf_callback(struct sk_buff *skb, struct ubuf_info *uarg, bool success) { @@ -59,11 +52,14 @@ void io_notif_set_extended(struct io_kiocb *notif) { struct io_notif_data *nd = io_notif_to_data(notif); - nd->zc_report = false; - nd->zc_used = false; - nd->zc_copied = false; - notif->io_task_work.func = io_notif_complete_tw_ext; - io_notif_to_data(notif)->uarg.callback = io_tx_ubuf_callback_ext; + if (nd->uarg.callback != io_tx_ubuf_callback_ext) { + nd->account_pages = 0; + nd->zc_report = false; + nd->zc_used = false; + nd->zc_copied = false; + nd->uarg.callback = io_tx_ubuf_callback_ext; + notif->io_task_work.func = io_notif_complete_tw_ext; + } } struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) @@ -81,10 +77,9 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) notif->task = current; io_get_task_refs(1); notif->rsrc_node = NULL; - notif->io_task_work.func = __io_notif_complete_tw; + notif->io_task_work.func = io_req_task_complete; nd = io_notif_to_data(notif); - nd->account_pages = 0; nd->uarg.flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; nd->uarg.callback = io_tx_ubuf_callback; refcount_set(&nd->uarg.refcnt, 1);