From patchwork Fri May 24 22:58:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13673783 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83F4E84FB3 for ; Fri, 24 May 2024 23:05:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716591913; cv=none; b=pjbMMJT3Bv+YRl+GFgMYGnnSW3KBuQlAegf2M0oMcgwXEHEkBftwMc2xeD84Kv/xLuPf1WGoPCP2rJxB7q+kzCt2vbQVdID9MgiolpVVRkqk4NyONvO6+329Mbjb+MqbUPFZER2HK8FWaY1fw93so1khcG03Zdhsy1cAoN0Wc9U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716591913; c=relaxed/simple; bh=5h1TIwk485xzuo65BPvZ2GQeWf4HXjNF6I31wjce8EI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aSwzegMQYdGcQBum48EUPfJPpHd0ZnluyUil4TNaapjHVfDKcmas3OHHpuFuoeCT7WQN7uM9URb9jdV9Owq8sXksi4UdosmQS6ejnPcuz7vauZw+KtegYiAXP6+O4n/SJAYeHyOvfhd3VRrW70e4cDQcEdhkagOJtTRBzzG4n8U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ShBbjrC0; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ShBbjrC0" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-2bf8a874296so8260a91.2 for ; Fri, 24 May 2024 16:05:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1716591909; x=1717196709; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iYXbRi7M2c1HqxvzjiKFqL80MUJ6pePfMWM0r/nvfyU=; b=ShBbjrC0F6omhbjsEuFZ1000RlDejtYdkJIMa9dkDl4Kd+R6IGx3fT5m353t7waXMj yaeWqeDu9bRRdmlsw0WhfbjMWxJVGEyJP2qZzcveYVEpaAUp+GaHijSp0OEekEFPreYf g9eGMjpUFNBK3H5xOdbeifu+zSgF68k3vK+avxuTdQN4xT+pYF2GJmVOqVQHxVox0XDp RuXgq26jQ67MlhJakG54jW5vw9+EMtX//YgTZ5tNl5/CmkCZ/Vm52SRhZhVxirWO7Lgk H9D/a2DCRBDkWoNNp31inJJ0INu8yVnHV+a8Jz2MTT9V/Rx90H22SWw//VG8kOU+jdxJ fpKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716591909; x=1717196709; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iYXbRi7M2c1HqxvzjiKFqL80MUJ6pePfMWM0r/nvfyU=; b=gPnteg/Uqqm/zq8mw+XFe330BE/fvKqSdFQoKVNkcJ9N5r4GXezUICP25z5r30hfcr Z3Usju+iOnHyXJ3nXwvusdNTlbFk2xErqdub6aL/SAdrNwsU5pXfiRFgOTwgJ71dV6ke 0Fv/Ans+uF6G/JtARbmZDlTyAqfeFsGtrDcL19SKojAksG2rXFQLhOIa1XpQRw679YhR c8YE/JmZ3vlnBo6zoQKQ52ZY+mv05gWQAZisW8zIaYHbG6AABLjoCDDCZVDMXusWlhRu QmcEFG4foQGr9nv7DtV+tGszwm1nNeK04AGZx5JvH6njzEqLQOzIabM7IFmYSVGjj1hc CxKw== X-Gm-Message-State: AOJu0YyCTE4FZKtb5oDuafEoeoQIJ376NMM5Ow0j2ZtuvwOtECfw3Fwc +oXSRegu318vaqPf/1o6Zma/Hhfj8yKzqlv1BIVJf2wwA4MBCS5fxT42oEHGAr5XA8e7GL8T3pL M X-Google-Smtp-Source: AGHT+IHD0vyii1QhUAlW8nJGnfNGqF63b9R1oYz8Q/Q2N1zLqKGbf5F2a4HkoD5D7KvT12+e1Zitvg== X-Received: by 2002:a17:902:f688:b0:1f3:358:cc30 with SMTP id d9443c01a7336-1f4486d1fbamr40538705ad.2.1716591909372; Fri, 24 May 2024 16:05:09 -0700 (PDT) Received: from localhost.localdomain ([2620:10d:c090:400::5:7713]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1f44c7c59besm19147625ad.106.2024.05.24.16.05.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 May 2024 16:05:08 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/3] io_uring/msg_ring: split fd installing into a helper Date: Fri, 24 May 2024 16:58:46 -0600 Message-ID: <20240524230501.20178-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240524230501.20178-1-axboe@kernel.dk> References: <20240524230501.20178-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 No functional changes in this patch, just in preparation for needing to complete the fd install with the ctx lock already held. Signed-off-by: Jens Axboe --- io_uring/msg_ring.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index 81c4a9d43729..feff2b0822cf 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -173,25 +173,23 @@ static struct file *io_msg_grab_file(struct io_kiocb *req, unsigned int issue_fl return file; } -static int io_msg_install_complete(struct io_kiocb *req, unsigned int issue_flags) +static int __io_msg_install_complete(struct io_kiocb *req) { struct io_ring_ctx *target_ctx = req->file->private_data; struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); struct file *src_file = msg->src_file; int ret; - if (unlikely(io_double_lock_ctx(target_ctx, issue_flags))) - return -EAGAIN; - ret = __io_fixed_fd_install(target_ctx, src_file, msg->dst_fd); if (ret < 0) - goto out_unlock; + return ret; msg->src_file = NULL; req->flags &= ~REQ_F_NEED_CLEANUP; if (msg->flags & IORING_MSG_RING_CQE_SKIP) - goto out_unlock; + return ret; + /* * If this fails, the target still received the file descriptor but * wasn't notified of the fact. This means that if this request @@ -199,8 +197,20 @@ static int io_msg_install_complete(struct io_kiocb *req, unsigned int issue_flag * later IORING_OP_MSG_RING delivers the message. */ if (!io_post_aux_cqe(target_ctx, msg->user_data, ret, 0)) - ret = -EOVERFLOW; -out_unlock: + return -EOVERFLOW; + + return ret; +} + +static int io_msg_install_complete(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_ring_ctx *target_ctx = req->file->private_data; + int ret; + + if (unlikely(io_double_lock_ctx(target_ctx, issue_flags))) + return -EAGAIN; + + ret = __io_msg_install_complete(req); io_double_unlock_ctx(target_ctx); return ret; } From patchwork Fri May 24 22:58:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13673784 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 92B5B8529E for ; Fri, 24 May 2024 23:05:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716591914; cv=none; b=Ah3LuQjEllmHV0HlBHGd8ygAHrhZMabhNaR9HLLJxLKlAi1BE7RJ1BBSf95u86e7rAgKHa3wWxBe/cNGAgJEGFIvks6bnyihT7acL51b6ZZ+RZUKz158KDJF5CD61R3gWMJAlyisZSs2yrTKlsaEUEQTsZdnCHCEWdekbD19kis= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716591914; c=relaxed/simple; bh=U2IGbEXXa16a5LAv/By4sPwNPP5H0PGpeF24X++4DSU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SwI1NqAfac9lh/1+Z4ymZjtno+09vaiU76Pe8rmnVA9iSkRm5AZhcTZoQIqDsyHaI5N0XuGsmyOibGu+ygttUpicQnKSTZQqPjjk4soc5Tj4sOjuHTuBoZVntJtWdRqo5WkQtP0W5J1hfs9x8FmXWPLeb2AUljyf0Ly7T9v6Hr8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=isYc021G; arc=none smtp.client-ip=209.85.216.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="isYc021G" Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-2bdf439c664so388877a91.3 for ; Fri, 24 May 2024 16:05:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1716591911; x=1717196711; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uThJqRi9Hnh0v6X3DkxTRbLlnshZaUAZF244xM8v0NM=; b=isYc021GQmsNfxLLHtFYh9/iO88Gd5xx/OXM/aSa/Bl34fmSfXHghvXA1K634IOAXx E+qezXwNHrOiH4Z0cOVYMaxX8T/6LnYfQ+x2G50LLCQ6DZN3s4293G+l+n4wHIxtr7Uz Mq8f9iFFnCtyJK5+3H/cXByqHhBg4XIEKm/xfLfjlB9znx1k9ZCLR5Xc4COFYcyWDW3r y76ObY+LrmRdTVrUTpU73FzpvI6ruUUWZY+GvmSuWuJcmI5G3bO8M0VuFfScBdwciTwR DDFX6TjA+fRMDQIPCV7aOS/YFZsqX5XWy0pvZsqos/+o/CQg46O3ZNiI1bx/9nV+EIkH fZyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716591911; x=1717196711; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uThJqRi9Hnh0v6X3DkxTRbLlnshZaUAZF244xM8v0NM=; b=La8LwdCqeiaVoc463F0LZlI/+VOG6k3PDVxEZDmnsSeXma/TQ4yhlo4v+WSd87Y7os qNCw7cYyfIB6ltS5iVg/zNavN3GC0VgMla9fsWGAjKgc/4uTBa4fAWwH1Oc3ZrgfbGWx iErRu1do9lh1Y26MBhXktoJd3hs3YHWJLov0Y5aYCbb3ssBBdIA/au0Jznn2jhtteDkd dD5PzoafoGNo/H9AZYg8TbuZc232lld+k/QewV+YLOwjNry5YuThadNBXGikbBMXww3a PI5OuG6mzBBXfhEb/9oSa1iDLcfIkh0sdyZFktwRXF5sWW/hRKFhCck4fNcCq6n+XTJ+ RqOA== X-Gm-Message-State: AOJu0Yw6E7r3IVxHV7V2SWWVLfd3ekaoXI25r1j4QR+/WA+UW49aANXa howklIDdsdwFcJqITj6k0Am1Ur31Bju4KwtBLVw4MbkdX/P2em4CZSQXztuahAficl2E0tyMWxC c X-Google-Smtp-Source: AGHT+IHIlnjBN3gV+kcJYKx1KENTg5z1eBzdyN43TVGmGBl0bcwVmxb2ivhKzh0N6JQTTjDClnIx0g== X-Received: by 2002:a17:902:c94d:b0:1f2:f6ca:fbcb with SMTP id d9443c01a7336-1f4483e8e63mr39972565ad.0.1716591911148; Fri, 24 May 2024 16:05:11 -0700 (PDT) Received: from localhost.localdomain ([2620:10d:c090:400::5:7713]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1f44c7c59besm19147625ad.106.2024.05.24.16.05.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 May 2024 16:05:09 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/3] io_uring/msg_ring: avoid double indirection task_work for data messages Date: Fri, 24 May 2024 16:58:47 -0600 Message-ID: <20240524230501.20178-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240524230501.20178-1-axboe@kernel.dk> References: <20240524230501.20178-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If IORING_SETUP_SINGLE_ISSUER is set, then we can't post CQEs remotely to the target ring. Instead, task_work is queued for the target ring, which is used to post the CQE. To make matters worse, once the target CQE has been posted, task_work is then queued with the originator to fill the completion. This obviously adds a bunch of overhead and latency. Instead of relying on generic kernel task_work for this, fill an overflow entry on the target ring and flag it as such that the target ring will flush it. This avoids both the task_work for posting the CQE, and it means that the originator CQE can be filled inline as well. In local testing, this reduces the latency on the sender side by 5-6x. Signed-off-by: Jens Axboe --- io_uring/msg_ring.c | 77 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 74 insertions(+), 3 deletions(-) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index feff2b0822cf..3f89ff3a40ad 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -123,6 +123,69 @@ static void io_msg_tw_complete(struct callback_head *head) io_req_queue_tw_complete(req, ret); } +static struct io_overflow_cqe *io_alloc_overflow(struct io_ring_ctx *target_ctx) +{ + bool is_cqe32 = target_ctx->flags & IORING_SETUP_CQE32; + size_t cqe_size = sizeof(struct io_overflow_cqe); + struct io_overflow_cqe *ocqe; + + if (is_cqe32) + cqe_size += sizeof(struct io_uring_cqe); + + ocqe = kmalloc(cqe_size, GFP_ATOMIC | __GFP_ACCOUNT); + if (!ocqe) + return NULL; + + if (is_cqe32) + ocqe->cqe.big_cqe[0] = ocqe->cqe.big_cqe[1] = 0; + + return ocqe; +} + +/* + * Entered with the target uring_lock held, and will drop it before + * returning. Adds a previously allocated ocqe to the overflow list on + * the target, and marks it appropriately for flushing. + */ +static void io_msg_add_overflow(struct io_msg *msg, + struct io_ring_ctx *target_ctx, + struct io_overflow_cqe *ocqe, int ret) + __releases(target_ctx->uring_lock) +{ + spin_lock(&target_ctx->completion_lock); + + if (list_empty(&target_ctx->cq_overflow_list)) { + set_bit(IO_CHECK_CQ_OVERFLOW_BIT, &target_ctx->check_cq); + atomic_or(IORING_SQ_TASKRUN, &target_ctx->rings->sq_flags); + } + + ocqe->cqe.user_data = msg->user_data; + ocqe->cqe.res = ret; + list_add_tail(&ocqe->list, &target_ctx->cq_overflow_list); + spin_unlock(&target_ctx->completion_lock); + mutex_unlock(&target_ctx->uring_lock); + wake_up_state(target_ctx->submitter_task, TASK_INTERRUPTIBLE); +} + +static bool io_msg_fill_remote(struct io_msg *msg, unsigned int issue_flags, + struct io_ring_ctx *target_ctx, u32 flags) +{ + struct io_overflow_cqe *ocqe; + + ocqe = io_alloc_overflow(target_ctx); + if (!ocqe) + return false; + + if (unlikely(io_double_lock_ctx(target_ctx, issue_flags))) { + kfree(ocqe); + return false; + } + + ocqe->cqe.flags = flags; + io_msg_add_overflow(msg, target_ctx, ocqe, msg->len); + return true; +} + static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) { struct io_ring_ctx *target_ctx = req->file->private_data; @@ -137,12 +200,20 @@ static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) if (target_ctx->flags & IORING_SETUP_R_DISABLED) return -EBADFD; - if (io_msg_need_remote(target_ctx)) - return io_msg_exec_remote(req, io_msg_tw_complete); - if (msg->flags & IORING_MSG_RING_FLAGS_PASS) flags = msg->cqe_flags; + if (io_msg_need_remote(target_ctx)) { + /* + * Try adding an overflow entry to the target, and only if + * that fails, resort to using more expensive task_work to + * have the target_ctx owner fill the CQE. + */ + if (!io_msg_fill_remote(msg, issue_flags, target_ctx, flags)) + return io_msg_exec_remote(req, io_msg_tw_complete); + return 0; + } + ret = -EOVERFLOW; if (target_ctx->flags & IORING_SETUP_IOPOLL) { if (unlikely(io_double_lock_ctx(target_ctx, issue_flags))) From patchwork Fri May 24 22:58:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13673785 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0468E128375 for ; Fri, 24 May 2024 23:05:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716591915; cv=none; b=Ka2KXoIgvS/przk34g5gkNHaUEtvvkYXoej+VYWip7TRURedaZ86C3APYz2IJugEHetgcgnpr0ugva2x5iVQVqLuKs817Bz6xmFgX1GoEtVb55SQC8H+cdM06JLbt7G/snbI0+ashdsPzQ/Z56hBtYkza6mI9jzb1PUWJyUgVcc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716591915; c=relaxed/simple; bh=HuwTchKXVEOVFWPvm1c8IrnxAYgzECvApfsMp9S2qiA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IMy3p5WY0yjhxsB3RIWlHFXdXnMJAoBp41ySF6kCQLzmVdEpK5UZ865OPRv8cHydGbOxE6RxBpuLldc+LLvN0ZwORh+h5IZgGHybU+7JBLuqQ7yS94phGzktPboHV22ud/72bJuEsazQLfflbD61HKiY0EDkHOqkozDKJZYzaIc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=AIpsMpA7; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="AIpsMpA7" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2bf5ba9a2c5so255375a91.0 for ; Fri, 24 May 2024 16:05:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1716591913; x=1717196713; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5tTCqtzG38P5ezjXGz+gaZMEKifQ9r85IputMtEIhqE=; b=AIpsMpA7uMzDdie+YHXHkvUIe1c6jIgUPTVV1pZ2IkgKzyjLC0s6uUW7B1+hEHKrw+ hz+vQZ9o28msrsmB0UeOdRmYr8W29e1YWlJR3zB4BMF7N2a2gjat9cUcpDK3ZR97STFN zyoPAb8hxS7hDUVzJIr+4aOmLDeaN+iNW00X/m+ohjMaobjq0ajEMMw+1Imj2l5wmJyU JnDm4OVcVSXU2O1Bnjo/ov4Mq+a/8GVK7QUzufcbEEwMk8Zs76FQyh1Uw9HSgeI3dTn1 k2im7YrszVpSk/oJqDXv/w80n6NLK/IZF1HqLRPmJivWrI7GTDECg6Rhh0XIsP5Fxsz5 R8tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716591913; x=1717196713; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5tTCqtzG38P5ezjXGz+gaZMEKifQ9r85IputMtEIhqE=; b=r9+gHKCe8q+FMzeyEppXRvmT0rKOHyA2tDtUPO5smWujyIKka3SNXCPUj9Wyks16Ac bkLqhUfdQlrZ3Zl7o60OFLf5Q8QgGADWg14ysSjkUlSzJ11sWDdmlQNfEP8K7LdPaRwt rO9aW5KbFX7T8YfU4caKg8x+Cm4hT6H4mn2LcA71SMSaAhNm2aP9S07IMsGxmQcMJue+ tmr92P2dlQKF48egORG/9XeSzcP3uYCBZ0JFcleSDOI6KQN+4tdQ4bPA5MmiCoZmSqsj I4y6uJ35DxGlTU7iAthHY0Q+1G83f1TIVCNO36eYnBrPCWjJe7fgqV49fUolScv97xeF feVA== X-Gm-Message-State: AOJu0YxIG1LDv/mZkjWP4FssK8fU7I+fW7j/QBy8FVKyO0c8SMPEIhwG IPLxgFRL7KB1zpgfcXm9j0gTfV6/m5XXdD3X87qICZr5XveTpNa8YzSGrQhe4fQ/0+To992I9Vo 1 X-Google-Smtp-Source: AGHT+IHaLUK3HyBDLm6geJTOxcEhm+y2CEUUteAZTFKpKZ0Z0nkF8zP+ciRt5mhakuBomYa+A1GpLw== X-Received: by 2002:a17:902:e5cd:b0:1f2:efdf:a403 with SMTP id d9443c01a7336-1f4498ec503mr40824855ad.4.1716591912828; Fri, 24 May 2024 16:05:12 -0700 (PDT) Received: from localhost.localdomain ([2620:10d:c090:400::5:7713]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1f44c7c59besm19147625ad.106.2024.05.24.16.05.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 May 2024 16:05:11 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/3] io_uring/msg_ring: avoid double indirection task_work for fd passing Date: Fri, 24 May 2024 16:58:48 -0600 Message-ID: <20240524230501.20178-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240524230501.20178-1-axboe@kernel.dk> References: <20240524230501.20178-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Like what was done for MSG_RING data passing avoiding a double task_work roundtrip for IORING_SETUP_SINGLE_ISSUER, implement the same model for fd passing. File descriptor passing is separately locked anyway, so the only remaining issue is CQE posting, just like it was for data passing. And for that, we can use the same approach. Signed-off-by: Jens Axboe --- io_uring/msg_ring.c | 48 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index 3f89ff3a40ad..499702425711 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -299,6 +299,41 @@ static void io_msg_tw_fd_complete(struct callback_head *head) io_req_queue_tw_complete(req, ret); } +static int io_msg_install_remote(struct io_msg *msg, unsigned int issue_flags, + struct io_ring_ctx *target_ctx) +{ + bool skip_cqe = msg->flags & IORING_MSG_RING_CQE_SKIP; + struct io_overflow_cqe *ocqe; + int ret; + + if (!skip_cqe) { + ocqe = io_alloc_overflow(target_ctx); + if (!ocqe) + return -ENOMEM; + } + + if (unlikely(io_double_lock_ctx(target_ctx, issue_flags))) { + kfree(ocqe); + return -EAGAIN; + } + + ret = __io_fixed_fd_install(target_ctx, msg->src_file, msg->dst_fd); + if (ret < 0) + goto out; + + msg->src_file = NULL; + + if (!skip_cqe) { + ocqe->cqe.flags = 0; + io_msg_add_overflow(msg, target_ctx, ocqe, ret); + return 0; + } +out: + mutex_unlock(&target_ctx->uring_lock); + kfree(ocqe); + return ret; +} + static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) { struct io_ring_ctx *target_ctx = req->file->private_data; @@ -320,8 +355,17 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) req->flags |= REQ_F_NEED_CLEANUP; } - if (io_msg_need_remote(target_ctx)) - return io_msg_exec_remote(req, io_msg_tw_fd_complete); + if (io_msg_need_remote(target_ctx)) { + int ret; + + ret = io_msg_install_remote(msg, issue_flags, target_ctx); + if (ret == -EAGAIN) + return io_msg_exec_remote(req, io_msg_tw_fd_complete); + else if (ret < 0) + return ret; + req->flags &= ~REQ_F_NEED_CLEANUP; + return 0; + } return io_msg_install_complete(req, issue_flags); }