From patchwork Tue Jun 18 18:48:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13702849 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B81C715ECCD for ; Tue, 18 Jun 2024 18:56:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718737001; cv=none; b=ThELyObgqqMKYDui5ln82q6WT+SMDzp9PXAOKh4s6oC7YcIPgTcAUcPNiP5Wraq/Gi/z1/uRMvv4QPxogpR2EmbU3RvHawcsgVbJPhb7qG7BD9CvV5AIGyTQU+ULUcnF63gBW3QJfW3+cOUe8ssAOKIuQDAloL0aW7CAiGk8sOo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718737001; c=relaxed/simple; bh=Dg7eZOnIgLth5/bbTSu+ZhOWjL8S1TMLSv1i4QJE9bY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aBux6srKJLPGnCSb8XrysmJXfEXm4FJ8Z7h69T6aczJ6oBx+qtp0vhI/ZTP9466i1f+fPHRrfTUyzBpH8gBOoaxSfzTPu4Ab5PunoZGNnu3YpzeCPTFd1UGrON2MjBkng1FiDwyyDf0vC1Og28aLeN8LuB8FjuwXCNFxCGh/5U0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=qZA0489F; arc=none smtp.client-ip=209.85.167.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="qZA0489F" Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3d21cb3833cso198439b6e.2 for ; Tue, 18 Jun 2024 11:56:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1718736998; x=1719341798; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T1EmDC2bGMmU0XeSvchLBl6+1crQIfO4g6MqrY1wcIM=; b=qZA0489F2bv7bK0rZKWSiZJOyW3hkphIDPx2B/UPzCKq0YNijL9JFY1TcFOdENyr+a K8tNbxtrmrewpOYNz+8ONRJG7y8af8JRjtRiuKH3rdVTg3TWuXdqPKaSGGuclLGSN1ZB 4yZxSC8Tnisf9yCK064+Wc2ZAeIj5ftP18ctLTFBj4b7BkD9hYZLykBNxpx9bTy/J8/i MF6cgAMBQw+JneODjL/t/sWsV3tZLDmNyLJ+e8Iqxb3l40o6QkEisyJU3nuBxxseIXn2 tn7DDHu3qgguG+QJ6ARU9QSsi5ZMPXrtY88afQYQi7pUMzNP9tXrVWLnfseoPwcNZ/rM t2qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718736998; x=1719341798; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T1EmDC2bGMmU0XeSvchLBl6+1crQIfO4g6MqrY1wcIM=; b=nglW/JVA0KFGmHE0aF1oyWspjG5o888f/OY/GGu2fl6D3o9bDHhWG8xpD8qDOi9Gmh ILw4c11d98UklqkrKdGE5lWT6WtebSW0qo9OmnKvck/Jw0p6bsjKXjbKAqoR/WUxQJw5 Uclw46xo8bWgpS1CemqMruHgayCY+2AYquW4gyLCwv+dV0/2a+WwpgDsCtPDMEhGxaki MpIGDmppaW+EocHG0orGD7aJIU8fPrFl3jSRoX0KpQEk7dyuJm7+y5W/ydkcrZFJQS/W Xcu+Ke60aYqNddGHBSCadCLcJevqhFGiH2Oedwl/51bWGKyXSctrBwCy3V04OPKqZ8K8 J1ew== X-Gm-Message-State: AOJu0Yw4kh5ERp3KKHE5ayCbmn+qczHwMEagN7qJmapKE08jJAO3ItOE xF2hibbNlChhjj5uWW8xuXLoxskYI4ZMeuHYfAquBWb4sH8gGQOt7vFp5YvGpY2u6AI0n9kCPwD Q X-Google-Smtp-Source: AGHT+IFu3gYPd84wGTjZbeDH+YWLvT4rEBMmk69RJTTtKTWIk62ifXIgJ4jsJ9G0T0eTU76e2Exlvw== X-Received: by 2002:a05:6870:ec8e:b0:254:d417:34ff with SMTP id 586e51a60fabf-25c94da7916mr664194fac.4.1718736996396; Tue, 18 Jun 2024 11:56:36 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-2567a9f7d6fsm3255492fac.20.2024.06.18.11.56.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Jun 2024 11:56:35 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 1/5] io_uring/msg_ring: tighten requirement for remote posting Date: Tue, 18 Jun 2024 12:48:40 -0600 Message-ID: <20240618185631.71781-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240618185631.71781-1-axboe@kernel.dk> References: <20240618185631.71781-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Currently this is gated on whether or not the target ring needs a local completion - and if so, whether or not we're running on the right task. The use case for same thread cross posting is probably a lot less relevant than remote posting. And since we're going to improve this situation anyway, just gate it on local posting and ignore what task we're currently running on. Signed-off-by: Jens Axboe --- io_uring/msg_ring.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index 81c4a9d43729..9fdb0cc19bfd 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -68,9 +68,7 @@ void io_msg_ring_cleanup(struct io_kiocb *req) static inline bool io_msg_need_remote(struct io_ring_ctx *target_ctx) { - if (!target_ctx->task_complete) - return false; - return current != target_ctx->submitter_task; + return target_ctx->task_complete; } static int io_msg_exec_remote(struct io_kiocb *req, task_work_func_t func) From patchwork Tue Jun 18 18:48:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13702850 Received: from mail-oa1-f44.google.com (mail-oa1-f44.google.com [209.85.160.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A37A715E5CA for ; Tue, 18 Jun 2024 18:56:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718737003; cv=none; b=lt6JOp1QIKjDIfUi4iGZ9wAgDqAMYuIDupUzjbU/dLd9sWHQZSDgLQuI5Gp17ZJPFqIbijDhyUzXS8/rFssP6kfsPt30c5YTkRZqIoKa5Ylhuvw1gmoe9SMujyHiB2FTUhrODvbzSLaQXpBSwGFKA2Fcq1t8InsKlk1iZyfbc3Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718737003; c=relaxed/simple; bh=sUZOHfP9JoKBzBl6ELl7LD1QpDsPwL4mR0POXRW8ipQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=r8gLfmD2MyA7wBekq6yCfi2sXvTtBHYE7Q/IcqNiAE4F7UPT773KTM7Fsqao+hTZdH1Nl84rLEj8CvLRazbraYL82WG3J4qT2sszyaUD4lowMDxhQ3VqEPmnk0/q30mYzBhE4kYE2DZGe5QuxGI1kvDzPEb7E/xmdpljNYa00jY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Jvp9NjoL; arc=none smtp.client-ip=209.85.160.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Jvp9NjoL" Received: by mail-oa1-f44.google.com with SMTP id 586e51a60fabf-24c582673a5so473447fac.2 for ; Tue, 18 Jun 2024 11:56:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1718737000; x=1719341800; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hc6I5c736RCR/T+6zHjTPji7tCumWOIEIjK9x9cxQIM=; b=Jvp9NjoLIl389vXvIRDd7rehLAq9T2u/n8bXApH23dp8CkXsWZprSp0aPXX39pAS4S QthoKzJq0eoz0MH3N6csCH9Cx5v2U65g6SFT7Hno5eTpVfdjOYASnPicreMUezBPrUQG KUzH5wjxdHIuv5yOSgjwEIiXldJ/1UWbZddwFb7b0b4hYXKjLBrDtho6dM+CLzeYnu6U 3ej613Q42dJOWo4VFXqzESIPV+6vhVArsMntO0ieyJfR7YsXLhOLIJmwjzDaX2Gfh+8g ToXsM7/JwheopuNZGQ3Wk5XcN/sJwJhDcYrKojHas4KA33OOAY0pDR5HJvWbqRqqkrOx Sgjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718737000; x=1719341800; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hc6I5c736RCR/T+6zHjTPji7tCumWOIEIjK9x9cxQIM=; b=APonOjXdQfWoalefxtdkEjT4vrxrwXbmu+ZvKUUqfMGsL52I1ipOqbNGX9ScuncevR 67IyKcu8Mw1T4etQ9Rnxh2n7PtB+Yxx6fH1rVW+kp5P3YuJ5W2b+TqopMP5y1HHlIwnf h0utHl5CemDjkR90dkAe0R1RS7EFJdW2VQy8YuCL6KhSy73Q2eMSW2yWs2BnyxeEi4yD 3RRBNckWoMNqWlFzj0fgu+LBzzPFp5/od/EQnGr/18uIZ1IuVQerRmDnRZt9mhbIEzxM J19S2iU2pYNFkPcnhDtmLeoDUqqSqVt/2a7oKOwxpaYg5E/Cci8tS5JNr2FPN4blwhWw RKkQ== X-Gm-Message-State: AOJu0Yz54OKOixlWXzL7zRwueN6FE1gPMF6Q4C4vOEQg+hX2woAQ2lAz phsD7AnAryqtI+UC8mmIkf46k0AY0hATjERaZp8xZZtJRoFd9WleuvesnwzEg5gu6+N/uyJxbGN C X-Google-Smtp-Source: AGHT+IF6zSZm4r5S6oUXuzRMWrKrVuL8PUPNvQEMBVnAOeqRvYwCPaw2xSGB8lfALfbxGYY4laMD2Q== X-Received: by 2002:a05:6870:148d:b0:258:4dcb:7d48 with SMTP id 586e51a60fabf-25c943e5c01mr809096fac.0.1718736999617; Tue, 18 Jun 2024 11:56:39 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-2567a9f7d6fsm3255492fac.20.2024.06.18.11.56.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Jun 2024 11:56:38 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 2/5] io_uring: add remote task_work execution helper Date: Tue, 18 Jun 2024 12:48:41 -0600 Message-ID: <20240618185631.71781-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240618185631.71781-1-axboe@kernel.dk> References: <20240618185631.71781-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 All our task_work handling is targeted at the state in the io_kiocb itself, which is what it is being used for. However, MSG_RING rolls its own task_work handling, ignoring how that is usually done. In preparation for switching MSG_RING to be able to use the normal task_work handling, add io_req_task_work_add_remote() which allows the caller to pass in the target io_ring_ctx. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 24 ++++++++++++++++-------- io_uring/io_uring.h | 2 ++ 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 438c44ca3abd..85b2ce54328c 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1098,9 +1098,10 @@ void tctx_task_work(struct callback_head *cb) WARN_ON_ONCE(ret); } -static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) +static inline void io_req_local_work_add(struct io_kiocb *req, + struct io_ring_ctx *ctx, + unsigned flags) { - struct io_ring_ctx *ctx = req->ctx; unsigned nr_wait, nr_tw, nr_tw_prev; struct llist_node *head; @@ -1114,6 +1115,8 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) if (req->flags & (REQ_F_LINK | REQ_F_HARDLINK)) flags &= ~IOU_F_TWQ_LAZY_WAKE; + guard(rcu)(); + head = READ_ONCE(ctx->work_llist.first); do { nr_tw_prev = 0; @@ -1195,13 +1198,18 @@ static void io_req_normal_work_add(struct io_kiocb *req) void __io_req_task_work_add(struct io_kiocb *req, unsigned flags) { - if (req->ctx->flags & IORING_SETUP_DEFER_TASKRUN) { - rcu_read_lock(); - io_req_local_work_add(req, flags); - rcu_read_unlock(); - } else { + if (req->ctx->flags & IORING_SETUP_DEFER_TASKRUN) + io_req_local_work_add(req, req->ctx, flags); + else io_req_normal_work_add(req); - } +} + +void io_req_task_work_add_remote(struct io_kiocb *req, struct io_ring_ctx *ctx, + unsigned flags) +{ + if (WARN_ON_ONCE(!(ctx->flags & IORING_SETUP_DEFER_TASKRUN))) + return; + io_req_local_work_add(req, ctx, flags); } static void __cold io_move_task_work_from_local(struct io_ring_ctx *ctx) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index cd43924eed04..7a8641214509 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -73,6 +73,8 @@ struct file *io_file_get_fixed(struct io_kiocb *req, int fd, unsigned issue_flags); void __io_req_task_work_add(struct io_kiocb *req, unsigned flags); +void io_req_task_work_add_remote(struct io_kiocb *req, struct io_ring_ctx *ctx, + unsigned flags); bool io_alloc_async_data(struct io_kiocb *req); void io_req_task_queue(struct io_kiocb *req); void io_req_task_complete(struct io_kiocb *req, struct io_tw_state *ts); From patchwork Tue Jun 18 18:48:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13702851 Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 151E215ECE0 for ; Tue, 18 Jun 2024 18:56:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718737003; cv=none; b=FnKIKEtUnuXDe1AmR/ePc/qb1k1N5PKKiD38ypD2BDxfCiB/2+nSDIE03QILIJZ6/PwtDSIxizN30dqfCltMJBNT63kHZNKnLgUdvIM81kII674OnyCjZTeV7AXCY8zjr8n+gNTvtQp9nU2jbgmKvcVLN5MfVnGmm3/gcLXq124= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718737003; c=relaxed/simple; bh=tRPO2ZimasNp6SbR6mX4GdyVsUhzV0yZwZPDdD//dJ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=c1l6AMxu2MaIHA2UsJIUAAodiI4NVWD74XjbXCj7MU+PTYrJvswK3AG+6eH/8jh0Y8BCoDpFgjzSHtf6Wnx5QOhSnfro46c3FMPh2PmYRqEN7r1GjhFZGAtWETfmRZIp+551o+com8KIQffUCelMjuq39gw9GMYjMMk3kaOYjGs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=0gjfTgis; arc=none smtp.client-ip=209.85.210.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="0gjfTgis" Received: by mail-ot1-f53.google.com with SMTP id 46e09a7af769-6f96064f38cso216386a34.0 for ; Tue, 18 Jun 2024 11:56:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1718737000; x=1719341800; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bDkIUVphJ5Gu6fW81wBdY4aANtGjmFYFCl9rBMHNECY=; b=0gjfTgisSTOLQKnkD0BgSACjOwoPrln3KzmsLlOrTgl/1VgEFWw7ZCYZbD+AKBX9M3 plxG5HgBJ4s/YvcfU7zMgI2ybPxhv5AdfSnpXUlAKAAcnTITeABULvm6L3CNjee/MX+S Kw0o5OYqhZdjz/iMFdY1O8f2Sbb60EomzzHaD8dAJoOsgbo87yoGcPdwas3mRR4wpHnN KyPiMrlxIWsQIjkyGxY5U4ediElEJ26/1YFQgZMPcuI7Dcm38gXFK2b03DjiplaSIKjY B+uQGVrpmw66eUr25wmEBToxip5Yi+HAzRsQjTWZoH7EtR700bzRFMj7Q4YFbP4qnwbl e2zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718737000; x=1719341800; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bDkIUVphJ5Gu6fW81wBdY4aANtGjmFYFCl9rBMHNECY=; b=dyPsEQHqHPTDSRPYlXRcBKAChW5sAyjrwKQtOj429wHrMyz/AWCgBVy/H4PRF0Bk6E oLnUdntDAb6q/BHdbMeXSYs8+aZ8gCw7QFtfN+gXSnuOq3uzVKvljeIHa/rSP9i2/ok2 BBIhKc3dS5aF9qdb0XsoOGRJvsKzIklxeHzDEFv5GmR9LoLEvuxUPq/d2wOJ+fFZ0kFX 2LSCweE3WFirbYAgYacNRIo1DwTdVoh43fTEhjFtEjJR4DfhCiIj0Ggq26I+CYGz3xll gYKSe7ZmBNGubNBs/0WTl400ND1C4jM8TWuf3v0YcbPUxxDVILx4FnFq8lniClzcc+5w sG7g== X-Gm-Message-State: AOJu0Yx5V08snNiBCdnBTqjKXRgqJ7xKdSHoHHIb+TBMWWi2bcu7wfqe 38E2GJaN0+Wrv8Zjil20R5aAQNGR3ymL52xAUaBqTmriXYRS8JcNERBgVhyb6bSh3QwARWSRYup V X-Google-Smtp-Source: AGHT+IEWIPLgrgcTLXUGx/ahHkyz2xQKNEIxjnDJf8kWDkAfT0w3hLSQ+G0nD80up8kA64KgSi/6Ag== X-Received: by 2002:a05:6870:6490:b0:254:7dbe:1b89 with SMTP id 586e51a60fabf-25c948c42c8mr742385fac.1.1718737000461; Tue, 18 Jun 2024 11:56:40 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-2567a9f7d6fsm3255492fac.20.2024.06.18.11.56.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Jun 2024 11:56:40 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 3/5] io_uring: add io_add_aux_cqe() helper Date: Tue, 18 Jun 2024 12:48:42 -0600 Message-ID: <20240618185631.71781-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240618185631.71781-1-axboe@kernel.dk> References: <20240618185631.71781-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This helper will post a CQE, and can be called from task_work where we now that the ctx is already properly locked and that deferred completions will get flushed later on. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 23 +++++++++++++++++++++-- io_uring/io_uring.h | 1 + 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 85b2ce54328c..cdeb94d2a26b 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -801,19 +801,38 @@ static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, return false; } -bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) +static bool __io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, + u32 cflags) { bool filled; - io_cq_lock(ctx); filled = io_fill_cqe_aux(ctx, user_data, res, cflags); if (!filled) filled = io_cqring_event_overflow(ctx, user_data, res, cflags, 0, 0); + return filled; +} + +bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) +{ + bool filled; + + io_cq_lock(ctx); + filled = __io_post_aux_cqe(ctx, user_data, res, cflags); io_cq_unlock_post(ctx); return filled; } +/* + * Must be called from inline task_work so we now a flush will happen later, + * and obviously with ctx->uring_lock held (tw always has that). + */ +void io_add_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) +{ + __io_post_aux_cqe(ctx, user_data, res, cflags); + ctx->submit_state.cq_flush = true; +} + /* * A helper for multishot requests posting additional CQEs. * Should only be used from a task_work including IO_URING_F_MULTISHOT. diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 7a8641214509..e1ce908f0679 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -65,6 +65,7 @@ bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow); int io_run_task_work_sig(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); +void io_add_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); From patchwork Tue Jun 18 18:48:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13702852 Received: from mail-oa1-f44.google.com (mail-oa1-f44.google.com [209.85.160.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47E7E15ECE3 for ; Tue, 18 Jun 2024 18:56:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718737004; cv=none; b=NEFnrdRtCMUvSYijui/pquZQJNTNFAyykfeuJX7f/4h8dlo5lDbwrevM2rG3854+UgVFGLVyKRr/aYGZIhXQtMcBqfZjM7kByHaSOUgZ1TRP7fN7tpLgfW/Lj66CVekH7RXIjndRn4qdiQ4X1HW/wFF/DLsQIWuKnfZnISyMC3c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718737004; c=relaxed/simple; bh=tU23bRJ3dVyD1aJPTclIDtk2SIjAEcnouQm0D7I2J0k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cRfYa9CeINEImUcWsUSuZmw1uYQ4BPUVUXtMO38jhNQIK3MTpPxyJraDIAQlI1EbE2BFQY+JZrbRHPZxHen+af5oJ2jBK1dVnpBRFyMwe4VFhOIcIIIZeJBOUifnqdRAT/yktOWeeiIlB3y/484jwbmH7Nb7fxRa+r7XI1HXHsQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=d7Q545mf; arc=none smtp.client-ip=209.85.160.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="d7Q545mf" Received: by mail-oa1-f44.google.com with SMTP id 586e51a60fabf-25c98395938so7520fac.3 for ; Tue, 18 Jun 2024 11:56:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1718737002; x=1719341802; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BIasNGfmpgFLYp8BWXtQg7WqBx1vlCg1C/16+esEzSY=; b=d7Q545mfZmBN3f6QtF/W/I3OAbu59CofZcKR0T17U/Cc6BO8/FQg4+FQ90hW+66Ocf lGJsTbdIulANA51HyQaYMjngO87G/H2szKkvaFX4Ph6rRAPDc5MTTQTk4Zfil+g+4FYr YOvnAwbaoX+wyVyLzh0G5mjCd3ZsdTCNBOi+jz3cT8MNHYVhS4l5jMzMuTsBxZiFiKYg /dWRNmHNndpF2UmqtygY21nWhJSfKGNsbuDZsXb5qTjtBvryTkCZFI+RqNWpDPsvKqv7 EQJXCjgLTMN8qwRU7ELTjk1q0fQJbNRksIaH/N/oKYWWJEZd3AG38NiJvoimiV4GEEQq ZKiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718737002; x=1719341802; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BIasNGfmpgFLYp8BWXtQg7WqBx1vlCg1C/16+esEzSY=; b=rQJ85X2oDaoIGE2ggYZhQRX6opkZrPkErZuJ/qpsXttdTM1AZWlSoIObD+udATSRWZ uv+uGVT/abOAnJj058rSF9F2fj8Pk8B7mVzDO1ZwasYgd8yQ/wS4Vzao/VyZw4R7HLly UABEdc3imzBUG77tsxuJQX25fW4F8HCxDYzWDyCKueNWbRZNWbwvX/WGJCJmZX32X7YN Z8BMOE7SraLtqMZ0UZ4J3vOIPXofX4K9qp+VicxUfhKKisP9xmBb4lvupswb4vjgEGiy 2GQum7Y4qLN8YOINnLI+lc4PSPiu/TKeuZxLvlT1zoZYOZENdlqJgm0qg+GF0uKf51tk Sa/A== X-Gm-Message-State: AOJu0Yz58FWKYVYi5v0UVJ8GcvE+9pHk6RDcHgUx5BU6S5aUvrTTFPlL k+pnMrnEJ97nJ3cUe+nSEjowne8FryWCiC51qau5JsQj2joHlkQZfN3Bm5hjPVkA+HCNqKOAygG D X-Google-Smtp-Source: AGHT+IEIveQUz5X9WkVpqscuNZzgHgQjow5mb24Qtlnd92BEahMFUNUauWgYvhwqp4b4bnoBrx9eEg== X-Received: by 2002:a05:6870:d1c9:b0:259:8928:85ec with SMTP id 586e51a60fabf-25c9496379bmr751910fac.2.1718737002120; Tue, 18 Jun 2024 11:56:42 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-2567a9f7d6fsm3255492fac.20.2024.06.18.11.56.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Jun 2024 11:56:40 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 4/5] io_uring/msg_ring: improve handling of target CQE posting Date: Tue, 18 Jun 2024 12:48:43 -0600 Message-ID: <20240618185631.71781-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240618185631.71781-1-axboe@kernel.dk> References: <20240618185631.71781-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the exported helper for queueing task_work for message passing, rather than rolling our own. Note that this is only done for strict data messages for now, file descriptor passing messages still rely on the kernel task_work. It could get converted at some point if it's performance critical. This improves peak performance of message passing by about 5x in some basic testing, with 2 threads just sending messages to each other. Before this change, it was capped at around 700K/sec, with the change it's at over 4M/sec. Signed-off-by: Jens Axboe --- io_uring/msg_ring.c | 90 +++++++++++++++++++++++---------------------- 1 file changed, 47 insertions(+), 43 deletions(-) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index 9fdb0cc19bfd..ad7d67d44461 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -13,7 +13,6 @@ #include "filetable.h" #include "msg_ring.h" - /* All valid masks for MSG_RING */ #define IORING_MSG_RING_MASK (IORING_MSG_RING_CQE_SKIP | \ IORING_MSG_RING_FLAGS_PASS) @@ -71,54 +70,43 @@ static inline bool io_msg_need_remote(struct io_ring_ctx *target_ctx) return target_ctx->task_complete; } -static int io_msg_exec_remote(struct io_kiocb *req, task_work_func_t func) +static void io_msg_tw_complete(struct io_kiocb *req, struct io_tw_state *ts) { - struct io_ring_ctx *ctx = req->file->private_data; - struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); - struct task_struct *task = READ_ONCE(ctx->submitter_task); - - if (unlikely(!task)) - return -EOWNERDEAD; + struct io_ring_ctx *ctx = req->ctx; - init_task_work(&msg->tw, func); - if (task_work_add(task, &msg->tw, TWA_SIGNAL)) - return -EOWNERDEAD; + io_add_aux_cqe(ctx, req->cqe.user_data, req->cqe.res, req->cqe.flags); + kmem_cache_free(req_cachep, req); + percpu_ref_put(&ctx->refs); +} - return IOU_ISSUE_SKIP_COMPLETE; +static void io_msg_remote_post(struct io_ring_ctx *ctx, struct io_kiocb *req, + int res, u32 cflags, u64 user_data) +{ + req->cqe.user_data = user_data; + io_req_set_res(req, res, cflags); + percpu_ref_get(&ctx->refs); + req->ctx = ctx; + req->task = READ_ONCE(ctx->submitter_task); + req->io_task_work.func = io_msg_tw_complete; + io_req_task_work_add_remote(req, ctx, IOU_F_TWQ_LAZY_WAKE); } -static void io_msg_tw_complete(struct callback_head *head) +static int io_msg_data_remote(struct io_kiocb *req) { - struct io_msg *msg = container_of(head, struct io_msg, tw); - struct io_kiocb *req = cmd_to_io_kiocb(msg); struct io_ring_ctx *target_ctx = req->file->private_data; - int ret = 0; - - if (current->flags & PF_EXITING) { - ret = -EOWNERDEAD; - } else { - u32 flags = 0; - - if (msg->flags & IORING_MSG_RING_FLAGS_PASS) - flags = msg->cqe_flags; - - /* - * If the target ring is using IOPOLL mode, then we need to be - * holding the uring_lock for posting completions. Other ring - * types rely on the regular completion locking, which is - * handled while posting. - */ - if (target_ctx->flags & IORING_SETUP_IOPOLL) - mutex_lock(&target_ctx->uring_lock); - if (!io_post_aux_cqe(target_ctx, msg->user_data, msg->len, flags)) - ret = -EOVERFLOW; - if (target_ctx->flags & IORING_SETUP_IOPOLL) - mutex_unlock(&target_ctx->uring_lock); - } + struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); + struct io_kiocb *target; + u32 flags = 0; - if (ret < 0) - req_set_fail(req); - io_req_queue_tw_complete(req, ret); + target = kmem_cache_alloc(req_cachep, GFP_KERNEL); + if (unlikely(!target)) + return -ENOMEM; + + if (msg->flags & IORING_MSG_RING_FLAGS_PASS) + flags = msg->cqe_flags; + + io_msg_remote_post(target_ctx, target, msg->len, flags, msg->user_data); + return 0; } static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) @@ -136,7 +124,7 @@ static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) return -EBADFD; if (io_msg_need_remote(target_ctx)) - return io_msg_exec_remote(req, io_msg_tw_complete); + return io_msg_data_remote(req); if (msg->flags & IORING_MSG_RING_FLAGS_PASS) flags = msg->cqe_flags; @@ -216,6 +204,22 @@ static void io_msg_tw_fd_complete(struct callback_head *head) io_req_queue_tw_complete(req, ret); } +static int io_msg_fd_remote(struct io_kiocb *req) +{ + struct io_ring_ctx *ctx = req->file->private_data; + struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); + struct task_struct *task = READ_ONCE(ctx->submitter_task); + + if (unlikely(!task)) + return -EOWNERDEAD; + + init_task_work(&msg->tw, io_msg_tw_fd_complete); + if (task_work_add(task, &msg->tw, TWA_SIGNAL)) + return -EOWNERDEAD; + + return IOU_ISSUE_SKIP_COMPLETE; +} + static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) { struct io_ring_ctx *target_ctx = req->file->private_data; @@ -238,7 +242,7 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) } if (io_msg_need_remote(target_ctx)) - return io_msg_exec_remote(req, io_msg_tw_fd_complete); + return io_msg_fd_remote(req); return io_msg_install_complete(req, issue_flags); } From patchwork Tue Jun 18 18:48:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13702853 Received: from mail-oa1-f50.google.com (mail-oa1-f50.google.com [209.85.160.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9FB615ECDE for ; Tue, 18 Jun 2024 18:56:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718737006; cv=none; b=sTahrzpUQL3ixfTFLFTOxcdV5aD3BWUC1V80kv5ZiWDh7nUgiSjhaqiTuEqhurQHkrVlpw/+DF0Ur7ZoUCOZHTf1mxLyjxvpVohhGAgrYH0f5+MV84B/RVm0X2MMpJcBL5sE1l00ki6WC7NniqhrgMlAuNQkIxCmFVo2TvrinZA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718737006; c=relaxed/simple; bh=WEDhBoYN5pLLdRV/+qAT7wA3ka28+zo2raRt0QVG6Hk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LXWqi3CA4kyI7zgAz9V6ppyDXd1SOVzmv4xTp7/JjAw360UrN9BgATxruY+i9kf45n2hoSf17Sp/qRxQ5eAOm10o9BjPCO2ezAhNQhAu9ALhmwfLrf8dZ2O8oXyJwrH4m8fJfAMn4Djmd2uxG8xS8b4vY4Tay6cEtZZ0zD+waDE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=VbNfz30o; arc=none smtp.client-ip=209.85.160.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="VbNfz30o" Received: by mail-oa1-f50.google.com with SMTP id 586e51a60fabf-25075f3f472so798356fac.2 for ; Tue, 18 Jun 2024 11:56:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1718737003; x=1719341803; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PXJyIJemjNC4yU1UV6G6l8eOfwcbX9iQfdK4+39fx6Q=; b=VbNfz30oSgYKD42AtwOkWs5kthv80Clss3q+sjpb9HZWLo8CkwxFMtwjJybZCoEpqb qIQ9UrG/OFaFWWhc4jdmThajddqJ2w71IEt7vcSrcwFuE1wZvwmCPSoDsUcsajIe0Qkx ejD6EQg7zyQnX0YvTcscchTR8l6xDOfdQjXWmM/XWOUXfmW5svDmR1jsmOFomR+vlGtI 7fneCcIwpG+x8B7zOzJchFhe5GHhoJHWVnW2TK8kQjY7RV9vHmzmsIUBUZfX8qYHvh2f logwZ3H2uY3apwBfpmwWS2qReAubD/iq0qAmmqRIQ0yYbJJt0J0ZOlQJkrioI3LwHWrb SShw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718737003; x=1719341803; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PXJyIJemjNC4yU1UV6G6l8eOfwcbX9iQfdK4+39fx6Q=; b=RZIUiznX9TBj2Ag/LYmO5YaMKFfl7rKmPd4ja3KtMW33LSpUK3VS/SMlIhjp1BhpP6 H3ivF07DriL5pwp6lgO1gSCIdbuFjzAKvr8L795yJnalPT+jHdRPS4KP3ouR3xg6wiHs ZcP0P7gxQAYpKhEbAAhFiXOJNbLwWMT8PQp1BFpRXaxjiffl9RYTqPgbZTAjXDrxQeD0 vghbPOZgdVNbzSzuHTv5suKt8P7BwcyLlEXMiGBJCUS93MDKo7ePvNf9qdsdR7ttxStx xHLrVAeR1BodI5qCBpPRVUpA4Bv3wPadCjqV/ZJOkLZ4oNO1LJOnhFgGXWvQXU5X808D OLnw== X-Gm-Message-State: AOJu0Yzma+dzfn5/nQAM4gtv+IEncBwrmvKOmRRLz2D9VzQTb0BYqb2g reYIQaDk9uGjK2RzGFb4siqthjvTF/H4kcrYx3wn/XIB4lnJd5TugDvl7eypF6aJxj8wh15H7bF p X-Google-Smtp-Source: AGHT+IFnG9Qpqarvqyd8b59T09u0HxIYK1LWUKhFkxwOXUp6jee4erPuKg8LqqewlNFmZNGF0Jbo9w== X-Received: by 2002:a05:6870:a2cc:b0:254:affe:5a08 with SMTP id 586e51a60fabf-25c94983780mr709673fac.2.1718737003295; Tue, 18 Jun 2024 11:56:43 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-2567a9f7d6fsm3255492fac.20.2024.06.18.11.56.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Jun 2024 11:56:42 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 5/5] io_uring/msg_ring: add an alloc cache for io_kiocb entries Date: Tue, 18 Jun 2024 12:48:44 -0600 Message-ID: <20240618185631.71781-6-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240618185631.71781-1-axboe@kernel.dk> References: <20240618185631.71781-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 With slab accounting, allocating and freeing memory has considerable overhead. Add a basic alloc cache for the io_kiocb allocations that msg_ring needs to do. Unlike other caches, this one is used by the sender, grabbing it from the remote ring. When the remote ring gets the posted completion, it'll free it locally. Hence it is separately locked, using ctx->msg_lock. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 3 +++ io_uring/io_uring.c | 6 ++++++ io_uring/msg_ring.c | 31 +++++++++++++++++++++++++++++-- io_uring/msg_ring.h | 1 + 4 files changed, 39 insertions(+), 2 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 1052a68fd68d..ede42dce1506 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -397,6 +397,9 @@ struct io_ring_ctx { struct callback_head poll_wq_task_work; struct list_head defer_list; + struct io_alloc_cache msg_cache; + spinlock_t msg_lock; + #ifdef CONFIG_NET_RX_BUSY_POLL struct list_head napi_list; /* track busy poll napi_id */ spinlock_t napi_lock; /* napi_list lock */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cdeb94d2a26b..7ed1e009aaec 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -95,6 +95,7 @@ #include "futex.h" #include "napi.h" #include "uring_cmd.h" +#include "msg_ring.h" #include "memmap.h" #include "timeout.h" @@ -315,6 +316,9 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) sizeof(struct io_async_rw)); ret |= io_alloc_cache_init(&ctx->uring_cache, IO_ALLOC_CACHE_MAX, sizeof(struct uring_cache)); + spin_lock_init(&ctx->msg_lock); + ret |= io_alloc_cache_init(&ctx->msg_cache, IO_ALLOC_CACHE_MAX, + sizeof(struct io_kiocb)); ret |= io_futex_cache_init(ctx); if (ret) goto err; @@ -351,6 +355,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) io_alloc_cache_free(&ctx->netmsg_cache, io_netmsg_cache_free); io_alloc_cache_free(&ctx->rw_cache, io_rw_cache_free); io_alloc_cache_free(&ctx->uring_cache, kfree); + io_alloc_cache_free(&ctx->msg_cache, io_msg_cache_free); io_futex_cache_free(ctx); kfree(ctx->cancel_table.hbs); kfree(ctx->cancel_table_locked.hbs); @@ -2599,6 +2604,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_alloc_cache_free(&ctx->netmsg_cache, io_netmsg_cache_free); io_alloc_cache_free(&ctx->rw_cache, io_rw_cache_free); io_alloc_cache_free(&ctx->uring_cache, kfree); + io_alloc_cache_free(&ctx->msg_cache, io_msg_cache_free); io_futex_cache_free(ctx); io_destroy_buffers(ctx); mutex_unlock(&ctx->uring_lock); diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index ad7d67d44461..47a754e83b49 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -11,6 +11,7 @@ #include "io_uring.h" #include "rsrc.h" #include "filetable.h" +#include "alloc_cache.h" #include "msg_ring.h" /* All valid masks for MSG_RING */ @@ -75,7 +76,13 @@ static void io_msg_tw_complete(struct io_kiocb *req, struct io_tw_state *ts) struct io_ring_ctx *ctx = req->ctx; io_add_aux_cqe(ctx, req->cqe.user_data, req->cqe.res, req->cqe.flags); - kmem_cache_free(req_cachep, req); + if (spin_trylock(&ctx->msg_lock)) { + if (io_alloc_cache_put(&ctx->msg_cache, req)) + req = NULL; + spin_unlock(&ctx->msg_lock); + } + if (req) + kfree(req); percpu_ref_put(&ctx->refs); } @@ -91,6 +98,19 @@ static void io_msg_remote_post(struct io_ring_ctx *ctx, struct io_kiocb *req, io_req_task_work_add_remote(req, ctx, IOU_F_TWQ_LAZY_WAKE); } +static struct io_kiocb *io_msg_get_kiocb(struct io_ring_ctx *ctx) +{ + struct io_kiocb *req = NULL; + + if (spin_trylock(&ctx->msg_lock)) { + req = io_alloc_cache_get(&ctx->msg_cache); + spin_unlock(&ctx->msg_lock); + } + if (req) + return req; + return kmem_cache_alloc(req_cachep, GFP_KERNEL | __GFP_NOWARN); +} + static int io_msg_data_remote(struct io_kiocb *req) { struct io_ring_ctx *target_ctx = req->file->private_data; @@ -98,7 +118,7 @@ static int io_msg_data_remote(struct io_kiocb *req) struct io_kiocb *target; u32 flags = 0; - target = kmem_cache_alloc(req_cachep, GFP_KERNEL); + target = io_msg_get_kiocb(req->ctx); if (unlikely(!target)) return -ENOMEM; @@ -296,3 +316,10 @@ int io_msg_ring(struct io_kiocb *req, unsigned int issue_flags) io_req_set_res(req, ret, 0); return IOU_OK; } + +void io_msg_cache_free(const void *entry) +{ + struct io_kiocb *req = (struct io_kiocb *) entry; + + kmem_cache_free(req_cachep, req); +} diff --git a/io_uring/msg_ring.h b/io_uring/msg_ring.h index 3987ee6c0e5f..3030f3942f0f 100644 --- a/io_uring/msg_ring.h +++ b/io_uring/msg_ring.h @@ -3,3 +3,4 @@ int io_msg_ring_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_msg_ring(struct io_kiocb *req, unsigned int issue_flags); void io_msg_ring_cleanup(struct io_kiocb *req); +void io_msg_cache_free(const void *entry);