From patchwork Mon Oct 14 20:49:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13835472 Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED8AD1D14EE for ; Mon, 14 Oct 2024 20:54:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728939266; cv=none; b=dtBfJ6M8iLpA+KPnQtP+/5axfXI50j7z1GX6DDWCM4kV7tyLEgilC+DcPYiM+jfSbkr9ImUI74P69kR11DGQ0DTdoWJgIx3TdOInSB66K1b/GA8ArXtgGT7Icqb3UiGXXZiycn9tk/kqPNLMpt4XIJCpbSUynBZhX+/8Jj4YBG8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728939266; c=relaxed/simple; bh=yiv1mUHkC+sXc2a72PzGwHO43QZ7YSGFs7GvNRT8g0g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qA55lEXvC72xrtGTNILGS6DpT3dSQF6DukGDcn4OdAxEIG+hyWbliBHjaRkbA5WNJ8V5MkGs74WqGV2orUXgdBnTSjpVz1s9+Yy2KOfldvdd14djbQCf7GOUaSQcwoyUHS2boHklHc8ptxBmzW69elZuyDRJ+0DFU8zF+WaSTLU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=W++ycGZ/; arc=none smtp.client-ip=209.85.166.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="W++ycGZ/" Received: by mail-io1-f41.google.com with SMTP id ca18e2360f4ac-8354b8df4c9so167338939f.1 for ; Mon, 14 Oct 2024 13:54:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1728939263; x=1729544063; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5slwh/hrSipEcpmMDPFHvZDFXrV7QCLK/2Ol7f9mbnQ=; b=W++ycGZ/uxRR9BQdTLlBOyoiICMRXTxcGPw6IJeOnISIjqBuyJKvwHxLXP6Ib/RS6j TbP2PGk9qU00KvogTPC3EDPSzyFQylW6da6vr1FTgKYcXsy2NS6xP46BZdLeL4B/Y1UQ 4pAKzcLQOAb+c3q1ITOjjpuFB6EdPpk4WIwxB2ROW1tpZpO90NxWFJobS8EAIuCnAkEf AlizcQ3ZK2ZaFQJPHdSWvXZGUsfIy7seoCO5neqtaPy6cgrThI9Hk1QZ6PKq2m5TE5IP AF59wyrQ9evblGysH5TNn/ZqwURDx0WNMLtgDfg8opZt/rxsoAsbCH6cpVs8EfAKAYaw i3Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728939263; x=1729544063; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5slwh/hrSipEcpmMDPFHvZDFXrV7QCLK/2Ol7f9mbnQ=; b=UTx8yjD+j9/kcMKX+IA0KI0MYSIJJTj4w0ikYVGiSaIcJ6hj3rwD+1/M4itSdTOqbC DKvMGcUMD1e5EnreMQNZyxx7M4K1n6Nxvg+fkAXK1tRaKcm3N7wZkX9KFT9ftw3J54mi zGaKWfPPcfHVhw56n+ywRGencH/DM10DftxAbulInxu3QMlOZ7gjC4yF3ZDm1ub63xnY TdW+vu14UixrER+ZHYU46zc8VLqd5qWvTRtMJV1nAKHk5DNBt5ioe0XX/VL05HijW0gH g+7ELX0nk80Yyp7GZUaXJ3n4tTDgN4KnzHizqldK8NnKpqI58gWQs+5ckv7+hcDKbn56 6uIQ== X-Gm-Message-State: AOJu0Yy8qwCDDQf2lxTdyl+tOpVBUgGsmuVMjxhW3cMsvrXoziwv9a01 I/GAc/PvMAfvtuTAlU942WdZ9cCsXITj4gfsVcWYUM9yxjJFW0lSS2vL2xQdYdqwDM/7spwLUl8 F X-Google-Smtp-Source: AGHT+IFSuc7YQf3Aa0fzWX5GyVIUgLw3a7mAA3Ta0OWWuFdkcoYUhMW4t12Q/zL2113qDt9hGtf1/w== X-Received: by 2002:a05:6e02:19cb:b0:3a2:463f:fd9e with SMTP id e9e14a558f8ab-3a3bcdbb642mr73411195ab.6.1728939262395; Mon, 14 Oct 2024 13:54:22 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a3afdb3629sm62644895ab.21.2024.10.14.13.54.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Oct 2024 13:54:21 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/3] io_uring: add support for ignoring inline completions for waits Date: Mon, 14 Oct 2024 14:49:46 -0600 Message-ID: <20241014205416.456078-2-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241014205416.456078-1-axboe@kernel.dk> References: <20241014205416.456078-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring treats all completions the same - they post a completion event, or more, and anyone waiting on event completions will see each event as it gets posted. However, some events may be more interesting that others. For a request and response type model, it's not uncommon to have send/write events that are submitted with a recv/read type of request. While the app does want to see a successful send/write completion eventually, it need not handle it upfront as it would want to do with a recv/read, as it isn't time sensitive. Generally, a send/write completion will just mean that a buffer can get recycled/reused, whereas a recv/read completion needs acting upon (and a response sent). This can be somewhat tricky to handle if many requests and responses are being handled, and the app generally needs to track the number of pending sends/writes to be able to sanely wait on just new incoming recv/read requests. And even with that, an application would still like to see a completion for a short/failed send/write immediately. Add infrastructure to account inline completions, such that they can be deducted from the 'wait_nr' being passed in via a submit_and_wait() type of situation. Inline completions are ones that complete directly inline from submission, such as a send to a socket where there's enough space to accomodate the data being sent. No functional changes in this patch, as no opcode supports setting REQ_F_IGNORE_INLINE just yet. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 4 ++++ include/uapi/linux/io_uring.h | 1 + io_uring/io_uring.c | 12 +++++++++--- io_uring/io_uring.h | 2 ++ 4 files changed, 16 insertions(+), 3 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 9c7e1d3f06e5..6eb8b739ea0d 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -206,6 +206,7 @@ struct io_submit_state { bool need_plug; bool cq_flush; unsigned short submit_nr; + unsigned short inline_completions; struct blk_plug plug; }; @@ -465,6 +466,7 @@ enum { REQ_F_BL_EMPTY_BIT, REQ_F_BL_NO_RECYCLE_BIT, REQ_F_BUFFERS_COMMIT_BIT, + REQ_F_IGNORE_INLINE_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -541,6 +543,8 @@ enum { REQ_F_BL_NO_RECYCLE = IO_REQ_FLAG(REQ_F_BL_NO_RECYCLE_BIT), /* buffer ring head needs incrementing on put */ REQ_F_BUFFERS_COMMIT = IO_REQ_FLAG(REQ_F_BUFFERS_COMMIT_BIT), + /* if set, ignore these completions for when waiting on events */ + REQ_F_IGNORE_INLINE = IO_REQ_FLAG(REQ_F_IGNORE_INLINE_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 86cb385fe0b5..1967f5ab2317 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -554,6 +554,7 @@ struct io_uring_params { #define IORING_FEAT_REG_REG_RING (1U << 13) #define IORING_FEAT_RECVSEND_BUNDLE (1U << 14) #define IORING_FEAT_MIN_TIMEOUT (1U << 15) +#define IORING_FEAT_IGNORE_INLINE (1U << 16) /* * io_uring_register(2) opcodes and arguments diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d7ad4ea5f40b..706822db7447 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2239,6 +2239,7 @@ static void io_submit_state_start(struct io_submit_state *state, state->plug_started = false; state->need_plug = max_ios > 2; state->submit_nr = max_ios; + state->inline_completions = 0; /* set only head, no need to init link_last in advance */ state->link.head = NULL; } @@ -3285,6 +3286,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, size_t, argsz) { struct io_ring_ctx *ctx; + int inline_complete = 0; struct file *file; long ret; @@ -3349,6 +3351,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, mutex_unlock(&ctx->uring_lock); goto out; } + inline_complete = ctx->submit_state.inline_completions; if (flags & IORING_ENTER_GETEVENTS) { if (ctx->syscall_iopoll) goto iopoll_locked; @@ -3386,8 +3389,10 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, ret2 = io_get_ext_arg(flags, argp, &ext_arg); if (likely(!ret2)) { - min_complete = min(min_complete, - ctx->cq_entries); + if (min_complete > ctx->cq_entries) + min_complete = ctx->cq_entries; + else + min_complete += inline_complete; ret2 = io_cqring_wait(ctx, min_complete, flags, &ext_arg); } @@ -3674,7 +3679,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, IORING_FEAT_EXT_ARG | IORING_FEAT_NATIVE_WORKERS | IORING_FEAT_RSRC_TAGS | IORING_FEAT_CQE_SKIP | IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING | - IORING_FEAT_RECVSEND_BUNDLE | IORING_FEAT_MIN_TIMEOUT; + IORING_FEAT_RECVSEND_BUNDLE | IORING_FEAT_MIN_TIMEOUT | + IORING_FEAT_IGNORE_INLINE; if (copy_to_user(params, p, sizeof(*p))) { ret = -EFAULT; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 9d70b2cf7b1e..bd1d4b6e46f0 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -357,6 +357,8 @@ static inline void io_req_complete_defer(struct io_kiocb *req) lockdep_assert_held(&req->ctx->uring_lock); wq_list_add_tail(&req->comp_list, &state->compl_reqs); + if (req->flags & REQ_F_IGNORE_INLINE) + state->inline_completions++; } static inline void io_commit_cqring_flush(struct io_ring_ctx *ctx) From patchwork Mon Oct 14 20:49:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13835473 Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EFC051CDFD2 for ; Mon, 14 Oct 2024 20:54:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728939267; cv=none; b=PSsBioSiC5AJWpSKvGpDzBS1xSCtPb2Z63xfH8/refCu9zZqCug1fOwsPLqp8ia5dea11zZo+cyBBL/zY8/C0XT7tZzJdlqH99EpNrt/sRxgBklJqCdJEXaPFY+6WllcCuA/qjK0vH5hYChQG8HW7do/gvm1XsD1WfXPUxumW60= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728939267; c=relaxed/simple; bh=X6yM1ysWrwvLR/A4bHLQWE0RVyfG/maA7GFnPoGJlKU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sKaG2WQXIgsPsweNSwV+Gw8gftCFydQbD8EtZKDvUF2HPkKBlw55gP3xvYQ24P+OdD/DtfjEnJJGkZ9q6QbVKWNM+Y1nxOw2X6yA1md0q2DYUhRHgvVJV5eNVfONY2odjfYgWYGNopHyn2ztyV62CNqNcloTNBF3Vys7VuvnPWs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=zlolkWwF; arc=none smtp.client-ip=209.85.166.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="zlolkWwF" Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-835496c8cefso281402239f.0 for ; Mon, 14 Oct 2024 13:54:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1728939264; x=1729544064; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jbP3JkI7Mm5xzwedpdCbAEUYoYaLTWmGdma8DF8byKk=; b=zlolkWwFawliHk9YdzeklQSvtXG6hdNzE7iGx8TihbQwqjyOWzN+6/nJlfhSy/YRnI uWUeqcvRnh1TU+xfWxMSNxFpcKg6J+BQHmN11UU1L7c9DOo22IXQ3iEiD1B/stvBfpuc UphsB7plQXTWwpQNsCsJ4EDM2CiLw8oOE7um/RzyO165zY2Bx9V/q1/zYOLWqRzBlaEX QWo2nSjuPb96t5qm8++KpnQG63N4eHLEhQfkhpZU4FVoFW7agxxPVzqtArqr9kNZClj/ 2jXWMJClN3oEW+SdBlkdfu8pPx1mCSa5VyhEA54mTnjXqBVImt2ZJckUk7Uxawq8uB30 /euw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728939264; x=1729544064; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jbP3JkI7Mm5xzwedpdCbAEUYoYaLTWmGdma8DF8byKk=; b=Zev1SDoxw8vxu9/pYvXm0KqvoNktssJVZmZnBxASBJ9alQfSEpljbxGNGgwk/Zo9ZD KsJiQNviFYbjs7dHX0za6BfOwUd42RABghpKj1T/RheT407cRB5T9pUhX/obcw2z6Gc5 Dju05yRAairiOpoF1CJvNN4dXgFz9O59+8wXWTjJw+gWO4PUdxHpVZ2lEKmOB1olZ2T4 MFx3DVi17xe1LxNGYpjx2/R66MzZ3iZsCmnRwIf0jiG/rf/GGvTgLN0hA1qwIq5IDouw Blv0RvjwKPh/Qglme/cyKHe9+J+sipOrgJObURSy6pg++0om9GrO8NnTC+bdfUPIZE7r /JhQ== X-Gm-Message-State: AOJu0YwCPIT3IpggPkpcDrmYXld1WM+fTKu8+ZZF3vwaPHlwFCG+6FuB e1TqNeswAGB7K88cxg0LHV6l5PMF2Hca51FtVHYNzOBYzewR/lNTJKVVV6e8jbCKV7gOZcw2NyG b X-Google-Smtp-Source: AGHT+IEewsMAewlw5yNX7Jrz3K7W1i9RSTAJy5C8cKoSmPx1CKig4Ntos3WVnFMZKt/bMWpPMq47ww== X-Received: by 2002:a05:6e02:1a23:b0:39f:5e12:1dde with SMTP id e9e14a558f8ab-3a3b60516b2mr87456075ab.21.1728939264526; Mon, 14 Oct 2024 13:54:24 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a3afdb3629sm62644895ab.21.2024.10.14.13.54.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Oct 2024 13:54:22 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/3] io_uring/net: add IORING_SEND_IGNORE_INLINE support to send/sendmsg Date: Mon, 14 Oct 2024 14:49:47 -0600 Message-ID: <20241014205416.456078-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241014205416.456078-1-axboe@kernel.dk> References: <20241014205416.456078-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If IORING_SEND_IGNORE_INLINE is set for a send/sendmsg request, then a successful inline completion of such a request will be ignored for a submit_and_wait() type of submissions. In other words, if an application submits a send for socketA with a recv for socketB, it can now do: io_uring_submit_and_wait(ring, 1); and have the inline send completion be ignored from the number of items to wait for. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 7 +++++++ io_uring/net.c | 9 ++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 1967f5ab2317..e026ade027c1 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -362,12 +362,19 @@ enum io_uring_op { * the starting buffer ID in cqe->flags as per * usual for provided buffer usage. The buffers * will be contigious from the starting buffer ID. + * + * IORING_SEND_IGNORE_INLINE If set for a send[msg] request, then the + * completion will NOT be included in the count + * waited for by an application, if completed + * inline as part of submission. It will still + * generate a CQE. */ #define IORING_RECVSEND_POLL_FIRST (1U << 0) #define IORING_RECV_MULTISHOT (1U << 1) #define IORING_RECVSEND_FIXED_BUF (1U << 2) #define IORING_SEND_ZC_REPORT_USAGE (1U << 3) #define IORING_RECVSEND_BUNDLE (1U << 4) +#define IORING_SEND_IGNORE_INLINE (1U << 5) /* * cqe.res for IORING_CQE_F_NOTIF if diff --git a/io_uring/net.c b/io_uring/net.c index 18507658a921..11ff58a5c145 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -400,7 +400,8 @@ static int io_sendmsg_prep_setup(struct io_kiocb *req, int is_msg) return ret; } -#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE) +#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE | \ + IORING_SEND_IGNORE_INLINE) int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { @@ -434,6 +435,8 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->buf_group = req->buf_index; req->buf_list = NULL; } + if (sr->flags & IORING_SEND_IGNORE_INLINE) + req->flags |= REQ_F_IGNORE_INLINE; #ifdef CONFIG_COMPAT if (req->ctx->compat) @@ -550,6 +553,8 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) ret = __sys_sendmsg_sock(sock, &kmsg->msg, flags); if (ret < min_ret) { + req->flags &= ~REQ_F_IGNORE_INLINE; + if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) return -EAGAIN; if (ret > 0 && io_net_retry(sock, flags)) { @@ -647,6 +652,8 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) kmsg->msg.msg_flags = flags; ret = sock_sendmsg(sock, &kmsg->msg); if (ret < min_ret) { + req->flags &= ~REQ_F_IGNORE_INLINE; + if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) return -EAGAIN; From patchwork Mon Oct 14 20:49:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13835474 Received: from mail-il1-f178.google.com (mail-il1-f178.google.com [209.85.166.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA94A1D07B8 for ; Mon, 14 Oct 2024 20:54:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728939270; cv=none; b=kIe1296s0xn9yeDmGlRybt/LleXyNo/dR2+t82k+MqNCbOmfRIpn5uu7slPlJFOwtivZd4TXHQP5IIsnXwxRCB9DQ7ZUu5mnwF0JpNK3irsne2/sS0LOtPoI6wxtAByPbjqQAewOJ1DutOVhr7GkmyWGp5JW7tJ+Ts/OgrFJAD4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728939270; c=relaxed/simple; bh=ODSbDyCmnWf2+n/n5tqJDeqWl2buYBZm5bsrlsObWwI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qOGnwI0s9ZpgJhu4GQNnLQlNSj+k1a4w01EcUIGP14KxdGRfQxCHKp2chbkmZ/VP/ZV2Vd2BERe2DzbKZkC4xAAx7nh7HtwsnD5yWEGwZ4hVJ9zldu0skxsxabzowuNU4jRoJSbxUZuutJwFw2k29rcf/WVPyjIZBqL/+DI9wEI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=mDs5rEHC; arc=none smtp.client-ip=209.85.166.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="mDs5rEHC" Received: by mail-il1-f178.google.com with SMTP id e9e14a558f8ab-3a3a5cd2a3bso26693105ab.3 for ; Mon, 14 Oct 2024 13:54:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1728939266; x=1729544066; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EusEAECScs7P5s7DtUS2E5FvEVP9+InNt62nDGrPub0=; b=mDs5rEHCcjb3F929VjOuvhysXqYcHU8F171g0ThcZcMR+yz6rRSFA30rciIiK4Yhhy TnDvzhsm9sWHlOSO7ilqHcFk89zlLzmaLZE1DBbrZGU1+/3n6iE7MNBXW+iLoUqV8ani 3SEoFWBCFwFBY14Wb4PwyvaR0nSXtcLfvEEPpMcrk7Pc9Aw5HfHpSTc15zgvzu6ZH1+M m4CHEjYd+uhwfawHfnYDYFQ/oXebn4swJmDFkrGzFfM5eibZXBicc8ZaCDM2dlWcgusl MI5I1PScAN+LFXwAhDtNsGX3AJIWBsWzdCCF748eOkXCK3UR0lm2x05ZE8qzdAWjnVH3 wobQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728939266; x=1729544066; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EusEAECScs7P5s7DtUS2E5FvEVP9+InNt62nDGrPub0=; b=dNpHlpPWEFVkOjiWmua1qWijqgC6XSI6BjLLwbFrObHrM8w0k/yTtKxUa5h71BuNSr iAVQav4uesCAdXLduEhbBD78lwpKaqAxPjWjxwyH94a8ERnMr0ARyX+3qEkIuRlJ0agj 4CVHSuCkZDXNXCevVbySj8Tbc272+JPykbwrMe+7EFhpOClp/bSwWPgww+ZYCqgJFLdu //0qoyRe46x6FUCGMK0GRpFaOdE/ZiLUH8Rzi5X+3Bd6Du6wL+89/mI3FiA0eatIYHsR KU5LZg3BAWB1y4fIiTXTTU+VhndyVFxRYbTQzAcjaXHvFTxKZizKAtAEE4krbtoNDubf utSw== X-Gm-Message-State: AOJu0YwGpAIqV/rullmKmc/6JRAHUBMZNCP/LYSNQNjE9Bhq+ggY7j4h 0c0VGk0GAYkWMmtvhNraaR/675+3AQfAyleCFWS+fzyI6nxO9lSftiboEeW1AIAuMZqanO7bFVL G X-Google-Smtp-Source: AGHT+IH3Nesm57A05j40aHouwcgTf3EsWBk/ILh63b2xj9GgyOif5rjhnLlHGVucFjzugDtz+/GIXw== X-Received: by 2002:a92:c544:0:b0:3a0:4e2b:9ab9 with SMTP id e9e14a558f8ab-3a3b5f7a9b9mr100211895ab.5.1728939266509; Mon, 14 Oct 2024 13:54:26 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a3afdb3629sm62644895ab.21.2024.10.14.13.54.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Oct 2024 13:54:24 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/3] io_uring/net: add IORING_SEND_IGNORE_INLINE support to zerocopy send Date: Mon, 14 Oct 2024 14:49:48 -0600 Message-ID: <20241014205416.456078-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241014205416.456078-1-axboe@kernel.dk> References: <20241014205416.456078-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If IORING_SEND_IGNORE_INLINE is set for a send zerocopy request, then a successful inline completion of such a request will be ignored for a submit_and_wait() type of submissions. In other words, if an application submits a send for socketA with a recv for socketB, it can now do: io_uring_submit_and_wait(ring, 1); and have the inline send completion be ignored from the number of items to wait for. Note that this only applies to the direct zerocopy send completion, it does not include the notification when it's safe to reuse the buffer. Those happen out-of-line anyway. Signed-off-by: Jens Axboe --- io_uring/net.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/io_uring/net.c b/io_uring/net.c index 11ff58a5c145..79f980182a10 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1223,7 +1223,8 @@ void io_send_zc_cleanup(struct io_kiocb *req) } #define IO_ZC_FLAGS_COMMON (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_FIXED_BUF) -#define IO_ZC_FLAGS_VALID (IO_ZC_FLAGS_COMMON | IORING_SEND_ZC_REPORT_USAGE) +#define IO_ZC_FLAGS_VALID (IO_ZC_FLAGS_COMMON | IORING_SEND_ZC_REPORT_USAGE | \ + IORING_SEND_IGNORE_INLINE) int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { @@ -1259,6 +1260,8 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) nd->zc_used = false; nd->zc_copied = false; } + if (zc->flags & IORING_SEND_IGNORE_INLINE) + req->flags |= REQ_F_IGNORE_INLINE; } if (zc->flags & IORING_RECVSEND_FIXED_BUF) { @@ -1406,6 +1409,8 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) ret = sock_sendmsg(sock, &kmsg->msg); if (unlikely(ret < min_ret)) { + req->flags &= ~REQ_F_IGNORE_INLINE; + if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) return -EAGAIN;