From patchwork Fri Apr 12 12:55:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13627720 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 466F45A0F1; Fri, 12 Apr 2024 12:55:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926539; cv=none; b=DJ4HK0vupYlR1SlmYRRdmZkh0KeX5YcYK0nP4fpCLKuiiVzmPXepNPe/abHi8RWWP4LdGw5JRgUmvhxjx0mFO5kI5F1eGDNcZKNDG6cphG1sBF7AQG/gLLORl5zOgTkNIIBI+g5GEXc8ASoKYWRSGAvsly/EzPd7MaEvBa5IGdY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926539; c=relaxed/simple; bh=+J6ZIjWMtQpY2a/JESoUxz7LG42hzE5YzesreEyeq9c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HNIFfBQ5XMd91qXAGK0/oSfcIRe1q/x5uiMk15wbIp5D8cDQ2wlLsFI2wiXJ59ibWzHzmFwe/3j/2d0tjlIJaYe4tIpa9qIZ26xd4O5EhNbSgxylRg99KB2/D+pUlQCJRGROmBCdhVCs+6WSV0GEOTylTbA0TNALIfx6gvtSTGs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lmcVR+0H; arc=none smtp.client-ip=209.85.218.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lmcVR+0H" Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a4715991c32so109284566b.1; Fri, 12 Apr 2024 05:55:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712926535; x=1713531335; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W+ZC7kUGHtaoxOSEWHEf09FVIWFyHEq4CFf55bkSPKQ=; b=lmcVR+0HOPdlp13B+3s98sd4FGuoRXXuBGDgxFLIzKnoQq0Xi280pKAebG/pUKiMGC 5rwqIvOYUjAgenKLU40cit1COZ3CAh1/leQzuQuGjBC51uTUolxgZ8uenF4QOx8T1jyO C33zuYmBZlQZImwHNmmVXfCfvxVvio8bIInQ6lHN31ibjBw4770mDaEcak1g1Du0Zc8+ l/1dbbu8JqEyM6Q5AzsaDyf+Xval+lXSnA3J+peQ+nRWIr8WY9MWfAG4Uz5cT2oWQRt0 d514H9iNpJWXJE97nBUA8RKuqkyrg3G3xaYqo+EFKpTE5hfUzKOMCtUlt3V2sxfuY/RT xMug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712926535; x=1713531335; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W+ZC7kUGHtaoxOSEWHEf09FVIWFyHEq4CFf55bkSPKQ=; b=VuD6/WQjkV7ESm4vcljCB+WD/NAbJuJLJ+EB5dw2ZvCQfWpuMxRTqM8l2qylO7RfAs /tkwk5jT7AqwiRJjRxcWLL++b4kOThWXcSfkmRshwIX7VilU57zIbvLVbeSRiP673bkU HuwtRUDAQgk6+CqIWAsOxWR5EB3tNfbGl3KstpxC6j5hYHSqC6L/USEU3X2p4H18ye/P /zJL2an01Pzj/xLkzW0p18QaB0SdfE92eLCBGKPIxPaBc2JjRmWwrED/nwoZxRdkjKTN +BZa1SOy2LM9ip7fIFVrvTZRBtMRHwmU7CPQWHUM7c0LfVphcKvjsr5OQu69qRAvysSZ Ddaw== X-Forwarded-Encrypted: i=1; AJvYcCV6xcLbLc01s+JFxlRouxbDyBSaCTEICTNXFb8hKembG0wC/+rnX7/AzC7MfD+dn/8vji7ffgFuQua4sutVGHMcaXujGzpZ X-Gm-Message-State: AOJu0YzCpMDMPMaoPKIlp4kR5qaaS5WOZfzi/z997Oq5NKNo5GNu3fXU pcjVk8sVwwE+qGqkWjmX/T/HU8fSrOyIAAjt/QisInrLC2Tclmi0rZw0vA== X-Google-Smtp-Source: AGHT+IGic95rkODjnCp3+iDkZNikLHY1laWxB/nSxwKlIaRWZKcnyrG/ntgHs0sWENYiUz5IuWjWeA== X-Received: by 2002:a17:906:d20d:b0:a51:ad60:ea32 with SMTP id w13-20020a170906d20d00b00a51ad60ea32mr1455939ejz.27.1712926535420; Fri, 12 Apr 2024 05:55:35 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id qw17-20020a1709066a1100b00a473774b027sm1790903ejc.207.2024.04.12.05.55.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Apr 2024 05:55:34 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, "David S . Miller" , Jakub Kicinski , David Ahern , Eric Dumazet , Willem de Bruijn Subject: [RFC 1/6] net: extend ubuf_info callback to ops structure Date: Fri, 12 Apr 2024 13:55:22 +0100 Message-ID: <62a4e09968a9a0f73780876dc6fb0f784bee5fae.1712923998.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We'll need to associate additional callbacks with ubuf_info, introduce a structure holding ubuf_info callbacks. Apart from a more smarter io_uring notification management introduced in next patches, it can be used to generalise msg_zerocopy_put_abort() and also store ->sg_from_iter, which is currently passed in struct msghdr. Signed-off-by: Pavel Begunkov Reviewed-by: David Ahern --- drivers/net/tap.c | 2 +- drivers/net/tun.c | 2 +- drivers/vhost/net.c | 8 ++++++-- include/linux/skbuff.h | 19 +++++++++++-------- io_uring/notif.c | 8 ++++++-- net/core/skbuff.c | 17 +++++++++++------ 6 files changed, 36 insertions(+), 20 deletions(-) diff --git a/drivers/net/tap.c b/drivers/net/tap.c index 9f0495e8df4d..bfdd3875fe86 100644 --- a/drivers/net/tap.c +++ b/drivers/net/tap.c @@ -754,7 +754,7 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control, skb_zcopy_init(skb, msg_control); } else if (msg_control) { struct ubuf_info *uarg = msg_control; - uarg->callback(NULL, uarg, false); + uarg->ops->complete(NULL, uarg, false); } dev_queue_xmit(skb); diff --git a/drivers/net/tun.c b/drivers/net/tun.c index 0b3f21cba552..b7401d990680 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -1906,7 +1906,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile, skb_zcopy_init(skb, msg_control); } else if (msg_control) { struct ubuf_info *uarg = msg_control; - uarg->callback(NULL, uarg, false); + uarg->ops->complete(NULL, uarg, false); } skb_reset_network_header(skb); diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index c64ded183f8d..f16279351db5 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -380,7 +380,7 @@ static void vhost_zerocopy_signal_used(struct vhost_net *net, } } -static void vhost_zerocopy_callback(struct sk_buff *skb, +static void vhost_zerocopy_complete(struct sk_buff *skb, struct ubuf_info *ubuf_base, bool success) { struct ubuf_info_msgzc *ubuf = uarg_to_msgzc(ubuf_base); @@ -408,6 +408,10 @@ static void vhost_zerocopy_callback(struct sk_buff *skb, rcu_read_unlock_bh(); } +static const struct ubuf_info_ops vhost_ubuf_ops = { + .complete = vhost_zerocopy_complete, +}; + static inline unsigned long busy_clock(void) { return local_clock() >> 10; @@ -879,7 +883,7 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) vq->heads[nvq->upend_idx].len = VHOST_DMA_IN_PROGRESS; ubuf->ctx = nvq->ubufs; ubuf->desc = nvq->upend_idx; - ubuf->ubuf.callback = vhost_zerocopy_callback; + ubuf->ubuf.ops = &vhost_ubuf_ops; ubuf->ubuf.flags = SKBFL_ZEROCOPY_FRAG; refcount_set(&ubuf->ubuf.refcnt, 1); msg.msg_control = &ctl; diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 9d24aec064e8..a110e97e074a 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -527,6 +527,11 @@ enum { #define SKBFL_ALL_ZEROCOPY (SKBFL_ZEROCOPY_FRAG | SKBFL_PURE_ZEROCOPY | \ SKBFL_DONT_ORPHAN | SKBFL_MANAGED_FRAG_REFS) +struct ubuf_info_ops { + void (*complete)(struct sk_buff *, struct ubuf_info *, + bool zerocopy_success); +}; + /* * The callback notifies userspace to release buffers when skb DMA is done in * lower device, the skb last reference should be 0 when calling this. @@ -536,8 +541,7 @@ enum { * The desc field is used to track userspace buffer index. */ struct ubuf_info { - void (*callback)(struct sk_buff *, struct ubuf_info *, - bool zerocopy_success); + const struct ubuf_info_ops *ops; refcount_t refcnt; u8 flags; }; @@ -1662,14 +1666,13 @@ static inline void skb_set_end_offset(struct sk_buff *skb, unsigned int offset) } #endif +extern const struct ubuf_info_ops msg_zerocopy_ubuf_ops; + struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, struct ubuf_info *uarg); void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref); -void msg_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *uarg, - bool success); - int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk, struct sk_buff *skb, struct iov_iter *from, size_t length); @@ -1757,13 +1760,13 @@ static inline void *skb_zcopy_get_nouarg(struct sk_buff *skb) static inline void net_zcopy_put(struct ubuf_info *uarg) { if (uarg) - uarg->callback(NULL, uarg, true); + uarg->ops->complete(NULL, uarg, true); } static inline void net_zcopy_put_abort(struct ubuf_info *uarg, bool have_uref) { if (uarg) { - if (uarg->callback == msg_zerocopy_callback) + if (uarg->ops == &msg_zerocopy_ubuf_ops) msg_zerocopy_put_abort(uarg, have_uref); else if (have_uref) net_zcopy_put(uarg); @@ -1777,7 +1780,7 @@ static inline void skb_zcopy_clear(struct sk_buff *skb, bool zerocopy_success) if (uarg) { if (!skb_zcopy_is_nouarg(skb)) - uarg->callback(skb, uarg, zerocopy_success); + uarg->ops->complete(skb, uarg, zerocopy_success); skb_shinfo(skb)->flags &= ~SKBFL_ALL_ZEROCOPY; } diff --git a/io_uring/notif.c b/io_uring/notif.c index b561bd763435..7caaebf94312 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -24,7 +24,7 @@ void io_notif_tw_complete(struct io_kiocb *notif, struct io_tw_state *ts) io_req_task_complete(notif, ts); } -static void io_tx_ubuf_callback(struct sk_buff *skb, struct ubuf_info *uarg, +static void io_tx_ubuf_complete(struct sk_buff *skb, struct ubuf_info *uarg, bool success) { struct io_notif_data *nd = container_of(uarg, struct io_notif_data, uarg); @@ -43,6 +43,10 @@ static void io_tx_ubuf_callback(struct sk_buff *skb, struct ubuf_info *uarg, } } +static const struct ubuf_info_ops io_ubuf_ops = { + .complete = io_tx_ubuf_complete, +}; + struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) __must_hold(&ctx->uring_lock) { @@ -62,7 +66,7 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) nd->zc_report = false; nd->account_pages = 0; nd->uarg.flags = IO_NOTIF_UBUF_FLAGS; - nd->uarg.callback = io_tx_ubuf_callback; + nd->uarg.ops = &io_ubuf_ops; refcount_set(&nd->uarg.refcnt, 1); return notif; } diff --git a/net/core/skbuff.c b/net/core/skbuff.c index b99127712e67..749abab23a67 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1708,7 +1708,7 @@ static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) return NULL; } - uarg->ubuf.callback = msg_zerocopy_callback; + uarg->ubuf.ops = &msg_zerocopy_ubuf_ops; uarg->id = ((u32)atomic_inc_return(&sk->sk_zckey)) - 1; uarg->len = 1; uarg->bytelen = size; @@ -1734,7 +1734,7 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, u32 bytelen, next; /* there might be non MSG_ZEROCOPY users */ - if (uarg->callback != msg_zerocopy_callback) + if (uarg->ops != &msg_zerocopy_ubuf_ops) return NULL; /* realloc only when socket is locked (TCP, UDP cork), @@ -1845,8 +1845,8 @@ static void __msg_zerocopy_callback(struct ubuf_info_msgzc *uarg) sock_put(sk); } -void msg_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *uarg, - bool success) +static void msg_zerocopy_complete(struct sk_buff *skb, struct ubuf_info *uarg, + bool success) { struct ubuf_info_msgzc *uarg_zc = uarg_to_msgzc(uarg); @@ -1855,7 +1855,7 @@ void msg_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *uarg, if (refcount_dec_and_test(&uarg->refcnt)) __msg_zerocopy_callback(uarg_zc); } -EXPORT_SYMBOL_GPL(msg_zerocopy_callback); + void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref) { @@ -1865,10 +1865,15 @@ void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref) uarg_to_msgzc(uarg)->len--; if (have_uref) - msg_zerocopy_callback(NULL, uarg, true); + msg_zerocopy_complete(NULL, uarg, true); } EXPORT_SYMBOL_GPL(msg_zerocopy_put_abort); +const struct ubuf_info_ops msg_zerocopy_ubuf_ops = { + .complete = msg_zerocopy_complete, +}; +EXPORT_SYMBOL_GPL(msg_zerocopy_ubuf_ops); + int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, struct msghdr *msg, int len, struct ubuf_info *uarg) From patchwork Fri Apr 12 12:55:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13627721 Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [209.85.208.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C1145A0FE; Fri, 12 Apr 2024 12:55:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926539; cv=none; b=duZDNrH5vsVPgpzgsmhf9Q25nZgJfaW2qcjiPihuA2KHvH58oRj2t56MULJS020pknL3kIDKRxJLYwvp7u5R0EoUWcOaSyelEtDXz63UID/L4Y4Zn4XEtuH8tlUNBOWT+rtaj0X4INN651GW8t1gRcoJxi33Q9swxf/Anyw0N/I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926539; c=relaxed/simple; bh=nnVzZ/5kAa/izHjn2UfPI1FOnktlC7X+kBBZVKz0ESw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uJKAlbwUw2bL+uTaRJRv0Vv3duLpZVI6qCi0MAVv08YknZNSjg1W9Qg5/PBXNgrKde6PLqAQRGqYYZKcaJSBb7caXVYJ/WlHEBZu73yGUGjDB8RyaBbdbzS8NnIaov8p/VbbZJJ8Fv51Xbai8D5t8s7ZekqZGlLaLrx2amelL9w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Y7/v0226; arc=none smtp.client-ip=209.85.208.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Y7/v0226" Received: by mail-ed1-f54.google.com with SMTP id 4fb4d7f45d1cf-56fe8093c9eso845272a12.3; Fri, 12 Apr 2024 05:55:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712926536; x=1713531336; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v9q0kwJKFg8QYJez2kJDUkUykIMbBt4afcht1V97p18=; b=Y7/v0226ceVplJw6TPTzeAZniOuGSVqWb0AzYv97XRH8/ME+TxBKuU8aRzEfOUEsQh suzTCYPAlgoDj5KJGdWa+bB9TztC4OfIAfA5rgLMR9lypcBfHNzJEnU+K8OxMo/TKC8d UKZfKVH7jzYF8aKVt54WkOFpQJI/XlGb3R/FxbvLxi+rfuBZEBzJ7f+JqSjR7Vubq/eH E8J48fTOeeFkKI3RJLYkVi43jzDCLAaJ90JQ0AerxGXZXiKFEmGCrk3lUSMhMpL3nuqh G8J8uVb44JywqxkmWIRSNFZ/rQs9X4279aGKhzLBTGYwfbzuarWIFBoJ+vSakEX3nLRX TIbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712926536; x=1713531336; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v9q0kwJKFg8QYJez2kJDUkUykIMbBt4afcht1V97p18=; b=Jwh+YkCUcUG+s9n3d1MdnjeAX8DILt/45S9GUmI7wEdKiLhn8rBhxlZAc3dlrFZjM9 Q0G7e1aQd/Z3x2m+5+3qIz7j/SCuXJ6GkeW4G5XAakkMh39qHmE3IQvbnSDClWlAAw4I ii7EvRfa9wNbysKGfhLElmZ1g6DikoaAw9n2tyczYav8GNrhTvzOtu/vgurD8SLk0dRc kkU9xC35XX8DNkuWZiFplj3aqN1kzItdMoZntSwE6Ha5XF9+OA4OEaImU2YfiVIu7BuU zsxlw7iJGh63gAxtU+dW7w0k0RN8my7V74acclcdOS5uOilGw+Twp5n6kS3434L1ywdQ YjFA== X-Forwarded-Encrypted: i=1; AJvYcCXMm0ownYKawU0s3kZlnkGWJDTth0usrORikmg5MN5pj6zF5iiNHxEil9htY1g/sUjavXr87TENWRfuGge9KTjgsy6ffgo+ X-Gm-Message-State: AOJu0YxVLN0PgQzTBZ7YwgMLeBMOV92yh7eyORVV18HU5TnXf0MkwstF 6kVnBDJ5sFXGsyhqsoACnX7DLj9UWph2K16TNwCHo3TGz+sSIjg6aaCiAw== X-Google-Smtp-Source: AGHT+IGT4Aj0LXY5I8Fl7BlsyVEv6USGyEQhdi1UaM+tYMczu/nO4qEDAJPcHW7xL823e93oDoh+Zg== X-Received: by 2002:a17:906:af8b:b0:a51:8d60:215a with SMTP id mj11-20020a170906af8b00b00a518d60215amr1540878ejb.27.1712926536515; Fri, 12 Apr 2024 05:55:36 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id qw17-20020a1709066a1100b00a473774b027sm1790903ejc.207.2024.04.12.05.55.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Apr 2024 05:55:36 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, "David S . Miller" , Jakub Kicinski , David Ahern , Eric Dumazet , Willem de Bruijn Subject: [RFC 2/6] net: add callback for setting a ubuf_info to skb Date: Fri, 12 Apr 2024 13:55:23 +0100 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 At the moment an skb can only have one ubuf_info associated with it, which might be a performance problem for zerocopy sends in cases like TCP via io_uring. Add a callback for assigning ubuf_info to skb, this way we will implement smarter assignment later like linking ubuf_info together. Note, it's an optional callback, which should be compatible with skb_zcopy_set(), that's because the net stack might potentially decide to clone an skb and take another reference to ubuf_info whenever it wishes. Also, a correct implementation should always be able to bind to an skb without prior ubuf_info, otherwise we could end up in a situation when the send would not be able to progress. Signed-off-by: Pavel Begunkov Reviewed-by: David Ahern --- include/linux/skbuff.h | 2 ++ net/core/skbuff.c | 20 ++++++++++++++------ 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index a110e97e074a..ced69f37977f 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -530,6 +530,8 @@ enum { struct ubuf_info_ops { void (*complete)(struct sk_buff *, struct ubuf_info *, bool zerocopy_success); + /* has to be compatible with skb_zcopy_set() */ + int (*link_skb)(struct sk_buff *skb, struct ubuf_info *uarg); }; /* diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 749abab23a67..1922e3d09c7f 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1881,11 +1881,18 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, struct ubuf_info *orig_uarg = skb_zcopy(skb); int err, orig_len = skb->len; - /* An skb can only point to one uarg. This edge case happens when - * TCP appends to an skb, but zerocopy_realloc triggered a new alloc. - */ - if (orig_uarg && uarg != orig_uarg) - return -EEXIST; + if (uarg->ops->link_skb) { + err = uarg->ops->link_skb(skb, uarg); + if (err) + return err; + } else { + /* An skb can only point to one uarg. This edge case happens + * when TCP appends to an skb, but zerocopy_realloc triggered + * a new alloc. + */ + if (orig_uarg && uarg != orig_uarg) + return -EEXIST; + } err = __zerocopy_sg_from_iter(msg, sk, skb, &msg->msg_iter, len); if (err == -EFAULT || (err == -EMSGSIZE && skb->len == orig_len)) { @@ -1899,7 +1906,8 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, return err; } - skb_zcopy_set(skb, uarg, NULL); + if (!uarg->ops->link_skb) + skb_zcopy_set(skb, uarg, NULL); return skb->len - orig_len; } EXPORT_SYMBOL_GPL(skb_zerocopy_iter_stream); From patchwork Fri Apr 12 12:55:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13627722 Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F40D5A78A; Fri, 12 Apr 2024 12:55:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926541; cv=none; b=W1jGXDexmgkfbfdklBq4xJLeJEVYDTVzUhtEOrHQO+aCaV3zxlDMkE0wa/f/PUGZA+fjZlqJQ37tsJaMBD4D6xIjb8blxxargZRrsyC59KXCrk0WOktcli+n86nGFZ82+M/3edUsdjPlCQ/TTKatAaIIcfKY+b5pdXxm+xNSUYA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926541; c=relaxed/simple; bh=Q8na8989lPX4iGF0V/+vD30mjBtKTgqBFdLY9v2vZhg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nAf5UMl8L/BqtWCP/EkN5oKidqx0AXF9wMfKNniGA5gdLn2sYsVkXSZLV5qMNUw3KNqCf6cJ/BYP1o7xaeckT/Xl314+cu5Ti75d0EweCcdtSssuVmAztr3odvvTKEJ7cdzNXi+TBAlwP9Ldntw4O/qiAIgpC3QD4T9IK7qQneQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CqZYA03l; arc=none smtp.client-ip=209.85.218.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CqZYA03l" Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-a4702457ccbso104473566b.3; Fri, 12 Apr 2024 05:55:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712926537; x=1713531337; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YVTfXmYQqBxum6R+kJh5DuD+n6/0VKxhzLhhkJxZnzw=; b=CqZYA03lTLNFnWzYBW0QNZaLAuwldnoc1aFzfKfnZutSo9J3QncrplDhwzpJZKkPbm s/Wra2kxGQP9+8xJe1r4zjuUWU28NjFiUBZiCIbX6tanX/1dxAA1tHzJAt7lZH39OO1t 2KY4zk/pF8xxunAFP3WE5r3c+8SJbW2V+iD9ZQdL7DVktkKNBICX2oxE1oboZd8Sks0k 0moSx3FIYteehiGwxiEDtEdiOxgJBg6Zq23x38gfVWtGXbosAeHDapRNlgZ6aS4Yv2Lj lN+YsvuZolVTG/pZ0/fNbCmZgqc/NMePbi66gA9AoIfjKCc1pHrOx4cOwwVo/ZIseKTd outA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712926537; x=1713531337; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YVTfXmYQqBxum6R+kJh5DuD+n6/0VKxhzLhhkJxZnzw=; b=FtR9TunXDMydM4LX4UqQ8V8GwXUbaxNsfU8KjuYZcx3M8wDHVFuhNGnkXtAzT4YSXa J4fMI8nOb2VhIcljBVkhAV9E303eF4oYzikv354+wF/1HjGSF0JTiaz1uJgKOR+WBeRn /5cs1GZnZgY8n0NrJhGhC1zVcx4cZHbWZefD+1Q+g6YQjVAPdIc2Shwo7DQW2a+MbU7g B0jLnzKOaJfZRfdUJIGwjB0liQCEGpIIdFYs4NQ/fAYRcRvcD8p+AWYYI4Lwv1xL84S2 YdqarzZ0RpWOO3/jUzXCNPgSp52hGUipE8pLSubEUUP06IO3zXbR+zheD5LePw2Ft4IE dkAA== X-Forwarded-Encrypted: i=1; AJvYcCVrqyZk/6yx3+AgLHWbltM0TgemBkuSZIUL7qe3aABCUi3ko5yZP9VjPFKVWvHYlcK2DsWhOi5fh6ejH5HIGvFR3UBX/oVa X-Gm-Message-State: AOJu0Yz0UswXDyESLLkgraEXhlJEYxsCXi4O7T1Y2TzQvrUNB00QkReX QoCH8l9D3HsDI18F1jjHGyeLmRL68jr0eZQWpZPiQvSBDYZEslGsDrLi+Q== X-Google-Smtp-Source: AGHT+IHLSWvEjOsq6SbkhT/2e/UDGw8ttbjpSUM29gvy3OTJNkR5p4MvxipTNIjZ+EjXAkvXsBFrTA== X-Received: by 2002:a17:906:354d:b0:a52:3b53:d58d with SMTP id s13-20020a170906354d00b00a523b53d58dmr871663eja.19.1712926537483; Fri, 12 Apr 2024 05:55:37 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id qw17-20020a1709066a1100b00a473774b027sm1790903ejc.207.2024.04.12.05.55.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Apr 2024 05:55:36 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, "David S . Miller" , Jakub Kicinski , David Ahern , Eric Dumazet , Willem de Bruijn Subject: [RFC 3/6] io_uring/notif: refactor io_tx_ubuf_complete() Date: Fri, 12 Apr 2024 13:55:24 +0100 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Flip the dec_and_test if, so when we add more stuff later there is less churn. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 7caaebf94312..5a8b2fdd67fd 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -37,10 +37,11 @@ static void io_tx_ubuf_complete(struct sk_buff *skb, struct ubuf_info *uarg, WRITE_ONCE(nd->zc_copied, true); } - if (refcount_dec_and_test(&uarg->refcnt)) { - notif->io_task_work.func = io_notif_tw_complete; - __io_req_task_work_add(notif, IOU_F_TWQ_LAZY_WAKE); - } + if (!refcount_dec_and_test(&uarg->refcnt)) + return; + + notif->io_task_work.func = io_notif_tw_complete; + __io_req_task_work_add(notif, IOU_F_TWQ_LAZY_WAKE); } static const struct ubuf_info_ops io_ubuf_ops = { From patchwork Fri Apr 12 12:55:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13627723 Received: from mail-ej1-f47.google.com (mail-ej1-f47.google.com [209.85.218.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 649065B5C5; Fri, 12 Apr 2024 12:55:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926541; cv=none; b=GlWXmhMVhPu8roivIRvUExmHQmOm0vDVP6mYYKJzfJ9nGqa0sk5Pcv7/FFV5NE/dn2fKaXr2VBH//wKgPacth8fX7X+e56GmOnHJye92HZFas+ssfXU4lYFKD4wvxXypRrVoSFLgvzugWcvtMeqX42zbeGmp2P5fqtZzCeZ0e3k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926541; c=relaxed/simple; bh=FRaoxelBlM5M3zFc8SaZ82xGErMRNCCauey/s2vGoj4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=It1RbzzGL60F55JNPAPPrg+8ruo2skw3y83APtiIFru8cjdJcmIhNpsenhQnrd4rRgr5/S6YUgPFeKKXq8kZzUZLVp58xqlwH4edzG6OG7vtKFDHKg2TnCCyXzQ00vGa3Eh9UQD3/FvTwUbKBAowo8ox5BGCPM+20r7QteydVD4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=h1woaI+B; arc=none smtp.client-ip=209.85.218.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="h1woaI+B" Received: by mail-ej1-f47.google.com with SMTP id a640c23a62f3a-a523dad53e0so48729766b.1; Fri, 12 Apr 2024 05:55:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712926538; x=1713531338; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Kh76fmEmFUyOwiJFHT8HJmKwt5ojCDL2FalDYbT9SG4=; b=h1woaI+BZomA6mfeZR4nQS4q6MyvQb5lMwI0CcmPy4Iz2BBhzx1id8veh+qRwy0NLv nNx+nZ0QaLG0MD77/ufonp96jbxz/xkrlPzJhkIWK1RSYN+wcNua5Q2WqG+RTe5mtcJg Ti/rjaRuLF/X9cryl41T8AvLUvZR6pJcne0vBGYqUGL4eJFeaX1CK5SPfDsu0tZIRSYy 1BiIBw5yjI3Wumk4e/YNDLmQvmBCnvaHdS5xIWRnSPfQ+EKIFQf0k2ljqCT6ZJFIEXxb 0efrB2SkS0rB2E1F5TzXftItRCQdaR36iqq9it75ATFcA5t2O310GZnaN3591wlNbtLl SUDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712926538; x=1713531338; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Kh76fmEmFUyOwiJFHT8HJmKwt5ojCDL2FalDYbT9SG4=; b=TBZJAc+nPPe0eGmH54w4vGAP8zI5G/ERxaIO55RoIRosU2S1UQCvXrWIvJb54bF4eE HFAzrB2PtpSMDRogaV89cU6+MTzDKr//5E/aINvB0bqIzkMyCslIoyDsmeuFdEipPF8Z a3zVuUndN0XJ9uaJC/pQdStpa0lYxe4hEexxqwPiH93qxlJpjntTzBX8vLOhMkWo74mq Koj68dmlOBK0bM4DSLbVXNFcgp5E0Mw4pGSEsQdKbJJFTK4Il/uJJTokSFgN7TTMTkRc 3pYFhfZVLv0YPJ7hjTqRzb8dXlVBUNqTxzx6dDEAmiUx8iPr+SD5vVLDaQKmW3PQt6TU xlZQ== X-Forwarded-Encrypted: i=1; AJvYcCUArVAFgb2S94EMfMo/NG+yXrbCAT9blyTRPJWrAKcSrsWRoWur130MFHLk720ZbYoyScZ4apsdzYAA9qv1ts1QRTYzcp5S X-Gm-Message-State: AOJu0YyP2ymT10jLSu94/5QoidnK/Rdld4L3QUDrRNFce+CnhtUlv584 IpcrEmEMEh9nVL41kGbm/mw+Wmhj+ii2xOdXvsQ/e+zOBNyHXLkSCcaQpA== X-Google-Smtp-Source: AGHT+IEx1N42p+GXLxy5yQ0lGqS7VgCNIuL2uYZAgsOpi+s0Fjx5W4y9pKQ3qPgNNMTv2rwrGyHJbg== X-Received: by 2002:a17:907:9494:b0:a51:8672:66e4 with SMTP id dm20-20020a170907949400b00a51867266e4mr5573351ejc.22.1712926538551; Fri, 12 Apr 2024 05:55:38 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id qw17-20020a1709066a1100b00a473774b027sm1790903ejc.207.2024.04.12.05.55.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Apr 2024 05:55:37 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, "David S . Miller" , Jakub Kicinski , David Ahern , Eric Dumazet , Willem de Bruijn Subject: [RFC 4/6] io_uring/notif: remove ctx var from io_notif_tw_complete Date: Fri, 12 Apr 2024 13:55:25 +0100 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We don't need ctx in the hottest path, i.e. registered buffers, let's get it only when we need it. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 5a8b2fdd67fd..53532d78a947 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -12,13 +12,12 @@ void io_notif_tw_complete(struct io_kiocb *notif, struct io_tw_state *ts) { struct io_notif_data *nd = io_notif_to_data(notif); - struct io_ring_ctx *ctx = notif->ctx; if (unlikely(nd->zc_report) && (nd->zc_copied || !nd->zc_used)) notif->cqe.res |= IORING_NOTIF_USAGE_ZC_COPIED; - if (nd->account_pages && ctx->user) { - __io_unaccount_mem(ctx->user, nd->account_pages); + if (nd->account_pages && notif->ctx->user) { + __io_unaccount_mem(notif->ctx->user, nd->account_pages); nd->account_pages = 0; } io_req_task_complete(notif, ts); From patchwork Fri Apr 12 12:55:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13627724 Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 439FF5B694; Fri, 12 Apr 2024 12:55:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926542; cv=none; b=r4OVGyw+rdsR/g2oSPz9dGD+3vWea4EwoPchFcm3ppIq/uiRr5mtt+AOLgm0Vnp34DROPwtRdNrKKuTy0TOkiIV+29c4K0ZeBG9rdWQ9U0q7JXDSB3eP8P/WUMYIbGAejcIM+nrBlPzHZVSQ/MUH66Yg/3qsbxF/MwdRArAotZ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926542; c=relaxed/simple; bh=pVOXkwinZJmVU9Lj8rrjvm7TE4CrfVtgro80Y4ih9XE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ueeOj4ocio5DVm4Mwh563j2eNURmomiJ5KXu92LhZp9NyWMAHlCGmRaq4n9s2RkafOI7c7EivVByP9BaQPmAfXVab5gMioGFIpbyVTqlXlOc5s8pxQr+IwCMKVAq8yNiVrRORvDtiNoNraVStxwUZXvbN8EIOcdf2WUpcK8HK7I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RKXiFf2V; arc=none smtp.client-ip=209.85.208.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RKXiFf2V" Received: by mail-ed1-f45.google.com with SMTP id 4fb4d7f45d1cf-56e1f3462caso1045686a12.3; Fri, 12 Apr 2024 05:55:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712926539; x=1713531339; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qQqp5Wk/zImF2BSrOJkJRMSM61A+Z72QenbLjcVFSpY=; b=RKXiFf2V9IgDtBmx1Gi7lOJJJ4zxm2g07dsxkd+SdmgAADuSuDt7zSPDTB1d9oQCd3 QWUQ8XHyRwzARf7R6pmQQMP2C1tkIsybFQtPEAW/EZ5/tmHzQcV1uX/GOX+sn1oSVtix KIOxbsswyiGLly5jjZWLXVOolPESSnpN7ctBiXUzHQSMTmcmwXLzYFVzyzj5bgzghODD gmtdudKOLvJd1tiYcxKTGnSG6ue7MppuRICAq3FFgRhs8nb8KNTRc5BqXFAQUAjj/kap PxQSZXE/4ExoG76KmGs8Aviz75rFsueX7CzS2qZUFn4knn2fCiV0jvRIkKj16FpKosWR oH+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712926539; x=1713531339; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qQqp5Wk/zImF2BSrOJkJRMSM61A+Z72QenbLjcVFSpY=; b=rBMvLmG/KkbxS/uMYcOJS4iz9Xc2/cfAT9aYi4IrwG/vJsHrnXJ38OlG9fPJ6SMJbq eGbCWJQO5naTFyAC6LhbWhQMWN6Y8ye82F4cmFEcU5DuLbDz+88UuTWww1DIY9y065wR o0OgVN6rQxv3Lkpj6f4C6k4cqeilm0F6Pn71M+DjH/p7+6sbcTBdAAleC/VDtR+LZP4E /d84vvaqZ//QToFbAzFvx5X+eLGeGXHF1o0VSETQp8YuKqXfvLRuXqhOZikL0p3Aoarj 7VWxcWq688upvSNVieKusvY3fHqMoz6iHIgNghE7rXPRy0vWoX/KDbHPuZBIhBewHcO8 MJ9w== X-Forwarded-Encrypted: i=1; AJvYcCUotfg407EOn964jlD1/QJcvXRjKaJ8ZLC/7FEsETa4BYnHhxa3xDCDr5qZUd+aO2ScDo6klCHtP6uptu0NyjuI+MIBFuOG X-Gm-Message-State: AOJu0Ywhl0/zzHeHyE5WB3/5LoCgdyZlLEr5QZuiEvC6WQYRoUNqlF6q bx9sAEJX3wf3XgkDzdEpTD1mJrYKuUFcm8+mk4mkutgejD2NV6hu0KBYUA== X-Google-Smtp-Source: AGHT+IG/5PNR3yIorzHwk2G8aSFu+OH9+dZdrWWXo1o8Ak7I10dAxVZUdUHqZaN9HeWj71SEpyd41Q== X-Received: by 2002:a17:906:134f:b0:a51:d742:b390 with SMTP id x15-20020a170906134f00b00a51d742b390mr1912083ejb.30.1712926539452; Fri, 12 Apr 2024 05:55:39 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id qw17-20020a1709066a1100b00a473774b027sm1790903ejc.207.2024.04.12.05.55.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Apr 2024 05:55:38 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, "David S . Miller" , Jakub Kicinski , David Ahern , Eric Dumazet , Willem de Bruijn Subject: [RFC 5/6] io_uring/notif: simplify io_notif_flush() Date: Fri, 12 Apr 2024 13:55:26 +0100 Message-ID: <3149fcf08c6143c7d809423c213c497deeb9a138.1712923998.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_notif_flush() is partially duplicating io_tx_ubuf_complete(), so instead of duplicating it, make the flush call io_tx_ubuf_complete. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 6 +++--- io_uring/notif.h | 9 +++------ 2 files changed, 6 insertions(+), 9 deletions(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 53532d78a947..26680176335f 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -9,7 +9,7 @@ #include "notif.h" #include "rsrc.h" -void io_notif_tw_complete(struct io_kiocb *notif, struct io_tw_state *ts) +static void io_notif_tw_complete(struct io_kiocb *notif, struct io_tw_state *ts) { struct io_notif_data *nd = io_notif_to_data(notif); @@ -23,8 +23,8 @@ void io_notif_tw_complete(struct io_kiocb *notif, struct io_tw_state *ts) io_req_task_complete(notif, ts); } -static void io_tx_ubuf_complete(struct sk_buff *skb, struct ubuf_info *uarg, - bool success) +void io_tx_ubuf_complete(struct sk_buff *skb, struct ubuf_info *uarg, + bool success) { struct io_notif_data *nd = container_of(uarg, struct io_notif_data, uarg); struct io_kiocb *notif = cmd_to_io_kiocb(nd); diff --git a/io_uring/notif.h b/io_uring/notif.h index 52e124a9957c..394e1d33daa6 100644 --- a/io_uring/notif.h +++ b/io_uring/notif.h @@ -20,7 +20,8 @@ struct io_notif_data { }; struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx); -void io_notif_tw_complete(struct io_kiocb *notif, struct io_tw_state *ts); +void io_tx_ubuf_complete(struct sk_buff *skb, struct ubuf_info *uarg, + bool success); static inline struct io_notif_data *io_notif_to_data(struct io_kiocb *notif) { @@ -32,11 +33,7 @@ static inline void io_notif_flush(struct io_kiocb *notif) { struct io_notif_data *nd = io_notif_to_data(notif); - /* drop slot's master ref */ - if (refcount_dec_and_test(&nd->uarg.refcnt)) { - notif->io_task_work.func = io_notif_tw_complete; - __io_req_task_work_add(notif, IOU_F_TWQ_LAZY_WAKE); - } + io_tx_ubuf_complete(NULL, &nd->uarg, true); } static inline int io_notif_account_mem(struct io_kiocb *notif, unsigned len) From patchwork Fri Apr 12 12:55:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13627725 Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D63235C61A; Fri, 12 Apr 2024 12:55:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926544; cv=none; b=AABxc1E/y3WNc0IX3UHzcdCRh41b9XbW6NMQoI+hwUjOEW4z48i0dNbp+x2cSP0YY1jLAPlwliQOWQYhrVwNmLSxvavmGM2n4lxXr95CVXTxBZ+ouNa2kyh56KhU7iTTTm3kR6ICGTOGezzBU9Aod+G8jFIzCyFtMmjsNOAIl7I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712926544; c=relaxed/simple; bh=rr1yCQeGebZn9uchHo/5phlmbfa/+1Hr89YHFY9SqYQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Fb4OroNDENglymqDUBATieeLD0JWNkI8JpyLM5CQyEMaQUxuIee/I8A3AxGLMAzjeLkH7zW52utpwE4kHbdmFQ8irQyK1tk5wm99tnvEdzYf8RL6WixtczceGazQcpktw7vhZ6rNK9gsyJOQmXWFlR35kw5vTEfQrUnVBagm7HU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VEFTQt27; arc=none smtp.client-ip=209.85.218.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VEFTQt27" Received: by mail-ej1-f54.google.com with SMTP id a640c23a62f3a-a51a7d4466bso104907866b.2; Fri, 12 Apr 2024 05:55:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712926540; x=1713531340; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zRv+s3gr19+ls832LKX6BTVWxWU26v5TpvDr6QThETk=; b=VEFTQt27vJ8kKR+Yx58dFtHD94ewZTiKKCISY/x/nDE6VM5MDqsMQjMEsXiY1QWEyi 9ibjvcgUw7kMlx4aS299kof9AUmUcyvVSsGR6gg/lyYEoYoV2UHQh2KzQncaFDRZuzfb IHPLOz51qOHTdN9ow/ygNZyIm+mqA+hnEejEYESzLt0uIh2n+Ig5wW/aGwQkk/KkIlef kadD1MNNBi7ID93gOwfHHW1AbDDX+DPBJARgZ0nqfusQYitGdUisWZ1DsiozOCVW9arJ 2p9dc3Siqg/FT5Nu6SCPl/trrgTb+nYkuz70fFRUBv/NrGjclx9IZl23DIReL/WgClz6 p+6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712926540; x=1713531340; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zRv+s3gr19+ls832LKX6BTVWxWU26v5TpvDr6QThETk=; b=mbYRRxCu2g7IXCP6fQiVuu5tJWDBIIEIwbBoZRNGSbYkc9feSGV+YXnboBoL8EO2Nt TpHi8poN9m8ja0pQKiqe7ovCR6cyOJ0A6NFKdyFoZzgWP+AgDfOfZb7okymzV8FTmRE4 8g5nd9+1afhBlUY8LAw/9v7RDSUWRR53Exg7gs51Iqz5X5CaZxSWmiijRe7shupfW3gp HfcyEZomvxdfX7i1GHixM47wvgZfWWMSDD+KmNBY4k9AK2izrBWWQ0S4d0OYpcwwQN+D iPw3/IKCGd0KcqAiInKvhYB8dtt9gBtoOs9OARjUuHmV7VyCk/tnSgkFCN7YcW43Y/aa HOMw== X-Forwarded-Encrypted: i=1; AJvYcCXb2zfhu89MhzuaRWxqasrbcPt7UT3VBqyqPv9GYzAzgClOKgo9f0WMi8O6lLa4wBpfEOBJzlCGtVyWo7USbJWyV7TpH0Rt X-Gm-Message-State: AOJu0Yw6wC3UNKmqV3B7w06Dt0Ne837texBJ4Bu/jvaiRLF2wc2g/kSs lD+iYkCpe9R4uEEDyfSkgGny7g2yclsJBnqs42otvF3qu2nJTqDo+kr32Q== X-Google-Smtp-Source: AGHT+IGU1Cj2Qd4ICEikKVxaLwMb3o/vkuQOcLgb/6LyzlNUnruzZMmMfXmI+cimfdm3M3GRFXP42A== X-Received: by 2002:a17:906:5a90:b0:a52:24d6:3024 with SMTP id l16-20020a1709065a9000b00a5224d63024mr1533334ejq.12.1712926540369; Fri, 12 Apr 2024 05:55:40 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id qw17-20020a1709066a1100b00a473774b027sm1790903ejc.207.2024.04.12.05.55.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Apr 2024 05:55:39 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, "David S . Miller" , Jakub Kicinski , David Ahern , Eric Dumazet , Willem de Bruijn Subject: [RFC 6/6] io_uring/notif: implement notification stacking Date: Fri, 12 Apr 2024 13:55:27 +0100 Message-ID: <3e2ef5f6d39c4631f5bae86b503a5397d6707563.1712923998.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The network stack allows only one ubuf_info per skb, and unlike MSG_ZEROCOPY, each io_uring zerocopy send will carry a separate ubuf_info. That means that send requests can't reuse a previosly allocated skb and need to get one more or more of new ones. That's fine for large sends, but otherwise it would spam the stack with lots of skbs carrying just a little data each. To help with that implement linking notification (i.e. an io_uring wrapper around ubuf_info) into a list. Each is refcounted by skbs and the stack as usual. additionally all non head entries keep a reference to the head, which they put down when their refcount hits 0. When the head have no more users, it'll efficiently put all notifications in a batch. As mentioned previously about ->io_link_skb, the callback implementation always allows to bind to an skb without a ubuf_info. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 71 +++++++++++++++++++++++++++++++++++++++++++----- io_uring/notif.h | 4 +++ 2 files changed, 68 insertions(+), 7 deletions(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 26680176335f..d58cdc01e691 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -9,18 +9,28 @@ #include "notif.h" #include "rsrc.h" +static const struct ubuf_info_ops io_ubuf_ops; + static void io_notif_tw_complete(struct io_kiocb *notif, struct io_tw_state *ts) { struct io_notif_data *nd = io_notif_to_data(notif); - if (unlikely(nd->zc_report) && (nd->zc_copied || !nd->zc_used)) - notif->cqe.res |= IORING_NOTIF_USAGE_ZC_COPIED; + do { + notif = cmd_to_io_kiocb(nd); - if (nd->account_pages && notif->ctx->user) { - __io_unaccount_mem(notif->ctx->user, nd->account_pages); - nd->account_pages = 0; - } - io_req_task_complete(notif, ts); + lockdep_assert(refcount_read(&nd->uarg.refcnt) == 0); + + if (unlikely(nd->zc_report) && (nd->zc_copied || !nd->zc_used)) + notif->cqe.res |= IORING_NOTIF_USAGE_ZC_COPIED; + + if (nd->account_pages && notif->ctx->user) { + __io_unaccount_mem(notif->ctx->user, nd->account_pages); + nd->account_pages = 0; + } + + nd = nd->next; + io_req_task_complete(notif, ts); + } while (nd); } void io_tx_ubuf_complete(struct sk_buff *skb, struct ubuf_info *uarg, @@ -39,12 +49,56 @@ void io_tx_ubuf_complete(struct sk_buff *skb, struct ubuf_info *uarg, if (!refcount_dec_and_test(&uarg->refcnt)) return; + if (nd->head != nd) { + io_tx_ubuf_complete(skb, &nd->head->uarg, success); + return; + } notif->io_task_work.func = io_notif_tw_complete; __io_req_task_work_add(notif, IOU_F_TWQ_LAZY_WAKE); } +static int io_link_skb(struct sk_buff *skb, struct ubuf_info *uarg) +{ + struct io_notif_data *nd, *prev_nd; + struct io_kiocb *prev_notif, *notif; + struct ubuf_info *prev_uarg = skb_zcopy(skb); + + nd = container_of(uarg, struct io_notif_data, uarg); + notif = cmd_to_io_kiocb(nd); + + if (!prev_uarg) { + net_zcopy_get(&nd->uarg); + skb_zcopy_init(skb, &nd->uarg); + return 0; + } + /* handle it separately as we can't link a notif to itself */ + if (unlikely(prev_uarg == &nd->uarg)) + return 0; + /* we can't join two links together, just request a fresh skb */ + if (unlikely(nd->head != nd || nd->next)) + return -EEXIST; + /* don't mix zc providers */ + if (unlikely(prev_uarg->ops != &io_ubuf_ops)) + return -EEXIST; + + prev_nd = container_of(prev_uarg, struct io_notif_data, uarg); + prev_notif = cmd_to_io_kiocb(nd); + + /* make sure all noifications can be finished in the same task_work */ + if (unlikely(notif->ctx != prev_notif->ctx || + notif->task != prev_notif->task)) + return -EEXIST; + + nd->head = prev_nd->head; + nd->next = prev_nd->next; + prev_nd->next = nd; + net_zcopy_get(&nd->head->uarg); + return 0; +} + static const struct ubuf_info_ops io_ubuf_ops = { .complete = io_tx_ubuf_complete, + .link_skb = io_link_skb, }; struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) @@ -65,6 +119,9 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) nd = io_notif_to_data(notif); nd->zc_report = false; nd->account_pages = 0; + nd->next = NULL; + nd->head = nd; + nd->uarg.flags = IO_NOTIF_UBUF_FLAGS; nd->uarg.ops = &io_ubuf_ops; refcount_set(&nd->uarg.refcnt, 1); diff --git a/io_uring/notif.h b/io_uring/notif.h index 394e1d33daa6..6d2e8b674b43 100644 --- a/io_uring/notif.h +++ b/io_uring/notif.h @@ -14,6 +14,10 @@ struct io_notif_data { struct file *file; struct ubuf_info uarg; unsigned long account_pages; + + struct io_notif_data *next; + struct io_notif_data *head; + bool zc_report; bool zc_used; bool zc_copied;