From patchwork Tue Jul 30 18:41:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13747774 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B03B7169397 for ; Tue, 30 Jul 2024 18:41:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722364912; cv=none; b=JUa6DgnIrEUOU+1LCGIk4MLv5M6cNJZ5BNKhPQeerU0Vc6yjvUEZrU9RpYAePJJjX4Im2Gd8Fji+POs/71CEmF6M7k/CyKFzBheajACucvcaqkj0HkWONTQo0TgI7fSP0kZhROQ602hjBl5dDlt5V81mw5b06ZkZKOduDQFWyfA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722364912; c=relaxed/simple; bh=8U98s/x9klh3R1w6W5giSFPu50LKMrDr3xkDem20ovU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=F7mzSSxrlXjVBNuAja4OlUEaDM10L+z2tAzqP4yuIOEOfSi7ZXINp/VxfHWoNEgglzB5I2u6DGNgFshrMyqWHX8Kr2ZKZ2h8MVhfFc2oeT1tgci0mKQTRJbJfMHBYXRfIN80xWVDaf9rOUfG6ttaKSMoK4+k9hAgRReTRoeYDbg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=N+Pw+dTI; arc=none smtp.client-ip=209.85.167.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="N+Pw+dTI" Received: by mail-oi1-f179.google.com with SMTP id 5614622812f47-3db1d4dab7fso2997459b6e.1 for ; Tue, 30 Jul 2024 11:41:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1722364909; x=1722969709; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BikDbh1/9SQRd/kB9M3Eiu6ynP/hqf/DKnY/EEbSCvw=; b=N+Pw+dTIBOonZAMRwaWpPTXXLjcLZGhKYt7Qw15QeFMjczQ0OtfUyzUtKaeD7SP91g kqdajdep9ypI8z34Ug+5Wizauo+xpj53/G5cMquY8odJjl1GUWdxtP62ez8Nskh8pZi+ KWz3Bde3bFQ75tgrhQzj2CzfiK+udABX4Zc5te8o4Pt5R5OahuhXQ1DzrEZC5tzwxoE0 geghNVldj/RVAaFNIlm9j/UYHF5AIOD5iHk9Ii7jq237ssoPBpKsR6YsKlNJCpgcvps9 vvqb7oekb3mW/CXRwN5gILDqf+iCQGj4XrgKo1DLQ4sI4nwnGXE140J/iOAMxdqIGXeR pEEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722364909; x=1722969709; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BikDbh1/9SQRd/kB9M3Eiu6ynP/hqf/DKnY/EEbSCvw=; b=tb2lQxR7ulF0pU5heR+GLfFkColpH5fgDsl7+Zx5PsGdq3FVGq0LKQ55qeqxCz4lI4 Nd8h0cCwldYXGzDr3/QIp92LlebzbmLPXqC+YEb6teVt3rrWoW2eiEE+U8lZdRQr5mRq maCqiCLcvgjc1DFznIqyCfpgGePwoY7qgGXI2xXG3mur3qfLVIdqYfJ/sl6EIh9Cbtrt Q8tQGAQh3qlq7+lLAPyCxUeuKCuz5SOQC0cEddUSnHRKh/fYINWjF2FN5G2n3wu8EjQV HJJSS+VN2Vt2KAsxXyuXLoO2CuWjuRNYXwKe61gf/yqtDjsOxidZBpvbaBb3I4IfMa1i a2KA== X-Gm-Message-State: AOJu0Ywzf7Y6TjulgUfMlq69kH+h+LNe1qX3NYnjJ5XgwNAYw2kpTDLW XleDZ3rfe6PjW9NGIPQEMtGDPj7AX3O7SXDcoSg2YnJTy6Kz3JX0gMMicVc5xBY6hF1CRLwT0SC M X-Google-Smtp-Source: AGHT+IHmEFKIeORqg/Jp5LW/faNuIRskuviRnwXbCec0ZGE8Hd/p/5viL9PkgWoYAhj4ZtwgZOhnlQ== X-Received: by 2002:a05:6358:591a:b0:1ac:671a:f39f with SMTP id e5c5f4694b2df-1adbe30dbcbmr1511840855d.11.1722364909193; Tue, 30 Jul 2024 11:41:49 -0700 (PDT) Received: from n191-036-066.byted.org ([139.177.233.173]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb3fb07bf7sm66132716d6.137.2024.07.30.11.41.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jul 2024 11:41:48 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: linux-api@vger.kernel.org, willemdebruijn.kernel@gmail.com, almasrymina@google.com, edumazet@google.com, davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, axboe@kernel.dk, shuah@kernel.org, linux-kselftest@vger.kernel.org, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v8 1/3] sock: support copying cmsgs to the user space in sendmsg Date: Tue, 30 Jul 2024 18:41:18 +0000 Message-Id: <20240730184120.4089835-2-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240730184120.4089835-1-zijianzhang@bytedance.com> References: <20240730184120.4089835-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang Users can pass msg_control as a placeholder to recvmsg, and get some info from the kernel upon returning of it, but it's not available for sendmsg. Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg creates a kernel copy of msg_control and passes that to the callees, put_cmsg in sendmsg path will write into this kernel buffer. If users want to get info after returning of sendmsg, they typically have to call recvmsg on the MSG_ERRQUEUE of the socket, incurring extra system call overhead. This commit supports copying cmsg from the kernel space to the user space upon returning of sendmsg to mitigate this overhead. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- include/linux/socket.h | 8 ++++++ include/net/sock.h | 2 +- net/core/sock.c | 6 ++-- net/ipv4/ip_sockglue.c | 2 +- net/ipv6/datagram.c | 2 +- net/socket.c | 63 ++++++++++++++++++++++++++++++++++++++---- 6 files changed, 72 insertions(+), 11 deletions(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index df9cdb8bbfb8..40173c919d0f 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -71,6 +71,7 @@ struct msghdr { void __user *msg_control_user; }; bool msg_control_is_user : 1; + bool msg_control_copy_to_user : 1; bool msg_get_inq : 1;/* return INQ after receive */ unsigned int msg_flags; /* flags on received message */ __kernel_size_t msg_controllen; /* ancillary data buffer length */ @@ -168,6 +169,11 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr return __cmsg_nxthdr(__msg->msg_control, __msg->msg_controllen, __cmsg); } +static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg) +{ + return 0; +} + static inline size_t msg_data_left(struct msghdr *msg) { return iov_iter_count(&msg->msg_iter); @@ -396,6 +402,8 @@ struct timespec64; struct __kernel_timespec; struct old_timespec32; +DECLARE_STATIC_KEY_FALSE(tx_copy_cmsg_to_user_key); + struct scm_timestamping_internal { struct timespec64 ts[3]; }; diff --git a/include/net/sock.h b/include/net/sock.h index cce23ac4d514..9c728287d21d 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1804,7 +1804,7 @@ static inline void sockcm_init(struct sockcm_cookie *sockc, }; } -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg, +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg, struct sockcm_cookie *sockc); int sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct sockcm_cookie *sockc); diff --git a/net/core/sock.c b/net/core/sock.c index 9abc4fe25953..b2cbe753af1d 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2826,8 +2826,8 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, } EXPORT_SYMBOL(sock_alloc_send_pskb); -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg, - struct sockcm_cookie *sockc) +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg __always_unused, + struct cmsghdr *cmsg, struct sockcm_cookie *sockc) { u32 tsflags; @@ -2881,7 +2881,7 @@ int sock_cmsg_send(struct sock *sk, struct msghdr *msg, return -EINVAL; if (cmsg->cmsg_level != SOL_SOCKET) continue; - ret = __sock_cmsg_send(sk, cmsg, sockc); + ret = __sock_cmsg_send(sk, msg, cmsg, sockc); if (ret) return ret; } diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c index cf377377b52d..6360b8ba9c84 100644 --- a/net/ipv4/ip_sockglue.c +++ b/net/ipv4/ip_sockglue.c @@ -267,7 +267,7 @@ int ip_cmsg_send(struct sock *sk, struct msghdr *msg, struct ipcm_cookie *ipc, } #endif if (cmsg->cmsg_level == SOL_SOCKET) { - err = __sock_cmsg_send(sk, cmsg, &ipc->sockc); + err = __sock_cmsg_send(sk, msg, cmsg, &ipc->sockc); if (err) return err; continue; diff --git a/net/ipv6/datagram.c b/net/ipv6/datagram.c index fff78496803d..c9ae30acf895 100644 --- a/net/ipv6/datagram.c +++ b/net/ipv6/datagram.c @@ -777,7 +777,7 @@ int ip6_datagram_send_ctl(struct net *net, struct sock *sk, } if (cmsg->cmsg_level == SOL_SOCKET) { - err = __sock_cmsg_send(sk, cmsg, &ipc6->sockc); + err = __sock_cmsg_send(sk, msg, cmsg, &ipc6->sockc); if (err) return err; continue; diff --git a/net/socket.c b/net/socket.c index fcbdd5bc47ac..4b65ac92045a 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2537,8 +2537,49 @@ static int copy_msghdr_from_user(struct msghdr *kmsg, return err < 0 ? err : 0; } -static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, - unsigned int flags, struct used_address *used_address, +DEFINE_STATIC_KEY_FALSE(tx_copy_cmsg_to_user_key); + +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys, + struct user_msghdr __user *umsg) +{ + struct compat_msghdr __user *umsg_compat = + (struct compat_msghdr __user *)umsg; + unsigned int flags = msg_sys->msg_flags; + struct msghdr msg_user = *msg_sys; + unsigned long cmsg_ptr; + struct cmsghdr *cmsg; + int err; + + msg_user.msg_control_is_user = true; + msg_user.msg_control_user = umsg->msg_control; + cmsg_ptr = (unsigned long)msg_user.msg_control; + for_each_cmsghdr(cmsg, msg_sys) { + if (!CMSG_OK(msg_sys, cmsg)) + break; + if (!cmsg_copy_to_user(cmsg)) + continue; + err = put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type, + cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg)); + if (err) + return err; + } + + err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), + COMPAT_FLAGS(umsg)); + if (err) + return err; + if (MSG_CMSG_COMPAT & flags) + err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr, + &umsg_compat->msg_controllen); + else + err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr, + &umsg->msg_controllen); + return err; +} + +static int ____sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg, + struct msghdr *msg_sys, unsigned int flags, + struct used_address *used_address, unsigned int allowed_msghdr_flags) { unsigned char ctl[sizeof(struct cmsghdr) + 20] @@ -2549,6 +2590,8 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, ssize_t err; err = -ENOBUFS; + if (static_branch_unlikely(&tx_copy_cmsg_to_user_key)) + msg_sys->msg_control_copy_to_user = false; if (msg_sys->msg_controllen > INT_MAX) goto out; @@ -2606,6 +2649,16 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, used_address->name_len); } + if (static_branch_unlikely(&tx_copy_cmsg_to_user_key)) { + if (msg_sys->msg_control_copy_to_user && msg && err >= 0) { + ssize_t len = err; + + err = sendmsg_copy_cmsg_to_user(msg_sys, msg); + if (!err) + err = len; + } + } + out_freectl: if (ctl_buf != ctl) sock_kfree_s(sock->sk, ctl_buf, ctl_len); @@ -2648,8 +2701,8 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg, if (err < 0) return err; - err = ____sys_sendmsg(sock, msg_sys, flags, used_address, - allowed_msghdr_flags); + err = ____sys_sendmsg(sock, msg, msg_sys, flags, used_address, + allowed_msghdr_flags); kfree(iov); return err; } @@ -2660,7 +2713,7 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg, long __sys_sendmsg_sock(struct socket *sock, struct msghdr *msg, unsigned int flags) { - return ____sys_sendmsg(sock, msg, flags, NULL, 0); + return ____sys_sendmsg(sock, NULL, msg, flags, NULL, 0); } long __sys_sendmsg(int fd, struct user_msghdr __user *msg, unsigned int flags, From patchwork Tue Jul 30 18:41:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13747775 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f172.google.com (mail-yb1-f172.google.com [209.85.219.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5426B1AA3D9 for ; Tue, 30 Jul 2024 18:41:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722364916; cv=none; b=V5IfQBw5K/vh+0bxXj/JQt2Ot8mjZtZC9LBDgl69BdYZKQ5w+LH9Gm0NJK+l4/88CPN5WDh77+qc0oQz/PiTFldhFFIkdo/8/DdLDS5kBa5JibaGRyXtCy5kZtsCjNLq5YBIO66r+5jU6Y1Mr6/J7cyt2APcJXC/I4WCW/KjYYI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722364916; c=relaxed/simple; bh=dRcVp0Z7DoqqrWdM+KQxiL42PINqxpG662XC4ml1djE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Qs1JJ6xFTW9WVheqmG9V4co8PvShw4vT3lOLBgh7MCMIid4hUzDQikQqNyaQUgRI206BtWJlgBtWAq97iqr964z1wIJSj46SwdwUhCFXYJCDaRqGtwX5nfATPiPcHXkTMaKtG24VsLS1pKSSW/DM7KdXoKU+ibkALciUQrnruMM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=azuwJn1D; arc=none smtp.client-ip=209.85.219.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="azuwJn1D" Received: by mail-yb1-f172.google.com with SMTP id 3f1490d57ef6-dfef5980a69so3877254276.3 for ; Tue, 30 Jul 2024 11:41:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1722364913; x=1722969713; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rPnJjIq98SiA+JXhukXzURjAkFL/DxtFbjsUcIfNX/M=; b=azuwJn1DruXA7Z3idgopnUCvXwCiQkBpIVYex79mAflg36qoatXMT+BBcedHxltL9H DEqQByKt5NFTjm/mNC6l/3IyVsLJk50ijryFoGvmGLRhN7nKYleRBCF5coeYAEZDOklG k562fT6wYRcNEo/wTl+Itlpetx2aD3Y493d5ZKJNHgZLcgZnA2GAejq2yW2eCcf4kxn7 y1mjaCjNZiHKKebRHgf8IlV8umEHvUshGgSRuPHJrEfTT6CyaDZrsB8FZ9gxVd6XbFn+ DePdaJqzFhwaZByxSW+PanVwoOVmxfEBiDY7ZmnNFWkn7D967g0OocTg6+ml+8PlozM3 OBHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722364913; x=1722969713; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rPnJjIq98SiA+JXhukXzURjAkFL/DxtFbjsUcIfNX/M=; b=mOAQLTH8lOGc+Ng3qu1VjoRKIY0xkfdJr8SVuIs4cd152bh7tb+rTcKEJvcL9Mw9Ue JARVgnSpr7nFF0EWqkIXAnIxbQDKf3SgvfJ2WEtMc45TaNsHHU3zm7iBq4Qhk/cS851Y Z0k9To2Gn/pg9Wk8vFquTZkPlhPH27KOOyFFIpl0PIDMTVa8nfD3Jkh02DGrT85U/mBa xdxuFIaRWyyvwdE3OrpMiPJa8Ixoar0N/FuikphqvsMeH2CppPI603o9bgKUyZGDf5fi E2DCvVdyScalOg/Nv71J5VNVWatcbVsVU7srw2ZVF1vzZcDuH5/aMgmnjXZ/gllovlr7 wS7A== X-Gm-Message-State: AOJu0YwYTCl6th5aNhHsTpaUBhVXvQ2YOwlc5+A8U3LkfGB9zsJdYgGs dDNNxzfdxnmvnqWtxUpmQ+uIMJis+fWYQczlYSkrfS3Maev02EOl9B5+tw4QpjRc+xvl+C6JwbK v X-Google-Smtp-Source: AGHT+IEryLVpO7mWHAYjPE/Ggq/E9tDXYa/U1l8LGbTJENtvyJIBzIceolRcnGS9FGQKyr5d7l2SVw== X-Received: by 2002:a05:6902:1029:b0:e0b:4aa4:1704 with SMTP id 3f1490d57ef6-e0b5446cb77mr15436368276.18.1722364912943; Tue, 30 Jul 2024 11:41:52 -0700 (PDT) Received: from n191-036-066.byted.org ([139.177.233.173]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb3fb07bf7sm66132716d6.137.2024.07.30.11.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jul 2024 11:41:52 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: linux-api@vger.kernel.org, willemdebruijn.kernel@gmail.com, almasrymina@google.com, edumazet@google.com, davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, axboe@kernel.dk, shuah@kernel.org, linux-kselftest@vger.kernel.org, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v8 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control Date: Tue, 30 Jul 2024 18:41:19 +0000 Message-Id: <20240730184120.4089835-3-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240730184120.4089835-1-zijianzhang@bytedance.com> References: <20240730184120.4089835-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang The MSG_ZEROCOPY flag enables copy avoidance for socket send calls. However, zerocopy is not a free lunch. Apart from the management of user pages, the combination of poll + recvmsg to receive notifications incurs unignorable overhead in the applications. We try to mitigate this overhead with a new notification mechanism based on msg_control. Leveraging the general framework to copy cmsgs to the user space, we copy zerocopy notifications to the user upon returning of sendmsgs. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- arch/alpha/include/uapi/asm/socket.h | 2 + arch/mips/include/uapi/asm/socket.h | 2 + arch/parisc/include/uapi/asm/socket.h | 2 + arch/sparc/include/uapi/asm/socket.h | 2 + include/linux/socket.h | 2 +- include/uapi/asm-generic/socket.h | 2 + include/uapi/linux/socket.h | 23 +++++++++ net/core/sock.c | 72 +++++++++++++++++++++++++-- 8 files changed, 102 insertions(+), 5 deletions(-) diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi/asm/socket.h index e94f621903fe..7c32d9dbe47f 100644 --- a/arch/alpha/include/uapi/asm/socket.h +++ b/arch/alpha/include/uapi/asm/socket.h @@ -140,6 +140,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/asm/socket.h index 60ebaed28a4c..3f7fade998cb 100644 --- a/arch/mips/include/uapi/asm/socket.h +++ b/arch/mips/include/uapi/asm/socket.h @@ -151,6 +151,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/uapi/asm/socket.h index be264c2b1a11..77f5bee0fdc9 100644 --- a/arch/parisc/include/uapi/asm/socket.h +++ b/arch/parisc/include/uapi/asm/socket.h @@ -132,6 +132,8 @@ #define SO_PASSPIDFD 0x404A #define SO_PEERPIDFD 0x404B +#define SCM_ZC_NOTIFICATION 0x404C + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi/asm/socket.h index 682da3714686..eb44fc515b45 100644 --- a/arch/sparc/include/uapi/asm/socket.h +++ b/arch/sparc/include/uapi/asm/socket.h @@ -133,6 +133,8 @@ #define SO_PASSPIDFD 0x0055 #define SO_PEERPIDFD 0x0056 +#define SCM_ZC_NOTIFICATION 0x0057 + #if !defined(__KERNEL__) diff --git a/include/linux/socket.h b/include/linux/socket.h index 40173c919d0f..71e3c6ebfed5 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -171,7 +171,7 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg) { - return 0; + return __cmsg->cmsg_type == SCM_ZC_NOTIFICATION; } static inline size_t msg_data_left(struct msghdr *msg) diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h index 8ce8a39a1e5f..02e9159c7944 100644 --- a/include/uapi/asm-generic/socket.h +++ b/include/uapi/asm-generic/socket.h @@ -135,6 +135,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__)) diff --git a/include/uapi/linux/socket.h b/include/uapi/linux/socket.h index d3fcd3b5ec53..b5b5fa9febb1 100644 --- a/include/uapi/linux/socket.h +++ b/include/uapi/linux/socket.h @@ -2,6 +2,8 @@ #ifndef _UAPI_LINUX_SOCKET_H #define _UAPI_LINUX_SOCKET_H +#include + /* * Desired design of maximum size and alignment (see RFC2553) */ @@ -35,4 +37,25 @@ struct __kernel_sockaddr_storage { #define SOCK_TXREHASH_DISABLED 0 #define SOCK_TXREHASH_ENABLED 1 +#define ZC_NOTIFICATION_MAX 16 + +/* + * A zc_info_elem represents a completion notification for sendmsgs in range + * lo to high, zerocopy represents whether the underlying transmission is + * zerocopy or not. + */ +struct zc_info_elem { + __u32 lo; + __u32 hi; + __u8 zerocopy; +}; + +/* + * zc_info is the struct used for the SCM_ZC_NOTIFICATION control message. + */ +struct zc_info { + __u32 size; /* size of the zc_info_elem arr */ + struct zc_info_elem arr[]; +}; + #endif /* _UAPI_LINUX_SOCKET_H */ diff --git a/net/core/sock.c b/net/core/sock.c index b2cbe753af1d..37b1b12623ee 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -1481,10 +1481,12 @@ int sk_setsockopt(struct sock *sk, int level, int optname, ret = -EOPNOTSUPP; } if (!ret) { - if (val < 0 || val > 1) + if (val < 0 || val > 1) { ret = -EINVAL; - else + } else { sock_valbool_flag(sk, SOCK_ZEROCOPY, valbool); + static_branch_enable(&tx_copy_cmsg_to_user_key); + } } break; @@ -2826,8 +2828,8 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, } EXPORT_SYMBOL(sock_alloc_send_pskb); -int __sock_cmsg_send(struct sock *sk, struct msghdr *msg __always_unused, - struct cmsghdr *cmsg, struct sockcm_cookie *sockc) +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg, + struct sockcm_cookie *sockc) { u32 tsflags; @@ -2863,6 +2865,68 @@ int __sock_cmsg_send(struct sock *sk, struct msghdr *msg __always_unused, case SCM_RIGHTS: case SCM_CREDENTIALS: break; + case SCM_ZC_NOTIFICATION: { + struct zc_info *zc = CMSG_DATA(cmsg); + struct sk_buff_head *q, local_q; + int cmsg_data_len, i = 0; + unsigned long flags; + struct sk_buff *skb; + + if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS) + return -EINVAL; + + cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr); + if (cmsg_data_len < sizeof(struct zc_info)) + return -EINVAL; + + if (zc->size > ZC_NOTIFICATION_MAX || + (cmsg_data_len - sizeof(struct zc_info)) != + (zc->size * sizeof(struct zc_info_elem))) + return -EINVAL; + + q = &sk->sk_error_queue; + skb_queue_head_init(&local_q); + + /* Get zerocopy error messages from sk_error_queue, and add them + * to a local queue for later processing. This minimizes the + * code while the spinlock is held and irq is disabled. + */ + spin_lock_irqsave(&q->lock, flags); + skb = skb_peek(q); + while (skb && i < zc->size) { + struct sk_buff *skb_next = skb_peek_next(skb, q); + struct sock_exterr_skb *serr = SKB_EXT_ERR(skb); + + if (serr->ee.ee_errno != 0 || + serr->ee.ee_origin != SO_EE_ORIGIN_ZEROCOPY) { + skb = skb_next; + continue; + } + + __skb_unlink(skb, q); + __skb_queue_tail(&local_q, skb); + skb = skb_next; + i++; + } + spin_unlock_irqrestore(&q->lock, flags); + + i = 0; + while ((skb = skb_peek(&local_q)) != NULL) { + struct sock_exterr_skb *serr = SKB_EXT_ERR(skb); + + zc->arr[i].hi = serr->ee.ee_data; + zc->arr[i].lo = serr->ee.ee_info; + zc->arr[i].zerocopy = !(serr->ee.ee_code + & SO_EE_CODE_ZEROCOPY_COPIED); + __skb_unlink(skb, &local_q); + consume_skb(skb); + i++; + } + + zc->size = i; + msg->msg_control_copy_to_user = true; + break; + } default: return -EINVAL; } From patchwork Tue Jul 30 18:41:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13747776 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68FA81AA3F1 for ; Tue, 30 Jul 2024 18:41:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722364919; cv=none; b=VNxhZ4I+YJ5ff5O6b9IEidFOydM0mIIq0kQlvzjC/iQ5xZwccLTdGWe3fati1TE7vw/Gj3uoLLZdzdSijFIzpaiGdiNVejk+ZbkhEHbp4TfVrvdBHwB62P+oVb/v9CiiGxXEzHdOpdyBKCwN+jJ3/FZi5nHe383PpfPtODftjYc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722364919; c=relaxed/simple; bh=eVwjcXh08v/BloRCWi+Au5NPRXGgBtqnMqSrhA86WPE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=at0AW1CxDUcYhMRFvZwialk2aT2G4LOkGxL7tC+ijv0OsLf1l+SZq6qxzUe1+59d5unQHOtnEkQYqJ2pMqvs2VvkwE4O3og396PHHn36yyRvuFlg+jk7P7EVhyVrmGHG7YocWBVpbVrWkDLUMcWdeTPsLCmTAYuyLYK9u5osbYU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=hXeM36iD; arc=none smtp.client-ip=209.85.219.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="hXeM36iD" Received: by mail-qv1-f54.google.com with SMTP id 6a1803df08f44-6b7a4668f07so25982366d6.1 for ; Tue, 30 Jul 2024 11:41:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1722364916; x=1722969716; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NFMubqDRuXqmoPI75vXrJ4qmre3+FeuST3BNycNrURg=; b=hXeM36iD6OHIsil52wi4n4FE7dcH7zsMPiZ52xBq1wkrJ9bR9xcA2JXbKrb1Qs/Bhe Sg+W9KcYrRGs/cSbbm7sM2Yeeuj8rKg7bs8FKl2PSthrEEPsHKRsy0whzDjlpb+bQk7H qTWZmXRUQl5YiYhQPWr6R22sXQTgHrfRmrxC52AejtOJbCQGECA1JoF0snwkdGvxc6MY j9/wi9ukXD9NnBUwKsZDYqvEf+GCfGWBKGo0bdjFBUIHoNkYXsLs6kB1m7Reo22TGmeg kL/YotP9qHGB8OBwEXZnQLuO77mydYYqnBClkoV0NF4PqmVOCTSFh4VQe4fEANnAzgTa IQMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722364916; x=1722969716; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NFMubqDRuXqmoPI75vXrJ4qmre3+FeuST3BNycNrURg=; b=tP54dDcXnKmfaXShRXbNnu9voxFolEQOLpIHARxekezLhTPvWCRmuiw5FfDh1lYNWB p2c4NzWq38pxUYvYXXOvH8UVIGvauWUhi2UddusNeOET9syjLzrwmMBcUtm9vPZnk3eu PPVdr+mNsNm02f8qyAvWZ/KQc85+zrkq3nFf10Zbf4Sf8QFbockKaFvXJYT32lQwW0bB lwusajQkqtj3R3ocKmJ2j5Il5jY9XqNuk+5qjBoEcyEAznc0DN65RffU4alNOVq1IhcC 3c9zRCjR+pcQYOB5kBFwNF1s7VJxQTHzAIP4R6upIhJt8nu7qn6Si4BuZgi42/k1DWfd be4Q== X-Gm-Message-State: AOJu0YwlJNd5BHSfojrBbi4Z2N5IoinwV51bw5SKWlxZWFNYwoaVseaC Jj8OU3S82nidnawSbO0+dxzf1GPbtOb+inbH0RUV1qj7nxfvXr2z73vo8HfcP8dbnvfUHtJ4Wzn 3 X-Google-Smtp-Source: AGHT+IGqMyyvOhsKtdiERws+A5taOq1D0dtDaWZ6YlDnhO029w/RAeUiX/89Qw+mJRbm+r6IQGTcBg== X-Received: by 2002:a05:6214:dc6:b0:6b5:434:cc86 with SMTP id 6a1803df08f44-6bb559f9338mr224388956d6.10.1722364915909; Tue, 30 Jul 2024 11:41:55 -0700 (PDT) Received: from n191-036-066.byted.org ([139.177.233.173]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb3fb07bf7sm66132716d6.137.2024.07.30.11.41.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jul 2024 11:41:54 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: linux-api@vger.kernel.org, willemdebruijn.kernel@gmail.com, almasrymina@google.com, edumazet@google.com, davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, axboe@kernel.dk, shuah@kernel.org, linux-kselftest@vger.kernel.org, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v8 3/3] selftests: add MSG_ZEROCOPY msg_control notification test Date: Tue, 30 Jul 2024 18:41:20 +0000 Message-Id: <20240730184120.4089835-4-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240730184120.4089835-1-zijianzhang@bytedance.com> References: <20240730184120.4089835-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang We update selftests/net/msg_zerocopy.c to accommodate the new mechanism, cfg_notification_limit has the same semantics for both methods. Test results are as follows, we update skb_orphan_frags_rx to the same as skb_orphan_frags to support zerocopy in the localhost test. cfg_notification_limit = 1, both method get notifications after 1 calling of sendmsg. In this case, the new method has around 17% cpu savings in TCP and 23% cpu savings in UDP. +---------------------+---------+---------+---------+---------+ | Test Type / Protocol| TCP v4 | TCP v6 | UDP v4 | UDP v6 | +---------------------+---------+---------+---------+---------+ | ZCopy (MB) | 7523 | 7706 | 7489 | 7304 | +---------------------+---------+---------+---------+---------+ | New ZCopy (MB) | 8834 | 8993 | 9053 | 9228 | +---------------------+---------+---------+---------+---------+ | New ZCopy / ZCopy | 117.42% | 116.70% | 120.88% | 126.34% | +---------------------+---------+---------+---------+---------+ cfg_notification_limit = 32, both get notifications after 32 calling of sendmsg, which means more chances to coalesce notifications, and less overhead of poll + recvmsg for the original method. In this case, the new method has around 7% cpu savings in TCP and slightly better cpu usage in UDP. In the env of selftest, notifications of TCP are more likely to be out of order than UDP, it's easier to coalesce more notifications in UDP. The original method can get one notification with range of 32 in a recvmsg most of the time. In TCP, most notifications' range is around 2, so the original method needs around 16 recvmsgs to get notified in one round. That's the reason for the "New ZCopy / ZCopy" diff in TCP and UDP here. +---------------------+---------+---------+---------+---------+ | Test Type / Protocol| TCP v4 | TCP v6 | UDP v4 | UDP v6 | +---------------------+---------+---------+---------+---------+ | ZCopy (MB) | 8842 | 8735 | 10072 | 9380 | +---------------------+---------+---------+---------+---------+ | New ZCopy (MB) | 9366 | 9477 | 10108 | 9385 | +---------------------+---------+---------+---------+---------+ | New ZCopy / ZCopy | 106.00% | 108.28% | 100.31% | 100.01% | +---------------------+---------+---------+---------+---------+ In conclusion, when notification interval is small or notifications are hard to be coalesced, the new mechanism is highly recommended. Otherwise, the performance gain from the new mechanism is very limited. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- tools/testing/selftests/net/msg_zerocopy.c | 101 ++++++++++++++++++-- tools/testing/selftests/net/msg_zerocopy.sh | 1 + 2 files changed, 95 insertions(+), 7 deletions(-) diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c index 7ea5fb28c93d..cf227f0011b5 100644 --- a/tools/testing/selftests/net/msg_zerocopy.c +++ b/tools/testing/selftests/net/msg_zerocopy.c @@ -66,6 +66,10 @@ #define SO_ZEROCOPY 60 #endif +#ifndef SCM_ZC_NOTIFICATION +#define SCM_ZC_NOTIFICATION 78 +#endif + #ifndef SO_EE_CODE_ZEROCOPY_COPIED #define SO_EE_CODE_ZEROCOPY_COPIED 1 #endif @@ -74,6 +78,14 @@ #define MSG_ZEROCOPY 0x4000000 #endif +#define ZC_INFO_ARR_SIZE (ZC_NOTIFICATION_MAX * sizeof(struct zc_info_elem)) +#define ZC_INFO_SIZE (sizeof(struct zc_info) + ZC_INFO_ARR_SIZE) + +enum notification_type { + MSG_ZEROCOPY_NOTIFY_ERRQUEUE = 1, + MSG_ZEROCOPY_NOTIFY_SENDMSG = 2, +}; + static int cfg_cork; static bool cfg_cork_mixed; static int cfg_cpu = -1; /* default: pin to last cpu */ @@ -86,7 +98,7 @@ static int cfg_runtime_ms = 4200; static int cfg_verbose; static int cfg_waittime_ms = 500; static int cfg_notification_limit = 32; -static bool cfg_zerocopy; +static enum notification_type cfg_zerocopy; static socklen_t cfg_alen; static struct sockaddr_storage cfg_dst_addr; @@ -97,6 +109,8 @@ static long packets, bytes, completions, expected_completions; static int zerocopied = -1; static uint32_t next_completion; static uint32_t sends_since_notify; +static char zc_ckbuf[CMSG_SPACE(ZC_INFO_SIZE)]; +static bool added_zcopy_info; static unsigned long gettimeofday_ms(void) { @@ -182,7 +196,26 @@ static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie) memcpy(CMSG_DATA(cm), &cookie, sizeof(cookie)); } -static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) +static void add_zcopy_info(struct msghdr *msg) +{ + struct zc_info *zc_info; + struct cmsghdr *cm; + + if (!msg->msg_control) + error(1, errno, "NULL user arg"); + cm = (struct cmsghdr *)msg->msg_control; + cm->cmsg_len = CMSG_LEN(ZC_INFO_SIZE); + cm->cmsg_level = SOL_SOCKET; + cm->cmsg_type = SCM_ZC_NOTIFICATION; + + zc_info = (struct zc_info *)CMSG_DATA(cm); + zc_info->size = ZC_NOTIFICATION_MAX; + + added_zcopy_info = true; +} + +static bool do_sendmsg(int fd, struct msghdr *msg, + enum notification_type do_zerocopy, int domain) { int ret, len, i, flags; static uint32_t cookie; @@ -200,6 +233,12 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) msg->msg_controllen = CMSG_SPACE(sizeof(cookie)); msg->msg_control = (struct cmsghdr *)ckbuf; add_zcopy_cookie(msg, ++cookie); + } else if (do_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG && + sends_since_notify + 1 >= cfg_notification_limit) { + memset(&msg->msg_control, 0, sizeof(msg->msg_control)); + msg->msg_controllen = CMSG_SPACE(ZC_INFO_SIZE); + msg->msg_control = (struct cmsghdr *)zc_ckbuf; + add_zcopy_info(msg); } } @@ -218,7 +257,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) if (do_zerocopy && ret) expected_completions++; } - if (do_zerocopy && domain == PF_RDS) { + if (msg->msg_control) { msg->msg_control = NULL; msg->msg_controllen = 0; } @@ -466,6 +505,44 @@ static void do_recv_completions(int fd, int domain) sends_since_notify = 0; } +static void do_recv_completions2(void) +{ + struct cmsghdr *cm = (struct cmsghdr *)zc_ckbuf; + struct zc_info *zc_info; + __u32 hi, lo, range; + __u8 zerocopy; + int i; + + zc_info = (struct zc_info *)CMSG_DATA(cm); + for (i = 0; i < zc_info->size; i++) { + hi = zc_info->arr[i].hi; + lo = zc_info->arr[i].lo; + zerocopy = zc_info->arr[i].zerocopy; + range = hi - lo + 1; + + if (cfg_verbose && lo != next_completion) + fprintf(stderr, "gap: %u..%u does not append to %u\n", + lo, hi, next_completion); + next_completion = hi + 1; + + if (zerocopied == -1) { + zerocopied = zerocopy; + } else if (zerocopied != zerocopy) { + fprintf(stderr, "serr: inconsistent\n"); + zerocopied = zerocopy; + } + + completions += range; + sends_since_notify -= range; + + if (cfg_verbose >= 2) + fprintf(stderr, "completed: %u (h=%u l=%u)\n", + range, hi, lo); + } + + added_zcopy_info = false; +} + /* Wait for all remaining completions on the errqueue */ static void do_recv_remaining_completions(int fd, int domain) { @@ -553,11 +630,16 @@ static void do_tx(int domain, int type, int protocol) else do_sendmsg(fd, &msg, cfg_zerocopy, domain); - if (cfg_zerocopy && sends_since_notify >= cfg_notification_limit) + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_ERRQUEUE && + sends_since_notify >= cfg_notification_limit) do_recv_completions(fd, domain); + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG && + added_zcopy_info) + do_recv_completions2(); + while (!do_poll(fd, POLLOUT)) { - if (cfg_zerocopy) + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_ERRQUEUE) do_recv_completions(fd, domain); } @@ -715,7 +797,7 @@ static void parse_opts(int argc, char **argv) cfg_payload_len = max_payload_len; - while ((c = getopt(argc, argv, "46c:C:D:i:l:mp:rs:S:t:vz")) != -1) { + while ((c = getopt(argc, argv, "46c:C:D:i:l:mnp:rs:S:t:vz")) != -1) { switch (c) { case '4': if (cfg_family != PF_UNSPEC) @@ -749,6 +831,9 @@ static void parse_opts(int argc, char **argv) case 'm': cfg_cork_mixed = true; break; + case 'n': + cfg_zerocopy = MSG_ZEROCOPY_NOTIFY_SENDMSG; + break; case 'p': cfg_port = strtoul(optarg, NULL, 0); break; @@ -768,7 +853,7 @@ static void parse_opts(int argc, char **argv) cfg_verbose++; break; case 'z': - cfg_zerocopy = true; + cfg_zerocopy = MSG_ZEROCOPY_NOTIFY_ERRQUEUE; break; } } @@ -779,6 +864,8 @@ static void parse_opts(int argc, char **argv) error(1, 0, "-D required for PF_RDS\n"); if (!cfg_rx && !saddr) error(1, 0, "-S required for PF_RDS\n"); + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG) + error(1, 0, "PF_RDS does not support ZC_NOTIF_SENDMSG"); } setup_sockaddr(cfg_family, daddr, &cfg_dst_addr); setup_sockaddr(cfg_family, saddr, &cfg_src_addr); diff --git a/tools/testing/selftests/net/msg_zerocopy.sh b/tools/testing/selftests/net/msg_zerocopy.sh index 89c22f5320e0..022a6936d86f 100755 --- a/tools/testing/selftests/net/msg_zerocopy.sh +++ b/tools/testing/selftests/net/msg_zerocopy.sh @@ -118,4 +118,5 @@ do_test() { do_test "${EXTRA_ARGS}" do_test "-z ${EXTRA_ARGS}" +do_test "-n ${EXTRA_ARGS}" echo ok