From patchwork Wed Jun 26 19:34:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13713307 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86B8E18FC6B for ; Wed, 26 Jun 2024 19:34:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719430451; cv=none; b=bqy06O/DgULYE/m4vHpNYRW6qMG9OT4YmJph4I97GhHKVRhZwAX6tA39IuCnVdJarXn8weOqx/2e3g/bHwcrAdAfIbi94l+NdFbIH8Qjlqme3cO/FK5pioWJrS4ckM8Dtts53tcrWEOvWZAgrwrOKBPJJ7+G2VUyuhozZuTLo4M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719430451; c=relaxed/simple; bh=UNwIgy0eaP1U/PjG+dxmb5r6nXFEuXCnnd8Z+VVWbpM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KRFyRhHPQGaO4SOnABGsOI+sBQTgKiX2S/Sx9yVFdW9BQuaX6a8oKg9NY2tgzMn2C2UY+Lw+4oVcG+D8r2FQ1W/Ri7bS/ISxZY9rFwALbQD0xuQ+yJaQBx67yce9S6p+2WmCjLzZi/QCwbFmNmfC63edrfLUtxf8EQnzOLEU/vw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LHN8wo1V; arc=none smtp.client-ip=209.85.128.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LHN8wo1V" Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-643efaf0786so39005337b3.1 for ; Wed, 26 Jun 2024 12:34:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1719430448; x=1720035248; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Pq1qBmFA90YYxxuCBnsZZeZJpNA0DQPz8KGklT3/y7Q=; b=LHN8wo1VcEHNEo69QA46DpLQIkA0Y/js1sdVcnyVP5S/qHk4vZnf9JEpXq+g+4M739 QE5ddl2k8XejcA8vgnnlR/9aqMnj7d3xLI+VrM9gPxzlB/XvejZzxy24+lWXgHGNW4fy ZDukkDOpJQ8wDiDgTAyINieKIHnih+sDJKP9iv3wxIntsVEbSQWbosDyJkPeg+y6VcYa eLEtDouJdPrVbyx8W4oBGDlJp1t21L6oDmDqO+iIL+j/tGLiwh4Wula+A1AkkU6vuEvP 8AZpjIf8xvExENBdAn357YTqfbjoUOylwTK09VtMtlbGLglho+e/r220k1gmINIfKd24 MP7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719430448; x=1720035248; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Pq1qBmFA90YYxxuCBnsZZeZJpNA0DQPz8KGklT3/y7Q=; b=XOYvBCbGpcJsL2HDKB0SypmfvbbSXpM+J+u95IuBS0ZERPyMEpEFVb4PXJOYd6GUt9 EE6xY5z0fA8b69dZfW4CXmWzFsCpxLZXsS1eHnfJ2taXmGRY2piCr2uKvqP/oPWSYWHC x1d9Ig6hgQTkBEj2S1fLQC0uyNVk0pd8G4XLVDsgzS1JjLu1eW8V/eDy88zTvAjDOOm/ A1psqPv9d6nUSYym5oTx3OZTmc470YKBi+jB2XFlVkbsrUjy6apyBf1zkNX5OZFcZmui 9WdiPzGXVpWB8/gAbv5KzXed20xm1g770qUvnUcAulSe3H5Mma7jn8b7XKIMlcs5uRbP KZDA== X-Gm-Message-State: AOJu0YzgE09AxeiVUjz5nJ5R9CDEMWKTJAdvTGFEBWV2dAvgm9C/Djfl cgOBPF9TiolljXlYmEN/QedpWTOzLmm1kqJbJO9ziku5ZJ+mfFaPbeupr7szueFRFBI8joe8DDy V X-Google-Smtp-Source: AGHT+IHFauWbFT7DxMYusahgE5j/+KdBP0c4hHDoJtmihacmCcD9D8vHxir9r89/G2anNjZgxQcPpQ== X-Received: by 2002:a05:690c:3747:b0:62c:dcb2:a75b with SMTP id 00721157ae682-64342d9ab69mr100195917b3.38.1719430448148; Wed, 26 Jun 2024 12:34:08 -0700 (PDT) Received: from n191-036-066.byted.org ([130.44.212.94]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b53df48dedsm40112286d6.67.2024.06.26.12.34.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 12:34:07 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v6 1/4] selftests: fix OOM problem in msg_zerocopy selftest Date: Wed, 26 Jun 2024 19:34:00 +0000 Message-Id: <20240626193403.3854451-2-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240626193403.3854451-1-zijianzhang@bytedance.com> References: <20240626193403.3854451-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang In selftests/net/msg_zerocopy.c, it has a while loop keeps calling sendmsg on a socket with MSG_ZEROCOPY flag, and it will recv the notifications until the socket is not writable. Typically, it will start the receiving process after around 30+ sendmsgs. However, because of the commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale") the sender is always writable and does not get any chance to run recv notifications. The selftest always exits with OUT_OF_MEMORY because the memory used by opt_skb exceeds the core.sysctl_optmem_max. According to our experiments, this problem can be mitigated by open the DEBUG_LOCKDEP configuration for the kernel. But it will makes the notifications disordered even in good commits before commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale"). We introduce "cfg_notification_limit" to force sender to receive notifications after some number of sendmsgs. And, notifications may not come in order, because of the reason we present above. We have order checking code managed by cfg_verbose. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- tools/testing/selftests/net/msg_zerocopy.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c index bdc03a2097e8..7ea5fb28c93d 100644 --- a/tools/testing/selftests/net/msg_zerocopy.c +++ b/tools/testing/selftests/net/msg_zerocopy.c @@ -85,6 +85,7 @@ static bool cfg_rx; static int cfg_runtime_ms = 4200; static int cfg_verbose; static int cfg_waittime_ms = 500; +static int cfg_notification_limit = 32; static bool cfg_zerocopy; static socklen_t cfg_alen; @@ -95,6 +96,7 @@ static char payload[IP_MAXPACKET]; static long packets, bytes, completions, expected_completions; static int zerocopied = -1; static uint32_t next_completion; +static uint32_t sends_since_notify; static unsigned long gettimeofday_ms(void) { @@ -208,6 +210,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) error(1, errno, "send"); if (cfg_verbose && ret != len) fprintf(stderr, "send: ret=%u != %u\n", ret, len); + sends_since_notify++; if (len) { packets++; @@ -435,7 +438,7 @@ static bool do_recv_completion(int fd, int domain) /* Detect notification gaps. These should not happen often, if at all. * Gaps can occur due to drops, reordering and retransmissions. */ - if (lo != next_completion) + if (cfg_verbose && lo != next_completion) fprintf(stderr, "gap: %u..%u does not append to %u\n", lo, hi, next_completion); next_completion = hi + 1; @@ -460,6 +463,7 @@ static bool do_recv_completion(int fd, int domain) static void do_recv_completions(int fd, int domain) { while (do_recv_completion(fd, domain)) {} + sends_since_notify = 0; } /* Wait for all remaining completions on the errqueue */ @@ -549,6 +553,9 @@ static void do_tx(int domain, int type, int protocol) else do_sendmsg(fd, &msg, cfg_zerocopy, domain); + if (cfg_zerocopy && sends_since_notify >= cfg_notification_limit) + do_recv_completions(fd, domain); + while (!do_poll(fd, POLLOUT)) { if (cfg_zerocopy) do_recv_completions(fd, domain); @@ -708,7 +715,7 @@ static void parse_opts(int argc, char **argv) cfg_payload_len = max_payload_len; - while ((c = getopt(argc, argv, "46c:C:D:i:mp:rs:S:t:vz")) != -1) { + while ((c = getopt(argc, argv, "46c:C:D:i:l:mp:rs:S:t:vz")) != -1) { switch (c) { case '4': if (cfg_family != PF_UNSPEC) @@ -736,6 +743,9 @@ static void parse_opts(int argc, char **argv) if (cfg_ifindex == 0) error(1, errno, "invalid iface: %s", optarg); break; + case 'l': + cfg_notification_limit = strtoul(optarg, NULL, 0); + break; case 'm': cfg_cork_mixed = true; break; From patchwork Wed Jun 26 19:34:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13713308 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-ot1-f46.google.com (mail-ot1-f46.google.com [209.85.210.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D09BB19149A for ; Wed, 26 Jun 2024 19:34:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719430452; cv=none; b=LMzg6qwjQwCHPdkZ2tlxpjEHOZIFR21jeHQ9ZDj/1MKxwpCGph1+AFpmyRYyHFoB4qRXq82wTlQq/6xmCWFqDgf7xj+bupF02DbTkvlcqQ8Bg8RE6OEdouhPvtbhoQ6blYRCQ1UdlkHl1CPXiVr42FOzRz4VpuFTdU1l0MmJ9ac= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719430452; c=relaxed/simple; bh=QvsyaMOFCyqiKKN4mxhDmXGb9oJ+DFav1kcdjknZmMc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nb1HgQ63C0a8Q8kdptS8gcjxl7LSXXwbYhhVVysQp8wsqXd63z8o8Nv1TcRFuAXHL5ooGPK8yT/9L+rYY9pB7IgHRJKYy6uYBPfEIXsw8TGYLWw07Ba3D+R2eKan0TiZTiYacjEGHZt6i9g1R53JmyLePJ0bEKxJDGP1615xqyI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ZhR07o6w; arc=none smtp.client-ip=209.85.210.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ZhR07o6w" Received: by mail-ot1-f46.google.com with SMTP id 46e09a7af769-6f9a4f9923aso3838966a34.0 for ; Wed, 26 Jun 2024 12:34:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1719430449; x=1720035249; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zU2+TnVnaRnjEbMWnSevTqs6sTsVoRAJHOXUXgLcrc4=; b=ZhR07o6wEQWbGaeSIn17rzoUAuX3jaTSHENyT2+SMbGcosssDEYlGCo9rf1aHsyJnK to9v8x4ij2AhweMhi5ElotkU3z9/mznBpkLuZqm+KEJGYNGrLulktjHuZ446yrc0EgZf VWt3H6TmcVhHW4W1mCzAumMwEk60xg0VQkqWIfSgAqorxkW3voNlLZHY1j/09IEHRtzo ZbhaZBJ1ganUhiczta+pdRoOO+2uPQQatBLyf9s21LGHHpGvJDCnlpL56YXPIHjOmm93 cpCTqYy6hDidD05c9UHCYSymxBznZkpUUdTX63nnO/URuc/81E06zNK7LZI/MvYN4yc9 k59g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719430449; x=1720035249; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zU2+TnVnaRnjEbMWnSevTqs6sTsVoRAJHOXUXgLcrc4=; b=HAe4EPM1f4M4UNeZ9TE/v3hFyX7UqPwVAoiGTdow/fGrIOQqX+0B5VBzzjTbHECjy2 RwD2Nxpky3DLxNE4kPOUTzEYl1LtRSSUCRP8d6QsqN88ISE2OZBoLBgxKoxb80qtqPyb fGt0euRTWsmpCHsLccU5idJ66kmYi6khHOSz6oWyEgbh3lj+JyP5onSVGczieBlIJRRA +LFkoGXIgglDdyuTfmZJVUtVXcCF1hpcttUIHJAcx3XKonB4gNTIbKiJP+5bRysNyhh2 QeztpCQJ6g8PIkAQTkTS7hkNwtpEnIoPs+REXZfDTyrqrY1Ww8VAqgNrXSjIP3A5jD7t WbNw== X-Gm-Message-State: AOJu0Yz7lcpAp6N2RpAusxsEILEUY4osCjytIj3nrittjJxsrhwXWs+m WSMLWHFBtJQflg5tJM69mV2X0y+uAeTlJT+nAMyffSnjZnN6CO8Baibx8PBMQFKTwh5XD4vwaz1 f X-Google-Smtp-Source: AGHT+IH2lMBNsKrXm2dzagRmjTycs4dJ65V2hTmwp51cUNRCgQFXqTZqcA45S9hYNCqK5duhqt9hZg== X-Received: by 2002:a9d:6b03:0:b0:700:d67c:9bc8 with SMTP id 46e09a7af769-700d67c9e6bmr2039484a34.10.1719430449477; Wed, 26 Jun 2024 12:34:09 -0700 (PDT) Received: from n191-036-066.byted.org ([130.44.212.94]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b53df48dedsm40112286d6.67.2024.06.26.12.34.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 12:34:09 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v6 2/4] sock: support copy cmsg to userspace in TX path Date: Wed, 26 Jun 2024 19:34:01 +0000 Message-Id: <20240626193403.3854451-3-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240626193403.3854451-1-zijianzhang@bytedance.com> References: <20240626193403.3854451-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang Since ____sys_sendmsg creates a kernel copy of msg_control and passes that to the callees, put_cmsg will write into this kernel buffer. If people want to piggyback some information like timestamps upon returning of sendmsg. ____sys_sendmsg will have to copy_to_user to the original buf, which is not supported. As a result, users typically have to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system call overhead. This commit supports copying cmsg to userspace in TX path by introducing a flag MSG_CMSG_COPY_TO_USER in struct msghdr to guide the copy logic upon returning of ___sys_sendmsg. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- include/linux/socket.h | 6 ++++++ net/core/sock.c | 2 ++ net/ipv4/ip_sockglue.c | 2 ++ net/ipv6/datagram.c | 3 +++ net/socket.c | 45 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 58 insertions(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 89d16b90370b..35adc30c9db6 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -168,6 +168,11 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr return __cmsg_nxthdr(__msg->msg_control, __msg->msg_controllen, __cmsg); } +static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg) +{ + return 0; +} + static inline size_t msg_data_left(struct msghdr *msg) { return iov_iter_count(&msg->msg_iter); @@ -329,6 +334,7 @@ struct ucred { #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ #define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ +#define MSG_CMSG_COPY_TO_USER 0x10000000 /* Copy cmsg to user space */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file descriptor received through diff --git a/net/core/sock.c b/net/core/sock.c index 9abc4fe25953..4a766a91ff5c 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2879,6 +2879,8 @@ int sock_cmsg_send(struct sock *sk, struct msghdr *msg, for_each_cmsghdr(cmsg, msg) { if (!CMSG_OK(msg, cmsg)) return -EINVAL; + if (cmsg_copy_to_user(cmsg)) + msg->msg_flags |= MSG_CMSG_COPY_TO_USER; if (cmsg->cmsg_level != SOL_SOCKET) continue; ret = __sock_cmsg_send(sk, cmsg, sockc); diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c index cf377377b52d..464d08b27fa8 100644 --- a/net/ipv4/ip_sockglue.c +++ b/net/ipv4/ip_sockglue.c @@ -249,6 +249,8 @@ int ip_cmsg_send(struct sock *sk, struct msghdr *msg, struct ipcm_cookie *ipc, for_each_cmsghdr(cmsg, msg) { if (!CMSG_OK(msg, cmsg)) return -EINVAL; + if (cmsg_copy_to_user(cmsg)) + msg->msg_flags |= MSG_CMSG_COPY_TO_USER; #if IS_ENABLED(CONFIG_IPV6) if (allow_ipv6 && cmsg->cmsg_level == SOL_IPV6 && diff --git a/net/ipv6/datagram.c b/net/ipv6/datagram.c index fff78496803d..b0341faf7f83 100644 --- a/net/ipv6/datagram.c +++ b/net/ipv6/datagram.c @@ -776,6 +776,9 @@ int ip6_datagram_send_ctl(struct net *net, struct sock *sk, goto exit_f; } + if (cmsg_copy_to_user(cmsg)) + msg->msg_flags |= MSG_CMSG_COPY_TO_USER; + if (cmsg->cmsg_level == SOL_SOCKET) { err = __sock_cmsg_send(sk, cmsg, &ipc6->sockc); if (err) diff --git a/net/socket.c b/net/socket.c index e416920e9399..6523cf5a7f32 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2621,6 +2621,39 @@ static int sendmsg_copy_msghdr(struct msghdr *msg, return 0; } +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys, + struct user_msghdr __user *umsg) +{ + struct compat_msghdr __user *umsg_compat = + (struct compat_msghdr __user *)umsg; + unsigned long cmsg_ptr = (unsigned long)umsg->msg_control; + unsigned int flags = msg_sys->msg_flags; + struct msghdr msg_user = *msg_sys; + struct cmsghdr *cmsg; + int err; + + msg_user.msg_control = umsg->msg_control; + msg_user.msg_control_is_user = true; + for_each_cmsghdr(cmsg, msg_sys) { + if (!CMSG_OK(msg_sys, cmsg)) + break; + if (cmsg_copy_to_user(cmsg)) + put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type, + cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg)); + } + + err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), COMPAT_FLAGS(umsg)); + if (err) + return err; + if (MSG_CMSG_COMPAT & flags) + err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr, + &umsg_compat->msg_controllen); + else + err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr, + &umsg->msg_controllen); + return err; +} + static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg, struct msghdr *msg_sys, unsigned int flags, struct used_address *used_address, @@ -2638,6 +2671,18 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg, err = ____sys_sendmsg(sock, msg_sys, flags, used_address, allowed_msghdr_flags); + if (err < 0) + goto out; + + if (msg_sys->msg_flags & MSG_CMSG_COPY_TO_USER) { + ssize_t len = err; + + err = sendmsg_copy_cmsg_to_user(msg_sys, msg); + if (err) + goto out; + err = len; + } +out: kfree(iov); return err; } From patchwork Wed Jun 26 19:34:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13713309 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 414831922FB for ; Wed, 26 Jun 2024 19:34:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719430455; cv=none; b=uDhfJ+QS9WT2D4tVOZTqPjk+3xd0M/IP5TmQasYmg4Z2UZVZKtbP7TFcM3/pU8x04O543HnLN93LVH44LgOGxUzaEBx1yQM5s6yd+HSPMjDGjO7rN5K1dFIiQe0xb9gInVmiHysQvoUmv9KIt2+O9Eu8FgBX2AtWmnb6rymgzoQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719430455; c=relaxed/simple; bh=FEWr2EERd6GJQ+Iezz/TtEgJb3LtqCl8v9s45kJ6oo0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RE51KRmm0AYW4KZDBN4e8XbhP0rbTXAueTUFtYio/14jw9oSHzU3lSpymvcd1MI8QPvSOkESaBa5bD8XAWab+8L7dWLT1dXKQvY+EeK3wbTqDypBYh5IEZi8mPvVT9sQ/no2nJ2j13x3mPpu4UXevwT4y4DUkZnsGlao6xe2S9Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=IDJCtfN7; arc=none smtp.client-ip=209.85.219.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="IDJCtfN7" Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-6b4febddfd3so32682566d6.1 for ; Wed, 26 Jun 2024 12:34:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1719430451; x=1720035251; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t4AFgorcTiGvVwLshUYUFIPxmEyC/yndHcuM4/2TN4s=; b=IDJCtfN7hslL+RF78o7U03W56M27UYkBsBvq88rU2GJO5luN5ukjWj13SpSq4Wudfj YI6nRomU4M9i0QevLrH2/W2sLzE9yKP8x+oSWwPY7H3b1CXpIu4hUoPG+/jAgL++US7y ci+cVSgRo7FKZ6zNrk02z8Z6PCcHq/otKkOd7tQKespka/h7e/AZkuaeaRW+eYYVpPXc CnwOWwhS4lbfzQNRuk6NNBrRXtvAhV6edJt7cnOOE2XFgI2iXmNlXUQIS+i1QVFTTpGb KDgBRlOBJsqjtf5qQWSxAQTNL4wDSbKmBzb6Iy2LbGzHA6XN6viSN38I/Mprk2W8CLdH coIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719430451; x=1720035251; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t4AFgorcTiGvVwLshUYUFIPxmEyC/yndHcuM4/2TN4s=; b=EWt3AR4k6OYflrutI2oFd5tBrH7A0KNwK1n+vEZibRQ3zmoVWwODopoBtceGOeZj3W MFHd6Qg74k4zU0WQT2K7TFKbJPwh/PIQXb1FWEprRNIWWHftkwV3aRe8IZtnLh4PaRGe ybMi1im89VbuxcubuSkY/heP3UCQOdS8oOTzMin24qNi7umB/beUZTVWhA2iI3opTmid Njo8jhIwqG3nawUCvFDS1kSyha5qdStnRTc6BJ8ygbRVi9FCEBm2Nwz7dLOs1PbBDhKi yS7clsvU/nHAbvd7b9A1nrkuxzpeUawfiz0Ar/tqeqaMOkPdkXK0XMUnLEoHgiVqt57k ITVw== X-Gm-Message-State: AOJu0YwShAE5T7h3oat6B0BT2rEB1uXn0SBiFpiMlYWOgvRDNSst4zOO nNLz3bzgQkzmutG9DOwkOvnF/h+BD86kuTcDuOrzx52xk4l9P4wQv2zAu75dQslryEKLTixucbP u X-Google-Smtp-Source: AGHT+IEaXmNUcIXz5KNaEjd7nPiH0XLdNTKiPxzOD4W9OzzBbZJL7p3Uj47RxnBHhK4tQ7L2Atxw2Q== X-Received: by 2002:a0c:e910:0:b0:6b5:4aa9:9682 with SMTP id 6a1803df08f44-6b54aa99d93mr89762796d6.41.1719430451043; Wed, 26 Jun 2024 12:34:11 -0700 (PDT) Received: from n191-036-066.byted.org ([130.44.212.94]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b53df48dedsm40112286d6.67.2024.06.26.12.34.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 12:34:10 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v6 3/4] sock: add MSG_ZEROCOPY notification mechanism based on msg_control Date: Wed, 26 Jun 2024 19:34:02 +0000 Message-Id: <20240626193403.3854451-4-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240626193403.3854451-1-zijianzhang@bytedance.com> References: <20240626193403.3854451-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang The MSG_ZEROCOPY flag enables copy avoidance for socket send calls. However, zerocopy is not a free lunch. Apart from the management of user pages, the combination of poll + recvmsg to receive notifications incurs unignorable overhead in the applications. The overhead of such sometimes might be more than the CPU savings from zerocopy. We try to solve this problem with a new notification mechanism based on msgcontrol. This new mechanism aims to reduce the overhead associated with receiving notifications by embedding them directly into user arguments passed with each sendmsg control message. By doing so, we can significantly reduce the complexity and overhead for managing notifications. In an ideal pattern, the user will keep calling sendmsg with SCM_ZC_NOTIFICATION msg_control, and the notification will be delivered as soon as possible. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- arch/alpha/include/uapi/asm/socket.h | 2 ++ arch/mips/include/uapi/asm/socket.h | 2 ++ arch/parisc/include/uapi/asm/socket.h | 2 ++ arch/sparc/include/uapi/asm/socket.h | 2 ++ include/linux/socket.h | 2 +- include/uapi/asm-generic/socket.h | 2 ++ include/uapi/linux/socket.h | 10 +++++++ net/core/sock.c | 42 +++++++++++++++++++++++++++ 8 files changed, 63 insertions(+), 1 deletion(-) diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi/asm/socket.h index e94f621903fe..7761a4e0ea2c 100644 --- a/arch/alpha/include/uapi/asm/socket.h +++ b/arch/alpha/include/uapi/asm/socket.h @@ -140,6 +140,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/asm/socket.h index 60ebaed28a4c..89edc51380f0 100644 --- a/arch/mips/include/uapi/asm/socket.h +++ b/arch/mips/include/uapi/asm/socket.h @@ -151,6 +151,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/uapi/asm/socket.h index be264c2b1a11..2911b43e6a9d 100644 --- a/arch/parisc/include/uapi/asm/socket.h +++ b/arch/parisc/include/uapi/asm/socket.h @@ -132,6 +132,8 @@ #define SO_PASSPIDFD 0x404A #define SO_PEERPIDFD 0x404B +#define SCM_ZC_NOTIFICATION 0x404C + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi/asm/socket.h index 682da3714686..dc045e87cc8e 100644 --- a/arch/sparc/include/uapi/asm/socket.h +++ b/arch/sparc/include/uapi/asm/socket.h @@ -133,6 +133,8 @@ #define SO_PASSPIDFD 0x0055 #define SO_PEERPIDFD 0x0056 +#define SCM_ZC_NOTIFICATION 0x0057 + #if !defined(__KERNEL__) diff --git a/include/linux/socket.h b/include/linux/socket.h index 35adc30c9db6..f2f013166525 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -170,7 +170,7 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg) { - return 0; + return __cmsg->cmsg_type == SCM_ZC_NOTIFICATION; } static inline size_t msg_data_left(struct msghdr *msg) diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h index 8ce8a39a1e5f..7474c8a244bc 100644 --- a/include/uapi/asm-generic/socket.h +++ b/include/uapi/asm-generic/socket.h @@ -135,6 +135,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__)) diff --git a/include/uapi/linux/socket.h b/include/uapi/linux/socket.h index d3fcd3b5ec53..26bee6291c6c 100644 --- a/include/uapi/linux/socket.h +++ b/include/uapi/linux/socket.h @@ -2,6 +2,8 @@ #ifndef _UAPI_LINUX_SOCKET_H #define _UAPI_LINUX_SOCKET_H +#include + /* * Desired design of maximum size and alignment (see RFC2553) */ @@ -35,4 +37,12 @@ struct __kernel_sockaddr_storage { #define SOCK_TXREHASH_DISABLED 0 #define SOCK_TXREHASH_ENABLED 1 +#define SOCK_ZC_INFO_MAX 16 + +struct zc_info_elem { + __u32 lo; + __u32 hi; + __u8 zerocopy; +}; + #endif /* _UAPI_LINUX_SOCKET_H */ diff --git a/net/core/sock.c b/net/core/sock.c index 4a766a91ff5c..1b2ce72e1338 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2863,6 +2863,48 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg, case SCM_RIGHTS: case SCM_CREDENTIALS: break; + case SCM_ZC_NOTIFICATION: { + struct zc_info_elem *zc_info_kern = CMSG_DATA(cmsg); + int cmsg_data_len, zc_info_elem_num; + struct sock_exterr_skb *serr; + struct sk_buff_head *q; + unsigned long flags; + struct sk_buff *skb; + int i = 0; + + if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS) + return -EINVAL; + + cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr); + if (cmsg_data_len % sizeof(struct zc_info_elem)) + return -EINVAL; + + zc_info_elem_num = cmsg_data_len / sizeof(struct zc_info_elem); + if (!zc_info_elem_num || zc_info_elem_num > SOCK_ZC_INFO_MAX) + return -EINVAL; + + q = &sk->sk_error_queue; + spin_lock_irqsave(&q->lock, flags); + skb = skb_peek(q); + while (skb && i < zc_info_elem_num) { + struct sk_buff *skb_next = skb_peek_next(skb, q); + + serr = SKB_EXT_ERR(skb); + if (serr->ee.ee_errno == 0 && + serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) { + zc_info_kern[i].hi = serr->ee.ee_data; + zc_info_kern[i].lo = serr->ee.ee_info; + zc_info_kern[i].zerocopy = !(serr->ee.ee_code + & SO_EE_CODE_ZEROCOPY_COPIED); + __skb_unlink(skb, q); + consume_skb(skb); + } + skb = skb_next; + i++; + } + spin_unlock_irqrestore(&q->lock, flags); + break; + } default: return -EINVAL; } From patchwork Wed Jun 26 19:34:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13713310 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A141E192B66 for ; Wed, 26 Jun 2024 19:34:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719430457; cv=none; b=ZZucQn8yjEbyb9sZr983DKHaJ/XtVaCSAJSGbEhAe4wmwnJiyYl+M5FGLjqGeOwIArBj53Kg7QuVVk21rVKPiVOg5RBZwz8zPkDQH+mG66EuDh/nNm0wcvgdF5XVNOlD4y26wgbEN/Fz9yyv/oqOvNQwx0QN6boIBF2aKUfnS6U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719430457; c=relaxed/simple; bh=l759AUoZn+Lkd8NE19BHlW8jwuV/0b0Hmv2kt2g0v6E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=uuJRWfxCnpmjF5KF5j+IeN+/ATWFObCzyhVg5LRQQREMCybzSukgHIUBAlUxSNUOubuEPYw/Mxfp21WNEojdecBnv2JE43kzKbt9sQsqz6g+pNUR+Ax1Ym4gMJlETAd7vTlHH+5IgU4r5XwMHMGTCopRzLBBEIZRPKWajpSGXko= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=GOhFnTTU; arc=none smtp.client-ip=209.85.222.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="GOhFnTTU" Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-79c05313eb5so205914285a.1 for ; Wed, 26 Jun 2024 12:34:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1719430454; x=1720035254; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0xJmOuJxFnxzNhLDPuw3aSaK2S5jGXnhr5CJos9rvB4=; b=GOhFnTTUI/Gu27IBE9NUpgDE+m7WdrOyag0flmVsvH9BMRT3tSFtwiUsVtAfq2/OtI /UAlUOKpdmLpBhVSoMoLsNK4OU/2v3g7LC1cp+JW5F5ryaPlSFd2nU5cOLKYc5/ul/Na SComuK6uM+abssL+jzLTTIdzwCp9JrDrJD71WDrv22lOwkbflIDaSnUURvSoLLYPH3CP k6XcJHOj950pS6S/g4cGwKu4ja3LUNCM0U+pSqvgTGyvQHNGIwc4nIzsIdQ2gdu2XiPy ZN6pTP9dfu4TIyea6FobBUMbkmFabRnsJjdLpe2hFdo1w0nG+SPhIA7mvyYKpijPyNim wQPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719430454; x=1720035254; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0xJmOuJxFnxzNhLDPuw3aSaK2S5jGXnhr5CJos9rvB4=; b=Gdbbj0A2Aa1zrHWZrqOxAWxZsOfIrFRpKjJdOdumeORbd/N3AtDXmpL6qS8TP9wo01 AeC+giJNHfM4t2J0M2AhRzl/xoTYwtlc5Kc04LzDeYK6ZBoNaqr2C4osSjiRYIGNi9r4 b/xl4BELRRRzh87CRAQwgt9WhfsZHs2GSdNKIfyzyeMfzeW6uaQv617KLTskttEHQNTQ T5J7BTeIEsXisXn96lWfoNEAel1KcYK34L0BHb3nTjR1D5k4+r3EuFkC6eKL/fXcdCK5 j3nY4eXG1irwWbYtG5Ry66c1v/3YdghC/oKEtzJIzkwJUzZQdGPFlezBj7BPniVYRpm3 DnTw== X-Gm-Message-State: AOJu0YwsvKENjdL0Uyx/oWMkPRvHsYUmpKh/W/U2axj/Cpb+y5/lu+0d +G1WaK58Ksk7fZkXQgnmtt4Y7Mr0KCbaOJrPAn7fc/3ncl4xqfQEx1LGR8NS5Q0ZK88bN6Jtk7r S X-Google-Smtp-Source: AGHT+IEeKGI8ybjs8fMAQf/A2RMj8JjBS4dN3qtnQfy7KXN/aUFvxweFG/PEzO5TjIGXZ/U5w5UHUA== X-Received: by 2002:ad4:5ecc:0:b0:6b4:ff5d:3ca with SMTP id 6a1803df08f44-6b540aa8c39mr144269686d6.40.1719430452987; Wed, 26 Jun 2024 12:34:12 -0700 (PDT) Received: from n191-036-066.byted.org ([130.44.212.94]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b53df48dedsm40112286d6.67.2024.06.26.12.34.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 12:34:12 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v6 4/4] selftests: add MSG_ZEROCOPY msg_control notification test Date: Wed, 26 Jun 2024 19:34:03 +0000 Message-Id: <20240626193403.3854451-5-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240626193403.3854451-1-zijianzhang@bytedance.com> References: <20240626193403.3854451-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang We update selftests/net/msg_zerocopy.c to accommodate the new mechanism. Test result from selftests/net/msg_zerocopy.c, cfg_notification_limit = 1, in this case the original method approximately aligns with the semantics of new one. In this case, the new flag has around 13% cpu savings in TCP and 18% cpu savings in UDP. +---------------------+---------+---------+---------+---------+ | Test Type / Protocol| TCP v4 | TCP v6 | UDP v4 | UDP v6 | +---------------------+---------+---------+---------+---------+ | ZCopy (MB) | 5147 | 4885 | 7489 | 7854 | +---------------------+---------+---------+---------+---------+ | New ZCopy (MB) | 5859 | 5505 | 9053 | 9236 | +---------------------+---------+---------+---------+---------+ | New ZCopy / ZCopy | 113.83% | 112.69% | 120.88% | 117.59% | +---------------------+---------+---------+---------+---------+ cfg_notification_limit = 32, it means less poll + recvmsg overhead, the new mechanism performs 8% better in TCP. For UDP, no obvious performance gain is observed and sometimes may lead to degradation. Thus, if users don't need to retrieve the notification ASAP in UDP, the original mechanism is preferred. +---------------------+---------+---------+---------+---------+ | Test Type / Protocol| TCP v4 | TCP v6 | UDP v4 | UDP v6 | +---------------------+---------+---------+---------+---------+ | ZCopy (MB) | 6272 | 6138 | 12138 | 10055 | +---------------------+---------+---------+---------+---------+ | New ZCopy (MB) | 6774 | 6620 | 11504 | 10355 | +---------------------+---------+---------+---------+---------+ | New ZCopy / ZCopy | 108.00% | 107.85% | 94.78% | 102.98% | +---------------------+---------+---------+---------+---------+ Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- tools/testing/selftests/net/msg_zerocopy.c | 109 ++++++++++++++++++-- tools/testing/selftests/net/msg_zerocopy.sh | 1 + 2 files changed, 102 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c index 7ea5fb28c93d..b8a1002aa6ae 100644 --- a/tools/testing/selftests/net/msg_zerocopy.c +++ b/tools/testing/selftests/net/msg_zerocopy.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* Evaluate MSG_ZEROCOPY * * Send traffic between two processes over one of the supported @@ -66,6 +67,10 @@ #define SO_ZEROCOPY 60 #endif +#ifndef SCM_ZC_NOTIFICATION +#define SCM_ZC_NOTIFICATION 78 +#endif + #ifndef SO_EE_CODE_ZEROCOPY_COPIED #define SO_EE_CODE_ZEROCOPY_COPIED 1 #endif @@ -74,6 +79,15 @@ #define MSG_ZEROCOPY 0x4000000 #endif +enum notification_type { + MSG_ZEROCOPY_NOTIFY_ERRQUEUE = 1, + MSG_ZEROCOPY_NOTIFY_SENDMSG = 2, +}; + +#define INVALID_ZEROCOPY_VAL 2 + +#define ZC_NOTIF_ARR_SZ (sizeof(struct zc_info_elem) * SOCK_ZC_INFO_MAX) + static int cfg_cork; static bool cfg_cork_mixed; static int cfg_cpu = -1; /* default: pin to last cpu */ @@ -85,14 +99,16 @@ static bool cfg_rx; static int cfg_runtime_ms = 4200; static int cfg_verbose; static int cfg_waittime_ms = 500; -static int cfg_notification_limit = 32; -static bool cfg_zerocopy; +static int cfg_notification_limit = 16; +static enum notification_type cfg_zerocopy; static socklen_t cfg_alen; static struct sockaddr_storage cfg_dst_addr; static struct sockaddr_storage cfg_src_addr; static char payload[IP_MAXPACKET]; +static char zc_ckbuf[CMSG_SPACE(ZC_NOTIF_ARR_SZ)]; +static bool added_zcopy_info; static long packets, bytes, completions, expected_completions; static int zerocopied = -1; static uint32_t next_completion; @@ -169,6 +185,25 @@ static int do_accept(int fd) return fd; } +static void add_zcopy_info(struct msghdr *msg) +{ + int i; + struct cmsghdr *cm; + struct zc_info_elem *zc_info; + + if (!msg->msg_control) + error(1, errno, "NULL user arg"); + cm = (struct cmsghdr *)msg->msg_control; + zc_info = (struct zc_info_elem *)CMSG_DATA(cm); + + cm->cmsg_len = CMSG_LEN(ZC_NOTIF_ARR_SZ); + cm->cmsg_level = SOL_SOCKET; + cm->cmsg_type = SCM_ZC_NOTIFICATION; + for (i = 0; i < SOCK_ZC_INFO_MAX; i++) + zc_info[i].zerocopy = INVALID_ZEROCOPY_VAL; + added_zcopy_info = true; +} + static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie) { struct cmsghdr *cm; @@ -182,7 +217,8 @@ static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie) memcpy(CMSG_DATA(cm), &cookie, sizeof(cookie)); } -static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) +static bool do_sendmsg(int fd, struct msghdr *msg, + enum notification_type do_zerocopy, int domain) { int ret, len, i, flags; static uint32_t cookie; @@ -200,6 +236,12 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) msg->msg_controllen = CMSG_SPACE(sizeof(cookie)); msg->msg_control = (struct cmsghdr *)ckbuf; add_zcopy_cookie(msg, ++cookie); + } else if (do_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG && + sends_since_notify >= cfg_notification_limit) { + memset(&msg->msg_control, 0, sizeof(msg->msg_control)); + msg->msg_controllen = sizeof(zc_ckbuf); + msg->msg_control = (struct cmsghdr *)zc_ckbuf; + add_zcopy_info(msg); } } @@ -218,7 +260,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) if (do_zerocopy && ret) expected_completions++; } - if (do_zerocopy && domain == PF_RDS) { + if (msg->msg_control) { msg->msg_control = NULL; msg->msg_controllen = 0; } @@ -392,6 +434,48 @@ static bool do_recvmsg_completion(int fd) return ret; } +static void do_recv_completions2(void) +{ + int i; + __u32 hi, lo, range; + __u8 zerocopy; + struct cmsghdr *cm = (struct cmsghdr *)zc_ckbuf; + struct zc_info_elem *zc_info = (struct zc_info_elem *)CMSG_DATA(cm); + + if (!added_zcopy_info) + return; + + added_zcopy_info = false; + for (i = 0; i < SOCK_ZC_INFO_MAX && zc_info[i].zerocopy != INVALID_ZEROCOPY_VAL; i++) { + struct zc_info_elem elem = zc_info[i]; + + hi = elem.hi; + lo = elem.lo; + zerocopy = elem.zerocopy; + range = hi - lo + 1; + + if (cfg_verbose && lo != next_completion) + fprintf(stderr, "gap: %u..%u does not append to %u\n", + lo, hi, next_completion); + next_completion = hi + 1; + + if (zerocopied == -1) + zerocopied = zerocopy; + else if (zerocopied != zerocopy) { + fprintf(stderr, "serr: inconsistent\n"); + zerocopied = zerocopy; + } + + completions += range; + + if (cfg_verbose >= 2) + fprintf(stderr, "completed: %u (h=%u l=%u)\n", + range, hi, lo); + } + + sends_since_notify -= i; +} + static bool do_recv_completion(int fd, int domain) { struct sock_extended_err *serr; @@ -553,11 +637,15 @@ static void do_tx(int domain, int type, int protocol) else do_sendmsg(fd, &msg, cfg_zerocopy, domain); - if (cfg_zerocopy && sends_since_notify >= cfg_notification_limit) + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_ERRQUEUE && + sends_since_notify >= cfg_notification_limit) do_recv_completions(fd, domain); + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG) + do_recv_completions2(); + while (!do_poll(fd, POLLOUT)) { - if (cfg_zerocopy) + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_ERRQUEUE) do_recv_completions(fd, domain); } @@ -715,7 +803,7 @@ static void parse_opts(int argc, char **argv) cfg_payload_len = max_payload_len; - while ((c = getopt(argc, argv, "46c:C:D:i:l:mp:rs:S:t:vz")) != -1) { + while ((c = getopt(argc, argv, "46c:C:D:i:l:mnp:rs:S:t:vz")) != -1) { switch (c) { case '4': if (cfg_family != PF_UNSPEC) @@ -749,6 +837,9 @@ static void parse_opts(int argc, char **argv) case 'm': cfg_cork_mixed = true; break; + case 'n': + cfg_zerocopy = MSG_ZEROCOPY_NOTIFY_SENDMSG; + break; case 'p': cfg_port = strtoul(optarg, NULL, 0); break; @@ -768,7 +859,7 @@ static void parse_opts(int argc, char **argv) cfg_verbose++; break; case 'z': - cfg_zerocopy = true; + cfg_zerocopy = MSG_ZEROCOPY_NOTIFY_ERRQUEUE; break; } } @@ -779,6 +870,8 @@ static void parse_opts(int argc, char **argv) error(1, 0, "-D required for PF_RDS\n"); if (!cfg_rx && !saddr) error(1, 0, "-S required for PF_RDS\n"); + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG) + error(1, 0, "PF_RDS does not support MSG_ZEROCOPY_NOTIFY_SENDMSG"); } setup_sockaddr(cfg_family, daddr, &cfg_dst_addr); setup_sockaddr(cfg_family, saddr, &cfg_src_addr); diff --git a/tools/testing/selftests/net/msg_zerocopy.sh b/tools/testing/selftests/net/msg_zerocopy.sh index 89c22f5320e0..022a6936d86f 100755 --- a/tools/testing/selftests/net/msg_zerocopy.sh +++ b/tools/testing/selftests/net/msg_zerocopy.sh @@ -118,4 +118,5 @@ do_test() { do_test "${EXTRA_ARGS}" do_test "-z ${EXTRA_ARGS}" +do_test "-n ${EXTRA_ARGS}" echo ok