From patchwork Fri Apr 19 21:48:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13636780 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-oo1-f48.google.com (mail-oo1-f48.google.com [209.85.161.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F354D29CE5 for ; Fri, 19 Apr 2024 21:48:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713563309; cv=none; b=oy2DTufhLUo01hJexcEsNryf5UEUvzUjHzTpMJf4Tc6yGtc0iPgywVU3qs/+GLT+DRUs3336i+f6AAfS4ECXotgplI5CuJsw+m6INtUDmMQPYY7b3hsrzufkqQXnrIp23sIeCyrhQWSify8vEFXTXoNIAlVupgZaz/uXlfg7/pA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713563309; c=relaxed/simple; bh=Tsp/2uIiI1mt1pjhsi7BwZApj/a3QDivPNmsTKAr8wE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=a2vMjf2sE9XcilqcexXwXHspGQ1IdtGWTtsOu6P38yUJH0YDCDh6e3iAqsTtVKDc8jL/8X79a3/MhlpdCtfAm3UNlW/CYpGQbi4+G4RW8EsrGEJ+2fFEHoM/QeVFxDjWDI9Yo2ZqX1+k94NSkGNDupcFw0mOGveU02/JZs4Wwz0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Dsgh1puK; arc=none smtp.client-ip=209.85.161.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Dsgh1puK" Received: by mail-oo1-f48.google.com with SMTP id 006d021491bc7-5aa362cc2ccso1722531eaf.3 for ; Fri, 19 Apr 2024 14:48:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1713563306; x=1714168106; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T1ABXgzFmtpaGn3PNm5y7bfO143XmsdVxQTUgriYetc=; b=Dsgh1puKu1ZAMx6L2x98KbZVPUsY89o2IWeFzNkyamuV16FkUdyx9LU1Etq4wqPc07 oJhVYOy4suks2UIxuD00R1CmDAGcuos3vDbOXWoQpuX35GFoRXTUz9UVdjVA7TNUUcb2 p/7cb6TZfq+kJC5Jw/X8RLK/gTCM6DnFXey6D0eAGJ8uK14uToQvONucUeJQjkQBtCfl Ux7q+gXfZyH/kKLZs1GrFoxgxhgEWF1/GNW1lSJDSZHWZFNyTD82gMTKzCQFGmWCaR13 S7ekcjXNKWuelX4GIRlzi6UaD3KT90w0fMSjfMfl19d3Njm/gesWMKJ2pHicTePGqv0B bWtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713563306; x=1714168106; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T1ABXgzFmtpaGn3PNm5y7bfO143XmsdVxQTUgriYetc=; b=m/9x0wif6PxLSy59dRkSiQt6MGCP/mHwrnqN6qURx6TknWB1g24bbai2GH8eZ1O6Pn /nF8fqFcPI5Z3FXJfDauWxE8Gwxa8nR8PFL645IsNDVprfBFxCmds0HDKbogwB4G3XZ9 SHygBroZyck9KdhVK5cqMuZ+2oRZeJD8Hom9rRNU+fz3iFciVNSDQEQSQdHgBqaC0weE cz+uLEGa3q5T5curQPyOFB23s6KJxXSvfgxo3n9zZzEH0i8QX4ye/e03M1km6t+Fom90 dPS+rxylOww/t5Mv2OWAEnrDjZbc9/U9JQh8g9R9ccMk2KGEVqakqPg5ibNMXUASQoN+ WojA== X-Gm-Message-State: AOJu0Yy6+AQX4i4Rm0beHBZdgZ5toRYEtV+/GSvcM+Ja85VXuodFz3XZ oHkStUoQSWU/ps1t+vKgHNGvneUSoxlwuRWIDMDIRyQf8mbJ6ZSucXW15GE7/GNlhx4++1c7p2I N X-Google-Smtp-Source: AGHT+IEJKBRBvSduRvLUhY8OdxSG1wNCDcUkptlsVNgfmC2Wsw2pihOr/2/o3Dk7e3c8PRSD2FuMIQ== X-Received: by 2002:a05:6358:f1c1:b0:186:68f8:3311 with SMTP id kr1-20020a056358f1c100b0018668f83311mr3465228rwb.27.1713563306298; Fri, 19 Apr 2024 14:48:26 -0700 (PDT) Received: from n191-036-066.byted.org ([130.44.212.119]) by smtp.gmail.com with ESMTPSA id n11-20020a0ce48b000000b0069b6c831e86sm1897511qvl.97.2024.04.19.14.48.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Apr 2024 14:48:26 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, davem@davemloft.net, kuba@kernel.org, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v2 1/3] selftests: fix OOM problem in msg_zerocopy selftest Date: Fri, 19 Apr 2024 21:48:17 +0000 Message-Id: <20240419214819.671536-2-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240419214819.671536-1-zijianzhang@bytedance.com> References: <20240419214819.671536-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang In selftests/net/msg_zerocopy.c, it has a while loop keeps calling sendmsg on a socket with MSG_ZEROCOPY flag, and it will recv the notifications until the socket is not writable. Typically, it will start the receiving process after around 30+ sendmsgs. However, because of the commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale") the sender is always writable and does not get any chance to run recv notifications. The selftest always exits with OUT_OF_MEMORY because the memory used by opt_skb exceeds the core.sysctl_optmem_max. According to our experiments, this problem can be solved by open the DEBUG_LOCKDEP configuration for the kernel. But it will makes the notificatoins disordered even in good commits before commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale"). We introduce "cfg_notification_limit" to force sender to receive notifications after some number of sendmsgs. And, notifications may not come in order, because of the reason we present above. We have order checking code managed by cfg_verbose. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- tools/testing/selftests/net/msg_zerocopy.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c index bdc03a2097e8..6c18b07cab30 100644 --- a/tools/testing/selftests/net/msg_zerocopy.c +++ b/tools/testing/selftests/net/msg_zerocopy.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* Evaluate MSG_ZEROCOPY * * Send traffic between two processes over one of the supported @@ -85,6 +86,7 @@ static bool cfg_rx; static int cfg_runtime_ms = 4200; static int cfg_verbose; static int cfg_waittime_ms = 500; +static int cfg_notification_limit = 32; static bool cfg_zerocopy; static socklen_t cfg_alen; @@ -95,6 +97,8 @@ static char payload[IP_MAXPACKET]; static long packets, bytes, completions, expected_completions; static int zerocopied = -1; static uint32_t next_completion; +/* The number of sendmsgs which have not received notified yet */ +static uint32_t sendmsg_counter; static unsigned long gettimeofday_ms(void) { @@ -208,6 +212,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) error(1, errno, "send"); if (cfg_verbose && ret != len) fprintf(stderr, "send: ret=%u != %u\n", ret, len); + sendmsg_counter++; if (len) { packets++; @@ -435,7 +440,7 @@ static bool do_recv_completion(int fd, int domain) /* Detect notification gaps. These should not happen often, if at all. * Gaps can occur due to drops, reordering and retransmissions. */ - if (lo != next_completion) + if (cfg_verbose && lo != next_completion) fprintf(stderr, "gap: %u..%u does not append to %u\n", lo, hi, next_completion); next_completion = hi + 1; @@ -460,6 +465,7 @@ static bool do_recv_completion(int fd, int domain) static void do_recv_completions(int fd, int domain) { while (do_recv_completion(fd, domain)) {} + sendmsg_counter = 0; } /* Wait for all remaining completions on the errqueue */ @@ -549,6 +555,9 @@ static void do_tx(int domain, int type, int protocol) else do_sendmsg(fd, &msg, cfg_zerocopy, domain); + if (cfg_zerocopy && sendmsg_counter >= cfg_notification_limit) + do_recv_completions(fd, domain); + while (!do_poll(fd, POLLOUT)) { if (cfg_zerocopy) do_recv_completions(fd, domain); @@ -708,7 +717,7 @@ static void parse_opts(int argc, char **argv) cfg_payload_len = max_payload_len; - while ((c = getopt(argc, argv, "46c:C:D:i:mp:rs:S:t:vz")) != -1) { + while ((c = getopt(argc, argv, "46c:C:D:i:mp:rs:S:t:vzl:n")) != -1) { switch (c) { case '4': if (cfg_family != PF_UNSPEC) @@ -760,6 +769,9 @@ static void parse_opts(int argc, char **argv) case 'z': cfg_zerocopy = true; break; + case 'l': + cfg_notification_limit = strtoul(optarg, NULL, 0); + break; } } From patchwork Fri Apr 19 21:48:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13636782 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com [209.85.219.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA25A13CFA7 for ; Fri, 19 Apr 2024 21:48:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713563311; cv=none; b=qFRsFoSPJXTd9ayAboezdClfqaxWeBkhEO/0Nc4dVaYcEHRGDnnYwQa4W10yqEpA/ijlaA3iNM4MgVKsQu+nf0emyyULjJO5CboDVQ9q/axGfcM5OqAq93jmBsk/gyKowDtuqks6a7Aty+PsUG8PhSRABrNJpw9D6ntRM+dK92Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713563311; c=relaxed/simple; bh=27dDV2+hcSergvTkiDbgg6Gzr8j20a9omaPrucnV/KA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=uKTP2fmYdnZxrcmYfB5GFYzwLwBgV2qlXm41PQi1DMzTDUSTX4xjL1j9tJedFoZ0uh1VvL4uKWW/o4HxLpIRvdZYkHIgGShlfYZmpxnCqC2Ve8i8hY9T7DAZndpwi9OelNc0UqPiJS9sA6vrH3AcM4Ye74jBd0F3Po9V2RAwXps= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=YwlKupJ3; arc=none smtp.client-ip=209.85.219.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="YwlKupJ3" Received: by mail-yb1-f178.google.com with SMTP id 3f1490d57ef6-dc74435c428so2729211276.2 for ; Fri, 19 Apr 2024 14:48:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1713563308; x=1714168108; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YJ8EyOML5tOe6Fqb/5cPCFDvCIFtk8FIAkixbWgIWbo=; b=YwlKupJ3fg0s5HGOINxz1IZeusRC8lnn/OxPM6vYeVWf2z3vNOM68BToFGYueOXsnB DwaBcLXh4nnITJQdrBqyVgZb7HMK1Ccg0/3aB2gjvPvMtWFKKl0IPO6gF7T4HUB4UmUo +y0x6Sa//XcSqm6ELRwyBA5FIViF16eWaadSx4W2jiHxIwt61QIIpvDG0hty/qUXpKWi Twwl61facPVM8074QaBCY5wlXaxvOP11+CkzvIkfiOfqm8jjYZi/ErboRvyMsfohDWcv xAhXCWf3ULTiTsEuIkyX2DUKGVjN29JOKd0A2EPvF3/k0NU8m6jTnVQqv9HmU/uw7/Aj ITig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713563308; x=1714168108; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YJ8EyOML5tOe6Fqb/5cPCFDvCIFtk8FIAkixbWgIWbo=; b=eHWSN4L5z36WSR1YD/d8QBD+bGMHHcol0wqvKFxo4JC7RffOKBlMllvh4km8Qy3MLF MCv8lp/iZVimi155bwSCWyxF7Yv9OD9Tase6xEMkfm0rG39yi4a12SlHcUNtDHkKbN3O 8FLmCfoLuOVvdVVMLud6T1f8LpUa4DtRWPCA2fm/PY6ReHtafKpLZ3Yhb6QEVKI+6gtS dILTV9GkNrVaUd3sVd3joSQQ+uVAp2FVoqbqcb9vVpWCRgra1dhKWyppW4EPl5T90RmQ qyD6CAUFOlWY66CiE4BrxkiMstgNaiuf22DE6tm6MSVDDxCyoPAyfsUaRAM6NQiiS8Jv Bxcg== X-Gm-Message-State: AOJu0YzUzPUtSpGDt9sSe7Lmal8cU1tf9yl5kmDNjIgr63XRZLbKo061 9Nxl2018hLKYfthfEok1+dcrodnu/o0FXuFiB89w4BFCZsDLRs02FpQCLAYkxyye0Wbx+8d0JDr 9 X-Google-Smtp-Source: AGHT+IH/ZoMtsicY1i8XsT8ZGsmAUE9Ytgbki6F6zrC3xFAqrPtcWOzKyC8Je7Ijl2hV3Z6oYIo8Zw== X-Received: by 2002:a5b:94d:0:b0:dcc:588f:1523 with SMTP id x13-20020a5b094d000000b00dcc588f1523mr3344046ybq.49.1713563308062; Fri, 19 Apr 2024 14:48:28 -0700 (PDT) Received: from n191-036-066.byted.org ([130.44.212.119]) by smtp.gmail.com with ESMTPSA id n11-20020a0ce48b000000b0069b6c831e86sm1897511qvl.97.2024.04.19.14.48.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Apr 2024 14:48:27 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, davem@davemloft.net, kuba@kernel.org, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v2 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control Date: Fri, 19 Apr 2024 21:48:18 +0000 Message-Id: <20240419214819.671536-3-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240419214819.671536-1-zijianzhang@bytedance.com> References: <20240419214819.671536-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang The MSG_ZEROCOPY flag enables copy avoidance for socket send calls. However, zerocopy is not a free lunch. Apart from the management of user pages, the combination of poll + recvmsg to receive notifications incurs unignorable overhead in the applications. The overhead of such sometimes might be more than the CPU savings from zerocopy. We try to solve this problem with a new notification mechanism based on msgcontrol. This new mechanism aims to reduce the overhead associated with receiving notifications by embedding them directly into user arguments passed with each sendmsg control message. By doing so, we can significantly reduce the complexity and overhead for managing notifications. In an ideal pattern, the user will keep calling sendmsg with SO_ZC_NOTIFICATION msg_control, and the notification will be delivered as soon as possible. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- arch/alpha/include/uapi/asm/socket.h | 2 + arch/mips/include/uapi/asm/socket.h | 2 + arch/parisc/include/uapi/asm/socket.h | 2 + arch/sparc/include/uapi/asm/socket.h | 2 + include/uapi/asm-generic/socket.h | 2 + include/uapi/linux/socket.h | 16 ++++++ net/core/sock.c | 70 +++++++++++++++++++++++++++ 7 files changed, 96 insertions(+) diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi/asm/socket.h index e94f621903fe..b24622a9cd47 100644 --- a/arch/alpha/include/uapi/asm/socket.h +++ b/arch/alpha/include/uapi/asm/socket.h @@ -140,6 +140,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SO_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/asm/socket.h index 60ebaed28a4c..638a4ebbffa7 100644 --- a/arch/mips/include/uapi/asm/socket.h +++ b/arch/mips/include/uapi/asm/socket.h @@ -151,6 +151,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SO_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/uapi/asm/socket.h index be264c2b1a11..393f1a6e9562 100644 --- a/arch/parisc/include/uapi/asm/socket.h +++ b/arch/parisc/include/uapi/asm/socket.h @@ -132,6 +132,8 @@ #define SO_PASSPIDFD 0x404A #define SO_PEERPIDFD 0x404B +#define SO_ZC_NOTIFICATION 0x404C + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi/asm/socket.h index 682da3714686..4fe127b0682b 100644 --- a/arch/sparc/include/uapi/asm/socket.h +++ b/arch/sparc/include/uapi/asm/socket.h @@ -133,6 +133,8 @@ #define SO_PASSPIDFD 0x0055 #define SO_PEERPIDFD 0x0056 +#define SO_ZC_NOTIFICATION 0x0057 + #if !defined(__KERNEL__) diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h index 8ce8a39a1e5f..acbbbe7ac06a 100644 --- a/include/uapi/asm-generic/socket.h +++ b/include/uapi/asm-generic/socket.h @@ -135,6 +135,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SO_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__)) diff --git a/include/uapi/linux/socket.h b/include/uapi/linux/socket.h index d3fcd3b5ec53..60e4db759d49 100644 --- a/include/uapi/linux/socket.h +++ b/include/uapi/linux/socket.h @@ -2,6 +2,8 @@ #ifndef _UAPI_LINUX_SOCKET_H #define _UAPI_LINUX_SOCKET_H +#include + /* * Desired design of maximum size and alignment (see RFC2553) */ @@ -35,4 +37,18 @@ struct __kernel_sockaddr_storage { #define SOCK_TXREHASH_DISABLED 0 #define SOCK_TXREHASH_ENABLED 1 +#define SOCK_ZC_INFO_MAX 256 + +struct zc_info_elem { + __u32 lo; + __u32 hi; + __u8 zerocopy; +}; + +struct zc_info_usr { + __u64 usr_addr; + unsigned int length; + struct zc_info_elem info[]; +}; + #endif /* _UAPI_LINUX_SOCKET_H */ diff --git a/net/core/sock.c b/net/core/sock.c index fe9195186c13..13f06480f2d8 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2809,6 +2809,13 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg, struct sockcm_cookie *sockc) { u32 tsflags; + int ret, zc_info_size, i = 0; + unsigned long flags; + struct sk_buff_head *q, local_q; + struct sk_buff *skb, *tmp; + struct sock_exterr_skb *serr; + struct zc_info_usr *zc_info_usr_p, *zc_info_kern_p; + void __user *usr_addr; switch (cmsg->cmsg_type) { case SO_MARK: @@ -2842,6 +2849,69 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg, case SCM_RIGHTS: case SCM_CREDENTIALS: break; + case SO_ZC_NOTIFICATION: + if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS) + return -EINVAL; + + zc_info_usr_p = (struct zc_info_usr *)CMSG_DATA(cmsg); + if (zc_info_usr_p->length <= 0 || zc_info_usr_p->length > SOCK_ZC_INFO_MAX) + return -EINVAL; + + zc_info_size = struct_size(zc_info_usr_p, info, zc_info_usr_p->length); + if (cmsg->cmsg_len != CMSG_LEN(zc_info_size)) + return -EINVAL; + + usr_addr = (void *)(uintptr_t)(zc_info_usr_p->usr_addr); + if (!access_ok(usr_addr, zc_info_size)) + return -EFAULT; + + zc_info_kern_p = kmalloc(zc_info_size, GFP_KERNEL); + if (!zc_info_kern_p) + return -ENOMEM; + + q = &sk->sk_error_queue; + skb_queue_head_init(&local_q); + spin_lock_irqsave(&q->lock, flags); + skb = skb_peek(q); + while (skb && i < zc_info_usr_p->length) { + struct sk_buff *skb_next = skb_peek_next(skb, q); + + serr = SKB_EXT_ERR(skb); + if (serr->ee.ee_errno == 0 && + serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) { + zc_info_kern_p->info[i].hi = serr->ee.ee_data; + zc_info_kern_p->info[i].lo = serr->ee.ee_info; + zc_info_kern_p->info[i].zerocopy = !(serr->ee.ee_code + & SO_EE_CODE_ZEROCOPY_COPIED); + __skb_unlink(skb, q); + __skb_queue_tail(&local_q, skb); + i++; + } + skb = skb_next; + } + spin_unlock_irqrestore(&q->lock, flags); + + zc_info_kern_p->usr_addr = zc_info_usr_p->usr_addr; + zc_info_kern_p->length = i; + + ret = copy_to_user(usr_addr, + zc_info_kern_p, + struct_size(zc_info_kern_p, info, i)); + kfree(zc_info_kern_p); + + if (unlikely(ret)) { + spin_lock_irqsave(&q->lock, flags); + skb_queue_reverse_walk_safe(&local_q, skb, tmp) { + __skb_unlink(skb, &local_q); + __skb_queue_head(q, skb); + } + spin_unlock_irqrestore(&q->lock, flags); + return -EFAULT; + } + + while ((skb = __skb_dequeue(&local_q))) + consume_skb(skb); + break; default: return -EINVAL; } From patchwork Fri Apr 19 21:48:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13636783 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-oo1-f46.google.com (mail-oo1-f46.google.com [209.85.161.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B7E529CE5 for ; Fri, 19 Apr 2024 21:48:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713563313; cv=none; b=kyjuEw2Uty6n2h4+mkiAv/05F5bZmquIlxPlbbrNOnnfL4hfF+yD3BJfD2ESO0KZWcetzPoKVAS+N+vngshD2dJudWPn4Eu1WV8kVD0Lj+y5Lb4J1kG77KZem3tDE+G2Tf60QbpxuOnEml3pINoG0TkReC2o86/MNz9K6VBw0Jg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713563313; c=relaxed/simple; bh=iNB7EvpT5jMjOl9+PhJ8pViFZodRUC4Yt9ETf97iSLI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qZTrmwVuwpU/JdamNeQnh5fuuRlnsQgJqlnjpUsPUKpv1gGQR9PWSP/cTDdGGgqRqKAlh0vSsRamCC+tfwO6G+XkF3yE1h5BP1KLIpm1UQ2B1H6c23OUtDfeJhCcHTBdj/3xg/okhqErf8sh8rOBXrJoblsvj7gjDI9bZPZw+eU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=gaHFz7CW; arc=none smtp.client-ip=209.85.161.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="gaHFz7CW" Received: by mail-oo1-f46.google.com with SMTP id 006d021491bc7-5aa22ebd048so1729583eaf.1 for ; Fri, 19 Apr 2024 14:48:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1713563309; x=1714168109; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yDJUxA7Q6wYpaDVK4Xmfi/I5kvBQuGvrQepbc4EWlAI=; b=gaHFz7CW944mhpq6iCKjqrK82JK3n7wfaSZxhgx6np52/NUf9vcKAc4DUtezmqJl+S uDr4pK/p4q1SA/2jvmB6gP8t3RJXKK0J1L2Sc1qGM6lrRGHh21+txgCfwy8L2UIcMGvp kz2kxw58L1k+bgkMTUu7EBdM2x/7hZTsPuU62K5yiz8s5UOmFEVEzyXu99ke5OYKg+Sl c90m+wwjp1FF3q7llCl1HbsP+eRd3n+VixKsC87V4zxC2rolpaI+xpiIUkWlKF95mr4/ HhOO/6xlPsZcvgkIBnXNwQffmi/ounai32F5qNZB0nQRtAO17xSKqQFEoyqkbyFmxWi7 5sfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713563309; x=1714168109; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yDJUxA7Q6wYpaDVK4Xmfi/I5kvBQuGvrQepbc4EWlAI=; b=ikib7cc7hbM8rU6dOxB17Rd7kn5n/bsSGfpp4pwVcKsRjaYpgDb963ht6QZ7f8/9HD c9AEYz/atTZjSbwW+YEU3Q+f0JdX9Cmad6Jn4FP59BEd3uha4gEmfBhPeivQdqcAt6ba LX4zm6UMMztrvZA9Oh5/UE2A9p/SycXkU4MOEysjZUHaVDRAtwwlQM4O8uPs1tsUa3n0 dU3iq8P/a7iGP+u8DhfdGGlrFqKmLg5u3nvJ0IJPJsa8mZHNIkCfs+oBKM30q+820Wsd aowNKdikHohQuUM+PNQNvt7rd8V8gh7OfQrngYem1kzsQtUYLKGoRbic+3BYjk9k9JzU jw7g== X-Gm-Message-State: AOJu0YyhfXb36BOC3keN4XWdg6Gpajc5n/vGDTjO5QTu2WAcmRC7LdnS jswiOoepBeLK/4xvMN9hmrBEYO6z9XjvJ35Stuy4PWAsR1+wkl20zoNimCTlJL+a6ZB+dAD0sp8 C X-Google-Smtp-Source: AGHT+IE7QjvNCNG6qfB6Zq0pJrliQoyngxdeLWuB9EoE4bsDA8fmybo0+jU33kMdiQIccHaRhgYpnQ== X-Received: by 2002:a05:6359:1909:b0:181:7b22:d845 with SMTP id mf9-20020a056359190900b001817b22d845mr3375030rwb.16.1713563309520; Fri, 19 Apr 2024 14:48:29 -0700 (PDT) Received: from n191-036-066.byted.org ([130.44.212.119]) by smtp.gmail.com with ESMTPSA id n11-20020a0ce48b000000b0069b6c831e86sm1897511qvl.97.2024.04.19.14.48.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Apr 2024 14:48:29 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, davem@davemloft.net, kuba@kernel.org, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v2 3/3] selftests: add MSG_ZEROCOPY msg_control notification test Date: Fri, 19 Apr 2024 21:48:19 +0000 Message-Id: <20240419214819.671536-4-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240419214819.671536-1-zijianzhang@bytedance.com> References: <20240419214819.671536-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang We update selftests/net/msg_zerocopy.c to accommodate the new mechanism. Test result from selftests/net/msg_zerocopy.c, cfg_notification_limit = 1, in this case MSG_ZEROCOPY approximately aligns with the semantics of MSG_ZEROCOPY_UARG. In this case, the new flag has around 13% cpu savings in TCP and 18% cpu savings in UDP. +---------------------+---------+---------+---------+---------+ | Test Type / Protocol| TCP v4 | TCP v6 | UDP v4 | UDP v6 | +---------------------+---------+---------+---------+---------+ | ZCopy (MB) | 5147 | 4885 | 7489 | 7854 | +---------------------+---------+---------+---------+---------+ | New ZCopy (MB) | 5859 | 5505 | 9053 | 9236 | +---------------------+---------+---------+---------+---------+ | New ZCopy / ZCopy | 113.83% | 112.69% | 120.88% | 117.59% | +---------------------+---------+---------+---------+---------+ cfg_notification_limit = 32, it means less poll + recvmsg overhead, the new mechanism performs 8% better in TCP. For UDP, no obvious performance gain is observed and sometimes may lead to degradation. Thus, if users don't need to retrieve the notification ASAP in UDP, the original mechanism is preferred. +---------------------+---------+---------+---------+---------+ | Test Type / Protocol| TCP v4 | TCP v6 | UDP v4 | UDP v6 | +---------------------+---------+---------+---------+---------+ | ZCopy (MB) | 6272 | 6138 | 12138 | 10055 | +---------------------+---------+---------+---------+---------+ | New ZCopy (MB) | 6774 | 6620 | 11504 | 10355 | +---------------------+---------+---------+---------+---------+ | New ZCopy / ZCopy | 108.00% | 107.85% | 94.78% | 102.98% | +---------------------+---------+---------+---------+---------+ Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- tools/testing/selftests/net/msg_zerocopy.c | 91 +++++++++++++++++++-- tools/testing/selftests/net/msg_zerocopy.sh | 1 + 2 files changed, 86 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c index 6c18b07cab30..aa60d5c3f0a7 100644 --- a/tools/testing/selftests/net/msg_zerocopy.c +++ b/tools/testing/selftests/net/msg_zerocopy.c @@ -67,6 +67,10 @@ #define SO_ZEROCOPY 60 #endif +#ifndef SO_ZC_NOTIFICATION +#define SO_ZC_NOTIFICATION 78 +#endif + #ifndef SO_EE_CODE_ZEROCOPY_COPIED #define SO_EE_CODE_ZEROCOPY_COPIED 1 #endif @@ -75,6 +79,12 @@ #define MSG_ZEROCOPY 0x4000000 #endif +#define ZC_MSGERR_NOTIFICATION 1 +#define ZC_MSGCTL_NOTIFICATION 2 + +#define SOCK_ZC_INFO_NUM 8 +#define ZC_INFO_SIZE(x) (sizeof(struct zc_info_usr) + x * sizeof(struct zc_info_elem)) + static int cfg_cork; static bool cfg_cork_mixed; static int cfg_cpu = -1; /* default: pin to last cpu */ @@ -87,13 +97,14 @@ static int cfg_runtime_ms = 4200; static int cfg_verbose; static int cfg_waittime_ms = 500; static int cfg_notification_limit = 32; -static bool cfg_zerocopy; +static int cfg_zerocopy; /* Either 1 or 2, mode for SO_ZEROCOPY */ static socklen_t cfg_alen; static struct sockaddr_storage cfg_dst_addr; static struct sockaddr_storage cfg_src_addr; static char payload[IP_MAXPACKET]; +static char zc_ckbuf[CMSG_SPACE(ZC_INFO_SIZE(SOCK_ZC_INFO_NUM))]; static long packets, bytes, completions, expected_completions; static int zerocopied = -1; static uint32_t next_completion; @@ -171,6 +182,23 @@ static int do_accept(int fd) return fd; } +static void add_zcopy_info(struct msghdr *msg) +{ + struct cmsghdr *cm; + struct zc_info_usr *zc_info_usr_p; + + if (!msg->msg_control) + error(1, errno, "NULL user arg"); + cm = (void *)msg->msg_control; + cm->cmsg_len = CMSG_LEN(ZC_INFO_SIZE(SOCK_ZC_INFO_NUM)); + cm->cmsg_level = SOL_SOCKET; + cm->cmsg_type = SO_ZC_NOTIFICATION; + + zc_info_usr_p = (struct zc_info_usr *)CMSG_DATA(cm); + zc_info_usr_p->usr_addr = (__u64)zc_info_usr_p; + zc_info_usr_p->length = SOCK_ZC_INFO_NUM; +} + static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie) { struct cmsghdr *cm; @@ -184,7 +212,7 @@ static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie) memcpy(CMSG_DATA(cm), &cookie, sizeof(cookie)); } -static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) +static bool do_sendmsg(int fd, struct msghdr *msg, int do_zerocopy, int domain) { int ret, len, i, flags; static uint32_t cookie; @@ -202,6 +230,11 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) msg->msg_controllen = CMSG_SPACE(sizeof(cookie)); msg->msg_control = (struct cmsghdr *)ckbuf; add_zcopy_cookie(msg, ++cookie); + } else if (do_zerocopy == ZC_MSGCTL_NOTIFICATION) { + memset(&msg->msg_control, 0, sizeof(msg->msg_control)); + msg->msg_controllen = CMSG_SPACE(ZC_INFO_SIZE(SOCK_ZC_INFO_NUM)); + msg->msg_control = (struct cmsghdr *)zc_ckbuf; + add_zcopy_info(msg); } } @@ -220,7 +253,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) if (do_zerocopy && ret) expected_completions++; } - if (do_zerocopy && domain == PF_RDS) { + if (msg->msg_control) { msg->msg_control = NULL; msg->msg_controllen = 0; } @@ -394,6 +427,43 @@ static bool do_recvmsg_completion(int fd) return ret; } +static void do_recv_completions2(struct cmsghdr *cmsg) +{ + int i; + __u32 hi, lo, range; + __u8 zerocopy; + struct zc_info_usr *zc_info_usr_p = (struct zc_info_usr *)CMSG_DATA(cmsg); + + for (i = 0; i < zc_info_usr_p->length; ++i) { + struct zc_info_elem elem = zc_info_usr_p->info[i]; + + hi = elem.hi; + lo = elem.lo; + zerocopy = elem.zerocopy; + range = hi - lo + 1; + + if (cfg_verbose && lo != next_completion) + fprintf(stderr, "gap: %u..%u does not append to %u\n", + lo, hi, next_completion); + next_completion = hi + 1; + + if (zerocopied == -1) + zerocopied = zerocopy; + else if (zerocopied != zerocopy) { + fprintf(stderr, "serr: inconsistent\n"); + zerocopied = zerocopy; + } + + completions += range; + + if (cfg_verbose >= 2) + fprintf(stderr, "completed: %u (h=%u l=%u)\n", + range, hi, lo); + } + + sendmsg_counter -= zc_info_usr_p->length; +} + static bool do_recv_completion(int fd, int domain) { struct sock_extended_err *serr; @@ -555,11 +625,15 @@ static void do_tx(int domain, int type, int protocol) else do_sendmsg(fd, &msg, cfg_zerocopy, domain); - if (cfg_zerocopy && sendmsg_counter >= cfg_notification_limit) + if (cfg_zerocopy == ZC_MSGERR_NOTIFICATION && + sendmsg_counter >= cfg_notification_limit) do_recv_completions(fd, domain); + if (cfg_zerocopy == ZC_MSGCTL_NOTIFICATION) + do_recv_completions2((struct cmsghdr *)zc_ckbuf); + while (!do_poll(fd, POLLOUT)) { - if (cfg_zerocopy) + if (cfg_zerocopy == ZC_MSGERR_NOTIFICATION) do_recv_completions(fd, domain); } @@ -767,11 +841,14 @@ static void parse_opts(int argc, char **argv) cfg_verbose++; break; case 'z': - cfg_zerocopy = true; + cfg_zerocopy = ZC_MSGERR_NOTIFICATION; break; case 'l': cfg_notification_limit = strtoul(optarg, NULL, 0); break; + case 'n': + cfg_zerocopy = ZC_MSGCTL_NOTIFICATION; + break; } } @@ -781,6 +858,8 @@ static void parse_opts(int argc, char **argv) error(1, 0, "-D required for PF_RDS\n"); if (!cfg_rx && !saddr) error(1, 0, "-S required for PF_RDS\n"); + if (cfg_zerocopy == ZC_MSGCTL_NOTIFICATION) + error(1, 0, "PF_RDS does not support ZC_MSGCTL_NOTIFICATION"); } setup_sockaddr(cfg_family, daddr, &cfg_dst_addr); setup_sockaddr(cfg_family, saddr, &cfg_src_addr); diff --git a/tools/testing/selftests/net/msg_zerocopy.sh b/tools/testing/selftests/net/msg_zerocopy.sh index 89c22f5320e0..022a6936d86f 100755 --- a/tools/testing/selftests/net/msg_zerocopy.sh +++ b/tools/testing/selftests/net/msg_zerocopy.sh @@ -118,4 +118,5 @@ do_test() { do_test "${EXTRA_ARGS}" do_test "-z ${EXTRA_ARGS}" +do_test "-n ${EXTRA_ARGS}" echo ok