From patchwork Thu May 9 15:49:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthieu Baerts X-Patchwork-Id: 13659839 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E4BE3BBE3; Thu, 9 May 2024 15:49:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715269785; cv=none; b=Cq6qX4VkWOurigYSm87cmVBG0zS2t13kAtE0C5B5QhWw1YBG6Jy/EAA0kegdlrOTctCR0Ho2l3V1hGxBAb30tkwswMb/rOPyBLrMr8SO1RXU/bnYB1YVld/VkBy2VAWSU8GNLxqJeZgZFKaP518aD4Eo7Sa9RPZ9/8xH3myflic= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715269785; c=relaxed/simple; bh=aBZwDb+Zg5BqvF/86zIslJHhnpqdqLhKSQhEm5hsQhA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=WqVfm/ECVi+FFv+2bHbw6eq/ZXt0JmrTBnr/nfE1Tkyu5GDbls/RJf26zPN5svWuIUXjhltue7L5X01ePh5wEdyHFlB1YiZ1EuXYYdlZVJ6NDJvAyV9qCDv6ap+w5kdJbb42lahd99T7ozbXTnafUzRS21I4F6WkDJMiZC8FvKg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CNOXsy4N; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CNOXsy4N" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0474CC32781; Thu, 9 May 2024 15:49:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715269784; bh=aBZwDb+Zg5BqvF/86zIslJHhnpqdqLhKSQhEm5hsQhA=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=CNOXsy4NSI7SFKgDsgrzigjPBQbQduMdXLBoa2xASiTX+RtLMSnySCEph2WtQTxbb Ck7NHszkOXqNylKBSLgI+dqZ5KSASgyUPYmEC6H6bJCIZz6qP6LMdkVkWPXsQfEYNi jrbfS5STswKahUrbX+PStblPtAFftt6bwCgIeQD60h0mdC2q51YYaFR3N0occGzFmM ZqLv04SxDoR0NqY9uV2c+0qZyFt50sZGT8J2NEEI67ITSoc8sxh6wk5RuGufXIl0Qy driukg+cRZP6pztwbcMmfQCZT/K8HaACLdngtOxcHoSuUBmmO1JQhWor7PeGYO26ZX 70E4ywPe/RnGA== From: "Matthieu Baerts (NGI0)" Date: Thu, 09 May 2024 17:49:11 +0200 Subject: [PATCH bpf-next v2 1/2] selftests/bpf: Add mptcp subflow example Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240509-upstream-bpf-next-20240506-mptcp-subflow-test-v2-1-4048c2948665@kernel.org> References: <20240509-upstream-bpf-next-20240506-mptcp-subflow-test-v2-0-4048c2948665@kernel.org> In-Reply-To: <20240509-upstream-bpf-next-20240506-mptcp-subflow-test-v2-0-4048c2948665@kernel.org> To: mptcp@lists.linux.dev, Mat Martineau , Geliang Tang , Andrii Nakryiko , Eduard Zingerman , Mykola Lysenko , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Shuah Khan Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, "Matthieu Baerts (NGI0)" , Nicolas Rybowski , Geliang Tang X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4287; i=matttbe@kernel.org; h=from:subject:message-id; bh=Wp/wcV4mGM06ft4X8e14lXn7YUnldR2CxDh3hdNKAEM=; b=owEBbQKS/ZANAwAIAfa3gk9CaaBzAcsmYgBmPPCOwVvg/x4jF7OA+J3Jk/KXMeOctkCAHdA74 gTUdw3sGjSJAjMEAAEIAB0WIQToy4X3aHcFem4n93r2t4JPQmmgcwUCZjzwjgAKCRD2t4JPQmmg c6SkD/0ShGdZ61JjqIV00c5XurF6y8jTtqIX3ZpUuQKQr4zLjdgvitz0KSkWkf+jE1jNqqcVoz7 fMNhvAN0lawYuO6cqrIraGKlrRrqD0m5LH2rsB64dYL3yuUxWhU89g043Xc+R90lGo2eWQI8cZG 16PQDo5ucIqgcFFuYD/zK2tVAbiRPBPcoepvLrKhY4T/nCY6jHgKj9xtaBuvc15BYZQVZVsBcnC YQb5UseDK5Y3b0iUHC91sOaQKdY+crdQHaMSdT6Zctqgab9vblwBG/rqZ3u/+C8t4WlmSY496Dp dOLEBERinviHAncSjDwaMBO2AsCU4C0oooGJZ/ZOEkdK2gNAMZudOi0FEXbVngGirVFmuTKsKqH 02unBEv0/YSxnQ/4c159DwBaqNTWflFli2WmvvDhpcW/c47YwCqpakV+1BeUm20HGWhiMYT4LHR JZ63wwts6445kUzLAH93rmYuPtyuMpXh08wbh4VmvCQzKH+Uvo96I9bFDcCShLb7d+LGnAXidQl BQoe0gtO4DmQ0xz2xgWIg4LzZk+r6ZSMHFWXTzy7UIq84Kw7PUSQdhi1LwqjCe3dXA1nbbTsr3f UtTqY6uFh4F5EwefNFM+Q/W4H1Xbns0ZoB3DdTB2FzCJ4E3DyeaN/4URLBlQIdeg4R4wuw7ohhs HyhL8n8hK12PLMA== X-Developer-Key: i=matttbe@kernel.org; a=openpgp; fpr=E8CB85F76877057A6E27F77AF6B7824F4269A073 X-Patchwork-Delegate: bpf@iogearbox.net From: Nicolas Rybowski Move Nicolas' patch into bpf selftests directory. This example adds a different mark (SO_MARK) on each subflow, and changes the TCP CC only on the first subflow. From the userspace, an application can do a setsockopt() on an MPTCP socket, and typically the same value will be propagated to all subflows (paths). If someone wants to have different values per subflow, the recommended way is to use BPF. So it is good to add such example here, and make sure there is no regressions. This example shows how it is possible to: Identify the parent msk of an MPTCP subflow. Put different sockopt for each subflow of a same MPTCP connection. Here especially, two different behaviours are implemented: A socket mark (SOL_SOCKET SO_MARK) is put on each subflow of a same MPTCP connection. The order of creation of the current subflow defines its mark. The TCP CC algorithm of the very first subflow of an MPTCP connection is set to "reno". This is just to show it is possible to identify an MPTCP connection, and set socket options, from different SOL levels, per subflow. "reno" has been picked because it is built-in and usually not set as default one. It is easy to verify with 'ss' that these modifications have been applied correctly. That's what the next patch is going to do. Nicolas' code comes from: commit 4d120186e4d6 ("bpf:examples: update mptcp_set_mark_kern.c") from the MPTCP repo https://github.com/multipath-tcp/mptcp_net-next (the "scripts" branch), and it has been adapted by Geliang. Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/76 Co-developed-by: Geliang Tang Signed-off-by: Geliang Tang Signed-off-by: Nicolas Rybowski Reviewed-by: Mat Martineau Signed-off-by: Matthieu Baerts (NGI0) --- Notes: - v1 -> v2: - The commit message has been updated: why setting multiple socket options, why reno, the verification is done in the next patch (different author). (Alexei) --- tools/testing/selftests/bpf/progs/mptcp_subflow.c | 70 +++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/mptcp_subflow.c b/tools/testing/selftests/bpf/progs/mptcp_subflow.c new file mode 100644 index 000000000000..de9dbba37133 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/mptcp_subflow.c @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020, Tessares SA. */ +/* Copyright (c) 2024, Kylin Software */ + +#include // SOL_SOCKET, SO_MARK, ... +#include // TCP_CONGESTION +#include +#include +#include "bpf_tcp_helpers.h" + +char _license[] SEC("license") = "GPL"; + +#ifndef SOL_TCP +#define SOL_TCP 6 +#endif + +#ifndef TCP_CA_NAME_MAX +#define TCP_CA_NAME_MAX 16 +#endif + +char cc[TCP_CA_NAME_MAX] = "reno"; + +/* Associate a subflow counter to each token */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u32)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, 100); +} mptcp_sf SEC(".maps"); + +SEC("sockops") +int mptcp_subflow(struct bpf_sock_ops *skops) +{ + __u32 init = 1, key, mark, *cnt; + struct mptcp_sock *msk; + struct bpf_sock *sk; + int err; + + if (skops->op != BPF_SOCK_OPS_TCP_CONNECT_CB) + return 1; + + sk = skops->sk; + if (!sk) + return 1; + + msk = bpf_skc_to_mptcp_sock(sk); + if (!msk) + return 1; + + key = msk->token; + cnt = bpf_map_lookup_elem(&mptcp_sf, &key); + if (cnt) { + /* A new subflow is added to an existing MPTCP connection */ + __sync_fetch_and_add(cnt, 1); + mark = *cnt; + } else { + /* A new MPTCP connection is just initiated and this is its primary subflow */ + bpf_map_update_elem(&mptcp_sf, &key, &init, BPF_ANY); + mark = init; + } + + /* Set the mark of the subflow's socket based on appearance order */ + err = bpf_setsockopt(skops, SOL_SOCKET, SO_MARK, &mark, sizeof(mark)); + if (err < 0) + return 1; + if (mark == 1) + err = bpf_setsockopt(skops, SOL_TCP, TCP_CONGESTION, cc, TCP_CA_NAME_MAX); + + return 1; +} From patchwork Thu May 9 15:49:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthieu Baerts X-Patchwork-Id: 13659840 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D9263BBE3; Thu, 9 May 2024 15:49:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715269790; cv=none; b=MNDoYEvoXTyG5icETN50RpjBQZNi/aKE4C7AeNQ+NlySDiRiJqBdw3Umj03+L58qkxzFZbho+5m8GoaX2rlnPeYwXUmX4T0fuVPQ5z41+Cb/XFm+bj1AuJUYCYYyDaWjt6jCU1uAPXSNiGaRkinsgK0ijAEJMhYgdVxaSVSXuvg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715269790; c=relaxed/simple; bh=fwWU1y6Fq/raMfRnXEAPArvOPb7WTnJwfEnbbmHKMMo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=WUoNPF55jlTkfpO0bGIAEyuH/GPBZ9x7MWGNiR4D4/E5wjBZEXcKBE2IzXOFv/nk6Q/vmETvTps/gG782/kND38JdzyCWRRL/UDMrQsWfr4ds+iolOEvBxSH7JGPX83WVCD7MkwpIRx5j59oxZ845Z+n0D3yjn6xHfsXHNo+2co= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DGZLxob+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DGZLxob+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3ECA9C4AF08; Thu, 9 May 2024 15:49:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715269789; bh=fwWU1y6Fq/raMfRnXEAPArvOPb7WTnJwfEnbbmHKMMo=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=DGZLxob+DWdDmZ5oINztgCw1wZDvVf7ay3e08FbmMxGfabdWnVGjsr/lJ9KG426S7 wg+KfoQ4Z/+iqifBNK0JoLlorU5KZOd3cpJyuJVz74UrafrYr4I65+pD0cQYVZaeGN dWUhGRz4ZOMRspdGYR/iXnafviYJTUSE51AkUCP4oV7LN0cEyfT1xD2eYIlSxtaSPE twYlWky0+v6Cg/yahrotiJx2RSOlKVpc+6atB1XCKhquNQ4eRPcrof1fNTrg+eVBzH PTdK7gKRXAde0AD9m4KlGAJYY3iiMpxmiBgvltsSKaRQR+AlaWDbopABONHHbgJgHY fNvr8wHGfrJbg== From: "Matthieu Baerts (NGI0)" Date: Thu, 09 May 2024 17:49:12 +0200 Subject: [PATCH bpf-next v2 2/2] selftests/bpf: Add mptcp subflow subtest Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240509-upstream-bpf-next-20240506-mptcp-subflow-test-v2-2-4048c2948665@kernel.org> References: <20240509-upstream-bpf-next-20240506-mptcp-subflow-test-v2-0-4048c2948665@kernel.org> In-Reply-To: <20240509-upstream-bpf-next-20240506-mptcp-subflow-test-v2-0-4048c2948665@kernel.org> To: mptcp@lists.linux.dev, Mat Martineau , Geliang Tang , Andrii Nakryiko , Eduard Zingerman , Mykola Lysenko , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Shuah Khan Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, "Matthieu Baerts (NGI0)" , Geliang Tang X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4451; i=matttbe@kernel.org; h=from:subject:message-id; bh=LJOvAclIqDeTdcev8kuUSRjihzhLlQDft5mKWsBKgVA=; b=kA0DAAgB9reCT0JpoHMByyZiAGY88I6gJC+qoBHY0rayM2/i1+mgE4r9HTrzf1Byy6x2nsshU 4kCMwQAAQgAHRYhBOjLhfdodwV6bif3eva3gk9CaaBzBQJmPPCOAAoJEPa3gk9CaaBz13YP/RDY XXwOC+86HWPGvJ10RvVL4e/1himpNFmzBbou2tqHBkr19qD0Ji2xmqnnI4LsY+QLsxhBEcM+8BZ ATdMxxXkhvTHKcyKOaqnpY0mNvZd/4ByQYia0Ykh2nDzsVwRXV3S3z4QEPpG1Qq/3OLBSrd/YJX bHU0hGII1occkeYG0LOZU/I0I00Yb5PjqswKtce49ruVgFfpIr+HmVGyvY70dwGY8Q8gMHoWhGq Sx1oilBRsglXGT7llNl/YZQ8advZdFmvjRkgyZYaah+2ESHHeXksAFTTlSkaaDFdESc+0iKaxcp hUc5buOlJidc0cq6lahyh82cnpNGtEYWd/QvvWe3u50/yFGKSojgP4NQbuWVUVVe7KXIr1WhNDH R0hT5CssUmARyzdtDO3Wzq2fbYagkCjPyk59eyCb5pVgWavOfXHCEOGrPdz13+XEObun2h1XZ/W 3Gw0halfqVsMr+MgL1l/HoTqDz/6wVIf+y6UX386ooBx3xCYt9OScNCYoE+zfOemsc1QPBbc3wj hSq+44GN/deXfASymVFiAGxLea0fRceRmWotMTM8/FfIDGflHcnqyjSh4tjMTB45hnnicDgucLw ayVYb6vXpR/sxXTdWQE94pDgUUmYnwEKsOP5fAj0x+2m+m9/qEtDYCaqGk8QTza1ytt9TRPEGAg Wo0/h X-Developer-Key: i=matttbe@kernel.org; a=openpgp; fpr=E8CB85F76877057A6E27F77AF6B7824F4269A073 X-Patchwork-Delegate: bpf@iogearbox.net From: Geliang Tang This patch adds a subtest named test_subflow to load and verify the newly added mptcp subflow example in test_mptcp. Add a helper endpoint_init() to add a new subflow endpoint. Add another helper ss_search() to verify the fwmark and congestion values set by mptcp_subflow prog using setsockopts. Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/76 Signed-off-by: Geliang Tang Reviewed-by: Mat Martineau Signed-off-by: Matthieu Baerts (NGI0) --- tools/testing/selftests/bpf/prog_tests/mptcp.c | 109 +++++++++++++++++++++++++ 1 file changed, 109 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/mptcp.c b/tools/testing/selftests/bpf/prog_tests/mptcp.c index 274d2e033e39..6039b0ff3801 100644 --- a/tools/testing/selftests/bpf/prog_tests/mptcp.c +++ b/tools/testing/selftests/bpf/prog_tests/mptcp.c @@ -9,8 +9,12 @@ #include "network_helpers.h" #include "mptcp_sock.skel.h" #include "mptcpify.skel.h" +#include "mptcp_subflow.skel.h" #define NS_TEST "mptcp_ns" +#define ADDR_1 "10.0.1.1" +#define ADDR_2 "10.0.1.2" +#define PORT_1 10001 #ifndef IPPROTO_MPTCP #define IPPROTO_MPTCP 262 @@ -340,10 +344,115 @@ static void test_mptcpify(void) close(cgroup_fd); } +static int endpoint_init(char *flags) +{ + SYS(fail, "ip -net %s link add veth1 type veth peer name veth2", NS_TEST); + SYS(fail, "ip -net %s addr add %s/24 dev veth1", NS_TEST, ADDR_1); + SYS(fail, "ip -net %s link set dev veth1 up", NS_TEST); + SYS(fail, "ip -net %s addr add %s/24 dev veth2", NS_TEST, ADDR_2); + SYS(fail, "ip -net %s link set dev veth2 up", NS_TEST); + SYS(fail, "ip -net %s mptcp endpoint add %s %s", NS_TEST, ADDR_2, flags); + + return 0; +fail: + return -1; +} + +static int _ss_search(char *src, char *dst, char *port, char *keyword) +{ + char cmd[128]; + int n; + + n = snprintf(cmd, sizeof(cmd), + "ip netns exec %s ss -Menita src %s dst %s %s %d | grep -q '%s'", + NS_TEST, src, dst, port, PORT_1, keyword); + if (n < 0 || n >= sizeof(cmd)) + return -1; + + return system(cmd); +} + +static int ss_search(char *src, char *keyword) +{ + return _ss_search(src, ADDR_1, "dport", keyword); +} + +static void run_subflow(char *new) +{ + int server_fd, client_fd, err; + char cc[TCP_CA_NAME_MAX]; + socklen_t len = sizeof(cc); + + server_fd = start_mptcp_server(AF_INET, ADDR_1, PORT_1, 0); + if (!ASSERT_GE(server_fd, 0, "start_mptcp_server")) + return; + + client_fd = connect_to_fd(server_fd, 0); + if (!ASSERT_GE(client_fd, 0, "connect to fd")) + goto fail; + + err = getsockopt(server_fd, SOL_TCP, TCP_CONGESTION, cc, &len); + if (!ASSERT_OK(err, "getsockopt(srv_fd, TCP_CONGESTION)")) + goto fail; + + send_byte(client_fd); + + ASSERT_OK(ss_search(ADDR_1, "fwmark:0x1"), "ss_search fwmark:0x1"); + ASSERT_OK(ss_search(ADDR_2, "fwmark:0x2"), "ss_search fwmark:0x2"); + ASSERT_OK(ss_search(ADDR_1, new), "ss_search new cc"); + ASSERT_OK(ss_search(ADDR_2, cc), "ss_search default cc"); + + close(client_fd); +fail: + close(server_fd); +} + +static void test_subflow(void) +{ + int cgroup_fd, prog_fd, err; + struct mptcp_subflow *skel; + struct nstoken *nstoken; + + cgroup_fd = test__join_cgroup("/mptcp_subflow"); + if (!ASSERT_GE(cgroup_fd, 0, "join_cgroup: mptcp_subflow")) + return; + + skel = mptcp_subflow__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel_open_load: mptcp_subflow")) + goto close_cgroup; + + err = mptcp_subflow__attach(skel); + if (!ASSERT_OK(err, "skel_attach: mptcp_subflow")) + goto skel_destroy; + + prog_fd = bpf_program__fd(skel->progs.mptcp_subflow); + err = bpf_prog_attach(prog_fd, cgroup_fd, BPF_CGROUP_SOCK_OPS, 0); + if (!ASSERT_OK(err, "prog_attach")) + goto skel_destroy; + + nstoken = create_netns(); + if (!ASSERT_OK_PTR(nstoken, "create_netns: mptcp_subflow")) + goto skel_destroy; + + if (!ASSERT_OK(endpoint_init("subflow"), "endpoint_init")) + goto close_netns; + + run_subflow(skel->data->cc); + +close_netns: + cleanup_netns(nstoken); +skel_destroy: + mptcp_subflow__destroy(skel); +close_cgroup: + close(cgroup_fd); +} + void test_mptcp(void) { if (test__start_subtest("base")) test_base(); if (test__start_subtest("mptcpify")) test_mptcpify(); + if (test__start_subtest("subflow")) + test_subflow(); }