| Message ID | 20220224155010.23676-4-fw@strlen.de (mailing list archive) |
|---|---|
| State | Rejected, archived |
| Headers | show |
| Series | mptcp: replace per-addr listener sockets | expand |
| Context | Check | Description |
|---|---|---|
| matttbe/build | success | Build and static analysis OK |
| matttbe/checkpatch | success | total: 0 errors, 0 warnings, 0 checks, 396 lines checked |
| matttbe/KVM_Validation__normal | warning | Unstable: 1 failed test(s): selftest_mptcp_join |
| matttbe/KVM_Validation__debug | success | Success! ✅ |
Hi Florian, Paolo, I had responded as below to v2 of this series but wondered if perhaps it got lost in the barrage of emails. So thought I'd resend it as I still have questions regarding our rationale here. On 2/24/22 7:50 AM, Florian Westphal wrote: > Currently mptcp adds kernel-based listener socket for all > netlink-configured mptcp address endpoints. > > This has caveats because kernel may interfere with unrelated programs > that use same address/port pairs. > I assume that this refers to a potential race between a kernel listener and the application which Paolo had raised. But I'm not sure if these changes eliminate that possibility. Pasting Paolo's example below from the prior discussion: """ s1 = socket() bind(s1, A) listen(s1) // at this point incoming MPTCP connection can be established on s1 // and ADD_ADDR sub-options could be sent back s2 = socket() bind(s2, B) listen(s2) """ Over a newly established MPTCP connection following listen(s1), the PM can issue an ADD_ADDR with B. In light of this change there would be no listener created for B. But if the remote endpoint immediately established a subflow in response (to the ADD_ADDR), then that would create a subflow (connection) socket at B. It appears (and correct me if I'm wrong) that bind(s2, B) would fail after this point (?). In other words, a subflow creation at an address could race with a subsequent bind() at that address causing startup issues in the application. The only difference now from our prior discussion is that previously the announcement (and the act of creating a listener) could race with the bind(). If this assessment is correct, then these changes aren't really sidestepping the above race which motivated this alternate approach in the first place. Hence, I had the following questions: What are the benefit(s) of this alternate approach? If the above race is still a worry, then isn't it a problem that we have no way to prevent it with this approach? Whereas with kernel listeners we could choose to not create a listener with the netlink API (and thus avoid a race), but in this approach the behavior is baked into the code and outcomes could vary at runtime. I do like the fact that we don't need code to manage kernel listeners with this approach even if it does not resolve the above race, but it comes with other caveats as we've been discussing like potential clashes with TCP listeners and TCP code changes that would have to be accepted upstream. So given the tradeoffs, I am considering the relative merits of the kernel listener approach: deterministic, documentable behavior and wholly contained in the MPTCP layer. I wanted to take a step back to make sure we're all on the same page as to why we're considering these changes. Hope it makes sense :) > RFC 8684 says: > Demultiplexing subflow SYNs MUST be done using the token; this is > unlike traditional TCP, where the destination port is used for > demultiplexing SYN packets. Once a subflow is set up, demultiplexing > packets is done using the 5-tuple, as in traditional TCP. > > This patch deviates from this in that it retains the existing checks of > verifying the incoming requests destination vs. the list of announced > addresses. If the request is to an address that was not assigned, its > treated like an invalid token, i.e. we send a tcp reset with mptcp > error specific code is returned. > > The checks that do this are moved from subflow specific code to the new > hook, this allows us to perform the check at an earlier stage. > > Furthermore, TCP-only listeners take precedence: An MPTCP peer MUST NOT > announce addr:port pairs that are already in use by a non-mptcp listener. > > This could be changed, but it requires move of mptcp_handle_join() hook > *before* the tcp port demux, i.e. an additional conditional in hotpath. > > As-is, the additional conditional (syn && !rst && ...) is placed in the > 'no socket found' path. > > The pernet "listening" socket is hidden from userspace, its not part of > any hashes and not bound to any address/port. > > TPROXY-like semantics apply: If tcp demux cannot find a port for a given > packet, check if the packet is a syn packet with a valid join token. > > If so, the pernet listener is returned and tcp processing resumes. > Otherwise, handling is identical. > > Signed-off-by: Florian Westphal <fw@strlen.de> > --- > include/net/mptcp.h | 19 +++- > net/ipv6/tcp_ipv6.c | 19 ++-- > net/mptcp/ctrl.c | 229 ++++++++++++++++++++++++++++++++++++++++++- > net/mptcp/protocol.c | 2 +- > net/mptcp/protocol.h | 2 +- > net/mptcp/subflow.c | 8 +- > 6 files changed, 258 insertions(+), 21 deletions(-) > > diff --git a/include/net/mptcp.h b/include/net/mptcp.h > index b914e63afc13..b8939d7ea12e 100644 > --- a/include/net/mptcp.h > +++ b/include/net/mptcp.h > @@ -189,6 +189,7 @@ int mptcp_subflow_init_cookie_req(struct request_sock *req, > struct sk_buff *skb); > > __be32 mptcp_get_reset_option(const struct sk_buff *skb); > +struct sock *__mptcp_handle_join(int af, struct sk_buff *skb); > > static inline __be32 mptcp_reset_option(const struct sk_buff *skb) > { > @@ -198,10 +199,20 @@ static inline __be32 mptcp_reset_option(const struct sk_buff *skb) > return htonl(0u); > } > > -static inline struct sock *mptcp_handle_join4(struct sk_buff *skb) > +static inline struct sock *mptcp_handle_join(struct sk_buff *skb, int af) > { > + const struct tcphdr *th = tcp_hdr(skb); > + > + if (th->syn && !th->ack && !th->rst && !th->fin) > + return __mptcp_handle_join(af, skb); > + > return NULL; > } > + > +static inline struct sock *mptcp_handle_join4(struct sk_buff *skb) > +{ > + return mptcp_handle_join(skb, AF_INET); > +} > #else > > static inline void mptcp_init(void) > @@ -284,14 +295,18 @@ static inline struct sock *mptcp_handle_join4(struct sk_buff *skb) { return NULL > > #if IS_ENABLED(CONFIG_MPTCP_IPV6) > int mptcpv6_init(void); > +int mptcpv6_init_net(struct net *net); > +void mptcpv6_exit_net(struct net *net); > void mptcpv6_handle_mapped(struct sock *sk, bool mapped); > > static inline struct sock *mptcp_handle_join6(struct sk_buff *skb) > { > - return NULL; > + return mptcp_handle_join(skb, AF_INET6); > } > #elif IS_ENABLED(CONFIG_IPV6) > static inline int mptcpv6_init(void) { return 0; } > +static inline int mptcpv6_init_net(struct net *net) { return 0; } > +static inline void mptcpv6_exit_net(struct net *net) { } > static inline void mptcpv6_handle_mapped(struct sock *sk, bool mapped) { } > static inline struct sock *mptcp_handle_join6(struct sk_buff *skb) { return NULL; } > #endif > diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c > index 2f7a621aa24d..b414e2f77fa3 100644 > --- a/net/ipv6/tcp_ipv6.c > +++ b/net/ipv6/tcp_ipv6.c > @@ -2256,13 +2256,22 @@ static struct inet_protosw tcpv6_protosw = { > > static int __net_init tcpv6_net_init(struct net *net) > { > - return inet_ctl_sock_create(&net->ipv6.tcp_sk, PF_INET6, > - SOCK_RAW, IPPROTO_TCP, net); > + int err = inet_ctl_sock_create(&net->ipv6.tcp_sk, PF_INET6, > + SOCK_RAW, IPPROTO_TCP, net); > + if (err) > + return err; > + > + err = mptcpv6_init_net(net); > + if (err) > + inet_ctl_sock_destroy(net->ipv6.tcp_sk); > + > + return err; > } > > static void __net_exit tcpv6_net_exit(struct net *net) > { > inet_ctl_sock_destroy(net->ipv6.tcp_sk); > + mptcpv6_exit_net(net); > } > > static struct pernet_operations tcpv6_net_ops = { > @@ -2287,15 +2296,9 @@ int __init tcpv6_init(void) > if (ret) > goto out_tcpv6_protosw; > > - ret = mptcpv6_init(); > - if (ret) > - goto out_tcpv6_pernet_subsys; > - > out: > return ret; > > -out_tcpv6_pernet_subsys: > - unregister_pernet_subsys(&tcpv6_net_ops); > out_tcpv6_protosw: > inet6_unregister_protosw(&tcpv6_protosw); > out_tcpv6_protocol: > diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c > index ae20b7d92e28..c7370c5147df 100644 > --- a/net/mptcp/ctrl.c > +++ b/net/mptcp/ctrl.c > @@ -12,6 +12,7 @@ > #include <net/netns/generic.h> > > #include "protocol.h" > +#include "mib.h" > > #define MPTCP_SYSCTL_PATH "net/mptcp" > > @@ -21,6 +22,12 @@ static int mptcp_pernet_id; > static int mptcp_pm_type_max = __MPTCP_PM_TYPE_MAX; > #endif > > +struct mptcp_join_sk { > + struct sock *sk; > + struct inet_bind_bucket *tb; > + struct inet_bind_hashbucket head; > +}; > + > struct mptcp_pernet { > #ifdef CONFIG_SYSCTL > struct ctl_table_header *ctl_table_hdr; > @@ -32,6 +39,18 @@ struct mptcp_pernet { > u8 checksum_enabled; > u8 allow_join_initial_addr_port; > u8 pm_type; > + > + /* pernet listener to handle mptcp join requests > + * based on the mptcp token. > + * > + * Has to be pernet because tcp uses > + * sock_net(sk_listener) to obtain the net namespace for > + * the syn/ack route lookup. > + */ > + struct mptcp_join_sk join4; > +#if IS_ENABLED(CONFIG_MPTCP_IPV6) > + struct mptcp_join_sk join6; > +#endif > }; > > static struct mptcp_pernet *mptcp_get_pernet(const struct net *net) > @@ -185,13 +204,190 @@ static void mptcp_pernet_del_table(struct mptcp_pernet *pernet) {} > > #endif /* CONFIG_SYSCTL */ > > +static void add_mptcp_rst(struct sk_buff *skb) > +{ > + struct mptcp_ext *ext = skb_ext_add(skb, SKB_EXT_MPTCP); > + > + if (ext) { > + memset(ext, 0, sizeof(*ext)); > + ext->reset_reason = MPTCP_RST_EMPTCP; > + } > +} > + > +struct sock *__mptcp_handle_join(int af, struct sk_buff *skb) > +{ > + struct mptcp_options_received mp_opt; > + struct mptcp_pernet *pernet; > + struct mptcp_sock *msk; > + struct socket *ssock; > + struct sock *lsk; > + struct net *net; > + > + /* paranoia check: don't allow 0 destination port, > + * else __inet_inherit_port will insert the child socket > + * into the phony hash slot of the pernet listener. > + */ > + if (tcp_hdr(skb)->dest == 0) > + return NULL; > + > + mptcp_get_options(skb, &mp_opt); > + > + if (!(mp_opt.suboptions & OPTIONS_MPTCP_MPJ)) > + return NULL; > + > + net = dev_net(skb_dst(skb)->dev); > + if (!mptcp_is_enabled(net)) > + return NULL; > + > + /* RFC8684: If the token is unknown [..], the receiver will send > + * back a reset (RST) signal, analogous to an unknown port in TCP, > + * containing an MP_TCPRST option (Section 3.6) [..] > + */ > + msk = mptcp_token_get_sock(net, mp_opt.token); > + if (!msk) { > + add_mptcp_rst(skb); > + return NULL; > + } > + > + if (!mptcp_pm_sport_in_anno_list(msk, af, skb)) { > + sock_put((struct sock *)msk); > + MPTCP_INC_STATS(net, MPTCP_MIB_MISMATCHPORTSYNRX); > + add_mptcp_rst(skb); > + return NULL; > + } > + > + sock_put((struct sock *)msk); > + pernet = mptcp_get_pernet(net); > + > + switch (af) { > + case AF_INET: > + lsk = pernet->join4.sk; > + break; > +#if IS_ENABLED(CONFIG_MPTCP_IPV6) > + case AF_INET6: > + lsk = pernet->join6.sk; > + break; > +#endif > + default: > + WARN_ON_ONCE(1); > + return NULL; > + } > + > + ssock = __mptcp_nmpc_socket(mptcp_sk(lsk)); > + if (WARN_ON(!ssock)) > + return NULL; > + > + return ssock->sk; > +} > + > +static struct socket *mptcp_create_join_listen_socket(struct net *net, int af) > +{ > + struct socket *s, *ssock; > + int err; > + > + err = sock_create_kern(net, af, SOCK_STREAM, IPPROTO_MPTCP, &s); > + if (err) > + return ERR_PTR(err); > + > + ssock = __mptcp_nmpc_socket(mptcp_sk(s->sk)); > + if (!ssock) { > + err = -EINVAL; > + goto out; > + } > + > + ssock->sk->sk_max_ack_backlog = SOMAXCONN; > + inet_sk_state_store(ssock->sk, TCP_LISTEN); > + > + s->sk->sk_max_ack_backlog = SOMAXCONN; > + inet_sk_state_store(s->sk, TCP_LISTEN); > + > + s->sk->sk_net_refcnt = 1; > + get_net_track(net, &s->sk->ns_tracker, GFP_KERNEL); > + sock_inuse_add(net, 1); > + > + return s; > +out: > + sock_release(s); > + return ERR_PTR(err); > +} > + > +static int mptcp_init_join_sk(struct net *net, struct sock *sk, struct mptcp_join_sk *join_sk) > +{ > + struct socket *ssock = __mptcp_nmpc_socket(mptcp_sk(sk)); > + struct inet_hashinfo *table = ssock->sk->sk_prot->h.hashinfo; > + struct inet_bind_bucket *tb; > + > + spin_lock_init(&join_sk->head.lock); > + INIT_HLIST_HEAD(&join_sk->head.chain); > + > + /* Our "listen socket" isn't bound to any address or port. > + * Conceptually, SYN packet with mptcp join request are steered to > + * this pernet socket just like TPROXY steals arbitrary connection > + * requests to assign them to listening socket with different > + * address or port. > + * > + * The bind_bucket is needed for sake of __inet_inherit_port(), > + * so it can place the new child socket in the correct > + * bind_bucket slot. > + * > + * A phony head is used to hide this socket from normal sk loookup. > + */ > + tb = inet_bind_bucket_create(table->bind_bucket_cachep, > + net, &join_sk->head, 0, 0); > + if (!tb) > + return -ENOMEM; > + > + inet_csk(ssock->sk)->icsk_bind_hash = tb; > + return 0; > +} > + > static int __net_init mptcp_net_init(struct net *net) > { > struct mptcp_pernet *pernet = mptcp_get_pernet(net); > + struct socket *sock; > + int err; > > mptcp_pernet_set_defaults(pernet); > > - return mptcp_pernet_new_table(net, pernet); > + err = mptcp_pernet_new_table(net, pernet); > + if (err) > + return err; > + > + sock = mptcp_create_join_listen_socket(net, AF_INET); > + if (IS_ERR(sock)) { > + err = PTR_ERR(sock); > + goto out_table; > + } > + > + err = mptcp_init_join_sk(net, sock->sk, &pernet->join4); > + if (err) { > + sock_release(sock); > + goto out_table; > + } > + > + /* struct sock is still reachable via sock->sk_socket backpointer */ > + pernet->join4.sk = sock->sk; > + return err; > + > +out_table: > + if (!net_eq(net, &init_net)) > + mptcp_pernet_del_table(pernet); > + return err; > +} > + > +static void __net_exit mptcp_exit_join_sk(struct mptcp_join_sk *jsk) > +{ > + struct socket *ssock = __mptcp_nmpc_socket(mptcp_sk(jsk->sk)); > + struct inet_bind_bucket *tb; > + struct inet_hashinfo *table; > + > + table = ssock->sk->sk_prot->h.hashinfo; > + > + tb = inet_csk(ssock->sk)->icsk_bind_hash; > + inet_bind_bucket_destroy(table->bind_bucket_cachep, tb); > + > + ssock = jsk->sk->sk_socket; > + sock_release(ssock); > } > > /* Note: the callback will only be called per extra netns */ > @@ -200,6 +396,7 @@ static void __net_exit mptcp_net_exit(struct net *net) > struct mptcp_pernet *pernet = mptcp_get_pernet(net); > > mptcp_pernet_del_table(pernet); > + mptcp_exit_join_sk(&pernet->join4); > } > > static struct pernet_operations mptcp_pernet_ops = { > @@ -219,12 +416,36 @@ void __init mptcp_init(void) > } > > #if IS_ENABLED(CONFIG_MPTCP_IPV6) > -int __init mptcpv6_init(void) > +int __net_init mptcpv6_init_net(struct net *net) > { > + struct mptcp_pernet *pernet = mptcp_get_pernet(net); > + struct socket *sock; > int err; > > - err = mptcp_proto_v6_init(); > + if (net_eq(net, &init_net)) { > + err = mptcp_proto_v6_init(); > + if (err) > + return err; > + } > + > + sock = mptcp_create_join_listen_socket(net, AF_INET6); > + if (IS_ERR(sock)) > + return PTR_ERR(sock); > > - return err; > + err = mptcp_init_join_sk(net, sock->sk, &pernet->join6); > + if (err) { > + sock_release(sock); > + return err; > + } > + > + pernet->join6.sk = sock->sk; > + return 0; > +} > + > +void __net_exit mptcpv6_exit_net(struct net *net) > +{ > + struct mptcp_pernet *pernet = mptcp_get_pernet(net); > + > + mptcp_exit_join_sk(&pernet->join6); > } > #endif > diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c > index 3cb975227d12..bc7108ed453c 100644 > --- a/net/mptcp/protocol.c > +++ b/net/mptcp/protocol.c > @@ -3794,7 +3794,7 @@ static struct inet_protosw mptcp_v6_protosw = { > .flags = INET_PROTOSW_ICSK, > }; > > -int __init mptcp_proto_v6_init(void) > +int __net_init mptcp_proto_v6_init(void) > { > int err; > > diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h > index 6b2d7f60c8ad..7ec2513e1c2f 100644 > --- a/net/mptcp/protocol.h > +++ b/net/mptcp/protocol.h > @@ -648,7 +648,7 @@ static inline bool mptcp_has_another_subflow(struct sock *ssk) > > void __init mptcp_proto_init(void); > #if IS_ENABLED(CONFIG_MPTCP_IPV6) > -int __init mptcp_proto_v6_init(void); > +int __net_init mptcp_proto_v6_init(void); > #endif > > struct sock *mptcp_sk_clone(const struct sock *sk, > diff --git a/net/mptcp/subflow.c b/net/mptcp/subflow.c > index 77da5f744a17..67a4c698602d 100644 > --- a/net/mptcp/subflow.c > +++ b/net/mptcp/subflow.c > @@ -116,6 +116,9 @@ static void subflow_init_req(struct request_sock *req, const struct sock *sk_lis > > static bool subflow_use_different_sport(struct mptcp_sock *msk, const struct sock *sk) > { > + if (inet_sk(sk)->inet_sport == 0) > + return true; > + > return inet_sk(sk)->inet_sport != inet_sk((struct sock *)msk)->inet_sport; > } > > @@ -216,11 +219,6 @@ static int subflow_check_req(struct request_sock *req, > pr_debug("syn inet_sport=%d %d", > ntohs(inet_sk(sk_listener)->inet_sport), > ntohs(inet_sk((struct sock *)subflow_req->msk)->inet_sport)); > - if (!mptcp_pm_sport_in_anno_list(subflow_req->msk, > - sk_listener->sk_family, skb)) { > - SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_MISMATCHPORTSYNRX); > - return -EPERM; > - } > SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_JOINPORTSYNRX); > } >
Kishen Maloor <kishen.maloor@intel.com> wrote: > Hi Florian, Paolo, > > I had responded as below to v2 of this series but wondered if perhaps it got lost > in the barrage of emails. So thought I'd resend it as I still have questions > regarding our rationale here. > > On 2/24/22 7:50 AM, Florian Westphal wrote: > > Currently mptcp adds kernel-based listener socket for all > > netlink-configured mptcp address endpoints. > > > > This has caveats because kernel may interfere with unrelated programs > > that use same address/port pairs. > > > > I assume that this refers to a potential race between a kernel listener and > the application which Paolo had raised. But I'm not sure if these changes > eliminate that possibility. Pasting Paolo's example below from the prior discussion: > > """ > s1 = socket() > bind(s1, A) > listen(s1) > // at this point incoming MPTCP connection can be established on s1 > // and ADD_ADDR sub-options could be sent back > > s2 = socket() > bind(s2, B) > listen(s2) > """ > > Over a newly established MPTCP connection following listen(s1), the PM can issue an > ADD_ADDR with B. In light of this change there would be no listener created for B. > But if the remote endpoint immediately established a subflow in response (to the > ADD_ADDR), then that would create a subflow (connection) socket at B. > It appears (and correct me if I'm wrong) that bind(s2, B) would fail after this point (?). Why would that fail? You can bind x:y even if there is an established connection from x:y to q:r. > In other words, a subflow creation at an address could race with a subsequent bind() > at that address causing startup issues in the application. I don't think so. Its fairly common to implement "graceful restart/update" via "close(listen); fork(); exit();"-sequence. child continues to handle existing connections while new process can start & to serve new clients.
On 3/8/22 10:45 AM, Florian Westphal wrote: > Kishen Maloor <kishen.maloor@intel.com> wrote: >> Hi Florian, Paolo, >> >> I had responded as below to v2 of this series but wondered if perhaps it got lost >> in the barrage of emails. So thought I'd resend it as I still have questions >> regarding our rationale here. >> >> On 2/24/22 7:50 AM, Florian Westphal wrote: >>> Currently mptcp adds kernel-based listener socket for all >>> netlink-configured mptcp address endpoints. >>> >>> This has caveats because kernel may interfere with unrelated programs >>> that use same address/port pairs. >>> >> >> I assume that this refers to a potential race between a kernel listener and >> the application which Paolo had raised. But I'm not sure if these changes >> eliminate that possibility. Pasting Paolo's example below from the prior discussion: >> >> """ >> s1 = socket() >> bind(s1, A) >> listen(s1) >> // at this point incoming MPTCP connection can be established on s1 >> // and ADD_ADDR sub-options could be sent back >> >> s2 = socket() >> bind(s2, B) >> listen(s2) >> """ >> >> Over a newly established MPTCP connection following listen(s1), the PM can issue an >> ADD_ADDR with B. In light of this change there would be no listener created for B. >> But if the remote endpoint immediately established a subflow in response (to the >> ADD_ADDR), then that would create a subflow (connection) socket at B. >> It appears (and correct me if I'm wrong) that bind(s2, B) would fail after this point (?). > > Why would that fail? You can bind x:y even if there is an established > connection from x:y to q:r. If I establish an MPTCP connection using mptcp_connect individually as Client and Server, then I am unable to bind a 3rd (new) Server process at the Client's addr+port [1]. Why is this the case? > >> In other words, a subflow creation at an address could race with a subsequent bind() >> at that address causing startup issues in the application. > > I don't think so. Its fairly common to implement "graceful restart/update" via > "close(listen); fork(); exit();"-sequence. child continues to handle existing > connections while new process can start & to serve new clients. What you're saying may be true for a previously bound addr+port that was in the LISTEN state at which a new listener may be bound after the old one was closed, and in the presence of ongoing connections. However, I am viewing this matter in light of your changes wherein a connection socket is established without there having been a bound listener at that addr+port. In that case, I was wondering if the above situation [1] would apply on a subsequent bind(). If it does, then we could encounter a race such as I described.
Kishen Maloor <kishen.maloor@intel.com> wrote: > >> Over a newly established MPTCP connection following listen(s1), the PM can issue an > >> ADD_ADDR with B. In light of this change there would be no listener created for B. > >> But if the remote endpoint immediately established a subflow in response (to the > >> ADD_ADDR), then that would create a subflow (connection) socket at B. > >> It appears (and correct me if I'm wrong) that bind(s2, B) would fail after this point (?). > > > > Why would that fail? You can bind x:y even if there is an established > > connection from x:y to q:r. > > If I establish an MPTCP connection using mptcp_connect individually as > Client and Server, then I am unable to bind a 3rd (new) Server process at the Client's > addr+port [1]. Why is this the case? Whats [1]? I suspect this patch series needs following addition in patch 3: diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c --- a/net/mptcp/ctrl.c +++ b/net/mptcp/ctrl.c @@ -337,6 +337,8 @@ static int mptcp_init_join_sk(struct net *net, struct sock *sk, struct mptcp_joi if (!tb) return -ENOMEM; + ssock->sk->sk_reuse = 1; + ssock->sk->sk_reuseport = 1; inet_csk(ssock->sk)->icsk_bind_hash = tb; return 0; } After that, follwing sequence should work: 1. bind(0.0.0.0, p1) // listen, accept etc, initial subflow established 2. announce p2 3. receive join on addr, p2 4. bind(0.0.0.0, p2) 4) should work because sk used for endpoint in 3) has reuse flag set and is not in listen state. cf. include/net/inet_hashtables.h, line 47: 2) If all sockets have sk->sk_reuse set, and none of them TCP_LISTEN state, the port may be shared. > However, I am viewing this matter in light of your changes wherein a connection > socket is established without there having been a bound listener at that > addr+port. In that case, I was wondering if the above situation [1] would apply on a > subsequent bind(). If it does, then we could encounter a race such as I described. If so, we should consider dropping support for address announcements with ports, it seems too fragile.
On 3/9/22 4:53 AM, Florian Westphal wrote: > Kishen Maloor <kishen.maloor@intel.com> wrote: >>>> Over a newly established MPTCP connection following listen(s1), the PM can issue an >>>> ADD_ADDR with B. In light of this change there would be no listener created for B. >>>> But if the remote endpoint immediately established a subflow in response (to the >>>> ADD_ADDR), then that would create a subflow (connection) socket at B. >>>> It appears (and correct me if I'm wrong) that bind(s2, B) would fail after this point (?). >>> >>> Why would that fail? You can bind x:y even if there is an established >>> connection from x:y to q:r. >> >> If I establish an MPTCP connection using mptcp_connect individually as >> Client and Server, then I am unable to bind a 3rd (new) Server process at the Client's >> addr+port [1]. Why is this the case? > > Whats [1]? > I suspect this patch series needs following addition in patch 3: > > diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c > --- a/net/mptcp/ctrl.c > +++ b/net/mptcp/ctrl.c > @@ -337,6 +337,8 @@ static int mptcp_init_join_sk(struct net *net, struct sock *sk, struct mptcp_joi > if (!tb) > return -ENOMEM; > > + ssock->sk->sk_reuse = 1; > + ssock->sk->sk_reuseport = 1; > inet_csk(ssock->sk)->icsk_bind_hash = tb; > return 0; > } > > After that, follwing sequence should work: > > 1. bind(0.0.0.0, p1) // listen, accept etc, initial subflow established > 2. announce p2 > 3. receive join on addr, p2 > 4. bind(0.0.0.0, p2) > > 4) should work because sk used for endpoint in 3) has reuse flag set > and is not in listen state. > > cf. include/net/inet_hashtables.h, line 47: > 2) If all sockets have sk->sk_reuse set, and none of them > TCP_LISTEN state, the port may be shared. > Wouldn't 4) fail if the socket being bound at the time does not have the SO_REUSExxx flag(s) set? If so, that would be application level thing and in that situation we don't have a way to avoid a race. Whereas when we require an explicit listener, we could have the kernel take a step back (and not create a listener) to break the race. (By the way, [1] was just an annotation so I could refer back to it in my statement below :)) >> However, I am viewing this matter in light of your changes wherein a connection >> socket is established without there having been a bound listener at that >> addr+port. In that case, I was wondering if the above situation [1] would apply on a >> subsequent bind(). If it does, then we could encounter a race such as I described. > > If so, we should consider dropping support for address announcements > with ports, it seems too fragile. Well, this isn't about port-based endpoints and I have thus far assumed in this conversation that we're just reusing the subflow port. So, A and B as taken from Paolo's original race condition scenario in my mind refer to addrA:portX and addrB:portX.
Kishen Maloor <kishen.maloor@intel.com> wrote: > On 3/9/22 4:53 AM, Florian Westphal wrote: > > Kishen Maloor <kishen.maloor@intel.com> wrote: > >>>> Over a newly established MPTCP connection following listen(s1), the PM can issue an > >>>> ADD_ADDR with B. In light of this change there would be no listener created for B. > >>>> But if the remote endpoint immediately established a subflow in response (to the > >>>> ADD_ADDR), then that would create a subflow (connection) socket at B. > >>>> It appears (and correct me if I'm wrong) that bind(s2, B) would fail after this point (?). > >>> > >>> Why would that fail? You can bind x:y even if there is an established > >>> connection from x:y to q:r. > >> > >> If I establish an MPTCP connection using mptcp_connect individually as > >> Client and Server, then I am unable to bind a 3rd (new) Server process at the Client's > >> addr+port [1]. Why is this the case? > > > > Whats [1]? > > I suspect this patch series needs following addition in patch 3: > > > > diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c > > --- a/net/mptcp/ctrl.c > > +++ b/net/mptcp/ctrl.c > > @@ -337,6 +337,8 @@ static int mptcp_init_join_sk(struct net *net, struct sock *sk, struct mptcp_joi > > if (!tb) > > return -ENOMEM; > > > > + ssock->sk->sk_reuse = 1; > > + ssock->sk->sk_reuseport = 1; > > inet_csk(ssock->sk)->icsk_bind_hash = tb; > > return 0; > > } > > > > After that, follwing sequence should work: > > > > 1. bind(0.0.0.0, p1) // listen, accept etc, initial subflow established > > 2. announce p2 > > 3. receive join on addr, p2 > > 4. bind(0.0.0.0, p2) > > > > 4) should work because sk used for endpoint in 3) has reuse flag set > > and is not in listen state. > > > > cf. include/net/inet_hashtables.h, line 47: > > 2) If all sockets have sk->sk_reuse set, and none of them > > TCP_LISTEN state, the port may be shared. > > > > Wouldn't 4) fail if the socket being bound at the time does not have the SO_REUSExxx flag(s) set? Yes, it needs SO_REUSEADDR set. > If so, that would be application level thing and in that situation we don't have a way to > avoid a race. Whereas when we require an explicit listener, we could have the kernel take a step > back (and not create a listener) to break the race. Uh, what? Sorry, I am totally lost. I have no idea what the problem is that we're solving here. EOD, I am out of ideas. Feel free to toss this patchset, I have no idea what to do.
On 3/9/22 1:37 PM, Florian Westphal wrote: > Kishen Maloor <kishen.maloor@intel.com> wrote: >> On 3/9/22 4:53 AM, Florian Westphal wrote: >>> Kishen Maloor <kishen.maloor@intel.com> wrote: >>>>>> Over a newly established MPTCP connection following listen(s1), the PM can issue an >>>>>> ADD_ADDR with B. In light of this change there would be no listener created for B. >>>>>> But if the remote endpoint immediately established a subflow in response (to the >>>>>> ADD_ADDR), then that would create a subflow (connection) socket at B. >>>>>> It appears (and correct me if I'm wrong) that bind(s2, B) would fail after this point (?). >>>>> >>>>> Why would that fail? You can bind x:y even if there is an established >>>>> connection from x:y to q:r. >>>> >>>> If I establish an MPTCP connection using mptcp_connect individually as >>>> Client and Server, then I am unable to bind a 3rd (new) Server process at the Client's >>>> addr+port [1]. Why is this the case? >>> >>> Whats [1]? >>> I suspect this patch series needs following addition in patch 3: >>> >>> diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c >>> --- a/net/mptcp/ctrl.c >>> +++ b/net/mptcp/ctrl.c >>> @@ -337,6 +337,8 @@ static int mptcp_init_join_sk(struct net *net, struct sock *sk, struct mptcp_joi >>> if (!tb) >>> return -ENOMEM; >>> >>> + ssock->sk->sk_reuse = 1; >>> + ssock->sk->sk_reuseport = 1; >>> inet_csk(ssock->sk)->icsk_bind_hash = tb; >>> return 0; >>> } >>> >>> After that, follwing sequence should work: >>> >>> 1. bind(0.0.0.0, p1) // listen, accept etc, initial subflow established >>> 2. announce p2 >>> 3. receive join on addr, p2 >>> 4. bind(0.0.0.0, p2) >>> >>> 4) should work because sk used for endpoint in 3) has reuse flag set >>> and is not in listen state. >>> >>> cf. include/net/inet_hashtables.h, line 47: >>> 2) If all sockets have sk->sk_reuse set, and none of them >>> TCP_LISTEN state, the port may be shared. >>> >> >> Wouldn't 4) fail if the socket being bound at the time does not have the SO_REUSExxx flag(s) set? > > Yes, it needs SO_REUSEADDR set. > >> If so, that would be application level thing and in that situation we don't have a way to >> avoid a race. Whereas when we require an explicit listener, we could have the kernel take a step >> back (and not create a listener) to break the race. > > Uh, what? Sorry, I am totally lost. I have no idea what the problem is > that we're solving here. > > EOD, I am out of ideas. Feel free to toss this patchset, I have no idea > what to do. Sorry, what isn't clear? This series was meant to address perceived pitfalls of kernel listeners, such as race conditions which were discussed. So, I believe we are trying to assess if these changes indeed do that, or if anything further needs to be done. It seems we've thus far identified one change as a result of this conversation to avoid a race, i.e. setting the _reuse_ flags inside mptcp_init_join_sk(). Is there anything else to do, or are we now confident in these changes? I think the group needs to make a call on our path forward (one way or the other) so that we can also progress on other related work. On a separate note, I brought together my entire userspace PM series and this on a tree to test and found a couple of glitches in this patchset. A few userspace PM subflows tests would fail due to that. I can respond later on this thread with the specific modifications after I clean things up on my end.
On Wed, 9 Mar 2022, Kishen Maloor wrote: > On 3/9/22 1:37 PM, Florian Westphal wrote: >> Kishen Maloor <kishen.maloor@intel.com> wrote: >>> On 3/9/22 4:53 AM, Florian Westphal wrote: >>>> Kishen Maloor <kishen.maloor@intel.com> wrote: >>>>>>> Over a newly established MPTCP connection following listen(s1), the PM can issue an >>>>>>> ADD_ADDR with B. In light of this change there would be no listener created for B. >>>>>>> But if the remote endpoint immediately established a subflow in response (to the >>>>>>> ADD_ADDR), then that would create a subflow (connection) socket at B. >>>>>>> It appears (and correct me if I'm wrong) that bind(s2, B) would fail after this point (?). >>>>>> >>>>>> Why would that fail? You can bind x:y even if there is an established >>>>>> connection from x:y to q:r. >>>>> >>>>> If I establish an MPTCP connection using mptcp_connect individually as >>>>> Client and Server, then I am unable to bind a 3rd (new) Server process at the Client's >>>>> addr+port [1]. Why is this the case? >>>> >>>> Whats [1]? >>>> I suspect this patch series needs following addition in patch 3: >>>> >>>> diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c >>>> --- a/net/mptcp/ctrl.c >>>> +++ b/net/mptcp/ctrl.c >>>> @@ -337,6 +337,8 @@ static int mptcp_init_join_sk(struct net *net, struct sock *sk, struct mptcp_joi >>>> if (!tb) >>>> return -ENOMEM; >>>> >>>> + ssock->sk->sk_reuse = 1; >>>> + ssock->sk->sk_reuseport = 1; >>>> inet_csk(ssock->sk)->icsk_bind_hash = tb; >>>> return 0; >>>> } >>>> >>>> After that, follwing sequence should work: >>>> >>>> 1. bind(0.0.0.0, p1) // listen, accept etc, initial subflow established >>>> 2. announce p2 >>>> 3. receive join on addr, p2 >>>> 4. bind(0.0.0.0, p2) >>>> >>>> 4) should work because sk used for endpoint in 3) has reuse flag set >>>> and is not in listen state. >>>> >>>> cf. include/net/inet_hashtables.h, line 47: >>>> 2) If all sockets have sk->sk_reuse set, and none of them >>>> TCP_LISTEN state, the port may be shared. >>>> >>> >>> Wouldn't 4) fail if the socket being bound at the time does not have the SO_REUSExxx flag(s) set? >> >> Yes, it needs SO_REUSEADDR set. >> >>> If so, that would be application level thing and in that situation we don't have a way to >>> avoid a race. Whereas when we require an explicit listener, we could have the kernel take a step >>> back (and not create a listener) to break the race. >> >> Uh, what? Sorry, I am totally lost. I have no idea what the problem is >> that we're solving here. >> >> EOD, I am out of ideas. Feel free to toss this patchset, I have no idea >> what to do. > > Sorry, what isn't clear? > > This series was meant to address perceived pitfalls of kernel listeners, such as > race conditions which were discussed. So, I believe we are trying to assess if > these changes indeed do that, or if anything further needs to be done. > Responding to both Kishen and Florian: I think Kishen's summary here is accurate - the way the current PM code uses listening sockets, and the expanded use of listening sockets in the userspace PM code, led to some hard-to-troubleshoot situations where in-kernel listeners interfered with applications expecting to bind() the same addresses / ports. And I think what Kishen found while testing the userspace PM patches is that the "pernet listen socket" approach still has some bind() failure corner cases that application programs (and programmers) might find surprising, and that the kernel can't solve these cases. In comparison, with the in-kernel (explicit) listener sockets, the PM code can see some of those failures and work around them to some degree. > It seems we've thus far identified one change as a result of this conversation to avoid a > race, i.e. setting the _reuse_ flags inside mptcp_init_join_sk(). Is there anything else to do, or > are we now confident in these changes? Kishen, to be sure I'm parsing the above question correctly, "these changes" == the "replace per-addr listener sockets" series this email thread is part of? If so, yes I think the reuse flags are part of moving forward with this, in addition to what you mention below. > > I think the group needs to make a call on our path forward (one way or the other) so that > we can also progress on other related work. > > On a separate note, I brought together my entire userspace PM series and this on a > tree to test and found a couple of glitches in this patchset. A few userspace > PM subflows tests would fail due to that. I can respond later on this thread with the > specific modifications after I clean things up on my end. We will talk about it again at the 10-March meeting, but it sounds like you have some modifications to propose that might be important to consider in a final decision. Do the fixes you have to this patchset involve the bind() issues above or are they totally separate? -- Mat Martineau Intel
On 3/9/22 4:37 PM, Mat Martineau wrote: > On Wed, 9 Mar 2022, Kishen Maloor wrote: > >> On 3/9/22 1:37 PM, Florian Westphal wrote: >>> Kishen Maloor <kishen.maloor@intel.com> wrote: >>>> On 3/9/22 4:53 AM, Florian Westphal wrote: >>>>> Kishen Maloor <kishen.maloor@intel.com> wrote: >>>>>>>> Over a newly established MPTCP connection following listen(s1), the PM can issue an >>>>>>>> ADD_ADDR with B. In light of this change there would be no listener created for B. >>>>>>>> But if the remote endpoint immediately established a subflow in response (to the >>>>>>>> ADD_ADDR), then that would create a subflow (connection) socket at B. >>>>>>>> It appears (and correct me if I'm wrong) that bind(s2, B) would fail after this point (?). >>>>>>> >>>>>>> Why would that fail? You can bind x:y even if there is an established >>>>>>> connection from x:y to q:r. >>>>>> >>>>>> If I establish an MPTCP connection using mptcp_connect individually as >>>>>> Client and Server, then I am unable to bind a 3rd (new) Server process at the Client's >>>>>> addr+port [1]. Why is this the case? >>>>> >>>>> Whats [1]? >>>>> I suspect this patch series needs following addition in patch 3: >>>>> >>>>> diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c >>>>> --- a/net/mptcp/ctrl.c >>>>> +++ b/net/mptcp/ctrl.c >>>>> @@ -337,6 +337,8 @@ static int mptcp_init_join_sk(struct net *net, struct sock *sk, struct mptcp_joi >>>>> if (!tb) >>>>> return -ENOMEM; >>>>> >>>>> + ssock->sk->sk_reuse = 1; >>>>> + ssock->sk->sk_reuseport = 1; >>>>> inet_csk(ssock->sk)->icsk_bind_hash = tb; >>>>> return 0; >>>>> } >>>>> >>>>> After that, follwing sequence should work: >>>>> >>>>> 1. bind(0.0.0.0, p1) // listen, accept etc, initial subflow established >>>>> 2. announce p2 >>>>> 3. receive join on addr, p2 >>>>> 4. bind(0.0.0.0, p2) >>>>> >>>>> 4) should work because sk used for endpoint in 3) has reuse flag set >>>>> and is not in listen state. >>>>> >>>>> cf. include/net/inet_hashtables.h, line 47: >>>>> 2) If all sockets have sk->sk_reuse set, and none of them >>>>> TCP_LISTEN state, the port may be shared. >>>>> >>>> >>>> Wouldn't 4) fail if the socket being bound at the time does not have the SO_REUSExxx flag(s) set? >>> >>> Yes, it needs SO_REUSEADDR set. >>> >>>> If so, that would be application level thing and in that situation we don't have a way to >>>> avoid a race. Whereas when we require an explicit listener, we could have the kernel take a step >>>> back (and not create a listener) to break the race. >>> >>> Uh, what? Sorry, I am totally lost. I have no idea what the problem is >>> that we're solving here. >>> >>> EOD, I am out of ideas. Feel free to toss this patchset, I have no idea >>> what to do. >> >> Sorry, what isn't clear? >> >> This series was meant to address perceived pitfalls of kernel listeners, such as >> race conditions which were discussed. So, I believe we are trying to assess if >> these changes indeed do that, or if anything further needs to be done. >> > > Responding to both Kishen and Florian: > > I think Kishen's summary here is accurate - the way the current PM code uses listening sockets, and the expanded use of listening sockets in the userspace PM code, led to some hard-to-troubleshoot situations where in-kernel listeners interfered with applications expecting to bind() the same addresses / ports. > > And I think what Kishen found while testing the userspace PM patches is that the "pernet listen socket" approach still has some bind() failure corner cases that application programs (and programmers) might find surprising, and that the kernel can't solve these cases. In comparison, with the in-kernel (explicit) listener sockets, the PM code can see some of those failures and work around them to some degree. > > >> It seems we've thus far identified one change as a result of this conversation to avoid a >> race, i.e. setting the _reuse_ flags inside mptcp_init_join_sk(). Is there anything else to do, or >> are we now confident in these changes? > > Kishen, to be sure I'm parsing the above question correctly, "these changes" == the "replace per-addr listener sockets" series this email thread is part of? > That's correct. I was referring to this "replace per-addr listener sockets" series. > If so, yes I think the reuse flags are part of moving forward with this, in addition to what you mention below. > >> >> I think the group needs to make a call on our path forward (one way or the other) so that >> we can also progress on other related work. >> >> On a separate note, I brought together my entire userspace PM series and this on a >> tree to test and found a couple of glitches in this patchset. A few userspace >> PM subflows tests would fail due to that. I can respond later on this thread with the >> specific modifications after I clean things up on my end. > > We will talk about it again at the 10-March meeting, but it sounds like you have some modifications to propose that might be important to consider in a final decision. Do the fixes you have to this patchset involve the bind() issues above or are they totally separate? > My changes are separate and fix a couple of logical errors in this patchset to make things work. The bind() related issues discussed in this thread however was just me analyzing for situations that could lead to races with the application, and was thinking along the lines of Paolo's prior assessments. > > -- > Mat Martineau > Intel
On Wed, 9 Mar 2022, Florian Westphal wrote: > Kishen Maloor <kishen.maloor@intel.com> wrote: >> On 3/9/22 4:53 AM, Florian Westphal wrote: >>> Kishen Maloor <kishen.maloor@intel.com> wrote: >>>>>> Over a newly established MPTCP connection following listen(s1), the PM can issue an >>>>>> ADD_ADDR with B. In light of this change there would be no listener created for B. >>>>>> But if the remote endpoint immediately established a subflow in response (to the >>>>>> ADD_ADDR), then that would create a subflow (connection) socket at B. >>>>>> It appears (and correct me if I'm wrong) that bind(s2, B) would fail after this point (?). >>>>> >>>>> Why would that fail? You can bind x:y even if there is an established >>>>> connection from x:y to q:r. >>>> >>>> If I establish an MPTCP connection using mptcp_connect individually as >>>> Client and Server, then I am unable to bind a 3rd (new) Server process at the Client's >>>> addr+port [1]. Why is this the case? >>> >>> Whats [1]? >>> I suspect this patch series needs following addition in patch 3: >>> >>> diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c >>> --- a/net/mptcp/ctrl.c >>> +++ b/net/mptcp/ctrl.c >>> @@ -337,6 +337,8 @@ static int mptcp_init_join_sk(struct net *net, struct sock *sk, struct mptcp_joi >>> if (!tb) >>> return -ENOMEM; >>> >>> + ssock->sk->sk_reuse = 1; >>> + ssock->sk->sk_reuseport = 1; >>> inet_csk(ssock->sk)->icsk_bind_hash = tb; >>> return 0; >>> } >>> >>> After that, follwing sequence should work: >>> >>> 1. bind(0.0.0.0, p1) // listen, accept etc, initial subflow established >>> 2. announce p2 >>> 3. receive join on addr, p2 >>> 4. bind(0.0.0.0, p2) >>> >>> 4) should work because sk used for endpoint in 3) has reuse flag set >>> and is not in listen state. >>> >>> cf. include/net/inet_hashtables.h, line 47: >>> 2) If all sockets have sk->sk_reuse set, and none of them >>> TCP_LISTEN state, the port may be shared. >>> >> >> Wouldn't 4) fail if the socket being bound at the time does not have the SO_REUSExxx flag(s) set? > > Yes, it needs SO_REUSEADDR set. > >> If so, that would be application level thing and in that situation we don't have a way to >> avoid a race. Whereas when we require an explicit listener, we could have the kernel take a step >> back (and not create a listener) to break the race. > > Uh, what? Sorry, I am totally lost. I have no idea what the problem is > that we're solving here. > > EOD, I am out of ideas. Feel free to toss this patchset, I have no idea > what to do. > Hi Florian - After the meeting discussion today, I think we should shelve the pernet listeners for now. This series did get us a lot closer to "handle MP_JOINs everywhere" behavior, but the corner cases seemed to be pulling us in to more TCP changes. More details: https://lore.kernel.org/mptcp/48686ee-4d79-c9fd-35d5-593b9ec9742b@linux.intel.com/ -- Mat Martineau Intel
diff --git a/include/net/mptcp.h b/include/net/mptcp.h index b914e63afc13..b8939d7ea12e 100644 --- a/include/net/mptcp.h +++ b/include/net/mptcp.h @@ -189,6 +189,7 @@ int mptcp_subflow_init_cookie_req(struct request_sock *req, struct sk_buff *skb); __be32 mptcp_get_reset_option(const struct sk_buff *skb); +struct sock *__mptcp_handle_join(int af, struct sk_buff *skb); static inline __be32 mptcp_reset_option(const struct sk_buff *skb) { @@ -198,10 +199,20 @@ static inline __be32 mptcp_reset_option(const struct sk_buff *skb) return htonl(0u); } -static inline struct sock *mptcp_handle_join4(struct sk_buff *skb) +static inline struct sock *mptcp_handle_join(struct sk_buff *skb, int af) { + const struct tcphdr *th = tcp_hdr(skb); + + if (th->syn && !th->ack && !th->rst && !th->fin) + return __mptcp_handle_join(af, skb); + return NULL; } + +static inline struct sock *mptcp_handle_join4(struct sk_buff *skb) +{ + return mptcp_handle_join(skb, AF_INET); +} #else static inline void mptcp_init(void) @@ -284,14 +295,18 @@ static inline struct sock *mptcp_handle_join4(struct sk_buff *skb) { return NULL #if IS_ENABLED(CONFIG_MPTCP_IPV6) int mptcpv6_init(void); +int mptcpv6_init_net(struct net *net); +void mptcpv6_exit_net(struct net *net); void mptcpv6_handle_mapped(struct sock *sk, bool mapped); static inline struct sock *mptcp_handle_join6(struct sk_buff *skb) { - return NULL; + return mptcp_handle_join(skb, AF_INET6); } #elif IS_ENABLED(CONFIG_IPV6) static inline int mptcpv6_init(void) { return 0; } +static inline int mptcpv6_init_net(struct net *net) { return 0; } +static inline void mptcpv6_exit_net(struct net *net) { } static inline void mptcpv6_handle_mapped(struct sock *sk, bool mapped) { } static inline struct sock *mptcp_handle_join6(struct sk_buff *skb) { return NULL; } #endif diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 2f7a621aa24d..b414e2f77fa3 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -2256,13 +2256,22 @@ static struct inet_protosw tcpv6_protosw = { static int __net_init tcpv6_net_init(struct net *net) { - return inet_ctl_sock_create(&net->ipv6.tcp_sk, PF_INET6, - SOCK_RAW, IPPROTO_TCP, net); + int err = inet_ctl_sock_create(&net->ipv6.tcp_sk, PF_INET6, + SOCK_RAW, IPPROTO_TCP, net); + if (err) + return err; + + err = mptcpv6_init_net(net); + if (err) + inet_ctl_sock_destroy(net->ipv6.tcp_sk); + + return err; } static void __net_exit tcpv6_net_exit(struct net *net) { inet_ctl_sock_destroy(net->ipv6.tcp_sk); + mptcpv6_exit_net(net); } static struct pernet_operations tcpv6_net_ops = { @@ -2287,15 +2296,9 @@ int __init tcpv6_init(void) if (ret) goto out_tcpv6_protosw; - ret = mptcpv6_init(); - if (ret) - goto out_tcpv6_pernet_subsys; - out: return ret; -out_tcpv6_pernet_subsys: - unregister_pernet_subsys(&tcpv6_net_ops); out_tcpv6_protosw: inet6_unregister_protosw(&tcpv6_protosw); out_tcpv6_protocol: diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c index ae20b7d92e28..c7370c5147df 100644 --- a/net/mptcp/ctrl.c +++ b/net/mptcp/ctrl.c @@ -12,6 +12,7 @@ #include <net/netns/generic.h> #include "protocol.h" +#include "mib.h" #define MPTCP_SYSCTL_PATH "net/mptcp" @@ -21,6 +22,12 @@ static int mptcp_pernet_id; static int mptcp_pm_type_max = __MPTCP_PM_TYPE_MAX; #endif +struct mptcp_join_sk { + struct sock *sk; + struct inet_bind_bucket *tb; + struct inet_bind_hashbucket head; +}; + struct mptcp_pernet { #ifdef CONFIG_SYSCTL struct ctl_table_header *ctl_table_hdr; @@ -32,6 +39,18 @@ struct mptcp_pernet { u8 checksum_enabled; u8 allow_join_initial_addr_port; u8 pm_type; + + /* pernet listener to handle mptcp join requests + * based on the mptcp token. + * + * Has to be pernet because tcp uses + * sock_net(sk_listener) to obtain the net namespace for + * the syn/ack route lookup. + */ + struct mptcp_join_sk join4; +#if IS_ENABLED(CONFIG_MPTCP_IPV6) + struct mptcp_join_sk join6; +#endif }; static struct mptcp_pernet *mptcp_get_pernet(const struct net *net) @@ -185,13 +204,190 @@ static void mptcp_pernet_del_table(struct mptcp_pernet *pernet) {} #endif /* CONFIG_SYSCTL */ +static void add_mptcp_rst(struct sk_buff *skb) +{ + struct mptcp_ext *ext = skb_ext_add(skb, SKB_EXT_MPTCP); + + if (ext) { + memset(ext, 0, sizeof(*ext)); + ext->reset_reason = MPTCP_RST_EMPTCP; + } +} + +struct sock *__mptcp_handle_join(int af, struct sk_buff *skb) +{ + struct mptcp_options_received mp_opt; + struct mptcp_pernet *pernet; + struct mptcp_sock *msk; + struct socket *ssock; + struct sock *lsk; + struct net *net; + + /* paranoia check: don't allow 0 destination port, + * else __inet_inherit_port will insert the child socket + * into the phony hash slot of the pernet listener. + */ + if (tcp_hdr(skb)->dest == 0) + return NULL; + + mptcp_get_options(skb, &mp_opt); + + if (!(mp_opt.suboptions & OPTIONS_MPTCP_MPJ)) + return NULL; + + net = dev_net(skb_dst(skb)->dev); + if (!mptcp_is_enabled(net)) + return NULL; + + /* RFC8684: If the token is unknown [..], the receiver will send + * back a reset (RST) signal, analogous to an unknown port in TCP, + * containing an MP_TCPRST option (Section 3.6) [..] + */ + msk = mptcp_token_get_sock(net, mp_opt.token); + if (!msk) { + add_mptcp_rst(skb); + return NULL; + } + + if (!mptcp_pm_sport_in_anno_list(msk, af, skb)) { + sock_put((struct sock *)msk); + MPTCP_INC_STATS(net, MPTCP_MIB_MISMATCHPORTSYNRX); + add_mptcp_rst(skb); + return NULL; + } + + sock_put((struct sock *)msk); + pernet = mptcp_get_pernet(net); + + switch (af) { + case AF_INET: + lsk = pernet->join4.sk; + break; +#if IS_ENABLED(CONFIG_MPTCP_IPV6) + case AF_INET6: + lsk = pernet->join6.sk; + break; +#endif + default: + WARN_ON_ONCE(1); + return NULL; + } + + ssock = __mptcp_nmpc_socket(mptcp_sk(lsk)); + if (WARN_ON(!ssock)) + return NULL; + + return ssock->sk; +} + +static struct socket *mptcp_create_join_listen_socket(struct net *net, int af) +{ + struct socket *s, *ssock; + int err; + + err = sock_create_kern(net, af, SOCK_STREAM, IPPROTO_MPTCP, &s); + if (err) + return ERR_PTR(err); + + ssock = __mptcp_nmpc_socket(mptcp_sk(s->sk)); + if (!ssock) { + err = -EINVAL; + goto out; + } + + ssock->sk->sk_max_ack_backlog = SOMAXCONN; + inet_sk_state_store(ssock->sk, TCP_LISTEN); + + s->sk->sk_max_ack_backlog = SOMAXCONN; + inet_sk_state_store(s->sk, TCP_LISTEN); + + s->sk->sk_net_refcnt = 1; + get_net_track(net, &s->sk->ns_tracker, GFP_KERNEL); + sock_inuse_add(net, 1); + + return s; +out: + sock_release(s); + return ERR_PTR(err); +} + +static int mptcp_init_join_sk(struct net *net, struct sock *sk, struct mptcp_join_sk *join_sk) +{ + struct socket *ssock = __mptcp_nmpc_socket(mptcp_sk(sk)); + struct inet_hashinfo *table = ssock->sk->sk_prot->h.hashinfo; + struct inet_bind_bucket *tb; + + spin_lock_init(&join_sk->head.lock); + INIT_HLIST_HEAD(&join_sk->head.chain); + + /* Our "listen socket" isn't bound to any address or port. + * Conceptually, SYN packet with mptcp join request are steered to + * this pernet socket just like TPROXY steals arbitrary connection + * requests to assign them to listening socket with different + * address or port. + * + * The bind_bucket is needed for sake of __inet_inherit_port(), + * so it can place the new child socket in the correct + * bind_bucket slot. + * + * A phony head is used to hide this socket from normal sk loookup. + */ + tb = inet_bind_bucket_create(table->bind_bucket_cachep, + net, &join_sk->head, 0, 0); + if (!tb) + return -ENOMEM; + + inet_csk(ssock->sk)->icsk_bind_hash = tb; + return 0; +} + static int __net_init mptcp_net_init(struct net *net) { struct mptcp_pernet *pernet = mptcp_get_pernet(net); + struct socket *sock; + int err; mptcp_pernet_set_defaults(pernet); - return mptcp_pernet_new_table(net, pernet); + err = mptcp_pernet_new_table(net, pernet); + if (err) + return err; + + sock = mptcp_create_join_listen_socket(net, AF_INET); + if (IS_ERR(sock)) { + err = PTR_ERR(sock); + goto out_table; + } + + err = mptcp_init_join_sk(net, sock->sk, &pernet->join4); + if (err) { + sock_release(sock); + goto out_table; + } + + /* struct sock is still reachable via sock->sk_socket backpointer */ + pernet->join4.sk = sock->sk; + return err; + +out_table: + if (!net_eq(net, &init_net)) + mptcp_pernet_del_table(pernet); + return err; +} + +static void __net_exit mptcp_exit_join_sk(struct mptcp_join_sk *jsk) +{ + struct socket *ssock = __mptcp_nmpc_socket(mptcp_sk(jsk->sk)); + struct inet_bind_bucket *tb; + struct inet_hashinfo *table; + + table = ssock->sk->sk_prot->h.hashinfo; + + tb = inet_csk(ssock->sk)->icsk_bind_hash; + inet_bind_bucket_destroy(table->bind_bucket_cachep, tb); + + ssock = jsk->sk->sk_socket; + sock_release(ssock); } /* Note: the callback will only be called per extra netns */ @@ -200,6 +396,7 @@ static void __net_exit mptcp_net_exit(struct net *net) struct mptcp_pernet *pernet = mptcp_get_pernet(net); mptcp_pernet_del_table(pernet); + mptcp_exit_join_sk(&pernet->join4); } static struct pernet_operations mptcp_pernet_ops = { @@ -219,12 +416,36 @@ void __init mptcp_init(void) } #if IS_ENABLED(CONFIG_MPTCP_IPV6) -int __init mptcpv6_init(void) +int __net_init mptcpv6_init_net(struct net *net) { + struct mptcp_pernet *pernet = mptcp_get_pernet(net); + struct socket *sock; int err; - err = mptcp_proto_v6_init(); + if (net_eq(net, &init_net)) { + err = mptcp_proto_v6_init(); + if (err) + return err; + } + + sock = mptcp_create_join_listen_socket(net, AF_INET6); + if (IS_ERR(sock)) + return PTR_ERR(sock); - return err; + err = mptcp_init_join_sk(net, sock->sk, &pernet->join6); + if (err) { + sock_release(sock); + return err; + } + + pernet->join6.sk = sock->sk; + return 0; +} + +void __net_exit mptcpv6_exit_net(struct net *net) +{ + struct mptcp_pernet *pernet = mptcp_get_pernet(net); + + mptcp_exit_join_sk(&pernet->join6); } #endif diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 3cb975227d12..bc7108ed453c 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -3794,7 +3794,7 @@ static struct inet_protosw mptcp_v6_protosw = { .flags = INET_PROTOSW_ICSK, }; -int __init mptcp_proto_v6_init(void) +int __net_init mptcp_proto_v6_init(void) { int err; diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h index 6b2d7f60c8ad..7ec2513e1c2f 100644 --- a/net/mptcp/protocol.h +++ b/net/mptcp/protocol.h @@ -648,7 +648,7 @@ static inline bool mptcp_has_another_subflow(struct sock *ssk) void __init mptcp_proto_init(void); #if IS_ENABLED(CONFIG_MPTCP_IPV6) -int __init mptcp_proto_v6_init(void); +int __net_init mptcp_proto_v6_init(void); #endif struct sock *mptcp_sk_clone(const struct sock *sk, diff --git a/net/mptcp/subflow.c b/net/mptcp/subflow.c index 77da5f744a17..67a4c698602d 100644 --- a/net/mptcp/subflow.c +++ b/net/mptcp/subflow.c @@ -116,6 +116,9 @@ static void subflow_init_req(struct request_sock *req, const struct sock *sk_lis static bool subflow_use_different_sport(struct mptcp_sock *msk, const struct sock *sk) { + if (inet_sk(sk)->inet_sport == 0) + return true; + return inet_sk(sk)->inet_sport != inet_sk((struct sock *)msk)->inet_sport; } @@ -216,11 +219,6 @@ static int subflow_check_req(struct request_sock *req, pr_debug("syn inet_sport=%d %d", ntohs(inet_sk(sk_listener)->inet_sport), ntohs(inet_sk((struct sock *)subflow_req->msk)->inet_sport)); - if (!mptcp_pm_sport_in_anno_list(subflow_req->msk, - sk_listener->sk_family, skb)) { - SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_MISMATCHPORTSYNRX); - return -EPERM; - } SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_JOINPORTSYNRX); }
Currently mptcp adds kernel-based listener socket for all netlink-configured mptcp address endpoints. This has caveats because kernel may interfere with unrelated programs that use same address/port pairs. RFC 8684 says: Demultiplexing subflow SYNs MUST be done using the token; this is unlike traditional TCP, where the destination port is used for demultiplexing SYN packets. Once a subflow is set up, demultiplexing packets is done using the 5-tuple, as in traditional TCP. This patch deviates from this in that it retains the existing checks of verifying the incoming requests destination vs. the list of announced addresses. If the request is to an address that was not assigned, its treated like an invalid token, i.e. we send a tcp reset with mptcp error specific code is returned. The checks that do this are moved from subflow specific code to the new hook, this allows us to perform the check at an earlier stage. Furthermore, TCP-only listeners take precedence: An MPTCP peer MUST NOT announce addr:port pairs that are already in use by a non-mptcp listener. This could be changed, but it requires move of mptcp_handle_join() hook *before* the tcp port demux, i.e. an additional conditional in hotpath. As-is, the additional conditional (syn && !rst && ...) is placed in the 'no socket found' path. The pernet "listening" socket is hidden from userspace, its not part of any hashes and not bound to any address/port. TPROXY-like semantics apply: If tcp demux cannot find a port for a given packet, check if the packet is a syn packet with a valid join token. If so, the pernet listener is returned and tcp processing resumes. Otherwise, handling is identical. Signed-off-by: Florian Westphal <fw@strlen.de> --- include/net/mptcp.h | 19 +++- net/ipv6/tcp_ipv6.c | 19 ++-- net/mptcp/ctrl.c | 229 ++++++++++++++++++++++++++++++++++++++++++- net/mptcp/protocol.c | 2 +- net/mptcp/protocol.h | 2 +- net/mptcp/subflow.c | 8 +- 6 files changed, 258 insertions(+), 21 deletions(-)