| Message ID | 20250312083907.1931644-1-edumazet@google.com (mailing list archive) |
|---|---|
| State | New |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [net-next] tcp: cache RTAX_QUICKACK metric in a hot cache line | expand |
On Wed, Mar 12, 2025 at 9:39 AM Eric Dumazet <edumazet@google.com> wrote: > > tcp_in_quickack_mode() is called from input path for small packets. > > It calls __sk_dst_get() which reads sk->sk_dst_cache which has been > put in sock_read_tx group (for good reasons). > > Then dst_metric(dst, RTAX_QUICKACK) also needs extra cache line misses. > > Cache RTAX_QUICKACK in icsk->icsk_ack.dst_quick_ack to no longer pull > these cache lines for the cases a delayed ACK is scheduled. > > After this patch TCP receive path does not longer access sock_read_tx > group. > > Signed-off-by: Eric Dumazet <edumazet@google.com> Reviewed-by: Jason Xing <kerneljasonxing@gmail.com>
On Wed, Mar 12, 2025 at 4:39 AM Eric Dumazet <edumazet@google.com> wrote: > > tcp_in_quickack_mode() is called from input path for small packets. > > It calls __sk_dst_get() which reads sk->sk_dst_cache which has been > put in sock_read_tx group (for good reasons). > > Then dst_metric(dst, RTAX_QUICKACK) also needs extra cache line misses. > > Cache RTAX_QUICKACK in icsk->icsk_ack.dst_quick_ack to no longer pull > these cache lines for the cases a delayed ACK is scheduled. > > After this patch TCP receive path does not longer access sock_read_tx > group. > > Signed-off-by: Eric Dumazet <edumazet@google.com> > --- Great! Indeed, from double-checking the code, it looks like with this change there are no fetches of dst RTAX_ values outside of the connection initialization code paths. That's great. Thanks, Eric! Reviewed-by: Neal Cardwell <ncardwell@google.com> neal
diff --git a/include/net/inet_connection_sock.h b/include/net/inet_connection_sock.h index d9978ffacc970efd308d0011a094aec41b561e65..f736d3097e43d97ee32f5d31f0e566536fe05a35 100644 --- a/include/net/inet_connection_sock.h +++ b/include/net/inet_connection_sock.h @@ -117,7 +117,8 @@ struct inet_connection_sock { #define ATO_BITS 8 __u32 ato:ATO_BITS, /* Predicted tick of soft clock */ lrcv_flowlabel:20, /* last received ipv6 flowlabel */ - unused:4; + dst_quick_ack:1, /* cache dst RTAX_QUICKACK */ + unused:3; unsigned long timeout; /* Currently scheduled timeout */ __u32 lrcvtime; /* timestamp of last received data packet */ __u16 last_seg_size; /* Size of last incoming segment */ diff --git a/net/core/sock.c b/net/core/sock.c index a0598518ce898f53825f15ec78249103a3ff8306..323892066def8ba517ff59f98f2e4ab47edd4e63 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2565,8 +2565,12 @@ void sk_setup_caps(struct sock *sk, struct dst_entry *dst) u32 max_segs = 1; sk->sk_route_caps = dst->dev->features; - if (sk_is_tcp(sk)) + if (sk_is_tcp(sk)) { + struct inet_connection_sock *icsk = inet_csk(sk); + sk->sk_route_caps |= NETIF_F_GSO; + icsk->icsk_ack.dst_quick_ack = dst_metric(dst, RTAX_QUICKACK); + } if (sk->sk_route_caps & NETIF_F_GSO) sk->sk_route_caps |= NETIF_F_GSO_SOFTWARE; if (unlikely(sk->sk_gso_disabled)) diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 4e221234808898131a462bc93ee4c9c0ae04309e..5bf8868ca2b56919b15e0c99de83210ed05ad6a7 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -333,9 +333,8 @@ static void tcp_enter_quickack_mode(struct sock *sk, unsigned int max_quickacks) static bool tcp_in_quickack_mode(struct sock *sk) { const struct inet_connection_sock *icsk = inet_csk(sk); - const struct dst_entry *dst = __sk_dst_get(sk); - return (dst && dst_metric(dst, RTAX_QUICKACK)) || + return icsk->icsk_ack.dst_quick_ack || (icsk->icsk_ack.quick && !inet_csk_in_pingpong_mode(sk)); }
tcp_in_quickack_mode() is called from input path for small packets. It calls __sk_dst_get() which reads sk->sk_dst_cache which has been put in sock_read_tx group (for good reasons). Then dst_metric(dst, RTAX_QUICKACK) also needs extra cache line misses. Cache RTAX_QUICKACK in icsk->icsk_ack.dst_quick_ack to no longer pull these cache lines for the cases a delayed ACK is scheduled. After this patch TCP receive path does not longer access sock_read_tx group. Signed-off-by: Eric Dumazet <edumazet@google.com> --- include/net/inet_connection_sock.h | 3 ++- net/core/sock.c | 6 +++++- net/ipv4/tcp_input.c | 3 +-- 3 files changed, 8 insertions(+), 4 deletions(-)