From patchwork Sun Mar 2 12:42:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13997799 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qv1-f73.google.com (mail-qv1-f73.google.com [209.85.219.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3AED2189BB0 for ; Sun, 2 Mar 2025 12:42:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740919362; cv=none; b=ZZOA66eU+VWlgJag1iezXIVQ0dv6OVhNBQqiy5L+AcVFaLJYK55Ccb4o4UH4+kyeYY8H/U1BZzGr/C17LQoSeuIhAj/wYUsqt0mAmR9a+RjonbvVs0gGPRiIaxzxA9gJPZZNSLDnSCmI0i9sLSRBiVVcslGVfWPEJAe4CGH78ZU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740919362; c=relaxed/simple; bh=mu8fr4NZwnln/8Dj8PQO9wWaoxOrMuLXb1ng7HtG/Xw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=DjVMttHMIWP7R9N33RTVLET4zGO9irDLKNFkt2FBpJlvFKS8Bky++p09+k25/DxPNMt1JIJz5PgZ8lJuqzWsT/+T7Ex/hSP348ufYQcr2LQrPlbXC3xkqdks8htyfJ923pBfXxL8jrFLcGmPB/N1qImUspuy60kk07lFrNZ+rcA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sQxydo7l; arc=none smtp.client-ip=209.85.219.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sQxydo7l" Received: by mail-qv1-f73.google.com with SMTP id 6a1803df08f44-6e8c8641ad6so6477466d6.3 for ; Sun, 02 Mar 2025 04:42:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740919360; x=1741524160; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Dj9rZcr8i8dq43YUEnSl7PBqvKsMqeBhFWze2LAWP6k=; b=sQxydo7lGlmd8AVb6OG1CIqcPbwI3HbFD+chgNUWWBNl/mKflKWd/NsvPdyXu62zor 7+qbZrfopX6FNHq2ChUv6baqYbmRrVAsbJ+NNbfIbibX4z6HHOawqIWxLaJF/QNP/txr 763/PfQF2wM2zKE7b1ZPEmt/AVgIU4qv1iuH0y/am8jjPkyMBfp9X0ott96YqeryKuoW rwP85y7gS13qSBMtIJJNUYhcOakRkBWlJDlqait80Y7Ww1e5nwrOAGOdoGQIFo5Vo8BF bjuTL7wng/o9vdyIIoU5akny1iweV/wrTw4nJ0Apw36oEseVJF9ZRmxWxIpK9/Hec4eF HRVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740919360; x=1741524160; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Dj9rZcr8i8dq43YUEnSl7PBqvKsMqeBhFWze2LAWP6k=; b=gmWue7stQFLQY0KFkBvBwpw1oXVesyMFOLVLtOg2bIOq8iHlBjot6SR5WmXYhWgKrb l7CwRZRJ/Uv3mP8mIxRxX2cH8nvN/9ZhQUUwUwwo/JLvbc/WYsJl08VtjXzYnkRtVfCf 2VotDUf7zt+/ALXsq+gk5cUoPJ7oOB7r8nv4NHN2ennV+p/x3M4aYZByZtTIvFOFTJha LTdprWVzGzE34r0P6O2yFylnxqhDUzxpAz4Dt11oVXoc+qEBeVqqcZ01hMhNAAqlc/v3 AnfXoTGSJmA2ZP0wJywbqhWE8crOigO+SdzMSczXdxd0+eFUQ5N9wvo7yX4ZHneZ0D9p Jsfw== X-Forwarded-Encrypted: i=1; AJvYcCXk0/aIQYZPoWYOM/sh/2XZmpOn9Ab7pVOEbyn5viFqDrwD3KKCaCGhWNviZzNgt8URzCx3Sr0=@vger.kernel.org X-Gm-Message-State: AOJu0Yz9N845D1dJlKEdpRGlv2qISQ+d8bQzzc25IER3bNzy1GvdHP40 1Vae2pQAkJh82CO0hcRszHprr8x0ebCCsREPCJh7MF+w8/aRmj1clUldG7rS6gW8wWsMCLfN+OV I5x21MHUeQQ== X-Google-Smtp-Source: AGHT+IF0+KIfwAvj5s940/ynCmLdKtOxsr25I8sQdPNQ5jfi5CxTZckRmtzudvhBzPuKkCbGi6UhdBlLDK3bUQ== X-Received: from qvxf30.prod.google.com ([2002:ad4:559e:0:b0:6e4:6d4f:eb58]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6214:e88:b0:6e2:4d70:1fbd with SMTP id 6a1803df08f44-6e8a0c8221bmr151205666d6.6.1740919360143; Sun, 02 Mar 2025 04:42:40 -0800 (PST) Date: Sun, 2 Mar 2025 12:42:34 +0000 In-Reply-To: <20250302124237.3913746-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250302124237.3913746-1-edumazet@google.com> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog Message-ID: <20250302124237.3913746-2-edumazet@google.com> Subject: [PATCH net-next 1/4] tcp: use RCU in __inet{6}_check_established() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni , Neal Cardwell Cc: Kuniyuki Iwashima , Jason Xing , Simon Horman , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org When __inet_hash_connect() has to try many 4-tuples before finding an available one, we see a high spinlock cost from __inet_check_established() and/or __inet6_check_established(). This patch adds an RCU lookup to avoid the spinlock acquisition when the 4-tuple is found in the hash table. Note that there are still spin_lock_bh() calls in __inet_hash_connect() to protect inet_bind_hashbucket, this will be fixed later in this series. Signed-off-by: Eric Dumazet Reviewed-by: Jason Xing Tested-by: Jason Xing Reviewed-by: Kuniyuki Iwashima --- net/ipv4/inet_hashtables.c | 19 ++++++++++++++++--- net/ipv6/inet6_hashtables.c | 19 ++++++++++++++++--- 2 files changed, 32 insertions(+), 6 deletions(-) diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c index 9bfcfd016e18275fb50fea8d77adc8a64fb12494..46d39aa2199ec3a405b50e8e85130e990d2c26b7 100644 --- a/net/ipv4/inet_hashtables.c +++ b/net/ipv4/inet_hashtables.c @@ -551,11 +551,24 @@ static int __inet_check_established(struct inet_timewait_death_row *death_row, unsigned int hash = inet_ehashfn(net, daddr, lport, saddr, inet->inet_dport); struct inet_ehash_bucket *head = inet_ehash_bucket(hinfo, hash); - spinlock_t *lock = inet_ehash_lockp(hinfo, hash); - struct sock *sk2; - const struct hlist_nulls_node *node; struct inet_timewait_sock *tw = NULL; + const struct hlist_nulls_node *node; + struct sock *sk2; + spinlock_t *lock; + + rcu_read_lock(); + sk_nulls_for_each(sk2, node, &head->chain) { + if (sk2->sk_hash != hash || + !inet_match(net, sk2, acookie, ports, dif, sdif)) + continue; + if (sk2->sk_state == TCP_TIME_WAIT) + break; + rcu_read_unlock(); + return -EADDRNOTAVAIL; + } + rcu_read_unlock(); + lock = inet_ehash_lockp(hinfo, hash); spin_lock(lock); sk_nulls_for_each(sk2, node, &head->chain) { diff --git a/net/ipv6/inet6_hashtables.c b/net/ipv6/inet6_hashtables.c index 9ec05e354baa69d14e88da37f5a9fce11e874e35..3604a5cae5d29a25d24f9513308334ff8e64b083 100644 --- a/net/ipv6/inet6_hashtables.c +++ b/net/ipv6/inet6_hashtables.c @@ -276,11 +276,24 @@ static int __inet6_check_established(struct inet_timewait_death_row *death_row, const unsigned int hash = inet6_ehashfn(net, daddr, lport, saddr, inet->inet_dport); struct inet_ehash_bucket *head = inet_ehash_bucket(hinfo, hash); - spinlock_t *lock = inet_ehash_lockp(hinfo, hash); - struct sock *sk2; - const struct hlist_nulls_node *node; struct inet_timewait_sock *tw = NULL; + const struct hlist_nulls_node *node; + struct sock *sk2; + spinlock_t *lock; + + rcu_read_lock(); + sk_nulls_for_each(sk2, node, &head->chain) { + if (sk2->sk_hash != hash || + !inet6_match(net, sk2, saddr, daddr, ports, dif, sdif)) + continue; + if (sk2->sk_state == TCP_TIME_WAIT) + break; + rcu_read_unlock(); + return -EADDRNOTAVAIL; + } + rcu_read_unlock(); + lock = inet_ehash_lockp(hinfo, hash); spin_lock(lock); sk_nulls_for_each(sk2, node, &head->chain) { From patchwork Sun Mar 2 12:42:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13997800 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qv1-f74.google.com (mail-qv1-f74.google.com [209.85.219.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33A531E3DE5 for ; Sun, 2 Mar 2025 12:42:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740919363; cv=none; b=Px58zkxNc8r4T7UqgaCTokduq7f0Xd0QQn51Q6n/9wHyPUQ0Z4uYwt4/MGcjf5BI12qLxzvU+tphsE0T3RgWA30+d5dcJOORQHjNAVVtPkVZrwzvWXL9Qom1BdIcSQWcUtUFY3LtT9SdybOT/bFLt9bwl46qEVTCKGKDIpLHEIk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740919363; c=relaxed/simple; bh=p93rH5PdMjPPgGz6GV5q/VnBL2KH+Cct81/A/oCWYfI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=j9T4Ps/Q8FJgBSOhnQnvltt5JQpEgdDIQFpuDyZJUAlRbxDWVzJMd1659pZgiPnYXZawns/2gsEraQpd39seqS/cfm5puyYxFqJ5Rzy14ILwwthzhFRvnfhS9ckMlK3FG5iMuQRcUVQ+wWZAJd4rYunjyNl6qTaYND6gpM/RuZs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=kxeHDecU; arc=none smtp.client-ip=209.85.219.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="kxeHDecU" Received: by mail-qv1-f74.google.com with SMTP id 6a1803df08f44-6e8993deec9so89313716d6.3 for ; Sun, 02 Mar 2025 04:42:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740919361; x=1741524161; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=s0JdeHQMm3xtUgmOsCtNWrZ/gZJbYB8dsrL51tkmKLQ=; b=kxeHDecUZGZyMUi7JW/YyLR5Ur65t9YAS7MVZb+JD+sRp3Er7SatSxWpgyJyR61uK6 Xxl36Iln6v8+e4aSZeZDklrOSPxFvtOq6noT/iclQ73H5vHzfbOn4vqDL8ycaWdhPb+V HnAHp5EHkWRHmrDHvGe2YsqodxqNrD9HZ5FR+12GL8QTESEAi0akkfuBVVyeOyTE+Ys/ lxvhIm8FKDb7tUUQnp5M+P6U/F92WM7pARZW9wMEBH1zroTJUhZqn8KhhwWunEtJvhtz elXl73lm94rJ+pdPBQGMDKs5aH+CZ+ITp3efPN04C0sjDUPY8V7iXXWBxX7by1QZVj0u FQlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740919361; x=1741524161; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=s0JdeHQMm3xtUgmOsCtNWrZ/gZJbYB8dsrL51tkmKLQ=; b=bJgByXnMyJCPszMEFx7KNy/W0ydjY+vJlSHxYRTxQlAVKsfiwsncrGt5/3KE5hAhYK JNgTMp54CrbUuS8v+qFKaYq6wAtoC7oMWgtpUxcP5bPPQ1fz05AOCICU6XomcGsJb/xX 5FPcX+aRU9SCUIPTgpiCMqZon2eQL0DEj+gonsmxvhvOQU+yozQwz7GmNGpgqMvAiqWo 9VKZj7ls7SP8KQ/tx0PGMPxGrMupgnEHtfX0nS9qVAVdSmdno2d/+5UHBaekmJuNCZln ZNmWUoG5tNJsM9JI3HoJSjUuhYvNdeVZSY8WCobeF7LIZqF6js2+Uwkk1nSfSsvwsBac HMwQ== X-Forwarded-Encrypted: i=1; AJvYcCUB0F/1PRKRaiXtFv2IDgt71Wv42nPZb1yjjibjAfsPSBulckgFjcYQLQlx9CIYqfOtJ/CP8tA=@vger.kernel.org X-Gm-Message-State: AOJu0YyOVNloDSh1zLKEqcGgPKZ6EAKaC+6amRdg+cSURdxY9k1EMRz/ +nn4vkY2WsCnBJi0He9hLrMTZ7sKO4t+wyt+Hg5XXYh3npFcY/Pz7Pl+D2NzNRuLOtpgJy84xfR hzz3fDWGluw== X-Google-Smtp-Source: AGHT+IFUFVcnKbe1r9qAGX9bBRN0X4yUWdEs+MT8sUa8UNadSW4x4KoYCkOYHHG6ALf39iIwuhb6WIj6VHAkIw== X-Received: from qtbfy11.prod.google.com ([2002:a05:622a:5a0b:b0:471:a4e2:f6f2]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:a05:622a:178d:b0:472:614:19a0 with SMTP id d75a77b69052e-474bc0edbf0mr137297261cf.35.1740919361415; Sun, 02 Mar 2025 04:42:41 -0800 (PST) Date: Sun, 2 Mar 2025 12:42:35 +0000 In-Reply-To: <20250302124237.3913746-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250302124237.3913746-1-edumazet@google.com> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog Message-ID: <20250302124237.3913746-3-edumazet@google.com> Subject: [PATCH net-next 2/4] tcp: optimize inet_use_bhash2_on_bind() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni , Neal Cardwell Cc: Kuniyuki Iwashima , Jason Xing , Simon Horman , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org There is no reason to call ipv6_addr_type(). Instead, use highly optimized ipv6_addr_any() and ipv6_addr_v4mapped(). Signed-off-by: Eric Dumazet Reviewed-by: Jason Xing Reviewed-by: Kuniyuki Iwashima --- net/ipv4/inet_connection_sock.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index bf9ce0c196575910b4b03fca13001979d4326297..b4e514da22b64f02cbd9f6c10698db359055e0cc 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -157,12 +157,10 @@ static bool inet_use_bhash2_on_bind(const struct sock *sk) { #if IS_ENABLED(CONFIG_IPV6) if (sk->sk_family == AF_INET6) { - int addr_type = ipv6_addr_type(&sk->sk_v6_rcv_saddr); - - if (addr_type == IPV6_ADDR_ANY) + if (ipv6_addr_any(&sk->sk_v6_rcv_saddr)) return false; - if (addr_type != IPV6_ADDR_MAPPED) + if (!ipv6_addr_v4mapped(&sk->sk_v6_rcv_saddr)) return true; } #endif From patchwork Sun Mar 2 12:42:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13997801 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qv1-f73.google.com (mail-qv1-f73.google.com [209.85.219.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D14371E500C for ; Sun, 2 Mar 2025 12:42:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740919365; cv=none; b=GgySvS2OW7ZNUZ2P0EXR+9ZUsMrE3b3dhfwVUSTuNaSqNK3OxczyNL6pKd8mC9Xb+KZCJyWBfQqgg82BOyJsCGJuRLF3Hmf/rU3wV488u8155rGarkkkr5vEoFvIvQwVGheKkvipL0WJm9X62n7EnyykxooZUvpII41zJFGIHww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740919365; c=relaxed/simple; bh=RT9lXwDB4DAszEcWZmwZYOGTxbyIsDNtILzbHsxNRQ0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=GRJrajRZSYS7PmUjrhvdKYE7g6dWxQIYca0S3pZ9uJy9r5S7czmtnRP7X7/QLeFq68UIQqL2sb7rd+P3J0iSdMJ7wP9gKc8XonCntcXK6PXy1Zlfw4a/nhrKgY3dvEc/ssRz+viLAk6cqA8tpdWwt1k1EWiKJ6p+aqEQE47Y5k4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Fhvgm0Bo; arc=none smtp.client-ip=209.85.219.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Fhvgm0Bo" Received: by mail-qv1-f73.google.com with SMTP id 6a1803df08f44-6e8b876f694so22547686d6.1 for ; Sun, 02 Mar 2025 04:42:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740919363; x=1741524163; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2DeiUAO4/StKObvmrt6w+tGgiNpuxOkqU3ZU2aa9xmY=; b=Fhvgm0BoP2ZrGr5IRafeLcKCa1M6MoC5sZ2TkXws4PmmtdNR0FkqMwWW1mUsDpznvo Zq3hXG4BmRlFRgDbXI2EdIqoN+AozlkGLohz8MWyKtv+ga7eBsM6dLy3ixGgHvjM9GcA DIrglBoIhTpwHcGDHw9A6g5RigY5EOdBniWcMkhfveZe6TWVxFb/MqZXfYU4E0V3LaGB U5fcJvnDNdWTAGPgKX0PF1hj4xIhhKRoOcFEwEwSkzPlLl+Y+UXF1b9mM0CW9GGvwU21 JGc8zn5qqxIbKD9Yi/Ka171mKxOX+i6Q/Ufrt/RjtBr/wDx65PJKGt0KOhMZae8bkcHO +dVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740919363; x=1741524163; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2DeiUAO4/StKObvmrt6w+tGgiNpuxOkqU3ZU2aa9xmY=; b=dOda3abGu4SIfcpS4i4B9KX1/SjlLboUrxXyQAu9WGp7Qc397ZzFb7O/5JDstHQMS4 52KwO2lAj0w9Rf7DnH7kjYZ3bLyNPAVYdOlBW4RUjDn5BhiVumNVkJM8O+nHLNaiBJgS 9XFyWmN9bcq34Qw3JKK0kmot2q315HS0n3kmxYmSRaAHpXSb2wgfF0x3OiAjK9UX8dn+ D9pH4ha893CPrRM9vUHz6abQJzWanX7S3Ymj0oig4cTo2kfudNgiWYGoeLEffPCj9UOn G3Kc84+6wEXcPSJOoz0EJ0ghuMergcdJO3eFglHByHCCr9g3BgtY2xoVFUvuVwEXkwbb U+rw== X-Forwarded-Encrypted: i=1; AJvYcCW2L3VewzHRTHCndRPJFqSt/H6//N3Csy4KkJ9lAdJYGCzQ2G3bAYNWc1CbDI7uzkyNP2bxlos=@vger.kernel.org X-Gm-Message-State: AOJu0YxbNjqLBMyiUzPJsvejlDvnzSvlNsNf5jU8yW+biEJzP7nFaKqS I1XeHNplilYwwjdrvbixXh8LFg/Jyln57pe87r+qGbxJ4rGMrc/fcoTIKhmCk2kEf49uDfJ3IoC tgAhepoEQ7A== X-Google-Smtp-Source: AGHT+IGvCxubxk91szR6C96RduoJOAtQ/Ox2mr6DJgXyEzkBpOcJ38cE7b2Aay+KB4miWSdtmVdj3XWRMNjDOw== X-Received: from qvbmi8.prod.google.com ([2002:a05:6214:5588:b0:6e8:8dcf:4a50]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6214:f28:b0:6d8:af37:ae5b with SMTP id 6a1803df08f44-6e8a0d9de8dmr138341606d6.43.1740919362776; Sun, 02 Mar 2025 04:42:42 -0800 (PST) Date: Sun, 2 Mar 2025 12:42:36 +0000 In-Reply-To: <20250302124237.3913746-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250302124237.3913746-1-edumazet@google.com> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog Message-ID: <20250302124237.3913746-4-edumazet@google.com> Subject: [PATCH net-next 3/4] tcp: add RCU management to inet_bind_bucket From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni , Neal Cardwell Cc: Kuniyuki Iwashima , Jason Xing , Simon Horman , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Add RCU protection to inet_bind_bucket structure. - Add rcu_head field to the structure definition. - Use kfree_rcu() at destroy time, and remove inet_bind_bucket_destroy() first argument. - Use hlist_del_rcu() and hlist_add_head_rcu() methods. Signed-off-by: Eric Dumazet Reviewed-by: Jason Xing Reviewed-by: Kuniyuki Iwashima --- include/net/inet_hashtables.h | 4 ++-- net/ipv4/inet_connection_sock.c | 2 +- net/ipv4/inet_hashtables.c | 14 +++++++------- net/ipv4/inet_timewait_sock.c | 2 +- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/include/net/inet_hashtables.h b/include/net/inet_hashtables.h index 5eea47f135a421ce8275d4cd83c5771b3f448e5c..73c0e4087fd1a6d0d2a40ab0394165e07b08ed6d 100644 --- a/include/net/inet_hashtables.h +++ b/include/net/inet_hashtables.h @@ -89,6 +89,7 @@ struct inet_bind_bucket { bool fast_ipv6_only; struct hlist_node node; struct hlist_head bhash2; + struct rcu_head rcu; }; struct inet_bind2_bucket { @@ -226,8 +227,7 @@ struct inet_bind_bucket * inet_bind_bucket_create(struct kmem_cache *cachep, struct net *net, struct inet_bind_hashbucket *head, const unsigned short snum, int l3mdev); -void inet_bind_bucket_destroy(struct kmem_cache *cachep, - struct inet_bind_bucket *tb); +void inet_bind_bucket_destroy(struct inet_bind_bucket *tb); bool inet_bind_bucket_match(const struct inet_bind_bucket *tb, const struct net *net, unsigned short port, diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index b4e514da22b64f02cbd9f6c10698db359055e0cc..e93c660340770a76446f97617ba23af32dc136fb 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -598,7 +598,7 @@ int inet_csk_get_port(struct sock *sk, unsigned short snum) if (bhash2_created) inet_bind2_bucket_destroy(hinfo->bind2_bucket_cachep, tb2); if (bhash_created) - inet_bind_bucket_destroy(hinfo->bind_bucket_cachep, tb); + inet_bind_bucket_destroy(tb); } if (head2_lock_acquired) spin_unlock(&head2->lock); diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c index 46d39aa2199ec3a405b50e8e85130e990d2c26b7..b737e13f8459c53428980221355344327c4bc8dd 100644 --- a/net/ipv4/inet_hashtables.c +++ b/net/ipv4/inet_hashtables.c @@ -76,7 +76,7 @@ struct inet_bind_bucket *inet_bind_bucket_create(struct kmem_cache *cachep, tb->fastreuse = 0; tb->fastreuseport = 0; INIT_HLIST_HEAD(&tb->bhash2); - hlist_add_head(&tb->node, &head->chain); + hlist_add_head_rcu(&tb->node, &head->chain); } return tb; } @@ -84,11 +84,11 @@ struct inet_bind_bucket *inet_bind_bucket_create(struct kmem_cache *cachep, /* * Caller must hold hashbucket lock for this tb with local BH disabled */ -void inet_bind_bucket_destroy(struct kmem_cache *cachep, struct inet_bind_bucket *tb) +void inet_bind_bucket_destroy(struct inet_bind_bucket *tb) { if (hlist_empty(&tb->bhash2)) { - __hlist_del(&tb->node); - kmem_cache_free(cachep, tb); + hlist_del_rcu(&tb->node); + kfree_rcu(tb, rcu); } } @@ -201,7 +201,7 @@ static void __inet_put_port(struct sock *sk) } spin_unlock(&head2->lock); - inet_bind_bucket_destroy(hashinfo->bind_bucket_cachep, tb); + inet_bind_bucket_destroy(tb); spin_unlock(&head->lock); } @@ -285,7 +285,7 @@ int __inet_inherit_port(const struct sock *sk, struct sock *child) error: if (created_inet_bind_bucket) - inet_bind_bucket_destroy(table->bind_bucket_cachep, tb); + inet_bind_bucket_destroy(tb); spin_unlock(&head2->lock); spin_unlock(&head->lock); return -ENOMEM; @@ -1162,7 +1162,7 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row, spin_unlock(&head2->lock); if (tb_created) - inet_bind_bucket_destroy(hinfo->bind_bucket_cachep, tb); + inet_bind_bucket_destroy(tb); spin_unlock(&head->lock); if (tw) diff --git a/net/ipv4/inet_timewait_sock.c b/net/ipv4/inet_timewait_sock.c index 337390ba85b4082701f78f1a0913ba47c1741378..aded4bf1bc16d9f1d9fd80d60f41027dd53f38eb 100644 --- a/net/ipv4/inet_timewait_sock.c +++ b/net/ipv4/inet_timewait_sock.c @@ -39,7 +39,7 @@ void inet_twsk_bind_unhash(struct inet_timewait_sock *tw, tw->tw_tb = NULL; tw->tw_tb2 = NULL; inet_bind2_bucket_destroy(hashinfo->bind2_bucket_cachep, tb2); - inet_bind_bucket_destroy(hashinfo->bind_bucket_cachep, tb); + inet_bind_bucket_destroy(tb); __sock_put((struct sock *)tw); } From patchwork Sun Mar 2 12:42:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13997802 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qt1-f202.google.com (mail-qt1-f202.google.com [209.85.160.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 268BB1E3DE5 for ; Sun, 2 Mar 2025 12:42:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740919366; cv=none; b=DHcQdVB70reWyM8SFOxKcbG/Wk+/7ezcQ+YTFOQHXB+1CDRtKW92ejU3vyd+vVsR1uXHuadKWPTPqbm4eDNk5irF1A8fCvuj49nK1jkuGaFu51pPvR3Eo3l4Whf9uFVDpEbmKuSpUHgQAYe/XPjk6Ctg6+97W7wWhVZoZIjc83E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740919366; c=relaxed/simple; bh=ItXP/zyRAh+/67PmK7Z+y9460lXos/4E+fsmUnOBZBA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=q+muPrvc5msZpCwt00rQ1cSNCBEwyNlD1aLQPz9qe+7y/dr7MbAfCP54ci4UgWepibJ7GFcPMG5vqFaYqy4kp2wkUbwbRumxrmzr7JR2jQK+Ofv6TWvVo0qiyQMWTv07UL3EGE6NHHKLn6CZl2UV0f3+hDUPHOq7xzMWzPe3tBI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=heFFxyzk; arc=none smtp.client-ip=209.85.160.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="heFFxyzk" Received: by mail-qt1-f202.google.com with SMTP id d75a77b69052e-471ec7f1969so92485911cf.1 for ; Sun, 02 Mar 2025 04:42:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740919364; x=1741524164; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bIbXbn1qyNwXG/XGNxdRDEuAyo93hNv8FKSURSQEbM4=; b=heFFxyzkco9dM3swFNshNCfwIIGaQfk/VQihNIyjTgRgFJIqbCRrDDDscndB8BOudQ 7fA2ywGdrx2vAsdkl84LW4Jd06huEFaZ0cbqZvSQG8Dt/WSpuGQlEzQ2qZgkJyoJoKGV 8Pf3EiQ/W0uUITZtUhOAr1YdmBpNoN+1erDcRws/bpbgIEgkv+oVyb+KFfW1XbjuY6zA 4S5rT9cxTGDkzR45o42/fVCE9IlpXYjU+Rqa7MFJLBMi92l0zGcJFlNnj8X4jT/hGUK6 RG970mrjxXCK7xLDjVGA+ayssHAMDiiC9XJ62Hse8uBKvP+Nf9Oyf3pMEi42I2ZW48Hm GHEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740919364; x=1741524164; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bIbXbn1qyNwXG/XGNxdRDEuAyo93hNv8FKSURSQEbM4=; b=S9fuVkKZAwcpCuf3EB00pfkq3+Ng+Jp7PTKtU6EeGEYKmasR4rOuxq4Glf53bE9xn+ uQ52TPnvbuqQnsYwc2mUOYbQczGF8xrAzspC4YqInxYAX16RYMedgtAKFpOXraeyaJIp p1NZdK6ZMq0fgPUbylNztAWOYfRQIKj9DW+7R2bQjFrk/0dJvDqS50OtRZq1y3Vvh//S G3vaqyNvlDub8+UkaGuTkqrgU1yY1BL1oxfseRWsiSncam7O7IotpHKmYA0Vcs9Jdz3C DOQQO5Ll5pACSlOrG3/yw0HLASuDdwSdVPatSss3q09M0UAT64rY+LMWcg07qejzbpuV zKNw== X-Forwarded-Encrypted: i=1; AJvYcCVxP2C6KPYnM66UGsAvkGvd5HWJ7PIP8N3+ttQqq/XxKXJypXRsCvIsVgVXUI273gPnXWAESn4=@vger.kernel.org X-Gm-Message-State: AOJu0YwOUmAPM9gl51Tq5UK2QBnA2piX2lFvDUGFhKesnwjjF7mYG5xu TEQpvZUUXbbp4VYlYwom+U2O6i4YGj5HdSRUUExuFb9BuV1rrJcnA9z2BmD7O0RTvw6cl9NGJwg HUU1fPJS/JQ== X-Google-Smtp-Source: AGHT+IHof4z7aGHpuS92kAvQXiZc6abSJZ/Z16oiPqjR33sve5qy+MbHUoXtRbDD/TQ8hkxyarz6mnFTA2T4ng== X-Received: from qtbfi5.prod.google.com ([2002:a05:622a:58c5:b0:472:3db:8d3c]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:ac8:5f07:0:b0:471:fa00:fb9b with SMTP id d75a77b69052e-473d8f99c26mr199701181cf.7.1740919364071; Sun, 02 Mar 2025 04:42:44 -0800 (PST) Date: Sun, 2 Mar 2025 12:42:37 +0000 In-Reply-To: <20250302124237.3913746-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250302124237.3913746-1-edumazet@google.com> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog Message-ID: <20250302124237.3913746-5-edumazet@google.com> Subject: [PATCH net-next 4/4] tcp: use RCU lookup in __inet_hash_connect() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni , Neal Cardwell Cc: Kuniyuki Iwashima , Jason Xing , Simon Horman , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org When __inet_hash_connect() has to try many 4-tuples before finding an available one, we see a high spinlock cost from the many spin_lock_bh(&head->lock) performed in its loop. This patch adds an RCU lookup to avoid the spinlock cost. check_established() gets a new @rcu_lookup argument. First reason is to not make any changes while head->lock is not held. Second reason is to not make this RCU lookup a second time after the spinlock has been acquired. Tested: Server: ulimit -n 40000; neper/tcp_crr -T 200 -F 30000 -6 --nolog Client: ulimit -n 40000; neper/tcp_crr -T 200 -F 30000 -6 --nolog -c -H server Before series: utime_start=0.288582 utime_end=1.548707 stime_start=20.637138 stime_end=2002.489845 num_transactions=484453 latency_min=0.156279245 latency_max=20.922042756 latency_mean=1.546521274 latency_stddev=3.936005194 num_samples=312537 throughput=47426.00 perf top on the client: 49.54% [kernel] [k] _raw_spin_lock 25.87% [kernel] [k] _raw_spin_lock_bh 5.97% [kernel] [k] queued_spin_lock_slowpath 5.67% [kernel] [k] __inet_hash_connect 3.53% [kernel] [k] __inet6_check_established 3.48% [kernel] [k] inet6_ehashfn 0.64% [kernel] [k] rcu_all_qs After this series: utime_start=0.271607 utime_end=3.847111 stime_start=18.407684 stime_end=1997.485557 num_transactions=1350742 latency_min=0.014131929 latency_max=17.895073144 latency_mean=0.505675853 # Nice reduction of latency metrics latency_stddev=2.125164772 num_samples=307884 throughput=139866.80 # 190 % increase perf top on client: 56.86% [kernel] [k] __inet6_check_established 17.96% [kernel] [k] __inet_hash_connect 13.88% [kernel] [k] inet6_ehashfn 2.52% [kernel] [k] rcu_all_qs 2.01% [kernel] [k] __cond_resched 0.41% [kernel] [k] _raw_spin_lock Signed-off-by: Eric Dumazet Reviewed-by: Jason Xing Tested-by: Jason Xing Reviewed-by: Kuniyuki Iwashima --- include/net/inet_hashtables.h | 3 +- net/ipv4/inet_hashtables.c | 52 +++++++++++++++++++++++------------ net/ipv6/inet6_hashtables.c | 24 ++++++++-------- 3 files changed, 50 insertions(+), 29 deletions(-) diff --git a/include/net/inet_hashtables.h b/include/net/inet_hashtables.h index 73c0e4087fd1a6d0d2a40ab0394165e07b08ed6d..b12797f13c9a3d66fab99c877d059f9c29c30d11 100644 --- a/include/net/inet_hashtables.h +++ b/include/net/inet_hashtables.h @@ -529,7 +529,8 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row, struct sock *sk, u64 port_offset, int (*check_established)(struct inet_timewait_death_row *, struct sock *, __u16, - struct inet_timewait_sock **)); + struct inet_timewait_sock **, + bool rcu_lookup)); int inet_hash_connect(struct inet_timewait_death_row *death_row, struct sock *sk); diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c index b737e13f8459c53428980221355344327c4bc8dd..d1b5f45ee718410fdf3e78c113c7ebd4a1ddba40 100644 --- a/net/ipv4/inet_hashtables.c +++ b/net/ipv4/inet_hashtables.c @@ -537,7 +537,8 @@ EXPORT_SYMBOL_GPL(__inet_lookup_established); /* called with local bh disabled */ static int __inet_check_established(struct inet_timewait_death_row *death_row, struct sock *sk, __u16 lport, - struct inet_timewait_sock **twp) + struct inet_timewait_sock **twp, + bool rcu_lookup) { struct inet_hashinfo *hinfo = death_row->hashinfo; struct inet_sock *inet = inet_sk(sk); @@ -556,17 +557,17 @@ static int __inet_check_established(struct inet_timewait_death_row *death_row, struct sock *sk2; spinlock_t *lock; - rcu_read_lock(); - sk_nulls_for_each(sk2, node, &head->chain) { - if (sk2->sk_hash != hash || - !inet_match(net, sk2, acookie, ports, dif, sdif)) - continue; - if (sk2->sk_state == TCP_TIME_WAIT) - break; - rcu_read_unlock(); - return -EADDRNOTAVAIL; + if (rcu_lookup) { + sk_nulls_for_each(sk2, node, &head->chain) { + if (sk2->sk_hash != hash || + !inet_match(net, sk2, acookie, ports, dif, sdif)) + continue; + if (sk2->sk_state == TCP_TIME_WAIT) + break; + return -EADDRNOTAVAIL; + } + return 0; } - rcu_read_unlock(); lock = inet_ehash_lockp(hinfo, hash); spin_lock(lock); @@ -1007,7 +1008,8 @@ static u32 *table_perturb; int __inet_hash_connect(struct inet_timewait_death_row *death_row, struct sock *sk, u64 port_offset, int (*check_established)(struct inet_timewait_death_row *, - struct sock *, __u16, struct inet_timewait_sock **)) + struct sock *, __u16, struct inet_timewait_sock **, + bool rcu_lookup)) { struct inet_hashinfo *hinfo = death_row->hashinfo; struct inet_bind_hashbucket *head, *head2; @@ -1025,7 +1027,7 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row, if (port) { local_bh_disable(); - ret = check_established(death_row, sk, port, NULL); + ret = check_established(death_row, sk, port, NULL, false); local_bh_enable(); return ret; } @@ -1061,6 +1063,21 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row, continue; head = &hinfo->bhash[inet_bhashfn(net, port, hinfo->bhash_size)]; + rcu_read_lock(); + hlist_for_each_entry_rcu(tb, &head->chain, node) { + if (!inet_bind_bucket_match(tb, net, port, l3mdev)) + continue; + if (tb->fastreuse >= 0 || tb->fastreuseport >= 0) { + rcu_read_unlock(); + goto next_port; + } + if (!check_established(death_row, sk, port, &tw, true)) + break; + rcu_read_unlock(); + goto next_port; + } + rcu_read_unlock(); + spin_lock_bh(&head->lock); /* Does not bother with rcv_saddr checks, because @@ -1070,12 +1087,12 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row, if (inet_bind_bucket_match(tb, net, port, l3mdev)) { if (tb->fastreuse >= 0 || tb->fastreuseport >= 0) - goto next_port; + goto next_port_unlock; WARN_ON(hlist_empty(&tb->bhash2)); if (!check_established(death_row, sk, - port, &tw)) + port, &tw, false)) goto ok; - goto next_port; + goto next_port_unlock; } } @@ -1089,8 +1106,9 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row, tb->fastreuse = -1; tb->fastreuseport = -1; goto ok; -next_port: +next_port_unlock: spin_unlock_bh(&head->lock); +next_port: cond_resched(); } diff --git a/net/ipv6/inet6_hashtables.c b/net/ipv6/inet6_hashtables.c index 3604a5cae5d29a25d24f9513308334ff8e64b083..9be315496459fcb391123a07ac887e2f59d27360 100644 --- a/net/ipv6/inet6_hashtables.c +++ b/net/ipv6/inet6_hashtables.c @@ -263,7 +263,8 @@ EXPORT_SYMBOL_GPL(inet6_lookup); static int __inet6_check_established(struct inet_timewait_death_row *death_row, struct sock *sk, const __u16 lport, - struct inet_timewait_sock **twp) + struct inet_timewait_sock **twp, + bool rcu_lookup) { struct inet_hashinfo *hinfo = death_row->hashinfo; struct inet_sock *inet = inet_sk(sk); @@ -281,17 +282,18 @@ static int __inet6_check_established(struct inet_timewait_death_row *death_row, struct sock *sk2; spinlock_t *lock; - rcu_read_lock(); - sk_nulls_for_each(sk2, node, &head->chain) { - if (sk2->sk_hash != hash || - !inet6_match(net, sk2, saddr, daddr, ports, dif, sdif)) - continue; - if (sk2->sk_state == TCP_TIME_WAIT) - break; - rcu_read_unlock(); - return -EADDRNOTAVAIL; + if (rcu_lookup) { + sk_nulls_for_each(sk2, node, &head->chain) { + if (sk2->sk_hash != hash || + !inet6_match(net, sk2, saddr, daddr, + ports, dif, sdif)) + continue; + if (sk2->sk_state == TCP_TIME_WAIT) + break; + return -EADDRNOTAVAIL; + } + return 0; } - rcu_read_unlock(); lock = inet_ehash_lockp(hinfo, hash); spin_lock(lock);