From patchwork Wed Jan 8 16:22:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13931302 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qt1-f201.google.com (mail-qt1-f201.google.com [209.85.160.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1FB81F3D54 for ; Wed, 8 Jan 2025 16:22:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736353381; cv=none; b=nz96HYWV95UFuiVODC/MlLelVOQ5MxWU9T9QHjNg3PUMEL+NSquzMVoe2w5JaLPGhFkmr/slxF7fgoTJbOqY2tfoB0IT5bA63QaIikUZf12wNCpraPZ8MaF4sWzV99AVYvEnTKdLjHXeeNp7f6+fkjrqsx2G12ttv1JasppNldc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736353381; c=relaxed/simple; bh=gO8FZqQuv0MMHUrKJJi/pFY0mGOd+fBug1vcRBxPlBU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UQhxcDIiwTi4/ETd0W2RYBzF4bq83H9pUqyO8Tm9Tx1BIq3dvu91Ldkjy7LlklGL1eYHFgbumehLW6wcIU2kYKQ3O/PqV4BvKje+MrRhrBfzQMDHT94JE1Mo0VaQrMLuURnMnCDJXT4bBIMTRCKIvhjpv2Si76EiljODuYCApFk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=AlI6XCjx; arc=none smtp.client-ip=209.85.160.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="AlI6XCjx" Received: by mail-qt1-f201.google.com with SMTP id d75a77b69052e-467b0b0aed4so358198071cf.2 for ; Wed, 08 Jan 2025 08:22:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736353378; x=1736958178; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UlHLQ3o8A1G5EivvnS/riuO3x8kL0WjN5265qdyloXo=; b=AlI6XCjxl4lV3ouh8AYrqp67ta5W8AwRXn4wK6o9xzf5hRq5NDTmVNzuZTQ76hJtiP la7YIvezCxZnIuzFCIyLO4PsoxRqBgKtASRrxiclcdHVGadB/IpOmEPwzj7baLXWVjsD q2ATbRwGVCzmYITnxMd5GdglX7gwQB7I6vJAAXSTNqcqjMcJVsH73yhQL9udWIWrqudc +1sFufxCfLdlEo3JFxs/FoO1/oSqGFWoaMa9Iq1mzHO8mlZj4TRwH+E7WnqxzpjDfkxH 7aQJQBMYaUWH9msNzA+rTmRqLF6LFEMTqi89tol0eRRHcP2ivVb1B9BeAjX0dEb+oSvc T8dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736353378; x=1736958178; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UlHLQ3o8A1G5EivvnS/riuO3x8kL0WjN5265qdyloXo=; b=pMJ2eZurnSTFk6EnuXrE9ojR3Xgl9Jv6Gz0Pv/Lk3FiQj80L6JfoctiZqMa4/FPTYg fiu+bc143FnliYKKL4dGfpDFC5SbymPQhoQR1jz3/XMkieQOaZdA2D01kcQvpiO0Ib5s lguzZTXHPlkdQ7Dx9em8FgxbZorZgSdAzGmU803Xbjbtx+l0mSl9gptPIeKkxqGKUDPa ivBT1ft6Kp4SqNhixbp+ZZMpC1V0lmof7F/i2wwSfr/SQ5hWuCxU/EgpZ4tbojNJMosC cYl8UdNCA0gIPXwoQRYMxI6bUAOYbnkDnLp+VtPs5ON7G/YePnP/ErqcDB8Rn/lcpaNd JaMA== X-Gm-Message-State: AOJu0Yz4iXS9h3AD3hZdJeZ0jEeJ6JRzq77F1rM8DBBDi1E+EHzab+fN yZ3/bml4B3kphhZxxyM3f2dgJ2kszACVv5vTtEVbcONITinZvBjzV6PjWrg3Xb6HmkwBZjnb1qq C1iuCaarnAg== X-Google-Smtp-Source: AGHT+IER3D8Lo7We55c5NFeeiOmidG1Z0nHt/Q5a14U3TlcF0Yht6SHL506p55ReHJu6dyaQISHSVENXwbknDg== X-Received: from qtjf12.prod.google.com ([2002:ac8:134c:0:b0:467:8f83:cafe]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:a05:622a:214:b0:467:5eb6:5153 with SMTP id d75a77b69052e-46c70ff77f9mr45828611cf.19.1736353378655; Wed, 08 Jan 2025 08:22:58 -0800 (PST) Date: Wed, 8 Jan 2025 16:22:52 +0000 In-Reply-To: <20250108162255.1306392-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250108162255.1306392-1-edumazet@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250108162255.1306392-2-edumazet@google.com> Subject: [PATCH v2 net-next 1/4] net: expedite synchronize_net() for cleanup_net() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, Simon Horman , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org cleanup_net() is the single thread responsible for netns dismantles, and a serious bottleneck. Before we can get per-netns RTNL, make sure all synchronize_net() called from this thread are using rcu_synchronize_expedited(). Signed-off-by: Eric Dumazet --- include/net/net_namespace.h | 2 ++ net/core/dev.c | 7 ++++++- net/core/net_namespace.c | 5 +++++ 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/include/net/net_namespace.h b/include/net/net_namespace.h index 5a2a0df8ad91b677b515b392869c6c755be5c868..a3009bdd7efec0a3b665cbf51c159c323458410a 100644 --- a/include/net/net_namespace.h +++ b/include/net/net_namespace.h @@ -565,4 +565,6 @@ void net_ns_init(void); static inline void net_ns_init(void) {} #endif +extern struct task_struct *cleanup_net_task; + #endif /* __NET_NET_NAMESPACE_H */ diff --git a/net/core/dev.c b/net/core/dev.c index efbe2c4d94580a2ce9401adb85784c9c1c862df9..76ad68b129eed0407686e8696102aeed9a8b30ec 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -11415,6 +11415,11 @@ struct net_device *alloc_netdev_dummy(int sizeof_priv) } EXPORT_SYMBOL_GPL(alloc_netdev_dummy); +static bool from_cleanup_net(void) +{ + return current == cleanup_net_task; +} + /** * synchronize_net - Synchronize with packet receive processing * @@ -11424,7 +11429,7 @@ EXPORT_SYMBOL_GPL(alloc_netdev_dummy); void synchronize_net(void) { might_sleep(); - if (rtnl_is_locked()) + if (from_cleanup_net() || rtnl_is_locked()) synchronize_rcu_expedited(); else synchronize_rcu(); diff --git a/net/core/net_namespace.c b/net/core/net_namespace.c index b5cd3ae4f04cf28d43f8401a3dafebac4a297123..cb39a12b2f8295c605f08b5589932932150a1644 100644 --- a/net/core/net_namespace.c +++ b/net/core/net_namespace.c @@ -588,6 +588,8 @@ static void unhash_nsid(struct net *net, struct net *last) static LLIST_HEAD(cleanup_list); +struct task_struct *cleanup_net_task; + static void cleanup_net(struct work_struct *work) { const struct pernet_operations *ops; @@ -596,6 +598,8 @@ static void cleanup_net(struct work_struct *work) LIST_HEAD(net_exit_list); LIST_HEAD(dev_kill_list); + cleanup_net_task = current; + /* Atomically snapshot the list of namespaces to cleanup */ net_kill_list = llist_del_all(&cleanup_list); @@ -670,6 +674,7 @@ static void cleanup_net(struct work_struct *work) put_user_ns(net->user_ns); net_free(net); } + cleanup_net_task = NULL; } /** From patchwork Wed Jan 8 16:22:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13931303 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qt1-f202.google.com (mail-qt1-f202.google.com [209.85.160.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4448D1F9439 for ; Wed, 8 Jan 2025 16:23:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736353382; cv=none; b=q9ejW+velFIa3y5AfBOROv5UR6jz4nDLUi+EglK92K3aZ6YucVmEV3YtmTUmYKVgJE4vexptfA2b8AQi7X0m1Hi66qRw4tN2U7EtvDktui9q8xMbrkkOhbkrQ6FbYTbZuxUnxbuekXzQafG7COQhxMdlxDMjECeGVB30BXBiCIA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736353382; c=relaxed/simple; bh=FbDbWtrkeO1zLkrOE6W8TLnoL+pS/UMS4N66Svjvmv4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YB37GhzaeNeIz9x3cXWu6IaXybCj4n+xum7oZPriF2XDhANvWk+p9tBmad6HlZ7oNRQpX6XlxkMeNQA7ml5hfrQfqEaXb5/22RYbtACMH+x+0ZrjzJWiF0/SjvUOUEEnqNT83YniQh0VZ8tKfPrp0xYtmtK74w0JP4wgZxRjcFU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Lhyt5iYv; arc=none smtp.client-ip=209.85.160.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Lhyt5iYv" Received: by mail-qt1-f202.google.com with SMTP id d75a77b69052e-4678b42cbfeso335686031cf.3 for ; Wed, 08 Jan 2025 08:23:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736353380; x=1736958180; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=24m9Jbnx67pDSoU0/Ox88pWvSYxaM1gsBFAZIN3yQRY=; b=Lhyt5iYvo2/hFg/RGjaymGmXJntuYx6UN7+3fIhM9MC6xqTp9Bv5oXCIdJCXnmEx/Z lKSj8NyJsFjvX6bgH2uP084ZuoVTvXWlaiw7etMo4XF51cu2I5NaAB9qDLnfTPJd7D99 kKQ8d0MXmvPvGbBN6/eR5uQtFgCCt7W9JfxbMrQcp7RWqegnH6rLOqcvM/Mhr75kxiTp 6jjpU2d+PRhJPpAGuHxCg34Kw9OU+ZvMTnegz8kyKdyhmSk2wFlnhO0cAGawJCjhE+eb TGD0xzn5dDAZ/QeZllJzXPECc2v5YJEn22MSghw+r23zLQSTDFG+WvBsgYPLRLlJDy7/ o4sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736353380; x=1736958180; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=24m9Jbnx67pDSoU0/Ox88pWvSYxaM1gsBFAZIN3yQRY=; b=xCr3FJuSnD6Q4wIJVBDdfAZT2v4kdcyOkc12ifyMyNcBe7C+ajziRTo1Mbb21v+QjA YsmLWO3xvhGyoLcnz+tRogN5sCosIrrfZ3qL4YO8ysQmet7MPt/OFb2xchIZfu2FnBH4 XKHoq6CUVtH+LnzTn3WZFMDyFbWkcE4/8RZ8sg6V8Q+gvo8T09Fzk1EqeMwRO3jIU9WE oEyLW1p1LpLijwxs32tGL4t5zxA6U2Ft365a9Cdp2Ze5fYHygm/tJgphm9a1AvEjsn78 IxRJSHBRV8gZ+rcsaMcoNLobYc179iUrx/RLxXpxgNFBpC5SL8ZRKQ1BdzicHu0bqATA jYNg== X-Gm-Message-State: AOJu0YwyvMLgQvDYkE0bVk/KS8T0vQYqGG9+kvgsVdGXKFNhGR/ZmaUn MwP9gio4xp8xAGxpv2VkKeAGmlXjKBmo2J5xnb6UeBDKL1/1hLPNz4WH5JlmzM3GonDka+jK9kZ t7a0V5KXKVg== X-Google-Smtp-Source: AGHT+IGgtcS3+OOOtwHbf1dIGYX5GuC7oeABEuwMJpxaA+3OnOKbtjxnNpAE4nqMZryLJehIRE357/m7MFVNpQ== X-Received: from qtnd3.prod.google.com ([2002:ac8:51c3:0:b0:467:64d0:3f08]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:ac8:5793:0:b0:467:70ce:75ea with SMTP id d75a77b69052e-46c7102bfdbmr49580191cf.23.1736353380222; Wed, 08 Jan 2025 08:23:00 -0800 (PST) Date: Wed, 8 Jan 2025 16:22:53 +0000 In-Reply-To: <20250108162255.1306392-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250108162255.1306392-1-edumazet@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250108162255.1306392-3-edumazet@google.com> Subject: [PATCH v2 net-next 2/4] net: no longer assume RTNL is held in flush_all_backlogs() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, Simon Horman , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org flush_all_backlogs() uses per-cpu and static data to hold its temporary data, on the assumption it is called under RTNL protection. Following patch in the series will break this assumption. Use instead a dynamically allocated piece of memory. In the unlikely case the allocation fails, use a boot-time allocated memory. Signed-off-by: Eric Dumazet --- net/core/dev.c | 53 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 35 insertions(+), 18 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index 76ad68b129eed0407686e8696102aeed9a8b30ec..8ff288cf25dceb5856496388f83f409fcb6f8e5d 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -5978,8 +5978,6 @@ void netif_receive_skb_list(struct list_head *head) } EXPORT_SYMBOL(netif_receive_skb_list); -static DEFINE_PER_CPU(struct work_struct, flush_works); - /* Network device is going away, flush any packets still pending */ static void flush_backlog(struct work_struct *work) { @@ -6036,36 +6034,54 @@ static bool flush_required(int cpu) return true; } +struct flush_backlogs { + cpumask_t flush_cpus; + struct work_struct w[]; +}; + +static struct flush_backlogs *flush_backlogs_alloc(void) +{ + return kmalloc(struct_size_t(struct flush_backlogs, w, nr_cpu_ids), + GFP_KERNEL); +} + +static struct flush_backlogs *flush_backlogs_fallback; +static DEFINE_MUTEX(flush_backlogs_mutex); + static void flush_all_backlogs(void) { - static cpumask_t flush_cpus; + struct flush_backlogs *ptr = flush_backlogs_alloc(); unsigned int cpu; - /* since we are under rtnl lock protection we can use static data - * for the cpumask and avoid allocating on stack the possibly - * large mask - */ - ASSERT_RTNL(); + if (!ptr) { + mutex_lock(&flush_backlogs_mutex); + ptr = flush_backlogs_fallback; + } + cpumask_clear(&ptr->flush_cpus); cpus_read_lock(); - cpumask_clear(&flush_cpus); for_each_online_cpu(cpu) { if (flush_required(cpu)) { - queue_work_on(cpu, system_highpri_wq, - per_cpu_ptr(&flush_works, cpu)); - cpumask_set_cpu(cpu, &flush_cpus); + INIT_WORK(&ptr->w[cpu], flush_backlog); + queue_work_on(cpu, system_highpri_wq, &ptr->w[cpu]); + __cpumask_set_cpu(cpu, &ptr->flush_cpus); } } /* we can have in flight packet[s] on the cpus we are not flushing, * synchronize_net() in unregister_netdevice_many() will take care of - * them + * them. */ - for_each_cpu(cpu, &flush_cpus) - flush_work(per_cpu_ptr(&flush_works, cpu)); + for_each_cpu(cpu, &ptr->flush_cpus) + flush_work(&ptr->w[cpu]); cpus_read_unlock(); + + if (ptr != flush_backlogs_fallback) + kfree(ptr); + else + mutex_unlock(&flush_backlogs_mutex); } static void net_rps_send_ipi(struct softnet_data *remsd) @@ -12259,12 +12275,13 @@ static int __init net_dev_init(void) * Initialise the packet receive queues. */ + flush_backlogs_fallback = flush_backlogs_alloc(); + if (!flush_backlogs_fallback) + goto out; + for_each_possible_cpu(i) { - struct work_struct *flush = per_cpu_ptr(&flush_works, i); struct softnet_data *sd = &per_cpu(softnet_data, i); - INIT_WORK(flush, flush_backlog); - skb_queue_head_init(&sd->input_pkt_queue); skb_queue_head_init(&sd->process_queue); #ifdef CONFIG_XFRM_OFFLOAD From patchwork Wed Jan 8 16:22:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13931304 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qt1-f201.google.com (mail-qt1-f201.google.com [209.85.160.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83BA51FF1C0 for ; Wed, 8 Jan 2025 16:23:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736353384; cv=none; b=F9bNzSfNx38bhs5IL7GufV4yBOr5bXM/P7ptAF/R0TbK1nxBx3Z7oKHJspcYDUwYbpTbtwPN6CQUUiXjzHWXm+NgDNO5s7jAbGyvaHTeCAqxyHnX+2/eV+kU8RVtuX5eJj77Fj/nI3cf1VkPeOFmT93avpgtlxihwbb7SjEFkC0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736353384; c=relaxed/simple; bh=rSqJj4XwTdwc39+3g8QMOFoAWAV1qaibloUAJCEd9D8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Z1QIvADRG6CZ9E+fkNsAvNG7QObcRHvFcKZ2LMy4ipAQ4Y4J6vAOhwfJ7TKGripKq8D8LQu8WabFfLaybH+DWJh69jsTAakmehokpj09df5c/OfzXvvfAkhrsLrKt9hWENUgsIt2rmAe9vYqK0jXLlflHEOut+DcWuMFz3jagAc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=a3N6Gdrc; arc=none smtp.client-ip=209.85.160.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="a3N6Gdrc" Received: by mail-qt1-f201.google.com with SMTP id d75a77b69052e-4679db55860so337860101cf.3 for ; Wed, 08 Jan 2025 08:23:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736353381; x=1736958181; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Xhbjwta6TiDMc7v+7DNtDB0Tr9ebF3Qj7Hv8pFJ6cq4=; b=a3N6GdrcTeXSPnq7XmvG57Y2/9qvhrJfNpY1YasIKnUl0sjlF19W+ewPr+FHyHkH7C xMNUeMpj4IVhk93ClMOabQXECpTY7aa9wrUy/qhC7LUqk/uN+CS0iUh/AeYIGam65I2Z HVVz+ODb0VZcAttNdSSvXj33aBJ4/fHAgtkzRPZGfxP+XTWquT0YXYSNKy8BEM5cSzCe bXobaEG4PeeLncgUzRT92L6OMCz9aWJeH0n9LfrPiDcCDK3vGSKtBb0EaO8om4fJTA3d A1ZIPsUmh9pbWNe2Wzo6KwTJKuUQn2CAWMCH9fJWC7kCmuM6gYz6Ps2RbzrerQGisM0d WC2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736353381; x=1736958181; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Xhbjwta6TiDMc7v+7DNtDB0Tr9ebF3Qj7Hv8pFJ6cq4=; b=rFFytGNpnEysC8pF3n4OeA5mo4/tn7vdFmVrOTgCnq2DcmTBwq6YWjBB81fkD8FcRg eC7Tw1az8GODJfq5TKd1YbGZKIQsJJslYuKRKmIj0rsfSJ15g+rnVWRqrEJG4DerO61T Qp0YbMZk46DAAh7xx8srjhHA0g771vBZE+waB0GLSyX4lRqJGk2IFPtqb/tW9IBWnKvV 0s+MLf5RdgDahI9mRxcovbr7EoemhUcYuCM0kOJs9xARuNTdxIgwhJw10QUeniRXojNZ jLMcti2lgNJgfaBdy0w90OkMmWv7/A0eVai17RUUJYYYNCA9OHlddsYk15JJS1uX2MMs uPyw== X-Gm-Message-State: AOJu0YwFj+x8+X8yBqY8/wK9HUYQ+bMTqA1Q937H7oO0GEHbV1MtT+ts 7alU93hWrBE1c663Uozkf2qzNRKQ2/4wakKQRhpXUDS0p5OcCU01rE9aD+E46rzSk5xleegZIFx xjoh5gBJv4Q== X-Google-Smtp-Source: AGHT+IEv8p0Thx3D4GtVYuFmbhZLcdXVvixTic9daCsqvjXKUpOsi4pJUZ7mazF+JuA9GANCbf61us1JpwPuvA== X-Received: from qtjw4.prod.google.com ([2002:ac8:7e84:0:b0:467:7076:37d7]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:ac8:5807:0:b0:46c:77bc:ce5e with SMTP id d75a77b69052e-46c77bcd922mr20213391cf.7.1736353381550; Wed, 08 Jan 2025 08:23:01 -0800 (PST) Date: Wed, 8 Jan 2025 16:22:54 +0000 In-Reply-To: <20250108162255.1306392-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250108162255.1306392-1-edumazet@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250108162255.1306392-4-edumazet@google.com> Subject: [PATCH v2 net-next 3/4] net: no longer hold RTNL while calling flush_all_backlogs() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, Simon Horman , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org flush_all_backlogs() is called from unregister_netdevice_many_notify() as part of netdevice dismantles. This is currently called under RTNL, and can last up to 20ms on busy hosts. There is no reason to hold RTNL at this stage, if our caller is cleanup_net() : netns are no more visible, devices are in NETREG_UNREGISTERING state and no other thread could mess our state while RTNL is temporarily released. Signed-off-by: Eric Dumazet --- net/core/dev.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/net/core/dev.c b/net/core/dev.c index 8ff288cf25dceb5856496388f83f409fcb6f8e5d..86fa9ae29d31a25dca8204c75fd39974ef84707d 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -11436,6 +11436,18 @@ static bool from_cleanup_net(void) return current == cleanup_net_task; } +static void rtnl_drop_if_cleanup_net(void) +{ + if (from_cleanup_net()) + __rtnl_unlock(); +} + +static void rtnl_acquire_if_cleanup_net(void) +{ + if (from_cleanup_net()) + rtnl_lock(); +} + /** * synchronize_net - Synchronize with packet receive processing * @@ -11548,8 +11560,10 @@ void unregister_netdevice_many_notify(struct list_head *head, unlist_netdevice(dev); WRITE_ONCE(dev->reg_state, NETREG_UNREGISTERING); } - flush_all_backlogs(); + rtnl_drop_if_cleanup_net(); + flush_all_backlogs(); + rtnl_acquire_if_cleanup_net(); synchronize_net(); list_for_each_entry(dev, head, unreg_list) { From patchwork Wed Jan 8 16:22:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13931305 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qk1-f201.google.com (mail-qk1-f201.google.com [209.85.222.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93DC81FF1DE for ; Wed, 8 Jan 2025 16:23:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736353386; cv=none; b=MEzR5L6AtPglsJfNidtbcz5dC/4mQeBw7hlIdw215ERMo8K+lm66FZGeFrN1VdfVV42bzX1KQP5f0Symcb02UjkSI2CQYZt00/Jz6JYPH7Q1PkIR9jKm68SqwsAM6pNDaPetC3niHG9o96+6XcYC+et9vueBf4SYFWqs0WIRuSA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736353386; c=relaxed/simple; bh=wismHMKSvkDLHvKmME2OpAhqeD4lUtTjzazscDtaTdw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=L3WEU0sYznES8KZ2fCS2EXrjobR7hODjnrEcH5xyDyqmVH+tS/vtP/uLp7YtxvfdjEod46gjlaO6u5ZJm5YSPi9XlzBWQrwconD4/PiXU5FBq6mv2pR8bM5D4pDkNzhxadllWziTcH2PcKwz77ZJsvEjdxiCbcwdjPVv8L7or/c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=kJiYJ6hZ; arc=none smtp.client-ip=209.85.222.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="kJiYJ6hZ" Received: by mail-qk1-f201.google.com with SMTP id af79cd13be357-7b6e7f07332so832657985a.1 for ; Wed, 08 Jan 2025 08:23:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736353383; x=1736958183; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pbzDviwearz7TxyF2UixIH7xoEZaBfrGaEL+pcXWhJQ=; b=kJiYJ6hZ5rHXok7dvJaNmYTaXX70Rj+t5oECGYg/7HJu7y2GHKuzqsG9Ty0caWnjiv D5yMBaC/bIj9BMo5oBwwD/sUohoWLZsJUHK518FM0iflDpCUKthdgp0NR7gNtyEv341T bq56p3JCEGMzvUY7dr0EAA7pBm6xW5jcsvivRXZT0BusHqzR3FAPfRToSMLdW/xCGW7g XJZ4cNupZLdykgk+oU0hNCssRYkG2zvTGTrsHTlmNIprCG+fHWcFvtY9Ci6yly3tzxv4 QB4Gakk1fSIwv/c0REBl4PPRA/couKZYp54mi1bktSL/uzhnpw29+CSr11Hj26/JjLgv jKMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736353383; x=1736958183; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pbzDviwearz7TxyF2UixIH7xoEZaBfrGaEL+pcXWhJQ=; b=X1bkqjxXM4mmD2qsJtGGECInQN9vvg2GYhClzxQralG0A8AoACcLHuiZbhwfc3tqnC GAi8ns3rQ9xFgNId1QMYA/OVOiWOQx9c6jw/5vcrJpFUnbbUt3fymgYNtMBKgjSJTJy3 i0As+HsL51HJ+hxipPk52v3ym8e9gbgpRL1gjxUe53OWAdPksD/3SsXo6Ka4EQL0BNlZ z4p8/3fz0PSZTMufB9qNuy4pgciOKlBd95dMrnqVOBQY3Zn5+ZidODjtqUEanE3C34yF 1JSMd6UKtEDyEjXk41C44/tN2evNonbp6imzPUU7zwKdqqL/6elNE3xfr8/zY8NbGkkH zULA== X-Gm-Message-State: AOJu0YydxarQ8EE/xuRBJvpe1h8dvfcVNfKrizIhym8n4BSWB6MQ0urc OLcEnoWa4fKcgHBYQXbcXpxiwWYIvFFWZ15MeGRw0nluHR5UYxqtQcsUD/rHf4EMCwMM/BriDz6 IT9thHD6L6Q== X-Google-Smtp-Source: AGHT+IFwzjUWqgEdwXHUHJjfvt/nO42t+9163f8nVWhcCTbo4bCGLODKqe9HLFxwlLw1c9BzQmUoFK6PI3aZFQ== X-Received: from qkbbk28.prod.google.com ([2002:a05:620a:1a1c:b0:7b6:c94b:252f]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:a05:620a:25cf:b0:7b6:f113:b658 with SMTP id af79cd13be357-7bcd96e47b2mr476641185a.12.1736353382939; Wed, 08 Jan 2025 08:23:02 -0800 (PST) Date: Wed, 8 Jan 2025 16:22:55 +0000 In-Reply-To: <20250108162255.1306392-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250108162255.1306392-1-edumazet@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250108162255.1306392-5-edumazet@google.com> Subject: [PATCH v2 net-next 4/4] net: reduce RTNL hold duration in unregister_netdevice_many_notify() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, Simon Horman , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Two synchronize_net() calls are currently done while holding RTNL. This is source of RTNL contention in workloads adding and deleting many network namespaces per second, because synchronize_rcu() and synchronize_rcu_expedited() can use 10+ ms in some cases. For cleanup_net() use, temporarily release RTNL while calling synchronize_net(). This should be safe, because devices are no longer visible to other threads after the call to unlist_netdevice(), and the netns are in dismantle phase. Signed-off-by: Eric Dumazet --- net/core/dev.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/net/core/dev.c b/net/core/dev.c index 86fa9ae29d31a25dca8204c75fd39974ef84707d..dde5d40e8f6b1ad5309ebc6a6e163ddb96f66f73 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -11563,8 +11563,8 @@ void unregister_netdevice_many_notify(struct list_head *head, rtnl_drop_if_cleanup_net(); flush_all_backlogs(); - rtnl_acquire_if_cleanup_net(); synchronize_net(); + rtnl_acquire_if_cleanup_net(); list_for_each_entry(dev, head, unreg_list) { struct sk_buff *skb = NULL; @@ -11624,7 +11624,9 @@ void unregister_netdevice_many_notify(struct list_head *head, #endif } + rtnl_drop_if_cleanup_net(); synchronize_net(); + rtnl_acquire_if_cleanup_net(); list_for_each_entry(dev, head, unreg_list) { netdev_put(dev, &dev->dev_registered_tracker);