From patchwork Tue Jan 7 17:38:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13929450 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-ua1-f74.google.com (mail-ua1-f74.google.com [209.85.222.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8651018C924 for ; Tue, 7 Jan 2025 17:38:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736271525; cv=none; b=G7IRFO4vmcbMrpD054VigjpJjjwYHW7PcBVhyUxqD4OHpRXhpNYCQ75MKwi8j4CuJ5H8L0CTxjhbAvDplq4tCTaa8prtnyMrCGEQBTQTFWBBA3XoogTqSPgI2bxpOAoWnIpS/NLw04wwDEVxtYI6hjusROY+U5TVZNWHhpQW11Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736271525; c=relaxed/simple; bh=kJN2SbSkvYGInmp4MBJGgCNomp540c6bMOIcV9wYj38=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=r9noRe0aCcdVTRaGsA35mCFEt34W7cJfg2rBv6fmCQjXlN7LiPPFiCmqcGIqczOfu6E/2y2QPFqYrmN1m96zZ+KhvWDeNdf7z3HVwxnnwr7cS6KghbAblkYVwxlUPc7x8m4NtCYnMxPlO+itEI7wIRqmogtZS2d8DFD9+WIzxxk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ocSN6NVd; arc=none smtp.client-ip=209.85.222.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ocSN6NVd" Received: by mail-ua1-f74.google.com with SMTP id a1e0cc1a2514c-85ca702c67cso14530241.0 for ; Tue, 07 Jan 2025 09:38:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736271522; x=1736876322; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KD5QT7C2cc1SRoGoZkCYCHl9+3wPP/aOnYKm+wH6DBQ=; b=ocSN6NVda/1tX+hShduHUlIJG0nhu1znHnCyEbc1LvCxvOn1ck3DtHjmvG592mgGOh UML2twjIEehCGAOkPUghjq8jtj3gbqbLjUY3Coy+/C1KuSzRL1kXUUkendBjrxftzSdw SancjfFhses8E0ofI7GU5EZyZgMuaAoiylxTVgNP2IbApJXRL9CP2rA1HEp8kOAzYV4E r4wTCwjmAU8VhY8hTC/2KNMvVRg6SbeFdCDLYZ8zZdzIq/7dr9gWvIMfh3aZgg5tlGFB 9N+HAAerJnvN1Gmz+c3R0mgoQRoLB1eX5h4hROb62/D6mL5r8o+UY6863+PtuqPLjBFY OyHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736271522; x=1736876322; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KD5QT7C2cc1SRoGoZkCYCHl9+3wPP/aOnYKm+wH6DBQ=; b=maBIjUTqMlurjDnmWiLGe/elXFEo+DES+n5OhPHzdraZR+j/Fk96gKycoYmDAzKJb8 WP+nahu28zI7ooqLLJDQsbTD4OgXAA7IFjDE4JNuPF0dbf825n2ychJXaqzvZcKBeRDY lFsYnCXscFD3Ctz28inp1qXa2JM61VYMgYoPXN6HoSjJL3Zf6sVlD/u1k4wqAo97whXK QOoMYN7Xm0dGjomA5c/qBsE5DAseH51WzqaHD7SdEw2L5teVMEBOhmwuto0GWp7Ag+LZ XK01EHFNvG83eYMacdUh0ajhvH6BrWo+IFmmaYzj0f83P92hVi4Leuqn8DH43dVMuvQP JHVQ== X-Gm-Message-State: AOJu0Yyvz7N3Mk/ZTJJXcgul8ZRrrPsvaKjkLbfso5l/464XgwibolGI yjgyng0SsLMJ8LBHs+EkubrX/K3aCVyqpL9XLjwHdwYq3FhKj6MzB2grATX4L3nwB0oIM28ZXvf yyba/vx7WUw== X-Google-Smtp-Source: AGHT+IGfgGUnsWPoMrv3jWYhpqv5ffoBnU+hTXTZ882dS/IFA4XGqCST3arKFaCVkqLBwstnWdFkumZ7dsj9Gw== X-Received: from vsbbq9.prod.google.com ([2002:a05:6102:5349:b0:4b2:ad82:19b6]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6102:5088:b0:4b2:9e5d:baf with SMTP id ada2fe7eead31-4b3c1eed846mr3248008137.11.1736271522408; Tue, 07 Jan 2025 09:38:42 -0800 (PST) Date: Tue, 7 Jan 2025 17:38:35 +0000 In-Reply-To: <20250107173838.1130187-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250107173838.1130187-1-edumazet@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250107173838.1130187-2-edumazet@google.com> Subject: [PATCH net-next 1/4] net: no longer assume RTNL is held in flush_all_backlogs() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, Simon Horman , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org flush_all_backlogs() uses per-cpu data to hold its temporary daya, on the assumption it is called under RTNL protection. Following patch in the series will break this assumption. Use instead a dynamically allocated piece of memory. In the unlikely case the allocation fails, use a boot-time allocated memory. Signed-off-by: Eric Dumazet --- net/core/dev.c | 53 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 35 insertions(+), 18 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index 073f682a9653a212198b12bae17fafe7b46f96e9..9e1eb272e4feaf40dc87defd54d691634e0902e5 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -5961,8 +5961,6 @@ void netif_receive_skb_list(struct list_head *head) } EXPORT_SYMBOL(netif_receive_skb_list); -static DEFINE_PER_CPU(struct work_struct, flush_works); - /* Network device is going away, flush any packets still pending */ static void flush_backlog(struct work_struct *work) { @@ -6019,36 +6017,54 @@ static bool flush_required(int cpu) return true; } +struct flush_backlogs { + cpumask_t flush_cpus; + struct work_struct w[]; +}; + +static struct flush_backlogs *flush_backlogs_alloc(void) +{ + return kmalloc(struct_size_t(struct flush_backlogs, w, nr_cpu_ids), + GFP_KERNEL); +} + +static struct flush_backlogs *flush_backlogs_fallback; +static DEFINE_MUTEX(flush_backlogs_mutex); + static void flush_all_backlogs(void) { - static cpumask_t flush_cpus; + struct flush_backlogs *ptr = flush_backlogs_alloc(); unsigned int cpu; - /* since we are under rtnl lock protection we can use static data - * for the cpumask and avoid allocating on stack the possibly - * large mask - */ - ASSERT_RTNL(); + if (!ptr) { + mutex_lock(&flush_backlogs_mutex); + ptr = flush_backlogs_fallback; + } + cpumask_clear(&ptr->flush_cpus); cpus_read_lock(); - cpumask_clear(&flush_cpus); for_each_online_cpu(cpu) { if (flush_required(cpu)) { - queue_work_on(cpu, system_highpri_wq, - per_cpu_ptr(&flush_works, cpu)); - cpumask_set_cpu(cpu, &flush_cpus); + INIT_WORK(&ptr->w[cpu], flush_backlog); + queue_work_on(cpu, system_highpri_wq, &ptr->w[cpu]); + __cpumask_set_cpu(cpu, &ptr->flush_cpus); } } /* we can have in flight packet[s] on the cpus we are not flushing, * synchronize_net() in unregister_netdevice_many() will take care of - * them + * them. */ - for_each_cpu(cpu, &flush_cpus) - flush_work(per_cpu_ptr(&flush_works, cpu)); + for_each_cpu(cpu, &ptr->flush_cpus) + flush_work(&ptr->w[cpu]); cpus_read_unlock(); + + if (ptr != flush_backlogs_fallback) + kfree(ptr); + else + mutex_unlock(&flush_backlogs_mutex); } static void net_rps_send_ipi(struct softnet_data *remsd) @@ -12237,12 +12253,13 @@ static int __init net_dev_init(void) * Initialise the packet receive queues. */ + flush_backlogs_fallback = flush_backlogs_alloc(); + if (!flush_backlogs_fallback) + goto out; + for_each_possible_cpu(i) { - struct work_struct *flush = per_cpu_ptr(&flush_works, i); struct softnet_data *sd = &per_cpu(softnet_data, i); - INIT_WORK(flush, flush_backlog); - skb_queue_head_init(&sd->input_pkt_queue); skb_queue_head_init(&sd->process_queue); #ifdef CONFIG_XFRM_OFFLOAD From patchwork Tue Jan 7 17:38:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13929451 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-vs1-f73.google.com (mail-vs1-f73.google.com [209.85.217.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 605AD18E750 for ; Tue, 7 Jan 2025 17:38:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.217.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736271526; cv=none; b=Dp2TW49T8M0mnmFfFEovSjv9shVCF+byTfso2wkc8ksuka3AvmXnDvhCcq9EPXP3wE4/ing0OIJclz5KWNOHDxpnX78xPeul8hMsORlsepAoR/mwKx1pp5w17lvhlbyCkitLo0vv0CigRjgTfe4zOC0hzA1V5zzKE8gxX6c9hHc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736271526; c=relaxed/simple; bh=FeWLCRIJSynfSqJYBMEnPDrSf1/Yq5W6089igqo+zJw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=TrVn1YDKfPWPNB82x1SjjxVzxMDBuVMxS1WjwIOzQyvn8K6pOdQiHVh5hcRR3CrltS3sV8a6+0FVygAYy2Nz9BEtj3SreISq/02airlbP64mNztOXIl+CI6/XnsuBv0SU97yd+4j2xPaW9vBg6LoE3M1KgqY635U/NrRxbcRRIc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=J74lyIaS; arc=none smtp.client-ip=209.85.217.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="J74lyIaS" Received: by mail-vs1-f73.google.com with SMTP id ada2fe7eead31-4b11b247407so1014860137.1 for ; Tue, 07 Jan 2025 09:38:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736271524; x=1736876324; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FQf5Rn3U0tejDnmzZj+gGv2suCDKUxoYIgY5TsACDAE=; b=J74lyIaSJydfmoJChyqXSbmQn2TGJHdQmJ6g6t+pfsIzDezXc7RAl9WRZA2C22tO0M RZ9ZhRP7wp598kVUuy6TOaA1vnNw2W864h7yIIyXZ0nSr0JQxa/v2fnuXVRWaJXGO+e+ Pd2RfNsaqAufzDDEm56z1hshJOrO1shcfigoROVFTIOYgbD3NElBprgeom4y6SZsTyAK 06zghFP7TpPpWNGJuOieBaWj7cAQXtRLjvEPjI6L041NVef0I+/aryTvQpHLPoK2xp8J 8vCtsK2UGTplJ3YaU9SWRgeYvMLYA5BWkqBc7KadWl0DgzGUJiQSxGZXtXlY7qIvPLPB +KJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736271524; x=1736876324; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FQf5Rn3U0tejDnmzZj+gGv2suCDKUxoYIgY5TsACDAE=; b=WOS8PKPx6gwk5V5PxDRwexmzdRhYG1PoFMLzvAXdBSWiGwnRkwuhNP7K56HJkNtIBt 7ORvjaf4F8JYecgTo9tWrftfsC9nUebao9Ojrmv8R2vl/LbrQMawOvl4qtTazkplvdqo 7BjdDowYKI+FPE4gE2UucmpE9Xmf/LJ+v98C55kSJXPHPSSguDRGa88sqWKayzPIBriR E3ZMebbnw9CYzTucCHRAMm2vOBopa4t0WGhQZ0DxCM4xiL/atgZ0kK4Fl5XecV7B5R2m QcUYvnXx7KsPaOmyRTLmHlx1eeVmXiWCJftbpMnYhCEaxQk1/3b8V/+w6DQwAb0tabt+ UF7g== X-Gm-Message-State: AOJu0YxrNxj16jn9eh2T2oqOgt7b8KQXr36sXoIHSkUaB+P3BoygyQBt yhPqVwcaDW5eFlMiHe/vFaFvj4DCrnyc74BMn9Bt05OQ2pqEDl5LRywNdVRhqHPEXj+6LVGtfME S1L3Y2Vc6jA== X-Google-Smtp-Source: AGHT+IHMYSktWYBj/911l6Rv1c6+xVTTKhgM1F+kwsGtNI0RHoF3IpMtDrgBR42uRvL8aRBon8doZyPZT5gGPw== X-Received: from vsvg17.prod.google.com ([2002:a05:6102:1591:b0:4af:e0b1:8690]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6102:3fa5:b0:4b1:130f:9fbb with SMTP id ada2fe7eead31-4b2cc488b98mr57416090137.26.1736271524223; Tue, 07 Jan 2025 09:38:44 -0800 (PST) Date: Tue, 7 Jan 2025 17:38:36 +0000 In-Reply-To: <20250107173838.1130187-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250107173838.1130187-1-edumazet@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250107173838.1130187-3-edumazet@google.com> Subject: [PATCH net-next 2/4] net: no longer hold RTNL while calling flush_all_backlogs() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, Simon Horman , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org flush_all_backlogs() is called from unregister_netdevice_many_notify() as part of netdevice dismantles. This is currently called under RTNL, and can last up to 20ms on busy hosts. There is no reason to block RTNL at this stage. Signed-off-by: Eric Dumazet --- net/core/dev.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/net/core/dev.c b/net/core/dev.c index 9e1eb272e4feaf40dc87defd54d691634e0902e5..ef6426aad84dc00740a1716c8fd4cfd48ee17cf3 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -11526,8 +11526,10 @@ void unregister_netdevice_many_notify(struct list_head *head, unlist_netdevice(dev); WRITE_ONCE(dev->reg_state, NETREG_UNREGISTERING); } - flush_all_backlogs(); + __rtnl_unlock(); + flush_all_backlogs(); + rtnl_lock(); synchronize_net(); list_for_each_entry(dev, head, unreg_list) { From patchwork Tue Jan 7 17:38:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13929452 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qv1-f73.google.com (mail-qv1-f73.google.com [209.85.219.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0355B1E47DB for ; Tue, 7 Jan 2025 17:38:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736271528; cv=none; b=q0S+DdypQoJvC3eyYSunOgUopyg4jY7DFprkQ7IiUt/WTRAOMSwGA22M479aKzxrsSfFdFUjWdh1KyWjz6vZl8AYCaxzAzIvxCTVSqFGo2K4icti/E/QdmQ6SDnEICf6CgS6HPNSuWiTlmvs2oK5kmNIgz6VjerIXRC5LBeiZzU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736271528; c=relaxed/simple; bh=G+jmvAfKpB8Ie4RA8MBpaCEM29TIaye/RR8nb8RsKu0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=n+wBFalYwpSqRwngpNM2nEERVA2nYf5X3H9e1mlcBASG4HlS6ch5rohdabEta/iCtOhCPPXxmfXFj5ZErK8YqnTZ3E3DaK4ECyLluN/DRKBWc5M5YDwtwgycjqm9JFlEwJIEfBCoh89rf8TM32x6tWb/BVJAB+KAz6pu24p7Mrs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=R/VdiG90; arc=none smtp.client-ip=209.85.219.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="R/VdiG90" Received: by mail-qv1-f73.google.com with SMTP id 6a1803df08f44-6d8f0b1023bso177251296d6.0 for ; Tue, 07 Jan 2025 09:38:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736271526; x=1736876326; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xFBuXtAygN/R9fNwpJcUlBWbvTZ+VC1nOIUEZqN6qNI=; b=R/VdiG90I8pITV+xEVhxqpRis43kpX281UFKAal5JufzlgA7rQySShfjbWpa0rmCXr G/hoOWHc+8MW4ltWgJkV3+vDEnGcy/m7o91kEKjQRKkfLvqYbnD9xpOF7RgFxq6Ga/fX 9Pj97DlPki9h964WRwBiAnBKdlmBUELOMv8mEZjiRVUqVFZEXDwmKwwmh3moUpT6Seo/ 3z0fThcPLX+aceVm2qe4/9IYkeLc86rBwcf9YeBWFIx8uzOVteEgh3EID4TFXmX2JMzv C8rSFcPN7x8bQnnM8RFocDw+Ee1C3sxVtB3zopl2zUJy8DZdlC57/1DooUzTWUGZrDs9 x3Xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736271526; x=1736876326; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xFBuXtAygN/R9fNwpJcUlBWbvTZ+VC1nOIUEZqN6qNI=; b=InJAnaOwcbIvL/2GyMATHUPnpKjW/JyMavWcbtxsU2CelaV0Irq/JvVi9iJc6ZTp7g FmPf8wz5672LjIj2j+MOZ7+QJllUb7FBOoYUVWRRnEWSrLXghWHqSBm+PuhDTkSNNhbM ngLIKUTLc6YFAUKz0/U2EsC8I60ASIb0QVHvCS2LundMBwQgcgcFoaPLfe2vvAPJMEsL 9/9XpTEJOtsmnAwWbWb9cqwwm90pQspdGFhSu+/gcrrBaQJwb7WWCxTTyoZO8mp7biVW TK95eLGkiJpxdsh0WdxpC+iV3eK1/QMO0oAzbwII8B0/rw1MMuIpDUU9tMpc7KGmDV/d MNgA== X-Gm-Message-State: AOJu0Yw13HKQyVCJeDdlxNk5ziu87GYZzXYVsh83jGIxzTckEoM8/Fmq RCTFV7D6Bx8LCsHlZeMVOfH/BlSFkeuJp8yEqP7MWW3Pp3J40Z3qBm5479wQQE4iYNX0jlTA3jU sZX1u3ngj5Q== X-Google-Smtp-Source: AGHT+IEAfy8i/NJk3i4lUpt/yLMtGzjXcNQ8iQW5av3lKm9/MdrOsDS8wmyYd/mBlpatUBhTAxkvDa0TKBtk2Q== X-Received: from qvad5.prod.google.com ([2002:a0c:f105:0:b0:6d9:2e33:ce28]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6214:401c:b0:6d8:a188:369f with SMTP id 6a1803df08f44-6dd2332bf80mr843328396d6.14.1736271525966; Tue, 07 Jan 2025 09:38:45 -0800 (PST) Date: Tue, 7 Jan 2025 17:38:37 +0000 In-Reply-To: <20250107173838.1130187-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250107173838.1130187-1-edumazet@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250107173838.1130187-4-edumazet@google.com> Subject: [PATCH net-next 3/4] net: expedite synchronize_net() for cleanup_net() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, Simon Horman , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org cleanup_net() is the single thread responsible for netns dismantles, and a potential bottleneck. Before we can get per-netns RTNL, make sure all synchronize_net() called from this thread are using rcu_synchronize_expedited(). Signed-off-by: Eric Dumazet --- include/net/net_namespace.h | 2 ++ net/core/dev.c | 2 +- net/core/net_namespace.c | 5 +++++ 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/include/net/net_namespace.h b/include/net/net_namespace.h index 5a2a0df8ad91b677b515b392869c6c755be5c868..a3009bdd7efec0a3b665cbf51c159c323458410a 100644 --- a/include/net/net_namespace.h +++ b/include/net/net_namespace.h @@ -565,4 +565,6 @@ void net_ns_init(void); static inline void net_ns_init(void) {} #endif +extern struct task_struct *cleanup_net_task; + #endif /* __NET_NET_NAMESPACE_H */ diff --git a/net/core/dev.c b/net/core/dev.c index ef6426aad84dc00740a1716c8fd4cfd48ee17cf3..342ab7d6001da8983db450f50327fc7915b0a8ba 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -11423,7 +11423,7 @@ EXPORT_SYMBOL_GPL(alloc_netdev_dummy); void synchronize_net(void) { might_sleep(); - if (rtnl_is_locked()) + if (current == cleanup_net_task || rtnl_is_locked()) synchronize_rcu_expedited(); else synchronize_rcu(); diff --git a/net/core/net_namespace.c b/net/core/net_namespace.c index b5cd3ae4f04cf28d43f8401a3dafebac4a297123..cb39a12b2f8295c605f08b5589932932150a1644 100644 --- a/net/core/net_namespace.c +++ b/net/core/net_namespace.c @@ -588,6 +588,8 @@ static void unhash_nsid(struct net *net, struct net *last) static LLIST_HEAD(cleanup_list); +struct task_struct *cleanup_net_task; + static void cleanup_net(struct work_struct *work) { const struct pernet_operations *ops; @@ -596,6 +598,8 @@ static void cleanup_net(struct work_struct *work) LIST_HEAD(net_exit_list); LIST_HEAD(dev_kill_list); + cleanup_net_task = current; + /* Atomically snapshot the list of namespaces to cleanup */ net_kill_list = llist_del_all(&cleanup_list); @@ -670,6 +674,7 @@ static void cleanup_net(struct work_struct *work) put_user_ns(net->user_ns); net_free(net); } + cleanup_net_task = NULL; } /** From patchwork Tue Jan 7 17:38:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13929453 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7670018C937 for ; Tue, 7 Jan 2025 17:38:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736271529; cv=none; b=jcK5aVWbIFZHq68MT//7K7oHIQTf+s8SiBzXST8BkNwj7RmApsM9sZoCYsZ2rnn+YOgw3V5MK8rNYeii/2PDd9UhbkT1Y/MQ8TZcpPBXJeAOZs0Jdb/SUuSt6rvE6iwluBspsjpVA7qnbmvjpm1kb3kfzf+fET/AhabmvL2PA+E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736271529; c=relaxed/simple; bh=vvHJsNZfs8a27k8zTsQo7kKPdJPs3tLoWM/n0R9OT+s=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=BC3c6C4y4oAlT7ZibZcfdKlOVAvCBIB/fU+YOkCcDZ7uMQpNfbPToDZFKU6cJ1glo5bGeY4pxscXJks6HCYZtD3s+wwpImNAtVolHBfTCc534Hd0BM/qfFX7Pi/KRKFkJN3UMPC8b1P8zXpzpDbOoQNder39jl5qhnzZ+RnGgcE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=buDNxbPl; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="buDNxbPl" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e549de22484so8884233276.2 for ; Tue, 07 Jan 2025 09:38:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736271527; x=1736876327; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CcwIt0Uo0z7zPC6nu2T1OIPBfOjdzNaM9QUuptz60Aw=; b=buDNxbPlSH5+CW6gZ17yDwLlazzOHOXecJGQYYTpNbRt6WlFW7bPlihxE2v38PP+Qu at0fZY8tbXINJXyifB2XDX56n7/pOaXpNwsuVLGDQQEEYUYxHV6pRSx0AmrB4lxtvZ/9 th2pzbCuBsBdxCR1I2/Nx3kCGKR72ZBy9EbcuHiCsaZQWJGWgdD04rKHILGlZxglj+wa QiuDxxyaV0PeR6024eT7Avfp4utMBeNMEJuyJEtrsxQdvsi15yxOpuBRz/ocWLg2BnXr JGSOUYzRxLOHig9WaPt4gj5Gz6KBD3FKb1lKqnZYbzepmJmnZ5puZTs/WkVxThrOjuq4 LNWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736271527; x=1736876327; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CcwIt0Uo0z7zPC6nu2T1OIPBfOjdzNaM9QUuptz60Aw=; b=i2sTEayqkqnSjb+bjBo3lKmtsDV5VNydaQRNCzskTSDFm2HHZTPmcKb7GHXNQxKLr9 m5eMUyTd+qgtNmFNUiOn6cEpDoVTupLsRyGCxhFu4uN8iCMseqwZojWX4T6wp8iqaBnw AhLdDbthegD9HZ0F5OrzmDQ4UID+zhe4M9UUw15scqHnk68AGKq0qx8rsNA46ZEKdTs4 QlpaDp2Ifa6FHcJ6SvuMHiy60MM58cD+RvMRT0w/ILYiLuwo/QaYGER/Tv6G0VI3ouwZ 0f/1OHqNV7jQ9tyGXNIk470a1k8P5XiY136jZYNzVw3Si67v2Tb4/2K696A+X72fT2ga mrAw== X-Gm-Message-State: AOJu0Yw2lRxvN15SC0l7zMqjO6OSIgnaBcvQcxvm8xzA/Sfhra4XMwOY p9qFiLa4kR4LVanuV4bCVCua8dV/fSATno2oUWFYceK/kAXTqV2nxaBmOd+UYEMiFxyVUA7vkZr x24OG+KIqqQ== X-Google-Smtp-Source: AGHT+IH6IKOwZKwMYvyXUzv00azDOf3cBObuiSj86dq6wZWpS8p/vM8G7Lk7gWIy+sYetGJuJhOJxyKvZ7iWJg== X-Received: from ywbir11.prod.google.com ([2002:a05:690c:6c0b:b0:6f0:1645:7f09]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:a05:690c:102:b0:6f2:9533:8fb1 with SMTP id 00721157ae682-6f3f80cdf5cmr419580497b3.7.1736271527475; Tue, 07 Jan 2025 09:38:47 -0800 (PST) Date: Tue, 7 Jan 2025 17:38:38 +0000 In-Reply-To: <20250107173838.1130187-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250107173838.1130187-1-edumazet@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250107173838.1130187-5-edumazet@google.com> Subject: [PATCH net-next 4/4] net: reduce RTNL hold duration in unregister_netdevice_many_notify() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, Simon Horman , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Two synchronize_net() calls are currently done while holding RTNL. This is source of RTNL contention in workloads adding and deleting many network namespaces per second, because synchronize_rcu() and synchronize_rcu_expedited() can use 10+ ms in some cases. Signed-off-by: Eric Dumazet --- net/core/dev.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/net/core/dev.c b/net/core/dev.c index 342ab7d6001da8983db450f50327fc7915b0a8ba..9e93b13b9a76bd256d93d05a13d21dca883d6ab8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -11529,8 +11529,8 @@ void unregister_netdevice_many_notify(struct list_head *head, __rtnl_unlock(); flush_all_backlogs(); - rtnl_lock(); synchronize_net(); + rtnl_lock(); list_for_each_entry(dev, head, unreg_list) { struct sk_buff *skb = NULL; @@ -11590,7 +11590,9 @@ void unregister_netdevice_many_notify(struct list_head *head, #endif } + __rtnl_unlock(); synchronize_net(); + rtnl_lock(); list_for_each_entry(dev, head, unreg_list) { netdev_put(dev, &dev->dev_registered_tracker);