From patchwork Wed Feb 28 12:05:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13575399 X-Patchwork-Delegate: kuba@kernel.org Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CFA37353D for ; Wed, 28 Feb 2024 12:10:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709122206; cv=none; b=PmDmopFKZeqwFOHX8NG/RIISLiu9fBaxVxsdUQMGJ0+azIuOIF2ut5D3U2K2hSgTj9FiGUlpnvkmWXupRhv99bePBcXJRguz8m7/Kp//DS9kgy5HGogWBzi1Wgvgt3HFYAhRwagqf9jMXSGRMxrpi1s1R1eeAZjJ5XuyeR4xJ+0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709122206; c=relaxed/simple; bh=d5cZxgHOSxYl2L5vyXHl0TuAk0hcCh84UFLszr2LNgU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gn3lhUWN5NupSEJzepygjS/T7ncaqMFMl78RsSQ8k+M2wSlRznrmJI1Rm8xyTIpSkRXhQMcLWxNPmX2+miXGcro/74TzhSMiYJtspJWmTPMk45nQ2XMu2w6ESSQD95dVzEf+E0LL4DatTneN/RldnVXkfUcbxiZmDix45n2kb08= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=sm1CWNs5; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Bth0fIfn; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="sm1CWNs5"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Bth0fIfn" From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1709122203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yosrrn4m374EGK/14DqRIz1O1n9eFp/cpKcHUCkz69g=; b=sm1CWNs5xijf56G5hGtNkkZXm+2aqYQ9FvSI6moGHkRiZHK5kYxKEdh3Y1aAIm2SvqoHDq hM31KBZUpmV5+LwmW5SP2WsFLHFLOr3xXynrZhbZnTyBnIPgKmH6yAoAoqV9ol57vpoQoL nibkUSwqu02Yen3sUOGAjwNGnvE8tERS5aJ5qqtzXxpYTCnMFk+DZ8R3dpoHtqzYFmcfPW lPsTxuCbrwDwyqwVa5o81IlNiPANsAEA28yqAsCCEnIiFfvp/BVoDKwJdCh/z1nc2eAOXU 9rDvfIt8nIvimQ39B2tYnQ57W+17zG/artHOJAuYXkA8XTC3ndrISfVRaJTwPw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1709122203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yosrrn4m374EGK/14DqRIz1O1n9eFp/cpKcHUCkz69g=; b=Bth0fIfnY4ZtqU1xWR+cta+csMrKMfovQGP6rXWj7zbEPBxFuDJ17lMkB+MsBcdZCi2wZF UYLpEcGLmcKHqWDA== To: netdev@vger.kernel.org Cc: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Jesper Dangaard Brouer , Paolo Abeni , Thomas Gleixner , Wander Lairson Costa , Yan Zhai , Sebastian Andrzej Siewior Subject: [PATCH v3 net-next 1/4] net: Remove conditional threaded-NAPI wakeup based on task state. Date: Wed, 28 Feb 2024 13:05:48 +0100 Message-ID: <20240228121000.526645-2-bigeasy@linutronix.de> In-Reply-To: <20240228121000.526645-1-bigeasy@linutronix.de> References: <20240228121000.526645-1-bigeasy@linutronix.de> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org A NAPI thread is scheduled by first setting NAPI_STATE_SCHED bit. If successful (the bit was not yet set) then the NAPI_STATE_SCHED_THREADED is set but only if thread's state is not TASK_INTERRUPTIBLE (is TASK_RUNNING) followed by task wakeup. If the task is idle (TASK_INTERRUPTIBLE) then the NAPI_STATE_SCHED_THREADED bit is not set. The thread is no relying on the bit but always leaving the wait-loop after returning from schedule() because there must have been a wakeup. The smpboot-threads implementation for per-CPU threads requires an explicit condition and does not support "if we get out of schedule() then there must be something to do". Removing this optimisation simplifies the following integration. Set NAPI_STATE_SCHED_THREADED unconditionally on wakeup and rely on it in the wait path by removing the `woken' condition. Signed-off-by: Sebastian Andrzej Siewior --- net/core/dev.c | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index 275fd5259a4a9..dbc2f96ac37eb 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4435,13 +4435,7 @@ static inline void ____napi_schedule(struct softnet_data *sd, */ thread = READ_ONCE(napi->thread); if (thread) { - /* Avoid doing set_bit() if the thread is in - * INTERRUPTIBLE state, cause napi_thread_wait() - * makes sure to proceed with napi polling - * if the thread is explicitly woken from here. - */ - if (READ_ONCE(thread->__state) != TASK_INTERRUPTIBLE) - set_bit(NAPI_STATE_SCHED_THREADED, &napi->state); + set_bit(NAPI_STATE_SCHED_THREADED, &napi->state); wake_up_process(thread); return; } @@ -6700,8 +6694,6 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) static int napi_thread_wait(struct napi_struct *napi) { - bool woken = false; - set_current_state(TASK_INTERRUPTIBLE); while (!kthread_should_stop()) { @@ -6710,15 +6702,13 @@ static int napi_thread_wait(struct napi_struct *napi) * Testing SCHED bit is not enough because SCHED bit might be * set by some other busy poll thread or by napi_disable(). */ - if (test_bit(NAPI_STATE_SCHED_THREADED, &napi->state) || woken) { + if (test_bit(NAPI_STATE_SCHED_THREADED, &napi->state)) { WARN_ON(!list_empty(&napi->poll_list)); __set_current_state(TASK_RUNNING); return 0; } schedule(); - /* woken being true indicates this thread owns this napi. */ - woken = true; set_current_state(TASK_INTERRUPTIBLE); } __set_current_state(TASK_RUNNING); From patchwork Wed Feb 28 12:05:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13575400 X-Patchwork-Delegate: kuba@kernel.org Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CF707353A for ; Wed, 28 Feb 2024 12:10:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709122207; cv=none; b=cq+W5Cy4g6ZVEp/7JgQVmx8KjARnuWfSQo1mmmwuewj6gaewbp94zztgZAJWp7NgN55eE7Eyi1OL8GBfhsV4MGNblG5DlewYiilMB8K5/lZkFrzDoIFjMVi0PSlMj0qhOkvDLXqsqE7RwWkb2BOlgRA0q5TYa0mBd7EG7xFK3f8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709122207; c=relaxed/simple; bh=24XfI8bRwfjgfWr+OxhkQpYmSBw56uH2f0q2PvDrA+Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fVAW/N4UbSSBe4V5AslVuDeN/taroL4EhulC5pJQrf3SkEqYQNB3QFB/yfj1XSyaZ2eOKrqaMywkE76MSQCwAtmMFvsR597ATMP6goHuoOEjwDqCOp4Cce1n7Uuw0L9OY+F978RhQaDnKRuE6bQ6JJsYA3OvGuDu2PUVVTuwy5E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=F6dDnA0I; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=mSHc6WYo; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="F6dDnA0I"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="mSHc6WYo" From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1709122203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Znkp3ifTqRg2GmbG7mmVzJfLRgA/pM5U2UkA17ifj5U=; b=F6dDnA0IrYGJk629aHWKhbD/POOAEP6BnIsNVAAJ2eblC+jF4j3e/YdtwWl8quE/RFGJka USXiCQnRw5V8rJfK7XI+7FRwXpvoM7NCvDAxA1QT/CiWv/FqqxOh76e2ixqj8QDOiHkCzS 5ir2w5LNdWK8EXlzLzMKnUlQLBfnzvNeWFIaKs5DJvOIQIBZymupiVXzhW8qUKeXR/D2rj FqSp+K4buoEzkDoQyIhFfqgroQPM//1QZAsBT5xjieB/30p/4nchXlEDC2Wiapo1X0HaUc EqDRTXE8DsF2Fyrwt76OKgGnCMlgNIf6QZ9JDXLHUr/r17evfzYKXGpO/P5Qdg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1709122203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Znkp3ifTqRg2GmbG7mmVzJfLRgA/pM5U2UkA17ifj5U=; b=mSHc6WYoTzZUYjBphLyY57sHIf+yX0XMJNOcqA8UfjNZ5S79SlFUpTIXMFF7ebWbPte7iy KZxR7duTtFFZccBA== To: netdev@vger.kernel.org Cc: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Jesper Dangaard Brouer , Paolo Abeni , Thomas Gleixner , Wander Lairson Costa , Yan Zhai , Sebastian Andrzej Siewior Subject: [PATCH v3 net-next 2/4] net: Allow to use SMP threads for backlog NAPI. Date: Wed, 28 Feb 2024 13:05:49 +0100 Message-ID: <20240228121000.526645-3-bigeasy@linutronix.de> In-Reply-To: <20240228121000.526645-1-bigeasy@linutronix.de> References: <20240228121000.526645-1-bigeasy@linutronix.de> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Backlog NAPI is a per-CPU NAPI struct only (with no device behind it) used by drivers which don't do NAPI them self, RPS and parts of the stack which need to avoid recursive deadlocks while processing a packet. The non-NAPI driver use the CPU local backlog NAPI. If RPS is enabled then a flow for the skb is computed and based on the flow the skb can be enqueued on a remote CPU. Scheduling/ raising the softirq (for backlog's NAPI) on the remote CPU isn't trivial because the softirq is only scheduled on the local CPU and performed after the hardirq is done. In order to schedule a softirq on the remote CPU, an IPI is sent to the remote CPU which schedules the backlog-NAPI on the then local CPU. On PREEMPT_RT interrupts are force-threaded. The soft interrupts are raised within the interrupt thread and processed after the interrupt handler completed still within the context of the interrupt thread. The softirq is handled in the context where it originated. With force-threaded interrupts enabled, ksoftirqd is woken up if a softirq is raised from hardirq context. This is the case if it is raised from an IPI. Additionally there is a warning on PREEMPT_RT if the softirq is raised from the idle thread. This was done for two reasons: - With threaded interrupts the processing should happen in thread context (where it originated) and ksoftirqd is the only thread for this context if raised from hardirq. Using the currently running task instead would "punish" a random task. - Once ksoftirqd is active it consumes all further softirqs until it stops running. This changed recently and is no longer the case. Instead of keeping the backlog NAPI in ksoftirqd (in force-threaded/ PREEMPT_RT setups) I am proposing NAPI-threads for backlog. The "proper" setup with threaded-NAPI is not doable because the threads are not pinned to an individual CPU and can be modified by the user. Additionally a dummy network device would have to be assigned. Also CPU-hotplug has to be considered if additional CPUs show up. All this can be probably done/ solved but the smpboot-threads already provide this infrastructure. Sending UDP packets over loopback expects that the packet is processed within the call. Delaying it by handing it over to the thread hurts performance. It is not beneficial to the outcome if the context switch happens immediately after enqueue or after a while to process a few packets in a batch. There is no need to always use the thread if the backlog NAPI is requested on the local CPU. This restores the loopback throuput. The performance drops mostly to the same value after enabling RPS on the loopback comparing the IPI and the tread result. Create NAPI-threads for backlog if request during boot. The thread runs the inner loop from napi_threaded_poll(), the wait part is different. It checks for NAPI_STATE_SCHED (the backlog NAPI can not be disabled). The NAPI threads for backlog are optional, it has to be enabled via the boot argument "thread_backlog_napi". It is mandatory for PREEMPT_RT to avoid the wakeup of ksoftirqd from the IPI. Signed-off-by: Sebastian Andrzej Siewior --- net/core/dev.c | 149 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 112 insertions(+), 37 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index dbc2f96ac37eb..b449b60546554 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -78,6 +78,7 @@ #include #include #include +#include #include #include #include @@ -196,6 +197,31 @@ static inline struct hlist_head *dev_index_hash(struct net *net, int ifindex) return &net->dev_index_head[ifindex & (NETDEV_HASHENTRIES - 1)]; } +#ifndef CONFIG_PREEMPT_RT + +static DEFINE_STATIC_KEY_FALSE(use_backlog_threads_key); + +static int __init setup_backlog_napi_threads(char *arg) +{ + static_branch_enable(&use_backlog_threads_key); + return 0; +} +early_param("thread_backlog_napi", setup_backlog_napi_threads); + +static bool use_backlog_threads(void) +{ + return static_branch_unlikely(&use_backlog_threads_key); +} + +#else + +static bool use_backlog_threads(void) +{ + return true; +} + +#endif + static inline void rps_lock_irqsave(struct softnet_data *sd, unsigned long *flags) { @@ -4403,6 +4429,7 @@ EXPORT_SYMBOL(__dev_direct_xmit); /************************************************************************* * Receiver routines *************************************************************************/ +static DEFINE_PER_CPU(struct task_struct *, backlog_napi); int netdev_max_backlog __read_mostly = 1000; EXPORT_SYMBOL(netdev_max_backlog); @@ -4435,12 +4462,16 @@ static inline void ____napi_schedule(struct softnet_data *sd, */ thread = READ_ONCE(napi->thread); if (thread) { + if (use_backlog_threads() && thread == raw_cpu_read(backlog_napi)) + goto use_local_napi; + set_bit(NAPI_STATE_SCHED_THREADED, &napi->state); wake_up_process(thread); return; } } +use_local_napi: list_add_tail(&napi->poll_list, &sd->poll_list); WRITE_ONCE(napi->list_owner, smp_processor_id()); /* If not called from net_rx_action() @@ -4686,6 +4717,11 @@ static void napi_schedule_rps(struct softnet_data *sd) #ifdef CONFIG_RPS if (sd != mysd) { + if (use_backlog_threads()) { + __napi_schedule_irqoff(&sd->backlog); + return; + } + sd->rps_ipi_next = mysd->rps_ipi_list; mysd->rps_ipi_list = sd; @@ -5943,7 +5979,7 @@ static void net_rps_action_and_irq_enable(struct softnet_data *sd) #ifdef CONFIG_RPS struct softnet_data *remsd = sd->rps_ipi_list; - if (remsd) { + if (!use_backlog_threads() && remsd) { sd->rps_ipi_list = NULL; local_irq_enable(); @@ -5958,7 +5994,7 @@ static void net_rps_action_and_irq_enable(struct softnet_data *sd) static bool sd_has_rps_ipi_waiting(struct softnet_data *sd) { #ifdef CONFIG_RPS - return sd->rps_ipi_list != NULL; + return !use_backlog_threads() && sd->rps_ipi_list; #else return false; #endif @@ -6002,7 +6038,7 @@ static int process_backlog(struct napi_struct *napi, int quota) * We can use a plain write instead of clear_bit(), * and we dont need an smp_mb() memory barrier. */ - napi->state = 0; + napi->state &= NAPIF_STATE_THREADED; again = false; } else { skb_queue_splice_tail_init(&sd->input_pkt_queue, @@ -6737,40 +6773,42 @@ static void skb_defer_free_flush(struct softnet_data *sd) } } +static void napi_threaded_poll_loop(struct napi_struct *napi) +{ + struct softnet_data *sd; + + for (;;) { + bool repoll = false; + void *have; + + local_bh_disable(); + sd = this_cpu_ptr(&softnet_data); + sd->in_napi_threaded_poll = true; + + have = netpoll_poll_lock(napi); + __napi_poll(napi, &repoll); + netpoll_poll_unlock(have); + + sd->in_napi_threaded_poll = false; + barrier(); + + skb_defer_free_flush(sd); + local_bh_enable(); + + if (!repoll) + break; + + cond_resched(); + } +} + static int napi_threaded_poll(void *data) { struct napi_struct *napi = data; - struct softnet_data *sd; - void *have; - while (!napi_thread_wait(napi)) { - for (;;) { - bool repoll = false; + while (!napi_thread_wait(napi)) + napi_threaded_poll_loop(napi); - local_bh_disable(); - sd = this_cpu_ptr(&softnet_data); - sd->in_napi_threaded_poll = true; - - have = netpoll_poll_lock(napi); - __napi_poll(napi, &repoll); - netpoll_poll_unlock(have); - - sd->in_napi_threaded_poll = false; - barrier(); - - if (sd_has_rps_ipi_waiting(sd)) { - local_irq_disable(); - net_rps_action_and_irq_enable(sd); - } - skb_defer_free_flush(sd); - local_bh_enable(); - - if (!repoll) - break; - - cond_resched(); - } - } return 0; } @@ -11391,7 +11429,7 @@ static int dev_cpu_dead(unsigned int oldcpu) list_del_init(&napi->poll_list); if (napi->poll == process_backlog) - napi->state = 0; + napi->state &= NAPIF_STATE_THREADED; else ____napi_schedule(sd, napi); } @@ -11399,12 +11437,14 @@ static int dev_cpu_dead(unsigned int oldcpu) raise_softirq_irqoff(NET_TX_SOFTIRQ); local_irq_enable(); + if (!use_backlog_threads()) { #ifdef CONFIG_RPS - remsd = oldsd->rps_ipi_list; - oldsd->rps_ipi_list = NULL; + remsd = oldsd->rps_ipi_list; + oldsd->rps_ipi_list = NULL; #endif - /* send out pending IPI's on offline CPU */ - net_rps_send_ipi(remsd); + /* send out pending IPI's on offline CPU */ + net_rps_send_ipi(remsd); + } /* Process offline CPU's input_pkt_queue */ while ((skb = __skb_dequeue(&oldsd->process_queue))) { @@ -11742,6 +11782,38 @@ static int net_page_pool_create(int cpuid) return 0; } +static int backlog_napi_should_run(unsigned int cpu) +{ + struct softnet_data *sd = per_cpu_ptr(&softnet_data, cpu); + struct napi_struct *napi = &sd->backlog; + + return test_bit(NAPI_STATE_SCHED_THREADED, &napi->state); +} + +static void run_backlog_napi(unsigned int cpu) +{ + struct softnet_data *sd = per_cpu_ptr(&softnet_data, cpu); + + napi_threaded_poll_loop(&sd->backlog); +} + +static void backlog_napi_setup(unsigned int cpu) +{ + struct softnet_data *sd = per_cpu_ptr(&softnet_data, cpu); + struct napi_struct *napi = &sd->backlog; + + napi->thread = this_cpu_read(backlog_napi); + set_bit(NAPI_STATE_THREADED, &napi->state); +} + +static struct smp_hotplug_thread backlog_threads = { + .store = &backlog_napi, + .thread_should_run = backlog_napi_should_run, + .thread_fn = run_backlog_napi, + .thread_comm = "backlog_napi/%u", + .setup = backlog_napi_setup, +}; + /* * This is called single threaded during boot, so no need * to take the rtnl semaphore. @@ -11794,10 +11866,13 @@ static int __init net_dev_init(void) init_gro_hash(&sd->backlog); sd->backlog.poll = process_backlog; sd->backlog.weight = weight_p; + INIT_LIST_HEAD(&sd->backlog.poll_list); if (net_page_pool_create(i)) goto out; } + if (use_backlog_threads()) + smpboot_register_percpu_thread(&backlog_threads); dev_boot_phase = 0; From patchwork Wed Feb 28 12:05:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13575401 X-Patchwork-Delegate: kuba@kernel.org Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71E3E7353F for ; Wed, 28 Feb 2024 12:10:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709122207; cv=none; b=IJKCRK5YaKaY0qW/+akUoSm4f7a/jjG1CEAtR7wLMQpVxnpkKY3cJwIMmHOIZGw8RsVHN5ysASmIqYt1hxQcrzPmWJ6OAxc1dNtNaJo6QPY0K6aXOnv5plDNhie1Knuk5kZ/tONum31ZfIfArZRSvuj5sD9LwYwJaLGE/WZro9o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709122207; c=relaxed/simple; bh=CgKiabh5Lt6zyfssdKEFyuIHwHEXUsJjgNB+tEwXYXU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mONw/LIXKYhu2Jxooj9jnHPxdjaiZzSI8aVNNImoRFthNpcOEHSY8JtZdZaMYmUsIZLAs0EjWIjT76IvUbQ+6Ov10S9kb/LF6j6VJU5G5GSY1/jXS0eRgAuGPbl3esm8ENzmkVYtQaqN/qkeUkIJflg45OUnNUs5YYEHcysJOmw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=DNTn7aW6; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=LWtzUQTW; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="DNTn7aW6"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="LWtzUQTW" From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1709122203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4oEYqKhwveXGIGItueiCCGQMV594VcZ8EGsFFNqBsag=; b=DNTn7aW6cvGrQrlZzCFv3f3mO/qxUaO2mneVdDMaZXGqm8dWbEQE5pf7w8RMvlKSrv8iHc hMQ6jowCZVAEGbd08lWT0dC4f1qDBPWqsG8VdH58CDkU8yYbzTYLkNSLa7ymsIIc/EK2EX 0PLG+7GCpbshJekn2cmZmMh/H+3LpxJmKugqMhEHCUrhOjBLfqqlyAfovQJAF27bcQEpiR 8kDKVddEwf+kHZoSPkn2WJH7qkGZPBzP36+8EnuSsiKJM8Tnk2SP0RYlLDwOTVcAh1nxbr RveXJZsBzXJrlyS/pwBCfJfn4p9WADMu4DfeuZsZW7CN0gNhjXYKgx7YsHX3ow== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1709122203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4oEYqKhwveXGIGItueiCCGQMV594VcZ8EGsFFNqBsag=; b=LWtzUQTW8sC/8pPld3ZAe8xP+IKitEBIf9Er60ayRddyB0w3E4nTgRPFcT4OiN7gCN2LxC XE3FG8pu14YXD0DA== To: netdev@vger.kernel.org Cc: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Jesper Dangaard Brouer , Paolo Abeni , Thomas Gleixner , Wander Lairson Costa , Yan Zhai , Sebastian Andrzej Siewior Subject: [PATCH v3 net-next 3/4] net: Use backlog-NAPI to clean up the defer_list. Date: Wed, 28 Feb 2024 13:05:50 +0100 Message-ID: <20240228121000.526645-4-bigeasy@linutronix.de> In-Reply-To: <20240228121000.526645-1-bigeasy@linutronix.de> References: <20240228121000.526645-1-bigeasy@linutronix.de> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org The defer_list is a per-CPU list which is used to free skbs outside of the socket lock and on the CPU on which they have been allocated. The list is processed during NAPI callbacks so ideally the list is cleaned up. Should the amount of skbs on the list exceed a certain water mark then the softirq is triggered remotely on the target CPU by invoking a remote function call. The raise of the softirqs via a remote function call leads to waking the ksoftirqd on PREEMPT_RT which is undesired. The backlog-NAPI threads already provide the infrastructure which can be utilized to perform the cleanup of the defer_list. The NAPI state is updated with the input_pkt_queue.lock acquired. It order not to break the state, it is needed to also wake the backlog-NAPI thread with the lock held. This requires to acquire the use the lock in rps_lock_irq*() if the backlog-NAPI threads are used even with RPS disabled. Move the logic of remotely starting softirqs to clean up the defer_list into kick_defer_list_purge(). Make sure a lock is held in rps_lock_irq*() if backlog-NAPI threads are used. Schedule backlog-NAPI for defer_list cleanup if backlog-NAPI is available. Signed-off-by: Sebastian Andrzej Siewior --- include/linux/netdevice.h | 1 + net/core/dev.c | 28 ++++++++++++++++++++++++---- net/core/skbuff.c | 4 ++-- 3 files changed, 27 insertions(+), 6 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 09023e44db4e2..fe5f4d7d397e2 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3368,6 +3368,7 @@ static inline void dev_xmit_recursion_dec(void) __this_cpu_dec(softnet_data.xmit.recursion); } +void kick_defer_list_purge(unsigned int cpu); void __netif_schedule(struct Qdisc *q); void netif_schedule_queue(struct netdev_queue *txq); diff --git a/net/core/dev.c b/net/core/dev.c index b449b60546554..aa831d40a1f54 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -225,7 +225,7 @@ static bool use_backlog_threads(void) static inline void rps_lock_irqsave(struct softnet_data *sd, unsigned long *flags) { - if (IS_ENABLED(CONFIG_RPS)) + if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads()) spin_lock_irqsave(&sd->input_pkt_queue.lock, *flags); else if (!IS_ENABLED(CONFIG_PREEMPT_RT)) local_irq_save(*flags); @@ -233,7 +233,7 @@ static inline void rps_lock_irqsave(struct softnet_data *sd, static inline void rps_lock_irq_disable(struct softnet_data *sd) { - if (IS_ENABLED(CONFIG_RPS)) + if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads()) spin_lock_irq(&sd->input_pkt_queue.lock); else if (!IS_ENABLED(CONFIG_PREEMPT_RT)) local_irq_disable(); @@ -242,7 +242,7 @@ static inline void rps_lock_irq_disable(struct softnet_data *sd) static inline void rps_unlock_irq_restore(struct softnet_data *sd, unsigned long *flags) { - if (IS_ENABLED(CONFIG_RPS)) + if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads()) spin_unlock_irqrestore(&sd->input_pkt_queue.lock, *flags); else if (!IS_ENABLED(CONFIG_PREEMPT_RT)) local_irq_restore(*flags); @@ -250,7 +250,7 @@ static inline void rps_unlock_irq_restore(struct softnet_data *sd, static inline void rps_unlock_irq_enable(struct softnet_data *sd) { - if (IS_ENABLED(CONFIG_RPS)) + if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads()) spin_unlock_irq(&sd->input_pkt_queue.lock); else if (!IS_ENABLED(CONFIG_PREEMPT_RT)) local_irq_enable(); @@ -4736,6 +4736,26 @@ static void napi_schedule_rps(struct softnet_data *sd) __napi_schedule_irqoff(&mysd->backlog); } +void kick_defer_list_purge(unsigned int cpu) +{ + struct softnet_data *sd; + unsigned long flags; + + sd = &per_cpu(softnet_data, cpu); + + if (use_backlog_threads()) { + rps_lock_irqsave(sd, &flags); + + if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) + __napi_schedule_irqoff(&sd->backlog); + + rps_unlock_irq_restore(sd, &flags); + + } else if (!cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) { + smp_call_function_single_async(cpu, &sd->defer_csd); + } +} + #ifdef CONFIG_NET_FLOW_LIMIT int netdev_flow_limit_table_len __read_mostly = (1 << 12); #endif diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 1f918e602bc4f..0c9676157a2c1 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -7042,8 +7042,8 @@ nodefer: __kfree_skb(skb); /* Make sure to trigger NET_RX_SOFTIRQ on the remote CPU * if we are unlucky enough (this seems very unlikely). */ - if (unlikely(kick) && !cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) - smp_call_function_single_async(cpu, &sd->defer_csd); + if (unlikely(kick)) + kick_defer_list_purge(cpu); } static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, From patchwork Wed Feb 28 12:05:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13575402 X-Patchwork-Delegate: kuba@kernel.org Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7ABC1F608 for ; Wed, 28 Feb 2024 12:10:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709122207; cv=none; b=s6XvernhdW1T5yHCDcwAdxRSip3hDXQI12SUBiQFvQhXgWa42XQ+zjMRxDkDKxfZissdc+3F0uq+OypYuECKY2Z8xwfSwsq8hhdM8eCJrUbrk8RrbnntUr6m1mQfMUqwJPQV5/KNPA/OIA5gPTD3cDfqg7JAidvRWK3Lq+ftkvI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709122207; c=relaxed/simple; bh=RAx/rSugBr3FyIsaKAEaz2zQddIN2D2tvsipZ45oJgI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tsqhbTk0dWZW5i+OhUPa36S238z7l3Y7azw3x7uPbAW4unPbq4vlt1Z9zVN0WMSRMD/ZTWA2po9jUE/apYQVPtr3EQhJ7HdHiXoBVWxTAw5c9c3W23kclbk4MciA5MQM0MEXNaUOKho1vRhO+jPqNZDbRhuiQ2T+kIhvgmibkrA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=x6F77Anq; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=yM5b0WSb; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="x6F77Anq"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="yM5b0WSb" From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1709122203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Tuhi7EF/VKrzMUN1WRvLgd9pf+l+ddOY554nuYzU+OE=; b=x6F77AnqayxZcnD+v9k86B/ePAEBkpZRt+QMgw2j+7iOf2F4auoVYzNOvhxJcFa03twtlh QF9LVkD0GUJaKxkSZDaQcKQ/bLICTmJwSyA49yvLDLlFul+soSy6ddYgprJPyYaQg+i/wA mThAxAiEYKZb6ArgtyHwbRajq0duEQ/Yu1wQ7SfSTO9c+mYZL3oPbuZdLhhMn8FyNJKHfy Iw28W3/LI/vn8gitykf7SK9dZXNyKuK9yU4ajtn/tubbuksEuE1cMCJyC6xwIdLjJ+KstG 8dXoYRRmuC1K5qPy/cl4dOkHzl+zjq6taqse+k5c0opVNGdOKwUy+VTcBL3Tuw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1709122203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Tuhi7EF/VKrzMUN1WRvLgd9pf+l+ddOY554nuYzU+OE=; b=yM5b0WSb+KVfSFhQkA5LnpO4d47oHRA5WD61Kg4RRNGAF3RyQjz26lTALZSs2XmPOGQH7t ljxBI2+36uHBWZBQ== To: netdev@vger.kernel.org Cc: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Jesper Dangaard Brouer , Paolo Abeni , Thomas Gleixner , Wander Lairson Costa , Yan Zhai , Sebastian Andrzej Siewior Subject: [PATCH v3 net-next 4/4] net: Rename rps_lock to backlog_lock. Date: Wed, 28 Feb 2024 13:05:51 +0100 Message-ID: <20240228121000.526645-5-bigeasy@linutronix.de> In-Reply-To: <20240228121000.526645-1-bigeasy@linutronix.de> References: <20240228121000.526645-1-bigeasy@linutronix.de> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org The rps_lock.*() functions use the inner lock of a sk_buff_head for locking. This lock is used if RPS is enabled, otherwise the list is accessed lockless and disabling interrupts is enough for the synchronisation because it is only accessed CPU local. Not only the list is protected but also the NAPI state protected. With the addition of backlog threads, the lock is also needed because of the cross CPU access even without RPS. The clean up of the defer_list list is also done via backlog threads (if enabled). It has been suggested to rename the locking function since it is no longer just RPS. Rename the rps_lock*() functions to backlog_lock*(). Suggested-by: Jakub Kicinski Signed-off-by: Sebastian Andrzej Siewior --- net/core/dev.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index aa831d40a1f54..6ee4231f00400 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -222,8 +222,8 @@ static bool use_backlog_threads(void) #endif -static inline void rps_lock_irqsave(struct softnet_data *sd, - unsigned long *flags) +static inline void backlog_lock_irq_save(struct softnet_data *sd, + unsigned long *flags) { if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads()) spin_lock_irqsave(&sd->input_pkt_queue.lock, *flags); @@ -231,7 +231,7 @@ static inline void rps_lock_irqsave(struct softnet_data *sd, local_irq_save(*flags); } -static inline void rps_lock_irq_disable(struct softnet_data *sd) +static inline void backlog_lock_irq_disable(struct softnet_data *sd) { if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads()) spin_lock_irq(&sd->input_pkt_queue.lock); @@ -239,8 +239,8 @@ static inline void rps_lock_irq_disable(struct softnet_data *sd) local_irq_disable(); } -static inline void rps_unlock_irq_restore(struct softnet_data *sd, - unsigned long *flags) +static inline void backlog_unlock_irq_restore(struct softnet_data *sd, + unsigned long *flags) { if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads()) spin_unlock_irqrestore(&sd->input_pkt_queue.lock, *flags); @@ -248,7 +248,7 @@ static inline void rps_unlock_irq_restore(struct softnet_data *sd, local_irq_restore(*flags); } -static inline void rps_unlock_irq_enable(struct softnet_data *sd) +static inline void backlog_unlock_irq_enable(struct softnet_data *sd) { if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads()) spin_unlock_irq(&sd->input_pkt_queue.lock); @@ -4744,12 +4744,12 @@ void kick_defer_list_purge(unsigned int cpu) sd = &per_cpu(softnet_data, cpu); if (use_backlog_threads()) { - rps_lock_irqsave(sd, &flags); + backlog_lock_irq_save(sd, &flags); if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) __napi_schedule_irqoff(&sd->backlog); - rps_unlock_irq_restore(sd, &flags); + backlog_unlock_irq_restore(sd, &flags); } else if (!cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) { smp_call_function_single_async(cpu, &sd->defer_csd); @@ -4811,7 +4811,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, reason = SKB_DROP_REASON_NOT_SPECIFIED; sd = &per_cpu(softnet_data, cpu); - rps_lock_irqsave(sd, &flags); + backlog_lock_irq_save(sd, &flags); if (!netif_running(skb->dev)) goto drop; qlen = skb_queue_len(&sd->input_pkt_queue); @@ -4820,7 +4820,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, enqueue: __skb_queue_tail(&sd->input_pkt_queue, skb); input_queue_tail_incr_save(sd, qtail); - rps_unlock_irq_restore(sd, &flags); + backlog_unlock_irq_restore(sd, &flags); return NET_RX_SUCCESS; } @@ -4835,7 +4835,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, drop: sd->dropped++; - rps_unlock_irq_restore(sd, &flags); + backlog_unlock_irq_restore(sd, &flags); dev_core_stats_rx_dropped_inc(skb->dev); kfree_skb_reason(skb, reason); @@ -5900,7 +5900,7 @@ static void flush_backlog(struct work_struct *work) local_bh_disable(); sd = this_cpu_ptr(&softnet_data); - rps_lock_irq_disable(sd); + backlog_lock_irq_disable(sd); skb_queue_walk_safe(&sd->input_pkt_queue, skb, tmp) { if (skb->dev->reg_state == NETREG_UNREGISTERING) { __skb_unlink(skb, &sd->input_pkt_queue); @@ -5908,7 +5908,7 @@ static void flush_backlog(struct work_struct *work) input_queue_head_incr(sd); } } - rps_unlock_irq_enable(sd); + backlog_unlock_irq_enable(sd); skb_queue_walk_safe(&sd->process_queue, skb, tmp) { if (skb->dev->reg_state == NETREG_UNREGISTERING) { @@ -5926,14 +5926,14 @@ static bool flush_required(int cpu) struct softnet_data *sd = &per_cpu(softnet_data, cpu); bool do_flush; - rps_lock_irq_disable(sd); + backlog_lock_irq_disable(sd); /* as insertion into process_queue happens with the rps lock held, * process_queue access may race only with dequeue */ do_flush = !skb_queue_empty(&sd->input_pkt_queue) || !skb_queue_empty_lockless(&sd->process_queue); - rps_unlock_irq_enable(sd); + backlog_unlock_irq_enable(sd); return do_flush; #endif @@ -6048,7 +6048,7 @@ static int process_backlog(struct napi_struct *napi, int quota) } - rps_lock_irq_disable(sd); + backlog_lock_irq_disable(sd); if (skb_queue_empty(&sd->input_pkt_queue)) { /* * Inline a custom version of __napi_complete(). @@ -6064,7 +6064,7 @@ static int process_backlog(struct napi_struct *napi, int quota) skb_queue_splice_tail_init(&sd->input_pkt_queue, &sd->process_queue); } - rps_unlock_irq_enable(sd); + backlog_unlock_irq_enable(sd); } return work;