From patchwork Fri Mar 29 15:42:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13610730 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 078B91C0DCF for ; Fri, 29 Mar 2024 15:42:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726951; cv=none; b=pViy3YrBVrFv+hOgmHJr9oKo1hQ212gHjdN3rg6SJZt6ijQPoHTGnIrk95MgvQ0CJG5n1AyBBWG6mzXiHfC8dUORudyTYBvKpcvrvrssivJ42Ei0fwuRe4KuklcfQsHuvNHPK5vxG93dIhipz0CwSm7G+xTvxhjhYJatshXzhWo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726951; c=relaxed/simple; bh=lFeGl7dZQs/ZceANoJRiP1nMhVUnouVH1jtKb+pAUk8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=g06RHoFluFgVA2n0LBIiQFjEwkhkVXcGPBUWfWK6LnCSzFLzO3SKFaG3nAZrcp+GCj6LS/3rC+ocd41DvlBXaQLy5FQC8LOS/5wXVTyZ64NCLVaX5z00twxuC+jF0PAwDroxzFdpiVmHgc3xHRkcxy/Dy+zrGL3ksLXzR2TJmqs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ygSHdrxs; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ygSHdrxs" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-610c23abd1fso38980727b3.0 for ; Fri, 29 Mar 2024 08:42:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711726949; x=1712331749; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Jply39cwIs/FhQ97Ad44YD+TULK2Sf78Gyu9nOqDg5M=; b=ygSHdrxsY54NRdIZetlj/Y69ULykQcTwQjtpZFXHBiX3xmrHlpTJjt9qgcuxO0Pkbq onbNtcNWVuu2J07YFGKu+GymD1hwOlRbM2D9SIKekNMYjf76e6F0anAsdG/5k48oqoCp NZnsgIm88YvhEj7bXc1RWcdyp5ywHjjHvavGTAG1iJZGeFmiSlRAAlvLvW75Igg8CqO6 HFi+9OvlcaTNUAlDafq9PhLH0QFUQE3FWem0/YFhlRkaIf81fWbUVqfihT4wlobpfLVF eq2a/UekUM39+u0xiKcPbuN82StWTJx5MsV5fcxND5xggfgwBxplOqSjD0LUCVnxAMdp GlYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711726949; x=1712331749; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Jply39cwIs/FhQ97Ad44YD+TULK2Sf78Gyu9nOqDg5M=; b=Qf6MiI6I7sHRHP4Eq7VeVoOoCPD0PuLPE2m2qgNgEWYH8WT9dfCRDWxw5lsuDxehEp 1CkHbr5j0rd43b7+NeDJgT0BAE7LQKi7my/f1we8k9RmPKX781BXYncxv/FtGb1kSwVP eR4ajMPj5P6I6LbD9yxbdbo0ks9IwVjSJnd2hl6XfN7ojM5An9TEP2NpT9OECvDGcVIZ IjOiPBP5WXGGGia10vFzI/q5v2J6/i94hGwge7qS4vwkysxs31Laf9D7eKCny3B8mAci mzgtfqS1CqpCVLK6/RhpG3/KBLP1+QzAfSMiuiYKliyhDuJGd6TLeKvG2YluGLapi+cs 4y/w== X-Gm-Message-State: AOJu0YyccQyEgoCMqVc1tDIeGj0MaWoxaaTDDdkhLZy8Lr1eFWzCS8D6 F2jRXP5uvK8sfQgtaQQBGQ48JhnMjGUYODBep/mN/RrQt479nURwhMSyclQwIO4S6LWNSyUdrNm 4/bLCempPFw== X-Google-Smtp-Source: AGHT+IGObWcfuDTpxaB2YNl3qmbcFeEonQuchJ4L065Op3Hp8qzbDUspWMck2EOKbWmAaypyLD5Jxxl4ewHcrA== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:250e:b0:dda:ce5f:b4c3 with SMTP id dt14-20020a056902250e00b00ddace5fb4c3mr780544ybb.1.1711726949119; Fri, 29 Mar 2024 08:42:29 -0700 (PDT) Date: Fri, 29 Mar 2024 15:42:18 +0000 In-Reply-To: <20240329154225.349288-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240329154225.349288-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240329154225.349288-2-edumazet@google.com> Subject: [PATCH v2 net-next 1/8] net: move kick_defer_list_purge() to net/core/dev.h From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org kick_defer_list_purge() is defined in net/core/dev.c and used from net/core/skubff.c Because we need softnet_data, include from net/core/dev.h Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 1 - net/core/dev.h | 6 +++--- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index e41d30ebaca61e48a2ceb43edf777fa8b9859ef2..cb37817d6382c29117afd8ce54db6dba94f8c930 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3287,7 +3287,6 @@ static inline void dev_xmit_recursion_dec(void) __this_cpu_dec(softnet_data.xmit.recursion); } -void kick_defer_list_purge(struct softnet_data *sd, unsigned int cpu); void __netif_schedule(struct Qdisc *q); void netif_schedule_queue(struct netdev_queue *txq); diff --git a/net/core/dev.h b/net/core/dev.h index 2bcaf8eee50c179db2ca59880521b9be6ecd45c8..9d0f8b4587f81f4c12487f1783d8ba5cc49fc1d6 100644 --- a/net/core/dev.h +++ b/net/core/dev.h @@ -4,11 +4,9 @@ #include #include +#include struct net; -struct net_device; -struct netdev_bpf; -struct netdev_phys_item_id; struct netlink_ext_ack; struct cpumask; @@ -150,4 +148,6 @@ static inline void xdp_do_check_flushed(struct napi_struct *napi) { } #endif struct napi_struct *napi_by_id(unsigned int napi_id); +void kick_defer_list_purge(struct softnet_data *sd, unsigned int cpu); + #endif From patchwork Fri Mar 29 15:42:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13610731 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7EE4A1C0DCC for ; Fri, 29 Mar 2024 15:42:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726953; cv=none; b=P4R1yb9FW375TdnAKsX+D70PQLZvMx0JY7Mt61SmwU0Gud9EcLv0z37IjIxgzw4Gly8Z3FEqUOeFYNdiDsfYk+txr+iTB/ByU/vZ97t77QpofBWczfuqbAGLPnslE1fxEFn/BDxm6617J7ez6tBBYF7Rl9JxZ7vmPbaeXwKPHV4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726953; c=relaxed/simple; bh=u6CKHjdRZ3Q3jFE+0JJbKpQLd3hkwh+kwA+sSojO8Vs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Ftl32I3ZnNUUA9ajdg1KIMwwSK98yahXEjBU/mGPctB4BprjQ5lqqQ3fvLH0OQvxMjDBA1u9R6LUyKuX7QSg77pc0r2AhYHoxaPfUqGGrPmFd+9B9ZxOFu22NAjgZjtSV0bL0I1KeQBj2+e+NwQeq4UzCcUNHddvBRasU61rpiY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=V/57zyWG; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="V/57zyWG" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60f9d800a29so33195237b3.0 for ; Fri, 29 Mar 2024 08:42:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711726950; x=1712331750; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zkkkR4bJVGdAmtjki7FbOrCs0xkI3kc/cNsgAoiErHk=; b=V/57zyWGmdJ74af3iBhnmc4mMLqMuJYJK8OesnfvTDj8w4vTidEhc+vbWCtGgOftAJ wup/RkfbVrU4JphGUdgT73UiYq8MP5hbC3FHfG/berG4ML0mBLCZpRlrErxNcMqt17UI lcYjXzHXrlu/tSM04QVJb9Pv6UwgcYioUi1IwvDbgRAjIEwaDNrY2JtYUpFO+RZLRLiK Dj7c0K55lZRO6SAlthj940A/g9eGLfOfJDKWjmrpA6b4J+uz9Ulu6WrhT+9/ywbd8tO6 t7VsJlT14CpgvQLnvN+Zukjlbme5kkp7MGYqCGgz01Uk5zbZs9FhsOltGD4vkNtM/BO6 YV4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711726950; x=1712331750; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zkkkR4bJVGdAmtjki7FbOrCs0xkI3kc/cNsgAoiErHk=; b=YLSxrKVhHdymK2owZR57dKb6c8xkPsG0Km5wlUoSXBr3zuXxH7Cc8lAzeVCkkAiuiY 2QlACttmTJVFscC/3IwJsqW8upLixhzjwMISAX7r/Fsc+pp1P9n689qe8ruri7JyV0Il Yng2s03n+eigwxgJFl3uXi+7w/MdKBCePWNWV3aC1fTENBzHa/bmjagPfjgrJclZB+VV O4tBUcyWOk+lhugktFWIH/ghv0tocgzGYlq48G1EaOYcHV1a3915T7OYNSDzPWtEahQG x40ccmyP9hg0dZ0LP8T+mBRcUfnnqMZdyGkJ/rYaEDxUVZxT3eNB/w+v+14XXT/AD85y Ervw== X-Gm-Message-State: AOJu0YwgvAEkT69IrjHfL07ZSzZ+6asiK7hL/dvQMmKXDpz0hWgjKLSd xBg5S8k8SX8lD246uKqfwxUitBUIgWOBFYKJLyDEILgEgwZkNTUu41ZRf8TraLVG0pLvJCWOtTE B/VG3CNbv9A== X-Google-Smtp-Source: AGHT+IGjll7A2Nr2rtwocEmp671U6FBxfcSmDJeqrqPHRiHBVwV0W1kC+FFRW9y50kbv6OE4FdVlcTqvoMDSyw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a81:4e10:0:b0:611:5ca6:59c3 with SMTP id c16-20020a814e10000000b006115ca659c3mr553411ywb.6.1711726950558; Fri, 29 Mar 2024 08:42:30 -0700 (PDT) Date: Fri, 29 Mar 2024 15:42:19 +0000 In-Reply-To: <20240329154225.349288-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240329154225.349288-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240329154225.349288-3-edumazet@google.com> Subject: [PATCH v2 net-next 2/8] net: move dev_xmit_recursion() helpers to net/core/dev.h From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Move dev_xmit_recursion() and friends to net/core/dev.h They are only used from net/core/dev.c and net/core/filter.c. Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 17 ----------------- net/core/dev.h | 17 +++++++++++++++++ 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index cb37817d6382c29117afd8ce54db6dba94f8c930..70775021cc269e0983f538619115237b0067d408 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3270,23 +3270,6 @@ static inline int dev_recursion_level(void) return this_cpu_read(softnet_data.xmit.recursion); } -#define XMIT_RECURSION_LIMIT 8 -static inline bool dev_xmit_recursion(void) -{ - return unlikely(__this_cpu_read(softnet_data.xmit.recursion) > - XMIT_RECURSION_LIMIT); -} - -static inline void dev_xmit_recursion_inc(void) -{ - __this_cpu_inc(softnet_data.xmit.recursion); -} - -static inline void dev_xmit_recursion_dec(void) -{ - __this_cpu_dec(softnet_data.xmit.recursion); -} - void __netif_schedule(struct Qdisc *q); void netif_schedule_queue(struct netdev_queue *txq); diff --git a/net/core/dev.h b/net/core/dev.h index 9d0f8b4587f81f4c12487f1783d8ba5cc49fc1d6..8572d2c8dc4adce75c98868c888363e6a32e0f52 100644 --- a/net/core/dev.h +++ b/net/core/dev.h @@ -150,4 +150,21 @@ static inline void xdp_do_check_flushed(struct napi_struct *napi) { } struct napi_struct *napi_by_id(unsigned int napi_id); void kick_defer_list_purge(struct softnet_data *sd, unsigned int cpu); +#define XMIT_RECURSION_LIMIT 8 +static inline bool dev_xmit_recursion(void) +{ + return unlikely(__this_cpu_read(softnet_data.xmit.recursion) > + XMIT_RECURSION_LIMIT); +} + +static inline void dev_xmit_recursion_inc(void) +{ + __this_cpu_inc(softnet_data.xmit.recursion); +} + +static inline void dev_xmit_recursion_dec(void) +{ + __this_cpu_dec(softnet_data.xmit.recursion); +} + #endif From patchwork Fri Mar 29 15:42:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13610732 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D89081C0DCF for ; Fri, 29 Mar 2024 15:42:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726954; cv=none; b=dIZxBaFkYS5hDTIEJD8xvhsZy+Ma7t7afF7+bbtME4WnWLg7x1UgSjhsOK6EP5hKSUKb8PY5LY99hcvkuaX9w7z4mSok6sOLiApoEAH3W53j8gBXtUy1ahc8IW438qCSHykTbMsEZRVCpLpiI8cJM6UR4x9GdUNX1p+xZpxXUP4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726954; c=relaxed/simple; bh=kA41VnKBM5QkuJWHFLJdysv5A55/K/SIu0cl2mwOaHM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=C0w8Cs3rKvUQWCzMTgWxYk05nlpsDv7GLcVwXU1Hd8CJXKfwu2CveGWP8dOCLDfUg7gQPKM2Ourhm/mSpfNyIxv2ORG+ONFtvvXyWUmL2Z9V8fc7XdUX2R70HfdqHayjWlwP6qrwwJOJak/ULbtDh7SQnf0+mQ+AUCW+oesc6b8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ARmsfNVv; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ARmsfNVv" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60cbba6fa0bso38253857b3.3 for ; Fri, 29 Mar 2024 08:42:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711726952; x=1712331752; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZLMqQ3fdvniunS4oceCdUDaRaaFXRuKKKp9GbMATOUk=; b=ARmsfNVvG9fnf08FiKmiMH9CCbNj5PcR+JORvkPkWMoqrOsGv3A6cj6cIOdoF+oxDG 8KpuYWTOEg8LbG9iEObL+ZzwR2tSGYF5ZFKfKsEAxg30T16CiZOCa5FbMSRoXV10yTaE AfzqrLy0unhk6l5k5bkcbvJwAN/Efs1TYA+i7TqpZk1eYFNTSzy2BFHn6YC2JJhWNGEW Hy5BEesxZ6pki/bW1xoPHYlKsdMHtmo1erxsSjBW8DXilWk/cM9BoFHO5O9STOQr9jtH VBUaQzpO755TJ03j9acSOx/+BlznGUfmKFDBWmtNIZmxCMoLj+kKVULtPSXLSSMWjqS7 7Juw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711726952; x=1712331752; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZLMqQ3fdvniunS4oceCdUDaRaaFXRuKKKp9GbMATOUk=; b=qRDQdrGh04tY1+RJNwNURx43H7Fgb3B4V+Av7tq4Pl366ziOdspaGWmaNIsdbIddX3 poQhFnvDqfVCUz8WwAo7DCe6WNuYJIlit20H2AQMJtVRkPQxd/xaygWQFudS/hxCmCiB +JgDnExC2QvENy0kPn9hc24buIzRohmEXn4to8ithxFJqIsz3URfjP+8/w7c/Qu1mvhd YH4ORURSysbNaKd6tc/URIQ4/GW9RQadBi71BxB48kqCNhu06xmRmYbmbJCvsdZvY69/ UB729WPsBHhPo9dZB4CxOU1L24yZYAmgNa8k1XXUBAM9AKpqKfImyc5UDRercN2D2PQz tnpg== X-Gm-Message-State: AOJu0Yy4CEpIecEXHTzPv3QvyY1Wn0d3V3ZfzXyz9jK6v8DCmz5FULnD bhjBTXgHkOvrSf81FRoBvPy2AhU1nZMXydacHfmlc6/SOAXxUwdxsjao8R8hWeKnNur6uKRAsAO 3LjGdzStGrA== X-Google-Smtp-Source: AGHT+IHjtDW0RzTV+E7nqcUcfFxFk0xs8rqTJEbto8bdPcnmWICXSOPYST3HBnsOX/qDFx0Z1Pjc5Orwj3oW/g== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1002:b0:dc9:5ef8:2b2d with SMTP id w2-20020a056902100200b00dc95ef82b2dmr742106ybt.4.1711726951947; Fri, 29 Mar 2024 08:42:31 -0700 (PDT) Date: Fri, 29 Mar 2024 15:42:20 +0000 In-Reply-To: <20240329154225.349288-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240329154225.349288-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240329154225.349288-4-edumazet@google.com> Subject: [PATCH v2 net-next 3/8] net: enqueue_to_backlog() change vs not running device From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org If the device attached to the packet given to enqueue_to_backlog() is not running, we drop the packet. But we accidentally increase sd->dropped, giving false signals to admins: sd->dropped should be reserved to cpu backlog pressure, not to temporary glitches at device dismantles. While we are at it, perform the netif_running() test before we get the rps lock, and use REASON_DEV_READY drop reason instead of NOT_SPECIFIED. Signed-off-by: Eric Dumazet --- net/core/dev.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index c136e80dea6182faac153f0cc4149bf8698b6676..4ad7836365e68f700b26dba2c50515a8c18329cf 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4801,12 +4801,13 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, unsigned long flags; unsigned int qlen; - reason = SKB_DROP_REASON_NOT_SPECIFIED; + reason = SKB_DROP_REASON_DEV_READY; + if (!netif_running(skb->dev)) + goto bad_dev; + sd = &per_cpu(softnet_data, cpu); backlog_lock_irq_save(sd, &flags); - if (!netif_running(skb->dev)) - goto drop; qlen = skb_queue_len(&sd->input_pkt_queue); if (qlen <= READ_ONCE(net_hotdata.max_backlog) && !skb_flow_limit(skb, qlen)) { @@ -4827,10 +4828,10 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, } reason = SKB_DROP_REASON_CPU_BACKLOG; -drop: sd->dropped++; backlog_unlock_irq_restore(sd, &flags); +bad_dev: dev_core_stats_rx_dropped_inc(skb->dev); kfree_skb_reason(skb, reason); return NET_RX_DROP; From patchwork Fri Mar 29 15:42:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13610733 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 323BE3BB55 for ; Fri, 29 Mar 2024 15:42:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726955; cv=none; b=m7qWdtm54cpo27kT1CgBNIYfuXQVFfLAv1kWvdX06AAVYdZWE2Pv42olANLx7UWcfsaZFJvGrFcWlJ9uIKX7a65tQSXay3zozfe0dWYBOtLo/m91443Psm3E/Nj3gnj05WnKrYoHsufvc+qLYr8Kt6mx5wXK8sK4zpa/fTu791Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726955; c=relaxed/simple; bh=49NfDQcjQcBykvTuhA4xf8SLni3I9f0ooXUsLaE12ms=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=b/BN6OZvT8plgSvpqi1MogVXRegy+1MRBg3LkHWZTUsEHeMBUeMiru3rDRU3WMqoQ/EdCDsT61pNVbF7iW4wYgpDMY4tm4jk9X4iHlAffAwy1Ec+ywHbaFtqLm6W9GyyMvVBzJ+MdPchdMPSMU6dCdP/XVWhkKhu2h9sjxz+tEw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=nB9MEdVV; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="nB9MEdVV" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dd8e82dd47eso3154828276.2 for ; Fri, 29 Mar 2024 08:42:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711726953; x=1712331753; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5MaqyQyrTnGAJTp/NlfiIi6i7imHnUe1aPUmKIFthC8=; b=nB9MEdVVbbcjy0pENeZsPl4SnI0Z6JCJ541F9j9rjsffWfXw4uALD+hneYFle+Ww5R KP8WyUuHypv2p/MrQCNuxrJ2IBQjwMChLZ6sRf4H1uX1HaS6m35ldMlEDer+C9qpAHKW rX0+BHywCFvLNY8ncSk7m2MlvLsehhS8WHymXIRGms89JIi2Qy0EBC2x0tl90bnoMGF/ sg3DWZHBJaFZyAdt/6bblnmu3oMCFoW2OiRBu1v2DsrcIYC83WhMjOyOf4tpYSleLghZ 8z7z9Fv+NCVERqKkBPFaUdol5nt1H8jmvAHdHwD74W4C/8C5s8+ile40id8kRMyoo4Ns rRWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711726953; x=1712331753; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5MaqyQyrTnGAJTp/NlfiIi6i7imHnUe1aPUmKIFthC8=; b=MFDc05Vp1H4mz/qGWyuN5EsZnBK5oZuEiPOY1DAOOJSXZ0tzBPgZRLg571ehvvYwnP mSNm6iSByWEh/U1CCFXsCF0drE8NArfZ/+HEFPkI57yc1MCuUEr7KovhwofpR+PrBgoF YGg2qyKuWg1yXME+9un6hV0cyVYvVO/K/tl7osRPYBqyg5svX4550NOp+WtFCOlfmZnN rvJyIorm0fUkb1V1uvwWDKUQnhh4nBaIM6/Qr+EAe+TItDqZcvu1PYh/pjxSfKPJj9k5 zevR0rdH+pfxK0OcUQwnbey6Dps0sGa69+mxRu4gQYgmYsoSZBlU2UObIp/HIIpWJJ8r 8Wlw== X-Gm-Message-State: AOJu0YzE6ZYNe8BaTLFPMjuNjJ2x9LI940AK+V7FawkabLwrxwdAZE4h jlBg/l8JQ//IpLh9Szo7WTRmV1EdfrNp9sql53zWFi6rhxWaWWIf609FaSlG5iumIgYvmGW7NaN dcj4qoJ+0tQ== X-Google-Smtp-Source: AGHT+IHjfLd3cRPZ46/iquC6wR6a1FhsZ/fwOTdYycEMGGC70NvC8Im2Z2IBvK+rOTFtFqAcknIFE6nalpuYpA== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1b09:b0:dc6:c94e:fb85 with SMTP id eh9-20020a0569021b0900b00dc6c94efb85mr163968ybb.2.1711726953123; Fri, 29 Mar 2024 08:42:33 -0700 (PDT) Date: Fri, 29 Mar 2024 15:42:21 +0000 In-Reply-To: <20240329154225.349288-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240329154225.349288-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240329154225.349288-5-edumazet@google.com> Subject: [PATCH v2 net-next 4/8] net: make softnet_data.dropped an atomic_t From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org If under extreme cpu backlog pressure enqueue_to_backlog() has to drop a packet, it could do this without dirtying a cache line and potentially slowing down the target cpu. Move sd->dropped into a separate cache line, and make it atomic. In non pressure mode, this field is not touched, no need to consume valuable space in a hot cache line. Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 3 ++- net/core/dev.c | 13 +++++++++---- net/core/net-procfs.c | 3 ++- 3 files changed, 13 insertions(+), 6 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 70775021cc269e0983f538619115237b0067d408..1c31cd2691d32064613836141fbdeeebc831b21f 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3236,10 +3236,11 @@ struct softnet_data { unsigned int input_queue_tail; #endif unsigned int received_rps; - unsigned int dropped; struct sk_buff_head input_pkt_queue; struct napi_struct backlog; + atomic_t dropped ____cacheline_aligned_in_smp; + /* Another possibly contended cache line */ spinlock_t defer_lock ____cacheline_aligned_in_smp; int defer_count; diff --git a/net/core/dev.c b/net/core/dev.c index 4ad7836365e68f700b26dba2c50515a8c18329cf..02c98f115243202c409ee00c16e08fb0cf4d9ab9 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4800,17 +4800,22 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, struct softnet_data *sd; unsigned long flags; unsigned int qlen; + int max_backlog; reason = SKB_DROP_REASON_DEV_READY; if (!netif_running(skb->dev)) goto bad_dev; + reason = SKB_DROP_REASON_CPU_BACKLOG; sd = &per_cpu(softnet_data, cpu); + qlen = skb_queue_len_lockless(&sd->input_pkt_queue); + max_backlog = READ_ONCE(net_hotdata.max_backlog); + if (unlikely(qlen > max_backlog)) + goto cpu_backlog_drop; backlog_lock_irq_save(sd, &flags); qlen = skb_queue_len(&sd->input_pkt_queue); - if (qlen <= READ_ONCE(net_hotdata.max_backlog) && - !skb_flow_limit(skb, qlen)) { + if (qlen <= max_backlog && !skb_flow_limit(skb, qlen)) { if (qlen) { enqueue: __skb_queue_tail(&sd->input_pkt_queue, skb); @@ -4826,11 +4831,11 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, napi_schedule_rps(sd); goto enqueue; } - reason = SKB_DROP_REASON_CPU_BACKLOG; - sd->dropped++; backlog_unlock_irq_restore(sd, &flags); +cpu_backlog_drop: + atomic_inc(&sd->dropped); bad_dev: dev_core_stats_rx_dropped_inc(skb->dev); kfree_skb_reason(skb, reason); diff --git a/net/core/net-procfs.c b/net/core/net-procfs.c index a97eceb84e61ec347ad132ff0f22c8ce82f12e90..fa6d3969734a6ec154c3444d1b25ee93edfc5588 100644 --- a/net/core/net-procfs.c +++ b/net/core/net-procfs.c @@ -144,7 +144,8 @@ static int softnet_seq_show(struct seq_file *seq, void *v) seq_printf(seq, "%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x " "%08x %08x\n", - sd->processed, sd->dropped, sd->time_squeeze, 0, + sd->processed, atomic_read(&sd->dropped), + sd->time_squeeze, 0, 0, 0, 0, 0, /* was fastroute */ 0, /* was cpu_collision */ sd->received_rps, flow_limit_count, From patchwork Fri Mar 29 15:42:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13610734 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qk1-f202.google.com (mail-qk1-f202.google.com [209.85.222.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7B5A45BEA for ; Fri, 29 Mar 2024 15:42:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726957; cv=none; b=i5zBJS0s1r6y6Bzon+USDda02WyD6+OaIH7I9WeBQl3gPdwneVaiWQKk9CWABXY4vvrn4LBy0Xm0KnZGRUWhJ7Peq0kSK21tu0cbEjxVhMdGLgw+b9MrkmAzb62WLNVM4bAistO7X/nq/JLcfyuWxXlx6+e3rGHg3Lz1FxoBl7o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726957; c=relaxed/simple; bh=eOGR39DCoGOQg+oNMRIwC8g01mQpk7BViTnhKr2Lp7E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=fQcWL7nrg5Dej8rJKtR2dquAnCg/jCM1z73Y5kUr5+Ya19w7eiPzQ4Z0ivjWyDRXHN/OP1DZcSQ+1qjwUsgiYKkcXJmy5U3YiqGovHOd1EUIEUkgykxHTjJV6wAVHphgdM5oBdoqxZVcGqbaNZ9PSrIJOJWToHLcVFfjduaJPEw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=mGZcE/gi; arc=none smtp.client-ip=209.85.222.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mGZcE/gi" Received: by mail-qk1-f202.google.com with SMTP id af79cd13be357-78a087e8b70so373415985a.1 for ; Fri, 29 Mar 2024 08:42:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711726955; x=1712331755; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sMXzAltYqOJOzOEzXcMQc99mG0oMwOhh00KlVSFtR2g=; b=mGZcE/gizSnX+v+bEChcHUbpo/owq0xb2jIid1PJIpQLOPvJSv1uRtOocWWYrTPZCc 2OoeW5LqAZTQcIYS3+Djrrfvya/Zw8sxz1MhSrVLx8JVzKeq1yMuGgCBn65+UanA7KPK opbf0WUXm1iOBhRAwy1ocGv+rkLS50DEP+//1paYlE5K/2Y746sJUWtRDMLqOkYOUiMr 1swY/u3jNJkoCbdlyu5qcgh4DFQuBuamOeEKz+QIu3mQs+4SzmwEwkt+0/iRScPrN0XV oYAhfeWyj3Au8GH+Ezq3RueeM2lKpubP1qxeVYGc1mQGPbg0yFujCWKZsiN0/pU03hOR /FzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711726955; x=1712331755; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sMXzAltYqOJOzOEzXcMQc99mG0oMwOhh00KlVSFtR2g=; b=eviOFdNdYnNZ/v47i+kOeRZK127kDyCxEdSMclsD/MlVsH2KccO2WCKpVdKPSWB0zq 0vNc5w1pU015MOq+zOcK6devz5H/4UZatXyvoMeSGYuLET1j8I388uXxpQ2cwQZYTApj jZ6n+ec/7YkkqYSQjsokUbtD0sXiinv2YhOMxgqrzyo+gvQ6apfN5GX0NieHHUUI5c2c IWYSvDX7gGv84sCp372b5U7mR7fR9N/Bvx6ZULh5Ey76OH71dQBQRCuWZ2NxkMpMSXRX Yw3UVeHyU7vezDUKdTBw9HQUZWENLPHOHxS+YoUVQlTJlkszZUfqstz5qTxIWci75CO4 EFyw== X-Gm-Message-State: AOJu0Yz6Qd7M39Ee+TYzJAKI3OXSKGHmaR3LbRYU5Cng5ZqCVgFTOPf5 7huk2jgthW0UJgPR0g6Umx0p/lndQQG9ipg/0muUlLyjGmO88Da1TCBoApHja6l9zpvoRvjmd4g G4/WYFzlehA== X-Google-Smtp-Source: AGHT+IFO6e1huPIUMg0iN+arYwtte3emedADA72AYRMmN3msuI8LKpVGzxMnX+8f2yNW7ypLQx0TX8rYEYlcwQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:620a:a9b:b0:78a:5d7c:b608 with SMTP id v27-20020a05620a0a9b00b0078a5d7cb608mr40897qkg.2.1711726954760; Fri, 29 Mar 2024 08:42:34 -0700 (PDT) Date: Fri, 29 Mar 2024 15:42:22 +0000 In-Reply-To: <20240329154225.349288-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240329154225.349288-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240329154225.349288-6-edumazet@google.com> Subject: [PATCH v2 net-next 5/8] net: enqueue_to_backlog() cleanup From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org We can remove a goto and a label by reversing a condition. Signed-off-by: Eric Dumazet --- net/core/dev.c | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index 02c98f115243202c409ee00c16e08fb0cf4d9ab9..0a8ccb0451c30a39f8f8b45d26b7e5548b8bfba4 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4816,20 +4816,18 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, backlog_lock_irq_save(sd, &flags); qlen = skb_queue_len(&sd->input_pkt_queue); if (qlen <= max_backlog && !skb_flow_limit(skb, qlen)) { - if (qlen) { -enqueue: - __skb_queue_tail(&sd->input_pkt_queue, skb); - input_queue_tail_incr_save(sd, qtail); - backlog_unlock_irq_restore(sd, &flags); - return NET_RX_SUCCESS; + if (!qlen) { + /* Schedule NAPI for backlog device. We can use + * non atomic operation as we own the queue lock. + */ + if (!__test_and_set_bit(NAPI_STATE_SCHED, + &sd->backlog.state)) + napi_schedule_rps(sd); } - - /* Schedule NAPI for backlog device - * We can use non atomic operation since we own the queue lock - */ - if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) - napi_schedule_rps(sd); - goto enqueue; + __skb_queue_tail(&sd->input_pkt_queue, skb); + input_queue_tail_incr_save(sd, qtail); + backlog_unlock_irq_restore(sd, &flags); + return NET_RX_SUCCESS; } backlog_unlock_irq_restore(sd, &flags); From patchwork Fri Mar 29 15:42:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13610735 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51BA81E491 for ; Fri, 29 Mar 2024 15:42:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726958; cv=none; b=WocxkSHFDrJ2/PJLYSDAPW1kgZAnjEyXNi2uXJFlHYysx6v4MRclCs3C0wHBqRXBAOKkhC+c0x2eHqsKFAbWh3SxYmT/ZP42f4kLJAampS7Gir9A5OXaeinVWq0AU/lkusyAzdT5p04ENM/6Hy35Tb/J6Y/F9mknOAyX8R3NES8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726958; c=relaxed/simple; bh=4DWTyyt6+BzClySBH46Kl3XhnA2GwwDCvNYLsMyp53I=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Zjy9R4sROxJOet8RTWheIt8F2uK8QabqJjqNFm1vLcfj1+R92RYoFwxv9eD/KI8qzGfEJKdC6fj5kxBbgqVSmpF6/Gw/Tytz2rbB14MLJ4dizTbzascf3SxrSD44tvjf25H+r2U1HnSiVjX4QKrdlIEQ22tJRCb2VPJiHIyLNeU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=eLINtWdt; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="eLINtWdt" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-ddaf2f115f2so3010024276.3 for ; Fri, 29 Mar 2024 08:42:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711726956; x=1712331756; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4m8dG+cGHACwpf060OQmHT6LiXTZOTDb4U/m0fLPXTY=; b=eLINtWdtx4KaVKPszmtF4uoqJxQ/mEdmt0pv4gFJTCSUzDseg0xXtcNIrvrp2PvOLA Bkz7Ay+pSggmRLTpyYlTZUasVcmclNEiD7KrDCoUx/qmep/JUks+xn4GtrYJ9nKEIBI2 Ke+HBuKNJbsGmhlpTxLUC4wod5ZNYre6htiWJSRULP+jePV8o30dUH0vddCP1fWVptt8 /IGLosaZQl1tG8m2mmwMVMbrOhkIkzIxxeeFu6MQLTHjjbH3AfGlN33zRgQmZE0rboZT 5ftQ98GrKZIp7TJmKjsnKk+f3p+uOfpTQCQ8zvzWSDny48cnyjrGe84InxVlSNp4j09D bKxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711726956; x=1712331756; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4m8dG+cGHACwpf060OQmHT6LiXTZOTDb4U/m0fLPXTY=; b=tBaKlAko8NKpZNF+0O8u3xnpoZZWM2ip/kid1kVmG1ZvAR1yAoTAYM/P9VRN0+ROxF rbq9bVAMpo6s6+va3IDg2DqKlFAI66c3hG02otlgGGAFzI0FkbO3W1dq7XmVor0amCcH ehq3CZ48vR2xQG4LLiY22MpjTJIl5NNXdIeNuSFMCpdzysvGJXKSePCe4Sirmt9dszWJ E+Ot4XG2GbXdP85SStvJUBojsQIKgLS7Lj8JCLBKOZy5f7fwUtV10DltP0RDiZ5H36Od QoLvwCB8td+Hh/5VBb996opcSb2pRbKf3WMN/JH+vko6sTg8RplNStR7rxQW4wsrwvwG 2dpw== X-Gm-Message-State: AOJu0YyZCMMc7ylTlSlMyOOvjI/UI+hld/m9M56IVEn6Okcn/MNWAE6H KFfCs+4Q9kWCCWyYCTf158UZBmgpwPRSwne+UN/sDzK7qvUxHanu80U9AASsfhvkE5t3dICHxPO Zi5RTReoiuQ== X-Google-Smtp-Source: AGHT+IFGijK8Z2IE14RgDanAXGPGs2NB31ZYcfErnq2zkEJ0H4XCfRiDRrgyXSx3h1z7QFkX+Y3h35O8i0Fspw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:2502:b0:dcc:79ab:e522 with SMTP id dt2-20020a056902250200b00dcc79abe522mr192060ybb.11.1711726956278; Fri, 29 Mar 2024 08:42:36 -0700 (PDT) Date: Fri, 29 Mar 2024 15:42:23 +0000 In-Reply-To: <20240329154225.349288-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240329154225.349288-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240329154225.349288-7-edumazet@google.com> Subject: [PATCH v2 net-next 6/8] net: rps: change input_queue_tail_incr_save() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org input_queue_tail_incr_save() is incrementing the sd queue_tail and save it in the flow last_qtail. Two issues here : - no lock protects the write on last_qtail, we should use appropriate annotations. - We can perform this write after releasing the per-cpu backlog lock, to decrease this lock hold duration (move away the cache line miss) Also move input_queue_head_incr() and rps helpers to include/net/rps.h, while adding rps_ prefix to better reflect their role. v2: Fixed a build issue (Jakub and kernel build bots) Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 15 --------------- include/net/rps.h | 23 +++++++++++++++++++++++ net/core/dev.c | 20 ++++++++++++-------- 3 files changed, 35 insertions(+), 23 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 1c31cd2691d32064613836141fbdeeebc831b21f..14f19cc2616452d7e6afbbaa52f8ad3e61a419e9 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3249,21 +3249,6 @@ struct softnet_data { call_single_data_t defer_csd; }; -static inline void input_queue_head_incr(struct softnet_data *sd) -{ -#ifdef CONFIG_RPS - sd->input_queue_head++; -#endif -} - -static inline void input_queue_tail_incr_save(struct softnet_data *sd, - unsigned int *qtail) -{ -#ifdef CONFIG_RPS - *qtail = ++sd->input_queue_tail; -#endif -} - DECLARE_PER_CPU_ALIGNED(struct softnet_data, softnet_data); static inline int dev_recursion_level(void) diff --git a/include/net/rps.h b/include/net/rps.h index 7660243e905b92651a41292e04caf72c5f12f26e..10ca25731c1ef766715fe7ee415ad0b71ec643a8 100644 --- a/include/net/rps.h +++ b/include/net/rps.h @@ -122,4 +122,27 @@ static inline void sock_rps_record_flow(const struct sock *sk) #endif } +static inline u32 rps_input_queue_tail_incr(struct softnet_data *sd) +{ +#ifdef CONFIG_RPS + return ++sd->input_queue_tail; +#else + return 0; +#endif +} + +static inline void rps_input_queue_tail_save(u32 *dest, u32 tail) +{ +#ifdef CONFIG_RPS + WRITE_ONCE(*dest, tail); +#endif +} + +static inline void rps_input_queue_head_incr(struct softnet_data *sd) +{ +#ifdef CONFIG_RPS + sd->input_queue_head++; +#endif +} + #endif /* _NET_RPS_H */ diff --git a/net/core/dev.c b/net/core/dev.c index 0a8ccb0451c30a39f8f8b45d26b7e5548b8bfba4..79073bbc9a644049cacf8433310f4641745049e9 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4611,7 +4611,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, if (unlikely(tcpu != next_cpu) && (tcpu >= nr_cpu_ids || !cpu_online(tcpu) || ((int)(per_cpu(softnet_data, tcpu).input_queue_head - - rflow->last_qtail)) >= 0)) { + READ_ONCE(rflow->last_qtail))) >= 0)) { tcpu = next_cpu; rflow = set_rps_cpu(dev, skb, rflow, next_cpu); } @@ -4666,7 +4666,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index, cpu = READ_ONCE(rflow->cpu); if (rflow->filter == filter_id && cpu < nr_cpu_ids && ((int)(per_cpu(softnet_data, cpu).input_queue_head - - rflow->last_qtail) < + READ_ONCE(rflow->last_qtail)) < (int)(10 * flow_table->mask))) expire = false; } @@ -4801,6 +4801,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, unsigned long flags; unsigned int qlen; int max_backlog; + u32 tail; reason = SKB_DROP_REASON_DEV_READY; if (!netif_running(skb->dev)) @@ -4825,8 +4826,11 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, napi_schedule_rps(sd); } __skb_queue_tail(&sd->input_pkt_queue, skb); - input_queue_tail_incr_save(sd, qtail); + tail = rps_input_queue_tail_incr(sd); backlog_unlock_irq_restore(sd, &flags); + + /* save the tail outside of the critical section */ + rps_input_queue_tail_save(qtail, tail); return NET_RX_SUCCESS; } @@ -5904,7 +5908,7 @@ static void flush_backlog(struct work_struct *work) if (skb->dev->reg_state == NETREG_UNREGISTERING) { __skb_unlink(skb, &sd->input_pkt_queue); dev_kfree_skb_irq(skb); - input_queue_head_incr(sd); + rps_input_queue_head_incr(sd); } } backlog_unlock_irq_enable(sd); @@ -5913,7 +5917,7 @@ static void flush_backlog(struct work_struct *work) if (skb->dev->reg_state == NETREG_UNREGISTERING) { __skb_unlink(skb, &sd->process_queue); kfree_skb(skb); - input_queue_head_incr(sd); + rps_input_queue_head_incr(sd); } } local_bh_enable(); @@ -6041,7 +6045,7 @@ static int process_backlog(struct napi_struct *napi, int quota) rcu_read_lock(); __netif_receive_skb(skb); rcu_read_unlock(); - input_queue_head_incr(sd); + rps_input_queue_head_incr(sd); if (++work >= quota) return work; @@ -11455,11 +11459,11 @@ static int dev_cpu_dead(unsigned int oldcpu) /* Process offline CPU's input_pkt_queue */ while ((skb = __skb_dequeue(&oldsd->process_queue))) { netif_rx(skb); - input_queue_head_incr(oldsd); + rps_input_queue_head_incr(oldsd); } while ((skb = skb_dequeue(&oldsd->input_pkt_queue))) { netif_rx(skb); - input_queue_head_incr(oldsd); + rps_input_queue_head_incr(oldsd); } return 0; From patchwork Fri Mar 29 15:42:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13610736 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8BDB45BEA for ; Fri, 29 Mar 2024 15:42:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726960; cv=none; b=UStsotenNhWNFrQ2L4xQa9T7jOmbQxEfegGEsnNuajGwOlmTyTPVsstBcyBkr2fp9PQXP8pAjQ7wT199EGQLJxs1iYggt0kYB+2ICwnkbNhhG7q3yOF8ZnVhjEZ7Rhi7CXJfaec+kV4VzKl/1RkHXYv1qsKrfM0bofDDrrofnsA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726960; c=relaxed/simple; bh=Rh2Np/oxrvww/pqriWwAwgIkuhL/a6F/FV9Z7mjLlW0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UECdEj7SeGzJgu5KvWo5QP/PcI8n0RjXx7BKSSlBYO3Ts0RAT/2FEONDGSJuE6OHrukJIABpXfW+J7OkOwepBWkrPlAzAun++OpciE1+GjeSySOuNQBdYlegcYySJWs0Nuhx6MDE59dFpCzaWENQPt7I5TyIAlhQlAGUnZqPR5w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4mwpcM9E; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4mwpcM9E" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60f9d800a29so33197087b3.0 for ; Fri, 29 Mar 2024 08:42:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711726958; x=1712331758; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0G6bhQ6kaT728Eab6u6g1DkBmh7dQX51z7FtoBdHYBs=; b=4mwpcM9EXUm8iANwDNHHGMmYeqFEvzBZ+m55HHRmE/T4Mltcrfoa+2/G1g0caIpICs j+/5XIXewBoS1hBDcuDRHGWBAsCJ/qTw05PKLWE3pdZc183zui3x3rzYZt7xFx88xTzv LlmVwi65ikmec5TEAhU6u4L4l3jemCj0AfdJGAYMFyKJ0e5IYBUY+QY0+SOrAN3QK/2X OKRgN0N3ulEOtUAk85SFIxUg/CxZw1+F+zAPggBcmyMQMkocNpsmZ/sGlJaQeco4jZOq 4XStz/frj2JBuAqAfdEbARyECKwckNSOq5Vxsc4ojkQjwWkOtrcezk07xtCHekzz+ybJ EeWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711726958; x=1712331758; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0G6bhQ6kaT728Eab6u6g1DkBmh7dQX51z7FtoBdHYBs=; b=JbC7cYq3lbqSLqwn7Y9qRO58z+iIGnoT2lTyilhiM3gImfH54RUpgsOTU6rsWEsFRK OiPSFc2CBe4BTEifdZeUXQP3SMUTdsUqVeRBDvePqv4I8aKYahVOut3Bvc/aNWkVskp7 HwNFtfSMv0u5a4V9AL8poDiaPYnbC0NkSCv5v86Oydt17tUPjvj0Kpm6/6r7b9Kv9aMi zV87ptncall4ZDqE7j75bggEFBs3r4urQt5ZQvjXYC0QwswTpJ3ZYUkqqZY0jLETStj5 CsZED7JuPNYJrXPat+5HcgT4aic+sNphHYEzldDVSj0cC2Awh/Q6lH3qRDqa96yptP4l ykfw== X-Gm-Message-State: AOJu0Yw1+X9nAtIDAKvYTHZit6yK+dQAWQvRUSmEg87Ht+LOO+cfS+3N fhCRpqW3VAQMk7+2bGilx4zlOI1ik8V5RmR3Uw1W469GXnPZ2ihJ2iBs3KZDVU6CSWyo1e9h/pK Vi2aFxUrg7A== X-Google-Smtp-Source: AGHT+IF+5zEeqKi8VzsZSSLseOPFpjHOXzblatzM+elVpqOvGbjx1j+F2b5CWWFXjAFL37k7lqZfoh8kVH+p3Q== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:260f:b0:dbe:387d:a8ef with SMTP id dw15-20020a056902260f00b00dbe387da8efmr164191ybb.1.1711726957962; Fri, 29 Mar 2024 08:42:37 -0700 (PDT) Date: Fri, 29 Mar 2024 15:42:24 +0000 In-Reply-To: <20240329154225.349288-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240329154225.349288-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240329154225.349288-8-edumazet@google.com> Subject: [PATCH v2 net-next 7/8] net: rps: add rps_input_queue_head_add() helper From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org process_backlog() can batch increments of sd->input_queue_head, saving some memory bandwidth. Also add READ_ONCE()/WRITE_ONCE() annotations around sd->input_queue_head accesses. Signed-off-by: Eric Dumazet --- include/net/rps.h | 9 +++++++-- net/core/dev.c | 13 ++++++++----- 2 files changed, 15 insertions(+), 7 deletions(-) diff --git a/include/net/rps.h b/include/net/rps.h index 10ca25731c1ef766715fe7ee415ad0b71ec643a8..a93401d23d66e45210acc73f0326087813b69d59 100644 --- a/include/net/rps.h +++ b/include/net/rps.h @@ -138,11 +138,16 @@ static inline void rps_input_queue_tail_save(u32 *dest, u32 tail) #endif } -static inline void rps_input_queue_head_incr(struct softnet_data *sd) +static inline void rps_input_queue_head_add(struct softnet_data *sd, int val) { #ifdef CONFIG_RPS - sd->input_queue_head++; + WRITE_ONCE(sd->input_queue_head, sd->input_queue_head + val); #endif } +static inline void rps_input_queue_head_incr(struct softnet_data *sd) +{ + rps_input_queue_head_add(sd, 1); +} + #endif /* _NET_RPS_H */ diff --git a/net/core/dev.c b/net/core/dev.c index 79073bbc9a644049cacf8433310f4641745049e9..818699dea9d7040ee74532ccdebf01c4fd6887cc 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4528,7 +4528,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb, out: #endif rflow->last_qtail = - per_cpu(softnet_data, next_cpu).input_queue_head; + READ_ONCE(per_cpu(softnet_data, next_cpu).input_queue_head); } rflow->cpu = next_cpu; @@ -4610,7 +4610,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, */ if (unlikely(tcpu != next_cpu) && (tcpu >= nr_cpu_ids || !cpu_online(tcpu) || - ((int)(per_cpu(softnet_data, tcpu).input_queue_head - + ((int)(READ_ONCE(per_cpu(softnet_data, tcpu).input_queue_head) - READ_ONCE(rflow->last_qtail))) >= 0)) { tcpu = next_cpu; rflow = set_rps_cpu(dev, skb, rflow, next_cpu); @@ -4665,7 +4665,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index, rflow = &flow_table->flows[flow_id]; cpu = READ_ONCE(rflow->cpu); if (rflow->filter == filter_id && cpu < nr_cpu_ids && - ((int)(per_cpu(softnet_data, cpu).input_queue_head - + ((int)(READ_ONCE(per_cpu(softnet_data, cpu).input_queue_head) - READ_ONCE(rflow->last_qtail)) < (int)(10 * flow_table->mask))) expire = false; @@ -6045,9 +6045,10 @@ static int process_backlog(struct napi_struct *napi, int quota) rcu_read_lock(); __netif_receive_skb(skb); rcu_read_unlock(); - rps_input_queue_head_incr(sd); - if (++work >= quota) + if (++work >= quota) { + rps_input_queue_head_add(sd, work); return work; + } } @@ -6070,6 +6071,8 @@ static int process_backlog(struct napi_struct *napi, int quota) backlog_unlock_irq_enable(sd); } + if (work) + rps_input_queue_head_add(sd, work); return work; } From patchwork Fri Mar 29 15:42:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13610737 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A0881E491 for ; Fri, 29 Mar 2024 15:42:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726961; cv=none; b=Mim1/bcpTtJ8n12fukmSjStyekti855BkG8THeqpUMYFjUX8dhX2LS92A8sor5+dKZnmnsIvn4/dg8458LB6Ii54gnBzO2aIPBBpQ0SVOvDB5rfMSeZpMA1iJV6r9gQ0H1Tqcs6ft7lFx46KJhGvKoAXjojrtlzXmWzSuGYyWcc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711726961; c=relaxed/simple; bh=cDQ06vC2AO2hdTXvL2nVcFw/pVw+7ML0rIty1NNvoeY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=M/9Kk9S2a6yo7W2B8pzJeXMSSExuarmL9S0mpXq2NlfWCX0aF84EnKHEMh13EuxTW+Rb07lgGxCdzsahrPrrh7yo6L1U5sg2nuIJPrQf2qiwmmHnrUHEBngjQ08GwD2pAz2ZxZKfJy9ZeZ2Ujubf2RKQNQq/PMt5iuyqFX9EOe0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=I9elXpTd; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="I9elXpTd" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60a0b18e52dso25311597b3.1 for ; Fri, 29 Mar 2024 08:42:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711726959; x=1712331759; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=P4Rf3n0ThGg2QZAaciZ/MGFx+ZUjVEIBegGBLhJw3P8=; b=I9elXpTdsdNewa+StcwKecW9CR/Tyvsah4F6oVIYtApcj0BxAoESpCaRQIfsHiwZ/f Xq3FmzZOImpR/q+rHxBJjxcSZjZpJKEJwZhlwjKuRgL6eSSAbEJG6SxPD6mBdATkg9dn xYdKoG7S6ONXNo+ILAgaYpksaWpRaQYL1YXIdmrqwZAV9wH+ptXX8K/hNiXJycXRE5X+ PvQcxzAd3hO9CIpdGKvM8XuJmUQH4VXlVsL7FjYr7rOCPeiY5QcDvahPPtfh5n/oytQL 9c1xSZd0DxVZbbHFfoPZQUrmFHGmjoS7V12qVl3wxvoVst5PpkZC5+/DD6sBvgSZLYK4 fiqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711726959; x=1712331759; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=P4Rf3n0ThGg2QZAaciZ/MGFx+ZUjVEIBegGBLhJw3P8=; b=vyKCL6orvjcnQL/sXRRZoC8/eYkE2Q30LhLpI/ReHpSVLt0y0V0CGrWScU+SxQaihh FnaVmbdFw8cmNFFCqZFkLdUjgYzkaF0MYL+z/Y8YUiKE8qvYrgS5l8QldIVpapaLM/1S lzuuPQ3p/MJcyAcVXauwQ53yL8vAvSi1jypVXa5enQ97ZtX+deR5c2KYAWNMFlfGHBsy TRtWi5wTG6Xk3ooQUE3XcTPWhvtocnSDXD1cf/Iv3hT+ojHwOHRB5DYo8xBMALFs/FdF tY67lUSNDBqKe7pucdELooks2okT147pJ48sQPhPhuhHWXMGHUYhxVSSQ2OJh2IO3pLM fusw== X-Gm-Message-State: AOJu0YwZqWHAdruQfJexWetN5uMj+k8j6IyZvmNYSv7ZY2RdgCaFqkCv vWiDzVEChFEhqu60vG/Fpi+p1/sgxQ55uQMSV/AADdoDfs6Kr7p6jSjQV7UO9IMLA7K355qZNJk CALuBk6E4WQ== X-Google-Smtp-Source: AGHT+IEqi7N+Kwc3DkeWAKaR1NwSZpc4syao/BJMYMJnkRVoLDXhCPAY9rGWv1wmcLmuOFFU4IYnVITfEl8Hqg== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a0d:ccd3:0:b0:614:6883:43bd with SMTP id o202-20020a0dccd3000000b00614688343bdmr36502ywd.1.1711726959523; Fri, 29 Mar 2024 08:42:39 -0700 (PDT) Date: Fri, 29 Mar 2024 15:42:25 +0000 In-Reply-To: <20240329154225.349288-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240329154225.349288-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240329154225.349288-9-edumazet@google.com> Subject: [PATCH v2 net-next 8/8] net: rps: move received_rps field to a better location From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Commit 14d898f3c1b3 ("dev: Move received_rps counter next to RPS members in softnet data") was unfortunate: received_rps is dirtied by a cpu and never read by other cpus in fast path. Its presence in the hot RPS cache line (shared by many cpus) is hurting RPS/RFS performance. Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 14f19cc2616452d7e6afbbaa52f8ad3e61a419e9..274d8db48b4858c70b43ea4628544e924ba6a263 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3203,6 +3203,7 @@ struct softnet_data { struct softnet_data *rps_ipi_list; #endif + unsigned int received_rps; bool in_net_rx_action; bool in_napi_threaded_poll; @@ -3235,7 +3236,6 @@ struct softnet_data { unsigned int cpu; unsigned int input_queue_tail; #endif - unsigned int received_rps; struct sk_buff_head input_pkt_queue; struct napi_struct backlog;