From patchwork Thu Jan 23 23:12:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samiullah Khawaja X-Patchwork-Id: 13948724 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C6D71CDFD3 for ; Thu, 23 Jan 2025 23:12:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737673961; cv=none; b=RZvcUNTSthbIKMlPQisa3C43cUJwOaEL7KV6f+wr5Kq2lbDxobC87aLnfhNAgNDRui7gb4Ehzq5YC28MncIzVFjx1yPq2iJeatoD1Wd2bcbEOLDEVw2BkSZovezDEz3GvtW9r7BxNtGJWpnYL9fTlNE0PCsEiZ4UDiLz/TcrSto= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737673961; c=relaxed/simple; bh=U1CUqeVeq1RCiemc7JFaF3fNONtCk7ot7Y8fwT12dZ8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=GHnSBrdAgudpW6y3ygabfgvrL7k1OEwiJprUxDoedd+wBhthflJ5SvSxJ1RguoUdLzF7tphuvBB10rLjGI3W0CNirTQPoShTtX6jUkq4y2e3SQTLGGcu7RRFlsuuw5UVuFeTnzBjsvONdyl9OP8PozD1H8J4W0MnI2duhd77PrA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LCgnEYLn; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LCgnEYLn" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ef909597d9so4213848a91.3 for ; Thu, 23 Jan 2025 15:12:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737673959; x=1738278759; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mBxbfT81fwwX1zSvujFdf84ewj4LqCM86p7iXIivu70=; b=LCgnEYLn0647ymee9694Mj2/Mc839Hoi8HirxA8vm9zTJLmZafS6xfSwFYkmEzgYgN 58GlOAf1DhVeCTwgJdOXE05tK2LevhHzBKEy1GpQYP+hb4zQTPrP8f9P3mu9JKgpGKye ik56Kk6h765qK9Zv505Hmc12eGBE7xsvpi/Bx9MSRnAfti7k4FHtp6Ebv9cn6As3euND ndSYaaG/NxqhA6dXpV/OdXbQumU8MkKlRMelWkPrN0owLgeKs0gJrY4CRUZpglL9fUcO g+bRg2qIAWV8wR3Fl8t7L5D3NxNCIRwV+pR+BNuX1IOqdrM6Ef6fCZHjl6BwnfKLsqGm kr1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737673959; x=1738278759; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mBxbfT81fwwX1zSvujFdf84ewj4LqCM86p7iXIivu70=; b=Fya4A9xtnAgPSI+HBsfttXmtLKRsaGgz9I50mGBVJpLnnSeKQZXas/IL7ZubdvFL5F ygRwFWD8GxCLc70LofmauAKdJG4zS2HOqYW0qn7ws3uyev/5LLwPwesiq47slO07Xpv6 +buavaK9ruQH1BnXwpcaO5TYY1oiGouyFETUxBpsBivOLLHgcSVRuVihS8WsIY0+Iypv gjW0HPdBUHbvEBMrytqFNpewbJc5Ai2Ln85FXNfOzobSbeVamovM8x/OhKATKVsaqAd2 m/7qL2fQK1sES/NDqozrWXaDtLeIcyaeG5UfrnUEcf0cQL7lxLJG53CqworXsWuYHS9P IgTQ== X-Gm-Message-State: AOJu0YxVnrJEXXgjrISrNomJulN2a8dlYIJhFNUvHIUWz1oswGeOZAI+ jq8+DzPvUeioa4E2ifMgkR2uJR0En10nu6xZXW6ZLWnOpRQDMInbxTdcg/Fqi9K02CK5YRoWKYQ fvAdhgky08Q== X-Google-Smtp-Source: AGHT+IH/Aveic4cCN6YaWe9eD9AGkJG/5MKUt68DE/okhYijiIb1a+uTiAww16KrG+n8663022AS7D88DzYbBQ== X-Received: from pfbch15.prod.google.com ([2002:a05:6a00:288f:b0:725:ec78:5008]) (user=skhawaja job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:9991:b0:1e1:f281:8d36 with SMTP id adf61e73a8af0-1eb21486389mr42231833637.10.1737673958865; Thu, 23 Jan 2025 15:12:38 -0800 (PST) Date: Thu, 23 Jan 2025 23:12:33 +0000 In-Reply-To: <20250123231236.2657321-1-skhawaja@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250123231236.2657321-1-skhawaja@google.com> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250123231236.2657321-2-skhawaja@google.com> Subject: [PATCH net-next v2 1/4] Add support to set napi threaded for individual napi From: Samiullah Khawaja To: Jakub Kicinski , "David S . Miller " , Eric Dumazet , Paolo Abeni , almasrymina@google.com Cc: netdev@vger.kernel.org, skhawaja@google.com X-Patchwork-Delegate: kuba@kernel.org A net device has a threaded sysctl that can be used to enable threaded napi polling on all of the NAPI contexts under that device. Allow enabling threaded napi polling at individual napi level using netlink. Extend the netlink operation `napi-set` and allow setting the threaded attribute of a NAPI. This will enable the threaded polling on a napi context. Tested using following command in qemu/virtio-net: ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \ --do napi-set --json '{"id": 66, "threaded": 1}' Signed-off-by: Samiullah Khawaja --- Documentation/netlink/specs/netdev.yaml | 10 ++++++++ Documentation/networking/napi.rst | 13 ++++++++++- include/linux/netdevice.h | 10 ++++++++ include/uapi/linux/netdev.h | 1 + net/core/dev.c | 31 +++++++++++++++++++++++++ net/core/netdev-genl-gen.c | 5 ++-- net/core/netdev-genl.c | 9 +++++++ tools/include/uapi/linux/netdev.h | 1 + 8 files changed, 77 insertions(+), 3 deletions(-) diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlink/specs/netdev.yaml index cbb544bd6c84..785240d60df6 100644 --- a/Documentation/netlink/specs/netdev.yaml +++ b/Documentation/netlink/specs/netdev.yaml @@ -268,6 +268,14 @@ attribute-sets: doc: The timeout, in nanoseconds, of how long to suspend irq processing, if event polling finds events type: uint + - + name: threaded + doc: Whether the napi is configured to operate in threaded polling + mode. If this is set to `1` then the NAPI context operates + in threaded polling mode. + type: u32 + checks: + max: 1 - name: queue attributes: @@ -659,6 +667,7 @@ operations: - defer-hard-irqs - gro-flush-timeout - irq-suspend-timeout + - threaded dump: request: attributes: @@ -711,6 +720,7 @@ operations: - defer-hard-irqs - gro-flush-timeout - irq-suspend-timeout + - threaded kernel-family: headers: [ "linux/list.h"] diff --git a/Documentation/networking/napi.rst b/Documentation/networking/napi.rst index 6083210ab2a4..41926e7a3dd4 100644 --- a/Documentation/networking/napi.rst +++ b/Documentation/networking/napi.rst @@ -413,7 +413,18 @@ dependent). The NAPI instance IDs will be assigned in the opposite order than the process IDs of the kernel threads. Threaded NAPI is controlled by writing 0/1 to the ``threaded`` file in -netdev's sysfs directory. +netdev's sysfs directory. It can also be enabled for a specific napi using +netlink interface. + +For example, using the script: + +.. code-block:: bash + + $ kernel-source/tools/net/ynl/pyynl/cli.py \ + --spec Documentation/netlink/specs/netdev.yaml \ + --do napi-set \ + --json='{"id": 66, + "threaded": 1}' .. rubric:: Footnotes diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 8da4c61f97b9..6afba24b18d1 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -352,6 +352,7 @@ struct napi_config { u64 gro_flush_timeout; u64 irq_suspend_timeout; u32 defer_hard_irqs; + bool threaded; unsigned int napi_id; }; @@ -572,6 +573,15 @@ static inline bool napi_complete(struct napi_struct *n) int dev_set_threaded(struct net_device *dev, bool threaded); +/* + * napi_set_threaded - set napi threaded state + * @napi: NAPI context + * @threaded: whether this napi does threaded polling + * + * Return 0 on success and negative errno on failure. + */ +int napi_set_threaded(struct napi_struct *napi, bool threaded); + void napi_disable(struct napi_struct *n); void napi_disable_locked(struct napi_struct *n); diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h index e4be227d3ad6..829648b2ef65 100644 --- a/include/uapi/linux/netdev.h +++ b/include/uapi/linux/netdev.h @@ -125,6 +125,7 @@ enum { NETDEV_A_NAPI_DEFER_HARD_IRQS, NETDEV_A_NAPI_GRO_FLUSH_TIMEOUT, NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT, + NETDEV_A_NAPI_THREADED, __NETDEV_A_NAPI_MAX, NETDEV_A_NAPI_MAX = (__NETDEV_A_NAPI_MAX - 1) diff --git a/net/core/dev.c b/net/core/dev.c index afa2282f2604..3885f3095873 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6787,6 +6787,30 @@ static void init_gro_hash(struct napi_struct *napi) napi->gro_bitmask = 0; } +int napi_set_threaded(struct napi_struct *napi, bool threaded) +{ + if (napi->dev->threaded) + return -EINVAL; + + if (threaded) { + if (!napi->thread) { + int err = napi_kthread_create(napi); + + if (err) + return err; + } + } + + if (napi->config) + napi->config->threaded = threaded; + + /* Make sure kthread is created before THREADED bit is set. */ + smp_mb__before_atomic(); + assign_bit(NAPI_STATE_THREADED, &napi->state, threaded); + + return 0; +} + int dev_set_threaded(struct net_device *dev, bool threaded) { struct napi_struct *napi; @@ -6798,6 +6822,11 @@ int dev_set_threaded(struct net_device *dev, bool threaded) return 0; if (threaded) { + /* Check if threaded is set at napi level already */ + list_for_each_entry(napi, &dev->napi_list, dev_list) + if (test_bit(NAPI_STATE_THREADED, &napi->state)) + return -EINVAL; + list_for_each_entry(napi, &dev->napi_list, dev_list) { if (!napi->thread) { err = napi_kthread_create(napi); @@ -6880,6 +6909,8 @@ static void napi_restore_config(struct napi_struct *n) napi_hash_add(n); n->config->napi_id = n->napi_id; } + + napi_set_threaded(n, n->config->threaded); } static void napi_save_config(struct napi_struct *n) diff --git a/net/core/netdev-genl-gen.c b/net/core/netdev-genl-gen.c index 996ac6a449eb..a1f80e687f53 100644 --- a/net/core/netdev-genl-gen.c +++ b/net/core/netdev-genl-gen.c @@ -92,11 +92,12 @@ static const struct nla_policy netdev_bind_rx_nl_policy[NETDEV_A_DMABUF_FD + 1] }; /* NETDEV_CMD_NAPI_SET - do */ -static const struct nla_policy netdev_napi_set_nl_policy[NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT + 1] = { +static const struct nla_policy netdev_napi_set_nl_policy[NETDEV_A_NAPI_THREADED + 1] = { [NETDEV_A_NAPI_ID] = { .type = NLA_U32, }, [NETDEV_A_NAPI_DEFER_HARD_IRQS] = NLA_POLICY_FULL_RANGE(NLA_U32, &netdev_a_napi_defer_hard_irqs_range), [NETDEV_A_NAPI_GRO_FLUSH_TIMEOUT] = { .type = NLA_UINT, }, [NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT] = { .type = NLA_UINT, }, + [NETDEV_A_NAPI_THREADED] = NLA_POLICY_MAX(NLA_U32, 1), }; /* Ops table for netdev */ @@ -187,7 +188,7 @@ static const struct genl_split_ops netdev_nl_ops[] = { .cmd = NETDEV_CMD_NAPI_SET, .doit = netdev_nl_napi_set_doit, .policy = netdev_napi_set_nl_policy, - .maxattr = NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT, + .maxattr = NETDEV_A_NAPI_THREADED, .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, }, }; diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index 715f85c6b62e..208c3dd768ec 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -183,6 +183,9 @@ netdev_nl_napi_fill_one(struct sk_buff *rsp, struct napi_struct *napi, if (napi->irq >= 0 && nla_put_u32(rsp, NETDEV_A_NAPI_IRQ, napi->irq)) goto nla_put_failure; + if (nla_put_u32(rsp, NETDEV_A_NAPI_THREADED, !!napi->thread)) + goto nla_put_failure; + if (napi->thread) { pid = task_pid_nr(napi->thread); if (nla_put_u32(rsp, NETDEV_A_NAPI_PID, pid)) @@ -321,8 +324,14 @@ netdev_nl_napi_set_config(struct napi_struct *napi, struct genl_info *info) { u64 irq_suspend_timeout = 0; u64 gro_flush_timeout = 0; + u32 threaded = 0; u32 defer = 0; + if (info->attrs[NETDEV_A_NAPI_THREADED]) { + threaded = nla_get_u32(info->attrs[NETDEV_A_NAPI_THREADED]); + napi_set_threaded(napi, !!threaded); + } + if (info->attrs[NETDEV_A_NAPI_DEFER_HARD_IRQS]) { defer = nla_get_u32(info->attrs[NETDEV_A_NAPI_DEFER_HARD_IRQS]); napi_set_defer_hard_irqs(napi, defer); diff --git a/tools/include/uapi/linux/netdev.h b/tools/include/uapi/linux/netdev.h index e4be227d3ad6..829648b2ef65 100644 --- a/tools/include/uapi/linux/netdev.h +++ b/tools/include/uapi/linux/netdev.h @@ -125,6 +125,7 @@ enum { NETDEV_A_NAPI_DEFER_HARD_IRQS, NETDEV_A_NAPI_GRO_FLUSH_TIMEOUT, NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT, + NETDEV_A_NAPI_THREADED, __NETDEV_A_NAPI_MAX, NETDEV_A_NAPI_MAX = (__NETDEV_A_NAPI_MAX - 1) From patchwork Thu Jan 23 23:12:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samiullah Khawaja X-Patchwork-Id: 13948725 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11D8E1D54FE for ; Thu, 23 Jan 2025 23:12:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737673963; cv=none; b=lhyp1HpW50dyzK/+rY3wBA1SOlRQVm4loQsaESmBVF/5+FhE5Jo6NHPNK3wBrrYjOvMULvgVYSg1b0tUOGfBZDJMQaAiGMx6Y3H8RTnxNaNlkMvuCG8McMc+LKo7brR8Z5/37VzI12QOyvd8l8UbXVoB2qxEfHoOaSWKYQD+YsU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737673963; c=relaxed/simple; bh=/TSdSsi2B1Xs//iGhQKSjID2cGhxTDaksQdeJMMIQoc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=T38ChnZPzxhP8QIg2kRXf5J4i088X6G2sh0XfOG19ZeexVOHX2mYzi+psioXGmpV7bKeXODaJHmITxP9gVQvYkbv0xb6qAH7Wfm9oCmjlFXdQmhe9lzHGJNSBPH2VfYK+1F+LcR8zbWT2h6vMkDG3rUhVDNyY4F+24qzMCJrvc0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=RmGzEgfq; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="RmGzEgfq" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2efa74481fdso3038218a91.1 for ; Thu, 23 Jan 2025 15:12:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737673960; x=1738278760; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6cwA3MrIP+OTaNNbDIH6Ue9DgL0CpRgZj8Iu5e2x/FQ=; b=RmGzEgfqvlqv9fo0wgq8T2dTlNR3lAuAUC4J1cH40Afk9ks3cozxv0imQjOVA/HhEd 6o1J80LKs5+NNo6KrJf0W2CHSX2264Aw15oiAurWMxWR5e2jN38IGelvgnDIRnr4XjJo a9ftYgpd7grr8Sx5KC4hJvHuAOhw4U2NAGWq/tWmWqZjAK8o/nXpeUp+zYlX1NMI8lMD wA/LuNltMOqfYFTtlksQscoMgBeBCu+klEqc/UUiBAHUt1zVSGoaoZUnquI/xKfLx5pS kti1qd1WCwYmopGWoW1dST9KK7thERTq2HhWD6yWkJvJc5WWt6UWeDz1RsqOLPkKmZEB E7qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737673960; x=1738278760; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6cwA3MrIP+OTaNNbDIH6Ue9DgL0CpRgZj8Iu5e2x/FQ=; b=wRBXyF+jyxEL9bQNiqflLzZZaRdb1B1NOGbTh6i0rwy32M4q91peLNcmHObVaok/HB wdgYrzrMJZOK/hYekfZpQmlyGXfPG+6WOW79vxdawo5kcJb+doC2vyI12FWSCCnbs8ms LDe3Rj7W2wgbI4MvKgbeGHuXCNiye2uMeFeFkhBjIJZlqNYfVEglzqeg3etLOQ+i9ktM txeeut6TPcm5s1i/v+aUVDnG23B+Y3lWRKqzhMLSFZHyokZ5rA8mRK9F8xxjkohUL7YD g9iQeIGhsSLC50YVVLJMsfk86CtsG8OHr39dOU/It8xFgEcd/P+RSZ/9h5/Z1hbS+Vy/ gO1g== X-Gm-Message-State: AOJu0YytH+5jmU1N3jimA7hTfKdRm6xiKK85/O4HgTbBjj3f3IVs0Zn3 OUY77TwUlwQrc0IZZOo+lI6dBn/Pui6OQSOyuQ68VHM8HviGujnRf97n1KApXJkp9Ck22MRrdoT M1f8JkMXXEw== X-Google-Smtp-Source: AGHT+IFm3kWUgUekI0ubZJ5oGjx7UZidUw/AmU2JmROQnTE3BdT1PbtpMLHANmoBKZblGhHFsJ48t/V3+ty3nw== X-Received: from pfbea18.prod.google.com ([2002:a05:6a00:4c12:b0:725:ceac:b47e]) (user=skhawaja job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:92a0:b0:729:9f1:69bc with SMTP id d2e1a72fcca58-72dafa410aemr40305152b3a.12.1737673960287; Thu, 23 Jan 2025 15:12:40 -0800 (PST) Date: Thu, 23 Jan 2025 23:12:34 +0000 In-Reply-To: <20250123231236.2657321-1-skhawaja@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250123231236.2657321-1-skhawaja@google.com> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250123231236.2657321-3-skhawaja@google.com> Subject: [PATCH net-next v2 2/4] net: Create separate gro_flush helper function From: Samiullah Khawaja To: Jakub Kicinski , "David S . Miller " , Eric Dumazet , Paolo Abeni , almasrymina@google.com Cc: netdev@vger.kernel.org, skhawaja@google.com X-Patchwork-Delegate: kuba@kernel.org Move multiple copies of same code snippet doing `gro_flush` and `gro_normal_list` into a separate helper function. Signed-off-by: Samiullah Khawaja --- net/core/dev.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index 3885f3095873..484947ad5410 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6484,6 +6484,17 @@ static void skb_defer_free_flush(struct softnet_data *sd) } } +static void __napi_gro_flush_helper(struct napi_struct *napi) +{ + if (napi->gro_bitmask) { + /* flush too old packets + * If HZ < 1000, flush all packets. + */ + napi_gro_flush(napi, HZ >= 1000); + } + gro_normal_list(napi); +} + #if defined(CONFIG_NET_RX_BUSY_POLL) static void __busy_poll_stop(struct napi_struct *napi, bool skip_schedule) @@ -6494,14 +6505,8 @@ static void __busy_poll_stop(struct napi_struct *napi, bool skip_schedule) return; } - if (napi->gro_bitmask) { - /* flush too old packets - * If HZ < 1000, flush all packets. - */ - napi_gro_flush(napi, HZ >= 1000); - } + __napi_gro_flush_helper(napi); - gro_normal_list(napi); clear_bit(NAPI_STATE_SCHED, &napi->state); } @@ -7170,14 +7175,7 @@ static int __napi_poll(struct napi_struct *n, bool *repoll) return work; } - if (n->gro_bitmask) { - /* flush too old packets - * If HZ < 1000, flush all packets. - */ - napi_gro_flush(n, HZ >= 1000); - } - - gro_normal_list(n); + __napi_gro_flush_helper(n); /* Some drivers may have called napi_schedule * prior to exhausting their budget. From patchwork Thu Jan 23 23:12:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samiullah Khawaja X-Patchwork-Id: 13948727 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF1D31D5AD4 for ; Thu, 23 Jan 2025 23:12:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737673965; cv=none; b=dkldQ6PbHrxsFSbhuJdaz+L3k0UmCrY2NC0Wq9rDBjMZoe8fuEKalCf+zGS6McDnkJXNiXILmKO2akFV2YWwZYKQ5lmgxQXSFWRfk68bETb152J7IlwRc1csbjtCGOrIKBu7D/j/ETsG2FiGhtcZ4lDwdUW/8p1Z5eRsHmLSShU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737673965; c=relaxed/simple; bh=xZ8QfA/CCZxiyc/phROKsmKJUsYLt9gpee2s/QtL5ds=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=TVkxkOjyYkTLdcRuDbyEjeJJ6KqQmr0/lyuVfCdB5LFt/3IyXYTInY4SKX5yFQBb24jHkqGjIpcxC/Nqy2cgHK7OnOvS6+0QdQ0lHfb56717K9ozEPYuUdQdCK3NSVwfl3QQ6//3DjC00SG3rEg+w+y8dKUbftGmIzADJ39DOY0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sLVuiOKt; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sLVuiOKt" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f46b7851fcso4238933a91.1 for ; Thu, 23 Jan 2025 15:12:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737673962; x=1738278762; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Eh4nnX1aD2le7gIboLyXAYEu1bW8nY+X/o+Mz+vnzjo=; b=sLVuiOKt3oTHE1rz2ey7J8tQC7lw0+tIdD5NQUzbW2bhadcjwSlf7ChenAfKbhhsAS vLvY4jv/q5bHkytVYY0DUMpXLq+DpfA8Byq+O8uQVrAcuDlauRVH+XVYTQbOPUBUzfJo daNNyqVpgJZwzIZbbii4v0ZzokHX4HxLyiDyUhzVHETFhiVinUwRL7wz28WAYBGWVm90 v/jxqCuHq2xe6MZCW7GWvj8/ypcfBeJ4b5ffRzgO1tIrVwkXdVzl8sHNQpEROZhzLBmF qGyt96C3VWvURvSxqZItYpaoKbWxqrn2EzIRabFrmsax+dcu/TVsdvO11/N9t1F4vhhB iL1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737673962; x=1738278762; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Eh4nnX1aD2le7gIboLyXAYEu1bW8nY+X/o+Mz+vnzjo=; b=J5dxRZkzw/Fx3KdCF8wEoSGP4jiRuU6NL2u7RNHZd+UeOa0wurylOcfWgIuvc0CFia c2x4sjZD8OSY+SaSxRrrTrizKsZaDPnnSyooPB1mdsLiRoDNyS6c4HmuyGu2IbqOeL2A DYBi8NYZ7dZpFacVJrKgib8075bB+zpdcbxYSdCogYFW7Hba9HEDpUKlGXiOtgEuSUaY WTuwwoMmZlSG3oI1ynQA/LzfjwjnNJvzm2nyQLxmGWAfvROiFX8Tgrb4/f/Xao5V1x1+ CaCh6onDgAdiIBJldeJ/9sk1tx1yVKzK5FDp19PGwjc7oEKvEvfkQ+lIQmp1JY6DC0WK mY+A== X-Gm-Message-State: AOJu0Yx6XtY/q5BV6B9D0+9RTM2tBE5uIEpue6AsX5AvbJAv9Jy1zF9h RGXx1HPk6txgTiKQFeiV5RCjOoP5Ss1oZYw+xSWge3kxhElaaIjS8bSlOUb1dPoxxfZTDWV/WgQ IVgLgnrWFTg== X-Google-Smtp-Source: AGHT+IFxpdthSe5BncCeTVj4UG7To8SnKpE9Lc9j3f6Py0fo5ssSpqj0wNqijkRjc2MRuf200iEoPfKEHsLb/g== X-Received: from pjbov5.prod.google.com ([2002:a17:90b:2585:b0:2f5:5240:4f0f]) (user=skhawaja job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:51c1:b0:2ef:2d9f:8e58 with SMTP id 98e67ed59e1d1-2f782d7f9cemr35686557a91.34.1737673961768; Thu, 23 Jan 2025 15:12:41 -0800 (PST) Date: Thu, 23 Jan 2025 23:12:35 +0000 In-Reply-To: <20250123231236.2657321-1-skhawaja@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250123231236.2657321-1-skhawaja@google.com> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250123231236.2657321-4-skhawaja@google.com> Subject: [PATCH net-next v2 3/4] Extend napi threaded polling to allow kthread based busy polling From: Samiullah Khawaja To: Jakub Kicinski , "David S . Miller " , Eric Dumazet , Paolo Abeni , almasrymina@google.com Cc: netdev@vger.kernel.org, skhawaja@google.com X-Patchwork-Delegate: kuba@kernel.org Add a new state to napi state enum: - STATE_THREADED_BUSY_POLL Threaded busy poll is enabled/running for this napi. Following changes are introduced in the napi scheduling and state logic: - When threaded busy poll is enabled through sysfs it also enables NAPI_STATE_THREADED so a kthread is created per napi. It also sets NAPI_STATE_THREADED_BUSY_POLL bit on each napi to indicate that we are supposed to busy poll for each napi. - When napi is scheduled with STATE_SCHED_THREADED and associated kthread is woken up, the kthread owns the context. If NAPI_STATE_THREADED_BUSY_POLL and NAPI_SCHED_THREADED both are set then it means that we can busy poll. - To keep busy polling and to avoid scheduling of the interrupts, the napi_complete_done returns false when both SCHED_THREADED and THREADED_BUSY_POLL flags are set. Also napi_complete_done returns early to avoid the STATE_SCHED_THREADED being unset. - If at any point STATE_THREADED_BUSY_POLL is unset, the napi_complete_done will run and unset the SCHED_THREADED bit also. This will make the associated kthread go to sleep as per existing logic. Signed-off-by: Samiullah Khawaja Reviewed-by: Willem de Bruijn --- Documentation/ABI/testing/sysfs-class-net | 3 +- Documentation/netlink/specs/netdev.yaml | 12 ++-- Documentation/networking/napi.rst | 67 ++++++++++++++++- .../net/ethernet/atheros/atl1c/atl1c_main.c | 2 +- include/linux/netdevice.h | 20 ++++-- include/uapi/linux/netdev.h | 6 ++ net/core/dev.c | 72 ++++++++++++++++--- net/core/net-sysfs.c | 2 +- net/core/netdev-genl-gen.c | 2 +- net/core/netdev-genl.c | 2 +- tools/include/uapi/linux/netdev.h | 6 ++ 11 files changed, 168 insertions(+), 26 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-class-net b/Documentation/ABI/testing/sysfs-class-net index ebf21beba846..15d7d36a8294 100644 --- a/Documentation/ABI/testing/sysfs-class-net +++ b/Documentation/ABI/testing/sysfs-class-net @@ -343,7 +343,7 @@ Date: Jan 2021 KernelVersion: 5.12 Contact: netdev@vger.kernel.org Description: - Boolean value to control the threaded mode per device. User could + Integer value to control the threaded mode per device. User could set this value to enable/disable threaded mode for all napi belonging to this device, without the need to do device up/down. @@ -351,4 +351,5 @@ Description: == ================================== 0 threaded mode disabled for this dev 1 threaded mode enabled for this dev + 2 threaded mode enabled, and busy polling enabled. == ================================== diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlink/specs/netdev.yaml index 785240d60df6..db3bf1eb9a63 100644 --- a/Documentation/netlink/specs/netdev.yaml +++ b/Documentation/netlink/specs/netdev.yaml @@ -78,6 +78,10 @@ definitions: name: qstats-scope type: flags entries: [ queue ] + - + name: napi-threaded + type: enum + entries: [ disable, enable, busy-poll-enable ] attribute-sets: - @@ -271,11 +275,11 @@ attribute-sets: - name: threaded doc: Whether the napi is configured to operate in threaded polling - mode. If this is set to `1` then the NAPI context operates - in threaded polling mode. + mode. If this is set to `enable` then the NAPI context operates + in threaded polling mode. If this is set to `busy-poll-enable` + then the NAPI kthread also does busypolling. type: u32 - checks: - max: 1 + enum: napi-threaded - name: queue attributes: diff --git a/Documentation/networking/napi.rst b/Documentation/networking/napi.rst index 41926e7a3dd4..edecc21f0bca 100644 --- a/Documentation/networking/napi.rst +++ b/Documentation/networking/napi.rst @@ -232,7 +232,9 @@ are not well known). Busy polling is enabled by either setting ``SO_BUSY_POLL`` on selected sockets or using the global ``net.core.busy_poll`` and ``net.core.busy_read`` sysctls. An io_uring API for NAPI busy polling -also exists. +also exists. Threaded polling of NAPI also has a mode to busy poll for +packets (:ref:`threaded busy polling`) using the same +thread that is used for NAPI processing. epoll-based busy polling ------------------------ @@ -395,6 +397,69 @@ Therefore, setting ``gro_flush_timeout`` and ``napi_defer_hard_irqs`` is the recommended usage, because otherwise setting ``irq-suspend-timeout`` might not have any discernible effect. +.. _threaded_busy_poll: + +Threaded NAPI busy polling +-------------------------- + +Threaded napi allows processing of packets from each NAPI in a kthread in +kernel. Threaded napi busy polling extends this and adds support to do +continuous busy polling of this napi. This can be used to enable busy polling +independent of userspace application or the API (epoll, io_uring, raw sockets) +being used in userspace to process the packets. + +It can be enabled for each NAPI using netlink interface or at device level using +the threaded NAPI sysctl. + +For example, using following script: + +.. code-block:: bash + + $ kernel-source/tools/net/ynl/pyynl/cli.py \ + --spec Documentation/netlink/specs/netdev.yaml \ + --do napi-set \ + --json='{"id": 66, + "threaded": "busy-poll-enable"}' + + +Enabling it for each NAPI allows finer control to enable busy pollling for +only a set of NIC queues which will get traffic with low latency requirements. + +Depending on application requirement, user might want to set affinity of the +kthread that is busy polling each NAPI. User might also want to set priority +and the scheduler of the thread depending on the latency requirements. + +For a hard low-latency application, user might want to dedicate the full core +for the NAPI polling so the NIC queue descriptors are picked up from the queue +as soon as they appear. For more relaxed low-latency requirement, user might +want to share the core with other threads. + +Once threaded busy polling is enabled for a NAPI, PID of the kthread can be +fetched using netlink interface so the affinity, priority and scheduler +configuration can be done. + +For example, following script can be used to fetch the pid: + +.. code-block:: bash + + $ kernel-source/tools/net/ynl/pyynl/cli.py \ + --spec Documentation/netlink/specs/netdev.yaml \ + --do napi-get \ + --json='{"id": 66}' + +This will output something like following, the pid `258` is the PID of the +kthread that is polling this NAPI. + +.. code-block:: bash + + $ {'defer-hard-irqs': 0, + 'gro-flush-timeout': 0, + 'id': 66, + 'ifindex': 2, + 'irq-suspend-timeout': 0, + 'pid': 258, + 'threaded': 'enable'} + .. _threaded: Threaded NAPI diff --git a/drivers/net/ethernet/atheros/atl1c/atl1c_main.c b/drivers/net/ethernet/atheros/atl1c/atl1c_main.c index c571614b1d50..513328476770 100644 --- a/drivers/net/ethernet/atheros/atl1c/atl1c_main.c +++ b/drivers/net/ethernet/atheros/atl1c/atl1c_main.c @@ -2688,7 +2688,7 @@ static int atl1c_probe(struct pci_dev *pdev, const struct pci_device_id *ent) adapter->mii.mdio_write = atl1c_mdio_write; adapter->mii.phy_id_mask = 0x1f; adapter->mii.reg_num_mask = MDIO_CTRL_REG_MASK; - dev_set_threaded(netdev, true); + dev_set_threaded(netdev, NETDEV_NAPI_THREADED_ENABLE); for (i = 0; i < adapter->rx_queue_count; ++i) netif_napi_add(netdev, &adapter->rrd_ring[i].napi, atl1c_clean_rx); diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 6afba24b18d1..9d6bb0d719b3 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -352,7 +352,7 @@ struct napi_config { u64 gro_flush_timeout; u64 irq_suspend_timeout; u32 defer_hard_irqs; - bool threaded; + u8 threaded; unsigned int napi_id; }; @@ -410,6 +410,8 @@ enum { NAPI_STATE_PREFER_BUSY_POLL, /* prefer busy-polling over softirq processing*/ NAPI_STATE_THREADED, /* The poll is performed inside its own thread*/ NAPI_STATE_SCHED_THREADED, /* Napi is currently scheduled in threaded mode */ + NAPI_STATE_THREADED_BUSY_POLL, /* The threaded napi poller will busy poll */ + NAPI_STATE_SCHED_THREADED_BUSY_POLL, /* The threaded napi poller is busy polling */ }; enum { @@ -423,8 +425,14 @@ enum { NAPIF_STATE_PREFER_BUSY_POLL = BIT(NAPI_STATE_PREFER_BUSY_POLL), NAPIF_STATE_THREADED = BIT(NAPI_STATE_THREADED), NAPIF_STATE_SCHED_THREADED = BIT(NAPI_STATE_SCHED_THREADED), + NAPIF_STATE_THREADED_BUSY_POLL = BIT(NAPI_STATE_THREADED_BUSY_POLL), + NAPIF_STATE_SCHED_THREADED_BUSY_POLL + = BIT(NAPI_STATE_SCHED_THREADED_BUSY_POLL), }; +#define NAPIF_STATE_THREADED_BUSY_POLL_MASK \ + (NAPIF_STATE_THREADED | NAPIF_STATE_THREADED_BUSY_POLL) + enum gro_result { GRO_MERGED, GRO_MERGED_FREE, @@ -571,16 +579,18 @@ static inline bool napi_complete(struct napi_struct *n) return napi_complete_done(n, 0); } -int dev_set_threaded(struct net_device *dev, bool threaded); +int dev_set_threaded(struct net_device *dev, + enum netdev_napi_threaded threaded); /* * napi_set_threaded - set napi threaded state * @napi: NAPI context - * @threaded: whether this napi does threaded polling + * @threaded: threading mode * * Return 0 on success and negative errno on failure. */ -int napi_set_threaded(struct napi_struct *napi, bool threaded); +int napi_set_threaded(struct napi_struct *napi, + enum netdev_napi_threaded threaded); void napi_disable(struct napi_struct *n); void napi_disable_locked(struct napi_struct *n); @@ -2404,7 +2414,7 @@ struct net_device { struct sfp_bus *sfp_bus; struct lock_class_key *qdisc_tx_busylock; bool proto_down; - bool threaded; + u8 threaded; /* priv_flags_slow, ungrouped to save space */ unsigned long see_all_hwtstamp_requests:1; diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h index 829648b2ef65..c2a9dbb361f6 100644 --- a/include/uapi/linux/netdev.h +++ b/include/uapi/linux/netdev.h @@ -74,6 +74,12 @@ enum netdev_qstats_scope { NETDEV_QSTATS_SCOPE_QUEUE = 1, }; +enum netdev_napi_threaded { + NETDEV_NAPI_THREADED_DISABLE, + NETDEV_NAPI_THREADED_ENABLE, + NETDEV_NAPI_THREADED_BUSY_POLL_ENABLE, +}; + enum { NETDEV_A_DEV_IFINDEX = 1, NETDEV_A_DEV_PAD, diff --git a/net/core/dev.c b/net/core/dev.c index 484947ad5410..8a5fde81f0b8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -78,6 +78,7 @@ #include #include #include +#include #include #include #include @@ -6403,7 +6404,8 @@ bool napi_complete_done(struct napi_struct *n, int work_done) * the guarantee we will be called later. */ if (unlikely(n->state & (NAPIF_STATE_NPSVC | - NAPIF_STATE_IN_BUSY_POLL))) + NAPIF_STATE_IN_BUSY_POLL | + NAPIF_STATE_SCHED_THREADED_BUSY_POLL))) return false; if (work_done) { @@ -6792,8 +6794,10 @@ static void init_gro_hash(struct napi_struct *napi) napi->gro_bitmask = 0; } -int napi_set_threaded(struct napi_struct *napi, bool threaded) +int napi_set_threaded(struct napi_struct *napi, + enum netdev_napi_threaded threaded) { + unsigned long val; if (napi->dev->threaded) return -EINVAL; @@ -6811,14 +6815,20 @@ int napi_set_threaded(struct napi_struct *napi, bool threaded) /* Make sure kthread is created before THREADED bit is set. */ smp_mb__before_atomic(); - assign_bit(NAPI_STATE_THREADED, &napi->state, threaded); + val = 0; + if (threaded == NETDEV_NAPI_THREADED_BUSY_POLL_ENABLE) + val |= NAPIF_STATE_THREADED_BUSY_POLL; + if (threaded) + val |= NAPIF_STATE_THREADED; + set_mask_bits(&napi->state, NAPIF_STATE_THREADED_BUSY_POLL_MASK, val); return 0; } -int dev_set_threaded(struct net_device *dev, bool threaded) +int dev_set_threaded(struct net_device *dev, enum netdev_napi_threaded threaded) { struct napi_struct *napi; + unsigned long val; int err = 0; netdev_assert_locked_or_invisible(dev); @@ -6826,17 +6836,22 @@ int dev_set_threaded(struct net_device *dev, bool threaded) if (dev->threaded == threaded) return 0; + val = 0; if (threaded) { /* Check if threaded is set at napi level already */ list_for_each_entry(napi, &dev->napi_list, dev_list) if (test_bit(NAPI_STATE_THREADED, &napi->state)) return -EINVAL; + val |= NAPIF_STATE_THREADED; + if (threaded == NETDEV_NAPI_THREADED_BUSY_POLL_ENABLE) + val |= NAPIF_STATE_THREADED_BUSY_POLL; + list_for_each_entry(napi, &dev->napi_list, dev_list) { if (!napi->thread) { err = napi_kthread_create(napi); if (err) { - threaded = false; + threaded = NETDEV_NAPI_THREADED_DISABLE; break; } } @@ -6855,9 +6870,13 @@ int dev_set_threaded(struct net_device *dev, bool threaded) * polled. In this case, the switch between threaded mode and * softirq mode will happen in the next round of napi_schedule(). * This should not cause hiccups/stalls to the live traffic. + * + * Switch to busy_poll threaded napi will occur after the threaded + * napi is scheduled. */ list_for_each_entry(napi, &dev->napi_list, dev_list) - assign_bit(NAPI_STATE_THREADED, &napi->state, threaded); + set_mask_bits(&napi->state, + NAPIF_STATE_THREADED_BUSY_POLL_MASK, val); return err; } @@ -7235,7 +7254,7 @@ static int napi_thread_wait(struct napi_struct *napi) return -1; } -static void napi_threaded_poll_loop(struct napi_struct *napi) +static void napi_threaded_poll_loop(struct napi_struct *napi, bool busy_poll) { struct bpf_net_context __bpf_net_ctx, *bpf_net_ctx; struct softnet_data *sd; @@ -7264,22 +7283,53 @@ static void napi_threaded_poll_loop(struct napi_struct *napi) } skb_defer_free_flush(sd); bpf_net_ctx_clear(bpf_net_ctx); + + /* Push the skbs up the stack if busy polling. */ + if (busy_poll) + __napi_gro_flush_helper(napi); local_bh_enable(); - if (!repoll) + /* If busy polling then do not break here because we need to + * call cond_resched and rcu_softirq_qs_periodic to prevent + * watchdog warnings. + */ + if (!repoll && !busy_poll) break; rcu_softirq_qs_periodic(last_qs); cond_resched(); + + if (!repoll) + break; } } static int napi_threaded_poll(void *data) { struct napi_struct *napi = data; + bool busy_poll_sched; + unsigned long val; + bool busy_poll; + + while (!napi_thread_wait(napi)) { + /* Once woken up, this means that we are scheduled as threaded + * napi and this thread owns the napi context, if busy poll + * state is set then we busy poll this napi. + */ + val = READ_ONCE(napi->state); + busy_poll = val & NAPIF_STATE_THREADED_BUSY_POLL; + busy_poll_sched = val & NAPIF_STATE_SCHED_THREADED_BUSY_POLL; + + /* Do not busy poll if napi is disabled. */ + if (unlikely(val & NAPIF_STATE_DISABLE)) + busy_poll = false; + + if (busy_poll != busy_poll_sched) + assign_bit(NAPI_STATE_SCHED_THREADED_BUSY_POLL, + &napi->state, busy_poll); - while (!napi_thread_wait(napi)) - napi_threaded_poll_loop(napi); + napi_threaded_poll_loop(napi, busy_poll); + } return 0; } @@ -12497,7 +12547,7 @@ static void run_backlog_napi(unsigned int cpu) { struct softnet_data *sd = per_cpu_ptr(&softnet_data, cpu); - napi_threaded_poll_loop(&sd->backlog); + napi_threaded_poll_loop(&sd->backlog, false); } static void backlog_napi_setup(unsigned int cpu) diff --git a/net/core/net-sysfs.c b/net/core/net-sysfs.c index 07cb99b114bd..beb496bcb633 100644 --- a/net/core/net-sysfs.c +++ b/net/core/net-sysfs.c @@ -657,7 +657,7 @@ static int modify_napi_threaded(struct net_device *dev, unsigned long val) if (list_empty(&dev->napi_list)) return -EOPNOTSUPP; - if (val != 0 && val != 1) + if (val > NETDEV_NAPI_THREADED_BUSY_POLL_ENABLE) return -EOPNOTSUPP; ret = dev_set_threaded(dev, val); diff --git a/net/core/netdev-genl-gen.c b/net/core/netdev-genl-gen.c index a1f80e687f53..b572beba42e7 100644 --- a/net/core/netdev-genl-gen.c +++ b/net/core/netdev-genl-gen.c @@ -97,7 +97,7 @@ static const struct nla_policy netdev_napi_set_nl_policy[NETDEV_A_NAPI_THREADED [NETDEV_A_NAPI_DEFER_HARD_IRQS] = NLA_POLICY_FULL_RANGE(NLA_U32, &netdev_a_napi_defer_hard_irqs_range), [NETDEV_A_NAPI_GRO_FLUSH_TIMEOUT] = { .type = NLA_UINT, }, [NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT] = { .type = NLA_UINT, }, - [NETDEV_A_NAPI_THREADED] = NLA_POLICY_MAX(NLA_U32, 1), + [NETDEV_A_NAPI_THREADED] = NLA_POLICY_MAX(NLA_U32, 2), }; /* Ops table for netdev */ diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index 208c3dd768ec..7ae5f3ed0961 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -329,7 +329,7 @@ netdev_nl_napi_set_config(struct napi_struct *napi, struct genl_info *info) if (info->attrs[NETDEV_A_NAPI_THREADED]) { threaded = nla_get_u32(info->attrs[NETDEV_A_NAPI_THREADED]); - napi_set_threaded(napi, !!threaded); + napi_set_threaded(napi, threaded); } if (info->attrs[NETDEV_A_NAPI_DEFER_HARD_IRQS]) { diff --git a/tools/include/uapi/linux/netdev.h b/tools/include/uapi/linux/netdev.h index 829648b2ef65..c2a9dbb361f6 100644 --- a/tools/include/uapi/linux/netdev.h +++ b/tools/include/uapi/linux/netdev.h @@ -74,6 +74,12 @@ enum netdev_qstats_scope { NETDEV_QSTATS_SCOPE_QUEUE = 1, }; +enum netdev_napi_threaded { + NETDEV_NAPI_THREADED_DISABLE, + NETDEV_NAPI_THREADED_ENABLE, + NETDEV_NAPI_THREADED_BUSY_POLL_ENABLE, +}; + enum { NETDEV_A_DEV_IFINDEX = 1, NETDEV_A_DEV_PAD, From patchwork Thu Jan 23 23:12:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samiullah Khawaja X-Patchwork-Id: 13948726 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF2331D5AD8 for ; Thu, 23 Jan 2025 23:12:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737673965; cv=none; b=DahrFyrKEQD6kJSyW2LnXMI8p12cBvE40ccqAwokk02om1ZWYJJe1S1sQ8bzRLLGVuRwhzx6PbGbpTe23hF8Qh+8uHQFoLHUgN15Iq5YlAOR7OZganodNzLKZgxJ78v7StRvpcEUWfT+MjaXwHry7FqNd35J3N4sFArbmtUSA9s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737673965; c=relaxed/simple; bh=172Kk9RdKO6Ntth20RvJAgF6oxpCsPrh3O92h4h98ic=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=RXZ1JEfNdqVPW/9R/H5TF7xTzcsZa/0x4Ai5PMDTDcY/gZIcz5KvWuNX7dWSrDQFz+KcRqc/bdVigWPxxzbuozvzebz7De017DbOjIRSAnvZucbMQAhcf03E9elw/zeSNMwWALiV9u/iVPtLwIT2DxnyjkIm4SJAq9tzTRI0jgo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ivANrdgf; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ivANrdgf" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ef909597d9so4213969a91.3 for ; Thu, 23 Jan 2025 15:12:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737673963; x=1738278763; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TYPNvZTUoIwJrSMLoTHaNaezqP1C9WRuC6S2ZKlVyFk=; b=ivANrdgfRncMbKDDuYI4dVTMA/hwZ+1H/OedfulTHE/liRy108o9yoArkkj+onxSTq N7gx102VWuXwucibO5VoHk0xCNh8cqW9azfLMAsPlji2Rs6+XJZWmUNR4zAi5YF2uBEL WVu6ns07Td/l0sa3cVrNWjHNlMSBGojSO44Lq1Lm2cIFBmIDm4Ye/C4KORjW6DVhalTw rcICYpsuUYcTal44OilTvzUNlxMseEOzGv5SQWjqI5NirMvn8x5uWZzcU+ZQhNteU7XA Y4VabA0szkAGqomkixN4zwixX+SGmLUImhA0s9dPfU44fApdlnLX0X31n9pPj53Zk9w5 pItg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737673963; x=1738278763; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TYPNvZTUoIwJrSMLoTHaNaezqP1C9WRuC6S2ZKlVyFk=; b=SC8pvtLhbrQsUNMvLp5zTKyNcx9jpd2TiIAfa2Qivuu5Th2y4YRhFl+4j0v7076E4E 5aC95C830XnQS86Dq9dYRXVEbf0DGCxw9ApiiEnG623QnXeBMHkPbwgECP+jm0xopp7G ejpThEWzJkJwyuNSdxDoIGSpc9dlt4503F2WjXCypZM2EIwSwYyRfB55WBC+e/rn5mlA bQWqFUuEGZh4ObWR71ZNJsU2g/ydO7Eu4W0UHlPWdGIhOvhFKw4WDysM4lcD154uvCb5 B/k38abCJIYWp7VZLapW0GCnsd6Jz18NbxQqRaL68mJdZeYfjHw0Az/HUCiW80CL9wdv V1JA== X-Gm-Message-State: AOJu0YzP239nOvBDa8rfmF1K2ecbgJWLeOYngxhGTcCyosOH8+dfJK0a kjoWIeZ+NuFVADGi21cYoWh6NaAw99qu1LjnZTBx8fkOufewb/63NguCqWatoEh1ndTHnrd6IKG 1XWcVW7eBhA== X-Google-Smtp-Source: AGHT+IFfWesATEV3uzhlZYVZ22AMZRfwTZ7JgDJ4ilhj+5brJql7R6SAM+xlaAzFIg40x0EYZ4cvk4clWB9Dkw== X-Received: from pfbcg7.prod.google.com ([2002:a05:6a00:2907:b0:725:eb9c:47e4]) (user=skhawaja job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:a95:b0:72d:8fa2:9998 with SMTP id d2e1a72fcca58-72dafa44feamr42612736b3a.14.1737673963105; Thu, 23 Jan 2025 15:12:43 -0800 (PST) Date: Thu, 23 Jan 2025 23:12:36 +0000 In-Reply-To: <20250123231236.2657321-1-skhawaja@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250123231236.2657321-1-skhawaja@google.com> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250123231236.2657321-5-skhawaja@google.com> Subject: [PATCH net-next v2 4/4] selftests: Add napi threaded busy poll test in `busy_poller` From: Samiullah Khawaja To: Jakub Kicinski , "David S . Miller " , Eric Dumazet , Paolo Abeni , almasrymina@google.com Cc: netdev@vger.kernel.org, skhawaja@google.com X-Patchwork-Delegate: kuba@kernel.org Add testcase to run busy poll test with threaded napi busy poll enabled. Signed-off-by: Samiullah Khawaja --- tools/testing/selftests/net/busy_poll_test.sh | 25 ++++++++++++++++++- tools/testing/selftests/net/busy_poller.c | 14 ++++++++--- 2 files changed, 35 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/net/busy_poll_test.sh b/tools/testing/selftests/net/busy_poll_test.sh index 7db292ec4884..aeca610dc989 100755 --- a/tools/testing/selftests/net/busy_poll_test.sh +++ b/tools/testing/selftests/net/busy_poll_test.sh @@ -27,6 +27,9 @@ NAPI_DEFER_HARD_IRQS=100 GRO_FLUSH_TIMEOUT=50000 SUSPEND_TIMEOUT=20000000 +# NAPI threaded busy poll config +NAPI_THREADED_POLL=2 + setup_ns() { set -e @@ -62,6 +65,9 @@ cleanup_ns() test_busypoll() { suspend_value=${1:-0} + napi_threaded_value=${2:-0} + prefer_busy_poll_value=${3:-$PREFER_BUSY_POLL} + tmp_file=$(mktemp) out_file=$(mktemp) @@ -73,10 +79,11 @@ test_busypoll() -b${SERVER_IP} \ -m${MAX_EVENTS} \ -u${BUSY_POLL_USECS} \ - -P${PREFER_BUSY_POLL} \ + -P${prefer_busy_poll_value} \ -g${BUSY_POLL_BUDGET} \ -i${NSIM_SV_IFIDX} \ -s${suspend_value} \ + -t${napi_threaded_value} \ -o${out_file}& wait_local_port_listen nssv ${SERVER_PORT} tcp @@ -109,6 +116,15 @@ test_busypoll_with_suspend() return $? } +test_busypoll_with_napi_threaded() +{ + # Only enable napi threaded poll. Set suspend timeout and prefer busy + # poll to 0. + test_busypoll 0 ${NAPI_THREADED_POLL} 0 + + return $? +} + ### ### Code start ### @@ -154,6 +170,13 @@ if [ $? -ne 0 ]; then exit 1 fi +test_busypoll_with_napi_threaded +if [ $? -ne 0 ]; then + echo "test_busypoll_with_napi_threaded failed" + cleanup_ns + exit 1 +fi + echo "$NSIM_SV_FD:$NSIM_SV_IFIDX" > $NSIM_DEV_SYS_UNLINK echo $NSIM_CL_ID > $NSIM_DEV_SYS_DEL diff --git a/tools/testing/selftests/net/busy_poller.c b/tools/testing/selftests/net/busy_poller.c index 04c7ff577bb8..f7407f09f635 100644 --- a/tools/testing/selftests/net/busy_poller.c +++ b/tools/testing/selftests/net/busy_poller.c @@ -65,15 +65,16 @@ static uint32_t cfg_busy_poll_usecs; static uint16_t cfg_busy_poll_budget; static uint8_t cfg_prefer_busy_poll; -/* IRQ params */ +/* NAPI params */ static uint32_t cfg_defer_hard_irqs; static uint64_t cfg_gro_flush_timeout; static uint64_t cfg_irq_suspend_timeout; +static enum netdev_napi_threaded cfg_napi_threaded_poll = NETDEV_NAPI_THREADED_DISABLE; static void usage(const char *filepath) { error(1, 0, - "Usage: %s -p -b -m -u -P -g -o -d -r -s -i", + "Usage: %s -p -b -m -u -P -g -o -d -r -s -t -i", filepath); } @@ -86,7 +87,7 @@ static void parse_opts(int argc, char **argv) if (argc <= 1) usage(argv[0]); - while ((c = getopt(argc, argv, "p:m:b:u:P:g:o:d:r:s:i:")) != -1) { + while ((c = getopt(argc, argv, "p:m:b:u:P:g:o:d:r:s:i:t:")) != -1) { /* most options take integer values, except o and b, so reduce * code duplication a bit for the common case by calling * strtoull here and leave bounds checking and casting per @@ -168,6 +169,12 @@ static void parse_opts(int argc, char **argv) cfg_ifindex = (int)tmp; break; + case 't': + if (tmp == ULLONG_MAX || tmp > 2) + error(1, ERANGE, "napi threaded poll value must be 0-2"); + + cfg_napi_threaded_poll = (enum netdev_napi_threaded)tmp; + break; } } @@ -246,6 +253,7 @@ static void setup_queue(void) cfg_gro_flush_timeout); netdev_napi_set_req_set_irq_suspend_timeout(set_req, cfg_irq_suspend_timeout); + netdev_napi_set_req_set_threaded(set_req, cfg_napi_threaded_poll); if (netdev_napi_set(ys, set_req)) error(1, 0, "can't set NAPI params: %s\n", yerr.msg);