From patchwork Fri Feb 21 11:52:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13985288 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 633671E9B01; Fri, 21 Feb 2025 11:52:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740138747; cv=none; b=G43MWwOYdbMlRzDkwRvL6f7K4gEHrUrTvgJVjssLJX6scQDdPYaXjhTPHE/R1xznyQfIwYUiUBGTxxNP9kAsfBOwU1cljPgTH9900omJwvYhAAlkHDFApqINh8x/4VaTQU6KkiiKRicoVrR1rvhC0VOprluE0+PTQoAQ85Z4Hy0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740138747; c=relaxed/simple; bh=Tvy7kNNTc9PkCWr3xmyAxYGN6dt8yiG1fuySpSPdgzU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=snsJ/HrpvZEp6eQk2bH6TLoYW8pbBUxRQNt0PHLRgI4PFzZ8bbeitySsV6S2hsLMiaNrujG12/wO0AdOEC6u+sWk7e67F4B+sVgLEbDZbZIEqmnJgDk5EIXyXGMzySLbXBto6Ozd3ZL+EHbVeKO8uHtzKJXGzXpoW9gkU5oOMuM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=V+VpJcMK; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=fuJg4mEe; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="V+VpJcMK"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="fuJg4mEe" From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1740138743; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h4xG8g916RPInqOf64SIRf7WG6TVcyPly6/a8EygfS8=; b=V+VpJcMKKQNVlMRnL70fVcxXeJ1NuP9r3RAUW+loQADDBVmYcMsp0UA7Yeie9i024PEBz0 Uf/nMzgMNZxaBro2rj87dWIng72IGKc9oklemnC71ER3JGf03GdCbSUqJEac4MluSdFrOm Fv0mDLQbfjYKyv+dAR2hbe+HFih+j/bgjUeoOYBlYXRF6mMNRoFqz3f61B0fqslYA3vycv yKN9hbSp3o5tY7TRb97yUD0V7a9E/3aUXaFRtsfLBUPcMhPJDg7YlpB0aGQ9yub5nxf+Dt C2u7/Pf8DkBuJkt2T9wjugByaXKvVAmgWX1fKuJVryg9DABn1AoHEHrP4FPUPA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1740138743; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h4xG8g916RPInqOf64SIRf7WG6TVcyPly6/a8EygfS8=; b=fuJg4mEeBPJChOeEAue2toQj3cIqctOWvhf3hMNBcKfZ/iKsaaYjhF25JIeq+t4hqCad/J VveHoBt1Fso5NYCg== To: linux-rdma@vger.kernel.org, netdev@vger.kernel.org Cc: "David S. Miller" , Andrew Lunn , Eric Dumazet , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Leon Romanovsky , Paolo Abeni , Saeed Mahameed , Simon Horman , Tariq Toukan , Thomas Gleixner , Yunsheng Lin , Sebastian Andrzej Siewior Subject: [PATCH net-next 1/2] page_pool: Convert page_pool_recycle_stats to u64_stats_t. Date: Fri, 21 Feb 2025 12:52:20 +0100 Message-ID: <20250221115221.291006-2-bigeasy@linutronix.de> In-Reply-To: <20250221115221.291006-1-bigeasy@linutronix.de> References: <20250221115221.291006-1-bigeasy@linutronix.de> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Using u64 for statistics can lead to inconsistency on 32bit because an update and a read requires to access two 32bit values. This can be avoided by using u64_stats_t for the counters and u64_stats_sync for the required synchronisation on 32bit platforms. The synchronisation is a NOP on 64bit architectures. Use u64_stats_t for the counters in page_pool_recycle_stats. Add U64_STATS_ZERO, a static initializer for u64_stats_t. Signed-off-by: Sebastian Andrzej Siewior --- Documentation/networking/page_pool.rst | 4 +- .../ethernet/mellanox/mlx5/core/en_stats.c | 12 ++--- include/linux/u64_stats_sync.h | 5 ++ include/net/page_pool/types.h | 13 +++-- net/core/page_pool.c | 50 +++++++++++++------ net/core/page_pool_user.c | 10 ++-- 6 files changed, 61 insertions(+), 33 deletions(-) diff --git a/Documentation/networking/page_pool.rst b/Documentation/networking/page_pool.rst index 9d958128a57cb..00431bc88825f 100644 --- a/Documentation/networking/page_pool.rst +++ b/Documentation/networking/page_pool.rst @@ -184,8 +184,8 @@ Stats struct page_pool_stats stats = { 0 }; if (page_pool_get_stats(page_pool, &stats)) { /* perhaps the driver reports statistics with ethool */ - ethtool_print_allocation_stats(&stats.alloc_stats); - ethtool_print_recycle_stats(&stats.recycle_stats); + ethtool_print_allocation_stats(u64_stats_read(&stats.alloc_stats)); + ethtool_print_recycle_stats(u64_stats_read(&stats.recycle_stats)); } #endif diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c index 611ec4b6f3709..baff961970f25 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c @@ -501,7 +501,7 @@ static void mlx5e_stats_update_stats_rq_page_pool(struct mlx5e_channel *c) { struct mlx5e_rq_stats *rq_stats = c->rq.stats; struct page_pool *pool = c->rq.page_pool; - struct page_pool_stats stats = { 0 }; + struct page_pool_stats stats = { }; if (!page_pool_get_stats(pool, &stats)) return; @@ -513,11 +513,11 @@ static void mlx5e_stats_update_stats_rq_page_pool(struct mlx5e_channel *c) rq_stats->pp_alloc_waive = stats.alloc_stats.waive; rq_stats->pp_alloc_refill = stats.alloc_stats.refill; - rq_stats->pp_recycle_cached = stats.recycle_stats.cached; - rq_stats->pp_recycle_cache_full = stats.recycle_stats.cache_full; - rq_stats->pp_recycle_ring = stats.recycle_stats.ring; - rq_stats->pp_recycle_ring_full = stats.recycle_stats.ring_full; - rq_stats->pp_recycle_released_ref = stats.recycle_stats.released_refcnt; + rq_stats->pp_recycle_cached = u64_stats_read(&stats.recycle_stats.cached); + rq_stats->pp_recycle_cache_full = u64_stats_read(&stats.recycle_stats.cache_full); + rq_stats->pp_recycle_ring = u64_stats_read(&stats.recycle_stats.ring); + rq_stats->pp_recycle_ring_full = u64_stats_read(&stats.recycle_stats.ring_full); + rq_stats->pp_recycle_released_ref = u64_stats_read(&stats.recycle_stats.released_refcnt); } #else static void mlx5e_stats_update_stats_rq_page_pool(struct mlx5e_channel *c) diff --git a/include/linux/u64_stats_sync.h b/include/linux/u64_stats_sync.h index 457879938fc19..086bd4a51cfe9 100644 --- a/include/linux/u64_stats_sync.h +++ b/include/linux/u64_stats_sync.h @@ -94,6 +94,8 @@ static inline void u64_stats_inc(u64_stats_t *p) local64_inc(&p->v); } +#define U64_STATS_ZERO(_member, _name) {} + static inline void u64_stats_init(struct u64_stats_sync *syncp) { } static inline void __u64_stats_update_begin(struct u64_stats_sync *syncp) { } static inline void __u64_stats_update_end(struct u64_stats_sync *syncp) { } @@ -141,6 +143,9 @@ static inline void u64_stats_inc(u64_stats_t *p) seqcount_init(&__s->seq); \ } while (0) +#define U64_STATS_ZERO(_member, _name) \ + _member.seq = SEQCNT_ZERO(#_name#_member.seq) + static inline void __u64_stats_update_begin(struct u64_stats_sync *syncp) { preempt_disable_nested(); diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 7f405672b089d..c5ad80a542b7d 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -6,6 +6,7 @@ #include #include #include +#include #include #define PP_FLAG_DMA_MAP BIT(0) /* Should page_pool do the DMA @@ -114,6 +115,7 @@ struct page_pool_alloc_stats { /** * struct page_pool_recycle_stats - recycling (freeing) statistics + * @syncp: synchronisations point for updates. * @cached: recycling placed page in the page pool cache * @cache_full: page pool cache was full * @ring: page placed into the ptr ring @@ -121,11 +123,12 @@ struct page_pool_alloc_stats { * @released_refcnt: page released (and not recycled) because refcnt > 1 */ struct page_pool_recycle_stats { - u64 cached; - u64 cache_full; - u64 ring; - u64 ring_full; - u64 released_refcnt; + struct u64_stats_sync syncp; + u64_stats_t cached; + u64_stats_t cache_full; + u64_stats_t ring; + u64_stats_t ring_full; + u64_stats_t released_refcnt; }; /** diff --git a/net/core/page_pool.c b/net/core/page_pool.c index f5e908c9e7ad8..36fa14a1e8441 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -37,21 +37,27 @@ DEFINE_STATIC_KEY_FALSE(page_pool_mem_providers); #define BIAS_MAX (LONG_MAX >> 1) #ifdef CONFIG_PAGE_POOL_STATS -static DEFINE_PER_CPU(struct page_pool_recycle_stats, pp_system_recycle_stats); +static DEFINE_PER_CPU(struct page_pool_recycle_stats, pp_system_recycle_stats) = { + U64_STATS_ZERO(.syncp, pp_system_recycle_stats), +}; /* alloc_stat_inc is intended to be used in softirq context */ #define alloc_stat_inc(pool, __stat) (pool->alloc_stats.__stat++) /* recycle_stat_inc is safe to use when preemption is possible. */ #define recycle_stat_inc(pool, __stat) \ do { \ - struct page_pool_recycle_stats __percpu *s = pool->recycle_stats; \ - this_cpu_inc(s->__stat); \ + struct page_pool_recycle_stats *s = this_cpu_ptr(pool->recycle_stats); \ + u64_stats_update_begin(&s->syncp); \ + u64_stats_inc(&s->__stat); \ + u64_stats_update_end(&s->syncp); \ } while (0) #define recycle_stat_add(pool, __stat, val) \ do { \ - struct page_pool_recycle_stats __percpu *s = pool->recycle_stats; \ - this_cpu_add(s->__stat, val); \ + struct page_pool_recycle_stats *s = this_cpu_ptr(pool->recycle_stats); \ + u64_stats_update_begin(&s->syncp); \ + u64_stats_add(&s->__stat, val); \ + u64_stats_update_end(&s->syncp); \ } while (0) static const char pp_stats[][ETH_GSTRING_LEN] = { @@ -82,6 +88,7 @@ static const char pp_stats[][ETH_GSTRING_LEN] = { bool page_pool_get_stats(const struct page_pool *pool, struct page_pool_stats *stats) { + unsigned int start; int cpu = 0; if (!stats) @@ -99,11 +106,19 @@ bool page_pool_get_stats(const struct page_pool *pool, const struct page_pool_recycle_stats *pcpu = per_cpu_ptr(pool->recycle_stats, cpu); - stats->recycle_stats.cached += pcpu->cached; - stats->recycle_stats.cache_full += pcpu->cache_full; - stats->recycle_stats.ring += pcpu->ring; - stats->recycle_stats.ring_full += pcpu->ring_full; - stats->recycle_stats.released_refcnt += pcpu->released_refcnt; + do { + start = u64_stats_fetch_begin(&pcpu->syncp); + u64_stats_add(&stats->recycle_stats.cached, + u64_stats_read(&pcpu->cached)); + u64_stats_add(&stats->recycle_stats.cache_full, + u64_stats_read(&pcpu->cache_full)); + u64_stats_add(&stats->recycle_stats.ring, + u64_stats_read(&pcpu->ring)); + u64_stats_add(&stats->recycle_stats.ring_full, + u64_stats_read(&pcpu->ring_full)); + u64_stats_add(&stats->recycle_stats.released_refcnt, + u64_stats_read(&pcpu->released_refcnt)); + } while (u64_stats_fetch_retry(&pcpu->syncp, start)); } return true; @@ -139,11 +154,11 @@ u64 *page_pool_ethtool_stats_get(u64 *data, const void *stats) *data++ = pool_stats->alloc_stats.empty; *data++ = pool_stats->alloc_stats.refill; *data++ = pool_stats->alloc_stats.waive; - *data++ = pool_stats->recycle_stats.cached; - *data++ = pool_stats->recycle_stats.cache_full; - *data++ = pool_stats->recycle_stats.ring; - *data++ = pool_stats->recycle_stats.ring_full; - *data++ = pool_stats->recycle_stats.released_refcnt; + *data++ = u64_stats_read(&pool_stats->recycle_stats.cached); + *data++ = u64_stats_read(&pool_stats->recycle_stats.cache_full); + *data++ = u64_stats_read(&pool_stats->recycle_stats.ring); + *data++ = u64_stats_read(&pool_stats->recycle_stats.ring_full); + *data++ = u64_stats_read(&pool_stats->recycle_stats.released_refcnt); return data; } @@ -247,9 +262,14 @@ static int page_pool_init(struct page_pool *pool, #ifdef CONFIG_PAGE_POOL_STATS if (!(pool->slow.flags & PP_FLAG_SYSTEM_POOL)) { + unsigned int cpu; + pool->recycle_stats = alloc_percpu(struct page_pool_recycle_stats); if (!pool->recycle_stats) return -ENOMEM; + + for_each_possible_cpu(cpu) + u64_stats_init(&per_cpu_ptr(pool->recycle_stats, cpu)->syncp); } else { /* For system page pool instance we use a singular stats object * instead of allocating a separate percpu variable for each diff --git a/net/core/page_pool_user.c b/net/core/page_pool_user.c index 6677e0c2e2565..0d038c0c8996d 100644 --- a/net/core/page_pool_user.c +++ b/net/core/page_pool_user.c @@ -149,15 +149,15 @@ page_pool_nl_stats_fill(struct sk_buff *rsp, const struct page_pool *pool, nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_WAIVE, stats.alloc_stats.waive) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHED, - stats.recycle_stats.cached) || + u64_stats_read(&stats.recycle_stats.cached)) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHE_FULL, - stats.recycle_stats.cache_full) || + u64_stats_read(&stats.recycle_stats.cache_full)) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING, - stats.recycle_stats.ring) || + u64_stats_read(&stats.recycle_stats.ring)) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_RING_FULL, - stats.recycle_stats.ring_full) || + u64_stats_read(&stats.recycle_stats.ring_full)) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_RELEASED_REFCNT, - stats.recycle_stats.released_refcnt)) + u64_stats_read(&stats.recycle_stats.released_refcnt))) goto err_cancel_msg; genlmsg_end(rsp, hdr); From patchwork Fri Feb 21 11:52:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13985289 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A18E209689; Fri, 21 Feb 2025 11:52:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740138747; cv=none; b=acyjcCYb0RQbLIlnOFyfPW182z0c+JIkgtsooFpX+UGdaKXJHoriTPaTd7Ys354SMX1MW/2Touqk40TKHIu8DHlNzvKxHnUAY54p+nuVU/KNVmsLCRaM5V0zlrg0GStprU6DbYipt6+b+VRiM58y1stmqAj/GWZogYjWIm/mLd8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740138747; c=relaxed/simple; bh=G0uOksgba5mq341N+bmoOPfw9IfLvA66dMORHlHUX/c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iqm1mLKjo62iRX9+G5X+hRHoGg9uxrIVJ/z/BBTUOo9DOCAa4Jp4M+WCk2V3jqaAv6VWko5U3iYlD4VY5UTRsZp37E0TlL98rBGCIdsSnqp6rnFXEANp0xXb91fqCp0AZYyJ0g7ljVyvrQG5VH5ZXIPFf++JaRkYczpK/haWmws= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=YWmWoz9s; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=fu9I59Q3; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="YWmWoz9s"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="fu9I59Q3" From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1740138744; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VSgM5Id043YlplzTovi+8LMCEBi0aLDg6OqYa1PA7g0=; b=YWmWoz9sBV0Cklcm8U5it2O3Ak+LXDewbWXI9uVoVYNWthbqPxiNP27tbyXJILwtdHu63p NpF/SEMslGfRca972Y4xjPywgM/+B9lDBdK9XNJOqYZm/xMlaTjjOzfjvugj6QR2opyeRQ wMBDV4f6iOK5zn9BNYL7ZlWDn0/zDboR8iNhPZJJa0QbqSG3CWiH7P9rR8hwCFkBulmdqU gyRb2XebrSaIc0pMmGOzy/rV8RrTODwEJgjeAO8gW7yGQI9dzyMbE8QFTSi1JjNdd/ycU9 +VUqdQ16rFm5RUBV+GP1goI7HADEzpQGnGnbbunO7Motddy0Ixcp9pokDgOv4w== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1740138744; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VSgM5Id043YlplzTovi+8LMCEBi0aLDg6OqYa1PA7g0=; b=fu9I59Q3E8ae2SGI6zl44NsEW8D0Jqm9ycR68G/iLkeXSKmzw9SeI26CHq2+Hh2TKQjgFY 9Qcr1Giarj+licDg== To: linux-rdma@vger.kernel.org, netdev@vger.kernel.org Cc: "David S. Miller" , Andrew Lunn , Eric Dumazet , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Leon Romanovsky , Paolo Abeni , Saeed Mahameed , Simon Horman , Tariq Toukan , Thomas Gleixner , Yunsheng Lin , Sebastian Andrzej Siewior Subject: [PATCH net-next 2/2] page_pool: Convert page_pool_alloc_stats to u64_stats_t. Date: Fri, 21 Feb 2025 12:52:21 +0100 Message-ID: <20250221115221.291006-3-bigeasy@linutronix.de> In-Reply-To: <20250221115221.291006-1-bigeasy@linutronix.de> References: <20250221115221.291006-1-bigeasy@linutronix.de> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Using u64 for statistics can lead to inconsistency on 32bit because an update and a read requires to access two 32bit values. This can be avoided by using u64_stats_t for the counters and u64_stats_sync for the required synchronisation on 32bit platforms. The synchronisation is a NOP on 64bit architectures. Use u64_stats_t for the counters in page_pool_recycle_stats. Signed-off-by: Sebastian Andrzej Siewior --- .../ethernet/mellanox/mlx5/core/en_stats.c | 12 ++--- include/net/page_pool/types.h | 14 +++--- net/core/page_pool.c | 45 +++++++++++++------ net/core/page_pool_user.c | 12 ++--- 4 files changed, 52 insertions(+), 31 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c index baff961970f25..afb5c135b68c1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c @@ -506,12 +506,12 @@ static void mlx5e_stats_update_stats_rq_page_pool(struct mlx5e_channel *c) if (!page_pool_get_stats(pool, &stats)) return; - rq_stats->pp_alloc_fast = stats.alloc_stats.fast; - rq_stats->pp_alloc_slow = stats.alloc_stats.slow; - rq_stats->pp_alloc_slow_high_order = stats.alloc_stats.slow_high_order; - rq_stats->pp_alloc_empty = stats.alloc_stats.empty; - rq_stats->pp_alloc_waive = stats.alloc_stats.waive; - rq_stats->pp_alloc_refill = stats.alloc_stats.refill; + rq_stats->pp_alloc_fast = u64_stats_read(&stats.alloc_stats.fast); + rq_stats->pp_alloc_slow = u64_stats_read(&stats.alloc_stats.slow); + rq_stats->pp_alloc_slow_high_order = u64_stats_read(&stats.alloc_stats.slow_high_order); + rq_stats->pp_alloc_empty = u64_stats_read(&stats.alloc_stats.empty); + rq_stats->pp_alloc_waive = u64_stats_read(&stats.alloc_stats.waive); + rq_stats->pp_alloc_refill = u64_stats_read(&stats.alloc_stats.refill); rq_stats->pp_recycle_cached = u64_stats_read(&stats.recycle_stats.cached); rq_stats->pp_recycle_cache_full = u64_stats_read(&stats.recycle_stats.cache_full); diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index c5ad80a542b7d..f45d55e6e8643 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -96,6 +96,7 @@ struct page_pool_params { #ifdef CONFIG_PAGE_POOL_STATS /** * struct page_pool_alloc_stats - allocation statistics + * @syncp: synchronisations point for updates. * @fast: successful fast path allocations * @slow: slow path order-0 allocations * @slow_high_order: slow path high order allocations @@ -105,12 +106,13 @@ struct page_pool_params { * the cache due to a NUMA mismatch */ struct page_pool_alloc_stats { - u64 fast; - u64 slow; - u64 slow_high_order; - u64 empty; - u64 refill; - u64 waive; + struct u64_stats_sync syncp; + u64_stats_t fast; + u64_stats_t slow; + u64_stats_t slow_high_order; + u64_stats_t empty; + u64_stats_t refill; + u64_stats_t waive; }; /** diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 36fa14a1e8441..d69a03609613b 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -42,7 +42,14 @@ static DEFINE_PER_CPU(struct page_pool_recycle_stats, pp_system_recycle_stats) = }; /* alloc_stat_inc is intended to be used in softirq context */ -#define alloc_stat_inc(pool, __stat) (pool->alloc_stats.__stat++) +#define alloc_stat_inc(pool, __stat) \ + do { \ + struct page_pool_alloc_stats *s = &pool->alloc_stats; \ + u64_stats_update_begin(&s->syncp); \ + u64_stats_inc(&s->__stat); \ + u64_stats_update_end(&s->syncp); \ + } while (0) + /* recycle_stat_inc is safe to use when preemption is possible. */ #define recycle_stat_inc(pool, __stat) \ do { \ @@ -88,19 +95,30 @@ static const char pp_stats[][ETH_GSTRING_LEN] = { bool page_pool_get_stats(const struct page_pool *pool, struct page_pool_stats *stats) { + const struct page_pool_alloc_stats *alloc_stats; unsigned int start; int cpu = 0; if (!stats) return false; + alloc_stats = &pool->alloc_stats; /* The caller is responsible to initialize stats. */ - stats->alloc_stats.fast += pool->alloc_stats.fast; - stats->alloc_stats.slow += pool->alloc_stats.slow; - stats->alloc_stats.slow_high_order += pool->alloc_stats.slow_high_order; - stats->alloc_stats.empty += pool->alloc_stats.empty; - stats->alloc_stats.refill += pool->alloc_stats.refill; - stats->alloc_stats.waive += pool->alloc_stats.waive; + do { + start = u64_stats_fetch_begin(&alloc_stats->syncp); + u64_stats_add(&stats->alloc_stats.fast, + u64_stats_read(&alloc_stats->fast)); + u64_stats_add(&stats->alloc_stats.slow, + u64_stats_read(&alloc_stats->slow)); + u64_stats_add(&stats->alloc_stats.slow_high_order, + u64_stats_read(&alloc_stats->slow_high_order)); + u64_stats_add(&stats->alloc_stats.empty, + u64_stats_read(&alloc_stats->empty)); + u64_stats_add(&stats->alloc_stats.refill, + u64_stats_read(&alloc_stats->refill)); + u64_stats_add(&stats->alloc_stats.waive, + u64_stats_read(&alloc_stats->waive)); + } while (u64_stats_fetch_retry(&alloc_stats->syncp, start)); for_each_possible_cpu(cpu) { const struct page_pool_recycle_stats *pcpu = @@ -148,12 +166,12 @@ u64 *page_pool_ethtool_stats_get(u64 *data, const void *stats) { const struct page_pool_stats *pool_stats = stats; - *data++ = pool_stats->alloc_stats.fast; - *data++ = pool_stats->alloc_stats.slow; - *data++ = pool_stats->alloc_stats.slow_high_order; - *data++ = pool_stats->alloc_stats.empty; - *data++ = pool_stats->alloc_stats.refill; - *data++ = pool_stats->alloc_stats.waive; + *data++ = u64_stats_read(&pool_stats->alloc_stats.fast); + *data++ = u64_stats_read(&pool_stats->alloc_stats.slow); + *data++ = u64_stats_read(&pool_stats->alloc_stats.slow_high_order); + *data++ = u64_stats_read(&pool_stats->alloc_stats.empty); + *data++ = u64_stats_read(&pool_stats->alloc_stats.refill); + *data++ = u64_stats_read(&pool_stats->alloc_stats.waive); *data++ = u64_stats_read(&pool_stats->recycle_stats.cached); *data++ = u64_stats_read(&pool_stats->recycle_stats.cache_full); *data++ = u64_stats_read(&pool_stats->recycle_stats.ring); @@ -278,6 +296,7 @@ static int page_pool_init(struct page_pool *pool, pool->recycle_stats = &pp_system_recycle_stats; pool->system = true; } + u64_stats_init(&pool->alloc_stats.syncp); #endif if (ptr_ring_init(&pool->ring, ring_qsize, GFP_KERNEL) < 0) { diff --git a/net/core/page_pool_user.c b/net/core/page_pool_user.c index 0d038c0c8996d..c368cb141147f 100644 --- a/net/core/page_pool_user.c +++ b/net/core/page_pool_user.c @@ -137,17 +137,17 @@ page_pool_nl_stats_fill(struct sk_buff *rsp, const struct page_pool *pool, nla_nest_end(rsp, nest); if (nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_FAST, - stats.alloc_stats.fast) || + u64_stats_read(&stats.alloc_stats.fast)) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW, - stats.alloc_stats.slow) || + u64_stats_read(&stats.alloc_stats.slow)) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_SLOW_HIGH_ORDER, - stats.alloc_stats.slow_high_order) || + u64_stats_read(&stats.alloc_stats.slow_high_order)) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_EMPTY, - stats.alloc_stats.empty) || + u64_stats_read(&stats.alloc_stats.empty)) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_REFILL, - stats.alloc_stats.refill) || + u64_stats_read(&stats.alloc_stats.refill)) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_ALLOC_WAIVE, - stats.alloc_stats.waive) || + u64_stats_read(&stats.alloc_stats.waive)) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHED, u64_stats_read(&stats.recycle_stats.cached)) || nla_put_uint(rsp, NETDEV_A_PAGE_POOL_STATS_RECYCLE_CACHE_FULL,