From patchwork Tue May 7 11:20:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 13656701 X-Patchwork-Delegate: kuba@kernel.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1398B15B991; Tue, 7 May 2024 11:21:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715080881; cv=none; b=V4yg1ckn+QsY1SsTIAGKf58hiw7LJiKFH1sGMfGAhoCmRa8JgidqAztWXBMiriFXxLhgLQG00ds8meXdNOW5n6KSf4LW4l0C4NZi31oVUTaSYKG3LXI3/iRiz6EssDl9V6YsauGigNey6paTl7eJkqL6kjT2JfSYxsTRF2p0P1w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715080881; c=relaxed/simple; bh=oRr6YJO6YzkVYsSaAlJ1Vw+FpsjgZgQfNHdpN/CxiJ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lnNh1eb0iCYVWfu4CVs4v1atOuZl/auT7u4NHCY3/QGnnC3Kx3mGMAFQs6i5bPovEO+zAbzVNPs+x0m/cDZvot/xeL5eeE5cGgDcM0EkvLHD0FGbny3IzQm9J/jNg03JXhmnNCrMUpBk8QOb5IwwZGIbqwwLhcZw5rCmwdOdGiw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=WbjKFdRa; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="WbjKFdRa" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1715080880; x=1746616880; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=oRr6YJO6YzkVYsSaAlJ1Vw+FpsjgZgQfNHdpN/CxiJ0=; b=WbjKFdRau0+V3OzuA5iEx/c14//EcxzG0siSfnyJkiUoyAfgNcG+eGZg lnwsKj/wehZc3Gqc61FM8Y3MJaXSId72iUwWKnd1VtDKy8dAfYODK+Zsu 1AoOT0188kFN4stwD4+G0SsEdNGm9BgIIS5fjpNRmu1jeE5B2kyNqdP9G r22R07z4ppQcIRO8RbdaCug3jZzflc3ia9aMVy3NQEAe5pDqZMUK1ZZ6p oOdmV0qGIIrpCBhAZAPWgAK1Yzgxrlv5rUuwUzG7oe9o/GCgx2angl4xg JzXcqyn76ptHDSdVvGIbk14wMBLv3cala1QAPZO7kzOJW7LpL4gh972OB g==; X-CSE-ConnectionGUID: ibE+cDKpT+mjg7LPi5ZqEg== X-CSE-MsgGUID: D6Au7NnYSuyzM/BD66Cu+A== X-IronPort-AV: E=McAfee;i="6600,9927,11065"; a="21472659" X-IronPort-AV: E=Sophos;i="6.08,261,1712646000"; d="scan'208";a="21472659" Received: from orviesa004.jf.intel.com ([10.64.159.144]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 May 2024 04:21:19 -0700 X-CSE-ConnectionGUID: t3M2l3TqT8az4/KxNF4hHA== X-CSE-MsgGUID: 5Z4avV/USV+DYiUWPZTT/Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,261,1712646000"; d="scan'208";a="33316324" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa004.jf.intel.com with ESMTP; 07 May 2024 04:21:17 -0700 From: Alexander Lobakin To: Christoph Hellwig Cc: Alexander Lobakin , Eric Dumazet , Jakub Kicinski , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , "Rafael J. Wysocki" , Magnus Karlsson , nex.sw.ncis.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH v6 4/7] page_pool: make sure frag API fields don't span between cachelines Date: Tue, 7 May 2024 13:20:23 +0200 Message-ID: <20240507112026.1803778-5-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240507112026.1803778-1-aleksander.lobakin@intel.com> References: <20240507112026.1803778-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org After commit 5027ec19f104 ("net: page_pool: split the page_pool_params into fast and slow") that made &page_pool contain only "hot" params at the start, cacheline boundary chops frag API fields group in the middle again. To not bother with this each time fast params get expanded or shrunk, let's just align them to `4 * sizeof(long)`, the closest upper pow-2 to their actual size (2 longs + 1 int). This ensures 16-byte alignment for the 32-bit architectures and 32-byte alignment for the 64-bit ones, excluding unnecessary false-sharing. ::page_state_hold_cnt is used quite intensively on hotpath no matter if frag API is used, so move it to the newly created hole in the first cacheline. Signed-off-by: Alexander Lobakin --- include/net/page_pool/types.h | 12 +++++++++++- net/core/page_pool.c | 10 ++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 5e43a08d3231..5460cbab5de0 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -130,12 +130,22 @@ struct page_pool { struct page_pool_params_fast p; int cpuid; + u32 pages_state_hold_cnt; bool has_init_callback; + /* The following block must stay within one cacheline. On 32-bit + * systems, sizeof(long) == sizeof(int), so that the block size is + * ``3 * sizeof(long)``. On 64-bit systems, the actual size is + * ``2 * sizeof(long) + sizeof(int)``. The closest pow-2 to both of + * them is ``4 * sizeof(long)``, so just use that one for simplicity. + * Having it aligned to a cacheline boundary may be excessive and + * doesn't bring any good. + */ + __cacheline_group_begin(frag) __aligned(4 * sizeof(long)); long frag_users; struct page *frag_page; unsigned int frag_offset; - u32 pages_state_hold_cnt; + __cacheline_group_end(frag); struct delayed_work release_dw; void (*disconnect)(void *pool); diff --git a/net/core/page_pool.c b/net/core/page_pool.c index dd364d738c00..95eac12e8790 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -172,12 +172,22 @@ static void page_pool_producer_unlock(struct page_pool *pool, spin_unlock_bh(&pool->ring.producer_lock); } +static void page_pool_struct_check(void) +{ + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_users); + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_page); + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_offset); + CACHELINE_ASSERT_GROUP_SIZE(struct page_pool, frag, 4 * sizeof(long)); +} + static int page_pool_init(struct page_pool *pool, const struct page_pool_params *params, int cpuid) { unsigned int ring_qsize = 1024; /* Default */ + page_pool_struct_check(); + memcpy(&pool->p, ¶ms->fast, sizeof(pool->p)); memcpy(&pool->slow, ¶ms->slow, sizeof(pool->slow));