From patchwork Sun Mar 9 12:47:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 14008399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A944C28B25 for ; Sun, 9 Mar 2025 12:50:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EAA2E280002; Sun, 9 Mar 2025 08:50:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E58BE280001; Sun, 9 Mar 2025 08:50:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFA4B280002; Sun, 9 Mar 2025 08:50:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A7395280001 for ; Sun, 9 Mar 2025 08:50:15 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 13B2212265B for ; Sun, 9 Mar 2025 12:50:17 +0000 (UTC) X-FDA: 83201995674.25.6C763B3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id C9C151C000C for ; Sun, 9 Mar 2025 12:50:14 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DYbwvg66; spf=pass (imf18.hostedemail.com: domain of toke@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=toke@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741524614; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=zY0n1mTsfg2sZHOCrorEXerzqFHGZByck2FJCEmzyfg=; b=tIPW7BxfP/G53KA4B88LOxz3iZujM2XT77FRnq7tvu7euzThaSyu6cswfNP+DrgZkULKRJ 6Q757yCL84GBKUBTGcOrD9gdojVfE87XOXPs8xC4GwChTtDSOyKRgNKKtJ2r/E6fGKZEiK +zwRVxjS5I6irgaaoGvBw8uz1Ik5ESk= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DYbwvg66; spf=pass (imf18.hostedemail.com: domain of toke@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=toke@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741524614; a=rsa-sha256; cv=none; b=hTx/gd0gDr9r5lNR9+GJWnyK7LXaV8fpnAFwu9B5HE7DDl6Ti7w5LR1JSDJN0HiyYfS4AK LEDVj1ca3Ddx6looi8EI4HJ9Hk8ee2EEno9Gla2nzKgbmvNyBeBoDRVxPIznLuiC34WMtN mnwYHXDCE3u/zP+ZAJ0qS7U4a0mR5d0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741524614; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=zY0n1mTsfg2sZHOCrorEXerzqFHGZByck2FJCEmzyfg=; b=DYbwvg662g8ENcztxmi+vQwBaqeXjilR/bM3+MXQVl/ylefnW2dA0IDN53QBpgCI8/N1UJ vNZ4dBvxgrVThHFsBlozT71Rzo+S2WixX7iJ8LwNVvW77v41zsXtMkqI8Om0zuHEiD/3lt DKd5mxmEi64TnpigBNTPyjlsSir4gp4= Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-112-zR3qSw2WNoCQrVkHi8gAYw-1; Sun, 09 Mar 2025 08:50:12 -0400 X-MC-Unique: zR3qSw2WNoCQrVkHi8gAYw-1 X-Mimecast-MFC-AGG-ID: zR3qSw2WNoCQrVkHi8gAYw_1741524611 Received: by mail-lf1-f71.google.com with SMTP id 2adb3069b0e04-549927ec579so1214008e87.3 for ; Sun, 09 Mar 2025 05:50:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741524611; x=1742129411; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=zY0n1mTsfg2sZHOCrorEXerzqFHGZByck2FJCEmzyfg=; b=Azl9LHjEd9WVKOlgPA/LJyoqdvnoHTiQaO5t9mEVZTzbnrF27QOdbTjPXq7f+DUpSE pmolZtsIA30tXM6r0OSYZOIMUotc/bJtlrJzs8OpHG9nqlfyIgVoDZ0Xf4tt2PpDII6i O7HFX+MD3iSyltYCMRps/uK1LnKFl2AtjGGQsggWBCGJhB48CM3751fOuZ1kbu+u6hWi d4Hnom5ffLevUWB8NPu8ztCeCM8kxCmbr7FnKeJ5ceC2Whvwi3IrSQALVUYl2DX1OleG DkuHCwcx17oU5Zu7rnHi1BT/XDxdVAB3MFM5BLMA/hL+ab56X4LdYfoUosOMYyZlP5dB dYmg== X-Forwarded-Encrypted: i=1; AJvYcCVp3zfV4DU0x9Lf4sxAY+a+a2oNtOWyYAxzI8VDAbMkBSkODfC93BSVC1UdU+Bbzwc6hhB4pQjVZQ==@kvack.org X-Gm-Message-State: AOJu0YxL70HTe4Te9co89yA5QjiFker/QnmqXmClLQ6HcITd+RnzXYCE 6nvAokcTNyW/R1WHpIBD90DmqQ88hMLzcqvEfep3cb9ZOuBl+1kkJOn/ypZdzmGKzMhak1AY700 Syf8wX7s3W6vzWMQrB+qDvlFTX0EfdllnYvasypaJyfCapT8J X-Gm-Gg: ASbGnctt/DYZLGoj64x3kf7/hB1cKKC3ZKfCw0FhV3DhlOHcnizyaI4EfxLXj6UkyCS IctTTowHVCvaLR9jdQ/+frfA4nGhcYScXSTzaYzgsZ0Gw7GYBoq5Qkj/K8g2Ooj+JxuPVIQOhVC uGFmKsHeXKIAi8XYjp/B4Uytvig3GYiAvR9M0j0viCZxnDeIhWnjhD4roZtMSsAi276g1Mbj3Lp j1bOmHCsAHkZMrPEura9OP3k0Sc00OBlDKzhTu8OdDunt9SVBUDdtib12ifg4F14sTSHhfIPpce WPnhzK0ZKGjH X-Received: by 2002:a05:6512:3dab:b0:545:ea9:1a1d with SMTP id 2adb3069b0e04-54990e2c0femr2738957e87.1.1741524610661; Sun, 09 Mar 2025 05:50:10 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGyrUvBveDFCHFXQ28XqNGPRJsR1btM2RLgidtLkHfCkg2GXey9OMd8zjOyVd4eptehm/8QMQ== X-Received: by 2002:a05:6512:3dab:b0:545:ea9:1a1d with SMTP id 2adb3069b0e04-54990e2c0femr2738941e87.1.1741524610134; Sun, 09 Mar 2025 05:50:10 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5498ae46167sm1082018e87.32.2025.03.09.05.50.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Mar 2025 05:50:08 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 6E5A518FA191; Sun, 09 Mar 2025 13:50:07 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Jesper Dangaard Brouer , Ilias Apalodimas , Andrew Morton , "David S. Miller" Cc: Yunsheng Lin , =?utf-8?q?Toke_H=C3=B8iland-J?= =?utf-8?q?=C3=B8rgensen?= , Yonglong Liu , Mina Almasry , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , netdev@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH net-next v2] page_pool: Track DMA-mapped pages and unmap them when destroying the pool Date: Sun, 9 Mar 2025 13:47:17 +0100 Message-ID: <20250309124719.21285-1-toke@redhat.com> X-Mailer: git-send-email 2.48.1 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: wzWRRidYLeZZ9JXkR4DQzXR7NALOEoPbzz6W64Ny1yY_1741524611 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: C9C151C000C X-Stat-Signature: 6ne3w3szgn8ikyd71n91xgub1mdkoxgx X-Rspamd-Server: rspam09 X-HE-Tag: 1741524614-518176 X-HE-Meta: U2FsdGVkX1/fk4OebBvMhdCdVMRuoDPfGY9alEHU6SZN7CFkpKQYvRdg6XlZf1KLEcsewsF30BPVsdbyP0GIhMBxdVe1Q1cZ0/6XYl8xSukQb4GoHv+VOnRLrxYEMxvqPoivjx9FCs+jVPBMun5LZQDYq4EuD2IlBS8WuzMNYZ08m3jvWs1FVLe3d4JMji7HkOo6yq49Vkt63AxDpJVu/L9ZYvQaKpE2FX+gjAF8J/ZWm0MvZmcXofDCXVxPknys4x1G+ZJtxG4EK8H1FuiGprUhS5Z2hqd1Da0mZI235odIGAhUWCHTmc8zpQP51QLyqDxAw1QQDmlfsmGqb3XopUm2M3FvAgrSlJUthDKXRTv+bCa0kQTl2/6aXZzW/rU6iIyyJB2ZHEHbx/dQg+hkTHEZJM8gAJ+nh237xhmBcOnXEUK/AXUtFs8H9BRoQ5ibjPJuJSNO8PpkbldAAcbwcIpO/6PDvZOVu1slW25nD5iQwJjvusDPKV6y6KkstQuzBRmy917GWxSHtzahBhphXYLlhE++JVetWb209mmrfc5S2UiDwUpKNwx+U3lMC9/Pptpw/nrVNUe+ka0JqipjOVntLQ3gSr37Gtcp2ZaknXHU6yppvgd5Vg3Hjmw1drObTn+fYi1e70i74G8+5XeILWdEQ1MbnBHTR4MdXlsw1fQZdZWtBIookSil7UOqR9Wr4Y1t+uw7ehe6tZ6WpJqdQC62Z9c2LxeFaOgBezJQ5m0EVAZzRqz/n+kz7vQWW6TAWlSeV59OpXzSVakAAcSL/vA5nQ99p0uTvywdYxqb/d3VeGCCaAP7FGPAGT6LIRklyd7el+0OhUeZe4xpOojqOYUVtHF8vIiDfjaCOuT9R71em6sq9pGj3PIG2tGwsvwzz369lJavMRzgtwMBx6hcLMnXJ1K3yDoFMN90sMMeSyTuD2ke0Z8bBGycylVDqnh8GnJX6dJmZCZvxhvB5Uo OkTCEJcK Io4gKp4RBHgflP42uarypqkzBBBAncuMQXiJFPkkqAinZeFW/TaZkawsQgsM49D5Zik0GylwmEHc/Rk6xGMwfqqPo9G7qfyj4lk+Ag+N3NIBKikrHlNmSbCxTPrqtO6g2isviVXKZYIs9ESZvLVwQiqtXTzDZp6KeG1uuUTxoaGIfy376e6vyhOrS3U4X0vVlMGUw7V2EyK30FxkMVz3LoXuZgJuP2Ln6VeeBscGIIwIeIUIlY00KIn74jPgAWsf9+Pa5pWI23v5om+a7E0vZ7DUlkecKbzQtIO3ojVPj12gKBJkuZiW/4L0jMZYSu/gjX4PonABJxdvEFs6zgej8BPQFJmYycEzoH+5CH0rA/YZzojkKXutlNrUVi/3gpz1midUMz5o3QcPmKWQHUDs52pndNuCzSO1nHps+sVG4ARmAOsjLknVHp5h4Of0jXCffws5SGpYfvIlUvd3CLT0kCyV2RBAKs/Q30V9t2eNTAm/o3Pb/StytThIUZYNnMNegQghFKwhVKj0nr8MhFwCcs2LMCUQxgLguWQVpOP721+o/aH7/NoY8si/ViQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When enabling DMA mapping in page_pool, pages are kept DMA mapped until they are released from the pool, to avoid the overhead of re-mapping the pages every time they are used. This causes problems when a device is torn down, because the page pool can't unmap the pages until they are returned to the pool. This causes resource leaks and/or crashes when there are pages still outstanding while the device is torn down, because page_pool will attempt an unmap of a non-existent DMA device on the subsequent page return. To fix this, implement a simple tracking of outstanding dma-mapped pages in page pool using an xarray. This was first suggested by Mina[0], and turns out to be fairly straight forward: We simply store pointers to pages directly in the xarray with xa_alloc() when they are first DMA mapped, and remove them from the array on unmap. Then, when a page pool is torn down, it can simply walk the xarray and unmap all pages still present there before returning, which also allows us to get rid of the get/put_device() calls in page_pool. Using xa_cmpxchg(), no additional synchronisation is needed, as a page will only ever be unmapped once. To avoid having to walk the entire xarray on unmap to find the page reference, we stash the ID assigned by xa_alloc() into the page structure itself, using the upper bits of the pp_magic field. This requires a couple of defines to avoid conflicting with the POINTER_POISON_DELTA define, but this is all evaluated at compile-time, so should not affect run-time performance. Since all the tracking is performed on DMA map/unmap, no additional code is needed in the fast path, meaning the performance overhead of this tracking is negligible. The extra memory needed to track the pages is neatly encapsulated inside xarray, which uses the 'struct xa_node' structure to track items. This structure is 576 bytes long, with slots for 64 items, meaning that a full node occurs only 9 bytes of overhead per slot it tracks (in practice, it probably won't be this efficient, but in any case it should be an acceptable overhead). [0] https://lore.kernel.org/all/CAHS8izPg7B5DwKfSuzz-iOop_YRbk3Sd6Y4rX7KBG9DcVJcyWg@mail.gmail.com/ Fixes: ff7d6b27f894 ("page_pool: refurbish version of page_pool code") Reported-by: Yonglong Liu Suggested-by: Mina Almasry Reviewed-by: Jesper Dangaard Brouer Tested-by: Jesper Dangaard Brouer Signed-off-by: Toke Høiland-Jørgensen Reviewed-by: Mina Almasry --- v2: - Stash the id in the pp_magic field of struct page instead of overwriting the mapping field. This version is compile-tested only. include/net/page_pool/types.h | 31 +++++++++++++++++++++++ mm/page_alloc.c | 3 ++- net/core/netmem_priv.h | 35 +++++++++++++++++++++++++- net/core/page_pool.c | 46 +++++++++++++++++++++++++++++------ 4 files changed, 105 insertions(+), 10 deletions(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 36eb57d73abc..d879a505ca4d 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -6,6 +6,7 @@ #include #include #include +#include #include #define PP_FLAG_DMA_MAP BIT(0) /* Should page_pool do the DMA @@ -54,6 +55,34 @@ struct pp_alloc_cache { netmem_ref cache[PP_ALLOC_CACHE_SIZE]; }; +/* + * DMA mapping IDs + * + * When DMA-mapping a page, we allocate an ID (from an xarray) and stash this in + * the upper bits of page->pp_magic. The number of bits available here is + * constrained by the size of an unsigned long, and the definition of + * PP_SIGNATURE. + */ +#define PP_DMA_INDEX_SHIFT (1 + __fls(PP_SIGNATURE - POISON_POINTER_DELTA)) +#define _PP_DMA_INDEX_BITS MIN(32, BITS_PER_LONG - PP_DMA_INDEX_SHIFT - 1) + +#if POISON_POINTER_DELTA > 0 +/* PP_SIGNATURE includes POISON_POINTER_DELTA, so limit the size of the DMA + * index to not overlap with that if set + */ +#define PP_DMA_INDEX_BITS MIN(_PP_DMA_INDEX_BITS, \ + __ffs(POISON_POINTER_DELTA) - PP_DMA_INDEX_SHIFT) +#else +#define PP_DMA_INDEX_BITS _PP_DMA_INDEX_BITS +#endif + +#define PP_DMA_INDEX_MASK GENMASK(PP_DMA_INDEX_BITS + PP_DMA_INDEX_SHIFT - 1, \ + PP_DMA_INDEX_SHIFT) +#define PP_DMA_INDEX_LIMIT XA_LIMIT(1, BIT(PP_DMA_INDEX_BITS) - 1) + +/* For check in page_alloc.c */ +#define PP_MAGIC_MASK (~(PP_DMA_INDEX_MASK | 0x3)) + /** * struct page_pool_params - page pool parameters * @fast: params accessed frequently on hotpath @@ -221,6 +250,8 @@ struct page_pool { void *mp_priv; const struct memory_provider_ops *mp_ops; + struct xarray dma_mapped; + #ifdef CONFIG_PAGE_POOL_STATS /* recycle stats are per-cpu to avoid locking */ struct page_pool_recycle_stats __percpu *recycle_stats; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 579789600a3c..96776e7b2301 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -55,6 +55,7 @@ #include #include #include +#include #include #include "internal.h" #include "shuffle.h" @@ -873,7 +874,7 @@ static inline bool page_expected_state(struct page *page, page->memcg_data | #endif #ifdef CONFIG_PAGE_POOL - ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) | + ((page->pp_magic & PP_MAGIC_MASK) == PP_SIGNATURE) | #endif (page->flags & check_flags))) return false; diff --git a/net/core/netmem_priv.h b/net/core/netmem_priv.h index 7eadb8393e00..59face70f40d 100644 --- a/net/core/netmem_priv.h +++ b/net/core/netmem_priv.h @@ -3,10 +3,19 @@ #ifndef __NETMEM_PRIV_H #define __NETMEM_PRIV_H -static inline unsigned long netmem_get_pp_magic(netmem_ref netmem) +static inline unsigned long _netmem_get_pp_magic(netmem_ref netmem) { return __netmem_clear_lsb(netmem)->pp_magic; } +static inline unsigned long netmem_get_pp_magic(netmem_ref netmem) +{ + return _netmem_get_pp_magic(netmem) & ~PP_DMA_INDEX_MASK; +} + +static inline void netmem_set_pp_magic(netmem_ref netmem, unsigned long pp_magic) +{ + __netmem_clear_lsb(netmem)->pp_magic = pp_magic; +} static inline void netmem_or_pp_magic(netmem_ref netmem, unsigned long pp_magic) { @@ -28,4 +37,28 @@ static inline void netmem_set_dma_addr(netmem_ref netmem, { __netmem_clear_lsb(netmem)->dma_addr = dma_addr; } + +static inline unsigned long netmem_get_dma_index(netmem_ref netmem) +{ + unsigned long magic; + + if (WARN_ON_ONCE(netmem_is_net_iov(netmem))) + return 0; + + magic = _netmem_get_pp_magic(netmem); + + return (magic & PP_DMA_INDEX_MASK) >> PP_DMA_INDEX_SHIFT; +} + +static inline void netmem_set_dma_index(netmem_ref netmem, + unsigned long id) +{ + unsigned long magic; + + if (WARN_ON_ONCE(netmem_is_net_iov(netmem))) + return; + + magic = netmem_get_pp_magic(netmem) | (id << PP_DMA_INDEX_SHIFT); + netmem_set_pp_magic(netmem, magic); +} #endif diff --git a/net/core/page_pool.c b/net/core/page_pool.c index acef1fcd8ddc..dceef9b82198 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -226,6 +226,8 @@ static int page_pool_init(struct page_pool *pool, return -EINVAL; pool->dma_map = true; + + xa_init_flags(&pool->dma_mapped, XA_FLAGS_ALLOC1); } if (pool->slow.flags & PP_FLAG_DMA_SYNC_DEV) { @@ -275,9 +277,6 @@ static int page_pool_init(struct page_pool *pool, /* Driver calling page_pool_create() also call page_pool_destroy() */ refcount_set(&pool->user_cnt, 1); - if (pool->dma_map) - get_device(pool->p.dev); - if (pool->slow.flags & PP_FLAG_ALLOW_UNREADABLE_NETMEM) { /* We rely on rtnl_lock()ing to make sure netdev_rx_queue * configuration doesn't change while we're initializing @@ -325,7 +324,7 @@ static void page_pool_uninit(struct page_pool *pool) ptr_ring_cleanup(&pool->ring, NULL); if (pool->dma_map) - put_device(pool->p.dev); + xa_destroy(&pool->dma_mapped); #ifdef CONFIG_PAGE_POOL_STATS if (!pool->system) @@ -470,9 +469,11 @@ page_pool_dma_sync_for_device(const struct page_pool *pool, __page_pool_dma_sync_for_device(pool, netmem, dma_sync_size); } -static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem) +static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem, gfp_t gfp) { dma_addr_t dma; + int err; + u32 id; /* Setup DMA mapping: use 'struct page' area for storing DMA-addr * since dma_addr_t can be either 32 or 64 bits and does not always fit @@ -486,9 +487,19 @@ static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem) if (dma_mapping_error(pool->p.dev, dma)) return false; + if (in_softirq()) + err = xa_alloc(&pool->dma_mapped, &id, netmem_to_page(netmem), + PP_DMA_INDEX_LIMIT, gfp); + else + err = xa_alloc_bh(&pool->dma_mapped, &id, netmem_to_page(netmem), + PP_DMA_INDEX_LIMIT, gfp); + if (err) + goto unmap_failed; + if (page_pool_set_dma_addr_netmem(netmem, dma)) goto unmap_failed; + netmem_set_dma_index(netmem, id); page_pool_dma_sync_for_device(pool, netmem, pool->p.max_len); return true; @@ -511,7 +522,7 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool, if (unlikely(!page)) return NULL; - if (pool->dma_map && unlikely(!page_pool_dma_map(pool, page_to_netmem(page)))) { + if (pool->dma_map && unlikely(!page_pool_dma_map(pool, page_to_netmem(page), gfp))) { put_page(page); return NULL; } @@ -557,7 +568,7 @@ static noinline netmem_ref __page_pool_alloc_pages_slow(struct page_pool *pool, */ for (i = 0; i < nr_pages; i++) { netmem = pool->alloc.cache[i]; - if (dma_map && unlikely(!page_pool_dma_map(pool, netmem))) { + if (dma_map && unlikely(!page_pool_dma_map(pool, netmem, gfp))) { put_page(netmem_to_page(netmem)); continue; } @@ -659,6 +670,8 @@ void page_pool_clear_pp_info(netmem_ref netmem) static __always_inline void __page_pool_release_page_dma(struct page_pool *pool, netmem_ref netmem) { + struct page *old, *page = netmem_to_page(netmem); + unsigned long id; dma_addr_t dma; if (!pool->dma_map) @@ -667,6 +680,17 @@ static __always_inline void __page_pool_release_page_dma(struct page_pool *pool, */ return; + id = netmem_get_dma_index(netmem); + if (!id) + return; + + if (in_softirq()) + old = xa_cmpxchg(&pool->dma_mapped, id, page, NULL, 0); + else + old = xa_cmpxchg_bh(&pool->dma_mapped, id, page, NULL, 0); + if (old != page) + return; + dma = page_pool_get_dma_addr_netmem(netmem); /* When page is unmapped, it cannot be returned to our pool */ @@ -674,6 +698,7 @@ static __always_inline void __page_pool_release_page_dma(struct page_pool *pool, PAGE_SIZE << pool->p.order, pool->p.dma_dir, DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING); page_pool_set_dma_addr_netmem(netmem, 0); + netmem_set_dma_index(netmem, 0); } /* Disconnects a page (from a page_pool). API users can have a need @@ -1083,8 +1108,13 @@ static void page_pool_empty_alloc_cache_once(struct page_pool *pool) static void page_pool_scrub(struct page_pool *pool) { + unsigned long id; + void *ptr; + page_pool_empty_alloc_cache_once(pool); - pool->destroy_cnt++; + if (!pool->destroy_cnt++) + xa_for_each(&pool->dma_mapped, id, ptr) + __page_pool_release_page_dma(pool, page_to_netmem(ptr)); /* No more consumers should exist, but producers could still * be in-flight.