From patchwork Wed Nov 20 10:34:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yunsheng Lin X-Patchwork-Id: 13881008 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1C8BD63920 for ; Wed, 20 Nov 2024 10:41:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 580576B0083; Wed, 20 Nov 2024 05:41:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 508586B0085; Wed, 20 Nov 2024 05:41:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3837F6B0088; Wed, 20 Nov 2024 05:41:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1B6B76B0083 for ; Wed, 20 Nov 2024 05:41:46 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C6623409D9 for ; Wed, 20 Nov 2024 10:41:45 +0000 (UTC) X-FDA: 82806131562.16.6F5144B Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by imf16.hostedemail.com (Postfix) with ESMTP id D562A18000A for ; Wed, 20 Nov 2024 10:40:51 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf16.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732099168; a=rsa-sha256; cv=none; b=fpo6D2uVPLNYyd6M7g7a2jHxyRFDOW1d+0Mh7zFp08QRzqJ1quB4cknv5M+cYakXU2oKsJ 2rL0YfzIi5/J6jZIizajRY5sPilvVv2f5BsGbLIxuRWyvyfxixmVhWyZtYfhgsNscKfuLA qMEZJfin2C30kXRpm6BDTf4s8pKfS40= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf16.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732099168; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W3htf//MwczDWqYyeeDY9pqQ6AhvVfaILiMVE/4mHVw=; b=CQ8u1TIoI1r+rtMj0jNV0Y7hi++yUnY4gi+cZ+37mmIgaLW4ecFjN2XD6dWjeaaSKB2KNn XtKFqKvL26BB3CgdeVw+Bj9J5VkltA/9iWt19vcuJmeUf2hJS2dYBfRoRt5atwZhfjEFgZ pY9NTslRwRSzaWZkXRB+T4YNFH5J/Ts= Received: from mail.maildlp.com (unknown [172.19.162.254]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4XtdBs1qkjz1V4dw; Wed, 20 Nov 2024 18:39:01 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id 819D01800F2; Wed, 20 Nov 2024 18:41:39 +0800 (CST) Received: from localhost.localdomain (10.90.30.45) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Wed, 20 Nov 2024 18:41:39 +0800 From: Yunsheng Lin To: , , CC: , , , Yunsheng Lin , Robin Murphy , Alexander Duyck , Andrew Morton , IOMMU , MM , Jesper Dangaard Brouer , Ilias Apalodimas , Eric Dumazet , Simon Horman , , Subject: [PATCH RFC v4 3/3] page_pool: skip dma sync operation for inflight pages Date: Wed, 20 Nov 2024 18:34:55 +0800 Message-ID: <20241120103456.396577-4-linyunsheng@huawei.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20241120103456.396577-1-linyunsheng@huawei.com> References: <20241120103456.396577-1-linyunsheng@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.90.30.45] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemf200006.china.huawei.com (7.185.36.61) X-Rspamd-Queue-Id: D562A18000A X-Stat-Signature: w3xb76g7qxf5c9j8em8qwj44ck98qq5d X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1732099251-665470 X-HE-Meta: U2FsdGVkX19tu5b/vWnr6gmE3j+vbP0aVrx1SM3yAFRi/E/4DGI+88BIJm7UI/vhnj/VjfcDJjUQmYWWCSk/ZQNoUOuURlsl6YMfbp7BNAEsGc7ExWS0EoaVx6ATU1dVULoMtZIwslgZw8h3yP7hVv4zetyipfdA6LR6YcoqOBnVTLQzL/tIIsvFIa4/aA+Iy2qTOkN7uEm7QOmaBoRl8pH8dXn/ecg2vCd90CHiHrVFzm4jNCoKmakPaY5SjEa6QO1/UKaU+YR+W7smWhSWdQ2xGjwW1o7Yr7hm8Gk4BpiF/OpZ6mzJWIr+iauJVCeUwL6jAfL3hipy6SG+gXO+/1CzNA/R/Lt4XqKYMEeeL2fU8FlctnzLntdKVQhQAxxn1rixot3fF+PRJyJ3OlK3qPpGHKHXtOOmX79ReZiCcp9RQBZEn0kqrw36apNdNqsLeAPYBa3RkVSsoJvAMttxOmLRchcwlo+GRn752T+jb3bUe/xRILZO6CibCY3RrQq+ooMOA/rqTMcX2aDv1ODIJex9HxFJB6P8gT+GMEstPg6C/n4gxVwlcLqSYnjWAdYRehNdgNx5eNr/deIVsnOubDyx3oyvD7K9MKM6mZzlBJh8uXXlgLMAp2GFG2E8H0tPWGtcQbDg32GGIWOqnPWcM2yAJpjhAWOQO8XL/G8FItGRSKPIbhYU9N8MlNQSykWLlujDn+AtHfBQNmJZisEMR4l0dpYpMHUaaGCUtYmBZX4FTJTemBY93YdmMAks4hqZN7uQc4Ty7DGo4PUjH4Rs49rN3cF99eHnx01iqzVCh6p2KPRYlfupFWmUHXI1VWBL5iKdH2/W1urZnF3VexGbyROtft/LATf6eigl7ni1QO/KFlG2I4Zh78c0zw+jsffGDezp7XFh2JzCJgePcVDxc3X/mMvuxQX3tHxe+v3FMWQ7ygW6t5HBZzhSdJeEclrKRj0T8KG+m9E/lESly4+ PIoxuufe FxnAT2OlDDdygdpLlTqAhoIFGco/VQiTfuVObPUgMqFy77vRf1BnaQ9CotLhb+nEUd+2Dfd98rrZqvO14BzX2jjiGp82JaB/CNdtqlAcIIusf6Lqmc2mLUMSVFiJczVL4o43XOCaoNNen1vObgH97iStb9k6Qo8/dmdl6leR21UtY/p5ANQVL1TjtQwYhv26KXmBn1SB6corr4J/N4yDWZqQA0QweXOVDhiYA+NN7DPGO8uxp/MEXdPXs9R2YHJ1B5LUZtoJfdGYcPPtnc02KIn2J3Vd+af+12VOFrCG4J28kVl7FZJz2j1dBO2/112zqVCwF17wKoYufeKOAJnP3dRFwRta4aFXNJURHSRJIXLhLQZb9V8JniuifBWKHJAQnbV870DOFC6QPxF2Vp+YI57C8Ek3pcNN9wHmNsUKme1BF7cR7ombx5Ql9GHkzfaJdZ1BgJe/2C56rocVZ1rrMThIk9WYaJVt/nzfsbjfW18LJpY4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Skip dma sync operation for inflight pages before the page_pool_destroy() returns to the driver as DMA API expects to be called with a valid device bound to a driver as mentioned in [1]. After page_pool_destroy() is called, the page is not expected to be recycled back to pool->alloc cache and dma sync operation is not needed when the page is not recyclable or pool->ring is full, so only skip the dma sync operation for the infilght pages by clearing the pool->dma_sync under protection of rcu lock when page is recycled to pool->ring to ensure that there is no dma sync operation called after page_pool_destroy() is returned. 1. https://lore.kernel.org/all/caf31b5e-0e8f-4844-b7ba-ef59ed13b74e@arm.com/ CC: Robin Murphy CC: Alexander Duyck CC: Andrew Morton CC: IOMMU CC: MM Fixes: f71fec47c2df ("page_pool: make sure struct device is stable") Signed-off-by: Yunsheng Lin --- net/core/page_pool.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 33a314abbba4..0bde7c6c781a 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -712,7 +712,8 @@ static void page_pool_return_page(struct page_pool *pool, netmem_ref netmem) rcu_read_unlock(); } -static bool page_pool_recycle_in_ring(struct page_pool *pool, netmem_ref netmem) +static bool page_pool_recycle_in_ring(struct page_pool *pool, netmem_ref netmem, + unsigned int dma_sync_size) { int ret; /* BH protection not needed if current is softirq */ @@ -723,10 +724,13 @@ static bool page_pool_recycle_in_ring(struct page_pool *pool, netmem_ref netmem) if (!ret) { recycle_stat_inc(pool, ring); - return true; + + rcu_read_lock(); + page_pool_dma_sync_for_device(pool, netmem, dma_sync_size); + rcu_read_unlock(); } - return false; + return !ret; } /* Only allow direct recycling in special circumstances, into the @@ -779,10 +783,11 @@ __page_pool_put_page(struct page_pool *pool, netmem_ref netmem, if (likely(__page_pool_page_can_be_recycled(netmem))) { /* Read barrier done in page_ref_count / READ_ONCE */ - page_pool_dma_sync_for_device(pool, netmem, dma_sync_size); - - if (allow_direct && page_pool_recycle_in_cache(netmem, pool)) + if (allow_direct && page_pool_recycle_in_cache(netmem, pool)) { + page_pool_dma_sync_for_device(pool, netmem, + dma_sync_size); return 0; + } /* Page found as candidate for recycling */ return netmem; @@ -845,7 +850,7 @@ void page_pool_put_unrefed_netmem(struct page_pool *pool, netmem_ref netmem, netmem = __page_pool_put_page(pool, netmem, dma_sync_size, allow_direct); - if (netmem && !page_pool_recycle_in_ring(pool, netmem)) { + if (netmem && !page_pool_recycle_in_ring(pool, netmem, dma_sync_size)) { /* Cache full, fallback to free pages */ recycle_stat_inc(pool, ring_full); page_pool_return_page(pool, netmem); @@ -903,14 +908,18 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data, /* Bulk producer into ptr_ring page_pool cache */ in_softirq = page_pool_producer_lock(pool); + rcu_read_lock(); for (i = 0; i < bulk_len; i++) { if (__ptr_ring_produce(&pool->ring, data[i])) { /* ring full */ recycle_stat_inc(pool, ring_full); break; } + page_pool_dma_sync_for_device(pool, (__force netmem_ref)data[i], + -1); } recycle_stat_add(pool, ring, i); + rcu_read_unlock(); page_pool_producer_unlock(pool, in_softirq); /* Hopefully all pages was return into ptr_ring */ @@ -1200,6 +1209,8 @@ void page_pool_destroy(struct page_pool *pool) if (!page_pool_release(pool)) return; + pool->dma_sync = false; + /* Paired with rcu lock in page_pool_napi_local() to enable clearing * of pool->p.napi in page_pool_disable_direct_recycling() is seen * before returning to driver to free the napi instance.