From patchwork Tue Feb 4 18:34:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13959791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E67E6C02193 for ; Tue, 4 Feb 2025 20:01:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=MBzLypQ9el0mk86gS0hss1rPgtw8saK5TrgmtmxW/bY=; b=xDzl7dJpXPBylv KNlUa4k5KfjDWcnqX6byOh0WwPrENQnIdIom1VKgMMJNUTvFVMhW7c4Uid9kMSborBWSAP6RRcgRj 2v3UyOQgwocidobMhqGBGhQpWH5sePAaUwezxp1zUS40RTPDT39UAODGa/ZFvNJszNSmqjxq0WkdV 8lY3QvLUBB3V5NEzJy5mLwuiIVYHqfOWYj0OYfCjPuhldmm6KgxN0hnhoGUxkmG+XIZLAP/AwLlQZ hgCgQa5s6NMyAKrAFvHS+sU5yq+g9OntF/m9p4HfUk/TN+agcAbkHYofW9TAh0pyq7nLY92edlNuq iS5//qiLwGj4biZwAXxQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfP6j-00000001Srz-24Wx; Tue, 04 Feb 2025 20:01:17 +0000 Received: from mail-bn8nam11on20617.outbound.protection.outlook.com ([2a01:111:f403:2414::617] helo=NAM11-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfNlY-00000001EOm-0SmA; Tue, 04 Feb 2025 18:35:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tvVGHSdZMlK/hMF+WOKUpqGRe+Agz0X80DH05ep7d0/na/CNgNde6qX3dFYJ0tzV+DdrK58l/Eu+kNjC7N+XzqqAhNOe4YDnMM2TES2NK5EIVfoc43fBqp9hgUlvJKCaySCCtbPsyJ5JuI09nPuVvlR5RcB26fyBc7mABMK6vskar/h7hjqRks9EY4VwtK/XPiHx48TFPYtwKgCjxu4qtAe4xupkWlV+rwm50madVu5xK2EELLANaUEr+MO4/q36sVziLQFOzw+JYZ42DTXfzyhyh6pAk4jQXxT7AR6k2MrXgTTZA9phqUQHfOqn6TS3kRccZ4EK6mzDZwZZ8ttUfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xAlfrFK8+1fOZhSDoLW+fnhg1BB82j87BA7quiVLBas=; b=xlXsx0KvVLj/HJ2Hc62Rgv99br/iKeBRMkxbJCbMKK62yqqkc5gfnZbvU15INP2B/G/iJqSdei+wc0ej5Mmm+oNRgXR3mbOLoVajaRnbSKRNjzwg6gERHi9XP9VW6HgPEIwA6NyM1mbXiBBbBy8ZuJdV5uTeQshQ6R0+YsBTk928mJ2R7vhPKo5KMy6g5J2iejhuKQ1SarkH69JGIyI25k9eRiU++UNrp83seG5H9l8S64VNs7vnijTFOJ+Ep88dQTYEjKqTslHRERyTN/Blad2Lo6nBvinVtWuDG092PD9forvQHAKmwQTlbrya3nzFylaJignLH8pva80Kx8yDGw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xAlfrFK8+1fOZhSDoLW+fnhg1BB82j87BA7quiVLBas=; b=jyN+MUlkQ+ptifqdrL7Rxe3cMfsGs5FpIl/eVWQc/Vn+wblfdEFkoScRjhunx++F8Y9NUskL5UBRKiY9QjVP3k0Vr4sHtukOK0P/5UGgJ7rsrqOR9j/ay5aQ8vNOZAufCF330AI65ZOw+ZLEA8kWqTBgDebaNJSnjFlSPw3e7eIt7Z5wsQUPudmdmZISI/b/MEiQUXXS3LAZy1e00wHgTeVdY1BXsL1/R/5jrxaANsR82umGxG40L1OhINYZmUUe4Uzq9Z0V5CHy6HWNWmJ7HudLrcqqNnmKltndH1YoymAUUv4QT4KJ2h+QkxLn6jxV0+3ndEVnqsq0K3gOvJwsEg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CH3PR12MB8659.namprd12.prod.outlook.com (2603:10b6:610:17c::13) by IA0PR12MB8226.namprd12.prod.outlook.com (2603:10b6:208:403::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8398.24; Tue, 4 Feb 2025 18:35:11 +0000 Received: from CH3PR12MB8659.namprd12.prod.outlook.com ([fe80::6eb6:7d37:7b4b:1732]) by CH3PR12MB8659.namprd12.prod.outlook.com ([fe80::6eb6:7d37:7b4b:1732%6]) with mapi id 15.20.8398.025; Tue, 4 Feb 2025 18:35:11 +0000 From: Jason Gunthorpe To: Alim Akhtar , Alyssa Rosenzweig , Albert Ou , asahi@lists.linux.dev, Lu Baolu , David Woodhouse , Heiko Stuebner , iommu@lists.linux.dev, Jernej Skrabec , Jonathan Hunter , Joerg Roedel , Krzysztof Kozlowski , linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-rockchip@lists.infradead.org, linux-samsung-soc@vger.kernel.org, linux-sunxi@lists.linux.dev, linux-tegra@vger.kernel.org, Marek Szyprowski , Hector Martin , Palmer Dabbelt , Paul Walmsley , Robin Murphy , Samuel Holland , Suravee Suthikulpanit , Sven Peter , Thierry Reding , Tomasz Jeznach , Krishna Reddy , Chen-Yu Tsai , Will Deacon Cc: Bagas Sanjaya , Joerg Roedel , Pasha Tatashin , patches@lists.linux.dev, David Rientjes , Matthew Wilcox Subject: [PATCH 06/19] iommu/pages: De-inline the substantial functions Date: Tue, 4 Feb 2025 14:34:47 -0400 Message-ID: <6-v1-416f64558c7c+2a5-iommu_pages_jgg@nvidia.com> In-Reply-To: <0-v1-416f64558c7c+2a5-iommu_pages_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR14CA0028.namprd14.prod.outlook.com (2603:10b6:208:23e::33) To CH3PR12MB8659.namprd12.prod.outlook.com (2603:10b6:610:17c::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB8659:EE_|IA0PR12MB8226:EE_ X-MS-Office365-Filtering-Correlation-Id: 10ae9e99-4ea1-4802-76d6-08dd454aa4a0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|366016|921020; X-Microsoft-Antispam-Message-Info: n9qXpLPioRXiyQRkFOYZvzTqFFm0QtaEPCLwnLKBWxWfQ3RpxLcFZroCAmYp4yx+kHPjjHCEqM2uz+D/ppAzcru70+tBH6v1J5cll7LgHG40qGJGhYXQVsYrAk0pabDyWC0xaw6N7yEQPNb9an38Q1x0N8s7dOt7DGPuRnxHvBsTCJyQYeTOcqm7FGT/NxOuoHD7Jgl8z+vNbenRBIavoeym0VtkaKsV6Ek4MpEVZtBBOoy7uSUi6baAUBB1PsFxq5qk2F3ISg1fXkLWl5n2vOrOL4ka2md9eZnCQLBpcxH7BjnEGKTf4pSDHYnyBbP6ws6qdwWbROfHf3ISpIelPt/2nntg9UAs4BymYL0m34+QNLBT317urijPrXv3ZIVSREJsWHVywuTTTdGe3d5dYLB0lJteFxrrj3XYqtLBG73NlZFyZavuTWGI+0I1t2Im130SHlejCVkhecwvaU8vb9zQy/rELhVktZLXnRPESIpwV6ErReNSW5Vl0Zo4lhYFYVvzSoooEZVw9lprS0AuXDlJoOg+1IlX2hDaPTMeZj34nyYaGqiF+OvPtQexiuZZv0SVuHThDAv+8gwLOhrFh0xvzEtE49o2y0fDsKQeFUJSYi1CmUQGfbH/7UKRqPoRDoEBaeMcPZZOyDW90lFVDrmXYJQgauzPx47kHyp5EqiT9swH4EzjpotLu3g8jeJDmCTiWbQLqLLYk/t60iniH9e3Az0N/hIwqYqEzUhQxYykysEE84icT0M/YkO+QxpA7vwWuGiQ0vAYW31cIPzkQ5eR5HD8DTsSe8CHl9XiHmyZi0q5JytF+4pWQBnUokbTZXJ7tqJv2i3yiscfsaZee0/gsh8Omfyap3JdgEgIVFhLYiYu2DJfPfe02RnjRFvHN6PrFtV+i/6bjNHLIa4+OXLX0o53G6+H5r6wGhD5wuI7TtVvMO2I+Z4Ds0CCy3IJ7gR1pw741RQ1iogcwlMzKHFx+Oq+rLsmQu58YF8yPZMvPRRuvq0UHhKGhXbz6FJPUMmqUS08Vx6S/o9pGaPe/xK9ztqmxtrPZzNqmkJyFku5y2F4SPuOM4WnDug0DnTQ5V8OuCOGXZaj3xXHRiLu+ofEOLdZOUNLJvZ4BNDEhrs5gKPjfNhOoIEyxuyRntU0NQPD5WIdICUICpQJqZKXd9URYZgMxuXs82kxTzAFm3ObF3bHDHWFLn+ykvEUdPAglNOX6BgVQzLOP8BKOf50mjN34bbCZjIoYY556v+HYGmLD09nMZYgPkuYRL6rWr+dZHaM8suEeU4syK2WpNo/14D9nAr+5h5IZqaBhrWXaV2EzmN6WJF2eLas8nObjDZPi3xGJxjuSgwso99ilYd92TraWcXDoJkVHT8S6pwsvDzT+xsxeLx7yqTBu3pacFKg8C4g0NP9S9UsEWf1S512Xe84xYfBhVzp2dLaUPIT9y8= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR12MB8659.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(366016)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: y8b2DOF4lerJ0pH9OGtSZO9nnGpJ1K3o1yh1zeoA32LgBj6ZI/eWzCDj2bO2Ak6wcs0XsOeo4Vtx55fbwXEtsBswgv7Y7Txd5UFrBaE3cL/g5XBwRH7jdNlonuq7Ul17j5lG4+V+Uc2m9pOllBdCnB6yWRM18ze1SSqB75Uv/NnaYVosXUb/HhHJCqI5DvGHsETWEdJzJCF+nvHyo19pz2C+XQRsFWG+J6nAibXtxxs/oJju5GZOEPCVTKFEVxEO1H9J+a+w1wE3RK+EKzuA3afAcsdzw8ceY4UogAt7kQluWmLIlc1dykUSE92pD2mTHGL1+zyeo+r0+EvfSSSScPTP66NIBO/bgvWbCI3eU/JJrzVNA8eDElyl2boeQIqEqgSyg550NysmE0Po1xYMVg+RIXHesqGWNxE83m5SAiNHp7Kk+q/AsqpkXSEBlhPKYVD0OY23C2d7sS3tjVdgQC1EEVnTKV2+qBallcY+vjnRXGHgdMgZ/jO92LLVu6UzxnS+AKwjjfio8HH37yr8n2vbdxYq9xxe136CWxVyctMiBEFu1nSXV5ErmEhw/a1Y7pa+rhEV3U/uF8XOoLviLmzIB1/1Tcp7KuQ9KioEkjoFL6OY0iR067CJYICfWTgv/FaXTNNU/kMhwwDLlm/vWOIAjD3ns5tRs+uCcOC6486BZB2JiqNtTUAgMX92p89K7SktmN2goJCZReeG+zFGHERD0mctB+dL0EFyDU8TWt6Zu2UgzV8iRR1fDAnj5AZX5eaDNzlVNnEdhj1mMzGJeElBGb+jJ5X9PCvv9vqP9upZH9/LKH0OKQJkr7GpDaw3sCAIkDWC+kvgyQcR0SAinojOoI8lQboBvx3POv02fhHpQpzU7HTbdMrN4UX8/yxylelwgPAv5Maj7kJCrYyv2VTskpu6ffJBq2ZLR7H1hHdU2vIJlw7fFlyTJKvdFZ9KqrSC1143P0TEHIz1sA23ZJggU08meCF1XF6uXqMrWEw9OX1+Xz166j+o5CNcqOKD/OPDB0ZhkbRVH3SMaBa5b90856g8hV2X0kNUOQLVvH3xH7slErU5soSujd/I4doqrnoxwilG3JR/KLNfYyhEn0aq2FNvcLocV8DtF6zMKRy1ILoeq2RNOdLSthS8Y4tnyuNqO70OZg03nKs8TtlPk5ZCGnfy15fc+kqZzwj44roGCOyANMzU7S1xHZEHgkMYgTDcGniXo1hW2RMk4ajwxz1mhO+nGd1/6KkzO5w12R2fb6chysVGbiJOZzMH9PoB8rNjlo2lKW7uSiUAknYhi+a2oJbLalar6PbN4jwYa/4g28qrxQOCSX0o1u+DStsBS3EB36VrYH7uISmQ20dbLV9FoykZThFBGcuNzVsCfn7MUcabiqcV1Dy2mh47BUj450qRC5libL+pv6ONkOyxs9fNpXLWQiyjsGTir4fmkenSqlZDv3huCwbD7jtxLZl/j2CwNdHr8sgfCI6DMOpN1AC3gD1/480gOzyUxoxbbzzTuvLVnF00G2pFbJ/9bWebVNeNZ3RuxqtSZYnQkd42dJX2qPoFCIzZkLDRzjyz2qM= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 10ae9e99-4ea1-4802-76d6-08dd454aa4a0 X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8659.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Feb 2025 18:35:05.0347 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lp941imDkWsqoCsh1gjC+kvfnoAfedHOnlmMVrB47aBam2zoa4Z4eUFx6T+oZB8D X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8226 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_103520_290877_3289FE11 X-CRM114-Status: GOOD ( 21.40 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org These are called in a lot of places and are not trivial. Move them to the core module. Tidy some of the comments and function arguments, fold __iommu_alloc_account() into its only caller, change __iommu_free_account() into __iommu_free_page() to remove some duplication. Signed-off-by: Jason Gunthorpe --- drivers/iommu/Makefile | 1 + drivers/iommu/iommu-pages.c | 84 ++++++++++++++++++++++++++++++ drivers/iommu/iommu-pages.h | 101 ++---------------------------------- 3 files changed, 90 insertions(+), 96 deletions(-) create mode 100644 drivers/iommu/iommu-pages.c diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile index 5e5a83c6c2aae2..fe91d770abe16c 100644 --- a/drivers/iommu/Makefile +++ b/drivers/iommu/Makefile @@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-y += amd/ intel/ arm/ iommufd/ riscv/ obj-$(CONFIG_IOMMU_API) += iommu.o +obj-$(CONFIG_IOMMU_SUPPORT) += iommu-pages.o obj-$(CONFIG_IOMMU_API) += iommu-traces.o obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o diff --git a/drivers/iommu/iommu-pages.c b/drivers/iommu/iommu-pages.c new file mode 100644 index 00000000000000..0fece3758408ae --- /dev/null +++ b/drivers/iommu/iommu-pages.c @@ -0,0 +1,84 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, Google LLC. + * Pasha Tatashin + */ +#include "iommu-pages.h" +#include +#include + +/** + * iommu_alloc_pages_node - Allocate a zeroed page of a given order from + * specific NUMA node + * @nid: memory NUMA node id + * @gfp: buddy allocator flags + * @order: page order + * + * Returns the virtual address of the allocated page. The page must be + * freed either by calling iommu_free_page() or via iommu_put_pages_list(). + */ +void *iommu_alloc_pages_node(int nid, gfp_t gfp, unsigned int order) +{ + const unsigned long pgcnt = 1UL << order; + struct page *page; + + page = alloc_pages_node(nid, gfp | __GFP_ZERO | __GFP_COMP, order); + if (unlikely(!page)) + return NULL; + + /* + * All page allocations that should be reported to as "iommu-pagetables" + * to userspace must use one of the functions below. This includes + * allocations of page-tables and other per-iommu_domain configuration + * structures. + * + * This is necessary for the proper accounting as IOMMU state can be + * rather large, i.e. multiple gigabytes in size. + */ + mod_node_page_state(page_pgdat(page), NR_IOMMU_PAGES, pgcnt); + mod_lruvec_page_state(page, NR_SECONDARY_PAGETABLE, pgcnt); + + return page_address(page); +} +EXPORT_SYMBOL_GPL(iommu_alloc_pages_node); + +static void __iommu_free_page(struct page *page) +{ + unsigned int order = folio_order(page_folio(page)); + const unsigned long pgcnt = 1UL << order; + + mod_node_page_state(page_pgdat(page), NR_IOMMU_PAGES, -pgcnt); + mod_lruvec_page_state(page, NR_SECONDARY_PAGETABLE, -pgcnt); + put_page(page); +} + +/** + * iommu_free_page - free page of any order + * @virt: virtual address of the page to be freed. + * + * Frees the allocation from iommu_alloc_pages_node() + */ +void iommu_free_page(void *virt) +{ + if (!virt) + return; + __iommu_free_page(virt_to_page(virt)); +} +EXPORT_SYMBOL_GPL(iommu_free_page); + +/** + * iommu_put_pages_list - free a list of pages. + * @head: the head of the lru list to be freed. + * + * Frees a list of pages allocated by iommu_alloc_pages_node(). + */ +void iommu_put_pages_list(struct list_head *head) +{ + while (!list_empty(head)) { + struct page *p = list_entry(head->prev, struct page, lru); + + list_del(&p->lru); + __iommu_free_page(p); + } +} +EXPORT_SYMBOL_GPL(iommu_put_pages_list); diff --git a/drivers/iommu/iommu-pages.h b/drivers/iommu/iommu-pages.h index b15e0f85b3f739..6045e1f90a40cb 100644 --- a/drivers/iommu/iommu-pages.h +++ b/drivers/iommu/iommu-pages.h @@ -7,67 +7,12 @@ #ifndef __IOMMU_PAGES_H #define __IOMMU_PAGES_H -#include -#include -#include +#include +#include -/* - * All page allocations that should be reported to as "iommu-pagetables" to - * userspace must use one of the functions below. This includes allocations of - * page-tables and other per-iommu_domain configuration structures. - * - * This is necessary for the proper accounting as IOMMU state can be rather - * large, i.e. multiple gigabytes in size. - */ - -/** - * __iommu_alloc_account - account for newly allocated page. - * @page: head struct page of the page. - * @order: order of the page - */ -static inline void __iommu_alloc_account(struct page *page, int order) -{ - const long pgcnt = 1l << order; - - mod_node_page_state(page_pgdat(page), NR_IOMMU_PAGES, pgcnt); - mod_lruvec_page_state(page, NR_SECONDARY_PAGETABLE, pgcnt); -} - -/** - * __iommu_free_account - account a page that is about to be freed. - * @page: head struct page of the page. - * @order: order of the page - */ -static inline void __iommu_free_account(struct page *page) -{ - unsigned int order = folio_order(page_folio(page)); - const long pgcnt = 1l << order; - - mod_node_page_state(page_pgdat(page), NR_IOMMU_PAGES, -pgcnt); - mod_lruvec_page_state(page, NR_SECONDARY_PAGETABLE, -pgcnt); -} - -/** - * iommu_alloc_pages_node - allocate a zeroed page of a given order from - * specific NUMA node. - * @nid: memory NUMA node id - * @gfp: buddy allocator flags - * @order: page order - * - * returns the virtual address of the allocated page - */ -static inline void *iommu_alloc_pages_node(int nid, gfp_t gfp, int order) -{ - struct page *page = - alloc_pages_node(nid, gfp | __GFP_ZERO | __GFP_COMP, order); - - if (unlikely(!page)) - return NULL; - - __iommu_alloc_account(page, order); - - return page_address(page); -} +void *iommu_alloc_pages_node(int nid, gfp_t gfp, unsigned int order); +void iommu_free_page(void *virt); +void iommu_put_pages_list(struct list_head *head); /** * iommu_alloc_pages - allocate a zeroed page of a given order @@ -104,40 +49,4 @@ static inline void *iommu_alloc_page(gfp_t gfp) return iommu_alloc_pages_node(numa_node_id(), gfp, 0); } -/** - * iommu_free_page - free page of any order - * @virt: virtual address of the page to be freed. - */ -static inline void iommu_free_page(void *virt) -{ - struct page *page; - - if (!virt) - return; - - page = virt_to_page(virt); - __iommu_free_account(page); - put_page(page); -} - -/** - * iommu_put_pages_list - free a list of pages. - * @page: the head of the lru list to be freed. - * - * There are no locking requirement for these pages, as they are going to be - * put on a free list as soon as refcount reaches 0. Pages are put on this LRU - * list once they are removed from the IOMMU page tables. However, they can - * still be access through debugfs. - */ -static inline void iommu_put_pages_list(struct list_head *page) -{ - while (!list_empty(page)) { - struct page *p = list_entry(page->prev, struct page, lru); - - list_del(&p->lru); - __iommu_free_account(p); - put_page(p); - } -} - #endif /* __IOMMU_PAGES_H */