From patchwork Fri Feb 14 17:07:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13975389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 09B5AC02198 for ; Fri, 14 Feb 2025 17:29:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=64rPo0typhWn9EiuPyy3CdGiIksUJQiXRmT4BBfuuqs=; b=WSDG940NOl8vt8A7INP+79zTpW aLXmoViw1DCLEFmHME12ACodmj2rAbbhjRXQRbRFMTTjxDy7se0s4Ejg2/vLnIsuTOyh1mX7ibGb+ RlRCv39euX1zpN7RlcNufEI4Z+vraJK5c7vtjF5uOsAVzK3PV08Ah/68E3wCs2ylDJhTcQvNz34sj UtvMQWvJ0KTyvFv/IZa6evmuIhgIqK4JWnS7zmmSxIu/40WKbhukOrrTbJ8KRXbHLCVABRaMVSrXU 1k0IctjiV1yytp6ivbm5bm3cHBgvhcklUxLo63EdFwdRibLPiK5j2Y5tjsiG+SUr65O5CzCXFaO8l Ex2rPMwg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tizV9-0000000FiLl-3QcR; Fri, 14 Feb 2025 17:29:19 +0000 Received: from mail-mw2nam12on20604.outbound.protection.outlook.com ([2a01:111:f403:200a::604] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tizAn-0000000Fctu-2tfI; Fri, 14 Feb 2025 17:08:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tD2DaKnAq7CAgWpgCwaPaOy+m3HR7slhc6tOU2bzr2VIu3dbiFSh3hJtSNaLCKHdxEygy68TFYe22aoD8a1bQjHnYHWnhO0hjDAT7M7UlIIOv03ZXBGgEfj2C/NFkD+sjmkJlTV5HB6ikF28WzDK0STBtzJr3uH9ngOlYS65RHihXpLDOAAq5Ll0dbEKilzPFkdFkuVr/N9CGrENj8XJ3U+9bdZKd+zGwzBRJaBiC/Ek6eqnuvV1Q2iWsnaInQRFHf0Ta9vUyoXQC3R8jRFNQvm/JXY4nxbizUh8Z2fbxj6HDW4FoVvL4M0QVThy1+V24e1sDmPGWprEEQfb254qBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=64rPo0typhWn9EiuPyy3CdGiIksUJQiXRmT4BBfuuqs=; b=Sr1E76LJYFSnHhGn1cczx0oe66v5x6x7vlZu4wU76ZNVI+v7AnJ8rpnJLdT7giohltbWBgSM//j68L8cFpi7kUafYstwyziIzlHaX57alRYOCNw2m6eIUGXUgSclHCgCB8vOrJUEdOM4aMaBrxCktsfjznkRT0pBTT7CT9L7RTQckNk7JYlYZlDWUYbCHnK258dKh9csSvOkosBM6KQCTOoB6qMl8j8GePrU0Hrb2ZEqw31i5MF51MtEXhxvGJ0xtDkiWfLqanGYFdM8W2mAGAL91KJ8/rkFkZLCC2QS34QqayJ6IWM8x/AHYz83nWPku9OAr4S+kAcCCcDu/XNTew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=64rPo0typhWn9EiuPyy3CdGiIksUJQiXRmT4BBfuuqs=; b=TywOvOYo2Meqx52g4UUGQtOpYncU+03hjoGBWpqsXQJjx4hzMZV25c6nmDGbSO9/E5dB8nDCBylH2aumunHwXKMVm8qbtlqury43um9+B/fCrfro8NzKnaTYJsmxP84i2y0I9dVIIMXgiv4cYCpUL7EM99YW9IfVKt0L/DO5virZiJY23NdxVC1pecg4EHiOcqyrUByRL41p37kTdhDJbZnrs92AVq0/GFagvuNZZy/e2RMq0NqM/55g4a92ryzA4JZZ+LfD+ucbhfRd92oeYzrQxLOdW94NEq4qsSd/1QcWoxjqQ/7PdY1mJMh+w+9UVU5fOmr9/EWVC+8kB093SA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CH3PR12MB8659.namprd12.prod.outlook.com (2603:10b6:610:17c::13) by PH7PR12MB9175.namprd12.prod.outlook.com (2603:10b6:510:2e6::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.15; Fri, 14 Feb 2025 17:08:08 +0000 Received: from CH3PR12MB8659.namprd12.prod.outlook.com ([fe80::6eb6:7d37:7b4b:1732]) by CH3PR12MB8659.namprd12.prod.outlook.com ([fe80::6eb6:7d37:7b4b:1732%6]) with mapi id 15.20.8445.013; Fri, 14 Feb 2025 17:08:07 +0000 From: Jason Gunthorpe To: Alim Akhtar , Alyssa Rosenzweig , Albert Ou , asahi@lists.linux.dev, Lu Baolu , David Woodhouse , Heiko Stuebner , iommu@lists.linux.dev, Jernej Skrabec , Jonathan Hunter , Joerg Roedel , Krzysztof Kozlowski , linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-rockchip@lists.infradead.org, linux-samsung-soc@vger.kernel.org, linux-sunxi@lists.linux.dev, linux-tegra@vger.kernel.org, Marek Szyprowski , Hector Martin , Palmer Dabbelt , Paul Walmsley , Robin Murphy , Samuel Holland , Suravee Suthikulpanit , Sven Peter , Thierry Reding , Tomasz Jeznach , Krishna Reddy , Chen-Yu Tsai , Will Deacon Cc: Bagas Sanjaya , Joerg Roedel , Pasha Tatashin , patches@lists.linux.dev, David Rientjes , Matthew Wilcox Subject: [PATCH v2 07/23] iommu/pages: De-inline the substantial functions Date: Fri, 14 Feb 2025 13:07:40 -0400 Message-ID: <7-v2-545d29711869+a76b5-iommu_pages_jgg@nvidia.com> In-Reply-To: <0-v2-545d29711869+a76b5-iommu_pages_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR02CA0043.namprd02.prod.outlook.com (2603:10b6:207:3d::20) To CH3PR12MB8659.namprd12.prod.outlook.com (2603:10b6:610:17c::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB8659:EE_|PH7PR12MB9175:EE_ X-MS-Office365-Filtering-Correlation-Id: d6ec3555-f750-4023-fa78-08dd4d1a224e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024|7416014|921020; X-Microsoft-Antispam-Message-Info: Xg4xiXQxEwGa3HHNKGWuoE/pEA9D9WeHwrCnSERfcVFymZTIYPicK3dcoRNPMFbqbtC75uJPaQn2Nt8/XHSgRoEcCGJmag+JUXafeo8j9FMwwgPMebg/Bmp09Jh6lzEkY9wLI9ilKhJIHDjXk+C+LgL3PBn/gjorHv+kN+paJqkOv7RxqkS8Rd64J+Gp/xtTXScqlIXPpUs/S2Ljii9srArswGP9qQhjgGyEQJJEeSIorbMvAie/pSpDOepVB0EpVqRfhtBpSE81Qm/raLgDix2m9BDFdXLmJFas/idKHpm7Q3bpA6WczKASjfQk8eBkrOmLN8MoOOtiRBQaq7heThjapQitVaBrAYv7HoEhOicxqIUHj5KYmcScw9yYuaW82cDt4K3D1dKH5OMPxxl8CAoe5PScvs7z+nvQT6Rb0dM4YOr+n4PeWAs/5le9pZbXAPQSB3yabu7vmDe9VZnn41nJ1TJGU5WYs5yVZSmjmZqjyPfsu/XaIskALBNdEZcahnq2VA7Vj5EKfHW2xkBB4MtjV+CjHztMV7GwkviFWLgCGe4Kw2zlZH+dCWxeIWWcOL0tHury/W7QZlkMKfGT1eXEAncUkxC754TbuNBDHyU9cWvnluJ62VRglnlsSmZ2AhJooy1kWMYOLPVV0fcZYmLHYXycdJ+xfn/471aKUMuuKXb2eM4cZMrngGGm1fjJ2mnZS2lMNBqiQ0/Sz//9USAQLlrlmyfRdwidThJXPUN1XAXf8rspY7ZhOKExouWX7QTsOLzVju9eUI/dm/xlu+qZfXKb6k9P2oH1u+fIh5vLAXbqpeDEHf7TC+ayBoDzEMKEMR14MZ5iJi1MGMxz3fuMfSJiPgUe+Y4UrS/EBWJEhZ+Gj9QCjEOV5YhXeIeBSlGOvmt33VXUc7q7L99Old7kKloOAmgQZgEPMEqGWDrv7aEIzjVwuHQ/i+TM1A41kus8iGsazvUW+DdejCqukNFc2rUdBwaGt6f2t55omMcB0QQdorpNkviGBuEtV7yCh1w5L1Yx8DSWI7Sk5EiVQYfeqtz1p4lpIA8I5MjHLY2QOYKyzKsf+W/V0qy9OjQvDRKuQlWvwx92nIl/SjGs2JkVn9bNKIEjAEWx3HOyb/J3e6I1aQKX3Y07RfvGmrmTnqItrpzdNwiwS8+7C/PgbyxZXBhU+i/K9ryoRHyddp58H3nPoqYExKF8mh3D+g9dE3mmKo22+xlmPj4oDrpqLW3r7lGnpXOe3HEDzhZ50LqivFqaUiXwysEY4kHeVQmve4pCJ3sCRy28k95+7K6/369/6SFsO5yhZSO6/kJ1nU96FPWq4QN9JBeuICgDCyykShJ4UKwoHgzJGHCFiHTkULV9QGSa4gPOweH48rWe9TWdwLKA44ATQhZoJhj93OHbuhQe1yF7F2ObU5dotE/IHsRn24rjbW9a02Pb2V1qZFU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR12MB8659.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024)(7416014)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: FiTJrMUg6ouAnII6HK2NfNi6YoK//SNB34dpc9xtEIt1vchJvn7se7q6wS+/U9+qV7T3e/jZZ2rl6qPxr009XuxFFvV54QQjdZSziAX1VcbhKjuHXGsaXLEK0DiZ/JEF9ZLXYTBo9W8lHEee57oF3mpLeQP1JiVdNogIJ+iVX7BfDm3rLgjqbAVy7jo8i6JICkXY54waQuNPy5YzBuZWcawrY67SZtuY9sH6Uh6CG6oJHhEvMAiEsqvI2A2Lye99J90kjQpLEQP9tegTQfkPHmqNflBoSoZQtBHHlxWHd9jtLOL6Cji0cwsRMnf3d3PL4rJR8GxnPcNTNfMl7B+h9LUM0sXifkF720bcmEvCx9RAu6onPtsup0urjucLUhjWTySyRxtB/1N+HN7Y2gTAZSRnK+FyhATrJ5ZKHU5jl8RjSQmEcN6YDOzEb/6iWsEH7dC6PTkUd/fw8/1oMul4eYrlfzP7QTiCv4WfkLhnIRS5vv7y6JybvdBUkljxSDdJ0Z+VvyeGJJGTld6zVPc2N2mDkkaNlCI5tZSCipJ26eOG398Dno+ZRaVMLp/RM3tvo275Ed+DwVChJgfb2wGPrdssxXEWXNtaNYgvQtpj9gPKlPCshEadMukazCaEWEC/e9sw+7A/t5yfXXdv4I8no0V3vTFYrbOJtcAfNMi1i/ZrxJJpZYKwdN2c0/WMZgPlTHlq/Y0n8DqFDbUSrfmsl1xaJOxHug8TSsrmoLQsPxC9GoOf5H1in7Ja7XJbPE1b2OXxw+nIruYLs5eKQTtJiWzKW2X2684H7N9hPzu4w5iXBS4KFaCt4cCIkzoq9xPgY/5oQO93GvfCw/zNoR1yQ2t6fuLi5HMNOfwjbHkb4Y8dhEuYOsD2dXXFrlY2BVPmmZwRDrFtiAb4WRcwPtwt5eFql8uPnlEDAQ55fOGpeeeBJGIvQD3Mu4m16EFXn3Bw4QfQuuSG/6qPIarWDudI+WSF2fSQMmaOd95biUD3S9kB1tDmad/4m+rAr0KLnme5PZI6M00+wWCtPNTT7OXW/f9WeyuNRMIub5xQslo214Eod5nN/uS2Qb3bfvadCArCQINONPSHLFLL7nbBQ2mWB45FUEFmnKUKcXn6rR6w+CDoyCdqdb3LNISCOYHzgd/Lyjuf7IzimjHw7SbmjNX9z6S14RxQr/+imhtu/VKz2PUAl4HPdHsj7KzTDBbf8j5f0WNW2OBNTfkq0xcKz6XuDOIjHN1DVgPvLstPkhdLnwLIPTSIFgjmOz4+HaA+mDZ13EG5xE2wisxxtTLYN0verJUWxJdzUj68NCilKA3GVIQBqRNOTuuUhw1AQoLnL+0KVqIJp9AaKu2vg+AIGUY0sQBTiHBPMKrY80UyB4Wk7nYobwbZZU57naTZEvj7L4Am03kU+vLwjI44vpYlPtLPELvBhjiggP1h9sJmzVl7hE5voSfoNIG87LBYKaeC1lKqNHU2WbwQYRGpN3F/1jmcf4xdVwKKgKJe3axJwd82d3aEYWuRXud0HM70A83zqotXGhoSon9iE6bBEMzSzK9/82YowbQ+Q5ZghPmczH1dTnM= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: d6ec3555-f750-4023-fa78-08dd4d1a224e X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8659.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2025 17:07:59.8871 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XuziWKnua4+EhAuKTkUsS1TpltADr7BHwdBw/7whRtB+xurVQKUC+y8OIh4tABtG X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB9175 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250214_090817_787890_C1A0AF1E X-CRM114-Status: GOOD ( 20.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org These are called in a lot of places and are not trivial. Move them to the core module. Tidy some of the comments and function arguments, fold __iommu_alloc_account() into its only caller, change __iommu_free_account() into __iommu_free_page() to remove some duplication. Signed-off-by: Jason Gunthorpe --- drivers/iommu/Makefile | 1 + drivers/iommu/iommu-pages.c | 84 +++++++++++++++++++++++++++++ drivers/iommu/iommu-pages.h | 103 ++---------------------------------- 3 files changed, 90 insertions(+), 98 deletions(-) create mode 100644 drivers/iommu/iommu-pages.c diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile index 5e5a83c6c2aae2..fe91d770abe16c 100644 --- a/drivers/iommu/Makefile +++ b/drivers/iommu/Makefile @@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-y += amd/ intel/ arm/ iommufd/ riscv/ obj-$(CONFIG_IOMMU_API) += iommu.o +obj-$(CONFIG_IOMMU_SUPPORT) += iommu-pages.o obj-$(CONFIG_IOMMU_API) += iommu-traces.o obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o diff --git a/drivers/iommu/iommu-pages.c b/drivers/iommu/iommu-pages.c new file mode 100644 index 00000000000000..dbf7205bb23dcc --- /dev/null +++ b/drivers/iommu/iommu-pages.c @@ -0,0 +1,84 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, Google LLC. + * Pasha Tatashin + */ +#include "iommu-pages.h" +#include +#include + +/** + * iommu_alloc_pages_node - Allocate a zeroed page of a given order from + * specific NUMA node + * @nid: memory NUMA node id + * @gfp: buddy allocator flags + * @order: page order + * + * Returns the virtual address of the allocated page. The page must be + * freed either by calling iommu_free_page() or via iommu_put_pages_list(). + */ +void *iommu_alloc_pages_node(int nid, gfp_t gfp, unsigned int order) +{ + const unsigned long pgcnt = 1UL << order; + struct page *page; + + page = alloc_pages_node(nid, gfp | __GFP_ZERO | __GFP_COMP, order); + if (unlikely(!page)) + return NULL; + + /* + * All page allocations that should be reported to as "iommu-pagetables" + * to userspace must use one of the functions below. This includes + * allocations of page-tables and other per-iommu_domain configuration + * structures. + * + * This is necessary for the proper accounting as IOMMU state can be + * rather large, i.e. multiple gigabytes in size. + */ + mod_node_page_state(page_pgdat(page), NR_IOMMU_PAGES, pgcnt); + mod_lruvec_page_state(page, NR_SECONDARY_PAGETABLE, pgcnt); + + return page_address(page); +} +EXPORT_SYMBOL_GPL(iommu_alloc_pages_node); + +static void __iommu_free_page(struct page *page) +{ + unsigned int order = folio_order(page_folio(page)); + const unsigned long pgcnt = 1UL << order; + + mod_node_page_state(page_pgdat(page), NR_IOMMU_PAGES, -pgcnt); + mod_lruvec_page_state(page, NR_SECONDARY_PAGETABLE, -pgcnt); + put_page(page); +} + +/** + * iommu_free_pages - free pages + * @virt: virtual address of the page to be freed. + * + * The page must have have been allocated by iommu_alloc_pages_node() + */ +void iommu_free_pages(void *virt) +{ + if (!virt) + return; + __iommu_free_page(virt_to_page(virt)); +} +EXPORT_SYMBOL_GPL(iommu_free_pages); + +/** + * iommu_put_pages_list - free a list of pages. + * @head: the head of the lru list to be freed. + * + * Frees a list of pages allocated by iommu_alloc_pages_node(). + */ +void iommu_put_pages_list(struct list_head *head) +{ + while (!list_empty(head)) { + struct page *p = list_entry(head->prev, struct page, lru); + + list_del(&p->lru); + __iommu_free_page(p); + } +} +EXPORT_SYMBOL_GPL(iommu_put_pages_list); diff --git a/drivers/iommu/iommu-pages.h b/drivers/iommu/iommu-pages.h index fcd17b94f7b830..e3c35aa14ad716 100644 --- a/drivers/iommu/iommu-pages.h +++ b/drivers/iommu/iommu-pages.h @@ -7,67 +7,12 @@ #ifndef __IOMMU_PAGES_H #define __IOMMU_PAGES_H -#include -#include -#include +#include +#include -/* - * All page allocations that should be reported to as "iommu-pagetables" to - * userspace must use one of the functions below. This includes allocations of - * page-tables and other per-iommu_domain configuration structures. - * - * This is necessary for the proper accounting as IOMMU state can be rather - * large, i.e. multiple gigabytes in size. - */ - -/** - * __iommu_alloc_account - account for newly allocated page. - * @page: head struct page of the page. - * @order: order of the page - */ -static inline void __iommu_alloc_account(struct page *page, int order) -{ - const long pgcnt = 1l << order; - - mod_node_page_state(page_pgdat(page), NR_IOMMU_PAGES, pgcnt); - mod_lruvec_page_state(page, NR_SECONDARY_PAGETABLE, pgcnt); -} - -/** - * __iommu_free_account - account a page that is about to be freed. - * @page: head struct page of the page. - * @order: order of the page - */ -static inline void __iommu_free_account(struct page *page) -{ - unsigned int order = folio_order(page_folio(page)); - const long pgcnt = 1l << order; - - mod_node_page_state(page_pgdat(page), NR_IOMMU_PAGES, -pgcnt); - mod_lruvec_page_state(page, NR_SECONDARY_PAGETABLE, -pgcnt); -} - -/** - * iommu_alloc_pages_node - allocate a zeroed page of a given order from - * specific NUMA node. - * @nid: memory NUMA node id - * @gfp: buddy allocator flags - * @order: page order - * - * returns the virtual address of the allocated page - */ -static inline void *iommu_alloc_pages_node(int nid, gfp_t gfp, int order) -{ - struct page *page = - alloc_pages_node(nid, gfp | __GFP_ZERO | __GFP_COMP, order); - - if (unlikely(!page)) - return NULL; - - __iommu_alloc_account(page, order); - - return page_address(page); -} +void *iommu_alloc_pages_node(int nid, gfp_t gfp, unsigned int order); +void iommu_free_pages(void *virt); +void iommu_put_pages_list(struct list_head *head); /** * iommu_alloc_pages - allocate a zeroed page of a given order @@ -104,42 +49,4 @@ static inline void *iommu_alloc_page(gfp_t gfp) return iommu_alloc_pages_node(numa_node_id(), gfp, 0); } -/** - * iommu_free_pages - free pages - * @virt: virtual address of the page to be freed. - * - * The page must have have been allocated by iommu_alloc_pages_node() - */ -static inline void iommu_free_pages(void *virt) -{ - struct page *page; - - if (!virt) - return; - - page = virt_to_page(virt); - __iommu_free_account(page); - put_page(page); -} - -/** - * iommu_put_pages_list - free a list of pages. - * @page: the head of the lru list to be freed. - * - * There are no locking requirement for these pages, as they are going to be - * put on a free list as soon as refcount reaches 0. Pages are put on this LRU - * list once they are removed from the IOMMU page tables. However, they can - * still be access through debugfs. - */ -static inline void iommu_put_pages_list(struct list_head *page) -{ - while (!list_empty(page)) { - struct page *p = list_entry(page->prev, struct page, lru); - - list_del(&p->lru); - __iommu_free_account(p); - put_page(p); - } -} - #endif /* __IOMMU_PAGES_H */