From patchwork Fri Jan 10 06:00:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 13933700 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CDAFE7719D for ; Fri, 10 Jan 2025 06:02:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F3EF16B00AA; Fri, 10 Jan 2025 01:02:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EEBA56B00AB; Fri, 10 Jan 2025 01:02:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCC1F6B00AC; Fri, 10 Jan 2025 01:02:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A6BD46B00AA for ; Fri, 10 Jan 2025 01:02:45 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 6B281A0868 for ; Fri, 10 Jan 2025 06:02:45 +0000 (UTC) X-FDA: 82990498290.03.5596037 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2060.outbound.protection.outlook.com [40.107.220.60]) by imf05.hostedemail.com (Postfix) with ESMTP id 937AD100002 for ; Fri, 10 Jan 2025 06:02:42 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="glHJs/oT"; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf05.hostedemail.com: domain of apopple@nvidia.com designates 40.107.220.60 as permitted sender) smtp.mailfrom=apopple@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736488962; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZaV7+Qz08NZ4CyqSz0ni0zOI32D7DGy3IeDms5nDk1g=; b=UqJ9SFlnQWGUnLQoSRG9gsNo2NjQ0z61yQLp5+uBbu5ptB7klA5X8XwPJxdnNIUm2GZcH/ 6vXcEXzfXOInuPueOqY9YxOjlL9RYkKZ40qkfcdAnNATKSQdJTgIX9pxnVbcrazLp5Wru/ pnAeZqGgbbpnfPuTbn9Ioxv73gZDIZs= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1736488962; a=rsa-sha256; cv=pass; b=DF8Kpewygye20w1e9II3Eb4PCzrBsmqIIBQjDMKJLrRHnydncrjDBXBR38MXItF7u7JlJe th9WszliIEaK9EbveSrkRFC643+VDO8iguLtjdkdaxrPbxv2Fq26P0XFz568kM2PLKfMhA j6yGlng6QXthDbK+g8vrWx2bAfcs98M= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="glHJs/oT"; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf05.hostedemail.com: domain of apopple@nvidia.com designates 40.107.220.60 as permitted sender) smtp.mailfrom=apopple@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UJTb5Ip60sL5Fx4Xa/jJg95JhtBQqJoyskIPAgUuO2uhR+53QuygAqpJcfvTWetlXl/q0J37/V5LiSmPDsMjBSiG3Kj2ne/L1TAVQR5pDx8of+kmX0aDsZ3ui7RO75yfSpzHCkR553/+EkgHwZk9vLBbfg27b+8635Ik9W6PKqyLJJltn5lbuaSbNAGZKcW7T4aIWPbKIpbH+4UTpWEYQkGiz1UgTSMp4R27Sd9SZizwibSXlEotq55iTGXzOoUZhG8UMI9klfnMFz+yTYeerRaMxNxltNCU8DcI20IQdaSnT2Q/l/Mz6+vKfYLsapX5OHa75MlO7RF+zrgW5vZwGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZaV7+Qz08NZ4CyqSz0ni0zOI32D7DGy3IeDms5nDk1g=; b=vVgXHTmTHC+K3Hveq21q4VFXR8bng+1fj5HD1ByQiGv0zUp2mGf1WI8kwqtgPV1w3EJlNFJWwCs8VKb/1hG6iR0EoVB4gl0Xw1bbvXARCp10p+oJQpd6NTIMeSEytfbU1yuQGQX040Ggf10NM2aTNerxPeYZ8uqiWVGW4b1EhGvWDx8GMcRcQmkN9MzMLZOzzKmej7U6+dBQxen3BCRxaZ3uRiiJAPSCSGrOmAMtujWJiysuMFIZl7E2XDhqIE0m+5KYOUcgi1wqoN1vTDjni6e9UWimq5+Ot8rgqx3BA0XAjK7p50OaLOcCCc841QYdAaTOIOHtM65B9oay0CVoeA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZaV7+Qz08NZ4CyqSz0ni0zOI32D7DGy3IeDms5nDk1g=; b=glHJs/oTXDvrQJrYprVGcEodBXCjUxnqRtRnzMCgJrC/zmYfZf4Dlvc7HXw0DzswJPSVSQ1cO0HF0YU3ZKt0CfbzZnJuZsyCQQKbdkTV+iFBkihFwWiAG40Px4mVsGWmI2mvdh4rR862wvY3+ecRp59I66TsFyBZM5i7YCrNtJkw/PLncGT/GYyckCS7awgMKEmad2OnwixgCV7CKkpATxef/bojnfpmhWRHgwin3YMgQySp6XpZnM3C4YqiOVs4LeUVDphrByL6hij7i3H9+ySIe7ZcmIoL8+/WHsTTZ5s6cZwBRH9BcsRXt0VlCbrINdjmrCH58hNfjjd2Wt0l7Q== Received: from DS0PR12MB7726.namprd12.prod.outlook.com (2603:10b6:8:130::6) by BY5PR12MB4132.namprd12.prod.outlook.com (2603:10b6:a03:209::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8335.10; Fri, 10 Jan 2025 06:02:40 +0000 Received: from DS0PR12MB7726.namprd12.prod.outlook.com ([fe80::953f:2f80:90c5:67fe]) by DS0PR12MB7726.namprd12.prod.outlook.com ([fe80::953f:2f80:90c5:67fe%7]) with mapi id 15.20.8335.011; Fri, 10 Jan 2025 06:02:40 +0000 From: Alistair Popple To: akpm@linux-foundation.org, dan.j.williams@intel.com, linux-mm@kvack.org Cc: alison.schofield@intel.com, Alistair Popple , lina@asahilina.net, zhang.lyra@gmail.com, gerald.schaefer@linux.ibm.com, vishal.l.verma@intel.com, dave.jiang@intel.com, logang@deltatee.com, bhelgaas@google.com, jack@suse.cz, jgg@ziepe.ca, catalin.marinas@arm.com, will@kernel.org, mpe@ellerman.id.au, npiggin@gmail.com, dave.hansen@linux.intel.com, ira.weiny@intel.com, willy@infradead.org, djwong@kernel.org, tytso@mit.edu, linmiaohe@huawei.com, david@redhat.com, peterx@redhat.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, nvdimm@lists.linux.dev, linux-cxl@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, jhubbard@nvidia.com, hch@lst.de, david@fromorbit.com, chenhuacai@kernel.org, kernel@xen0n.name, loongarch@lists.linux.dev Subject: [PATCH v6 15/26] huge_memory: Add vmf_insert_folio_pud() Date: Fri, 10 Jan 2025 17:00:43 +1100 Message-ID: <60fcfaa3df47885b1df9b064ecb3d4e366fc07e7.1736488799.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: X-ClientProxiedBy: SY6PR01CA0067.ausprd01.prod.outlook.com (2603:10c6:10:ea::18) To DS0PR12MB7726.namprd12.prod.outlook.com (2603:10b6:8:130::6) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR12MB7726:EE_|BY5PR12MB4132:EE_ X-MS-Office365-Filtering-Correlation-Id: 1bb4a122-00ed-4c5f-24df-08dd313c63a8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016; X-Microsoft-Antispam-Message-Info: 6yWKBGsW1wYjTL4yu4YkfiOwWOETa00YGGWVnFn1amPCB4ALPZdUW0yPMZYncNSFSvdEKLvgM8p/7GWHYgdyQ0k3mLdY1pVF/BuPM+GURxtIUN+YhV5wxqeZJmqFsKhiGlxarVjp0zvuOCAERloNIk7YXz2Czg5BDEikddnyCW/7FtGyKLv9b0zhNx134U7FtvkyVeaWisgEmTNebgc+uFk+ulUYdpExWeUFJKrd+3RsEViNzXMHXp3LRAwaW6EIuOU5/wZ8AoB0l7mtbefeJPoEuyyrGpuIURxeXw8Dnh6qqcXHc1cA2Zve2/dsVhu4zxSJ+aZcqDBsp3RWMYqra6DPcAijdZzUY1iIRKiiTUC7vDmwYMjOKyq7qMxyKW01zK4Zhmuh+KUvWy91YGblkOBxg8p48eVDAb0C4cLFzz4k8PVZsBLUv88nYD4I0DJonhhjdeg82fkLObevTW+fTY48LIgzUizHlSi29q0rVRYmwMfMfpVQABlBTsE3+ooyRauOlzep4wWkbfzMTlkE7AxNu5ozAP7J8fZZA+FrnLEv0ceMrvAYCHXy4NXJQEagKyo4pWY+guAu6ecAVy88AIxrveqQjjYZcl/1nQ5hiIv02e31siYTyh0lanmlLuT30NttYXyH+oFs7LInkKbIBGGa7Fo2FTj8i1gYJClxapWLBXyE7ijbZ7SHDZKLelqVBcS3R6GS23kmcfZSGTINdyyVJWQA+zcmcqqIrOdr0t1K8YrBmSaC5izxw6nHSVKWoQzEv8br+wCQt8YCgTXmHIAGpoQvgetN3lVd6iP6jDH8lXAPVDE2/r1HNzn4kE2mXiS0IrvwxRSVVwUOypC5NXQVp5b5m4+1hh1MAwSPmtkeBOUWOI8UZE+mlpF7570zcChSYallqLeTjvQI0vZyVmXCsYmZi/CGQ9yC+Ka5/uEaCRXL4lXRspQWToFV2NY3XcuYGDT2sfgiXpRoG/jONlb/ZzEjMPPzm+4dHFQHskKDEOWK1H5KV1w1m4fHwcA0jamPF/egIVSefsjBSlHsqI24C2dKY3UQ3aEumFPfxaNHRVoNGeMzKznSBHdwe3qWd7yIz86ZaJPrbxbIdy/VmYws2NH4Rglq5WTOglVYIAUPOwV9YRPV6S3FChUH2RcWo7na20ys1Us35gDjPEzE+t8wm+g5HElJkpEvD7qgeDCsmBBSsnz2qxPW18oUU0NFzwYCBIDiSBqBuXT30oi58emtCUw6DbOKPVeFRWRLLhIzwv4SbebE4oy0GJd2vNMkFBs7Ar60DMegHnmYAvKBGyB4Wn9gwfrPid9xUrgpLquLzvkC1qH6oqbQ/qyGAEdVFOHA5aW3qdH2ypPQ2pnN/cvSeWRAK3Vnf8NkbvHAyJo1xzWfW65lYjBUscXxt0Gp X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR12MB7726.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: G0S/uFECHFZpJvzvwcpfey1+iaBSHsAqcIB1nGNEAyk6+wb8uFn+lfPjiBKnxqej1TtH+O0zQVTiaunOvG89LqIg6AW1ooDwEbT2EgCrcgkIjn8I3NsLDiLSgFY93HzbgrNkNfkvMkOldmG4mZrLQCAHy7Rg044yZy+lRQbynz+SMv7FtU1EERyWsm0NBi4tZx3CngSYdx1yZuX0AHWAs1D5B7MRdixx1zth25DPvL4tTcaUGu0hzdlKOb+ptFYlm6h32B1PCDDZhgsrnPqFevsmjvcXFOGVvoUzci8rxDz7JTq6sHfnvLgRC9j6lU3dpg9kjmBCIDaZHxQb5SzTxkJ/Gi3EQXcUYYUp/PozGSVl9eSDSixAj4qgxwHxjtyUnAsrmMwGWBVuYhO7bA4W74BeKXR6fWAoXqDu+IQumG93u4Sd601OrLrkn/Pfw9mUNuirGGssL0hfB2+5tvatRgES31JLSGlTwIMEtsZdTIP0FTYdvVTF8fWvdGUgMwylDT763gqDYULH3mQxlQ4dOwgEO1Dr6EEQQ/smWIZVL6VbX1gej97745f1CpIyo6AOBg0aUyPV6HtvyeGqFXk5UuExQuG1Zs8DW/uVe2+9/xaNWnHhm3uXlMuhL27js2Xqc+xHTMOBoVPCoXLPAHBOU6gyndtb2FTOTv0hT63W5+3P0sqMkbr9ahFCS+poL66EpaLIqgp27VIRgBWMHozWZpbp0uPJy/T9v0oEzSBO7A+iKZdQH18802CVx6iB4ghFyRrapen6XT9HZzw7sY80uoInNiEUsktHIq+kiwW3YPvues2qp4D5+4GDhEN5qt0R8KZZGujYT+APtJnh3i+o4ay4TkMqVODaMUAaOlnaidd+53Y9Op7oMolxWeo5Dn/DRspdWy5cgMEo+k/dqR7VZm5KOMnu1F5evc2uT7rB5rLUH17FVF4i3niLjgdyziVbioPu8RGZ9GQf/eUF8c8eKq1ImYlgzWZY1eqAiVEQqPrSZvScPKqU1Nir0qHSmxhmFDw2t3mdhJ2lC3gu2s7F+MKBLnseuL0U/YCAbLwYsFZiYtMu7pduf/E8SdS86yBBSsS2Hg9q0gZwGGjdevDPE1Bt+MatJqV5y+DCMEPcQBO5LO43ajqizFGbsguSCse6iwskXeEx/SM9Df8m7b4lZ0nDQ4A5XAQpUP5OF1iNhljggEPDqJRSczLSGG4iz7p3LFshIgBzGVWBMV/4Dwd5gJ3yJLyoifYWiu6BSN0+jxjB204NWz0iFAkZaO3+vdQvrWPSOjRKxze6wVthnWFXpIhP6D/TLb2Pc3JMsPZNekLZCypna1J5HR3e2XnR/O7GDB9dVldrQ6vCv7CZkNaihoI1gOCWZEfRBxnDV0AW4rbtHxSQh1QWYt74ApiluJt7XEQqqJsrKJhiSn/WklEBT94KjQsPf56sItZGJ25lrKvkpZFyo0eD/BMEFq/xVeQkI/7weo4tuEtT5ZzcrFCK9VZ45sN4Wi7jyBWpUHvGDbX6dvhMNwhvxWIp6/3XrD2IVsJT5vm07/r6LSrz/SMyzwcxD0SH69APvReMEKOSln7fiHu2JPJrKNR9gYLCY5xg X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1bb4a122-00ed-4c5f-24df-08dd313c63a8 X-MS-Exchange-CrossTenant-AuthSource: DS0PR12MB7726.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jan 2025 06:02:39.9463 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FMcmMd+UXyHm+/TF7jdNNKJx4n9ksgHfx2s75jz62yfD4KubpO0S7HMowIec7M1IKtVz7W2HOzvcrBIUIBBh1A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4132 X-Stat-Signature: 53zbtgsxzphm41yaatb6xhtwkef1hyj8 X-Rspamd-Queue-Id: 937AD100002 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1736488962-127578 X-HE-Meta: U2FsdGVkX1+avuRKM/SDoHq+t9L6BitG4V/cgpJUNc8f3eLT0yu/TQEWL8//G91J85jYVKWt+JaEXKGg1nUA5yDU0Vhw/X+Y4+enxFxsjl1ValnjrpjhqMMBwCfMBRM5p2rE16WWZW/exCivPOe0hqTqzHfSVcY81lrf2raqa3S4CcdVGEOJs4IEd1d3y/62XSow4ebYx1a6y4jtfN8U/eE4GR0RMcLuj20iS4pXi0nYKXBG+IC2Dz/FKBZabSzowitQqzj8oLl/UhgqLKm1NgVsrTXrke7Y2NnqUtmetJKBBsx9bAX0dqM8tZpNWWtgKsxsUaKvqGTp1Ty1vCY+9rzYK9XNlTxsC87W5CxIems0eFnnmzjLvXKUbJRbZ7qCoV3YJpyHk3JnCL+HwQ0VJLF+QE4xqjyLG+sFkhudbOcYE9P2enS2/a6oA+ob9y63PVfnXqB67rq7Wx4QoXkAU7wiPuNjUJ7quBGiNPHNg6oOgP05kvPaLzRHIBRXtK8eJyYjQU2UeY589FcM4QCCkOP5fnOJgT+Cy5ryrCPs2MrhvzAytyv2uzwNFxsSU3eARLkjf4OBKoRvj+dS0nmuPXIeJ2JcEpEPMR5h6FSkOZMYyrtky+nUiFPl43KbhaS/oUTjVEMaaGNTCcvk0Cqrj6elIcctWXk9Yqigq3pFtJoXSfHeGRyULPGMO/7vycVDlLjp+qGCRKpCIdicpVnn2rmY6706zmqVJcMXGXsWdRKEI+V+oGnO6KgKre3p3uIxoP1TIxzkSxkqhQmGIyzy6orXXyFjFj0XQoddfJOLTKOASCtsgzAWAXhooWM1IQ9Vn4Lyp4pcrkmCvl5QMn5ZDRL9V9A5cvL6glVu3VJlAJD9j5TIqwl7RIfPgCNvoz6IfGk0AoRF8jKLk/p+1rGtHL6MYoW6ON4wKvn5xt2+ZZD9ja8Kn2zGEsB4u97JmsCWrI3sId4F5e3h7/b6Vvx PHOAIGTR 5YAbmI/4CLnhC4M3qwA9qn4f8HvQhAvS3kPcCWkPN24SwDLDExWpD8qx5ormj0JOTrBUI+ihOKbveEkLRyQ7cAQfky3yF7KihjcAQn4KIgOP8EJvV2oRN3z9Eq348KkkVm531chAkKoWFQrayIcpYjmVpbIcokJHJ+nvv+kbGHGucjXmUB4FjrhYhFG1nqr52aAj03SBNSNwze8+nqCeEtXdoU3FLc4nOUK55ppGInDHBh9MZHaMgtbT5jc83s+ShhrVneAYdjXMYcjLzMFQC2hGEL6Rm5TE/h3A2puQ1K6l/VbbgYlMRO3oRW0RcOqrmy6+xPOHd87g9FlotWyeqL4FlgKpeiR9ys9uWmUK0xNpTpnv5jF8ECtvVT7IMp1h6FiqPRNf+2PpirQdUg/mszFgyGeQGHuDiJi7iL1cN4WxguFQEhup4TLTLWNEYh1Ja9E2vJ7eahrrQHiwFIs7kZJTmOQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently DAX folio/page reference counts are managed differently to normal pages. To allow these to be managed the same as normal pages introduce vmf_insert_folio_pud. This will map the entire PUD-sized folio and take references as it would for a normally mapped page. This is distinct from the current mechanism, vmf_insert_pfn_pud, which simply inserts a special devmap PUD entry into the page table without holding a reference to the page for the mapping. Signed-off-by: Alistair Popple --- Changes for v5: - Removed is_huge_zero_pud() as it's unlikely to ever be implemented. - Minor code clean-up suggested by David. --- include/linux/huge_mm.h | 1 +- mm/huge_memory.c | 89 ++++++++++++++++++++++++++++++++++++------ 2 files changed, 78 insertions(+), 12 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 93e509b..5bd1ff7 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -39,6 +39,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write); vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write); +vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio, bool write); enum transparent_hugepage_flag { TRANSPARENT_HUGEPAGE_UNSUPPORTED, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 120cd2c..256adc3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1482,19 +1482,17 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, struct mm_struct *mm = vma->vm_mm; pgprot_t prot = vma->vm_page_prot; pud_t entry; - spinlock_t *ptl; - ptl = pud_lock(mm, pud); if (!pud_none(*pud)) { if (write) { if (WARN_ON_ONCE(pud_pfn(*pud) != pfn_t_to_pfn(pfn))) - goto out_unlock; + return; entry = pud_mkyoung(*pud); entry = maybe_pud_mkwrite(pud_mkdirty(entry), vma); if (pudp_set_access_flags(vma, addr, pud, entry, 1)) update_mmu_cache_pud(vma, addr, pud); } - goto out_unlock; + return; } entry = pud_mkhuge(pfn_t_pud(pfn, prot)); @@ -1508,9 +1506,6 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, } set_pud_at(mm, addr, pud, entry); update_mmu_cache_pud(vma, addr, pud); - -out_unlock: - spin_unlock(ptl); } /** @@ -1528,6 +1523,7 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write) unsigned long addr = vmf->address & PUD_MASK; struct vm_area_struct *vma = vmf->vma; pgprot_t pgprot = vma->vm_page_prot; + spinlock_t *ptl; /* * If we had pud_special, we could avoid all these restrictions, @@ -1545,10 +1541,48 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write) track_pfn_insert(vma, &pgprot, pfn); + ptl = pud_lock(vma->vm_mm, vmf->pud); insert_pfn_pud(vma, addr, vmf->pud, pfn, write); + spin_unlock(ptl); + return VM_FAULT_NOPAGE; } EXPORT_SYMBOL_GPL(vmf_insert_pfn_pud); + +/** + * vmf_insert_folio_pud - insert a pud size folio mapped by a pud entry + * @vmf: Structure describing the fault + * @folio: folio to insert + * @write: whether it's a write fault + * + * Return: vm_fault_t value. + */ +vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio, bool write) +{ + struct vm_area_struct *vma = vmf->vma; + unsigned long addr = vmf->address & PUD_MASK; + pud_t *pud = vmf->pud; + struct mm_struct *mm = vma->vm_mm; + spinlock_t *ptl; + + if (addr < vma->vm_start || addr >= vma->vm_end) + return VM_FAULT_SIGBUS; + + if (WARN_ON_ONCE(folio_order(folio) != PUD_ORDER)) + return VM_FAULT_SIGBUS; + + ptl = pud_lock(mm, pud); + if (pud_none(*vmf->pud)) { + folio_get(folio); + folio_add_file_rmap_pud(folio, &folio->page, vma); + add_mm_counter(mm, mm_counter_file(folio), HPAGE_PUD_NR); + } + insert_pfn_pud(vma, addr, vmf->pud, pfn_to_pfn_t(folio_pfn(folio)), write); + spin_unlock(ptl); + + return VM_FAULT_NOPAGE; +} +EXPORT_SYMBOL_GPL(vmf_insert_folio_pud); #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ void touch_pmd(struct vm_area_struct *vma, unsigned long addr, @@ -2146,7 +2180,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, zap_deposited_table(tlb->mm, pmd); spin_unlock(ptl); } else if (is_huge_zero_pmd(orig_pmd)) { - zap_deposited_table(tlb->mm, pmd); + if (!vma_is_dax(vma) || arch_needs_pgtable_deposit()) + zap_deposited_table(tlb->mm, pmd); spin_unlock(ptl); } else { struct folio *folio = NULL; @@ -2634,12 +2669,23 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, orig_pud = pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); arch_check_zapped_pud(vma, orig_pud); tlb_remove_pud_tlb_entry(tlb, pud, addr); - if (vma_is_special_huge(vma)) { + if (!vma_is_dax(vma) && vma_is_special_huge(vma)) { spin_unlock(ptl); /* No zero page support yet */ } else { - /* No support for anonymous PUD pages yet */ - BUG(); + struct page *page = NULL; + struct folio *folio; + + /* No support for anonymous PUD pages or migration yet */ + VM_WARN_ON_ONCE(vma_is_anonymous(vma) || !pud_present(orig_pud)); + + page = pud_page(orig_pud); + folio = page_folio(page); + folio_remove_rmap_pud(folio, page, vma); + add_mm_counter(tlb->mm, mm_counter_file(folio), -HPAGE_PUD_NR); + + spin_unlock(ptl); + tlb_remove_page_size(tlb, page, HPAGE_PUD_SIZE); } return 1; } @@ -2647,6 +2693,10 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, static void __split_huge_pud_locked(struct vm_area_struct *vma, pud_t *pud, unsigned long haddr) { + struct folio *folio; + struct page *page; + pud_t old_pud; + VM_BUG_ON(haddr & ~HPAGE_PUD_MASK); VM_BUG_ON_VMA(vma->vm_start > haddr, vma); VM_BUG_ON_VMA(vma->vm_end < haddr + HPAGE_PUD_SIZE, vma); @@ -2654,7 +2704,22 @@ static void __split_huge_pud_locked(struct vm_area_struct *vma, pud_t *pud, count_vm_event(THP_SPLIT_PUD); - pudp_huge_clear_flush(vma, haddr, pud); + old_pud = pudp_huge_clear_flush(vma, haddr, pud); + + if (!vma_is_dax(vma)) + return; + + page = pud_page(old_pud); + folio = page_folio(page); + + if (!folio_test_dirty(folio) && pud_dirty(old_pud)) + folio_mark_dirty(folio); + if (!folio_test_referenced(folio) && pud_young(old_pud)) + folio_set_referenced(folio); + folio_remove_rmap_pud(folio, page, vma); + folio_put(folio); + add_mm_counter(vma->vm_mm, mm_counter_file(folio), + -HPAGE_PUD_NR); } void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud,