From patchwork Thu Aug 15 15:11:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13764908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34548C52D7C for ; Thu, 15 Aug 2024 15:13:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5A928D0001; Thu, 15 Aug 2024 11:13:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B070B6B013E; Thu, 15 Aug 2024 11:13:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95AAD8D0001; Thu, 15 Aug 2024 11:13:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 70CF06B013C for ; Thu, 15 Aug 2024 11:13:31 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 267F41606E2 for ; Thu, 15 Aug 2024 15:13:31 +0000 (UTC) X-FDA: 82454823822.24.7A88344 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2065.outbound.protection.outlook.com [40.107.94.65]) by imf27.hostedemail.com (Postfix) with ESMTP id 44E1940025 for ; Thu, 15 Aug 2024 15:13:27 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=HlHAzKMm; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf27.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.65 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723734735; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ihOu2iVN4hwGcHHGXbybEwpG6CY6bCTOfEKx143ZTj4=; b=dJbTeQmakSr9/xEk/1BNoqj9gdUFcbjst/Ob8wVGaebwn4gUGrZ0UK3zBjnmhZT7uxZWst WcOX78gRM5FhuH4gLEinaeBctx6IG3YIlrZxZPACjFG8v+6oPQUZXD5T0oYkYjxllNO6bk LekLQLAMAcEgawoAjpkIJNscqGdwb7Q= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1723734735; a=rsa-sha256; cv=pass; b=NauI2Yd2KWCZxobkx1Wuca+vRzxaCbq+tU0NEr1EUXwWRPOtBloV1I1i7uCdTSOViGne2o 2pO7pDDzn9WhWitQG/zkBzLIpEadQMlqc/QwbjRJNeDMGkbziENoQlTTW/Z8IkNBwjxAte /nMWfOugY0q75AFpO0eS3fJ80xMHa4Y= ARC-Authentication-Results: i=2; imf27.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=HlHAzKMm; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf27.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.65 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=h/lJTt19gWlwQyLniK90pj2stX4iYDjUy9Ra2am6llOdxY0xnSjjDOGpscS45myUC3T3vd5LIKL1WbNlvY3OiXFlV95KkI1AZZcI8+enlq580Sn5QS3mLpJm77yImPeq3uwiuJm1URJM/tU1RQdx+DqXGeqwrxNGHtoG/YnzJO4yrbzyxIu4AnS8FyHaJVheh4RhunKA4l3sWoUDALsFFQpgBAAGgqqUKsrhpvaD/XDnBEpyCA6IA0it7TlnTleoea9ENiqzqX2+mhTkgszcmndgHF04pq2X6U39lGPcim0Jo4lKd4f+EcKS8eNutrdon4kOhBjOtYf8ykZsP6OsCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ihOu2iVN4hwGcHHGXbybEwpG6CY6bCTOfEKx143ZTj4=; b=VOHovd+bdujtNa9v5rhCUGJ89hgeLsLAZN3jyHUW27A1Utldupatur3xfSGmUiB0fVnLMDwRXp7hIVukDApXnLhgf/nyMgbHIeWDocgLfYxsrfHE42jIkYOdTma8mvS+aelv1TmeM5f7k1vO07K5ENRhjMr8yAGh1Ar4rY9hgoLbCqaqvJFFv79qgUbyg8HXCdtSRLV41fjhdngIVJhnz4CpeRDWjgQX4/vcZ0Fi4h6LRoSjQhtG80841Bpq8pO5+gt5mHw2Pfn1nFZ6GYav7BW4WWV4v+RMi6cgYavPTPR5YC8sb99DtVQ7Hca5qFuLroN2vHdQmwBpkmmjt+N7CQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ihOu2iVN4hwGcHHGXbybEwpG6CY6bCTOfEKx143ZTj4=; b=HlHAzKMmBrvPzM9aUQJqBr9OXS+kstH6QVd9dvoU8IXWU1vMmm7+Ur3/6fB3wbtVzHSU9RYb50AULa0OR/ZpJGYlZpGwlpioRjT0aqDwrrwVGgcEnekv2cLjohc8OMw57yB1Fj2U1E6bYemKnELtgWNnKmnJ2uTDF3KWJ1xxpV4w4Hwc0bsqDqPGitne/8SDKGtOFj9+tnNXu7JORQe5sWI9RCxgo26rVcDIYWm2maIfMrDuqj5r8YxH3SZe05YYrdaFl8YcgRH6+dKTqbyRt49Il8BqafckFSQSKpCV9xoH1k9zpw3wsqnpMDvLOB+IaXwIQk/KWe4gPP1Yom2fgg== Received: from CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) by DS0PR12MB6631.namprd12.prod.outlook.com (2603:10b6:8:d1::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7875.18; Thu, 15 Aug 2024 15:11:42 +0000 Received: from CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8]) by CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8%3]) with mapi id 15.20.7875.016; Thu, 15 Aug 2024 15:11:42 +0000 From: Jason Gunthorpe To: Cc: Alejandro Jimenez , Lu Baolu , David Hildenbrand , Christoph Hellwig , iommu@lists.linux.dev, Joao Martins , Kevin Tian , kvm@vger.kernel.org, linux-mm@kvack.org, Pasha Tatashin , Peter Xu , Ryan Roberts , Sean Christopherson , Tina Zhang Subject: [PATCH 11/16] iommupt: Add the 64 bit ARMv8 page table format Date: Thu, 15 Aug 2024 12:11:27 -0300 Message-ID: <11-v1-01fa10580981+1d-iommu_pt_jgg@nvidia.com> In-Reply-To: <0-v1-01fa10580981+1d-iommu_pt_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0239.namprd13.prod.outlook.com (2603:10b6:208:2bf::34) To CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB7763:EE_|DS0PR12MB6631:EE_ X-MS-Office365-Filtering-Correlation-Id: 74004e44-e112-4ed6-fabb-08dcbd3c8e58 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: ziSufx5ojokGC33D/nzHI6hCJjOTxu4RQLTpTihna/7dLbKElMM5Wpt73mo4Oj8k2+H+AGIknfyQfCCJK7neX5cUaNkr9YlJNkMK5M8lfbNlHwoqXmaY9LKTvuMqrl9SzQwwZJnWwXsgKJk38UUDn1BPSI6SYJVfSyDWo20M7q3niB1d4amafu6x+yBZWxfcIq1BzrvvBJaHjWzZPVhR+65HQfZdXUlLG7rb3MJE3OY6hGYVQTjrp5wE+gdoCd37zjHv7ampzIYShI16ir6VQdqdOEKi0/cCH8R8kZFuVoDapWkcjpmagKyzgXJlG+7KRk/EeSPSLnCDUn2oqhUfGKnPehc/3fGm3v63eAOe6GFNamZszGi9pRSqQtRGORxtihcbUtxCscquL3F7XdMGUQtCxFazzhQ5SH9qna5x2AtK6y/JOqmUhXOmN9lyxO/2A65HVehJT8zT6jMR2vgsSURMiqGV0m2WAP22tIDH9KPY7t+DSfIvqIF6GDL8udshDFr6+cxTqsl5XOeJLy6e5fflHedhNMF9F5YxqpU9Y/RoorfFsjc91TRzCzBi/DGeAZAeaidyVoJTI4QSGnCGGY08g+pejx3LBraK/dHXJKK5pvxrgCuI0v8rkwhKkpzXEN72gB5EGiQqNI9a5W4UByj9vDNMoSLOgPfR8feG7VzedDAITUL2HAvWhaivCktJ9eFCB7AGVdAuMdmJxxH8S3YvgfFHQ2S5OqLfmQfp+Df0mWoSKJ7kOW1Fpu43C5pjGf7OZI2/ufNMtMWxIAurfcTvD5AKCFR2rZ1LKpjQ2Ks4sIdPOL65Ll5GnjeWhotQ1TZ3kxghDtDeCabCUWWxOhNThpqQYaoCcpaoN7cfdwZ0krEhgX5QyyK8bRpTTydPnFm1B0hm8ywJJ3pcC7u9q/YSM8iYKPcyVsidUeGJmbxoa+ZDkURVEP1/nkTtvaGJpoUurVOzLxZ7czDztHhlXeI2Hq/bARQ/C4YRqxsQ9VCimGlIbXo5+f7ADlTQfzphfd2ryXutE1wjs5sMkfQQ/0pIYb8alGbmOvJNAd8UqDgJkMfi74/ixE9x7cB5nvZi/x6u9XWsly9nd8PIiSaR6mrzOL6KA2r6jgEU48Vgpnog6ock6runmw0rvJaMKcMq9U1dcnx2PNSTCFGVSN3kUHQpVPAxD4G/r/nGU/6C8WcgqNihlsHU+MODjqimaS2v7Gt9KLhYB2rBU+cgXYP1ovKY5zq7tUJuTka/W0iK1LAm7ohbEr6JXqDhIVHi6F+vGsrtVjYrhROpV3RjW3cV2OJXX5nymihpPUGXGc82zxRChij43u8wx5TSUb1+iyoO6KS7nS1IezSHyirmt/3BQQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR12MB7763.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 67P6Uz96U3O+h4yLkkIm67jLvwMgQ6S0kQJ3g9lQpqiji4orSXeAr6C8ZaooZpQsMOJl2W1/f5yVamB6dazXPAPMt0HL2kvs5v7Ozx0MoSr9BCwLpS5Wpl6gZ/Dk+aEktx7S7aT7bwXj7Aq9HU58AFe+4NvqHGc6gHYDy+CPtnJY+YsqRHs1PL8SySsZmHnNDRukPnyOt79FBV4hS1G0WL0EWLE2klEMdY6IoOBYF3VQWSm1m1v34XGnLh8Xm/4TAbGkwkaaSmJ1h3v0hi8fKh+xcSQEwc+TFXhRYny9ClKxCIVu4+ORA5JhMSCJar3b9mpg0UR/68RZ9YSnw+7U3rMoI5Yl2mWtvy+lij3RlYH0U/CHo3INXoEipVLjLwN6THo2l/g+Xqr9rSl90RRhdwTV3ZYAjjmkwji4vk3PWw2Bm4prIkI4wMI9nKJiDyCxiAU2ovU1P4O9G9blbTPfhLxxP8iXo/pPcti3hQxEE9hAbu9cU6Xfop4szC8mddiz9+3k0+jqHvkrAIsr+xiqAs97XwWvABz/PqzL3FNBS5ec+L46KGiY7ADSwKmE3t0AWiVuc84Ppf6CprPlL5t8k6cJ90sDPTX/TzEDuexCD6ECjWkyTT7YV/CwOFPM/AuiYqKMZB/e0ATZcZsuVkiDMSQUm8O1K+bD+qP4+SGV+lU2A6aGN2kYRQNAxBiTljSGvC6k+xnswbDnAY2gGGyiPAMq8CvfrjUs4J+XtKWr3KXYBi5whQ7Z003LNefwXK/hlsSXNnkAp2/K8UHwsNnmoJg7vMfB8pE1LIMqM4tfw4TzLER7iGkFTqN7NAGggby3UTvxl+TGEOBhvs4xp2mhZnGUlJph87ALdxVFIXNPxZa8W30nkKa5oGJ8jr8/1Qoh8ZgqCHx2Q/O1TBM7jWSIQs63bEHlvSsaX5+xFJKj2CAHmrhSsBukr4bUovJ9bx6PQvsTEdCknIB15HUCx7pvdl3mZVej6PAsowC2XLZDPON2ipqp5TlvrV+GsNDSXQGnjXcLh3BxuXv7cL9cEpkCEBEIXhQFl2c3PXxHyi74oxJC/1QEZ1auhlRsVI0Q+wSoTe9/Jr4LFLAGTw/pGba3lhXIkhtOyM9X1EWIgD+5s6xxCVlblZ8sooexDrpUMJiptJssmcUOl+F40ehl/npPvAfaq4Mwq9yGoMvLutpOEHpvcVsvQ8dzv84MVRDir6cmroWnDSNZTE85Tv5meu0H7yHvtAtaIf+CpxfXtRQCBzaLLNW8abJ6M9UjA361lUlpa+Wddf7jUZAhHJT5ouI/0gKcWonWyNAhPi4OpuA+JvUmUTR8GRdDSs/CLx7GJ/CkXfgBhOecVHFZoFYXd9ibrpkXHGiDrz1PHB74u8AbK31HnHIXHBlcCoZPsNB0JksXWyGSR36weA0rs4oqlOlY3BuVYhzrSUoT007P+IZALhozcBzFSdwpYVBVvJEGpreP1JkUExkv7P42pPWucFiAqx1Jp+gg7vWmqwHM991z5ZuWDzV1wZkxux/NVl12FepsshL16UDpjc3hdHJ7qVmRGxD/02y66rjNpemU7btkSTg= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 74004e44-e112-4ed6-fabb-08dcbd3c8e58 X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB7763.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Aug 2024 15:11:36.6136 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wka/EjPR5hlw01zTVC4Vv7inIkQnRmTjmhVi0T38+qyxR731zAIcWE/haEvSCawx X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB6631 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 44E1940025 X-Stat-Signature: jirjy4i5hcc4ikkr4gzq495urbi4r8jx X-HE-Tag: 1723734807-968944 X-HE-Meta: U2FsdGVkX19EdKcT01qinlfj42XYqNZ4Nd577uw+FDxnFI8Wd0OOBxsbCBPlTexVzjxTnIULTKK3NcKa1FaLD4i2ehxl+L88R3m6AmpCSkhwp/+Hpln2NcE+Xjo+J/K8WBcTuHgYJbrSYq4WFgM0TaCnCKZgbcLRFIvbPcgHdfsu2z6VaMxzttS2gHYLBEK/6juT8euPHMG2eSKIvrn7Y5hrf5SmCtxn/AftJy40qAPgE1yghHygNq/exvKbCAL+qJ9s2qlj17HMz66rPWTec74MfWJmqYT2WyfyTVJBQhXmeyB74/gBRpl4Q+4fuKClDFuxtql08ofITJEyB/6ee4CVJ3FfEHTeLKw1gdPiSLZ7iBpmpkRX9P19Jwl2jEUx+RPvNIMNGTjaS/Zaumme8+xWLfGFfREjqozThaPr471mnoSeL5JxC8MADOayyhirT9LevKDHLsXYyUetF/rZeEmfuriyVyXILGa10Jh/y0kcWjbNSGosPkOk5RDu6DIg9tFLuDTEIT8deN5za3cWDgJHJMF8+x/qaHxhuhh4Ok8mBA9vSTGnjk33kFaJR0LGv0ACOXTPN/QoMBp37nqn8G1turSJi63v+EEQ+NQroLtLtg1Tlkn/1ov9D5UY6fVShSOg1+mzg6P7Fxii+NXDVyuWY9LffeqPvWyeVrSGsmOdLi2FjvZPAfAUyWDTis52QZ7AtT2akmAczxADDIIkF4v/9XSEYlqwK4OfRLycV16fh78LIyJcDAw2eNDfGzdAaj04X77mdj2PG9+EZjfUYpWo2Gqud40IBDY3q702/LSONwgruEsD999fXOM/Fipuh/+bqncUGv744sM/AZ8wTdnfIzSDYFMrwSsGYYrpUbwE8THWesxfX1X+gwZVqvQCCZXaRAavv5GSEhHEK9vxHcDW5VsP7L2YohDHDw/yCwS+WrvREmlvITRnKfAkOLQs8gCir2MjlJdFZZqXPJ8 gPCgBl+a 3sLqtpWV0pyUgpNzdG49zNpz2eRgoMaH4gQEN+Jh3gz+6pFbO1QYjTZqqgvqcAPG/vvYkZsMDjxhlWVXt+AKcWpJ8CNoU8sw9Glq6roQccgwvQT4eXX/yP91nALKSfuOR0MJIIwImiIZPaF6HeLAOyGhysyBcyrhhI3vorOamEAUYg4WwJ01WgP1GJ4qkcgGPXPJaEYoLgQK4nNdHJwtysLE0LOjyBYlVmJx8OS5pSjzD146oYn5uGL5Exq3qPSPNjbzvdZlQbrCKqeFIWCnSlDSh8q3tbEALGsV+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The features, format and naming is taking from the ARMv8 VMSAv8-64 chapter. ARMv8 uses almost all the features of the common implementation: - Contigous pages - Leaf pages at many levels - Variable top level - Variable size top level, including super-sized (concatenated tables) - Dirty tracking - low or high starting VA Compared to the io-pgtable version this also implements the contiguous page hint, and supports dirty readback from the S2. The common algorithms use a bit in the folio to keep track of the cache invalidation race, while the io-pgtable version uses a SW bit in the table PTE. In part as an demonstration, to be evaluated with performace data, ARMv8 is multi-compiled for each of the 4k/16k/64k granule size. This gives 3x the .text usage with an unmeasured performance improvement. It shows how Generic PT can be used to optimize code gen. FIXME: Not every detail around the variable VA width is fully completed and tested yet. Signed-off-by: Jason Gunthorpe --- drivers/iommu/generic_pt/Kconfig | 39 ++ drivers/iommu/generic_pt/fmt/Makefile | 4 + drivers/iommu/generic_pt/fmt/armv8.h | 621 ++++++++++++++++++ drivers/iommu/generic_pt/fmt/defs_armv8.h | 28 + .../iommu/generic_pt/fmt/iommu_armv8_16k.c | 13 + drivers/iommu/generic_pt/fmt/iommu_armv8_4k.c | 13 + .../iommu/generic_pt/fmt/iommu_armv8_64k.c | 13 + include/linux/generic_pt/common.h | 22 + include/linux/generic_pt/iommu.h | 73 ++ 9 files changed, 826 insertions(+) create mode 100644 drivers/iommu/generic_pt/fmt/armv8.h create mode 100644 drivers/iommu/generic_pt/fmt/defs_armv8.h create mode 100644 drivers/iommu/generic_pt/fmt/iommu_armv8_16k.c create mode 100644 drivers/iommu/generic_pt/fmt/iommu_armv8_4k.c create mode 100644 drivers/iommu/generic_pt/fmt/iommu_armv8_64k.c diff --git a/drivers/iommu/generic_pt/Kconfig b/drivers/iommu/generic_pt/Kconfig index 3ac9b2324ebd98..260fff5daa6e57 100644 --- a/drivers/iommu/generic_pt/Kconfig +++ b/drivers/iommu/generic_pt/Kconfig @@ -29,10 +29,49 @@ config IOMMU_PT Generic library for building IOMMU page tables if IOMMU_PT +config IOMMU_PT_ARMV8_4K + tristate "IOMMU page table for 64 bit ARMv8 4k page size" + depends on !GENERIC_ATOMIC64 # for cmpxchg64 + default n + help + Enable support for the ARMv8 VMSAv8-64 and the VMSAv8-32 long + descriptor pagetable format. This format supports both stage-1 and + stage-2, as well as address spaces up to 48-bits in size. 4K + granule size version. + + If unsure, say N here. + +config IOMMU_PT_ARMV8_16K + tristate "IOMMU page table for 64 bit ARMv8 16k page size" + depends on !GENERIC_ATOMIC64 # for cmpxchg64 + default n + help + Enable support for the ARMv8 VMSAv8-64 and the VMSAv8-32 long + descriptor pagetable format. This format supports both stage-1 and + stage-2, as well as address spaces up to 48-bits in size. 4K + granule size version. + + If unsure, say N here. + +config IOMMU_PT_ARMV8_64K + tristate "IOMMU page table for 64 bit ARMv8 64k page size" + depends on !GENERIC_ATOMIC64 # for cmpxchg64 + default n + help + Enable support for the ARMv8 VMSAv8-64 and the VMSAv8-32 long + descriptor pagetable format. This format supports both stage-1 and + stage-2, as well as address spaces up to 48-bits in size. 4K + granule size version. + + If unsure, say N here. + config IOMMUT_PT_KUNIT_TEST tristate "IOMMU Page Table KUnit Test" if !KUNIT_ALL_TESTS select IOMMU_IO_PGTABLE depends on KUNIT + depends on IOMMU_PT_ARMV8_4K || !IOMMU_PT_ARMV8_4K + depends on IOMMU_PT_ARMV8_16K || !IOMMU_PT_ARMV8_16K + depends on IOMMU_PT_ARMV8_64K || !IOMMU_PT_ARMV8_64K default KUNIT_ALL_TESTS endif endif diff --git a/drivers/iommu/generic_pt/fmt/Makefile b/drivers/iommu/generic_pt/fmt/Makefile index 0c35b9ae4dfb34..9a9173ce85e075 100644 --- a/drivers/iommu/generic_pt/fmt/Makefile +++ b/drivers/iommu/generic_pt/fmt/Makefile @@ -1,5 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 +iommu_pt_fmt-$(CONFIG_IOMMU_PT_ARMV8_4K) += armv8_4k +iommu_pt_fmt-$(CONFIG_IOMMU_PT_ARMV8_16K) += armv8_16k +iommu_pt_fmt-$(CONFIG_IOMMU_PT_ARMV8_64K) += armv8_64k + IOMMU_PT_KUNIT_TEST := define create_format obj-$(2) += iommu_$(1).o diff --git a/drivers/iommu/generic_pt/fmt/armv8.h b/drivers/iommu/generic_pt/fmt/armv8.h new file mode 100644 index 00000000000000..73bccbfa72b19e --- /dev/null +++ b/drivers/iommu/generic_pt/fmt/armv8.h @@ -0,0 +1,621 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES + * + * The page table format described by the ARMv8 VMSAv8-64 chapter in the + * Architecture Reference Manual. With the right cfg this will also implement + * the VMSAv8-32 Long Descriptor format. + * + * This was called io-pgtable-arm.c and ARM_xx_LPAE_Sx. + * + * NOTE! The level numbering is consistent with the Generic Page Table API, but + * is backwards from what the ARM documents use. What ARM calls level 3 this + * calls level 0. + * + * Present in io-pgtable-arm.c but not here: + * ARM_MALI_LPAE + * IO_PGTABLE_QUIRK_ARM_OUTER_WBWA + */ +#ifndef __GENERIC_PT_FMT_ARMV8_H +#define __GENERIC_PT_FMT_ARMV8_H + +#include "defs_armv8.h" +#include "../pt_defs.h" + +#include +#include +#include +#include +#include +#include +#include + +#if ARMV8_GRANUAL_SIZE == 4096 +enum { + PT_MAX_TOP_LEVEL = 3, + PT_GRANUAL_LG2SZ = 12, +}; +#elif ARMV8_GRANUAL_SIZE == 16384 +enum { + PT_MAX_TOP_LEVEL = 3, + PT_GRANUAL_LG2SZ = 14, +}; +#elif ARMV8_GRANUAL_SIZE == 65536 +enum { + PT_MAX_TOP_LEVEL = 2, + PT_GRANUAL_LG2SZ = 16, +}; +#else +#error "Invalid ARMV8_GRANUAL_SIZE" +#endif + +enum { + PT_MAX_OUTPUT_ADDRESS_LG2 = 48, + /* + * Currently only support up to 48 bits of usable address, the 64k 52 + * bit mode is not supported. + */ + PT_MAX_VA_ADDRESS_LG2 = 48, + PT_TABLEMEM_LG2SZ = PT_GRANUAL_LG2SZ, + PT_ENTRY_WORD_SIZE = sizeof(u64), +}; + +/* Common PTE bits */ +enum { + ARMV8PT_FMT_VALID = BIT(0), + ARMV8PT_FMT_PAGE = BIT(1), + ARMV8PT_FMT_TABLE = BIT(1), + ARMV8PT_FMT_NS = BIT(5), + ARMV8PT_FMT_SH = GENMASK(9, 8), + ARMV8PT_FMT_AF = BIT(10), + + ARMV8PT_FMT_OA52 = GENMASK_ULL(15, 12), + ARMV8PT_FMT_OA48 = GENMASK_ULL(47, PT_GRANUAL_LG2SZ), + + ARMV8PT_FMT_DBM = BIT_ULL(51), + ARMV8PT_FMT_CONTIG = BIT_ULL(52), + ARMV8PT_FMT_UXN = BIT_ULL(53), + ARMV8PT_FMT_PXN = BIT_ULL(54), + ARMV8PT_FMT_NSTABLE = BIT_ULL(63), +}; + +/* S1 PTE bits */ +enum { + ARMV8PT_FMT_ATTRINDX = GENMASK(4, 2), + ARMV8PT_FMT_AP = GENMASK(7, 6), + ARMV8PT_FMT_nG = BIT(11), +}; + +enum { + ARMV8PT_MAIR_ATTR_IDX_CACHE = 1, + ARMV8PT_MAIR_ATTR_IDX_DEV = 2, + + ARMV8PT_SH_IS = 3, + ARMV8PT_SH_OS = 2, + + ARMV8PT_AP_UNPRIV = 1, + ARMV8PT_AP_RDONLY = 2, +}; + +/* S2 PTE bits */ +enum { + ARMV8PT_FMT_S2MEMATTR = GENMASK(5, 2), + ARMV8PT_FMT_S2AP = GENMASK(7, 6), +}; + +enum { + /* + * For !S2FWB these code to: + * 1111 = Normal outer write back cachable / Inner Write Back Cachable + * Permit S1 to override + * 0101 = Normal Non-cachable / Inner Non-cachable + * 0001 = Device / Device-nGnRE + * For S2FWB these code to: + * 0110 Force Normal Write Back + * 0101 Normal* is forced Normal-NC, Device unchanged + * 0001 Force Device-nGnRE + */ + ARMV8PT_MEMATTR_FWB_WB = 6, + ARMV8PT_MEMATTR_OIWB = 0xf, + ARMV8PT_MEMATTR_NC = 5, + ARMV8PT_MEMATTR_DEV = 1, + + ARMV8PT_S2AP_READ = 1, + ARMV8PT_S2AP_WRITE = 2, +}; + +#define common_to_armv8pt(common_ptr) \ + container_of_const(common_ptr, struct pt_armv8, common) +#define to_armv8pt(pts) common_to_armv8pt((pts)->range->common) + +static inline pt_oaddr_t armv8pt_oa(const struct pt_state *pts) +{ + u64 entry = pts->entry; + pt_oaddr_t oa; + + oa = log2_mul(FIELD_GET(ARMV8PT_FMT_OA48, entry), PT_GRANUAL_LG2SZ); + + /* LPA support on 64K page size */ + if (PT_GRANUAL_SIZE == SZ_64K) + oa |= ((pt_oaddr_t)FIELD_GET(ARMV8PT_FMT_OA52, entry)) << 52; + return oa; +} + +static inline pt_oaddr_t armv8pt_table_pa(const struct pt_state *pts) +{ + return armv8pt_oa(pts); +} +#define pt_table_pa armv8pt_table_pa + +/* + * Return a block or page entry pointing at a physical address Returns the + * address adjusted for the item in a contiguous case. + */ +static inline pt_oaddr_t armv8pt_item_oa(const struct pt_state *pts) +{ + return armv8pt_oa(pts); +} +#define pt_item_oa armv8pt_item_oa + +static inline bool armv8pt_can_have_leaf(const struct pt_state *pts) +{ + /* + * See D5-18 Translation granule sizes, with block and page sizes, and + * output address ranges + */ + if ((PT_GRANUAL_SIZE == SZ_4K && pts->level > 2) || + (PT_GRANUAL_SIZE == SZ_16K && pts->level > 1) || + (PT_GRANUAL_SIZE == SZ_64K && pts_feature(pts, PT_FEAT_ARMV8_LPA) && pts->level > 2) || + (PT_GRANUAL_SIZE == SZ_64K && !pts_feature(pts, PT_FEAT_ARMV8_LPA) && pts->level > 1)) + return false; + return true; +} +#define pt_can_have_leaf armv8pt_can_have_leaf + +static inline unsigned int armv8pt_table_item_lg2sz(const struct pt_state *pts) +{ + return PT_GRANUAL_LG2SZ + + (PT_TABLEMEM_LG2SZ - ilog2(sizeof(u64))) * pts->level; +} +#define pt_table_item_lg2sz armv8pt_table_item_lg2sz + +/* Number contigous entries that ARMV8PT_FMT_CONTIG will join at this level */ +static inline unsigned short +armv8pt_contig_count_lg2(const struct pt_state *pts) +{ + if (PT_GRANUAL_SIZE == SZ_4K) + return ilog2(16); /* 64KB, 2MB */ + else if (PT_GRANUAL_SIZE == SZ_16K && pts->level == 1) + return ilog2(32); /* 1GB */ + else if (PT_GRANUAL_SIZE == SZ_16K && pts->level == 0) + return ilog2(128); /* 2M */ + else if (PT_GRANUAL_SIZE == SZ_64K) + return ilog2(32); /* 2M, 16G */ + return ilog2(1); +} +#define pt_contig_count_lg2 armv8pt_contig_count_lg2 + +static inline unsigned int +armv8pt_entry_num_contig_lg2(const struct pt_state *pts) +{ + if (pts->entry & ARMV8PT_FMT_CONTIG) + return armv8pt_contig_count_lg2(pts); + return ilog2(1); +} +#define pt_entry_num_contig_lg2 armv8pt_entry_num_contig_lg2 + +static inline pt_vaddr_t armv8pt_full_va_prefix(const struct pt_common *common) +{ + if (pt_feature(common, PT_FEAT_ARMV8_TTBR1)) + return PT_VADDR_MAX; + return 0; +} +#define pt_full_va_prefix armv8pt_full_va_prefix + +static inline unsigned int armv8pt_num_items_lg2(const struct pt_state *pts) +{ + /* FIXME S2 concatenated tables */ + return PT_GRANUAL_LG2SZ - ilog2(sizeof(u64)); +} +#define pt_num_items_lg2 armv8pt_num_items_lg2 + +static inline enum pt_entry_type armv8pt_load_entry_raw(struct pt_state *pts) +{ + const u64 *tablep = pt_cur_table(pts, u64); + u64 entry; + + pts->entry = entry = READ_ONCE(tablep[pts->index]); + if (!(entry & ARMV8PT_FMT_VALID)) + return PT_ENTRY_EMPTY; + if (pts->level != 0 && (entry & ARMV8PT_FMT_TABLE)) + return PT_ENTRY_TABLE; + + /* + * Suppress returning VALID for levels that cannot have a page to remove + * code. + */ + if (!armv8pt_can_have_leaf(pts)) + return PT_ENTRY_EMPTY; + + /* Must be a block or page, don't check the page bit on level 0 */ + return PT_ENTRY_OA; +} +#define pt_load_entry_raw armv8pt_load_entry_raw + +static inline void +armv8pt_install_leaf_entry(struct pt_state *pts, pt_oaddr_t oa, + unsigned int oasz_lg2, + const struct pt_write_attrs *attrs) +{ + unsigned int isz_lg2 = pt_table_item_lg2sz(pts); + u64 *tablep = pt_cur_table(pts, u64); + u64 entry; + + PT_WARN_ON(log2_mod(oa, oasz_lg2)); + + entry = ARMV8PT_FMT_VALID | + FIELD_PREP(ARMV8PT_FMT_OA48, log2_div(oa, PT_GRANUAL_LG2SZ)) | + FIELD_PREP(ARMV8PT_FMT_OA52, oa >> 48) | attrs->descriptor_bits; + + /* + * On the last level the leaf is called a page and has the page/table bit set, + * on other levels it is called a block and has it clear. + */ + if (pts->level == 0) + entry |= ARMV8PT_FMT_PAGE; + + if (oasz_lg2 != isz_lg2) { + u64 *end; + + PT_WARN_ON(oasz_lg2 != isz_lg2 + armv8pt_contig_count_lg2(pts)); + PT_WARN_ON(log2_mod(pts->index, armv8pt_contig_count_lg2(pts))); + + entry |= ARMV8PT_FMT_CONTIG; + tablep += pts->index; + end = tablep + log2_to_int(armv8pt_contig_count_lg2(pts)); + for (; tablep != end; tablep++) { + WRITE_ONCE(*tablep, entry); + entry += FIELD_PREP( + ARMV8PT_FMT_OA48, + log2_to_int(isz_lg2 - PT_GRANUAL_LG2SZ)); + } + } else { + WRITE_ONCE(tablep[pts->index], entry); + } + pts->entry = entry; +} +#define pt_install_leaf_entry armv8pt_install_leaf_entry + +static inline bool armv8pt_install_table(struct pt_state *pts, + pt_oaddr_t table_pa, + const struct pt_write_attrs *attrs) +{ + u64 *tablep = pt_cur_table(pts, u64); + u64 entry; + + entry = ARMV8PT_FMT_VALID | ARMV8PT_FMT_TABLE | + FIELD_PREP(ARMV8PT_FMT_OA48, + log2_div(table_pa, PT_GRANUAL_LG2SZ)) | + FIELD_PREP(ARMV8PT_FMT_OA52, table_pa >> 48); + + if (pts_feature(pts, PT_FEAT_ARMV8_NS)) + entry |= ARMV8PT_FMT_NSTABLE; + + return pt_table_install64(&tablep[pts->index], entry, pts->entry); +} +#define pt_install_table armv8pt_install_table + +static inline void armv8pt_attr_from_entry(const struct pt_state *pts, + struct pt_write_attrs *attrs) +{ + attrs->descriptor_bits = + pts->entry & + (ARMV8PT_FMT_SH | ARMV8PT_FMT_AF | ARMV8PT_FMT_UXN | + ARMV8PT_FMT_PXN | ARMV8PT_FMT_ATTRINDX | ARMV8PT_FMT_AP | + ARMV8PT_FMT_nG | ARMV8PT_FMT_S2MEMATTR | ARMV8PT_FMT_S2AP); +} +#define pt_attr_from_entry armv8pt_attr_from_entry + +static inline void armv8pt_clear_entry(struct pt_state *pts, + unsigned int num_contig_lg2) +{ + u64 *tablep = pt_cur_table(pts, u64); + u64 *end; + + PT_WARN_ON(log2_mod(pts->index, num_contig_lg2)); + + tablep += pts->index; + end = tablep + log2_to_int(num_contig_lg2); + for (; tablep != end; tablep++) + WRITE_ONCE(*tablep, 0); +} +#define pt_clear_entry armv8pt_clear_entry + +/* + * Call fn over all the items in an entry. If the entry is contiguous this + * iterates over the entire contiguous entry, including items preceding + * pts->va. always_inline avoids an indirect function call. + */ +static __always_inline bool armv8pt_reduce_contig(const struct pt_state *pts, + bool (*fn)(u64 *tablep, + u64 entry)) +{ + u64 *tablep = pt_cur_table(pts, u64); + + if (pts->entry & ARMV8PT_FMT_CONTIG) { + unsigned int num_contig_lg2 = armv8pt_contig_count_lg2(pts); + u64 *end; + + tablep += log2_set_mod(pts->index, 0, num_contig_lg2); + end = tablep + log2_to_int(num_contig_lg2); + for (; tablep != end; tablep++) + if (fn(tablep, READ_ONCE(*tablep))) + return true; + return false; + } + return fn(tablep + pts->index, pts->entry); +} + +static inline bool armv8pt_check_is_dirty_s1(u64 *tablep, u64 entry) +{ + return (entry & (ARMV8PT_FMT_DBM | + FIELD_PREP(ARMV8PT_FMT_AP, ARMV8PT_AP_RDONLY))) == + ARMV8PT_FMT_DBM; +} + +static bool armv6pt_clear_dirty_s1(u64 *tablep, u64 entry) +{ + WRITE_ONCE(*tablep, + entry | FIELD_PREP(ARMV8PT_FMT_AP, ARMV8PT_AP_RDONLY)); + return false; +} + +static inline bool armv8pt_check_is_dirty_s2(u64 *tablep, u64 entry) +{ + const u64 DIRTY = ARMV8PT_FMT_DBM | + FIELD_PREP(ARMV8PT_FMT_S2AP, ARMV8PT_S2AP_WRITE); + + return (entry & DIRTY) == DIRTY; +} + +static bool armv6pt_clear_dirty_s2(u64 *tablep, u64 entry) +{ + WRITE_ONCE(*tablep, entry & ~(u64)FIELD_PREP(ARMV8PT_FMT_S2AP, + ARMV8PT_S2AP_WRITE)); + return false; +} + +static inline bool armv8pt_entry_write_is_dirty(const struct pt_state *pts) +{ + if (!pts_feature(pts, PT_FEAT_ARMV8_S2)) + return armv8pt_reduce_contig(pts, armv8pt_check_is_dirty_s1); + else + return armv8pt_reduce_contig(pts, armv8pt_check_is_dirty_s2); +} +#define pt_entry_write_is_dirty armv8pt_entry_write_is_dirty + +static inline void armv8pt_entry_set_write_clean(struct pt_state *pts) +{ + if (!pts_feature(pts, PT_FEAT_ARMV8_S2)) + armv8pt_reduce_contig(pts, armv6pt_clear_dirty_s1); + else + armv8pt_reduce_contig(pts, armv6pt_clear_dirty_s2); +} +#define pt_entry_set_write_clean armv8pt_entry_set_write_clean + +/* --- iommu */ +#include +#include + +#define pt_iommu_table pt_iommu_armv8 + +/* The common struct is in the per-format common struct */ +static inline struct pt_common *common_from_iommu(struct pt_iommu *iommu_table) +{ + return &container_of(iommu_table, struct pt_iommu_table, iommu) + ->armpt.common; +} + +static inline struct pt_iommu *iommu_from_common(struct pt_common *common) +{ + return &container_of(common, struct pt_iommu_table, armpt.common)->iommu; +} + +static inline int armv8pt_iommu_set_prot(struct pt_common *common, + struct pt_write_attrs *attrs, + unsigned int iommu_prot) +{ + bool is_s1 = !pt_feature(common, PT_FEAT_ARMV8_S2); + u64 pte = 0; + + if (is_s1) { + u64 ap = 0; + + if (!(iommu_prot & IOMMU_WRITE) && (iommu_prot & IOMMU_READ)) + ap |= ARMV8PT_AP_RDONLY; + if (!(iommu_prot & IOMMU_PRIV)) + ap |= ARMV8PT_AP_UNPRIV; + pte = ARMV8PT_FMT_nG | FIELD_PREP(ARMV8PT_FMT_AP, ap); + + if (iommu_prot & IOMMU_MMIO) + pte |= FIELD_PREP(ARMV8PT_FMT_ATTRINDX, + ARMV8PT_MAIR_ATTR_IDX_DEV); + else if (iommu_prot & IOMMU_CACHE) + pte |= FIELD_PREP(ARMV8PT_FMT_ATTRINDX, + ARMV8PT_MAIR_ATTR_IDX_CACHE); + } else { + u64 s2ap = 0; + + if (iommu_prot & IOMMU_READ) + s2ap |= ARMV8PT_S2AP_READ; + if (iommu_prot & IOMMU_WRITE) + s2ap |= ARMV8PT_S2AP_WRITE; + pte = FIELD_PREP(ARMV8PT_FMT_S2AP, s2ap); + + if (iommu_prot & IOMMU_MMIO) + pte |= FIELD_PREP(ARMV8PT_FMT_S2MEMATTR, + ARMV8PT_MEMATTR_DEV); + else if ((iommu_prot & IOMMU_CACHE) && + pt_feature(common, PT_FEAT_ARMV8_S2FWB)) + pte |= FIELD_PREP(ARMV8PT_FMT_S2MEMATTR, + ARMV8PT_MEMATTR_FWB_WB); + else if (iommu_prot & IOMMU_CACHE) + pte |= FIELD_PREP(ARMV8PT_FMT_S2MEMATTR, + ARMV8PT_MEMATTR_OIWB); + else + pte |= FIELD_PREP(ARMV8PT_FMT_S2MEMATTR, + ARMV8PT_MEMATTR_NC); + } + + /* + * For DBM the writable entry starts out dirty to avoid the HW doing + * memory accesses to dirty it. We can just leave the DBM bit + * permanently set with no cost. + */ + if (pt_feature(common, PT_FEAT_ARMV8_DBM) && (iommu_prot & IOMMU_WRITE)) + pte |= ARMV8PT_FMT_DBM; + + if (iommu_prot & IOMMU_CACHE) + pte |= FIELD_PREP(ARMV8PT_FMT_SH, ARMV8PT_SH_IS); + else + pte |= FIELD_PREP(ARMV8PT_FMT_SH, ARMV8PT_SH_OS); + + /* FIXME for mali: + pte |= ARM_LPAE_PTE_SH_OS; + */ + + if (iommu_prot & IOMMU_NOEXEC) + pte |= ARMV8PT_FMT_UXN | ARMV8PT_FMT_PXN; + + if (pt_feature(common, PT_FEAT_ARMV8_NS)) + pte |= ARMV8PT_FMT_NS; + + // FIXME not on mali: + pte |= ARMV8PT_FMT_AF; + + attrs->descriptor_bits = pte; + return 0; +} +#define pt_iommu_set_prot armv8pt_iommu_set_prot + +static inline int armv8pt_iommu_fmt_init(struct pt_iommu_armv8 *iommu_table, + struct pt_iommu_armv8_cfg *cfg) +{ + struct pt_armv8 *armv8pt = &iommu_table->armpt; + unsigned int levels; + + /* Atomicity of dirty bits conflicts with an incoherent cache */ + if ((cfg->features & PT_FEAT_ARMV8_DBM) && + (cfg->features & PT_FEAT_DMA_INCOHERENT)) + return -EOPNOTSUPP; + + /* FIXME are these inputs supposed to be an exact request, or a HW capability? */ + + if (cfg->ias_lg2 <= PT_GRANUAL_LG2SZ) + return -EINVAL; + + if ((PT_GRANUAL_SIZE == SZ_64K && cfg->oas_lg2 > 52) || + (PT_GRANUAL_SIZE != SZ_64K && cfg->oas_lg2 > 48)) + return -EINVAL; + + /*if (cfg->ias > 48) + table->feat_lva = true; */ + + cfg->ias_lg2 = min(cfg->ias_lg2, PT_MAX_VA_ADDRESS_LG2); + + levels = DIV_ROUND_UP(cfg->ias_lg2 - PT_GRANUAL_LG2SZ, + PT_GRANUAL_LG2SZ - ilog2(sizeof(u64))); + if (levels > PT_MAX_TOP_LEVEL + 1) + return -EINVAL; + + /* + * Table D5-6 PA size implications for the VTCR_EL2.{T0SZ, SL0} + * Single level is not supported without FEAT_TTST, which we are not + * implementing. + */ + if (pt_feature(&armv8pt->common, PT_FEAT_ARMV8_S2) && + PT_GRANUAL_SIZE == SZ_4K && levels == 1) + return -EINVAL; + + /* FIXME - test me S2 concatenated translation tables + if (levels > 1 && cfg->is_s2 && + cfg->ias_lg2 - (ARMV8PT_LVL0_ITEM_LG2SZ * (levels - 1)) <= 4) + levels--; + */ + pt_top_set_level(&armv8pt->common, levels - 1); + armv8pt->common.max_vasz_lg2 = cfg->ias_lg2; + armv8pt->common.max_oasz_lg2 = cfg->oas_lg2; + return 0; +} +#define pt_iommu_fmt_init armv8pt_iommu_fmt_init + +#if defined(GENERIC_PT_KUNIT) +static inline void armv8pt_kunit_setup_cfg(struct pt_iommu_armv8_cfg *cfg) +{ + cfg->ias_lg2 = 48; + cfg->oas_lg2 = 48; + + cfg->features &= ~(BIT(PT_FEAT_ARMV8_TTBR1) | BIT(PT_FEAT_ARMV8_S2) | + BIT(PT_FEAT_ARMV8_DBM) | BIT(PT_FEAT_ARMV8_S2FWB) | + BIT(PT_FEAT_ARMV8_NS)); +} +#define pt_kunit_setup_cfg armv8pt_kunit_setup_cfg +#endif + +#if defined(GENERIC_PT_KUNIT) && IS_ENABLED(CONFIG_IOMMU_IO_PGTABLE_LPAE) +#include + +static struct io_pgtable_ops * +armv8pt_iommu_alloc_io_pgtable(struct pt_iommu_armv8_cfg *cfg, + struct device *iommu_dev, + struct io_pgtable_cfg **unused_pgtbl_cfg) +{ + struct io_pgtable_cfg pgtbl_cfg = {}; + enum io_pgtable_fmt fmt; + + pgtbl_cfg.ias = cfg->ias_lg2; + pgtbl_cfg.oas = cfg->oas_lg2; + if (PT_GRANUAL_SIZE == SZ_64K) + pgtbl_cfg.pgsize_bitmap |= SZ_64K | SZ_512M; + if (PT_GRANUAL_SIZE == SZ_16K) + pgtbl_cfg.pgsize_bitmap |= SZ_16K | SZ_32M; + if (PT_GRANUAL_SIZE == SZ_4K) + pgtbl_cfg.pgsize_bitmap |= SZ_4K | SZ_2M | SZ_1G; + pgtbl_cfg.coherent_walk = true; + + if (cfg->features & BIT(PT_FEAT_ARMV8_S2)) + fmt = ARM_64_LPAE_S2; + else + fmt = ARM_64_LPAE_S1; + + return alloc_io_pgtable_ops(fmt, &pgtbl_cfg, NULL); +} +#define pt_iommu_alloc_io_pgtable armv8pt_iommu_alloc_io_pgtable + +static void armv8pt_iommu_setup_ref_table(struct pt_iommu_armv8 *iommu_table, + struct io_pgtable_ops *pgtbl_ops) +{ + struct io_pgtable_cfg *pgtbl_cfg = + &io_pgtable_ops_to_pgtable(pgtbl_ops)->cfg; + struct pt_common *common = &iommu_table->armpt.common; + + /* FIXME should determine the level from the pgtbl_cfg */ + if (pt_feature(common, PT_FEAT_ARMV8_S2)) + pt_top_set(common, __va(pgtbl_cfg->arm_lpae_s2_cfg.vttbr), + pt_top_get_level(common)); + else + pt_top_set(common, __va(pgtbl_cfg->arm_lpae_s1_cfg.ttbr), + pt_top_get_level(common)); +} +#define pt_iommu_setup_ref_table armv8pt_iommu_setup_ref_table + +static u64 armv8pt_kunit_cmp_mask_entry(struct pt_state *pts) +{ + if (pts->type == PT_ENTRY_TABLE) + return pts->entry & (~(u64)(ARMV8PT_FMT_OA48)); + return pts->entry & (~(u64)ARMV8PT_FMT_CONTIG); +} +#define pt_kunit_cmp_mask_entry armv8pt_kunit_cmp_mask_entry +#endif + +#endif diff --git a/drivers/iommu/generic_pt/fmt/defs_armv8.h b/drivers/iommu/generic_pt/fmt/defs_armv8.h new file mode 100644 index 00000000000000..751372a6024e4a --- /dev/null +++ b/drivers/iommu/generic_pt/fmt/defs_armv8.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES + * + * VMSAv8-64 translation table in AArch64 mode + * + */ +#ifndef __GENERIC_PT_FMT_DEFS_ARMV8_H +#define __GENERIC_PT_FMT_DEFS_ARMV8_H + +#include +#include + +/* Header self-compile default defines */ +#ifndef ARMV8_GRANUAL_SIZE +#define ARMV8_GRANUAL_SIZE 4096 +#endif + +typedef u64 pt_vaddr_t; +typedef u64 pt_oaddr_t; + +struct armv8pt_write_attrs { + u64 descriptor_bits; + gfp_t gfp; +}; +#define pt_write_attrs armv8pt_write_attrs + +#endif diff --git a/drivers/iommu/generic_pt/fmt/iommu_armv8_16k.c b/drivers/iommu/generic_pt/fmt/iommu_armv8_16k.c new file mode 100644 index 00000000000000..46a5aead0007fc --- /dev/null +++ b/drivers/iommu/generic_pt/fmt/iommu_armv8_16k.c @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES + */ +#define PT_FMT armv8 +#define PT_FMT_VARIANT 16k +#define PT_SUPPORTED_FEATURES \ + (BIT(PT_FEAT_DMA_INCOHERENT) | BIT(PT_FEAT_ARMV8_LPA) | \ + BIT(PT_FEAT_ARMV8_S2) | BIT(PT_FEAT_ARMV8_DBM) | \ + BIT(PT_FEAT_ARMV8_S2FWB)) +#define ARMV8_GRANUAL_SIZE 16384 + +#include "iommu_template.h" diff --git a/drivers/iommu/generic_pt/fmt/iommu_armv8_4k.c b/drivers/iommu/generic_pt/fmt/iommu_armv8_4k.c new file mode 100644 index 00000000000000..2143104dfe0d4d --- /dev/null +++ b/drivers/iommu/generic_pt/fmt/iommu_armv8_4k.c @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES + */ +#define PT_FMT armv8 +#define PT_FMT_VARIANT 4k +#define PT_SUPPORTED_FEATURES \ + (BIT(PT_FEAT_DMA_INCOHERENT) | BIT(PT_FEAT_ARMV8_LPA) | \ + BIT(PT_FEAT_ARMV8_S2) | BIT(PT_FEAT_ARMV8_DBM) | \ + BIT(PT_FEAT_ARMV8_S2FWB)) +#define ARMV8_GRANUAL_SIZE 4096 + +#include "iommu_template.h" diff --git a/drivers/iommu/generic_pt/fmt/iommu_armv8_64k.c b/drivers/iommu/generic_pt/fmt/iommu_armv8_64k.c new file mode 100644 index 00000000000000..df008e716b6017 --- /dev/null +++ b/drivers/iommu/generic_pt/fmt/iommu_armv8_64k.c @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES + */ +#define PT_FMT armv8 +#define PT_FMT_VARIANT 64k +#define PT_SUPPORTED_FEATURES \ + (BIT(PT_FEAT_DMA_INCOHERENT) | BIT(PT_FEAT_ARMV8_LPA) | \ + BIT(PT_FEAT_ARMV8_S2) | BIT(PT_FEAT_ARMV8_DBM) | \ + BIT(PT_FEAT_ARMV8_S2FWB)) +#define ARMV8_GRANUAL_SIZE 65536 + +#include "iommu_template.h" diff --git a/include/linux/generic_pt/common.h b/include/linux/generic_pt/common.h index 6a865dbf075192..6c8296b1dd1a65 100644 --- a/include/linux/generic_pt/common.h +++ b/include/linux/generic_pt/common.h @@ -100,4 +100,26 @@ enum { PT_FEAT_FMT_START, }; +struct pt_armv8 { + struct pt_common common; +}; + +enum { + /* Use the upper address space instead of lower */ + PT_FEAT_ARMV8_TTBR1 = PT_FEAT_FMT_START, + /* + * Large Physical Address extension allows larger page sizes on 64k. + * Larger physical addresess are always supported + */ + PT_FEAT_ARMV8_LPA, + /* Use the Stage 2 format instead of Stage 1 */ + PT_FEAT_ARMV8_S2, + /* Use Dirty Bit Modifier, necessary for IOMMU dirty tracking */ + PT_FEAT_ARMV8_DBM, + /* For S2 uses the Force Write Back coding of the S2MEMATTR */ + PT_FEAT_ARMV8_S2FWB, + /* Set the NS and NSTable bits in all entries */ + PT_FEAT_ARMV8_NS, +}; + #endif diff --git a/include/linux/generic_pt/iommu.h b/include/linux/generic_pt/iommu.h index f77f6aef3f5958..64af0043d127bc 100644 --- a/include/linux/generic_pt/iommu.h +++ b/include/linux/generic_pt/iommu.h @@ -204,4 +204,77 @@ static inline void pt_iommu_deinit(struct pt_iommu *iommu_table) iommu_table->ops->deinit(iommu_table); } +struct pt_iommu_armv8 { + struct pt_iommu iommu; + struct pt_armv8 armpt; +}; + +struct pt_iommu_armv8_cfg { + struct device *iommu_device; + unsigned int features; + /* Input Address Size lg2 */ + u8 ias_lg2; + /* Output Address Size lg2 */ + u8 oas_lg2; +}; + +int pt_iommu_armv8_4k_init(struct pt_iommu_armv8 *table, + struct pt_iommu_armv8_cfg *cfg, gfp_t gfp); +int pt_iommu_armv8_16k_init(struct pt_iommu_armv8 *table, + struct pt_iommu_armv8_cfg *cfg, gfp_t gfp); +int pt_iommu_armv8_64k_init(struct pt_iommu_armv8 *table, + struct pt_iommu_armv8_cfg *cfg, gfp_t gfp); + +static size_t __pt_iommu_armv8_granuals_to_lg2(size_t granual_sizes) +{ + size_t supported_granuals = 0; + + if (IS_ENABLED(CONFIG_IOMMU_PT_ARMV8_4K)) + supported_granuals |= BIT(12); + if (IS_ENABLED(CONFIG_IOMMU_PT_ARMV8_16K)) + supported_granuals |= BIT(14); + if (IS_ENABLED(CONFIG_IOMMU_PT_ARMV8_64K)) + supported_granuals |= BIT(16); + + granual_sizes &= supported_granuals; + if (!granual_sizes) + return 0; + + /* Prefer the CPU page size if possible */ + if (granual_sizes & PAGE_SIZE) + return PAGE_SHIFT; + + /* + * Otherwise prefer the largest page size smaller than the CPU page + * size + */ + if (granual_sizes % PAGE_SIZE) + return ilog2(rounddown_pow_of_two(granual_sizes % PAGE_SIZE)); + + /* Otherwise use the smallest page size available */ + return __ffs(granual_sizes); +} + +static inline int pt_iommu_armv8_init(struct pt_iommu_armv8 *table, + struct pt_iommu_armv8_cfg *cfg, + size_t granual_sizes, gfp_t gfp) +{ + switch (__pt_iommu_armv8_granuals_to_lg2(granual_sizes)) { + case 12: + if (!IS_ENABLED(CONFIG_IOMMU_PT_ARMV8_4K)) + return -EOPNOTSUPP; + return pt_iommu_armv8_4k_init(table, cfg, gfp); + case 14: + if (!IS_ENABLED(CONFIG_IOMMU_PT_ARMV8_16K)) + return -EOPNOTSUPP; + return pt_iommu_armv8_16k_init(table, cfg, gfp); + case 16: + if (!IS_ENABLED(CONFIG_IOMMU_PT_ARMV8_64K)) + return -EOPNOTSUPP; + return pt_iommu_armv8_64k_init(table, cfg, gfp); + default: + return -EOPNOTSUPP; + } +} + #endif