From patchwork Wed Mar 27 18:07:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607328 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 65B3ECD1288 for ; Wed, 27 Mar 2024 19:18:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=g3SvHxftZ4th35QWUcggtih68fmXLRO+InJg0Fr6cVk=; b=f+PbzGiTwzLWbW BMOvmYBqsVjyR1+oNmEBwA08q3DqeyxhUeynQK4iRHn6Yf280d6yKMydelYGAjaBLqKiT4ngQ8tr6 WaqXpH2XHyOrEWi3jffbkrk6xtUEEKd+cGEXjrm5pEOGmrwHABwZSOQsvgWO/tWN4L1jpSFv8ENjN jOKImXqvbwWiA03+SMVw44o/AiPmO1r/es91tDIhfQEunG4ac/kGcceJH/C423GV0CG7QwwJqkFay 1jSmzNt7G/glEyRcblqXQCjbCyKDPVSgVi6ZIxPb3PPRrLgCqPWFYzU3CxbbTMQuwCOyODjQUn1SQ 50n7remDFadJ0YTvAzbA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn5-0000000AkyF-06j0; Wed, 27 Mar 2024 19:18:27 +0000 Received: from mail-dm6nam10on20611.outbound.protection.outlook.com ([2a01:111:f400:7e88::611] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhd-0000000ATdz-1kzJ for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:47 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gc5vBK2ZEdNOWa65gUnUI/PzUf5NGwE5N/4dCD8WUhryZ81+fjhCeJCjz9qidNNymBEhq1bqQBBffxIGJJn/BqLRjOnAxp6gWwUEd/TIPqhy5KBJOutWWHrBVlzbuXWgWUR0smXHF6CJzIGC3i2bUZ4NHJPUDCLlDPG7au5dkz3/NFDjUHeJyfBF1/sJ8TaFjYMxfGkpvB2DIk6qtyYj4bEFM5bBYJW3G6YQDthm5qo+Tspnm1q85sQ8pNNIuPE2a6WhaprW1Q8WUCsXcwSUC6BZd4y4O7vJ8KmnHk1moUNj55CPNddf8GTQrc5WXlHOEuHYnYWzB+xfkoUA9g3vtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YNsVxxAr5oOUMzPY6oB1kMo+IdzMqLmUtep2e7LaAM4=; b=KTxl+AX+tGzuOWCpbEHgxnD64a8eoaQHU5AS2K+53uBZFY9NRoI26FDttRN96L+Nj/OhRz4fPUwo4Vbphn8yVV+SmdAqGuWsdebZWU+D3vA0nxTJ8BYqr51GeugSgiKMeusiURiAEGAah6mbxPfFzd5CEaXLu9nHfK+vqSQSnGSGFM0UBZpP+Uc1FmjQyi9T5fU2IkYtYvquvg+OntZW8pa/WBnsNIMvg5r17dR2KONKHwt7stK2YkMD96knvBNu2rsvqZdcYPNc7VzZqMOMsOzkP+OxhbxrkNnmrxLBPbhBM31Ptd87aExN/pq4TfQC0A6DZfwG9UaoJWeM9MuLXQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YNsVxxAr5oOUMzPY6oB1kMo+IdzMqLmUtep2e7LaAM4=; b=J7pxMqNCmaXJNYaMZIG8exLchX3VY3+Xfd1C4tKJaEuMdHw0YCSOGU9MPkWT6z4hHrT/gQzFgD9YfqQcu6GqcGy5Zt83i/lbbxSZCvYvq7h92FoXr41ZwG8ULoYnKw2iWCdK2WrtB9PB3+/BmEcULS+2T5PFdNV0ad0ANciw3MQ9Jg3jgna9fErdbbOJjCL5vXBYbvn/S3MknsjvdwzoWV/NSXqIovk96kdDfxegWqFaa5jyEKgQRtOXx+g0T3N5Zh1iG/cESEQ4lmF9aFpWmiUtcapo9hVcLPJDr27SdjB2A6/TdA5m9YgkoUpBCNop2guK91xN3jnNx8XNFYuIbA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by IA1PR12MB6044.namprd12.prod.outlook.com (2603:10b6:208:3d4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.33; Wed, 27 Mar 2024 18:08:33 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:33 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 01/29] iommu: Validate the PASID in iommu_attach_device_pasid() Date: Wed, 27 Mar 2024 15:07:47 -0300 Message-ID: <1-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR20CA0020.namprd20.prod.outlook.com (2603:10b6:208:e8::33) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|IA1PR12MB6044:EE_ X-MS-Office365-Filtering-Correlation-Id: a050baea-c1ec-441c-3f90-08dc4e88e22c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vfyaMpaGsUzbA788ubvGJcwVQVjW9zAU4idRKpqoiGnpS85nU2pYInuxEuQBrTQk8o8LqsahxIHnVDQY+Q+MirIRbILhGBFJYtcXRNDGajNHH5F+n11vhzp5cCfwGoRpBerhPwjY8oFD+tYaDlo5VblYz+35t3fDwz/ssAF534BM3fkK0Ia5do9dH7ZfvMGCeUiHNv1XlvRzes2utB6/V2Aw8GBPsxXYgO0z9+hW+epOmNNLUMMGOykKKRU9v6JcbDVP6A6no4RHyJpHk7Y+6zGDT0plMA3O7aLJDU+0epasnVVcj+g1/dktMmu0O885nS8sUH5FwEqgfykoiaGXuVuDXXyllroz+xpULAVBEjnr3GhUJpnb5RWWn85vOO2WEkHmtZE3tRwdi4bBBHifzZR2XLrrmLTyBr0cj2MqoPT8iTeLgjiuz8kOvNsIO97Df/P3yDfxHk4tMjqz6ub94pdlmIcvJyWzI0ntTZNlu0R/Z/BWWgzDhuPKQ+C19Ui8oCXv//92hWQJhalRwoqQyN5PJ7emebZ3YtPzhubr9rcZjgszuYGUYwdc5owG1y7F2YAMdQVhY7EHB6srq8d6lk0aRL3l2V8aZ7XXDkWD0PIp2l4nyXA2xpRNjlwBVCyV7vFuTtdgGMJy00I+EYsBlrhDbIP+WjJWrLgzPbTE2RU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7416005)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qDYX4rQFxMoT/quDN7WBKNAL0NZGr6y8E3ygZnP8LZc/q0cGaOYOFMqwZLwDzuR2bxRBpqEexSbTFmuQe3j5H7uGoAxJ5FU9SCefp0D77wb+DBl/p6790aPzsIUnFHMHRxHAu0IIrODNKYk0Kd4tzKVRUCOgU1U8VDrgQ9BTTP11WV2rFPtH+3idp89FMzCMo2Yi0kUoItMXcpQgAf0k+vNC2IuRbyxf2KHWgYjZ35qEm0mvXNfPoIExzNlw2xtfJfNO3grL8+jcGTUiWXGxD2SzkanzUpVbM/mZ+i8YkcJaVsX3PIDqBmKRXhW/od8TJ2DervKhvTsgSjWrVZHze43sfEVB5CU0zkHTEd5tM7Bu+1pwxKnl/FPwu7As0DEZqGFuLM5jVM0bT3a936xI31xTV5z18JIbTFOw+CCNw5EWvXmuJzqY1U5csJNyGfA5xPBbh7OADJ5NhSk2VfUQyKD7vc6vxFxjYXIhnmx/w6TaYNyEyzISMWMpmCbOVQjcvSMqhjJuIDiYOun261vxStY4WYfugHM32qSpfLkDHx+kFIIaB4Ed4+lIUbofDG1KVtwWuYO5H2YtEVPECGcOBigikweDNX3KAHMjGvdiDxPPJPBGFuSCBbJZokVNV0AH6XN/TkAyoK/Y47630Bd2BhiBJYmD7P7WPuoHMwP6qQuM0hafT2iQKgbnOfgC7XMuv6Vhkw9DaTJVFFqCgQaC66sDZl1DZTnpyI0rp2KBp0Ft9gBWENzdcpEtUNanA/5gNWOR0dxHbZ6kJOlDCeiyTiXXfNxVRVnkobnDtfhG82r6hz1ay5szozF50HPldLA1gSawddQhl5Pqf0fXL9FlN8c9rYsjJ1w03yOxOVuDef3ETn5aQ/O4wGjHRvyzy5lnugolZGx4LhoUZpF2SztjcSn54Fmj3d28XpYmbkYt7NV76rOV44cFOYB3UOPTzLQxJrW5Gm2KUwgzd3O2UkLc6JnMVR886nrnrq8r+pXgSJI6BQWEQc3P2prv5pufs0dPrGri7y5nPiJMTrpJstNsMBnHUcENTZ0iTE5BlityRkZkpDodNzSGnferdr7WU8DhW29InBZzAPfEKbJKy2k6Lt6X4/fgS8UcavqQey3F6efGuqx8vowqaVxN250AOqW6oC2kY2yeNLstP458U8g7krx8XrL0W9fzAJgNsErm+UwYvIkHj0eknOga2KqGXrCaRJXJe4+BPzqGnzGd8lR54mtfNb7FmWukahbiwC5QSXe+T96kk5LBr7JoM2CnwXHN5JyTdTWdFLsdiGn1cIcfWwtvYss80DYlgA/KWm7+3wDKelLpHJ8gIGfio+/7FN/py/lSAWnVYVWbk2xUmKXgFsKU+6agaCTNAFh2PuVwtn3sssJsFnnNaRVU19eJ0qZqjLTe3pKejbpppjHpv32jenyJl8k6rPhK1UBm6N5/w85B+CFlqBDXFzICZCXX8oihHlSdz4nNq2GEhgtIyVjdMLiPEE+wRGPa6oi7Y8ckMENfWRJs9WkxFN7AuzRkKpQ3/TUOD1N3dgt1vyoQHR/AZqh2brHgezqEMQVoowzTlVb+QauR6Mdn5Rss0kHdVvNL X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: a050baea-c1ec-441c-3f90-08dc4e88e22c X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:19.7766 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: YWXi3VqY+pLmszmSYMt9jtILmsAP6+n4DBt6ozZVOI5WzaUnW00bpisOgiZeQGWp X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6044 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110845_636187_21DDDC3F X-CRM114-Status: GOOD ( 13.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SVA code checks that the PASID is valid for the device when assigning the PASID to the MM, but the normal PAGING related path does not check it. Devices that don't support PASID or PASID values too large for the device should not invoke the driver callback. The drivers should rely on the core code for this enforcement. Fixes: 16603704559c ("iommu: Add attach/detach_dev_pasid iommu interfaces") Signed-off-by: Jason Gunthorpe --- drivers/iommu/iommu.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 098869007c69e5..a95a483def2d2a 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -3354,6 +3354,7 @@ int iommu_attach_device_pasid(struct iommu_domain *domain, { /* Caller must be a probed driver on dev */ struct iommu_group *group = dev->iommu_group; + struct group_device *device; void *curr; int ret; @@ -3363,10 +3364,18 @@ int iommu_attach_device_pasid(struct iommu_domain *domain, if (!group) return -ENODEV; - if (!dev_has_iommu(dev) || dev_iommu_ops(dev) != domain->owner) + if (!dev_has_iommu(dev) || dev_iommu_ops(dev) != domain->owner || + pasid == IOMMU_NO_PASID) return -EINVAL; mutex_lock(&group->mutex); + for_each_group_device(group, device) { + if (pasid >= device->dev->iommu->max_pasids) { + ret = -EINVAL; + goto out_unlock; + } + } + curr = xa_cmpxchg(&group->pasid_array, pasid, NULL, domain, GFP_KERNEL); if (curr) { ret = xa_err(curr) ? : -EBUSY; From patchwork Wed Mar 27 18:07:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AA549C54E67 for ; Wed, 27 Mar 2024 19:18:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RgKyCIMl2fbkdIr+GD6MKaYYYsDw4FEE2i6XlpfEzyY=; b=gKJFa1kXWVlJht M/g8u3oEVvbZNbGPgAgk+b4yrejuzUIqp8M6xf1GEJHQAW4ihTQ/Z0LGO/z9osISZP9XFEYG96Ex5 74A16kuhzKF6pLcXG0p3qRaJi/KDiIqeuujYouApmt2jUriSDVhXolfYyV3h5xdBDeDVa0wAyLjU+ D9E6ivXBJ8hSvFd1dW6rZQkzFE2+TFkuoROQgl0AETIs+oYj12c3TLP7/IsozQt7tDcvkyKflm6Pv Kl8WYP0BTcoVmfmIFdpv1aa0q31LYGoPrhlp+fjZxa2LsqUWc907L7yj8Oyda0alPQ0KySOMgxtVV dSrMI3ukgWYgzTbLgeug==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn0-0000000AkvH-3shB; Wed, 27 Mar 2024 19:18:22 +0000 Received: from mail-dm6nam10on20604.outbound.protection.outlook.com ([2a01:111:f400:7e88::604] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhQ-0000000ATY9-1Co8 for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:33 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HNv3lPqx801rlxjKGeym4/WsB0BhwzVyfHXFdmr9fkz7CtQSk6GPu3yHSLnwPlNKHo2ESXI0wcMFPFW37JeZ0vgYdhReKHH9TMwtsDyPJiOqs9ePdWjTzm2HUYRp15glaF3DvPDt7NlTIitD8fSxyI9hXWcqd1DAaTHvFyBppnj4x17C/IDPZF1yMnX8/JcKlEoSHgqSzzN7ZeSrKaXhNa20rM1sgPh6TXGC0IhjDokx1AFPB+uwkZISlk0HJILeC7A3/xcbtz/0l6awVhFKcQWohBOK0z8c0mgkIslDCPgSy15nGPz998cEXk74vf7/9aXD3EO01jJn9OOI8xIlVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+q+NyN7EmQpLSiCvfUmAP8PFcDmU7QJ/E6ybS8LRe0Y=; b=cBEB1kUkN9PV3ytrRnjaTeLQaMzkNuam5ci89VvfiTU0Oxmihc1gQNEinai0bKbpk+RR/NV/BLnPPFfxxXH25dV6urNawmKRyLARQwO9ODvw+I5ahQ7zE8RXdNs/Z00Xw7Om0doQyN3mF84QCiuCVbEByldd6SvXwyCqPSdSGaZfnrwfs6BGuVNyUqXedCiYazp4dGiU1y3iC7grIPJsxUXWG3gMyELWsgaizy2qsORB1/p9SMs9QFcdP3kgoFkJdE0mQNz6VOauaPDrg+VuAFhFJgRB2KBPWtdzchJl5C2Bg0yh2UpmKUnvsVUQkMM+oB5rtx8z8T9z0P8XkS/fSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+q+NyN7EmQpLSiCvfUmAP8PFcDmU7QJ/E6ybS8LRe0Y=; b=M/OwlqHpkLPW2P0wz12nvrltXy17GVo5Eb1onmJ8vaF6Ds1JiAjv0SxByET0zvDuuoVKYrvLj8HWQn//227qzQhRgWcAaoAMlS2QPRIouqgj7Mz0m2C7T4cS/XHyoPI1BQ6K0W7WurN6xHw3dRW2h+CbsYoWC15NCqtUIUDZ8XRcsNjO/rP+8ELzTj6tQd6PziBe7Puo0smm/waaFSU90kHRnOuo9jvim7bqZKV/8FOZJPEBZ4UHWEXht08Ghs4MmHSME80A3ZBHpgZdJc9Pnsyw2CUbmGZ5rQNmqHE0UA50aBTT4W+c9rLoNrcHnICbsOS+dIoZjpFRRWRZHGP6sA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:19 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:19 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 02/29] iommu/arm-smmu-v3: Add cpu_to_le64() around STRTAB_STE_0_V Date: Wed, 27 Mar 2024 15:07:48 -0300 Message-ID: <2-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0310.namprd13.prod.outlook.com (2603:10b6:208:2c1::15) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 8d84f27a-1e5d-4ebc-b2a0-08dc4e88e0da X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ldJs1LAGHsSeN/dvAUkTEVWTxZD/e2trYP9Fa3lATVbVhf9jEpye3rBHSsgMCBm0npf6tZ9x9OlA+tCDhB1jh7VzIOVx18AtzUVTlq7EkgyrE+vbZ0mKlJBgxGE/dQj50WQDTRzAcxzzKshetP5SkkueBdEKKF5MVr37iN7AsDnIoVcuThilzo2HLk/CBk1EDrekh/txaQtne6VTgQ0QCEC/x3BPl8EynFXB03OW7uqJ20EFzwUZAUlBPL9rfDERIv6UpyP+2YlXfPNOlBp5HwT4PHeeBaWYrGN8NhocXGLlWut92ySxRAYUcr1W5xX8ctfkI6bRGI9CqGMJfs+H5vOlcIKi8ooq/UxBY4CELoNqN+yRU6KbxhljhipIRQzrC9nGbqO5YnocMQTgr7Aunge1BXT2MnzWgN5VezI3F/90IV9bZ6QpsTThlpPFQyUsL7gNZQquHHvma9ZdZBuK3PhVqSEuV4bwA9pPttFF7Jy30Tmjh3gJJGBvNFgHuGMHRsJf1VwRhoVidrWD9PdvZTapEorVaHWh+7U/lWJeo885CQZ9IfoXTGDH0Fteh83veQRq8P2rLdj8N3DEy1ZUKbHSJctk4fWTgiNbuSfVCzzS5ksG8SbKIoS/VcfYxE8uJA8Wzt8PHhEb7izbi/I82+dSwmFZqv9Dg3nHjCN/7Mg= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: zLUWM2e3hImP+aEQHZYo2t8EjOiE4g7T+GJcCRkDg8SzeRU60dNw4DRlIRvUAtSn0yR3PzMvGbBZ9Fu9YUkHgTSh/8E2TUjD5B6AfKzcjjpJXJMRqRC29qqWjOAWeVOXYgd0xCeawvBghKdrY3oJHTBsApWGJTq5ZaQG2PlIxCA7kut2vQOfDD58mDTdchCzCwhTHWmm5UjvfwmH2W36PR/6hjrrPUQtBEjQidgND6HfdHyDlV5LKfvNtezIS7kHixNrZpNQ82u7Hs7rCDh+28jZodEl0KoRXSLXp+MvWfv+CWvXwsxuE1OpqQPWOpEAzLeOoOwEkKgZrLzCUktZHaKYTehVB6Z2HVCliXGqtCj4HLamaRxb/Je37ZhKt+4tgeY+A1BgTyEuur50+Vo7FlJhswgIOCPfGHs6PS0rM0MF81FKzUSgBbIUs/qF9A10yd24IV4syJT3dugt78tbZ3v1eDctunJW89c/BvxaMOhDQ8S6sjRNPv1ZaeMXMAZo7ybLHvM/4HxLPkdfYr4vyDD7LFP51+TasbF6ijopS/uoQyFnPxmIdrBIr4ZCYstequ9zcvJ3gCGDMfj7jJBpGQXTFvUa1C+GLB34pwa1DDtYF4v5tkVb4gSZhTGVLB9e7hr5jjxOVDGflBWnCiNxayoeEbk8m1dE3unkWM5RtGsMTtthMU5d3OuCZpO7xA7+bAMVCdr2pogdUPUhdWGbGp+xm16eA2AWJrX0sVHJY9kHKXnajUO1piKprcvzS2zza3SjC15XOp2heKITxBSdNCbPO3dvQfunB6SYBjTpVTk2puVjSJ95MpWnCQS9hrhMkduYkva2HKAxKoWtoVGwplVYKWOhHqCeRDV8MxNh3Hj/KDV5e5fvUjFDQN62813zYuwiBGxveUZ6+2DbqIZMt1hkoFenEnVQ0dIwxTcILV4haczqDhDMH3QRKIAXFxUXA+WFhGyt0ndzYm0Li9fCaKoDFvhOFuICOrdHP7QwfonyqVl/sH9pDV03B9Ju1yGxPDrIL93dANP6T6xN6D8O3v7mPS+ADYQoWC3I4niGIb4cIQK1Vc4In+m8u2o/Hq9x5h80gRWE9pqUvTa+Bv489xSMQG57unpGEzGhvDSNtdhRVnUU/Ped1wSoKQDca6rNiYgbUzkp/BDDWyK/xp/vaQJvUtslgPH06n8/Ttiafaf3hpjc++YWaqW8fJJM8S8/5MBugHjRtVlU9QKhtJTyIs/9/50jhY3zZ1l+rUYTbm6jKMNxouNBfLibbLRP3saQkA9/5NaMusgATTGOACRTWbbWZW/X2IGLHpQ3ORv2Q+QFJZmazniWYvL1SUh6QqyImHjlaSKLGa503ldGz6ynPId8oQLwQKM1q4Dxbk5LV1Ri0Zxj/bKhyffUR2++0eK7R5/cifY13/qjDbC2WwuNf+Y5/ze/eXcq35isfI2aGAiNa43PfAiTGFvgNH1S00A1O1Rd4OLgKVYfZl+IPstjuOoNY8O05RSNV40/6r3TO4vnVu486D/VAejXgMGnb7kyCRR48M/oJY7JA67RONDRMWT9HUEtagaXLM+FS6JZv7t/mSD6WdPRSI+8GZJguax/ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8d84f27a-1e5d-4ebc-b2a0-08dc4e88e0da X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:17.6065 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2cG8P9oMYKn7mnF4I6nuFVY02/qOpQxaZyta7v0U34JQqaAHbD0GB5bercgU8hAF X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110832_371088_7B0A4042 X-CRM114-Status: UNSURE ( 8.54 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org STRTAB_STE_0_V is a CPU value, it needs conversion for sparse to be clean. The missing annotation was a mistake introduced by splitting the ops out from the STE writer. Fixes: 7da51af9125c ("iommu/arm-smmu-v3: Make STE programming independent of the callers") Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202403011441.5WqGrYjp-lkp@intel.com/ Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 5ed036225e69bb..fa3f3e7d9b0cba 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1139,7 +1139,8 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, * requires a breaking update, zero the V bit, write all qwords * but 0, then set qword 0 */ - unused_update.data[0] = entry->data[0] & (~STRTAB_STE_0_V); + unused_update.data[0] = entry->data[0] & + cpu_to_le64(~STRTAB_STE_0_V); entry_set(smmu, sid, entry, &unused_update, 0, 1); entry_set(smmu, sid, entry, target, 1, num_entry_qwords - 1); entry_set(smmu, sid, entry, target, 0, 1); From patchwork Wed Mar 27 18:07:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A8DF3C54E67 for ; Wed, 27 Mar 2024 19:18:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ywewusmc0LK+r9YbqYwra4i5ft8TRmlT3UcTzwPoyTU=; b=YZgfD/CcUvy3Gf 6l6eNTBhNaCUrhAXBJGMDWSPIW+OeN5t+sEbjhTVBCgoNnVwoLHp876KZtHZLVs5c3uWx/JC/mG6A EqQmXJXUWRZZfjsu/rpaaAuF9QqUYcoMsz7avbzMBHnajp2CYpipGiuL1f6ASYOAWcmsHx9hG1yg4 Z0SvDpXYhdOIWZ9nU6UrUOFxiBHOb1gzsVyMtSEtIyOSk37RjLAMLAYxfKsj9BI5Ze3ZM3Bh79PPB hT0iihFh16EJaqg58rl39Z7ViuzAddI0G1Z5aqCkgUN3VNoUYJ3jatj0gh2D3VDGRWXNCHQS0pe8A OHMF8UGVgDyB/7F56ADg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn2-0000000Akw2-0rdN; Wed, 27 Mar 2024 19:18:24 +0000 Received: from mail-dm6nam10on20604.outbound.protection.outlook.com ([2a01:111:f400:7e88::604] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhV-0000000ATY9-2mVi for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Xnvh1FLhRnn49a01NqAfug52JiQXfsVPj51foa1z7zDL5eSOH0pblPomijiK42DW03SCrogeids1aPC4bM261weRAHkTq2UI5W89vg0RzUvCIIX0oWWza6i0DCvKCdLFt/cI0u7P91NjiqYOEBeN/RFFplSRkfg5oA8FUWiy4713bB8oT2RoQ6RuNIV7y+kuWNObLb7OYa6nZsQIP5XbnX7K7IbRb/InNW3TQ2B+P//0nY+EWUGEW2FW5OV7qd6jMBrxM2H+doqFJtRJGJ2AXFyPvG9QcpvCP17TLlmBfR/cylvOjlwD05qFfDAa+e6TmPOflQtNwRAaeIiHPw9+0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=EYplJVT6i2QO8Qki5MFmA2QlS4i7sXa+pl68wMcz9v4=; b=PactBvjsMJHLVMEekzPy+YxeYAc6P4pyrRxGoc72vxv4KiXOESCPGmluIhvD1CU94AlcTs+H5W1AC2vr7YQq/q61AaWfGe3qZPzMdXrWuHz0FyladTwghH+LSBipeQXosAFf9CrDPCAKMqtjjsxk93l8kK7cWaq+aRSvzF9rREIJcKWDxkeNHCu/f+5FwXnPokV90eIJ0VCIzUKoB80jwMRmLSPEuJwodKGn9CFzyPViGKdeg7HCCfWqqTWpmseXEJ4DexO+pxDPILZmiidlXeFiP5nlSsGyPXswxXS1Fca/us1/3/emp7uIIxD7+AUow+A8W4cca1rvbm8HcrWHYQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EYplJVT6i2QO8Qki5MFmA2QlS4i7sXa+pl68wMcz9v4=; b=IdAN5VgsBXYxYGJt/2biJb2/cQJSCAjUIB0cs0Ouqkqv1iHp/p/R0XrQb85TtNWaFAXz9pFEIjJu0v7up+KeB4xU2kByGDLjAaq8YkXcf1/z+bxqP/4qOTSjFLFTPie439kgfiek5xeAyNJDOU195KyLhVRAhb+pPrZKMESPu9tg4W9l4vi44ZesIcN2NDp3da31tf0ocWvvWA+pY82Su1BQ1DoXONVy8OY1J8KWIACCfgkApiVfd1sNy3euGXZ0MV2ZhRUQt5Z7q+z7Em1NwR84eRrrow4Kv9JtU7WCr1wJN9uqEiwvYeXt3nR3xdiCsiVr2IwtWl9SGDqGjs8EwA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:23 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:23 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 03/29] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID Date: Wed, 27 Mar 2024 15:07:49 -0300 Message-ID: <3-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0312.namprd13.prod.outlook.com (2603:10b6:208:2c1::17) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 9b05ab97-48a3-45fe-893d-08dc4e88e14a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: yS83FIXJw/rNUHZ0fQ6cB5R/E2wyGRo6nKfH5c6iS5LxkH+7bEE7GS43f3RvlhtKMAOag5G84PtQDDzfmRO7no52ol4jSy0KJaN7JPn5r5fsQ8t6ESIUZD6GLC1gz9v0/v8wKz/G1OcnD3GGse1aQLQESYrh15bWKFn2q4DIB1Vbqun8M5ayDuU66x13kCsx3ostOLyn3HiA8C55YgKOSu1jtjz+Kx+oBfwfXN9OOVPfF08NjjhmnznDIQW2ctLWV2q1Si6rNW0of9lto24YFEDqrCgv80ysOf4ijsmUqd5ryMfYLh9i9CqXBAqXcU7Xqkrlrgp0zbNnG3Ke4s8B6jG73YD5XyvD+OroKvsb6CY4Gwri3CBngZYtIDNFMsURyvTFjLJXaNc9PbFk7jNp7dlc2VI57EPaTXkATZFWa1slBywBDmJx+Efe3udY6rn80oYVpBnwgms/RIWQ9RRC+8Abfs7PvcrVUhpv/gusiPX2gDrixrkFVwHvFaCvgJ/OeGDMFschEuN4JPaFIgGSHH/P80AQ/JM9vU0jSFKRDhsmkrNy5OrMRSgazViH/LCPEVjYgSlPlppWZ5TJSqp5rUeThA4HPIRHbcQjZzsbbEoabUkkPsd3nvyVrNECRBFYyVbJt2JX+rNW40Eaek+yHIR4pvoHDgh2frGlfzsj9Ug= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: I10IeHzrwokqWPwR8pV/AkVxrifCcG/zzKkfbBJOSb7QAclE5tU/T+77qxTiCf+QlNRo4TO5rAzaq7qLSVTv9q4lQJGdea2BFtGxZ6u/H5LJ6fzRs3wLp+pHW/EMrZUH3NqNeo/f4KjT1amCHZ0bxin9x2lk2QlWm1jiDmyoumzdQdvhh5tM0KKnWF9EtqKct03u7DLVqPdph1TsFDsxjMihj2Zmp+34v0s95Lu4/NVU14kt/60Buk2XFCxuoNmfj5OHATr0Nuy5v+X2/gPwgPQMgGspcJADpxt4aN0lQv2Zi136Ddrs5ZiQ3Rp6MfJaw9ojp8688ZF6bZ+98tGoW3P9eCMUgcaeycZP/GMrO4NASyfelnLxglukpZU1Q2SHMsOkoLSPlBHPasBDF+yR/dKJF+GPoi7iGC5dxPkx6NxIIy2jCX9bZKx7IUZGsLvYzo+3nQO7XIS5vlUuumIX9qz6JWSLd4RbQJvzLDQywC/zjSdpV/N3ySYkFaIKwOTSxbbMERuHVAj6c0LjvkcZ41fYg6KCvf//59F34sGVpSox8nY9cHBmfJ+szZ5M63DQM0DIh1Xfl/ufF8di4bso9QcszM7Dlpsi5BQ48kZvIgOkoHEZ/FHLFxPKgwXcbCnTN8rojG5IjoJFciMighvZrRjUB1dmGvgnTFz9oS5YKkkKV22tcECuNbsmUECS6Pa4S3CKASXBnYL7IvZMRuk/DGzUdCprvCxsGP1rurJb5gohzL63JymuXIqWKX6lehV74+eI0dFg2uxrawVWF8zq+1smpy2Er1hhauJR18ASQ7/tOa2ZbRmsbTbrLu5yZcAlhM6WoZymS4aIxLnrcC6vVGSH4RbkIkynjbkoxwp5kpNFgfDmLKy0p7FtL+ICqTdYVGyGTBeP7PEt7DwpH87kP9h6Xc8hRanoJy+KOL5BQ1bpagn6eTS8S1ZPGdlzRp1iblDOxLSwb45cuQCfvrCvnNmy1a9HuJBv7AoFEydBwrqvzL2tiqXiXcXn6o+P4mYd9Dxx5HUdgluee1WnkP3O846NBunYAtPhnCglmovUlIHqqs5mp+XSXcPwbvze6oDR0e1hFcM09qO/f5zwIl4TDHa1XjEmn+daroqWGdx7BZUXhXEDdjNhv7tlmRZMDx6FlnUOV6rP3Jr/InJn6L3Y+jFYh1akE/ybOF7QB9hm99nvVObsi8aNRitM8L2Abns2J5MqSPcT2FG9EuSd3XEBoeaeOK00U4jYOeVh95IE9raJbQy6i99Z3f9uLiFMfr7m6amUFSvgSwJVYrUudiNjsfsS9EEgux9uPMT5TfI9E6j9b52/Dp+v6RnAr0ycvtB0XVowRDTBJiRSUuEIinxQZs4WFcjZ87GobpfmaDpqev+FSYsl26lV+wNzqMSAoBvc2gi6P/NPxWDzuNQdFpuV8uTgp1KYNsPGp01Ql3IuBXMq+UcjTsniIWISxyiWEfdRGkY6ZNnNIPGeNwza+ODl8GRvnQjIGuqHhl7lMWKKZpZMzIgps3TjQJ3ZWog/AmEp2rTcCxgD7Ud8g6iUPljja5diUGSe9MsxY5xjVtm1gyuGYiZmBfvTBeXUwi69FWBK X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9b05ab97-48a3-45fe-893d-08dc4e88e14a X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:18.3188 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XkFZWYEIma5zz8NaP3JvgZJ8BPxTqJyFVuzTDmqDvrnd3s7J3XDK+XawEVu2kDgk X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110837_829196_84F4BD75 X-CRM114-Status: GOOD ( 10.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SVA code is wired to assume that the SVA is programmed onto the mm->pasid. The current core code always does this, so it is fine. Add a check for clarity. Fixes: 386fa64fd52b ("arm-smmu-v3/sva: Add SVA domain support") Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Mostafa Saleh Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 2cd433a9c8a0fa..41b44baef15e80 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -569,6 +569,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, int ret = 0; struct mm_struct *mm = domain->mm; + if (mm_get_enqcmd_pasid(mm) != id) + return -EINVAL; + mutex_lock(&sva_lock); ret = __arm_smmu_sva_bind(dev, id, mm); mutex_unlock(&sva_lock); From patchwork Wed Mar 27 18:07:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 59702CD1284 for ; Wed, 27 Mar 2024 19:18:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2kDfjl444Rgv9G3UuoOpPjjIADcMRNquiTlMtvgULzw=; b=U/Fl0VkhifW7YL NpN2+9LwPyBiQ5rqIkmkdMc/pstP665ornGK7kEHhmxs+go+/dHzhu/y4DLVPZSyZSyeVkjd8U1Jl 7OWJzVVIb4cgJ+a3SNFJh03/yp/siKGH0spZ/efWckqt5wRlIJLZNmedN0XfPXOTABJqolCILEtWX LZOpIfnD64CzG+JSo5xvJSrLzRFmFRB8ODcriMHBVKDJu++hIsR8SPdcNjZ+uphZiawEfUcl5hnxJ wCME6gRWG3LxcnG7VqiK1/kr8QyDaCw91o6q6S9ip283RHcvdyE2LJzAFGZDnr44oUu1ag8DsvXDK odRQO2NeRvxSxIEDPRfQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn4-0000000AkxN-0vM6; Wed, 27 Mar 2024 19:18:26 +0000 Received: from mail-dm6nam10on2043.outbound.protection.outlook.com ([40.107.93.43] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhb-0000000ATfM-2C7L for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EF0oKuAqeRTIDYXYC60eMAl4J9aM8/tUwAWvKW6l/+Ni5Y20x280fL7yLMk1rKjQBLXDEwTK4+jjyasl4WmSCSDf401Aa8wjoW9Cu8WvkTNDhHP2xtcDbyb2IOpYtbEJ3DdombQiTz+4Nz6p5EJHRDXEMwYNbt/IDAOoiQmPT5An1BpKw9TUuihglV+kZQ46C2XVFQFIgMxVP6fvlj7aa25a31M/6iNxZle1b704XmPJqDqeb1pm0nDPvz0P9us2po/Fai5snktNqh+T+zQitpk4vaT83Zz/2mCQd8191H6a88clVqWMW6cl2OKuUy6863vDtmp0PeA0qXO3p/5vZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CjcPxWTDoKE5QE2U1JWsA/frcjjI8u6RrYDI05HXnBY=; b=KV6bBYlSm7s8FyqfOaav+26/r0me0YeeeXdSjmflTqFwlKhuVjQVC8FSGdij8IFCa+e3q1tnCSN5e9FYhvdt/I7uuA3uSSf/Rf2fMa8ddWyepgpyu8YjCTH1iLLgHs7ASv46DHlLTw1g+anJPhBJSVVNal4PfC7qauiMRM9KGJENAUHaBgbM9lTyiOzFkBvFvPLtavps9QCxWUIh0avn/OryI9S0Ovk0sBZRp3uyvQsknmrnubedL1nbw/D7Rcdz/hEwwy8OAmTU/zElwvbpHGhejSgQ7Hy6gTrFXXOkcml+nbllmdTrq+4KckkyHLNvae5BObGPbVaX4ZzHoIoLVQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CjcPxWTDoKE5QE2U1JWsA/frcjjI8u6RrYDI05HXnBY=; b=kb8RWZJvIuYD6+OxZY8bCU04D4Pa0/Z0fC11LoIGb84y/9DeE2Mta5DdzfZ4WAldoEo7/UKHhvJxBGgl8aLM7TLb+T+pJVKV6CJeVN0Pasa28GbeOMdXse5Sv2jAmR6kVoVaYjqTMOKJoObNlgtDOoTk655UWtdYj96zprNAX9wPY2RNWCfxzwEbqvnYc2FXr/M3IEQWE+I/q+/EnBw4Q4GtFmGTwuP3W6C5mXfxFCyG6BHAjh9lABuxQWO61ogbjoFDT0vp9TOwResyMy2hqK/+Mw9YeeeH0vzyf5hff4w6Sp9soYucdI/EjFYWOurWb2J+OzqpG9eIFzjTyfw/dQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:20 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:19 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 04/29] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain Date: Wed, 27 Mar 2024 15:07:50 -0300 Message-ID: <4-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR05CA0011.namprd05.prod.outlook.com (2603:10b6:208:91::21) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: ddd7296e-d957-4d45-9275-08dc4e88e0de X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: G+2KXiu9Gb+fPkHUXsW67hu/RjXz6t07jViQ0VQ7jFNFxajXTGS8VQBV2RL+eLMHufQWaLn4KN/FrZK6QXL+a0/GTrrhlnxNLNsBia2qUbEk5F/8CUb7qrYMl6MKl23l107870dAuFSJFxi38Gd8qdcDkwz8WEXpk5rvV9jOt6HKH3o8uJNSaspIj0gl0lJA3vEcHhItDuAboBYWRQqgQwMlunA0AJ+VcZdXND1f5CzvS+C1U66SkZym7nW3k5DePY/Tiyaukx66swuAOBXS/nLqEXeaC1m9uo8vItdPth+eWW/wX0/vunPAduU2ltgOLd1U9d6cHDsfy0b5cRMdGxeA6U4+Qsoeyx+YGVtIAWy8H80E9o3A49DH1zGJufRmM1SqkB8gXbA69Oj2CBLM8uY+uOonvvdrBSlWIS3MlLgAH2PrNdfcaurm8Puxh4WJ7sKjX5Z5TRTMj2y8zH5NUnw6ygMyPmuX1kfj/BLydKvn34Aceyj3Pquu2bVRAOMskn+Mn3ZkhpUOOpPtGAY3JaPaaX/Ou+HG2LgM2LCGmuegb1RzyMLaEFNPd7ER/pAozrDiI0CoQSvfDd7BByB0YXp4W/xCOXND/ix0C3UkYKNoij+o1MugQ5+SzgMq3Yzg4GBh+NWOKL8UzXzcVyzxz8EnKVUw29uIoRIwdodY6VI= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ksISuH4XfcgxbAZFvSPqnMu0QlNJf4Z2antcxyKpTlQCOVCSQWeYnxXDVvDf45MmEioKhG7r/lNp+jIl3k8OXUQVhl7cRS+xLDgiRNR4kGlY3LcEdftHj85kHKayGXkoCD4ZPpgJhpL4fortR7e2wqBuF6vzlF39o4ptd0CoAaBx0OOG5ZPD/k/M2251zX9c8nvK+LqEDyLsf2IysKLW9lBaiVu+5de8m7B4AdzilDyvCoNz0Vz+xVqeAfGw06HdERgU3R77fdIezycaO/CKN5B/IJYSotdboAMBPtYyQ8nogp0V6+pTfXJDBPbwOKm8XDKtFCVtJK7L1q0Lsbltyj7bAFoBp7/I1GFpQ8bVQXRuCdlcsVHaBfB+3yfIADDWFAFPH/kJW++QMvqEpRXsOJKyh0kkwjWsXY7AICTarBoazoHLyX/JKLiyLD9K3bntpbo8FGz0HAs8Sl+Cae3ToKUcDX4amt+PGIWojJdSG/d9S/0sy1n+Uid51zrO4tVJ12hn2sJJiCCVCqr0zhyG9JeQFqsE+h9YDRR7EhtKxs15kqm3eKJjmLtO+acvr5b2hx6ObRYfPECU4YOODEr7x2UfEo8vG3GruULmanUvytddr3QM0E7RQv4iOKJW7/xmCVxqcdcrzXojNpJe/xL0mekuPBertVCKvYdB8LoKQ+O38Iw+D36bN8g1pO5v19uxBImxy+JRCNsOtbBIE+zTLFcXZANK+ZDQSnrXMze57TBXf6r0N1kwrB0Dt5vpq/nOHQexusM0Q73J4XiICogBQfmoKWy7BTXesygmO5Ujuvx3fhuZjDsdAyQGxC4cpYVlZBNVb0nDcQBV6wT1o7pNp7AQqYq8bbebRhmFJcc+A/wPsDEnM+GI7Q03OyIdOSagUzesIzdp7DrQ0zbxTPKIpy3WPr6gDATw1DChSDwWaV00c1yVZfOrQsg7kolJrSaFc0myci2y2RHXyNdDI4ohurKrlMX3sG7GrCvPo0fil8QJAyc+kM70QKSFTYYlfMjyHBTVlFLbSceVLnerXvScHUjNpBNyhREhKPKyN08bcky3AwC9GyoT5OPOchgyqXjynS8YWpmvRse7bJOLdTqFe+SfyBpQnNJWS12ncWhheYZqC6o/wVtrWduAZc8X01y2ELL9nOp15gq2l4jdyPOyjHHfth3Hhkozk+K6eDV/IJxWnhDcsSC5Jix+yprTuuerTh2JXzpj3tWsBpxFMWCxnJKT0+C3QzK6Xc4kDI4fSCECR59b0041mR15YNq5eiyQZByR4zLyFDWlOBLGy4KKhVigurCWndi8s6MpeW6xgAROL0vXQ1UB10TN/DuGDxGCnrxfTQjs9/16pH+o7GaAA2C1iyu3y/Fec1wZEXWPCpxeGsiHwUVc+r9p5eJJMZK6hg9PNXy3jLOgJr3+2x4R2xYJzb26rtQodULDn5n7mEPaGMMeb5mDOL7ym9vnHDGNJDwFYIhLoTCimWotfvrR2j51VOtH/g49J95qqdqwTuxu+l/P8ZynNe+NsTbOj3JAGCXmKMQ48NUELKmwOwN9OqXt5HPUFHRXHBp2bc3FveQM5yGQ7ulp9oaP+lONcO8J X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ddd7296e-d957-4d45-9275-08dc4e88e0de X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:17.6153 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: DXqo4Pubz945+lY15SSSbZnxPewPOJecW3PamLQwz2X1pYm0QUOHcXbd/PT4iLnf X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110843_728471_268E80FB X-CRM114-Status: GOOD ( 11.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org At this point we know which master we are going to change the PCI config on, this is the only device we need to invalidate. Switch arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master(). Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Reviewed-by: Nicolin Chen Reviewed-by: Moritz Fischer Reviewed-by: Mostafa Saleh Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index fa3f3e7d9b0cba..7480f70701a045 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2412,7 +2412,10 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master, pdev = to_pci_dev(master->dev); atomic_inc(&smmu_domain->nr_ats_masters); - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0); + /* + * ATC invalidation of PASID 0 causes the entire ATC to be flushed. + */ + arm_smmu_atc_inv_master(master); if (pci_enable_ats(pdev, stu)) dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu); } From patchwork Wed Mar 27 18:07:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607262 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E17B8C54E67 for ; Wed, 27 Mar 2024 18:13:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4maJsvZwrRKHCEkWTmNkux2qAsboEGiuHo+WVkyieLs=; b=YsE/iaiDlhGMiZ BB9vgQi9OsIaagMRJpf6ELEdsZ7lo36kVd5ukEWKVLp6dIjFDAWK3L+Y1Qy5ht3kWMWv7F/6g+p61 juvH/Vf8LSQ5/fGDIhiDlFhTol5eOF59d0xFUUZm14rT/co/0XOS1HiRvNyMGl6dGWvGlV1D5nxrl j9+HQYJLx87EhCx3CbPPlQ7t47qKrZA/ACVsHym3LhbM1ggkl+EhdHLP0hS1oHa05RAgNYBXh2Or2 1ah8W9PBEZle7PI8aJXNRCB95aPWmswaVs0/b7T9KFg2HMgARY+U1mDctoSIj3ShgUvTpPVWRsC80 keefjN2NQB8ce6Ys6LQA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXmA-0000000AVys-02yz; Wed, 27 Mar 2024 18:13:26 +0000 Received: from mail-dm6nam10on20611.outbound.protection.outlook.com ([2a01:111:f400:7e88::611] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhg-0000000ATdz-08Nu for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:51 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PNWQtYUMMDgJrIa4DqOrDxSt4WqnkIJ85wX0O9l6U+uloa9Dr+WOPxSgwL9yBMS3eLkl6COmEWYeZWDftVNm2JxpU9A/friXyOFzZxKCpTjiY6+Zzgy/AmD7nSnzDA+FvEs37vcBrmKmSDIYO80gAHhFfBsLJDjbGWbY4gQk9lotduHukPhpJPlJi4mh92XwK4kf61JiH7u1s8+urKL8m1iaBOfmMV8zRDMfkWHqxBueL50oqKdgALz+IdKt8fFDquWqSJjH6RnF7g4RG2fl6mITn0Bbvb2++pJJBEpw8ooDfgrc85JL30z2tWiV7XjHdHYQHLrgOxGKKJTO+b+WOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hyL/S52DRL+W1JlOeobeWf4tel8CHEt6/GhtyCTxIxk=; b=cQ3IcCbAR2EI53+P9z57qPIepvrDt9b0GmJur9IT8ssbc7AAb2Twg+IjXe4xjfAsrpC1NzcX7zySm1r3bvxSHnlMJBRQVzMFTwim+G9GvZoV2vRkNNCOYNiVtgqt1bCbjumpa6iU20KBlrL1FhcrEjnrZdhNAspRUP+88fglG2ei+imnUMvydPtd6uWYHxhdNpJqT/sHiO5TJ0pBPaWLj01jTXGEuWF3SQMr9Oy0Ri/sDtXMRnTwoX4VQuJoWH1wQMDMlQjwB/G/LyTM7DTx8F42RbYuN+Wz53IG8GivqN6TxBbSCb/xlipZOn5IMF+CyT/A8TeXBE2a8+Ec1fbDlg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hyL/S52DRL+W1JlOeobeWf4tel8CHEt6/GhtyCTxIxk=; b=OTWy3qy7FYdaP4TxfFV99C/MNWJuneB53xqA9CAAl1XXbbg0uSa+MoK7SHyxcSlOIxTNUD8/+VnP/lHFMw25mFX2ptv4S5K7a30tCNMaFxeQaGsdDrDlyhcj2EWzt/6uQ1M8uNPZqgs0kNGtn57Fa5sYtwM3pSatSWqLHof3ldv503FCDY7v+hIr4JHN3/7HLJb/GCbbawvlwo7ntK73OjnvR9YuJWuRPvy59Qebl42JhjKulGxKkLhgRh2uoBi3yzrwjK04oTQ+cwHvifGgTX2w5gyP6YRL1UOuwZmhCIhUfh5KxMLPCenqffItmTST2m1s2/PItdlIrHz3Wgmw0w== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by IA1PR12MB6044.namprd12.prod.outlook.com (2603:10b6:208:3d4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.33; Wed, 27 Mar 2024 18:08:32 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:32 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 05/29] iommu/arm-smmu-v3: Add a type for the CD entry Date: Wed, 27 Mar 2024 15:07:51 -0300 Message-ID: <5-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0046.namprd05.prod.outlook.com (2603:10b6:208:236::15) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|IA1PR12MB6044:EE_ X-MS-Office365-Filtering-Correlation-Id: 58bafbf6-c3e6-41ce-e36c-08dc4e88e222 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZdMd3ejNZawLe+BKoGxyE07HCvagD1diw0OVj6oIbeijOoZmmns0Q9Sv29HdMtXM0Q5r41vnVirT9n4rBpR7ZV/lHRpEt4xxdLDATh3uMnfJWNx0Fjt6rJ5zHEPtndMDsZih05z6YAx04cePZB9nj9aOOXFXOWzEeH7KCxujrsVKR+2x9tW9VuYidn3cVVfU5DHFhAZgET1fYfwH8MEthyRVM5FNv0HQ7VXv6Oz2fHlbingwEHcA243VXLMRGaqiYFZNIJt3ox3Mu6Iys9P6xZZy2cPouOpZNoaPtO9H72pO0Hn7cAN5fjY5gcdP/QZf1VDX5DdurzyKfr5jYsaF/ckrOoHg/8Ly0vlkqvrmtg4zqzsT5saQYquwbmlKZtOlnaYaRx/VyvLOPOce6eOOUdyaRQB1pmbIYKohskAOmnyjS5tVFiwZnMsNEsdtQ+9OL1qJGxmiFHF+cFL7HnnBP8oLhumv5zbXs/yj3NAKRTGK3h0265pmpWX9EqskgvA2XemE8lpvrVzQod3QxDZMJWixuoSYc7Jc+vQiSfPfDjTr5+cAMhXnP7voEwS4NxrsogiZY8sppWUOwGsuSpg0bnSpOmWed9O67j1pJ8JrQTHkNu2UKrgpSfZ1wxW07tkEm6t3fS1x+6y8PObAt0OcpNBobzu0FcDJpHWgCW6UMAI= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7416005)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 0l8rTyPwI/pYbJqjriFRCw8ABZof/vjV4sQ1VEUuoqkkmjl8g4yKIe719hRk/fh6L/IqgTwiDUyTnHIxd5cncsx/ohTZE08E/FGhpxmdadlewFVBN/rNVU6uA1UqhXqFNdmWfVYxpwDZ/V5zwGxKEqm4rWroV6+t74n3C/7pkCVf7zo/fBLQWw72yL3gFcTrFgtIewE2WbMcBPjhO4KFTBGAKiLPdm1RHMWGxM14IAcFqdJV8wua6VvaSqeW6jGf6kqyWzbLNcJZHfgLiNDyFMJiZe1/S1Ws/X+RXFoEfPnqNtlo934nK6pI09WkGAWrlvQcMo/FAIrZwpAaWVCQ850T+E/RhUSqmC2yxLeeoLd4ryuwTdUHrIThvRjGS3SFg27MbA906bWV3edh+qTwQnjz/r6feClSXF86faX8NsncPLWheSomVLfS3qM2xejQNTuAfAwT/g+5pvlBKTXZudmyp4itTpLoQ7E8qn3L2HzdZuH30LW96J/pCMZeCpaSFjfqhyfNBcREr2SinVjBgBXY6X9I5zgwtPZMlko+F+R7JvurA8LR+hMJMSz2eq7XDHp5k1z1D3s9udAJeP8hFEk1D0jROyMXGAFNwUe2FzdUGLwjZfKnodfrq8iY0ULB342L4Da6nMQgoPYNJqq7c+B0soa+GhX90xKOMPn3xDCJxJDQSMC16fY1xDqHcBr4XdH8sXUm34k21yl11iBRhTurhrxVqnD4PBycXcbWvSPSsAOkfLoIFjm8ggG/ejV3XAUGwA/7WL/TsXGU0k/MQ1HrOTeL/X/okvBb5FRlc0A+/hoXJtlrAwzIyrnAnZEF37jUAcaghtiGfYaigCVsDIeH22Ue3x7c6HDnXJ0toP3DJt4x555whKAximhHbn1ndWgmTTZJUfSDERtqZS5Rvbemnk08GRAn27oqY6M+7vG2f1BnEhaQ3oLITl7OpopcZZXEGllv4oFGt3QeYBHy8k/VkPaEp5yY/0ZY3XxUH+P8CxYWLb0oBDLdbcadX3bgmbZz4ioly/OXaDr2z9xMJrLjujJXUBvNyDkHfIoWMjiL9mxc4HI7SWBut+nz3e+/dRbWJwD+ZoiULlA6HYJqO38ZwECxj/YCkls2AQEUtNfzOFh/iesD/7vXGBKWcUQIvrLct/PKm41R25Sf5TKzpNUNe44bq3fnkEUMl9a5y3g60VZV7XWg/LATIOqvkAUuKc52vU2vlwszrfqti15L9cyRNAZEhtUibJUAiMTxrSH57rfilbzYv7CSCvuqII7IQ/fnkGEuu1DbY2hRHQPOcdj80Fe36T8rTIhOEAwkpjnGrim24I55Ii+Mf0PYTaziMLLtCLJZFbwpMvyx8KdFBM9xMuokgP4g61rSN3EBhfXnaD3CaqnWMkA1Aqs4ru95QvQcI9+EPWzF1QZQVZl3MI47J9PvuYr8AIMiU87KrfIMhvv2slJ6nWz/0WymakJ1c/VzIsthsWGfTYW6AgVszx505d7r2813UcPIFdMWS/mQ15jT8MZQ9Oxm148oYrcm6z/RDdCYuRdiOXX9XzpTuQucHtEnY9OXlwe7BDF2Uh0AfeBKQfagGrie1t2hJ2uG X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 58bafbf6-c3e6-41ce-e36c-08dc4e88e222 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:19.7069 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fPThecL/FthzAwKOzPf2XbaAVKRGoYVsavp/WJLMNIK/iLZnNn3ROefQl2BuLdCI X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6044 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110848_262775_A90E1A8A X-CRM114-Status: GOOD ( 13.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Instead of passing a naked __le16 * around to represent a CD table entry wrap it in a "struct arm_smmu_cd" with an array of the correct size. This makes it much clearer which functions will comprise the "CD API". Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Reviewed-by: Moritz Fischer Reviewed-by: Nicolin Chen Reviewed-by: Mostafa Saleh Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++--------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++- 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 7480f70701a045..6f62a38052b504 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1214,7 +1214,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst, WRITE_ONCE(*dst, cpu_to_le64(val)); } -static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid) +static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, + u32 ssid) { __le64 *l1ptr; unsigned int idx; @@ -1223,7 +1224,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid) struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR) - return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS; + return (struct arm_smmu_cd *)(cd_table->cdtab + + ssid * CTXDESC_CD_DWORDS); idx = ssid >> CTXDESC_SPLIT; l1_desc = &cd_table->l1_desc[idx]; @@ -1237,7 +1239,7 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid) arm_smmu_sync_cd(master, ssid, false); } idx = ssid & (CTXDESC_L2_ENTRIES - 1); - return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS; + return &l1_desc->l2ptr[idx]; } int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, @@ -1256,7 +1258,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, */ u64 val; bool cd_live; - __le64 *cdptr; + struct arm_smmu_cd *cdptr; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; struct arm_smmu_device *smmu = master->smmu; @@ -1267,7 +1269,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, if (!cdptr) return -ENOMEM; - val = le64_to_cpu(cdptr[0]); + val = le64_to_cpu(cdptr->data[0]); cd_live = !!(val & CTXDESC_CD_0_V); if (!cd) { /* (5) */ @@ -1284,9 +1286,9 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, * this substream's traffic */ } else { /* (1) and (2) */ - cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); - cdptr[2] = 0; - cdptr[3] = cpu_to_le64(cd->mair); + cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); + cdptr->data[2] = 0; + cdptr->data[3] = cpu_to_le64(cd->mair); /* * STE may be live, and the SMMU might read dwords of this CD in any @@ -1318,7 +1320,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, * field within an aligned 64-bit span of a structure can be altered * without first making the structure invalid. */ - WRITE_ONCE(cdptr[0], cpu_to_le64(val)); + WRITE_ONCE(cdptr->data[0], cpu_to_le64(val)); arm_smmu_sync_cd(master, ssid, true); return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 23baf117e7e4b5..7078ed569fd4d3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -282,6 +282,11 @@ struct arm_smmu_ste { #define CTXDESC_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 12) #define CTXDESC_CD_DWORDS 8 + +struct arm_smmu_cd { + __le64 data[CTXDESC_CD_DWORDS]; +}; + #define CTXDESC_CD_0_TCR_T0SZ GENMASK_ULL(5, 0) #define CTXDESC_CD_0_TCR_TG0 GENMASK_ULL(7, 6) #define CTXDESC_CD_0_TCR_IRGN0 GENMASK_ULL(9, 8) @@ -591,7 +596,7 @@ struct arm_smmu_ctx_desc { }; struct arm_smmu_l1_ctx_desc { - __le64 *l2ptr; + struct arm_smmu_cd *l2ptr; dma_addr_t l2ptr_dma; }; From patchwork Wed Mar 27 18:07:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3F7C0C54E67 for ; Wed, 27 Mar 2024 19:18:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cHU8wpB7FOUT6jid8GnYgty4/us7zLZNadC2rb0XTqc=; b=1g/vARivPWg7TX ejx0/uIW7pb5Wn/5FCUpq7vks/GBsWxBWDC3K945fwqXVYJhnL0uSDg8Kdcps8bi6Woc7kB/Wx45I WPk7Ldkpy880KAOYCTXkzIUjs65DJE8FNNWgpxI5VbmL0zSCwcV9pFLd9F1ipDyjCSsWygp5BYEuS stkynuYxQzPVV3afaofgNj5Q42DYyjDRHTErd8Vk4sp/nPRgpus26clERffkbXzoDy80l6y8BHC4C QnL155Rd/BMc0pNGCqCgVAi7ss+GHGMUTuYfM2pV6TMBys81CUxpCkbqMZwfvLCgJFEcr2ifeC2+1 kCIwnoYpi6G29A7ibs/w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn7-0000000Al0I-3g1s; Wed, 27 Mar 2024 19:18:29 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhv-0000000AU2t-0cxK for linux-arm-kernel@bombadil.infradead.org; Wed, 27 Mar 2024 18:09:03 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=eic8gb+150fVmfgrRI8wCitBYk4nxNVChg4sW3oQ14g=; b=sDGOxx55/fuJNYBHPZ4FuCxM11 HVkzUEF5P1z7R+nWgcmlw+Lyh/fV4490xxbGgardRXLwHIQIWcX/Gm6c2qqaqltQAJ3Fajf8Peh7w N1RO6TE+iX8H8Uu+/s2OsAkd+2fZbqcn+bZCZBPzJNAMqmgoQmc84jUWkCS3Ieqtsjdjuil8npiYt OFse2tooATIAM+nNmcRzPthRc+SuNdYAO5Q61nO+uyxFXZQEV2tuoXsFZSTPTDTQJB9leZuOH951G tKBWwXRIihgW9cFBosvP/jHZ1FjW96MA2Q4+XW+pAlj2USf5nf1NRfYk7kajfTZToruFOfw7PdXWv QwV+b4/A==; Received: from mail-dm6nam10on20605.outbound.protection.outlook.com ([2a01:111:f400:7e88::605] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by casper.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhs-00000004QF8-0saz for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:09:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Jr9x3P4uI1eEjowRF9SSK+A4bd826jPPK02TyU1U3Ymm7Ocg+ssBd1aojOIgn3pFoDFA2IpyccqTdfKw0VoEsVFYbwVvO5vUbjn6+QbFR9c5sX5IUO1aMgqlD1kL0TGZw52P3r4e1oWmXAu4ZzvU2RwlSA1K0hpBucPiNuKfCso8IZmv1Zl2CSJyCR0xJ1YxrS9Eoj72zR7MsmebKD6rsBdxuaim88qoNhv023yyli8ykz10vjSf+DO36MFv8H5Zo+hWUuerN5xeM/ATYKO66RdKs/0ul0z4wZKzkvWPvvU5p56CBLVxQ6iFr67I3alF7mLjFaeXwAAn2hZje7ydIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eic8gb+150fVmfgrRI8wCitBYk4nxNVChg4sW3oQ14g=; b=ZgD65PL1YPZZkR1wXFFv3aPy8OI2BqLk/6J4c+BVC7BguFgxB6w8JmAFP8gj2cBC8N4XpOgqtXxtb5FH1oKHVyGSZwwTbab6Vejgv+mX4quh8SHNSk922Ocup/SpDqiZT8pzbSBHbCj0P3wkUwpN37Mn7U6YmQHqf8j3ZRxY+AZ1OFyazhFM/REKIoT988ecofcikAWbg9F9wwjaMBIHXtLigkgTOQ9icqqFc9+0XzbWijYhz8s6joSM80h706/ffHoz4TDgPzyEgjXE/afh960zsiBG34gWlktJDOErsxrIjx1qqAFjQYCkb1LDDxQnvfLaq0Oo7dz1RzS41UvdMA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eic8gb+150fVmfgrRI8wCitBYk4nxNVChg4sW3oQ14g=; b=oCPFD+imMHBgT6HRCwLoQ1kDR6Nh8dVSmOmbBpU7RtKkqtBhO9Mc18aJXxGaZ0f/o2gqnRFv6lQaKYiaQ0LQHIN17+NTeMSYDuISo92cQahP2Y54HvBQvxK2EjyZFTVvzZ3mpcCtYroMZWnqdW3EuLwIaW9ME1oN+ooDesj7Mop3zLOZfeD2P+FW+8MUeoJXwO1LCOtzI0Ij/YequW6OtttKComrjQ5jA9I70T3vbnjfSTqCw9xQtwTxlc/EewkMBbVbI+rqptz3PlHqw0BqBKKLyC96MINYkEbuIY/ZX765XmKnkz3ek9Upf/7/6i/mURTvnquxmjf8dsfSEFn7mA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by IA1PR12MB6044.namprd12.prod.outlook.com (2603:10b6:208:3d4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.33; Wed, 27 Mar 2024 18:08:35 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:35 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 06/29] iommu/arm-smmu-v3: Add an ops indirection to the STE code Date: Wed, 27 Mar 2024 15:07:52 -0300 Message-ID: <6-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0314.namprd13.prod.outlook.com (2603:10b6:208:2c1::19) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|IA1PR12MB6044:EE_ X-MS-Office365-Filtering-Correlation-Id: fa47be6d-f440-4a68-0493-08dc4e88e25a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vtbIsyxw5PklOQcI1zxnBlAq33RHAT6myC2h+QI2Wne7hmHb/hH6h/AndtC2Q008myL2caWpiOIihY84GWgalcjVO8UsrTIseOrSWpG/c/4qs/HHvbqO7DLdaWZUYewzIA7g0TxTC2Sqbig0B/CtXcgxVdNa9BE2VWUQdqhOKbqKxtoJZ1FvzeLasM0AKAlPtuFBdKIoMgY6tPDNrnfZbzqX2hwxg6e8yCZn4gF/tg91eufH4UefikdrU5HP72HwppWvJO6E+BWktp9WEXtLoey1btXu8UQfZyVHQ9L89HYOpHghEboVsk7tAhA2pMctTsaZUphf9wr8spSalIDR6og+OTSgOHAbhX+fO9PhaUT7jW+iPZTvb7B2BOf/AgIjcum9t3jrddQ4Du4z0igH3hsZIxzdmMdm1s9KPZ1nF4G57U5E/00C6f8SJafhj6XweEHDNL86iUfEIByYGxjZoRZC4RrrlZOi/rSDAdmBzvlpYQu287a4WmtCwdONXbvzUzX2b/nFg1Wh6/lhf4vkQ+5fbwMDcf2BBupuTWlWGEGXHZgj0ySwNsZVqj9EL+e1cvPs1B3YX/upavB2cJvDiCDyaYi0ToQVVkqlkEuwG31Goi5cqBrfJG3jLhdwNk5OQQUxJh0sthTj8ByintF+WZGQOwG7vZhap6SESLnEx/g= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7416005)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Vtkp9osnhayqI4axTxOBCyU6qhEv9O4nFggH0OrJcoE/ULUHcBIoyDEfs0tiObZdAyGVvfblhoF2ED2LVYuT9TaBWYw6dUSf2RfY7XM/ObRGiKunjE8EIP7UBJwksxvOc+1gyfbKCZKId+LmpKESDs3c+yEXivZx78JS+76yjfyRpjuczXU+XLJvpsRm3XhBqP1qXM//w5i82uWWOujQV+fyMNcKRo7M/gFnePKSqCmVBCw81TLzH7tFLHav8jdGLnU0NtrF7DC8ZJU8WK2Eq7CA6lk9lhroOdsr2MJQA28FQvnbPKeOz9jA2APBKw0YhWV0+RzTL1BBCUUPOOtA82oPeIq/ftGk5T98Cvnz2FVEmkuPKAwfDQ3It47pTBxFpMXIu7zheRkiSqXAgfJqfOvnaXNA8hKsG/XIqq/j49NpsoLrO2i7aC0PkMGjhZE6mMkqFlU4S5RRSm3jN9rtFOuF+GLBqaeu9cC+zb9KxsMmETV28Q0aAxqNWStkvGiBEdtsBRTY0oWq+B4Ev3lk1etm19enWe4S8gkfwg7j6hFgG479PKf1VTH477FDxjsgsucy/J+iETr9tjo0+KSkLczKKF//CTByfAOhSA7WyxxpGd47d7SfzJDr3hCzBgnPsnO5X7RtEG0EiVYfJMShCOiAtX13FOv0roVo96/fF4o3AsOw1ZGzHSaimPYEmcmvAp96ygpD0VV4ytDIE/OvVTOksBNIJEwi7hmzX9L/WECvFvR/cZGkxC8bqitELf7N2AIFbCJnwrKDEwWsRLX9LBdLg8yBZXhN1RHEFZJxqZCE6E+IHlwNVC2XT/TKVQSPqFw0P6Utw4CraqTdKYhVa6qZkdDqwV1LI8T3WTwU3iGVHpgMNHXhXnNy4IdyGKI1uplbb55PY/F+s36wlG3tY7vOc5SW3p7sf0EjgnFduw+Z6fF+c2SWZPUOtXsRL1SalH12GTiMYVScnY3iM6M93UPPmkOzKpyjFDjK4/Tk2+Inag8tCTl4+q4V/ioPzpL9ODyVoDq9hdwnopIzP4RcsPZHeDI8z2qn+xcWLu/kuryS0XAPNP92q3gIZnUU+LyZCbLp/BdNXi9+Yxrs0+nspSe2Ox0p8AYi1pcZJX1tHxHgbaQzpD4JGaFFTIrspoLZYF7N0+pmcS4Vg/NM0Oc0kYruZ2gRjOYojO3npKua/f13Y+gTGKUvPLLJfojGnXoQElZo5ixqAL4SqFAuQNQjjVWOUincINVC/eLutfab8GeVAldcFjHKuz2iY+f7WB7rs7U4cKTgvLbaDx+0DkfvTjOb7wGRBFfRoEk8jdcyEKFxmoXp6vOh9fzZFsMPMMYRZmpHtYMDg7a1IdDaZ6tWd8jd9NookrT5gP3sirg8RXEzBsOSE9Z+dNK4mdvb+M6rl1ssQrOMEzL/kn2ytemeiM3rbf2l6Q7mYghLS8eW2yVZbdE+U5GQYBDYeY0FEspBWz2O3vdwHqkpZ9pSsz8dzRUkLbvi197oE5LfQ3Vq96fjBRV4GOCubJE8YA4QNzx2wTm3lcF6/x9BxYtz/yKU5t1YXB1vA8EE1X8Ngq2m5vZFYx4XmjkaNxE7sk5KNTz6 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: fa47be6d-f440-4a68-0493-08dc4e88e25a X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:20.1405 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vudIufNpslvH9E+vhEx+Z3qZ/v1f1pCqAG0frVkgTworZAvfTGEk5j0eY6IML53M X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6044 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_180900_272348_4209215E X-CRM114-Status: GOOD ( 19.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Prepare to put the CD code into the same mechanism. Add an ops indirection around all the STE specific code and make the worker functions independent of the entry content being processed. get_used and sync ops are provided to hook the correct code. Signed-off-by: Michael Shavit Reviewed-by: Michael Shavit Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 170 ++++++++++++-------- 1 file changed, 102 insertions(+), 68 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 6f62a38052b504..19fa511cec2c05 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -47,8 +47,20 @@ enum arm_smmu_msi_index { ARM_SMMU_MAX_MSIS, }; -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, - ioasid_t sid); +struct arm_smmu_entry_writer_ops; +struct arm_smmu_entry_writer { + const struct arm_smmu_entry_writer_ops *ops; + struct arm_smmu_master *master; +}; + +struct arm_smmu_entry_writer_ops { + __le64 v_bit; + void (*get_used)(const __le64 *entry, __le64 *used); + void (*sync)(struct arm_smmu_entry_writer *writer); +}; + +#define NUM_ENTRY_QWORDS 8 +static_assert(sizeof(struct arm_smmu_ste) == NUM_ENTRY_QWORDS * sizeof(u64)); static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = { [EVTQ_MSI_INDEX] = { @@ -977,43 +989,42 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) * would be nice if this was complete according to the spec, but minimally it * has to capture the bits this driver uses. */ -static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent, - struct arm_smmu_ste *used_bits) +static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) { - unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent->data[0])); + unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0])); - used_bits->data[0] = cpu_to_le64(STRTAB_STE_0_V); - if (!(ent->data[0] & cpu_to_le64(STRTAB_STE_0_V))) + used_bits[0] = cpu_to_le64(STRTAB_STE_0_V); + if (!(ent[0] & cpu_to_le64(STRTAB_STE_0_V))) return; - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_CFG); + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_CFG); /* S1 translates */ if (cfg & BIT(0)) { - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT | - STRTAB_STE_0_S1CTXPTR_MASK | - STRTAB_STE_0_S1CDMAX); - used_bits->data[1] |= + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT | + STRTAB_STE_0_S1CTXPTR_MASK | + STRTAB_STE_0_S1CDMAX); + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR | STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH | STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW | STRTAB_STE_1_EATS); - used_bits->data[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID); + used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID); } /* S2 translates */ if (cfg & BIT(1)) { - used_bits->data[1] |= + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_EATS | STRTAB_STE_1_SHCFG); - used_bits->data[2] |= + used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR | STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI | STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2R); - used_bits->data[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK); + used_bits[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK); } if (cfg == STRTAB_STE_0_CFG_BYPASS) - used_bits->data[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG); + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG); } /* @@ -1022,57 +1033,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent, * unused_update is an intermediate value of entry that has unused bits set to * their new values. */ -static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry, - const struct arm_smmu_ste *target, - struct arm_smmu_ste *unused_update) +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer, + const __le64 *entry, const __le64 *target, + __le64 *unused_update) { - struct arm_smmu_ste target_used = {}; - struct arm_smmu_ste cur_used = {}; + __le64 target_used[NUM_ENTRY_QWORDS] = {}; + __le64 cur_used[NUM_ENTRY_QWORDS] = {}; u8 used_qword_diff = 0; unsigned int i; - arm_smmu_get_ste_used(entry, &cur_used); - arm_smmu_get_ste_used(target, &target_used); + writer->ops->get_used(entry, cur_used); + writer->ops->get_used(target, target_used); - for (i = 0; i != ARRAY_SIZE(target_used.data); i++) { + for (i = 0; i != NUM_ENTRY_QWORDS; i++) { /* * Check that masks are up to date, the make functions are not * allowed to set a bit to 1 if the used function doesn't say it * is used. */ - WARN_ON_ONCE(target->data[i] & ~target_used.data[i]); + WARN_ON_ONCE(target[i] & ~target_used[i]); /* Bits can change because they are not currently being used */ - unused_update->data[i] = (entry->data[i] & cur_used.data[i]) | - (target->data[i] & ~cur_used.data[i]); + unused_update[i] = (entry[i] & cur_used[i]) | + (target[i] & ~cur_used[i]); /* * Each bit indicates that a used bit in a qword needs to be * changed after unused_update is applied. */ - if ((unused_update->data[i] & target_used.data[i]) != - target->data[i]) + if ((unused_update[i] & target_used[i]) != target[i]) used_qword_diff |= 1 << i; } return used_qword_diff; } -static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid, - struct arm_smmu_ste *entry, - const struct arm_smmu_ste *target, unsigned int start, +static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry, + const __le64 *target, unsigned int start, unsigned int len) { bool changed = false; unsigned int i; for (i = start; len != 0; len--, i++) { - if (entry->data[i] != target->data[i]) { - WRITE_ONCE(entry->data[i], target->data[i]); + if (entry[i] != target[i]) { + WRITE_ONCE(entry[i], target[i]); changed = true; } } if (changed) - arm_smmu_sync_ste_for_sid(smmu, sid); + writer->ops->sync(writer); return changed; } @@ -1102,17 +1111,14 @@ static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid, * V=0 process. This relies on the IGNORED behavior described in the * specification. */ -static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, - struct arm_smmu_ste *entry, - const struct arm_smmu_ste *target) +static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, + __le64 *entry, const __le64 *target) { - unsigned int num_entry_qwords = ARRAY_SIZE(target->data); - struct arm_smmu_device *smmu = master->smmu; - struct arm_smmu_ste unused_update; + __le64 unused_update[NUM_ENTRY_QWORDS]; u8 used_qword_diff; used_qword_diff = - arm_smmu_entry_qword_diff(entry, target, &unused_update); + arm_smmu_entry_qword_diff(writer, entry, target, unused_update); if (hweight8(used_qword_diff) == 1) { /* * Only one qword needs its used bits to be changed. This is a @@ -1128,22 +1134,21 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, * writing it in the next step anyways. This can save a sync * when the only change is in that qword. */ - unused_update.data[critical_qword_index] = - entry->data[critical_qword_index]; - entry_set(smmu, sid, entry, &unused_update, 0, num_entry_qwords); - entry_set(smmu, sid, entry, target, critical_qword_index, 1); - entry_set(smmu, sid, entry, target, 0, num_entry_qwords); + unused_update[critical_qword_index] = + entry[critical_qword_index]; + entry_set(writer, entry, unused_update, 0, NUM_ENTRY_QWORDS); + entry_set(writer, entry, target, critical_qword_index, 1); + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS); } else if (used_qword_diff) { /* * At least two qwords need their inuse bits to be changed. This * requires a breaking update, zero the V bit, write all qwords * but 0, then set qword 0 */ - unused_update.data[0] = entry->data[0] & - cpu_to_le64(~STRTAB_STE_0_V); - entry_set(smmu, sid, entry, &unused_update, 0, 1); - entry_set(smmu, sid, entry, target, 1, num_entry_qwords - 1); - entry_set(smmu, sid, entry, target, 0, 1); + unused_update[0] = entry[0] & (~writer->ops->v_bit); + entry_set(writer, entry, unused_update, 0, 1); + entry_set(writer, entry, target, 1, NUM_ENTRY_QWORDS - 1); + entry_set(writer, entry, target, 0, 1); } else { /* * No inuse bit changed. Sanity check that all unused bits are 0 @@ -1151,18 +1156,7 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, * compute_qword_diff(). */ WARN_ON_ONCE( - entry_set(smmu, sid, entry, target, 0, num_entry_qwords)); - } - - /* It's likely that we'll want to use the new STE soon */ - if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) { - struct arm_smmu_cmdq_ent - prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG, - .prefetch = { - .sid = sid, - } }; - - arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd); + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS)); } } @@ -1435,17 +1429,57 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc) WRITE_ONCE(*dst, cpu_to_le64(val)); } -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, u32 sid) +struct arm_smmu_ste_writer { + struct arm_smmu_entry_writer writer; + u32 sid; +}; + +static void arm_smmu_ste_writer_sync_entry(struct arm_smmu_entry_writer *writer) { + struct arm_smmu_ste_writer *ste_writer = + container_of(writer, struct arm_smmu_ste_writer, writer); struct arm_smmu_cmdq_ent cmd = { .opcode = CMDQ_OP_CFGI_STE, .cfgi = { - .sid = sid, + .sid = ste_writer->sid, .leaf = true, }, }; - arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); + arm_smmu_cmdq_issue_cmd_with_sync(writer->master->smmu, &cmd); +} + +static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = { + .sync = arm_smmu_ste_writer_sync_entry, + .get_used = arm_smmu_get_ste_used, + .v_bit = cpu_to_le64(STRTAB_STE_0_V), +}; + +static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, + struct arm_smmu_ste *ste, + const struct arm_smmu_ste *target) +{ + struct arm_smmu_device *smmu = master->smmu; + struct arm_smmu_ste_writer ste_writer = { + .writer = { + .ops = &arm_smmu_ste_writer_ops, + .master = master, + }, + .sid = sid, + }; + + arm_smmu_write_entry(&ste_writer.writer, ste->data, target->data); + + /* It's likely that we'll want to use the new STE soon */ + if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) { + struct arm_smmu_cmdq_ent + prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG, + .prefetch = { + .sid = sid, + } }; + + arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd); + } } static void arm_smmu_make_abort_ste(struct arm_smmu_ste *target) From patchwork Wed Mar 27 18:07:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E9879C47DD9 for ; Wed, 27 Mar 2024 19:18:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=W1+mS0hORV6ztHfWM9Q9Md1mO9DknBb9vOhArP0CYAU=; b=izpSZd3ITabOMM 0BGgoH+XaGnP1UTA/xrSNOFt6s/ndSDp2pu5K1SNUagHvq+1dnvkTU2kAhnmktH4Xim5cZkP5I2gX tzx1iMAA/d36o6nfgSjPIcnPV+m6kDLDIlIr2yZZtcHdJ2JaPLsV3ns0rq0jzBMarI6Zz7aTT6tmp n+wyo5q3Q9Ej+srs/PIBCOuvU3vEd15mZNT6FbRZ/fdIdB/zPAOTqelhmvqGcEjXJ3gpZSjXHGc1M QewslFSZ6SB71mrHS8EC9Ap31AD32LE9NzuEmP/+/qZ/caVl/UZ5XaQJd0SIFe+fig0LyQEnIMuyB bXfoQ5RHQ3VC6zXLHphA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn1-0000000AkvV-2Dy4; Wed, 27 Mar 2024 19:18:23 +0000 Received: from mail-dm6nam10on20604.outbound.protection.outlook.com ([2a01:111:f400:7e88::604] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhT-0000000ATY9-2qcV for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GBoIqat1yo51Kcf1SS79QTHNBLVQsE7ZPDQsBVYnEe+GiHdsUjN4e6gEeXbU9mkbZQKAtKU7DsKBTv5WvlILiPVfapolPQinQ1VhJFbINm242vuTPWnePU2cXB+DtamK+8ewYBE+4K28MY35ZZc8F2XNe+mUTpH4qqSvhLCivnt4yWjHP9kehFO35s1aoVRsNAmGu5Gxjpm9LXYWs95+X4+I3BzPX40IMgJ0t7sbzlVje3nPSF7RnIEWXKwWAaBRHGzBmXr2usfcYRQN5AMZGJciMafaM+LXlsGUy4VT3or6Z2c3NbIhl/I/OheFQ9tMXn5Gblpy1cjhzHSfeuMfWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FosaEK32/1ZODeT9QLnadln0gjum6Vu9SMmO+5zATo0=; b=kCQ55rjrskujkhaXn6gz37D6Envk3cMHEmwfCDok0C2D2Vyqn6jgdLBcjwQSLdFK62GBO7UYbrKkLj+VaFRAXjiaccVahXtd4w1CGLTVEgQgmKCLCKgyiJrRfpLlBv0o4C55jALxZygVHJw0gkfTQjx5NQY6Oi5usLQ2t6zP51tKKxkXqQLj1jWLKXYmy2FaV9bo0OQU9ixNclHgAgzCi2B6WdrRj8KgX9y3QFs5PGQ1Zhz16uPUGrAvj/KT29YQ9IRM9zIg3lzjXV3bIGBFZ8uPJUeNhaGD9VnLxySVcB6akRv8SgmKMtpAcddsCEJC4FgLKwtIISS5Kr4Wi0RnVA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FosaEK32/1ZODeT9QLnadln0gjum6Vu9SMmO+5zATo0=; b=Wm55HyO01KwImMISxtFGjBG3NLiIfsKOetgtFoQm1QGDwe6NVypwghkUA5/0RbLiBzvSznOYz/imst3Cy3hV9YbUgQzfVSX3wiaj8qHpJTKGcawZzBsF+5ZQTXkgORCKXNlwnayZ3SoadoZH6o7MMV8KAH/p3a77uIV7uhpspZe3av/1LEKnLwZda5QpRmbVGmOJDRWElNsMb3vmTLU155dKxc77cZJfhGS7YY+hBtqXU/tLSaSVyMMLIGErwYFs58F8cIXapVqDX6qH5pw+Cu+sKfcbV2D4UAYs+Sq6oNNU00++54uTqckSrBD4CvskYt8XI1bZWFNHxVwfRb1+0g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:20 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:20 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 07/29] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry() Date: Wed, 27 Mar 2024 15:07:53 -0300 Message-ID: <7-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0312.namprd13.prod.outlook.com (2603:10b6:208:2c1::17) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 25e95adb-424d-41fc-bb0f-08dc4e88e102 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mQxOS+iOxdFZSh3fsU56FLF+dtHoopM+hSnlmAOBA4Liqa6dISSGOJ7bpKLowS5D5ceMFAdHWRquZU98e41mz5JKIWfwHp+sRJuBKw1ZFVYqYTGB3kLWM/+WQfDBk/sBgws8aTJDZKpIlg5PviWRhdPwCxaqZkiGt7fWoUJn6EkQm4/jAlGh/5++ot13AZJVrw2YLojNL6WCpgVhTfvPr8uYDFqbZGwQRWDhZlovfs7uGe02/fr20jT94ehHM4kCuc/TSySWtqfSFvFQ98y0Z7+ClC3FM2Pc9pVYE+fkQKlahZyOquJ26QcoIvVQJbHSnNT0rktiJ7ff/8HGfBfm5mdPcwzWVuV57mMCPz9p9Ml0THPn+5QI3jkWz6rEVd17AgsnR6HNU2hQkzX61dAzahiWF6kl3C8jX/gcKT098uPDEohJTCYdv1fzWMt9OpFsZz3404TdjWeiABEFBDIJyH9YDU7Iz1cmSGao+DKFw5LaKpYLxmxfao3CFiRxmPhNplk5/uCbQQdMuKkC0y2aBwx/t18nX0d+38rLek9ZRIc1QVqdoZyIFXg2N5XmEDlw+0G3ku9VcTV/v+3Z2CzIRA3Lpv7iGi73KFL5X7WwIeV8pndHgDBxqH9u0MrzrAJd3EdVZNVLeL5FsQKymxU+mK8z2bT7BlKCpDQokaMCFcE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IFsaRKzH9giRoU9YdEwBX/7/pRrxYL7gqBprHmWSSfFAMb9d6DGFIQ0us9cJOxJ2IZlXXDOEB501/dm24MNiavcqJZ0kCXmqiTGuHVLinHeKsfMJK2gryNbeACEIW1SXI6N2lshzFhzKCub/40d32l88aOzTpw9RAloYHOHh1bq+UNXRc/eajTbEQuS+PjNU9uVACYL0fo+Fbsn0eUr8LjgcvokcYAfalJrefM8KyMxHF0gvI33UoTSCkHWT6jH+eh+pPOZeTFSHYzgvCC/XvNzFb02KozbZ+lBe9dRLOOUeSNDAKPm4W0ADErm8ryJZL+9tMvlBcmsv/lkSt4T/1L6eGv+QNEUE21yESOzFblr3PIWgcBn59jvjssLASDZaVZZvJNWpHLVcdQSkKS2roN8g7X/72wQX+qk4t4CbBD9ngQKCxlYODxmV2iPC96Www5Ppx1EAzkaScXaDUZLN9jQwtEjaE0CGAdGxRMstGCYq2oyX6DnyG9+zQm4EsmAkhY0r3YTL61QaO543X8KPK2OBNQnU6QK9tuMs92SQVZImb6GT0/pL5o4XrPkXTwyv/W6HKFl3luf+9wbewPCmVMdNLpbLRS2Sgv7Jr73gqCNpWRchGDotlwKEu6vrtGCcW5Eee3MXGNz11TFO0dssjyxibTjGHr2c33dPq7LuIabh4whY0IrqOzO6tX7/elTGALEGCOaVbyaKqvW0GFjb+2/+zCbn9iDHv7tCRErWbqakaTOjuIB0ArnR+Hq6u1mAGOVFt+VD6fALF8pD5W88vSs4fTfhO0ugBOfnUPUi9UIAzyofvn3exQDm6rERHaXdtQqW96gzuoIjGoRF30FXOocuGmGqFKNC0iy2HRwyVngO2ELPjkV9rfkLY9CQQ8mqOHxarONGBaL/qN6b5ay4ROjR+qinA1JBpQJ2pjxle33YRB8qO9hc3ikDaCWcubc2RM7L/4CnQBTnSybEVbgZoy6nbTPqAwqxp0mOh0M+jtmnuWPCdk/1QKAK461b2eMxVP75wGVu2I0zvBJUCxPdBD0zpm6duaPvYko6rcRS2wpl1MHGo93zZt5Toqm03WTQwrLrxHvMWz6MQQJcHb7m+WKQB8NcGCHRvt9kVjsDvyTosfDw+HsMbDwMR/nh2JGMntHcAS4aV292f4Ki7kUHvAyLcyys9z/O4iFQqsV1wvlaHtxDVzW9JmSCgoIp9mBn/Td1ue2iCPciQ50HVpbnyU9Xj3B+t8DigQvMK4gFjUcVdOiuk6Ak1LGWfhQSZ17U0o1pDGaymHRyF6nJbk23JhBw0v3OJLaxHJEYaGSZHkYTG1A+rr3X19k9q1PeDwFdrGV96StqcNbPUXAtoZ+J7xDW6uuLOQTi+E5GGWLW/Wx0w7FTUuDI4DXNDcFADWi/EjzsDH7MrLql2v/lYeEclDevQLiOsuEarH/ULbwnaNkGsmdN3c3mXs79eupWsEWQRr/493lDii6r1h5QryWgXEcETrnsvebArhYuO8o25vajgpMz3H2BQIlHqsWgtgoiA9dd1HJPbiizKxGaTLozuJgN++Y7G1z+aYNQTX05yKcAJnvSkRix01WJwwFPx67z X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 25e95adb-424d-41fc-bb0f-08dc4e88e102 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:17.8492 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: OLEasHP6uke5v+0HnMxZrX5Smd1TTOIJI7Rwf+6KkLvxjAvI+vXEMuRbCwg4yhoi X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110835_884816_7123B590 X-CRM114-Status: GOOD ( 19.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org CD table entries and STE's have the same essential programming sequence, just with different types. Have arm_smmu_write_ctx_desc() generate a target CD and call arm_smmu_write_entry() to do the programming. Due to the way the target CD is generated by modifying the existing CD this alone is not enough for the CD callers to be freed of the ordering requirements. The following patches will make the rest of the CD flow mirror the STE flow with precise CD contents generated in all cases. Signed-off-by: Michael Shavit Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Moritz Fischer Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 94 ++++++++++++++++----- 1 file changed, 74 insertions(+), 20 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 19fa511cec2c05..453437ca4bfc2b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -61,6 +61,7 @@ struct arm_smmu_entry_writer_ops { #define NUM_ENTRY_QWORDS 8 static_assert(sizeof(struct arm_smmu_ste) == NUM_ENTRY_QWORDS * sizeof(u64)); +static_assert(sizeof(struct arm_smmu_cd) == NUM_ENTRY_QWORDS * sizeof(u64)); static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = { [EVTQ_MSI_INDEX] = { @@ -1236,6 +1237,67 @@ static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, return &l1_desc->l2ptr[idx]; } +struct arm_smmu_cd_writer { + struct arm_smmu_entry_writer writer; + unsigned int ssid; +}; + +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits) +{ + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V); + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V))) + return; + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd)); + + /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */ + if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) { + used_bits[0] &= ~cpu_to_le64( + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 | + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 | + CTXDESC_CD_0_TCR_SH0); + used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK); + } +} + +static void arm_smmu_cd_writer_sync_entry(struct arm_smmu_entry_writer *writer) +{ + struct arm_smmu_cd_writer *cd_writer = + container_of(writer, struct arm_smmu_cd_writer, writer); + + arm_smmu_sync_cd(writer->master, cd_writer->ssid, true); +} + +static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = { + .sync = arm_smmu_cd_writer_sync_entry, + .get_used = arm_smmu_get_cd_used, + .v_bit = cpu_to_le64(CTXDESC_CD_0_V), +}; + +static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, + struct arm_smmu_cd *cdptr, + const struct arm_smmu_cd *target) +{ + struct arm_smmu_cd_writer cd_writer = { + .writer = { + .ops = &arm_smmu_cd_writer_ops, + .master = master, + }, + .ssid = ssid, + }; + + arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data); +} + +static void arm_smmu_clean_cd_entry(struct arm_smmu_cd *target) +{ + struct arm_smmu_cd used = {}; + int i; + + arm_smmu_get_cd_used(target->data, used.data); + for (i = 0; i != ARRAY_SIZE(target->data); i++) + target->data[i] &= used.data[i]; +} + int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, struct arm_smmu_ctx_desc *cd) { @@ -1252,17 +1314,20 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, */ u64 val; bool cd_live; - struct arm_smmu_cd *cdptr; + struct arm_smmu_cd target; + struct arm_smmu_cd *cdptr = ⌖ + struct arm_smmu_cd *cd_table_entry; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; struct arm_smmu_device *smmu = master->smmu; if (WARN_ON(ssid >= (1 << cd_table->s1cdmax))) return -E2BIG; - cdptr = arm_smmu_get_cd_ptr(master, ssid); - if (!cdptr) + cd_table_entry = arm_smmu_get_cd_ptr(master, ssid); + if (!cd_table_entry) return -ENOMEM; + target = *cd_table_entry; val = le64_to_cpu(cdptr->data[0]); cd_live = !!(val & CTXDESC_CD_0_V); @@ -1284,13 +1349,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, cdptr->data[2] = 0; cdptr->data[3] = cpu_to_le64(cd->mair); - /* - * STE may be live, and the SMMU might read dwords of this CD in any - * order. Ensure that it observes valid values before reading - * V=1. - */ - arm_smmu_sync_cd(master, ssid, true); - val = cd->tcr | #ifdef __BIG_ENDIAN CTXDESC_CD_0_ENDI | @@ -1304,18 +1362,14 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, if (cd_table->stall_enabled) val |= CTXDESC_CD_0_S; } - + cdptr->data[0] = cpu_to_le64(val); /* - * The SMMU accesses 64-bit values atomically. See IHI0070Ca 3.21.3 - * "Configuration structures and configuration invalidation completion" - * - * The size of single-copy atomic reads made by the SMMU is - * IMPLEMENTATION DEFINED but must be at least 64 bits. Any single - * field within an aligned 64-bit span of a structure can be altered - * without first making the structure invalid. + * Since the above is updating the CD entry based on the current value + * without zeroing unused bits it needs fixing before being passed to + * the programming logic. */ - WRITE_ONCE(cdptr->data[0], cpu_to_le64(val)); - arm_smmu_sync_cd(master, ssid, true); + arm_smmu_clean_cd_entry(&target); + arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target); return 0; } From patchwork Wed Mar 27 18:07:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 18614C54E67 for ; Wed, 27 Mar 2024 18:13:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wxgaf6cZG8GCR1SCFmZUy7ZI+yx0UZgZuYi8S3yutHs=; b=IjhRNmUDWVR4oB j5ybJ6dFiTysgv/FmJ5wxLWxRPGgLk0WVawInnfzJCM8u14xfEMw3+h+i0qANY+YZex99/JXa4/EW FZdLGuefY4w8KMGmhXctCDzxm9gkGgCfUNKbGUe7GvcfeTulFx0TpwqkM1ZPaq9bAfHwuzxqOEbl0 zMVkqznGifjVSRAyci8o0/xqNoFCQqVzn1QkjxA4kR6tX3+rGhLKEBwjOzE84h4yaRt/ZE4tQmU/l p/1adx+aQ7jj4cAuBChh0OW0/mHbWAhXPY7YSN6cVuaizesMDuHLWf7dygoUFjV/CSW0Sovj5rYRE rr+daHeug5Vl5qwM+BFQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXll-0000000AVe1-38MA; Wed, 27 Mar 2024 18:13:01 +0000 Received: from mail-dm6nam10on20611.outbound.protection.outlook.com ([2a01:111:f400:7e88::611] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhY-0000000ATdz-1M4k for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LxWwwAWwv+bjaIYHJubNv/E0zb8RTwlNeYmfdZ04zXieIKaGTV+96obnPcM1Yjwm5Ot0d+ri+mWdwS+IY4UuXn5aEyetx4DBK1z60U26KFd/sw2TSmsMZkvwP68qcW5+f7xR7ndCkkQkdrfdwlqFY1Mzwwpvv05BgDr4mkQEBQubAm3DpxNdE/IJ7f8d6pKnDzmxJ+3/eeA346w+037R3xA9GAV7TloMIv6D12w5zRW/EiH8f/cap7ofvtpUwMtgwyrkvV8XcA6u64lbKE4tfb2XzcMmZeSqxcKs/lTfwbK3AitCG24JZ8bJNobU3+Mc3yWqkQIsTR92pbDqko91Fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Xl1E50wiFJDauM5NmT6LROZcbPJ7yB/GkFKe6n06lg0=; b=lQ+4l13GSROk0JYkol1zkpXsfOoPr7U+9o62CnsAZ0m141IIWkKbLBu1dVkveeXMXOPWRevsbjP4kbplLqJXd1m19L6ngskZCP1eScCjVVnj9L19ol3IVxDPI7VfXG9wtwcRHK0GEftpSOR97j9W2HWmOlCYkAmyPeur4QVclRjguKDaXy9IDeGIoc2LgALJLWF8Nb88a+7rIep8WRdkY4t8WsUY1xeC7UO0npyv9Pm1ctu5AzBGhz5UHFDQ/81XJM+pHBlPdgN6Xa3c/rFXIvc6On4olttjWKdMF5/gz8FnnDiPII0i0onBQDMTE2wSK4SdoYd/IemSEuQTXB34tQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Xl1E50wiFJDauM5NmT6LROZcbPJ7yB/GkFKe6n06lg0=; b=fLN2ebqprovFJN5FjRKP2UoPtTPIQPokS6QWZ1AR5Y3KrkYij94+wanMTyEJjncs+dq2uIxLFTadd2SW0heXsCgvW+xhgQAdmiPKg+oWQmamvfFKGozdkqFjqtDUIf8atrV6Z2yoLmBrZSDQc1OfHHCDPkD+2E41pebqLoHiyhZ6WdlNYqAHFjXcnYzhNR84vM6Gt+ORHImDWMV9TSbhBkVlv8czisYIhS5UuoKn+rarw+SM80yUDlyhpe+qefBvJj4mjwm2Cy+gghD7QzXe1myfKO3J6SqgzUUIzacUYt3MQT47V9+ptg5oPQSqKxPGMEuyd9121alo5pLA2m39aA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by IA1PR12MB6044.namprd12.prod.outlook.com (2603:10b6:208:3d4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.33; Wed, 27 Mar 2024 18:08:32 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:32 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 08/29] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function Date: Wed, 27 Mar 2024 15:07:54 -0300 Message-ID: <8-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0039.namprd05.prod.outlook.com (2603:10b6:208:236::8) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|IA1PR12MB6044:EE_ X-MS-Office365-Filtering-Correlation-Id: 8f4b9850-e6ad-48b0-6f85-08dc4e88e1fa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OeK19my1H9mEDC8MC2Ti5N0qju+JIZ+1ggP0BWEYQ6/HGb6dUnTOkWuSmOg6cWHejMfx1JnrTBgauF2dKPT9yeICgkzR+YQZ+N8r1p8rYBqWCXDUYBYL8WWi3bMPxLHRwoP33sqjIEyKNzrcwUQ4cKwQ8JKmJ5zizrQ7T3QxY0VLyxSzyuy1DZlyi8w01f5svjqJAHTHdhq8de4NjcYjta699u0GSTniMgrndCW9wPj91SQcLfPYBazrihRDoI5FVbCdw7rBrQ/QnJCQqPuMas/YTip/PkXYPu3fL5Jj5mOZRmjvAkaeQzd+xQSMUzMIWA69iYvT5FUioC5ihvpgXZM3EtpXfp0AQ958i8S75Q8LOT8xjYsiMWEvkAteXR1oAfmOdd+MqPodJvde6KLhQoNV3HNBUPE+HpN3lX1rDTd1R/BgwtAPdhwYpvbNANYWDHDXFJanOdYljPddtRrqS3jn2DXpeE4Ofd5cjPtDQsejxRG+shbyVxhBMo5cwO9wtC0FdmeANuZLX3aSdov+y9ho/gpt/bDJEj0J7fgq4AfEjlTC54TUg5FGMrbow3wlkeZq0YSc143XhLBV/mgC0eWvT25MJ2uq1aOh/5ZBlikaMkjFcPPD5wMjS5hMSKgVJ5d9tL/c6aRurQOqyihGlbxxL2KQPLSs4vnrE/MqQl8= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7416005)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: vFkxJWa6xWoddUnYgx3lBf109W38V7t0hKho/kRyKjXqLr912K75Phsjd1380SEcnQ95k+Rk47mU1Oowa+u7o41/2FGU4d6y0fExCx5Az6VPVaAlnXkT99AszrgSJGaOIcN/ugxmFW8Vyui+UCJ3k/WMiWRtgG7e7QaiOC+9PZmV/OB6Bw/pR5OsFKa33bzi1dQJO/Mi8ts7ckcIN9sXf+fyCjkr08Z2yInVSz7Eg72kSfbgw3ASGh7rIZjVCEoVJgi7D3Pv1s9Oba2LbGw9z1NSoC+9pKRLc5ngqVA6DpYHOhy3UsJLyPra9+E1gUZtFpbe0Zhl8Enc12NpxpadnAADWvgZ5/Ix/vJxw8pbhEcwLlL/lwIcXm6C9ycQd7xF115+t/W4bS78L0bOmFyQ5C6aJ6rdSKNQ8cwGjz2o1o+BOMrvpT95i9l2K2XKUY3shusvhA7GHFBPLgXbXPjocSzP2uaoOPmzQC/MJYu7rD55yFTufb43I1LKTnD4xz6U0VWpqaCABQ4Ot3Uf/9ienkrfR3zCjg+b5yJ6J8Gjqlq1HJ4TT+MpA/WUykMm0kHq9T6lyOLDnf11fxHGTOgC5D9v5UrVx5goDgu3rCQZB20xfhPYyFFookaQCynPiJEb5wqEfFE5Ch/9L9urKx2aqIp/vp+nE4lwVNOrOtmJKpsx4wR14mTDo+BaFWpHQGh51mx2PZYvaa/N9gPFtyeaz54Eu9d3e68aVbdUE6e51KRv3zaY+FfAWLhRR7+8OEcg0e0aqX2gQpt0jWeXfHB2X2LYJFcp0zyD+Zk0wjooiDkPdt9G87GjkAncepcW1khHOjjDRZd2PfDKLU4Av2eLLtZR1Gbq6Bl8uOp85evm/UVEvFrrAuXIIywcFOBzvQqaFj82rAQFK3u81K0RTuOFUr6/kb010YjaLx9gj/RhSsDIpEM2y86JYD9GWh7iuQJtIEF1XYx5sHkCd9PJyaAi3ur3w94yJEeA12DWWu7WuUf6Mp6h3pRoX7wJx1batSzOfAAODZ8BGWcGB8Bia3L6x40h05zftQ7C5cXkGPHKJ9I35vF0InXif21wE7U3SULzYL82s0XniEsvMBiWIjySaS1VoodTZxBXJgYbnG5N6s/R6JfVPZiqxOzRuJJGObI5aS0qbC6RAtf7Bjqt3rliNrhAI/rnfNtxFxGASQlwOPLZuXlbuX1Qp0NdwIePczq25SDolWVvRVVN9Vc8ydmQ1t+ljynSJL5YjU0mpQswNUHru+e+FI7liQtWGHh13sb3AIBrj8eQkglS894oAnPztEYRSoi+AcoIKX12d638gqXSZTx5Fu4GxESBD9LtYgRepZsmp66jX+Lk3elc9FiYYwKHadcGoVQ/6TrIutVyNQGvX3E2+hCPXbLPrDkzMUZBCVlRW0npdI8DMbXF2riS6YXOax2CjaYOzXTNVzKhPEQt5VugUuFN1MWzO5m4a+fR9PkuGCaqXqj5qRufAR4pOzw8hRrL8QEqZXQ48MeMyGJTV8fwNQHcjYEUb97CqGDMUhiVM7grPfSOxep42V/L1yMUiQ0J5G69lK46thtO2yjIO8xzuWhjLvtMSK35AWSf X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8f4b9850-e6ad-48b0-6f85-08dc4e88e1fa X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:19.5127 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ypgSyV4TCJcpJ5FJ1+gqsCvYIHzh1WPr4t1nPfxTTLdykrdMc0ZQO5/DDB5TQCKg X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6044 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110840_517975_08DB5077 X-CRM114-Status: GOOD ( 19.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce arm_smmu_make_s1_cd() to build the CD from the paging S1 domain, and reorganize all the places programming S1 domain CD table entries to call it. Split arm_smmu_update_s1_domain_cd_entry() from arm_smmu_update_ctx_desc_devices() so that the S1 path has its own call chain separate from the unrelated SVA path. arm_smmu_update_s1_domain_cd_entry() only works on S1 domains attached to RIDs and refreshes all their CDs. Remove the forced clear of the CD during S1 domain attach, arm_smmu_write_cd_entry() will do this automatically if necessary. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 25 +++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 60 +++++++++++++------ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 9 +++ 3 files changed, 76 insertions(+), 18 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 41b44baef15e80..d159f60480935e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -53,6 +53,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); } +static void +arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) +{ + struct arm_smmu_master *master; + struct arm_smmu_cd target_cd; + unsigned long flags; + + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_for_each_entry(master, &smmu_domain->devices, domain_head) { + struct arm_smmu_cd *cdptr; + + /* S1 domains only support RID attachment right now */ + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); + if (WARN_ON(!cdptr)) + continue; + + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, + &target_cd); + } + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); +} + /* * Check if the CPU ASID is available on the SMMU side. If a private context * descriptor is using it, try to replace it. @@ -96,7 +119,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) * be some overlap between use of both ASIDs, until we invalidate the * TLB. */ - arm_smmu_update_ctx_desc_devices(smmu_domain, IOMMU_NO_PASID, cd); + arm_smmu_update_s1_domain_cd_entry(smmu_domain); /* Invalidate TLB entries previously associated with that context */ arm_smmu_tlb_inv_asid(smmu, asid); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 453437ca4bfc2b..fd1d4d774a7cf2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1209,8 +1209,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst, WRITE_ONCE(*dst, cpu_to_le64(val)); } -static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, - u32 ssid) +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, + u32 ssid) { __le64 *l1ptr; unsigned int idx; @@ -1273,9 +1273,9 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = { .v_bit = cpu_to_le64(CTXDESC_CD_0_V), }; -static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, - struct arm_smmu_cd *cdptr, - const struct arm_smmu_cd *target) +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, + struct arm_smmu_cd *cdptr, + const struct arm_smmu_cd *target) { struct arm_smmu_cd_writer cd_writer = { .writer = { @@ -1288,6 +1288,32 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data); } +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, + struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain) +{ + struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; + + memset(target, 0, sizeof(*target)); + + target->data[0] = cpu_to_le64( + cd->tcr | +#ifdef __BIG_ENDIAN + CTXDESC_CD_0_ENDI | +#endif + CTXDESC_CD_0_V | + CTXDESC_CD_0_AA64 | + (master->stall_enabled ? CTXDESC_CD_0_S : 0) | + CTXDESC_CD_0_R | + CTXDESC_CD_0_A | + CTXDESC_CD_0_ASET | + FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) + ); + + target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); + target->data[3] = cpu_to_le64(cd->mair); +} + static void arm_smmu_clean_cd_entry(struct arm_smmu_cd *target) { struct arm_smmu_cd used = {}; @@ -2646,29 +2672,29 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); switch (smmu_domain->stage) { - case ARM_SMMU_DOMAIN_S1: + case ARM_SMMU_DOMAIN_S1: { + struct arm_smmu_cd target_cd; + struct arm_smmu_cd *cdptr; + if (!master->cd_table.cdtab) { ret = arm_smmu_alloc_cd_tables(master); if (ret) goto out_list_del; - } else { - /* - * arm_smmu_write_ctx_desc() relies on the entry being - * invalid to work, clear any existing entry. - */ - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, - NULL); - if (ret) - goto out_list_del; } - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, &smmu_domain->cd); - if (ret) + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); + if (!cdptr) { + ret = -ENOMEM; goto out_list_del; + } + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, + &target_cd); arm_smmu_make_cdtable_ste(&target, master); arm_smmu_install_ste_for_dev(master, &target); break; + } case ARM_SMMU_DOMAIN_S2: arm_smmu_make_s2_domain_ste(&target, master, smmu_domain); arm_smmu_install_ste_for_dev(master, &target); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 7078ed569fd4d3..919f9f717bd3b2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -749,6 +749,15 @@ extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; extern struct arm_smmu_ctx_desc quiet_cd; +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, + u32 ssid); +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, + struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain); +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, + struct arm_smmu_cd *cdptr, + const struct arm_smmu_cd *target); + int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid, struct arm_smmu_ctx_desc *cd); void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); From patchwork Wed Mar 27 18:07:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607244 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A8C65C54E67 for ; Wed, 27 Mar 2024 18:08:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PK5vIChwjptqsvdciaIGqy20uBMb2q1hKTAHT1S0gXs=; b=uP/F6DgMNVELTf L1q0cLkUnovW8v0b6SgbqVrO8mn7o/7/7RvkZEtQL5FE8MCjtCdWB405Y414dwQOA6lbW88QFava1 uTZ6nS6+EdOuicGGXM8ze/xLwJnaOnYEAPFuzFO1gw9XYv4+pXNeUOXVTkhSmh3U8ND1HpworM/77 o/UQwWkFHmnNuLk93OMkUD355qYhOBMxioEdGAREdc6TSi+WrjgBNNr45tVyrFdT3zQuLxijbhTwf q/076en7d1Lh5LISL924EQn1WC1K0VWDzIXS/2yG+qKp7uX4uyTyz7jWOTYzh/WSfLso2BPQnOZPZ rg3srwjdfz0/Q3VTI4Mw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhZ-0000000ATiH-3tOU; Wed, 27 Mar 2024 18:08:41 +0000 Received: from mail-dm6nam10on20604.outbound.protection.outlook.com ([2a01:111:f400:7e88::604] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhO-0000000ATY9-3G0w for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:32 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ScLtmvoskazURKW2S2d8JCMwxjS5yBzmkICCGVU4KiXbBHJcCD0sTZDNiRxj2mFcq3eD5IzkeYuu1HfrixRhDFyc/1gM7QH+GN68R1osKzybfmGN+m3EeT1eDskx2Oo2r+5Sc0FjaBrGFb3+7E27hZ9Yn8cRy1lnYa3uaU/pfJBvAspAknq21/wacfAZblXYcrBKuqMmcjDoEN++HwaB8XQK2E5YsutbITKkloYLsg+hjiYcU5tA9fAbvu4ZzbTNBig7mU0TT74xlR/zWbeRHld/Q7xqnlevRu7b/2q9OUFDzGX9jFoWhRBPmYDCUbJyMw3nRDmusvCHsVqs2Har7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ha5ibzzndNSjw2W4R5NZXWHSoUPZ4A4iPkuxRpGnbE8=; b=AJ7uYGsmHs5yvrj7HUBnwYqyr2g6ZgxjSC7Ypr2GNZ6d014HxJHEfQ5g8PNLF6iMWF/skG8wnCOkQaX/Opa5+AlNYX62nLOmRgfjgB4TTs//5Brj19VsrD92GCBjcqt9ONOKCuPNsyF/uq6BeVUrz95jgabRt7XSYfWmobCCXseEv+p0N3xpQ1IAwYpkxlF+a2sfiois0uXfqiY5nr/w7ke4oKtl+P37bE7qd3hoqjf/7LSFZtcCuPtg8Xm58lWmi3VCWpj4A7gbPipwoCvAZ1qaJN5ryjO1jDk6v0GMkLUU5tYi3OkRHYh5mIUEGQjClqoi0atyP+w0BjZfygcqZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ha5ibzzndNSjw2W4R5NZXWHSoUPZ4A4iPkuxRpGnbE8=; b=K62c3gjlj30narVNHOp/MplPIiXILx7XwfRa4B2X4BR2FY07M28ZBBvquBjDHU6Jct90WTGEFkwAEMl8RGxhy08t9qWVOv3FeO1XnIw/g/HF8AVX5zW8Qty/tYI4M787M28Hhx0Aw/Dsj0mrjJyFb8LYv6Nx6YRy784XEBNNpsvzY4glOQRKulcljtJH3OgQ25huxPnl6l4uLWBAt6sb/WYsE5n2vID63SXhlXVqhsJwPBSGQUNZC7qQ3+5PUTfp/La1dsXVdWA8MTis3FI3aIhHb8sIWMqOt2kvz1ZusNInLsM2bLpvJHEM/z4K8hmmMJ+Axy4UzIIpd2Gj6W7eVg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:19 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:19 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 09/29] iommu/arm-smmu-v3: Consolidate clearing a CD table entry Date: Wed, 27 Mar 2024 15:07:55 -0300 Message-ID: <9-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR20CA0010.namprd20.prod.outlook.com (2603:10b6:208:e8::23) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 28991e9a-530d-4184-57c6-08dc4e88e0db X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: nEAGw/6ZMvsjmrqXPwLM6f9j8TCKXb4QsrjDkyCvG4mPf2ptSk6nfTub70zerccrwDftQhCHIl/PmZ8ZkQOTadMRtZJBEs//Ub3P95MgMqqpEMJUNnCem43NRFvq2kHf258kfl1B+NZap01RAAhwxQToUzCYbJ4Jawu7F193TKUZp/e/VWx10zvaX8R1AGmmicpC5hRQMatDai1b/ip2RnSBB9ShUI+FvBrUzr8JBr4KHvNJcvX+lJMezGNWj7yzhTAlKMMtUAq6QAw6Tw2fJ7Fs/P7lCfkFVy6BgiOxCwmONOfUJiFbFY7D4nt6m7QudVNgUlcrjHgfR/L6aJe3SgrtclkO/S4rVkPgc7PSC2qQn+elkFZt6HH94yj0gC8Mn3K9Zn8cpCjajCup4zRQjliUmsa2Sc71AXc/8TanJ0PTPNOKh/HBmCkXipaHZsIb5diVPkw+T3NQ+oPoAxd8Li+mKI44/GumhC0mc5vKI9Aliv19kKNI0tqx65Fu6q7S4EPLv+vGd1/sSghdddTkaDIhuhJSIxj3/yJx8hr/R0Nk4Yxz858Fn63en6rYwNpfnsKcKJPr6fhfYYyHI6MfuQ4DFoRpMDBHsFm4vAXZwXqUvkK0qi11MyfJgU/K3P7OKszPCBQ3SGaqgitT3zRLsLNk2VfxJl/J7MIXWTFSaHw= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 2ZFpQuy3utEgdLVcEcQvNbu2Hw6OKcLlZjHFx3AeLytcu/y0qv9qjk6IotvdM+JX0ofzeXXcJknuohQVIwnuFNQyIPrPJ1WytnqN30EC1AqxVuC4G1LxyBwDpER9NSm+O/i4c4bLsfYahfwOHHIW5YRfPURqHfQabyvlTZKjY2lKV0Xc7D8N2GQlgpp8XILNhgP2FnWDAzmxngdE4Hj+4+MJUBu4s9/Hpl+zHznqjijBQkpKFGdDWt1/NtXl+5K3g4Lg+gWNhHUD6jew+AFgJsUcRlqEWrK4ZbqwA9JPXbqDJGA/L6N21CB+xcrifHIleYHLhFiT/Q4etNKDgxEsQ8JnwquBAbEKp3HVhoO7jVaG4CA57+px5GPssJTdrw5Ii2yt+PzsGi1C+WknDUJavMdnrUdTSFGaN2eMd8XM3k7OJ+hsyUKYOrt21v/lugtozdUpWQ1tM5H5dS9uS5a7hpB0bKKbwOB/ZOfxK/gVeyMpLpMz7tx7OfmJjrobKYulBHswFhdjTrff9nOFemqdwC31JLwwN8G6ffVl3uaoNLYmOhLWm2ETnolMuqCjEcIRoBpWC5qHfWfuR05SWNTmNzcILC5SZyDXBf3rL5zMIoTu0nmbdeEjCcGCLkirD/9aug7/83abEGmLAJUc3Br9diLPPfRfzQQB0u3cDJR5SzeMXw8Q+mfsxBgNTlE2ozzMezf4Mx6PyoSG+UupAD5AzkufnSoX2e85oW+ecRqQv2BYAZ76qTzSPXiZ+N67UPaF7xzTyBSFsN9gQdEW4/rCWPRkPxHZTmMB6HnEn+Jt2uSnPA1C0wgKtJfsRs4N23fqX9jIN6O4pktFEIXT2Tv/KYuOUJ25iHbGOr9Vt57Tc6NLeay31387MY9E2X3bSjfJQixVCgl4q1R98Bto/LLs/NQsOLR3CdrM4R+FIWxEyhCDHgSasBJwqdG/zyed/nYVFoHvaVOxBOPwGc1m+ROr5UVBGGcAbnTltyNCbKa9JpGpB8QoDCDYaf70PG1l3JLsozSJlaW+U9WCHAoOaHrfLpw01Vw0RjdoZEsGCq3A5enmTIYjmDXjGWOEAnFM0zxnNxWLVb8NP0+S/voQgY3gLIIaaEfN4XcmKf7diw637up/bG08nnl21KdRP1vNe3qotZRuKKwdfynkQRLj6HocaWLutHOdU05/jkVVf/pRqAxpRO2uUrdPwEd4Nq1eSFfz7R3cfc9sLvWttV0Km+MhBZr45LH+gx0H6WXbSQKm7flHullMTZN81m26PBTdiS5WU4n4oxmQFhoPqMxsqvc5F8yKS9/+EY08JKWBELpNWRe7FpHOuPIEJB4x41KDj6n35UwXKusHLc0sy310pDJlqdLnD909M5PrXWOB92ocj2UX2SBP7MvvaJGZDLAsweOtXn8VjpyaKrUTKvhAILhck/I4zur6iL27IFbUIOY+r6kiQw0V2u2V70bv2jSm1T3UAqoNLbeZC4Doj3QbI6FTjv0L4B2cUO+zSrGNWc7CFVTcTDUZLKNWfmW6rvf9PBq9BZfkUytxOiENpN1R1Vcmj3WDsXNdKcHin/6/K3YIko0Sz1ExwOPFj0I5a4krYAbY X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 28991e9a-530d-4184-57c6-08dc4e88e0db X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:17.5958 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: g4GosL3pEHEhbo1vJ16ykvNDUmdIB6F9RzwPnQ7deA5009KvJpwwhMuKX5uThsBr X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110830_845323_CF26B70A X-CRM114-Status: GOOD ( 14.01 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence. If we are clearing an entry and for some reason it is not already allocated in the CD table then something has gone wrong. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Reviewed-by: Nicolin Chen Reviewed-by: Moritz Fischer Reviewed-by: Mostafa Saleh Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 ++++++++++++++----- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 3 files changed, 17 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index d159f60480935e..7cf286f7a009fb 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -569,7 +569,7 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, mutex_lock(&sva_lock); - arm_smmu_write_ctx_desc(master, id, NULL); + arm_smmu_clear_cd(master, id); list_for_each_entry(t, &master->bonds, list) { if (t->mm == mm) { diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index fd1d4d774a7cf2..54571f2a4acd5b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1314,6 +1314,19 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, target->data[3] = cpu_to_le64(cd->mair); } +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid) +{ + struct arm_smmu_cd target = {}; + struct arm_smmu_cd *cdptr; + + if (!master->cd_table.cdtab) + return; + cdptr = arm_smmu_get_cd_ptr(master, ssid); + if (WARN_ON(!cdptr)) + return; + arm_smmu_write_cd_entry(master, ssid, cdptr, &target); +} + static void arm_smmu_clean_cd_entry(struct arm_smmu_cd *target) { struct arm_smmu_cd used = {}; @@ -2698,9 +2711,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) case ARM_SMMU_DOMAIN_S2: arm_smmu_make_s2_domain_ste(&target, master, smmu_domain); arm_smmu_install_ste_for_dev(master, &target); - if (master->cd_table.cdtab) - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, - NULL); + arm_smmu_clear_cd(master, IOMMU_NO_PASID); break; } @@ -2748,8 +2759,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev, * arm_smmu_domain->devices to avoid races updating the same context * descriptor from arm_smmu_share_asid(). */ - if (master->cd_table.cdtab) - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL); + arm_smmu_clear_cd(master, IOMMU_NO_PASID); return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 919f9f717bd3b2..d32da11058aab6 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -749,6 +749,7 @@ extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; extern struct arm_smmu_ctx_desc quiet_cd; +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid); struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid); void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, From patchwork Wed Mar 27 18:07:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4A59CC47DD9 for ; Wed, 27 Mar 2024 18:08:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=F5lWRiascYoCNG+ajVXK4iP4Nroyp8YJIPG3NvtMLRM=; b=Q9rjL/N1GZL8l9 ll6ucYmN1ODHx27XNK4r84kUatVP+vrzMi/iC+o5UWDohyrYAvh3QmBChzqkByLlIWSFQ7n5DNIN2 06zO4OAoS7dz+4nKMH3Q5Qg8x2Q1DIbSHJJN9vWkzj0Sdztu1SeqlZjhw3wHe10MZ/S5QxnazePry GfjAbAR9gcqJSPooDXmAbVsG/LPCHLQhJT8DU38LgKWy1H3ivjaq/cX3RnniR5mQI9uYrBJn3lFjl MyaGhN0o22OrWdAeWrhivjZz39xTDiXN3cWcq1rDEisbTIcZIIlZHQA2pwCv6bWfq9n7igFlfDftp YOSfHlxsoAbOhCMOvq2Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhP-0000000ATbP-2kBA; Wed, 27 Mar 2024 18:08:31 +0000 Received: from mail-dm6nam10on20604.outbound.protection.outlook.com ([2a01:111:f400:7e88::604] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhM-0000000ATY9-2Qw2 for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:30 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MR9RnnUUbBwhP2aCyt88wUCoK188q7NfCygnXpgNVmH/LcyMwHTtm6b0K9JuhbI5qwAFmOT9ATMS+6Exax4dyex01fjs7hc4E/f9AkJsli5kxDfTyUcOXiH3xVP3nZ0RMbDu1x0h5VFKM4iyGKPhdtrj/p2wJYLGgRhvcLBgt3rBKsjB3z5cvJijtl1juBN2lzS21U2Q2ATNZCVCR1n1xH3iUvK74QrSncmdC+PZ3dnjlgXoRds/Ft5BSDQ4+66UWp+XD+dqYg8JwjoA996XvMBQsQFh2R7XHKEdD8Utt9Rm43ubT1aIXakx2vW8d/WzPxXDpV1SE8hv1VXJ8/JvTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2yfv0ZAW0JRpnvf+9YtpH+CzkLMQq2+WNeiPASvI/ZE=; b=WERMPZ9VKEMLVNS6vGS4l5Gb1toyMayB4a+O66m0Iz1G/2dMOh8P+hkpJijpQQzRcKCL2U9EIssTtL3DL9t8nx4Rg/nE7fwOeCIIIcSXipWzIPyEDEtz4qXShkiMAVFAV+BhnuYhJtWijcEMDmCm0FqlNUTlPMw6n9KjV03BW/ZMlqwT+EYc1Kq8eOoZrVbuI+pRebnd4uKEHsAnQSEZpUr1t6P9iclQCam1g6Je50kLlcPYFfyPJRRUi3v9EeAiIKWhxDzGrFGLUHaWrA/fSYUUEV45o+q3xjQ19HSZfFnuhuVK44lcmG/uzmBYMZB+jSMXQ8T0A9YKBjCvi4/scA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2yfv0ZAW0JRpnvf+9YtpH+CzkLMQq2+WNeiPASvI/ZE=; b=m5pVj1mXYmCv/kJ90rxwD7LGW9z6phpFnQxLJLpyEJ/+ddmrbeQzoI5sxk0TH2HEqiNMuJTJssciPKVaV8R5GYNuEpiyqi4UndXsD2UecCkC+GywfHf5MyBQQWkU1RWgwwZkDmxfsSJzFI7dUaXUGo88XXqMr9WVlZ5BSQF+dzmGUCKHLR2uh6ZeVRgRMkMMeCZOxJyNiYGWmJOIv/Rv2qPKG98Z0RDGCtNXu5UVXKpt3l2bP2w0NGkXZBiW1ze0yRUMuKkpBMKjueGLbS0fKuvycBfrP5lLPNsrznXtMHMBJaMVoe2E53/D3as4e53e6d51m3/0LW1SvZSVX8Mhpw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:18 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:17 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 10/29] iommu/arm-smmu-v3: Make arm_smmu_alloc_cd_ptr() Date: Wed, 27 Mar 2024 15:07:56 -0300 Message-ID: <10-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR20CA0002.namprd20.prod.outlook.com (2603:10b6:208:e8::15) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: b592a4db-53ad-4fcf-7355-08dc4e88e0a2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KY+v5blNwPmKdbLQW8EkWSo276nIAdx4WXUXUTdwcr2anhUinpjgduLx5UzRd/+Y9WB89JXUc08fHP6Qcq9v7toPwxRnoV3SB+jwEAy0ZXPsZpcDVK4sau9jQDtvKPp6PDEN5sFdaBwQuS/48iYEBmwST7l4lkqhNduS67x3S1JHPlZJLKP1vZUoFLM/tVvA49Yb69SYAwOE9iWjdIEJoRRxrDU6FznWJ/4ky77ZcKxSDioqDYfcUYzDAXtIoW6tQKg3xF+iB/9WiCqBxQXaNWNt1jemZvvqKiE9p9CfTFPsg7DchxK3AS9XuoT45PbCsQFTXOyK+Zl12BPaDrxgb8fJnOPpYXLsCaahYxRGa7tDE+Jf1qpUAZ9o3dO9YRssAzH4SKubAuUHG6ZryLDLqnYLUWs9MKVctcT850JLHDagZL7NPvKQhurBGedPY45YyNKDQW/njWYEFvUwq6LJjwgEJ+92NsU/QqaHVjbW0TU2bn4shKT8yzBXvbh+q6ZHUcTUhe1cEcqBcD3WvWLtmcD20Ji4gGOZXjuz4UeWLC20jgXCliSlAz8rmJOSnPOKWsdO5XgJu28KcDZ+dRazxOAA2f4SdXoCmWA/kD/roP2SiaaN8eCRrz5/8YbjZwd6Mn7oFoyL6TdV50Rb38ibtXD+vtV47I43vOxPamqAe04= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: JWQ0cPi5UGdiW4AtPGdC883W8SMi99/P23hofy3cuz5YFhPKqZU3A+bTABCr7QUtQTtIcu1Dh7rXd3UX6dQ2XfLmPeOK/V+yLH+9lwGYg6DrIQzuTUGNz7R+FRep2e2/28a1a4DqLLdykx8n8alMsWs5o3/U6xzYKkmdsD4Plb60ntLf4PfrS+qJSHr7h4cElVjshMdZn/jogOhZWucGNlhtP0iywwt+LVO7omBfdxASGpAXPYKaV54ucCJxnDtziSvhmCULe1DwlX3GbrKcNp3lRMu1UDK1q1x0pTU4Q7ca9ytXqLK28vKVopH7JvoKN5jEH9mdx2FQnjk83nYzF8AiCAgjES4zQXQFbbs1IfMj6mybf2uj14ixl6giPsolD9BU/jkD8e0FQfP7JntFu+eKuAmAuM54/3A6MxV6KgUVjNgCCKxL2+c88G2H+R+cABePzc/0pidTC8yomhKbylVymSsy/IPzbYegeVUkbBmHP1GiRuhadYhb3Yii7+nNJ9Pj4kRvvkSNJmHVgi/e7WcwhOjWLfmV1JNba1WBJ2wYm+eELw8jDIM2KDKLbUjpubE1c1CTQwd5yQ0W5GFo9yPafMTyZxd1qTTwLurPnPN3zOCEAvXE5aVYvoDPu0dBR4vnGSmQ/j7DKWM/VfQ8wMNjkeVczsTTTsP4nFiUYSjiudtpAGWI6zi45hTXCkEFWGSyOOTPci/pUbQQRrmNkflghnY27dvA2bCs54sPZoiyOyMXNFYHUNIMJGaqZQ7TskO0H22yV7M6sVKa5cXv86TAm2NCeAdwMtF1zqIhAhEHUhIDisxSLn4Tn2YTqGk6wsukA7t/08wkbKowCkb/aCcbDtnQLG1TlKFE0YUEwx1LjPBMvWCrnDM4+aoQSzJu9lrjktEWU3BTEm1BkZ4xW8hw2tZqr1dhdln+T4vfD+g0fYVnrd2yre1OyGjtj5NOZErvcpAbP+bUnzahmChcJP/5H5/LPDF3xqDYC/zSSkMXdc3O/5HY+QyWvmgmRQwZN3OZe0ch1rvimmL++I+Z0dap8CVmGxSxuoeAGmH9W1KC1E39HkDA5WGiBAChxedTVLpm/0F5Vya/48wBfMIqAWFVH3p4YAOJiH/Fw+zODJiWjcapAtAa+b/Kim9bBn4yMMaqJge99ku1hWHC+t5UMD/GkS9D5x2QBR8zQNT912iXv3Y0JBnAQnD8LxQIIe1bebXnNV8txggbE7mz7FMQtc9AcjiPl3nYQ0KsXQJCWO8CZSecmnyk1hLJuOUOntZf3Rn8WMNnQdU2D+i0OyFqicMfipwf3D6+SrSEaVYBOHO3ByDAnAeuz7DkLhHUFiClCE6PL6EKKo3fUImXwztuaN59EH+r4w/vl7upAc6uUWXKmgirOhBvNpdISFnbXMq2Lm8iOTdS0yV2SAPLfX87A3ZRub/HH9v8jiLjT3MpvDbyu13+o1mxsmMhI9Dc3SVBF3aZSewSVqts7/VGi7GKgMv2WdPnNAG4nP/gIbr2D8zJELy/FbtQZg6g0ZHGEI6WdaF7VtAHzCTxK/xXl86PvfqXaCUk9bfRN1cyXwaBGEFd+Jw9uxV2QQwmox8OHKGj X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: b592a4db-53ad-4fcf-7355-08dc4e88e0a2 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:17.3149 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: dcGZkdvYAjERiHetleKvzTQz250Gb/VzC3ZD0Eqg2NDsW0WlIbOt1UnVzvjDMWWa X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110828_636780_2D881CAB X-CRM114-Status: GOOD ( 15.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Only the attach callers can perform an allocation for the CD table entry, the other callers must not do so, they do not have the correct locking and they cannot sleep. Split up the functions so this is clear. arm_smmu_get_cd_ptr() will return pointer to a CD table entry without doing any kind of allocation. arm_smmu_alloc_cd_ptr() will allocate the table and any required leaf. A following patch will add lockdep assertions to arm_smmu_alloc_cd_ptr() once the restructuring is completed and arm_smmu_alloc_cd_ptr() is never called in the wrong context. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 61 +++++++++++++-------- 1 file changed, 39 insertions(+), 22 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 54571f2a4acd5b..2ea4fe9d6594bc 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -103,6 +103,7 @@ static struct arm_smmu_option_prop arm_smmu_options[] = { static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain, struct arm_smmu_device *smmu); +static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master); static void parse_driver_options(struct arm_smmu_device *smmu) { @@ -1212,29 +1213,51 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst, struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid) { - __le64 *l1ptr; - unsigned int idx; struct arm_smmu_l1_ctx_desc *l1_desc; - struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; + if (!cd_table->cdtab) + return NULL; + if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR) return (struct arm_smmu_cd *)(cd_table->cdtab + ssid * CTXDESC_CD_DWORDS); - idx = ssid >> CTXDESC_SPLIT; - l1_desc = &cd_table->l1_desc[idx]; - if (!l1_desc->l2ptr) { - if (arm_smmu_alloc_cd_leaf_table(smmu, l1_desc)) - return NULL; + l1_desc = &cd_table->l1_desc[ssid / CTXDESC_L2_ENTRIES]; + if (!l1_desc->l2ptr) + return NULL; + return &l1_desc->l2ptr[ssid % CTXDESC_L2_ENTRIES]; +} - l1ptr = cd_table->cdtab + idx * CTXDESC_L1_DESC_DWORDS; - arm_smmu_write_cd_l1_desc(l1ptr, l1_desc); - /* An invalid L1CD can be cached */ - arm_smmu_sync_cd(master, ssid, false); +static struct arm_smmu_cd *arm_smmu_alloc_cd_ptr(struct arm_smmu_master *master, + u32 ssid) +{ + struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; + struct arm_smmu_device *smmu = master->smmu; + + if (!cd_table->cdtab) { + if (arm_smmu_alloc_cd_tables(master)) + return NULL; } - idx = ssid & (CTXDESC_L2_ENTRIES - 1); - return &l1_desc->l2ptr[idx]; + + if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_64K_L2) { + unsigned int idx = ssid >> CTXDESC_SPLIT; + struct arm_smmu_l1_ctx_desc *l1_desc; + + l1_desc = &cd_table->l1_desc[idx]; + if (!l1_desc->l2ptr) { + __le64 *l1ptr; + + if (arm_smmu_alloc_cd_leaf_table(smmu, l1_desc)) + return NULL; + + l1ptr = cd_table->cdtab + idx * CTXDESC_L1_DESC_DWORDS; + arm_smmu_write_cd_l1_desc(l1ptr, l1_desc); + /* An invalid L1CD can be cached */ + arm_smmu_sync_cd(master, ssid, false); + } + } + return arm_smmu_get_cd_ptr(master, ssid); } struct arm_smmu_cd_writer { @@ -1362,7 +1385,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, if (WARN_ON(ssid >= (1 << cd_table->s1cdmax))) return -E2BIG; - cd_table_entry = arm_smmu_get_cd_ptr(master, ssid); + cd_table_entry = arm_smmu_alloc_cd_ptr(master, ssid); if (!cd_table_entry) return -ENOMEM; @@ -2689,13 +2712,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) struct arm_smmu_cd target_cd; struct arm_smmu_cd *cdptr; - if (!master->cd_table.cdtab) { - ret = arm_smmu_alloc_cd_tables(master); - if (ret) - goto out_list_del; - } - - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); + cdptr = arm_smmu_alloc_cd_ptr(master, IOMMU_NO_PASID); if (!cdptr) { ret = -ENOMEM; goto out_list_del; From patchwork Wed Mar 27 18:07:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D3BDCD1283 for ; Wed, 27 Mar 2024 18:09:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qdRy5rGjpxASA9alW7lwdZneH+hRoFVj9z4b4oJ9MGo=; b=qouSkr+H3nXllu dDYNAMB2JmTz5bd8FBjqteYvS42r0+J3UmYzQapCL6e5MMcatXU/sVNRdVldhE4gdhxk2XR5UVbUP WZkZtEo7xXjm/cEQXfp8Nahdde8ZSFL08Pt3GEYWNd0/GKqiAeeP3omV/9xak/CcTe8XaOxKqr1Ro EeeQnTQdirgakf7K+X8LeC0IUxdF0ps+ZNnDAGhXjGUJyWTBnC8u8mqwdp8DjrJux9DYcDyZQqCim x77cXruY0xWzR0v5D6aG5y7eVPGPtgKzRWoQiQWUc/hK82kPPjUYELVTSlSVUfYjNlPz8Rfn7eQmq xOWKkbm7pDL8P9t4SilQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhd-0000000ATlG-1ns4; Wed, 27 Mar 2024 18:08:45 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhV-0000000ATe0-0oHX for linux-arm-kernel@bombadil.infradead.org; Wed, 27 Mar 2024 18:08:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=ZKIg8fkiyGwdfKLRu0dNuardwFyH3L09ZLZSUEodqQk=; b=QKj6Vro6OngVL9z0sqRmrdawSn 55ng5fc2rVgY8TnNWttHwZ+pvJL7myhl7DtvjuBUrqgZiXpnaMFIvZcZAQkfxV3dW2NaWq0ttSQNk NanTn0EkHclp5eEoAEA/hFp9ZqnX0JUcQ79O0rRpmHL9uww8rVuz53xhQt0R6760dv72gxl9zijCj jlabNBALOKjlDZ/Knd8qFKxVYXi5Ub7F4aDcZ4SrBqrX19KY+LIFBULmmavEe31HGSvqSnf8Mew9d hL3tjBZBRcDKAjvahGkdP1cOM94GcueyhdDGADsYsMWh/NSB3f+H0550GgsqwKIywRnkA3OkiJRoB zmKeLSLw==; Received: from mail-dm6nam10on2062e.outbound.protection.outlook.com ([2a01:111:f400:7e88::62e] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhR-00000000RvW-37BS for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:35 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cIpQcWfd3ZhknHeZAJtfd749zwHDt/SwfhqIO4seWLperwyiIK2/2dtjdWddvM11I8yCdlXjZPzxmeYaxgWFJiVGZJ1etq1IfebT04N3EznQy8id5reiaL9crWu3mNG0TW4jtlZaHa8828gXm9gyb+kXQcMoF3mI865b2CmMtd1F0h4Ie2DHqlUVOaylWc7tK/ODSuCrVasx1NGiOVBgssZ3JQ7LpW2orGHVHpwhxwOV91TTb0/s6cKuGKH6WnAWjFbuEdRT4WPpVDSTBh1BVQXB9ZwogoibFiW/hu9Q+43PIILb6r8MaAYIDbT6At0SC0qNlwjjPJUCr3A04H768w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZKIg8fkiyGwdfKLRu0dNuardwFyH3L09ZLZSUEodqQk=; b=MJDlE2Q09swmGvoijCX9hkWj3rai/tu+kPV4sKRv6NgvuF3EWcht4jQm28KYin3fE7+ZaoYpp+TA2qoHSI5f0TIgepZIZg/AIa1wz0MP1AA3aIEZygcBbe4aN3YDUM/l7ESZq4VWYXOymM/zo8kqBbMWP21cQZTe1zME5hL5E7MUptti2H+YC4mwRLY2uqVkVCO2qrE+XhTK7P1Z7qsLAIoxQiObIbD7+IWwNayRoZzAhjyv2nG4JFE07MWDsSAQbRZTSuMXCOAbaxTjfWjrt8LL8dUcizGnnEwKxJXMFm3fslDqlLLowPrg2UP5ee2cEh/joE4lMgp4vg713qM2TQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZKIg8fkiyGwdfKLRu0dNuardwFyH3L09ZLZSUEodqQk=; b=DHTPa9JCUB+VyRyUbyuW3JWjYA+q/97aZPjr72nA+lzNx61/2CHUDvWb7u+E3WDFmLQFqHLR2MfRYRx/DFh+rFE3KCbkz1XN6/GuiGfz6kMQFJgLoI/8RdzBzYVyculb9tVWr6a8399ShP9mMnPDEJ+aQOt1P0hRT6qnL7O9dv1wNkf4yOd/nUh9PhtETmfi/LagWBkSwXME2v6bQgpqyimQgChzCcgXAzv4AxOIFrdzkx14RusdGbJq+MuZlm6xjWod9KOAe9VPS/Go/uQW+C5+S7DXqGqeoZVdU1FGJiPqvRfItV5Jpu3XVUmEkIHvm6t8m1iyVde45Csj4jeRmw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:18 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:18 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 11/29] iommu/arm-smmu-v3: Allocate the CD table entry in advance Date: Wed, 27 Mar 2024 15:07:57 -0300 Message-ID: <11-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0039.namprd05.prod.outlook.com (2603:10b6:208:236::8) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 3108e2cf-a03c-4fc9-4821-08dc4e88e0ad X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jjMZO2P4bQdn03/a51G9zOqoVtFfDvdoixoNV8PIeO2E+6YPclNYqSakuZl/qjp5mBEhA42m+FkWB94Z7zWbjvW1vkUnqDs4PcdGTjlQUS4Xzt+4Pd1eMp0YTF/l9JOp4z2XSB6OIp4L0QHexjsIJazD4t92889xhP9oxdVKQvHdsaHqFuvwZTjhrv8+SWtgDJKE/jbxVw0gN+q0nWjORegKyinZFcthBrdhrlab/nHE0Ojf1LKUmkneE2E8MqjbHJeEjWvp3ceOcNngDxtCMF61XlwOCq5rZTO+QoZZQs9CP0yTjCh/AvYkCVD59p1H79IqjG3gHRdEJJX9wD1GiMaxkl+7c9Df8B5QMGxeJB51NVt5GMTF29ZOUaIPfw8kvppMMGDSZUYWsTka6DyFAYhYzKD/T+VqIw8Av5ra6MJiDrfabq0+/677ZRVjH2veSqz1JsftI4s6I29U17gjFbKiMeXmyG/OB2zgHraPg3otkeoStWPVmHs6jtnYKV2J7wb/9bJKKs8J8FQfbTjKXmYbpRXSMqEGqYTejRd/r5fN6o7PCkegVTK8pp7y103aqQzKewzrUcvDlhVjo08O0zMbCa/KpxgLlQ8GPsl1lpxELyJw+TT54rBy/NlLs2Yi6P2F4w69hb2SZPlQfiMQY/hEL/1REieJf/+Ri3TgDdQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: dKj0VupSrK1Qd2Y42xnKuLdrYBrJ/NS9RB0lYhCuL97M9j9J3nqCIVXxu+QPrA8zpwb9LHoWuYivusKnpSN4kqZoaJJ32RAYnZ/Wcki7NTXbPiEVD5N/W/KggiZffLgT1XYxFP6As3RBDXZlNfLnOT8IQ8daCRw/MOaCM2sgQvHG9WalPtZs4oVnYeASsOwkI3C76fJF2GWe+R3wSBEdPjL5uqjj/JtI7bUYWnUJTV8Nbp/9Vw+k09fcZExfUUFlB08u3oOIZg55vvQgVnGHrFjsr+zb9mgBXBAyscPStzYQhUdDlR79B765G0g+2HoW3tq0DFfVnyxsd42L75jEPAAjXxisKSo+F88YsLaOvhLFHSm1fKOmAflyT4CtE3pTjlUIM/e42UePRrYrmaVKjrhPzn0SPJ1dFqUcf5swmknyIQ68FI0VoBgpY6BmMEtAitRqaB3qXySq0bgTz+ieyiWe/d48AT1+Uge8iV0OyNvSO3kGvbsqV3niU1c9o7g2YuztFgPkx+gm64hG0W16DXCJ7hmbegxB0z3CnljpHEiiXSBj2IDhcTcBPvj/ua1+zxEMmnaJJ2/gvTeC/s0vjJlLaKkcLwkfEDDMmLQVQhA3w8HVmGBWWnRgs2dIwFx/6OA3Jxqo5s6eVWC+OMZUl2WPmBYG/j0jRi9d1gQ+J2mqBReIPp2KLUseq/8y8PBiTGL2RDklENVtZ63jZ79PX2RKqoF0D4ElbUMzz67iixTg4LFkpgbGXmLZCvie3ILq2lsFzqnwXy4+LfEW4l0dTpMCB0+bI9j3FmnM0t8VXHQA9pn4jakLezxX4MrtXWGDHerZJMgYxvwlVh9Utdp4eMMkhIUYDcCKQvut5nmRcty5YkzHRxbE8oALWM32B0mqDO+sQmnJWgMATjaFhbYrBY/Da92v7sESKskIvTRcb1N72a0kZpTRhQ2VynoKk+UcEpMeAsOBkMtD6LxJ6boxSlz1t1+705kkRXKqAIThXLcZElAxXev3SxPX0oTdnaaYwEwulhJIL5itv1GNFc9ieUEuv6ZF62HuvjGZeLZONjB/OS04l4tT5cIr987grTLc4FR4fIV/i/R7mNkG1OzrCJ2MmFMvNWfjNJN3r9VfKGzS1+hcGv/rOf/SWHwco+hmnQUx0XYfjni5kuxhslq4gqCzg3QNpp6PjyrXoG0HmFa8ukTvJKCRPXlhNoNPjXptBYBUkb2tGZt0m2ToCz85a8bvXeLFMLOmMsD4CC/h/NjPGJnjx+O4S59ypCzJQIRgHQfKxz3OUo3v0ZMjTeSo+2RsDWPjGxAz2QXyZsAj2gIcpGnJ96R4ovFerCp0rEn0e6H/ad8jIHvvzgvNFMloJj3QXpck5suYEfaEfocLuMY/IzlrA0pLugkFXBmZenq7GobtFDsJfEJ9twiNKeRbabEdIMqLwUXz6JpOIaJu0H+Q6fC0IOVXfcCWIcs5k9+m9yKufZH+DgKKeCxulMQRFXRR+HNeLq9P33tGfGf1eYtavtSh1WI2aJ3+z+1uMM12wIAHjy4CtrZSFxozowoPj3aI/c0z7PrNvni0A4LSv5VrR7Scwc6OnuHP5ni2/ibs X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3108e2cf-a03c-4fc9-4821-08dc4e88e0ad X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:17.3427 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: TF79dbTXHpfZQfWvXZihvpsVUvza8dLJARCV45c+5aU+c5Yjttdsiz3reYs518Q0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_180834_064066_FB8D7585 X-CRM114-Status: GOOD ( 16.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Avoid arm_smmu_attach_dev() having to undo the changes to the smmu_domain->devices list, acquire the cdptr earlier so we don't need to handle that error. Now there is a clear break in arm_smmu_attach_dev() where all the prep-work has been done non-disruptively and we commit to making the HW change, which cannot fail. This completes transforming arm_smmu_attach_dev() so that it does not disturb the HW if it fails. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Reviewed-by: Nicolin Chen Reviewed-by: Mostafa Saleh Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++-------------- 1 file changed, 8 insertions(+), 16 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 2ea4fe9d6594bc..2bf55ed4e32ced 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2663,6 +2663,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) struct arm_smmu_device *smmu; struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct arm_smmu_master *master; + struct arm_smmu_cd *cdptr; if (!fwspec) return -ENOENT; @@ -2691,6 +2692,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) if (ret) return ret; + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + cdptr = arm_smmu_alloc_cd_ptr(master, IOMMU_NO_PASID); + if (!cdptr) + return -ENOMEM; + } + /* * Prevent arm_smmu_share_asid() from trying to change the ASID * of either the old or new domain while we are working on it. @@ -2710,13 +2717,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) switch (smmu_domain->stage) { case ARM_SMMU_DOMAIN_S1: { struct arm_smmu_cd target_cd; - struct arm_smmu_cd *cdptr; - - cdptr = arm_smmu_alloc_cd_ptr(master, IOMMU_NO_PASID); - if (!cdptr) { - ret = -ENOMEM; - goto out_list_del; - } arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, @@ -2733,16 +2733,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } arm_smmu_enable_ats(master, smmu_domain); - goto out_unlock; - -out_list_del: - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_del_init(&master->domain_head); - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); - -out_unlock: mutex_unlock(&arm_smmu_asid_lock); - return ret; + return 0; } static int arm_smmu_attach_dev_ste(struct device *dev, From patchwork Wed Mar 27 18:07:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607258 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 943AFC47DD9 for ; Wed, 27 Mar 2024 18:13:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Yn+v8xJ+LJ0nHxtChypeVDoA0l+53EjIGNaBPskHLAo=; b=MyQt0iZrBhmpBg KVuo62ZHP2WFLhCYoTfTOD28Tc7ZKul5pzUEbX+tqb5pBNcWobBROAIvaqMlVDlgovXqXYoV7uBt7 grsPpgNLLfDUij5rDbHTFJvyMO/F0ttNFmLY6y917UeoZT0/rPFOL3mj9bqCfLr9vM8CMTWKbfEwD yL+G5q5SXCmStRQLN28gRwIHq1HS2f9dXB4SNuHsUW7axyXR5UfvZjpdTSE/LSdwP6akt1dUmmZ2q CH0BkZpnY3KGp43X5z82ZFN8zaerG8+iFAu+Kx67/QK8Q1luVr8WuC79WMFc9UbPcpIt+Epx6eLkD WKh7PDKqGaUiVTIY1oNQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXlo-0000000AVg2-3190; Wed, 27 Mar 2024 18:13:05 +0000 Received: from mail-dm6nam10on20611.outbound.protection.outlook.com ([2a01:111:f400:7e88::611] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXha-0000000ATdz-12cN for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:45 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=StTmEQES0fDRsthy/LR30nSlRpfL7duFNAgHAY4A+YJS5N/2V89qhjvCHLr5EKso5VtnG4QOjAKuDUWENJY7sa4dF3gIEB/ckG+VjFpjYq7FLtcFOlu8CWIC8W1BBXwBPRDsmweyk7E7+3BJmW+P8LA1eslVhzp7KvRmjrnq4wKOciH/UyUW9LC1BQXKfj1d8/5aOs37CG31IahERcR4kosK8tmubz3SbZBa2tplu0sBQu2q5EUlVZ8zsTyeVBRVzLM0qzlEHGf+U+bi2j6OS/BWyxszuQG6cM9hMphofvvuNQuIs6r74DG0K3NwTntPNtwpUwMDoLZPgK6v3mGEeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yxSHe/Gm4v6Fx9FqWD10FkfN1pRoQw7ctPe0LMinBtQ=; b=c/nCsR8jaG838a+eI6u8UV1hm47ZlsVbtKKiEZAxIqQcMQJKxSQ6Mm+zWgZBvBvp/Hkm2LitULFY5SKaWvbjIkeIm6U4TPiGNBhk08KVsQIjvhdVI68Y8GDaL3N3jBINH0j0fz5AAluk4fpnZN4BSP/Ys6h9Im8v+DympqDNpDEbaxjvnBFtRIF9ml3y0V1AXDc11gSgPanFoUyuaU+HpgGPkK51uai15dUKJIff287VrNY4TxjEdL2HktPlLXPI49f4uOIyjBPsMpjEjjvw7qiWRt79HiLbyXIDzUPwU3d8GMtcSOmwfVAgjLUhbvqzfmUqegWf8424659SiVf4zg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yxSHe/Gm4v6Fx9FqWD10FkfN1pRoQw7ctPe0LMinBtQ=; b=Hz/ezf5NjbA6zt+/ldV76OrrLtgq6CR/x9sn6+5/4vG1PBxz29df2n7qg6CIFTMl8t4BYEv6oese5iDt117yrDlFcsTSPzs9QYd3hRC4C3vp7NgPgM8TCAKsvFxr6vldc5NLbXZoEHdzFCkX2wnzr21ks896+f4D/15QvSjztmuDlhwkLmcqEl+kEt+3hayO9nYccmGGu4QlN6s1ySVJzVj5wJFnnux59TZ/l+N5TME61msTcwi2JNeKcTrmpPh/k7WMkh10OHP+hq4fd0kM9NuL0riPSKfkdmWRCGCfrsauW5kqKwe9naHpZn60VIxwaBRug5zRrwWY4y6LtZfxKA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by IA1PR12MB6044.namprd12.prod.outlook.com (2603:10b6:208:3d4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.33; Wed, 27 Mar 2024 18:08:32 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:32 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 12/29] iommu/arm-smmu-v3: Move the CD generation for SVA into a function Date: Wed, 27 Mar 2024 15:07:58 -0300 Message-ID: <12-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0310.namprd13.prod.outlook.com (2603:10b6:208:2c1::15) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|IA1PR12MB6044:EE_ X-MS-Office365-Filtering-Correlation-Id: 14bde0c3-6555-4f01-320d-08dc4e88e1ff X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ITdkibXyS0OCZ8f77lv0nNPfcoTny3MV1/4l+vzC3+TLU/bD0Nrr67n8t0cAF9JJsrRQsgbuQp8AKBVHCmWE28TqPvNPkjjZlN8KPjXGt4vlJE3uSqBsTB5nNtldwG8wP3s4IoBGDoMK0ylNCzYjDdseoRteauH5s3dMcKu4laVZNY3nQzUYW8y4c4MGZ6nz03EuMrqdV7hbsfbhEootoYYDxthYdWiXBGzNNkyO9ixcnheYK4mE32fNndabyJx0t6jz3uUqaDM6Qmv8e99DcAk7kdUy7g6HRtmhw0r6wlkYzRsVqfMq2/vw7MP6tEUdSZNdWIfR8oXQsWDY2cCvHAjl6TOsYwd7/M7al8DbFbpHz8dhi0RspRTmjvxYPXQzVHAc2XY+/n9uaBd9KgCANo64n1v+N4RX365cXB8Adl4v5u031h4jtNNy+X9MyOKwYcSomdxXyVfr3mUw2wzyF5qk2vrskoBOhYj5Z+w7YVQtcWI8F7tL3oaE++OoNIV9rBmidY8rvL7mRmJi60IOKRnEUz/8R5JV7GNbktbmf3IFJmst9ZdREZ/v/m8+pUAhB9181LPt49znMJ1e3yrB2FJgBOYxXmYu86xnIQVKRC4HA3RLRLAAFPr9EJmdPh3y23oJ0DCt5zc4T3D7xk0gUvxJZcWF5SggY2trABS25wU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7416005)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /2Z68MCYa99s6JvH0nd/BY+cRrRR7QQ3cuL0OxBOMHMXnx3U58qNKK/u4Xyzy72ADwlqeq6HrbEGoVU4t1tUqsYyivbqzAGykFyuDf7sqdSQy+WQGYzaWRWNMEnKoLbwDXPeDUdlabBDf9w4cT3nBVtMlfB75MsYWTH8455mMpWJAO8/UufXLGs75VkR2uEqjmkI9ihb7HpPfrPhDtc1v93sGOQkpktwfbXHKU4/b65z+07DwKek64SIzyOGvbAR7wh65OceI32RVLAuitEXgkXi20ijphRKe1RxNuHPGoCheKCvLyHWDBmptKCgmOvuj+BXkeR3GVg5pmG9n90F3iiS/dE0GFjB7s1DrASXGgD0BTcgLwgPlodIk2jXYzolIe7Ikofn3EmdPJ6KmpIfHmSPKHmNL7M9I9mvr4ElD/oBDFdz6odIqiVST8vc1sMrWIv719+u9pxDl3nAKNqgDUzBP5EMUtdVGX0rjE1V8Km9lJPxsifUr/RIklj/1CwQCQMwTES2hJXSty+hwy9M3AxIMsOMZE89df2sqEVZQEVOC413D+aXF++YGLhgzR+UyOfKCOlHwqcJp95AIm6etjmNf1vgU7fjuGZF8ft77jOej+R9B2OsmEu8bvCqMkSys2ug0y547teRUpIviUkz3Mev1A7FlssfFGSVIB0cgTKOpKaz4jtfvkyn5fCDYjJI7WU5BUSN3LiSBQ9+TrmM7FAmWTrlJEzNq8ICEh0WE4CKUdjucJ4ufOIHdtRk1gU5XUb+dFu2Kadd7xqL4jFaWwFKcWZSUETKPVd8SnG1CyAR5EEXJVTbfUUGykMjel+zYOiXR8RzO1TG3GqyLsyD0x3Gd5za+gZ8h9DRc7DHIJAierAiH5Fn8G99Mby5AUBbZLvJMi21DI5M7dQCTg/Q0Z1EjvHpTAp6sNKoYdZE8rHM/qIyox+UzH4RxcjYCdT3RsRs3NbEXSUgkboUD/1WsYc7R+1OBtDiwNigjwYM1ZxdcGS91XXArTIEvBJL2dFPtQLvqW8/YXnuziV+djr/MoRAnc1DjKw4cVH/HlDXkYbi4Zgd/ZsuTwwXsKe9D/LQmzAT+nSW99F0tHDx6bUmAZh8AhDhPIlDYl7XCEovPvsDMnE+JXHcc2hAnzC7x244ZKt5AKtszw4DMlLYJRtQ/6O8PFL8l8JeFH4Tl32UwjqtnokcbVw0Yc/WP7xJ+EV3PapvGjQsYofjNN/rP49TXBbLf+TvIlym3/UiNTK1lFuPlHbi+b6SZh5tUN/JvLoOTeGuYnMls8sRLtaevHYnqPYdBcn0oMbg4gpx4OpJ5XeARkOLi3abEiB8t3BC2DlsuRFKwGt1HyChWvc63bGSRyQBtrmPm75n9yEFRoFPYQzsgoPhFXpwVZ3uU+FWJKU8VZPTJXaTVa305nwAtx9T9El1sLglLpkIR/tgqWhquF0mzsnx+IJdku8xFKeq7lOvq1JwO//f24Pw2eE1X8D9rXom+OE5x4f3lajc2KsBSWWg42+7Kijx9Jk4bVm4goT6UWEccsdAygoiVSKzYc04iNhdxiy2NiZweciXeVt5lfIi8eLP0EcPdMYYIQO24UK/ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 14bde0c3-6555-4f01-320d-08dc4e88e1ff X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:19.6260 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Niu6dlZKAHYn4yQ/yeEDH6erPLIMU4C6qYFIwNkPUaJGCfTRrhbNBoTx2piaKdSC X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6044 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110842_371321_D95B5361 X-CRM114-Status: GOOD ( 25.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Pull all the calculations for building the CD table entry for a mmu_struct into arm_smmu_make_sva_cd(). Call it in the two places installing the SVA CD table entry. Open code the last caller of arm_smmu_update_ctx_desc_devices() and remove the function. Remove arm_smmu_write_ctx_desc() since all callers are gone. Add the locking assertions to arm_smmu_alloc_cd_ptr() since arm_smmu_update_ctx_desc_devices() was the last problematic caller. Remove quiet_cd since all users are gone, arm_smmu_make_sva_cd() creates the same value. The behavior of quiet_cd changes slightly, the old implementation edited the CD in place to set CTXDESC_CD_0_TCR_EPD0 assuming it was a SVA CD entry. This version generates a full CD entry with a 0 TTB0 and relies on arm_smmu_write_cd_entry() to install it hitlessly. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 156 +++++++++++------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 103 +----------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 +- 3 files changed, 108 insertions(+), 158 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 7cf286f7a009fb..80a7d559ef2d3f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -34,25 +34,6 @@ struct arm_smmu_bond { static DEFINE_MUTEX(sva_lock); -/* - * Write the CD to the CD tables for all masters that this domain is attached - * to. Note that this is only used to update existing CD entries in the target - * CD table, for which it's assumed that arm_smmu_write_ctx_desc can't fail. - */ -static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain, - int ssid, - struct arm_smmu_ctx_desc *cd) -{ - struct arm_smmu_master *master; - unsigned long flags; - - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { - arm_smmu_write_ctx_desc(master, ssid, cd); - } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); -} - static void arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) { @@ -128,11 +109,86 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) return NULL; } +static u64 page_size_to_cd(void) +{ + static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K || + PAGE_SIZE == SZ_64K); + if (PAGE_SIZE == SZ_64K) + return ARM_LPAE_TCR_TG0_64K; + if (PAGE_SIZE == SZ_16K) + return ARM_LPAE_TCR_TG0_16K; + return ARM_LPAE_TCR_TG0_4K; +} + +static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, + struct arm_smmu_master *master, + struct mm_struct *mm, u16 asid) +{ + u64 par; + + memset(target, 0, sizeof(*target)); + + par = cpuid_feature_extract_unsigned_field( + read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1), + ID_AA64MMFR0_EL1_PARANGE_SHIFT); + + target->data[0] = cpu_to_le64( + CTXDESC_CD_0_TCR_EPD1 | +#ifdef __BIG_ENDIAN + CTXDESC_CD_0_ENDI | +#endif + CTXDESC_CD_0_V | + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par) | + CTXDESC_CD_0_AA64 | + (master->stall_enabled ? CTXDESC_CD_0_S : 0) | + CTXDESC_CD_0_R | + CTXDESC_CD_0_A | + CTXDESC_CD_0_ASET | + FIELD_PREP(CTXDESC_CD_0_ASID, asid)); + + /* + * If no MM is passed then this creates a SVA entry that faults + * everything. arm_smmu_write_cd_entry() can hitlessly go between these + * two entries types since TTB0 is ignored by HW when EPD0 is set. + */ + if (mm) { + target->data[0] |= cpu_to_le64( + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, + 64ULL - vabits_actual) | + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, page_size_to_cd()) | + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, + ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, + ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS)); + + target->data[1] = cpu_to_le64(virt_to_phys(mm->pgd) & + CTXDESC_CD_1_TTB0_MASK); + } else { + target->data[0] |= cpu_to_le64(CTXDESC_CD_0_TCR_EPD0); + + /* + * Disable stall and immediately generate an abort if stall + * disable is permitted. This speeds up cleanup for an unclean + * exit if the device is still doing a lot of DMA. + */ + if (master->stall_enabled && + !(master->smmu->features & ARM_SMMU_FEAT_STALL_FORCE)) + target->data[0] &= + cpu_to_le64(~(CTXDESC_CD_0_S | CTXDESC_CD_0_R)); + } + + /* + * MAIR value is pretty much constant and global, so we can just get it + * from the current CPU register + */ + target->data[3] = cpu_to_le64(read_sysreg(mair_el1)); +} + static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) { u16 asid; int err = 0; - u64 tcr, par, reg; struct arm_smmu_ctx_desc *cd; struct arm_smmu_ctx_desc *ret = NULL; @@ -166,39 +222,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) if (err) goto out_free_asid; - tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - vabits_actual) | - FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | - FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) | - FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) | - CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; - - switch (PAGE_SIZE) { - case SZ_4K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_4K); - break; - case SZ_16K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_16K); - break; - case SZ_64K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_64K); - break; - default: - WARN_ON(1); - err = -EINVAL; - goto out_free_asid; - } - - reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); - par = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_EL1_PARANGE_SHIFT); - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par); - - cd->ttbr = virt_to_phys(mm->pgd); - cd->tcr = tcr; - /* - * MAIR value is pretty much constant and global, so we can just get it - * from the current CPU register - */ - cd->mair = read_sysreg(mair_el1); cd->asid = asid; cd->mm = mm; @@ -276,6 +299,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + struct arm_smmu_master *master; + unsigned long flags; mutex_lock(&sva_lock); if (smmu_mn->cleared) { @@ -287,8 +312,19 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, * but disable translation. */ - arm_smmu_update_ctx_desc_devices(smmu_domain, mm_get_enqcmd_pasid(mm), - &quiet_cd); + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_for_each_entry(master, &smmu_domain->devices, domain_head) { + struct arm_smmu_cd target; + struct arm_smmu_cd *cdptr; + + cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm)); + if (WARN_ON(!cdptr)) + continue; + arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid); + arm_smmu_write_cd_entry(master, mm_get_enqcmd_pasid(mm), cdptr, + &target); + } + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0); @@ -383,6 +419,8 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid, struct mm_struct *mm) { int ret; + struct arm_smmu_cd target; + struct arm_smmu_cd *cdptr; struct arm_smmu_bond *bond; struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct iommu_domain *domain = iommu_get_domain_for_dev(dev); @@ -409,9 +447,13 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid, goto err_free_bond; } - ret = arm_smmu_write_ctx_desc(master, pasid, bond->smmu_mn->cd); - if (ret) + cdptr = arm_smmu_alloc_cd_ptr(master, mm_get_enqcmd_pasid(mm)); + if (!cdptr) { + ret = -ENOMEM; goto err_put_notifier; + } + arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid); + arm_smmu_write_cd_entry(master, pasid, cdptr, &target); list_add(&bond->list, &master->bonds); return 0; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 2bf55ed4e32ced..af5ebedf0f0beb 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -89,12 +89,6 @@ struct arm_smmu_option_prop { DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa); DEFINE_MUTEX(arm_smmu_asid_lock); -/* - * Special value used by SVA when a process dies, to quiesce a CD without - * disabling it. - */ -struct arm_smmu_ctx_desc quiet_cd = { 0 }; - static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -1206,7 +1200,7 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst, u64 val = (l1_desc->l2ptr_dma & CTXDESC_L1_DESC_L2PTR_MASK) | CTXDESC_L1_DESC_V; - /* See comment in arm_smmu_write_ctx_desc() */ + /* The HW has 64 bit atomicity with stores to the L2 CD table */ WRITE_ONCE(*dst, cpu_to_le64(val)); } @@ -1229,12 +1223,15 @@ struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, return &l1_desc->l2ptr[ssid % CTXDESC_L2_ENTRIES]; } -static struct arm_smmu_cd *arm_smmu_alloc_cd_ptr(struct arm_smmu_master *master, - u32 ssid) +struct arm_smmu_cd *arm_smmu_alloc_cd_ptr(struct arm_smmu_master *master, + u32 ssid) { struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; struct arm_smmu_device *smmu = master->smmu; + might_sleep(); + iommu_group_mutex_assert(master->dev); + if (!cd_table->cdtab) { if (arm_smmu_alloc_cd_tables(master)) return NULL; @@ -1350,91 +1347,6 @@ void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid) arm_smmu_write_cd_entry(master, ssid, cdptr, &target); } -static void arm_smmu_clean_cd_entry(struct arm_smmu_cd *target) -{ - struct arm_smmu_cd used = {}; - int i; - - arm_smmu_get_cd_used(target->data, used.data); - for (i = 0; i != ARRAY_SIZE(target->data); i++) - target->data[i] &= used.data[i]; -} - -int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, - struct arm_smmu_ctx_desc *cd) -{ - /* - * This function handles the following cases: - * - * (1) Install primary CD, for normal DMA traffic (SSID = IOMMU_NO_PASID = 0). - * (2) Install a secondary CD, for SID+SSID traffic. - * (3) Update ASID of a CD. Atomically write the first 64 bits of the - * CD, then invalidate the old entry and mappings. - * (4) Quiesce the context without clearing the valid bit. Disable - * translation, and ignore any translation fault. - * (5) Remove a secondary CD. - */ - u64 val; - bool cd_live; - struct arm_smmu_cd target; - struct arm_smmu_cd *cdptr = ⌖ - struct arm_smmu_cd *cd_table_entry; - struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; - struct arm_smmu_device *smmu = master->smmu; - - if (WARN_ON(ssid >= (1 << cd_table->s1cdmax))) - return -E2BIG; - - cd_table_entry = arm_smmu_alloc_cd_ptr(master, ssid); - if (!cd_table_entry) - return -ENOMEM; - - target = *cd_table_entry; - val = le64_to_cpu(cdptr->data[0]); - cd_live = !!(val & CTXDESC_CD_0_V); - - if (!cd) { /* (5) */ - val = 0; - } else if (cd == &quiet_cd) { /* (4) */ - if (!(smmu->features & ARM_SMMU_FEAT_STALL_FORCE)) - val &= ~(CTXDESC_CD_0_S | CTXDESC_CD_0_R); - val |= CTXDESC_CD_0_TCR_EPD0; - } else if (cd_live) { /* (3) */ - val &= ~CTXDESC_CD_0_ASID; - val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); - /* - * Until CD+TLB invalidation, both ASIDs may be used for tagging - * this substream's traffic - */ - } else { /* (1) and (2) */ - cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); - cdptr->data[2] = 0; - cdptr->data[3] = cpu_to_le64(cd->mair); - - val = cd->tcr | -#ifdef __BIG_ENDIAN - CTXDESC_CD_0_ENDI | -#endif - CTXDESC_CD_0_R | CTXDESC_CD_0_A | - (cd->mm ? 0 : CTXDESC_CD_0_ASET) | - CTXDESC_CD_0_AA64 | - FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) | - CTXDESC_CD_0_V; - - if (cd_table->stall_enabled) - val |= CTXDESC_CD_0_S; - } - cdptr->data[0] = cpu_to_le64(val); - /* - * Since the above is updating the CD entry based on the current value - * without zeroing unused bits it needs fixing before being passed to - * the programming logic. - */ - arm_smmu_clean_cd_entry(&target); - arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target); - return 0; -} - static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master) { int ret; @@ -1443,7 +1355,6 @@ static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master) struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; - cd_table->stall_enabled = master->stall_enabled; cd_table->s1cdmax = master->ssid_bits; max_contexts = 1 << cd_table->s1cdmax; @@ -1541,7 +1452,7 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc) val |= FIELD_PREP(STRTAB_L1_DESC_SPAN, desc->span); val |= desc->l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK; - /* See comment in arm_smmu_write_ctx_desc() */ + /* The HW has 64 bit atomicity with stores to the L2 STE table */ WRITE_ONCE(*dst, cpu_to_le64(val)); } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index d32da11058aab6..5aefb0ee2b9bb7 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -608,8 +608,6 @@ struct arm_smmu_ctx_desc_cfg { u8 s1fmt; /* log2 of the maximum number of CDs supported by this table */ u8 s1cdmax; - /* Whether CD entries in this table have the stall bit set. */ - u8 stall_enabled:1; }; struct arm_smmu_s2_cfg { @@ -747,11 +745,12 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; -extern struct arm_smmu_ctx_desc quiet_cd; void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid); struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid); +struct arm_smmu_cd *arm_smmu_alloc_cd_ptr(struct arm_smmu_master *master, + u32 ssid); void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain); @@ -759,8 +758,6 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, struct arm_smmu_cd *cdptr, const struct arm_smmu_cd *target); -int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid, - struct arm_smmu_ctx_desc *cd); void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, From patchwork Wed Mar 27 18:07:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AE6B5CD1284 for ; Wed, 27 Mar 2024 18:13:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=q2srst18VAlFoh5kTSem0cdOzdiBhI4OcmEaQPGDmnA=; b=zHW8e+ExpO0XTS hkr8vGNJHUzdHYx5YfzKGGZ0DT63sJ0x9Oa08FkaFBbOKhw/6JRmnKN5+Xw52qClfPl9aRzmPhWUH shxF5x6QJCbwsKCJgR731C/wQwAZAAOCXsvMZNpkQ1DPCDcVa3WAA5SjqU1B52dgZWMGKBrz5UqJO ql2sQ6yEW+nwapT+587IANCDhzW0mtlke7E9IKz2IdD0l4smsSrvDQJ1VHpTpkGJMwb1f4G7c2SdG dpbLXDMslLTlVhwc17Y33dSW6gZCoIW9U+raSYlxdwbxiXHHeUiaV86k6vUVwoZC5/5/FrOL0JoWW 0HzwL6upsZFt1vXB4iUA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXlq-0000000AVhQ-2WQ5; Wed, 27 Mar 2024 18:13:06 +0000 Received: from mail-dm6nam10on20604.outbound.protection.outlook.com ([2a01:111:f400:7e88::604] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhb-0000000ATY9-25FI for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:47 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ha78aRk8Ym1iyEOasd3n9zttL54CfVoaa6OnglPGgqSQMtwSQ6SDbl4mp/X7aM9kdWNcxPuLG4V1MWB3hklb8YZmQBFrKM8+YSYSv31g1iy4y64UCtnPLZUfxDW3db0BwqQWXDPiv2yd0D/aISGiNkTETKOLdSmpR/+fMlbaTyW5zp5Z5/2Kr3F8ZpcMoP57BUHDghswZ/HByEGzBe4U7JgO5CipZec6v3hnnq0ddlnjNw1dOyOrke+V9KElkWkTpP97ob1coesKXV9WL6dSiZAV80dz0CHgw7rzksvqdl8Nhp5XKXPKa6FWD9vWKhpcpp4y8YtyiyEE8lCHjf5/bQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0e1UYuPTCAp8UJlsQZ32Rt4F34NQGChm2NGhgWKckdw=; b=VLNbyeI/cAMKiHlEHKAxMctK2dKw7KK0+JgC/XmxfHYINe/qHGdfyrQ9i9inhLKg5kNr7EHL+PsxTR96ZfOTUW/fOM9Zn54eb3vgLtIno2yYWUV8W3VP60ON0GpztcJToQvJrqadseVbxutsK2A6j5HPRSbrNimZISBfhMDuttm+yUtM0XRP7NSuiIektJolk6DfKIBmMEx6CzFq98fGiD7pdw+TzreG6bI/Wc/CvYGHOQYoAgW61RLGsLjk3pXVpGubosmc+WsJhxQZTFuSskeNBBE8vekej1ij952log4alKuflUsQejrlVrbSiR3PAIG+xf5SomNIVIc5y2MaVA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0e1UYuPTCAp8UJlsQZ32Rt4F34NQGChm2NGhgWKckdw=; b=AzaXUWPkJAg0zo803xDjww3dExyInJEZKtThZzPlJ6dl/oRIte9zKthp96yHeNk+GOF6WFeZloNkO5kct7T1M2CRVxeKKdPNK4CGZCMvQ8+PtjCjGC/Zb9S2ezwLwdK3k0gICFP9zbjc+8bTDhR3427NhybwU1orjT0HBwbDtWC41Ohv7WleasEFgkYSbNnt03sjS3bFm/mlD2PS2/O+EM+btO0oid52Ms146mif/7bjh0OMSilSi7H7iLwr7fqxT6vUcN73RnB2+GLxMimBJRWpMUsveeNas/ZQFCbymm/ZWonny+ZQ/wNwI9Mf48YQiSlSJvPe2SKDaPNtSw+3zQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:26 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:26 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 13/29] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd() Date: Wed, 27 Mar 2024 15:07:59 -0300 Message-ID: <13-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0046.namprd05.prod.outlook.com (2603:10b6:208:236::15) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 6779786e-a58d-490f-ac9f-08dc4e88e15c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: spufI9xcvTf/TgMxjUwXi/OPa9Dn7dVydqxGgrQqTBaG5ULmaovHVTaGf+C2zU8g1yhMlSe7awiNBWwZxZRKH0J2zbykfYS84XqJA0x79JgnlusF7R8lLX6lnQafWI+g6mz3aEYKeML4G1w9iQDoolJ3vhCfkrwjDxMtmacHdfSyXVmiIdKoNjHO/QThFRpKUPEiS8pXFMeaHk5jLr17vnwxTXkkBpn1/3LN3NW1+77jkM6MA++p0mv7NYODVwiDBlq/mwIHH5QcImzNQoe591io/vokmDieFjbBAyUV7hPXzxVmMK4hYDvznNfberm1fOI5+3a03tlDToIfvQ35unM9/RtrQVmCqmu//pbWzndDI1S9Nk7B9CY7Bev+T4yu3t3m6eZG6av5CBHII1oiY/91NQxFortxIk5D7fNcoMFiKHJrmhHZyVGzCrM+kE1v7lzhwtavjj+9N/74VXF+HneGOFIojg15JiEn0UHdxCWoKYvcQZ93sY17/0QGdBXUuxtYOPjfvRLKOSZpZ50bFyeuRa2QIEPjpinlc1O36TOzqGiwHCJFfoV44d7IsrQF2wc0Mb4z91yAtUnL9Pasqz8enadOByGZZrohilnkX0cT32rg615jAhYWk0mgAR+PqOl1niFrYPk5peT7skGwi6vHnWufQOrufGP6hnnZkcQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ssRdDe2z2FwRhMsr5MDsevNF8GBVleVG0fn+a6yi8H2tCFF7xffsYV7ZOyOtfo/Bfpz/o9NtRoh+thxPsJgS0xplTnq3MpQyYGlEsp4B/+2JIEPeJ4NB6G4YZ/bUSSWIb2Pumhw2LjVtRlQaGUAYaBWx7yvoipt5WMHVv2ftHqANCtDRLIyX/S2Ow/R1EWYk331jgVF/CFuo8sXBijeVpNpMXsfIDCXStFwd+TmJOOpRdbwUgVoVze1suL+Dhdzyi96HIi4O1j7gMYO3RRWXRSj8neCOf1ebb3PIb0Aws1ZBhMEEucO4u1MxKDRA0VmYE1s+Lj8RljU6XrdKxHmEBb715MgNIA9VIAny0gkHGdrYcV2145oKwdQ/68c80Gxu5A3UPeA3qkQVx44Vwx54wYvNs8O2Q3gDstDmLucy077Icrqo7alwZfPW0iz60HtZWeILcSikO6x/oAnn1kHKczhXaqqxydKF7O3cDd/x32TTBuy5TGVO6VEaxSJuiaXSBxMdJbRCMubnyWaaJ24DswV8cgv68ymbjwXsU61TnNs8SCyfOM6QCVVYQJE8mTBEo/fKDZ1w9HId1Ix7MaO70w4NHwslNJ7Ozmr/NG6H6o02UnhmNCgk/V4tezAj8RRs9mrbU7RcSis4fdbm5Q72CnQBZNxfNsZFsO4SMvmKsAcSoAXF31RMi7DMfo1Jv2yNi+g/3JIou0NEy6hUQMGBzMVwajSsLxnMOa+PRmVcLQ1y9RLrebIz+hr6R5Yfd4MvFharCf7uhR2HzzXHVpfCMArjggCzjkbK1c/FeFmgKvutNEMhTha5jLcMEmYW1Sgqi8fN/Fgvl6cS3rQRYoK97r+NRQO35CHSa3TTCddVlWQOMBkseE99pWpuT1tVDBGuKQAd3mkZiI1CDbvf0I3IDHD/4nFRYCmG5+L/DSnsBdW7AGZlZ7h8fOOzxkV9qwc+aP3zYga6yQjggGGNcPx68EU/kiY5NGJSQmvZIeAPFSrFVu2NktpqNjQwm7SALkV+eYHgCu9SZLziL60Amdg//gA4GsEWxatZHoXYGRfHzIABWrDBjN4QlnO2wCzpLRPmQrG9c1ICMk/fQ5LTJns4ktasGrM9g8J/02SAABYtbsmyXAymtjJZqGKm7D3Qv8bnDO3NgrV6c61yJyfhIpr19NZ19TQwZXhEsBXkjGhMuB8F/Q4uxnO7aHsrDR7FWI9ZQSeGSlknu8dJk5e1XzeIzu4Ru+CLxet4WhGBaskicMWd/d9zfQoHdQnBPU3vjG4iGvyirGV6RvIX5h4vn3dh/xPO6ffAqDJMyw7dxLipL6fNi0xxR/7DLmvd3TXVjlLDt5rZC+bLLhapoouab0PVfX6sd2geAe/ERSVC0zDM10l9Bi0nlz/PYgfbFVTb1RJ6EEyuZAEDQ8VYmn56y5FwpGUt6e1rG5/A1K5lLRjEhnNgL49B82ceSW1fihGc5+2i56FQ4arhyHWTfG2tfWQl5l8dLfPyyK8JS3oKEjUKPxDK9ewEZk1Blw2f3EodArZYPAQ/jL9491PCi64PdsNZwCbycrGXfh9FPpr8ulAfLRUCddXvTjUDLIi0DgyzOBnb X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6779786e-a58d-490f-ac9f-08dc4e88e15c X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:18.4602 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: TtFEz5gmhHjRnKW/1Eq+qO+IhBlt49rheg3FR/pX/QSPZzo9+sNoxjc3MTxMZ3bN X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110843_780997_91DB7B47 X-CRM114-Status: GOOD ( 13.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Half the code was living in arm_smmu_domain_finalise_s1(), just move it here and take the values directly from the pgtbl_ops instead of storing copies. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Reviewed-by: Mostafa Saleh Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 47 ++++++++------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 -- 2 files changed, 18 insertions(+), 32 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index af5ebedf0f0beb..49e51bc1a5c788 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1313,15 +1313,25 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, struct arm_smmu_domain *smmu_domain) { struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; + const struct io_pgtable_cfg *pgtbl_cfg = + &io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg; + typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = + &pgtbl_cfg->arm_lpae_s1_cfg.tcr; memset(target, 0, sizeof(*target)); target->data[0] = cpu_to_le64( - cd->tcr | + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) | + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) | + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) | + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) | + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) | #ifdef __BIG_ENDIAN CTXDESC_CD_0_ENDI | #endif + CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_V | + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) | CTXDESC_CD_0_AA64 | (master->stall_enabled ? CTXDESC_CD_0_S : 0) | CTXDESC_CD_0_R | @@ -1329,9 +1339,9 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, CTXDESC_CD_0_ASET | FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) ); - - target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); - target->data[3] = cpu_to_le64(cd->mair); + target->data[1] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.ttbr & + CTXDESC_CD_1_TTB0_MASK); + target->data[3] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.mair); } void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid) @@ -2286,13 +2296,11 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) } static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, - struct arm_smmu_domain *smmu_domain, - struct io_pgtable_cfg *pgtbl_cfg) + struct arm_smmu_domain *smmu_domain) { int ret; u32 asid; struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; - typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr; refcount_set(&cd->refs, 1); @@ -2300,31 +2308,13 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, mutex_lock(&arm_smmu_asid_lock); ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); - if (ret) - goto out_unlock; - cd->asid = (u16)asid; - cd->ttbr = pgtbl_cfg->arm_lpae_s1_cfg.ttbr; - cd->tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) | - FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) | - FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) | - FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) | - FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) | - FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) | - CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; - cd->mair = pgtbl_cfg->arm_lpae_s1_cfg.mair; - - mutex_unlock(&arm_smmu_asid_lock); - return 0; - -out_unlock: mutex_unlock(&arm_smmu_asid_lock); return ret; } static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu, - struct arm_smmu_domain *smmu_domain, - struct io_pgtable_cfg *pgtbl_cfg) + struct arm_smmu_domain *smmu_domain) { int vmid; struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; @@ -2348,8 +2338,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain, struct io_pgtable_cfg pgtbl_cfg; struct io_pgtable_ops *pgtbl_ops; int (*finalise_stage_fn)(struct arm_smmu_device *smmu, - struct arm_smmu_domain *smmu_domain, - struct io_pgtable_cfg *pgtbl_cfg); + struct arm_smmu_domain *smmu_domain); /* Restrict the stage to what we can actually support */ if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1)) @@ -2392,7 +2381,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain, smmu_domain->domain.geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1; smmu_domain->domain.geometry.force_aperture = true; - ret = finalise_stage_fn(smmu, smmu_domain, &pgtbl_cfg); + ret = finalise_stage_fn(smmu, smmu_domain); if (ret < 0) { free_io_pgtable_ops(pgtbl_ops); return ret; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 5aefb0ee2b9bb7..7bafec4c0c2fac 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc { struct arm_smmu_ctx_desc { u16 asid; - u64 ttbr; - u64 tcr; - u64 mair; refcount_t refs; struct mm_struct *mm; From patchwork Wed Mar 27 18:08:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B22B7C47DD9 for ; Wed, 27 Mar 2024 18:13:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cBPzdUDDovTSgkavHU5TGa92CETfO/iHI/6e3EcxgHw=; b=1UZsljt4ryJ+Wz 70lMtRVygAZmUIV1QPE5n8MQmbx7YlH6heqp0w56Ad6CovsOpCx7Fu1aHKL7PDWZabB11l1/5A0aR hIeWyCHki2A8u5+ekpXXjrS5oL3IHUNZXtCdkdFvS5bnpc9Q51vhGw0esAWq7Rh+bpkpVhtE0n5JX lWBjnG3LI+EX2GlfvoVHRZVS0bbGyyNxQPV0FVUdfnBT/bFOsYZpeGa0kpQVphPhpmjFghhCSy+iX TyujjxDwkWSrhVZ/yEZLb1axwCMfj3xhRapI3olE4ChRssGnVd7M+CpejB3UcefoPduCUO0Rd5RaG wL7M0EMXq8nF5wiP2F3A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXm3-0000000AVsM-0eif; Wed, 27 Mar 2024 18:13:19 +0000 Received: from mail-dm6nam10on20604.outbound.protection.outlook.com ([2a01:111:f400:7e88::604] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhf-0000000ATY9-1qXf for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:51 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NdHozNGdlY22BN9RreCR+Zbsh6kb8tJBIh68C0o+NWccMnbZfvPk+TbY9KLVDQ7W0suZUOL5jKz5B92oTpnWKkkQTXWRM4RniSpo/bjgX8db11YLlVOOU0xeEK6PKDGSUncK25gR2XWS/wbp7vstFJsSuyqsUWEorTFNu1KePB43PB8KgVjTNI2ZcuOj+9TcqqCkCn+MA/9bqR8g7J66J1sVHyWnd1ssaPWvTeeS3DejejoikMxmeFIvg8t0SSqWgY8lPgionXsOmwmCLFl0NtnaD1viI3E4qJtjW4hzsgAm4v7UH/liD4xX4MfSCJoxoCoabwh6IapjF/JxNgr92g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HiaHvS0G1kgtkjpC/cUsflX72SZnzdC3t/Ue1rEK3HI=; b=Oydfn4LGoTXeOcAZZbGopXZTlHEoUqcxchcvVy7d8ZlC+EUlcqMc1/ZGX0rR8O3KEoDGJ76npSPHF+/QU3uiUKyq7hg5ikOgqVKP1wnYQPX4NhxasmIv19CP1h+BxepsZeQn7s5MxmSKdAW6r0I0IMcffrMk+B+ZqB52faqmbwCfW0skbGkBIH+tfsUV8rbaMFVev87BhD6xBFkNBseP5UWmecJigheJopN2UaOGwu1007ZRoPXokKa1Tj7ruoMbxX8j42qmh9AYGe4x3PnpX5lTL2i2NOiqIutTsq8eLBBtVXF2CmwMf2N9BbwlOOe2VFN8MXiF1h65B6bjCYoNcw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HiaHvS0G1kgtkjpC/cUsflX72SZnzdC3t/Ue1rEK3HI=; b=tg7mDJ63rbNkowGt/27DNcujwu6o2eLOm5L6q5AEWrbXDiXnb/RT7mfG8G6mQLAzmL1spIXkrsjHfeo7JjhrKqlZVs6XFHQ3T1X9qwClHMK/7x3MEStwN7X8vsUThpcg0ZFvyc+DJCAPEZQZgT1YYGNKblt7NCNkUWJ7SGvchn7rndKiYvJQXynC1oThjTIMJvBkgDr9Mw5epIMAFvuI+n0Lh3/MP4hZfneMYWC7Oo3OTbnhIvlGvSuvyK0os2bdfbXY3kdetumJxgmJQfYWBgU6IHK+DvBUyVzMenBhUDKsbROSK480lUkTMryv2gOkNTJwI5Dtnw6ZmyinkVMVZQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:29 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:29 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 14/29] iommu/arm-smmu-v3: Start building a generic PASID layer Date: Wed, 27 Mar 2024 15:08:00 -0300 Message-ID: <14-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0048.namprd05.prod.outlook.com (2603:10b6:208:236::17) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 1f8606a7-d646-43fb-ad11-08dc4e88e187 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YcuuQBvjd6MXDTWQanWDLcaK1GmQnf85Ga7VtJKdrs/Oyceg8+f4qTUyQGDmDluByZKoOZwWgwiEd7AXPnGSaHi3ITvtzNhJIMSjVkCmT+WfQicv1WAdMg1kaXHJXORmI+W/8JNBSO3i2IQYtnMUu1dSEiO83LIlfNJJPlasztHLbbmC8CUssJgm0HqyMB/Loo9OkL273FUZoWyM8YfYFj04jBzVBprlZdQiuTJuiZfHAFHZYLcOWv4vhjixrueIPYBfA1AMyC24VPPJnioId2fOrKV7iJm/GuqZGzxQHrQZ8DiedVFa8HvnVHowIn4MwKMHssbCnvk6qvQnmZ2robxuuY/HwqLS27ToJk/57nkw0AirO4vtPtAghE0dYL+y48J2vnGWNQQCUcfP1ebO0BLHyTegWd1T9R+rmpZcOq8sdGaTJ1WXMuvnEJ4NtcocCYbK8Ui4G83TOI8m8eu1RrhblO1vkm9kz98mMrDnQgOtVW4bMVC9yzUDdHlPlDaiDpyPgbwhtDyXlqebPTR3/0IyuGsFwk7LdQpLEwvqTADM6v4ZfIW4J9sMVoI7DLWIUNpwb6eNa4GPq4JPkuEST8SYKGEBX1XxsTrGeiX9zNmtTBJhrHRChrfQtCV9fmKGhGABprcxVj5BShlmOGsoLV0Bndgm+ubdNMfZVJYtmnc= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: TPP62VnRlTHDqluNFff1WqILyMM4RSyLZ7upAiaBkYav1kpLL1oCVIRQmKOb1VwY75iqEVGTLNRugkW4W6XhxJ++YaS0dwFZFp+YL1Hr18W/nRSG6+rvwrRJZxhuz3s29Rqh2YpQ4YQTfohuBHSGOjfhyWSx+x5JQ6WMFWrsa5gto9RouS2d7wIIr+XhLIfMlAazq9kL4cHs6uzxYZhQmYEO4r8rDKb0loHeusZe9kqBbAZzTsXJym4ZErXBFy/oDgFKqSQqQnGNWjpxpkV6cGisY7xRjdY6hfBHCNKZhkqR9b9X1jji2+qNR2ScQHGrXB+JSt0PQYOlcDaA7RM6WuScjPm2kkp2715kkrbRGIlkkD7jAuxpp++CUJ1QexqGKLa8C8+/pTNrlUaQl8BANjH5zfc9S/7h/6fNHQ7sTTL6r+UC+/lVY9N7Dvq7Z7VXwCWWsa2T3OOPjtMatS+dCLXJkmj/1WM74FigTIJFp9KLal/sgGu2yhNwso2XHLpjlI1qZsKq1fgSpkeDaXX9mYdePs5C1L8Zk/u7qz3BTehRouOOGYvk05dFsQKYTx/OCUtMrtXL/5etxQF1ztaDX1JCeNWOLOf3AAfrMhPjQ9HJUl550awDg801wrWM8oCKsmYxahOzrntG7l0OpR3sNPw3XRqDm6ZQaSGGn1O05mgGUQBfJvI/x82kmswQvXlQoQ5Fg0WvkOE8BRjKMMRwwp7TTDjf81/hT7yYa8n6xGeAIltxmUJcwf0lvVsBkhL0hkXg9Yiyh93PekIz3Ra373n1fRGrAHRh7THDpz+bOHAHE22jTmeRsvCVDMCupesiLyNx27pBX2BlbwhHqWzPjUWStpPDFJSMfxpFd5MaANhoXW1UImVl27+vWNiYgBfSUI5lM3YvL+a4DlfeOg5FCoj1MzdOCygoP1j7AbDJl8mS1aUun6o566V72eOH2wymkGhvbT4seohlzObx3G8ISpw5Y/Nby2S8Hr9rN3HGu5rJkMhJ0XhjvkQ+X5MTwplJPvIr8NfBset8/FBPT0qfq6n0duxnRImCKvsQd8OH+DcvwYuZpIQJTzSLKHz8QdoysW9IegwhCawvclSnMb860eCcMMjvqA6Vsnv2VS01h+imZQLngzib0jjj96HWeEKHe1CpQZyW6dCYf+s3A/VlXEgn9kc1aE6DyrM8OjJdgKMIJU6Px/TmRPPO72Ahn37565g7vKBpW7hiZuTpJM+NdBRHEUA8gTLGJjL7kUhkCY1FY+pQ0nC80/cDlX4Ac2MtDq9sfI7LPaF0OmuC4e7l3b1Y6PVQSsdaiNR8cdWORAKpRvBAj7U/TzPbnIgc18JNqL1H4N4wC8m+VjJf1MeWuZJb4GpmEp0k09AbsQEjUmAt/YBwJAWFTSFlMPZ6y1s5I+XGJkpOeQANAQJU0JOSi31AvXvGqP/F8mZ2SU40ECgGNeMdA1XlC68miFoFdwceH2+T3N2lJxSXLejhAD4xk5BGjKTnD1ugpJTFK6eKuL1ZoYVezmYiVIFp66HDRWCUVr0kOlg4lyLoSqFhzsE383FyHcDF5TW/w/w51Fkn76dpjfLlxYwUXw43oUa/vWsC X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1f8606a7-d646-43fb-ad11-08dc4e88e187 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:18.7987 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: h+eioKAL+OG2/7y30XIvOKq54EWunIOZE3R3zwMZn0i+7+lRPQc5lowR5XyhMevp X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110847_869308_B478E384 X-CRM114-Status: GOOD ( 20.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add arm_smmu_set_pasid()/arm_smmu_remove_pasid() which are to be used by callers that already constructed the arm_smmu_cd they wish to program. These functions will encapsulate the shared logic to setup a CD entry that will be shared by SVA and S1 domain cases. Prior fixes had already moved most of this logic up into __arm_smmu_sva_bind(), move it to it's final home. Following patches will relieve some of the remaining SVA restrictions: - The RID domain is a S1 domain and has already setup the STE to point to the CD table - The programmed PASID is the mm_get_enqcmd_pasid() - Nothing changes while SVA is running (sva_enable) SVA invalidation will still iterate over the S1 domain's master list, later patches will resolve that. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 57 ++++++++++--------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 32 ++++++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 9 ++- 3 files changed, 67 insertions(+), 31 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 80a7d559ef2d3f..095d11df2a1966 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -415,29 +415,27 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) arm_smmu_free_shared_cd(cd); } -static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid, - struct mm_struct *mm) +static struct arm_smmu_bond *__arm_smmu_sva_bind(struct device *dev, + struct mm_struct *mm) { int ret; - struct arm_smmu_cd target; - struct arm_smmu_cd *cdptr; struct arm_smmu_bond *bond; struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct iommu_domain *domain = iommu_get_domain_for_dev(dev); struct arm_smmu_domain *smmu_domain; if (!(domain->type & __IOMMU_DOMAIN_PAGING)) - return -ENODEV; + return ERR_PTR(-ENODEV); smmu_domain = to_smmu_domain(domain); if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1) - return -ENODEV; + return ERR_PTR(-ENODEV); if (!master || !master->sva_enabled) - return -ENODEV; + return ERR_PTR(-ENODEV); bond = kzalloc(sizeof(*bond), GFP_KERNEL); if (!bond) - return -ENOMEM; + return ERR_PTR(-ENOMEM); bond->mm = mm; @@ -447,22 +445,12 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid, goto err_free_bond; } - cdptr = arm_smmu_alloc_cd_ptr(master, mm_get_enqcmd_pasid(mm)); - if (!cdptr) { - ret = -ENOMEM; - goto err_put_notifier; - } - arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid); - arm_smmu_write_cd_entry(master, pasid, cdptr, &target); - list_add(&bond->list, &master->bonds); - return 0; + return bond; -err_put_notifier: - arm_smmu_mmu_notifier_put(bond->smmu_mn); err_free_bond: kfree(bond); - return ret; + return ERR_PTR(ret); } bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) @@ -609,10 +597,9 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, struct arm_smmu_bond *bond = NULL, *t; struct arm_smmu_master *master = dev_iommu_priv_get(dev); + arm_smmu_remove_pasid(master, to_smmu_domain(domain), id); + mutex_lock(&sva_lock); - - arm_smmu_clear_cd(master, id); - list_for_each_entry(t, &master->bonds, list) { if (t->mm == mm) { bond = t; @@ -631,17 +618,33 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t id) { - int ret = 0; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct mm_struct *mm = domain->mm; + struct arm_smmu_bond *bond; + struct arm_smmu_cd target; + int ret; if (mm_get_enqcmd_pasid(mm) != id) return -EINVAL; mutex_lock(&sva_lock); - ret = __arm_smmu_sva_bind(dev, id, mm); - mutex_unlock(&sva_lock); + bond = __arm_smmu_sva_bind(dev, mm); + if (IS_ERR(bond)) { + mutex_unlock(&sva_lock); + return PTR_ERR(bond); + } - return ret; + arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid); + ret = arm_smmu_set_pasid(master, NULL, id, &target); + if (ret) { + list_del(&bond->list); + arm_smmu_mmu_notifier_put(bond->smmu_mn); + kfree(bond); + mutex_unlock(&sva_lock); + return ret; + } + mutex_unlock(&sva_lock); + return 0; } static void arm_smmu_sva_domain_free(struct iommu_domain *domain) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 49e51bc1a5c788..3922478799e130 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1223,8 +1223,8 @@ struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, return &l1_desc->l2ptr[ssid % CTXDESC_L2_ENTRIES]; } -struct arm_smmu_cd *arm_smmu_alloc_cd_ptr(struct arm_smmu_master *master, - u32 ssid) +static struct arm_smmu_cd *arm_smmu_alloc_cd_ptr(struct arm_smmu_master *master, + u32 ssid) { struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; struct arm_smmu_device *smmu = master->smmu; @@ -2417,6 +2417,10 @@ static void arm_smmu_install_ste_for_dev(struct arm_smmu_master *master, int i, j; struct arm_smmu_device *smmu = master->smmu; + master->cd_table.in_ste = + FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(target->data[0])) == + STRTAB_STE_0_CFG_S1_TRANS; + for (i = 0; i < master->num_streams; ++i) { u32 sid = master->streams[i].id; struct arm_smmu_ste *step = @@ -2637,6 +2641,30 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return 0; } +int arm_smmu_set_pasid(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain, ioasid_t pasid, + const struct arm_smmu_cd *cd) +{ + struct arm_smmu_cd *cdptr; + + /* The core code validates pasid */ + + if (!master->cd_table.in_ste) + return -ENODEV; + + cdptr = arm_smmu_alloc_cd_ptr(master, pasid); + if (!cdptr) + return -ENOMEM; + arm_smmu_write_cd_entry(master, pasid, cdptr, cd); + return 0; +} + +void arm_smmu_remove_pasid(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain, ioasid_t pasid) +{ + arm_smmu_clear_cd(master, pasid); +} + static int arm_smmu_attach_dev_ste(struct device *dev, struct arm_smmu_ste *ste) { diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 7bafec4c0c2fac..a3b94b839ee927 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -602,6 +602,7 @@ struct arm_smmu_ctx_desc_cfg { dma_addr_t cdtab_dma; struct arm_smmu_l1_ctx_desc *l1_desc; unsigned int num_l1_ents; + u8 in_ste; u8 s1fmt; /* log2 of the maximum number of CDs supported by this table */ u8 s1cdmax; @@ -746,8 +747,6 @@ extern struct mutex arm_smmu_asid_lock; void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid); struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid); -struct arm_smmu_cd *arm_smmu_alloc_cd_ptr(struct arm_smmu_master *master, - u32 ssid); void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain); @@ -755,6 +754,12 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, struct arm_smmu_cd *cdptr, const struct arm_smmu_cd *target); +int arm_smmu_set_pasid(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain, ioasid_t pasid, + const struct arm_smmu_cd *cd); +void arm_smmu_remove_pasid(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain, ioasid_t pasid); + void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, From patchwork Wed Mar 27 18:08:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607330 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 30F9FC54E67 for ; Wed, 27 Mar 2024 19:18:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ARbc4ItSc1aA+usVAG9CL93Q+bTyCs3Hp2GLQfCLrHE=; b=bp+GfmDtJLObr6 kOpRJjwvWu2izuF4jnf8Dr5NbfLSGw+0jrQJDXzxv+hdno0LQ/iOj8wNDptPdlDJKrvx7RtRI7VMs xPPflhjWDzR1ZnuhjG2gK1QDZsD8i+/ZnoAxZYrsqZXi3zv24/LsiLVd8bXjEdGXr4Wgjy16kDOu9 ikeJhECuw2/xGajEQyfY3fp2yZe9u8uHAHeHzGNWq7FmOTWbEvj04Iq0romCvQio3q6aw+gPXc+77 wZ4d8wqg0vdDuRY934+Bn5V5zCLLxYq9en23eM1cFHVUNnI+9yhpHDzESVviRPLY2LLFfRClEaW+s iBD8TH1wvIyTCMnJi2rw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn3-0000000Akwz-2Gap; Wed, 27 Mar 2024 19:18:25 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhX-0000000ATg8-2K9r for linux-arm-kernel@bombadil.infradead.org; Wed, 27 Mar 2024 18:08:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=3Py1jAEACkdkxTX90xty5HkM5BMc74Fbtk/JZIyPMq0=; b=Yproxy50eCLISc4R6J9n3dKQ1j bR6tl6p6FT/AX/FGu31r+TNH5g1hGJRUqi0OLfGcX3JAEI4cClv2L8FV6nJaQunwEMMUNgO8D9eEA T1S4e/3HC05UHg4epArW6v0szPAwcI7Qe/Tz0pLrXW3Zd/w+btZuKH7bnHpXbbg3BTIHrOQcgwAwt a0wWMu1aUqHa2x7DX297v+ccom6DmRpSytq3BmVKIxETOFNu4izZH4pf+ZRtnkGU4wgLSTrj5ch17 inebCe7esEGSGNMxp6JdBEfxo76OGXCSdnuDeYvNP3/1QDWox08CX3IP+ezl482c9/cSZruQgJH4U R/xrDoZA==; Received: from mail-dm6nam10on2062e.outbound.protection.outlook.com ([2a01:111:f400:7e88::62e] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhU-00000000RvW-1T99 for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:38 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hSgshfu3cyu4lNKFO/KrTLeajXumiQKvVuyC+gcvGc1CuvxNfq6c0ID/pS9W38sg7jz9jZiu1epwLynkg5hvh2mkz2VtmaUJ3Idn5rDY7IMgOpQllmct/5XCrhz9ivCXjO8EzjQ3BQs/EFbyxG6Lo3EYjGyRSv7Qb3krb2yygQyFEgf9QEXkPNMphVauzqQ3I2F8DoK9TSgcKDMABc3zBqf0lA9ik6ix1jc5quTU/fQOHmbaTssKjmUyeKRnhhQDqDezn/spdKSsUNGbMdflb2sAQedJ0JWXa3lCIlNk7Yw1MNmumkhSmlAAHNqso044KLFON+occhMsMgY5Dy81yw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3Py1jAEACkdkxTX90xty5HkM5BMc74Fbtk/JZIyPMq0=; b=ZP+ukkamRW5MOSCzsOR8URrqeGJmi+c5nBiChvrjeaI8LcARPake/4LRIaiQcIMVLpkTMeOshcKgZtbJMTJyhBWDndTaNKmSbKi4KPGJUscqA9iFUQs2XgvjEUNDENj3rL8wpb0r2FtFL0wMf0+HZF++cGtgL9PQUJFA0VI83e+8qTVMxatLosel2LqnEl8jtdRvp02hN2IQtPyUpfDvx+aQrmUcP+r1yUXgYUe6i/6TO44RN+wIkxn7ozLsio95dsXdVfhWmLvhsq4HNGBvKczH8r+5Q+8G0WuWakXMHfCaXPbM7flKalq7HLC0/MWuoWYt7zkLqA7hEzsJ9m22UQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3Py1jAEACkdkxTX90xty5HkM5BMc74Fbtk/JZIyPMq0=; b=KpfP7sPCY7Du8bBIESh8xc9uL2Kcoh+BNmX4L+eq5u5unjElssc2d24SiIaLlnriW049bURnEAIXCUEd7IfisoF9AkhMsMaPFzSYoMdAowqjGCu66AP1NqsFbxqrEFW9heqqoz1dILximtN1Av6LKb0JQk8bi967levDvzGiGi/2Wm2qsOgBmYeRWngtjwXQZrzrEXnGCz14CmwIVdJBDIQmt6VlvL23ovUpLw1R/eFKzO56VUP6y9zLeegCxM0FZRvnKaYeN2JvmO6qshqsqLuNErvMEJCwemdcQ1xxamuHwoUA7g2GrsMoKuZ3QC/sx2lacM8ttByfMZk+klUalw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:20 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:20 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 15/29] iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list Date: Wed, 27 Mar 2024 15:08:01 -0300 Message-ID: <15-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR20CA0012.namprd20.prod.outlook.com (2603:10b6:208:e8::25) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: f43e7238-d7a6-4299-f8db-08dc4e88e114 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ol52iCtuJ2na/I44b4/DAmeYU5mWvEftrNom8elUliT+M1UqoHW1XbpB7x7NgoSuQ34d2npKHC6cwRpjotTlVk/Baq2JoQe7LqCUHmgpU+qPdn2zUuQUprzx14V7cX5qAk/h5uL3XAzbt8W6XqKtldxmd86vryRdSfPe4anXsFhqVztB1H3vGzu6bLlfKbPx+e73QvrSoB/fQJdO3rN3DH96DdGKOuIZcryvMIFbJPhbGZj8asep5sT9oawuV0E6uxp8iMDN3V0WPLZUmLsyaxDgqAvxvAoQ/ZrSV2KsiIwkRuEl9BmK3eBHf5+RFATkft9QXpGg+ULIeP6OAPP1wHW5Swt3DGH8sxq4abGPZCttDATBtMddBlqCG8hymV7eQzJxK14sAV01F0n/Ds6IsF7HQD18OQnQAOed69BDoXgVXhZWCeQGTPNB+y8PaegjWopK+sivtF40T9VuVZ32C49cfmoL7r9SntlZyXyUhpUmLrsE0NPI7ODKnpkNwJ01ntqxPywiVwSrDcB9zbbslO5AdwcLgxpDFI4rAotJJ+cuGhB/JNyJzR8BXP/nOUpAsb41zQEKiHh1x2HqAfj/eN6VvWvVdO/cSNLyisjtih7KmCcP8amHbXWKeFTJA5MzRs2QhdFe13MoTJ442TZ4ydadp4azT9Zl3F0aqmN/vN8= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: szpPRRZcVAPmqJhiPUH7RQY2pXqidOSv/u8OKnzMdd35Nrz7Q5x1vetvIWTRls6PLUB5hcbe1JHhcDcgi6KzKrOyE6IE4Q2g6kZvvX5XfrBSi4OyndYxvItAq26nIGN1dgUw2UBktse2bhnzCkN9EQPV6VI/GGYyeWbVE8lfkCyrS5PKsdrhR1+qe6TN5nj0CsH7xqt5tzdl/nWzcziuIZvNFwcACK0tNq1EZphqZLrALH/GMUOhXERNfaCLgDi9/JrGitHulTbI2evM4rNvbRyqGoSwjT9hLXGqIAxCT9kTU6f7bvVPvP5vGJmhoCVtRAPRD1JAIXr+W9wsv5Akr4Q6h+25GUkZLR8855eavEi1VgiY6lOMQiBtBJx2XtSoBRal+ca9nLmCA2geC9RnsN4sWhc17nD0v5E40sBdIe8mYA1rjGfZO/PZgybkMfaNoIqvMGzC27AkSaUcuVOZnhSzRO7+W8bO2tqNCcOqn1m6S+VmLtRvRbkTSjJ8bGmnzcY9eG9+FnItCkOfUETbp2oqCLX3NfmA6hchdTSfD92M1Zmiv3I+1y+QZcIo/Lk5isj7nIopVLWm3jUXQ9JpubKbjY1mYhTI3rqQ2BWXBh67dg14E+uyOo3fIvIjf0VR3ZWJVTMhrP057BoHi+0bd3SmNtow+rFj69nTaC6zAfWXqjZidl4AZ9cpAitE3mIP5RI2NDwwNLTQGufHMeJEoCSjwf7wAKprXJGReo50c9UaHy1K9TZNhpkyv165yNjbeUbEHishZUJLc9ivBvQrarrUNV6sWnQKRbdM1JzINj3VeNo8AaiFujn2OqNgXeHyeNLmxq95IQUrsl0P1QLcUdbFLdV2o+6T5XkoGHsrVo8ppyLwD3+MpSLPx1+0cpH35v3GI1yeiHmagq41zp96kreo8O77OF5aU8yAvqzqkpizGMT4EFMkMU+gYECUIMgwq/n5+q5s5qtMHzMNuw2aXXIhVZG6dmZYiew5HnPxJKQg3tluSCH0y+mbEQV7N7T+M5Oz4NEm1T2/YVIOY2CtR0SwoGKwaF5j/3yhek2trFqQzOAa4V/gko4zruHi6vK3oFAHBkdRCPmsQ02p01inlIJADEUp1jkJ9MLl5IplcWgIUsnccdmD5Q039bJsux6gn8dd8O/M5FW6GdAMrbOg2DZFf12PxTrbogXAikRf9OSnYKqBbTy5XbKWpIUUHdmQZi4llc7Oua8jIhNgt8HMQGaqspZvf/KFauoJZmeUsWHlu4SJAmn88Hz1kFjOAy1xq3gE1ektm9WUmp9JKWAQyQbxUhfZ45usQzHaQwlqcFwfqfWjxsDUeJArVBUyQpYgO9zAEuzTaNhKTetdpM36x9za7ZGkUevbuZ06MdlpbDbMtxq07MUFYpYzUlVi6i+lXxm0ItissZNQLyEsWkHs/p49iYbp6kX7L818R+lVh+gNL4Wuj/KzMzw272L7b3CTdNnRPY+HPb8rWuWVjJMXr3Ee/GYam/qT3s3U/vUwOUET+nco1/xXpcYROyvvMGzb8zfrZz1Pme5hzuMZmFAiKIKRskCQFDhULnVa9PN2Y1HeBQvvnI31ep8i/aFCZkay X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f43e7238-d7a6-4299-f8db-08dc4e88e114 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:18.0235 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: raNFUZaalx2z1YoD6sU79TRzmVelm1Xh6K8PzDdcBHFN5cR/rjI4el+UG+FYvm6b X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_180836_595316_DB3244E0 X-CRM114-Status: GOOD ( 16.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The next patch will need to store the same master twice (with different SSIDs), so allocate memory for each list element. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 ++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 39 ++++++++++++++++--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 +++- 3 files changed, 47 insertions(+), 10 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 095d11df2a1966..63f98264b646bc 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -37,12 +37,13 @@ static DEFINE_MUTEX(sva_lock); static void arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) { - struct arm_smmu_master *master; + struct arm_smmu_master_domain *master_domain; struct arm_smmu_cd target_cd; unsigned long flags; spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { + list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) { + struct arm_smmu_master *master = master_domain->master; struct arm_smmu_cd *cdptr; /* S1 domains only support RID attachment right now */ @@ -299,7 +300,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); struct arm_smmu_domain *smmu_domain = smmu_mn->domain; - struct arm_smmu_master *master; + struct arm_smmu_master_domain *master_domain; unsigned long flags; mutex_lock(&sva_lock); @@ -313,7 +314,9 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) * but disable translation. */ spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { + list_for_each_entry(master_domain, &smmu_domain->devices, + devices_elm) { + struct arm_smmu_master *master = master_domain->master; struct arm_smmu_cd target; struct arm_smmu_cd *cdptr; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 3922478799e130..4411706019f036 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2012,10 +2012,10 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, unsigned long iova, size_t size) { + struct arm_smmu_master_domain *master_domain; int i; unsigned long flags; struct arm_smmu_cmdq_ent cmd; - struct arm_smmu_master *master; struct arm_smmu_cmdq_batch cmds; if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_ATS)) @@ -2043,7 +2043,10 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, cmds.num = 0; spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { + list_for_each_entry(master_domain, &smmu_domain->devices, + devices_elm) { + struct arm_smmu_master *master = master_domain->master; + if (!master->ats_enabled) continue; @@ -2539,9 +2542,26 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master) pci_disable_pasid(pdev); } +static struct arm_smmu_master_domain * +arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain, + struct arm_smmu_master *master) +{ + struct arm_smmu_master_domain *master_domain; + + lockdep_assert_held(&smmu_domain->devices_lock); + + list_for_each_entry(master_domain, &smmu_domain->devices, + devices_elm) { + if (master_domain->master == master) + return master_domain; + } + return NULL; +} + static void arm_smmu_detach_dev(struct arm_smmu_master *master) { struct iommu_domain *domain = iommu_get_domain_for_dev(master->dev); + struct arm_smmu_master_domain *master_domain; struct arm_smmu_domain *smmu_domain; unsigned long flags; @@ -2552,7 +2572,11 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master) arm_smmu_disable_ats(master, smmu_domain); spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_del_init(&master->domain_head); + master_domain = arm_smmu_find_master_domain(smmu_domain, master); + if (master_domain) { + list_del(&master_domain->devices_elm); + kfree(master_domain); + } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); master->ats_enabled = false; @@ -2566,6 +2590,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); struct arm_smmu_device *smmu; struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + struct arm_smmu_master_domain *master_domain; struct arm_smmu_master *master; struct arm_smmu_cd *cdptr; @@ -2602,6 +2627,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return -ENOMEM; } + master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL); + if (!master_domain) + return -ENOMEM; + master_domain->master = master; + /* * Prevent arm_smmu_share_asid() from trying to change the ASID * of either the old or new domain while we are working on it. @@ -2615,7 +2645,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) master->ats_enabled = arm_smmu_ats_supported(master); spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_add(&master->domain_head, &smmu_domain->devices); + list_add(&master_domain->devices_elm, &smmu_domain->devices); spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); switch (smmu_domain->stage) { @@ -2929,7 +2959,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) master->dev = dev; master->smmu = smmu; INIT_LIST_HEAD(&master->bonds); - INIT_LIST_HEAD(&master->domain_head); dev_iommu_priv_set(dev, master); ret = arm_smmu_insert_master(smmu, master); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index a3b94b839ee927..e4043e48a6e20d 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -696,7 +696,6 @@ struct arm_smmu_stream { struct arm_smmu_master { struct arm_smmu_device *smmu; struct device *dev; - struct list_head domain_head; struct arm_smmu_stream *streams; /* Locked by the iommu core using the group mutex */ struct arm_smmu_ctx_desc_cfg cd_table; @@ -730,12 +729,18 @@ struct arm_smmu_domain { struct iommu_domain domain; + /* List of struct arm_smmu_master_domain */ struct list_head devices; spinlock_t devices_lock; struct list_head mmu_notifiers; }; +struct arm_smmu_master_domain { + struct list_head devices_elm; + struct arm_smmu_master *master; +}; + static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) { return container_of(dom, struct arm_smmu_domain, domain); From patchwork Wed Mar 27 18:08:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91E7DCD1283 for ; Wed, 27 Mar 2024 18:13:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GbakEl87JoXEOY+dbkdNEe303JBFHbD5695Kp8w0CPc=; b=PtzoKVgnVnGbSO 7e2WEKgf4HOLrLRt/ND7SSvT5AYX8cgCp0unb45pfGoW+0Zds5qtFqG7NFFizWwddzyFC7LUG/jrI Io4T8uMJbQDMZekn1rmf277196dReMIYrXks6T0HnLVJGJUn4xeNnHjpTs8Ee91GmUen/KoWZ4ozB lOKi1e72nCjrY2nXB8NTKuPUVkbpemq9hTdMLtVG3q3YpAuuLJYx08i/ckcYh8T+ubygu1Sf8OxLV PfgaaBAA5nT7Hvqi9HbrqMNWzKRI4InXnkhp4w0K2dMYf2Sbcg3J3xblcs+6f18QHDC5JOxVili1l 63OqBAMjIaE2BKQvMZFw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXlr-0000000AViC-1fnd; Wed, 27 Mar 2024 18:13:07 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhf-0000000ATmU-0ELx for linux-arm-kernel@bombadil.infradead.org; Wed, 27 Mar 2024 18:08:47 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=3CIYtob+B1PrjFQ8McGfXGPUAIqLFU8bt9emLD10p/0=; b=h3DO9TZAeXSqbZxh7weQJBSDK0 Hs0iUPtl/3uIeuzb6YUeMhXylIEbn6XPqKUuZl+c/Szbmg8ZQ+QyzYOJyUEgOAUzZJrLU2BWlqIrn lEGcjzDejHm+lLFFQE1BAYcIOHRh1OnEy7r/ObnKTE1XcY7+83KcQTsU8mLzC7d9CyHdI5aOBq0Qv 8suYD3Wdz5OYvQvIBye4b50QKwebdciwYFhnnF3MX6Sbi8VmKEiNdDoTgME9LgMb7L6MTFYskvltY iap1dIDLb6wdKab7vw/cUmGrFmCZJsh0BUyZAFe5azWjAmrkYmnAMD6HOwEjJSenrvVmI8FDmdM1N EZaN86/w==; Received: from mail-dm6nam10on2062e.outbound.protection.outlook.com ([2a01:111:f400:7e88::62e] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhb-00000000RvW-2eiD for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:45 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MM2Ub9oxXp3bMswMENfaOtI0d/uaUDCqUdRQmxC8LsPfG74n3UP4QumvMz2MPqewo0t8FAm9T6559lkeTavPUDlhSSJ+wc5sLCyxtpzDAuitnUXyan7GyvEmsdH+GbIttV0UjUndt7cCNrK+IWbl1zrz5CTjKfy5/lmS6GNoLPXFiDMnvIQlXwKYPHnFyU93Nab/fUFqo7AX3SU46RGUTY2aFObYLKmvypuRhaJYVfrVW/m4/L8Ome0TCklAbI0YLvsN7B4e5ykWmgknimD0SCbwG0lvROjNGAygY0bOGhCXfqCupLCTyLiza8G/tZXj3rNujZaQzOCvLTG5c4sQRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3CIYtob+B1PrjFQ8McGfXGPUAIqLFU8bt9emLD10p/0=; b=h8LtzPZsj3M2aF1zspodiRVg2YXNu+LFF47bEkaR4VaWdF36oF9ZwPLpEqmTo3Eu9OQA6YguB9Vxfl62aEnPgEESJGc28aCVQVaEi8K4sC6JVJkQ1jgH0ENgqCZCeIflTqmg2srXxrdHDcr/r25mSeNIFjLWPZDwCtCKHalxGrF5z1vaaVGxtzXJUP91o4tyqtNDSgvmvryj/1zPwnZOJurLAhrveTo3PVrnNLFpF1Fur+MqUB3eUKCLekE2KzKxddcn9ttM6GSm7KhIjt39FNqGTZm8RnZ9/le3oBRjYlNqrxeZCIkipsBZZQVCHHOKc6yQhq/qXYxMDaqYxPoFNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3CIYtob+B1PrjFQ8McGfXGPUAIqLFU8bt9emLD10p/0=; b=SvDmpU2pW+tti65GbCe6ejQUVfQxHAUx+vF2S6y1q/w+xcLbSoxNrvTsPEsHExS/GxYY4oFOqNrbU62QSei2NW+ODpBrkEnPVO6JgK1UY/W9JHq2Lm4S2QVzLTstCNzyt2S7/5KYZH5XFHKPz4Bt9PDDKHlqunLGze6GusRbU41YbCon5AGGHsNgkqF9fQRdLbdt31mFrGx5Ef/p/sb4RLJ9J94RZN/86rXDRRPDOU2hhjmtaoEHYp2bOesH8f1HMcwTw9I0icyaUFyzur719QBx21m9wuyfLczfXrG1+jmpTJERT3R96w2HbPlwusx64k+ViIJNQL1LpkNxlPr/hw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:27 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:27 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 16/29] iommu/arm-smmu-v3: Make changing domains be hitless for ATS Date: Wed, 27 Mar 2024 15:08:02 -0300 Message-ID: <16-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR05CA0005.namprd05.prod.outlook.com (2603:10b6:208:91::15) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: ca4e096a-a6d6-4f8f-5aaa-08dc4e88e162 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4Igbtk3ZzGHsb6CvAUGzzTadCknsQ/c6JnBSzYw7e11k+QE0tKC3jDJ/BR5Cm0QSyCXWrrCsRImL2rLCExqqpj7GTm0TZ/nsw3SVjtS3rbM0reH9mzpdFfhaKZYG7FVS0qNrxGBCOj4PUVgIPCbyrnmkMsl+zym7toavX7ehl4HZkMQg+3OVQANTVAxVE7kWuvUmLF0jdBywH8dG4ABW0qsyUnnXpmVzMNqmhAHjyvy71G5dUr47VBatzqgbCvlMTkCZBc1Fqh0SNu+bol7VNxA36Dw6wG5RdB4dkA3FJQs8GChAoTEqdOHUsc7VBdc6jYhXQ85meh9pR4GOu0jQeMv8gd95BmB+s3BAYmK4OoSsCYC2nSca3s+R4JyIB7hN7AAdgcdlKHdS9J2s4X/KeM5Wthpkc94T/41yFVnMBjnNQwreADGKlbkwIXmpb9aArW5qu/BjJbM55kIzyBATfo6ztmHwCcsXZhBlS9bhDtTiIsLZVC+ap7GzxhxTeoGJkNkYyvhfZsCrMPTde2GI3+z8eJbPzfCIAsicrWOty1sQo/kqdPfMQhEiR4SHL/uAg6s02X3X2/vpWU9aJPKB1szImAghrdq37mKQ+ZXjYvWFzPaFiu6DNxDt1j88jHjzqnxHmWz8xsOqMOfljWCyOpbRk2rN3aVSPwWibuFM58w= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: niVM+lN3im6bCxlJo3px+14IaPYoAZ+47zTPE84GQ+iomXBOV82XrR3+CI7O5w1kqfpbFDtgln08990vkNvJ1s0uvnUbc9q9gwX0zwxUW9OR2FC/3ph+AJV0zLuFG7afbNCX5nuNbNWtiIjb1ilaCCsLYWJumjPKvpwKJvJWYzySzHQPFuOp9vxQ9YEdblLWr2YHfc/90g8boiNLWRoUtzJgVkqZ01j6gOp7HvHS1iIYOlJ7Y14mf9civRZCPkiAlrzAjDB3QJAmKv5JFRJFCUUC7Ijr71CvAIVRneGgVAfWE7E+8bEBr8e+avFWWMjynVgC9t6d8c21LiXUhwxAaKxIc6rAgEAuRNqcxa57r1xZK66GpnGE6yD3Md+Y2zF+jhpz5F87hP5IDPjCH8zl/vugIkcPB557pgu00GnqB/Y5y7OzEhZW3vg0AmFOsxF4cIpttpWJ/dOZ4yOn5oYb9BXVQLsqsmRAM23hKPWnJndjLE9SA3aCGU8wGLuRPR5ml5JrotRnM7wthpoXmKozt3mLVCmXl2kkdgvWVR0OOlrUWaHaBTqY31pUv5OMu/3c7eaGixbWlXIBMk4NY3/4FfhxPZzs8bRBO22rMxIqMLCs1My4MmW1XFXSVHtDYSqKh/xdl4e5uMHcLcB+CWRiueZfCRcA08lnQUnD/sTSKA0Fshurmz9/ONQT+TY8samo3f1CKr8hLRZB8JjG8akQ6puYbpeeg2wivXFfeU7Fcu5PDkTjj/wTXc16pLRnKISvddPgk2qOBgkqdxzLQkaZLvNABW8HKbersSO1F+4zizqHNkVlbeeYHvuFf8TAGKSv/LMYt93k+1vV0MMAbgrUU20QikGaCns9NwU52YxbKWAeRoXz2iGwigyvbMuy9aiHoGfX9J1eHi96rb6BOaHLJlbfavFGXaEiO/A1tGKVtIwIjvPzP10MUy72Go0RlAquEv1Nmt/fJA2tSRtlZ4LWqm6YX3FNpxzORdXfJzRONeGOj67/nTQY4C9xJv1YI54vp5UBZpc8e+fidPwMgHzCDW33ZhIzUtDFrZjHhKfkdCnIdtbAeo7sjIZpcZvOL1vx5bMOmRojJ28N/HcnR+fkp2JccGZw48UH03fTUY8k9CVW2gVitZ3aOpp5WDNaiCckhjCD7gIDdyXnOV+QCwhzyw+e36qMsUMwxi9+UFVAeh4v4DUrLwj6cybTncMPFc+1NkSwq+WvkFFGk69D8KCKQohiZbMjig0fNlHed1JjbKZJ7dTODo6QFLqsRWWhurfWxURqSDdREfKYcxA2WjHHjyf+CiUizozTMxMBBt/LbruLDnfvD3fEqqR7xMWkTUYbaLb6gli5kljuYPD+kN6a62V05o5TPuMiQYE9pARS3PIkDUU9iBFegXnfU9urNd5kwgFB1MVcnknLIdhP2QJqRYaX5umHdVx3fSW392+0TcUcDVVqKZYz6MHq9tSiVNuIWtgF53A3bhERl/hwGBJmXcNcxQKvJ0zRW+czK9EaACR+14b91ENEYx9rnIUodcETGe50iw6GLz/uXnj1JjREXrws7/pD2t2LbI1Acjcte+3mTBHKgK1xmjVGxAYmxok8 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ca4e096a-a6d6-4f8f-5aaa-08dc4e88e162 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:18.5474 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: BtXW3PwmZB1P04Upic9UpHyUqJnOP21lDfLo3akeZKyJ6mZpmxqsMPGbF2HJlwCd X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_180843_828619_FC0E35B8 X-CRM114-Status: GOOD ( 29.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The core code allows the domain to be changed on the fly without a forced stop in BLOCKED/IDENTITY. In this flow the driver should just continually maintain the ATS with no change while the STE is updated. ATS relies on a linked list smmu_domain->devices to keep track of which masters have the domain programmed, but this list is also used by arm_smmu_share_asid(), unrelated to ats. Create two new functions to encapsulate this combined logic: arm_smmu_attach_prepare() arm_smmu_attach_commit() The two functions can sequence both enabling ATS and disabling across the STE store. Have every update of the STE use this sequence. Installing a S1/S2 domain always enables the ATS if the PCIe device supports it. The enable flow is now ordered differently to allow it to be hitless: 1) Add the master to the new smmu_domain->devices list 2) Program the STE 3) Enable ATS at PCIe 4) Remove the master from the old smmu_domain This flow ensures that invalidations to either domain will generate an ATC invalidation to the device while the STE is being switched. Thus we don't need to turn off the ATS anymore for correctness. The disable flow is the reverse: 1) Disable ATS at PCIe 2) Program the STE 3) Invalidate the ATC 4) Remove the master from the old smmu_domain Move the nr_ats_masters adjustments to be close to the list manipulations. It is a count of the number of ATS enabled masters currently in the list. This is stricly before and after the STE/CD are revised, and done under the list's spin_lock. This is part of the bigger picture to allow changing the RID domain while a PASID is in use. If a SVA PASID is relying on ATS to function then changing the RID domain cannot just temporarily toggle ATS off without also wrecking the SVA PASID. The new infrastructure here is organized so that the PASID attach/detach flows will make use of it as well in following patches. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 202 ++++++++++++++------ 1 file changed, 144 insertions(+), 58 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 4411706019f036..fb18d9d500aeba 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1538,7 +1538,8 @@ static void arm_smmu_make_bypass_ste(struct arm_smmu_ste *target) } static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, - struct arm_smmu_master *master) + struct arm_smmu_master *master, + bool ats_enabled) { struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; struct arm_smmu_device *smmu = master->smmu; @@ -1561,7 +1562,7 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, STRTAB_STE_1_S1STALLD : 0) | FIELD_PREP(STRTAB_STE_1_EATS, - master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0)); + ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0)); if (smmu->features & ARM_SMMU_FEAT_E2H) { /* @@ -1589,7 +1590,8 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target, struct arm_smmu_master *master, - struct arm_smmu_domain *smmu_domain) + struct arm_smmu_domain *smmu_domain, + bool ats_enabled) { struct arm_smmu_s2_cfg *s2_cfg = &smmu_domain->s2_cfg; const struct io_pgtable_cfg *pgtbl_cfg = @@ -1605,7 +1607,7 @@ static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target, target->data[1] = cpu_to_le64( FIELD_PREP(STRTAB_STE_1_EATS, - master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) | + ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) | FIELD_PREP(STRTAB_STE_1_SHCFG, STRTAB_STE_1_SHCFG_INCOMING)); @@ -2455,22 +2457,16 @@ static bool arm_smmu_ats_supported(struct arm_smmu_master *master) return dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)); } -static void arm_smmu_enable_ats(struct arm_smmu_master *master, - struct arm_smmu_domain *smmu_domain) +static void arm_smmu_enable_ats(struct arm_smmu_master *master) { size_t stu; struct pci_dev *pdev; struct arm_smmu_device *smmu = master->smmu; - /* Don't enable ATS at the endpoint if it's not enabled in the STE */ - if (!master->ats_enabled) - return; - /* Smallest Translation Unit: log2 of the smallest supported granule */ stu = __ffs(smmu->pgsize_bitmap); pdev = to_pci_dev(master->dev); - atomic_inc(&smmu_domain->nr_ats_masters); /* * ATC invalidation of PASID 0 causes the entire ATC to be flushed. */ @@ -2479,22 +2475,6 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master, dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu); } -static void arm_smmu_disable_ats(struct arm_smmu_master *master, - struct arm_smmu_domain *smmu_domain) -{ - if (!master->ats_enabled) - return; - - pci_disable_ats(to_pci_dev(master->dev)); - /* - * Ensure ATS is disabled at the endpoint before we issue the - * ATC invalidation via the SMMU. - */ - wmb(); - arm_smmu_atc_inv_master(master); - atomic_dec(&smmu_domain->nr_ats_masters); -} - static int arm_smmu_enable_pasid(struct arm_smmu_master *master) { int ret; @@ -2558,39 +2538,147 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain, return NULL; } -static void arm_smmu_detach_dev(struct arm_smmu_master *master) +/* + * If the domain uses the smmu_domain->devices list return the arm_smmu_domain + * structure, otherwise NULL. These domains track attached devices so they can + * issue invalidations. + */ +static struct arm_smmu_domain * +to_smmu_domain_devices(struct iommu_domain *domain) { - struct iommu_domain *domain = iommu_get_domain_for_dev(master->dev); + /* The domain can be NULL only when processing the first attach */ + if (!domain) + return NULL; + if (domain->type & __IOMMU_DOMAIN_PAGING) + return to_smmu_domain(domain); + return NULL; +} + +static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, + struct iommu_domain *domain) +{ + struct arm_smmu_domain *smmu_domain = to_smmu_domain_devices(domain); struct arm_smmu_master_domain *master_domain; - struct arm_smmu_domain *smmu_domain; unsigned long flags; - if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING)) + if (!smmu_domain) return; - smmu_domain = to_smmu_domain(domain); - arm_smmu_disable_ats(master, smmu_domain); - spin_lock_irqsave(&smmu_domain->devices_lock, flags); master_domain = arm_smmu_find_master_domain(smmu_domain, master); if (master_domain) { list_del(&master_domain->devices_elm); kfree(master_domain); + if (master->ats_enabled) + atomic_dec(&smmu_domain->nr_ats_masters); } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); +} - master->ats_enabled = false; +struct attach_state { + bool want_ats; + bool disable_ats; + struct iommu_domain *old_domain; +}; + +/* + * Prepare to attach a domain to a master. If disable_ats is not set this will + * turn on ATS if supported. smmu_domain can be NULL if the domain being + * attached does not have a page table and does not require invalidation + * tracking. + */ +static int arm_smmu_attach_prepare(struct arm_smmu_master *master, + struct iommu_domain *domain, + struct attach_state *state) +{ + struct arm_smmu_domain *smmu_domain = + to_smmu_domain_devices(domain); + struct arm_smmu_master_domain *master_domain; + unsigned long flags; + + /* + * arm_smmu_share_asid() must not see two domains pointing to the same + * arm_smmu_master_domain contents otherwise it could randomly write one + * or the other to the CD. + */ + lockdep_assert_held(&arm_smmu_asid_lock); + + state->want_ats = !state->disable_ats && arm_smmu_ats_supported(master); + + if (smmu_domain) { + master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL); + if (!master_domain) + return -ENOMEM; + master_domain->master = master; + + /* + * During prepare we want the current smmu_domain and new + * smmu_domain to be in the devices list before we change any + * HW. This ensures that both domains will send ATS + * invalidations to the master until we are done. + * + * It is tempting to make this list only track masters that are + * using ATS, but arm_smmu_share_asid() also uses this to change + * the ASID of a domain, unrelated to ATS. + * + * Notice if we are re-attaching the same domain then the list + * will have two identical entries and commit will remove only + * one of them. + */ + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + if (state->want_ats) + atomic_inc(&smmu_domain->nr_ats_masters); + list_add(&master_domain->devices_elm, &smmu_domain->devices); + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + } + + if (!state->want_ats && master->ats_enabled) { + pci_disable_ats(to_pci_dev(master->dev)); + /* + * This is probably overkill, but the config write for disabling + * ATS should complete before the STE is configured to generate + * UR to avoid AER noise. + */ + wmb(); + } + return 0; +} + +/* + * Commit is done after the STE/CD are configured with the EATS setting. It + * completes synchronizing the PCI device's ATC and finishes manipulating the + * smmu_domain->devices list. + */ +static void arm_smmu_attach_commit(struct arm_smmu_master *master, + struct attach_state *state) +{ + lockdep_assert_held(&arm_smmu_asid_lock); + + if (state->want_ats && !master->ats_enabled) { + arm_smmu_enable_ats(master); + } else if (master->ats_enabled) { + /* + * The translation has changed, flush the ATC. At this point the + * SMMU is translating for the new domain and both the old&new + * domain will issue invalidations. + */ + arm_smmu_atc_inv_master(master); + } + master->ats_enabled = state->want_ats; + + arm_smmu_remove_master_domain(master, state->old_domain); } static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) { int ret = 0; - unsigned long flags; struct arm_smmu_ste target; struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); struct arm_smmu_device *smmu; struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); - struct arm_smmu_master_domain *master_domain; + struct attach_state state = { + .old_domain = iommu_get_domain_for_dev(dev), + }; struct arm_smmu_master *master; struct arm_smmu_cd *cdptr; @@ -2627,11 +2715,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return -ENOMEM; } - master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL); - if (!master_domain) - return -ENOMEM; - master_domain->master = master; - /* * Prevent arm_smmu_share_asid() from trying to change the ASID * of either the old or new domain while we are working on it. @@ -2640,13 +2723,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) */ mutex_lock(&arm_smmu_asid_lock); - arm_smmu_detach_dev(master); - - master->ats_enabled = arm_smmu_ats_supported(master); - - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_add(&master_domain->devices_elm, &smmu_domain->devices); - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + ret = arm_smmu_attach_prepare(master, domain, &state); + if (ret) { + mutex_unlock(&arm_smmu_asid_lock); + return ret; + } switch (smmu_domain->stage) { case ARM_SMMU_DOMAIN_S1: { @@ -2655,18 +2736,19 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, &target_cd); - arm_smmu_make_cdtable_ste(&target, master); + arm_smmu_make_cdtable_ste(&target, master, state.want_ats); arm_smmu_install_ste_for_dev(master, &target); break; } case ARM_SMMU_DOMAIN_S2: - arm_smmu_make_s2_domain_ste(&target, master, smmu_domain); + arm_smmu_make_s2_domain_ste(&target, master, smmu_domain, + state.want_ats); arm_smmu_install_ste_for_dev(master, &target); arm_smmu_clear_cd(master, IOMMU_NO_PASID); break; } - arm_smmu_enable_ats(master, smmu_domain); + arm_smmu_attach_commit(master, &state); mutex_unlock(&arm_smmu_asid_lock); return 0; } @@ -2695,10 +2777,13 @@ void arm_smmu_remove_pasid(struct arm_smmu_master *master, arm_smmu_clear_cd(master, pasid); } -static int arm_smmu_attach_dev_ste(struct device *dev, - struct arm_smmu_ste *ste) +static int arm_smmu_attach_dev_ste(struct iommu_domain *domain, + struct device *dev, struct arm_smmu_ste *ste) { struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct attach_state state = { + .old_domain = iommu_get_domain_for_dev(dev), + }; if (arm_smmu_master_sva_enabled(master)) return -EBUSY; @@ -2716,9 +2801,10 @@ static int arm_smmu_attach_dev_ste(struct device *dev, * the stream (STE.EATS == 0b00), causing F_BAD_ATS_TREQ and * F_TRANSL_FORBIDDEN events (IHI0070Ea 5.2 Stream Table Entry). */ - arm_smmu_detach_dev(master); - + state.disable_ats = true; + arm_smmu_attach_prepare(master, domain, &state); arm_smmu_install_ste_for_dev(master, ste); + arm_smmu_attach_commit(master, &state); mutex_unlock(&arm_smmu_asid_lock); /* @@ -2736,7 +2822,7 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain, struct arm_smmu_ste ste; arm_smmu_make_bypass_ste(&ste); - return arm_smmu_attach_dev_ste(dev, &ste); + return arm_smmu_attach_dev_ste(domain, dev, &ste); } static const struct iommu_domain_ops arm_smmu_identity_ops = { @@ -2754,7 +2840,7 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain, struct arm_smmu_ste ste; arm_smmu_make_abort_ste(&ste); - return arm_smmu_attach_dev_ste(dev, &ste); + return arm_smmu_attach_dev_ste(domain, dev, &ste); } static const struct iommu_domain_ops arm_smmu_blocked_ops = { From patchwork Wed Mar 27 18:08:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607332 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 97445CD1283 for ; Wed, 27 Mar 2024 19:18:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=aa1dn/GiOQSKWm63H2VCSTq3D4Nvwip9WrUgMJcGMUg=; b=zvih2tjHgTuOhz TaM4LK3fsZPrSuauNYAuB6oQZeLDEMnfd0SPN5zewtdMrw9li69MW/Zm4Lzfsd9HwAJSijtXYnXJV gZ3DNItJ0G3vbWJfYYYZ6EGeNyFJt0xD2RzvjUSiAqCKhBldQDywyNsvPLbavNBxaQfAfbMyhviX7 vKB5W1+k/e0sw0PQMSTGr2v9jcspiay/Ob/kF67QgLOXk3/agiZDeVAa0+JOxPF5fkPuZf4xrxzJD cDtrB0sa97+jjQqtjEkSGTk5N7Epp2iEF63WZvscEXXqJeFDuOP+jUBTuPK3IuO1dQn57bV8hRGUx xhgMAMVjNscOri41AgvA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn7-0000000Akzn-0oxu; Wed, 27 Mar 2024 19:18:29 +0000 Received: from mail-dm6nam10on20611.outbound.protection.outlook.com ([2a01:111:f400:7e88::611] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhp-0000000ATdz-1NMf for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lWBvQfNJamQ7fArcgtC3z0329PRRm9xhdmUGYTTs79fx4Bg6V6NLPjYRzEmluYJCelDAN4QTPRfGDL9ulBTzCVdW3IT+wgycatS/b98QEqw+13ecsOPVdIYL9jlMjlX/beonJVXK/dWYAP1DeK82l/C+TreG/UrlatulQFQ7Dk+L0XzSlQrEfTqxTGhWfQrPkwpr1XRuteSEooFkl2TV3fhIrbLGTvLeFeUPI/webCCuhrEoUoUnG5Y0+cyfxuAR+Mi5luKtD56dbcH4rd58qe+RYF6WB6UdCKqOLXi6M0L309mh6MZNJNXif25vNnkiDFlxsvXZbqQTt0Ch4SnN7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wuZ8uXi0Y7MfOGtYqppEmrCbTgTklmpT9Fao4yNrIhY=; b=UaLxEFxLW9439R3+CC2LvB1arlhxxegLnWMxwuoXnLp/iQLqE4cLcS4dz2d5X+Lg+RUCqZdcgFNgB6QurxTFBcRhqM/JweMjbDQMFxuxnRw5shjT6TIjGsjEP3MHp4xHwOVLunwkqhz+0qsE3V2ukgEI8hfGeqy8UrY5D/A9mLHeT01mNlr+HHPmwt7ppxqddTs78ErUgehKsl3pUiWxZd8AgI9ndLYjPFSWyYvHB3RsBP+YDmx2TKaPReSI0ZLjbDMMAWThsaXr4BtsY4fEshKNZ0GQBZEv1H90HuSarG1H1oQELk0GF86QN4BKc5wjN6bZ30LkJkqMK8hu0CTz0Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wuZ8uXi0Y7MfOGtYqppEmrCbTgTklmpT9Fao4yNrIhY=; b=SaMG4PQUAkg34cwHVBPItxnHXdkcl/q9YwwS+8y+GG143vr2CGDpMXkHlSU5QJ4K+upYhZEG+QLLQ5SUUqrxeEaBDm7RDdwoIjIuCLX4czIFjdh/fHfkXOcwipYi7TG1CPRcPPgP0FYp8FsQrshDYy4bvyxes2A1Yaen/RLCf+y2AhgM5mhHsPt+Elpo6blFE8BDBVzjWPuQTkuFguwBHdpptohjMaKV04wD52wg4MI78kNJuEVnjnZ89++a/BD2FuBsz6RbN4Vo649q1bLChH02113y98P9s+yDx5wov/7TcWtN5MMOzYMq+39daWcco+sy5ViMDhqrISQlaAOi/g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by IA1PR12MB6044.namprd12.prod.outlook.com (2603:10b6:208:3d4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.33; Wed, 27 Mar 2024 18:08:33 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:33 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 17/29] iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain Date: Wed, 27 Mar 2024 15:08:03 -0300 Message-ID: <17-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR05CA0015.namprd05.prod.outlook.com (2603:10b6:208:91::25) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|IA1PR12MB6044:EE_ X-MS-Office365-Filtering-Correlation-Id: f99f0824-4a9a-4be1-1327-08dc4e88e22c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vE5X32rUgi/4dqxw4a5U+rhJxTxaXFIn58dVTaW3ubHfrQlqrmoHENocAugxcTSe66/Xb/14HIO+cd3soE/MuMxMIYntdVDh9uDugep4Uats1MiYgwV7870PF3q39L5ufhDU6rzqNuIanGb3PHvJxZaJ9DTJwkzOzWi/qG4B2s84ha+4lytePpMTI+fvq1jaldSxBVqxn/qWoPptt0zfGrV40T396p2d4+VdXBEmEpOW8eSHoUs2Py/SLL2P+1zHMsBmN6t58ZaxA3quW8vGMzJ+Y9UqqUhChIikzdMIb9p2qCDj8T45tPV7gv7UIEDvvCo54CDQ46NuoFPNcVpPZYCAdBLfJzAxelEO5IKmpH33xiDyMCtv32rMgQLapYrM3ctLTGQ56bhxB85OWef6AXuhGIUZbAlgnwbLmLCakDz3HySOBf/86HK2XwWEiRTLo/CL6+rCohO+VkxavcZxuWcyDefZx9kk15QCxq2yCCzTp5t1ztCVxOvpuFPQbnv4V1T8d9rvhId1M+Z9nIf614dAjflQgquMU0edv1L09ZhiZQGfhuV5V6zLICpvnq95AWPLxEniCHrGKCu87LQlI5iotRfVSZs2AGeR7J0h7hzWE0umsHVPZy1uU67LPJySAFP1AsHZDfG9LASL4kilLkihra/JtysyFuNrSFA6kLQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7416005)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: GbKOtGa+cN+CjzkSAXugbWkbPaQ6MUWiV4+UX7kLUhnJIU4/jpZ7O1SP8oTSFkx3F1JoL0Vmxml68BQkzCFad1r9YBrEKLIOB8oh/jmdLtuaFid4quxrJYNghl58K2UjywyGBi8Crv/u9OG6/Ok+p8rFrKOa32F7cGu13VjfjzbsoKiO0Tb6iMJmzVKYmNeD6enFwRSKU+HwCNU9XZptIqEpRMmpuuZ65TCHjOq9fguknZazvchci+G1dR5mXIjNpGIWbkmfG3CHdwT1qZnbsbnL6WE+RqTKzGTNLqOuD1bQWonEd7/TR2ZOF/fBZWxOTQJU43+6AKpFYMVLoUJmy3E93G1b+qfxfKFWtMj4M2FYbj1glxWdFsIq8HrCkulQ1rpOXWCCkb1JZKhdsJiq0clPWrH6HpgjO/82BcKnfIyWkdERUGkcvmBLYCqYVF0Yk4AgUe5Hsz1UCCmEPVfRxHiDmTaFlRcScyTIVZh4hDOEHbqY4vxiCQUeqNkO9Zu/437gFCtd2v4PybLr4Jp5V6em3OXYcrMi5TztWkTptb4eEH7QlG8GfqSTrKC5M1ND0swCB4R7lAN3Pf0uoE6rgq5RyYY/XUsRlr54bvZrLekjVHe6nkTJ9iUE8Sc4AnntX8WbrV3Ql4jMJlTmHQvERn5qjzO/h0BJuDdIqrXn/qyvWboOmJGB3KKUrjm7ixE1GYQGe7kMEmSV6OvVtuIa8/ed1d4j2MwESQX2E9KhgKPGgBPp03Pg9rkOPY2tOS4pDfWcuvZhxmYDIhH7PKOY4jEtSa3KvQ/BjM9zMcRZXOVHe1Q/sTWraxupKkez7zWFNROamiJh6+/sTUJaoewWkquz0uegCAmotz8GNMrPEmDLJ+d2dnkybcy2cAx9iWPmuHCiOs33a+1ZpYZlvzIxPNp7MGirVtyJ5aj+ZcMHOYqEtPsNxL+bxNHlftuZV+XDJlB971xEQ/FZZpVgtjt01tAxYh0LzIFhaZTnpjMcVHbp4xtRzc1b7jd7CAGW4nt8TRvxF0e/kSmnbJw2LDhXeoN8UWKRVyqVCoraQwuJxambCDqWsnA6+taH/BHcxjcXJp3CiDxv18l+9GnsSp5getIlGRpHXj449pVMyMiqWCpQvyVKODNmmnvYi8hGn5v4EF/Tw93paPOB6RK81zZxOpu0OQI/hJmPKKJGJp7sr7BiY2PFlcwgspeq8877KKOsBe/i8vr7dZLnrglvQOm7wA9F1Bgb1MdlB1Hyp+RrYhCRLs3GQ3Vv3us0C5HSzFd9h1HYot7aeY97X60tMFEyzUJ7skbONcdegQbrT5T/5mymptHscjzU7R8Ri4HKznbQWKnchXP+PVJ5BZhyCTUrxp9xULAgQCcjuQmF19RqcMuu0N5lODh+2QDdsPV1KWcF4wY4PSP8/qHMibp8gNPKt+dpTpD2NUzUC9wIWOstvERt5b50mO/qcmMIOyuW42PjKtJz3i/7e5v3gvHovnRyyDCalaYJ+0B8i4b1/f6vwLfP47kWkDQ1R3VK/Z/nKvQoRLMxl49v/jGeoSeI5KATLvpR3z9IenpVoj4+71qdNllMpdMvZsxw1qJBtyEthzYh X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f99f0824-4a9a-4be1-1327-08dc4e88e22c X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:19.8256 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: DDSukouBalazORgrk5T9raLe1be1Qnup5vyo5n4iPsFQFHB7aIl8rSBaOWgVNQF6 X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6044 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110857_429890_2A7340A5 X-CRM114-Status: GOOD ( 16.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Prepare to allow a S1 domain to be attached to a PASID as well. Keep track of the SSID the domain is using on each master in the arm_smmu_master_domain. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 15 ++++--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 42 +++++++++++++++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 ++- 3 files changed, 43 insertions(+), 19 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 63f98264b646bc..be54dfb89e880e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -46,13 +46,12 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) struct arm_smmu_master *master = master_domain->master; struct arm_smmu_cd *cdptr; - /* S1 domains only support RID attachment right now */ - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); + cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid); if (WARN_ON(!cdptr)) continue; arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); - arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, + arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr, &target_cd); } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); @@ -292,8 +291,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, smmu_domain); } - arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), start, - size); + arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), start, + size); } static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) @@ -330,7 +329,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); - arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0); + arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0); smmu_mn->cleared = true; mutex_unlock(&sva_lock); @@ -409,8 +408,8 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) */ if (!smmu_mn->cleared) { arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); - arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0, - 0); + arm_smmu_atc_inv_domain_sva(smmu_domain, + mm_get_enqcmd_pasid(mm), 0, 0); } /* Frees smmu_mn */ diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index fb18d9d500aeba..00be1e2ebefaa9 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2011,8 +2011,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) return arm_smmu_cmdq_batch_submit(master->smmu, &cmds); } -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, - unsigned long iova, size_t size) +static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, + ioasid_t ssid, unsigned long iova, size_t size) { struct arm_smmu_master_domain *master_domain; int i; @@ -2040,8 +2040,6 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, if (!atomic_read(&smmu_domain->nr_ats_masters)) return 0; - arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd); - cmds.num = 0; spin_lock_irqsave(&smmu_domain->devices_lock, flags); @@ -2052,6 +2050,16 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, if (!master->ats_enabled) continue; + /* + * Non-zero ssid means SVA is co-opting the S1 domain to issue + * invalidations for SVA PASIDs. + */ + if (ssid != IOMMU_NO_PASID) + arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd); + else + arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, + &cmd); + for (i = 0; i < master->num_streams; i++) { cmd.atc.sid = master->streams[i].id; arm_smmu_cmdq_batch_add(smmu_domain->smmu, &cmds, &cmd); @@ -2062,6 +2070,19 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds); } +static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, + unsigned long iova, size_t size) +{ + return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova, + size); +} + +int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain, + ioasid_t ssid, unsigned long iova, size_t size) +{ + return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size); +} + /* IO_PGTABLE API */ static void arm_smmu_tlb_inv_context(void *cookie) { @@ -2083,7 +2104,7 @@ static void arm_smmu_tlb_inv_context(void *cookie) cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); } - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0); + arm_smmu_atc_inv_domain(smmu_domain, 0, 0); } static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd, @@ -2181,7 +2202,7 @@ static void arm_smmu_tlb_inv_range_domain(unsigned long iova, size_t size, * Unfortunately, this can't be leaf-only since we may have * zapped an entire table. */ - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova, size); + arm_smmu_atc_inv_domain(smmu_domain, iova, size); } void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, @@ -2524,7 +2545,8 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master) static struct arm_smmu_master_domain * arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain, - struct arm_smmu_master *master) + struct arm_smmu_master *master, + ioasid_t ssid) { struct arm_smmu_master_domain *master_domain; @@ -2532,7 +2554,8 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain, list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) { - if (master_domain->master == master) + if (master_domain->master == master && + master_domain->ssid == ssid) return master_domain; } return NULL; @@ -2565,7 +2588,8 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, return; spin_lock_irqsave(&smmu_domain->devices_lock, flags); - master_domain = arm_smmu_find_master_domain(smmu_domain, master); + master_domain = arm_smmu_find_master_domain(smmu_domain, master, + IOMMU_NO_PASID); if (master_domain) { list_del(&master_domain->devices_elm); kfree(master_domain); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index e4043e48a6e20d..5f49a5771ab027 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -739,6 +739,7 @@ struct arm_smmu_domain { struct arm_smmu_master_domain { struct list_head devices_elm; struct arm_smmu_master *master; + ioasid_t ssid; }; static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) @@ -770,8 +771,8 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, struct arm_smmu_domain *smmu_domain); bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd); -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, - unsigned long iova, size_t size); +int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain, + ioasid_t ssid, unsigned long iova, size_t size); #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); From patchwork Wed Mar 27 18:08:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6FEA9CD1283 for ; Wed, 27 Mar 2024 19:18:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rNlynJglhi797CHasnTszvZkYUPd4HCU1BUrqaB0EaU=; b=jBtTJFdeblm7Td 2KfnNjBRifZX3trFlR5WMusA6TNrDSJ6rVUZUsW0XY+B2oU6BwSgPdAzOq5+JkPdXeqlcnzxdwMG8 h6ac3EMTHGvlFFFtLGkvmvo/JHNVlRmmAS98j6/zHChIcz9llaG/jw/WtHkoTgJDPaNGJrD9d0/52 HvdFMANTsmjuQI4Qxvfc3y6wume8wMys0him1qgfM2Ew7zv+kNAvjiDPeAd6MoyAk4DtK7L4UqOHp 4xbEUVGC7zRoeZGHHcEdo6Bg9UdoYh+r9D99oykzwneTgphQWUX35YNqnN3j45I6jQYEJBX7SBazV Uu6PMC4NSfUzM/+x+o6w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn2-0000000AkwT-25NP; Wed, 27 Mar 2024 19:18:24 +0000 Received: from mail-dm6nam10on20611.outbound.protection.outlook.com ([2a01:111:f400:7e88::611] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhW-0000000ATdz-1cJI for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KXqgy4xZRokcglVsH0+AG31p9TjxhyNCrG/EFjqN4++1KArn0nTRdn2K/Byf38TzLsMBusvh/9ttOSYOtkV17xTuFyYdqUwBpECn4azuvjr40Vch185TmytsLbPkD+bDDmWjifXwLPEecxTD1Gb74Yvu6jaOumAyaRMosYPYZhAUfd7p4gBeZRb4QhieWEhZOMdiScDi1rnoNSIWmQ6u+n5sAHjQmdg8Ka0iM+l5X2bmWjYtQIV/DuDqKBtw0SENkW9iaDSJHkb9kWCFxRJOpOi3vLl1Z6Wy1zkSRBd4J98uhyjlb2zHeIZRHM6GzfpSQvjd3jWm02hVLWH1ul27fA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=v/hAtwLjsJOu8g9LAALDkiAlP1Ro+9RmW8hrd4iBZPk=; b=H1s8C7QT5uuYncS253aTk91rrsh8akpzaiPI9+41tC/cbikzOmMI19vvOkAD5bPrBpbg6iKlzmDIeQ7HqQBpPCvakOrQsRQPmH6+SvFudaf3fWb8Kv0WTPsniN7q4MCsY+YEIlJGg/eoH9xvhpCuI2HKTrPw9TULnEY7k4xXmpriykxjZ7Lj74xGTKCW3VhzF4ux/k/G8lnnRlLa41smNvs2ykOjroXQQC84dRps8o5HR0LGJlW18VqgCqSG2Rb/x8MyG+blqAKXUjVZZLdKb6g8uHLdCDl0NtLnrjWdc6uQmtNlpcNMFHZ8H3jqY2KZlawfxoHdS6Y72hVmYwpZnA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=v/hAtwLjsJOu8g9LAALDkiAlP1Ro+9RmW8hrd4iBZPk=; b=uAZDSdhIAmtjrODjbWRrHZGn1TizSyNJ5gJnlO3lahh6R8oAQene3lGpxSFGOUuq3jDAuk+RdVU9hFL9IUGq9oLEs2yQRyITHf9+wVX28eJDyJZOHuFMB4rHFOjEfuKWIWOJL1VYiTaL6vbzcV0HbfXkA4sHyIt+Nen72YKwUC/PQ05/Rl6OD6lcv3FQMRgKxEZ30Z1rBxf+dnsx5olCzYALBuBbMU7IWPjOui9rXIDVEk4d0qNJx6jIfi6NgtzONFu4+W5obHms53NVRTBxWzEDQtD1rJVpw7x3x5Zt5EIlyaaIC/fM0dtuRXGpt+DP1VrJL4Bd02YpqDXCBAqthA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by IA1PR12MB6044.namprd12.prod.outlook.com (2603:10b6:208:3d4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.33; Wed, 27 Mar 2024 18:08:31 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:30 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 18/29] iommu/arm-smmu-v3: Do not use master->sva_enable to restrict attaches Date: Wed, 27 Mar 2024 15:08:04 -0300 Message-ID: <18-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0042.namprd05.prod.outlook.com (2603:10b6:208:236::11) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|IA1PR12MB6044:EE_ X-MS-Office365-Filtering-Correlation-Id: 482c9163-76fd-4654-84a5-08dc4e88e1d8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9OGrJGlZgzQjXQcHA3ArmRm6f/lCsbadmX4rNk+Dje4pKF6vyjdPy3GFpOFdvkx9Pi+3EPje4hNAXC2BcQZHzs11CepdaEI+YwLgphiwJD5bX+OtYyruGeEHo9mOkPvtgEgFROHXOwhTFb0GtU3us2BgZCblLOfoWemDcTnLmSD3ev4mCEO4Lv72chg19Fo7R9ngCyhAoJG5Hw9BjIaKpkHE7xkrvTjOmk7qN8DJhshJRZz0uUbrXMT5kvEPcTsyYH1NnJULmlFF+357Ik/YdPZrJ9W2IepQ/QH3Xbga9bxqYlbgQZK+bnujKnNpnafzIU9cDfPmIZVVCfDy3wjhAOl3JqrJmj1MQtLxTMx3VqJKrUnHJlwvRy+6OYVlCGq57RVistlllsujllAlAZCtpm1GPxyeDGcynuWJEjpuLT8GBGV7Pm1UsQNKCdeyNVZEkomIDuQyjOrdvPoZrkB3fq5Gb1/4tCSvBc02HIL1aUnWwKOUXrbNhltDD7KBw7PE4PsT1TXQL2Yqrc/yZvxZPvnW7tYOvvNGdM307aIKcsbw+OG+FqIszUj3CrNli9tNxfxtIxkKMWbJm9DoQLMcIfIoI6PX+7ZZ7ZJiGZ7oDLcRwW6/3U7spOF4Z1TkCl2uvDjZ0MiThcYE+9vS61fKnE8RpV7/wfX1uu5mz33pzyM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7416005)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: wJkWJ7tf5MfrVM0NhKwCue/BRD/a1yxObblt0XuZrT8F5BXttU/iDuPl3QZ3poUaQsHIAH4k+NS5dJmMRYTnQCZv7naYl7VLDMhiPow/xKM7oWCePR7SA+xWQrDiIksqwBKudiULpe+vGfssJRrrOF3U36/l3KrYthEX08ss8xNBnbXUIwNcYUrdYb6jsq3mLM28Gen0bsVkCTUgX92yBx2CRwAwb20OczOQ2TAJahVg9rR6GqdRlFFN3csulJV9yfrvtcMHG3kya4xJopmiIt4CFlkdqG5IPhmJMNiNzK+lC8w821mxbCV8D75SQk1tVUukP6XtVBmxJ5jZTjbFSHNDSQ+Fa1vzxdl74/oRUWLQfzNSSBMpPQ7/QRRldIQDSKTTguo/B6WM533GwkxIkYOYj3PiVuHmlb+QKPi0ZNSkCwkP3h50ftb5hsxtd5f/CbRfxtgZCnvbJiwLxsealBnQcc8oJZj+ssFdVEzn7AW5BD/iHq3Q0mWHO07FLHuvkv+O4trGIrafDtRsHkX4F4g9jGeumIqbZCQ7RhmAqCVcfh+/SeH3E4pfMQ8Cj9f/i4nozOJSUGPw18Wv2WLzbVb11y1CmM2Uq2/fOmr9kPkrwy56cqavVej584AbJgPPvLf0mP9nD/JFPHQJRkIpAL+I9I3DtT6gLix5uXnagybDEYmFHiVeaHhjkkqcTsHwjv/XT1ebTL0/n47Dd1lNlX3vFydCzURrlgMHU8MKqbtkjdF+zbO8/nSXqQRBounQJp2gptGH65D7ROHzbR1YkjEDjPyS2xcONTCA0gpQMXSzmzE4q8r5M1fB23VIkvF84P/C1XF+n5TCFCwxEm4/Wn88En7TMDH/pRCE5vgR3gI5X4o675pUdP2NC04O4nP92VXQ8Q4/j5Vz74iP11KtmtBw0RMo3/gM0TAIO2jRbC5RTV1xFRZ+DFvkP/vsWeu8LNxdjyhvhu+Am+uCvX2tt6MEx6dQFTPu5D6dj9SuKMKmTfRNsJ0ydgSr6Va7JhR2bVLtVvbeeb64XdGXsPkKS8vzaNXxnE/1XIE+0xQpUxgNf1YWnxOG7Wayx/n/O7JkdrLcUqr+hZSv4ectmUpKcwIXCdQPMhf8pR6oX4cACK1vGE0F9wH5hUki7cGLXDtn7OH2gH7/xYgo5XLX1+E3/4bKyFG+RQTHk12gBcgL4o+w5xmiUudggNCiWqeiZKY5puRjdQpvEVpd2iECUllO2GlaoGW10lYjXQsYW9AZCCMFNYI6RPQi0pjzJdnvTPLKNxtBKL1YI0jTi6RjA5jbyC9ZVQtwqSzBb315qI238UkXmYe3xRasvdYUBpuS7Mg3WexlsNp4JK6oupr0TndroN5X75gITEUvvicrU0+8RmEIph+q9hTXx+JLEmN9jixi4w+B1la9M7KcywcjzGSeaBtsJZFgOBM4uRmb/zPNIoeiYt9+aovxyer7tUj0qfFHJtgSSL0kIWAC5Bi9us70SItblSyJRyAQ7o+NGqjtNNo/52Vm/cb86wypECUBQcyyi9DjCdfVuszlR39QgeWp7B/AVcMq4ENDaRwuicWEDcd2PBjK3nqvpMHWE4lJYz/u X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 482c9163-76fd-4654-84a5-08dc4e88e1d8 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:19.2218 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: HrQfFOFiu9nmpYBQ6AdlBHxg58bm1U9Rc73u5XXT+Lzg0YIcHsG6G+5fSy0Vl+2l X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6044 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110838_563048_48927AF3 X-CRM114-Status: GOOD ( 19.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We no longer need a master->sva_enable to control what attaches are allowed. Instead we can tell if the attach is legal based on the current configuration of the master. Keep track of the number of valid CD entries for SSID's in the cd_table and if the cd_table has been installed in the STE directly so we know what the configuration is. The attach logic is then made into: - SVA bind, check if the CD is installed - RID attach of S2, block if SSIDs are used - RID attach of IDENTITY/BLOCKING, block if SSIDs are used Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++++---------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++ 3 files changed, 20 insertions(+), 13 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index be54dfb89e880e..24e7cf759bbc35 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -626,7 +626,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, struct arm_smmu_cd target; int ret; - if (mm_get_enqcmd_pasid(mm) != id) + if (mm_get_enqcmd_pasid(mm) != id || !master->cd_table.in_ste) return -EINVAL; mutex_lock(&sva_lock); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 00be1e2ebefaa9..a77467f8837515 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1297,6 +1297,8 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, struct arm_smmu_cd *cdptr, const struct arm_smmu_cd *target) { + bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V); + bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V); struct arm_smmu_cd_writer cd_writer = { .writer = { .ops = &arm_smmu_cd_writer_ops, @@ -1305,6 +1307,13 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, .ssid = ssid, }; + if (ssid != IOMMU_NO_PASID && cur_valid != target_valid) { + if (cur_valid) + master->cd_table.used_ssids--; + else + master->cd_table.used_ssids++; + } + arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data); } @@ -2712,16 +2721,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) master = dev_iommu_priv_get(dev); smmu = master->smmu; - /* - * Checking that SVA is disabled ensures that this device isn't bound to - * any mm, and can be safely detached from its old domain. Bonds cannot - * be removed concurrently since we're holding the group mutex. - */ - if (arm_smmu_master_sva_enabled(master)) { - dev_err(dev, "cannot attach - SVA enabled\n"); - return -EBUSY; - } - mutex_lock(&smmu_domain->init_mutex); if (!smmu_domain->smmu) { @@ -2737,7 +2736,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) cdptr = arm_smmu_alloc_cd_ptr(master, IOMMU_NO_PASID); if (!cdptr) return -ENOMEM; - } + } else if (arm_smmu_ssids_in_use(&master->cd_table)) + return -EBUSY; /* * Prevent arm_smmu_share_asid() from trying to change the ASID @@ -2809,7 +2809,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain, .old_domain = iommu_get_domain_for_dev(dev), }; - if (arm_smmu_master_sva_enabled(master)) + if (arm_smmu_ssids_in_use(&master->cd_table)) return -EBUSY; /* diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 5f49a5771ab027..3da131e0173e1f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -602,12 +602,19 @@ struct arm_smmu_ctx_desc_cfg { dma_addr_t cdtab_dma; struct arm_smmu_l1_ctx_desc *l1_desc; unsigned int num_l1_ents; + unsigned int used_ssids; u8 in_ste; u8 s1fmt; /* log2 of the maximum number of CDs supported by this table */ u8 s1cdmax; }; +/* True if the cd table has SSIDS > 0 in use. */ +static inline bool arm_smmu_ssids_in_use(struct arm_smmu_ctx_desc_cfg *cd_table) +{ + return cd_table->used_ssids; +} + struct arm_smmu_s2_cfg { u16 vmid; }; From patchwork Wed Mar 27 18:08:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6F1C8C54E67 for ; Wed, 27 Mar 2024 18:13:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=78a6QDcthU8/Mev7B+vivMVFY0K9hP0X339da3RiVeo=; b=E3FS0CN5tttOxN 03UzDb1JOq0zK2kTLQnUqM8CCZ+voTssJLriQP0A92r532XuJflpAmq4lP/s2eTwgCcrzWrNkWYMl Wu2dzHFSEk4unhWV0pQTHd0NQOKeP5HO91ZjihvSR4PVdNXIW3EdlVWB9aDH2tWsMWfJVvoNf0zi3 DJaKcdtsWNUxHFodgMy+4zskTPW3tDB1aUf6Px9Uv3Aa2TZUaNgWcDuKTOkpZ2+OX2bCxflJcCRX4 9aF5+oJypQRKvJRlCUTneBTYGQ6h+o9rGixwemom1OHHALk02YEf4ny0SQqY2heB2azeC7bFejHtH G61WWWIjUbdffrm8QeSA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXmQ-0000000AWD6-1PPI; Wed, 27 Mar 2024 18:13:42 +0000 Received: from mail-dm6nam10on2043.outbound.protection.outlook.com ([40.107.93.43] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhl-0000000ATfM-0gah for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:55 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Gwxo9PAT+vK6N+T+6q2/VxIi7WlSIawjU4Twv451NBl4MzMV0oZ2+8i+wYTT0DqmB6kcRQahf4iR/UZfaWbgzRUls/Nmr+Wj+LIsT4dK+gXn5BPJiDlZLD8wJT++Tc+Y90eagBULxKMAvOr/GSXfYctrgIdj88ybJ8X2p2AL2U1d8hclW6c2yqqb2xpfBAdD6XyhDIDuErzv+QJoHGHMAZIw9yinLBXfGxIzcu5F+6i8+c3ZPjTP9HgMcuW/86ksfh03z+JPw5HRsxu+4Pvd7Qi5U+oj2tF4PG3h1fZFtSzd2qRifOy6m9k7b9HcoQ6v3Xj1xq3nla2+sFkkotb5WA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jQBvI8OVv5BuVAW5XzdwqugUxNAWCcD+pcM/BDWuAuE=; b=e0pFlcm18Sueq44MoSzQhe+t2C/IkWeaz2+D0HS+PLbOasP2Cza6y77/GEjPYVfQfLOe6mdcfAyoFgW3zE/ABELZavYT5C845WsaGg0xBuBDYf0OS1qyHkP3LwnRFYXdNg8ggtjnljs9HW66a8HtAwV8wLZo3Hou0EVZiW7qmRjqy8jLsggCl3qsZphATwEdAvFW6xc3ycWXngYPOOJPVSZizxrZsdaw9bs5x6hb8iu62dGPwBzz0X/CLV+PB2C3skGGnTTwlwXktfDGLr6nf91ET06bEBDOEGqcK9fUZHUbLUWfX0oW/luKbY+/AL98YkWFLU3PDVHjWOfeLCG3QA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jQBvI8OVv5BuVAW5XzdwqugUxNAWCcD+pcM/BDWuAuE=; b=pFY3frQH6mhJCsjCOS9uxWFwYPiS1zl5+zEQ9dDOenI/9Cov/hUYkpuDmz8ylBTuFBlwa7pWhG3bfMJ8thEgAv8aO2Ac3fES2tX/7QLYdNPxHciZoGDwKJE8k2F51DdD7po+Tp4qf/jLEB0lJamvEvXqtFoUk5a+s1y5csjrDwVY/EDZiwBuOcUzaWgx4rbH8gpOCXMlSujvJpVLikk4HvXmIgL7KNY0OlY9id9k6tarxmMkc5GR2PRd/JhBbATAKirqbXtGTi3l5xUtk27rudOp5l8f5zi+hJEZ/gPn+Ji2h6RAlCBMbN9Rc8bowpapcVtyaya0jEbnM13BDau4ww== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:30 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:30 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 19/29] iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*() interface Date: Wed, 27 Mar 2024 15:08:05 -0300 Message-ID: <19-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0315.namprd13.prod.outlook.com (2603:10b6:208:2c1::20) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 1260a44a-4284-4878-2059-08dc4e88e1c9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: izC8IRY2OiEu6cYW9sZKhVc3MlNX+/DqNgbpzm0DllyD84Sl3kB0XhIpRSrkvFDTMsOkDT6ieb6KOWLp2DteYKyHVYjgnhHgGBgqNlbOBIyHCmB9UTJm66MOctWdjEXiUhyD3IxYhonbnzDfU2LTUr/ogVXaknk8qDANM3h9R+JpnrHTSyOuay3ZUxBmRuFnc9NmplQpvCQFpU20yZR39vHO6XxHK/vZLK+lHK4iu2GWcxf21GSGOMVSZxdIQWljJDAdcv8Gzn1VT6Cq5KYVf2XOI3dob81vHBzPzCe4N6pSPBPJeHE4PQEsKqbuP6G8JwmNVzKsiPlTw6UTA7XQJybFG+6nlHUE/2nGhfoU7RIv5gdIdAdUDbe/OMlpDCXfEuZtGOeNtMZKM+3QiJEuF0clFdk/uZg3DJqRkLQlL7sWUJhoqLyGt9OhsCzOzq/9AD8c9fKGqi7WS7M70yBfwvfYDaRMOySdBjnjEfXAcHpzWMMiJmHXdMF8UW1h/457Dia8+bsjyq0QqC5Dy3sR6spnETLo/yyIEdmWrCfEVYkpi8JKh1WVPnY7KruT2sMq0FenLnoLQJvGbVe+Wu8NymCGBS3t+tdwHyegK9frhvBVP/wWRc+Bgj0Vx9PXLjMiCeWh/RC85g6Lb8EM3HYO1Q== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: UddpcAazUKuArODqZJmhGh3LLOejWtih9T6uX4+ny0ZothF11nhOufI0oV9pA/LSn588Pl8Lz12eYVs0U2B5VcM0y632RZH4jzL5jl6zo7qJWG5moeqNxRp4YHCcn1wTgohiqVw/vk815HFoIVySjXUXXHdnwkH21UfMOxOgNlAXcfrZcseZfB2pw0h3KcBTf429GWmg6DThs1H7KiZzUcGdy5jmn+Ok9rSSygDEpemDNNc/Y1HJIhvqvDeKLp718B4vS2tciB/sVn4W0oeIg2Ut1h01uDITAdzTVtvUrfGKMN/tP4yGDKjFP+PlI8tk4kBryApdJAPizoXOP8cSA6/7mwTQAKoyOcK4/it1aSJpy8dV+KjUYS0BsQIxG37v6WdBoVm8PDShmRzUbNhWKeZiENf+1Zh1NUp1cMIvfFcJAKBdTW7KD+f2KwP2iEd5ckWOR/cx1ixt0NmQRLBKxw3ulxUQGSRVVO/TGHTC6Hlu+GePw7BCQDr75lDq5G/MjUPOfHg+S4uDnEHP25AErhm8IqY45Fq7IBbKzSDWlHI4/QG5AdnlS6o9fSprswkFZ7yWrR5QpSs8kxl7KdLxYIGMJi5UHmNo3VTv8NjzVwX5jBXlnQMvDljQ0o+CSTe+zGO0dbxtzK/mJaj50Ik+G1cyXC6mJNoSSv3qcbOwkLcfVGUjTfXaGqNHNQ3lU/xNnDNh1ycqyjAg77BYcZGcE0xoE//l8xoMZIuWdypdUTREgutCIFlnkpYy3FZzgvVXxSEwwe8gw+QWR1KrdT4izDMca87WIEAZPIgUXikJbWjBMTgnGOiQCQvUwYfTZpfxfrQsN4lpEZ2M7RpoOkvEP0U8j25GHaYya/EEqS6riC9TMRyXE8P41ToSW9jT3z0RMkLxJahU+LX5H1MLYi4MCpSu2R1FYt1EceiPEC0zJrcfrpZIPASq8ZK4vO6gb+fMH87GFchZbdgTRW3UVMJYTknffXscVaEWQRN3N6UCBbyc5JecDFUd/PsMF9bGLEQ7vZpOZguN5u9I1BA4VSZEXxZl3J/sWxgQJRdmvJvApW5Zplk4WTTDSlxIB2z7MVr12BS/zTem2B8mESReO2qPYu/l9znGun/0BqHQsF/7XrmIp5BHvEj4/IG+VfsqTSad/NDbkWnDcwnV10VDzmLaGKf+BobGgmaUr+gzr8sKJU+2ap0QjjKDnIh3Rs+208yZFk6YRLHxKpl75fJQDMpN5C9CI2UYih4vHxzxdTHCPGsGH0br8gBc5gjcWUbAlc7LcjSXs+aNCiPaBsaSyK8oZX5f/jcdUaaI/sJyyHnKYVOzW2UOIUjYZ8lVYQV+xdV+JMsfJj5p21bcX8yO/9XFi/gm6b2yXkP6eXXled9EIg11tginJlg4rebsSOupBzXsNcVElA1iJ72jht5mUhZOgW3mT15mSvAsyyGhozg6F6fOImgP38ghRJi1YswlrvVDkoc2V7ICVDwSOT5DOi2/aOjW7fpTDBROUTQO0tJe0n5Ycz60CWdse4OFKAyFYkYNLfNInnAN5o1/0auSnfKvVcvqRNli6nbpfJjFnZVUsw7FO214tcoglFRbRFL5eC0i X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1260a44a-4284-4878-2059-08dc4e88e1c9 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:19.1502 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MvqAPyPr1XkhJNiKzBi23P6LFOx4BqhH8HTZAIX5Tg2a0l8Nbb6kBaKmMqcuzJHE X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110853_387162_F7FCFC4B X-CRM114-Status: GOOD ( 14.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allow creating and managing arm_smmu_mater_domain's with a non-zero SSID through the arm_smmu_attach_*() family of functions. This triggers ATC invalidation for the correct SSID in PASID cases and tracks the per-attachment SSID in the struct arm_smmu_master_domain. Generalize arm_smmu_attach_remove() to be able to remove SSID's as well by ensuring the ATC for the PASID is flushed properly. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 ++++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index a77467f8837515..0376c1bda8d8fa 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2003,13 +2003,14 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size, cmd->atc.size = log2_span; } -static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) +static int arm_smmu_atc_inv_master(struct arm_smmu_master *master, + ioasid_t ssid) { int i; struct arm_smmu_cmdq_ent cmd; struct arm_smmu_cmdq_batch cmds; - arm_smmu_atc_inv_to_cmd(IOMMU_NO_PASID, 0, 0, &cmd); + arm_smmu_atc_inv_to_cmd(ssid, 0, 0, &cmd); cmds.num = 0; for (i = 0; i < master->num_streams; i++) { @@ -2500,7 +2501,7 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master) /* * ATC invalidation of PASID 0 causes the entire ATC to be flushed. */ - arm_smmu_atc_inv_master(master); + arm_smmu_atc_inv_master(master, IOMMU_NO_PASID); if (pci_enable_ats(pdev, stu)) dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu); } @@ -2587,7 +2588,8 @@ to_smmu_domain_devices(struct iommu_domain *domain) } static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, - struct iommu_domain *domain) + struct iommu_domain *domain, + ioasid_t ssid) { struct arm_smmu_domain *smmu_domain = to_smmu_domain_devices(domain); struct arm_smmu_master_domain *master_domain; @@ -2597,8 +2599,7 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, return; spin_lock_irqsave(&smmu_domain->devices_lock, flags); - master_domain = arm_smmu_find_master_domain(smmu_domain, master, - IOMMU_NO_PASID); + master_domain = arm_smmu_find_master_domain(smmu_domain, master, ssid); if (master_domain) { list_del(&master_domain->devices_elm); kfree(master_domain); @@ -2612,6 +2613,7 @@ struct attach_state { bool want_ats; bool disable_ats; struct iommu_domain *old_domain; + ioasid_t ssid; }; /* @@ -2643,6 +2645,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master, if (!master_domain) return -ENOMEM; master_domain->master = master; + master_domain->ssid = state->ssid; /* * During prepare we want the current smmu_domain and new @@ -2695,11 +2698,14 @@ static void arm_smmu_attach_commit(struct arm_smmu_master *master, * SMMU is translating for the new domain and both the old&new * domain will issue invalidations. */ - arm_smmu_atc_inv_master(master); + if (state->want_ats) + arm_smmu_atc_inv_master(master, state->ssid); + else + arm_smmu_atc_inv_master(master, IOMMU_NO_PASID); } master->ats_enabled = state->want_ats; - arm_smmu_remove_master_domain(master, state->old_domain); + arm_smmu_remove_master_domain(master, state->old_domain, state->ssid); } static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) @@ -2711,6 +2717,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct attach_state state = { .old_domain = iommu_get_domain_for_dev(dev), + .ssid = IOMMU_NO_PASID, }; struct arm_smmu_master *master; struct arm_smmu_cd *cdptr; @@ -2807,6 +2814,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain, struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct attach_state state = { .old_domain = iommu_get_domain_for_dev(dev), + .ssid = IOMMU_NO_PASID, }; if (arm_smmu_ssids_in_use(&master->cd_table)) From patchwork Wed Mar 27 18:08:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7F841C54E67 for ; Wed, 27 Mar 2024 18:13:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=e7hWwxMLrJQ0LDkxnW5Ni3HVFjmW4bSQZPxYpSND3Yw=; b=NjejWYYbtsACDK wjEv6DgXMbyhGsn9CdCv3WTyZ1UkG6yPVg7beSXAlunFtYLcUpTFQHBohbq6cTcmEidNnL0CgjxZc HpcTSH0j+0e7CX+9T5ERw2Wo+IR7g7Rgh4kQwc4hse3vkapM+101DzTPe4sZHyPRWL6w5N4uAgjsG AVNXXlvSHgXau2pPKjkKqjh66nbz0CuWDm79yi13IVhR86pgDzJmk55IjI3TxzPy0F3+cNfeaLyeD sOT8FF+dVemcxHxiKUoqjVZ+eZCterjeaHT0dNXlLtc6expwM/pqWlM6jXMJo0unMrgvbnvMfpUpJ dhLfBlq3HKUCpefBSlrQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXlv-0000000AVlV-0yOc; Wed, 27 Mar 2024 18:13:11 +0000 Received: from mail-dm6nam10on2043.outbound.protection.outlook.com ([40.107.93.43] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhe-0000000ATfM-1ScJ for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:49 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=msjkGhaoPUe4vqHCaRAdkpFe/0p0lKYdrSRil/poIMgdCkj0YAb6ZVSkeuI7+LmBvO7Aetb28MOjktWIcEcRrjrO3rqVFrYY1JHNrw2+fZte3Mgr/iaiiN6h786TDydi7U5ZzuxfykFl0nxlT9p4vv/+BdESOghm3fTBjgx0XGIAvUmGMgrUCEPjCRd0WCSqvBXjoG/SaaDoMm2HMZVWf6i1RaqMveT5C5oKMVjhD8IufhRq+rnKxBTy14569ZHG6EXLMLs2rUX+LUSCu4RNoOvMlxeoDv4r9j1WH1an1llhLNpQ4TFybeNwimjDFTBSO2rTb7e8sjUVIbURsdciIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sOVipcU6YZzA7Y/k2PkAaFfvnxP5+bMlrG/Vhhx9CAQ=; b=XR0SmwzCFBrY64XBD8DhavKoL6+8aR0mGHSOaSX+oQ2bFH6EsVula4bG5IabblkND5uGOA52zMpwH/tCK1acfhOzYYL6b2CD6oPYWBPD1DrB1PW450ZQIuXt8R4aISLCLCeNRBeRM3WIJvqmc66JJfHQ8L4ggfAvvT4xRDooYzdmzfjWYys9WiH8P+S3TnN81j+JcAjklYcY29a2Njg4e/kOhwAEkpGTEzxnh0lQLlpNk4b2LI6wyrw3KFASt32c+X9Q5ZaaK7arC8RKYuLWevgSxXUFy68gxZliqW27lk+nU6xf4fOipw40i+gFYM6mkyFfBHZ1a7rt/HBbhvn0Eg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sOVipcU6YZzA7Y/k2PkAaFfvnxP5+bMlrG/Vhhx9CAQ=; b=Ipsuih0SLDkhDuNDvOSdXpr9au7HRRYpV4AxGIcGNFy/kcWepNRyv/VtrIollln7OYO61yC2sWAGdj8KSbPtA+nJ8yYUEgMnrMZHsDb/oSkovHO1gNiDu+qwv+qRbbMLJuEcy6/ahEqYlO/QaCxP9Fz8EwVzhAIUzelne2qYStEMuJzw2RozFS4JgId13rNAuiZmn+Pn+PmbaSQgz/1RSjqKzDaYowA/qzgDQIMNABTumB3yNrSBvn0LTrSQC9B+TVMofZkLfFJgocfdVbc9V6UCfD/MD9+9NUprwm8dMtgVcn3XK42uP91dLAzc64PmfFNI7Zp+pdqA1oBQ80/QJg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:28 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:28 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 20/29] iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain Date: Wed, 27 Mar 2024 15:08:06 -0300 Message-ID: <20-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0040.namprd05.prod.outlook.com (2603:10b6:208:236::9) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: b7281c64-d9df-41d2-dc65-08dc4e88e17e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: y+LaKuC5wTb2mfvwTdDjPcnYjZP+F4YDWvBxC66GYC4n0UnCYUFLz0dOZhiKi0ISGd6+z+1KgOPCUyIJ5lP2SAjyM3FvGQ8m2ALHv0WQzLTwUKfXfNW69hgxJIqCpRDVm0xvrEii1d6/R5j3vVwOyx4fbNMR5/KtYwN5ctNn0fgxo8POialCcqI9JEoRSK5iTMKQf8utC4WzFrvXqWwaYrVk2eZUAb3Cre9DdcgLpXUXOl19lcJHw6Bm1lf8GgScDtpTHBhC00SsD9L3aJ0+2Vd46oc0DdtUFeX0JiYZejDvb54XFsB6BEh+LjK+u5QupoB8au/Ue0v5j2ycFrQJH/ihnqlVvhRmFO4EM/BUKQA4EGILmjoi3f7SAOxvOafCBKgfL0C+ETMMobruGrQXd690tMbBCa0mdfSI54I0JfsOw3ugOyxoeSe0MtW3gj/YCutuX5RwOQrKvUd3HpPBgjSwYv2ISXyFf4kWt0mDgsNa1QPg5w2TInqhh0FZYmPekcVJOm97UdFgbJmdO9GiCkyqAMcdTgLqFtxkkVvumPlxE8KmIe8g2sy/00tCuCaL+CXhwlxjGoGlVpJqIM1rlFZC5o/J7tkLFRFTnvdHB2XyyV+ETwbvaOFiJkRBLLrWSKND4+bGpa7A4XF90qIozXX2Nn41f3f9FeGZoQU0Ha8= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: UpTJ5aeCi8X2ZN9/A+OsDICYDEtbwwT4HnLjP6jwm5BQ+vgDCobFOMmGvuFRcWJgGAExCanGyDpj3V6ksrIly47pIeZzjv7PQUN8Z0cDeIV0QRP4VH2Gt/w2glo+BOV9pTJtZqxXehrJQMve60Dr7VZXLavRhgfzQKXtSSSIelkOspQpy8oenHvr2KiaWtX29drO5PtTzqoSAwn1PcnQGWbyKtWc/CEgjsNqW75hq5UVz6tZyKC7M9ds/7MoKKzu8SBLqJUHUbvNeU5GlCanY9ZKcQC4PUWy/g/DacW+F7DsQLWTJqenNKkbJzR5CMIyhQHMWG73wqOKQ2CwBlUCpiZD68TW69i/R1BMaPftDawVCIp3mWwfecN+apSebZVX/dibkTrvSdVXOO3FBGFjmFAxxc8H0CdGYD7VkfMF3DCKQjOSn0pHxXZXhZz5WJEFICBpCtQjwABwBYl3cP3Vq065tJnK+WMbaOYVejiBU+uzgcZ9guWrAOh9cEy5pyT6otE8dooqOQtO5EabyxriwN8GV4pf4Nx/hC40nVbU50Mvac/PExdE4tfZxO229UV/zNWiSepiQ0Hc0fwiNDYUqC7FOwSibV7V3rXT5E9MWlBl6c6upLZkTYOWJZx9JeI7KC90/IG3lnSjm9lPbVv7d6fmaFH+dl+MLXWUnFR1/XiH36UDIFgNltWWjfdUmIqmo1DtfntWsXOCvXs6BjjSOrRUrNj0u7HA7jKiB+Iyq3TOI6WZvAP3gkMxDiKESVGefaZs7k07c2v/1OjgcFUckrEzu/pk3YadWTW3sZdD979KRNbeFiWUUyzeGUL16ajAhvzAfZR6BpuYmcMuYd4xUjXsD/XJKmY20JeCX5XgLNHtz3RUlrnRvuIkrF+KF4erGvGyTguDE5v0+jeKqcv4ZzNfjXtVlc4nGogTeZTC01eraAKLJ778H7OnHteT3olOvQrXXLV00zx6pPXDzc4+p/YWuAwktEMqBDpLO4vzouNMX+YkIU8lYUkxFI5M2vqPWlZlhQY8cuBKT3riaJdEAfUeO1eQeCi5i/maXaLKPBvQutdovuwHRFYMPPf3xipV03DaaS5zKP8U5N6OKa1j6aT6qi4R0mMGsATp1Ta6AL4PV8lmqEjZUb4J4fKOZLPoWqpgNJAFoeeR9RNnAOHrhqMIYyiWM68ZfJDaCVxJd6aRT3sanZ94Xk++VwKMtnr34J+KrLZcbrk4eaVoB3jdQi99mn96G/h9sWmZjNgjrr+oFVwnzZdEJzYjbm93tPzGT/r4//zUXYOoOw/eft7nCR33KcSq/7y+iBO/7R6W1Uyn7T8jVaKtXqW3pPqkCVAIgBSJIBIQfl73mTEXzQbes27fkPMCNLGY9NChYRUvA410TNhHUTUb5YAnOzofMeUdckt0quqfDzcu6PMmK6yfjoxEuzxnDDHuuV8v5aQb4DzhJbR2Epyxq5AGc3ak6dowtDDh3ZRLVuOSdl8JV7NHryZn0LnXBvKe92lDMt3N3h8fZZI+RWpExTVLwYv0nCBScvi1G5No5YO6hhZ3pkylgKzXF7hxPj+/nnJEGuVeR3g0vQAbCGPj0NBJ8MquDKXQ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: b7281c64-d9df-41d2-dc65-08dc4e88e17e X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:18.6520 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mB+YA7XRLnOLzLK68fU/+4Xh2eZCu4U4p5mxVQNut4+TVrjOZ7OjYJRdk5OFOw6l X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110846_682721_E9DB450D X-CRM114-Status: GOOD ( 17.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently the SVA domain is a naked struct iommu_domain, allocate a struct arm_smmu_domain instead. This is necessary to be able to use the struct arm_master_domain mechanism. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 21 +++++----- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 38 ++++++++++--------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 +- 3 files changed, 36 insertions(+), 27 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 24e7cf759bbc35..3e7aad0960bfd2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -637,7 +637,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, } arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid); - ret = arm_smmu_set_pasid(master, NULL, id, &target); + ret = arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target); if (ret) { list_del(&bond->list); arm_smmu_mmu_notifier_put(bond->smmu_mn); @@ -651,7 +651,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, static void arm_smmu_sva_domain_free(struct iommu_domain *domain) { - kfree(domain); + kfree(to_smmu_domain(domain)); } static const struct iommu_domain_ops arm_smmu_sva_domain_ops = { @@ -659,14 +659,17 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = { .free = arm_smmu_sva_domain_free }; -struct iommu_domain *arm_smmu_sva_domain_alloc(void) +struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type) { - struct iommu_domain *domain; + struct arm_smmu_domain *smmu_domain; - domain = kzalloc(sizeof(*domain), GFP_KERNEL); - if (!domain) - return NULL; - domain->ops = &arm_smmu_sva_domain_ops; + if (type != IOMMU_DOMAIN_SVA) + return ERR_PTR(-EOPNOTSUPP); - return domain; + smmu_domain = arm_smmu_domain_alloc(); + if (IS_ERR(smmu_domain)) + return ERR_CAST(smmu_domain); + smmu_domain->domain.ops = &arm_smmu_sva_domain_ops; + + return &smmu_domain->domain; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 0376c1bda8d8fa..9611ac239fea8c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2270,23 +2270,10 @@ static bool arm_smmu_capable(struct device *dev, enum iommu_cap cap) } } -static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) -{ - - if (type == IOMMU_DOMAIN_SVA) - return arm_smmu_sva_domain_alloc(); - return ERR_PTR(-EOPNOTSUPP); -} - -static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev) +struct arm_smmu_domain *arm_smmu_domain_alloc(void) { struct arm_smmu_domain *smmu_domain; - /* - * Allocate the domain and initialise some of its data structures. - * We can't really do anything meaningful until we've added a - * master. - */ smmu_domain = kzalloc(sizeof(*smmu_domain), GFP_KERNEL); if (!smmu_domain) return ERR_PTR(-ENOMEM); @@ -2296,6 +2283,23 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev) spin_lock_init(&smmu_domain->devices_lock); INIT_LIST_HEAD(&smmu_domain->mmu_notifiers); + return smmu_domain; +} + +static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev) +{ + struct arm_smmu_domain *smmu_domain; + + smmu_domain = arm_smmu_domain_alloc(); + if (IS_ERR(smmu_domain)) + return ERR_CAST(smmu_domain); + + /* + * Allocate the domain and initialise some of its data structures. + * We can't really do anything meaningful until we've added a + * master. + */ + if (dev) { struct arm_smmu_master *master = dev_iommu_priv_get(dev); int ret; @@ -2309,7 +2313,7 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev) return &smmu_domain->domain; } -static void arm_smmu_domain_free(struct iommu_domain *domain) +static void arm_smmu_domain_free_paging(struct iommu_domain *domain) { struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct arm_smmu_device *smmu = smmu_domain->smmu; @@ -3273,7 +3277,7 @@ static struct iommu_ops arm_smmu_ops = { .identity_domain = &arm_smmu_identity_domain, .blocked_domain = &arm_smmu_blocked_domain, .capable = arm_smmu_capable, - .domain_alloc = arm_smmu_domain_alloc, + .domain_alloc = arm_smmu_sva_domain_alloc, .domain_alloc_paging = arm_smmu_domain_alloc_paging, .probe_device = arm_smmu_probe_device, .release_device = arm_smmu_release_device, @@ -3295,7 +3299,7 @@ static struct iommu_ops arm_smmu_ops = { .iotlb_sync = arm_smmu_iotlb_sync, .iova_to_phys = arm_smmu_iova_to_phys, .enable_nesting = arm_smmu_enable_nesting, - .free = arm_smmu_domain_free, + .free = arm_smmu_domain_free_paging, } }; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 3da131e0173e1f..9db84d5940466a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -757,6 +757,8 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; +struct arm_smmu_domain *arm_smmu_domain_alloc(void); + void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid); struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid); @@ -789,7 +791,7 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master); int arm_smmu_master_disable_sva(struct arm_smmu_master *master); bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master); void arm_smmu_sva_notifier_synchronize(void); -struct iommu_domain *arm_smmu_sva_domain_alloc(void); +struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type); void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t id); #else /* CONFIG_ARM_SMMU_V3_SVA */ From patchwork Wed Mar 27 18:08:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 13527C47DD9 for ; Wed, 27 Mar 2024 18:13:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=aRS6JmXJLdSyKAOUWHLHSFONMZvZzoysi/eyiXe6Psg=; b=L4IxnZo1QyOak/ bkYFDS2x4Y+ijiC9WNoqWvOB0KvDEXZ5RxV/d5SZd+BKe17XuLK+HoGFtWST3zERKFlBC71NBl0q2 r7/YzYG0l5xguxceBhn1g6Wnei0mkHFoz6vByGOXvX424Vfw5pAHBms4xgxwFxoQbk/xZU8Vt7Swk 1yTXVwlcF8+NnduxWUePRX9M9PKN0iAstFUJBvnpKdRg2rFLcAE3ggNmTY75VeWheiki3Fb/J5pb7 Plr7RVMBBK1FDYufxYwy8LMazm1nW8PywGgJAMRcKZIvlllsSKNLshm/kYPondH4c0BVM5KVfzQzw +OV+ehLQypYZQVe/fDrw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXlm-0000000AVeT-3DZz; Wed, 27 Mar 2024 18:13:02 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXha-0000000ATiP-3Rq3 for linux-arm-kernel@bombadil.infradead.org; Wed, 27 Mar 2024 18:08:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=g5v/9484J7yDEom6gZ4kYML6HpwQWLCC2pzh9JwczH8=; b=XV/R7dnstG+phCpba7bAHJbhhm h200Ah+/lE7qqLLVRabMzbLmUjXh2JdGWwkHt6XckYVO/LgjA9y2d0OW0sxzYDGqNXfWY5aQIxWRy kMcrLpb9ZAAAiK9VelDbabhfeCG1JcKAizZ/yJeEgCSvb55JvwgmCjpccOsIrP4sjjEpPurOO2HdK 8xGoRf0Mz/siOQQ7lrDCoUfe+jtyvR9pTMTDHMP89HF3iK8pl7YvxMv8OQICn9oBX+f4NtBgZepD1 zPKq8hErYaFB8PChHJI1Og5CmP86XjR8clXTHSgQJE4mqrYH4y6QNDwoiKCMsj3RH28xneN7xzyH4 41qgPK8A==; Received: from mail-dm6nam10on2062e.outbound.protection.outlook.com ([2a01:111:f400:7e88::62e] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhW-00000000RvW-38vd for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:41 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CO9sLFb5Dn/UkeIcd4sbDGS2nDl8zdPdAlVhkK3VRsjQbyIoz8JT6U+4muezrmSslBTyZsCLT+RgqZJe0Ucwd9x1CGb7KgfpO/BNhDEMiFusglCGEdf1EF4YgKXsIPbyQlBZACCuUI7FcW5mbo+qlhPwQsMboNH/kjf6kHTgb2ymimF/QaQSNcWr7bwsYMw8kalWkgza2C2KFYcovo8L9Eq4Hkm6bzbGyHRNifxFm3yKjRcR6jyHmLn97qz9I9lfH4NL3XrwNusUaMPu7PH3Y2rZpjNnFCqLwYoGTydd45YtMRLfHrWJ3uIp6ouWRmep8g2mKdVBQ/sLI7eJZ4Qi0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=g5v/9484J7yDEom6gZ4kYML6HpwQWLCC2pzh9JwczH8=; b=jxSZil+pdQSryPkCLKybpmhqKxgp1WHOxWYAgnmdzIhw6iAJYi0xIzOAaq51KpRF2UFgibT7pT9jOGCEQ/n2rp3G++Mfy+dPETf+qYgEQW45dkIuIfFYtuDksaNfDOrzokZkKII7PILMdUkYrdAp0f4m1O7owfGHBgAPJKtkHG5osQeyOR/Cc1O3eNd7n9eqlyTArEV2t1X3DjJQ05+y/zlrQcAaaX/vy6iEnJk0CqtY14qDR1VT33Yw5Ks7E0C2MycdnEashlcrerHDt0+y7PHjfu2AAqejUZ51OLrRedLJ5bQao0byQVUvTDd69jM4YMBB1z8TMyTk9183tjixww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=g5v/9484J7yDEom6gZ4kYML6HpwQWLCC2pzh9JwczH8=; b=QE4k2jdusKRhWoo3FKWQXRX+c/IkBqNeqX+YCofxlsA+3FpBN2wE5vH6cptDYPuZVrjrSvITxiaS7oYFN9R2aLrz0esnaLgUfi5RVY7QKVW0nbFTbTBue9CHv9ld3+jgR3ex61Wvh81S7D+yWuAHCnnYd57TZaf56C+HBiAMfJql7h5kmatjE/Ren0jlKgx1VBmGt32G2aOrtOnEEyF+NQneEjeUw5CTe1xnvbwNC1LJzeFbacm661Drb3qpBq/1DMmYHHs2ovJHi9kcaF3bLy5d7IUxgII9UOIvpGtMoI5z8MTA+KbqxREZBquwAF/CMLNnDgUEi4BQ8L5bLij5tg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:26 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:26 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 21/29] iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA Date: Wed, 27 Mar 2024 15:08:07 -0300 Message-ID: <21-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0303.namprd13.prod.outlook.com (2603:10b6:208:2c1::8) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: aaf91855-7165-483d-8687-08dc4e88e15b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: klSwySpAjlnp6WnR5mZt7F1mef7D+aPBSf7fGdy0/p3r6hqR3prs2cwtcpBdoTCMtYhAZ6dQjsihDqYeerMVRtoRv5MR5nxWBD8wPcBfFDnsCYABTqqX6qKXuyqax0tDAp9OPY+04v2uxlJ5TridvisN/4u5KMEG+ReZNUH4AU5RYzmLTm3paXmdaAfIBoA1IyuQ7hOVB8mBZij1lM9WFa3VCOzrlvt1Jbh1xxcYoO+vSq28uwnc2sAusCc8QylcHefoa2iFaSziiFslDgm2aDKCsl44WzUfQB6aBqg6SGGme/GnRF/Iq79YlcGHLzg7riniEg+ENA54EGIGhnCdsZWmqBNkqy568n/LAsfx1BZrp4dP7UatZEOd0Q5/SN0m0Sl9h4d/0iRIcn8sYvoPhvfI3fI61fae2zBlDTwwR3MoiJqXxE0lT/pcJ59w5cB1UJmKbBPmEjrH4uyL4oUPqPFVZocoOzs1qMIvkDnbedICsuZNY7K8dJ5ogwdisVsOd+nY/oVUaMeFulheFF0tpNk9ED37ojvOn85pp/WA68xYXaTBGiOK36ATTyDWtgCPnEkUfCPObF6218VTFee0aNoOtYdfzYNdFIlxbwFIEsqFjmKiENvnGjcYQu3x5iVUcL+ZUVSvDviz5RyHC2vqCb7bd2TSBwmMEV6FCvwPpwg= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: F5Naphl2DTXFAa8BikszPzeNngMqHjB1eqa8Ad1O/DRSDUBg2gcAx0kJxxoprzYozFTvgQmGFdaR0/deWYBz8B2paiKU6THAcs5iMkyPc8eWyPj32AAaX+ahD9VIOIWbSlpeR0PzNQxFrZsHOXwO2bot/5GlzYZyzPwyAbHMGudCwG2X4SmjlAVLX3srCyNzughdZaNhgIEyg5ilzMmIdZ9v8NGrPDADWsrED2ye1IQN9DitMLYP1KKfqgs9CqEUkRcpm0uFV2E5c5sYn4D/kEvUrAND1g4fd5sCS1zzXANeabOPvGHWMFVg6fjzd/1/3ZUd9xpfZIBuwusQYtgE0cYjdJ3C0vt0lhgWuTFUOKGgHPh7yXejAytDN7iXgeJvqIQo91uFX1xDn87Y4i54atxqUTuWPBAbsax6rAoMV2uqK/mz+shk+BAm+VZt8WKqVFyV83PPUpXVGwPQ0qElNCB5wIZhRzAY97Q46VVRxBeE7ETJQCYOwg/PBmNWWn0NohqaxBI9Wp28BPashIqBs9no3SsxApEYG0FI3R0KB+jUQQJfiy9QKKYPLP2MKGekWirdRjuwJbGcO7AOczaxZQmfAZnWje7Qb99dPR+NUZp5fpyVyBzHoTWeoleu4VFoZu70pjwxu17MKB9QI2eA3sZVieNRXyWkwEaFrUpych+udRgw/h1StM3naeAwNmJlUB2jKMT+CiA9Hvbh/G7CzuDMwukqYHGadAU0bTa40/AXxhJNBmORTPCzY+TJyZYs4N/tKVTymTw/sGgmT2LmpDNhIniVMxNeNCPK/NPJi6n76/HZByQJasugUpu548s5kEU1c5t/CzqOunNC8KeHwF+aTWDjVZGyau3SQFued8L3KfM0LLuWB8KzeUeS8JM9ffw2U+/IM7A4VSLsGciNIfHeRSQRzP0xSittjgj8panlaOsgWN4OqWPpdawV8ZPNY8qqjwEg/DXi7AL+nHYo4wx28yg5ThQYyqUkPqhUxfMfFfg7VJiqgIZ/PF5CP+XFTfmZPX2qpnfVLmod9mKtx7HB6iCuqI9fZn9d02PIO7ewtJ8DsE2b7+I39xg80Et8Eyho52szCvUhi1ytcIdDD0SkdQeKXY3g3dGPWYEGcJtzFGdNgYNoKPRr6DRigOgW9UCyeUM8N4NHnwFwcvnzgCfW6N5wEFOvGwA6bYJZ8QbBwp2qT6Md6SutoMB/N9HqNa5d7E3yJ30uLdcSWohK2ne1zecgB72fofkF+tElV8LywDlQzBjtItzvGTVJh9C6unLPd73p1UUQmPLrCoa3A4Xg53TlB8sJzE3Bz6eJZwzGxgRwMoGEI4I8JAHcyTYk0ouI3Z1OyvB+HUwx3W2FG0aThcPkGSEUPit0q24aiEmj/AICEdBLVwRijLGMIhG7gWZlVtfVB6Cm/Wfr+I9rAtV4ZJekeJg0n7SGHD8Td8q6U27trSWAqaDNOPJQEwLtH4+nvV0yueKjTC/iMW1pkPuIqgajFpNXfIcai4/0IBArqicyR2xXMKjR2qYJbyQqTYys7EatzoTH6h3p4Pg/lAcncxnG7CuTcCpdQ2+FxOls55IIJ6wqqc2mTco9tSjL X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: aaf91855-7165-483d-8687-08dc4e88e15b X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:18.4526 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XKFluHDaH2DPj4E8rQNELH0Jer4Liqpnl3tc8G5MVCZ44GkPoeKe0qEvqspqN0E4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_180839_872150_71B8D38C X-CRM114-Status: GOOD ( 14.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently the smmu_domain->devices list is unused for SVA domains. Fill it in with the SSID and master of every arm_smmu_set_pasid() using the same logic as the RID attach. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 29 +++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 9611ac239fea8c..3d9109ad60c19c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2586,7 +2586,8 @@ to_smmu_domain_devices(struct iommu_domain *domain) /* The domain can be NULL only when processing the first attach */ if (!domain) return NULL; - if (domain->type & __IOMMU_DOMAIN_PAGING) + if ((domain->type & __IOMMU_DOMAIN_PAGING) || + domain->type == IOMMU_DOMAIN_SVA) return to_smmu_domain(domain); return NULL; } @@ -2792,7 +2793,15 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid, const struct arm_smmu_cd *cd) { + struct attach_state state = { + /* + * For now the core code prevents calling this when a domain is + * already attached, no need to set old_domain. + */ + .ssid = pasid, + }; struct arm_smmu_cd *cdptr; + int ret; /* The core code validates pasid */ @@ -2802,14 +2811,30 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, cdptr = arm_smmu_alloc_cd_ptr(master, pasid); if (!cdptr) return -ENOMEM; + + mutex_lock(&arm_smmu_asid_lock); + ret = arm_smmu_attach_prepare(master, &smmu_domain->domain, &state); + if (ret) + goto out_unlock; + arm_smmu_write_cd_entry(master, pasid, cdptr, cd); - return 0; + + arm_smmu_attach_commit(master, &state); + +out_unlock: + mutex_unlock(&arm_smmu_asid_lock); + return ret; } void arm_smmu_remove_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid) { + mutex_lock(&arm_smmu_asid_lock); arm_smmu_clear_cd(master, pasid); + if (master->ats_enabled) + arm_smmu_atc_inv_master(master, pasid); + arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid); + mutex_unlock(&arm_smmu_asid_lock); } static int arm_smmu_attach_dev_ste(struct iommu_domain *domain, From patchwork Wed Mar 27 18:08:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E4BFDC47DD9 for ; Wed, 27 Mar 2024 18:08:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YsDJl9yvHticO3ryH7+NPY/yxLJHdb9sXmZpCi3Wsrs=; b=2AA9Al/EccUuia bpowHv9xaghqrFU4T4kaKWO5yYpbeby1KZoavasrqA2ZjepWFkpy+Smg463x7ml2qiO+GlYASzKOW NsdoJE9h96WjUVPmGHN7FJUtjuUXdXM+wPmeYa+lxTL1CmmF6/EsKEGsLWV1MlcUo9TomllqkHT9n 1Xi6rq2SS6nB81W6SJRJ/0hKZ9xsYZ7PS9VIbCMXl8Dzqv0BGfOewzJhWB4vb9Z07yj1wM+qJIWso NdEDDal9VPme0aIFUp/QYfc/vcE9xsd2koUr9D8vgq+0HgzCRlSVUcY68jcCajp4BS7AdrRnmnA1G VxYSDN4SDAOddKyxqvCg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhb-0000000ATjS-1HkO; Wed, 27 Mar 2024 18:08:43 +0000 Received: from mail-dm6nam10on20604.outbound.protection.outlook.com ([2a01:111:f400:7e88::604] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhS-0000000ATY9-0UAX for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:35 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=l4QQZBEUiIUJkHOYZ52XIEm9WkdAqGPwQcNZ2mLgJ/XxyRWiO7kQf/6AnC9HQrrOPnZBynp/uVZu+58y+Tt0n7h8w2YWL5mEN6E5WgpwaBU2c16d9Ph5mhwgjvireoCcLTtF+L/66U0s78JF38y1djwqo6BYxdSorQodIgjWPStRF9HIeuzp/gQTJJZxKAG1MFwS22MtmEnDM0/Hbr5r3NaBsm2CGlIoDPOrP1FYNlZN+gZtneCl96BKEOuMuFnioZYJ8WlL2CJO/9nn6YL5rgBtdYe4F7khuj7RPDaeDvuX4Y+usbjS0J2TK5hPz7bf+SOqjqWima0dP5OL6yLFSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=w7FRZs9o5O65w7A5fHtmKCVArx1qpU0Mag+3cPgbb/k=; b=fkjOrJ6BPU2g+v14gHqlrd6ylqmCS+VV36q80YAb5OtLZNzqBaJFQQ2MqWPDW0MThT4MxCehpHarw/HwzrqWQPwiCiyqpkDRNIa4wDk5TsMGXj16sP2UX97ikyyBgy9YKzgGKq3QN4T5dXoBJQWlQlvP2KiANvV9YAOGXfTINUS1CEQoj56IiXR34jNa90tefMXVnBEobNQ+Q7olmm9PmmpebWObV8bXov6QtX6JIozIR5pFHY4ATJctTi85NggJrvoxJqnm7JO6jD/wWM0fLSWf1DSeuca0co8FTijrs7U+2g8iVFN2uGdm55MuahBr8sVlTLinKkt1crPhlycRxQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=w7FRZs9o5O65w7A5fHtmKCVArx1qpU0Mag+3cPgbb/k=; b=AoRyAWdL0Q6OjX3uUPCehh3EtwfRKTaXDbehCTOTDIDQUR8Q+QFaEx3Xk10iZ8V7jwNx+WuHNERVykLb4aRwsB9khm+fd34lLrcR0kEg1Lz2SPDuq2YyO9fwlVZoXHh3V+/Xa4grXExt41yoYw1DMOnLdv0xWioLscRdZpCng8WUaS27UOs9D1F8kP+t5JPtO75qKs+ks1lQZWzjcaqae7OPHdBDA4E2XnHsGd2oeTnQJ786gLrLhs11kz9UW79jFccqGjec8QyM/xwlD9IkUIxOalfHAY8TPsa4wiOWW8merDbQB1/1GUESLMqyDREUrvzADbOZdoHLI53tAVFXZQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:19 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:19 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 22/29] iommu: Add ops->domain_alloc_sva() Date: Wed, 27 Mar 2024 15:08:08 -0300 Message-ID: <22-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR20CA0002.namprd20.prod.outlook.com (2603:10b6:208:e8::15) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 0d41a608-348b-40aa-e0d3-08dc4e88e0a6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2AKnZS+74mTCOI4SpDIpJ7EJj0CejOqX0+nTggYOMzqRni/et+OwGYrFneeGuWCcE+hQ2XRzDJ3xBYeUCArfEke8AXexWKKdswNGW/kw6K6slT70lnXYO5lbwQSz8fWyhWWEnUPMmE7OLc3KZ2duGQkTj/FPP/38j/PCJ4OjYBo1nlRsuo0eBZu8JP29zkVjixbTnWVZjfpsQO4tEdoQ4phd6A5f9HKVRGaZNFocggrrXH1FdFJJgIp4I3hywOMbjJ582yOvotBlaf6F6+fyrkDmT3OcCSsSkKkWXFLSJqIjp+XJDN6HYeHoyPeXnGd+b0oBftoRmwLRVuteWmcPNI3RRyJL0Ze7QDUdHtdsVgmijx1AbDxD24ap4pdHPcgkwZBxXWfT2EBQJJZP+/ddtNAhIXXXDOemmDD+iwikq+BictmDEnEfQ5wXyjzAszvke4NCByzXnLNsP+ZhSALpOwfSOJqZPPBy8hOxCERU9J2GMGrgFpgm4AClGQ+VYiDCHdmVoPapqbw/wp0N0NuJz8YXmFMeOAr8KeYaK/VAxwrisgA2F/CWu6nS0nGjxH3QPIuWO9Zm0kglCKKH3sNqH4cOGsHhQZW283KOMRyxSiY1gZsdtn3HqnghDqjjUOmcGQmwJrFf8dv8BrUc1dvaQffX5KO//wYAsWUDeQpNh3g= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: EBCsPcwmwoAiPuPDF59TBGdUyeLg5cAnEkU1a3hLZXNJIAdOgGFvaT6wfADQFBPW/6CEHwd9X/6BZxWvXLOTZdqVhLB63ZkDC6ZFytiEIlGWZ8gcJshw02jJBbVCIsgheqMTtKDb/8LhYCl/L1TDFkwwbYqRjCMzv8Re+thMkeolbinXS7oVNv8/PKXxx9lls/oryZuz+bbV1Nj1S4RhKjFxtDEfGGb5l4JT/1TZPt6ZlP1RFLlY6A5lFd/lCMkbc21E5L7JEtNXCpiWCxXfJjVZnMmpQz9EMZ8VuyTunqZoX9Zk83oFH0G+ofUYrVlVFmgAjJS/4jq48HUc9Z6DwlxlGGVgeS4XTXvHe6E+boFEXUKA+2nN4MLS7iO9/2cVkov2G2Qyd8B7pmbOtfevqxvyrcqozgPTaCitQgsQAAQIbiSGajHWVz0c8dxWVxPBnnD+HDwXtQX0Wy3jkKDg4VoK1u2umkcyVfcm3t5HbGmGX2qb188HVDNhCP4jWgVdD6MpbDoPNnSHBdHxwdiJZXrYqV2/OEEdTq7V+YJLtElN/0O7kS4vXAIFt7eXoE5N+AOF6WqfvqmVI8MIfAeod/RaoDZ67HA+JmRw2JuQl9EviAGzv9ZDi76GJm8exQH6EXvnkoeuM+vTTThzVRGO7qCJYHWNPR49ZDe9ntAIczcN+G8BnhIjA/3Dt0q/BWkm2cuep3BTKYADGfhZpKQMecFJnZr9HNMxkqk1rbeutRW1Vlnem+Jtjo1xFyHW2F2FBR2tmrtBsjJ4+ILuEdj1S+2JZJ8sRXB+8CBh7qZQXDAoaqP+3kwJ8VIVlB1gQumSNVYwbYZJIBBr7g2Mqbf70UtrxoTqKU7S8WJvQor3FQJ9dlD0inovpXsTED06+SbmI3ZoagFR7ewYM7tlOnoYKBxEz9V/+0uCgFy1SRx0aMRjwWiT9GVWBRq3FnXjSsl2uOLtkkdhqW3u/zN3Xa3ATT5OEHPZXrZ4jGibjdROpm5p0NHQxksU8neAYQX0zgbAjwS3Xlg0gFnUYuLQmjM14m3CNigk73TLqJi2g3Dn973lU6vqu3CasCZTSbgyjbQCovIXCVAYkWwAg3fh1fQ3WImsKXnWLXmNQYsLvqXkR3ZJ9APZjNBm2EUFrUDTGsJHtX+jbvZNhYipwdEmsXECb8tnGj+wcqWHOL2V7W5rd6DgpYmtNlFTRlXDEjNYV0G9VJfII0ItyoPVQsQ4zEn8OYmnrKStNcQ+MIwaQCdzJUcs1aJsu13k4wK+V1h2C2ZkzvzcKbV/VGFWg9uj6wNKgN8a3hvAbHBmypKVhOaoPOhcm8pOBJbbdnok9Hhs6O6P/EazIcaGszi3NpixIfjr2d7Z0YciUauU636PkAvPnp9rBzBGU1agco/POeCFHUCow1Mjno3cusDMQxCr2Ho4sey/UZdMnc2DcLmLVFmvwH16BfbYyEcKC2SZ6/RcC1rWkReirT9FDA4qQmve4FW+eNZWhLz75bXgpaGeY4FP1TySGMcL7AYrmx4HQMJuoG5n3DOHoncEW3tVdkQ/5Zhlpn+rgVYtbaM6rcC+oRYmXNXKoK9oGLLv3JDLRlDL6gwz X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0d41a608-348b-40aa-e0d3-08dc4e88e0a6 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:17.3534 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cfM/cUhJRjO/RLvZNr+bkEdVGJwVrBYd5npSkvCb8nv1NYhUDcl640BAwjQeoH0I X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110834_222415_9C75505A X-CRM114-Status: GOOD ( 20.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Make a new op that receives the device and the mm_struct that the SVA domain should be created for. Unlike domain_alloc_paging() the dev argument is never NULL here. This allows drivers to fully initialize the SVA domain and allocate the mmu_notifier during allocation. It allows the notifier lifetime to follow the lifetime of the iommu_domain. Since we have only one call site, upgrade the new op to return ERR_PTR instead of NULL. Change SMMUv3 to use the new op. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 +++++++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 2 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +++++- drivers/iommu/iommu-sva.c | 16 +++++++++++----- include/linux/iommu.h | 3 +++ 5 files changed, 27 insertions(+), 11 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 3e7aad0960bfd2..e337e40ac5de31 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -659,17 +659,20 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = { .free = arm_smmu_sva_domain_free }; -struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type) +struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, + struct mm_struct *mm) { + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_domain *smmu_domain; - if (type != IOMMU_DOMAIN_SVA) - return ERR_PTR(-EOPNOTSUPP); - smmu_domain = arm_smmu_domain_alloc(); if (IS_ERR(smmu_domain)) return ERR_CAST(smmu_domain); + + smmu_domain->domain.type = IOMMU_DOMAIN_SVA; smmu_domain->domain.ops = &arm_smmu_sva_domain_ops; + smmu_domain->smmu = smmu; return &smmu_domain->domain; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 3d9109ad60c19c..7b001afda17aa8 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3302,8 +3302,8 @@ static struct iommu_ops arm_smmu_ops = { .identity_domain = &arm_smmu_identity_domain, .blocked_domain = &arm_smmu_blocked_domain, .capable = arm_smmu_capable, - .domain_alloc = arm_smmu_sva_domain_alloc, .domain_alloc_paging = arm_smmu_domain_alloc_paging, + .domain_alloc_sva = arm_smmu_sva_domain_alloc, .probe_device = arm_smmu_probe_device, .release_device = arm_smmu_release_device, .device_group = arm_smmu_device_group, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 9db84d5940466a..107a39f1dfe869 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -791,7 +791,8 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master); int arm_smmu_master_disable_sva(struct arm_smmu_master *master); bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master); void arm_smmu_sva_notifier_synchronize(void); -struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type); +struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, + struct mm_struct *mm); void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t id); #else /* CONFIG_ARM_SMMU_V3_SVA */ @@ -837,5 +838,8 @@ static inline void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, ioasid_t id) { } + +#define arm_smmu_sva_domain_alloc NULL + #endif /* CONFIG_ARM_SMMU_V3_SVA */ #endif /* _ARM_SMMU_V3_H */ diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 640acc804e8cdc..18a35e798b729c 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -108,8 +108,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm /* Allocate a new domain and set it on device pasid. */ domain = iommu_sva_domain_alloc(dev, mm); - if (!domain) { - ret = -ENOMEM; + if (IS_ERR(domain)) { + ret = PTR_ERR(domain); goto out_free_handle; } @@ -283,9 +283,15 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev, const struct iommu_ops *ops = dev_iommu_ops(dev); struct iommu_domain *domain; - domain = ops->domain_alloc(IOMMU_DOMAIN_SVA); - if (!domain) - return NULL; + if (ops->domain_alloc_sva) { + domain = ops->domain_alloc_sva(dev, mm); + if (IS_ERR(domain)) + return domain; + } else { + domain = ops->domain_alloc(IOMMU_DOMAIN_SVA); + if (!domain) + return ERR_PTR(-ENOMEM); + } domain->type = IOMMU_DOMAIN_SVA; mmgrab(mm); diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 2e925b5eba534c..8aabe83af8f266 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -518,6 +518,7 @@ static inline int __iommu_copy_struct_from_user_array( * Upon failure, ERR_PTR must be returned. * @domain_alloc_paging: Allocate an iommu_domain that can be used for * UNMANAGED, DMA, and DMA_FQ domain types. + * @domain_alloc_sva: Allocate an iommu_domain for Shared Virtual Addressing. * @probe_device: Add device to iommu driver handling * @release_device: Remove device from iommu driver handling * @probe_finalize: Do final setup work after the device is added to an IOMMU @@ -558,6 +559,8 @@ struct iommu_ops { struct device *dev, u32 flags, struct iommu_domain *parent, const struct iommu_user_data *user_data); struct iommu_domain *(*domain_alloc_paging)(struct device *dev); + struct iommu_domain *(*domain_alloc_sva)(struct device *dev, + struct mm_struct *mm); struct iommu_device *(*probe_device)(struct device *dev); void (*release_device)(struct device *dev); From patchwork Wed Mar 27 18:08:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8F060C47DD9 for ; Wed, 27 Mar 2024 19:18:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JmDXvAlA6EBSwUQDxbBlcJ71t/jESlc04tW3pEwqSbE=; b=f1WbsphecZqMnL ZpCIhXZ+aQX2omdgftRjugiKLwdrS4sPLjiM7DRuWVAO7KWABzivSVYj83CiflAYn3wm+fSAqqCCB g5jaToM+Ioy30IDGPnXg8DqoaZUw+9+jnuSFzMO5nqpn9+TCriJsYIv5o635V1xGrCCnnmryYwZoN OrKZGC518beXgIiucBx9WzMc6OodZND38gAtzMdGfLToxBwYFLY8qoofMkdsx4vO7pFC5qA0DWWVt YLVu+RIQrmE9Wf6PKrS6KVFat4m1O5AgiHq34kAf4jvy2EA6AiCe+EkZQN8/6Mb1P51XD84no5MO6 Xv4TkTGRiT8G+V3FoqxQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYnA-0000000Al1y-0rXD; Wed, 27 Mar 2024 19:18:32 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhx-0000000AU55-1bfJ for linux-arm-kernel@bombadil.infradead.org; Wed, 27 Mar 2024 18:09:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=BtmEO240KNaODknEMwPSl9Gj5wYZzHrGdqD7hDx3g0E=; b=FM5Pcs6NfzOjtuo4PWt+cpefa5 mmSwyOzuAQeGPltRHMF2X3MMJpZ0Dh+4YPvcvCEl2LKupgZWe+70ybGR5xf+7WOh06UvOk24Dfa0A vbstcbrYwEmzB7WG42uBHH4d681LsL5VxDtP+ezjCGyVG8Di14tarUbuBSdQ4Eh6nNf5BNf08VSA/ f/7VHno81vjxK7rEOUB9PG3imrcwDHiGhYhfmZTe7UsRsV4Rn5olRu6q102jk1vNU9R0583Ol4igj QlzSGCRFBCm1GjK/5FYWgXTPIwa+fiyNWhlGQIm+UF9OBZJ7k6FQJK69yh+OZyYQZp6a9cOu3MLvj yFBE8qLg==; Received: from mail-dm6nam10on20605.outbound.protection.outlook.com ([2a01:111:f400:7e88::605] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by casper.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhu-00000004QF8-1Fsf for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:09:04 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lEFCRLkOazLrvE09ZLjmqFm5Ufa4EYqqM067zcq4NZI+6gnZyar7dEN7kJZkqd2CghpKO9DfD1lENcIxl4UV93t4Oh07WWIdiFK1V+YEKBkE45Ue0ZaZm5iE/0CT0598JNaB5GYX5Vn4iGf5F7EmUQan10MsGV7vllbP078fXQ3MLhEK7blgmgjmjE4YBUI0u5wzlTnflKfuKmL78EQpX7pBQj4m95Yj/CiyHGiq2bxm1FtubOmBtk3YulD4feDtNZ/ZjjFk6oAwoh1JrZVOhf9TdapdqqB3oslqmEkGKPwRmjy1SA2pS2rkdPgWx1LVa1uukPGgRBeqpU+pd85A5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BtmEO240KNaODknEMwPSl9Gj5wYZzHrGdqD7hDx3g0E=; b=hkbe8tiO5J7I+WEYrKDPUidSi/j7yFeDTZYxDlKPpNnLEJI+y2Si1XbR6UwRS6hsFSiQuPWw8zt9+ywL5JupVSkFbpYm2hJGPhU9PBptvVPVgiQo0tmoz86rWdylsXevrdG6mL5U6r1VsxgaRXbVY6ihpV+toABNSw5z+WFn1/HqYKSbcAL3oSHesST2LrvJoAMfqgtYukA4TbdC1CTdBH2197NrAiwrBe6kysDijTdoA+73/aqj/o807NK+JCGw5hvtUTiuf81yr39VLGrxynnEZZQaxLsMkzAt7qJWl3FKZpGuCbZb3KAK+dZIW6wmxENRE6AclKMjk+4ZzQwYHA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BtmEO240KNaODknEMwPSl9Gj5wYZzHrGdqD7hDx3g0E=; b=tH4BH694SCBZCR86aJfJOq+Cvh68ayJnvRPQYJvkEEm446+i4Iwlhaxwcl5MjCIL6nx+X5Fg9QiSMlo7mtChi09YYSGKL0EzjZbFesI+2ZyOUnitWqZjYHJd0I9botgnKxEkz4+2qIBYuBNUTHn37fF/SAuXegZGMDUiAYm6a1YIBNOkJpBj24E8R2m3+lDHDR71XsHeMIqNhFFTY/nUFRC7AJxhCOY6R4DuioYbYQ2lWi3uyQz18HglAQ/hFY2e4fn7XFFweKMhKYAPJPG4wVcNP+5i+bcqMOT2fabwEANdWEF+ZgkzNo/kG16sbWOc+2VNzssEe3qvESYhkaNbqg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by IA1PR12MB6044.namprd12.prod.outlook.com (2603:10b6:208:3d4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.33; Wed, 27 Mar 2024 18:08:34 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:34 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 23/29] iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain Date: Wed, 27 Mar 2024 15:08:09 -0300 Message-ID: <23-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0050.namprd05.prod.outlook.com (2603:10b6:208:236::19) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|IA1PR12MB6044:EE_ X-MS-Office365-Filtering-Correlation-Id: dfea69fc-c8b1-4cf2-93bf-08dc4e88e237 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: j6xdSwg4mLIxACbGa7qBF3lLoFsFBihP+uM/68k4U25DqQ9DHy6BLW4+0rKl1QgdfQT5wUj8q48a5Brpcia5icOmis2h8mfxa0ho0GRGaSKjTkIBesCGo37AAOECkx3Te4KNdhaliIyFTRVB5AWCFkGbtJGzph93nOPcscfkoNljhZ2lWGToLbTvqNLjK7x8wpUZ2hFL1Lh0AZjLMpslpLdKjycvAhnIBdVft7COkX4ZXwyZWMb7mn4F5UYFVBPMIrGkLOT7zRhpzp7X25yJLTJQNgu32TPkUXt76SdUm50+tttB0R0Yp2N6W2vVX9IhFmCST3VilD7/91l+zQm5kqeIi2RdXikETdkpAhu8W+XaxzrAz1pozPg9QWKbXKItY7CdzTMF3ABaou6d4sBxSKidDNX5u2z/AdUHrORd7S60pIJ54XviZfrOtI3Ai+tCpyx+B6bHtRNYeTNQDhyFk3tIQxaXYnC5yWnG+NgfFKiNdGSPIug1ZVf9qHiyh/7NVtCJzr13ihwc2muV1POdA93WxED8QSshwsls0iEOPL2rJKzcMS5fVWdEC/yUjm2NhTo4R1zTC9E/aMU3PwpEcDh4wCiNls/hs4qcDIs0iVJbSc1a/wU2W2/DOmxyZUP+w8liqLu1QixkjDOlnEFNMRwn2RYFncnE2vfDQqOYk7s= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7416005)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ZpcMsG1mLc5dx0yIYVPvAjH4Viu/wRKFJmeVlZQ4Q+vGBeIP8nGElNiyOB62dAFfxMrN6Fxue6tDC1x0+oC4mVGxLjjrjnTeV1ErYDEaRoC37M8LwyMAomXBvIWHD0WoUuN3+R//uilulWEfAnyd/IZ9byt6jB3EDbE6pE1vTHxWUw7HmRaVEL1X0kYy42G8jNsjjmxpv10Lr1tioqljX808gJkwH0BTFjorYUtiMn7zBawdRSjlyKPV9/kxJAMHmj1dNBJeERTKIMTwSGPhpTUtrko8srlWu8eBsgsfPdIb4cWDpVk62a6L4BwIkQpYsTpfHAyYUy4cYfA/btedb2iVcG3G7KKPAXgDcC/1tsXGKVMigVV/uQ6guAdA1D1N+AyQ4wENYX+2lJVyK3Elr1mk3kLarlCF1XV3eqFuPzhC/Fqzu2lySGUcUW5rd0D8Rzai/0sM+LPixvDpSRmPtnej4W9dF2D8pkO/KlMl7oKjkTy/LaCjf1u3+Im4mFKjuQ4nUxrFZOJzJCcS/LTgqyHBYxiANsFZspWlFggqICVN9KNjaFsl8BvprcybhHOyDv/b84jX6MvNnfN4f6kgUEiBgbJpaem92MeaJYw7aeSWcyyXy6Gfi/ObEHbg85o6/GrHoFoa1f8OZUc7VCvDtwuuf0vG2yK4Yhur/mU3CSbl9wyQdwUZ47QyQcaM+Ehyzmjqbfie3sIcCu6SYW0vubxpfZq+AAs3baQt2WtyGNQY96sHwTkunxGT63RbmM/pwK42MqHK4bxmvWHUg4EpTX0hmHkoGnuqMx6a/NJ3OHvaqnBgh01dvbtXG0ULRz9XDlQ6CXw1ZTL//4SjMEGDvDgTmPBuU9gFDpSrpKJF7UIMaeAMihoBKMB9yz+ONIu3G1liX/LBH8Gy+7Uk+yTRNOrSm+27gs2eq87DD1zO8VWSkgpsRKeAFfn9Qu1CCVvOVB1Ezu4OYxoHX4HmG7ccfWwLyOK0/KnVXbIQp1IZFWgeNhMCo9adQAjlHpnxAHEpOPNqZXdjFTRv5vbd4iYiDy7QCs1OHu/0loJ68CE6E9ELJhDFw2ClpDgBNdGY9ARSVfhchIJhuMViZAgGZ423ExKB3gqlarpWYU2K/lquFlieOpW3yJ7L//Wtads2iTtFeRHMkzF3Dt6csp23V0+JgdmwLhTjmiZcMuxihe/8rI+7hn+XKQgWZ1nq99zOXTrrGSOgugogud4BSCYrmt133r6sHlUz/3utayeCR2OOVJ9RrUseQInwBJA24lTxTgHw7IxkzGfIXGTRzYWYhtZ+Y5zEPhReDoSOfNmmifKmUGwuf8KGnCyQ8dKtcEU3sMXOrOWb7f+F24q9IFf2alvrs9oT1cnzo7XFSiUs9wIIKGQ7Y1uw9BEuW40SMkk9V9HPeKkLG3ayZX3KD1i2YgGhOeq8Cy6NLHBmtw4DmmVsMhQepEHChI0Ej/IKgEAoHUGECjc7qdrvSkLaX0/frOKY1eyz2F3y+JQ7SAuoz2Qk8NlQXpSBx8aANSwvrSG72aOQjNgWSwye1Y6L1ZVKjP1qLeYwAvqlfJHbSGHO6wax36GCJbA/rXIvsavZTrowjr9o X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: dfea69fc-c8b1-4cf2-93bf-08dc4e88e237 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:19.9845 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XZu06/4ZtsXbW9fAOhDpYAO0UUzSYG/KSGWCQrnvn9mPL/VZjZhnV1f2sqL+gyC9 X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6044 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_180902_387192_B2DA96DC X-CRM114-Status: GOOD ( 20.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This removes all the notifier de-duplication logic in the driver and relies on the core code to de-duplicate and allocate only one SVA domain per mm per smmu instance. This naturally gives a 1:1 relationship between SVA domain and mmu notifier. Remove all of the previous mmu_notifier, bond, shared cd, and cd refcount logic entirely. For the purpose of organizing patches lightly remove BTM support. The next patches will add it back in. BTM is a performance optimization so this is bisection friendly functionally invisible change. Note that the current code never even enables ARM_SMMU_FEAT_BTM so none of this code is ever even used. The bond/shared_cd/btm/asid allocator are tightly wound together and changing them all at once would make this patch too big. The core issue is that having a single SVA domain per-smmu instance conflicts with the design of having a global ASID table that BTM currently needs, as we would end up having to assign multiple SVA domains to the same ASID. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 396 ++++-------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 78 +--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +- 3 files changed, 97 insertions(+), 391 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index e337e40ac5de31..fd0b9f230f89e3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -12,29 +12,9 @@ #include "arm-smmu-v3.h" #include "../../io-pgtable-arm.h" -struct arm_smmu_mmu_notifier { - struct mmu_notifier mn; - struct arm_smmu_ctx_desc *cd; - bool cleared; - refcount_t refs; - struct list_head list; - struct arm_smmu_domain *domain; -}; - -#define mn_to_smmu(mn) container_of(mn, struct arm_smmu_mmu_notifier, mn) - -struct arm_smmu_bond { - struct mm_struct *mm; - struct arm_smmu_mmu_notifier *smmu_mn; - struct list_head list; -}; - -#define sva_to_bond(handle) \ - container_of(handle, struct arm_smmu_bond, sva) - static DEFINE_MUTEX(sva_lock); -static void +static void __maybe_unused arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) { struct arm_smmu_master_domain *master_domain; @@ -57,58 +37,6 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); } -/* - * Check if the CPU ASID is available on the SMMU side. If a private context - * descriptor is using it, try to replace it. - */ -static struct arm_smmu_ctx_desc * -arm_smmu_share_asid(struct mm_struct *mm, u16 asid) -{ - int ret; - u32 new_asid; - struct arm_smmu_ctx_desc *cd; - struct arm_smmu_device *smmu; - struct arm_smmu_domain *smmu_domain; - - cd = xa_load(&arm_smmu_asid_xa, asid); - if (!cd) - return NULL; - - if (cd->mm) { - if (WARN_ON(cd->mm != mm)) - return ERR_PTR(-EINVAL); - /* All devices bound to this mm use the same cd struct. */ - refcount_inc(&cd->refs); - return cd; - } - - smmu_domain = container_of(cd, struct arm_smmu_domain, cd); - smmu = smmu_domain->smmu; - - ret = xa_alloc(&arm_smmu_asid_xa, &new_asid, cd, - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); - if (ret) - return ERR_PTR(-ENOSPC); - /* - * Race with unmap: TLB invalidations will start targeting the new ASID, - * which isn't assigned yet. We'll do an invalidate-all on the old ASID - * later, so it doesn't matter. - */ - cd->asid = new_asid; - /* - * Update ASID and invalidate CD in all associated masters. There will - * be some overlap between use of both ASIDs, until we invalidate the - * TLB. - */ - arm_smmu_update_s1_domain_cd_entry(smmu_domain); - - /* Invalidate TLB entries previously associated with that context */ - arm_smmu_tlb_inv_asid(smmu, asid); - - xa_erase(&arm_smmu_asid_xa, asid); - return NULL; -} - static u64 page_size_to_cd(void) { static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K || @@ -122,7 +50,8 @@ static u64 page_size_to_cd(void) static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, struct arm_smmu_master *master, - struct mm_struct *mm, u16 asid) + struct mm_struct *mm, u16 asid, + bool btm_invalidation) { u64 par; @@ -143,7 +72,7 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, (master->stall_enabled ? CTXDESC_CD_0_S : 0) | CTXDESC_CD_0_R | CTXDESC_CD_0_A | - CTXDESC_CD_0_ASET | + (btm_invalidation ? 0 : CTXDESC_CD_0_ASET) | FIELD_PREP(CTXDESC_CD_0_ASID, asid)); /* @@ -185,69 +114,6 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, target->data[3] = cpu_to_le64(read_sysreg(mair_el1)); } -static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) -{ - u16 asid; - int err = 0; - struct arm_smmu_ctx_desc *cd; - struct arm_smmu_ctx_desc *ret = NULL; - - /* Don't free the mm until we release the ASID */ - mmgrab(mm); - - asid = arm64_mm_context_get(mm); - if (!asid) { - err = -ESRCH; - goto out_drop_mm; - } - - cd = kzalloc(sizeof(*cd), GFP_KERNEL); - if (!cd) { - err = -ENOMEM; - goto out_put_context; - } - - refcount_set(&cd->refs, 1); - - mutex_lock(&arm_smmu_asid_lock); - ret = arm_smmu_share_asid(mm, asid); - if (ret) { - mutex_unlock(&arm_smmu_asid_lock); - goto out_free_cd; - } - - err = xa_insert(&arm_smmu_asid_xa, asid, cd, GFP_KERNEL); - mutex_unlock(&arm_smmu_asid_lock); - - if (err) - goto out_free_asid; - - cd->asid = asid; - cd->mm = mm; - - return cd; - -out_free_asid: - arm_smmu_free_asid(cd); -out_free_cd: - kfree(cd); -out_put_context: - arm64_mm_context_put(mm); -out_drop_mm: - mmdrop(mm); - return err < 0 ? ERR_PTR(err) : ret; -} - -static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) -{ - if (arm_smmu_free_asid(cd)) { - /* Unpin ASID */ - arm64_mm_context_put(cd->mm); - mmdrop(cd->mm); - kfree(cd); - } -} - /* * Cloned from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, this * is used as a threshold to replace per-page TLBI commands to issue in the @@ -262,8 +128,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, unsigned long start, unsigned long end) { - struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); - struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + struct arm_smmu_domain *smmu_domain = + container_of(mn, struct arm_smmu_domain, mmu_notifier); size_t size; /* @@ -280,34 +146,27 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, size = 0; } - if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM)) { + if (!smmu_domain->btm_invalidation) { if (!size) arm_smmu_tlb_inv_asid(smmu_domain->smmu, - smmu_mn->cd->asid); + smmu_domain->cd.asid); else arm_smmu_tlb_inv_range_asid(start, size, - smmu_mn->cd->asid, + smmu_domain->cd.asid, PAGE_SIZE, false, smmu_domain); } - arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), start, - size); + arm_smmu_atc_inv_domain(smmu_domain, start, size); } static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { - struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); - struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + struct arm_smmu_domain *smmu_domain = + container_of(mn, struct arm_smmu_domain, mmu_notifier); struct arm_smmu_master_domain *master_domain; unsigned long flags; - mutex_lock(&sva_lock); - if (smmu_mn->cleared) { - mutex_unlock(&sva_lock); - return; - } - /* * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, * but disable translation. @@ -319,25 +178,27 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) struct arm_smmu_cd target; struct arm_smmu_cd *cdptr; - cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm)); + cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid); if (WARN_ON(!cdptr)) continue; - arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid); - arm_smmu_write_cd_entry(master, mm_get_enqcmd_pasid(mm), cdptr, + arm_smmu_make_sva_cd(&target, master, NULL, + smmu_domain->cd.asid, + smmu_domain->btm_invalidation); + arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr, &target); } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); - arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0); - - smmu_mn->cleared = true; - mutex_unlock(&sva_lock); + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); + arm_smmu_atc_inv_domain(smmu_domain, 0, 0); } static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) { - kfree(mn_to_smmu(mn)); + struct arm_smmu_domain *smmu_domain = + container_of(mn, struct arm_smmu_domain, mmu_notifier); + + kfree(smmu_domain); } static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { @@ -346,115 +207,6 @@ static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { .free_notifier = arm_smmu_mmu_notifier_free, }; -/* Allocate or get existing MMU notifier for this {domain, mm} pair */ -static struct arm_smmu_mmu_notifier * -arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, - struct mm_struct *mm) -{ - int ret; - struct arm_smmu_ctx_desc *cd; - struct arm_smmu_mmu_notifier *smmu_mn; - - list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) { - if (smmu_mn->mn.mm == mm) { - refcount_inc(&smmu_mn->refs); - return smmu_mn; - } - } - - cd = arm_smmu_alloc_shared_cd(mm); - if (IS_ERR(cd)) - return ERR_CAST(cd); - - smmu_mn = kzalloc(sizeof(*smmu_mn), GFP_KERNEL); - if (!smmu_mn) { - ret = -ENOMEM; - goto err_free_cd; - } - - refcount_set(&smmu_mn->refs, 1); - smmu_mn->cd = cd; - smmu_mn->domain = smmu_domain; - smmu_mn->mn.ops = &arm_smmu_mmu_notifier_ops; - - ret = mmu_notifier_register(&smmu_mn->mn, mm); - if (ret) { - kfree(smmu_mn); - goto err_free_cd; - } - - list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers); - return smmu_mn; - -err_free_cd: - arm_smmu_free_shared_cd(cd); - return ERR_PTR(ret); -} - -static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) -{ - struct mm_struct *mm = smmu_mn->mn.mm; - struct arm_smmu_ctx_desc *cd = smmu_mn->cd; - struct arm_smmu_domain *smmu_domain = smmu_mn->domain; - - if (!refcount_dec_and_test(&smmu_mn->refs)) - return; - - list_del(&smmu_mn->list); - - /* - * If we went through clear(), we've already invalidated, and no - * new TLB entry can have been formed. - */ - if (!smmu_mn->cleared) { - arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); - arm_smmu_atc_inv_domain_sva(smmu_domain, - mm_get_enqcmd_pasid(mm), 0, 0); - } - - /* Frees smmu_mn */ - mmu_notifier_put(&smmu_mn->mn); - arm_smmu_free_shared_cd(cd); -} - -static struct arm_smmu_bond *__arm_smmu_sva_bind(struct device *dev, - struct mm_struct *mm) -{ - int ret; - struct arm_smmu_bond *bond; - struct arm_smmu_master *master = dev_iommu_priv_get(dev); - struct iommu_domain *domain = iommu_get_domain_for_dev(dev); - struct arm_smmu_domain *smmu_domain; - - if (!(domain->type & __IOMMU_DOMAIN_PAGING)) - return ERR_PTR(-ENODEV); - smmu_domain = to_smmu_domain(domain); - if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1) - return ERR_PTR(-ENODEV); - - if (!master || !master->sva_enabled) - return ERR_PTR(-ENODEV); - - bond = kzalloc(sizeof(*bond), GFP_KERNEL); - if (!bond) - return ERR_PTR(-ENOMEM); - - bond->mm = mm; - - bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm); - if (IS_ERR(bond->smmu_mn)) { - ret = PTR_ERR(bond->smmu_mn); - goto err_free_bond; - } - - list_add(&bond->list, &master->bonds); - return bond; - -err_free_bond: - kfree(bond); - return ERR_PTR(ret); -} - bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) { unsigned long reg, fld; @@ -571,11 +323,6 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master) int arm_smmu_master_disable_sva(struct arm_smmu_master *master) { mutex_lock(&sva_lock); - if (!list_empty(&master->bonds)) { - dev_err(master->dev, "cannot disable SVA, device is bound\n"); - mutex_unlock(&sva_lock); - return -EBUSY; - } arm_smmu_master_sva_disable_iopf(master); master->sva_enabled = false; mutex_unlock(&sva_lock); @@ -592,66 +339,52 @@ void arm_smmu_sva_notifier_synchronize(void) mmu_notifier_synchronize(); } -void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, - struct device *dev, ioasid_t id) -{ - struct mm_struct *mm = domain->mm; - struct arm_smmu_bond *bond = NULL, *t; - struct arm_smmu_master *master = dev_iommu_priv_get(dev); - - arm_smmu_remove_pasid(master, to_smmu_domain(domain), id); - - mutex_lock(&sva_lock); - list_for_each_entry(t, &master->bonds, list) { - if (t->mm == mm) { - bond = t; - break; - } - } - - if (!WARN_ON(!bond)) { - list_del(&bond->list); - arm_smmu_mmu_notifier_put(bond->smmu_mn); - kfree(bond); - } - mutex_unlock(&sva_lock); -} - static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t id) { + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct arm_smmu_master *master = dev_iommu_priv_get(dev); - struct mm_struct *mm = domain->mm; - struct arm_smmu_bond *bond; struct arm_smmu_cd target; int ret; - if (mm_get_enqcmd_pasid(mm) != id || !master->cd_table.in_ste) + /* Prevent arm_smmu_mm_release from being called while we are attaching */ + if (!mmget_not_zero(domain->mm)) return -EINVAL; - mutex_lock(&sva_lock); - bond = __arm_smmu_sva_bind(dev, mm); - if (IS_ERR(bond)) { - mutex_unlock(&sva_lock); - return PTR_ERR(bond); - } + /* + * This does not need the arm_smmu_asid_lock because SVA domains never + * get reassigned + */ + arm_smmu_make_sva_cd(&target, master, domain->mm, smmu_domain->cd.asid, + smmu_domain->btm_invalidation); + ret = arm_smmu_set_pasid(master, smmu_domain, id, &target); - arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid); - ret = arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target); - if (ret) { - list_del(&bond->list); - arm_smmu_mmu_notifier_put(bond->smmu_mn); - kfree(bond); - mutex_unlock(&sva_lock); - return ret; - } - mutex_unlock(&sva_lock); - return 0; + mmput(domain->mm); + return ret; } static void arm_smmu_sva_domain_free(struct iommu_domain *domain) { - kfree(to_smmu_domain(domain)); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + /* + * Ensure the ASID is empty in the iommu cache before allowing reuse. + */ + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); + + /* + * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can + * still be called/running at this point. We allow the ASID to be + * reused, and if there is a race then it just suffers harmless + * unnecessary invalidation. + */ + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); + + /* + * Actual free is defered to the SRCU callback + * arm_smmu_mmu_notifier_free() + */ + mmu_notifier_put(&smmu_domain->mmu_notifier); } static const struct iommu_domain_ops arm_smmu_sva_domain_ops = { @@ -665,6 +398,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_domain *smmu_domain; + u32 asid; + int ret; smmu_domain = arm_smmu_domain_alloc(); if (IS_ERR(smmu_domain)) @@ -674,5 +409,22 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, smmu_domain->domain.ops = &arm_smmu_sva_domain_ops; smmu_domain->smmu = smmu; + ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain, + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); + if (ret) + goto err_free; + + smmu_domain->cd.asid = asid; + smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops; + ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm); + if (ret) + goto err_asid; + return &smmu_domain->domain; + +err_asid: + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); +err_free: + kfree(smmu_domain); + return ERR_PTR(ret); } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 7b001afda17aa8..a82a5e4a13bb44 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1446,22 +1446,6 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_master *master) cd_table->cdtab = NULL; } -bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd) -{ - bool free; - struct arm_smmu_ctx_desc *old_cd; - - if (!cd->asid) - return false; - - free = refcount_dec_and_test(&cd->refs); - if (free) { - old_cd = xa_erase(&arm_smmu_asid_xa, cd->asid); - WARN_ON(old_cd != cd); - } - return free; -} - /* Stream table manipulation functions */ static void arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc) @@ -2021,8 +2005,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master, return arm_smmu_cmdq_batch_submit(master->smmu, &cmds); } -static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, - ioasid_t ssid, unsigned long iova, size_t size) +int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, + unsigned long iova, size_t size) { struct arm_smmu_master_domain *master_domain; int i; @@ -2060,15 +2044,7 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, if (!master->ats_enabled) continue; - /* - * Non-zero ssid means SVA is co-opting the S1 domain to issue - * invalidations for SVA PASIDs. - */ - if (ssid != IOMMU_NO_PASID) - arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd); - else - arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, - &cmd); + arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, &cmd); for (i = 0; i < master->num_streams; i++) { cmd.atc.sid = master->streams[i].id; @@ -2080,19 +2056,6 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds); } -static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, - unsigned long iova, size_t size) -{ - return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova, - size); -} - -int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain, - ioasid_t ssid, unsigned long iova, size_t size) -{ - return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size); -} - /* IO_PGTABLE API */ static void arm_smmu_tlb_inv_context(void *cookie) { @@ -2281,7 +2244,6 @@ struct arm_smmu_domain *arm_smmu_domain_alloc(void) mutex_init(&smmu_domain->init_mutex); INIT_LIST_HEAD(&smmu_domain->devices); spin_lock_init(&smmu_domain->devices_lock); - INIT_LIST_HEAD(&smmu_domain->mmu_notifiers); return smmu_domain; } @@ -2324,7 +2286,7 @@ static void arm_smmu_domain_free_paging(struct iommu_domain *domain) if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { /* Prevent SVA from touching the CD while we're freeing it */ mutex_lock(&arm_smmu_asid_lock); - arm_smmu_free_asid(&smmu_domain->cd); + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); mutex_unlock(&arm_smmu_asid_lock); } else { struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; @@ -2342,11 +2304,9 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, u32 asid; struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; - refcount_set(&cd->refs, 1); - /* Prevent SVA from modifying the ASID until it is written to the CD */ mutex_lock(&arm_smmu_asid_lock); - ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd, + ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); cd->asid = (u16)asid; mutex_unlock(&arm_smmu_asid_lock); @@ -2805,6 +2765,9 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, /* The core code validates pasid */ + if (smmu_domain->smmu != master->smmu) + return -EINVAL; + if (!master->cd_table.in_ste) return -ENODEV; @@ -2826,9 +2789,18 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, return ret; } -void arm_smmu_remove_pasid(struct arm_smmu_master *master, - struct arm_smmu_domain *smmu_domain, ioasid_t pasid) +static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) { + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct arm_smmu_domain *smmu_domain; + struct iommu_domain *domain; + + domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA); + if (WARN_ON(IS_ERR(domain)) || !domain) + return; + + smmu_domain = to_smmu_domain(domain); + mutex_lock(&arm_smmu_asid_lock); arm_smmu_clear_cd(master, pasid); if (master->ats_enabled) @@ -3105,7 +3077,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) master->dev = dev; master->smmu = smmu; - INIT_LIST_HEAD(&master->bonds); dev_iommu_priv_set(dev, master); ret = arm_smmu_insert_master(smmu, master); @@ -3287,17 +3258,6 @@ static int arm_smmu_def_domain_type(struct device *dev) return 0; } -static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) -{ - struct iommu_domain *domain; - - domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA); - if (WARN_ON(IS_ERR(domain)) || !domain) - return; - - arm_smmu_sva_remove_dev_pasid(domain, dev, pasid); -} - static struct iommu_ops arm_smmu_ops = { .identity_domain = &arm_smmu_identity_domain, .blocked_domain = &arm_smmu_blocked_domain, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 107a39f1dfe869..3516869954ea33 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc { struct arm_smmu_ctx_desc { u16 asid; - - refcount_t refs; - struct mm_struct *mm; }; struct arm_smmu_l1_ctx_desc { @@ -711,7 +708,6 @@ struct arm_smmu_master { bool stall_enabled; bool sva_enabled; bool iopf_enabled; - struct list_head bonds; unsigned int ssid_bits; }; @@ -740,7 +736,8 @@ struct arm_smmu_domain { struct list_head devices; spinlock_t devices_lock; - struct list_head mmu_notifiers; + struct mmu_notifier mmu_notifier; + bool btm_invalidation; }; struct arm_smmu_master_domain { @@ -779,9 +776,8 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, struct arm_smmu_domain *smmu_domain); -bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd); -int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain, - ioasid_t ssid, unsigned long iova, size_t size); +int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, + unsigned long iova, size_t size); #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); @@ -793,8 +789,6 @@ bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master); void arm_smmu_sva_notifier_synchronize(void); struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, struct mm_struct *mm); -void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, - struct device *dev, ioasid_t id); #else /* CONFIG_ARM_SMMU_V3_SVA */ static inline bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) { From patchwork Wed Mar 27 18:08:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607264 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7BB2C47DD9 for ; Wed, 27 Mar 2024 18:13:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=P+KY76JC5xlIBnzyL8Daz2SiI+CpLCwPp/Udmdhytns=; b=QlZj2K61eJQWSO 55hNLvhqrqC9kJg1lkHet06fFwp+O/MWUXsRmnKDYEOJG47hRxIDHa8ujzrLsmd4uqQufZKT6n/m4 Xbw/7l08d53NUQlS8epyLCgA0ZctctgeP7bnXWlD1DI3Kcd066P3Sf5xM8twiW7PCIK8Q4ORzy53l SObPTMhe6OFqFq67uUj0lTznVx34pXF/A0I0s8OjvM3/EJtXC6iob7g0kTuyDLyNld6YuM2JSKkAc RkpviYXcgWCdqp6nPKspXYUtaT0xDF06wQ4z+adTvDyhDz3Pif71/+ERVQbN073CChGIgGNmLL8bd EXted5AiH5CANvb94qNg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXmL-0000000AW8m-1cxV; Wed, 27 Mar 2024 18:13:37 +0000 Received: from mail-dm6nam10on2043.outbound.protection.outlook.com ([40.107.93.43] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhh-0000000ATfM-1ogQ for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:52 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=msEp/dLkz2KG8UcCT3hjlr/5SkDNgHDG+gleXQaCHQ7mvfnz09a1qGDukVT7QJcAvgZsCef3hBB1wdORsadOInZ6ZS5rgAdORBiDD0Q7xZGVntTDNt6XQuzFuVXZ+DWoIv7csgoZfx1xlq3AIJWr2CQH36nJqtChUHVdK9H8C37di/cMbCA+Aetp7X4IrMNr26h//ZksqpirGoe0JmB6FjeObZtX9kht12Uy+ozu8J3+a+bZHp9JH6Bk5tqzcRWHIfObKDlFMvY6VS5eAU0ogYkL3/EssoVfA0FFZblJVadrMbfJ5YTm7DR1EZGbOG7ukq1WXdghedYbapTLN/DeLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qb9Bm5mZhRpKZ1E0v2HrnDeTRaYvYu3eMk6OkfeOjbs=; b=n5DgxYLziBAYW1KFgB740idRBxR5RTxmbSG98df+LfiffI3ovkpvkOBnnIg4Lz+h74EEOEubIlPjn1ePl+VEdKOm49TAWLtPhdTJEb6ppoDgf4l8Os9m9ZHbk9TzHrF6wi4LUW2ZN0O/zjotqwgWeIEjK5Xn3cgG8UuEiVv/gCsDRmY7IU28ZiNPgmArD7Z9w5I2WMfeuuNz15/ZQrEtLsMBD/mix1omfDTmnEE8Zo6DvVpfyeiRW+EptsbVkf7DVvJMAxvCp4PVHqRuduXegOfxsxRhlDu1WdMp9nyDqmaTUH0y62XrIY0hN9t/CuwtgfoaGrquC3D11CdWhQmazw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qb9Bm5mZhRpKZ1E0v2HrnDeTRaYvYu3eMk6OkfeOjbs=; b=VaSnwtoiYzsqbF2EFlIJ/xqPvTIEAcLipRY8ynI4BEUovdUNYw6269hDJNFh2/wLQu66XrJmF0tdy9Vkp6zoc0oAjp52PAvSWX+fyLg5RlSYQLnhMZCj7dgbVRnsHwIRL92GSfko5/cj/ehrIvWPXzX36r0p293QTqNc70A1hjfQJ2Zz24PZbnDY1CNzLszIcdum19mDssNFPshOxiIrueinJJphdy/1Dy9LhceJLNH7ImimQGHUuzdLQJSMw/WStaJMHG081WuTxM6xAV3/neaQFQ/YsotgGokpp14TSlrbQn8UqGODhhjN8rck9FqfZ3MYwwV5yq28X0l9gks8rw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:29 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:29 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 24/29] iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID Date: Wed, 27 Mar 2024 15:08:10 -0300 Message-ID: <24-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0306.namprd13.prod.outlook.com (2603:10b6:208:2c1::11) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: d6fc686f-aa2b-40bf-7447-08dc4e88e1a2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qFMGsZPSvX9HgZWAHev5As/ETDglHu7s+iaejNXm0n9LyU+vuYmemKr0vTncug4maufUhkCPYIcCKOV1RqLQOaouymt7AsS9YwarvJoOFUGzup0+hC3HTsKDPaGK+ZHb6mbmRsh+mytVH0dZnSJeqGBFM781beRXT9wM8FQn9k1wu+MJy9IgNxGYtCQOvBUUiZnVPFMsaJpVfni/GsKu/krUW7wHZRnR6delRHoHG/Jj/BIUzm2N1LGv31hQkeKvKxhu41+lrUAGEUs57q1S4SHDRzIfr/e/FbgoXF/VyBqki+xruKmdxOSMm+oCiwNa4P5scJaF1BpRkGm1NwMiYQU240XB9zdHtK8mPm+XaHC0Z5KeHTwrR3MsHtai1/y1tyr8cLOy3cdZRje6C5XBZ85eCBO0gP6S/8x7ST3DDnfBe9Lvh2kNIU3vgZdLC5uS3AAESzHwnmVYd3An2lE6O/3SnlaRGlsH996FboQ4csqrLDFaSRoWArPCTBooS2pLxdshg55ctNzC3akBgqjSdpWSQveonIQATvRWsPdJucl9GSRUV/xh0urDoFL3tvvclMccL563N+NeUE1ZXsQFm8rMDhs2aXixJeR+NbSF8UYna0seyIC/rRwgYToggWDg+Kv2IBeZ3lPJuFi7mhnoSBcoH/Fzi/ewnZS8x/WjCps= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: d6f0VRVwQchNc52BHVTDpdtS9h30k9iiWBEj8lkt3K6m6un7FkT9yZ6P2Y34q/ZNBxUOsv7newfD/7kTggnlVn3qDsDwDE8+TO4bS+VP6EmWQwGvJpW/MkzAXR2TDAPM4XWXW+F7LzJPK5e/IOGN7FTBnp+KcCV7nLLKrAp6weeGUxpQhUy15v3QBg9Pq7HkZHopfyoyQYG1/KOwzDls7j1o+yNOmmnKUaclzeI4SwrgM3xHTQ7MZF3Td/Og7aHb1aTIuKh76kI95gTF+MSYU87rSHZwDBVcx9zDOeKV3pvp2jDD3dwuVhfi/R0n152Tp1BWgWnAxzEH1s3R60q3A8I+AotfVjFZAfVEZ+xN+QMErPjPC0EA0GOjRReJAL3cRLVW3uSev/+XXeBt4AAaZtS+HTTCQz7+7hxoNydjYnySb7ic3KoZTmQwKwsqtEYCepNAhH19LEpxSENrEPEKiA2e3izHTbfBvusc2XVxqur/rZdQKDjBxdFPN6SG5U7b6qjNgjyNChuhh0pj3sMuZlUhGghJUhFY/XL5Cozm1PBwmVtxLGtr04yGRxSB4/eD90GLshv6PmoXVuqi2TUVt5QH/ewklwGq72aox5UVmCtfPVhvy1oE62mtSx59cpNNo8jAjiAuJUHpNDRjWifbVAOPib9sFy5e66L+TzsHPJytKtTLO1vBK0gq6k/jcOGmonMAxWKS1ecFO75J0HmWLcC3tfWPKSwfK0ASpQt2HsJoBqGMFwApDLALhWH6kzoSbqrsQjGNzdqcLyuq3Et3nN8BBPKNRp+8Qv1QvmV0WHhb018/NQ5LfJF6rM3LstTrXYj87Ev0OTaGj1yrAiCVz4hRuVOwTUTHC5SvCI3I74QfAatlYOLokzNoLCJWQ5vTZ7DEBCxtXSZeg1bg+EgKfjTI/KgsZZS5+PmSexsUHvYRT0iiTBm/uyJfcz8bhTYXDDC/vMcLTZGCHWao31EHNTKKhdVWq+XP2RJnnYUzdzC4WGMml8Zg0UTlPiLIFFYmGEV1ThdAlHcsyx+CFUYedR35z9B+gYP2hoUgeZ6O42lMFg126cKG+ufq/b9lGB5c2qcjdxxIiuYAwPhkGFiZppB89ppH7I+Y5EmdnM67k+o3RPGmzhTDP+Mm49xRL3h6AdmcLax3I71cWoVrr7fYKxZ5/Affw71Kk4FwAc1CksJJF9Fup2TsfyktSLUfaiJ8weLp/lHjaHYWVxs5zX4+3l1qMpJ8w31+km/nSTPuABoSbW/fMLd33z19hV7jpUFa5nc11ojBJKNL8cZHqUHCq4hHg0aRtAVKa9bkfQ+RGLemFbtFfR40eabaoOrCjHx0BDNtOqia8Hj0jTTWKoNFdVswEGzauQBawu30B8u2xYvgfhaob33hcwazCXMfC+TqXSI3E7obzaH9N+4JQsgCDKL92MKe/het64RuvgiPih1qa7JKhCH7Ry6n/e0dIgJ6MMeyq7Dk9gAR7dkocFdyBhtM06bpE9VygxHPDmXacQXMp4W8rn+5GW757sL9ABqe16+mXMbzIBUEcwqdo2O3yff3DeNb9yVrvqoIH7msOIMhF9dvrq/19EIfFA9MaY4J X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: d6fc686f-aa2b-40bf-7447-08dc4e88e1a2 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:18.9231 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6KRaA4uw5m5dVFP+43W60Y6sZdJbiZPQSTEFo3fpdib+NVGh5zQOFIC52shS2Aou X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110850_056806_592DCB0A X-CRM114-Status: GOOD ( 26.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SMMUv3 IOTLB is tagged with a VMID/ASID cache tag. Any time the underlying translation is changed these need to be invalidated. At boot time the IOTLB starts out empty and all cache tags are available for allocation. When a tag is taken out of the allocator the code assumes the IOTLB doesn't reference it, and immediately programs it into a STE/CD. If the cache is referencing the tag then it will have stale data and IOMMU will become incoherent. Thus, whenever an ASID/VMID is freed back to the allocator we need to know that the IOTLB doesn't have any references to it. Invalidation is a bit inconsistent, the SVA code open codes an invalidation prior to freeing while the paging code runs through: arm_smmu_domain_free() free_io_pgtable_ops() io_pgtable_tlb_flush_all)() arm_smmu_tlb_inv_context() To do it. Make arm_smmu_tlb_inv_context() able to invalidate all the domain types and call it from a new arm_smmu_domain_free_id() which also puts back the ID. Lightly reorganize things so arm_smmu_domain_free_id() is the only place that does the final flush prior to ASID/VMID free and that arm_smmu_tlb_inv_context() provides full invalidation for both arm_smmu_flush_iotlb_all() and arm_smmu_domain_free_id(). Remove the iommu_flush_ops::tlb_flush_all to avoid duplicate invalidation on free. Nothing else calls this besides free_io_pgtable_ops(). Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 9 +-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 68 ++++++++++++------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 3 files changed, 46 insertions(+), 32 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index fd0b9f230f89e3..9ec1a5869ac3b2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -367,18 +367,13 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain) { struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); - /* - * Ensure the ASID is empty in the iommu cache before allowing reuse. - */ - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); - /* * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can * still be called/running at this point. We allow the ASID to be * reused, and if there is a race then it just suffers harmless * unnecessary invalidation. */ - xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); + arm_smmu_domain_free_id(smmu_domain); /* * Actual free is defered to the SRCU callback @@ -423,7 +418,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, return &smmu_domain->domain; err_asid: - xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); + arm_smmu_domain_free_id(smmu_domain); err_free: kfree(smmu_domain); return ERR_PTR(ret); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index a82a5e4a13bb44..888972c97f56e1 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2057,27 +2057,19 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, } /* IO_PGTABLE API */ -static void arm_smmu_tlb_inv_context(void *cookie) +static void arm_smmu_tlb_inv_context(struct arm_smmu_domain *smmu_domain) { - struct arm_smmu_domain *smmu_domain = cookie; struct arm_smmu_device *smmu = smmu_domain->smmu; struct arm_smmu_cmdq_ent cmd; - /* - * NOTE: when io-pgtable is in non-strict mode, we may get here with - * PTEs previously cleared by unmaps on the current CPU not yet visible - * to the SMMU. We are relying on the dma_wmb() implicit during cmd - * insertion to guarantee those are observed before the TLBI. Do be - * careful, 007. - */ - if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + if ((smmu_domain->stage == ARM_SMMU_DOMAIN_S1 || + smmu_domain->domain.type == IOMMU_DOMAIN_SVA)) { arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid); - } else { - cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; - cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; + } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2) { + cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; + cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); } - arm_smmu_atc_inv_domain(smmu_domain, 0, 0); } static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd, @@ -2211,7 +2203,6 @@ static void arm_smmu_tlb_inv_walk(unsigned long iova, size_t size, } static const struct iommu_flush_ops arm_smmu_flush_ops = { - .tlb_flush_all = arm_smmu_tlb_inv_context, .tlb_flush_walk = arm_smmu_tlb_inv_walk, .tlb_add_page = arm_smmu_tlb_inv_page_nosync, }; @@ -2275,25 +2266,42 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev) return &smmu_domain->domain; } -static void arm_smmu_domain_free_paging(struct iommu_domain *domain) +/* + * Return the domain's ASID or VMID back to the allocator. All IDs in the + * allocator do not have an IOTLB entries referencing them. + */ +void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain) { - struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct arm_smmu_device *smmu = smmu_domain->smmu; - free_io_pgtable_ops(smmu_domain->pgtbl_ops); + arm_smmu_tlb_inv_context(smmu_domain); - /* Free the ASID or VMID */ - if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + if ((smmu_domain->stage == ARM_SMMU_DOMAIN_S1 || + smmu_domain->domain.type == IOMMU_DOMAIN_SVA) && + smmu_domain->cd.asid) { /* Prevent SVA from touching the CD while we're freeing it */ mutex_lock(&arm_smmu_asid_lock); xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); mutex_unlock(&arm_smmu_asid_lock); - } else { - struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; - if (cfg->vmid) - ida_free(&smmu->vmid_map, cfg->vmid); + } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 && + smmu_domain->s2_cfg.vmid) { + ida_free(&smmu->vmid_map, smmu_domain->s2_cfg.vmid); } +} +static void arm_smmu_domain_free_paging(struct iommu_domain *domain) +{ + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + /* + * At this point the page table is not programmed into any STE/CD and + * there is no possible concurrent HW walker running due to prior STE/CD + * invalidations. However entries tagged with the ASID/VMID may still be + * in the IOTLB. Invalidating the IOTLB should fully serialize any + * concurrent dirty bit write back before freeing the PTE memory. + */ + arm_smmu_domain_free_id(smmu_domain); + free_io_pgtable_ops(smmu_domain->pgtbl_ops); kfree(smmu_domain); } @@ -2914,8 +2922,18 @@ static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain) { struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); - if (smmu_domain->smmu) + /* + * NOTE: when io-pgtable is in non-strict mode, we may get here with + * PTEs previously cleared by unmaps on the current CPU not yet visible + * to the SMMU. We are relying on the dma_wmb() implicit during cmd + * insertion to guarantee those are observed before the TLBI. Do be + * careful, 007. + */ + + if (smmu_domain->smmu) { arm_smmu_tlb_inv_context(smmu_domain); + arm_smmu_atc_inv_domain(smmu_domain, 0, 0); + } } static void arm_smmu_iotlb_sync(struct iommu_domain *domain, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 3516869954ea33..a711a659576a95 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -772,6 +772,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, void arm_smmu_remove_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid); +void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain); void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, From patchwork Wed Mar 27 18:08:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3B23EC47DD9 for ; Wed, 27 Mar 2024 19:18:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rus2oyxWWKKGIegxJyQP19eK2nZtv2C+E7fdOevFbxg=; b=hd8lvFYGuHyO/R X13OwB8WsBkg34Zq2Njgg1q7lW+o6NMFmRyIcNMxqHV7WJbVqPzOI6VAF8cx5FKXB9Rl9qoKaS7DR YPJdojofACRO0MzORaabSbM7rfmDR1Eonvga23e/RyGFv3oF3Bj7DDp+yA47Y4uYX4mofaJG3SB7p cUAg+RKxZxA/NYGvhz3QylHslaHUcg0nbZhVvixE8LepGEtIAZffOQ+IGdKperMiFWM84AM4HpYxq 35UAyCCAip+LQ4hRLXqy4JnsqvWfnXWi+I2Dm3EeqC8IqGgYpXbWzKoUcoBvurw/n/wBLSX4Pfd86 COerF6DCn9nUZRIQM3DQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn5-0000000Akyb-37m5; Wed, 27 Mar 2024 19:18:27 +0000 Received: from mail-dm6nam10on20604.outbound.protection.outlook.com ([2a01:111:f400:7e88::604] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhj-0000000ATY9-2eBb for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:53 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BfHv6f1x8HHecrlbj1aFRuDxMTzgAdshYA7I6eD3bAJDgtbbtPWhq1UVdoc4xfnSoGPFs8CV4nSKcqBhZonjxJp5NHtf0qvcTkvJcjOeR5svL96jmuo1P5o4hZ2iUxQKk5f0OLCx5oEtq2jWp3iDFlkqnPbZOVv2MuNWO86Y/CDkKW/fIQFzLaRCEW0sJZy/qgngdi4mS9rFkhukhvzW4JcTIMobMpxMognzfmNuGHtv6+GX5LiexVrRYZO/GPRYTf/ql0uSWj6d95EDp9QfUOQBhhGA1UsIoxKydp4w5RLWbvY8HSvFB0hs25q19Pql/mQ35yHgi78EV9JkMao4Zg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YifcmNgQewXB0Krv0y3NQTC6WlItmItJ660UGTNJ4F0=; b=n6jQD9F4zQ/1H4dvRsiE5BPz3/ABwy8IxPuikflq+SAvCoI7ACH3+Yb0KaC+ogV8tTe6lfOjccW/QgkJBByZ2OR42vZ7rVpuTatquG5pJfkkK16079i6mor2NiGkNXgyzegw5uyzZpWM5EJNH+itoHt80lRRTPh/oJDyEgLCidzEeS6rt5M8XERdEqSVgOL38McmPQJGNiQZSuO3Rl0kGwFP9h88yKRNX+1i9zx+yjy9sTO4p6/6ob5Rth4v/Ogdw4pwjLZ1hzAnvkPCZsaSCI/SwmguCwCwmck+u5PF9F1e4LvHJLCYD2sMdJJ+qyLhJ185rS0TJ1KvGdkUsojbtg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YifcmNgQewXB0Krv0y3NQTC6WlItmItJ660UGTNJ4F0=; b=liC1Ks6SOTiyDYfjB2VT0yl7KNkwSizwgTfcb70rkXcLEReqsjpygCbJKnL/sMJ592nyekaZdQ3NI/H4KuYsG2hyY2tIdvIXas/7OQFNEUCFTFm+yT5OHzenFuvnMyS2BfGB8PkicLl4XrHwNJUotmgmtWU856857ieEqS5Qx4LGPHM2Nq4Z7Z6YHRXvIZEtgO9cqgYLAEHWh6SCnMw29+u1QC9dRQPoKKdrQV/arJZo5NHBYAFbcJGMwwFzdgy1Xx+zoaGbYZEH/Kbl4nI1RilWgcwcQi1vHCrp/ZQn8G2T8/RtaMYYvw+O6J7fdcngw3lFIT+mimlvF8iSbWgZwA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:30 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:30 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 25/29] iommu/arm-smmu-v3: Move the arm_smmu_asid_xa to per-smmu like vmid Date: Wed, 27 Mar 2024 15:08:11 -0300 Message-ID: <25-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0058.namprd05.prod.outlook.com (2603:10b6:208:236::27) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 3f7a7b22-cb5a-48cb-39d1-08dc4e88e1be X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KLFl0sK23SoCx+qyba2vegeQpOXw/+gNa108xCot5Us2P7wST1P36itg8nYQj2wDD0vy8rLZmgdfDIh/pQlXxZma/g0p7xF2fdJ72iv1ZI4I4NakmVCkn2j7IksA3LkCYLgJhOQgiIYSDw235QDsPT8uV07MKvqxhHr7ffmLmS64W0TElTUxw2hifts03VAZlZaEmXqPNDhaltQYi8TtZAf2LgrTP1pfM7lGkw2Q7f7RSnO2blayx5Klci5KJJQok8IZHlue1Z+ZEZCZV6xJatJvnr4lJXj4n1mu3+1X5VLEtnN6qcBNy20idRaqpUjlpqI05l7y7bWgJoUFv/BaFnlmh7KvkpuA+PnxFFkNgVGhPWpRq1pBw4olXL0C8LUU8BBEMNA/JYVeusgm6px7CWe347m9yPnGVpvADpyE0ce+xvJh6QjKo7wwZfxYIWyS971FaHB6j9asmXNHX4nwGfSyFcQYXpgSR+wBDFlPZLKTeDpb58lMllbI8i31K3HGYpNv5BwN+MoNxO6O4L9B/8TtPtnDLTNiPOZWIFFYp+gReeoBn03etSFF3rZcy8c1tKaUCemAm4YZFMBCiJPz0SEQ5K82WHN3YlVeA0CKNk2EwAw3DPjqsremQUF19jZMmn+T+23ESsVGsMgvwHZTnkqSPUVhR1JIunNi2l47LQY= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: jOOTn5ruM1mXkvBJ55Dec9Z3APVC6mQ5d24JrTXjd64oIIQhdFCFV10B+vf1jwgNTJ1g0PeaqCmbkoWVCT+6G6AiaVnknMN8idvYGwhh9NNJjONGvPy5xt5re1TwK30E4shDH8uUX5Ots6bb+Fh/3xVe4QDp/FtiieD/Fbkbh/ZD1ErFJrN0CNBSr07aeKiIRBSnM+HCe19U5J1WjspmoGmYwl3ykC9ApBysK8T5FjSOvWtV+ZU0T4lxmN34qvvCwGG6/LHPyPQXMkCml14dmRxit2nVvtp0XRiOc3mlGtOOgQ3dBF6DMTgSvQgW2zg/56zPyM6KmGdg5WitE+/nwF/OwQhpVRKh9UYDzvl/yeFWSdX/AiyVgZcPX+mkjBQl9uSShQIOGcedPLizXT1H5i8sGgFzKsNsqu2rKiuyaAZmuf4sK8alTco5t77TRNw1Ca119Xb1oo86jHyG5zpvW8b90rHNYtebkizAc+8Aag8DuGYZhdGz55/8YmwP2TXALd7il23SSKw0JjEtacGRGRH2/o0EDz3hkUWIi6u0BxTQT78IBRvGiFPOy5OJ9O8dtvVNCSxu5EI+/XxeazYPmTIobq1tErEZxw6mUVyTzbFFrEn9ycqSOUvSt4qkXxTEdRzTgpU2AL0czSagAlKO87ULeNSXCJ3+lfbZX4+msq9FBjpHpI76LHvGuILZMlJJhDc4F/IbCTnlpAFcCV5fvHnr1xCOqDsGLqFWvF+3iFv8iIsLkNmc0C9p2RuI9l0AVs5c+bd/sKx4LozavTsy4kz/rPsex/E6jsc+IxtFDTM7SOhyXRyxBpuzXGUr9mOQ7xjuu4m0DZR4S7TYF5MDK5uWE5bFsJA7T4pmxvqMS3ATDth5CQzJebz9etm4eJcHu8hYr34KG/CrupWTEUDiiDK8HX5Hu1Yh/GHKVGcZy7qjzDJB8gFqkRL3ys5gnLvhZAujeO9BYJGtKYU7Ubd8WugxNVDYkQwSQFLK4aX7ZMtM/98VxoWnP9kxuojkiDefGZZW7OFDOCWfH3hmxUYP39JZopSxesSmR5PDVQxMp/JgBUrzmmilXiV7DSJIebbXFCRDcHPsENw1FlFXl/BvYz4MqFf6TQo6MWTTwyBwyRYUsO+nlfP4uIS5qn049olgkad/jDS15djlY2fkwi4lQ9bp1nBRMccyKLw0vhq7P6/pUuEvLqva+xr4ofau91RIpfERK7HZvl3MG0s66JrepjdL0nTkqWZEJ/i99g+hYAg+Pj73j5IEdklD2gI4PIlTkicn0jPMuKe8tLb2jSuohUDck/goeWKf6/rgPnnEPYHxs2MqLBir9ecHwk+rhzyVNayyI6PhspL1fma/4UtpoSCqhh7c+KaBiduppxLAyOcGiAtGO7zG+QyCfSJjFqTL4VDEVSVMcJzy2b1nTfo1Ghy6hLYNcZtsNiG4g6yR6yVEj8XgE3tR6xDGtX7Q2R4mxk/b7cQQv2pgnadWZQXNOh5sZU4kpoN321ak994/Ty1j3WlBBdZPnuv990vkUNQ2ivielwiSsCt+IE1WoRG9pWt8nK2C7FO9DhhGZpNQeS3138lLJS1tgDNRgfRAoRq1 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3f7a7b22-cb5a-48cb-39d1-08dc4e88e1be X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:19.1280 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 399CA5WnwMm8oThpgF/VE9GEo/zxBgglKgh8HhSQkI/EUCzVBhFTWHNi6a6OFYTf X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110851_827540_A568DC21 X-CRM114-Status: GOOD ( 18.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SVA BTM and shared cd code was the only thing keeping this as a global array. Now that is out of the way we can move it to per-smmu. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 39 +++++++++---------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 +-- 3 files changed, 22 insertions(+), 24 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 9ec1a5869ac3b2..95d8d2d283c916 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -404,7 +404,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, smmu_domain->domain.ops = &arm_smmu_sva_domain_ops; smmu_domain->smmu = smmu; - ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain, + ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); if (ret) goto err_free; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 888972c97f56e1..bdcf9a7039f869 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -86,9 +86,6 @@ struct arm_smmu_option_prop { const char *prop; }; -DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa); -DEFINE_MUTEX(arm_smmu_asid_lock); - static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -2280,9 +2277,9 @@ void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain) smmu_domain->domain.type == IOMMU_DOMAIN_SVA) && smmu_domain->cd.asid) { /* Prevent SVA from touching the CD while we're freeing it */ - mutex_lock(&arm_smmu_asid_lock); - xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); - mutex_unlock(&arm_smmu_asid_lock); + mutex_lock(&smmu->asid_lock); + xa_erase(&smmu->asid_map, smmu_domain->cd.asid); + mutex_unlock(&smmu->asid_lock); } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 && smmu_domain->s2_cfg.vmid) { ida_free(&smmu->vmid_map, smmu_domain->s2_cfg.vmid); @@ -2313,11 +2310,11 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; /* Prevent SVA from modifying the ASID until it is written to the CD */ - mutex_lock(&arm_smmu_asid_lock); - ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain, + mutex_lock(&smmu->asid_lock); + ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); cd->asid = (u16)asid; - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&smmu->asid_lock); return ret; } @@ -2609,7 +2606,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master, * arm_smmu_master_domain contents otherwise it could randomly write one * or the other to the CD. */ - lockdep_assert_held(&arm_smmu_asid_lock); + lockdep_assert_held(&master->smmu->asid_lock); state->want_ats = !state->disable_ats && arm_smmu_ats_supported(master); @@ -2661,7 +2658,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master, static void arm_smmu_attach_commit(struct arm_smmu_master *master, struct attach_state *state) { - lockdep_assert_held(&arm_smmu_asid_lock); + lockdep_assert_held(&master->smmu->asid_lock); if (state->want_ats && !master->ats_enabled) { arm_smmu_enable_ats(master); @@ -2725,11 +2722,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) * This allows the STE and the smmu_domain->devices list to * be inconsistent during this routine. */ - mutex_lock(&arm_smmu_asid_lock); + mutex_lock(&smmu->asid_lock); ret = arm_smmu_attach_prepare(master, domain, &state); if (ret) { - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&smmu->asid_lock); return ret; } @@ -2753,7 +2750,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } arm_smmu_attach_commit(master, &state); - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&smmu->asid_lock); return 0; } @@ -2783,7 +2780,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, if (!cdptr) return -ENOMEM; - mutex_lock(&arm_smmu_asid_lock); + mutex_lock(&master->smmu->asid_lock); ret = arm_smmu_attach_prepare(master, &smmu_domain->domain, &state); if (ret) goto out_unlock; @@ -2793,7 +2790,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, arm_smmu_attach_commit(master, &state); out_unlock: - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&master->smmu->asid_lock); return ret; } @@ -2809,12 +2806,12 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) smmu_domain = to_smmu_domain(domain); - mutex_lock(&arm_smmu_asid_lock); + mutex_lock(&master->smmu->asid_lock); arm_smmu_clear_cd(master, pasid); if (master->ats_enabled) arm_smmu_atc_inv_master(master, pasid); arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid); - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&master->smmu->asid_lock); } static int arm_smmu_attach_dev_ste(struct iommu_domain *domain, @@ -2833,7 +2830,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain, * Do not allow any ASID to be changed while are working on the STE, * otherwise we could miss invalidations. */ - mutex_lock(&arm_smmu_asid_lock); + mutex_lock(&master->smmu->asid_lock); /* * The SMMU does not support enabling ATS with bypass/abort. When the @@ -2846,7 +2843,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain, arm_smmu_attach_prepare(master, domain, &state); arm_smmu_install_ste_for_dev(master, ste); arm_smmu_attach_commit(master, &state); - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&master->smmu->asid_lock); /* * This has to be done after removing the master from the @@ -3508,6 +3505,8 @@ static int arm_smmu_init_strtab(struct arm_smmu_device *smmu) smmu->strtab_cfg.strtab_base = reg; ida_init(&smmu->vmid_map); + xa_init_flags(&smmu->asid_map, XA_FLAGS_ALLOC1); + mutex_init(&smmu->asid_lock); return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index a711a659576a95..97c13f9313dcfe 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -673,6 +673,8 @@ struct arm_smmu_device { #define ARM_SMMU_MAX_ASIDS (1 << 16) unsigned int asid_bits; + struct xarray asid_map; + struct mutex asid_lock; #define ARM_SMMU_MAX_VMIDS (1 << 16) unsigned int vmid_bits; @@ -751,9 +753,6 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) return container_of(dom, struct arm_smmu_domain, domain); } -extern struct xarray arm_smmu_asid_xa; -extern struct mutex arm_smmu_asid_lock; - struct arm_smmu_domain *arm_smmu_domain_alloc(void); void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid); From patchwork Wed Mar 27 18:08:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 08072C47DD9 for ; Wed, 27 Mar 2024 19:18:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7Jel5J/RaNSMf6DFXFqz5jPYWSNI1sPxfqbYN7hCMic=; b=2eoyN4s4s1o5/v Ns0oSQkNX8RJyrnODVw8eAwThbvVWHHRCPPkCeVVgCgLYkbafzoqUxRHsFFuao+B8RqtrzSNNxYeX l+F6t5ru54Or1Y7tsSJJUAnU6diJz1p2k3ux7jEvGvD8VQBmiRgv2CnATVQoPWGq8p+LZJ4Ntrbai hMlpRl3O52MtAwFueSZ1daYA2Wz2HYwgNjd6TVShtBdfke2O79tKCSZJoispI6P5rNoR3aZ2KvSeG FwaVrGKdV+3dIF8FjcpNAm0+Gec0Tb6Wd0Eejs3fPym6/CVlY5fHxIOs3mukAs924Sea0T28amO7o 84UqReKxrGL/zFFVVF+w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn6-0000000AkzI-2Hce; Wed, 27 Mar 2024 19:18:28 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhm-0000000ATtH-0IO2 for linux-arm-kernel@bombadil.infradead.org; Wed, 27 Mar 2024 18:08:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=XTP6lyihEdsKvKw2lgOnHuzKPKxDE+Enop2SXZ/QG1c=; b=oFojcym/Qw80eVbpFd9LimQNRo cPtjj4K2iMew72tJs7T3Aa0abuUAX1+Py0rsm6vnWGZJijQB8te/S1AhxvDLhaCDAAxuv3nrz5ut1 86DJhRt6A973fwcus+9zX3BQFZTnQmw/xQ6LjS6KaP1guW7uXNbWQl5fblqCPhsDDwM5lRmrgUcjR ZXq8O9XjepwKVBYG36t2Og6usr1+ueqjXkvW7pG6klhKvm+mgXwTD2sz3SEE2+wLikjyAVS5SlU2m 9HrCAECf1dSPBvCl1FHEn5oi9vXnAkIgFscOHvJo93ARYNo4N/xn1KogiVdR3mIatXI3zi70+us83 ZZCIlfag==; Received: from mail-dm6nam10on2062e.outbound.protection.outlook.com ([2a01:111:f400:7e88::62e] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhi-00000000RvW-3axq for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:52 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TJcTqXlJ6hYoR+BKpirzqlgopBlDFW9Ld7sE9kV0nGAq07gw1eZhG/mTHn7ahUNn34tmFRZ49WeSoYuQHouoSjxkmqv+O4kjeUas0jkvjXB0o09HVUCUvAJ23TP6ahisyuSLEti7VwNmk5DERs7ZGfqfADXLGPz2Bk3WE12xKE9RzafKvOKY998HoGrcNdV/0nnKzkhmombI78GCi+iLjjvk3AcEDJDuYIzMDfPc+EJYIFaYDUiX6D5QNFspESkpVhdYjD9h+8i2OwvZmwHXVDI1Lx602W5KafcsYS/ZgUnHy/lJHxGJD3UAE39SyZQcFcqLuOARVf/424b8YAtogw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XTP6lyihEdsKvKw2lgOnHuzKPKxDE+Enop2SXZ/QG1c=; b=ZTduprZY2L1AtxULnLKsj8BWgcWqqxZTrpwaU2fAL8pXe1r30x174WYTupqQJF+Pu5wIcd+KbV12Fbyz7nBmvPzBLiWhXzmSIPkaJ+ucfWMfQ7ps/rSyVEf/hGrTiTZCQVtNHAyfn8k0SkFuJVfs58PZLC6G+qSLg++XSkLvnV6+YTD3ErWHe3Md4RyKNU9L5Xrr8vruU9gdZzHiYpxf/WSjFSVdlBiJtq1FgMdrJ9Bv8TOKimb2gWtck0WmtPbDYF60VKFqqfeJPtt/k0LYjBY+MWJDxnTwFd0qSD9PAd+S6Lh/X1hk/n13OsdAit38+A5AOoVJKGbt5zLkzZ20gQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XTP6lyihEdsKvKw2lgOnHuzKPKxDE+Enop2SXZ/QG1c=; b=lg/qj9K0v1p65dm0e3AyhyR/LNRN0jSTswGDyxkLpkMdYkmKWlfs2Nd5On6iCD6UWy+SZ98cFzvJQnnS8T0LCBCT+S2RGcJuFGC6nhimeEDiCrljIRf6R2I/vYAl3U9zsTV71rzRtex1ziBWniFO5djt18aukfJEg1TPm4DzgJqEJ3sld21eSjbVhCDKrCVKlQUE9T6aKUURxUYZKiCUa7TT8U+VlGj0VQuIAvC7TC3NVM2mRzaDpv1yvdnLgR/px3FI/1/8kMnk7alOzXXKmq+VYRKc6njBh4cVGROl9i+jLjstxK2JfoQuzkOic3yveEEmypbH7b745Vs9bM/Y7g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:29 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:29 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 26/29] iommu/arm-smmu-v3: Bring back SVA BTM support Date: Wed, 27 Mar 2024 15:08:12 -0300 Message-ID: <26-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0041.namprd05.prod.outlook.com (2603:10b6:208:236::10) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 4df2b653-3a64-43cc-80cd-08dc4e88e1a8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: C0DEJgV1YY1gQ0n2D7oMjFSejwi+Kh7RR8qj5N5Y0+6HLQ2CGqHdv1HRXb5y9uY8ogmhraR+xpZ8RW+bn4BzXMMWGOJi2kq1oi5fSlrMZkvTilB5XWK4ng34Yc7FLotZ552Fs6SU21NdjXFTzO7zl24Y31/Cn/e8DdIAZHnFuKtrLeA6Bm1x7ZGix3fSwBYnW7nnYuEQYKn5qySmiCmbxsVm6KGjrGcqcHgPC980EUPprQ15TR+JXyWOpDs1LsA5skO5SZGG2Bor1QXf4aeDZANEel74QtwK90bbMtpXuzHXWqvx9k8yKMUVBeQb+2I4zNUcYtiseZB4Wv/XrLEllWpJZuu5dGlq+8QgejZEJwPUQLzRe/C6b+LRk1cwvmXZXAUbeeks9iyvdYy0QP3YkaEpVFppjNs1xTcDWQsrrGlXQEV5xA9K0qcVjXXKyAJllkRMWnHtD4PYF6V5J8D9Dt7jzVi1JxwI4Ztb8zsKdlfLl/UPjhuSqGrIMe9EHDtYuwNWl0WJ2J1ffUEG0WetYnLxxw+JhQNzFp/rlrBZQCQxOnLZqcvpyOAwpXLQYw9rdcsqT3UQZexWAc9DdD7wLXNLHsaWsyQowuNXapX5bHeeYs0g69Shy1rbgcjKD2mj5xcx5RRuB4/RNwvQiPTZuQx0FsaaI/E2jB0726NrQHc= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: KdHUafB60ZZ5XYq07Tk1ObQMxj0EkLCmIbDNHzz2TrGrOlBFZ4XI+h0ZwGKyo2m1OTqXVA3hAqvdTiyKD8sgGq6iR00zHlMx8jI6lncu8u0BVnLw0UK89dtFESg1e/23idXnTHWI49Th5+x86Z0EKWkdvIY4pey2gXs7SS1P0S3BzQLYXiqNiDKBEiLUxE6yHUFw1VJe3rtDSpzjaRmatSdeYL8MnoqUZTTuoezBMz6Yxc8ENHI9oh9FfeYZ2eQWnsO1DRoQlHlWqihHCPvOuSjJ7/Gf26C/r+WGpDkeBq4QpBwr3lDpG8BeKGsNh5g/UQ0vEXeqqIbQJUAKxDkWNRjXUoJzzVlcOjK52C/YPUKkneS1O8GswP+RMTcGNiK8lt66vHMru/tlmS5L7sO/ETMS5nF0V/+M5LKpnG+ctrX1l50xjQeUJxDL/sulp75lHaY+Gn373Xn7IuuPp3ZkqWmr6stAyuX3MPIZA57SQp3bll0x0z4Sy15OO/0hm6FrWdYL1/Lzog9/rL8fXpclIiWf+zvuaz9zT1o43ypYVqpjvxXGrp81tu9vigWp3jeeEGuHWmERXVCQyhjqOTjaJDc5YBh7flLTFMcyO3lvVZp+dVJNtvjWZ1FEkZuI339Sw6+4YY3c+y9ZKMe631d+1oh9bKAnb0D1cJQQnHQ2tNJYxqYBquz5YlAzLRTuUpGpZIgutDZ7Yo5ChenDGsZYpKyrl7CJ+uQt0prTV/Rl+B4PjAOL31KzwjJNsMTT2KqGs1R85qDfTy7orYJljahvmShwrVhFfVu6JUINx5P82IKCnUr+P+9gbnBInPtomQdoV8PW22cm2VYq+LrY7uLPwA5F8hG2Zo6Jd+ThUiCoYHj1M2PQGiBlKOU8svRnNxSoL46TN7jFNsCr5mTrnLqCIQ/QlY+nN0J30PJ/4R7uF977m34PrC2i1pu1h4rnJZk86cPAnX01zbnQgahPrKRLlF1/SFZq4pesOFlAQrmxHYdYbXAYOKWj3KnsfDllgWyNyHrxRBe85rnfZE7KdOUKL0ufobK0+mlQJNgrnHhcCH4VZAwM/jQC7R5J8+H2pblqHWJ7fVolF1KOmUG+AOLNCqZBZ8oryQ3vsKb1f1yRKZjuyKm6Gs4EwuVTBgP8LtIb4CZ9Uf1wchoWu9q+oYYK7Jelk54NSk4Yghe2Q4JvQZJ5keLIoqIX9YHdH60z76xVjzFngU7Jgdk03E6gf7YSLCoIKMn7awuW3cs9a6LK1PJd+NU5XENEkfTeZUh67fasDqLXv2I65rmCHGduKxoai9CbUShjF3w6OTDsG0YeQCauo3fcR+0pDhWcX8OdCIXrxiKAajibwKJPMwSgcyjQlpkLgEXZALoJdh6yuqGkHebKAVJd25QIV1MNobDz0NYpMV6mvOGHLN0BxysbQWEpWTNhSM3CRW7C0Mbauj6Vp5Vk0bamV9k8tmdr9jRStWGnHh7j0c/OsBcZeI3gLxR+52liGiUicPguOW7L7c0Rv5NhfoF3nXmL9OBIa4J3SfUpMDFMWIGqd9VFiZ/CRkfR90s6p9/51tFqkpd1zwGJnN/Q8g4TBqWfDqA+kCgPjLEi X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4df2b653-3a64-43cc-80cd-08dc4e88e1a8 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:18.9887 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /IyMa5OqYX4cW8ARdfRnZ9MOmNavtYlt/RMc1LYjpGC+6eBl4copl0F+78YagsFZ X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_180851_095896_BD8D716B X-CRM114-Status: GOOD ( 29.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org BTM support is a feature where the CPU TLB invalidation can be forwarded to the IOMMU and also invalidate the IOTLB. For this to work the CPU and IOMMU ASID must be the same. Retain the prior SVA design here of keeping the ASID allocator for the IOMMU private to SMMU and force SVA domains to set an ASID that matches the CPU ASID. This requires changing the ASID assigned to a S1 domain if it happens to be overlapping with the required CPU ASID. We hold on to the CPU ASID so long as the SVA iommu_domain exists, so SVA domain conflict is not possible. With the asid per-smmu we no longer have a problem that two per-smmu iommu_domain's would need to share a CPU ASID entry in the IOMMU's xarray. Use the same ASID move algorithm for the S1 domains as before with some streamlining around how the xarray is being used. Do not synchronize the ASID's if BTM mode is not supported. Just leave BTM features off everywhere. Audit all the places that touch cd->asid and think carefully about how the locking works with the change to the cd->asid by the move algorithm. Use xarray internal locking during xa_alloc() instead of double locking. Add a note that concurrent S1 invalidation doesn't fully work. Note that this is all still dead code, ARM_SMMU_FEAT_BTM is never set. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 133 ++++++++++++++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 15 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +- 3 files changed, 129 insertions(+), 21 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 95d8d2d283c916..11e295f70a89b4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -14,12 +14,33 @@ static DEFINE_MUTEX(sva_lock); -static void __maybe_unused -arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) +static int arm_smmu_realloc_s1_domain_asid(struct arm_smmu_device *smmu, + struct arm_smmu_domain *smmu_domain) { struct arm_smmu_master_domain *master_domain; + u32 old_asid = smmu_domain->cd.asid; struct arm_smmu_cd target_cd; unsigned long flags; + int ret; + + lockdep_assert_held(&smmu->asid_lock); + + /* + * FIXME: The unmap and invalidation path doesn't take any locks but + * this is not fully safe. Since updating the CD tables is not atomic + * there is always a hole where invalidating only one ASID of two active + * ASIDs during unmap will cause the IOTLB to become stale. + * + * This approach is to hopefully shift the racing CPUs to the new ASID + * before we start programming the HW. This increases the chance that + * racing IOPTE changes will pick up an invalidation for the new ASID + * and we achieve eventual consistency. For the brief period where the + * old ASID is still in the CD entries it will become incoherent. + */ + ret = xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid, smmu_domain, + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); + if (ret) + return ret; spin_lock_irqsave(&smmu_domain->devices_lock, flags); list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) { @@ -35,6 +56,10 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) &target_cd); } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + + /* Clean the ASID we are about to assign to a new translation */ + arm_smmu_tlb_inv_asid(smmu, old_asid); + return 0; } static u64 page_size_to_cd(void) @@ -147,12 +172,12 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, } if (!smmu_domain->btm_invalidation) { + ioasid_t asid = READ_ONCE(smmu_domain->cd.asid); + if (!size) - arm_smmu_tlb_inv_asid(smmu_domain->smmu, - smmu_domain->cd.asid); + arm_smmu_tlb_inv_asid(smmu_domain->smmu, asid); else - arm_smmu_tlb_inv_range_asid(start, size, - smmu_domain->cd.asid, + arm_smmu_tlb_inv_range_asid(start, size, asid, PAGE_SIZE, false, smmu_domain); } @@ -181,6 +206,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid); if (WARN_ON(!cdptr)) continue; + + /* An SVA ASID never changes, no asid_lock required */ arm_smmu_make_sva_cd(&target, master, NULL, smmu_domain->cd.asid, smmu_domain->btm_invalidation); @@ -374,6 +401,8 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain) * unnecessary invalidation. */ arm_smmu_domain_free_id(smmu_domain); + if (smmu_domain->btm_invalidation) + arm64_mm_context_put(domain->mm); /* * Actual free is defered to the SRCU callback @@ -387,13 +416,97 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = { .free = arm_smmu_sva_domain_free }; +static int arm_smmu_share_asid(struct arm_smmu_device *smmu, + struct arm_smmu_domain *smmu_domain, + struct mm_struct *mm) +{ + struct arm_smmu_domain *old_s1_domain; + int ret; + + /* + * Notice that BTM is never currently enabled, this is all dead code. + * The specification cautions: + * + * Note: Arm expects that SMMU stage 2 address spaces are generally + * shared with their respective PE virtual machine stage 2 + * configuration. If broadcast invalidation is required to be avoided + * for a particular SMMU stage 2 address space, Arm recommends that a + * hypervisor configures the STE with a VMID that is not allocated for + * virtual machine use on the PEs + * + * However, in Linux, both KVM and SMMU think they own the VMID pool. + * Unfortunately the ARM design is problematic for Linux as we do not + * currently share the S2 table with KVM. This creates a situation where + * the S2 needs to have the same VMID as KVM in order to allow the guest + * to use BTM, however we must still invalidate the S2 directly since it + * is a different radix tree. What Linux would like is something like + * ASET for the STE to disable BTM only for the S2. + * + * Arguably in a system with BTM the driver should prefer to use a S1 + * table in all cases execpt when explicitly asked to create a nesting + * parent. Then it should use the VMID of KVM to enable BTM in the + * guest. We cannot optimize away the resulting double invalidation of + * the S2 :( Or we simply ignore BTM entirely as we are doing now. + */ + if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM)) + return xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid, + smmu_domain, + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), + GFP_KERNEL); + + /* At this point the caller ensures we have a mmget() */ + smmu_domain->cd.asid = arm64_mm_context_get(mm); + + mutex_lock(&smmu->asid_lock); + old_s1_domain = xa_store(&smmu->asid_map, smmu_domain->cd.asid, + smmu_domain, GFP_KERNEL); + if (xa_err(old_s1_domain)) { + ret = xa_err(old_s1_domain); + goto out_put_asid; + } + + /* + * In BTM mode the CPU ASID and the IOMMU ASID have to be the same. + * Unfortunately we run separate allocators for this and the IOMMU + * ASID can already have been assigned to a S1 domain. SVA domains + * always align to their CPU ASIDs. In this case we change + * the S1 domain's ASID, update the CD entry and flush the caches. + * + * This is a bit tricky, all the places writing to a S1 CD, reading the + * S1 ASID, or doing xa_erase must hold the asid_lock or xa_lock to + * avoid IOTLB incoherence. + */ + if (old_s1_domain) { + if (WARN_ON(old_s1_domain->domain.type == IOMMU_DOMAIN_SVA)) { + ret = -EINVAL; + goto out_restore_s1; + } + ret = arm_smmu_realloc_s1_domain_asid(smmu, old_s1_domain); + if (ret) + goto out_restore_s1; + } + + smmu_domain->btm_invalidation = true; + + ret = 0; + goto out_unlock; + +out_restore_s1: + xa_store(&smmu->asid_map, smmu_domain->cd.asid, old_s1_domain, + GFP_KERNEL); +out_put_asid: + arm64_mm_context_put(mm); +out_unlock: + mutex_unlock(&smmu->asid_lock); + return ret; +} + struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, struct mm_struct *mm) { struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_domain *smmu_domain; - u32 asid; int ret; smmu_domain = arm_smmu_domain_alloc(); @@ -404,12 +517,10 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, smmu_domain->domain.ops = &arm_smmu_sva_domain_ops; smmu_domain->smmu = smmu; - ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain, - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); + ret = arm_smmu_share_asid(smmu, smmu_domain, mm); if (ret) goto err_free; - smmu_domain->cd.asid = asid; smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops; ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm); if (ret) @@ -419,6 +530,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, err_asid: arm_smmu_domain_free_id(smmu_domain); + if (smmu_domain->btm_invalidation) + arm64_mm_context_put(mm); err_free: kfree(smmu_domain); return ERR_PTR(ret); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index bdcf9a7039f869..5a2c6d099008ed 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1324,6 +1324,8 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr; + lockdep_assert_held(&master->smmu->asid_lock); + memset(target, 0, sizeof(*target)); target->data[0] = cpu_to_le64( @@ -2061,7 +2063,7 @@ static void arm_smmu_tlb_inv_context(struct arm_smmu_domain *smmu_domain) if ((smmu_domain->stage == ARM_SMMU_DOMAIN_S1 || smmu_domain->domain.type == IOMMU_DOMAIN_SVA)) { - arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid); + arm_smmu_tlb_inv_asid(smmu, READ_ONCE(smmu_domain->cd.asid)); } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2) { cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; @@ -2305,17 +2307,10 @@ static void arm_smmu_domain_free_paging(struct iommu_domain *domain) static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, struct arm_smmu_domain *smmu_domain) { - int ret; - u32 asid; struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; - /* Prevent SVA from modifying the ASID until it is written to the CD */ - mutex_lock(&smmu->asid_lock); - ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain, - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); - cd->asid = (u16)asid; - mutex_unlock(&smmu->asid_lock); - return ret; + return xa_alloc(&smmu->asid_map, &cd->asid, smmu_domain, + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); } static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 97c13f9313dcfe..12eabafbb70c9c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -586,7 +586,7 @@ struct arm_smmu_strtab_l1_desc { }; struct arm_smmu_ctx_desc { - u16 asid; + u32 asid; }; struct arm_smmu_l1_ctx_desc { From patchwork Wed Mar 27 18:08:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607263 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2E37CC54E67 for ; Wed, 27 Mar 2024 18:13:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=EHjhasjBYi7M2PWTu7YJ7wBN+4u1caWqgoQnKYzmk9c=; b=N/xGFYtVkc6Zmb Z5SZXq2dlUdgnlZhqWnGPSduHKIpcq4YKpWi3PqA2N+556Dim3OuJ02jFFCDh8kxJmgKW5fz6yp12 9eM5qQy+ZP91lDbZBdf3yJ8zsiwfCs2W4cmc2GxKtNFU6jWSVVdk5m/OIuxKOBTn6EpF0h5BBKKb4 zmn+9UoVFTVhesdiHLbub+LA1Z0RlFrrh0XpD2vcgOZxdF/OJPaUg8ZlCQsntBXwluqmHJC+8/1Mu 1Ucpw3vJFj/+VS4nf4OT7hpFL3+Gzu4l39d+LxzryXrHS8ixm/H/Z0d/ayVI43HgAYxO1kywUqslm W1dpWl7zxsiRzbDrR50A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXmG-0000000AW4g-1SKA; Wed, 27 Mar 2024 18:13:32 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhj-0000000ATr8-3BBI for linux-arm-kernel@bombadil.infradead.org; Wed, 27 Mar 2024 18:08:52 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=7STOWmLSNyS2xjMxQ1maFHfALEUKGwqSgk/uQYibJkI=; b=NjkYe8xI2jOQLwLpU+9v5scFaJ FBSBZbWc/9I4BlIHLisQFdTWDLVY8wK8eN1xQPoNxlsQUpaJsJI5j0gWoUakAsXIkpUhlI2qVN/4Z A0Zz7qkJVw9v33IwQ9mXKM6BicVP7tx1wG+rLmnoM/cAxwoE5/Xz75j5yRazMyYpF+HNjjp5tDrOy mekoxDtigtqEuuQakBIJBLnPQyy6ethWwFBOvlNs3R7oVGsJJdu09rCEWfrBPPlvcTkINaAc4nYK2 2FvL1981DO84E0YWAalTzjdB32Bj2v2pvx7iTOSY/mAaVTEy8nrLmWaFDTAiR//Yh3PbisMYPv6TP j4DPC88A==; Received: from mail-dm6nam10on2062e.outbound.protection.outlook.com ([2a01:111:f400:7e88::62e] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhg-00000000RvW-1n8i for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:50 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PWr0SW3I/7tVSK8mu2s8GbK/Yd3H3zsi9m2763R8/0IX+BtI/PGvHiP3g5rWWAhPnMdlC3FC4v7Ne+8NzWj/AFP5GQPDFPA0if5Flu6BIFy8siwpN05pl85e3T5IcXNlnlaY0elrMVoZMAYiROPiqVWpjXsGIXc8I0gGU7dEotS763isTdmjmjPoKpFaYJ1kvm49OApAaPM0CN08IFUTLuIpReihSVgJ/kL/PORfmRs82SodmeIUnSlZV1oS3WP0c4tQ2U83515dNcu6UdxOQ8LYK4y4Q0uRnaxMDMyBT5mkU5ON1/S7xXeyl1NlFwzuCKqNyuVfy3ypzPCKl6pjpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7STOWmLSNyS2xjMxQ1maFHfALEUKGwqSgk/uQYibJkI=; b=il+TzTmbOV3xfAtzsx/DtKPIIbyNdPBtEo0imym8Kzdg1mi63VXRW4ukw69sjYPb8orIBmxXtXYvMMdGhJyWXU9mP9KE4qlAAGyxN3C32Yepk2o+NBNhbPXTmzS1ddgbsAOODoNbWaJ/AOOErvfDiJsF+29Pph40/FeOP3mLgT/Dnr2eUp6PpTw/xG4+miy/01JhOclqn8ooq7lcM6x8zEljHVnFqYprPtQQFUSXFLg+MsYuGfdrdfAXSXZAyjQ5WggI4OCljDQaHUNJ7iSzqvObI59eIx15TE7ZEGqw5f9FfEa6EU3rRJo8CMCQS+LSeOlFXEEI4Wfj+TiEJJxV5w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7STOWmLSNyS2xjMxQ1maFHfALEUKGwqSgk/uQYibJkI=; b=YPOya+o/cpWd+YMNY+9Vmh46jRASEsl3maBhGs4JIkaQ7yjOidHhnd46y/eF9kPBJGf+N09lVucvvhmcJ7bcDluxY06lACjXYPMn5E6iqTn21wWFH+GxB5/xriz3Ks00tmO+AECuw7bMIBjaKqZjhHzfvfXPlQ4edEY6sgzuJaxK8ECD+f5p36M4ErXpLwvUsztIylghZGLLF9Clx1G6V+TNhEfgXp2ZajuXB1UAFyUUWlcYilhtqL7rkrRDgVXZDYvfgkMwF3oRWnzosuS2L6tUGhI0W/GP55StVzwihshUfZCJ63ZWlFs5CgqQpYykDFrDpVfaF/q4Rlucpy/M1g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:29 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:29 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 27/29] iommu/arm-smmu-v3: Allow IDENTITY/BLOCKED to be set while PASID is used Date: Wed, 27 Mar 2024 15:08:13 -0300 Message-ID: <27-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0302.namprd13.prod.outlook.com (2603:10b6:208:2c1::7) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 5f7d57e0-d3de-4e17-fa17-08dc4e88e19b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qFX8F/p5iEXqE6tml/BXo+kyN4aKWbmKtdi1WXRo3146UuF5jsxqMmH7t+YCTfVF6z6gvng9m1nSd1a+Y8TEpFaPJJ5CynD+UT1z+l3jxzfqO+cLw03pC7u8VNpL1VVu1yoIcx8zyVEiUW5G3Ip6ZFzYPTT9PFS+VW9ycfFOJxL5MzQ05ySTJPRBPJoGTvEJn6uKc4aXEMTgU5XldzKQCyNjUp4uFzTOGFYznkHzA19aysrTdfZAWikCOhJrhcWCztadEhb5EHS+tHjB0Hc10Lxv7xIvJQXbeeYJvO3AGLa30veu9GLBrkDls0co/ktR7jGu5DalHuSXPKVABZaEaxRrWdLF+leee1QoUtvLnUUjCqltJKluqyV+1PmrlmWh+DDeh3dEzaWGnGLpDB7rXvdamZ4D+0Eix9ZS1JBTUNkAleh9AlZXq86zVEmoTDGw2MrjsHMAVjsLzb3mDDfqjm7SwCYlCew+Ul8DM9emCKQti9dv4M443fKK0b8X+m9aAGM6ostZ4v4i8zxUFQFTN94uSxQng9sTjjNwH/iIXEmJMrgUjcs/tdZUwTdnzgpCvrdLzLWxG2WILcd3yBgDhrUDZhTqwf/Bb3FeW198NSmeOyvDOXZicYEefcH01jjL960uGWF0lSZ7M8MGj65yyw+sRwND4EuxiCx0TOp7N7o= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 1UudNqNLMGj2t2xPzJI/uBrPR1WeNINg/jwgfljS3dN9JFZLoBFVKD2tPt5smKgHylMcZD0oqC9rfJTT3VfLdRlxKHb6g9HpE96IJYSz3PYDkA5LmpfCcVm51FZkHva/ytlB3QtYGPU9JyTEb65kmcR5GIWl+c0TNel/SxklXnRtfzQoBLfFq3cH7bq3Eh5zyqYg8HwEai4o36uHWR6xgJRtrH8Ko91mWvZZDdaogXshCv6Q67VeugdoAHgTsKfIkjlvNCI2hPPoI7fOFFLpUnLWvP3EqIgIH2Ix4emkf29P0yx8o/xeu7WvYiIwk3Kt34yZVuDJLDbfFR70AJ9s0Q3DOIyma5B7QTtQlBCOIbyOggdclzVjkQ0tXi8iwkMXA5z3jn4FNdas6gygU7gSfDtMBJjCM0eKYgpq2H2SvmMsNSnMR0mhk4eRe9Mgb7UtzAXwjthd1YKX113eLjQM+/DbCfWmcav7iBguKdZNnL8l/UBtBJW6mmonh/wEPh5o8Jf0yADk5dwuJ72F+Wgvw+bGM3uWODbH4PqevFckMUn54uGsnarke6wG0wKMzZLKHu4PSv8odEFn96upiL6OiVKrcXiZZOmg3l4l3ilThbeYsxi/A2QB7Orux3HxPCkYGxX0ALgCVx/zMRquC/gsRLYuIQv/fvPMnEOWA0u8v4oun3vXaWO+UiVhhLLCPHLiY4n34sfiFnmmdMJ7gUL14L9O8G0g+Vcee29/3DIRXXhNQWYaLIU4rbXpMdiEFQB+bQZPaUED2oLYauNzRDyjjCw3YE2MMKanwIIn7m+J4GxClA6WAYiRw/YiElH9/qaK9juX/3UYI/BcC+BwruPtQF7xtP5O/e30f2rIPBAQ4VkYuenZnL9jISOKNPS3xKG79vahbo+EMXPslo+X1RvS7fhrYpKcFaYynmjd8f6g3BlZkjI0coEcIDniRZGnimodM6X4sDQfH74W+1g+tHZ4m0mC6G/y6n+pAycWTBbTE+BwjKdrOlGTwP2IkoAFjYMxsSB0DnH+wEjwvtYCulFGphd083Db6NEbCcqD8M/gEMzR2+P8p1fMne+Yv55h7/bTxgIx/FjiUdREhM3T2kplfpKBHfTZvnJtsa0RKOjXkEQ9MqeOffReZN6CpLJUrKdY0FaMMEZnFW15CY6isISeU6P4uORbOb5Z2UXEakw1vO9jK71fzmmbLW33x/rtDKNSBtsy/kTZ7wc5TkXwJK+wiiXxUypcG7Vs3GouV3U5pUFP7YFDo1gMz71YsBvpfc5oNHuZENNzUbEHc3f/hBUev0TZLmPn2W3HwJKjA5ovL1mKPIVktTzXgIiI9FnjzuBJ6WDmwaOTR8bEjdZI+RWryRLU2wRQ6lmQkKpahNN6ohxmQFIEFZPAs+ZyMNiqxTS9+3d+CXeetjMQg9CL9qGQUTsV8Jc9Bd86cHI6T/riE4n21YZPgdbLNCbZXqMVSov+ywaUEpy8Ta2g7N9Xs4M5dnpL0tVpp3GC8wdyymlJKf4sCJWcDVSscgb+gTM4cAE4qR99CRyVhdxf/9Xt/l28helXLNRQLWzO09iFjQDWjdnMkb2PVjjw6oiKZKZpJMQb X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5f7d57e0-d3de-4e17-fa17-08dc4e88e19b X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:18.8791 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FqtanR7C6CH99JegimZGlnXeSrmesVYkBE6UU7c4RvDhYQgyz2dwzQEiJ8ykF0Rv X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_180848_661615_7156B71E X-CRM114-Status: GOOD ( 26.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The HW supports this, use the S1DSS bits to configure the behavior of SSID=0 which is the RID's translation. If SSID's are currently being used in the CD table then just update the S1DSS bits in the STE, remove the master_domain and leave ATS alone. For iommufd the driver design has a small problem that all the unused CD table entries are set with V=0 which will generate an event if VFIO userspace tries to use the CD entry. This patch extends this problem to include the RID as well if PASID is being used. For BLOCKED with used PASIDs the F_STREAM_DISABLED (STRTAB_STE_1_S1DSS_TERMINATE) event is generated on untagged traffic and a substream CD table entry with V=0 (removed pasid) will generate C_BAD_CD. Arguably there is no advantage to using S1DSS over the CD entry 0 with V=0. As we don't yet support PASID in iommufd this is a problem to resolve later, possibly by using EPD0 for unused CD table entries instead of V=0, and not using S1DSS for BLOCKED. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 65 +++++++++++++++------ 1 file changed, 47 insertions(+), 18 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 5a2c6d099008ed..69b628c4aaacdf 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1003,6 +1003,14 @@ static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW | STRTAB_STE_1_EATS); used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID); + + /* + * See 13.5 Summary of attribute/permission configuration fields + * for the SHCFG behavior. + */ + if (FIELD_GET(STRTAB_STE_1_S1DSS, le64_to_cpu(ent[1])) == + STRTAB_STE_1_S1DSS_BYPASS) + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG); } /* S2 translates */ @@ -1531,7 +1539,7 @@ static void arm_smmu_make_bypass_ste(struct arm_smmu_ste *target) static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, struct arm_smmu_master *master, - bool ats_enabled) + bool ats_enabled, unsigned int s1dss) { struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; struct arm_smmu_device *smmu = master->smmu; @@ -1545,7 +1553,7 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, FIELD_PREP(STRTAB_STE_0_S1CDMAX, cd_table->s1cdmax)); target->data[1] = cpu_to_le64( - FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) | + FIELD_PREP(STRTAB_STE_1_S1DSS, s1dss) | FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) | FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) | FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) | @@ -1556,6 +1564,10 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, FIELD_PREP(STRTAB_STE_1_EATS, ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0)); + if (s1dss == STRTAB_STE_1_S1DSS_BYPASS) + target->data[1] |= cpu_to_le64(FIELD_PREP( + STRTAB_STE_1_SHCFG, STRTAB_STE_1_SHCFG_INCOMING)); + if (smmu->features & ARM_SMMU_FEAT_E2H) { /* * To support BTM the streamworld needs to match the @@ -2732,7 +2744,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, &target_cd); - arm_smmu_make_cdtable_ste(&target, master, state.want_ats); + arm_smmu_make_cdtable_ste(&target, master, state.want_ats, + STRTAB_STE_1_S1DSS_SSID0); arm_smmu_install_ste_for_dev(master, &target); break; } @@ -2809,8 +2822,10 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) mutex_unlock(&master->smmu->asid_lock); } -static int arm_smmu_attach_dev_ste(struct iommu_domain *domain, - struct device *dev, struct arm_smmu_ste *ste) +static void arm_smmu_attach_dev_ste(struct iommu_domain *domain, + struct device *dev, + struct arm_smmu_ste *ste, + unsigned int s1dss) { struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct attach_state state = { @@ -2818,9 +2833,6 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain, .ssid = IOMMU_NO_PASID, }; - if (arm_smmu_ssids_in_use(&master->cd_table)) - return -EBUSY; - /* * Do not allow any ASID to be changed while are working on the STE, * otherwise we could miss invalidations. @@ -2828,14 +2840,29 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain, mutex_lock(&master->smmu->asid_lock); /* - * The SMMU does not support enabling ATS with bypass/abort. When the - * STE is in bypass (STE.Config[2:0] == 0b100), ATS Translation Requests - * and Translated transactions are denied as though ATS is disabled for - * the stream (STE.EATS == 0b00), causing F_BAD_ATS_TREQ and - * F_TRANSL_FORBIDDEN events (IHI0070Ea 5.2 Stream Table Entry). + * If the CD table is not in use we can use the provided STE, otherwise + * we use a cdtable STE with the provided S1DSS. */ - state.disable_ats = true; - arm_smmu_attach_prepare(master, domain, &state); + if (arm_smmu_ssids_in_use(&master->cd_table)) { + /* + * If a CD table has to be present then we need to run with ATS + * on even though the RID will fail ATS queries with UR. This is + * because we have no idea what the PASID's need. + */ + arm_smmu_attach_prepare(master, domain, &state); + arm_smmu_make_cdtable_ste(ste, master, state.want_ats, s1dss); + } else { + /* + * The SMMU does not support enabling ATS with bypass/abort. + * When the STE is in bypass (STE.Config[2:0] == 0b100), ATS + * Translation Requests and Translated transactions are denied + * as though ATS is disabled for the stream (STE.EATS == 0b00), + * causing F_BAD_ATS_TREQ and F_TRANSL_FORBIDDEN events + * (IHI0070Ea 5.2 Stream Table Entry). + */ + state.disable_ats = true; + arm_smmu_attach_prepare(master, domain, &state); + } arm_smmu_install_ste_for_dev(master, ste); arm_smmu_attach_commit(master, &state); mutex_unlock(&master->smmu->asid_lock); @@ -2846,7 +2873,6 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain, * descriptor from arm_smmu_share_asid(). */ arm_smmu_clear_cd(master, IOMMU_NO_PASID); - return 0; } static int arm_smmu_attach_dev_identity(struct iommu_domain *domain, @@ -2855,7 +2881,8 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain, struct arm_smmu_ste ste; arm_smmu_make_bypass_ste(&ste); - return arm_smmu_attach_dev_ste(domain, dev, &ste); + arm_smmu_attach_dev_ste(domain, dev, &ste, STRTAB_STE_1_S1DSS_BYPASS); + return 0; } static const struct iommu_domain_ops arm_smmu_identity_ops = { @@ -2873,7 +2900,9 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain, struct arm_smmu_ste ste; arm_smmu_make_abort_ste(&ste); - return arm_smmu_attach_dev_ste(domain, dev, &ste); + arm_smmu_attach_dev_ste(domain, dev, &ste, + STRTAB_STE_1_S1DSS_TERMINATE); + return 0; } static const struct iommu_domain_ops arm_smmu_blocked_ops = { From patchwork Wed Mar 27 18:08:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607334 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6274CCD1284 for ; Wed, 27 Mar 2024 19:18:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/KOiLLlyej8tL6ZWfP5r+eNxSrvH2XG310VnU9HA8WQ=; b=Ij83UmfwdlLtnb KHbQzVLFKY0YLnFIY5ArLhHFgAuiayWgM9hflcoYyP2OQsQso3VZk6CE9leeSA/feS+pFcQmljmju 9tpDwV8gSXApKqwa8AawJwZZu51LZtVKqP+S8o7LQd+3GjOH2SZahZKQkzkWdjmQMbULJMIOMaA2Q TwUwma7SzTLYjS0HDtJ59Wu1pvvavxQUU2csXURK/mjwe8M1KAQLu95mMbwLdxMexGbjNYPkXZfi5 bIMm0URkGt8o2+Qj8+e2eX2tjd5mRySnVLPYU7hU9P3pD8sCCFAlPdUQlzQlkUXJ5Nqqoqr/MypQp SplRGdvdE3djvXImJ01Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn9-0000000Al1E-0nmk; Wed, 27 Mar 2024 19:18:31 +0000 Received: from mail-dm6nam10on20611.outbound.protection.outlook.com ([2a01:111:f400:7e88::611] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhr-0000000ATdz-3a8C for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:09:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EJdIqfP1KPzr0zOVmxS6mxQ9ymq71lwRvhXBoGJYjp15lh3/OVNf9GntFf5kLMt/rqc5iOhdgW6YX+7tpTDtDHQcEAde+zip3zmwpomIySjlbqFsmgctpdmkLYixWVnG81IYua2MFgY2vdPsvQH/5WlrxmYqWDRAm1p8NSDoHGVeMlLCYz1GkRj08lY6O5pu4tJEZzO9T5+aYtbRQMs/UVb5qIYc15keVSxDQdVRb7g2Q5y5FL86sqikwYV0/HMz6u1vJibssVphSusZzmQR7UMbSXu9dsmG7Oy1G5evwuskfkCZyU5XwsbhaLPC5JM4ONVCBhjQYhGVOthB7+H1ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RfDowk3NmvxOw4FdVo6SbySPfzW2QPXgwjAU/zIKZGU=; b=JdJYEeedqnzfq8ZKIDeeBZiDlTkM3ognTfC0eQsJG7X4Dme0pwImiSbc+8mvU7rVjeaIZPZ3pCYqtM7jRVnlSAvcz3RH+PJJ+uGEZE0nlnpS6g0wJY3byDJdkm0l/FVh2hUflxxNIPlP/nYaJXuKZHWJniabyAwBqarUHj2ipWEBUCmbi2anvvAn0ShdkZfT+LU6prF3jHH10Cxt+i0Z7aIbALBJMpUe3frQ/KyEJ7vSOb3O3XApB9veW7rYcacgqwWAEUwPjONxUBvUwjCt3+bY/43WEec2PdD87Lktfeej7EeIqDKLGrM7u/PoDwBRbTB2qh4rWrhTGZk/F8ESFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RfDowk3NmvxOw4FdVo6SbySPfzW2QPXgwjAU/zIKZGU=; b=stxiZhvPzd5ouQ/XrR0+3B6u1SdADTeuZZYd8kO4Nq4sK+LutGFrJS3pfbU4DgZ9qD8sE+ayMLbYJLIh8K6fFT/iDzJQvVTvqKqqzwRCNSErX2POMsmhpI9RsWoDhhTfs9/Z/1sehSIIyEDW3jIM86Fk8wlw6WdxRKkULJfnOmUzqoLCmW7fXdqIrbcuBeVI0F5nJrCVs1hImcz2II2c12UHSMexVZruh0Voivlw3lxkdnKBxDLfQr3emjSNhGmZFsmT8f0vn6i5IZEQ0/LeeyRMNVekj5qkPfaBDwXviDMTBnN/lLvPwQI9PKtfgrRvQmIBYrIu/nFpLATFEUza/Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by IA1PR12MB6044.namprd12.prod.outlook.com (2603:10b6:208:3d4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.33; Wed, 27 Mar 2024 18:08:33 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:33 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 28/29] iommu/arm-smmu-v3: Allow a PASID to be set when RID is IDENTITY/BLOCKED Date: Wed, 27 Mar 2024 15:08:14 -0300 Message-ID: <28-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0317.namprd13.prod.outlook.com (2603:10b6:208:2c1::22) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|IA1PR12MB6044:EE_ X-MS-Office365-Filtering-Correlation-Id: 6737e32d-1934-4597-e5d7-08dc4e88e245 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: nutWMHUWVb5xtTYMpX3NvkG6KiKAL+ahJRigleqSFo5LMZBTPXPtfwBrrD2OK2Ss+JYGuphED44xAgYN+1JSxw4VniXWTXJ6L42KoKfX6p9fZ56XI6z5eQChGcxTD4cINvhFLjJr4Sfbo52UQEK0o5vxjCKtvt12vrsQpZKaEJTNWEHo1n9RW5Av1Xvrbd1GEpzGqwT7DqUCoepXNFea43qvk/3kTnk5660NUzTWVta0lYegWeytF1Sqj+v7jWQmowoqmyR/jyEwsXRyNnX2hBtIbfgaTX7mPJJ4ldozwjefgG9DYhD9u7AwsGJ2aOJPolSnCWBhMHS+JYmtCoqD3M8BW+IWgzAH54h7hkfu89cPdQ7ERgC0AczHk8pPu7x9PUASunXMbWOJu7WenOTruIfLcqDpBvPBcrRHT8DMQ7/MY6+1bUbAFuMcB3f4yzJKt1JpADshH330kiPb1c81wUxwzADRDhjlnAEJfN4928MsM4lpaVYZ+M5gxWe4vysYoPtUnkq+G9TxfxFJIKDiqZUpIHlZIMi6axtK/jaJHjU1KwObtSWIUbgvxCwaNDK97v2HYVPJXk6hSsCJJc19em+RliTlhEmOAcKe4KzUURIYaMkWaj5vie8sBy7xnmGok81vvH3er1pNiLz0+gu813T1uiC4EknRkoL7a/hnA1Y= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7416005)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: crcQugL9sZl7yo1Th6TQYB9OrwQugxpN0l1+9xw/7KzZ2sMPpC2ldfQbcQw7ufEb+/ioVB1SpkDFyjRMYpl2yS2kxF3+p1S225W7PkaNhCtFnopSY0XhIeutBWbE/PY4BvRbiRrOkm+t+HQATrlvq/EdFC6H7Ab6euoRxWv0VTthkzw4FnyG8hbdhrpocfGBzoh3MLPjocMEeQPgve6NmdwXR4+ynv02HKXGkf7FhLI5JKV67wICNwRtkmGsodU11kmw6zunpM+3Cp8d9UK+/toiAMgL69fB1Gq/EwduldhdExq3bRF6ZpHCf+uFKHHhzZYiCH4jvGlQsS3tpZ9WSnVkwtc1sXiw1Wd7pXVwrFVYUqsn6nipceJ4OmFiNOc3+2FYv9kHK45YEtfj4K3m5WNXVKww4iM/wVpmKNPzen2Aov8umvhqovmui7LNPAo/Ovyllib9slmceBkCEmhmyRQ4ScMtH5aFsd8ALnbYimjO/ClFDI1rGqhfAyRCcQAELdIHZFVhds8ZigRX336/tYEkYKXUD+L3ja+z8gWN7AUvO1vTCDj2VG5XpkyG0f6yuYJdxq3ZMrwX2cDhK32DPsfHs6Eqn/U8jMzu9hVIzIJEPuwZr+J6DdRRAhv2WtO7aFYbzt+TW+xWlOKKaC3yYs+tSv6IflJthjxr1q6JHzc1gcw2TeXtswFTgYWnW/MEWGJc+fJJWW3ThAkdQu4F8PwpePYnnE3DewLUR4UxKWgjWXMGJK5dCyDphcEvqcvRmFSRsDuwvuXqfFAhh5x8dNdywgnv0BXGbiHzKLdYrsjxK/eYwzM946R+7oiuKR15uL1J3+WQ1lAsGdhTUrdfQC7eyjmB6yIqEuxTgoSETvOwzRWYCKo8/eKE89oSPTKFMl1s7iUl9/LIG1L+udBMHYZ+xyHge5Nd49Kfn3UAki0EsxcnwSS3c8aRu7kF8lA9F5Zy/9pBTm1zuL3fF/ogr2Hr1xbxzGQuPQ7ehNnoxt9KgkCsvcFJArDvuSjFuA9qLkKxitIIj6JaOLNhoIJtlKG24ZY5E3XsGgDhQWdaXPlLTMu5IymH16vdclNC+aiCD52evF4ViyVvV1ByWkvxAJnjPHNrQ30xOS5rDKQjwPXFM29V5cJWpFRHmxdsCxvZGd9Am/ysYurV3BnCUCw72QN80mV8wz8W5VBXxESbheJOJPN3OmHUZGlUoteuvUmLCWp1FSzJQEUt8YGhth3S4jJUJSeBjYb6oZR7jfuZ3aJ/hWciMHTs7UffNUP6XIOQjHQ+gBscMrrcV32rC6+J0jXuJpDHZeFufSoTjb8mzh7HAgWYa5R6rWpMT834DP/aIOiG4hMRnAWK9JNpCd1nyO6DpeQAWmbKbtRH4T7jAQd+ouH/fLhfWlSLTnmm2tEKURttOeNKdhMxAJ8LMKcPaRquotGbEJXzwi2InEpbSltzjC+mDG/OBm4tl3xIWNBJt8DTxAc+79B7BL3hw2iWeVuyVtZqJESjtLolzZWfdT/+LnqqbgH6qyoeKhh1yS26Exg8Vv2CEPtI90YbjI7RMa4xkCujpx+BZzFVA6rqZ5Zu99+RNdSpqgJigeDIO3dR X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6737e32d-1934-4597-e5d7-08dc4e88e245 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:19.9479 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: o6BrWyifMEyllIBtEchuTbZsi9sHmE4Vg2NNMXKcOLaqpShNEiswdQsVoTnHb5xb X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6044 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110900_387652_C9CFC0B9 X-CRM114-Status: GOOD ( 22.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If the STE doesn't point to the CD table we can upgrade it by reprogramming the STE with the appropriate S1DSS. We may also need to turn on ATS at the same time. Keep track if the installed STE is pointing at the cd_table and the ATS state to trigger this path. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 49 ++++++++++++++++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 +- 2 files changed, 49 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 69b628c4aaacdf..f87525225c8a50 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2432,6 +2432,9 @@ static void arm_smmu_install_ste_for_dev(struct arm_smmu_master *master, master->cd_table.in_ste = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(target->data[0])) == STRTAB_STE_0_CFG_S1_TRANS; + master->ste_ats_enabled = + FIELD_GET(STRTAB_STE_1_EATS, le64_to_cpu(target->data[1])) == + STRTAB_STE_1_EATS_TRANS; for (i = 0; i < master->num_streams; ++i) { u32 sid = master->streams[i].id; @@ -2762,10 +2765,36 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return 0; } +static void arm_smmu_update_ste(struct arm_smmu_master *master, + struct iommu_domain *sid_domain, + bool want_ats) +{ + unsigned int s1dss = STRTAB_STE_1_S1DSS_TERMINATE; + struct arm_smmu_ste ste; + + if (master->cd_table.in_ste && master->ste_ats_enabled == want_ats) + return; + + if (sid_domain->type == IOMMU_DOMAIN_IDENTITY) + s1dss = STRTAB_STE_1_S1DSS_BYPASS; + else + WARN_ON(sid_domain->type != IOMMU_DOMAIN_BLOCKED); + + /* + * Change the STE into a cdtable one with SID IDENTITY/BLOCKED behavior + * using s1dss if necessary. The cd_table is already installed then + * the S1DSS is correct and this will just update the EATS. Otherwise + * it installs the entire thing. This will be hitless. + */ + arm_smmu_make_cdtable_ste(&ste, master, want_ats, s1dss); + arm_smmu_install_ste_for_dev(master, &ste); +} + int arm_smmu_set_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid, const struct arm_smmu_cd *cd) { + struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev); struct attach_state state = { /* * For now the core code prevents calling this when a domain is @@ -2781,8 +2810,10 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, if (smmu_domain->smmu != master->smmu) return -EINVAL; - if (!master->cd_table.in_ste) - return -ENODEV; + if (!master->cd_table.in_ste && + sid_domain->type != IOMMU_DOMAIN_IDENTITY && + sid_domain->type != IOMMU_DOMAIN_BLOCKED) + return -EINVAL; cdptr = arm_smmu_alloc_cd_ptr(master, pasid); if (!cdptr) @@ -2794,6 +2825,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, goto out_unlock; arm_smmu_write_cd_entry(master, pasid, cdptr, cd); + arm_smmu_update_ste(master, sid_domain, state.want_ats); arm_smmu_attach_commit(master, &state); @@ -2820,6 +2852,19 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) arm_smmu_atc_inv_master(master, pasid); arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid); mutex_unlock(&master->smmu->asid_lock); + + /* + * When the last user of the CD table goes away downgrade the STE back + * to a non-cd_table one. + */ + if (!arm_smmu_ssids_in_use(&master->cd_table)) { + struct iommu_domain *sid_domain = + iommu_get_domain_for_dev(master->dev); + + if (domain->type == IOMMU_DOMAIN_IDENTITY || + domain->type == IOMMU_DOMAIN_BLOCKED) + sid_domain->ops->attach_dev(sid_domain, dev); + } } static void arm_smmu_attach_dev_ste(struct iommu_domain *domain, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 12eabafbb70c9c..853cd17d06e671 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -706,7 +706,8 @@ struct arm_smmu_master { /* Locked by the iommu core using the group mutex */ struct arm_smmu_ctx_desc_cfg cd_table; unsigned int num_streams; - bool ats_enabled; + bool ats_enabled : 1; + bool ste_ats_enabled : 1; bool stall_enabled; bool sva_enabled; bool iopf_enabled; From patchwork Wed Mar 27 18:08:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607260 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 799B1C54E67 for ; Wed, 27 Mar 2024 18:13:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/pPzzNopuvEfp+DFgRAowq51G+5I9ZkKfDgmq6qU0I8=; b=McgwLj2D4TtGqw 6siGgiU6LZmNQ+sey8tEN1cHHbd+nzaAeY7LKCd3ONq6ArAFsv2fwq9wzMEYJEuooQAvdXCa/IyPH qSTyq/1bA01VFOCmOQA0zm10yQcUFLYhrsQDrcI6t6FOwyQovS+c9GcOJbpQafUnr3+uEPPYT8XwH rlI445VO7exSSf5BYiSAPJrQGbALZx/71gThCyT6eswc5MHCzKQGfa2kiQvkrnje1pgPvDQEPpbEp t3/vz9vl49cxxVxW0A2doMO1hPB272b2lg/2uHbIhy/UDyeGFUMVM5ADsYsoAfSKxdX3WK3g9IYMC 4WVw2MSLfpkVCqv7+AOQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXlz-0000000AVod-0uoS; Wed, 27 Mar 2024 18:13:15 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhh-0000000ATpE-1DYe for linux-arm-kernel@bombadil.infradead.org; Wed, 27 Mar 2024 18:08:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=JS8oOgns0RpldlkoAuexcNfWp7baY2eah7vEr5Y/FZs=; b=hp/L/VZI6G0iw4Dc+1LxotKjur TdJf13Bzokqil8yKYx6WjWdLRvsWSxXHkwUhci5HbOR/JaJclEIUp7bFBTheObGjGA1+ganRq9RqP O+Vwx73Vs8tr8LBNnLJoVcBu4V7JOjzYkBfzLCR8RKNmCQXUvS8HCrs4+Tz4fcRZH+hf8VRFQc7L8 bb+ez2mGJ6iglLg0kDH1CqOeMvI4hEmFGwdo1us3qj0h4TdnPRGdqsKK7IffGqDXa7OtuKYqSY8KE eTtRFePDaBesoywN9+mXOuxsDB2tL85qB5R1p+q0vwx+xUB/cuVfuJ3t1JAfjFZhWe6o+BJROd63a HxYnm4mw==; Received: from mail-dm6nam10on2062e.outbound.protection.outlook.com ([2a01:111:f400:7e88::62e] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhe-00000000RvW-0ueK for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:48 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mYHFioXMR+BXyIJDH0YLCGJO3a0Cjl4HEIGTYJJq2EvgmQcI/+oNxOzbtYXl83jyNYG9V3UmRGETyAmluO/7f1caozw2NKqrHdClNrNZVoOcRZbzXjHEZFd+0FEl32oIeR+4qXn0Stbi2VNGhuQZJC6248fwTl1PPRWC+6ayZypu3bm4RqL3DJm9xQPnA2Q18EBZqmC4ijGIuLspZF8MnTYSyE/6BJdaEV8kmsjKpkX5t8abiibnfxxdJJ7PxhPOziRUJ1rw2yYNX7yBAhwgTQC/KkEw66L3gztA6hstInY4otarwYRNSQpMPvcLILs4d92GjDiQT/dFEtgdH5w2wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JS8oOgns0RpldlkoAuexcNfWp7baY2eah7vEr5Y/FZs=; b=BPVA8PqEb10smbfi6MQ5xnyVpnkAJT9+4Hchp4gLlJxlnJYVi3DwzXJ/N9l+VCFBRCgfW4OkFV1z1tw9dKApDkxrA3d7cgXa7N7SLqWUmb3k2Qic28Ocswj85bMaW+SsWgIYrO6OoEecY36vPTp/4Wda3YnUj3k7UwYETgjEY/aZm5MeiR2j8C8Ezx1rQgQtfq10jnmVL72TJ1SD2R7YJSJ6btInjqbdZXndYHFJ/K3NBztKk05HRwOsAmlWyGw5lDK4SpGY9bjTe+QBYMvnc18J8Mu0QQTnvjaM788u8RffAAezDN3SN6DuekmAp6YGRFvC6dNi7KZT/u4qxVTD7Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JS8oOgns0RpldlkoAuexcNfWp7baY2eah7vEr5Y/FZs=; b=OCqhxdCIc6w2jmv3x9egU9si4PfeRxGA1WjlrdH10MlcABtTtmzkkbRBz0XiPkKz2v7iNHgRYxFbiX3TdcoOROoXuBDq5v/nAaCpckxJ8wfa8zbtYahfrf/kR5RlRPTDtfpyEev25+G3R3a9UMqclb8RprJSO7t6eT3plq1MFnODcMAf99Sl2V/6WycPWy4NSyWxEBKPWFWWPR+Y3i0xsLNuPVAZEeAS5Q50NYNoRjoDxyLXOmuZh3vEuCK7Oip3AovH2ZqH6ygrIu6CFQIvNxVQRgjH0LtUqK+OnQILV4e2YLZSRLTCdO5sLt3EcFSk3wwq9bLxvasFq3qHzDSHZg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:27 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:27 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 29/29] iommu/arm-smmu-v3: Allow setting a S1 domain to a PASID Date: Wed, 27 Mar 2024 15:08:15 -0300 Message-ID: <29-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0064.namprd05.prod.outlook.com (2603:10b6:208:236::33) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 13c07355-e622-4bb9-bc27-08dc4e88e17d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ANcPo2CZJHEEFlwK+AL3s2Uv59HhScPGKmsg7RHo1pTLAjlpO9/7k+QDrEdXneUh831s27m8S0myLwoDADchuOkaGerCEIVmTEmm6DDr36rl6OacADbuPwIOG5t5hg5PVe00S5D5EfTTeLfoaC1yIo3aCA2gqDIW+UJhlDTWgGi4GPblF1IItIYWq8fHQJptCzxP4gmHfQPu+7E7QICogsMUyzHwB4zT1NgTQk4SWY1QZC+g+rXYfo8Q1WeYLGVn0H+tOsAqTAYLTJVpE+6OxrZ8xk+pKTGaci+e3cSQwe+/wwSJvhyZciwoNEK2SBZlMkF9DqA9cUyg+PVKKxPcqILvnEK3cNIAxOSdZPFTSvyqSrA2292jn/vkFwek6UviQSFkjdW0jlhFScDNWUgJ3ikcMvSKlXXlROuA7kid3sQD3hyS0PfNLC+fDpnSMt0vV3O5SK0e8eO/9vt6ReFWranMci+aAyr3EZ9ANrCRgSHSwWEtJrl34Xdhp1RSpnvWXUJmse8NEqYdAc15HHonaiXQ+2XK2eBzZGPLAPAzeXNXwMmBSEu0lSzP6qNcfPrzvuyFWFEVPJwAqeo+OpgJfi4VrMU6nRFv2Iqbjt40bUgQo/rnUf39CRiyfsxar/WwU7rv0IbL2VnS3LsOmoFnqoF1EiAAKNknGSo0PnPygJw= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: EdNlyjGXK54A+fwGh0QxnbZ8RYOwL3vRcqahYS93JTDmQcikq3ZimJfUG2TY0A2AZ89WeCog3BN6eyfyiw6uk3FXBtx3pZPJCup9tphrO0VLna2cjhjVjCvlTKSuKfryG12bKBxkDM7aKwLsOhliFWCMHVXNhtiQfAnzh3RaNcCg96mSWKBtuK0yh+Qbw1piJK+cYOhtQ1XeqE3LuqkQJzxWB+cpeEEHqxcZ/4RnUOf1coNoWAxL6jwphJbE3i93Hg6ADzVZF1sDt+GVpmoF77sq7dkeWxgAwc9f0UsHB8grSG7OMukmWsdySqOjbwrpxDTu7OpmT9w1or6TObXdy2OMb+1mH8q/utbzS/BNSylnCklqr3t3DJazacH3upZTHGYJhS6/66DSX7CqmDZRYoPI5yDGthD7hgA6jKqkZ2gWyM3QWSm8QwlU0WrxrfxOHLMk73V23wwn2v6FybjNZht+YqGNLglV1j8Z1JygizjRjKfecQo8Hm8wQn6YsRe9eRksrr6p6zCGNk2S0E97L63jNWINKRMoPI4Q1ZF9s65lwytm40TCjqr5+FX9tIvh8dK3s4OQEa4EyzNegae2DGlCxdFNjy4umz+SyOqfCzRFv0+1ZRw2F2DH4C5WMy6UQD/8tUiKE1px7rjCNuxKIGDBEygL8WAH15F9bL5o+eo8pSEI1X+kqZHddV2Xctcc54f6ATWGAPkgSxNJCSvpg29uOZsGkfwMv/fsqxdpPeCO0TPrCYDS407C3TUbsIFLE8ZBMGnPe2oDe90Xpm3aoIWFF6/596Aw5Rp4laBUoA1A/5/MKQbNxHzFIXRLFASqlR3xl8VrJupg9faOeOlW3W7u+dZAP3vKlUZdsuzIno8bzE2b3jfGWP1xoTn+ENe/uuSfuTcn8F3QH/bj07uJpKhrE0SzIsID/IaIrebZRrbGTcFOQuqQ3WanaYzulLY1fxmrTr4iPfxtws3pZTZr7i6NgW/rNZEVOjj555eEMPwjXxqiLSwZYi7MgIvJQquk9BGiHWq3xwH9PGQ9zC3LmXl51v22nD7i1Ne+G5RLeYPE3XDPxl3soe76p4xhnRYNqJ3us5F3y8oGoMR0PiAxiAsUz885aekzSbvx7ffer7DafF1DSSeWV9x7L5WGe/0oRmvSTa/+eVEvHuy+1T2LiVaSl2t1lg0t2CmeaOT7glEvVNgFyW9nfUZa3I052zlbsQxkExtpEi0xXc3D95r+fHV8G5tx55HSwPdUE25nlGPFtepy2R793ddM4EOW2dscfDnzaHFWMgDEa6bvhucwqvqIaBdZy0QzbyC3wD8RZe0uzcb5qrCaQjudXOYY9DPaQKKEnVWsjjk8DBSTyv1pvn8MYH5wHv09W4sQKclLg1CF1GQ69ilGe77A2+z+dyIi/zssK0C92NUQXBXw1b0zQhM1oHAaJwt736bVsRGGWlIpEUec99AklxdFlZqMbBpBOVArYwAzNbkNoNTHiYqwke/2I81uf4FR//DqCpSTPjDh+VFyZPCdg2bNc61tLAddxvBkbRJ+MsE0vbQJVB5i3NXZjHzXXjCBXDmNx1cQG0bVNoyqJSst52qV2hL67yIx X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 13c07355-e622-4bb9-bc27-08dc4e88e17d X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:18.6290 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JCKSDw/n0H2GwDoB4i44KvnOixq+TH7YDVrL05cheOmbK5jdBLAtUHvvsvcPpCKk X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_180846_362934_819A8E9B X-CRM114-Status: GOOD ( 17.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SVA cleanup made the SSID logic entirely general so all we need to do is call it with the correct cd table entry for a S1 domain. This is slightly tricky because of the ASID and how the locking works, the simple fix is to just update the ASID once we get the right locks. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 45 +++++++++++++++++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +- 2 files changed, 42 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index f87525225c8a50..59f24602e24d68 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1332,8 +1332,6 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr; - lockdep_assert_held(&master->smmu->asid_lock); - memset(target, 0, sizeof(*target)); target->data[0] = cpu_to_le64( @@ -2765,6 +2763,36 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return 0; } +static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain, + struct device *dev, ioasid_t id) +{ + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct arm_smmu_device *smmu = master->smmu; + struct arm_smmu_cd target_cd; + int ret = 0; + + mutex_lock(&smmu_domain->init_mutex); + if (!smmu_domain->smmu) + ret = arm_smmu_domain_finalise(smmu_domain, smmu); + else if (smmu_domain->smmu != smmu) + ret = -EINVAL; + mutex_unlock(&smmu_domain->init_mutex); + if (ret) + return ret; + + if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1) + return -EINVAL; + + /* + * We can read cd.asid outside the lock because arm_smmu_set_pasid() + * will fix it + */ + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); + return arm_smmu_set_pasid(master, to_smmu_domain(domain), id, + &target_cd); +} + static void arm_smmu_update_ste(struct arm_smmu_master *master, struct iommu_domain *sid_domain, bool want_ats) @@ -2792,7 +2820,7 @@ static void arm_smmu_update_ste(struct arm_smmu_master *master, int arm_smmu_set_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid, - const struct arm_smmu_cd *cd) + struct arm_smmu_cd *cd) { struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev); struct attach_state state = { @@ -2824,6 +2852,14 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, if (ret) goto out_unlock; + /* + * We don't want to obtain to the asid_lock too early, so fix up the + * caller set ASID under the lock in case it changed. + */ + cd->data[0] &= ~cpu_to_le64(CTXDESC_CD_0_ASID); + cd->data[0] |= cpu_to_le64( + FIELD_PREP(CTXDESC_CD_0_ASID, smmu_domain->cd.asid)); + arm_smmu_write_cd_entry(master, pasid, cdptr, cd); arm_smmu_update_ste(master, sid_domain, state.want_ats); @@ -2840,7 +2876,7 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) struct arm_smmu_domain *smmu_domain; struct iommu_domain *domain; - domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA); + domain = iommu_get_domain_for_dev_pasid(dev, pasid, 0); if (WARN_ON(IS_ERR(domain)) || !domain) return; @@ -3362,6 +3398,7 @@ static struct iommu_ops arm_smmu_ops = { .owner = THIS_MODULE, .default_domain_ops = &(const struct iommu_domain_ops) { .attach_dev = arm_smmu_attach_dev, + .set_dev_pasid = arm_smmu_s1_set_dev_pasid, .map_pages = arm_smmu_map_pages, .unmap_pages = arm_smmu_unmap_pages, .flush_iotlb_all = arm_smmu_flush_iotlb_all, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 853cd17d06e671..890fc6628d5e0b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -768,7 +768,7 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, int arm_smmu_set_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid, - const struct arm_smmu_cd *cd); + struct arm_smmu_cd *cd); void arm_smmu_remove_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid);