From patchwork Wed Mar 27 18:07:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E9879C47DD9 for ; Wed, 27 Mar 2024 19:18:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=W1+mS0hORV6ztHfWM9Q9Md1mO9DknBb9vOhArP0CYAU=; b=izpSZd3ITabOMM 0BGgoH+XaGnP1UTA/xrSNOFt6s/ndSDp2pu5K1SNUagHvq+1dnvkTU2kAhnmktH4Xim5cZkP5I2gX tzx1iMAA/d36o6nfgSjPIcnPV+m6kDLDIlIr2yZZtcHdJ2JaPLsV3ns0rq0jzBMarI6Zz7aTT6tmp n+wyo5q3Q9Ej+srs/PIBCOuvU3vEd15mZNT6FbRZ/fdIdB/zPAOTqelhmvqGcEjXJ3gpZSjXHGc1M QewslFSZ6SB71mrHS8EC9Ap31AD32LE9NzuEmP/+/qZ/caVl/UZ5XaQJd0SIFe+fig0LyQEnIMuyB bXfoQ5RHQ3VC6zXLHphA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn1-0000000AkvV-2Dy4; Wed, 27 Mar 2024 19:18:23 +0000 Received: from mail-dm6nam10on20604.outbound.protection.outlook.com ([2a01:111:f400:7e88::604] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhT-0000000ATY9-2qcV for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:08:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GBoIqat1yo51Kcf1SS79QTHNBLVQsE7ZPDQsBVYnEe+GiHdsUjN4e6gEeXbU9mkbZQKAtKU7DsKBTv5WvlILiPVfapolPQinQ1VhJFbINm242vuTPWnePU2cXB+DtamK+8ewYBE+4K28MY35ZZc8F2XNe+mUTpH4qqSvhLCivnt4yWjHP9kehFO35s1aoVRsNAmGu5Gxjpm9LXYWs95+X4+I3BzPX40IMgJ0t7sbzlVje3nPSF7RnIEWXKwWAaBRHGzBmXr2usfcYRQN5AMZGJciMafaM+LXlsGUy4VT3or6Z2c3NbIhl/I/OheFQ9tMXn5Gblpy1cjhzHSfeuMfWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FosaEK32/1ZODeT9QLnadln0gjum6Vu9SMmO+5zATo0=; b=kCQ55rjrskujkhaXn6gz37D6Envk3cMHEmwfCDok0C2D2Vyqn6jgdLBcjwQSLdFK62GBO7UYbrKkLj+VaFRAXjiaccVahXtd4w1CGLTVEgQgmKCLCKgyiJrRfpLlBv0o4C55jALxZygVHJw0gkfTQjx5NQY6Oi5usLQ2t6zP51tKKxkXqQLj1jWLKXYmy2FaV9bo0OQU9ixNclHgAgzCi2B6WdrRj8KgX9y3QFs5PGQ1Zhz16uPUGrAvj/KT29YQ9IRM9zIg3lzjXV3bIGBFZ8uPJUeNhaGD9VnLxySVcB6akRv8SgmKMtpAcddsCEJC4FgLKwtIISS5Kr4Wi0RnVA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FosaEK32/1ZODeT9QLnadln0gjum6Vu9SMmO+5zATo0=; b=Wm55HyO01KwImMISxtFGjBG3NLiIfsKOetgtFoQm1QGDwe6NVypwghkUA5/0RbLiBzvSznOYz/imst3Cy3hV9YbUgQzfVSX3wiaj8qHpJTKGcawZzBsF+5ZQTXkgORCKXNlwnayZ3SoadoZH6o7MMV8KAH/p3a77uIV7uhpspZe3av/1LEKnLwZda5QpRmbVGmOJDRWElNsMb3vmTLU155dKxc77cZJfhGS7YY+hBtqXU/tLSaSVyMMLIGErwYFs58F8cIXapVqDX6qH5pw+Cu+sKfcbV2D4UAYs+Sq6oNNU00++54uTqckSrBD4CvskYt8XI1bZWFNHxVwfRb1+0g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.32; Wed, 27 Mar 2024 18:08:20 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:20 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 07/29] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry() Date: Wed, 27 Mar 2024 15:07:53 -0300 Message-ID: <7-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0312.namprd13.prod.outlook.com (2603:10b6:208:2c1::17) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: 25e95adb-424d-41fc-bb0f-08dc4e88e102 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mQxOS+iOxdFZSh3fsU56FLF+dtHoopM+hSnlmAOBA4Liqa6dISSGOJ7bpKLowS5D5ceMFAdHWRquZU98e41mz5JKIWfwHp+sRJuBKw1ZFVYqYTGB3kLWM/+WQfDBk/sBgws8aTJDZKpIlg5PviWRhdPwCxaqZkiGt7fWoUJn6EkQm4/jAlGh/5++ot13AZJVrw2YLojNL6WCpgVhTfvPr8uYDFqbZGwQRWDhZlovfs7uGe02/fr20jT94ehHM4kCuc/TSySWtqfSFvFQ98y0Z7+ClC3FM2Pc9pVYE+fkQKlahZyOquJ26QcoIvVQJbHSnNT0rktiJ7ff/8HGfBfm5mdPcwzWVuV57mMCPz9p9Ml0THPn+5QI3jkWz6rEVd17AgsnR6HNU2hQkzX61dAzahiWF6kl3C8jX/gcKT098uPDEohJTCYdv1fzWMt9OpFsZz3404TdjWeiABEFBDIJyH9YDU7Iz1cmSGao+DKFw5LaKpYLxmxfao3CFiRxmPhNplk5/uCbQQdMuKkC0y2aBwx/t18nX0d+38rLek9ZRIc1QVqdoZyIFXg2N5XmEDlw+0G3ku9VcTV/v+3Z2CzIRA3Lpv7iGi73KFL5X7WwIeV8pndHgDBxqH9u0MrzrAJd3EdVZNVLeL5FsQKymxU+mK8z2bT7BlKCpDQokaMCFcE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(366007)(1800799015)(7416005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IFsaRKzH9giRoU9YdEwBX/7/pRrxYL7gqBprHmWSSfFAMb9d6DGFIQ0us9cJOxJ2IZlXXDOEB501/dm24MNiavcqJZ0kCXmqiTGuHVLinHeKsfMJK2gryNbeACEIW1SXI6N2lshzFhzKCub/40d32l88aOzTpw9RAloYHOHh1bq+UNXRc/eajTbEQuS+PjNU9uVACYL0fo+Fbsn0eUr8LjgcvokcYAfalJrefM8KyMxHF0gvI33UoTSCkHWT6jH+eh+pPOZeTFSHYzgvCC/XvNzFb02KozbZ+lBe9dRLOOUeSNDAKPm4W0ADErm8ryJZL+9tMvlBcmsv/lkSt4T/1L6eGv+QNEUE21yESOzFblr3PIWgcBn59jvjssLASDZaVZZvJNWpHLVcdQSkKS2roN8g7X/72wQX+qk4t4CbBD9ngQKCxlYODxmV2iPC96Www5Ppx1EAzkaScXaDUZLN9jQwtEjaE0CGAdGxRMstGCYq2oyX6DnyG9+zQm4EsmAkhY0r3YTL61QaO543X8KPK2OBNQnU6QK9tuMs92SQVZImb6GT0/pL5o4XrPkXTwyv/W6HKFl3luf+9wbewPCmVMdNLpbLRS2Sgv7Jr73gqCNpWRchGDotlwKEu6vrtGCcW5Eee3MXGNz11TFO0dssjyxibTjGHr2c33dPq7LuIabh4whY0IrqOzO6tX7/elTGALEGCOaVbyaKqvW0GFjb+2/+zCbn9iDHv7tCRErWbqakaTOjuIB0ArnR+Hq6u1mAGOVFt+VD6fALF8pD5W88vSs4fTfhO0ugBOfnUPUi9UIAzyofvn3exQDm6rERHaXdtQqW96gzuoIjGoRF30FXOocuGmGqFKNC0iy2HRwyVngO2ELPjkV9rfkLY9CQQ8mqOHxarONGBaL/qN6b5ay4ROjR+qinA1JBpQJ2pjxle33YRB8qO9hc3ikDaCWcubc2RM7L/4CnQBTnSybEVbgZoy6nbTPqAwqxp0mOh0M+jtmnuWPCdk/1QKAK461b2eMxVP75wGVu2I0zvBJUCxPdBD0zpm6duaPvYko6rcRS2wpl1MHGo93zZt5Toqm03WTQwrLrxHvMWz6MQQJcHb7m+WKQB8NcGCHRvt9kVjsDvyTosfDw+HsMbDwMR/nh2JGMntHcAS4aV292f4Ki7kUHvAyLcyys9z/O4iFQqsV1wvlaHtxDVzW9JmSCgoIp9mBn/Td1ue2iCPciQ50HVpbnyU9Xj3B+t8DigQvMK4gFjUcVdOiuk6Ak1LGWfhQSZ17U0o1pDGaymHRyF6nJbk23JhBw0v3OJLaxHJEYaGSZHkYTG1A+rr3X19k9q1PeDwFdrGV96StqcNbPUXAtoZ+J7xDW6uuLOQTi+E5GGWLW/Wx0w7FTUuDI4DXNDcFADWi/EjzsDH7MrLql2v/lYeEclDevQLiOsuEarH/ULbwnaNkGsmdN3c3mXs79eupWsEWQRr/493lDii6r1h5QryWgXEcETrnsvebArhYuO8o25vajgpMz3H2BQIlHqsWgtgoiA9dd1HJPbiizKxGaTLozuJgN++Y7G1z+aYNQTX05yKcAJnvSkRix01WJwwFPx67z X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 25e95adb-424d-41fc-bb0f-08dc4e88e102 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:17.8492 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: OLEasHP6uke5v+0HnMxZrX5Smd1TTOIJI7Rwf+6KkLvxjAvI+vXEMuRbCwg4yhoi X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_110835_884816_7123B590 X-CRM114-Status: GOOD ( 19.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org CD table entries and STE's have the same essential programming sequence, just with different types. Have arm_smmu_write_ctx_desc() generate a target CD and call arm_smmu_write_entry() to do the programming. Due to the way the target CD is generated by modifying the existing CD this alone is not enough for the CD callers to be freed of the ordering requirements. The following patches will make the rest of the CD flow mirror the STE flow with precise CD contents generated in all cases. Signed-off-by: Michael Shavit Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Moritz Fischer Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 94 ++++++++++++++++----- 1 file changed, 74 insertions(+), 20 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 19fa511cec2c05..453437ca4bfc2b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -61,6 +61,7 @@ struct arm_smmu_entry_writer_ops { #define NUM_ENTRY_QWORDS 8 static_assert(sizeof(struct arm_smmu_ste) == NUM_ENTRY_QWORDS * sizeof(u64)); +static_assert(sizeof(struct arm_smmu_cd) == NUM_ENTRY_QWORDS * sizeof(u64)); static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = { [EVTQ_MSI_INDEX] = { @@ -1236,6 +1237,67 @@ static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, return &l1_desc->l2ptr[idx]; } +struct arm_smmu_cd_writer { + struct arm_smmu_entry_writer writer; + unsigned int ssid; +}; + +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits) +{ + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V); + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V))) + return; + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd)); + + /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */ + if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) { + used_bits[0] &= ~cpu_to_le64( + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 | + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 | + CTXDESC_CD_0_TCR_SH0); + used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK); + } +} + +static void arm_smmu_cd_writer_sync_entry(struct arm_smmu_entry_writer *writer) +{ + struct arm_smmu_cd_writer *cd_writer = + container_of(writer, struct arm_smmu_cd_writer, writer); + + arm_smmu_sync_cd(writer->master, cd_writer->ssid, true); +} + +static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = { + .sync = arm_smmu_cd_writer_sync_entry, + .get_used = arm_smmu_get_cd_used, + .v_bit = cpu_to_le64(CTXDESC_CD_0_V), +}; + +static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, + struct arm_smmu_cd *cdptr, + const struct arm_smmu_cd *target) +{ + struct arm_smmu_cd_writer cd_writer = { + .writer = { + .ops = &arm_smmu_cd_writer_ops, + .master = master, + }, + .ssid = ssid, + }; + + arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data); +} + +static void arm_smmu_clean_cd_entry(struct arm_smmu_cd *target) +{ + struct arm_smmu_cd used = {}; + int i; + + arm_smmu_get_cd_used(target->data, used.data); + for (i = 0; i != ARRAY_SIZE(target->data); i++) + target->data[i] &= used.data[i]; +} + int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, struct arm_smmu_ctx_desc *cd) { @@ -1252,17 +1314,20 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, */ u64 val; bool cd_live; - struct arm_smmu_cd *cdptr; + struct arm_smmu_cd target; + struct arm_smmu_cd *cdptr = ⌖ + struct arm_smmu_cd *cd_table_entry; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; struct arm_smmu_device *smmu = master->smmu; if (WARN_ON(ssid >= (1 << cd_table->s1cdmax))) return -E2BIG; - cdptr = arm_smmu_get_cd_ptr(master, ssid); - if (!cdptr) + cd_table_entry = arm_smmu_get_cd_ptr(master, ssid); + if (!cd_table_entry) return -ENOMEM; + target = *cd_table_entry; val = le64_to_cpu(cdptr->data[0]); cd_live = !!(val & CTXDESC_CD_0_V); @@ -1284,13 +1349,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, cdptr->data[2] = 0; cdptr->data[3] = cpu_to_le64(cd->mair); - /* - * STE may be live, and the SMMU might read dwords of this CD in any - * order. Ensure that it observes valid values before reading - * V=1. - */ - arm_smmu_sync_cd(master, ssid, true); - val = cd->tcr | #ifdef __BIG_ENDIAN CTXDESC_CD_0_ENDI | @@ -1304,18 +1362,14 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, if (cd_table->stall_enabled) val |= CTXDESC_CD_0_S; } - + cdptr->data[0] = cpu_to_le64(val); /* - * The SMMU accesses 64-bit values atomically. See IHI0070Ca 3.21.3 - * "Configuration structures and configuration invalidation completion" - * - * The size of single-copy atomic reads made by the SMMU is - * IMPLEMENTATION DEFINED but must be at least 64 bits. Any single - * field within an aligned 64-bit span of a structure can be altered - * without first making the structure invalid. + * Since the above is updating the CD entry based on the current value + * without zeroing unused bits it needs fixing before being passed to + * the programming logic. */ - WRITE_ONCE(cdptr->data[0], cpu_to_le64(val)); - arm_smmu_sync_cd(master, ssid, true); + arm_smmu_clean_cd_entry(&target); + arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target); return 0; }