From patchwork Wed Mar 27 18:07:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13607335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3F7C0C54E67 for ; Wed, 27 Mar 2024 19:18:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cHU8wpB7FOUT6jid8GnYgty4/us7zLZNadC2rb0XTqc=; b=1g/vARivPWg7TX ejx0/uIW7pb5Wn/5FCUpq7vks/GBsWxBWDC3K945fwqXVYJhnL0uSDg8Kdcps8bi6Woc7kB/Wx45I WPk7Ldkpy880KAOYCTXkzIUjs65DJE8FNNWgpxI5VbmL0zSCwcV9pFLd9F1ipDyjCSsWygp5BYEuS stkynuYxQzPVV3afaofgNj5Q42DYyjDRHTErd8Vk4sp/nPRgpus26clERffkbXzoDy80l6y8BHC4C QnL155Rd/BMc0pNGCqCgVAi7ss+GHGMUTuYfM2pV6TMBys81CUxpCkbqMZwfvLCgJFEcr2ifeC2+1 kCIwnoYpi6G29A7ibs/w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpYn7-0000000Al0I-3g1s; Wed, 27 Mar 2024 19:18:29 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhv-0000000AU2t-0cxK for linux-arm-kernel@bombadil.infradead.org; Wed, 27 Mar 2024 18:09:03 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=eic8gb+150fVmfgrRI8wCitBYk4nxNVChg4sW3oQ14g=; b=sDGOxx55/fuJNYBHPZ4FuCxM11 HVkzUEF5P1z7R+nWgcmlw+Lyh/fV4490xxbGgardRXLwHIQIWcX/Gm6c2qqaqltQAJ3Fajf8Peh7w N1RO6TE+iX8H8Uu+/s2OsAkd+2fZbqcn+bZCZBPzJNAMqmgoQmc84jUWkCS3Ieqtsjdjuil8npiYt OFse2tooATIAM+nNmcRzPthRc+SuNdYAO5Q61nO+uyxFXZQEV2tuoXsFZSTPTDTQJB9leZuOH951G tKBWwXRIihgW9cFBosvP/jHZ1FjW96MA2Q4+XW+pAlj2USf5nf1NRfYk7kajfTZToruFOfw7PdXWv QwV+b4/A==; Received: from mail-dm6nam10on20605.outbound.protection.outlook.com ([2a01:111:f400:7e88::605] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by casper.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpXhs-00000004QF8-0saz for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2024 18:09:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Jr9x3P4uI1eEjowRF9SSK+A4bd826jPPK02TyU1U3Ymm7Ocg+ssBd1aojOIgn3pFoDFA2IpyccqTdfKw0VoEsVFYbwVvO5vUbjn6+QbFR9c5sX5IUO1aMgqlD1kL0TGZw52P3r4e1oWmXAu4ZzvU2RwlSA1K0hpBucPiNuKfCso8IZmv1Zl2CSJyCR0xJ1YxrS9Eoj72zR7MsmebKD6rsBdxuaim88qoNhv023yyli8ykz10vjSf+DO36MFv8H5Zo+hWUuerN5xeM/ATYKO66RdKs/0ul0z4wZKzkvWPvvU5p56CBLVxQ6iFr67I3alF7mLjFaeXwAAn2hZje7ydIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eic8gb+150fVmfgrRI8wCitBYk4nxNVChg4sW3oQ14g=; b=ZgD65PL1YPZZkR1wXFFv3aPy8OI2BqLk/6J4c+BVC7BguFgxB6w8JmAFP8gj2cBC8N4XpOgqtXxtb5FH1oKHVyGSZwwTbab6Vejgv+mX4quh8SHNSk922Ocup/SpDqiZT8pzbSBHbCj0P3wkUwpN37Mn7U6YmQHqf8j3ZRxY+AZ1OFyazhFM/REKIoT988ecofcikAWbg9F9wwjaMBIHXtLigkgTOQ9icqqFc9+0XzbWijYhz8s6joSM80h706/ffHoz4TDgPzyEgjXE/afh960zsiBG34gWlktJDOErsxrIjx1qqAFjQYCkb1LDDxQnvfLaq0Oo7dz1RzS41UvdMA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eic8gb+150fVmfgrRI8wCitBYk4nxNVChg4sW3oQ14g=; b=oCPFD+imMHBgT6HRCwLoQ1kDR6Nh8dVSmOmbBpU7RtKkqtBhO9Mc18aJXxGaZ0f/o2gqnRFv6lQaKYiaQ0LQHIN17+NTeMSYDuISo92cQahP2Y54HvBQvxK2EjyZFTVvzZ3mpcCtYroMZWnqdW3EuLwIaW9ME1oN+ooDesj7Mop3zLOZfeD2P+FW+8MUeoJXwO1LCOtzI0Ij/YequW6OtttKComrjQ5jA9I70T3vbnjfSTqCw9xQtwTxlc/EewkMBbVbI+rqptz3PlHqw0BqBKKLyC96MINYkEbuIY/ZX765XmKnkz3ek9Upf/7/6i/mURTvnquxmjf8dsfSEFn7mA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by IA1PR12MB6044.namprd12.prod.outlook.com (2603:10b6:208:3d4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.33; Wed, 27 Mar 2024 18:08:35 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.031; Wed, 27 Mar 2024 18:08:35 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Lu Baolu , Eric Auger , Jean-Philippe Brucker , Joerg Roedel , Kevin Tian , kernel test robot , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Tony Zhu , Yi Liu , Zhangfei Gao Subject: [PATCH v6 06/29] iommu/arm-smmu-v3: Add an ops indirection to the STE code Date: Wed, 27 Mar 2024 15:07:52 -0300 Message-ID: <6-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0314.namprd13.prod.outlook.com (2603:10b6:208:2c1::19) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|IA1PR12MB6044:EE_ X-MS-Office365-Filtering-Correlation-Id: fa47be6d-f440-4a68-0493-08dc4e88e25a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vtbIsyxw5PklOQcI1zxnBlAq33RHAT6myC2h+QI2Wne7hmHb/hH6h/AndtC2Q008myL2caWpiOIihY84GWgalcjVO8UsrTIseOrSWpG/c/4qs/HHvbqO7DLdaWZUYewzIA7g0TxTC2Sqbig0B/CtXcgxVdNa9BE2VWUQdqhOKbqKxtoJZ1FvzeLasM0AKAlPtuFBdKIoMgY6tPDNrnfZbzqX2hwxg6e8yCZn4gF/tg91eufH4UefikdrU5HP72HwppWvJO6E+BWktp9WEXtLoey1btXu8UQfZyVHQ9L89HYOpHghEboVsk7tAhA2pMctTsaZUphf9wr8spSalIDR6og+OTSgOHAbhX+fO9PhaUT7jW+iPZTvb7B2BOf/AgIjcum9t3jrddQ4Du4z0igH3hsZIxzdmMdm1s9KPZ1nF4G57U5E/00C6f8SJafhj6XweEHDNL86iUfEIByYGxjZoRZC4RrrlZOi/rSDAdmBzvlpYQu287a4WmtCwdONXbvzUzX2b/nFg1Wh6/lhf4vkQ+5fbwMDcf2BBupuTWlWGEGXHZgj0ySwNsZVqj9EL+e1cvPs1B3YX/upavB2cJvDiCDyaYi0ToQVVkqlkEuwG31Goi5cqBrfJG3jLhdwNk5OQQUxJh0sthTj8ByintF+WZGQOwG7vZhap6SESLnEx/g= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7416005)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Vtkp9osnhayqI4axTxOBCyU6qhEv9O4nFggH0OrJcoE/ULUHcBIoyDEfs0tiObZdAyGVvfblhoF2ED2LVYuT9TaBWYw6dUSf2RfY7XM/ObRGiKunjE8EIP7UBJwksxvOc+1gyfbKCZKId+LmpKESDs3c+yEXivZx78JS+76yjfyRpjuczXU+XLJvpsRm3XhBqP1qXM//w5i82uWWOujQV+fyMNcKRo7M/gFnePKSqCmVBCw81TLzH7tFLHav8jdGLnU0NtrF7DC8ZJU8WK2Eq7CA6lk9lhroOdsr2MJQA28FQvnbPKeOz9jA2APBKw0YhWV0+RzTL1BBCUUPOOtA82oPeIq/ftGk5T98Cvnz2FVEmkuPKAwfDQ3It47pTBxFpMXIu7zheRkiSqXAgfJqfOvnaXNA8hKsG/XIqq/j49NpsoLrO2i7aC0PkMGjhZE6mMkqFlU4S5RRSm3jN9rtFOuF+GLBqaeu9cC+zb9KxsMmETV28Q0aAxqNWStkvGiBEdtsBRTY0oWq+B4Ev3lk1etm19enWe4S8gkfwg7j6hFgG479PKf1VTH477FDxjsgsucy/J+iETr9tjo0+KSkLczKKF//CTByfAOhSA7WyxxpGd47d7SfzJDr3hCzBgnPsnO5X7RtEG0EiVYfJMShCOiAtX13FOv0roVo96/fF4o3AsOw1ZGzHSaimPYEmcmvAp96ygpD0VV4ytDIE/OvVTOksBNIJEwi7hmzX9L/WECvFvR/cZGkxC8bqitELf7N2AIFbCJnwrKDEwWsRLX9LBdLg8yBZXhN1RHEFZJxqZCE6E+IHlwNVC2XT/TKVQSPqFw0P6Utw4CraqTdKYhVa6qZkdDqwV1LI8T3WTwU3iGVHpgMNHXhXnNy4IdyGKI1uplbb55PY/F+s36wlG3tY7vOc5SW3p7sf0EjgnFduw+Z6fF+c2SWZPUOtXsRL1SalH12GTiMYVScnY3iM6M93UPPmkOzKpyjFDjK4/Tk2+Inag8tCTl4+q4V/ioPzpL9ODyVoDq9hdwnopIzP4RcsPZHeDI8z2qn+xcWLu/kuryS0XAPNP92q3gIZnUU+LyZCbLp/BdNXi9+Yxrs0+nspSe2Ox0p8AYi1pcZJX1tHxHgbaQzpD4JGaFFTIrspoLZYF7N0+pmcS4Vg/NM0Oc0kYruZ2gRjOYojO3npKua/f13Y+gTGKUvPLLJfojGnXoQElZo5ixqAL4SqFAuQNQjjVWOUincINVC/eLutfab8GeVAldcFjHKuz2iY+f7WB7rs7U4cKTgvLbaDx+0DkfvTjOb7wGRBFfRoEk8jdcyEKFxmoXp6vOh9fzZFsMPMMYRZmpHtYMDg7a1IdDaZ6tWd8jd9NookrT5gP3sirg8RXEzBsOSE9Z+dNK4mdvb+M6rl1ssQrOMEzL/kn2ytemeiM3rbf2l6Q7mYghLS8eW2yVZbdE+U5GQYBDYeY0FEspBWz2O3vdwHqkpZ9pSsz8dzRUkLbvi197oE5LfQ3Vq96fjBRV4GOCubJE8YA4QNzx2wTm3lcF6/x9BxYtz/yKU5t1YXB1vA8EE1X8Ngq2m5vZFYx4XmjkaNxE7sk5KNTz6 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: fa47be6d-f440-4a68-0493-08dc4e88e25a X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 18:08:20.1405 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vudIufNpslvH9E+vhEx+Z3qZ/v1f1pCqAG0frVkgTworZAvfTGEk5j0eY6IML53M X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6044 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240327_180900_272348_4209215E X-CRM114-Status: GOOD ( 19.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Prepare to put the CD code into the same mechanism. Add an ops indirection around all the STE specific code and make the worker functions independent of the entry content being processed. get_used and sync ops are provided to hook the correct code. Signed-off-by: Michael Shavit Reviewed-by: Michael Shavit Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 170 ++++++++++++-------- 1 file changed, 102 insertions(+), 68 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 6f62a38052b504..19fa511cec2c05 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -47,8 +47,20 @@ enum arm_smmu_msi_index { ARM_SMMU_MAX_MSIS, }; -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, - ioasid_t sid); +struct arm_smmu_entry_writer_ops; +struct arm_smmu_entry_writer { + const struct arm_smmu_entry_writer_ops *ops; + struct arm_smmu_master *master; +}; + +struct arm_smmu_entry_writer_ops { + __le64 v_bit; + void (*get_used)(const __le64 *entry, __le64 *used); + void (*sync)(struct arm_smmu_entry_writer *writer); +}; + +#define NUM_ENTRY_QWORDS 8 +static_assert(sizeof(struct arm_smmu_ste) == NUM_ENTRY_QWORDS * sizeof(u64)); static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = { [EVTQ_MSI_INDEX] = { @@ -977,43 +989,42 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) * would be nice if this was complete according to the spec, but minimally it * has to capture the bits this driver uses. */ -static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent, - struct arm_smmu_ste *used_bits) +static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) { - unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent->data[0])); + unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0])); - used_bits->data[0] = cpu_to_le64(STRTAB_STE_0_V); - if (!(ent->data[0] & cpu_to_le64(STRTAB_STE_0_V))) + used_bits[0] = cpu_to_le64(STRTAB_STE_0_V); + if (!(ent[0] & cpu_to_le64(STRTAB_STE_0_V))) return; - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_CFG); + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_CFG); /* S1 translates */ if (cfg & BIT(0)) { - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT | - STRTAB_STE_0_S1CTXPTR_MASK | - STRTAB_STE_0_S1CDMAX); - used_bits->data[1] |= + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT | + STRTAB_STE_0_S1CTXPTR_MASK | + STRTAB_STE_0_S1CDMAX); + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR | STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH | STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW | STRTAB_STE_1_EATS); - used_bits->data[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID); + used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID); } /* S2 translates */ if (cfg & BIT(1)) { - used_bits->data[1] |= + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_EATS | STRTAB_STE_1_SHCFG); - used_bits->data[2] |= + used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR | STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI | STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2R); - used_bits->data[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK); + used_bits[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK); } if (cfg == STRTAB_STE_0_CFG_BYPASS) - used_bits->data[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG); + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG); } /* @@ -1022,57 +1033,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent, * unused_update is an intermediate value of entry that has unused bits set to * their new values. */ -static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry, - const struct arm_smmu_ste *target, - struct arm_smmu_ste *unused_update) +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer, + const __le64 *entry, const __le64 *target, + __le64 *unused_update) { - struct arm_smmu_ste target_used = {}; - struct arm_smmu_ste cur_used = {}; + __le64 target_used[NUM_ENTRY_QWORDS] = {}; + __le64 cur_used[NUM_ENTRY_QWORDS] = {}; u8 used_qword_diff = 0; unsigned int i; - arm_smmu_get_ste_used(entry, &cur_used); - arm_smmu_get_ste_used(target, &target_used); + writer->ops->get_used(entry, cur_used); + writer->ops->get_used(target, target_used); - for (i = 0; i != ARRAY_SIZE(target_used.data); i++) { + for (i = 0; i != NUM_ENTRY_QWORDS; i++) { /* * Check that masks are up to date, the make functions are not * allowed to set a bit to 1 if the used function doesn't say it * is used. */ - WARN_ON_ONCE(target->data[i] & ~target_used.data[i]); + WARN_ON_ONCE(target[i] & ~target_used[i]); /* Bits can change because they are not currently being used */ - unused_update->data[i] = (entry->data[i] & cur_used.data[i]) | - (target->data[i] & ~cur_used.data[i]); + unused_update[i] = (entry[i] & cur_used[i]) | + (target[i] & ~cur_used[i]); /* * Each bit indicates that a used bit in a qword needs to be * changed after unused_update is applied. */ - if ((unused_update->data[i] & target_used.data[i]) != - target->data[i]) + if ((unused_update[i] & target_used[i]) != target[i]) used_qword_diff |= 1 << i; } return used_qword_diff; } -static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid, - struct arm_smmu_ste *entry, - const struct arm_smmu_ste *target, unsigned int start, +static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry, + const __le64 *target, unsigned int start, unsigned int len) { bool changed = false; unsigned int i; for (i = start; len != 0; len--, i++) { - if (entry->data[i] != target->data[i]) { - WRITE_ONCE(entry->data[i], target->data[i]); + if (entry[i] != target[i]) { + WRITE_ONCE(entry[i], target[i]); changed = true; } } if (changed) - arm_smmu_sync_ste_for_sid(smmu, sid); + writer->ops->sync(writer); return changed; } @@ -1102,17 +1111,14 @@ static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid, * V=0 process. This relies on the IGNORED behavior described in the * specification. */ -static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, - struct arm_smmu_ste *entry, - const struct arm_smmu_ste *target) +static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, + __le64 *entry, const __le64 *target) { - unsigned int num_entry_qwords = ARRAY_SIZE(target->data); - struct arm_smmu_device *smmu = master->smmu; - struct arm_smmu_ste unused_update; + __le64 unused_update[NUM_ENTRY_QWORDS]; u8 used_qword_diff; used_qword_diff = - arm_smmu_entry_qword_diff(entry, target, &unused_update); + arm_smmu_entry_qword_diff(writer, entry, target, unused_update); if (hweight8(used_qword_diff) == 1) { /* * Only one qword needs its used bits to be changed. This is a @@ -1128,22 +1134,21 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, * writing it in the next step anyways. This can save a sync * when the only change is in that qword. */ - unused_update.data[critical_qword_index] = - entry->data[critical_qword_index]; - entry_set(smmu, sid, entry, &unused_update, 0, num_entry_qwords); - entry_set(smmu, sid, entry, target, critical_qword_index, 1); - entry_set(smmu, sid, entry, target, 0, num_entry_qwords); + unused_update[critical_qword_index] = + entry[critical_qword_index]; + entry_set(writer, entry, unused_update, 0, NUM_ENTRY_QWORDS); + entry_set(writer, entry, target, critical_qword_index, 1); + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS); } else if (used_qword_diff) { /* * At least two qwords need their inuse bits to be changed. This * requires a breaking update, zero the V bit, write all qwords * but 0, then set qword 0 */ - unused_update.data[0] = entry->data[0] & - cpu_to_le64(~STRTAB_STE_0_V); - entry_set(smmu, sid, entry, &unused_update, 0, 1); - entry_set(smmu, sid, entry, target, 1, num_entry_qwords - 1); - entry_set(smmu, sid, entry, target, 0, 1); + unused_update[0] = entry[0] & (~writer->ops->v_bit); + entry_set(writer, entry, unused_update, 0, 1); + entry_set(writer, entry, target, 1, NUM_ENTRY_QWORDS - 1); + entry_set(writer, entry, target, 0, 1); } else { /* * No inuse bit changed. Sanity check that all unused bits are 0 @@ -1151,18 +1156,7 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, * compute_qword_diff(). */ WARN_ON_ONCE( - entry_set(smmu, sid, entry, target, 0, num_entry_qwords)); - } - - /* It's likely that we'll want to use the new STE soon */ - if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) { - struct arm_smmu_cmdq_ent - prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG, - .prefetch = { - .sid = sid, - } }; - - arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd); + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS)); } } @@ -1435,17 +1429,57 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc) WRITE_ONCE(*dst, cpu_to_le64(val)); } -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, u32 sid) +struct arm_smmu_ste_writer { + struct arm_smmu_entry_writer writer; + u32 sid; +}; + +static void arm_smmu_ste_writer_sync_entry(struct arm_smmu_entry_writer *writer) { + struct arm_smmu_ste_writer *ste_writer = + container_of(writer, struct arm_smmu_ste_writer, writer); struct arm_smmu_cmdq_ent cmd = { .opcode = CMDQ_OP_CFGI_STE, .cfgi = { - .sid = sid, + .sid = ste_writer->sid, .leaf = true, }, }; - arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); + arm_smmu_cmdq_issue_cmd_with_sync(writer->master->smmu, &cmd); +} + +static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = { + .sync = arm_smmu_ste_writer_sync_entry, + .get_used = arm_smmu_get_ste_used, + .v_bit = cpu_to_le64(STRTAB_STE_0_V), +}; + +static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, + struct arm_smmu_ste *ste, + const struct arm_smmu_ste *target) +{ + struct arm_smmu_device *smmu = master->smmu; + struct arm_smmu_ste_writer ste_writer = { + .writer = { + .ops = &arm_smmu_ste_writer_ops, + .master = master, + }, + .sid = sid, + }; + + arm_smmu_write_entry(&ste_writer.writer, ste->data, target->data); + + /* It's likely that we'll want to use the new STE soon */ + if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) { + struct arm_smmu_cmdq_ent + prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG, + .prefetch = { + .sid = sid, + } }; + + arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd); + } } static void arm_smmu_make_abort_ste(struct arm_smmu_ste *target)