From patchwork Mon Feb 27 13:21:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ganapatrao Kulkarni X-Patchwork-Id: 13153448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 74E61C64ED6 for ; Mon, 27 Feb 2023 13:23:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TR1OBj9WaIKSKXyrOCCBxclGnc/OuDAPi4Litmj2PWg=; b=KhNuK2I1XW5XQI tif6OvwA7BnzqK/0IkoKrUZvlAZq2SyNxs0OWa2/Fnyu1TZUR/FOEVGb8lzXNetu6rEvCpXftTsrm e0fnzPYrUjI+IL3SpNGM4jf/fHIlBTlJosf5I00bx2V0Ky3ZGfQBksQyF6VCNt+Dh4NJP3crBiW1o kFZguyMHvsKHXWZq8YvtKlniq9pauMoY1ncH/8hxPHscwVIfOntOD264GHoXCd2mX4xk2mBYgmkOx bKgNd85CUMCZPPlQvhvu5N45SoaJyKW0MBHWieeNKHMwMvbJm5v7XV+Qh047EcwzX2nI1kou/H1B5 cmWib9OsiOKwd+etg+Uw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pWdSa-009jnw-Dr; Mon, 27 Feb 2023 13:22:32 +0000 Received: from mail-dm6nam10on20729.outbound.protection.outlook.com ([2a01:111:f400:7e88::729] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pWdSO-009jiD-Jz for linux-arm-kernel@lists.infradead.org; Mon, 27 Feb 2023 13:22:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Of/38FpxvJudbDV+xnBelDTY/5eYUo9z985lmPk7Eqr4bMRMmROCyIhMOWT8APsiJ3D8DBzzDtemguN2Vj5AmbR94/V7THJxpRHiVq8qH2mvunNXVv+zkj+0gX4FN32ouEO2Ezf/93q6ebvSlNKGmDw7Y3j8fmDSZ1HY149MGZL7TX2Ll6eNAvuQHmIBqI8kwBE7XyMUGayUQPj964tphjO76DPtn0W20/YOOdd/lD7KfNF00NMUK4VpL3kTXX29dpLR6zQxjUAyKDUI2rc/RuwRHL7NsjFaD1pDmoCRuOIDI4kRL9pdgrVBARNFX6JMBLlcX+yNrytDy4NvdMY2oQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CQ2KRr9Ghve1sjeFkGepMy25F0bIUg6LHFHFozopTew=; b=lIib8MG3zGtv1KwbBdefuVxJ2XDExJFXezDj118AFde7tlhNsG8/nK6UoeQUWTH3Pq7QSl2K7QfUci4pItMkOiDRjedxqyhISiPzGuW4vNKVKEiTG7ej67bzTCjd7c2sD4BXv/crlCtdsn0Q8Xs8T7yeZNEyHVaMYI/wBTJW/mWAIi3+DdLmTBo4/KE0YiZhLBhqLZi5N922kkUL0kb66ghWpgaCpm+jazarREBBpQtAjYipjSXl/rAJswiMxhLr7XyO+EqdN1I4XK4rRfw8toA9uEcOToZhC8wmNh90FX24ZcgnplqPBGbctqtj+qoVNLTEkzL4aYZUHNiJnY62wQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CQ2KRr9Ghve1sjeFkGepMy25F0bIUg6LHFHFozopTew=; b=k7RL8aPLyv9cC/sSviSxfGhYqf8C3D09hzQyAsFgpwUVEcQgMseBUMgJuZrMRRprzdBeHItYiZXk67eLEjGDUfAAy34rJqDNVBXYzBmktrJmrfQ2ZHwCvfRk54SlTP+UOGcQvFX3spNX94l5QsKD8TVrK7nbtvVHS4P2s6a+lc0= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from DM8PR01MB6824.prod.exchangelabs.com (2603:10b6:8:23::24) by MWHPR01MB2719.prod.exchangelabs.com (2603:10b6:300:f8::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.29; Mon, 27 Feb 2023 13:22:10 +0000 Received: from DM8PR01MB6824.prod.exchangelabs.com ([fe80::fc8b:5e5e:a850:7f0a]) by DM8PR01MB6824.prod.exchangelabs.com ([fe80::fc8b:5e5e:a850:7f0a%6]) with mapi id 15.20.6134.029; Mon, 27 Feb 2023 13:22:10 +0000 From: Ganapatrao Kulkarni To: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, joro@8bytes.org, bhelgaas@google.com, robin.murphy@arm.com, will@kernel.org Cc: jean-philippe@linaro.org, darren@os.amperecomputing.com, scott@os.amperecomputing.com, gankulkarni@os.amperecomputing.com Subject: [PATCH 1/2] PCI/ATS: Add a helper function to configure ATS STU of a PF. Date: Mon, 27 Feb 2023 05:21:50 -0800 Message-Id: <20230227132151.1907480-2-gankulkarni@os.amperecomputing.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20230227132151.1907480-1-gankulkarni@os.amperecomputing.com> References: <20230227132151.1907480-1-gankulkarni@os.amperecomputing.com> X-ClientProxiedBy: CH2PR05CA0013.namprd05.prod.outlook.com (2603:10b6:610::26) To DM8PR01MB6824.prod.exchangelabs.com (2603:10b6:8:23::24) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM8PR01MB6824:EE_|MWHPR01MB2719:EE_ X-MS-Office365-Filtering-Correlation-Id: 66b2b6d8-83ac-4931-0ec3-08db18c5a18c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3T3QlPI94M+YGeJBZGOLjybR3CZtM+TmDEHcoCc0HuWfYxd7No9sQRnqYTcUkxS+HOoMtWOXnvV/50KMtynaouqgz1oVm4OwExltBd7EeCBMez7j7W8I574RL0EYwm0xGoZspBcJDO5rSVUmK8CvmUWfCyK48ZUKnc4M+CQ9bTQ17egIZJF1xH8TDI1aKnGfarFQAJSuFUOvbfJZgGYhh1vgOaszHunqUfusdHm1P839j5vZIEqP51I2af/6ca8ee0XW6lVzwi8dqCl3th+KW+Tn1rXjTVNXbfMbBmm+1nW/tGz2c1i8N2zT/rIkv4o2QID9sBCgHglzrbskcO0s3KTKmMYFDEMkAv9TlNUMYszLmPFB0sS8Xu45nGnmJ/1JNaW3rqsK6ClrDRn4ZAGIQZZaN4sOzShNxUXXJz9EiSeeOsGJ0Nag8ER4gh1HqklTCYm8ulO0l6W4zMl7gEgdDlXhn5ce7vgDbVBKX2xdYtciNnwvp151VR0unNVKBQWUjtw8ZGqeXk7rTENJlijVUTipXgq07S0deRffYr4E3l/g7dHrdNHj5o4eMGRfLb3AaoyUCh2oer1zfd/qKfvw36jeKp7YU/CXjOJjuUFLlSHRWwXjPZFU9QNUnfUT8C95gQQwHDQy2AJ82TBx8AhiFoLhR2FhfV1l4Gk5YVRlOa67IGKo3xjVtM433SpZFcwublgZe4UhAK6t0mAosVCUeA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR01MB6824.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(39850400004)(376002)(366004)(346002)(136003)(396003)(451199018)(8936002)(2906002)(5660300002)(41300700001)(66946007)(66556008)(66476007)(4326008)(8676002)(316002)(478600001)(6486002)(52116002)(6666004)(107886003)(26005)(6512007)(186003)(1076003)(6506007)(2616005)(86362001)(83380400001)(38350700002)(38100700002);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: A9/fO0bkSHhbdc0wxkgEGC5gVKHSxm6WCSBlZ4pq1WVmMS2XacF0sa/MjtSTlLwBiXLu0ffBN84xrqPlncOtP40Pk0nwSd6AV0fz49uZGnnSLgm6nNPrTGO0ztrf8c7vzCF4ji2jZPBuu299yznhLnLsSWi3dHiawuvkyZZiegaKNE994UgiaJlug1wOESPgeERjItldsTV8S48bHQLECfagcUwr5vGVbzq2AleTcxkFoAfa3qSdRaquJcu1252f8fskeN2hFXED1R1PwLw/FktADuTyk38cmtlUpJ50n3Eh/OsQyahdDUqhjwy8trFqECWvgjHDzITdJngICBl9192cbp7nh4ofTBufRkfzemhaXViERQ8qJ+f5fd30jUfJ7d8FS1E081V81tjm6RA2L+PSrj0optwIViUiqjFzQ2q1cMGeSnuyi79LKWXwLHc2ALOQy9rWSJSWUsGImw/MnvYIR9xXCL4mb1mhL6Ng12qho5hZttZaP0H4CXBsG+RWouJMpAgvCJOs4l2NS1IuaRWDj/XWHvZsYZcTJlMmDucgpFHCt/ge+brY3zq6iw1aIdt7WkBKFJ4LugM+R7qbbO5EvtjY5hQQr8zLLO6cPOI2iSEdzZmrAILskAXMuSxMIGFHfE2B6jev2r0UTXZtBYy1QuLbMZ70YDyz2HZUdm9F1Ea9y7lmfCYg0W1wnlMEBJAHwQ0wb9W+Fy6MoVnMlDtXEs0hmHQxh4E6OKHFYQSCemI7OA6y4ULfLeCL4LSASrmwYiBiSUL2bj5hu0p7KgP2WHN+vZOLRJ8J2lADjEw40lj4K2JEgqMKfZ9MK1ddECD+YC0xCuL+wrszVVsJLu7njZ+3YacUy2GyQ+Njmvg5A+1VZHxmMB+PY+VSTFt7gEVoRBNS0oeNkm8+ii2uVUX5aFbiSlm/h3TonIa988AG6iqvDEVuzkXnMpqHCoJshWbDsR+FCJmufNtmVwGYtKo+WU367iW/B5JnvfpvHdN5ymVuLjacfcT239QuPy4x9Zhug6ztQyGJ5Ytf5DcI0rA0h5MO3yeJwYJEMj3G5wT7iC8X3BUGgYTcEKepUt9Jjxt3ifd1qEPO54fy2DKnJSaowHWEZ9BBlxCTYYThQ84Z524Eeb0odB549qAn4ULrZoS0twKeh9DRjJbFdYxRYx84ahD6AMqaXakd9mdbTYQMPEF1G3ongCjCrMEl2I8KKOJU/O2phplwjHD+u7YDCjT5Ri/rNNCklOECtUrK1yLMd4U0zXvFCMErKrkepNUMIWiTaTnMuoGYw2cheGHflB8cnTC/sDn1x062S3ORrAsruy6sFjOyueJoD/uzRTT3V1wDu6mRp5clvN0oyW2KrI+ri035eRLN8dx2v+ZcSGI5HeiXnYGE4vNh+9Tw73g9YEBepDpqQ0QQ90N+EOKzrwTVZ/zJspy6ndasAv0Z0d0cIZkisY30mRzHSvdnKF0a36FyYtcmKM47f4cgEFYdfyUjNjpmIKEZjRUVsT3myBQ+t3HsIESZnqhNPlPm+JKNE0K6vS4yhufue7CmfnRZM6k/HY1jcne3EHMdGS1ZRvqy4IFNYfIyXNv7sFcA9OJIquhOKZXgNO80W6LlyyjubotPAIHhgrFRqI6fDDEXd7k= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 66b2b6d8-83ac-4931-0ec3-08db18c5a18c X-MS-Exchange-CrossTenant-AuthSource: DM8PR01MB6824.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2023 13:22:10.2553 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2c9dg2mrrpPrdZrr3qBX3s7UPFKTMLgMYMDIwPnJaBjZKKI3jv9NJuMwKW1JlpeSPS54d88COSkV/F/HOAU86qVQ7guuxqLSARzBLKnBKYZdYToY8HC9Wgd+LF2xNiVY X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR01MB2719 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230227_052220_696123_72E50B69 X-CRM114-Status: GOOD ( 16.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As per PCI specification (PCI Express Base Specification Revision 6.0, Section 10.5) both PF and VFs of a PCI EP are permitted to be enabled independently for ATS capability, however the STU(Smallest Translation Unit) is shared between PF and VFs. For VFs, it is hardwired to Zero and the associated PF's value applies to VFs. In the current code, the STU is being configured while enabling the PF ATS. Hence, it is not able to enable ATS for VFs, if it is not enabled on the associated PF already. Adding a function pci_ats_stu_configure(), which can be called to configure the STU during PF enumeration. Latter enumerations of VFs can successfully enable ATS independently. Signed-off-by: Ganapatrao Kulkarni --- drivers/pci/ats.c | 32 ++++++++++++++++++++++++++++++-- include/linux/pci-ats.h | 1 + 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index f9cc2e10b676..70e1982efdb4 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -46,6 +46,34 @@ bool pci_ats_supported(struct pci_dev *dev) } EXPORT_SYMBOL_GPL(pci_ats_supported); +/** + * pci_ats_stu_configure - Configure STU of a PF. + * @dev: the PCI device + * @ps: the IOMMU page shift + * + * Returns 0 on success, or negative on failure. + */ +int pci_ats_stu_configure(struct pci_dev *dev, int ps) +{ + u16 ctrl; + + if (dev->ats_enabled || dev->is_virtfn) + return 0; + + if (!pci_ats_supported(dev)) + return -EINVAL; + + if (ps < PCI_ATS_MIN_STU) + return -EINVAL; + + dev->ats_stu = ps; + ctrl = PCI_ATS_CTRL_STU(dev->ats_stu - PCI_ATS_MIN_STU); + pci_write_config_word(dev, dev->ats_cap + PCI_ATS_CTRL, ctrl); + + return 0; +} +EXPORT_SYMBOL_GPL(pci_ats_stu_configure); + /** * pci_enable_ats - enable the ATS capability * @dev: the PCI device @@ -68,8 +96,8 @@ int pci_enable_ats(struct pci_dev *dev, int ps) return -EINVAL; /* - * Note that enabling ATS on a VF fails unless it's already enabled - * with the same STU on the PF. + * Note that enabling ATS on a VF fails unless it's already + * configured with the same STU on the PF. */ ctrl = PCI_ATS_CTRL_ENABLE; if (dev->is_virtfn) { diff --git a/include/linux/pci-ats.h b/include/linux/pci-ats.h index df54cd5b15db..9b40eb555124 100644 --- a/include/linux/pci-ats.h +++ b/include/linux/pci-ats.h @@ -8,6 +8,7 @@ /* Address Translation Service */ bool pci_ats_supported(struct pci_dev *dev); int pci_enable_ats(struct pci_dev *dev, int ps); +int pci_ats_stu_configure(struct pci_dev *dev, int ps); void pci_disable_ats(struct pci_dev *dev); int pci_ats_queue_depth(struct pci_dev *dev); int pci_ats_page_aligned(struct pci_dev *dev); From patchwork Mon Feb 27 13:21:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ganapatrao Kulkarni X-Patchwork-Id: 13153449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CDC19C64ED8 for ; Mon, 27 Feb 2023 13:23:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=j1UpZd56v//qMMbeLFGMojXiPKNTGrwqRF+EANjySvg=; b=d/ckVbsF/V1BwM 0CNjmCTTZcD8OrpfmexyMJ6Xgb5DHeychS6QcUAxawfqd188q5zsjBl8EBMzGuXsfvTLU249dhd39 YVOoMYttP4wZzD9uDk19uXrB+sAAmGGRstYL73J4aOkBTI1ii9F1ThgwdLcnDHM4B3uCfF4bSE1OL f94WirapaeTPT1PedieDd7BmQfbpozgdaPyGXBq/o+lKAP5Rl9KrsMc7nebIzjK3FC5aksmxHiu08 XHOzXaUQIY7jWFosB4vZlzDW6DPkB+lRnxpirfOiQFAMiOToB0S9E+pet5tuDR6wlW9ALA+ivlIYy 489+N93jGXTU8PNExhIg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pWdSl-009jro-Tb; Mon, 27 Feb 2023 13:22:44 +0000 Received: from mail-dm6nam10on20729.outbound.protection.outlook.com ([2a01:111:f400:7e88::729] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pWdSQ-009jiD-6I for linux-arm-kernel@lists.infradead.org; Mon, 27 Feb 2023 13:22:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YkzHNWvVIKWidWRT6zqWBC2gKfqU6ib1IQgCAV/iJa23NGhmwbW2d9hNZd7j3fih6DeZ7vJ6D2q2JY82JYsfIRTH95QTy3IctKXjtXexx9g5Cbr60EUp4KKE85/w7KUN4D3Y9bF9myTqzFKcS5Z2Qi3+wWpJxQWrhQR0kmyleZZac4A8/g5Js55I3KGQyWUnZ5CWFYA9vp8WoeUI/pSizzUJgLdJzvy7Svt0Zg0k7ITml91UTaqAgBgHfs0gIrTGIRoXskUIq42AWZ5O8eWKBGDIkOAQ/LgA17UJoY2nfnpBbFq2Q01nbDyLrsU9fIRIxEpSNmovwCEAoGMGcvyc4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zAt2Jdw+BFCu/M780DOdKTlmXUiMtgpBo93yRj4aDVQ=; b=j7LrawkJ7Y5ZJ9rPSfGEqozqu9pyptvKzCJAjjEMrMmZ0hr12c/l2DoBiwaSCGOsEBDXrkrH/HEsQqTzSriz2DZA3Itm7clTj1JU4RN6hX/HyU7YTI8sliRDGJVWyUhs9IPmGXArjRMEML3AdRQI6MV+BaSDR66xrT4SPudmEcPdL5TyVi7ard7Ocw6G4kQ4jW22qEulBxnl0+Stz1kH69ySWz00nOTVjH/qjoK1DZ3ggGzzlsRyqLhlZ6BbzuS5+xxpbST9R2MFD6ho0iWdi14KtdyaP3UL6SNQmYqzCQnskl1/mSYU5mMo7+0Em/gL3UXqcU8LfDflV9ya7Elqrw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zAt2Jdw+BFCu/M780DOdKTlmXUiMtgpBo93yRj4aDVQ=; b=sXTCyn2pDy2Q99AHP7Ygs/IizZ9ysvafFQLoVIhu0USCgfyT2Ood2TAbSGej40BM/qSMYR3odI4GzsFfwuLGscX2KABi2fbos9amiaCboDusHYC5k2oNpSqQDp7W3YGrhssqwxDK3PHXXGQRSiemYs09SeD+DDCHm90thAUoIP0= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from DM8PR01MB6824.prod.exchangelabs.com (2603:10b6:8:23::24) by MWHPR01MB2719.prod.exchangelabs.com (2603:10b6:300:f8::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.29; Mon, 27 Feb 2023 13:22:11 +0000 Received: from DM8PR01MB6824.prod.exchangelabs.com ([fe80::fc8b:5e5e:a850:7f0a]) by DM8PR01MB6824.prod.exchangelabs.com ([fe80::fc8b:5e5e:a850:7f0a%6]) with mapi id 15.20.6134.029; Mon, 27 Feb 2023 13:22:11 +0000 From: Ganapatrao Kulkarni To: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, joro@8bytes.org, bhelgaas@google.com, robin.murphy@arm.com, will@kernel.org Cc: jean-philippe@linaro.org, darren@os.amperecomputing.com, scott@os.amperecomputing.com, gankulkarni@os.amperecomputing.com Subject: [PATCH 2/2] iommu/arm-smmu-v3: Configure STU of a PF if ATS is not enabled. Date: Mon, 27 Feb 2023 05:21:51 -0800 Message-Id: <20230227132151.1907480-3-gankulkarni@os.amperecomputing.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20230227132151.1907480-1-gankulkarni@os.amperecomputing.com> References: <20230227132151.1907480-1-gankulkarni@os.amperecomputing.com> X-ClientProxiedBy: CH2PR05CA0013.namprd05.prod.outlook.com (2603:10b6:610::26) To DM8PR01MB6824.prod.exchangelabs.com (2603:10b6:8:23::24) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM8PR01MB6824:EE_|MWHPR01MB2719:EE_ X-MS-Office365-Filtering-Correlation-Id: 0ed7eea0-01be-4e14-a65f-08db18c5a23c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vDACYXgauCXvO23IAFtRUDFhFGf3AkqLAD1Xsuq5wNxRy3uq83IsXmoX/5kQ2Qt06gp9QBf6qgWyYnvO/mU3fZ9aaSLt++y0ULdl2w59ALvv2Z+8cBsAdrSc8TvQI92Be3gTdcRM7aC8O/+3PMP9BWYQWe7quFNvykgBODJDu3olXcy4WVIa4SnJepU+SzO3RHMh3e3CZ/ZP38X5vu3I0O3WMd0Xmx7rDL00ZMZsOReLLzwo9JyThCEO6UrHEY8i4sirclgCeDTp1MKXq7XxBnsKYxyKr8rswmkdMI8vhn64CxDwNs/7EPfsE0jUnc/5yEcU68Z3Sxs4pVd2Xm3tSqO1rR1l2LGOqEhWsFbIkPncKZ2X6NpW3cec5Q6I1rYmWWNw92Br6omsuUbtCIwK6jlNVDFMfs2roET4GzNDLySpQC+vZj0DWwdNsXRklZ5dqiVHlz/CzgQXMKzt+k+URCiB9hbFQXzLLMDHb9vR87PWSbPiFoL614jTj6st+EQj2rlS5fEX8Q/WySGkhiUAlMJNuP575l9hhg9tAez/pgpxtPfL1F+wOrRTeCt+i+0icfL9YlKbS5HiQeYVz6XTBhj4wFwa/LF/XrtL7qIlHkfzHCy+8N73pU/r4+PB4k9JfMK6sUR6CJgzlBAvi37ZVbJ6sQsYl5IkqM5sluknS72Cd1mvsMqE5Q5YKrf4J58Vtltv5Y4Dcq0YL1PC+0JWYw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR01MB6824.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(39850400004)(376002)(366004)(346002)(136003)(396003)(451199018)(8936002)(2906002)(5660300002)(41300700001)(66946007)(66556008)(66476007)(4326008)(8676002)(316002)(478600001)(6486002)(52116002)(6666004)(107886003)(26005)(6512007)(186003)(1076003)(6506007)(2616005)(86362001)(83380400001)(38350700002)(38100700002);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ROLTypb+l8NKAvLLGiOvozbTfjxtCEEFsZCEhE1iewhz5QTaGDfuX/pvLxYDy5QNhMm0X7BGS7vaw7LMa3fSJlbix8fViJWZhA+G7qYsWfNUmYSM3KIEpu5QvpX35qybCvLXBMQonVRAESoGthtR/NkbD3wKaWYzzRqyckzWxGpG9RKsmuzVHit2Sha+Chbj5gviSqqpz+RmVGsfhFkI3PV6TEoyzRULMYaQMX7xUk56kXFU9TElezVQfHnH/g0O1jf+KPPi4r12r3dBdUJd6IvAxOpwXKyF1bpIxkrSs6QEMQuHFecx0YDjaksn5jR0o+pulK4yiatQ1UDPcMaOmirLdSakHbh+WNDPjnoUS2+6K725YsgxaCoXIO65RvbrFjEH8HC/vFzDOhKVGhg/MCxB4+5EgueRz8rEmlcB6j6S+GVVzCTCV4abtyiBPWDPmFYnxMGwyzERkh+v46erw9txlXejSpIUsze1tvbd6LJhN/OCYJ4Geo74QyxbxybS4GXsinyfuhvOUAO+KAIvle/fZ+7vFmtiEidGLIsmFY/mTvGRGM8dXqkBhKqozYxwZJ6Phk90NN6z4dU1OHFNGcFQ1XzdDuj1biPkQJSkNwWmep0ulSX0pVtn0tS1j/OVwsGgTtnfpxtb/tQ3HLzX+JbOG2a7lArBTk69ziHnNHnzJdfIXKOAbnvspUaw2AjgqYhs3PVvxAIc36QVTKhM5kdHSj/7er3QgFazfoJZ3PhnNN/IATCudKZlMjpneqHBdv4ahjhte90Im++t1tOI1DPQJ1Q9pYY4gfmmzI32f4G7xnkWlN/QXNNuXjiqYx8WiRabblAuq7AIYGVZwu92uBfLV9CzA6FtPgGpPdOU70Wb75BKYfsU+eLs1cXpnokl2AJo9PbOSOGun9+qd68SPd3vmMrM3uMhdkYjTQvPth/R+YoRtseyvBp7k+x7BuafuDMYfYGgMJFvfq659Dpdd4iH155CnzvPkqmOnA/2UnmbmgE5vjUSQeH8I5Kf+kY32LWu0cTh79xqdyXU9khqlILaAR6AU1q7qCJKd9lpBLeoxjsDxTMN0yrHxv810MvAYyzi2D4nhNjX8dxgDhpZOO1Eufm6nEm/2H+nsEb5fMjivPRs1DfcamAWWnnwuzpB10QZhbZWh6DcRvitD5kEQLEHxMWnak+2e05+M14+ZQjIImqzOvWju4TunA7p8SvGUMAL+xjJK3mfhTiSil5Kfz2htQXbExpm7oOLBU7LNjVhVWYLTlK/LQtp14Vv11bfvGd4yvGiaXYEi++m9+KTJln9bfBHU/3Ut6VMb0XSGt356GIZ0iISDPneZOWCxfvFeazg2ukEA6AN596dUib9fZz/S8ABvCs5F6EEKL9IlCQ6UeTBfDbKSrQrKStLYhZ2z3TiTZq1fbg/veJm4x5UupbpHQUTY0MoEZ4Ptm7k2/jNxjG+jn2fDL5Dr7MZ7v3moTXnU5qUY7hZ8b6nhy2PnzzVj5f+Afo3chC/QpXQ8GF0MRq3sEsdKayvVhsfY2gyI+XoA//gcQ5/p0VNhovHUv/xT/E93skZ9xcte5zqD96f4rSHb4ZwqpIEaxONsXLKThKYOxnC9Slft5D+ZKhVshpQjAKuYmQsoFHRghZk6i4= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0ed7eea0-01be-4e14-a65f-08db18c5a23c X-MS-Exchange-CrossTenant-AuthSource: DM8PR01MB6824.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2023 13:22:11.3658 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: B6mlNueOaw4Z6XRfsR/bpERinI/5xJ7LNgu9lyfvMretx1cTyyaFX87zFnJU1KATKpIcAYTnQcaQLsF0dRPJsf23i6Ra4IRM7o5sTHQZ/48i3vNNA0cCdyaXOCdJnCuu X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR01MB2719 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230227_052222_287971_E772CCDD X-CRM114-Status: GOOD ( 14.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When the host kernel is booted with iommu passthrough mode, PF and VFs are enumerated with iommu/smmu domain set to bypass mode. In bypass mode, ATS is not enabled on all VFs and associated PF. When VFs are attached to a VM, the corresponding iommu domain is set to SMMU translation mode and smmu-v3 driver try to enable the ATS and fails due to invalid STU of a PF. Adding a fix to configure STU of a PF while enumerating in passthrough mode. Signed-off-by: Ganapatrao Kulkarni --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 25 ++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index f2425b0f0cd6..b218ef0bf001 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2292,6 +2292,23 @@ static bool arm_smmu_ats_supported(struct arm_smmu_master *master) return dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)); } +static void arm_smmu_ats_stu_init(struct arm_smmu_master *master) +{ + size_t stu; + struct pci_dev *pdev; + struct arm_smmu_device *smmu = master->smmu; + + if (master->ats_enabled) + return; + + /* Smallest Translation Unit: log2 of the smallest supported granule */ + stu = __ffs(smmu->pgsize_bitmap); + pdev = to_pci_dev(master->dev); + + if (pci_ats_stu_configure(pdev, stu)) + dev_err(master->dev, "Failed to configure ATS STU (%zu)\n", stu); +} + static void arm_smmu_enable_ats(struct arm_smmu_master *master) { size_t stu; @@ -2404,6 +2421,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) struct arm_smmu_device *smmu; struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct arm_smmu_master *master; + bool ats_supported; if (!fwspec) return -ENOENT; @@ -2446,9 +2464,10 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } master->domain = smmu_domain; + ats_supported = arm_smmu_ats_supported(master); if (smmu_domain->stage != ARM_SMMU_DOMAIN_BYPASS) - master->ats_enabled = arm_smmu_ats_supported(master); + master->ats_enabled = ats_supported; arm_smmu_install_ste_for_dev(master); @@ -2458,6 +2477,10 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) arm_smmu_enable_ats(master); + /* Configure ATS STU of a PF in passthrough */ + if (!master->ats_enabled && ats_supported) + arm_smmu_ats_stu_init(master); + out_unlock: mutex_unlock(&smmu_domain->init_mutex); return ret;