From patchwork Mon Feb 17 10:18:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Teddy Astie X-Patchwork-Id: 13977466 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7E461C0219E for ; Mon, 17 Feb 2025 10:18:29 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.890001.1299025 (Exim 4.92) (envelope-from ) id 1tjyCj-0008O2-IS; Mon, 17 Feb 2025 10:18:21 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 890001.1299025; Mon, 17 Feb 2025 10:18:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tjyCj-0008Nv-Fm; Mon, 17 Feb 2025 10:18:21 +0000 Received: by outflank-mailman (input) for mailman id 890001; Mon, 17 Feb 2025 10:18:19 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tjyCh-0008Nl-Hv for xen-devel@lists.xenproject.org; Mon, 17 Feb 2025 10:18:19 +0000 Received: from mail13.wdc04.mandrillapp.com (mail13.wdc04.mandrillapp.com [205.201.139.13]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 80f9017b-ed18-11ef-9aa6-95dc52dad729; Mon, 17 Feb 2025 11:18:17 +0100 (CET) Received: from pmta16.mandrill.prod.suw01.rsglab.com (localhost [127.0.0.1]) by mail13.wdc04.mandrillapp.com (Mailchimp) with ESMTP id 4YxJWr3KK3zNCd9S4 for ; Mon, 17 Feb 2025 10:18:16 +0000 (GMT) Received: from [37.26.189.201] by mandrillapp.com id 1f2b238252fe47ad80f706973b3bd3fa; Mon, 17 Feb 2025 10:18:16 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 80f9017b-ed18-11ef-9aa6-95dc52dad729 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mandrillapp.com; s=mte1; t=1739787496; x=1740057496; bh=qlPp+jB8VBLIbB0yYS5fKXDPMGM/r8nqip5koUnElg0=; h=From:Subject:To:Cc:Message-Id:Feedback-ID:Date:MIME-Version: Content-Type:Content-Transfer-Encoding:CC:Date:Subject:From; b=IzlFLGztd11tjJq2oDwOQfsfn+GdceLPiE0TznLvHz8lpB3TRpLxACiINgq/+Gy+4 Ut79pB9A7GR9XsnZw9vVKbRUForgpIA2uuEWII/qudDG+GuhUppd/TXAsPvCGsNIBT jDGuKA/JJNgHs50zwQc4D1IGMeR9CPuh7fiJDG1sZyo3UYz/dOPbPW+W9UhjC0tRju 7/XNanW1vGflhrG3Pt9I4B5p0U/67v273offN+fRWZjnKS4sxOoSzrNuainPMOtsdN JmriPT+gpQlbpPW/Yjic8y/q1rYh8w0aLlV5Ho7zqEzpd98Yttcb1NPaG8M2FDraDp SJfDLyRP1aPSQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vates.tech; s=mte1; t=1739787496; x=1740047996; i=teddy.astie@vates.tech; bh=qlPp+jB8VBLIbB0yYS5fKXDPMGM/r8nqip5koUnElg0=; h=From:Subject:To:Cc:Message-Id:Feedback-ID:Date:MIME-Version: Content-Type:Content-Transfer-Encoding:CC:Date:Subject:From; b=s2Ln421em8150/vDWcmurh2XfSJXiKLMpRFT5241APkhrh8XjgyTLlasK+C2WHe45 lXABQMpW/ZGutE+5hWEO9HFwlFlG6qzvjl8cpSqL2kP0mLV0y6zlIhb2PHznvfUaqM hp+skIzRATS3nDiDf+2Zwfkm98AXo2Ra8t7hUgB+tiGgjUW/oWHjbaZgNzalJsRhse OsKkCjCk7zrTIUYcOKFfbs7ZlwsfY9noIs3CBRFGOen0RRtc5QXhmVaQpR9Ug5NSpZ TJwvy5UquEgP2qsr1iBnDROOL56csykjgIQ7fX57Rz8igRuS9/LyARW3qE1JRmsTKF lJBnQ9r9L70Bw== From: "Teddy Astie" Subject: =?utf-8?q?=5BXEN_RFC_PATCH_v6_00/11=5D_IOMMU_subsystem_redesign_and?= =?utf-8?q?_PV-IOMMU_interface?= X-Mailer: git-send-email 2.47.2 X-Bm-Disclaimer: Yes X-Bm-Milter-Handled: 4ffbd6c1-ee69-4e1b-aabd-f977039bd3e2 X-Bm-Transport-Timestamp: 1739787495396 To: xen-devel@lists.xenproject.org Cc: "Teddy Astie" , "Andrew Cooper" , "Anthony PERARD" , "Michal Orzel" , "Jan Beulich" , "Julien Grall" , " =?utf-8?q?Roger_Pau_Monn=C3=A9?= " , "Stefano Stabellini" , "Bertrand Marquis" , "Volodymyr Babchuk" , "Shawn Anastasio" , "Lukasz Hawrylko" , "Daniel P. Smith" , " =?utf-8?q?Mateusz_M?= =?utf-8?q?=C3=B3wka?= " , " =?utf-8?q?Marek_Marczy?= =?utf-8?q?kowski-G=C3=B3recki?= " Message-Id: X-Native-Encoded: 1 X-Report-Abuse: =?utf-8?q?Please_forward_a_copy_of_this_message=2C_including?= =?utf-8?q?_all_headers=2C_to_abuse=40mandrill=2Ecom=2E_You_can_also_report_?= =?utf-8?q?abuse_here=3A_https=3A//mandrillapp=2Ecom/contact/abuse=3Fid=3D30?= =?utf-8?q?504962=2E1f2b238252fe47ad80f706973b3bd3fa?= X-Mandrill-User: md_30504962 Feedback-ID: 30504962:30504962.20250217:md Date: Mon, 17 Feb 2025 10:18:16 +0000 MIME-Version: 1.0 This work has been presented at Xen Summit 2024 during the IOMMU paravirtualization and Xen IOMMU subsystem rework design session. Operating systems may want to have access to a IOMMU in order to do DMA protection or implement certain features (e.g VFIO on Linux). VFIO support is mandatory for framework such as SPDK, which can be useful to implement an alternative storage backend for virtual machines [1]. In this patch series, we introduce in Xen the ability to manage several contexts per domain and provide a new hypercall interface to allow guests to manage IOMMU contexts. The VT-d and AMD-Vi driver is updated to support these new features. [1] Using SPDK with the Xen hypervisor - FOSDEM 2023 --- Cc: Marek Marczykowski-Górecki PCI Passthrough now work on my side, but things are still feels quite brittle. Changed in v2 : * fixed Xen crash when dumping IOMMU contexts (using X debug key) with DomUs without IOMMU * s/dettach/detach/ * removed some unused includes * fix dangling devices in contexts with detach Changed in v3 : * lock entirely map/unmap in hypercall * prevent IOMMU operations on dying contexts (fix race condition) * iommu_check_context+iommu_get_context -> iommu_get_context and check for NULL Changed in v4 : * Part of initialization logic is moved to domain or toolstack (IOMMU_init) + domain/toolstack now decides on "context count" and "pagetable pool size" + for now, all domains are able to initialize PV-IOMMU * introduce "dom0-iommu=no-dma" to make default context block all DMA (disables HAP and sync-pt), enforcing usage of PV-IOMMU for DMA Can be used to expose properly "Pre-boot DMA protection" * redesigned locking logic for contexts + contexts are accessed using iommu_get_context and released with iommu_put_context Changed in v5 : * various PCI Passthrough related fixes + rewrote parts of PCI Passthrough logic + various other related bug fixes * simplified VT-d DID (for hardware) management by only having one map instead of two (pseudo_domid map was previously used for old quarantine code then recycled for PV-IOMMU in addition to another map also tracing Domain<->VT-d DID, now there is only one map tracking both making things simpler) * reworked parts of Xen quarantine logic (needed for PCI Passthrough) * added cf_check annotations * some changes to PV-IOMMU headers (Alejandro) Changed in v6 : * reorganized the patch series to allow bissecting * it is splitted in various smaller patches * initial AMD-Vi port (it doesn't completely work with PV-IOMMU though, but builds at least) * AMD-Vi lacks support for iommu_lookup_page (needed for several PV-IOMMU ops) TODO: * fix some issues with no-dma+PV and grants * complete "no-dma" mode (expose to toolstack, add documentation, ...) * properly define nested mode and PASID support * consider per-iommu domid limit (allocate did on first attach/reattach ?) * fix ARM/PPC build issues * make new quarantine code more unity region aware (isolate devices with different reserved regions regions using separate 'contexts') * find a way to make PV-IOMMU work in DomUs (they don't see machine bdf) * there are corner cases with PV-IOMMU and to-domain Xen PCI Passthrough (e.g pci-assignable-remove will reassign to context 0, while the driver expects the device to to be in context X) Teddy Astie (11): docs/designs: Add a design document for IOMMU subsystem redesign docs/designs: Add a design document for PV-IOMMU x86/domain: Defer domain iommu initialization. iommu: Move IOMMU domain related structures to (arch_)iommu_context iommu: Simplify quarantine logic vtd: Remove MAP_ERROR_RECOVERY code path in domain_context_mapping_one iommu: Simplify hardware did management iommu: Introduce redesigned IOMMU subsystem x86/iommu: Introduce IOMMU arena iommu: Introduce PV-IOMMU iommu: Introduce no-dma feature docs/designs/iommu-contexts.md | 403 +++++ docs/designs/pv-iommu.md | 116 ++ xen/arch/arm/include/asm/iommu.h | 4 + xen/arch/ppc/include/asm/iommu.h | 3 + xen/arch/x86/domain.c | 10 +- xen/arch/x86/include/asm/arena.h | 54 + xen/arch/x86/include/asm/iommu.h | 59 +- xen/arch/x86/include/asm/pci.h | 17 - xen/arch/x86/mm/p2m-ept.c | 2 +- xen/arch/x86/pv/dom0_build.c | 6 +- xen/arch/x86/tboot.c | 3 +- xen/common/Makefile | 1 + xen/common/memory.c | 4 +- xen/common/pv-iommu.c | 539 +++++++ xen/drivers/passthrough/amd/iommu.h | 21 +- xen/drivers/passthrough/amd/iommu_cmd.c | 20 +- xen/drivers/passthrough/amd/iommu_init.c | 13 +- xen/drivers/passthrough/amd/iommu_map.c | 217 +-- xen/drivers/passthrough/amd/pci_amd_iommu.c | 346 ++-- xen/drivers/passthrough/iommu.c | 735 ++++++++- xen/drivers/passthrough/pci.c | 404 ++--- xen/drivers/passthrough/vtd/extern.h | 19 +- xen/drivers/passthrough/vtd/iommu.c | 1612 ++++++------------- xen/drivers/passthrough/vtd/iommu.h | 2 - xen/drivers/passthrough/vtd/qinval.c | 2 +- xen/drivers/passthrough/vtd/quirks.c | 21 +- xen/drivers/passthrough/vtd/vtd.h | 3 +- xen/drivers/passthrough/x86/Makefile | 1 + xen/drivers/passthrough/x86/arena.c | 157 ++ xen/drivers/passthrough/x86/iommu.c | 294 +++- xen/include/hypercall-defs.c | 6 + xen/include/public/pv-iommu.h | 343 ++++ xen/include/public/xen.h | 1 + xen/include/xen/iommu.h | 117 +- xen/include/xen/pci.h | 3 + 35 files changed, 3585 insertions(+), 1973 deletions(-) create mode 100644 docs/designs/iommu-contexts.md create mode 100644 docs/designs/pv-iommu.md create mode 100644 xen/arch/x86/include/asm/arena.h create mode 100644 xen/common/pv-iommu.c create mode 100644 xen/drivers/passthrough/x86/arena.c create mode 100644 xen/include/public/pv-iommu.h