From patchwork Tue Mar 12 16:25:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 13590291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 96082C54E60 for ; Tue, 12 Mar 2024 16:26:28 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.691955.1078611 (Exim 4.92) (envelope-from ) id 1rk4x5-0005Wj-VH; Tue, 12 Mar 2024 16:26:07 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 691955.1078611; Tue, 12 Mar 2024 16:26:07 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rk4x5-0005Wc-Rt; Tue, 12 Mar 2024 16:26:07 +0000 Received: by outflank-mailman (input) for mailman id 691955; Tue, 12 Mar 2024 16:26:06 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rk4x4-0005WW-0t for xen-devel@lists.xenproject.org; Tue, 12 Mar 2024 16:26:06 +0000 Received: from wfout3-smtp.messagingengine.com (wfout3-smtp.messagingengine.com [64.147.123.146]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 34ec0686-e08d-11ee-a1ee-f123f15fe8a2; Tue, 12 Mar 2024 17:26:01 +0100 (CET) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.west.internal (Postfix) with ESMTP id A04A81C00107; Tue, 12 Mar 2024 12:25:55 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Tue, 12 Mar 2024 12:25:56 -0400 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 12 Mar 2024 12:25:53 -0400 (EDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 34ec0686-e08d-11ee-a1ee-f123f15fe8a2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:subject:subject:to:to; s=fm1; t=1710260755; x=1710347155; bh=NwcXswoAPzQCq70LrrXUbpoKQWinPR2b G2OYKwUeiIk=; b=NdyzPkKfBUnMCRDXJkCF5Efyra+fwbVJBjS+/qpCdrw8U8f4 +QLm6/ZkLwx70P8eQzOuGrt0eDN2P0WiZTSCu+nBmgHa3x+cxhdIV3Dw82V5EwN5 tuN9waxgYPY5ZacOoPegaKWcaD8D0iFaNPu+I3rFpu6wb5zGvkos1crhpmm2KxaA IqkLe6OvINtXHvoo1OUeGRtdGOc7cYV86meh19WtRPPjI1cqY85MJY4pOoDmnJQm iclMOetVya90OvHhsRRde+DHTkhLx0w1Z8t1wWOCjeJyVKAN66daSj8/xCpg0PSD 2CBEvG903l17JqRsdh2LpNIafV8YjNRE8PrD+g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:message-id:mime-version:reply-to:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; t=1710260755; x=1710347155; bh=NwcXswoAPzQCq 70LrrXUbpoKQWinPR2bG2OYKwUeiIk=; b=TrL+78tDNL6FQSO6FRdSMc+tja5Ce x0VchrnpX1r/M+hLHObz1k07WF1XfmGPSu6bfXkRCckD1wUleXs+l1onRAZFzW/e vHkT6QPb394eblf2pbGctHlKLrmL74iHzr5Rw/AqqRX7tamdZgp4kBCd0u8IyP3P tJMMHYFbxZRqnj2G/3iUOhdBpZmACmGj+1OcUoZKRK7FMcd5jhKslxW1fedmQhFq YrLVU1LyYfVB7DJNBBSl8Z5Q9tQeyPGawwgz2mNP/w6+hxuUwpnwCk0MZsL8m26k 8ud0irLwN5gTMemH5HREfiZRUN4zrwZhJZwOEXPwHTv5JbcoH2lhDVW4A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrjeefgdekkecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffogggtgfesthekredtredtjeenucfhrhhomhepofgrrhgvkhcu ofgrrhgtiiihkhhofihskhhiqdfikphrvggtkhhiuceomhgrrhhmrghrvghksehinhhvih hsihgslhgvthhhihhnghhslhgrsgdrtghomheqnecuggftrfgrthhtvghrnhepleekhfdu leetleelleetteevfeefteffkeetteejheelgfegkeelgeehhfdthedvnecuvehluhhsth gvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghksehi nhhvihhsihgslhgvthhhihhnghhslhgrsgdrtghomh X-ME-Proxy: Feedback-ID: i1568416f:Fastmail From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Cc: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Andrew Cooper , George Dunlap , Jan Beulich , Julien Grall , Stefano Stabellini , Wei Liu , Paul Durrant , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= Subject: [PATCH v2 1/2] IOMMU: store name for extra reserved device memory Date: Tue, 12 Mar 2024 17:25:14 +0100 Message-ID: <20240312162541.384793-1-marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 It will be useful for error reporting in a subsequent patch. Signed-off-by: Marek Marczykowski-Górecki Acked-by: Jan Beulich --- New in v2 --- xen/drivers/char/xhci-dbc.c | 3 ++- xen/drivers/passthrough/iommu.c | 5 ++++- xen/include/xen/iommu.h | 3 ++- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/xen/drivers/char/xhci-dbc.c b/xen/drivers/char/xhci-dbc.c index 3bf389be7d0b..8e2037f1a5f7 100644 --- a/xen/drivers/char/xhci-dbc.c +++ b/xen/drivers/char/xhci-dbc.c @@ -1421,7 +1421,8 @@ void __init xhci_dbc_uart_init(void) iommu_add_extra_reserved_device_memory( PFN_DOWN(virt_to_maddr(&dbc_dma_bufs)), PFN_UP(sizeof(dbc_dma_bufs)), - uart->dbc.sbdf); + uart->dbc.sbdf, + "XHCI console"); serial_register_uart(SERHND_XHCI, &dbc_uart_driver, &dbc_uart); } } diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c index 996c31be1284..03587c0cd680 100644 --- a/xen/drivers/passthrough/iommu.c +++ b/xen/drivers/passthrough/iommu.c @@ -682,6 +682,7 @@ struct extra_reserved_range { unsigned long start; unsigned long nr; pci_sbdf_t sbdf; + const char *name; }; static unsigned int __initdata nr_extra_reserved_ranges; static struct extra_reserved_range __initdata @@ -689,7 +690,8 @@ static struct extra_reserved_range __initdata int __init iommu_add_extra_reserved_device_memory(unsigned long start, unsigned long nr, - pci_sbdf_t sbdf) + pci_sbdf_t sbdf, + const char *name) { unsigned int idx; @@ -700,6 +702,7 @@ int __init iommu_add_extra_reserved_device_memory(unsigned long start, extra_reserved_ranges[idx].start = start; extra_reserved_ranges[idx].nr = nr; extra_reserved_ranges[idx].sbdf = sbdf; + extra_reserved_ranges[idx].name = name; return 0; } diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h index 9621459c63ee..b7829dff4588 100644 --- a/xen/include/xen/iommu.h +++ b/xen/include/xen/iommu.h @@ -329,7 +329,8 @@ struct iommu_ops { */ extern int iommu_add_extra_reserved_device_memory(unsigned long start, unsigned long nr, - pci_sbdf_t sbdf); + pci_sbdf_t sbdf, + const char *name); /* * To be called by specific IOMMU driver during initialization, * to fetch ranges registered with iommu_add_extra_reserved_device_memory(). From patchwork Tue Mar 12 16:25:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 13590290 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3DBD7C54E58 for ; Tue, 12 Mar 2024 16:26:27 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.691956.1078621 (Exim 4.92) (envelope-from ) id 1rk4x7-0005kx-6A; Tue, 12 Mar 2024 16:26:09 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 691956.1078621; Tue, 12 Mar 2024 16:26:09 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rk4x7-0005kq-3E; Tue, 12 Mar 2024 16:26:09 +0000 Received: by outflank-mailman (input) for mailman id 691956; Tue, 12 Mar 2024 16:26:08 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rk4x6-0005fl-Nx for xen-devel@lists.xenproject.org; Tue, 12 Mar 2024 16:26:08 +0000 Received: from wfhigh4-smtp.messagingengine.com (wfhigh4-smtp.messagingengine.com [64.147.123.155]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 35fadccc-e08d-11ee-afdd-a90da7624cb6; Tue, 12 Mar 2024 17:26:01 +0100 (CET) Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfhigh.west.internal (Postfix) with ESMTP id B13131800098; Tue, 12 Mar 2024 12:25:58 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Tue, 12 Mar 2024 12:25:59 -0400 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 12 Mar 2024 12:25:56 -0400 (EDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 35fadccc-e08d-11ee-afdd-a90da7624cb6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1710260758; x=1710347158; bh=qF84mesdWl /lEGSQYlkmAm+QzNjClBgQx5p+iDQh+vg=; b=FfQK9LIzkOJqaMKwDtuRJehC+s 4fHEDLOEMNUPfMWk0fdeiilK94BFmMhB13NNOh1mbzHFZd6TV2szJcZwYzRBtmzD 0j4RwOwR+ulNKDZJXsQzsqgaqKLSkTPbtFFxBQPHwJZqrDKh6Hb6KkOIalY2wHb0 GJmFK5LNdUJSdP4HkwjrOL2vZe+T7VLwy7rV59poei7o+jB6bdVNB3Njn1OLGy/i 2//7tvUMTadqWkX1BUO41rLD0A+8HZNCQuvPZHESfmkZ2uA+zcqBICA0rF40uZ9u thBXjmNQNL2fWgGb6WZ8y9iYn5v35lEDduQJydtGEQHJfad3LBJoyPzx1DlA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1710260758; x= 1710347158; bh=qF84mesdWl/lEGSQYlkmAm+QzNjClBgQx5p+iDQh+vg=; b=o IdrArvWUvA6ePny+nor9rLKE/1485d1TMywDzezP8X7C8L51/p0nqoPErh9RIpgW f6FDy0dWEdKpcg9/prgHaSdfpHA9jkE6YKJrD9YlPVqA6dWMwXIv9hH2jGMbGB/9 RmSmemXGZU5njVGh1jWRkJIA5vknCFx82sZwk51gZcBT2vDCL99Y4Euxtnhjg0bm /oIW0h+0aQHKXRyVUNqcMIFAxGXvqseuykiYISuY1veWF8dJhL7ylftoohz562Tf D5uCvEdvcChccMG/XHeIKcrl66iqSmizczNA1E8y4wh7jXSiicxtG4aPttHWzNT0 1DwBzBz4hy3o/4WV0pdIA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrjeefgdekkecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfgggtgfesthekredtredtjeenucfhrhhomhepofgrrhgv khcuofgrrhgtiiihkhhofihskhhiqdfikphrvggtkhhiuceomhgrrhhmrghrvghksehinh hvihhsihgslhgvthhhihhnghhslhgrsgdrtghomheqnecuggftrfgrthhtvghrnhepgfeu udehgfdvfeehhedujeehfeduveeugefhkefhheelgeevudetueeiudfggfffnecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghk sehinhhvihhsihgslhgvthhhihhnghhslhgrsgdrtghomh X-ME-Proxy: Feedback-ID: i1568416f:Fastmail From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Cc: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Jan Beulich , Paul Durrant , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Subject: [PATCH v2 2/2] drivers/char: mark extra reserved device memory in memory map Date: Tue, 12 Mar 2024 17:25:15 +0100 Message-ID: <20240312162541.384793-2-marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240312162541.384793-1-marmarek@invisiblethingslab.com> References: <20240312162541.384793-1-marmarek@invisiblethingslab.com> MIME-Version: 1.0 The IOMMU driver checks if RMRR/IVMD are marked as reserved in memory map. This should be true for addresses coming from the firmware, but when extra pages used by Xen itself are included in the mapping, those are taken from usable RAM used. Mark those pages as reserved too. Not marking the pages as reserved didn't caused issues before due to another a bug in IOMMU driver code, that was fixed in 83afa3135830 ("amd-vi: fix IVMD memory type checks"). Failing to reserve memory will lead to panic in IOMMU setup code. And not including the page in IOMMU mapping will lead to broken console (due to IOMMU faults). The pages chosen by the XHCI console driver should still be usable by the CPU though, and the console code already can deal with too slow console by dropping characters (and console not printing anything is a special case of "slow"). When reserving fails print an error message showing which pages failed and who requested them. This should be enough hint to find why XHCI console doesn't work. Fixes: 3a1a7b809ffa "drivers/char: mark DMA buffers as reserved for the XHCI" Signed-off-by: Marek Marczykowski-Górecki --- Alternative error handling could be a panic, but with this version I think it can be avoided. And not panicing gives a better chance to actually see the error message (from the hopefully started dom0), especially as the affected driver is the console one. The reserve_e820_ram() is x86-specific. Is there some equivalent API for ARM, or maybe even some abstract one? That said, I have no way to test XHCI console on ARM, I don't know if such hardware even exists... Changes in v2: - move reserving to iommu_get_extra_reserved_device_memory() to cover all users of iommu_add_extra_reserved_device_memory() - change error handling to not panic, as in this code layout it can skip sending the pages to the IOMMU driver --- xen/drivers/passthrough/iommu.c | 19 +++++++++++++++++++ xen/include/xen/iommu.h | 5 ++++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c index 03587c0cd680..a311a37a2a03 100644 --- a/xen/drivers/passthrough/iommu.c +++ b/xen/drivers/passthrough/iommu.c @@ -21,6 +21,9 @@ #include #include #include +#ifdef CONFIG_X86 +#include +#endif unsigned int __read_mostly iommu_dev_iotlb_timeout = 1000; integer_param("iommu_dev_iotlb_timeout", iommu_dev_iotlb_timeout); @@ -715,6 +718,22 @@ int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func, for ( idx = 0; idx < nr_extra_reserved_ranges; idx++ ) { +#ifdef CONFIG_X86 + if ( !reserve_e820_ram( + &e820, + pfn_to_paddr(extra_reserved_ranges[idx].start), + pfn_to_paddr(extra_reserved_ranges[idx].start + + extra_reserved_ranges[idx].nr)) ) + { + printk(XENLOG_ERR "Failed to reserve [%"PRIx64"-%"PRIx64") for %s, " + "skipping IOMMU mapping for it, some functionality may be broken\n", + pfn_to_paddr(extra_reserved_ranges[idx].start), + pfn_to_paddr(extra_reserved_ranges[idx].start + + extra_reserved_ranges[idx].nr), + extra_reserved_ranges[idx].name); + continue; + } +#endif ret = func(extra_reserved_ranges[idx].start, extra_reserved_ranges[idx].nr, extra_reserved_ranges[idx].sbdf.sbdf, diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h index b7829dff4588..875eaeb90167 100644 --- a/xen/include/xen/iommu.h +++ b/xen/include/xen/iommu.h @@ -324,7 +324,8 @@ struct iommu_ops { }; /* - * To be called by Xen internally, to register extra RMRR/IVMD ranges. + * To be called by Xen internally, to register extra RMRR/IVMD ranges for RAM + * pages. * Needs to be called before IOMMU initialization. */ extern int iommu_add_extra_reserved_device_memory(unsigned long start, @@ -334,6 +335,8 @@ extern int iommu_add_extra_reserved_device_memory(unsigned long start, /* * To be called by specific IOMMU driver during initialization, * to fetch ranges registered with iommu_add_extra_reserved_device_memory(). + * This has a side effect of marking requested ranges as "reserverd" in the + * memory map. */ extern int iommu_get_extra_reserved_device_memory(iommu_grdm_t *func, void *ctxt);