| Message ID | 20200912032200.11489-1-baolu.lu@linux.intel.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Convert the intel iommu driver to the dma-iommu api | expand |
Hi, On 12/09/2020 04:21, Lu Baolu wrote: > Tom Murphy has almost done all the work. His latest patch series was > posted here. > > https://lore.kernel.org/linux-iommu/20200903201839.7327-1-murphyt7@tcd.ie/ > > Thanks a lot! > > This series is a follow-up with below changes: > > 1. Add a quirk for the i915 driver issue described in Tom's cover > letter. Last week I have copied you on an i915 series which appears to remove the need for this quirk. so if we get those i915 patches reviewed and merged, do you still want to pursue this quirk? > 2. Fix several bugs in patch "iommu: Allow the dma-iommu api to use > bounce buffers" to make the bounce buffer work for untrusted devices. > 3. Several cleanups in iommu/vt-d driver after the conversion. With the previous version of the series I hit a problem on Ivybridge where apparently the dma engine width is not respected. At least that is my layman interpretation of the errors. From the older thread: <3> [209.526605] DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (ffff008000) Relevant iommu boot related messages are: <6>[ 0.184234] DMAR: Host address width 36 <6>[ 0.184245] DMAR: DRHD base: 0x000000fed90000 flags: 0x0 <6>[ 0.184288] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a <6>[ 0.184308] DMAR: DRHD base: 0x000000fed91000 flags: 0x1 <6>[ 0.184337] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a <6>[ 0.184357] DMAR: RMRR base: 0x000000d8d28000 end: 0x000000d8d46fff <6>[ 0.184377] DMAR: RMRR base: 0x000000db000000 end: 0x000000df1fffff <6>[ 0.184398] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 <6>[ 0.184414] DMAR-IR: HPET id 0 under DRHD base 0xfed91000 <6>[ 0.184428] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. <6>[ 0.185173] DMAR-IR: Enabled IRQ remapping in x2apic mode <6>[ 0.878934] DMAR: No ATSR found <6>[ 0.878966] DMAR: dmar0: Using Queued invalidation <6>[ 0.879007] DMAR: dmar1: Using Queued invalidation <6>[ 0.915032] DMAR: Intel(R) Virtualization Technology for Directed I/O <6>[ 0.915060] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) <6>[ 0.915084] software IO TLB: mapped [mem 0xc80d4000-0xcc0d4000] (64MB) (Full boot log at https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/boot0.txt, failures at https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/igt@i915_selftest@live@blt.html.) Does this look familiar or at least plausible to you? Is this something your new series has fixed? Regards, Tvrtko > > Please review and test. > > Best regards, > baolu > > Lu Baolu (2): > iommu: Add quirk for Intel graphic devices in map_sg > iommu/vt-d: Cleanup after converting to dma-iommu ops > > Tom Murphy (4): > iommu: Handle freelists when using deferred flushing in iommu drivers > iommu: Add iommu_dma_free_cpu_cached_iovas() > iommu: Allow the dma-iommu api to use bounce buffers > iommu/vt-d: Convert intel iommu driver to the iommu ops > > .../admin-guide/kernel-parameters.txt | 5 - > drivers/iommu/dma-iommu.c | 229 ++++- > drivers/iommu/intel/Kconfig | 1 + > drivers/iommu/intel/iommu.c | 885 +++--------------- > include/linux/dma-iommu.h | 8 + > include/linux/iommu.h | 1 + > 6 files changed, 323 insertions(+), 806 deletions(-) >
Hi Tvrtko, On 9/14/20 4:04 PM, Tvrtko Ursulin wrote: > > Hi, > > On 12/09/2020 04:21, Lu Baolu wrote: >> Tom Murphy has almost done all the work. His latest patch series was >> posted here. >> >> https://lore.kernel.org/linux-iommu/20200903201839.7327-1-murphyt7@tcd.ie/ >> >> Thanks a lot! >> >> This series is a follow-up with below changes: >> >> 1. Add a quirk for the i915 driver issue described in Tom's cover >> letter. > > Last week I have copied you on an i915 series which appears to remove the need for this quirk. so if we get those i915 patches reviewed and merged, do you still want to pursue this quirk? It's up to the graphic guys. I don't know the details in i915 driver. I don't think my tests could cover all cases. > >> 2. Fix several bugs in patch "iommu: Allow the dma-iommu api to use >> bounce buffers" to make the bounce buffer work for untrusted devices. >> 3. Several cleanups in iommu/vt-d driver after the conversion. > > With the previous version of the series I hit a problem on Ivybridge where apparently the dma engine width is not respected. At least that is my layman interpretation of the errors. From the older thread: > > <3> [209.526605] DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (ffff008000) > > Relevant iommu boot related messages are: > > <6>[ 0.184234] DMAR: Host address width 36 > <6>[ 0.184245] DMAR: DRHD base: 0x000000fed90000 flags: 0x0 > <6>[ 0.184288] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a > <6>[ 0.184308] DMAR: DRHD base: 0x000000fed91000 flags: 0x1 > <6>[ 0.184337] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a > <6>[ 0.184357] DMAR: RMRR base: 0x000000d8d28000 end: 0x000000d8d46fff > <6>[ 0.184377] DMAR: RMRR base: 0x000000db000000 end: 0x000000df1fffff > <6>[ 0.184398] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 > <6>[ 0.184414] DMAR-IR: HPET id 0 under DRHD base 0xfed91000 > <6>[ 0.184428] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. > <6>[ 0.185173] DMAR-IR: Enabled IRQ remapping in x2apic mode > > <6>[ 0.878934] DMAR: No ATSR found > <6>[ 0.878966] DMAR: dmar0: Using Queued invalidation > <6>[ 0.879007] DMAR: dmar1: Using Queued invalidation > > <6>[ 0.915032] DMAR: Intel(R) Virtualization Technology for Directed I/O > <6>[ 0.915060] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) > <6>[ 0.915084] software IO TLB: mapped [mem 0xc80d4000-0xcc0d4000] (64MB) > > (Full boot log at https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/boot0.txt, failures at https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/igt@i915_selftest@live@blt.html.) > > Does this look familiar or at least plausible to you? Is this something your new series has fixed? This happens during attaching a domain to device. It has nothing to do with this patch series. I will look into this issue, but not in this email thread context. Best regards, baolu > > Regards, > > Tvrtko > >> >> Please review and test. >> >> Best regards, >> baolu >> >> Lu Baolu (2): >> iommu: Add quirk for Intel graphic devices in map_sg >> iommu/vt-d: Cleanup after converting to dma-iommu ops >> >> Tom Murphy (4): >> iommu: Handle freelists when using deferred flushing in iommu drivers >> iommu: Add iommu_dma_free_cpu_cached_iovas() >> iommu: Allow the dma-iommu api to use bounce buffers >> iommu/vt-d: Convert intel iommu driver to the iommu ops >> >> .../admin-guide/kernel-parameters.txt | 5 - >> drivers/iommu/dma-iommu.c | 229 ++++- >> drivers/iommu/intel/Kconfig | 1 + >> drivers/iommu/intel/iommu.c | 885 +++--------------- >> include/linux/dma-iommu.h | 8 + >> include/linux/iommu.h | 1 + >> 6 files changed, 323 insertions(+), 806 deletions(-) >>
On 15/09/2020 02:47, Lu Baolu wrote: > Hi Tvrtko, > > On 9/14/20 4:04 PM, Tvrtko Ursulin wrote: >> >> Hi, >> >> On 12/09/2020 04:21, Lu Baolu wrote: >>> Tom Murphy has almost done all the work. His latest patch series was >>> posted here. >>> >>> https://lore.kernel.org/linux-iommu/20200903201839.7327-1-murphyt7@tcd.ie/ >>> >>> >>> Thanks a lot! >>> >>> This series is a follow-up with below changes: >>> >>> 1. Add a quirk for the i915 driver issue described in Tom's cover >>> letter. >> >> Last week I have copied you on an i915 series which appears to remove >> the need for this quirk. so if we get those i915 patches reviewed and >> merged, do you still want to pursue this quirk? > > It's up to the graphic guys. I don't know the details in i915 driver. > I don't think my tests could cover all cases. I am the graphic guy. :) I just need some reviews (internally) for my series and then we can merge it, at which point you don't need the quirk patch any more. I'll try to accelerate this. With regards to testing, you could send your series with my patches on top to our trybot mailing list (intel-gfx-trybot@lists.freedesktop.org / https://patchwork.freedesktop.org/project/intel-gfx-trybot/series/?ordering=-last_updated) which would show you if it is still hitting the DMAR issues in i915. >> >>> 2. Fix several bugs in patch "iommu: Allow the dma-iommu api to use >>> bounce buffers" to make the bounce buffer work for untrusted devices. >>> 3. Several cleanups in iommu/vt-d driver after the conversion. >> >> With the previous version of the series I hit a problem on Ivybridge >> where apparently the dma engine width is not respected. At least that >> is my layman interpretation of the errors. From the older thread: >> >> <3> [209.526605] DMAR: intel_iommu_map: iommu width (39) is not >> sufficient for the mapped address (ffff008000) >> >> Relevant iommu boot related messages are: >> >> <6>[ 0.184234] DMAR: Host address width 36 >> <6>[ 0.184245] DMAR: DRHD base: 0x000000fed90000 flags: 0x0 >> <6>[ 0.184288] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap >> c0000020e60262 ecap f0101a >> <6>[ 0.184308] DMAR: DRHD base: 0x000000fed91000 flags: 0x1 >> <6>[ 0.184337] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap >> c9008020660262 ecap f0105a >> <6>[ 0.184357] DMAR: RMRR base: 0x000000d8d28000 end: 0x000000d8d46fff >> <6>[ 0.184377] DMAR: RMRR base: 0x000000db000000 end: 0x000000df1fffff >> <6>[ 0.184398] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 >> IOMMU 1 >> <6>[ 0.184414] DMAR-IR: HPET id 0 under DRHD base 0xfed91000 >> <6>[ 0.184428] DMAR-IR: Queued invalidation will be enabled to >> support x2apic and Intr-remapping. >> <6>[ 0.185173] DMAR-IR: Enabled IRQ remapping in x2apic mode >> >> <6>[ 0.878934] DMAR: No ATSR found >> <6>[ 0.878966] DMAR: dmar0: Using Queued invalidation >> <6>[ 0.879007] DMAR: dmar1: Using Queued invalidation >> >> <6>[ 0.915032] DMAR: Intel(R) Virtualization Technology for >> Directed I/O >> <6>[ 0.915060] PCI-DMA: Using software bounce buffering for IO >> (SWIOTLB) >> <6>[ 0.915084] software IO TLB: mapped [mem 0xc80d4000-0xcc0d4000] >> (64MB) >> >> (Full boot log at >> https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/boot0.txt, >> failures at >> https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/igt@i915_selftest@live@blt.html.) >> >> >> Does this look familiar or at least plausible to you? Is this >> something your new series has fixed? > > This happens during attaching a domain to device. It has nothing to do > with this patch series. I will look into this issue, but not in this > email thread context. I am not sure what step is attaching domain to device, but these type messages: <3> [209.526605] DMAR: intel_iommu_map: iommu width (39) is not >> sufficient for the mapped address (ffff008000) They definitely appear to happen at runtime, as i915 is getting exercised by userspace. Regards, Tvrtko
Hi Lu, On 2020-09-11 9:21 p.m., Lu Baolu wrote: > Tom Murphy has almost done all the work. His latest patch series was > posted here. > > https://lore.kernel.org/linux-iommu/20200903201839.7327-1-murphyt7@tcd.ie/ > > Thanks a lot! > > This series is a follow-up with below changes: > > 1. Add a quirk for the i915 driver issue described in Tom's cover > letter. > 2. Fix several bugs in patch "iommu: Allow the dma-iommu api to use > bounce buffers" to make the bounce buffer work for untrusted devices. > 3. Several cleanups in iommu/vt-d driver after the conversion. > I'm trying to test this on an old Sandy Bridge, but found that I get spammed with warnings on boot. I've put a sample of a few of them below. They all seem to be related to ioat. I had the same issue with Tom's v2 but never saw this on his v1. Thanks, Logan [ 44.057877] ------------[ cut here ]------------ [ 44.063167] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 [ 44.073351] Modules linked in: [ 44.076882] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 [ 44.087935] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 [ 44.096650] Workqueue: events work_for_cpu_fn [ 44.101644] RIP: 0010:__iommu_dma_unmap+0x115/0x130 [ 44.107225] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 [ 44.128443] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 [ 44.134413] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 [ 44.144487] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 [ 44.152613] RBP: 00000000fec00000 R08: 00000000000fec00 R09: 00000000000fedff [ 44.160733] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c39000 [ 44.168861] R13: ffff888472d85ee8 R14: 0000000000080000 R15: 0000000000000000 [ 44.176980] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 [ 44.186198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 44.192761] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 [ 44.200871] Call Trace: [ 44.203716] ? lockdep_hardirqs_on_prepare+0xad/0x180 [ 44.209509] iommu_dma_free+0x18/0x30 [ 44.213734] ioat_free_chan_resources+0x19e/0x300 [ 44.219133] ioat_dma_self_test+0x2a0/0x3d0 [ 44.223964] ioat_pci_probe+0x60e/0x903 [ 44.228387] local_pci_probe+0x42/0x80 [ 44.232721] work_for_cpu_fn+0x16/0x20 [ 44.237037] process_one_work+0x292/0x630 [ 44.241644] ? __schedule+0x344/0x970 [ 44.245868] worker_thread+0x227/0x3e0 [ 44.250185] ? process_one_work+0x630/0x630 [ 44.254989] kthread+0x136/0x150 [ 44.258718] ? kthread_use_mm+0x100/0x100 [ 44.263334] ret_from_fork+0x22/0x30 [ 44.267474] irq event stamp: 1881262 [ 44.271602] hardirqs last enabled at (1881272): [<ffffffff811b8465>] console_unlock+0x435/0x570 [ 44.281593] hardirqs last disabled at (1881281): [<ffffffff811b845b>] console_unlock+0x42b/0x570 [ 44.291588] softirqs last enabled at (1747140): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 [ 44.301285] softirqs last disabled at (1747144): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 [ 44.312153] ---[ end trace ed0f88fd959a5a39 ]--- [ 44.353963] ------------[ cut here ]------------ [ 44.359291] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 [ 44.369482] Modules linked in: [ 44.373030] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 [ 44.384097] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 [ 44.392825] Workqueue: events work_for_cpu_fn [ 44.397831] RIP: 0010:__iommu_dma_unmap+0x115/0x130 [ 44.403421] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 [ 44.424644] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 [ 44.430627] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 [ 44.438770] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 [ 44.446885] RBP: 00000000ffc00000 R08: 00000000000ffc00 R09: 00000000000ffdff [ 44.455000] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 [ 44.463119] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 [ 44.471235] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 [ 44.480440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 44.487004] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 [ 44.495128] Call Trace: [ 44.498008] iommu_dma_free+0x18/0x30 [ 44.502230] ioat_free_chan_resources+0x19e/0x300 [ 44.507631] ioat_dma_self_test+0x2a0/0x3d0 [ 44.512466] ioat_pci_probe+0x60e/0x903 [ 44.516889] local_pci_probe+0x42/0x80 [ 44.521217] work_for_cpu_fn+0x16/0x20 [ 44.525540] process_one_work+0x292/0x630 [ 44.530157] ? __schedule+0x344/0x970 [ 44.534393] worker_thread+0x227/0x3e0 [ 44.538720] ? process_one_work+0x630/0x630 [ 44.543531] kthread+0x136/0x150 [ 44.547270] ? kthread_use_mm+0x100/0x100 [ 44.551889] ret_from_fork+0x22/0x30 [ 44.556034] irq event stamp: 2145654 [ 44.560161] hardirqs last enabled at (2145664): [<ffffffff811b8465>] console_unlock+0x435/0x570 [ 44.570158] hardirqs last disabled at (2145673): [<ffffffff811b845b>] console_unlock+0x42b/0x570 [ 44.580153] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 [ 44.589854] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 [ 44.600729] ---[ end trace ed0f88fd959a5a3a ]--- [ 44.606043] ------------[ cut here ]------------ [ 44.611331] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 [ 44.621524] Modules linked in: [ 44.625050] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 [ 44.636119] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 [ 44.644834] Workqueue: events work_for_cpu_fn [ 44.649827] RIP: 0010:__iommu_dma_unmap+0x115/0x130 [ 44.655410] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 [ 44.676618] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 [ 44.682588] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 [ 44.690702] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 [ 44.698816] RBP: 00000000ffa00000 R08: 00000000000ffa00 R09: 00000000000ffbff [ 44.706934] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 [ 44.715054] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 [ 44.723174] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 [ 44.732384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 44.738941] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 [ 44.747060] Call Trace: [ 44.749921] ? lockdep_hardirqs_on_prepare+0xad/0x180 [ 44.755706] iommu_dma_free+0x18/0x30 [ 44.759931] ioat_free_chan_resources+0x19e/0x300 [ 44.765332] ioat_dma_self_test+0x2a0/0x3d0 [ 44.770156] ioat_pci_probe+0x60e/0x903 [ 44.774583] local_pci_probe+0x42/0x80 [ 44.778909] work_for_cpu_fn+0x16/0x20 [ 44.783232] process_one_work+0x292/0x630 [ 44.787846] ? __schedule+0x344/0x970 [ 44.792079] worker_thread+0x227/0x3e0 [ 44.796403] ? process_one_work+0x630/0x630 [ 44.801205] kthread+0x136/0x150 [ 44.804946] ? kthread_use_mm+0x100/0x100 [ 44.809563] ret_from_fork+0x22/0x30 [ 44.813704] irq event stamp: 2146044 [ 44.817831] hardirqs last enabled at (2146054): [<ffffffff811b8465>] console_unlock+0x435/0x570 [ 44.827821] hardirqs last disabled at (2146063): [<ffffffff811b845b>] console_unlock+0x42b/0x570 [ 44.837810] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 [ 44.847503] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 [ 44.858366] ---[ end trace ed0f88fd959a5a3b ]--- [ 44.863675] ------------[ cut here ]------------ [ 44.868969] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 [ 44.879148] Modules linked in: [ 44.882689] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 [ 44.893760] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 [ 44.902487] Workqueue: events work_for_cpu_fn [ 44.907493] RIP: 0010:__iommu_dma_unmap+0x115/0x130 [ 44.913080] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 [ 44.934290] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 [ 44.940254] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 [ 44.948379] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 [ 44.956505] RBP: 00000000ff800000 R08: 00000000000ff800 R09: 00000000000ff9ff [ 44.964627] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 [ 44.972748] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 [ 44.980871] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 [ 44.990084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 44.996644] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 [ 45.004759] Call Trace: [ 45.007600] ? lockdep_hardirqs_on_prepare+0xad/0x180 [ 45.013391] iommu_dma_free+0x18/0x30 [ 45.017609] ioat_free_chan_resources+0x19e/0x300 [ 45.022992] ioat_dma_self_test+0x2a0/0x3d0 [ 45.027809] ioat_pci_probe+0x60e/0x903 [ 45.032237] local_pci_probe+0x42/0x80 [ 45.036563] work_for_cpu_fn+0x16/0x20 [ 45.040886] process_one_work+0x292/0x630 [ 45.045500] ? __schedule+0x344/0x970 [ 45.049735] worker_thread+0x227/0x3e0 [ 45.054061] ? process_one_work+0x630/0x630 [ 45.058864] kthread+0x136/0x150 [ 45.062597] ? kthread_use_mm+0x100/0x100 [ 45.067200] ret_from_fork+0x22/0x30 [ 45.071335] irq event stamp: 2146432 [ 45.075458] hardirqs last enabled at (2146442): [<ffffffff811b8465>] console_unlock+0x435/0x570 [ 45.085440] hardirqs last disabled at (2146451): [<ffffffff811b845b>] console_unlock+0x42b/0x570 [ 45.095432] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 [ 45.105134] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 [ 45.116004] ---[ end trace ed0f88fd959a5a3c ]--- [ 45.121305] ------------[ cut here ]------------ [ 45.126596] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 [ 45.136772] Modules linked in: [ 45.142288] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 [ 45.153351] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 [ 45.162077] Workqueue: events work_for_cpu_fn [ 45.167074] RIP: 0010:__iommu_dma_unmap+0x115/0x130 [ 45.172663] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 [ 45.193874] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 [ 45.199850] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 [ 45.207970] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 [ 45.216095] RBP: 00000000ff600000 R08: 00000000000ff600 R09: 00000000000ff7ff [ 45.224207] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 [ 45.232327] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 [ 45.240446] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 [ 45.249662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 45.256225] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 [ 45.264344] Call Trace: [ 45.267205] ? lockdep_hardirqs_on_prepare+0xad/0x180 [ 45.273003] iommu_dma_free+0x18/0x30 [ 45.277231] ioat_free_chan_resources+0x19e/0x300 [ 45.282631] ioat_dma_self_test+0x2a0/0x3d0 [ 45.287457] ioat_pci_probe+0x60e/0x903 [ 45.291894] local_pci_probe+0x42/0x80 [ 45.296217] work_for_cpu_fn+0x16/0x20 [ 45.300539] process_one_work+0x292/0x630 [ 45.305162] ? __schedule+0x344/0x970 [ 45.309399] worker_thread+0x227/0x3e0 [ 45.313724] ? process_one_work+0x630/0x630 [ 45.318535] kthread+0x136/0x150 [ 45.322274] ? kthread_use_mm+0x100/0x100 [ 45.326893] ret_from_fork+0x22/0x30 [ 45.331034] irq event stamp: 2146822 [ 45.335161] hardirqs last enabled at (2146832): [<ffffffff811b8465>] console_unlock+0x435/0x570 [ 45.345154] hardirqs last disabled at (2146841): [<ffffffff811b845b>] console_unlock+0x42b/0x570 [ 45.355150] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 [ 45.364850] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 [ 45.375710] ---[ end trace ed0f88fd959a5a3d ]--- [ 45.381018] ------------[ cut here ]------------ [ 45.386308] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 [ 45.396494] Modules linked in: [ 45.400020] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 [ 45.411086] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 [ 45.419810] Workqueue: events work_for_cpu_fn [ 45.424819] RIP: 0010:__iommu_dma_unmap+0x115/0x130 [ 45.430406] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 [ 45.451630] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 [ 45.457610] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 [ 45.465730] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 [ 45.473853] RBP: 00000000ff400000 R08: 00000000000ff400 R09: 00000000000ff5ff [ 45.481973] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 [ 45.490092] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 [ 45.498211] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 [ 45.507418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 45.513969] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 [ 45.522081] Call Trace: [ 45.524934] ? lockdep_hardirqs_on_prepare+0xad/0x180 [ 45.530718] iommu_dma_free+0x18/0x30 [ 45.534989] ioat_free_chan_resources+0x19e/0x300 [ 45.540378] ioat_dma_self_test+0x2a0/0x3d0 [ 45.545193] ioat_pci_probe+0x60e/0x903 [ 45.549619] local_pci_probe+0x42/0x80 [ 45.553940] work_for_cpu_fn+0x16/0x20 [ 45.558258] process_one_work+0x292/0x630 [ 45.562871] ? __schedule+0x344/0x970 [ 45.567108] worker_thread+0x227/0x3e0 [ 45.571434] ? process_one_work+0x630/0x630 [ 45.576243] kthread+0x136/0x150 [ 45.579981] ? kthread_use_mm+0x100/0x100 [ 45.584598] ret_from_fork+0x22/0x30 [ 45.588733] irq event stamp: 2147210 [ 45.592860] hardirqs last enabled at (2147220): [<ffffffff811b8465>] console_unlock+0x435/0x570 [ 45.602845] hardirqs last disabled at (2147229): [<ffffffff811b845b>] console_unlock+0x42b/0x570 [ 45.612836] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 [ 45.622531] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 [ 45.633392] ---[ end trace ed0f88fd959a5a3e ]--- [ 45.638708] ------------[ cut here ]------------ [ 45.644003] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 [ 45.654191] Modules linked in: [ 45.657734] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 [ 45.668807] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 [ 45.677534] Workqueue: events work_for_cpu_fn [ 45.682538] RIP: 0010:__iommu_dma_unmap+0x115/0x130 [ 45.688126] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 [ 45.709343] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 [ 45.715321] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 [ 45.723440] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 [ 45.731555] RBP: 00000000ff200000 R08: 00000000000ff200 R09: 00000000000ff3ff [ 45.739665] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 [ 45.747777] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 [ 45.755897] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 [ 45.765106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 45.771661] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 [ 45.779778] Call Trace: [ 45.782638] ? lockdep_hardirqs_on_prepare+0xad/0x180 [ 45.788439] iommu_dma_free+0x18/0x30 [ 45.792664] ioat_free_chan_resources+0x19e/0x300 [ 45.798060] ioat_dma_self_test+0x2a0/0x3d0 [ 45.802886] ioat_pci_probe+0x60e/0x903 [ 45.807306] local_pci_probe+0x42/0x80 [ 45.811633] work_for_cpu_fn+0x16/0x20 [ 45.815946] process_one_work+0x292/0x630 [ 45.820556] ? __schedule+0x344/0x970 [ 45.824791] worker_thread+0x227/0x3e0 [ 45.829117] ? process_one_work+0x630/0x630 [ 45.833969] kthread+0x136/0x150 [ 45.837706] ? kthread_use_mm+0x100/0x100 [ 45.842325] ret_from_fork+0x22/0x30 [ 45.846475] irq event stamp: 2147602 [ 45.850608] hardirqs last enabled at (2147612): [<ffffffff811b8465>] console_unlock+0x435/0x570 [ 45.860603] hardirqs last disabled at (2147621): [<ffffffff811b845b>] console_unlock+0x42b/0x570 [ 45.870589] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 [ 45.880293] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 [ 45.891163] ---[ end trace ed0f88fd959a5a3f ]--- [ 45.896473] ------------[ cut here ]------------ [ 45.901762] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 [ 45.911956] Modules linked in: [ 45.915503] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 [ 45.926570] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 [ 45.935302] Workqueue: events work_for_cpu_fn [ 45.940312] RIP: 0010:__iommu_dma_unmap+0x115/0x130 [ 45.945905] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 [ 45.967128] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 [ 45.973111] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 [ 45.981236] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 [ 45.989363] RBP: 00000000ff000000 R08: 00000000000ff000 R09: 00000000000ff1ff [ 45.997489] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 [ 46.005613] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 [ 46.013738] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 [ 46.022952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 46.029521] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 [ 46.037643] Call Trace: [ 46.040508] ? lockdep_hardirqs_on_prepare+0xad/0x180 [ 46.046305] iommu_dma_free+0x18/0x30 [ 46.050538] ioat_free_chan_resources+0x19e/0x300 [ 46.055941] ioat_dma_self_test+0x2a0/0x3d0 [ 46.060765] ioat_pci_probe+0x60e/0x903 [ 46.065196] local_pci_probe+0x42/0x80 [ 46.069526] work_for_cpu_fn+0x16/0x20 [ 46.073854] process_one_work+0x292/0x630 [ 46.078477] ? __schedule+0x344/0x970 [ 46.082717] worker_thread+0x227/0x3e0 [ 46.087035] ? process_one_work+0x630/0x630 [ 46.091849] kthread+0x136/0x150 [ 46.095594] ? kthread_use_mm+0x100/0x100 [ 46.100218] ret_from_fork+0x22/0x30 [ 46.104366] irq event stamp: 2147992 [ 46.108499] hardirqs last enabled at (2148002): [<ffffffff811b8465>] console_unlock+0x435/0x570 [ 46.120483] hardirqs last disabled at (2148011): [<ffffffff811b845b>] console_unlock+0x42b/0x570 [ 46.130472] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 [ 46.140171] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 [ 46.151034] ---[ end trace ed0f88fd959a5a40 ]--- [ 46.156339] ------------[ cut here ]------------ [ 46.161628] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 [ 46.171817] Modules linked in: [ 46.175365] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 [ 46.186430] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 [ 46.195163] Workqueue: events work_for_cpu_fn [ 46.200173] RIP: 0010:__iommu_dma_unmap+0x115/0x130 [ 46.205767] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 [ 46.226991] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 [ 46.232976] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 [ 46.241101] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 [ 46.249226] RBP: 00000000fec00000 R08: 00000000000fec00 R09: 00000000000fedff [ 46.257352] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 [ 46.265474] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 [ 46.273602] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 [ 46.282822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 46.289388] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 [ 46.297513] Call Trace: [ 46.300358] ? lockdep_hardirqs_on_prepare+0xad/0x180 [ 46.306152] iommu_dma_free+0x18/0x30 [ 46.310379] ioat_free_chan_resources+0x19e/0x300 [ 46.315776] ioat_dma_self_test+0x2a0/0x3d0 [ 46.320607] ioat_pci_probe+0x60e/0x903 [ 46.325044] local_pci_probe+0x42/0x80 [ 46.329373] work_for_cpu_fn+0x16/0x20 [ 46.333699] process_one_work+0x292/0x630 [ 46.338321] ? __schedule+0x344/0x970 [ 46.342563] worker_thread+0x227/0x3e0 [ 46.346891] ? process_one_work+0x630/0x630 [ 46.351704] kthread+0x136/0x150 [ 46.355449] ? kthread_use_mm+0x100/0x100 [ 46.360069] ret_from_fork+0x22/0x30 [ 46.364204] irq event stamp: 2148380 [ 46.368338] hardirqs last enabled at (2148390): [<ffffffff811b8465>] console_unlock+0x435/0x570 [ 46.378336] hardirqs last disabled at (2148399): [<ffffffff811b845b>] console_unlock+0x42b/0x570 [ 46.388335] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 [ 46.398040] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 [ 46.408914] ---[ end trace ed0f88fd959a5a41 ]---
Hi Logan, On 2020/9/19 4:47, Logan Gunthorpe wrote: > Hi Lu, > > On 2020-09-11 9:21 p.m., Lu Baolu wrote: >> Tom Murphy has almost done all the work. His latest patch series was >> posted here. >> >> https://lore.kernel.org/linux-iommu/20200903201839.7327-1-murphyt7@tcd.ie/ >> >> Thanks a lot! >> >> This series is a follow-up with below changes: >> >> 1. Add a quirk for the i915 driver issue described in Tom's cover >> letter. >> 2. Fix several bugs in patch "iommu: Allow the dma-iommu api to use >> bounce buffers" to make the bounce buffer work for untrusted devices. >> 3. Several cleanups in iommu/vt-d driver after the conversion. >> > > I'm trying to test this on an old Sandy Bridge, but found that I get > spammed with warnings on boot. I've put a sample of a few of them below. > They all seem to be related to ioat. > > I had the same issue with Tom's v2 but never saw this on his v1. Have you verified whether this could be reproduced with the lasted upstream kernel (without this patch series)? Best regards, baolu
On 2020-09-20 12:36 a.m., Lu Baolu wrote: > Hi Logan, > > On 2020/9/19 4:47, Logan Gunthorpe wrote: >> Hi Lu, >> >> On 2020-09-11 9:21 p.m., Lu Baolu wrote: >>> Tom Murphy has almost done all the work. His latest patch series was >>> posted here. >>> >>> https://lore.kernel.org/linux-iommu/20200903201839.7327-1-murphyt7@tcd.ie/ >>> >>> Thanks a lot! >>> >>> This series is a follow-up with below changes: >>> >>> 1. Add a quirk for the i915 driver issue described in Tom's cover >>> letter. >>> 2. Fix several bugs in patch "iommu: Allow the dma-iommu api to use >>> bounce buffers" to make the bounce buffer work for untrusted devices. >>> 3. Several cleanups in iommu/vt-d driver after the conversion. >>> >> >> I'm trying to test this on an old Sandy Bridge, but found that I get >> spammed with warnings on boot. I've put a sample of a few of them below. >> They all seem to be related to ioat. >> >> I had the same issue with Tom's v2 but never saw this on his v1. > > Have you verified whether this could be reproduced with the lasted > upstream kernel (without this patch series)? Yes. Also, it's hitting a warning in the dma-iommu code which would not be hit without this patch set. Thanks, Logan
Hi Logan, On 9/21/20 11:48 PM, Logan Gunthorpe wrote: > > > On 2020-09-20 12:36 a.m., Lu Baolu wrote: >> Hi Logan, >> >> On 2020/9/19 4:47, Logan Gunthorpe wrote: >>> Hi Lu, >>> >>> On 2020-09-11 9:21 p.m., Lu Baolu wrote: >>>> Tom Murphy has almost done all the work. His latest patch series was >>>> posted here. >>>> >>>> https://lore.kernel.org/linux-iommu/20200903201839.7327-1-murphyt7@tcd.ie/ >>>> >>>> Thanks a lot! >>>> >>>> This series is a follow-up with below changes: >>>> >>>> 1. Add a quirk for the i915 driver issue described in Tom's cover >>>> letter. >>>> 2. Fix several bugs in patch "iommu: Allow the dma-iommu api to use >>>> bounce buffers" to make the bounce buffer work for untrusted devices. >>>> 3. Several cleanups in iommu/vt-d driver after the conversion. >>>> >>> >>> I'm trying to test this on an old Sandy Bridge, but found that I get >>> spammed with warnings on boot. I've put a sample of a few of them below. >>> They all seem to be related to ioat. >>> >>> I had the same issue with Tom's v2 but never saw this on his v1. >> >> Have you verified whether this could be reproduced with the lasted >> upstream kernel (without this patch series)? > > Yes. I am sorry. Just want to make sure I understand you correctly. :-) When you say "yes", do you mean you could reproduce this with pure upstream kernel (5.9-rc6)? > Also, it's hitting a warning in the dma-iommu code which would not > be hit without this patch set. Without this series, DMA APIs don't go through dma-iommu. Do you mind posting the warning message? Best regards, baolu
On 2020-09-18 21:47, Logan Gunthorpe wrote: > Hi Lu, > > On 2020-09-11 9:21 p.m., Lu Baolu wrote: >> Tom Murphy has almost done all the work. His latest patch series was >> posted here. >> >> https://lore.kernel.org/linux-iommu/20200903201839.7327-1-murphyt7@tcd.ie/ >> >> Thanks a lot! >> >> This series is a follow-up with below changes: >> >> 1. Add a quirk for the i915 driver issue described in Tom's cover >> letter. >> 2. Fix several bugs in patch "iommu: Allow the dma-iommu api to use >> bounce buffers" to make the bounce buffer work for untrusted devices. >> 3. Several cleanups in iommu/vt-d driver after the conversion. >> > > I'm trying to test this on an old Sandy Bridge, but found that I get > spammed with warnings on boot. I've put a sample of a few of them below. > They all seem to be related to ioat. > > I had the same issue with Tom's v2 but never saw this on his v1. I think this might have more to do with ioat being totally broken - AFAICS it appears to allocate descriptors with a size of 2MB, but free them with a size of 512KB. Try throwing CONFIG_DMA_API_DEBUG at it to confirm. Robin. > > Thanks, > > Logan > > > [ 44.057877] ------------[ cut here ]------------ > [ 44.063167] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 > [ 44.073351] Modules linked in: > [ 44.076882] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 > [ 44.087935] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 > [ 44.096650] Workqueue: events work_for_cpu_fn > [ 44.101644] RIP: 0010:__iommu_dma_unmap+0x115/0x130 > [ 44.107225] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 > [ 44.128443] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 > [ 44.134413] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 > [ 44.144487] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 > [ 44.152613] RBP: 00000000fec00000 R08: 00000000000fec00 R09: 00000000000fedff > [ 44.160733] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c39000 > [ 44.168861] R13: ffff888472d85ee8 R14: 0000000000080000 R15: 0000000000000000 > [ 44.176980] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 > [ 44.186198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 44.192761] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 > [ 44.200871] Call Trace: > [ 44.203716] ? lockdep_hardirqs_on_prepare+0xad/0x180 > [ 44.209509] iommu_dma_free+0x18/0x30 > [ 44.213734] ioat_free_chan_resources+0x19e/0x300 > [ 44.219133] ioat_dma_self_test+0x2a0/0x3d0 > [ 44.223964] ioat_pci_probe+0x60e/0x903 > [ 44.228387] local_pci_probe+0x42/0x80 > [ 44.232721] work_for_cpu_fn+0x16/0x20 > [ 44.237037] process_one_work+0x292/0x630 > [ 44.241644] ? __schedule+0x344/0x970 > [ 44.245868] worker_thread+0x227/0x3e0 > [ 44.250185] ? process_one_work+0x630/0x630 > [ 44.254989] kthread+0x136/0x150 > [ 44.258718] ? kthread_use_mm+0x100/0x100 > [ 44.263334] ret_from_fork+0x22/0x30 > [ 44.267474] irq event stamp: 1881262 > [ 44.271602] hardirqs last enabled at (1881272): [<ffffffff811b8465>] console_unlock+0x435/0x570 > [ 44.281593] hardirqs last disabled at (1881281): [<ffffffff811b845b>] console_unlock+0x42b/0x570 > [ 44.291588] softirqs last enabled at (1747140): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 > [ 44.301285] softirqs last disabled at (1747144): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 > [ 44.312153] ---[ end trace ed0f88fd959a5a39 ]--- > [ 44.353963] ------------[ cut here ]------------ > [ 44.359291] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 > [ 44.369482] Modules linked in: > [ 44.373030] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 > [ 44.384097] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 > [ 44.392825] Workqueue: events work_for_cpu_fn > [ 44.397831] RIP: 0010:__iommu_dma_unmap+0x115/0x130 > [ 44.403421] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 > [ 44.424644] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 > [ 44.430627] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 > [ 44.438770] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 > [ 44.446885] RBP: 00000000ffc00000 R08: 00000000000ffc00 R09: 00000000000ffdff > [ 44.455000] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 > [ 44.463119] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 > [ 44.471235] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 > [ 44.480440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 44.487004] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 > [ 44.495128] Call Trace: > [ 44.498008] iommu_dma_free+0x18/0x30 > [ 44.502230] ioat_free_chan_resources+0x19e/0x300 > [ 44.507631] ioat_dma_self_test+0x2a0/0x3d0 > [ 44.512466] ioat_pci_probe+0x60e/0x903 > [ 44.516889] local_pci_probe+0x42/0x80 > [ 44.521217] work_for_cpu_fn+0x16/0x20 > [ 44.525540] process_one_work+0x292/0x630 > [ 44.530157] ? __schedule+0x344/0x970 > [ 44.534393] worker_thread+0x227/0x3e0 > [ 44.538720] ? process_one_work+0x630/0x630 > [ 44.543531] kthread+0x136/0x150 > [ 44.547270] ? kthread_use_mm+0x100/0x100 > [ 44.551889] ret_from_fork+0x22/0x30 > [ 44.556034] irq event stamp: 2145654 > [ 44.560161] hardirqs last enabled at (2145664): [<ffffffff811b8465>] console_unlock+0x435/0x570 > [ 44.570158] hardirqs last disabled at (2145673): [<ffffffff811b845b>] console_unlock+0x42b/0x570 > [ 44.580153] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 > [ 44.589854] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 > [ 44.600729] ---[ end trace ed0f88fd959a5a3a ]--- > [ 44.606043] ------------[ cut here ]------------ > [ 44.611331] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 > [ 44.621524] Modules linked in: > [ 44.625050] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 > [ 44.636119] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 > [ 44.644834] Workqueue: events work_for_cpu_fn > [ 44.649827] RIP: 0010:__iommu_dma_unmap+0x115/0x130 > [ 44.655410] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 > [ 44.676618] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 > [ 44.682588] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 > [ 44.690702] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 > [ 44.698816] RBP: 00000000ffa00000 R08: 00000000000ffa00 R09: 00000000000ffbff > [ 44.706934] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 > [ 44.715054] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 > [ 44.723174] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 > [ 44.732384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 44.738941] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 > [ 44.747060] Call Trace: > [ 44.749921] ? lockdep_hardirqs_on_prepare+0xad/0x180 > [ 44.755706] iommu_dma_free+0x18/0x30 > [ 44.759931] ioat_free_chan_resources+0x19e/0x300 > [ 44.765332] ioat_dma_self_test+0x2a0/0x3d0 > [ 44.770156] ioat_pci_probe+0x60e/0x903 > [ 44.774583] local_pci_probe+0x42/0x80 > [ 44.778909] work_for_cpu_fn+0x16/0x20 > [ 44.783232] process_one_work+0x292/0x630 > [ 44.787846] ? __schedule+0x344/0x970 > [ 44.792079] worker_thread+0x227/0x3e0 > [ 44.796403] ? process_one_work+0x630/0x630 > [ 44.801205] kthread+0x136/0x150 > [ 44.804946] ? kthread_use_mm+0x100/0x100 > [ 44.809563] ret_from_fork+0x22/0x30 > [ 44.813704] irq event stamp: 2146044 > [ 44.817831] hardirqs last enabled at (2146054): [<ffffffff811b8465>] console_unlock+0x435/0x570 > [ 44.827821] hardirqs last disabled at (2146063): [<ffffffff811b845b>] console_unlock+0x42b/0x570 > [ 44.837810] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 > [ 44.847503] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 > [ 44.858366] ---[ end trace ed0f88fd959a5a3b ]--- > [ 44.863675] ------------[ cut here ]------------ > [ 44.868969] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 > [ 44.879148] Modules linked in: > [ 44.882689] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 > [ 44.893760] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 > [ 44.902487] Workqueue: events work_for_cpu_fn > [ 44.907493] RIP: 0010:__iommu_dma_unmap+0x115/0x130 > [ 44.913080] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 > [ 44.934290] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 > [ 44.940254] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 > [ 44.948379] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 > [ 44.956505] RBP: 00000000ff800000 R08: 00000000000ff800 R09: 00000000000ff9ff > [ 44.964627] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 > [ 44.972748] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 > [ 44.980871] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 > [ 44.990084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 44.996644] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 > [ 45.004759] Call Trace: > [ 45.007600] ? lockdep_hardirqs_on_prepare+0xad/0x180 > [ 45.013391] iommu_dma_free+0x18/0x30 > [ 45.017609] ioat_free_chan_resources+0x19e/0x300 > [ 45.022992] ioat_dma_self_test+0x2a0/0x3d0 > [ 45.027809] ioat_pci_probe+0x60e/0x903 > [ 45.032237] local_pci_probe+0x42/0x80 > [ 45.036563] work_for_cpu_fn+0x16/0x20 > [ 45.040886] process_one_work+0x292/0x630 > [ 45.045500] ? __schedule+0x344/0x970 > [ 45.049735] worker_thread+0x227/0x3e0 > [ 45.054061] ? process_one_work+0x630/0x630 > [ 45.058864] kthread+0x136/0x150 > [ 45.062597] ? kthread_use_mm+0x100/0x100 > [ 45.067200] ret_from_fork+0x22/0x30 > [ 45.071335] irq event stamp: 2146432 > [ 45.075458] hardirqs last enabled at (2146442): [<ffffffff811b8465>] console_unlock+0x435/0x570 > [ 45.085440] hardirqs last disabled at (2146451): [<ffffffff811b845b>] console_unlock+0x42b/0x570 > [ 45.095432] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 > [ 45.105134] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 > [ 45.116004] ---[ end trace ed0f88fd959a5a3c ]--- > [ 45.121305] ------------[ cut here ]------------ > [ 45.126596] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 > [ 45.136772] Modules linked in: > [ 45.142288] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 > [ 45.153351] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 > [ 45.162077] Workqueue: events work_for_cpu_fn > [ 45.167074] RIP: 0010:__iommu_dma_unmap+0x115/0x130 > [ 45.172663] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 > [ 45.193874] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 > [ 45.199850] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 > [ 45.207970] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 > [ 45.216095] RBP: 00000000ff600000 R08: 00000000000ff600 R09: 00000000000ff7ff > [ 45.224207] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 > [ 45.232327] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 > [ 45.240446] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 > [ 45.249662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 45.256225] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 > [ 45.264344] Call Trace: > [ 45.267205] ? lockdep_hardirqs_on_prepare+0xad/0x180 > [ 45.273003] iommu_dma_free+0x18/0x30 > [ 45.277231] ioat_free_chan_resources+0x19e/0x300 > [ 45.282631] ioat_dma_self_test+0x2a0/0x3d0 > [ 45.287457] ioat_pci_probe+0x60e/0x903 > [ 45.291894] local_pci_probe+0x42/0x80 > [ 45.296217] work_for_cpu_fn+0x16/0x20 > [ 45.300539] process_one_work+0x292/0x630 > [ 45.305162] ? __schedule+0x344/0x970 > [ 45.309399] worker_thread+0x227/0x3e0 > [ 45.313724] ? process_one_work+0x630/0x630 > [ 45.318535] kthread+0x136/0x150 > [ 45.322274] ? kthread_use_mm+0x100/0x100 > [ 45.326893] ret_from_fork+0x22/0x30 > [ 45.331034] irq event stamp: 2146822 > [ 45.335161] hardirqs last enabled at (2146832): [<ffffffff811b8465>] console_unlock+0x435/0x570 > [ 45.345154] hardirqs last disabled at (2146841): [<ffffffff811b845b>] console_unlock+0x42b/0x570 > [ 45.355150] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 > [ 45.364850] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 > [ 45.375710] ---[ end trace ed0f88fd959a5a3d ]--- > [ 45.381018] ------------[ cut here ]------------ > [ 45.386308] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 > [ 45.396494] Modules linked in: > [ 45.400020] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 > [ 45.411086] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 > [ 45.419810] Workqueue: events work_for_cpu_fn > [ 45.424819] RIP: 0010:__iommu_dma_unmap+0x115/0x130 > [ 45.430406] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 > [ 45.451630] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 > [ 45.457610] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 > [ 45.465730] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 > [ 45.473853] RBP: 00000000ff400000 R08: 00000000000ff400 R09: 00000000000ff5ff > [ 45.481973] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 > [ 45.490092] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 > [ 45.498211] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 > [ 45.507418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 45.513969] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 > [ 45.522081] Call Trace: > [ 45.524934] ? lockdep_hardirqs_on_prepare+0xad/0x180 > [ 45.530718] iommu_dma_free+0x18/0x30 > [ 45.534989] ioat_free_chan_resources+0x19e/0x300 > [ 45.540378] ioat_dma_self_test+0x2a0/0x3d0 > [ 45.545193] ioat_pci_probe+0x60e/0x903 > [ 45.549619] local_pci_probe+0x42/0x80 > [ 45.553940] work_for_cpu_fn+0x16/0x20 > [ 45.558258] process_one_work+0x292/0x630 > [ 45.562871] ? __schedule+0x344/0x970 > [ 45.567108] worker_thread+0x227/0x3e0 > [ 45.571434] ? process_one_work+0x630/0x630 > [ 45.576243] kthread+0x136/0x150 > [ 45.579981] ? kthread_use_mm+0x100/0x100 > [ 45.584598] ret_from_fork+0x22/0x30 > [ 45.588733] irq event stamp: 2147210 > [ 45.592860] hardirqs last enabled at (2147220): [<ffffffff811b8465>] console_unlock+0x435/0x570 > [ 45.602845] hardirqs last disabled at (2147229): [<ffffffff811b845b>] console_unlock+0x42b/0x570 > [ 45.612836] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 > [ 45.622531] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 > [ 45.633392] ---[ end trace ed0f88fd959a5a3e ]--- > [ 45.638708] ------------[ cut here ]------------ > [ 45.644003] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 > [ 45.654191] Modules linked in: > [ 45.657734] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 > [ 45.668807] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 > [ 45.677534] Workqueue: events work_for_cpu_fn > [ 45.682538] RIP: 0010:__iommu_dma_unmap+0x115/0x130 > [ 45.688126] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 > [ 45.709343] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 > [ 45.715321] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 > [ 45.723440] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 > [ 45.731555] RBP: 00000000ff200000 R08: 00000000000ff200 R09: 00000000000ff3ff > [ 45.739665] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 > [ 45.747777] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 > [ 45.755897] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 > [ 45.765106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 45.771661] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 > [ 45.779778] Call Trace: > [ 45.782638] ? lockdep_hardirqs_on_prepare+0xad/0x180 > [ 45.788439] iommu_dma_free+0x18/0x30 > [ 45.792664] ioat_free_chan_resources+0x19e/0x300 > [ 45.798060] ioat_dma_self_test+0x2a0/0x3d0 > [ 45.802886] ioat_pci_probe+0x60e/0x903 > [ 45.807306] local_pci_probe+0x42/0x80 > [ 45.811633] work_for_cpu_fn+0x16/0x20 > [ 45.815946] process_one_work+0x292/0x630 > [ 45.820556] ? __schedule+0x344/0x970 > [ 45.824791] worker_thread+0x227/0x3e0 > [ 45.829117] ? process_one_work+0x630/0x630 > [ 45.833969] kthread+0x136/0x150 > [ 45.837706] ? kthread_use_mm+0x100/0x100 > [ 45.842325] ret_from_fork+0x22/0x30 > [ 45.846475] irq event stamp: 2147602 > [ 45.850608] hardirqs last enabled at (2147612): [<ffffffff811b8465>] console_unlock+0x435/0x570 > [ 45.860603] hardirqs last disabled at (2147621): [<ffffffff811b845b>] console_unlock+0x42b/0x570 > [ 45.870589] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 > [ 45.880293] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 > [ 45.891163] ---[ end trace ed0f88fd959a5a3f ]--- > [ 45.896473] ------------[ cut here ]------------ > [ 45.901762] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 > [ 45.911956] Modules linked in: > [ 45.915503] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 > [ 45.926570] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 > [ 45.935302] Workqueue: events work_for_cpu_fn > [ 45.940312] RIP: 0010:__iommu_dma_unmap+0x115/0x130 > [ 45.945905] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 > [ 45.967128] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 > [ 45.973111] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 > [ 45.981236] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 > [ 45.989363] RBP: 00000000ff000000 R08: 00000000000ff000 R09: 00000000000ff1ff > [ 45.997489] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 > [ 46.005613] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 > [ 46.013738] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 > [ 46.022952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 46.029521] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 > [ 46.037643] Call Trace: > [ 46.040508] ? lockdep_hardirqs_on_prepare+0xad/0x180 > [ 46.046305] iommu_dma_free+0x18/0x30 > [ 46.050538] ioat_free_chan_resources+0x19e/0x300 > [ 46.055941] ioat_dma_self_test+0x2a0/0x3d0 > [ 46.060765] ioat_pci_probe+0x60e/0x903 > [ 46.065196] local_pci_probe+0x42/0x80 > [ 46.069526] work_for_cpu_fn+0x16/0x20 > [ 46.073854] process_one_work+0x292/0x630 > [ 46.078477] ? __schedule+0x344/0x970 > [ 46.082717] worker_thread+0x227/0x3e0 > [ 46.087035] ? process_one_work+0x630/0x630 > [ 46.091849] kthread+0x136/0x150 > [ 46.095594] ? kthread_use_mm+0x100/0x100 > [ 46.100218] ret_from_fork+0x22/0x30 > [ 46.104366] irq event stamp: 2147992 > [ 46.108499] hardirqs last enabled at (2148002): [<ffffffff811b8465>] console_unlock+0x435/0x570 > [ 46.120483] hardirqs last disabled at (2148011): [<ffffffff811b845b>] console_unlock+0x42b/0x570 > [ 46.130472] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 > [ 46.140171] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 > [ 46.151034] ---[ end trace ed0f88fd959a5a40 ]--- > [ 46.156339] ------------[ cut here ]------------ > [ 46.161628] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 > [ 46.171817] Modules linked in: > [ 46.175365] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 > [ 46.186430] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 > [ 46.195163] Workqueue: events work_for_cpu_fn > [ 46.200173] RIP: 0010:__iommu_dma_unmap+0x115/0x130 > [ 46.205767] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 > [ 46.226991] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 > [ 46.232976] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 > [ 46.241101] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 > [ 46.249226] RBP: 00000000fec00000 R08: 00000000000fec00 R09: 00000000000fedff > [ 46.257352] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c55000 > [ 46.265474] R13: ffff888472dc7268 R14: 0000000000080000 R15: 0000000000000000 > [ 46.273602] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 > [ 46.282822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 46.289388] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 > [ 46.297513] Call Trace: > [ 46.300358] ? lockdep_hardirqs_on_prepare+0xad/0x180 > [ 46.306152] iommu_dma_free+0x18/0x30 > [ 46.310379] ioat_free_chan_resources+0x19e/0x300 > [ 46.315776] ioat_dma_self_test+0x2a0/0x3d0 > [ 46.320607] ioat_pci_probe+0x60e/0x903 > [ 46.325044] local_pci_probe+0x42/0x80 > [ 46.329373] work_for_cpu_fn+0x16/0x20 > [ 46.333699] process_one_work+0x292/0x630 > [ 46.338321] ? __schedule+0x344/0x970 > [ 46.342563] worker_thread+0x227/0x3e0 > [ 46.346891] ? process_one_work+0x630/0x630 > [ 46.351704] kthread+0x136/0x150 > [ 46.355449] ? kthread_use_mm+0x100/0x100 > [ 46.360069] ret_from_fork+0x22/0x30 > [ 46.364204] irq event stamp: 2148380 > [ 46.368338] hardirqs last enabled at (2148390): [<ffffffff811b8465>] console_unlock+0x435/0x570 > [ 46.378336] hardirqs last disabled at (2148399): [<ffffffff811b845b>] console_unlock+0x42b/0x570 > [ 46.388335] softirqs last enabled at (2014254): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 > [ 46.398040] softirqs last disabled at (2014258): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 > [ 46.408914] ---[ end trace ed0f88fd959a5a41 ]--- > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu >
On 2020-09-15 09:31, Tvrtko Ursulin wrote: > > On 15/09/2020 02:47, Lu Baolu wrote: >> Hi Tvrtko, >> >> On 9/14/20 4:04 PM, Tvrtko Ursulin wrote: >>> >>> Hi, >>> >>> On 12/09/2020 04:21, Lu Baolu wrote: >>>> Tom Murphy has almost done all the work. His latest patch series was >>>> posted here. >>>> >>>> https://lore.kernel.org/linux-iommu/20200903201839.7327-1-murphyt7@tcd.ie/ >>>> >>>> >>>> Thanks a lot! >>>> >>>> This series is a follow-up with below changes: >>>> >>>> 1. Add a quirk for the i915 driver issue described in Tom's cover >>>> letter. >>> >>> Last week I have copied you on an i915 series which appears to remove >>> the need for this quirk. so if we get those i915 patches reviewed and >>> merged, do you still want to pursue this quirk? >> >> It's up to the graphic guys. I don't know the details in i915 driver. >> I don't think my tests could cover all cases. > > I am the graphic guy. :) I just need some reviews (internally) for my > series and then we can merge it, at which point you don't need the quirk > patch any more. I'll try to accelerate this. > > With regards to testing, you could send your series with my patches on > top to our trybot mailing list (intel-gfx-trybot@lists.freedesktop.org / > https://patchwork.freedesktop.org/project/intel-gfx-trybot/series/?ordering=-last_updated) > which would show you if it is still hitting the DMAR issues in i915. > >>> >>>> 2. Fix several bugs in patch "iommu: Allow the dma-iommu api to use >>>> bounce buffers" to make the bounce buffer work for untrusted devices. >>>> 3. Several cleanups in iommu/vt-d driver after the conversion. >>> >>> With the previous version of the series I hit a problem on Ivybridge >>> where apparently the dma engine width is not respected. At least that >>> is my layman interpretation of the errors. From the older thread: >>> >>> <3> [209.526605] DMAR: intel_iommu_map: iommu width (39) is not >>> sufficient for the mapped address (ffff008000) >>> >>> Relevant iommu boot related messages are: >>> >>> <6>[ 0.184234] DMAR: Host address width 36 >>> <6>[ 0.184245] DMAR: DRHD base: 0x000000fed90000 flags: 0x0 >>> <6>[ 0.184288] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap >>> c0000020e60262 ecap f0101a >>> <6>[ 0.184308] DMAR: DRHD base: 0x000000fed91000 flags: 0x1 >>> <6>[ 0.184337] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap >>> c9008020660262 ecap f0105a >>> <6>[ 0.184357] DMAR: RMRR base: 0x000000d8d28000 end: >>> 0x000000d8d46fff >>> <6>[ 0.184377] DMAR: RMRR base: 0x000000db000000 end: >>> 0x000000df1fffff >>> <6>[ 0.184398] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 >>> IOMMU 1 >>> <6>[ 0.184414] DMAR-IR: HPET id 0 under DRHD base 0xfed91000 >>> <6>[ 0.184428] DMAR-IR: Queued invalidation will be enabled to >>> support x2apic and Intr-remapping. >>> <6>[ 0.185173] DMAR-IR: Enabled IRQ remapping in x2apic mode >>> >>> <6>[ 0.878934] DMAR: No ATSR found >>> <6>[ 0.878966] DMAR: dmar0: Using Queued invalidation >>> <6>[ 0.879007] DMAR: dmar1: Using Queued invalidation >>> >>> <6>[ 0.915032] DMAR: Intel(R) Virtualization Technology for >>> Directed I/O >>> <6>[ 0.915060] PCI-DMA: Using software bounce buffering for IO >>> (SWIOTLB) >>> <6>[ 0.915084] software IO TLB: mapped [mem 0xc80d4000-0xcc0d4000] >>> (64MB) >>> >>> (Full boot log at >>> https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/boot0.txt, >>> failures at >>> https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/igt@i915_selftest@live@blt.html.) >>> >>> >>> Does this look familiar or at least plausible to you? Is this >>> something your new series has fixed? >> >> This happens during attaching a domain to device. It has nothing to do >> with this patch series. I will look into this issue, but not in this >> email thread context. > > I am not sure what step is attaching domain to device, but these type > messages: > > <3> [209.526605] DMAR: intel_iommu_map: iommu width (39) is not > >> sufficient for the mapped address (ffff008000) > > They definitely appear to happen at runtime, as i915 is getting > exercised by userspace. AFAICS this certainly might be related to this series - iommu-dma will constrain IOVA allocation based on the domain geometry that the driver reports, which in this case is set only once when first allocating the domain. Thus it looks like both the dmar_domain->gaw adjustment in prepare_domain_attach_device() and the domain_use_first_level() business in intel_alloc_iova() effectively get lost in this conversion, since the domain geometry never gets updated to reflect those additional constraints. Robin. > > Regards, > > Tvrtko > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu
On 2020-09-21 6:24 p.m., Lu Baolu wrote: >>>> I'm trying to test this on an old Sandy Bridge, but found that I get >>>> spammed with warnings on boot. I've put a sample of a few of them below. >>>> They all seem to be related to ioat. >>>> >>>> I had the same issue with Tom's v2 but never saw this on his v1. >>> >>> Have you verified whether this could be reproduced with the lasted >>> upstream kernel (without this patch series)? >> >> Yes. > > I am sorry. Just want to make sure I understand you correctly. :-) When > you say "yes", do you mean you could reproduce this with pure upstream > kernel (5.9-rc6)? I mean I've verified the bug does *not* exist without this patch set. > >> Also, it's hitting a warning in the dma-iommu code which would not >> be hit without this patch set. > > Without this series, DMA APIs don't go through dma-iommu. Do you mind > posting the warning message? It was on my original email but here it is again: [ 44.057877] ------------[ cut here ]------------ [ 44.063167] WARNING: CPU: 4 PID: 84 at drivers/iommu/dma-iommu.c:496 __iommu_dma_unmap+0x115/0x130 [ 44.073351] Modules linked in: [ 44.076882] CPU: 4 PID: 84 Comm: kworker/4:1 Tainted: G W 5.9.0-rc4-00006-g110da1e703a2 #216 [ 44.087935] Hardware name: Supermicro SYS-7047GR-TRF/X9DRG-QF, BIOS 3.0a 12/05/2013 [ 44.096650] Workqueue: events work_for_cpu_fn [ 44.101644] RIP: 0010:__iommu_dma_unmap+0x115/0x130 [ 44.107225] Code: 48 8b 0c 24 48 c7 44 24 10 00 00 00 00 48 c7 44 24 18 00 00 00 00 48 c7 44 24 20 00 00 00 00 48 c7 44 24 08 ff ff ff ff eb 8d <0f> 0b e9 74 ff ff ff e8 1f 36 66 00 66 66 2e 0f 1f 84 00 00 00 00 [ 44.128443] RSP: 0000:ffffc90002397c38 EFLAGS: 00010206 [ 44.134413] RAX: 0000000000200000 RBX: 0000000000080000 RCX: 0000000000000000 [ 44.144487] RDX: 0000000000000403 RSI: ffffffff82a7fb20 RDI: ffff888477804f30 [ 44.152613] RBP: 00000000fec00000 R08: 00000000000fec00 R09: 00000000000fedff [ 44.160733] R10: 0000000000000002 R11: 0000000000000004 R12: ffff888270c39000 [ 44.168861] R13: ffff888472d85ee8 R14: 0000000000080000 R15: 0000000000000000 [ 44.176980] FS: 0000000000000000(0000) GS:ffff888479800000(0000) knlGS:0000000000000000 [ 44.186198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 44.192761] CR2: 0000000000000000 CR3: 0000000002a20001 CR4: 00000000000606e0 [ 44.200871] Call Trace: [ 44.203716] ? lockdep_hardirqs_on_prepare+0xad/0x180 [ 44.209509] iommu_dma_free+0x18/0x30 [ 44.213734] ioat_free_chan_resources+0x19e/0x300 [ 44.219133] ioat_dma_self_test+0x2a0/0x3d0 [ 44.223964] ioat_pci_probe+0x60e/0x903 [ 44.228387] local_pci_probe+0x42/0x80 [ 44.232721] work_for_cpu_fn+0x16/0x20 [ 44.237037] process_one_work+0x292/0x630 [ 44.241644] ? __schedule+0x344/0x970 [ 44.245868] worker_thread+0x227/0x3e0 [ 44.250185] ? process_one_work+0x630/0x630 [ 44.254989] kthread+0x136/0x150 [ 44.258718] ? kthread_use_mm+0x100/0x100 [ 44.263334] ret_from_fork+0x22/0x30 [ 44.267474] irq event stamp: 1881262 [ 44.271602] hardirqs last enabled at (1881272): [<ffffffff811b8465>] console_unlock+0x435/0x570 [ 44.281593] hardirqs last disabled at (1881281): [<ffffffff811b845b>] console_unlock+0x42b/0x570 [ 44.291588] softirqs last enabled at (1747140): [<ffffffff817da3e5>] ioat_cleanup+0x65/0x470 [ 44.301285] softirqs last disabled at (1747144): [<ffffffff817d797a>] ioat_free_chan_resources+0x6a/0x300 [ 44.312153] ---[ end trace ed0f88fd959a5a39 ]---
On 2020-09-22 3:51 a.m., Robin Murphy wrote: > On 2020-09-18 21:47, Logan Gunthorpe wrote: >> Hi Lu, >> >> On 2020-09-11 9:21 p.m., Lu Baolu wrote: >>> Tom Murphy has almost done all the work. His latest patch series was >>> posted here. >>> >>> https://lore.kernel.org/linux-iommu/20200903201839.7327-1-murphyt7@tcd.ie/ >>> >>> Thanks a lot! >>> >>> This series is a follow-up with below changes: >>> >>> 1. Add a quirk for the i915 driver issue described in Tom's cover >>> letter. >>> 2. Fix several bugs in patch "iommu: Allow the dma-iommu api to use >>> bounce buffers" to make the bounce buffer work for untrusted devices. >>> 3. Several cleanups in iommu/vt-d driver after the conversion. >>> >> >> I'm trying to test this on an old Sandy Bridge, but found that I get >> spammed with warnings on boot. I've put a sample of a few of them below. >> They all seem to be related to ioat. >> >> I had the same issue with Tom's v2 but never saw this on his v1. > > I think this might have more to do with ioat being totally broken - > AFAICS it appears to allocate descriptors with a size of 2MB, but free > them with a size of 512KB. Try throwing CONFIG_DMA_API_DEBUG at it to > confirm. Ah, yes, nice catch. Looks like it was broken recently by the following commit, but nobody noticed and the dma-iommu patch set added a warning which caught it. a02254f8a676 ("dmaengine: ioat: Decreasing allocation chunk size 2M->512K") Reverting that fixes the issue. I'll try to send patch or two for this. Logan
On 9/22/20 7:05 PM, Robin Murphy wrote: >>>> With the previous version of the series I hit a problem on Ivybridge >>>> where apparently the dma engine width is not respected. At least >>>> that is my layman interpretation of the errors. From the older thread: >>>> >>>> <3> [209.526605] DMAR: intel_iommu_map: iommu width (39) is not >>>> sufficient for the mapped address (ffff008000) >>>> >>>> Relevant iommu boot related messages are: >>>> >>>> <6>[ 0.184234] DMAR: Host address width 36 >>>> <6>[ 0.184245] DMAR: DRHD base: 0x000000fed90000 flags: 0x0 >>>> <6>[ 0.184288] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap >>>> c0000020e60262 ecap f0101a >>>> <6>[ 0.184308] DMAR: DRHD base: 0x000000fed91000 flags: 0x1 >>>> <6>[ 0.184337] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap >>>> c9008020660262 ecap f0105a >>>> <6>[ 0.184357] DMAR: RMRR base: 0x000000d8d28000 end: >>>> 0x000000d8d46fff >>>> <6>[ 0.184377] DMAR: RMRR base: 0x000000db000000 end: >>>> 0x000000df1fffff >>>> <6>[ 0.184398] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 >>>> IOMMU 1 >>>> <6>[ 0.184414] DMAR-IR: HPET id 0 under DRHD base 0xfed91000 >>>> <6>[ 0.184428] DMAR-IR: Queued invalidation will be enabled to >>>> support x2apic and Intr-remapping. >>>> <6>[ 0.185173] DMAR-IR: Enabled IRQ remapping in x2apic mode >>>> >>>> <6>[ 0.878934] DMAR: No ATSR found >>>> <6>[ 0.878966] DMAR: dmar0: Using Queued invalidation >>>> <6>[ 0.879007] DMAR: dmar1: Using Queued invalidation >>>> >>>> <6>[ 0.915032] DMAR: Intel(R) Virtualization Technology for >>>> Directed I/O >>>> <6>[ 0.915060] PCI-DMA: Using software bounce buffering for IO >>>> (SWIOTLB) >>>> <6>[ 0.915084] software IO TLB: mapped [mem >>>> 0xc80d4000-0xcc0d4000] (64MB) >>>> >>>> (Full boot log at >>>> https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/boot0.txt, >>>> failures at >>>> https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/igt@i915_selftest@live@blt.html.) >>>> >>>> >>>> Does this look familiar or at least plausible to you? Is this >>>> something your new series has fixed? >>> >>> This happens during attaching a domain to device. It has nothing to do >>> with this patch series. I will look into this issue, but not in this >>> email thread context. >> >> I am not sure what step is attaching domain to device, but these type >> messages: >> >> <3> [209.526605] DMAR: intel_iommu_map: iommu width (39) is not >> >> sufficient for the mapped address (ffff008000) >> >> They definitely appear to happen at runtime, as i915 is getting >> exercised by userspace. > > AFAICS this certainly might be related to this series - iommu-dma will Oh! I looked at the wrong function. prepare_domain_attach_device() prints a similar message which made me believe that it was not caused by the this patches series. > constrain IOVA allocation based on the domain geometry that the driver > reports, which in this case is set only once when first allocating the > domain. Thus it looks like both the dmar_domain->gaw adjustment in > prepare_domain_attach_device() and the domain_use_first_level() business > in intel_alloc_iova() effectively get lost in this conversion, since the > domain geometry never gets updated to reflect those additional constraints. Sounds reasonable. I will look into the code and work out a fix. > > Robin. > Best regards, baolu
Hi Tvrtko, On 9/15/20 4:31 PM, Tvrtko Ursulin wrote: >>> With the previous version of the series I hit a problem on Ivybridge >>> where apparently the dma engine width is not respected. At least that >>> is my layman interpretation of the errors. From the older thread: >>> >>> <3> [209.526605] DMAR: intel_iommu_map: iommu width (39) is not >>> sufficient for the mapped address (ffff008000) >>> >>> Relevant iommu boot related messages are: >>> >>> <6>[ 0.184234] DMAR: Host address width 36 >>> <6>[ 0.184245] DMAR: DRHD base: 0x000000fed90000 flags: 0x0 >>> <6>[ 0.184288] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap >>> c0000020e60262 ecap f0101a >>> <6>[ 0.184308] DMAR: DRHD base: 0x000000fed91000 flags: 0x1 >>> <6>[ 0.184337] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap >>> c9008020660262 ecap f0105a >>> <6>[ 0.184357] DMAR: RMRR base: 0x000000d8d28000 end: >>> 0x000000d8d46fff >>> <6>[ 0.184377] DMAR: RMRR base: 0x000000db000000 end: >>> 0x000000df1fffff >>> <6>[ 0.184398] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 >>> IOMMU 1 >>> <6>[ 0.184414] DMAR-IR: HPET id 0 under DRHD base 0xfed91000 >>> <6>[ 0.184428] DMAR-IR: Queued invalidation will be enabled to >>> support x2apic and Intr-remapping. >>> <6>[ 0.185173] DMAR-IR: Enabled IRQ remapping in x2apic mode >>> >>> <6>[ 0.878934] DMAR: No ATSR found >>> <6>[ 0.878966] DMAR: dmar0: Using Queued invalidation >>> <6>[ 0.879007] DMAR: dmar1: Using Queued invalidation >>> >>> <6>[ 0.915032] DMAR: Intel(R) Virtualization Technology for >>> Directed I/O >>> <6>[ 0.915060] PCI-DMA: Using software bounce buffering for IO >>> (SWIOTLB) >>> <6>[ 0.915084] software IO TLB: mapped [mem 0xc80d4000-0xcc0d4000] >>> (64MB) >>> >>> (Full boot log at >>> https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/boot0.txt, >>> failures at >>> https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7054/fi-ivb-3770/igt@i915_selftest@live@blt.html.) >>> >>> >>> Does this look familiar or at least plausible to you? Is this >>> something your new series has fixed? >> >> This happens during attaching a domain to device. It has nothing to do >> with this patch series. I will look into this issue, but not in this >> email thread context. > > I am not sure what step is attaching domain to device, but these type > messages: > > <3> [209.526605] DMAR: intel_iommu_map: iommu width (39) is not > >> sufficient for the mapped address (ffff008000) > > They definitely appear to happen at runtime, as i915 is getting > exercised by userspace. Can you please check whether below change helps here? diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index c8323a9f8bde..0484c539debc 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -724,6 +724,7 @@ static int domain_update_device_node(struct dmar_domain *domain) /* Some capabilities may be different across iommus */ static void domain_update_iommu_cap(struct dmar_domain *domain) { + domain->geometry.aperture_end = __DOMAIN_MAX_ADDR(dmar_domain->gaw); domain_update_iommu_coherency(domain); domain->iommu_snooping = domain_update_iommu_snooping(NULL); domain->iommu_superpage = domain_update_iommu_superpage(domain, NULL); Best regards, baolu