| Message ID | 20220719174253.541965-1-olekstysh@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | PCI devices passthrough on Arm, part 3 | expand |
Hi Oleksandr, > On 19 Jul 2022, at 6:42 pm, Oleksandr Tyshchenko <olekstysh@gmail.com> wrote: > > From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> > > Hi, all! > > You can find previous discussion at [1]. > > 1. This patch series is focusing on vPCI and adds support for non-identity > PCI BAR mappings which is required while passing through a PCI device to > a guest. The highlights are: > > - Add relevant vpci register handlers when assigning PCI device to a domain > and remove those when de-assigning. This allows having different > handlers for different domains, e.g. hwdom and other guests. > > - Emulate guest BAR register values based on physical BAR values. > This allows creating a guest view of the registers and emulates > size and properties probe as it is done during PCI device enumeration by > the guest. > > - Instead of handling a single range set, that contains all the memory > regions of all the BARs and ROM, have them per BAR. > > - Take into account guest's BAR view and program its p2m accordingly: > gfn is guest's view of the BAR and mfn is the physical BAR value as set > up by the host bridge in the hardware domain. > This way hardware domain sees physical BAR values and guest sees > emulated ones. > > 2. The series also adds support for virtual PCI bus topology for guests: > - We emulate a single host bridge for the guest, so segment is always 0. > - The implementation is limited to 32 devices which are allowed on > a single PCI bus. > - The virtual bus number is set to 0, so virtual devices are seen > as embedded endpoints behind the root complex. > > 3. The series has been updated due to the new PCI(vPCI) locking scheme implemented > in the prereq series which is also on the review now [2]. > > 4. For unprivileged guests vpci_{read|write} has been re-worked > to not passthrough accesses to the registers not explicitly handled > by the corresponding vPCI handlers: without that passthrough > to guests is completely unsafe as Xen allows them full access to > the registers. During development this can be reverted for debugging purposes. > > !!! OT: please note, Oleksandr Andrushchenko who is the author of all this stuff > has managed to address allmost all review comments given for v6 and pushed the updated > version to the github (23.02.22). > So after receiving his agreement I just picked it up and did the following before > pushing V7: > - rebased on the recent staging (resolving a few conflicts) > - updated according to the recent changes (added cf_check specifiers where appropriate, etc) > and performed minor adjustments > - made sure that both current and prereq series [2] didn't really break x86 by testing > PVH Dom0 (vPCI) and PV Dom0 + HVM DomU (PCI passthrough to DomU) using Qemu > - my colleague Volodymyr Babchuk (who was involved in the prereq series) rechecked that > both series worked on Arm using real HW > > You can also find the series at [3]. > > [1] https://lore.kernel.org/xen-devel/20220204063459.680961-1-andr2000@gmail.com/ > [2] https://lore.kernel.org/xen-devel/20220718211521.664729-1-volodymyr_babchuk@epam.com/ > [3] https://github.com/otyshchenko1/xen/commits/vpci7 > I tested the whole series on ARM N1SDP board everything works as expected. So for the whole series: Tested-by: Rahul Singh <rahul.singh@arm.com> Regards, Rahul
On 26.07.22 16:47, Rahul Singh wrote: > Hi Oleksandr, Hello Rahul > >> On 19 Jul 2022, at 6:42 pm, Oleksandr Tyshchenko <olekstysh@gmail.com> wrote: >> >> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> >> >> Hi, all! >> >> You can find previous discussion at [1]. >> >> 1. This patch series is focusing on vPCI and adds support for non-identity >> PCI BAR mappings which is required while passing through a PCI device to >> a guest. The highlights are: >> >> - Add relevant vpci register handlers when assigning PCI device to a domain >> and remove those when de-assigning. This allows having different >> handlers for different domains, e.g. hwdom and other guests. >> >> - Emulate guest BAR register values based on physical BAR values. >> This allows creating a guest view of the registers and emulates >> size and properties probe as it is done during PCI device enumeration by >> the guest. >> >> - Instead of handling a single range set, that contains all the memory >> regions of all the BARs and ROM, have them per BAR. >> >> - Take into account guest's BAR view and program its p2m accordingly: >> gfn is guest's view of the BAR and mfn is the physical BAR value as set >> up by the host bridge in the hardware domain. >> This way hardware domain sees physical BAR values and guest sees >> emulated ones. >> >> 2. The series also adds support for virtual PCI bus topology for guests: >> - We emulate a single host bridge for the guest, so segment is always 0. >> - The implementation is limited to 32 devices which are allowed on >> a single PCI bus. >> - The virtual bus number is set to 0, so virtual devices are seen >> as embedded endpoints behind the root complex. >> >> 3. The series has been updated due to the new PCI(vPCI) locking scheme implemented >> in the prereq series which is also on the review now [2]. >> >> 4. For unprivileged guests vpci_{read|write} has been re-worked >> to not passthrough accesses to the registers not explicitly handled >> by the corresponding vPCI handlers: without that passthrough >> to guests is completely unsafe as Xen allows them full access to >> the registers. During development this can be reverted for debugging purposes. >> >> !!! OT: please note, Oleksandr Andrushchenko who is the author of all this stuff >> has managed to address allmost all review comments given for v6 and pushed the updated >> version to the github (23.02.22). >> So after receiving his agreement I just picked it up and did the following before >> pushing V7: >> - rebased on the recent staging (resolving a few conflicts) >> - updated according to the recent changes (added cf_check specifiers where appropriate, etc) >> and performed minor adjustments >> - made sure that both current and prereq series [2] didn't really break x86 by testing >> PVH Dom0 (vPCI) and PV Dom0 + HVM DomU (PCI passthrough to DomU) using Qemu >> - my colleague Volodymyr Babchuk (who was involved in the prereq series) rechecked that >> both series worked on Arm using real HW >> >> You can also find the series at [3]. >> >> [1] https://urldefense.com/v3/__https://lore.kernel.org/xen-devel/20220204063459.680961-1-andr2000@gmail.com/__;!!GF_29dbcQIUBPA!1P9LeytJC7d3tnSuQCjk7YqIqfZPpGlrc6ES1l1sUAPbfGbeYg2YM477xiUy0oTU9ys7qv9MHD6GNDWCeHHG_qsr-NY$ [lore[.]kernel[.]org] >> [2] https://urldefense.com/v3/__https://lore.kernel.org/xen-devel/20220718211521.664729-1-volodymyr_babchuk@epam.com/__;!!GF_29dbcQIUBPA!1P9LeytJC7d3tnSuQCjk7YqIqfZPpGlrc6ES1l1sUAPbfGbeYg2YM477xiUy0oTU9ys7qv9MHD6GNDWCeHHGbScTNb4$ [lore[.]kernel[.]org] >> [3] https://urldefense.com/v3/__https://github.com/otyshchenko1/xen/commits/vpci7__;!!GF_29dbcQIUBPA!1P9LeytJC7d3tnSuQCjk7YqIqfZPpGlrc6ES1l1sUAPbfGbeYg2YM477xiUy0oTU9ys7qv9MHD6GNDWCeHHGhpAmrcM$ [github[.]com] >> > I tested the whole series on ARM N1SDP board everything works as expected. Sounds great! > > So for the whole series: > Tested-by: Rahul Singh <rahul.singh@arm.com> Thank you for testing! > > Regards, > Rahul
From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> Hi, all! You can find previous discussion at [1]. 1. This patch series is focusing on vPCI and adds support for non-identity PCI BAR mappings which is required while passing through a PCI device to a guest. The highlights are: - Add relevant vpci register handlers when assigning PCI device to a domain and remove those when de-assigning. This allows having different handlers for different domains, e.g. hwdom and other guests. - Emulate guest BAR register values based on physical BAR values. This allows creating a guest view of the registers and emulates size and properties probe as it is done during PCI device enumeration by the guest. - Instead of handling a single range set, that contains all the memory regions of all the BARs and ROM, have them per BAR. - Take into account guest's BAR view and program its p2m accordingly: gfn is guest's view of the BAR and mfn is the physical BAR value as set up by the host bridge in the hardware domain. This way hardware domain sees physical BAR values and guest sees emulated ones. 2. The series also adds support for virtual PCI bus topology for guests: - We emulate a single host bridge for the guest, so segment is always 0. - The implementation is limited to 32 devices which are allowed on a single PCI bus. - The virtual bus number is set to 0, so virtual devices are seen as embedded endpoints behind the root complex. 3. The series has been updated due to the new PCI(vPCI) locking scheme implemented in the prereq series which is also on the review now [2]. 4. For unprivileged guests vpci_{read|write} has been re-worked to not passthrough accesses to the registers not explicitly handled by the corresponding vPCI handlers: without that passthrough to guests is completely unsafe as Xen allows them full access to the registers. During development this can be reverted for debugging purposes. !!! OT: please note, Oleksandr Andrushchenko who is the author of all this stuff has managed to address allmost all review comments given for v6 and pushed the updated version to the github (23.02.22). So after receiving his agreement I just picked it up and did the following before pushing V7: - rebased on the recent staging (resolving a few conflicts) - updated according to the recent changes (added cf_check specifiers where appropriate, etc) and performed minor adjustments - made sure that both current and prereq series [2] didn't really break x86 by testing PVH Dom0 (vPCI) and PV Dom0 + HVM DomU (PCI passthrough to DomU) using Qemu - my colleague Volodymyr Babchuk (who was involved in the prereq series) rechecked that both series worked on Arm using real HW You can also find the series at [3]. [1] https://lore.kernel.org/xen-devel/20220204063459.680961-1-andr2000@gmail.com/ [2] https://lore.kernel.org/xen-devel/20220718211521.664729-1-volodymyr_babchuk@epam.com/ [3] https://github.com/otyshchenko1/xen/commits/vpci7 Oleksandr Andrushchenko (11): xen/pci: arm: add stub for is_memory_hole vpci: add hooks for PCI device assign/de-assign vpci/header: implement guest BAR register handlers rangeset: add RANGESETF_no_print flag vpci/header: handle p2m range sets per BAR vpci/header: program p2m with guest BAR view vpci/header: emulate PCI_COMMAND register for guests vpci/header: reset the command register when adding devices vpci: add initial support for virtual PCI bus topology xen/arm: translate virtual PCI bus topology for guests xen/arm: account IO handlers for emulated PCI MSI-X xen/arch/arm/mm.c | 6 + xen/arch/arm/vpci.c | 31 ++- xen/common/rangeset.c | 5 +- xen/drivers/Kconfig | 4 + xen/drivers/passthrough/pci.c | 11 + xen/drivers/vpci/header.c | 458 ++++++++++++++++++++++++++-------- xen/drivers/vpci/msi.c | 4 + xen/drivers/vpci/msix.c | 4 + xen/drivers/vpci/vpci.c | 130 ++++++++++ xen/include/xen/rangeset.h | 5 +- xen/include/xen/sched.h | 8 + xen/include/xen/vpci.h | 42 +++- 12 files changed, 604 insertions(+), 104 deletions(-)