| Message ID | 20190923100931.29670-1-liuwe@microsoft.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Port Xen to Hyper-V | expand |
> -----Original Message----- > From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Wei Liu > Sent: 23 September 2019 11:09 > To: Xen Development List <xen-devel@lists.xenproject.org> > Cc: Wei Liu <liuwe@microsoft.com>; Wei Liu <wl@xen.org>; Andrew Cooper <Andrew.Cooper3@citrix.com>; > Michael Kelley <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne > <roger.pau@citrix.com> > Subject: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > Hi all > > In case you're wondering, I can already run a fully fledged Xen system on > Hyper-V with emulated disk and network. > > This is the very first stage for porting Xen to run on Hyper-V with all the > goodies Hyper-V has to offer. With this series, Xen can successfully detect > Hyper-V and prints out a message. I would like to first get the directory > structure and kconfig options agreed upon. > > There are two major areas to be worked on: > * Make Dom0 able to use Hyper-V's synthetic devices. > * Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS > and other interfaces. > > They aren't trivial, and time can be scarce on my side, so I intend to post > patches piece meal when they are ready. > > Questions and comments are welcome. Yay! Glad you found the time to do this :-) Making use of the synic, stimer, will be useful. To get the benefit for nested guests though, I assume we'll need to exposed enlightened VMCS, EPT flush, etc. through? Cheers, Paul > > Thanks, > Wei. > > Cc: Jan Beulich <jbeulich@suse.com> > Cc: Andrew Cooper <andrew.cooper3@citrix.com> > Cc: Wei Liu <wl@xen.org> > Cc: Roger Pau Monné <roger.pau@citrix.com> > > Wei Liu (8): > x86: introduce CONFIG_GUEST and move code > x86: include asm_defns.h directly in hypercall.h > x86: drop hypervisor_cpuid_base > x86: factor out xen variants for hypervisor setup code > x86: factor out hypervisor agnostic code > x86: make probe_xen return boolean value > x86: introduce CONFIG_HYPERV and hyperv directory > x86: be more verbose when running nested > > xen/arch/x86/Kconfig | 13 ++ > xen/arch/x86/Makefile | 2 +- > xen/arch/x86/guest/Makefile | 6 +- > xen/arch/x86/guest/hyperv/Makefile | 1 + > xen/arch/x86/guest/hyperv/hyperv.c | 63 ++++++++++ > xen/arch/x86/guest/hypervisor.c | 116 ++++++++++++++++++ > xen/arch/x86/guest/xen/Makefile | 4 + > xen/arch/x86/guest/{ => xen}/hypercall_page.S | 0 > xen/arch/x86/guest/{ => xen}/pvh-boot.c | 0 > xen/arch/x86/guest/{ => xen}/xen.c | 73 ++--------- > xen/arch/x86/pv/shim.c | 2 +- > xen/arch/x86/setup.c | 11 ++ > xen/include/asm-x86/guest.h | 2 + > xen/include/asm-x86/guest/hypercall.h | 2 + > xen/include/asm-x86/guest/hyperv.h | 48 ++++++++ > xen/include/asm-x86/guest/hypervisor.h | 58 +++++++++ > xen/include/asm-x86/guest/xen.h | 23 +--- > 17 files changed, 337 insertions(+), 87 deletions(-) > create mode 100644 xen/arch/x86/guest/hyperv/Makefile > create mode 100644 xen/arch/x86/guest/hyperv/hyperv.c > create mode 100644 xen/arch/x86/guest/hypervisor.c > create mode 100644 xen/arch/x86/guest/xen/Makefile > rename xen/arch/x86/guest/{ => xen}/hypercall_page.S (100%) > rename xen/arch/x86/guest/{ => xen}/pvh-boot.c (100%) > rename xen/arch/x86/guest/{ => xen}/xen.c (80%) > create mode 100644 xen/include/asm-x86/guest/hyperv.h > create mode 100644 xen/include/asm-x86/guest/hypervisor.h > > -- > 2.20.1 > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xenproject.org > https://lists.xenproject.org/mailman/listinfo/xen-devel
On Mon, Sep 23, 2019 at 10:48:45AM +0000, Paul Durrant wrote: > > -----Original Message----- > > From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Wei Liu > > Sent: 23 September 2019 11:09 > > To: Xen Development List <xen-devel@lists.xenproject.org> > > Cc: Wei Liu <liuwe@microsoft.com>; Wei Liu <wl@xen.org>; Andrew Cooper <Andrew.Cooper3@citrix.com>; > > Michael Kelley <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne > > <roger.pau@citrix.com> > > Subject: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > > > Hi all > > > > In case you're wondering, I can already run a fully fledged Xen system on > > Hyper-V with emulated disk and network. > > > > This is the very first stage for porting Xen to run on Hyper-V with all the > > goodies Hyper-V has to offer. With this series, Xen can successfully detect > > Hyper-V and prints out a message. I would like to first get the directory > > structure and kconfig options agreed upon. > > > > There are two major areas to be worked on: > > * Make Dom0 able to use Hyper-V's synthetic devices. > > * Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS > > and other interfaces. > > > > They aren't trivial, and time can be scarce on my side, so I intend to post > > patches piece meal when they are ready. > > > > Questions and comments are welcome. > > Yay! Glad you found the time to do this :-) Making use of the synic, > stimer, will be useful. To get the benefit for nested guests though, I > assume we'll need to exposed enlightened VMCS, EPT flush, etc. > through? Per TLFS, eVMCS should be used by L1 Xen. Not sure what you meant by EPT flush. If you mean things like HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen only as well. I'm not sure whether they should be expose to L2 guests. There are hypercalls for virtual TLB management. We may choose to expose them to L2 guests for better performance. Wei.
> -----Original Message----- > From: Wei Liu <wl@xen.org> > Sent: 23 September 2019 12:27 > To: Paul Durrant <Paul.Durrant@citrix.com> > Cc: 'Wei Liu' <wl@xen.org>; Xen Development List <xen-devel@lists.xenproject.org>; Wei Liu > <liuwe@microsoft.com>; Andrew Cooper <Andrew.Cooper3@citrix.com>; Michael Kelley > <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne <roger.pau@citrix.com> > Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > On Mon, Sep 23, 2019 at 10:48:45AM +0000, Paul Durrant wrote: > > > -----Original Message----- > > > From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Wei Liu > > > Sent: 23 September 2019 11:09 > > > To: Xen Development List <xen-devel@lists.xenproject.org> > > > Cc: Wei Liu <liuwe@microsoft.com>; Wei Liu <wl@xen.org>; Andrew Cooper > <Andrew.Cooper3@citrix.com>; > > > Michael Kelley <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne > > > <roger.pau@citrix.com> > > > Subject: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > > > > > Hi all > > > > > > In case you're wondering, I can already run a fully fledged Xen system on > > > Hyper-V with emulated disk and network. > > > > > > This is the very first stage for porting Xen to run on Hyper-V with all the > > > goodies Hyper-V has to offer. With this series, Xen can successfully detect > > > Hyper-V and prints out a message. I would like to first get the directory > > > structure and kconfig options agreed upon. > > > > > > There are two major areas to be worked on: > > > * Make Dom0 able to use Hyper-V's synthetic devices. > > > * Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS > > > and other interfaces. > > > > > > They aren't trivial, and time can be scarce on my side, so I intend to post > > > patches piece meal when they are ready. > > > > > > Questions and comments are welcome. > > > > Yay! Glad you found the time to do this :-) Making use of the synic, > > stimer, will be useful. To get the benefit for nested guests though, I > > assume we'll need to exposed enlightened VMCS, EPT flush, etc. > > through? > > Per TLFS, eVMCS should be used by L1 Xen. Yes, I guess it only needs to be used by L1, but Windows is using an increasing number of VMs for various purposes so I think making it stack well would be useful. > > Not sure what you meant by EPT flush. If you mean things like > HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen > only as well. I'm not sure whether they should be expose to L2 guests. > Yes, that's what I meant and, again, I was thinking about the nested Windows guest using VMs itself. > There are hypercalls for virtual TLB management. We may choose to expose > them to L2 guests for better performance. > Do you mean 'Direct Virtual Flush'? The hypercalls in question are already handled in viridian for the purposes of remote TLB flush, so I guess it would be a question of wiring them through. Paul > Wei.
On Mon, Sep 23, 2019 at 12:11:26PM +0000, Paul Durrant wrote: > > -----Original Message----- > > From: Wei Liu <wl@xen.org> > > Sent: 23 September 2019 12:27 > > To: Paul Durrant <Paul.Durrant@citrix.com> > > Cc: 'Wei Liu' <wl@xen.org>; Xen Development List <xen-devel@lists.xenproject.org>; Wei Liu > > <liuwe@microsoft.com>; Andrew Cooper <Andrew.Cooper3@citrix.com>; Michael Kelley > > <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne <roger.pau@citrix.com> > > Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > > > On Mon, Sep 23, 2019 at 10:48:45AM +0000, Paul Durrant wrote: > > > > -----Original Message----- > > > > From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Wei Liu > > > > Sent: 23 September 2019 11:09 > > > > To: Xen Development List <xen-devel@lists.xenproject.org> > > > > Cc: Wei Liu <liuwe@microsoft.com>; Wei Liu <wl@xen.org>; Andrew Cooper > > <Andrew.Cooper3@citrix.com>; > > > > Michael Kelley <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne > > > > <roger.pau@citrix.com> > > > > Subject: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > > > > > > > Hi all > > > > > > > > In case you're wondering, I can already run a fully fledged Xen system on > > > > Hyper-V with emulated disk and network. > > > > > > > > This is the very first stage for porting Xen to run on Hyper-V with all the > > > > goodies Hyper-V has to offer. With this series, Xen can successfully detect > > > > Hyper-V and prints out a message. I would like to first get the directory > > > > structure and kconfig options agreed upon. > > > > > > > > There are two major areas to be worked on: > > > > * Make Dom0 able to use Hyper-V's synthetic devices. > > > > * Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS > > > > and other interfaces. > > > > > > > > They aren't trivial, and time can be scarce on my side, so I intend to post > > > > patches piece meal when they are ready. > > > > > > > > Questions and comments are welcome. > > > > > > Yay! Glad you found the time to do this :-) Making use of the synic, > > > stimer, will be useful. To get the benefit for nested guests though, I > > > assume we'll need to exposed enlightened VMCS, EPT flush, etc. > > > through? > > > > Per TLFS, eVMCS should be used by L1 Xen. > > Yes, I guess it only needs to be used by L1, but Windows is using an > increasing number of VMs for various purposes so I think making it > stack well would be useful. > > > > > Not sure what you meant by EPT flush. If you mean things like > > HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen > > only as well. I'm not sure whether they should be expose to L2 guests. > > > > Yes, that's what I meant and, again, I was thinking about the nested > Windows guest using VMs itself. So this is basically nested on nested. I'm not sure how to make that work yet. I'm also not sure how well these interfaces stack. TLFS only describes one level of nesting. I will need to ask if we can make nested on nested work. > > > There are hypercalls for virtual TLB management. We may choose to expose > > them to L2 guests for better performance. > > > > Do you mean 'Direct Virtual Flush'? The hypercalls in question are already handled in viridian for the purposes of remote TLB flush, so I guess it would be a question of wiring them through. Yes. I meant that. Wei. > > Paul > > > Wei.
And a bit more thought. On Mon, Sep 23, 2019 at 01:54:31PM +0100, Wei Liu wrote: [...] > > > > > > Per TLFS, eVMCS should be used by L1 Xen. > > > > Yes, I guess it only needs to be used by L1, but Windows is using an > > increasing number of VMs for various purposes so I think making it > > stack well would be useful. > > > > > > > > Not sure what you meant by EPT flush. If you mean things like > > > HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen > > > only as well. I'm not sure whether they should be expose to L2 guests. > > > > > > > Yes, that's what I meant and, again, I was thinking about the nested > > Windows guest using VMs itself. > > So this is basically nested on nested. I'm not sure how to make that > work yet. I'm also not sure how well these interfaces stack. TLFS only > describes one level of nesting. I will need to ask if we can make nested > on nested work. > Since L2 Root is Windows in a Hyper-V setup, chances are what you want already works. Of course this depends on whether Windows, when running as Root or Child, uses VM related features. I'm not versed in Windows features. If you have specific things in mind, I can ask for more details and think about how to implement them in Xen. Wei.
> -----Original Message----- > From: Wei Liu <wl@xen.org> > Sent: 23 September 2019 14:34 > To: Paul Durrant <Paul.Durrant@citrix.com> > Cc: 'Wei Liu' <wl@xen.org>; Xen Development List <xen-devel@lists.xenproject.org>; Wei Liu > <liuwe@microsoft.com>; Andrew Cooper <Andrew.Cooper3@citrix.com>; Michael Kelley > <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne <roger.pau@citrix.com> > Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > And a bit more thought. > > On Mon, Sep 23, 2019 at 01:54:31PM +0100, Wei Liu wrote: > [...] > > > > > > > > Per TLFS, eVMCS should be used by L1 Xen. > > > > > > Yes, I guess it only needs to be used by L1, but Windows is using an > > > increasing number of VMs for various purposes so I think making it > > > stack well would be useful. > > > > > > > > > > > Not sure what you meant by EPT flush. If you mean things like > > > > HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen > > > > only as well. I'm not sure whether they should be expose to L2 guests. > > > > > > > > > > Yes, that's what I meant and, again, I was thinking about the nested > > > Windows guest using VMs itself. > > > > So this is basically nested on nested. I'm not sure how to make that > > work yet. I'm also not sure how well these interfaces stack. TLFS only > > describes one level of nesting. I will need to ask if we can make nested > > on nested work. > > > > Since L2 Root is Windows in a Hyper-V setup, chances are what you want > already works. Of course this depends on whether Windows, when running > as Root or Child, uses VM related features. > > I'm not versed in Windows features. If you have specific things in mind, > I can ask for more details and think about how to implement them in Xen. Well, eVMCS would be a major one. If the L2 Windows could be told to use that rather than L0 have to bounce individual VMREAD/VMWRITEs up to L1 then that would be a win. Paul > > Wei.
On Mon, Sep 23, 2019 at 01:47:14PM +0000, Paul Durrant wrote: > > -----Original Message----- > > From: Wei Liu <wl@xen.org> > > Sent: 23 September 2019 14:34 > > To: Paul Durrant <Paul.Durrant@citrix.com> > > Cc: 'Wei Liu' <wl@xen.org>; Xen Development List <xen-devel@lists.xenproject.org>; Wei Liu > > <liuwe@microsoft.com>; Andrew Cooper <Andrew.Cooper3@citrix.com>; Michael Kelley > > <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne <roger.pau@citrix.com> > > Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > > > And a bit more thought. > > > > On Mon, Sep 23, 2019 at 01:54:31PM +0100, Wei Liu wrote: > > [...] > > > > > > > > > > Per TLFS, eVMCS should be used by L1 Xen. > > > > > > > > Yes, I guess it only needs to be used by L1, but Windows is using an > > > > increasing number of VMs for various purposes so I think making it > > > > stack well would be useful. > > > > > > > > > > > > > > Not sure what you meant by EPT flush. If you mean things like > > > > > HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen > > > > > only as well. I'm not sure whether they should be expose to L2 guests. > > > > > > > > > > > > > Yes, that's what I meant and, again, I was thinking about the nested > > > > Windows guest using VMs itself. > > > > > > So this is basically nested on nested. I'm not sure how to make that > > > work yet. I'm also not sure how well these interfaces stack. TLFS only > > > describes one level of nesting. I will need to ask if we can make nested > > > on nested work. > > > > > > > Since L2 Root is Windows in a Hyper-V setup, chances are what you want > > already works. Of course this depends on whether Windows, when running > > as Root or Child, uses VM related features. > > > > I'm not versed in Windows features. If you have specific things in mind, > > I can ask for more details and think about how to implement them in Xen. > > Well, eVMCS would be a major one. If the L2 Windows could be told to > use that rather than L0 have to bounce individual VMREAD/VMWRITEs up > to L1 then that would be a win. Oh, I was thinking more about names of Windows specific high-level features. Yes I can definitely ask whether eVMCS is exposed to L2 Dom0 / Root. Wei. > > Paul > > > > > Wei.
> -----Original Message----- > From: Wei Liu <wl@xen.org> > Sent: 23 September 2019 15:21 > To: Paul Durrant <Paul.Durrant@citrix.com> > Cc: 'Wei Liu' <wl@xen.org>; Xen Development List <xen-devel@lists.xenproject.org>; Wei Liu > <liuwe@microsoft.com>; Andrew Cooper <Andrew.Cooper3@citrix.com>; Michael Kelley > <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne <roger.pau@citrix.com> > Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > On Mon, Sep 23, 2019 at 01:47:14PM +0000, Paul Durrant wrote: > > > -----Original Message----- > > > From: Wei Liu <wl@xen.org> > > > Sent: 23 September 2019 14:34 > > > To: Paul Durrant <Paul.Durrant@citrix.com> > > > Cc: 'Wei Liu' <wl@xen.org>; Xen Development List <xen-devel@lists.xenproject.org>; Wei Liu > > > <liuwe@microsoft.com>; Andrew Cooper <Andrew.Cooper3@citrix.com>; Michael Kelley > > > <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne <roger.pau@citrix.com> > > > Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > > > > > And a bit more thought. > > > > > > On Mon, Sep 23, 2019 at 01:54:31PM +0100, Wei Liu wrote: > > > [...] > > > > > > > > > > > > Per TLFS, eVMCS should be used by L1 Xen. > > > > > > > > > > Yes, I guess it only needs to be used by L1, but Windows is using an > > > > > increasing number of VMs for various purposes so I think making it > > > > > stack well would be useful. > > > > > > > > > > > > > > > > > Not sure what you meant by EPT flush. If you mean things like > > > > > > HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen > > > > > > only as well. I'm not sure whether they should be expose to L2 guests. > > > > > > > > > > > > > > > > Yes, that's what I meant and, again, I was thinking about the nested > > > > > Windows guest using VMs itself. > > > > > > > > So this is basically nested on nested. I'm not sure how to make that > > > > work yet. I'm also not sure how well these interfaces stack. TLFS only > > > > describes one level of nesting. I will need to ask if we can make nested > > > > on nested work. > > > > > > > > > > Since L2 Root is Windows in a Hyper-V setup, chances are what you want > > > already works. Of course this depends on whether Windows, when running > > > as Root or Child, uses VM related features. > > > > > > I'm not versed in Windows features. If you have specific things in mind, > > > I can ask for more details and think about how to implement them in Xen. > > > > Well, eVMCS would be a major one. If the L2 Windows could be told to > > use that rather than L0 have to bounce individual VMREAD/VMWRITEs up > > to L1 then that would be a win. > > Oh, I was thinking more about names of Windows specific high-level > features. Yes I can definitely ask whether eVMCS is exposed to L2 Dom0 / > Root. Oh, I see. The main ones are 'credential guard' and the more recent 'sandbox'. Paul > > Wei. > > > > > Paul > > > > > > > > Wei.
On Mon, Sep 23, 2019 at 02:39:10PM +0000, Paul Durrant wrote: > > -----Original Message----- > > From: Wei Liu <wl@xen.org> > > Sent: 23 September 2019 15:21 > > To: Paul Durrant <Paul.Durrant@citrix.com> > > Cc: 'Wei Liu' <wl@xen.org>; Xen Development List <xen-devel@lists.xenproject.org>; Wei Liu > > <liuwe@microsoft.com>; Andrew Cooper <Andrew.Cooper3@citrix.com>; Michael Kelley > > <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne <roger.pau@citrix.com> > > Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > > > On Mon, Sep 23, 2019 at 01:47:14PM +0000, Paul Durrant wrote: > > > > -----Original Message----- > > > > From: Wei Liu <wl@xen.org> > > > > Sent: 23 September 2019 14:34 > > > > To: Paul Durrant <Paul.Durrant@citrix.com> > > > > Cc: 'Wei Liu' <wl@xen.org>; Xen Development List <xen-devel@lists.xenproject.org>; Wei Liu > > > > <liuwe@microsoft.com>; Andrew Cooper <Andrew.Cooper3@citrix.com>; Michael Kelley > > > > <mikelley@microsoft.com>; Jan Beulich <jbeulich@suse.com>; Roger Pau Monne <roger.pau@citrix.com> > > > > Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V > > > > > > > > And a bit more thought. > > > > > > > > On Mon, Sep 23, 2019 at 01:54:31PM +0100, Wei Liu wrote: > > > > [...] > > > > > > > > > > > > > > Per TLFS, eVMCS should be used by L1 Xen. > > > > > > > > > > > > Yes, I guess it only needs to be used by L1, but Windows is using an > > > > > > increasing number of VMs for various purposes so I think making it > > > > > > stack well would be useful. > > > > > > > > > > > > > > > > > > > > Not sure what you meant by EPT flush. If you mean things like > > > > > > > HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen > > > > > > > only as well. I'm not sure whether they should be expose to L2 guests. > > > > > > > > > > > > > > > > > > > Yes, that's what I meant and, again, I was thinking about the nested > > > > > > Windows guest using VMs itself. > > > > > > > > > > So this is basically nested on nested. I'm not sure how to make that > > > > > work yet. I'm also not sure how well these interfaces stack. TLFS only > > > > > describes one level of nesting. I will need to ask if we can make nested > > > > > on nested work. > > > > > > > > > > > > > Since L2 Root is Windows in a Hyper-V setup, chances are what you want > > > > already works. Of course this depends on whether Windows, when running > > > > as Root or Child, uses VM related features. > > > > > > > > I'm not versed in Windows features. If you have specific things in mind, > > > > I can ask for more details and think about how to implement them in Xen. > > > > > > Well, eVMCS would be a major one. If the L2 Windows could be told to > > > use that rather than L0 have to bounce individual VMREAD/VMWRITEs up > > > to L1 then that would be a win. > > > > Oh, I was thinking more about names of Windows specific high-level > > features. Yes I can definitely ask whether eVMCS is exposed to L2 Dom0 / > > Root. > > Oh, I see. The main ones are 'credential guard' and the more recent 'sandbox'. > Got it. Wei.
On Mon, Sep 23, 2019 at 11:09:23AM +0100, Wei Liu wrote: > Hi all > > In case you're wondering, I can already run a fully fledged Xen system on > Hyper-V with emulated disk and network. > > This is the very first stage for porting Xen to run on Hyper-V with all the > goodies Hyper-V has to offer. With this series, Xen can successfully detect > Hyper-V and prints out a message. I would like to first get the directory > structure and kconfig options agreed upon. > > There are two major areas to be worked on: > * Make Dom0 able to use Hyper-V's synthetic devices. > * Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS > and other interfaces. > > They aren't trivial, and time can be scarce on my side, so I intend to post > patches piece meal when they are ready. > > Questions and comments are welcome. Thanks for doing this! In the past I've played with trying to get Xen to boot as a guest on HyperV gen2 instances, and I did manage to get it booting, Linux however would get extremely confused because gen2 lacks emulated devices and instead the guest is forced to use the HyperV PV devices AFAIK. Anyway, I had the following patch on my backlog which allowed me to boot on gen2 instances, posting it here in case it helps you if you plan to go that route. Roger. ---8<--- From 7c9fad735fdc3633ed401befda26c21dc7145af8 Mon Sep 17 00:00:00 2001 From: Roger Pau Monne <roger.pau@citrix.com> Date: Tue, 18 Sep 2018 11:36:42 +0200 Subject: [PATCH] x86/lapic: remove the PIT usage to calibrate the lapic timer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit And instead use NOW which is based on the TSC. This was already used when running in shim mode, since there's likely no PIT in that environment. Remove printing the CPU frequency, since it's already printed earlier at boot, and getting the CPU frequency against the TSC without any external reference timer is pointless. The motivation behind this change is to allow Xen to boot on HyperV gen2 instances, which lack a PIT. Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> --- I'm not sure about the reason behind the usage of the PIT instead of the TSC, maybe this was done because the TSC wasn't available until the Pentium? Xen certainly doesn't care about such hardware anymore, and the TSC is already used unconditionally in Xen. Linux seems to prefer to calibrate the lapic timer against the PM timer and has already dropped PIT usage for that. My early tests on a single box show no differences between the TSC or the PIT for lapic timer calibration, but that's a single box. The RFC is because I'm not sure I understand the motivation for using the PIT in the first place, so I might be missing something relevant that could make this patch moot. --- Cc: Jan Beulich <jbeulich@suse.com> Cc: Andrew Cooper <andrew.cooper3@citrix.com> Cc: Wei Liu <wei.liu2@citrix.com> --- xen/arch/x86/apic.c | 67 ++------------------------------------------- 1 file changed, 2 insertions(+), 65 deletions(-) diff --git a/xen/arch/x86/apic.c b/xen/arch/x86/apic.c index 88ada9d0ec..21dbeb2298 100644 --- a/xen/arch/x86/apic.c +++ b/xen/arch/x86/apic.c @@ -1010,46 +1010,6 @@ __next: /* used for system time scaling */ static u32 __read_mostly bus_scale; /* scaling factor: ns -> bus cycles */ -/* - * The timer chip is already set up at HZ interrupts per second here, - * but we do not accept timer interrupts yet. We only allow the BP - * to calibrate. - */ -static unsigned int __init get_8254_timer_count(void) -{ - /*extern spinlock_t i8253_lock;*/ - /*unsigned long flags;*/ - - unsigned int count; - - /*spin_lock_irqsave(&i8253_lock, flags);*/ - - outb_p(0x00, PIT_MODE); - count = inb_p(PIT_CH0); - count |= inb_p(PIT_CH0) << 8; - - /*spin_unlock_irqrestore(&i8253_lock, flags);*/ - - return count; -} - -/* next tick in 8254 can be caught by catching timer wraparound */ -static void __init wait_8254_wraparound(void) -{ - unsigned int curr_count, prev_count; - - curr_count = get_8254_timer_count(); - do { - prev_count = curr_count; - curr_count = get_8254_timer_count(); - - /* workaround for broken Mercury/Neptune */ - if (prev_count >= curr_count + 0x100) - curr_count = get_8254_timer_count(); - - } while (prev_count >= curr_count); -} - /* * This function sets up the local APIC timer, with a timeout of * 'clocks' APIC bus clock. During calibration we actually call @@ -1092,7 +1052,7 @@ static void setup_APIC_timer(void) local_irq_restore(flags); } -static void wait_tick_pvh(void) +static void wait_tick(void) { u64 lapse_ns = 1000000000ULL / HZ; s_time_t start, curr_time; @@ -1121,7 +1081,6 @@ static void wait_tick_pvh(void) static int __init calibrate_APIC_clock(void) { - unsigned long long t1, t2; long tt1, tt2; long result; int i; @@ -1138,33 +1097,15 @@ static int __init calibrate_APIC_clock(void) */ __setup_APIC_LVTT(1000000000); - if ( !xen_guest ) - /* - * The timer chip counts down to zero. Let's wait - * for a wraparound to start exact measurement: - * (the current tick might have been already half done) - */ - wait_8254_wraparound(); - else - wait_tick_pvh(); - - /* - * We wrapped around just now. Let's start: - */ - t1 = rdtsc_ordered(); tt1 = apic_read(APIC_TMCCT); /* * Let's wait LOOPS ticks: */ for (i = 0; i < LOOPS; i++) - if ( !xen_guest ) - wait_8254_wraparound(); - else - wait_tick_pvh(); + wait_tick(); tt2 = apic_read(APIC_TMCCT); - t2 = rdtsc_ordered(); /* * The APIC bus clock counter is 32 bits only, it @@ -1176,10 +1117,6 @@ static int __init calibrate_APIC_clock(void) result = (tt1-tt2)*APIC_DIVISOR/LOOPS; - apic_printk(APIC_VERBOSE, "..... CPU clock speed is %ld.%04ld MHz.\n", - ((long)(t2 - t1) / LOOPS) / (1000000 / HZ), - ((long)(t2 - t1) / LOOPS) % (1000000 / HZ)); - apic_printk(APIC_VERBOSE, "..... host bus clock speed is %ld.%04ld MHz.\n", result / (1000000 / HZ), result % (1000000 / HZ));
On Wed, Sep 25, 2019 at 01:02:50PM +0200, Roger Pau Monné wrote: > On Mon, Sep 23, 2019 at 11:09:23AM +0100, Wei Liu wrote: > > Hi all > > > > In case you're wondering, I can already run a fully fledged Xen system on > > Hyper-V with emulated disk and network. > > > > This is the very first stage for porting Xen to run on Hyper-V with all the > > goodies Hyper-V has to offer. With this series, Xen can successfully detect > > Hyper-V and prints out a message. I would like to first get the directory > > structure and kconfig options agreed upon. > > > > There are two major areas to be worked on: > > * Make Dom0 able to use Hyper-V's synthetic devices. > > * Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS > > and other interfaces. > > > > They aren't trivial, and time can be scarce on my side, so I intend to post > > patches piece meal when they are ready. > > > > Questions and comments are welcome. > > Thanks for doing this! > > In the past I've played with trying to get Xen to boot as a guest on > HyperV gen2 instances, and I did manage to get it booting, Linux > however would get extremely confused because gen2 lacks emulated > devices and instead the guest is forced to use the HyperV PV devices > AFAIK. How did you force Linux to use Hyper-V PV devices? AIUI Linux won't have access to VMBus (like XenBus but with some differences) and SynIC (Synthetic interrupt controller, think of it as an abstraction around event channel in Xen world) in the first place. But I haven't tried gen2 so I could be wrong. > > Anyway, I had the following patch on my backlog which allowed me to > boot on gen2 instances, posting it here in case it helps you if you > plan to go that route. The long term plan for the time source is to use the reference TSC page provided by Hyper-V, just like we did for Xen. Your patch seems to be an useful thing to do on its own anyway. Wei.
On Wed, Sep 25, 2019 at 04:36:34PM +0100, Wei Liu wrote: > On Wed, Sep 25, 2019 at 01:02:50PM +0200, Roger Pau Monné wrote: > > On Mon, Sep 23, 2019 at 11:09:23AM +0100, Wei Liu wrote: > > > Hi all > > > > > > In case you're wondering, I can already run a fully fledged Xen system on > > > Hyper-V with emulated disk and network. > > > > > > This is the very first stage for porting Xen to run on Hyper-V with all the > > > goodies Hyper-V has to offer. With this series, Xen can successfully detect > > > Hyper-V and prints out a message. I would like to first get the directory > > > structure and kconfig options agreed upon. > > > > > > There are two major areas to be worked on: > > > * Make Dom0 able to use Hyper-V's synthetic devices. > > > * Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS > > > and other interfaces. > > > > > > They aren't trivial, and time can be scarce on my side, so I intend to post > > > patches piece meal when they are ready. > > > > > > Questions and comments are welcome. > > > > Thanks for doing this! > > > > In the past I've played with trying to get Xen to boot as a guest on > > HyperV gen2 instances, and I did manage to get it booting, Linux > > however would get extremely confused because gen2 lacks emulated > > devices and instead the guest is forced to use the HyperV PV devices > > AFAIK. > > How did you force Linux to use Hyper-V PV devices? AIUI Linux won't > have access to VMBus (like XenBus but with some differences) and SynIC > (Synthetic interrupt controller, think of it as an abstraction around > event channel in Xen world) in the first place. But I haven't tried > gen2 so I could be wrong. > I try Gen2 this morning. I need to apply your patch and give Xen noapic option to boot. At the end I only get a blank screen so I don't think Linux has booted successfully. Wei.
On Wed, Sep 25, 2019 at 04:36:34PM +0100, Wei Liu wrote: > On Wed, Sep 25, 2019 at 01:02:50PM +0200, Roger Pau Monné wrote: > > On Mon, Sep 23, 2019 at 11:09:23AM +0100, Wei Liu wrote: > > > Hi all > > > > > > In case you're wondering, I can already run a fully fledged Xen system on > > > Hyper-V with emulated disk and network. > > > > > > This is the very first stage for porting Xen to run on Hyper-V with all the > > > goodies Hyper-V has to offer. With this series, Xen can successfully detect > > > Hyper-V and prints out a message. I would like to first get the directory > > > structure and kconfig options agreed upon. > > > > > > There are two major areas to be worked on: > > > * Make Dom0 able to use Hyper-V's synthetic devices. > > > * Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS > > > and other interfaces. > > > > > > They aren't trivial, and time can be scarce on my side, so I intend to post > > > patches piece meal when they are ready. > > > > > > Questions and comments are welcome. > > > > Thanks for doing this! > > > > In the past I've played with trying to get Xen to boot as a guest on > > HyperV gen2 instances, and I did manage to get it booting, Linux > > however would get extremely confused because gen2 lacks emulated > > devices and instead the guest is forced to use the HyperV PV devices > > AFAIK. > > How did you force Linux to use Hyper-V PV devices? I didn't, that's where I stopped. I just managed to get Xen booting on gen2, but Linux dom0 won't boot due to what you pointed out below. Roger.
On Thu, Sep 26, 2019 at 12:41:04PM +0200, Roger Pau Monné wrote: > On Wed, Sep 25, 2019 at 04:36:34PM +0100, Wei Liu wrote: > > On Wed, Sep 25, 2019 at 01:02:50PM +0200, Roger Pau Monné wrote: > > > On Mon, Sep 23, 2019 at 11:09:23AM +0100, Wei Liu wrote: > > > > Hi all > > > > > > > > In case you're wondering, I can already run a fully fledged Xen system on > > > > Hyper-V with emulated disk and network. > > > > > > > > This is the very first stage for porting Xen to run on Hyper-V with all the > > > > goodies Hyper-V has to offer. With this series, Xen can successfully detect > > > > Hyper-V and prints out a message. I would like to first get the directory > > > > structure and kconfig options agreed upon. > > > > > > > > There are two major areas to be worked on: > > > > * Make Dom0 able to use Hyper-V's synthetic devices. > > > > * Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS > > > > and other interfaces. > > > > > > > > They aren't trivial, and time can be scarce on my side, so I intend to post > > > > patches piece meal when they are ready. > > > > > > > > Questions and comments are welcome. > > > > > > Thanks for doing this! > > > > > > In the past I've played with trying to get Xen to boot as a guest on > > > HyperV gen2 instances, and I did manage to get it booting, Linux > > > however would get extremely confused because gen2 lacks emulated > > > devices and instead the guest is forced to use the HyperV PV devices > > > AFAIK. > > > > How did you force Linux to use Hyper-V PV devices? > > I didn't, that's where I stopped. I just managed to get Xen > booting on gen2, but Linux dom0 won't boot due to what you pointed out > below. OK. Thanks for clarification. I think when the port is complete a Xen system should be able to boot on both Gen1 and Gen2. Wei. > > Roger.
Hi all In case you're wondering, I can already run a fully fledged Xen system on Hyper-V with emulated disk and network. This is the very first stage for porting Xen to run on Hyper-V with all the goodies Hyper-V has to offer. With this series, Xen can successfully detect Hyper-V and prints out a message. I would like to first get the directory structure and kconfig options agreed upon. There are two major areas to be worked on: * Make Dom0 able to use Hyper-V's synthetic devices. * Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS and other interfaces. They aren't trivial, and time can be scarce on my side, so I intend to post patches piece meal when they are ready. Questions and comments are welcome. Thanks, Wei. Cc: Jan Beulich <jbeulich@suse.com> Cc: Andrew Cooper <andrew.cooper3@citrix.com> Cc: Wei Liu <wl@xen.org> Cc: Roger Pau Monné <roger.pau@citrix.com> Wei Liu (8): x86: introduce CONFIG_GUEST and move code x86: include asm_defns.h directly in hypercall.h x86: drop hypervisor_cpuid_base x86: factor out xen variants for hypervisor setup code x86: factor out hypervisor agnostic code x86: make probe_xen return boolean value x86: introduce CONFIG_HYPERV and hyperv directory x86: be more verbose when running nested xen/arch/x86/Kconfig | 13 ++ xen/arch/x86/Makefile | 2 +- xen/arch/x86/guest/Makefile | 6 +- xen/arch/x86/guest/hyperv/Makefile | 1 + xen/arch/x86/guest/hyperv/hyperv.c | 63 ++++++++++ xen/arch/x86/guest/hypervisor.c | 116 ++++++++++++++++++ xen/arch/x86/guest/xen/Makefile | 4 + xen/arch/x86/guest/{ => xen}/hypercall_page.S | 0 xen/arch/x86/guest/{ => xen}/pvh-boot.c | 0 xen/arch/x86/guest/{ => xen}/xen.c | 73 ++--------- xen/arch/x86/pv/shim.c | 2 +- xen/arch/x86/setup.c | 11 ++ xen/include/asm-x86/guest.h | 2 + xen/include/asm-x86/guest/hypercall.h | 2 + xen/include/asm-x86/guest/hyperv.h | 48 ++++++++ xen/include/asm-x86/guest/hypervisor.h | 58 +++++++++ xen/include/asm-x86/guest/xen.h | 23 +--- 17 files changed, 337 insertions(+), 87 deletions(-) create mode 100644 xen/arch/x86/guest/hyperv/Makefile create mode 100644 xen/arch/x86/guest/hyperv/hyperv.c create mode 100644 xen/arch/x86/guest/hypervisor.c create mode 100644 xen/arch/x86/guest/xen/Makefile rename xen/arch/x86/guest/{ => xen}/hypercall_page.S (100%) rename xen/arch/x86/guest/{ => xen}/pvh-boot.c (100%) rename xen/arch/x86/guest/{ => xen}/xen.c (80%) create mode 100644 xen/include/asm-x86/guest/hyperv.h create mode 100644 xen/include/asm-x86/guest/hypervisor.h