| Message ID | 1494084283-12723-2-git-send-email-eric.auger@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Sat, May 06 2017 at 4:24:20 pm BST, Eric Auger <eric.auger@redhat.com> wrote: > Add description for how to access ITS registers and how to save/restore > ITS tables into/from memory. > > Signed-off-by: Eric Auger <eric.auger@redhat.com> > > --- > v6 -> v7: > - rephrase ordering sequence + few cosmetic changes > > v5 -> v6: > - add restoration ordering > - 256B -> 256 Byte aligned > - DTE Size is number of bits for the EVENTID > - s/GITS_READR/GITS_CREADR > > v4 -> v5: > - take into account Christoffer's comments > - pending table save on GICV3 side now > > v3 -> v4: > - take into account Peter's comments: > - typos > - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0 > - add a validity bit in DTE > - document all fields in CTE and ITE > - document ABI revision > - take into account Andre's comments: > - document restrictions about GITS_CREADR writing and GITS_IIDR > - document -EBUSY error if one or more VCPUS are runnning > - document 64b registers only can be accessed with 64b access > - itt_addr field matches bits [51:8] of the itt_addr > > v1 -> v2: > - DTE and ITE now are 8 bytes > - DTE and ITE now indexed by deviceid/eventid > - use ITE name instead of ITTE > - mentions ITT_addr matches bits [51:8] of the actual address > - mentions LE layout > --- > Documentation/virtual/kvm/devices/arm-vgic-its.txt | 120 +++++++++++++++++++++ > 1 file changed, 120 insertions(+) > > diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt > index 6081a5b..ba132e9 100644 > --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt > +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt [...] > + ITS Table ABI REV0: > + ------------------- > + > + Revision 0 of the ABI only supports physical LPIs. Nit: these are no more physical than any other interrupt that KVM deals with. If you're hinting at the lack of GICv4 support, it wouldn't necessarily invalidate this ABI. It is actually even likely that the ABI could stay the same (until we start supporting GICv4 in a nested configuration). > + > + The device table and ITT are indexed by the deviceid and eventid, > + respectively. The collection table is not indexed by collectionid: > + CTEs are written in the table in the order of collection creation. All Is this order really relevant? Can we relax it? Would something break if collections were in a random order? Thanks, M.
Hi Marc, On 07/05/2017 13:54, Marc Zyngier wrote: > On Sat, May 06 2017 at 4:24:20 pm BST, Eric Auger <eric.auger@redhat.com> wrote: >> Add description for how to access ITS registers and how to save/restore >> ITS tables into/from memory. >> >> Signed-off-by: Eric Auger <eric.auger@redhat.com> >> >> --- >> v6 -> v7: >> - rephrase ordering sequence + few cosmetic changes >> >> v5 -> v6: >> - add restoration ordering >> - 256B -> 256 Byte aligned >> - DTE Size is number of bits for the EVENTID >> - s/GITS_READR/GITS_CREADR >> >> v4 -> v5: >> - take into account Christoffer's comments >> - pending table save on GICV3 side now >> >> v3 -> v4: >> - take into account Peter's comments: >> - typos >> - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0 >> - add a validity bit in DTE >> - document all fields in CTE and ITE >> - document ABI revision >> - take into account Andre's comments: >> - document restrictions about GITS_CREADR writing and GITS_IIDR >> - document -EBUSY error if one or more VCPUS are runnning >> - document 64b registers only can be accessed with 64b access >> - itt_addr field matches bits [51:8] of the itt_addr >> >> v1 -> v2: >> - DTE and ITE now are 8 bytes >> - DTE and ITE now indexed by deviceid/eventid >> - use ITE name instead of ITTE >> - mentions ITT_addr matches bits [51:8] of the actual address >> - mentions LE layout >> --- >> Documentation/virtual/kvm/devices/arm-vgic-its.txt | 120 +++++++++++++++++++++ >> 1 file changed, 120 insertions(+) >> >> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt >> index 6081a5b..ba132e9 100644 >> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt >> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt > [...] > >> + ITS Table ABI REV0: >> + ------------------- >> + >> + Revision 0 of the ABI only supports physical LPIs. > > Nit: these are no more physical than any other interrupt that KVM deals > with. If you're hinting at the lack of GICv4 support, it wouldn't > necessarily invalidate this ABI. It is actually even likely that the ABI > could stay the same (until we start supporting GICv4 in a nested > configuration). I understood vLPI are associated to vPE and not to collections. As such ITE would need to be updated, wouldn't it? > >> + >> + The device table and ITT are indexed by the deviceid and eventid, >> + respectively. The collection table is not indexed by collectionid: >> + CTEs are written in the table in the order of collection creation. All > > Is this order really relevant? Can we relax it? Would something break if > collections were in a random order? Christoffer asked me to mention the exact storage order or at least I understood his comment that way. Nothing would break if we change the order. Thanks Eric > > Thanks, > > M. >
On 07/05/17 18:05, Auger Eric wrote: > Hi Marc, > > On 07/05/2017 13:54, Marc Zyngier wrote: >> On Sat, May 06 2017 at 4:24:20 pm BST, Eric Auger <eric.auger@redhat.com> wrote: >>> Add description for how to access ITS registers and how to save/restore >>> ITS tables into/from memory. >>> >>> Signed-off-by: Eric Auger <eric.auger@redhat.com> >>> >>> --- >>> v6 -> v7: >>> - rephrase ordering sequence + few cosmetic changes >>> >>> v5 -> v6: >>> - add restoration ordering >>> - 256B -> 256 Byte aligned >>> - DTE Size is number of bits for the EVENTID >>> - s/GITS_READR/GITS_CREADR >>> >>> v4 -> v5: >>> - take into account Christoffer's comments >>> - pending table save on GICV3 side now >>> >>> v3 -> v4: >>> - take into account Peter's comments: >>> - typos >>> - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0 >>> - add a validity bit in DTE >>> - document all fields in CTE and ITE >>> - document ABI revision >>> - take into account Andre's comments: >>> - document restrictions about GITS_CREADR writing and GITS_IIDR >>> - document -EBUSY error if one or more VCPUS are runnning >>> - document 64b registers only can be accessed with 64b access >>> - itt_addr field matches bits [51:8] of the itt_addr >>> >>> v1 -> v2: >>> - DTE and ITE now are 8 bytes >>> - DTE and ITE now indexed by deviceid/eventid >>> - use ITE name instead of ITTE >>> - mentions ITT_addr matches bits [51:8] of the actual address >>> - mentions LE layout >>> --- >>> Documentation/virtual/kvm/devices/arm-vgic-its.txt | 120 +++++++++++++++++++++ >>> 1 file changed, 120 insertions(+) >>> >>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt >>> index 6081a5b..ba132e9 100644 >>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt >>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt >> [...] >> >>> + ITS Table ABI REV0: >>> + ------------------- >>> + >>> + Revision 0 of the ABI only supports physical LPIs. >> >> Nit: these are no more physical than any other interrupt that KVM deals >> with. If you're hinting at the lack of GICv4 support, it wouldn't >> necessarily invalidate this ABI. It is actually even likely that the ABI >> could stay the same (until we start supporting GICv4 in a nested >> configuration). > I understood vLPI are associated to vPE and not to collections. As such > ITE would need to be updated, wouldn't it? Even in a GICv4 setup (and ignoring nesting), a guest would only see a normal GICv3, as all the GICv4 state is the hypervisor's business, and not something meaningful to the guest. Also, it is very desirable to be able to migrate a GICv4 setup to a GICv3 setup without breaking everything. To precisely answer your question, the VPE setup is done by finding out which collection is mapped to which vcpu, and using that to map VLPIs to VPEs. None of that is observable by the guest. >> >>> + >>> + The device table and ITT are indexed by the deviceid and eventid, >>> + respectively. The collection table is not indexed by collectionid: >>> + CTEs are written in the table in the order of collection creation. All >> >> Is this order really relevant? Can we relax it? Would something break if >> collections were in a random order? > Christoffer asked me to mention the exact storage order or at least I > understood his comment that way. Nothing would break if we change the order. OK. If that's not a requirement, then we should probably drop it. Christoffer, what do you think? Thanks, M.
On Mon, May 08, 2017 at 10:14:21AM +0100, Marc Zyngier wrote: > On 07/05/17 18:05, Auger Eric wrote: > > Hi Marc, > > > > On 07/05/2017 13:54, Marc Zyngier wrote: > >> On Sat, May 06 2017 at 4:24:20 pm BST, Eric Auger <eric.auger@redhat.com> wrote: > >>> Add description for how to access ITS registers and how to save/restore > >>> ITS tables into/from memory. > >>> > >>> Signed-off-by: Eric Auger <eric.auger@redhat.com> > >>> > >>> --- > >>> v6 -> v7: > >>> - rephrase ordering sequence + few cosmetic changes > >>> > >>> v5 -> v6: > >>> - add restoration ordering > >>> - 256B -> 256 Byte aligned > >>> - DTE Size is number of bits for the EVENTID > >>> - s/GITS_READR/GITS_CREADR > >>> > >>> v4 -> v5: > >>> - take into account Christoffer's comments > >>> - pending table save on GICV3 side now > >>> > >>> v3 -> v4: > >>> - take into account Peter's comments: > >>> - typos > >>> - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0 > >>> - add a validity bit in DTE > >>> - document all fields in CTE and ITE > >>> - document ABI revision > >>> - take into account Andre's comments: > >>> - document restrictions about GITS_CREADR writing and GITS_IIDR > >>> - document -EBUSY error if one or more VCPUS are runnning > >>> - document 64b registers only can be accessed with 64b access > >>> - itt_addr field matches bits [51:8] of the itt_addr > >>> > >>> v1 -> v2: > >>> - DTE and ITE now are 8 bytes > >>> - DTE and ITE now indexed by deviceid/eventid > >>> - use ITE name instead of ITTE > >>> - mentions ITT_addr matches bits [51:8] of the actual address > >>> - mentions LE layout > >>> --- > >>> Documentation/virtual/kvm/devices/arm-vgic-its.txt | 120 +++++++++++++++++++++ > >>> 1 file changed, 120 insertions(+) > >>> > >>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt > >>> index 6081a5b..ba132e9 100644 > >>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt > >>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt > >> [...] > >> > >>> + ITS Table ABI REV0: > >>> + ------------------- > >>> + > >>> + Revision 0 of the ABI only supports physical LPIs. > >> > >> Nit: these are no more physical than any other interrupt that KVM deals > >> with. If you're hinting at the lack of GICv4 support, it wouldn't > >> necessarily invalidate this ABI. It is actually even likely that the ABI > >> could stay the same (until we start supporting GICv4 in a nested > >> configuration). > > I understood vLPI are associated to vPE and not to collections. As such > > ITE would need to be updated, wouldn't it? > > Even in a GICv4 setup (and ignoring nesting), a guest would only see a > normal GICv3, as all the GICv4 state is the hypervisor's business, and > not something meaningful to the guest. Also, it is very desirable to be > able to migrate a GICv4 setup to a GICv3 setup without breaking everything. > > To precisely answer your question, the VPE setup is done by finding out > which collection is mapped to which vcpu, and using that to map VLPIs to > VPEs. None of that is observable by the guest. > Doesn't GICv4 hardware has state to configure the virtual interrupt injection, which we must model if the guest hypervisor should think it runs on a GICv4 system and uses direct interrupt injection? For example, the vPE table entries could be cached by the ITS and would therefore need to be flushed if presenting a GICv4 to the guest. To clarify: I understand this comment to be about what we present to the guest, a GICv3 or a GICv4, and doesn't affect whether or not we run on a physical GICv3 or GICv4, so when Eric says 'physical LPIs' here, he really means 'physical from the point of view of the VM', which are of course virtual, but are physical in the virtual emulated GICv3. I tried to clarify this in a fix I will send shortly. > >> > >>> + > >>> + The device table and ITT are indexed by the deviceid and eventid, > >>> + respectively. The collection table is not indexed by collectionid: > >>> + CTEs are written in the table in the order of collection creation. All > >> > >> Is this order really relevant? Can we relax it? Would something break if > >> collections were in a random order? > > Christoffer asked me to mention the exact storage order or at least I > > understood his comment that way. Nothing would break if we change the order. > > OK. If that's not a requirement, then we should probably drop it. > > Christoffer, what do you think? > Yes, we should drop it. I have a fix for this in my queue. Thanks, -Christoffer
diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt index 6081a5b..ba132e9 100644 --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt @@ -32,7 +32,127 @@ Groups: KVM_DEV_ARM_VGIC_CTRL_INIT request the initialization of the ITS, no additional parameter in kvm_device_attr.addr. + + KVM_DEV_ARM_ITS_SAVE_TABLES + save the ITS table data into guest RAM, at the location provisioned + by the guest in corresponding registers/table entries. + + The layout of the tables in guest memory defines an ABI. The entries + are laid out in little endian format as described in the last paragraph. + + KVM_DEV_ARM_ITS_RESTORE_TABLES + restore the ITS tables from guest RAM to ITS internal structures. + + The GICV3 must be restored before the ITS and all ITS registers but + the GITS_CTLR must be restored before restoring the ITS tables. + + The GITS_IIDR read-only register must also be restored before + calling KVM_DEV_ARM_ITS_RESTORE_TABLES as the IIDR revision field + encodes the ABI revision. + + The expected ordering when restoring the GICv3/ITS is described in section + "ITS Restore Sequence". + Errors: -ENXIO: ITS not properly configured as required prior to setting this attribute -ENOMEM: Memory shortage when allocating ITS internal data + -EINVAL: Inconsistent restored data + -EFAULT: Invalid guest ram access + -EBUSY: One or more VCPUS are running + + KVM_DEV_ARM_VGIC_GRP_ITS_REGS + Attributes: + The attr field of kvm_device_attr encodes the offset of the + ITS register, relative to the ITS control frame base address + (ITS_base). + + kvm_device_attr.addr points to a __u64 value whatever the width + of the addressed register (32/64 bits). 64 bit registers can only + be accessed with full length. + + Writes to read-only registers are ignored by the kernel except for: + - GITS_CREADR. It must be restored otherwise commands in the queue + will be re-executed after restoring CWRITER. GITS_CREADR must be + restored before restoring the GITS_CTLR which is likely to enable the + ITS. Also it must be restored after GITS_CBASER since a write to + GITS_CBASER resets GITS_CREADR. + - GITS_IIDR. The Revision field encodes the table layout ABI revision. + In the future we might implement direct injection of virtual LPIs. + This will require an upgrade of the table layout and an evolution of + the ABI. GITS_IIDR must be restored before calling + KVM_DEV_ARM_ITS_RESTORE_TABLES. + + For other registers, getting or setting a register has the same + effect as reading/writing the register on real hardware. + Errors: + -ENXIO: Offset does not correspond to any supported register + -EFAULT: Invalid user pointer for attr->addr + -EINVAL: Offset is not 64-bit aligned + -EBUSY: one or more VCPUS are running + + ITS Restore Sequence: + ------------------------- + +The following ordering must be followed when restoring the GIC and the ITS: +a) restore all guest memory and create vcpus +b) restore all redistributors +c) initialize the ITS and then provide its base address + (KVM_DEV_ARM_VGIC_CTRL_INIT, KVM_DEV_ARM_VGIC_GRP_ADDR) +d) restore the ITS in the following order: + 1. Restore GITS_CBASER + 2. Restore all other GITS_ registers, except GITS_CTLR! + 3. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES) + 4. Restore GITS_CTLR + +Then vcpus can be started. + + ITS Table ABI REV0: + ------------------- + + Revision 0 of the ABI only supports physical LPIs. + + The device table and ITT are indexed by the deviceid and eventid, + respectively. The collection table is not indexed by collectionid: + CTEs are written in the table in the order of collection creation. All + entries are 8 bytes. + + Device Table Entry (DTE): + + bits: | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 | + values: | V | next | ITT_addr | Size | + + where; + - V indicates whether the entry is valid. If not, other fields + are not meaningful. + - next: equals to 0 if this entry is the last one; otherwise it + corresponds to the deviceid offset to the next DTE, capped by + 2^14 -1. + - ITT_addr matches bits [51:8] of the ITT address (256 Byte aligned). + - Size specifies the supported number of bits for the eventid, + minus one + + Collection Table Entry (CTE): + + bits: | 63| 62 .. 52 | 51 ... 16 | 15 ... 0 | + values: | V | RES0 | RDBase | ICID | + + where: + - V indicates whether the entry is valid. If not, other fields are + not meaningful. + - RES0: reserved field with Should-Be-Zero-or-Preserved behavior. + - RDBase is the PE number (GICR_TYPER.Processor_Number semantic), + - ICID is the collection ID + + Interrupt Translation Entry (ITE): + + bits: | 63 ... 48 | 47 ... 16 | 15 ... 0 | + values: | next | pINTID | ICID | + + where: + - next: equals to 0 if this entry is the last one; otherwise it corresponds + to the eventid offset to the next ITE capped by 2^16 -1. + - pINTID is the physical LPI ID; if zero, it means the entry is not valid + and other fields are not meaningful. + - ICID is the collection ID +
Add description for how to access ITS registers and how to save/restore ITS tables into/from memory. Signed-off-by: Eric Auger <eric.auger@redhat.com> --- v6 -> v7: - rephrase ordering sequence + few cosmetic changes v5 -> v6: - add restoration ordering - 256B -> 256 Byte aligned - DTE Size is number of bits for the EVENTID - s/GITS_READR/GITS_CREADR v4 -> v5: - take into account Christoffer's comments - pending table save on GICV3 side now v3 -> v4: - take into account Peter's comments: - typos - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0 - add a validity bit in DTE - document all fields in CTE and ITE - document ABI revision - take into account Andre's comments: - document restrictions about GITS_CREADR writing and GITS_IIDR - document -EBUSY error if one or more VCPUS are runnning - document 64b registers only can be accessed with 64b access - itt_addr field matches bits [51:8] of the itt_addr v1 -> v2: - DTE and ITE now are 8 bytes - DTE and ITE now indexed by deviceid/eventid - use ITE name instead of ITTE - mentions ITT_addr matches bits [51:8] of the actual address - mentions LE layout --- Documentation/virtual/kvm/devices/arm-vgic-its.txt | 120 +++++++++++++++++++++ 1 file changed, 120 insertions(+)