x86: shrink 'struct domain', was already PAGE_SIZE

Commit Message

Corneliu ZUZU Feb. 1, 2016, 7:42 a.m. UTC

The X86 domain structure already occupied PAGE_SIZE (4096).

Looking @ the memory layout of the structure, we could see that
overall most was occupied by (used the pahole tool on domain.o):
 * sizeof(domain.arch) = sizeof(arch_domain) = 3328 bytes.
 * sizeof(domain.arch.hvm_domain) = 2224 bytes.
 * sizeof(domain.arch.hvm_domain.pl_time) = 1088 bytes.
This patch attempts to free some space, by making the pl_time
field in hvm_domain dynamically allocated.
We xzalloc/xfree it @ hvm_domain_initialise/hvm_domain_destroy.

After this change, the domain structure shrunk w/ 1152 bytes (>1K!).

Signed-off-by: Corneliu ZUZU <[e-mail]>
---
 xen/arch/x86/hvm/hpet.c          |  5 ++---
 xen/arch/x86/hvm/hvm.c           |  9 ++++++++-
 xen/arch/x86/hvm/pmtimer.c       | 18 +++++++++---------
 xen/arch/x86/hvm/rtc.c           |  5 ++---
 xen/arch/x86/hvm/vpt.c           | 10 +++++-----
 xen/arch/x86/time.c              |  2 +-
 xen/include/asm-x86/hvm/domain.h |  2 +-
 xen/include/asm-x86/hvm/vpt.h    |  3 +++
 8 files changed, 31 insertions(+), 23 deletions(-)

Comments

Jan Beulich Feb. 1, 2016, 10:47 a.m. UTC | #1

>>> On 01.02.16 at 08:42, <[e-mail]> wrote:
> The X86 domain structure already occupied PAGE_SIZE (4096).
> 
> Looking @ the memory layout of the structure, we could see that
> overall most was occupied by (used the pahole tool on domain.o):
>  * sizeof(domain.arch) = sizeof(arch_domain) = 3328 bytes.
>  * sizeof(domain.arch.hvm_domain) = 2224 bytes.
>  * sizeof(domain.arch.hvm_domain.pl_time) = 1088 bytes.
> This patch attempts to free some space, by making the pl_time
> field in hvm_domain dynamically allocated.
> We xzalloc/xfree it @ hvm_domain_initialise/hvm_domain_destroy.
> 
> After this change, the domain structure shrunk w/ 1152 bytes (>1K!).
> 
> Signed-off-by: Corneliu ZUZU <[e-mail]>

Reviewed-by: Jan Beulich <[e-mail]>
albeit ...

> --- a/xen/include/asm-x86/hvm/vpt.h
> +++ b/xen/include/asm-x86/hvm/vpt.h
> @@ -139,6 +139,9 @@ struct pl_time {    /* platform time */
>      /* Ensures monotonicity in appropriate timer modes. */
>      uint64_t last_guest_time;
>      spinlock_t pl_time_lock;
> +    /* pl_time allocated dynamically, need to keep link to
> +     * containing domain */
> +    struct domain *domain;
>  };

... I wonder about the usefulness of this (malformed) comment. I'll
likely ditch it while committing.

Jan

Corneliu ZUZU Feb. 1, 2016, 10:58 a.m. UTC | #2

On 2/1/2016 12:47 PM, Jan Beulich wrote:
>>>> On 01.02.16 at 08:42, <[e-mail]> wrote:
>> The X86 domain structure already occupied PAGE_SIZE (4096).
>>
>> Looking @ the memory layout of the structure, we could see that
>> overall most was occupied by (used the pahole tool on domain.o):
>>   * sizeof(domain.arch) = sizeof(arch_domain) = 3328 bytes.
>>   * sizeof(domain.arch.hvm_domain) = 2224 bytes.
>>   * sizeof(domain.arch.hvm_domain.pl_time) = 1088 bytes.
>> This patch attempts to free some space, by making the pl_time
>> field in hvm_domain dynamically allocated.
>> We xzalloc/xfree it @ hvm_domain_initialise/hvm_domain_destroy.
>>
>> After this change, the domain structure shrunk w/ 1152 bytes (>1K!).
>>
>> Signed-off-by: Corneliu ZUZU <[e-mail]>
> Reviewed-by: Jan Beulich <[e-mail]>
> albeit ...
>
>> --- a/xen/include/asm-x86/hvm/vpt.h
>> +++ b/xen/include/asm-x86/hvm/vpt.h
>> @@ -139,6 +139,9 @@ struct pl_time {    /* platform time */
>>       /* Ensures monotonicity in appropriate timer modes. */
>>       uint64_t last_guest_time;
>>       spinlock_t pl_time_lock;
>> +    /* pl_time allocated dynamically, need to keep link to
>> +     * containing domain */
>> +    struct domain *domain;
>>   };
> ... I wonder about the usefulness of this (malformed) comment. I'll
> likely ditch it while committing.
>
> Jan
>

Agreed, sorry about that and thanks.

Corneliu.

Andrew Cooper Feb. 1, 2016, 11:07 a.m. UTC | #3

On 01/02/16 07:42, Corneliu ZUZU wrote:
> The X86 domain structure already occupied PAGE_SIZE (4096).
>
> Looking @ the memory layout of the structure, we could see that
> overall most was occupied by (used the pahole tool on domain.o):
>  * sizeof(domain.arch) = sizeof(arch_domain) = 3328 bytes.
>  * sizeof(domain.arch.hvm_domain) = 2224 bytes.
>  * sizeof(domain.arch.hvm_domain.pl_time) = 1088 bytes.
> This patch attempts to free some space, by making the pl_time
> field in hvm_domain dynamically allocated.
> We xzalloc/xfree it @ hvm_domain_initialise/hvm_domain_destroy.
>
> After this change, the domain structure shrunk w/ 1152 bytes (>1K!).
>
> Signed-off-by: Corneliu ZUZU <[e-mail]>

Reviewed-by: Andrew Cooper <[e-mail]>

Thankyou for doing this - it is exactly the kind of change I had in mind.

~Andrew

Corneliu ZUZU Feb. 1, 2016, 11:28 a.m. UTC | #4

On 2/1/2016 1:07 PM, Andrew Cooper wrote:
> On 01/02/16 07:42, Corneliu ZUZU wrote:
>> The X86 domain structure already occupied PAGE_SIZE (4096).
>>
>> Looking @ the memory layout of the structure, we could see that
>> overall most was occupied by (used the pahole tool on domain.o):
>>   * sizeof(domain.arch) = sizeof(arch_domain) = 3328 bytes.
>>   * sizeof(domain.arch.hvm_domain) = 2224 bytes.
>>   * sizeof(domain.arch.hvm_domain.pl_time) = 1088 bytes.
>> This patch attempts to free some space, by making the pl_time
>> field in hvm_domain dynamically allocated.
>> We xzalloc/xfree it @ hvm_domain_initialise/hvm_domain_destroy.
>>
>> After this change, the domain structure shrunk w/ 1152 bytes (>1K!).
>>
>> Signed-off-by: Corneliu ZUZU <[e-mail]>
> Reviewed-by: Andrew Cooper <[e-mail]>
>
> Thankyou for doing this - it is exactly the kind of change I had in mind.
>
> ~Andrew
>
Glad to hear that and thank you for the prompt review :) I really needed 
this for another patch.

Corneliu.

Patch

diff --git a/xen/arch/x86/hvm/hpet.c b/xen/arch/x86/hvm/hpet.c
index 5e020ae..de5c088 100644
--- a/xen/arch/x86/hvm/hpet.c
+++ b/xen/arch/x86/hvm/hpet.c
@@ -26,10 +26,9 @@ 
 #include <xen/event.h>
 #include <xen/trace.h>
 
-#define domain_vhpet(x) (&(x)->arch.hvm_domain.pl_time.vhpet)
+#define domain_vhpet(x) (&(x)->arch.hvm_domain.pl_time->vhpet)
 #define vcpu_vhpet(x)   (domain_vhpet((x)->domain))
-#define vhpet_domain(x) (container_of((x), struct domain, \
-                                      arch.hvm_domain.pl_time.vhpet))
+#define vhpet_domain(x) (container_of((x), struct pl_time, vhpet)->domain)
 #define vhpet_vcpu(x)   (pt_global_vcpu_target(vhpet_domain(x)))
 
 #define HPET_BASE_ADDRESS   0xfed00000ULL
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 674feea..cc667ae 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -1574,13 +1574,18 @@  int hvm_domain_initialise(struct domain *d)
     if ( rc != 0 )
         goto fail0;
 
+    d->arch.hvm_domain.pl_time = xzalloc(struct pl_time);
     d->arch.hvm_domain.params = xzalloc_array(uint64_t, HVM_NR_PARAMS);
     d->arch.hvm_domain.io_handler = xzalloc_array(struct hvm_io_handler,
                                                   NR_IO_HANDLERS);
     rc = -ENOMEM;
-    if ( !d->arch.hvm_domain.params || !d->arch.hvm_domain.io_handler )
+    if ( !d->arch.hvm_domain.pl_time ||
+         !d->arch.hvm_domain.params  || !d->arch.hvm_domain.io_handler )
         goto fail1;
 
+    /* need link to containing domain */
+    d->arch.hvm_domain.pl_time->domain = d;
+
     /* Set the default IO Bitmap. */
     if ( is_hardware_domain(d) )
     {
@@ -1637,6 +1642,7 @@  int hvm_domain_initialise(struct domain *d)
         xfree(d->arch.hvm_domain.io_bitmap);
     xfree(d->arch.hvm_domain.io_handler);
     xfree(d->arch.hvm_domain.params);
+    xfree(d->arch.hvm_domain.pl_time);
  fail0:
     hvm_destroy_cacheattr_region_list(d);
     return rc;
@@ -1667,6 +1673,7 @@  void hvm_domain_destroy(struct domain *d)
 {
     xfree(d->arch.hvm_domain.io_handler);
     xfree(d->arch.hvm_domain.params);
+    xfree(d->arch.hvm_domain.pl_time);
 
     hvm_destroy_cacheattr_region_list(d);
 
diff --git a/xen/arch/x86/hvm/pmtimer.c b/xen/arch/x86/hvm/pmtimer.c
index 9c2e4bd..b1a5565 100644
--- a/xen/arch/x86/hvm/pmtimer.c
+++ b/xen/arch/x86/hvm/pmtimer.c
@@ -66,7 +66,7 @@  static void pmt_update_sci(PMTState *s)
 
 void hvm_acpi_power_button(struct domain *d)
 {
-    PMTState *s = &d->arch.hvm_domain.pl_time.vpmt;
+    PMTState *s = &d->arch.hvm_domain.pl_time->vpmt;
 
     if ( !has_vpm(d) )
         return;
@@ -79,7 +79,7 @@  void hvm_acpi_power_button(struct domain *d)
 
 void hvm_acpi_sleep_button(struct domain *d)
 {
-    PMTState *s = &d->arch.hvm_domain.pl_time.vpmt;
+    PMTState *s = &d->arch.hvm_domain.pl_time->vpmt;
 
     if ( !has_vpm(d) )
         return;
@@ -152,7 +152,7 @@  static int handle_evt_io(
     int dir, unsigned int port, unsigned int bytes, uint32_t *val)
 {
     struct vcpu *v = current;
-    PMTState *s = &v->domain->arch.hvm_domain.pl_time.vpmt;
+    PMTState *s = &v->domain->arch.hvm_domain.pl_time->vpmt;
     uint32_t addr, data, byte;
     int i;
 
@@ -215,7 +215,7 @@  static int handle_pmt_io(
     int dir, unsigned int port, unsigned int bytes, uint32_t *val)
 {
     struct vcpu *v = current;
-    PMTState *s = &v->domain->arch.hvm_domain.pl_time.vpmt;
+    PMTState *s = &v->domain->arch.hvm_domain.pl_time->vpmt;
 
     if ( bytes != 4 )
     {
@@ -251,7 +251,7 @@  static int handle_pmt_io(
 
 static int pmtimer_save(struct domain *d, hvm_domain_context_t *h)
 {
-    PMTState *s = &d->arch.hvm_domain.pl_time.vpmt;
+    PMTState *s = &d->arch.hvm_domain.pl_time->vpmt;
     uint32_t x, msb = s->pm.tmr_val & TMR_VAL_MSB;
     int rc;
 
@@ -282,7 +282,7 @@  static int pmtimer_save(struct domain *d, hvm_domain_context_t *h)
 
 static int pmtimer_load(struct domain *d, hvm_domain_context_t *h)
 {
-    PMTState *s = &d->arch.hvm_domain.pl_time.vpmt;
+    PMTState *s = &d->arch.hvm_domain.pl_time->vpmt;
 
     if ( !has_vpm(d) )
         return -ENODEV;
@@ -345,7 +345,7 @@  int pmtimer_change_ioport(struct domain *d, unsigned int version)
 
 void pmtimer_init(struct vcpu *v)
 {
-    PMTState *s = &v->domain->arch.hvm_domain.pl_time.vpmt;
+    PMTState *s = &v->domain->arch.hvm_domain.pl_time->vpmt;
 
     if ( !has_vpm(v->domain) )
         return;
@@ -369,7 +369,7 @@  void pmtimer_init(struct vcpu *v)
 
 void pmtimer_deinit(struct domain *d)
 {
-    PMTState *s = &d->arch.hvm_domain.pl_time.vpmt;
+    PMTState *s = &d->arch.hvm_domain.pl_time->vpmt;
 
     if ( !has_vpm(d) )
         return;
@@ -383,5 +383,5 @@  void pmtimer_reset(struct domain *d)
         return;
 
     /* Reset the counter. */
-    d->arch.hvm_domain.pl_time.vpmt.pm.tmr_val = 0;
+    d->arch.hvm_domain.pl_time->vpmt.pm.tmr_val = 0;
 }
diff --git a/xen/arch/x86/hvm/rtc.c b/xen/arch/x86/hvm/rtc.c
index d391e38..d98d694 100644
--- a/xen/arch/x86/hvm/rtc.c
+++ b/xen/arch/x86/hvm/rtc.c
@@ -38,10 +38,9 @@ 
 #define MIN_PER_HOUR    60
 #define HOUR_PER_DAY    24
 
-#define domain_vrtc(x) (&(x)->arch.hvm_domain.pl_time.vrtc)
+#define domain_vrtc(x) (&(x)->arch.hvm_domain.pl_time->vrtc)
 #define vcpu_vrtc(x)   (domain_vrtc((x)->domain))
-#define vrtc_domain(x) (container_of((x), struct domain, \
-                                     arch.hvm_domain.pl_time.vrtc))
+#define vrtc_domain(x) (container_of((x), struct pl_time, vrtc)->domain)
 #define vrtc_vcpu(x)   (pt_global_vcpu_target(vrtc_domain(x)))
 #define epoch_year     1900
 #define get_year(x)    (x + epoch_year)
diff --git a/xen/arch/x86/hvm/vpt.c b/xen/arch/x86/hvm/vpt.c
index 4fa6566..358ec57 100644
--- a/xen/arch/x86/hvm/vpt.c
+++ b/xen/arch/x86/hvm/vpt.c
@@ -28,7 +28,7 @@ 
 
 void hvm_init_guest_time(struct domain *d)
 {
-    struct pl_time *pl = &d->arch.hvm_domain.pl_time;
+    struct pl_time *pl = d->arch.hvm_domain.pl_time;
 
     spin_lock_init(&pl->pl_time_lock);
     pl->stime_offset = -(u64)get_s_time();
@@ -37,7 +37,7 @@  void hvm_init_guest_time(struct domain *d)
 
 u64 hvm_get_guest_time_fixed(struct vcpu *v, u64 at_tsc)
 {
-    struct pl_time *pl = &v->domain->arch.hvm_domain.pl_time;
+    struct pl_time *pl = v->domain->arch.hvm_domain.pl_time;
     u64 now;
 
     /* Called from device models shared with PV guests. Be careful. */
@@ -505,7 +505,7 @@  void pt_adjust_global_vcpu_target(struct vcpu *v)
     pt_adjust_vcpu(&vpit->pt0, v);
     spin_unlock(&vpit->lock);
 
-    pl_time = &v->domain->arch.hvm_domain.pl_time;
+    pl_time = v->domain->arch.hvm_domain.pl_time;
 
     spin_lock(&pl_time->vrtc.lock);
     pt_adjust_vcpu(&pl_time->vrtc.pt, v);
@@ -540,9 +540,9 @@  void pt_may_unmask_irq(struct domain *d, struct periodic_time *vlapic_pt)
     if ( d )
     {
         pt_resume(&d->arch.vpit.pt0);
-        pt_resume(&d->arch.hvm_domain.pl_time.vrtc.pt);
+        pt_resume(&d->arch.hvm_domain.pl_time->vrtc.pt);
         for ( i = 0; i < HPET_TIMER_NUM; i++ )
-            pt_resume(&d->arch.hvm_domain.pl_time.vhpet.pt[i]);
+            pt_resume(&d->arch.hvm_domain.pl_time->vhpet.pt[i]);
     }
 
     if ( vlapic_pt )
diff --git a/xen/arch/x86/time.c b/xen/arch/x86/time.c
index 988403a..5819b9f 100644
--- a/xen/arch/x86/time.c
+++ b/xen/arch/x86/time.c
@@ -799,7 +799,7 @@  static void __update_vcpu_system_time(struct vcpu *v, int force)
 
         if ( is_hvm_domain(d) )
         {
-            struct pl_time *pl = &v->domain->arch.hvm_domain.pl_time;
+            struct pl_time *pl = v->domain->arch.hvm_domain.pl_time;
 
             stime += pl->stime_offset + v->arch.hvm_vcpu.stime_offset;
             if ( stime >= 0 )
diff --git a/xen/include/asm-x86/hvm/domain.h b/xen/include/asm-x86/hvm/domain.h
index a8cc2ad..2446586 100644
--- a/xen/include/asm-x86/hvm/domain.h
+++ b/xen/include/asm-x86/hvm/domain.h
@@ -91,7 +91,7 @@  struct hvm_domain {
     /* Cached CF8 for guest PCI config cycles */
     uint32_t                pci_cf8;
 
-    struct pl_time         pl_time;
+    struct pl_time         *pl_time;
 
     struct hvm_io_handler *io_handler;
     unsigned int          io_handler_count;
diff --git a/xen/include/asm-x86/hvm/vpt.h b/xen/include/asm-x86/hvm/vpt.h
index 495d669..47e0ef1 100644
--- a/xen/include/asm-x86/hvm/vpt.h
+++ b/xen/include/asm-x86/hvm/vpt.h
@@ -139,6 +139,9 @@  struct pl_time {    /* platform time */
     /* Ensures monotonicity in appropriate timer modes. */
     uint64_t last_guest_time;
     spinlock_t pl_time_lock;
+    /* pl_time allocated dynamically, need to keep link to
+     * containing domain */
+    struct domain *domain;
 };
 
 void pt_save_timer(struct vcpu *v);