From patchwork Fri Sep 20 06:27:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suleiman Souhlal X-Patchwork-Id: 11153787 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 42A4276 for ; Fri, 20 Sep 2019 06:27:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 16C6520882 for ; Fri, 20 Sep 2019 06:27:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NwItX0CD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2394873AbfITG1f (ORCPT ); Fri, 20 Sep 2019 02:27:35 -0400 Received: from mail-ua1-f74.google.com ([209.85.222.74]:55008 "EHLO mail-ua1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2393085AbfITG1f (ORCPT ); Fri, 20 Sep 2019 02:27:35 -0400 Received: by mail-ua1-f74.google.com with SMTP id t16so1208684uae.21 for ; Thu, 19 Sep 2019 23:27:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=tCwobJ4sibSbRL27xRdckc8ojKP82LRYHZd+OSAZkFc=; b=NwItX0CDO8BWB79lwMnI/3DOOTIfkoSTBe9lNRVdjukKaRivMeIvDHkWK6yp5k8u63 hEB341B4VTKGuk0Lvr27YGvMrtomaF+Hyrf1GkLEAbTbLOqrmZJ2DoKmskO2dalXC4LO VkvtJWhOvXAdrCPnilFMiGLU5d8z2c/EzKx73QIgZIZCj6ho3L2rns7kqDWcEOpZJ0S3 I2rlk0NbDpwaXmpFVRfwpgPU9+6z/1Ikt+w1M0dnORZdINjQ7hJeksL1pHqC4JTK+93p MV6qrub4k3YpL665sq4WiT9lRI+/WzmAJNb0yculC6qO9xyPgphDfaa9D0uXUvuABYSU hb8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=tCwobJ4sibSbRL27xRdckc8ojKP82LRYHZd+OSAZkFc=; b=YlGoO5ls7Wx81FZLhyIbpXwIsFVvDQ17tdLVK+GTHarSuyjS0xnK2gmZTZGLcgS0hg HeGT1gy5bmSHK+oPQ8XQaARsOVVf04yXJ+DZ9jTV/dWcKsk3IVGHCStVDmZOxPoPx4t+ hNU5fEfP38L4c9etEcgPy55DpPSwTBuidXdBVSaGTvQkdVpq1psfSToxCbjEuHLiwGKB 8FrDb0HNK3X7FNL1+9LkwiYtjzTO1iKVnx3rl6j3Kc+oXyEG6res/+3eleSsGzKPXY2e 8TdBtddGismmeVayEl7txIZMuteoUpCO4P3d2aa5Z2NrqoOVgGU/IPK79pObzHUuO4JX dVSQ== X-Gm-Message-State: APjAAAVn8gz95SnyOZA4gAeIL+qACye/OTuiSY7VD23W94M9R6oMRpde CkeQDII0OxhSB+VBvI6ds9r56LSo+bAs+g== X-Google-Smtp-Source: APXvYqwkpI1UTuytyD+va/BNDVesQIbTDdeVsObF4jwY61WcWFOHCDgl8oIjbxKI5sjw4aUWlO8ab6fCNgQ4rA== X-Received: by 2002:ab0:5ac6:: with SMTP id x6mr8614021uae.7.1568960852015; Thu, 19 Sep 2019 23:27:32 -0700 (PDT) Date: Fri, 20 Sep 2019 15:27:12 +0900 In-Reply-To: <20190920062713.78503-1-suleiman@google.com> Message-Id: <20190920062713.78503-2-suleiman@google.com> Mime-Version: 1.0 References: <20190920062713.78503-1-suleiman@google.com> X-Mailer: git-send-email 2.23.0.351.gc4317032e6-goog Subject: [RFC 1/2] kvm: Mechanism to copy host timekeeping parameters into guest. From: Suleiman Souhlal To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de Cc: john.stultz@linaro.org, sboyd@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Suleiman Souhlal Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This is used to synchronize time between host and guest. The guest can request the (guest) physical address it wants the data in through the MSR_KVM_TIMEKEEPER_EN MSR. We maintain a shadow copy of the timekeeper that gets updated whenever the timekeeper gets updated, and then copied into the guest. It currently assumes the host timekeeper is "tsc". Signed-off-by: Suleiman Souhlal --- arch/x86/include/asm/kvm_host.h | 3 + arch/x86/include/asm/pvclock-abi.h | 27 ++++++ arch/x86/include/uapi/asm/kvm_para.h | 1 + arch/x86/kvm/x86.c | 121 +++++++++++++++++++++++++++ 4 files changed, 152 insertions(+) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index bdc16b0aa7c6..b1b4c3a80b8d 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -666,7 +666,10 @@ struct kvm_vcpu_arch { struct pvclock_vcpu_time_info hv_clock; unsigned int hw_tsc_khz; struct gfn_to_hva_cache pv_time; + struct gfn_to_hva_cache pv_timekeeper_g2h; + struct pvclock_timekeeper pv_timekeeper; bool pv_time_enabled; + bool pv_timekeeper_enabled; /* set guest stopped flag in pvclock flags field */ bool pvclock_set_guest_stopped_request; diff --git a/arch/x86/include/asm/pvclock-abi.h b/arch/x86/include/asm/pvclock-abi.h index 1436226efe3e..2809008b9b26 100644 --- a/arch/x86/include/asm/pvclock-abi.h +++ b/arch/x86/include/asm/pvclock-abi.h @@ -40,6 +40,33 @@ struct pvclock_wall_clock { u32 nsec; } __attribute__((__packed__)); +struct pvclock_read_base { + u64 mask; + u64 cycle_last; + u32 mult; + u32 shift; + u64 xtime_nsec; + u64 base; +} __attribute__((__packed__)); + +struct pvclock_timekeeper { + u64 gen; + u64 flags; + struct pvclock_read_base tkr_mono; + struct pvclock_read_base tkr_raw; + u64 xtime_sec; + u64 ktime_sec; + u64 wall_to_monotonic_sec; + u64 wall_to_monotonic_nsec; + u64 offs_real; + u64 offs_boot; + u64 offs_tai; + u64 raw_sec; + u64 tsc_offset; +} __attribute__((__packed__)); + +#define PVCLOCK_TIMEKEEPER_ENABLED (1 << 0) + #define PVCLOCK_TSC_STABLE_BIT (1 << 0) #define PVCLOCK_GUEST_STOPPED (1 << 1) /* PVCLOCK_COUNTS_FROM_ZERO broke ABI and can't be used anymore. */ diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/asm/kvm_para.h index 2a8e0b6b9805..3ebb1d87db3a 100644 --- a/arch/x86/include/uapi/asm/kvm_para.h +++ b/arch/x86/include/uapi/asm/kvm_para.h @@ -50,6 +50,7 @@ #define MSR_KVM_STEAL_TIME 0x4b564d03 #define MSR_KVM_PV_EOI_EN 0x4b564d04 #define MSR_KVM_POLL_CONTROL 0x4b564d05 +#define MSR_KVM_TIMEKEEPER_EN 0x4b564d06 struct kvm_steal_time { __u64 steal; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 91602d310a3f..06a940a74005 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -157,6 +157,8 @@ module_param(force_emulation_prefix, bool, S_IRUGO); int __read_mostly pi_inject_timer = -1; module_param(pi_inject_timer, bint, S_IRUGO | S_IWUSR); +static atomic_t pv_timekeepers_nr; + #define KVM_NR_SHARED_MSRS 16 struct kvm_shared_msrs_global { @@ -2621,6 +2623,16 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) break; } + case MSR_KVM_TIMEKEEPER_EN: + if (kvm_gfn_to_hva_cache_init(vcpu->kvm, + &vcpu->arch.pv_timekeeper_g2h, data, + sizeof(struct pvclock_timekeeper))) + vcpu->arch.pv_timekeeper_enabled = false; + else { + vcpu->arch.pv_timekeeper_enabled = true; + atomic_inc(&pv_timekeepers_nr); + } + break; case MSR_KVM_ASYNC_PF_EN: if (kvm_pv_enable_async_pf(vcpu, data)) return 1; @@ -6965,6 +6977,109 @@ static struct perf_guest_info_callbacks kvm_guest_cbs = { .handle_intel_pt_intr = kvm_handle_intel_pt_intr, }; +static DEFINE_SPINLOCK(shadow_pvtk_lock); +static struct pvclock_timekeeper shadow_pvtk; + +static void +pvclock_copy_read_base(struct pvclock_read_base *pvtkr, + struct tk_read_base *tkr) +{ + pvtkr->cycle_last = tkr->cycle_last; + pvtkr->mult = tkr->mult; + pvtkr->shift = tkr->shift; + pvtkr->mask = tkr->mask; + pvtkr->xtime_nsec = tkr->xtime_nsec; + pvtkr->base = tkr->base; +} + +static void +kvm_copy_into_pvtk(struct kvm_vcpu *vcpu) +{ + struct pvclock_timekeeper *pvtk; + unsigned long flags; + + if (!vcpu->arch.pv_timekeeper_enabled) + return; + + pvtk = &vcpu->arch.pv_timekeeper; + if (pvclock_gtod_data.clock.vclock_mode == VCLOCK_TSC) { + pvtk->flags |= PVCLOCK_TIMEKEEPER_ENABLED; + spin_lock_irqsave(&shadow_pvtk_lock, flags); + pvtk->tkr_mono = shadow_pvtk.tkr_mono; + pvtk->tkr_raw = shadow_pvtk.tkr_raw; + + pvtk->xtime_sec = shadow_pvtk.xtime_sec; + pvtk->ktime_sec = shadow_pvtk.ktime_sec; + pvtk->wall_to_monotonic_sec = + shadow_pvtk.wall_to_monotonic_sec; + pvtk->wall_to_monotonic_nsec = + shadow_pvtk.wall_to_monotonic_nsec; + pvtk->offs_real = shadow_pvtk.offs_real; + pvtk->offs_boot = shadow_pvtk.offs_boot; + pvtk->offs_tai = shadow_pvtk.offs_tai; + pvtk->raw_sec = shadow_pvtk.raw_sec; + spin_unlock_irqrestore(&shadow_pvtk_lock, flags); + + pvtk->tsc_offset = kvm_x86_ops->read_l1_tsc_offset(vcpu); + } else + pvtk->flags &= ~PVCLOCK_TIMEKEEPER_ENABLED; + + BUILD_BUG_ON(offsetof(struct pvclock_timekeeper, gen) != 0); + + /* + * Make the gen count odd to indicate we are in the process of + * updating. + */ + vcpu->arch.pv_timekeeper.gen++; + vcpu->arch.pv_timekeeper.gen |= 1; + + /* + * See comment in kvm_guest_time_update() for why we have to do + * multiple writes. + */ + kvm_write_guest_cached(vcpu->kvm, &vcpu->arch.pv_timekeeper_g2h, + &vcpu->arch.pv_timekeeper, sizeof(vcpu->arch.pv_timekeeper.gen)); + + smp_wmb(); + + kvm_write_guest_cached(vcpu->kvm, &vcpu->arch.pv_timekeeper_g2h, + &vcpu->arch.pv_timekeeper, sizeof(vcpu->arch.pv_timekeeper)); + + smp_wmb(); + + vcpu->arch.pv_timekeeper.gen++; + + kvm_write_guest_cached(vcpu->kvm, &vcpu->arch.pv_timekeeper_g2h, + &vcpu->arch.pv_timekeeper, sizeof(vcpu->arch.pv_timekeeper.gen)); +} + +static void +update_shadow_pvtk(struct timekeeper *tk) +{ + struct pvclock_timekeeper *pvtk; + unsigned long flags; + + pvtk = &shadow_pvtk; + + if (atomic_read(&pv_timekeepers_nr) == 0 || + pvclock_gtod_data.clock.vclock_mode != VCLOCK_TSC) + return; + + spin_lock_irqsave(&shadow_pvtk_lock, flags); + pvclock_copy_read_base(&pvtk->tkr_mono, &tk->tkr_mono); + pvclock_copy_read_base(&pvtk->tkr_raw, &tk->tkr_raw); + + pvtk->xtime_sec = tk->xtime_sec; + pvtk->ktime_sec = tk->ktime_sec; + pvtk->wall_to_monotonic_sec = tk->wall_to_monotonic.tv_sec; + pvtk->wall_to_monotonic_nsec = tk->wall_to_monotonic.tv_nsec; + pvtk->offs_real = tk->offs_real; + pvtk->offs_boot = tk->offs_boot; + pvtk->offs_tai = tk->offs_tai; + pvtk->raw_sec = tk->raw_sec; + spin_unlock_irqrestore(&shadow_pvtk_lock, flags); +} + #ifdef CONFIG_X86_64 static void pvclock_gtod_update_fn(struct work_struct *work) { @@ -6993,6 +7108,7 @@ static int pvclock_gtod_notify(struct notifier_block *nb, unsigned long unused, struct timekeeper *tk = priv; update_pvclock_gtod(tk); + update_shadow_pvtk(tk); /* disable master clock if host does not trust, or does not * use, TSC based clocksource. @@ -7809,6 +7925,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) bool req_immediate_exit = false; + kvm_copy_into_pvtk(vcpu); + if (kvm_request_pending(vcpu)) { if (kvm_check_request(KVM_REQ_GET_VMCS12_PAGES, vcpu)) kvm_x86_ops->get_vmcs12_pages(vcpu); @@ -8891,6 +9009,9 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu) kvmclock_reset(vcpu); + if (vcpu->arch.pv_timekeeper_enabled) + atomic_dec(&pv_timekeepers_nr); + kvm_x86_ops->vcpu_free(vcpu); free_cpumask_var(wbinvd_dirty_mask); } From patchwork Fri Sep 20 06:27:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suleiman Souhlal X-Patchwork-Id: 11153789 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C3644112B for ; Fri, 20 Sep 2019 06:27:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8F598208C3 for ; Fri, 20 Sep 2019 06:27:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bsYHpr0X" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2394886AbfITG1l (ORCPT ); Fri, 20 Sep 2019 02:27:41 -0400 Received: from mail-qt1-f202.google.com ([209.85.160.202]:47644 "EHLO mail-qt1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2394879AbfITG1l (ORCPT ); Fri, 20 Sep 2019 02:27:41 -0400 Received: by mail-qt1-f202.google.com with SMTP id p56so6879696qtj.14 for ; Thu, 19 Sep 2019 23:27:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=CucnbTzStvcSIAgRMuzfV7pL2ynq6M2/TlEKsf9uX4w=; b=bsYHpr0XFWPiKYjovv3Vm02n9BUgmNaNZSZg2VDIoU0BYcP/TrtJXaCrFnMpsHIooY Eq+66t/3FUdko0cWY/eVQqH7ZHIl/TbBH43W3y8uOI8q4+GzmdbsWtMwRQMYwWC0446W vyPGQ+WE3Flzx0A848WSr/GfslkZ5VMhlUEzSuhxOWPZMSUUGpzcseLgGAszQtdl1i2q FcDiBM5wyDngAUMbr39JR4rzYscQut7vhw5HOqVoI04AHgaT7sIm5d992cfN39cuOiqS nIHUUQk8XwxEw2D5/sa+usYVZ17EVjNgTI4awW1F2mmtnSRJlTHc+e+XgyAe01l6q8eG D5bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=CucnbTzStvcSIAgRMuzfV7pL2ynq6M2/TlEKsf9uX4w=; b=j3mgHr3FG9OtIxSt7WTbnKKFRSpbZM+pZ+O1+3AH6ogTeE+VJvvjMwKk5ijB2X2AD7 2Gf2PCFQxvjfo5z0IQNsbEfaDZrinHCR0Ud/m0LgnCAiTVrhzjxqdSbdnl5BzlS0/vVV hKFgE7pfsjOi23Rb6PqzltJaf2aazTIU7mkWx6OW0DyL+2Mr7zTjU97wYeayzFgNDHf2 4iX9Aiu5M4IZGhZtbcUBFm8yYlTNmGFQzJjtf5RLwBo4GXJotHeBrTg1jLymFZX1AmKS AhchN6VATaXSdTbRVS5IEvdn1ZJ2scd3C9o5vQhApL6MGd7Yis0Af5fSXSRlctJ4cPIL U42g== X-Gm-Message-State: APjAAAXiPh9IhQrXe7ovcU+B6WP4LSXd/ErKn8ChiAKbhJZ1PTf46ueu ZJAOBIlk93ktquW8FaPuQbL04hJasTldVA== X-Google-Smtp-Source: APXvYqzrg03IpbCj13IBkNlMEaYZURlrk44PPVVELnwg1h+eklx0eOFDcPBCFusgE42xxp9jxy31c5/Lv2VhsA== X-Received: by 2002:ac8:47d0:: with SMTP id d16mr1632218qtr.176.1568960859307; Thu, 19 Sep 2019 23:27:39 -0700 (PDT) Date: Fri, 20 Sep 2019 15:27:13 +0900 In-Reply-To: <20190920062713.78503-1-suleiman@google.com> Message-Id: <20190920062713.78503-3-suleiman@google.com> Mime-Version: 1.0 References: <20190920062713.78503-1-suleiman@google.com> X-Mailer: git-send-email 2.23.0.351.gc4317032e6-goog Subject: [RFC 2/2] x86/kvmclock: Use host timekeeping. From: Suleiman Souhlal To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de Cc: john.stultz@linaro.org, sboyd@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Suleiman Souhlal Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When CONFIG_KVMCLOCK_HOST_TIMEKEEPING is enabled, and the host supports it, update our timekeeping parameters to be the same as the host. This lets us have our time synchronized with the host's, even in the presence of host NTP or suspend. When enabled, kvmclock uses raw tsc instead of pvclock. When enabled, syscalls that can change time, such as settimeofday(2) or adj_timex(2) are disabled in the guest. Signed-off-by: Suleiman Souhlal --- arch/x86/Kconfig | 9 +++ arch/x86/include/asm/kvmclock.h | 2 + arch/x86/kernel/kvmclock.c | 127 +++++++++++++++++++++++++++++++- kernel/time/timekeeping.c | 21 ++++++ 4 files changed, 155 insertions(+), 4 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 4195f44c6a09..37299377d9d7 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -837,6 +837,15 @@ config PARAVIRT_TIME_ACCOUNTING config PARAVIRT_CLOCK bool +config KVMCLOCK_HOST_TIMEKEEPING + bool "kvmclock uses host timekeeping" + depends on KVM_GUEST + ---help--- + Select this option to make the guest use the same timekeeping + parameters as the host. This means that time will be almost + exactly the same between the two. Only works if the host uses "tsc" + clocksource. + config JAILHOUSE_GUEST bool "Jailhouse non-root cell support" depends on X86_64 && PCI diff --git a/arch/x86/include/asm/kvmclock.h b/arch/x86/include/asm/kvmclock.h index eceea9299097..cad7eae62b54 100644 --- a/arch/x86/include/asm/kvmclock.h +++ b/arch/x86/include/asm/kvmclock.h @@ -4,4 +4,6 @@ extern struct clocksource kvm_clock; +bool kvm_clock_copy_into_tk(struct timekeeper *tk); + #endif /* _ASM_X86_KVM_CLOCK_H */ diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c index 904494b924c1..e3a33dcaffc8 100644 --- a/arch/x86/kernel/kvmclock.c +++ b/arch/x86/kernel/kvmclock.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -29,6 +30,9 @@ static int msr_kvm_system_time __ro_after_init = MSR_KVM_SYSTEM_TIME; static int msr_kvm_wall_clock __ro_after_init = MSR_KVM_WALL_CLOCK; static u64 kvm_sched_clock_offset __ro_after_init; +static bool pv_timekeeper_enabled; +static bool pv_timekeeper_present; + static int __init parse_no_kvmclock(char *arg) { kvmclock = 0; @@ -54,6 +58,8 @@ static struct pvclock_wall_clock wall_clock __bss_decrypted; static DEFINE_PER_CPU(struct pvclock_vsyscall_time_info *, hv_clock_per_cpu); static struct pvclock_vsyscall_time_info *hvclock_mem; +struct pvclock_timekeeper pv_timekeeper; + static inline struct pvclock_vcpu_time_info *this_cpu_pvti(void) { return &this_cpu_read(hv_clock_per_cpu)->pvti; @@ -94,7 +100,10 @@ static u64 kvm_clock_read(void) static u64 kvm_clock_get_cycles(struct clocksource *cs) { - return kvm_clock_read(); + if (pv_timekeeper_present) + return rdtsc_ordered(); + else + return kvm_clock_read(); } static u64 kvm_sched_clock_read(void) @@ -159,9 +168,26 @@ bool kvm_check_and_clear_guest_paused(void) return ret; } +int +kvm_clock_enable(struct clocksource *cs) +{ + if (pv_timekeeper_present) + pv_timekeeper_enabled = 1; + + return 0; +} + +void +kvm_clock_disable(struct clocksource *cs) +{ + pv_timekeeper_enabled = 0; +} + struct clocksource kvm_clock = { .name = "kvm-clock", .read = kvm_clock_get_cycles, + .enable = kvm_clock_enable, + .disable = kvm_clock_disable, .rating = 400, .mask = CLOCKSOURCE_MASK(64), .flags = CLOCK_SOURCE_IS_CONTINUOUS, @@ -197,6 +223,86 @@ static void kvm_setup_secondary_clock(void) } #endif +#ifdef CONFIG_KVMCLOCK_HOST_TIMEKEEPING +static void +pvclock_copy_into_read_base(struct pvclock_timekeeper *pvtk, + struct tk_read_base *tkr, struct pvclock_read_base *pvtkr) +{ + int shift_diff; + + tkr->mask = pvtkr->mask; + tkr->cycle_last = pvtkr->cycle_last + pvtk->tsc_offset; + tkr->mult = pvtkr->mult; + shift_diff = tkr->shift - pvtkr->shift; + tkr->shift = pvtkr->shift; + tkr->xtime_nsec = pvtkr->xtime_nsec; + tkr->base = pvtkr->base; +} + +static u64 +pvtk_read_begin(struct pvclock_timekeeper *pvtk) +{ + u64 gen; + + gen = pvtk->gen & ~1; + /* Make sure that the gen count is read before the data. */ + virt_rmb(); + + return gen; +} + +static bool +pvtk_read_retry(struct pvclock_timekeeper *pvtk, u64 gen) +{ + /* Make sure that the gen count is re-read after the data. */ + virt_rmb(); + return unlikely(gen != pvtk->gen); +} + +bool +kvm_clock_copy_into_tk(struct timekeeper *tk) +{ + struct pvclock_timekeeper *pvtk; + u64 gen; + + if (!pv_timekeeper_enabled) + return false; + + pvtk = &pv_timekeeper; + do { + gen = pvtk_read_begin(pvtk); + if (!(pv_timekeeper.flags & PVCLOCK_TIMEKEEPER_ENABLED)) + return false; + + pvclock_copy_into_read_base(pvtk, &tk->tkr_mono, + &pvtk->tkr_mono); + pvclock_copy_into_read_base(pvtk, &tk->tkr_raw, &pvtk->tkr_raw); + + tk->xtime_sec = pvtk->xtime_sec; + tk->ktime_sec = pvtk->ktime_sec; + tk->wall_to_monotonic.tv_sec = pvtk->wall_to_monotonic_sec; + tk->wall_to_monotonic.tv_nsec = pvtk->wall_to_monotonic_nsec; + tk->offs_real = pvtk->offs_real; + tk->offs_boot = pvtk->offs_boot; + tk->offs_tai = pvtk->offs_tai; + tk->raw_sec = pvtk->raw_sec; + } while (pvtk_read_retry(pvtk, gen)); + + return true; +} + +static void +kvm_register_timekeeper(void) +{ + unsigned long pa; + + pa = __pa(&pv_timekeeper); + wrmsrl(MSR_KVM_TIMEKEEPER_EN, pa); + if (pv_timekeeper.flags & PVCLOCK_TIMEKEEPER_ENABLED) + pv_timekeeper_present = 1; +} +#endif /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ + /* * After the clock is registered, the host will keep writing to the * registered memory location. If the guest happens to shutdown, this memory @@ -271,8 +377,10 @@ static int __init kvm_setup_vsyscall_timeinfo(void) flags = pvclock_read_flags(&hv_clock_boot[0].pvti); if (!(flags & PVCLOCK_TSC_STABLE_BIT)) return 0; - - kvm_clock.archdata.vclock_mode = VCLOCK_PVCLOCK; + if (pv_timekeeper_present) + kvm_clock.archdata.vclock_mode = VCLOCK_TSC; + else + kvm_clock.archdata.vclock_mode = VCLOCK_PVCLOCK; #endif kvmclock_init_mem(); @@ -312,6 +420,10 @@ void __init kvmclock_init(void) if (!kvm_para_available() || !kvmclock) return; +#ifdef CONFIG_KVMCLOCK_HOST_TIMEKEEPING + kvm_register_timekeeper(); +#endif + if (kvm_para_has_feature(KVM_FEATURE_CLOCKSOURCE2)) { msr_kvm_system_time = MSR_KVM_SYSTEM_TIME_NEW; msr_kvm_wall_clock = MSR_KVM_WALL_CLOCK_NEW; @@ -360,11 +472,18 @@ void __init kvmclock_init(void) * can use TSC as clocksource. * */ +#ifndef CONFIG_KVMCLOCK_HOST_TIMEKEEPING if (boot_cpu_has(X86_FEATURE_CONSTANT_TSC) && boot_cpu_has(X86_FEATURE_NONSTOP_TSC) && !check_tsc_unstable()) kvm_clock.rating = 299; +#endif + + if (pv_timekeeper_present) + clocksource_register_khz(&kvm_clock, + pvclock_tsc_khz(this_cpu_pvti())); + else + clocksource_register_hz(&kvm_clock, NSEC_PER_SEC); - clocksource_register_hz(&kvm_clock, NSEC_PER_SEC); pv_info.name = "KVM"; } diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index ca69290bee2a..c6e4785babb1 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -23,6 +23,10 @@ #include #include +#ifdef CONFIG_KVMCLOCK_HOST_TIMEKEEPING +#include +#endif + #include "tick-internal.h" #include "ntp_internal.h" #include "timekeeping_internal.h" @@ -1223,6 +1227,7 @@ EXPORT_SYMBOL_GPL(get_device_system_crosststamp); */ int do_settimeofday64(const struct timespec64 *ts) { +#ifndef CONFIG_KVMCLOCK_HOST_TIMEKEEPING struct timekeeper *tk = &tk_core.timekeeper; struct timespec64 ts_delta, xt; unsigned long flags; @@ -1261,9 +1266,13 @@ int do_settimeofday64(const struct timespec64 *ts) audit_tk_injoffset(ts_delta); return ret; +#else /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ + return -EINVAL; +#endif /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ } EXPORT_SYMBOL(do_settimeofday64); +#ifndef CONFIG_KVMCLOCK_HOST_TIMEKEEPING /** * timekeeping_inject_offset - Adds or subtracts from the current time. * @tv: pointer to the timespec variable containing the offset @@ -1307,6 +1316,7 @@ static int timekeeping_inject_offset(const struct timespec64 *ts) return ret; } +#endif /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ /* * Indicates if there is an offset between the system clock and the hardware @@ -1332,6 +1342,7 @@ int persistent_clock_is_local; */ void timekeeping_warp_clock(void) { +#ifndef CONFIG_KVMCLOCK_HOST_TIMEKEEPING if (sys_tz.tz_minuteswest != 0) { struct timespec64 adjust; @@ -1340,6 +1351,7 @@ void timekeeping_warp_clock(void) adjust.tv_nsec = 0; timekeeping_inject_offset(&adjust); } +#endif /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ } /** @@ -2107,6 +2119,9 @@ static void timekeeping_advance(enum timekeeping_adv_mode mode) clock_set |= accumulate_nsecs_to_secs(tk); write_seqcount_begin(&tk_core.seq); +#ifdef CONFIG_KVMCLOCK_HOST_TIMEKEEPING + kvm_clock_copy_into_tk(tk); +#endif /* * Update the real timekeeper. * @@ -2241,6 +2256,7 @@ ktime_t ktime_get_update_offsets_now(unsigned int *cwsseq, ktime_t *offs_real, return base; } +#ifndef CONFIG_KVMCLOCK_HOST_TIMEKEEPING /** * timekeeping_validate_timex - Ensures the timex is ok for use in do_adjtimex */ @@ -2305,6 +2321,7 @@ static int timekeeping_validate_timex(const struct __kernel_timex *txc) return 0; } +#endif /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ /** @@ -2312,6 +2329,7 @@ static int timekeeping_validate_timex(const struct __kernel_timex *txc) */ int do_adjtimex(struct __kernel_timex *txc) { +#ifndef CONFIG_KVMCLOCK_HOST_TIMEKEEPING struct timekeeper *tk = &tk_core.timekeeper; struct audit_ntp_data ad; unsigned long flags; @@ -2368,6 +2386,9 @@ int do_adjtimex(struct __kernel_timex *txc) ntp_notify_cmos_timer(); return ret; +#else /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ + return -EINVAL; +#endif /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ } #ifdef CONFIG_NTP_PPS