From patchwork Mon Aug 17 08:41:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11717469 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 947E3722 for ; Mon, 17 Aug 2020 09:12:30 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6A09A206C0 for ; Mon, 17 Aug 2020 09:12:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="vkg2mWCi" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6A09A206C0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FOu/SHEOr6q/uKuxrPW8jJvrBOLCiGs0LYhovu7CeAM=; b=vkg2mWCiAcRc+EA7cwafTxCBW GT/oX6DLHI51X1gLC+l9nb4N1j3l+3Gbyd7BAI4D2iFuSMcfgHQD+qUGFTTeY3Th0rHwcBd94Detf Q1W0yOatoejPPRG9ttE4nUNpdts7hTD6UExm848EfzCY2UW5yRwo5Qt8ZVSt1ZH+w9pcAERFJ0hhQ NOzl+PSmx6fZSBh8gvjc5cBOrL+Lv34rqsPoButOLV3YLD3gO8crZaBw+L4yxzEpc+HkOF2G7wCRA YsqswiAMOP6cvnsfVE2Dj1H62JH03YKvwT61HCa4KV3Yg8PTh1URMhQNrkuPDf3iFnmZRFdoPqw5s Dq39yiMMw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7b9h-0000PS-Im; Mon, 17 Aug 2020 09:10:13 +0000 Received: from szxga05-in.huawei.com ([45.249.212.191] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7aht-0002Cd-3E for linux-arm-kernel@lists.infradead.org; Mon, 17 Aug 2020 08:41:34 +0000 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 8E9C953B776B2C53A1D1; Mon, 17 Aug 2020 16:41:22 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.487.0; Mon, 17 Aug 2020 16:41:12 +0800 From: Keqian Zhu To: , , , Subject: [RFC PATCH 1/5] KVM: arm64: Document pvtime LPT interface Date: Mon, 17 Aug 2020 16:41:06 +0800 Message-ID: <20200817084110.2672-2-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200817084110.2672-1-zhukeqian1@huawei.com> References: <20200817084110.2672-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200817_044129_673056_35A3A1F6 X-CRM114-Status: GOOD ( 16.83 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.191 listed in list.dnswl.org] 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.191 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Marc Zyngier , Keqian Zhu , Steven Price , James Morse , Catalin Marinas , wanghaibin.wang@huawei.com, Will Deacon Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The "Arm Paravirtualized Time for Arm-Base Systems" specification DEN 0057A just contains "Stolen Time" now, as the details of "Live Physical Time" have not been fully agreed. This introduces a new LPT structure with more understandable fields, as scale_mult and rscale_mult are symmetrical. Signed-off-by: Steven Price Signed-off-by: Keqian Zhu --- Documentation/virt/kvm/arm/pvtime.rst | 78 ++++++++++++++++++++++++++++++----- Documentation/virt/kvm/devices/vm.rst | 30 ++++++++++++++ 2 files changed, 97 insertions(+), 11 deletions(-) diff --git a/Documentation/virt/kvm/arm/pvtime.rst b/Documentation/virt/kvm/arm/pvtime.rst index 94bffe2..fd11915 100644 --- a/Documentation/virt/kvm/arm/pvtime.rst +++ b/Documentation/virt/kvm/arm/pvtime.rst @@ -8,14 +8,17 @@ support for AArch64 guests: https://developer.arm.com/docs/den0057/a -KVM/arm64 implements the stolen time part of this specification by providing -some hypervisor service calls to support a paravirtualized guest obtaining a -view of the amount of time stolen from its execution. +KVM/arm64 implements the stolen time and live physical time (LPT) parts of this +specification by providing some hypervisor service calls to a paravirtualized +guest. With stolen time support, guest can obtain the amount of time stolen +from its execution. LPT represents time during which the guest is running and +it realizes cross-native-frequency migrations. -Two new SMCCC compatible hypercalls are defined: +Three new SMCCC compatible hypercalls are defined: * PV_TIME_FEATURES: 0xC5000020 * PV_TIME_ST: 0xC5000021 +* PV_TIME_LPT: 0xC5000022 These are only available in the SMC64/HVC64 calling convention as paravirtualized time is not available to 32 bit Arm guests. The existence of @@ -26,7 +29,8 @@ PV_TIME_FEATURES ============= ======== ========== Function ID: (uint32) 0xC5000020 PV_call_id: (uint32) The function to query for support. - Currently only PV_TIME_ST is supported. + Currently only PV_TIME_ST and PV_TIME_LPT are + supported. Return value: (int64) NOT_SUPPORTED (-1) or SUCCESS (0) if the relevant PV-time feature is supported by the hypervisor. ============= ======== ========== @@ -39,17 +43,23 @@ PV_TIME_ST NOT_SUPPORTED (-1) ============= ======== ========== -The IPA returned by PV_TIME_ST should be mapped by the guest as normal memory -with inner and outer write back caching attributes, in the inner shareable -domain. A total of 16 bytes from the IPA returned are guaranteed to be -meaningfully filled by the hypervisor (see structure below). +PV_TIME_LPT + ============= ======== ========== + Function ID: (uint32) 0xC5000022 + Return value: (int64) IPA of the LPT data structure for this guest. + On failure: + NOT_SUPPORTED (-1) + ============= ======== ========== -PV_TIME_ST returns the structure for the calling VCPU. +The IPA returned by PV_TIME_ST and PV_TIME_LPT should be mapped by the guest as +normal memory with inner and outer write back caching attributes, in the inner +shareable domain. Stolen Time ----------- -The structure pointed to by the PV_TIME_ST hypercall is as follows: +A total of 16 bytes from the IPA returned are guaranteed to be meaningfully +filled by the hypervisor. The structure description is as follows: +-------------+-------------+-------------+----------------------------+ | Field | Byte Length | Byte Offset | Description | @@ -78,3 +88,49 @@ the region using 64k pages and avoids conflicting attributes with other memory. For the user space interface see Documentation/virt/kvm/devices/vcpu.rst section "3. GROUP: KVM_ARM_VCPU_PVTIME_CTRL". + +Live Physical Time +----------- + +A total of 48 bytes from the IPA returned are guaranteed to be meaningfully +filled by the hypervisor. The structure description is as follows: + ++-----------------+-------------+-------------+----------------------------+ +| Field | Byte Length | Byte Offset | Description | ++=================+=============+=============+============================+ +| Revision | 4 | 0 | Must be 0 for version 1.0 | ++-----------------+-------------+-------------+----------------------------+ +| Attributes | 4 | 4 | Must be 0 | ++-----------------+-------------+-------------+----------------------------+ +| sequence_number | 8 | 8 | Bit 0: reserved | +| | | | Bits 1-63: number of runs, | +| | | | including the initial run. | ++-----------------+-------------+-------------+----------------------------| +| native_freq | 4 | 16 | Native frequency | ++-----------------+-------------+-------------+----------------------------| +| pv_freq | 4 | 20 | Paravirtualized frequency | +| | | | seen by guest | ++-----------------+-------------+-------------+----------------------------| +| scale_mult | 8 | 24 | Multiplier to scale native | +| | | | cycles to PV cycles | ++-----------------+-------------+-------------+----------------------------| +| rscale_mult | 8 | 32 | Multiplier to reversely | +| | | | scale PV cycles to native | +| | | | cycles | ++-----------------+-------------+-------------+----------------------------| +| fracbits | 4 | 40 | The bits of fraction of | +| | | | scale_mult | ++-----------------+-------------+-------------+----------------------------| +| rfracbits | 4 | 44 | The bits of fraction of | +| | | | rscale_mult | ++-----------------+-------------+-------------+----------------------------| + +All values in the structure are stored little-endian. + +The structure will be updated by the hypervisor prior to scheduling VCPUs. It +will be present within a reserved region of the normal memory given to the +guest. The guest should not attempt to write into this memory. There is a +structure per guest. + +For the user space interface see Documentation/virt/kvm/devices/vm.rst +section "6. GROUP: KVM_ARM_VM_PVTIME_LPT_CTRL". diff --git a/Documentation/virt/kvm/devices/vm.rst b/Documentation/virt/kvm/devices/vm.rst index 0aa5b1c..7e5a189 100644 --- a/Documentation/virt/kvm/devices/vm.rst +++ b/Documentation/virt/kvm/devices/vm.rst @@ -314,3 +314,33 @@ Allows userspace to query the status of migration mode. if it is enabled :Returns: -EFAULT if the given address is not accessible from kernel space; 0 in case of success. + +6. GROUP: KVM_ARM_VM_PVTIME_LPT_CTRL +==================================== + +:Architectures: ARM64 + +6.1. ATTRIBUTE: KVM_ARM_VM_PVTIME_LPT_IPA (r/w) +----------------------------------------------- + +Specifies the base address of the live physical time (LPT) structure for this +guest. The base address must be 64 byte aligned and exist within a valid guest +memory region. See Documentation/virt/kvm/arm/pvtime.rst for more information +including the layout of the LPT structure. + +Parameters: 64-bit base address +Returns: -ENXIO: LPT not implemented + -EEXIST: Base address already set for this guest + -EINVAL: Base address not 64 byte aligned or not within guest memory + +6.2. ATTRIBUTE: KVM_ARM_VM_PVTIME_LPT_FREQ (r/w) +------------------------------------------------ + +Specifies the paravirtualized (PV) frequency for this guest. The PV frequency +is independent with native frequency, so we can support cross-native-frequency +migration. + +Parameters: 32-bit PV frequency +Returns: -ENXIO: LPT not implemented + -EEXIST: PV frequency already set for this guest + -EINVAL: PV frequency is zero From patchwork Mon Aug 17 08:41:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11717463 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F239722 for ; Mon, 17 Aug 2020 09:11:17 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 284F22063A for ; Mon, 17 Aug 2020 09:11:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="2hLhTWD4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 284F22063A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=LzxUhhR1IjlA8zpRB506Vys6zKjL0d4snxTrIheEJ8k=; b=2hLhTWD4u2szmXzjoofJUqQIO zGt36HjQmPsX1nI5ahvqsK7K1J18HC7kNJLNgiHJnCPadnTzcmAWb5IzhyKT4dadJIz7+UM/chvtW PMMcvIFAJsd7ScgA/ijzRdF+m2204Fqsq8Ml1oiCyrMThdj2jA04ka2oDOGKPccCbEvDFB0l0B0ZD 8gbERtBc6goJzuxtZ4wEIvsEKyb4YGhOJsvRXDYbroM9JSzrz+lVQJ44HzsNR0xM1q3GXepUx54l0 57LVPk+TEXsE0/o2OSu8pLG1QRcn5sX0I/9EECh/mv10IX5Phsmtcrl+QfGbi0qdVkkv/378T44vk r7kw0Aasg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7b8j-0008MC-8p; Mon, 17 Aug 2020 09:09:13 +0000 Received: from szxga07-in.huawei.com ([45.249.212.35] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7ahr-0002CM-4r for linux-arm-kernel@lists.infradead.org; Mon, 17 Aug 2020 08:41:30 +0000 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 795BABA4CBE70EFE0EF9; Mon, 17 Aug 2020 16:41:22 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.487.0; Mon, 17 Aug 2020 16:41:13 +0800 From: Keqian Zhu To: , , , Subject: [RFC PATCH 2/5] KVM: arm64: Support Live Physical Time reporting Date: Mon, 17 Aug 2020 16:41:07 +0800 Message-ID: <20200817084110.2672-3-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200817084110.2672-1-zhukeqian1@huawei.com> References: <20200817084110.2672-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200817_044127_909926_FF15B721 X-CRM114-Status: GOOD ( 20.64 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.35 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.35 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Marc Zyngier , Keqian Zhu , Steven Price , James Morse , Catalin Marinas , wanghaibin.wang@huawei.com, Will Deacon Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Provide a method for a guest to derive a paravirtualized counter/timer which isn't dependent on the host's counter frequency. This allows a guest to be migrated onto a new host which doesn't have the same frequency without the virtual counter being disturbed. The host provides a shared structure which contains coefficients that can be used to map the real counter from the host (the Arm "virtual counter") to a paravirtualized view of time. On migration the new host updates the coefficients to ensure that the guests view of time (after using the coefficients) doesn't change and that the derived counter progresses at the same real frequency. Signed-off-by: Steven Price Signed-off-by: Keqian Zhu --- arch/arm64/include/asm/kvm_host.h | 10 +++ arch/arm64/kvm/arm.c | 7 +++ arch/arm64/kvm/hypercalls.c | 5 ++ arch/arm64/kvm/pvtime.c | 125 +++++++++++++++++++++++++++++++++++++- 4 files changed, 146 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index e21d4a0..0c6a564 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -95,6 +95,13 @@ struct kvm_arch { * supported. */ bool return_nisv_io_abort_to_user; + + /* Guest PV Live Physical Time state */ + struct { + u32 fpv; /* PV frequency */ + gpa_t base; /* Base IPA of shared structure */ + bool updated; /* Indicate whether it is updated by KVM */ + } lpt; }; #define KVM_NR_MEM_OBJS 40 @@ -506,6 +513,9 @@ int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run, gpa_t kvm_init_stolen_time(struct kvm_vcpu *vcpu); void kvm_update_stolen_time(struct kvm_vcpu *vcpu); +gpa_t kvm_init_lpt_time(struct kvm *kvm); +int kvm_update_lpt_time(struct kvm *kvm); + int kvm_arm_pvtime_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr); int kvm_arm_pvtime_get_attr(struct kvm_vcpu *vcpu, diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 90cb905..671f1461 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -135,6 +135,9 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) /* The maximum number of VCPUs is limited by the host's GIC model */ kvm->arch.max_vcpus = kvm_arm_default_max_vcpus(); + /* Should be setup by userspace before guest run */ + kvm->arch.lpt.base = GPA_INVALID; + return ret; out_free_stage2_pgd: kvm_free_stage2_pgd(kvm); @@ -528,6 +531,10 @@ static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu) vcpu->arch.has_run_once = true; + ret = kvm_update_lpt_time(kvm); + if (ret) + return ret; + if (likely(irqchip_in_kernel(kvm))) { /* * Map the VGIC hardware resources before running a vcpu the diff --git a/arch/arm64/kvm/hypercalls.c b/arch/arm64/kvm/hypercalls.c index 550dfa3..254491b 100644 --- a/arch/arm64/kvm/hypercalls.c +++ b/arch/arm64/kvm/hypercalls.c @@ -62,6 +62,11 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) if (gpa != GPA_INVALID) val = gpa; break; + case ARM_SMCCC_HV_PV_TIME_LPT: + gpa = kvm_init_lpt_time(vcpu->kvm); + if (gpa != GPA_INVALID) + val = gpa; + break; default: return kvm_psci_call(vcpu); } diff --git a/arch/arm64/kvm/pvtime.c b/arch/arm64/kvm/pvtime.c index 2b24e7f..24131ca 100644 --- a/arch/arm64/kvm/pvtime.c +++ b/arch/arm64/kvm/pvtime.c @@ -43,7 +43,9 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu) switch (feature) { case ARM_SMCCC_HV_PV_TIME_FEATURES: case ARM_SMCCC_HV_PV_TIME_ST: - val = SMCCC_RET_SUCCESS; + case ARM_SMCCC_HV_PV_TIME_LPT: + if (vcpu->kvm->arch.lpt.updated) + val = SMCCC_RET_SUCCESS; break; } @@ -134,3 +136,124 @@ int kvm_arm_pvtime_has_attr(struct kvm_vcpu *vcpu, } return -ENXIO; } + +static int pvclock_lpt_update_vtimer(struct kvm *kvm, + struct pvclock_vm_lpt_time *pvclock) +{ + u32 current_freq = arch_timer_get_rate(); + u64 current_time = kvm_phys_timer_read(); + u32 previous_freq; + struct kvm_vcpu *vcpu; + int i; + + /* The first run? */ + if (le64_to_cpu(pvclock->sequence_number) == 0) + return 0; + + /* PV frequency must not change! */ + if (le32_to_cpu(pvclock->pv_freq) != kvm->arch.lpt.fpv) + return -EFAULT; + + previous_freq = le32_to_cpu(pvclock->native_freq); + if (previous_freq == current_freq) + return 0; + + kvm_for_each_vcpu(i, vcpu, kvm) { + struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); + u64 cntvct, new_cntvct; + u32 cnt_tval, new_cnt_tval; + + /* Update cntvoff based on new cntvct */ + cntvct = current_time - vtimer->cntvoff; + new_cntvct = mul_u64_u32_div(cntvct, + current_freq, + previous_freq); + vtimer->cntvoff = current_time - new_cntvct; + + /* Update cnt_cval based on new cnt_tval */ + cnt_tval = vtimer->cnt_cval - cntvct; + new_cnt_tval = mul_u64_u32_div(cnt_tval, + current_freq, + previous_freq); + vtimer->cnt_cval = new_cntvct + new_cnt_tval; + } + + return 0; +} + +static void pvclock_lpt_update_structure(struct kvm *kvm, + struct pvclock_vm_lpt_time *pvclock) +{ + u64 sequence_number, scale_mult, rscale_mult; + u32 native_freq, pv_freq; + u32 scale_intbits, fracbits; + u32 rscale_intbits, rfracbits; + + sequence_number = le64_to_cpu(pvclock->sequence_number) + 2; + + native_freq = arch_timer_get_rate(); + pv_freq = kvm->arch.lpt.fpv; + + /* At least one bit for int part */ + scale_intbits = rscale_intbits = 1; + if (pv_freq >= native_freq) + scale_intbits = ilog2(pv_freq / native_freq) + 1; + else + rscale_intbits = ilog2(native_freq / pv_freq) + 1; + + fracbits = 64 - scale_intbits; + scale_mult = mul_u64_u32_div(BIT_ULL(fracbits), pv_freq, native_freq); + rfracbits = 64 - rscale_intbits; + rscale_mult = mul_u64_u32_div(BIT_ULL(rfracbits), native_freq, pv_freq); + + pvclock->sequence_number = cpu_to_le64(sequence_number); + pvclock->native_freq = cpu_to_le32(native_freq); + pvclock->pv_freq = cpu_to_le32(pv_freq); + pvclock->scale_mult = cpu_to_le64(scale_mult); + pvclock->rscale_mult = cpu_to_le64(rscale_mult); + pvclock->fracbits = cpu_to_le32(fracbits); + pvclock->rfracbits = cpu_to_le32(rfracbits); +} + +int kvm_update_lpt_time(struct kvm *kvm) +{ + u32 pv_freq = kvm->arch.lpt.fpv; + u64 lpt_ipa = kvm->arch.lpt.base; + struct pvclock_vm_lpt_time pvclock; + int ret = 0; + + /* Userspace does not enable LPT? */ + if (pv_freq == 0 && lpt_ipa == GPA_INVALID) + return 0; + + /* Userspace fault programming? */ + if (pv_freq == 0 || lpt_ipa == GPA_INVALID) + return -EINVAL; + + mutex_lock(&kvm->lock); + if (kvm->arch.lpt.updated) + goto unlock; + + ret = kvm_read_guest_lock(kvm, lpt_ipa, &pvclock, sizeof(pvclock)); + if (ret < 0) + goto unlock; + + ret = pvclock_lpt_update_vtimer(kvm, &pvclock); + if (ret < 0) + goto unlock; + + pvclock_lpt_update_structure(kvm, &pvclock); + + ret = kvm_write_guest_lock(kvm, lpt_ipa, &pvclock, sizeof(pvclock)); + if (!ret) + kvm->arch.lpt.updated = true; + +unlock: + mutex_unlock(&kvm->lock); + return ret; +} + +gpa_t kvm_init_lpt_time(struct kvm *kvm) +{ + return kvm->arch.lpt.base; +} From patchwork Mon Aug 17 08:41:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11717419 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 20336739 for ; Mon, 17 Aug 2020 09:09:13 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E9EF02063A for ; Mon, 17 Aug 2020 09:09:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Lp8gfVBx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E9EF02063A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=w9YFynjcJztiJPTDln50wliGECAjKJaUYyOViXXmb80=; b=Lp8gfVBxQ/xWQt64ceXVOD7pl MaC9qB0sUBh0D3fRFQ1YU9kJaZbcRlPEH/n8ukDCVmK9QC7h9mGUWswJIgcQ9pUdRjqdM0nOKsq4k Cq5ieE+b4bOFrG6x5Q+2OrQC/SAi/Em+VwmGZO/mQwoT9gdYWW8Mjymn1hEZLw0VbJtOMlMC4Kyh5 kV3KwPyS0TLxHbBFVSRWB9FY8i75LKZoS1iNfVYA5osrFTjkXQ9bMXKUeiP2L0DItcIYhehAsVxYB hx7KyKsJnenJOBLnrxxGDx5rmWBwMRmJiun0gfRKkmg+sJ9AkQ8jDoajE3fdyCKmEf2fdDXmWWdwB qX6aGJ3wA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7b8N-00086X-VP; Mon, 17 Aug 2020 09:08:52 +0000 Received: from szxga07-in.huawei.com ([45.249.212.35] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7ahr-0002CI-4u for linux-arm-kernel@lists.infradead.org; Mon, 17 Aug 2020 08:41:30 +0000 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 6AFD1575E938EDF8A34A; Mon, 17 Aug 2020 16:41:22 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.487.0; Mon, 17 Aug 2020 16:41:14 +0800 From: Keqian Zhu To: , , , Subject: [RFC PATCH 3/5] KVM: arm64: Provide VM device attributes for LPT time Date: Mon, 17 Aug 2020 16:41:08 +0800 Message-ID: <20200817084110.2672-4-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200817084110.2672-1-zhukeqian1@huawei.com> References: <20200817084110.2672-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200817_044127_503089_7159E579 X-CRM114-Status: GOOD ( 17.20 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.35 listed in list.dnswl.org] 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.35 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Marc Zyngier , Keqian Zhu , Steven Price , James Morse , Catalin Marinas , wanghaibin.wang@huawei.com, Will Deacon Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Allow user space to inform the KVM host what the PV frequency is and where in the physical memory map the paravirtualized LPT time structures should be located. User space can set attributes on the VM for that guest. The address is given in terms of the physical address visible to the guest and must be 64 byte aligned. The guest will discover the address via a hypercall. PV frequency is 32 bits and must not be 0. Signed-off-by: Steven Price Signed-off-by: Keqian Zhu --- arch/arm64/include/asm/kvm_host.h | 4 ++ arch/arm64/include/uapi/asm/kvm.h | 5 +++ arch/arm64/kvm/arm.c | 64 ++++++++++++++++++++++++++++ arch/arm64/kvm/pvtime.c | 87 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 160 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 0c6a564..cbe330c 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -523,6 +523,10 @@ int kvm_arm_pvtime_get_attr(struct kvm_vcpu *vcpu, int kvm_arm_pvtime_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr); +int kvm_arm_pvtime_lpt_set_attr(struct kvm *kvm, struct kvm_device_attr *attr); +int kvm_arm_pvtime_lpt_get_attr(struct kvm *kvm, struct kvm_device_attr *attr); +int kvm_arm_pvtime_lpt_has_attr(struct kvm *kvm, struct kvm_device_attr *attr); + static inline void kvm_arm_pvtime_vcpu_init(struct kvm_vcpu_arch *vcpu_arch) { vcpu_arch->steal.base = GPA_INVALID; diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index ba85bb2..7b045c7 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -325,6 +325,11 @@ struct kvm_vcpu_events { #define KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES 3 #define KVM_DEV_ARM_ITS_CTRL_RESET 4 +/* Device Control API on kvm fd */ +#define KVM_ARM_VM_PVTIME_LPT_CTRL 0 +#define KVM_ARM_VM_PVTIME_LPT_IPA 0 +#define KVM_ARM_VM_PVTIME_LPT_FREQ 1 + /* Device Control API on vcpu fd */ #define KVM_ARM_VCPU_PMU_V3_CTRL 0 #define KVM_ARM_VCPU_PMU_V3_IRQ 0 diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 671f1461..4a867e5 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1235,11 +1235,60 @@ static int kvm_vm_ioctl_set_device_addr(struct kvm *kvm, } } +static int kvm_arm_vm_set_attr(struct kvm *kvm, struct kvm_device_attr *attr) +{ + int ret; + + switch (attr->group) { + case KVM_ARM_VM_PVTIME_LPT_CTRL: + ret = kvm_arm_pvtime_lpt_set_attr(kvm, attr); + break; + default: + ret = -ENXIO; + break; + } + + return ret; +} + +static int kvm_arm_vm_get_attr(struct kvm *kvm, struct kvm_device_attr *attr) +{ + int ret; + + switch (attr->group) { + case KVM_ARM_VM_PVTIME_LPT_CTRL: + ret = kvm_arm_pvtime_lpt_get_attr(kvm, attr); + break; + default: + ret = -ENXIO; + break; + } + + return ret; +} + +static int kvm_arm_vm_has_attr(struct kvm *kvm, struct kvm_device_attr *attr) +{ + int ret; + + switch (attr->group) { + case KVM_ARM_VM_PVTIME_LPT_CTRL: + ret = kvm_arm_pvtime_lpt_has_attr(kvm, attr); + break; + default: + ret = -ENXIO; + break; + } + + return ret; +} + long kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { struct kvm *kvm = filp->private_data; void __user *argp = (void __user *)arg; + struct kvm_device_attr attr; switch (ioctl) { case KVM_CREATE_IRQCHIP: { @@ -1271,6 +1320,21 @@ long kvm_arch_vm_ioctl(struct file *filp, return 0; } + case KVM_SET_DEVICE_ATTR: { + if (copy_from_user(&attr, argp, sizeof(attr))) + return -EFAULT; + return kvm_arm_vm_set_attr(kvm, &attr); + } + case KVM_GET_DEVICE_ATTR: { + if (copy_from_user(&attr, argp, sizeof(attr))) + return -EFAULT; + return kvm_arm_vm_get_attr(kvm, &attr); + } + case KVM_HAS_DEVICE_ATTR: { + if (copy_from_user(&attr, argp, sizeof(attr))) + return -EFAULT; + return kvm_arm_vm_has_attr(kvm, &attr); + } default: return -EINVAL; } diff --git a/arch/arm64/kvm/pvtime.c b/arch/arm64/kvm/pvtime.c index 24131ca..3f93473 100644 --- a/arch/arm64/kvm/pvtime.c +++ b/arch/arm64/kvm/pvtime.c @@ -257,3 +257,90 @@ gpa_t kvm_init_lpt_time(struct kvm *kvm) { return kvm->arch.lpt.base; } + +int kvm_arm_pvtime_lpt_set_attr(struct kvm *kvm, struct kvm_device_attr *attr) +{ + u64 __user *user = (u64 __user *)attr->addr; + u64 ipa; + u32 freq; + int idx; + int ret = 0; + + switch (attr->attr) { + case KVM_ARM_VM_PVTIME_LPT_IPA: + if (get_user(ipa, user)) { + ret = -EFAULT; + break; + } + if (!IS_ALIGNED(ipa, 64)) { + ret = -EINVAL; + break; + } + if (kvm->arch.lpt.base != GPA_INVALID) { + ret = -EEXIST; + break; + } + /* Check the address is in a valid memslot */ + idx = srcu_read_lock(&kvm->srcu); + if (kvm_is_error_hva(gfn_to_hva(kvm, ipa >> PAGE_SHIFT))) + ret = -EINVAL; + srcu_read_unlock(&kvm->srcu, idx); + if (ret) + break; + + kvm->arch.lpt.base = ipa; + break; + case KVM_ARM_VM_PVTIME_LPT_FREQ: + if (get_user(freq, user)) { + ret = -EFAULT; + break; + } + if (freq == 0) { + ret = -EINVAL; + break; + } + if (kvm->arch.lpt.fpv != 0) { + ret = -EEXIST; + break; + } + kvm->arch.lpt.fpv = freq; + break; + default: + ret = -ENXIO; + break; + } + + return ret; +} + +int kvm_arm_pvtime_lpt_get_attr(struct kvm *kvm, struct kvm_device_attr *attr) +{ + u64 __user *user = (u64 __user *)attr->addr; + int ret = 0; + + switch (attr->attr) { + case KVM_ARM_VM_PVTIME_LPT_IPA: + if (put_user(kvm->arch.lpt.base, user)) + ret = -EFAULT; + break; + case KVM_ARM_VM_PVTIME_LPT_FREQ: + if (put_user(kvm->arch.lpt.fpv, user)) + ret = -EFAULT; + break; + default: + ret = -ENXIO; + break; + } + + return ret; +} + +int kvm_arm_pvtime_lpt_has_attr(struct kvm *kvm, struct kvm_device_attr *attr) +{ + switch (attr->attr) { + case KVM_ARM_VM_PVTIME_LPT_IPA: + case KVM_ARM_VM_PVTIME_LPT_FREQ: + return 0; + } + return -ENXIO; +} From patchwork Mon Aug 17 08:41:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11717447 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 71CEF722 for ; Mon, 17 Aug 2020 09:10:07 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 482F12063A for ; Mon, 17 Aug 2020 09:10:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Y2h8GO2K" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 482F12063A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FW9hK+RbNoZboZrHBS3PnEd3HX53+vvX+fAi+qcLcIQ=; b=Y2h8GO2Ke2csCcF4rcgdcgfZ7 +WKILQp6JVorLIjBNV4Hz1eRwx6oHKOpg8b6UaxKXdfOT39L1emlqhwaOFPjd5RA1q6O59aqHx9LK NdFW5TVIqjycQ06Fd2sqi1FWsJgxEaMs0xMp0CkAe0CcgqB6e07EZi9NYptkjwW8GLFZyYYzuLtl0 ddP4+P0aRTlLRO7eOD1MapMi/0qHOHXncFLv5rpabGJO8TyyepvcgsDd1MmI749Ct4qgTKr/j0Cy7 S/oNFth/c3kafvngB2vrapZgOgd7X/carH6uKqPz2zbXWuHToT3TR1FHC4u/oUD0eJ2ppVgFMtcPw nnD6j+kRg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7b9E-00008J-4h; Mon, 17 Aug 2020 09:09:44 +0000 Received: from szxga07-in.huawei.com ([45.249.212.35] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7ahs-0002CN-1j for linux-arm-kernel@lists.infradead.org; Mon, 17 Aug 2020 08:41:33 +0000 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 63C58374DE456567A6C9; Mon, 17 Aug 2020 16:41:22 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.487.0; Mon, 17 Aug 2020 16:41:14 +0800 From: Keqian Zhu To: , , , Subject: [RFC PATCH 4/5] clocksource: arm_arch_timer: Add pvtime LPT initialization Date: Mon, 17 Aug 2020 16:41:09 +0800 Message-ID: <20200817084110.2672-5-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200817084110.2672-1-zhukeqian1@huawei.com> References: <20200817084110.2672-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200817_044128_387372_308C99A9 X-CRM114-Status: GOOD ( 16.39 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.35 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.35 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Marc Zyngier , Keqian Zhu , Steven Price , James Morse , Catalin Marinas , wanghaibin.wang@huawei.com, Will Deacon Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Enable paravirtualized time to be used in a KVM guest if the host supports it. This allows the guest to derive a counter which is clocked at a persistent rate even when the guest is migrated. If we discover that the system supports SMCCC v1.1 then we probe to determine whether the hypervisor supports paravirtualized features and finally whether it supports "Live Physical Time" reporting. If so a shared structure is made available to the guest containing coefficients to calculate the derived clock. Signed-off-by: Steven Price Signed-off-by: Keqian Zhu --- drivers/clocksource/arm_arch_timer.c | 69 ++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/drivers/clocksource/arm_arch_timer.c b/drivers/clocksource/arm_arch_timer.c index 6c3e841..eb2e57a 100644 --- a/drivers/clocksource/arm_arch_timer.c +++ b/drivers/clocksource/arm_arch_timer.c @@ -26,6 +26,7 @@ #include #include +#include #include #include @@ -84,6 +85,66 @@ static int __init early_evtstrm_cfg(char *buf) } early_param("clocksource.arm_arch_timer.evtstrm", early_evtstrm_cfg); +/* PV-time LPT */ +#ifdef CONFIG_ARM64 +struct pvclock_vm_lpt_time *lpt_info; +EXPORT_SYMBOL_GPL(lpt_info); +DEFINE_STATIC_KEY_FALSE(pvclock_lpt_key_enabled); +EXPORT_SYMBOL_GPL(pvclock_lpt_key_enabled); + +static bool has_pv_lpt_clock(void) +{ + struct arm_smccc_res res; + + if (arm_smccc_1_1_get_conduit() == SMCCC_CONDUIT_NONE) + return false; + + arm_smccc_1_1_invoke(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, + ARM_SMCCC_HV_PV_TIME_FEATURES, &res); + if (res.a0 != SMCCC_RET_SUCCESS) + return false; + + arm_smccc_1_1_invoke(ARM_SMCCC_HV_PV_TIME_FEATURES, + ARM_SMCCC_HV_PV_TIME_LPT, &res); + return res.a0 == SMCCC_RET_SUCCESS; +} + +static int pvclock_lpt_init(void) +{ + struct arm_smccc_res res; + + if (!has_pv_lpt_clock()) + return 0; + + arm_smccc_1_1_invoke(ARM_SMCCC_HV_PV_TIME_LPT, 0, &res); + if (res.a0 == SMCCC_RET_NOT_SUPPORTED) + return 0; + + lpt_info = memremap(res.a0, sizeof(*lpt_info), MEMREMAP_WB); + if (!lpt_info) { + pr_warn("Failed to map pvclock LPT data structure\n"); + return -EFAULT; + } + + if (le32_to_cpu(lpt_info->revision) != 0 || + le32_to_cpu(lpt_info->attributes) != 0) { + pr_warn_once("Unexpected revision or attributes " + "in pvclock LPT data structure\n"); + return -EFAULT; + } + + static_branch_enable(&pvclock_lpt_key_enabled); + pr_info("Using pvclock LPT\n"); + return 0; +} +#else /* CONFIG_ARM64 */ +static int pvclock_lpt_init(void) +{ + return 0; +} +#endif /* CONFIG_ARM64 */ + + /* * Architected system timer support. */ @@ -1285,6 +1346,10 @@ static int __init arch_timer_of_init(struct device_node *np) arch_timer_populate_kvm_info(); + ret = pvclock_lpt_init(); + if (ret) + return ret; + rate = arch_timer_get_cntfrq(); arch_timer_of_configure_rate(rate, np); @@ -1613,6 +1678,10 @@ static int __init arch_timer_acpi_init(struct acpi_table_header *table) arch_timer_populate_kvm_info(); + ret = pvclock_lpt_init(); + if (ret) + return ret; + /* * When probing via ACPI, we have no mechanism to override the sysreg * CNTFRQ value. This *must* be correct. From patchwork Mon Aug 17 08:41:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11717457 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0CDC1739 for ; Mon, 17 Aug 2020 09:11:07 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C48372063A for ; Mon, 17 Aug 2020 09:11:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="jwpNzTQb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C48372063A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cGLf2PaKXmNo2P2C/6afIevL2P5/0B/vOCl65cCyxWg=; b=jwpNzTQbLdv0xE1hSEx2xDn2/ 93Gd4tSHz06bW0aY0oy0lhvxWMTnHgrmIMvCGTJCjpA3RMc7Qm0gIxuBU/YbW15JhOT6KXVRGo6Gg 66LqpWf7D8dWh6NWMnhAqRSyyL9fsvLIb+Kaid7D9MCrmkcNmVkFFlhIsjKFq8IsUGgeCbaaaD268 wRWRSlgTFn1CQ9LjlFMJy1C32QABKqP9zFD+B1GVbdHHZj1lfbqyWCuNg8DbVdyOZj1hpBrHjGEBh 6oDVykbGoPvrLAhi2n05EUjdo4y0Ywg6M+kcQ2TctMtqnZv1aIx/787VzsGsKFN4QHptFgpUHcrG/ 4OlHRqBJw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7bA9-0000fK-W5; Mon, 17 Aug 2020 09:10:42 +0000 Received: from szxga07-in.huawei.com ([45.249.212.35] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7ahr-0002CK-4r for linux-arm-kernel@lists.infradead.org; Mon, 17 Aug 2020 08:41:35 +0000 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 5AAED57CF9022F757412; Mon, 17 Aug 2020 16:41:22 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.174.187.22) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.487.0; Mon, 17 Aug 2020 16:41:15 +0800 From: Keqian Zhu To: , , , Subject: [RFC PATCH 5/5] clocksource: arm_arch_timer: Use pvtime LPT Date: Mon, 17 Aug 2020 16:41:10 +0800 Message-ID: <20200817084110.2672-6-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20200817084110.2672-1-zhukeqian1@huawei.com> References: <20200817084110.2672-1-zhukeqian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200817_044127_941773_DE421FE4 X-CRM114-Status: GOOD ( 23.41 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.35 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.35 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Marc Zyngier , Keqian Zhu , Steven Price , James Morse , Catalin Marinas , wanghaibin.wang@huawei.com, Will Deacon Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Enable paravirtualized time to be used in a KVM guest if the host supports it. This allows the guest to derive a counter which is clocked at a persistent rate even when the guest is migrated. If we discover that the system supports SMCCC v1.1 then we probe to determine whether the hypervisor supports paravirtualized features and finally whether it supports "Live Physical Time" reporting. If so a shared structure is made available to the guest containing coefficients to calculate the derived clock. The guest kernel uses the coefficients to present a clock to user space that is always clocked at the same rate whenever the guest is running ('live'), even if the physical clock changes (due to the guest being migrated). The existing workaround framework for CNTVCT is used to trap user space accesses to the timer registers so we can present the derived clock. Signed-off-by: Steven Price Signed-off-by: Keqian Zhu --- arch/arm64/include/asm/arch_timer.h | 179 ++++++++++++++++++++++++++++++++--- drivers/clocksource/arm_arch_timer.c | 59 +++++++----- 2 files changed, 204 insertions(+), 34 deletions(-) diff --git a/arch/arm64/include/asm/arch_timer.h b/arch/arm64/include/asm/arch_timer.h index 9f0ec21..bbaecf1 100644 --- a/arch/arm64/include/asm/arch_timer.h +++ b/arch/arm64/include/asm/arch_timer.h @@ -10,6 +10,7 @@ #include #include +#include #include #include @@ -64,25 +65,178 @@ struct arch_timer_erratum_workaround { DECLARE_PER_CPU(const struct arch_timer_erratum_workaround *, timer_unstable_counter_workaround); + +extern struct pvclock_vm_lpt_time *lpt_info; +DECLARE_STATIC_KEY_FALSE(pvclock_lpt_key_enabled); + +/* LPT read/write base layer */ + +#define lpt_read_base(target, trans, read) ({ \ + __le64 _seq_begin, _seq_end; \ + u64 _nval, _pval; \ + \ + do { \ + _seq_begin = READ_ONCE(lpt_info->sequence_number); \ + /* LPT structure can be treated as readonly device */ \ + rmb(); \ + \ + _nval = read(target); \ + _pval = trans(_nval); \ + \ + rmb(); \ + _seq_end = READ_ONCE(lpt_info->sequence_number); \ + } while (unlikely(_seq_begin != _seq_end)); \ + \ + _pval; \ +}) + +#define lpt_write_base(val, target, trans, write) ({ \ + __le64 _seq_begin, _seq_end; \ + u64 _pval = val; \ + u64 _nval; \ + \ + do { \ + _seq_begin = READ_ONCE(lpt_info->sequence_number); \ + /* LPT structure can be treated as readonly device */ \ + rmb(); \ + \ + _nval = trans(_pval); \ + write(_nval, target); \ + \ + rmb(); \ + _seq_end = READ_ONCE(lpt_info->sequence_number); \ + } while (unlikely(_seq_begin != _seq_end)); \ +}) + +#define lpt_read(target, trans, read) ({ \ + u64 _val; \ + \ + if (static_branch_unlikely(&pvclock_lpt_key_enabled)) { \ + _val = lpt_read_base(target, trans, read); \ + } else { \ + _val = read(target); \ + } \ + \ + _val; \ +}) + +#define lpt_write(val, target, trans, write) ({ \ + if (static_branch_unlikely(&pvclock_lpt_key_enabled)) { \ + lpt_write_base(val, target, trans, write); \ + } else { \ + write(val, target); \ + } \ +}) + +/* LPT read/write layer for timer and count */ + +static inline u64 native_to_pv_cycles(u64 cnt) +{ + u64 scale_mult = le64_to_cpu(lpt_info->scale_mult); + u32 fracbits = le32_to_cpu(lpt_info->fracbits); + + return mul_u64_u64_shr(scale_mult, cnt, fracbits); +} + +static inline u64 pv_to_native_cycles(u64 cnt) +{ + u64 rscale_mult = le64_to_cpu(lpt_info->rscale_mult); + u32 rfracbits = le32_to_cpu(lpt_info->rfracbits); + + return mul_u64_u64_shr(rscale_mult, cnt, rfracbits); +} + +#define arch_timer_read_mediated(reg) ({ \ + lpt_read(reg, native_to_pv_cycles, read_sysreg); \ +}) + +#define arch_timer_write_mediated(val, reg) ({ \ + u64 _val = val; \ + lpt_write(_val, reg, pv_to_native_cycles, write_sysreg); \ +}) + +#define mem_timer_read_mediated(addr) ({ \ + lpt_read(addr, native_to_pv_cycles, readl_relaxed); \ +}) + +#define mem_timer_write_mediated(val, addr) ({ \ + u64 _val = val; \ + lpt_write(_val, addr, pv_to_native_cycles, writel_relaxed); \ +}) + +/* LPT read/write layer for cntkctl_el1 */ + +static inline int cntkctl_evnti_shift(void) +{ + u32 native_freq = le32_to_cpu(lpt_info->native_freq); + u32 pv_freq = le32_to_cpu(lpt_info->pv_freq); + int div, shift; + + if (pv_freq >= native_freq) + div = pv_freq / native_freq; + else + div = native_freq / pv_freq; + + /* Find the closest power of two to the divisor */ + shift = fls(div); + if ((shift == 1) || (shift > 1 && !(shift & (1 << (shift - 2))))) + shift--; + + return pv_freq >= native_freq ? shift : -shift; +} + +static inline u64 parse_cntkctl(u64 val, bool native_to_pv) +{ + int evnti = (val >> ARCH_TIMER_EVT_TRIGGER_SHIFT) & 0xF; + + if (native_to_pv) + evnti = evnti + cntkctl_evnti_shift(); + else + evnti = evnti - cntkctl_evnti_shift(); + + evnti = min(15, max(0, evnti)); + val &= ~ARCH_TIMER_EVT_TRIGGER_MASK; + val |= evnti << ARCH_TIMER_EVT_TRIGGER_SHIFT; + + return val; +} + +#define TRANS_CNTKCTL_N(nval) ({ \ + parse_cntkctl(nval, true); \ +}) + +#define TRANS_CNTKCTL_P(pval) ({ \ + parse_cntkctl(pval, false); \ +}) + +#define arch_timer_read_cntkctl_mediated() ({ \ + lpt_read(cntkctl_el1, TRANS_CNTKCTL_N, read_sysreg); \ +}) + +#define arch_timer_write_cntkctl_mediated(val) ({ \ + u64 _val = val; \ + lpt_write(_val, cntkctl_el1, TRANS_CNTKCTL_P, write_sysreg); \ +}) + /* inline sysreg accessors that make erratum_handler() work */ static inline notrace u32 arch_timer_read_cntp_tval_el0(void) { - return read_sysreg(cntp_tval_el0); + return arch_timer_read_mediated(cntp_tval_el0); } static inline notrace u32 arch_timer_read_cntv_tval_el0(void) { - return read_sysreg(cntv_tval_el0); + return arch_timer_read_mediated(cntv_tval_el0); } static inline notrace u64 arch_timer_read_cntpct_el0(void) { - return read_sysreg(cntpct_el0); + return arch_timer_read_mediated(cntpct_el0); } static inline notrace u64 arch_timer_read_cntvct_el0(void) { - return read_sysreg(cntvct_el0); + return arch_timer_read_mediated(cntvct_el0); } #define arch_timer_reg_read_stable(reg) \ @@ -110,7 +264,7 @@ void arch_timer_reg_write_cp15(int access, enum arch_timer_reg reg, u32 val) write_sysreg(val, cntp_ctl_el0); break; case ARCH_TIMER_REG_TVAL: - write_sysreg(val, cntp_tval_el0); + arch_timer_write_mediated(val, cntp_tval_el0); break; } } else if (access == ARCH_TIMER_VIRT_ACCESS) { @@ -119,7 +273,7 @@ void arch_timer_reg_write_cp15(int access, enum arch_timer_reg reg, u32 val) write_sysreg(val, cntv_ctl_el0); break; case ARCH_TIMER_REG_TVAL: - write_sysreg(val, cntv_tval_el0); + arch_timer_write_mediated(val, cntv_tval_el0); break; } } @@ -151,17 +305,20 @@ u32 arch_timer_reg_read_cp15(int access, enum arch_timer_reg reg) static inline u32 arch_timer_get_cntfrq(void) { - return read_sysreg(cntfrq_el0); + if (static_branch_unlikely(&pvclock_lpt_key_enabled)) + return le32_to_cpu(lpt_info->pv_freq); + else + return read_sysreg(cntfrq_el0); } static inline u32 arch_timer_get_cntkctl(void) { - return read_sysreg(cntkctl_el1); + return arch_timer_read_cntkctl_mediated(); } static inline void arch_timer_set_cntkctl(u32 cntkctl) { - write_sysreg(cntkctl, cntkctl_el1); + arch_timer_write_cntkctl_mediated(cntkctl); isb(); } @@ -199,7 +356,7 @@ static __always_inline u64 __arch_counter_get_cntpct(void) u64 cnt; isb(); - cnt = read_sysreg(cntpct_el0); + cnt = arch_timer_read_mediated(cntpct_el0); arch_counter_enforce_ordering(cnt); return cnt; } @@ -219,7 +376,7 @@ static __always_inline u64 __arch_counter_get_cntvct(void) u64 cnt; isb(); - cnt = read_sysreg(cntvct_el0); + cnt = arch_timer_read_mediated(cntvct_el0); arch_counter_enforce_ordering(cnt); return cnt; } diff --git a/drivers/clocksource/arm_arch_timer.c b/drivers/clocksource/arm_arch_timer.c index eb2e57a..28277b0 100644 --- a/drivers/clocksource/arm_arch_timer.c +++ b/drivers/clocksource/arm_arch_timer.c @@ -26,7 +26,6 @@ #include #include -#include #include #include @@ -137,11 +136,24 @@ static int pvclock_lpt_init(void) pr_info("Using pvclock LPT\n"); return 0; } + +static bool pvclock_lpt_enabled(void) +{ + return static_branch_unlikely(&pvclock_lpt_key_enabled); +} #else /* CONFIG_ARM64 */ static int pvclock_lpt_init(void) { return 0; } + +static bool pvclock_lpt_enabled(void) +{ + return false; +} + +#define mem_timer_read_mediated(val, addr) (readl_relaxed(val, addr)) +#define mem_timer_write_mediated(val, addr) (writel_relaxed(val, addr)) #endif /* CONFIG_ARM64 */ @@ -160,7 +172,7 @@ void arch_timer_reg_write(int access, enum arch_timer_reg reg, u32 val, writel_relaxed(val, timer->base + CNTP_CTL); break; case ARCH_TIMER_REG_TVAL: - writel_relaxed(val, timer->base + CNTP_TVAL); + mem_timer_write_mediated(val, timer->base + CNTP_TVAL); break; } } else if (access == ARCH_TIMER_MEM_VIRT_ACCESS) { @@ -170,7 +182,7 @@ void arch_timer_reg_write(int access, enum arch_timer_reg reg, u32 val, writel_relaxed(val, timer->base + CNTV_CTL); break; case ARCH_TIMER_REG_TVAL: - writel_relaxed(val, timer->base + CNTV_TVAL); + mem_timer_write_mediated(val, timer->base + CNTV_TVAL); break; } } else { @@ -279,8 +291,8 @@ struct ate_acpi_oem_info { int _retries = 200; \ \ do { \ - _old = read_sysreg(reg); \ - _new = read_sysreg(reg); \ + _old = arch_timer_read_mediated(reg); \ + _new = arch_timer_read_mediated(reg); \ _retries--; \ } while (unlikely(_old != _new) && _retries); \ \ @@ -325,8 +337,8 @@ static u64 notrace fsl_a008585_read_cntvct_el0(void) int _retries = 50; \ \ do { \ - _old = read_sysreg(reg); \ - _new = read_sysreg(reg); \ + _old = arch_timer_read_mediated(reg); \ + _new = arch_timer_read_mediated(reg); \ _retries--; \ } while (unlikely((_new - _old) >> 5) && _retries); \ \ @@ -383,8 +395,8 @@ static u64 notrace arm64_858921_read_cntpct_el0(void) { u64 old, new; - old = read_sysreg(cntpct_el0); - new = read_sysreg(cntpct_el0); + old = arch_timer_read_mediated(cntpct_el0); + new = arch_timer_read_mediated(cntpct_el0); return (((old ^ new) >> 32) & 1) ? old : new; } @@ -392,8 +404,8 @@ static u64 notrace arm64_858921_read_cntvct_el0(void) { u64 old, new; - old = read_sysreg(cntvct_el0); - new = read_sysreg(cntvct_el0); + old = arch_timer_read_mediated(cntvct_el0); + new = arch_timer_read_mediated(cntvct_el0); return (((old ^ new) >> 32) & 1) ? old : new; } #endif @@ -411,7 +423,7 @@ static u64 notrace arm64_858921_read_cntvct_el0(void) int _retries = 150; \ \ do { \ - _val = read_sysreg(reg); \ + _val = arch_timer_read_mediated(reg); \ _retries--; \ } while (((_val + 1) & GENMASK(9, 0)) <= 1 && _retries); \ \ @@ -431,12 +443,14 @@ static u64 notrace sun50i_a64_read_cntvct_el0(void) static u32 notrace sun50i_a64_read_cntp_tval_el0(void) { - return read_sysreg(cntp_cval_el0) - sun50i_a64_read_cntpct_el0(); + return arch_timer_read_mediated(cntp_cval_el0) - + sun50i_a64_read_cntpct_el0(); } static u32 notrace sun50i_a64_read_cntv_tval_el0(void) { - return read_sysreg(cntv_cval_el0) - sun50i_a64_read_cntvct_el0(); + return arch_timer_read_mediated(cntv_cval_el0) - + sun50i_a64_read_cntvct_el0(); } #endif @@ -458,10 +472,10 @@ static void erratum_set_next_event_tval_generic(const int access, unsigned long if (access == ARCH_TIMER_PHYS_ACCESS) { cval = evt + arch_counter_get_cntpct(); - write_sysreg(cval, cntp_cval_el0); + arch_timer_write_mediated(cval, cntp_cval_el0); } else { cval = evt + arch_counter_get_cntvct(); - write_sysreg(cval, cntv_cval_el0); + arch_timer_write_mediated(cval, cntv_cval_el0); } arch_timer_reg_write(access, ARCH_TIMER_REG_CTRL, ctrl, clk); @@ -906,12 +920,11 @@ static void arch_counter_set_user_access(void) | ARCH_TIMER_VIRT_EVT_EN | ARCH_TIMER_USR_PCT_ACCESS_EN); - /* - * Enable user access to the virtual counter if it doesn't - * need to be workaround. The vdso may have been already + /* Trap user access to the virtual counter if we support LPT + * or it needs to be workaround. The vdso may have been already * disabled though. */ - if (arch_timer_this_cpu_has_cntvct_wa()) + if (pvclock_lpt_enabled() || arch_timer_this_cpu_has_cntvct_wa()) pr_info("CPU%d: Trapping CNTVCT access\n", smp_processor_id()); else cntkctl |= ARCH_TIMER_USR_VCT_ACCESS_EN; @@ -1029,9 +1042,9 @@ static u64 arch_counter_get_cntvct_mem(void) u32 vct_lo, vct_hi, tmp_hi; do { - vct_hi = readl_relaxed(arch_counter_base + CNTVCT_HI); - vct_lo = readl_relaxed(arch_counter_base + CNTVCT_LO); - tmp_hi = readl_relaxed(arch_counter_base + CNTVCT_HI); + vct_hi = mem_timer_read_mediated(arch_counter_base + CNTVCT_HI); + vct_lo = mem_timer_read_mediated(arch_counter_base + CNTVCT_LO); + tmp_hi = mem_timer_read_mediated(arch_counter_base + CNTVCT_HI); } while (vct_hi != tmp_hi); return ((u64) vct_hi << 32) | vct_lo;