From patchwork Wed Jul 22 16:44:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Brazdil X-Patchwork-Id: 11678947 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5F60513B6 for ; Wed, 22 Jul 2020 16:48:00 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2AD3F206F5 for ; Wed, 22 Jul 2020 16:48:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Jdz8jksI"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="O+j9vJMx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2AD3F206F5 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ngmSYNU9Vn4xlEDB0P/EeXoFb3wNh5qHN9uwbaUxn68=; b=Jdz8jksINaRASAgDCqSyxqya7 bEJiWoPACRkLqIxI4qMMWvQleF8YdwSVwnkZANIFf8HhZqtJCPfkfWBsqDePDNxL4PkfqfRZniDrC ZxI4SyNFdFvMR58RhsYjsd1/dzyfmSxcb1QLnx1xlMDQoKiIoKv+mnEpDRG1uItBhsm3irxN9bb3w wIShX/kxJ3lcwgq+r7bU3STODVNifGps5QV44I4VqbW+mXQPf0WDcxuX5stO8c7X8KSXClixMn6xh 843GO3YRqJRtKQlJeB4JltlAt1uPXZ1ivZXaFGgjYY+QgXltTziM8yv7dhq8t7X21115po+9ei/jl vDjk7qySg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jyHsi-0002HG-Ku; Wed, 22 Jul 2020 16:46:12 +0000 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jyHrm-0001wN-CK for linux-arm-kernel@lists.infradead.org; Wed, 22 Jul 2020 16:45:16 +0000 Received: by mail-wm1-x342.google.com with SMTP id f139so2619176wmf.5 for ; Wed, 22 Jul 2020 09:45:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=V96ak4i8UKYW1530XCTpq7OSsFZ4sEMYY/vy7g+OwDs=; b=O+j9vJMxDWg4Y7iNQtrBdo/PaCRmPCN0IrEKu0IAOXYNS4QWENM1txkn/akKPubHQt 2Q9bO5sSIjnp7XX8FPY3pwRcbHoMqdMJkGzXaT88liZXeU7cj6sVsFUDq1KO/ckOzlAz QczOXDyjtOBABv852W5Tzx6/2+oXjm/PTaAoZQsN+a7dWEbseiKUqyXAyZZWizDvGnjU 1FZUbG1JADmor77thW2lzkIlb3rOoNPKVh6tb9aPE4gepPFwYQCfH9Zb86Pby+wMTA/O NAZylWO739DSTe2kT8txIfBsXMeHOQKFBra06R5Omocjlvxtbruuk8RbcoziaI81HyVd QAPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=V96ak4i8UKYW1530XCTpq7OSsFZ4sEMYY/vy7g+OwDs=; b=qaKAtVrNSXdkxMOuF1qJVXi1US0GDxXZ5Ou/UzKJMtG2ze8Q0qfO9Bo/zL8uLTjKXs 5quT4LTL4x0KV+luVCseWbQ5ldgxZvOrM2alzfYaHtL74lcO7h57Sqx1z8hZD6fJlujO LGbanSjkFMtTpeAf6+bbWpYgWQzxcRrKHk2dGkTS/HO5wfhmFX0J9F9kJlHBv+MGZ3k3 j8loEPYJPFuC7RrnEY6RKrW3IqSKfAjRJswCP95osctcbCdTkRaK7xuI0CiaRIuEozG7 1/F+848WF9loIFoE5Rz4ERTfVBsGLmzzGuOxeG8Z8LVJ4Vw+W2zXhvhDLq+k2OZk+b4x VXhQ== X-Gm-Message-State: AOAM533C/jhf0Cewvgn1kFp5RVc02cgi4RWQtCkAsEQkUP5Rd6ZXwDTG NmJOkhCbQd07escTYzjEKM5FKQ== X-Google-Smtp-Source: ABdhPJzzu8w3H0zHy13NXlQtb/DWOK/nCH71MNq45VhKumCJJVIQ8gvuKohqxVj5jKeuYtcbVRZ2Vg== X-Received: by 2002:a1c:2045:: with SMTP id g66mr452060wmg.184.1595436313169; Wed, 22 Jul 2020 09:45:13 -0700 (PDT) Received: from localhost ([2a01:4b00:8523:2d03:b0ee:900a:e004:b9d0]) by smtp.gmail.com with ESMTPSA id h5sm521647wrc.97.2020.07.22.09.45.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 22 Jul 2020 09:45:12 -0700 (PDT) From: David Brazdil To: Catalin Marinas , Will Deacon , Marc Zyngier , Dennis Zhou , Tejun Heo , Christoph Lameter , Arnd Bergmann Subject: [PATCH 8/9] kvm: arm64: Set up hyp percpu data for nVHE Date: Wed, 22 Jul 2020 17:44:23 +0100 Message-Id: <20200722164424.42225-9-dbrazdil@google.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200722164424.42225-1-dbrazdil@google.com> References: <20200722164424.42225-1-dbrazdil@google.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200722_124514_498094_42AD61A9 X-CRM114-Status: GOOD ( 21.16 ) X-Spam-Score: -15.7 (---------------) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-15.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:342 listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -7.5 USER_IN_DEF_SPF_WL From: address is in the default SPF white-list 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.5 ENV_AND_HDR_SPF_MATCH Env and Hdr From used in default SPF WL Match -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Suzuki K Poulose , linux-kernel@vger.kernel.org, kernel-team@google.com, James Morse , Julien Thierry , David Brazdil , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Add hyp percpu section to linker script and rename the corresponding ELF sections of hyp/nvhe object files. This moves all nVHE-specific percpu variables to the new hyp percpu section. Allocate sufficient amount of memory for all percpu hyp regions at global KVM init time, and create corresponding hyp mappings. The base addresses of hyp percpu regions are kept in a dynamically allocated array in the kernel. Add NULL checks in PMU event-reset code as it may run before KVM memory is initialized. Signed-off-by: David Brazdil --- arch/arm64/include/asm/kvm_asm.h | 19 +++++++++-- arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/vmlinux.lds.S | 10 ++++++ arch/arm64/kvm/arm.c | 55 +++++++++++++++++++++++++++++-- arch/arm64/kvm/hyp/nvhe/Makefile | 2 ++ arch/arm64/kvm/pmu.c | 5 ++- 6 files changed, 86 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 3d69cab873e4..22e63e651702 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -72,8 +72,23 @@ #define CHOOSE_VHE_SYM(sym) sym #define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym) -#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym)) -#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu) +/* Array of percpu base addresses. Length of the array is nr_cpu_ids. */ +extern unsigned long *kvm_arm_hyp_percpu_base; + +/* + * Compute pointer to a symbol defined in nVHE percpu region. + * Returns NULL if percpu memory has not been allocated yet. + */ +#define this_cpu_ptr_nvhe(sym) per_cpu_ptr_nvhe(sym, smp_processor_id()) +#define per_cpu_ptr_nvhe(sym, cpu) \ + ({ \ + unsigned long base, off; \ + base = kvm_arm_hyp_percpu_base \ + ? kvm_arm_hyp_percpu_base[cpu] : 0; \ + off = (unsigned long)&kvm_nvhe_sym(sym) - \ + (unsigned long)&kvm_nvhe_sym(__per_cpu_start); \ + base ? (typeof(kvm_nvhe_sym(sym))*)(base + off) : NULL; \ + }) #ifndef __KVM_NVHE_HYPERVISOR__ /* diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index 3994169985ef..5062553a6847 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -18,5 +18,6 @@ extern char __exittext_begin[], __exittext_end[]; extern char __irqentry_text_start[], __irqentry_text_end[]; extern char __mmuoff_data_start[], __mmuoff_data_end[]; extern char __entry_tramp_text_start[], __entry_tramp_text_end[]; +extern char __kvm_nvhe___per_cpu_start[], __kvm_nvhe___per_cpu_end[]; #endif /* __ASM_SECTIONS_H */ diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 6827da7f3aa5..c678615ccd6d 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -16,6 +16,9 @@ #include "image.h" +#define __CONCAT3(x, y, z) x ## y ## z +#define CONCAT3(x, y, z) __CONCAT3(x, y, z) + OUTPUT_ARCH(aarch64) ENTRY(_text) @@ -187,6 +190,13 @@ SECTIONS PERCPU_SECTION(L1_CACHE_BYTES) + /* KVM nVHE per-cpu section */ + #undef PERCPU_SECTION_NAME + #undef PERCPU_SYMBOL_NAME + #define PERCPU_SECTION_NAME(suffix) CONCAT3(.hyp, PERCPU_SECTION_BASE_NAME, suffix) + #define PERCPU_SYMBOL_NAME(name) __kvm_nvhe_ ## name + PERCPU_SECTION(L1_CACHE_BYTES) + .rela.dyn : ALIGN(8) { *(.rela .rela*) } diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index dc557b380c87..bbbc5c1519a9 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -47,6 +47,7 @@ __asm__(".arch_extension virt"); #endif static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); +unsigned long *kvm_arm_hyp_percpu_base; /* The VMID used in the VTTBR */ static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1); @@ -1253,6 +1254,15 @@ long kvm_arch_vm_ioctl(struct file *filp, } } +#define kvm_hyp_percpu_base(cpu) ((unsigned long)per_cpu_ptr_nvhe(__per_cpu_start, cpu)) +#define kvm_hyp_percpu_array_size (nr_cpu_ids * sizeof(*kvm_arm_hyp_percpu_base)) +#define kvm_hyp_percpu_array_order (get_order(kvm_hyp_percpu_array_size)) +#define kvm_hyp_percpu_begin CHOOSE_NVHE_SYM(__per_cpu_start) +#define kvm_hyp_percpu_size ((unsigned long)CHOOSE_NVHE_SYM(__per_cpu_end) - \ + (unsigned long)CHOOSE_NVHE_SYM(__per_cpu_start)) +#define kvm_hyp_percpu_order (kvm_hyp_percpu_size \ + ? get_order(kvm_hyp_percpu_size) : 0) + static void cpu_init_hyp_mode(void) { phys_addr_t pgd_ptr; @@ -1268,8 +1278,8 @@ static void cpu_init_hyp_mode(void) * kernel's mapping to the linear mapping, and store it in tpidr_el2 * so that we can use adr_l to access per-cpu variables in EL2. */ - tpidr_el2 = ((unsigned long)this_cpu_ptr(&kvm_host_data) - - (unsigned long)kvm_ksym_ref(&kvm_host_data)); + tpidr_el2 = (unsigned long)this_cpu_ptr_nvhe(__per_cpu_start) - + (unsigned long)kvm_ksym_ref(kvm_hyp_percpu_begin); pgd_ptr = kvm_mmu_get_httbr(); hyp_stack_ptr = __this_cpu_read(kvm_arm_hyp_stack_page) + PAGE_SIZE; @@ -1501,8 +1511,11 @@ static void teardown_hyp_mode(void) int cpu; free_hyp_pgds(); - for_each_possible_cpu(cpu) + for_each_possible_cpu(cpu) { free_hyp_page(per_cpu(kvm_arm_hyp_stack_page, cpu)); + free_hyp_pages(kvm_hyp_percpu_base(cpu), kvm_hyp_percpu_order); + } + free_hyp_pages((unsigned long)kvm_arm_hyp_percpu_base, kvm_hyp_percpu_array_order); } /** @@ -1535,6 +1548,28 @@ static int init_hyp_mode(void) per_cpu(kvm_arm_hyp_stack_page, cpu) = stack_page; } + /* + * Allocate and initialize pages for Hypervisor-mode percpu regions. + */ + kvm_arm_hyp_percpu_base = (unsigned long*)alloc_hyp_pages( + GFP_KERNEL | __GFP_ZERO, kvm_hyp_percpu_array_order); + if (!kvm_arm_hyp_percpu_base) { + err = -ENOMEM; + goto out_err; + } + for_each_possible_cpu(cpu) { + unsigned long percpu_base; + + percpu_base = alloc_hyp_pages(GFP_KERNEL, kvm_hyp_percpu_order); + if (!percpu_base) { + err = -ENOMEM; + goto out_err; + } + + memcpy((void*)percpu_base, kvm_hyp_percpu_begin, kvm_hyp_percpu_size); + kvm_arm_hyp_percpu_base[cpu] = percpu_base; + } + /* * Map the Hyp-code called directly from the host */ @@ -1579,6 +1614,20 @@ static int init_hyp_mode(void) } } + /* + * Map Hyp percpu pages + */ + for_each_possible_cpu(cpu) { + char *percpu_begin = (char *)kvm_arm_hyp_percpu_base[cpu]; + char *percpu_end = percpu_begin + PAGE_ALIGN(kvm_hyp_percpu_size); + err = create_hyp_mappings(percpu_begin, percpu_end, PAGE_HYP); + + if (err) { + kvm_err("Cannot map hyp percpu region\n"); + goto out_err; + } + } + for_each_possible_cpu(cpu) { kvm_host_data_t *cpu_data; diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 0b34414557d6..1d415698e60e 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -23,6 +23,8 @@ $(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE quiet_cmd_hypcopy = HYPCOPY $@ cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ \ --rename-section=.text=.hyp.text \ + --rename-section=.data..percpu=.hyp.data..percpu \ + --rename-section=.data..percpu..read_mostly=.hyp.data..percpu..read_mostly \ $< $@ # Remove ftrace and Shadow Call Stack CFLAGS. diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c index 717941910723..b4c8302f0798 100644 --- a/arch/arm64/kvm/pmu.c +++ b/arch/arm64/kvm/pmu.c @@ -33,7 +33,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) { struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data); - if (!kvm_pmu_switch_needed(attr)) + if (!ctx || !kvm_pmu_switch_needed(attr)) return; if (!attr->exclude_host) @@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr) { struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data); + if (!ctx) + return; + ctx->pmu_events.events_host &= ~clr; ctx->pmu_events.events_guest &= ~clr; }