From patchwork Wed Jan 27 17:27:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B89AC433E0 for ; Wed, 27 Jan 2021 17:27:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B435460187 for ; Wed, 27 Jan 2021 17:27:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B435460187 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 134746B006C; Wed, 27 Jan 2021 12:27:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C13F6B0072; Wed, 27 Jan 2021 12:27:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2E9F6B0073; Wed, 27 Jan 2021 12:27:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0161.hostedemail.com [216.40.44.161]) by kanga.kvack.org (Postfix) with ESMTP id C40E16B006C for ; Wed, 27 Jan 2021 12:27:11 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 7E70E181AEF30 for ; Wed, 27 Jan 2021 17:27:11 +0000 (UTC) X-FDA: 77752235862.17.star24_0a0a8a927598 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 69EB8180D0186 for ; Wed, 27 Jan 2021 17:27:11 +0000 (UTC) X-HE-Tag: star24_0a0a8a927598 X-Filterd-Recvd-Size: 13622 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 17:27:10 +0000 (UTC) Received: by mail-qv1-f46.google.com with SMTP id l14so1439068qvp.2 for ; Wed, 27 Jan 2021 09:27:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ASygAqr57jUZ78nCjocRBoBMKp4WVYWpgcDuzWEf5/U=; b=lTW8AsSeGiTqhrvoe5+DBlxNbXH/Jl+2xyBWNDwB76Bg8MNCuvd6VNwkmJ4+h63cZX cuq1Hb5sgFA9btCgFJ1qlG7gFOO9yjiPdnvbbADDuICXgW4ePbFURxqoTvJbgj35gXLl cYTYx1IjAV587qwyGmaRdylhQw3z11+nFnsBEcq0EOLmQsJWymmY++T5KvljbsHwL5TZ JKP3aTXz5mdnMHXMT1rL/M2j51IVvCKZBnD/YmNSWiVDpKFPuhgDLUuH9M8INScH3Ywl j8bNzL4bTdELqLfTU0vHpDFzwa+6thecSerPkkUNYqHGh3aUQs9T6I/ptCm+mYIUblHA Hlfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ASygAqr57jUZ78nCjocRBoBMKp4WVYWpgcDuzWEf5/U=; b=OpM5VHG2aR89ib5gJaj93vXvStotMCLQig4r2G/ciprDtWlLKTl6pdDmWyV5ts2G8M 0K84cJTAQU59dNXBy93ujeLNg6K2gjIWwLa07bUnLurwFoOU/0V+1HUAdsm/3HHhgOAI mHqd2dBGtkur5BW45u4jJfuDcDmBaHxf4Yxlfqx1WYEnBKaIgbwaeWWP5TIOkuMs3HTl q+/wLIv4OGuMFUrCj3FRqgmYwZX3nHpZQfdrlWjc/rEp7jDzYTLTFXrCtMp2Eu4WD1yw LHMFlayk5nFFxbnHLwOf9Y44/M7sW/y6xAZi5ThHhsee4G5HtPyroKOfYSt0k2/lccNU DO/w== X-Gm-Message-State: AOAM532G/Uijmb9OkwAijNby6WQjMRra3J0Om1GZJpwtN8OWrQzyEU/N BH10IGL5mm164wYSqJHCCZJj0Q== X-Google-Smtp-Source: ABdhPJxIcrcK4iDLzYXxJWKQD6eG4blKjf34bscTuC6KLr/1IGH2MCDgmQpbj2CQsy/4e2VNcDPU/g== X-Received: by 2002:a0c:a8e7:: with SMTP id h39mr11424788qvc.45.1611768430096; Wed, 27 Jan 2021 09:27:10 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:09 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 1/6] arm64: kexec: add expandable argument to relocation function Date: Wed, 27 Jan 2021 12:27:01 -0500 Message-Id: <20210127172706.617195-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, kexec relocation function (arm64_relocate_new_kernel) accepts the following arguments: head: start of array that contains relocation information. entry: entry point for new kernel or purgatory. dtb_mem: first and only argument to entry. The number of arguments cannot be easily expended, because this function is also called from HVC_SOFT_RESTART, which preserves only three arguments (hypervisor abi). And, also arm64_relocate_new_kernel is written in assembly but called without stack, thus no place to move extra arguments to free registers. Soon, we will need to pass more arguments: once we enable MMU we will need to pass information about page tables. Add a new struct: kern_reloc_arg, and place it in kexec safe page (i.e memory that is not overwritten during relocation). Thus, make arm64_relocate_new_kernel to only take one argument, that contains all the needed information. Note: Another benefit of allowing this function to accept more arguments, is that kernel can actually accept up to 4 arguments (x0-x3), however currently only one is used, but if in the future we will need for more (for example, pass information about when previous kernel exited to have a precise measurement in time spent in purgatory), we won't be easilty do that if arm64_relocate_new_kernel can't accept more arguments. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 18 ++++++++++++++++++ arch/arm64/kernel/asm-offsets.c | 9 +++++++++ arch/arm64/kernel/cpu-reset.S | 11 +++-------- arch/arm64/kernel/cpu-reset.h | 8 +++----- arch/arm64/kernel/machine_kexec.c | 27 +++++++++++++++++++++++++-- arch/arm64/kernel/relocate_kernel.S | 21 ++++++++------------- 6 files changed, 66 insertions(+), 28 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 9befcd87e9a8..990185744148 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -90,12 +90,30 @@ static inline void crash_prepare_suspend(void) {} static inline void crash_post_resume(void) {} #endif +/* + * kern_reloc_arg is passed to kernel relocation function as an argument. + * head kimage->head, allows to traverse through relocation segments. + * entry_addr kimage->start, where to jump from relocation function (new + * kernel, or purgatory entry address). + * kern_arg0 first argument to kernel is its dtb address. The other + * arguments are currently unused, and must be set to 0 + */ +struct kern_reloc_arg { + phys_addr_t head; + phys_addr_t entry_addr; + phys_addr_t kern_arg0; + phys_addr_t kern_arg1; + phys_addr_t kern_arg2; + phys_addr_t kern_arg3; +}; + #define ARCH_HAS_KIMAGE_ARCH struct kimage_arch { void *dtb; phys_addr_t dtb_mem; phys_addr_t kern_reloc; + phys_addr_t kern_reloc_arg; /* Core ELF header buffer */ void *elf_headers; unsigned long elf_headers_mem; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 301784463587..6067a288f568 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -23,6 +23,7 @@ #include #include #include +#include int main(void) { @@ -150,6 +151,14 @@ int main(void) DEFINE(PTRAUTH_USER_KEY_APGA, offsetof(struct ptrauth_keys_user, apga)); DEFINE(PTRAUTH_KERNEL_KEY_APIA, offsetof(struct ptrauth_keys_kernel, apia)); BLANK(); +#endif +#ifdef CONFIG_KEXEC_CORE + DEFINE(KEXEC_KRELOC_HEAD, offsetof(struct kern_reloc_arg, head)); + DEFINE(KEXEC_KRELOC_ENTRY_ADDR, offsetof(struct kern_reloc_arg, entry_addr)); + DEFINE(KEXEC_KRELOC_KERN_ARG0, offsetof(struct kern_reloc_arg, kern_arg0)); + DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1)); + DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2)); + DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3)); #endif return 0; } diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S index 37721eb6f9a1..bbf70db43744 100644 --- a/arch/arm64/kernel/cpu-reset.S +++ b/arch/arm64/kernel/cpu-reset.S @@ -16,14 +16,11 @@ .pushsection .idmap.text, "awx" /* - * __cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) - Helper for - * cpu_soft_restart. + * __cpu_soft_restart(el2_switch, entry, arg) - Helper for cpu_soft_restart. * * @el2_switch: Flag to indicate a switch to EL2 is needed. * @entry: Location to jump to for soft reset. - * arg0: First argument passed to @entry. (relocation list) - * arg1: Second argument passed to @entry.(physical kernel entry) - * arg2: Third argument passed to @entry. (physical dtb address) + * arg: Entry argument * * Put the CPU into the same state as it would be if it had been reset, and * branch to what would be the reset vector. It must be executed with the @@ -47,9 +44,7 @@ SYM_CODE_START(__cpu_soft_restart) hvc #0 // no return 1: mov x8, x1 // entry - mov x0, x2 // arg0 - mov x1, x3 // arg1 - mov x2, x4 // arg2 + mov x0, x2 // arg br x8 SYM_CODE_END(__cpu_soft_restart) diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h index ed50e9587ad8..7a8720ff186f 100644 --- a/arch/arm64/kernel/cpu-reset.h +++ b/arch/arm64/kernel/cpu-reset.h @@ -11,12 +11,10 @@ #include void __cpu_soft_restart(unsigned long el2_switch, unsigned long entry, - unsigned long arg0, unsigned long arg1, unsigned long arg2); + unsigned long arg); static inline void __noreturn cpu_soft_restart(unsigned long entry, - unsigned long arg0, - unsigned long arg1, - unsigned long arg2) + unsigned long arg) { typeof(__cpu_soft_restart) *restart; @@ -25,7 +23,7 @@ static inline void __noreturn cpu_soft_restart(unsigned long entry, restart = (void *)__pa_symbol(__cpu_soft_restart); cpu_install_idmap(); - restart(el2_switch, entry, arg0, arg1, arg2); + restart(el2_switch, entry, arg); unreachable(); } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 90a335c74442..679db3f1e0c5 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -43,6 +43,7 @@ static void _kexec_image_info(const char *func, int line, pr_debug(" head: %lx\n", kimage->head); pr_debug(" nr_segments: %lu\n", kimage->nr_segments); pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc); + pr_debug(" kern_reloc_arg: %pa\n", &kimage->arch.kern_reloc_arg); for (i = 0; i < kimage->nr_segments; i++) { pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n", @@ -59,19 +60,42 @@ void machine_kexec_cleanup(struct kimage *kimage) /* Empty routine needed to avoid build errors. */ } +/* Allocates pages for kexec page table */ +static void *kexec_page_alloc(void *arg) +{ + struct kimage *kimage = (struct kimage *)arg; + struct page *page = kimage_alloc_control_pages(kimage, 0); + + if (!page) + return NULL; + + memset(page_address(page), 0, PAGE_SIZE); + + return page_address(page); +} + int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); + struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage); + + if (!kern_reloc_arg) + return -ENOMEM; memcpy(reloc_code, arm64_relocate_new_kernel, arm64_relocate_new_kernel_size); kimage->arch.kern_reloc = __pa(reloc_code); + kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg); + kern_reloc_arg->head = kimage->head; + kern_reloc_arg->entry_addr = kimage->start; + kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem; kexec_image_info(kimage); /* Flush the reloc_code in preparation for its execution. */ __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size); flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code + arm64_relocate_new_kernel_size); + __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg)); return 0; } @@ -192,8 +216,7 @@ void machine_kexec(struct kimage *kimage) * userspace (kexec-tools). * In kexec_file case, the kernel starts directly without purgatory. */ - cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start, - kimage->arch.dtb_mem); + cpu_soft_restart(kimage->arch.kern_reloc, kimage->arch.kern_reloc_arg); BUG(); /* Should never get here. */ } diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index b78ea5de97a4..c92228aeddca 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -8,7 +8,7 @@ #include #include - +#include #include #include #include @@ -26,13 +26,8 @@ * safe memory that has been set up to be preserved during the copy operation. */ SYM_CODE_START(arm64_relocate_new_kernel) - /* Setup the list loop variables. */ - mov x18, x2 /* x18 = dtb address */ - mov x17, x1 /* x17 = kimage_start */ - mov x16, x0 /* x16 = kimage_head */ - mov x14, xzr /* x14 = entry ptr */ - mov x13, xzr /* x13 = copy dest */ /* Check if the new image needs relocation. */ + ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 = kimage_head */ tbnz x16, IND_DONE_BIT, .Ldone raw_dcache_line_size x15, x1 /* x15 = dcache line size */ .Lloop: @@ -73,12 +68,12 @@ SYM_CODE_START(arm64_relocate_new_kernel) isb /* Start new image. */ - mov x0, x18 - mov x1, xzr - mov x2, xzr - mov x3, xzr - br x17 - + ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */ + ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3] + ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2] + ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1] + ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */ + br x4 SYM_CODE_END(arm64_relocate_new_kernel) .align 3 /* To keep the 64-bit values below naturally aligned. */ From patchwork Wed Jan 27 17:27:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40E19C433E6 for ; Wed, 27 Jan 2021 17:27:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D3B3764DA8 for ; Wed, 27 Jan 2021 17:27:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D3B3764DA8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 74F8C6B0072; Wed, 27 Jan 2021 12:27:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DB916B0073; Wed, 27 Jan 2021 12:27:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57CE36B0074; Wed, 27 Jan 2021 12:27:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0052.hostedemail.com [216.40.44.52]) by kanga.kvack.org (Postfix) with ESMTP id 395FB6B0072 for ; Wed, 27 Jan 2021 12:27:13 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id EF289181AEF30 for ; Wed, 27 Jan 2021 17:27:12 +0000 (UTC) X-FDA: 77752235904.05.ship93_310d81d27598 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id CABCA180301BC for ; Wed, 27 Jan 2021 17:27:12 +0000 (UTC) X-HE-Tag: ship93_310d81d27598 X-Filterd-Recvd-Size: 9669 Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 17:27:12 +0000 (UTC) Received: by mail-qt1-f170.google.com with SMTP id z6so1992135qtn.0 for ; Wed, 27 Jan 2021 09:27:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=iM7vyMjvp8HFtXYgFN4DAarGfV3AZjiwX6w+vbwvdyg=; b=QMOtRowsZT/Ru/xiPTLv/7JFh1hwOeF1FlFu0MyMoJe50YZ0ukuVerCY5joYBmYk/j dENZG3TuOn7AB27MiwjTj0TrcFWdGNygOEnZ4oGtC8PD8Du3kTYT6MOH+q217OZij3VA xMIee8OoRMSkHIpRftlWwViPysU80FoFq3KYkOC9aLDwgio7a8z5Ksf6AILaSVU0Gxha mSMwr5P5KU4POmhpF7+VQycvc+a+IS3S+icaRsghxQ5TaGvRsRibJJlSlzRdHCgkriq2 mJAX/hxhmlpNg6OOMCgLkRiiCUisawlzo9BxNh/nMUteSZE2Uaa4Vf0zsuQq+CBuEALt cl1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iM7vyMjvp8HFtXYgFN4DAarGfV3AZjiwX6w+vbwvdyg=; b=ALxO3gRTXR2fj2FdAfhf/2yRO34Mh2NsZ9+/hNI7cTsvWc+bNr8pSqR45j0AL5qdSC vSPAd7yVvnPgIwyLJB9ZO6PQx7OXRAeTBHopyZB/EoQWhcf+ohRK0KgTGRz8pPuhteY9 Cao3HC1czv8U5WZPU3gMLJlTNb6Fx0jOYrpjC2EEXeM2PE4UF8mHBSprzuXp6vOKjxYX pFz4eF7FaXXyAH0S8ffzGe+fy6zK9K710zy/gsLwr1hNXNQUZQQZ1QbJzb+icE7ic3uO 76hudvl/uE8A6bDmIvP3Ua5fGpNF3Xp3P4TZsVHTV5ZKQ5nEUwQeKLCO5m+XHWNwAaPD kneQ== X-Gm-Message-State: AOAM532z3mQ6LUr+rtWy9nDwOFD751AuVkhJ6incUDSaONFTwoGln9Ee bXhXIOlQYJU4od/OeL11oA3h+rPfKANtKcjv X-Google-Smtp-Source: ABdhPJz2pmeuW3zsZ09pL8xkdh0pAeu9616r0G+I0/+leO2zQYI1rzh9PJ65p7HB/lI/13nJfQJHcA== X-Received: by 2002:ac8:118c:: with SMTP id d12mr10787471qtj.262.1611768431629; Wed, 27 Jan 2021 09:27:11 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:11 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 2/6] arm64: kexec: use ld script for relocation function Date: Wed, 27 Jan 2021 12:27:02 -0500 Message-Id: <20210127172706.617195-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, relocation code declares start and end variables which are used to compute it size. The better way to do this is to use ld script incited, and put relocation function in its own section. Soon, relocation function will share the same page with EL2 vectors. So, proper marking is needed. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 4 ++++ arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/machine_kexec.c | 17 ++++++++--------- arch/arm64/kernel/relocate_kernel.S | 15 ++------------- arch/arm64/kernel/vmlinux.lds.S | 19 +++++++++++++++++++ 5 files changed, 34 insertions(+), 22 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 990185744148..7f4f9abdf049 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -90,6 +90,10 @@ static inline void crash_prepare_suspend(void) {} static inline void crash_post_resume(void) {} #endif +#if defined(CONFIG_KEXEC_CORE) +extern const char arm64_relocate_new_kernel[]; +#endif + /* * kern_reloc_arg is passed to kernel relocation function as an argument. * head kimage->head, allows to traverse through relocation segments. diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index 8ff579361731..ae873eb22205 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -19,5 +19,6 @@ extern char __exittext_begin[], __exittext_end[]; extern char __irqentry_text_start[], __irqentry_text_end[]; extern char __mmuoff_data_start[], __mmuoff_data_end[]; extern char __entry_tramp_text_start[], __entry_tramp_text_end[]; +extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[]; #endif /* __ASM_SECTIONS_H */ diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 679db3f1e0c5..361a4d082093 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -20,13 +20,10 @@ #include #include #include +#include #include "cpu-reset.h" -/* Global variables for the arm64_relocate_new_kernel routine. */ -extern const unsigned char arm64_relocate_new_kernel[]; -extern const unsigned long arm64_relocate_new_kernel_size; - /** * kexec_image_info - For debugging output. */ @@ -78,13 +75,15 @@ int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage); + long func_offset, reloc_size; if (!kern_reloc_arg) return -ENOMEM; - memcpy(reloc_code, arm64_relocate_new_kernel, - arm64_relocate_new_kernel_size); - kimage->arch.kern_reloc = __pa(reloc_code); + func_offset = arm64_relocate_new_kernel - __relocate_new_kernel_start; + reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; + memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); + kimage->arch.kern_reloc = __pa(reloc_code) + func_offset; kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg); kern_reloc_arg->head = kimage->head; kern_reloc_arg->entry_addr = kimage->start; @@ -92,9 +91,9 @@ int machine_kexec_post_load(struct kimage *kimage) kexec_image_info(kimage); /* Flush the reloc_code in preparation for its execution. */ - __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size); + __flush_dcache_area(reloc_code, reloc_size); flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code + - arm64_relocate_new_kernel_size); + reloc_size); __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg)); return 0; diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index c92228aeddca..d2a4a0b0d76b 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -14,6 +14,7 @@ #include #include +.pushsection ".kexec_relocate.text", "ax" /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. * @@ -75,16 +76,4 @@ SYM_CODE_START(arm64_relocate_new_kernel) ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */ br x4 SYM_CODE_END(arm64_relocate_new_kernel) - -.align 3 /* To keep the 64-bit values below naturally aligned. */ - -.Lcopy_end: -.org KEXEC_CONTROL_PAGE_SIZE - -/* - * arm64_relocate_new_kernel_size - Number of bytes to copy to the - * control_code_page. - */ -.globl arm64_relocate_new_kernel_size -arm64_relocate_new_kernel_size: - .quad .Lcopy_end - arm64_relocate_new_kernel +.popsection diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 4c0b0c89ad59..33b0d3c9fd3b 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -82,6 +83,16 @@ jiffies = jiffies_64; #define HIBERNATE_TEXT #endif +#ifdef CONFIG_KEXEC_CORE +#define KEXEC_TEXT \ + . = ALIGN(SZ_4K); \ + __relocate_new_kernel_start = .; \ + *(.kexec_relocate.text) \ + __relocate_new_kernel_end = .; +#else +#define KEXEC_TEXT +#endif + #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 #define TRAMP_TEXT \ . = ALIGN(PAGE_SIZE); \ @@ -142,6 +153,7 @@ SECTIONS HYPERVISOR_TEXT IDMAP_TEXT HIBERNATE_TEXT + KEXEC_TEXT TRAMP_TEXT *(.fixup) *(.gnu.warning) @@ -316,3 +328,10 @@ ASSERT((__entry_tramp_text_end - __entry_tramp_text_start) == PAGE_SIZE, * If padding is applied before .head.text, virt<->phys conversions will fail. */ ASSERT(_text == KIMAGE_VADDR, "HEAD is misaligned") + +#ifdef CONFIG_KEXEC_CORE +/* kexec relocation code should fit into one KEXEC_CONTROL_PAGE_SIZE */ +ASSERT(__relocate_new_kernel_end - (__relocate_new_kernel_start & ~(SZ_4K - 1)) + <= SZ_4K, "kexec relocation code is too big or misaligned") +ASSERT(KEXEC_CONTROL_PAGE_SIZE >= SZ_4K, "KEXEC_CONTROL_PAGE_SIZE is brokern") +#endif From patchwork Wed Jan 27 17:27:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCF53C433E0 for ; Wed, 27 Jan 2021 17:27:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 68AC764DA6 for ; Wed, 27 Jan 2021 17:27:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 68AC764DA6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0ED6F6B0073; Wed, 27 Jan 2021 12:27:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0282A6B0074; Wed, 27 Jan 2021 12:27:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E05806B0075; Wed, 27 Jan 2021 12:27:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C73AC6B0073 for ; Wed, 27 Jan 2021 12:27:14 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 94D278249980 for ; Wed, 27 Jan 2021 17:27:14 +0000 (UTC) X-FDA: 77752235988.18.heart89_040cb6d27598 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id 5E6AD100ED3CB for ; Wed, 27 Jan 2021 17:27:14 +0000 (UTC) X-HE-Tag: heart89_040cb6d27598 X-Filterd-Recvd-Size: 8746 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 17:27:13 +0000 (UTC) Received: by mail-qt1-f181.google.com with SMTP id c1so1983732qtc.1 for ; Wed, 27 Jan 2021 09:27:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=lbvVstnavugORmz+7RoYwB1ceirdtrsJhtHKQ5onWhE=; b=QV+R8Re8BFcvEa3KJeWhAPQnSVBkbq5dabdVCP2CjZZ8ZSwcwRvXqu83WHuCg68EzE pP/iQOyEwDyg1Iqy+YnkFzs5hZjA8YahurgylMOElXDiWUjz08sJoZL7cLIybd2FokX4 /AbrCaUi6+ckqv6Ai7V2lFdBNm0v1Ee6D30znoHnux3cOcDF8kaLEoB3XNte8NUC1/V9 mz4tMZapzAEf3Tz97RChN/qhHCaBbbUGMJcCsJihf073iEYNlIIChdftQDBryKBA5mDd ewU32BTqwSlRsSmXhTQGNNPz/Jn6wVmbysMfXLLufw9WsmDRthUkfBaXWHLYiETpzofh EMAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lbvVstnavugORmz+7RoYwB1ceirdtrsJhtHKQ5onWhE=; b=akt2Wxfd516MyfWlqI7+3hd6tbXOWvEkKHI1FzSHP64ViVBQcMSEUUBcHSPLGVvTZ5 ZH99dx0r8xP6DZsi0MFpVPKN8nYGPGdA3k27zB/UwGXLTsWZrUWbEfBbrw+jXz8l7tLG y/GjjI+zrkwMXKgXayUOnAFAwTvDvgS7gIW84uVGKOUY4ULoXgFt72cHh4nzmEFQjUJu aRzo8jHfQivCrzwbLEhPgmj64VaZzf5RL/40CICFKEyQC7/km4rUaO14ae9HoVBE9aB5 gAxJCn5ZpSGVLMv579BQa14AcVSsPwwkJttvABzqC1MeH2CEaj5XaX2wb08wpM0WxmBG /55w== X-Gm-Message-State: AOAM531xRNyeuTY/1W1FIa1qpNk1ID5bU9ZNpYQAr9aKh56LySMEZhDa f0Vbgqf+0iweF76/p/FQ3YWOAw== X-Google-Smtp-Source: ABdhPJz6VAe7Hg9z00hIz2UDa6TgtFcqklDUbjVYu90Qteg2VkXL4vXeuEwM1lTUmzQASkqOin5HYw== X-Received: by 2002:a05:622a:201:: with SMTP id b1mr10304370qtx.237.1611768433197; Wed, 27 Jan 2021 09:27:13 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:12 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 3/6] arm64: kexec: kexec may require EL2 vectors Date: Wed, 27 Jan 2021 12:27:03 -0500 Message-Id: <20210127172706.617195-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we have a EL2 mode without VHE, the EL2 vectors are needed in order to switch to EL2 and jump to new world with hypervisor privileges. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 5 +++++ arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/machine_kexec.c | 9 +++++++- arch/arm64/kernel/relocate_kernel.S | 35 +++++++++++++++++++++++++++++ 4 files changed, 49 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 7f4f9abdf049..b96d8a6aac80 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -92,6 +92,7 @@ static inline void crash_post_resume(void) {} #if defined(CONFIG_KEXEC_CORE) extern const char arm64_relocate_new_kernel[]; +extern const char arm64_kexec_el2_vectors[]; #endif /* @@ -101,6 +102,9 @@ extern const char arm64_relocate_new_kernel[]; * kernel, or purgatory entry address). * kern_arg0 first argument to kernel is its dtb address. The other * arguments are currently unused, and must be set to 0 + * el2_vector If present means that relocation routine will go to EL1 + * from EL2 to do the copy, and then back to EL2 to do the jump + * to new world. */ struct kern_reloc_arg { phys_addr_t head; @@ -109,6 +113,7 @@ struct kern_reloc_arg { phys_addr_t kern_arg1; phys_addr_t kern_arg2; phys_addr_t kern_arg3; + phys_addr_t el2_vector; }; #define ARCH_HAS_KIMAGE_ARCH diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 6067a288f568..8a9475be1b62 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -159,6 +159,7 @@ int main(void) DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1)); DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2)); DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3)); + DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector)); #endif return 0; } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 361a4d082093..41d1e3ca13f8 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -75,19 +75,26 @@ int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage); - long func_offset, reloc_size; + long func_offset, vector_offset, reloc_size; if (!kern_reloc_arg) return -ENOMEM; func_offset = arm64_relocate_new_kernel - __relocate_new_kernel_start; reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; + vector_offset = arm64_kexec_el2_vectors - __relocate_new_kernel_start; + memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code) + func_offset; kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg); kern_reloc_arg->head = kimage->head; kern_reloc_arg->entry_addr = kimage->start; kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem; + + /* Setup vector table only when EL2 is available, but no VHE */ + if (is_hyp_mode_available() && !is_kernel_in_hyp_mode()) + kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset; + kexec_image_info(kimage); /* Flush the reloc_code in preparation for its execution. */ diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index d2a4a0b0d76b..c6178b1a4e60 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -14,6 +14,17 @@ #include #include +.macro el1_sync_64 + .align 7 + br x4 /* Jump to new world from el2 */ +.endm + +.macro invalid_vector label +\label: + .align 7 + b \label +.endm + .pushsection ".kexec_relocate.text", "ax" /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. @@ -76,4 +87,28 @@ SYM_CODE_START(arm64_relocate_new_kernel) ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */ br x4 SYM_CODE_END(arm64_relocate_new_kernel) + +/* el2 vectors - switch el2 here while we restore the memory image. */ + .align 11 +SYM_CODE_START(arm64_kexec_el2_vectors) + invalid_vector el2_sync_invalid_sp0 /* Synchronous EL2t */ + invalid_vector el2_irq_invalid_sp0 /* IRQ EL2t */ + invalid_vector el2_fiq_invalid_sp0 /* FIQ EL2t */ + invalid_vector el2_error_invalid_sp0 /* Error EL2t */ + + invalid_vector el2_sync_invalid_spx /* Synchronous EL2h */ + invalid_vector el2_irq_invalid_spx /* IRQ EL2h */ + invalid_vector el2_fiq_invalid_spx /* FIQ EL2h */ + invalid_vector el2_error_invalid_spx /* Error EL2h */ + + el1_sync_64 /* Synchronous 64-bit EL1 */ + invalid_vector el1_irq_invalid_64 /* IRQ 64-bit EL1 */ + invalid_vector el1_fiq_invalid_64 /* FIQ 64-bit EL1 */ + invalid_vector el1_error_invalid_64 /* Error 64-bit EL1 */ + + invalid_vector el1_sync_invalid_32 /* Synchronous 32-bit EL1 */ + invalid_vector el1_irq_invalid_32 /* IRQ 32-bit EL1 */ + invalid_vector el1_fiq_invalid_32 /* FIQ 32-bit EL1 */ + invalid_vector el1_error_invalid_32 /* Error 32-bit EL1 */ +SYM_CODE_END(arm64_kexec_el2_vectors) .popsection From patchwork Wed Jan 27 17:27:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A248BC433DB for ; Wed, 27 Jan 2021 17:27:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0C60E64DA5 for ; Wed, 27 Jan 2021 17:27:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0C60E64DA5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B9A616B0074; Wed, 27 Jan 2021 12:27:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A39756B0075; Wed, 27 Jan 2021 12:27:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D6D06B0078; Wed, 27 Jan 2021 12:27:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0110.hostedemail.com [216.40.44.110]) by kanga.kvack.org (Postfix) with ESMTP id 6E8A96B0074 for ; Wed, 27 Jan 2021 12:27:16 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 266CD3632 for ; Wed, 27 Jan 2021 17:27:16 +0000 (UTC) X-FDA: 77752236072.05.body97_2e177d727598 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 08EF818017AA2 for ; Wed, 27 Jan 2021 17:27:16 +0000 (UTC) X-HE-Tag: body97_2e177d727598 X-Filterd-Recvd-Size: 10047 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 17:27:15 +0000 (UTC) Received: by mail-qt1-f179.google.com with SMTP id v3so1973363qtw.4 for ; Wed, 27 Jan 2021 09:27:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=SdcRL3eX4lOwXH8jydL/qdL0IVgZG3cGF34GsbvsmD8=; b=K3B0SdmdyKapkNJZMDzEgXozpmx8/0k4lvny4yseNM4bQmkE44PvsM9aK1V/2oYXRH L12qNPPnei6EWRQu+on8qdESTEknOP+uHmmDxPCWxucBOWENNtaR7pi86cpaTOj/N3DG DVivHIxQDW5asZvrdaCEGm/dr5D4LQBhigXNZcC1CVlGx9C60VVT+R1H1eSTzxA4DFnk SL/Uq8Xmx68QpPaQN+9N79M1ux+9p6ACjmro9sZwM+mEP3Htd8tjDb9N58tdwoTDJA4T hk0L2UYTRayDnWTuYIzdJPHhljf00W6pbGLcQHZ1rlHDfOQ3aJzz7sSpgBJJtfB2uTaj iVew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SdcRL3eX4lOwXH8jydL/qdL0IVgZG3cGF34GsbvsmD8=; b=HZz5buSXjTXl1for2ZwOE1lX4XKXhnKXk+s7U6AGQ35AZqXUB9kvG6RFBgFlIDluQ0 WBrkbENPusGQApLL4v116E4/iN3UqDVgx2ALsDOIodF9+P2zhgKTYu0N6ZLQ44w2JwaW hURlwo8Ewyy3oPnX4Awr2lAagQbzLeAYIsVErU21TRu783kTVebtC419qutsq3+jxd3N eHnHfI4sbO61HxWYV1JfG7lZ1FviZ2LnEP+k4SRBaTvuWjMrv0ZSsc3FijDokW2N99cL nrF7l9At9mAyRMa90jS7Izce3jlMyTRQOfHBbJEboQFhDZ1a3HM5U3dZ7zWByH2XCa2z 96Cw== X-Gm-Message-State: AOAM531mSLKTgWZii7NOSFiHv9v4jII3cANuy82tZhAbpf8LwZrTAhz8 IdnIaW4BdlUOrvhVCd72RkIeOA== X-Google-Smtp-Source: ABdhPJxlFV1txevvNgU5xnq79+y1SsCRubMlvuyhI8QVtE3ik/wKyoAO5LODelM9gvODmj3UENO8cQ== X-Received: by 2002:aed:2123:: with SMTP id 32mr10909736qtc.325.1611768434815; Wed, 27 Jan 2021 09:27:14 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:14 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 4/6] arm64: kexec: configure trans_pgd page table for kexec Date: Wed, 27 Jan 2021 12:27:04 -0500 Message-Id: <20210127172706.617195-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Configure a page table located in kexec-safe memory that has the following mappings: 1. identity mapping for text of relocation function with executable permission. 2. va mappings for all source ranges 3. va mappings for all destination ranges. Signed-off-by: Pavel Tatashin --- arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/kexec.h | 12 +++++ arch/arm64/kernel/asm-offsets.c | 6 +++ arch/arm64/kernel/machine_kexec.c | 89 ++++++++++++++++++++++++++++++- 4 files changed, 107 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index fc0ed9d6e011..440abd0c0ee1 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1134,7 +1134,7 @@ config CRASH_DUMP config TRANS_TABLE def_bool y - depends on HIBERNATION + depends on HIBERNATION || KEXEC_CORE config XEN_DOM0 def_bool y diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index b96d8a6aac80..049cde429b1b 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[]; * el2_vector If present means that relocation routine will go to EL1 * from EL2 to do the copy, and then back to EL2 to do the jump * to new world. + * trans_ttbr0 idmap for relocation function and its argument + * trans_ttbr1 map for source/destination addresses. + * trans_t0sz t0sz for idmap page in trans_ttbr0 + * src_addr start address for source pages. + * dst_addr start address for destination pages. + * copy_len Number of bytes that need to be copied */ struct kern_reloc_arg { phys_addr_t head; @@ -114,6 +120,12 @@ struct kern_reloc_arg { phys_addr_t kern_arg2; phys_addr_t kern_arg3; phys_addr_t el2_vector; + phys_addr_t trans_ttbr0; + phys_addr_t trans_ttbr1; + unsigned long trans_t0sz; + unsigned long src_addr; + unsigned long dst_addr; + unsigned long copy_len; }; #define ARCH_HAS_KIMAGE_ARCH diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 8a9475be1b62..06278611451d 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -160,6 +160,12 @@ int main(void) DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2)); DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3)); DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector)); + DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0)); + DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1)); + DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz)); + DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr)); + DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr)); + DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len)); #endif return 0; } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 41d1e3ca13f8..9588c91f67c6 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "cpu-reset.h" @@ -71,11 +72,89 @@ static void *kexec_page_alloc(void *arg) return page_address(page); } +/* + * Map source segments starting from src_va, and map destination + * segments starting from dst_va, and return size of copy in + * *copy_len argument. + * Relocation function essentially needs to do: + * memcpy(dst_va, src_va, copy_len); + */ +static int map_segments(struct kimage *kimage, pgd_t *pgdp, + struct trans_pgd_info *info, + unsigned long src_va, + unsigned long dst_va, + unsigned long *copy_len) +{ + unsigned long *ptr = 0; + unsigned long dest = 0; + unsigned long len = 0; + unsigned long entry, addr; + int rc; + + for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) { + addr = entry & PAGE_MASK; + + switch (entry & IND_FLAGS) { + case IND_DESTINATION: + dest = addr; + break; + case IND_INDIRECTION: + ptr = __va(addr); + break; + case IND_SOURCE: + rc = trans_pgd_map_page(info, pgdp, __va(addr), + src_va, PAGE_KERNEL); + if (rc) + return rc; + rc = trans_pgd_map_page(info, pgdp, __va(dest), + dst_va, PAGE_KERNEL); + if (rc) + return rc; + dest += PAGE_SIZE; + src_va += PAGE_SIZE; + dst_va += PAGE_SIZE; + len += PAGE_SIZE; + } + } + *copy_len = len; + + return 0; +} + +static int mmu_relocate_setup(struct kimage *kimage, void *reloc_code, + struct kern_reloc_arg *kern_reloc_arg) +{ + struct trans_pgd_info info = { + .trans_alloc_page = kexec_page_alloc, + .trans_alloc_arg = kimage, + }; + pgd_t *trans_pgd = kexec_page_alloc(kimage); + int rc; + + if (!trans_pgd) + return -ENOMEM; + + /* idmap relocation function */ + rc = trans_pgd_idmap_page(&info, &kern_reloc_arg->trans_ttbr0, + &kern_reloc_arg->trans_t0sz, reloc_code); + if (rc) + return rc; + + kern_reloc_arg->src_addr = _PAGE_OFFSET(VA_BITS_MIN); + kern_reloc_arg->dst_addr = _PAGE_OFFSET(VA_BITS_MIN - 1); + kern_reloc_arg->trans_ttbr1 = phys_to_ttbr(__pa(trans_pgd)); + + rc = map_segments(kimage, trans_pgd, &info, kern_reloc_arg->src_addr, + kern_reloc_arg->dst_addr, &kern_reloc_arg->copy_len); + return rc; +} + int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage); long func_offset, vector_offset, reloc_size; + int rc = 0; if (!kern_reloc_arg) return -ENOMEM; @@ -95,6 +174,14 @@ int machine_kexec_post_load(struct kimage *kimage) if (is_hyp_mode_available() && !is_kernel_in_hyp_mode()) kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset; + /* + * If relocation is not needed, we do not need to enable MMU in + * relocation routine, therefore do not create page tables for + * scenarios such as crash kernel + */ + if (!(kimage->head & IND_DONE)) + rc = mmu_relocate_setup(kimage, reloc_code, kern_reloc_arg); + kexec_image_info(kimage); /* Flush the reloc_code in preparation for its execution. */ @@ -103,7 +190,7 @@ int machine_kexec_post_load(struct kimage *kimage) reloc_size); __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg)); - return 0; + return rc; } /** From patchwork Wed Jan 27 17:27:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42F30C433E0 for ; Wed, 27 Jan 2021 17:27:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B16E664DA3 for ; Wed, 27 Jan 2021 17:27:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B16E664DA3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B95C66B0075; Wed, 27 Jan 2021 12:27:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AF5BF6B0078; Wed, 27 Jan 2021 12:27:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9748C6B007B; Wed, 27 Jan 2021 12:27:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0106.hostedemail.com [216.40.44.106]) by kanga.kvack.org (Postfix) with ESMTP id 73E696B0075 for ; Wed, 27 Jan 2021 12:27:18 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2D842180AD822 for ; Wed, 27 Jan 2021 17:27:18 +0000 (UTC) X-FDA: 77752236156.19.car89_220f69227598 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id D53801AD1B5 for ; Wed, 27 Jan 2021 17:27:17 +0000 (UTC) X-HE-Tag: car89_220f69227598 X-Filterd-Recvd-Size: 9272 Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 17:27:17 +0000 (UTC) Received: by mail-qk1-f182.google.com with SMTP id l27so2467380qki.9 for ; Wed, 27 Jan 2021 09:27:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=RDSr+b90kmy9FHMSi/QfI0SbQbMsMCPJRwK16BQ54DQ=; b=D0vmR3k2J3V59Tf9s9cd/7e81xmYwG30fQDjq0TRTMFYsuLttnJjFOhyv3py9kd7e8 tR5GuoAbb317sw2avBKR0hwLqYH9ztq8Nfx4j3Puh61MMGT3Mk5McC8tw6KOcxr0zlRw mV/fer9zN7LgR0lkpI88m/eb4RjED+TibTTVUsaJyqa2x2EyVDjCGgnfgdho0CaA7dVk l/m3T/XDxDWPoe5jkoOsIpUFsHV4f98IJvnq3Na4WzdaI4x424oe0lbOkQ/mLZyhS6Tp fMkcLsHgwzYABYXvHW/EoysJnRU6e+lZCq0osuDKHNa5I53k0XRd8dxF/APoFx4Jbv0U wDAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RDSr+b90kmy9FHMSi/QfI0SbQbMsMCPJRwK16BQ54DQ=; b=MHi/70XmfTutUn+I6OjhDi0DgLt6q0FExRKxiqSciGGmzJJRgJZZJWrPJDH4PFtH0P lNqCfRV3Dw7kCaLJ1f2PtNi6swijgV2TZMv19XlcaorsIPYr6df2L+AiPop1CvJhxuHU H+5jAT1eiCDlRzQ2nw8VSvbI7W3PWcXPToQm5G+lVLLnJPmyfIWfIxly+TEbBnbuMPoQ zDX0D3+n/vFaiaxbjBZTg1H7Fh5PPt3cvIsDPgae94fVYZgbnPn6LwngyGgzLWQzO5/u soCTqIfX73NFQmXAILN0vCde3AUd5MzyXye8amJAmCVBGBXfHEKDdA2ggmMONw31CPKC 0YSg== X-Gm-Message-State: AOAM533aKPWp93LAHzyQdYpWf/4RIMCWvsQZYFMoUUm+qCJy+YTM3oPT ftjWUjIKLn7mgk3sHr1diyUIqA== X-Google-Smtp-Source: ABdhPJy1F3OJMWVpbhDsfEHVOUOuSIdueuuHWt8/BvaPPfMVcsC5saRFBDHsiaILyCDjZvE/5KZc7w== X-Received: by 2002:a37:2741:: with SMTP id n62mr11561676qkn.43.1611768436393; Wed, 27 Jan 2021 09:27:16 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:15 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 5/6] arm64: kexec: enable MMU during kexec relocation Date: Wed, 27 Jan 2021 12:27:05 -0500 Message-Id: <20210127172706.617195-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now, that we have transitional page tables configured, temporarily enable MMU to allow faster relocation of segments to final destination. The performance data: for a moderate size kernel + initramfs: 25M the relocation was taking 0.382s, with enabled MMU it now takes 0.019s only or x20 improvement. The time is proportional to the size of relocation, therefore if initramfs is larger, 100M it could take over a second. Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/relocate_kernel.S | 131 ++++++++++++++++++---------- 1 file changed, 87 insertions(+), 44 deletions(-) diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index c6178b1a4e60..9c60981a6911 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -4,6 +4,8 @@ * * Copyright (C) Linaro. * Copyright (C) Huawei Futurewei Technologies. + * Copyright (C) 2020, Microsoft Corporation. + * Pavel Tatashin */ #include @@ -14,6 +16,54 @@ #include #include +.macro tlb_invalidate + dsb sy + dsb ish + tlbi vmalle1 + dsb ish + isb +.endm + +.macro turn_off_mmu tmp1, tmp2 + mrs \tmp1, sctlr_el1 + mov_q \tmp2, SCTLR_ELx_FLAGS + bic \tmp1, \tmp1, \tmp2 + pre_disable_mmu_workaround + msr sctlr_el1, \tmp1 + isb +.endm + +.macro turn_on_mmu tmp1, tmp2 + mrs \tmp1, sctlr_el1 + mov_q \tmp2, SCTLR_ELx_FLAGS + orr \tmp1, \tmp1, \tmp2 + msr sctlr_el1, \tmp1 + ic iallu + dsb nsh + isb +.endm + +/* + * Set ttbr0 and ttbr1, called while MMU is disabled, so no need to temporarily + * set zero_page table. Invalidate TLB after new tables are set. + */ +.macro set_ttbr arg, tmp1, tmp2 + ldr \tmp1, [\arg, #KEXEC_KRELOC_TRANS_TTBR0] + msr ttbr0_el1, \tmp1 + ldr \tmp1, [\arg, #KEXEC_KRELOC_TRANS_TTBR1] + offset_ttbr1 \tmp1, \tmp2 + msr ttbr1_el1, \tmp1 + isb +.endm + +/* Set T0SZ to match the requirements of idmap page */ +.macro set_tcr_t0sz arg, tmp1, tmp2 + ldr \tmp2, [\arg, #KEXEC_KRELOC_TRANS_T0SZ] + mrs \tmp1, tcr_el1 + bfi \tmp1, \tmp2, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH + msr tcr_el1, \tmp1 +.endm + .macro el1_sync_64 .align 7 br x4 /* Jump to new world from el2 */ @@ -36,56 +86,49 @@ * symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. The * machine_kexec() routine will copy arm64_relocate_new_kernel to the kexec * safe memory that has been set up to be preserved during the copy operation. + * + * This function temporarily enables MMU if kernel relocation is needed. + * Also, if we enter this function at EL2 on non-VHE kernel, we temporarily go + * to EL1 to enable MMU, and escalate back to EL2 at the end to do the jump to + * the new kernel. This is determined by presence of el2_vector. */ SYM_CODE_START(arm64_relocate_new_kernel) - /* Check if the new image needs relocation. */ - ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 = kimage_head */ - tbnz x16, IND_DONE_BIT, .Ldone - raw_dcache_line_size x15, x1 /* x15 = dcache line size */ -.Lloop: - and x12, x16, PAGE_MASK /* x12 = addr */ - - /* Test the entry flags. */ -.Ltest_source: - tbz x16, IND_SOURCE_BIT, .Ltest_indirection - - /* Invalidate dest page to PoC. */ - mov x2, x13 - add x20, x2, #PAGE_SIZE - sub x1, x15, #1 - bic x2, x2, x1 -2: dc ivac, x2 - add x2, x2, x15 - cmp x2, x20 - b.lo 2b - dsb sy - - copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 - b .Lnext -.Ltest_indirection: - tbz x16, IND_INDIRECTION_BIT, .Ltest_destination - mov x14, x12 /* ptr = addr */ - b .Lnext -.Ltest_destination: - tbz x16, IND_DESTINATION_BIT, .Lnext - mov x13, x12 /* dest = addr */ -.Lnext: - ldr x16, [x14], #8 /* entry = *ptr++ */ - tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */ -.Ldone: - /* wait for writes from copy_page to finish */ - dsb nsh - ic iallu - dsb nsh - isb - - /* Start new image. */ - ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */ + mov x20, xzr /* x20 will hold vector value */ + ldr x11, [x0, #KEXEC_KRELOC_COPY_LEN] + cbz x11, 5f /* Check if need to relocate */ + ldr x20, [x0, #KEXEC_KRELOC_EL2_VECTOR] + cbz x20, 2f /* need to reduce to EL1? */ + msr vbar_el2, x20 /* el2_vector present, means */ + adr x1, 2f /* we will do copy in el1 but */ + msr elr_el2, x1 /* do final jump from el2 */ + eret /* Reduce to EL1 */ +2: set_tcr_t0sz x0, x1, x2 /* Set t0sz for idmaped page */ + set_ttbr x0, x1, x2 /* Set our page tables */ + tlb_invalidate + ldr x1, [x0, #KEXEC_KRELOC_DST_ADDR]; /* arg is not idmapped so */ + ldr x2, [x0, #KEXEC_KRELOC_SRC_ADDR]; /* read before MMU is on */ + turn_on_mmu x3, x4 /* Turn MMU back on */ + mov x12, x1 /* x12 dst backup */ +3: copy_page x1, x2, x3, x4, x5, x6, x7, x8, x9, x10 + sub x11, x11, #PAGE_SIZE + cbnz x11, 3b /* page copy loop */ + raw_dcache_line_size x2, x3 /* x2 = dcache line size */ + sub x3, x2, #1 /* x3 = dcache_size - 1 */ + bic x12, x12, x3 +4: dc cvau, x12 /* Flush D-cache */ + add x12, x12, x2 + cmp x12, x1 /* Compare to dst + len */ + b.ne 4b /* D-cache flush loop */ + turn_off_mmu x1, x2 /* Turn off MMU */ + tlb_invalidate /* Invalidate TLB */ +5: ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */ ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3] ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2] ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1] ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */ - br x4 + cbnz x20, 6f /* need to escalate to el2? */ + br x4 /* Jump to new world */ +6: hvc #0 /* enters kexec_el1_sync */ SYM_CODE_END(arm64_relocate_new_kernel) /* el2 vectors - switch el2 here while we restore the memory image. */ From patchwork Wed Jan 27 17:27:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B4CEC433E0 for ; Wed, 27 Jan 2021 17:27:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9BFCB64DA6 for ; Wed, 27 Jan 2021 17:27:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9BFCB64DA6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 096756B0078; Wed, 27 Jan 2021 12:27:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 01F466B007B; Wed, 27 Jan 2021 12:27:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E04336B007D; Wed, 27 Jan 2021 12:27:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0150.hostedemail.com [216.40.44.150]) by kanga.kvack.org (Postfix) with ESMTP id A19996B0078 for ; Wed, 27 Jan 2021 12:27:43 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4A3DA180AD822 for ; Wed, 27 Jan 2021 17:27:43 +0000 (UTC) X-FDA: 77752237206.01.team85_05149b127598 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 32E101004A081 for ; Wed, 27 Jan 2021 17:27:19 +0000 (UTC) X-HE-Tag: team85_05149b127598 X-Filterd-Recvd-Size: 5752 Received: from mail-qv1-f52.google.com (mail-qv1-f52.google.com [209.85.219.52]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 17:27:18 +0000 (UTC) Received: by mail-qv1-f52.google.com with SMTP id p5so1420847qvs.7 for ; Wed, 27 Jan 2021 09:27:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=cfSnNSnfVduFxCPOvWfs9p+QSvXlJihsz/H5U3Y4H1U=; b=JSF/HlHCJzShZn7ALunS0BkmdIvbsb31kLfU+RnIPOJjHHHbnPC+hKp5GzFf5aNfG0 PCnd0hUx8KnQTQJAs7hc3j8UjBo0nhsavr2YgBKj48WT2dlprkP86SX5YogstiQFgEW1 fQXmcUFNK/whrTvXsAd2ttbgakW8eugEXjKNmuKRJBUbncvkUaYlDCqNXZhKNp4QGgZi s9l3aBEqnDiNAea7t1MlSQvIt3uOdGGZZwDbfriUGm56jfCILeRPYfkcC4aBIVfBChfU myBdivbvXZCSHraxlvN/kcYnz6cOQzbdEGPqEKU9R4Tk+vISJcmeJUo2Io2EH78AfPDL BLQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cfSnNSnfVduFxCPOvWfs9p+QSvXlJihsz/H5U3Y4H1U=; b=emQPSYN+UlkkDhr5YVnqhV1MIsAAKN9iEebyif8nYjP9svjPp+jbr+q03Rd0dZaQ5I GBB2yy2+nhB7tOAtD7o7LZBF9w7B90CR8a6Am89XW6rEBSou/U9rX9wheLn9OymLMU3T WIf+YdIuOPgzUMLoYVmoUo1X0hiF6PbhT51wVhKVxcG1ymliRTE8WqC7WGLWOU6esPv3 jGvRQbJURq0bwvJiXJEhpcAh+XfU7bqFEFWTssWBMm2pWj6dbcThRWWXsfXB7rgVH8sj hXoc6L/buqql6bmF9Ucao3JIEDCn8QuXR/AQoxKSjnVEwVprl3Ec3P16LTKUmzxRr8H8 faVw== X-Gm-Message-State: AOAM5307U+q4ngTup7mZKYtH8Ded6tiSfHDeDc2l4+ghyvKx2hdhFb6X UC6iO7LA371jc5WUMLwnP9ibqQ== X-Google-Smtp-Source: ABdhPJwHsa1PlzOcEErpv+fEXO2+H7UWl48S+Yfvc0TUYpC69d2qgqgALhqP+OIOSgJyhuzsv33wBA== X-Received: by 2002:a0c:99c4:: with SMTP id y4mr11377443qve.9.1611768437920; Wed, 27 Jan 2021 09:27:17 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:17 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 6/6] arm64: kexec: remove head from relocation argument Date: Wed, 27 Jan 2021 12:27:06 -0500 Message-Id: <20210127172706.617195-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now, that relocation is done using virtual addresses, reloc_arg->head is not needed anymore. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 2 -- arch/arm64/kernel/asm-offsets.c | 1 - arch/arm64/kernel/machine_kexec.c | 1 - 3 files changed, 4 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 049cde429b1b..2fa4109bd582 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -97,7 +97,6 @@ extern const char arm64_kexec_el2_vectors[]; /* * kern_reloc_arg is passed to kernel relocation function as an argument. - * head kimage->head, allows to traverse through relocation segments. * entry_addr kimage->start, where to jump from relocation function (new * kernel, or purgatory entry address). * kern_arg0 first argument to kernel is its dtb address. The other @@ -113,7 +112,6 @@ extern const char arm64_kexec_el2_vectors[]; * copy_len Number of bytes that need to be copied */ struct kern_reloc_arg { - phys_addr_t head; phys_addr_t entry_addr; phys_addr_t kern_arg0; phys_addr_t kern_arg1; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 06278611451d..94f050ad6471 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -153,7 +153,6 @@ int main(void) BLANK(); #endif #ifdef CONFIG_KEXEC_CORE - DEFINE(KEXEC_KRELOC_HEAD, offsetof(struct kern_reloc_arg, head)); DEFINE(KEXEC_KRELOC_ENTRY_ADDR, offsetof(struct kern_reloc_arg, entry_addr)); DEFINE(KEXEC_KRELOC_KERN_ARG0, offsetof(struct kern_reloc_arg, kern_arg0)); DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1)); diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 9588c91f67c6..07da8d623d8e 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -166,7 +166,6 @@ int machine_kexec_post_load(struct kimage *kimage) memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code) + func_offset; kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg); - kern_reloc_arg->head = kimage->head; kern_reloc_arg->entry_addr = kimage->start; kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem;