From patchwork Thu Sep 30 14:30:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6596C433FE for ; Thu, 30 Sep 2021 14:31:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5AE7161A2C for ; Thu, 30 Sep 2021 14:31:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5AE7161A2C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 4DC179400A1; Thu, 30 Sep 2021 10:31:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4666394003A; Thu, 30 Sep 2021 10:31:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 243709400A1; Thu, 30 Sep 2021 10:31:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0236.hostedemail.com [216.40.44.236]) by kanga.kvack.org (Postfix) with ESMTP id 1379D94003A for ; Thu, 30 Sep 2021 10:31:18 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BEE9130C69 for ; Thu, 30 Sep 2021 14:31:17 +0000 (UTC) X-FDA: 78644477394.08.BC15B8D Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by imf21.hostedemail.com (Postfix) with ESMTP id 72F2CD03459D for ; Thu, 30 Sep 2021 14:31:17 +0000 (UTC) Received: by mail-qt1-f173.google.com with SMTP id e16so5787388qte.13 for ; Thu, 30 Sep 2021 07:31:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ZgbetCXmU2I4AxLznQ7nGzSUyVlFTduK3mMsOCeyR2E=; b=NvUabrPKnpzcu5TeS9l5XS+8SzQvkmleB8d/hakxdXIDBVUCJ9US6MWCXxhdTzzOig rr/9DOSe6KjgBFl4joofHr//byFKjS0nSWBIndIorTah/fBqaOu/+2XRbb7FbH0UuTwv Qs+9FnH6mov9SV8ogh3PfFtbpic6oXECOAXGs49oQpo/gRXLDCIhgwUHOgU2toON5fDa ZxNRn+d1qmt4sgmUbA+DVxfghB99G1F4g68Ox7xXTW6zqsK4O3qyumJD+UElSbXa4FRJ ayCH2de8a5bJEypbyBdzpWcQ30ht6ipCMWAJuJAaNNHEDZVYI6LaWbsuZo+3w/8KAA/g cSew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZgbetCXmU2I4AxLznQ7nGzSUyVlFTduK3mMsOCeyR2E=; b=C1u1LTqQQZhCLeKk3f7gcxMqD+obPV2AiAWv0R2q07Evd9JjJRbUNCCMQOj8X3joym lD4I0i9s2icFW1SIY3udmDsqxYYHIsVAH0O0bturGQpD+wq5ldJs5TYjr5wEn0c91ky4 I868X8XRWKwrLeyh6J+9npHhv0RXjqxx5m3rwkDPEqUJ3wGeimzMLp38lLiB/T1s47VZ 1rTU2mPgW6nhoaz2Z2/Iv8nuuwNLdpKMHRT8BCxLLPER3Eh7Jrqm7bcfNQZe4hE45X6B 2S9xeo46H/sLvjQUiAzhokpPyLY7OORGqjf1oMXCDIfBAjay7IDzELT6QCPsc2wqw+7h LOpw== X-Gm-Message-State: AOAM531uINqOP+chh+J47mfVOyzro7hkgKDLBofTG+5Qf+OVOTSdCcfP kIW7siV+jAGfomPxAQRY280iyw== X-Google-Smtp-Source: ABdhPJxvgW9/FtyJRraTI54gn/Z60LChe52Uu0d4MLes/qY0mj18LYUViK9I0CDtWXyZAWETFAM8yw== X-Received: by 2002:ac8:4c84:: with SMTP id j4mr6839446qtv.411.1633012276760; Thu, 30 Sep 2021 07:31:16 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:16 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 01/15] arm64: kernel: add helper for booted at EL2 and not VHE Date: Thu, 30 Sep 2021 14:30:59 +0000 Message-Id: <20210930143113.1502553-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 72F2CD03459D X-Stat-Signature: fuyko6xz3fehnsyoctih5j58asujqpmq Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=NvUabrPK; dmarc=none; spf=pass (imf21.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-HE-Tag: 1633012277-236761 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace places that contain logic like this: is_hyp_mode_available() && !is_kernel_in_hyp_mode() With a dedicated boolean function is_hyp_nvhe(). This will be needed later in kexec in order to sooner switch back to EL2. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/include/asm/virt.h | 5 +++++ arch/arm64/kernel/cpu-reset.h | 3 +-- arch/arm64/kernel/hibernate.c | 2 +- arch/arm64/kernel/sdei.c | 2 +- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h index 7379f35ae2c6..a9457e96203c 100644 --- a/arch/arm64/include/asm/virt.h +++ b/arch/arm64/include/asm/virt.h @@ -128,6 +128,11 @@ static __always_inline bool is_protected_kvm_enabled(void) return cpus_have_final_cap(ARM64_KVM_PROTECTED_MODE); } +static inline bool is_hyp_nvhe(void) +{ + return is_hyp_mode_available() && !is_kernel_in_hyp_mode(); +} + #endif /* __ASSEMBLY__ */ #endif /* ! __ASM__VIRT_H */ diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h index 9a7b1262ef17..81b3d0fe7a63 100644 --- a/arch/arm64/kernel/cpu-reset.h +++ b/arch/arm64/kernel/cpu-reset.h @@ -20,8 +20,7 @@ static inline void __noreturn __nocfi cpu_soft_restart(unsigned long entry, { typeof(__cpu_soft_restart) *restart; - unsigned long el2_switch = !is_kernel_in_hyp_mode() && - is_hyp_mode_available(); + unsigned long el2_switch = is_hyp_nvhe(); restart = (void *)__pa_symbol(function_nocfi(__cpu_soft_restart)); cpu_install_idmap(); diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 46a0b4d6e251..4c9533f4c0c4 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -49,7 +49,7 @@ extern int in_suspend; /* Do we need to reset el2? */ -#define el2_reset_needed() (is_hyp_mode_available() && !is_kernel_in_hyp_mode()) +#define el2_reset_needed() (is_hyp_nvhe()) /* temporary el2 vectors in the __hibernate_exit_text section. */ extern char hibernate_el2_vectors[]; diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c index 47f77d1234cb..d20620a1c51a 100644 --- a/arch/arm64/kernel/sdei.c +++ b/arch/arm64/kernel/sdei.c @@ -202,7 +202,7 @@ unsigned long sdei_arch_get_entry_point(int conduit) * dropped to EL1 because we don't support VHE, then we can't support * SDEI. */ - if (is_hyp_mode_available() && !is_kernel_in_hyp_mode()) { + if (is_hyp_nvhe()) { pr_err("Not supported on this hardware/boot configuration\n"); goto out_err; } From patchwork Thu Sep 30 14:31:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14EA1C433F5 for ; Thu, 30 Sep 2021 14:31:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AE90461A2A for ; Thu, 30 Sep 2021 14:31:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AE90461A2A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 8117A9400A2; Thu, 30 Sep 2021 10:31:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7756694003A; Thu, 30 Sep 2021 10:31:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57A489400A2; Thu, 30 Sep 2021 10:31:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0252.hostedemail.com [216.40.44.252]) by kanga.kvack.org (Postfix) with ESMTP id 40EB394003A for ; Thu, 30 Sep 2021 10:31:19 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id EFAD2181B3E96 for ; Thu, 30 Sep 2021 14:31:18 +0000 (UTC) X-FDA: 78644477436.27.32AA187 Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) by imf06.hostedemail.com (Postfix) with ESMTP id 9C1F1801ACBD for ; Thu, 30 Sep 2021 14:31:18 +0000 (UTC) Received: by mail-qv1-f54.google.com with SMTP id jo30so3689430qvb.3 for ; Thu, 30 Sep 2021 07:31:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=+4++pnD78fv084ZJJb31cAwxAfjQdVgmdWKFv3ofNWM=; b=PwSkFB5lUIJlNNvHorXkdi5cHBr8TW91BMYYJo/Le3xqGzArqbpyx08n0Ln1FNN8V9 FAFTqIVj/f6d2W2gsY177xVTv/r9AbnqH4GQjLc5PtOUs8+4OQmadzz3A1Xs43xchghI Wh4vRLWrbaVRb9Pb5gtLeFoVmTZXCEB68pVST4gG3jNkJaIE4sHBDcgxA2xRJF9rcc5i CM08jBrUniz8B0qRG8PFfjRS7s22iK1OMmYBeFI8eXLY2MI7XDHeoUOCgT3O2uDFTCHN esQ15Pe1NoByE+7fmDdFl6Uf/mCv2brr7uhwPGQwntr7Nt5pqreknxdkGI2gzzHDVK1Y ipZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+4++pnD78fv084ZJJb31cAwxAfjQdVgmdWKFv3ofNWM=; b=25alpirunkKeLH/IgzoZR5/f1ZBAGP3H5KUg2Ky9G0mEzMIImfG0A1ofJs9SG4FjB3 Rn4zK2WJRwiWZAZEzhyDmygIj9iJulJSg8zre1OVg0WTxFMFLmhYiwe6xC70XPDupQa6 oZF6vkEaIOueAdECWqJSqZkYurxRDXqnXfoCRmXWVvZp85oQZkTNevXtULXPHJamexXO luyEI875Yl8bIdeG4fRrQdW7y2vy5Ky6dYZ30qmNaP2uwpH0B/wteMZxbVN7X8eWZOuk 3s8CkXspsQkmzD5KsipgCfKoi6dKAlXBGsCGE8owcFbOe8rmbXrMFCO8fYz+iRBSqww+ +1yw== X-Gm-Message-State: AOAM532JlDu+KeWyzEtLtV2Va4GG3OM0Ed5E847vpNNp1h20Ab3lvvKb 7+wSGY0PXFR5WhMTCPNP9FUimw== X-Google-Smtp-Source: ABdhPJzkISSDNj6DI6kaZcbl1IAI6lqibUXU1DRHLZaDVciLYY9K0DrqkYiXrO51Bv8hV2XWzusKEg== X-Received: by 2002:a0c:f38b:: with SMTP id i11mr1177471qvk.20.1633012277774; Thu, 30 Sep 2021 07:31:17 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:17 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 02/15] arm64: trans_pgd: hibernate: Add trans_pgd_copy_el2_vectors Date: Thu, 30 Sep 2021 14:31:00 +0000 Message-Id: <20210930143113.1502553-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9C1F1801ACBD X-Stat-Signature: q8cr8d318i9ijtohu4wd4acepk6fwnkt Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=PwSkFB5l; spf=pass (imf06.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.219.54 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none X-Rspamd-Server: rspam06 X-HE-Tag: 1633012278-190648 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Users of trans_pgd may also need a copy of vector table because it is also may be overwritten if a linear map can be overwritten. Move setup of EL2 vectors from hibernate to trans_pgd, so it can be later shared with kexec as well. Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/include/asm/trans_pgd.h | 9 +++-- arch/arm64/include/asm/virt.h | 2 ++ arch/arm64/kernel/hibernate-asm.S | 52 --------------------------- arch/arm64/kernel/hibernate.c | 26 ++++++-------- arch/arm64/mm/Makefile | 1 + arch/arm64/mm/trans_pgd-asm.S | 58 ++++++++++++++++++++++++++++++ arch/arm64/mm/trans_pgd.c | 27 ++++++++++++-- 7 files changed, 103 insertions(+), 72 deletions(-) create mode 100644 arch/arm64/mm/trans_pgd-asm.S diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h index 5d08e5adf3d5..7b04d32b102c 100644 --- a/arch/arm64/include/asm/trans_pgd.h +++ b/arch/arm64/include/asm/trans_pgd.h @@ -1,8 +1,8 @@ /* SPDX-License-Identifier: GPL-2.0 */ /* - * Copyright (c) 2020, Microsoft Corporation. - * Pavel Tatashin + * Copyright (c) 2021, Microsoft Corporation. + * Pasha Tatashin */ #ifndef _ASM_TRANS_TABLE_H @@ -36,4 +36,9 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0, unsigned long *t0sz, void *page); +int trans_pgd_copy_el2_vectors(struct trans_pgd_info *info, + phys_addr_t *el2_vectors); + +extern char trans_pgd_stub_vectors[]; + #endif /* _ASM_TRANS_TABLE_H */ diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h index a9457e96203c..3c8af033a997 100644 --- a/arch/arm64/include/asm/virt.h +++ b/arch/arm64/include/asm/virt.h @@ -67,6 +67,8 @@ */ extern u32 __boot_cpu_mode[2]; +#define ARM64_VECTOR_TABLE_LEN SZ_2K + void __hyp_set_vectors(phys_addr_t phys_vector_base); void __hyp_reset_vectors(void); diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S index 81c0186a5e32..a30a2c3f905e 100644 --- a/arch/arm64/kernel/hibernate-asm.S +++ b/arch/arm64/kernel/hibernate-asm.S @@ -112,56 +112,4 @@ alternative_insn "dc cvau, x4", "dc civac, x4", ARM64_WORKAROUND_CLEAN_CACHE hvc #0 3: ret SYM_CODE_END(swsusp_arch_suspend_exit) - -/* - * Restore the hyp stub. - * This must be done before the hibernate page is unmapped by _cpu_resume(), - * but happens before any of the hyp-stub's code is cleaned to PoC. - * - * x24: The physical address of __hyp_stub_vectors - */ -SYM_CODE_START_LOCAL(el1_sync) - msr vbar_el2, x24 - eret -SYM_CODE_END(el1_sync) - -.macro invalid_vector label -SYM_CODE_START_LOCAL(\label) - b \label -SYM_CODE_END(\label) -.endm - - invalid_vector el2_sync_invalid - invalid_vector el2_irq_invalid - invalid_vector el2_fiq_invalid - invalid_vector el2_error_invalid - invalid_vector el1_sync_invalid - invalid_vector el1_irq_invalid - invalid_vector el1_fiq_invalid - invalid_vector el1_error_invalid - -/* el2 vectors - switch el2 here while we restore the memory image. */ - .align 11 -SYM_CODE_START(hibernate_el2_vectors) - ventry el2_sync_invalid // Synchronous EL2t - ventry el2_irq_invalid // IRQ EL2t - ventry el2_fiq_invalid // FIQ EL2t - ventry el2_error_invalid // Error EL2t - - ventry el2_sync_invalid // Synchronous EL2h - ventry el2_irq_invalid // IRQ EL2h - ventry el2_fiq_invalid // FIQ EL2h - ventry el2_error_invalid // Error EL2h - - ventry el1_sync // Synchronous 64-bit EL1 - ventry el1_irq_invalid // IRQ 64-bit EL1 - ventry el1_fiq_invalid // FIQ 64-bit EL1 - ventry el1_error_invalid // Error 64-bit EL1 - - ventry el1_sync_invalid // Synchronous 32-bit EL1 - ventry el1_irq_invalid // IRQ 32-bit EL1 - ventry el1_fiq_invalid // FIQ 32-bit EL1 - ventry el1_error_invalid // Error 32-bit EL1 -SYM_CODE_END(hibernate_el2_vectors) - .popsection diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 4c9533f4c0c4..b96ef9060e4c 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -51,9 +51,6 @@ extern int in_suspend; /* Do we need to reset el2? */ #define el2_reset_needed() (is_hyp_nvhe()) -/* temporary el2 vectors in the __hibernate_exit_text section. */ -extern char hibernate_el2_vectors[]; - /* hyp-stub vectors, used to restore el2 during resume from hibernate. */ extern char __hyp_stub_vectors[]; @@ -434,6 +431,7 @@ int swsusp_arch_resume(void) void *zero_page; size_t exit_size; pgd_t *tmp_pg_dir; + phys_addr_t el2_vectors; void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *, void *, phys_addr_t, phys_addr_t); struct trans_pgd_info trans_info = { @@ -461,6 +459,14 @@ int swsusp_arch_resume(void) return -ENOMEM; } + if (el2_reset_needed()) { + rc = trans_pgd_copy_el2_vectors(&trans_info, &el2_vectors); + if (rc) { + pr_err("Failed to setup el2 vectors\n"); + return rc; + } + } + exit_size = __hibernate_exit_text_end - __hibernate_exit_text_start; /* * Copy swsusp_arch_suspend_exit() to a safe page. This will generate @@ -473,26 +479,14 @@ int swsusp_arch_resume(void) return rc; } - /* - * The hibernate exit text contains a set of el2 vectors, that will - * be executed at el2 with the mmu off in order to reload hyp-stub. - */ - dcache_clean_inval_poc((unsigned long)hibernate_exit, - (unsigned long)hibernate_exit + exit_size); - /* * KASLR will cause the el2 vectors to be in a different location in * the resumed kernel. Load hibernate's temporary copy into el2. * * We can skip this step if we booted at EL1, or are running with VHE. */ - if (el2_reset_needed()) { - phys_addr_t el2_vectors = (phys_addr_t)hibernate_exit; - el2_vectors += hibernate_el2_vectors - - __hibernate_exit_text_start; /* offset */ - + if (el2_reset_needed()) __hyp_set_vectors(el2_vectors); - } hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1, resume_hdr.reenter_kernel, restore_pblist, diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index f188c9092696..ff1e800ba7a1 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -7,6 +7,7 @@ obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o +obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o KASAN_SANITIZE_physaddr.o += n diff --git a/arch/arm64/mm/trans_pgd-asm.S b/arch/arm64/mm/trans_pgd-asm.S new file mode 100644 index 000000000000..8c4bffe3089d --- /dev/null +++ b/arch/arm64/mm/trans_pgd-asm.S @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +/* + * Copyright (c) 2021, Microsoft Corporation. + * Pasha Tatashin + */ + +#include +#include +#include + +.macro invalid_vector label +SYM_CODE_START_LOCAL(\label) + .align 7 + b \label +SYM_CODE_END(\label) +.endm + +.macro el1_sync_vector +SYM_CODE_START_LOCAL(el1_sync) + .align 7 + cmp x0, #HVC_SET_VECTORS /* Called from hibernate */ + b.ne 1f + msr vbar_el2, x1 + mov x0, xzr + eret +1: /* Unexpected argument, set an error */ + mov_q x0, HVC_STUB_ERR + eret +SYM_CODE_END(el1_sync) +.endm + +SYM_CODE_START(trans_pgd_stub_vectors) + invalid_vector hyp_stub_el2t_sync_invalid // Synchronous EL2t + invalid_vector hyp_stub_el2t_irq_invalid // IRQ EL2t + invalid_vector hyp_stub_el2t_fiq_invalid // FIQ EL2t + invalid_vector hyp_stub_el2t_error_invalid // Error EL2t + + invalid_vector hyp_stub_el2h_sync_invalid // Synchronous EL2h + invalid_vector hyp_stub_el2h_irq_invalid // IRQ EL2h + invalid_vector hyp_stub_el2h_fiq_invalid // FIQ EL2h + invalid_vector hyp_stub_el2h_error_invalid // Error EL2h + + el1_sync_vector // Synchronous 64-bit EL1 + invalid_vector hyp_stub_el1_irq_invalid // IRQ 64-bit EL1 + invalid_vector hyp_stub_el1_fiq_invalid // FIQ 64-bit EL1 + invalid_vector hyp_stub_el1_error_invalid // Error 64-bit EL1 + + invalid_vector hyp_stub_32b_el1_sync_invalid // Synchronous 32-bit EL1 + invalid_vector hyp_stub_32b_el1_irq_invalid // IRQ 32-bit EL1 + invalid_vector hyp_stub_32b_el1_fiq_invalid // FIQ 32-bit EL1 + invalid_vector hyp_stub_32b_el1_error_invalid // Error 32-bit EL1 + .align 11 +SYM_INNER_LABEL(__trans_pgd_stub_vectors_end, SYM_L_LOCAL) +SYM_CODE_END(trans_pgd_stub_vectors) + +# Check the trans_pgd_stub_vectors didn't overflow +.org . - (__trans_pgd_stub_vectors_end - trans_pgd_stub_vectors) + SZ_2K diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 527f0a39c3da..26bd8f2d95af 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -5,8 +5,8 @@ * * This file derived from: arch/arm64/kernel/hibernate.c * - * Copyright (c) 2020, Microsoft Corporation. - * Pavel Tatashin + * Copyright (c) 2021, Microsoft Corporation. + * Pasha Tatashin * */ @@ -322,3 +322,26 @@ int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0, return 0; } + +/* + * Create a copy of the vector table so we can call HVC_SET_VECTORS or + * HVC_SOFT_RESTART from contexts where the table may be overwritten. + */ +int trans_pgd_copy_el2_vectors(struct trans_pgd_info *info, + phys_addr_t *el2_vectors) +{ + void *hyp_stub = trans_alloc(info); + + if (!hyp_stub) + return -ENOMEM; + *el2_vectors = virt_to_phys(hyp_stub); + memcpy(hyp_stub, &trans_pgd_stub_vectors, ARM64_VECTOR_TABLE_LEN); + caches_clean_inval_pou((unsigned long)hyp_stub, + (unsigned long)hyp_stub + + ARM64_VECTOR_TABLE_LEN); + dcache_clean_inval_poc((unsigned long)hyp_stub, + (unsigned long)hyp_stub + + ARM64_VECTOR_TABLE_LEN); + + return 0; +} From patchwork Thu Sep 30 14:31:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 711E8C433F5 for ; Thu, 30 Sep 2021 14:31:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 26E9D61A2A for ; Thu, 30 Sep 2021 14:31:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 26E9D61A2A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id C07A09400A3; Thu, 30 Sep 2021 10:31:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B6ABA94003A; Thu, 30 Sep 2021 10:31:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9227D9400A3; Thu, 30 Sep 2021 10:31:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0126.hostedemail.com [216.40.44.126]) by kanga.kvack.org (Postfix) with ESMTP id 7C42694003A for ; Thu, 30 Sep 2021 10:31:20 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 378B51831AFE4 for ; Thu, 30 Sep 2021 14:31:20 +0000 (UTC) X-FDA: 78644477520.17.8130C61 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) by imf29.hostedemail.com (Postfix) with ESMTP id A9635900067C for ; Thu, 30 Sep 2021 14:31:19 +0000 (UTC) Received: by mail-qv1-f45.google.com with SMTP id x9so3661368qvn.12 for ; Thu, 30 Sep 2021 07:31:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=e39R8ZsIeIwklE8cpr9UY0qKnjifmGWiOSmlNRknozY=; b=Dvw31wgsS8G8oxRKiLY9AhC7vsIrGYUqnkg0rdzHudMErK/hqx/f2xKLuM+d+wFz94 JUmD5np8vuwG2sGXNGsbm1qYTnL5SwmKk3BvG6nraqjwNtTgysYN2X6REwY/fvMqNdAz WLoSnNhuaCUYjqopKhwffkXgoe7lX17Re98nrYZngFujO2+9S5P4HTM+0zeBJdgd5nLG 2YrkHbtNBlzYOOJByqwcPTwT9QKuRm31Mkw6DL6XZxRQc6JsHrc/vF3FUunfcRX4Tuxg zJNVbfpaez0v6k+uebd2NVVfyA93/+wABN7kKOsZMg78c45ftAn4QhH3QWl22IAt60Vj 3CeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=e39R8ZsIeIwklE8cpr9UY0qKnjifmGWiOSmlNRknozY=; b=vhRYB8VoIuV9bDFjsR5Jd06Oxnjp+0R9Wb+92EdPojKANc0q8vUZnDvfehRCLf0AY5 Xn3FcMIMeVGWkCVzZi4k9r/qiXai2m4Z+3hAEsbUbiujpmjca/1ryzGLj8PA8VkqHNk7 +EWPMaFkwdNUuVGOWB4sgp7DLY37XiT5i2QHFfrbbSy++03RC3dAoNDevJMnAMGQyyQ5 4caMcS/9KDANiURcv4A35PBE9jPkaqo77Vw5c3DEUFq7/hX6Oc5BEkdz21Y4FVenAZ2w 5xB27QSlZc8wwoqFVQssF8pk+WayNYv0qm1flUeOVyO05F4eXALvcsaoT2V1rxWmQYJF CNcQ== X-Gm-Message-State: AOAM532ZNLqfCfjINlk7/L1pp1Ph+tu+u8kEEQ56p3gvmzR3HmQG5OWY wuBKAwcxkXnqcN1wwZEYnJ2rfA== X-Google-Smtp-Source: ABdhPJyKP/5js7mH/GPTqK5GuDttLVlJNJvbT8t4LfCfXnHZzGw54YFenib9JG+PlW6v9j7cikxMqQ== X-Received: by 2002:a05:6214:2d1:: with SMTP id g17mr5468778qvu.63.1633012278962; Thu, 30 Sep 2021 07:31:18 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:18 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 03/15] arm64: hibernate: abstract ttrb0 setup function Date: Thu, 30 Sep 2021 14:31:01 +0000 Message-Id: <20210930143113.1502553-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A9635900067C X-Stat-Signature: iuremwgnfm6jbox31a43uaar8bz7985a Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=Dvw31wgs; spf=pass (imf29.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.219.45 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none X-Rspamd-Server: rspam06 X-HE-Tag: 1633012279-452128 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, only hibernate sets custom ttbr0 with safe idmaped function. Kexec, is also going to be using this functionality when relocation code is going to be idmapped. Move the setup sequence to a dedicated cpu_install_ttbr0() for custom ttbr0. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/include/asm/mmu_context.h | 24 ++++++++++++++++++++++++ arch/arm64/kernel/hibernate.c | 21 +-------------------- 2 files changed, 25 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index f4ba93d4ffeb..6770667b34a3 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -115,6 +115,30 @@ static inline void cpu_install_idmap(void) cpu_switch_mm(lm_alias(idmap_pg_dir), &init_mm); } +/* + * Load our new page tables. A strict BBM approach requires that we ensure that + * TLBs are free of any entries that may overlap with the global mappings we are + * about to install. + * + * For a real hibernate/resume/kexec cycle TTBR0 currently points to a zero + * page, but TLBs may contain stale ASID-tagged entries (e.g. for EFI runtime + * services), while for a userspace-driven test_resume cycle it points to + * userspace page tables (and we must point it at a zero page ourselves). + * + * We change T0SZ as part of installing the idmap. This is undone by + * cpu_uninstall_idmap() in __cpu_suspend_exit(). + */ +static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) +{ + cpu_set_reserved_ttbr0(); + local_flush_tlb_all(); + __cpu_set_tcr_t0sz(t0sz); + + /* avoid cpu_switch_mm() and its SW-PAN and CNP interactions */ + write_sysreg(ttbr0, ttbr0_el1); + isb(); +} + /* * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index b96ef9060e4c..2758f75d6809 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -212,26 +212,7 @@ static int create_safe_exec_page(void *src_start, size_t length, if (rc) return rc; - /* - * Load our new page tables. A strict BBM approach requires that we - * ensure that TLBs are free of any entries that may overlap with the - * global mappings we are about to install. - * - * For a real hibernate/resume cycle TTBR0 currently points to a zero - * page, but TLBs may contain stale ASID-tagged entries (e.g. for EFI - * runtime services), while for a userspace-driven test_resume cycle it - * points to userspace page tables (and we must point it at a zero page - * ourselves). - * - * We change T0SZ as part of installing the idmap. This is undone by - * cpu_uninstall_idmap() in __cpu_suspend_exit(). - */ - cpu_set_reserved_ttbr0(); - local_flush_tlb_all(); - __cpu_set_tcr_t0sz(t0sz); - write_sysreg(trans_ttbr0, ttbr0_el1); - isb(); - + cpu_install_ttbr0(trans_ttbr0, t0sz); *phys_dst_addr = virt_to_phys(page); return 0; From patchwork Thu Sep 30 14:31:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBC9AC433F5 for ; Thu, 30 Sep 2021 14:31:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8122461A0C for ; Thu, 30 Sep 2021 14:31:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8122461A0C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 6FF699400A4; Thu, 30 Sep 2021 10:31:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 687DE94003A; Thu, 30 Sep 2021 10:31:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DDED9400A4; Thu, 30 Sep 2021 10:31:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0105.hostedemail.com [216.40.44.105]) by kanga.kvack.org (Postfix) with ESMTP id 3BC3294003A for ; Thu, 30 Sep 2021 10:31:21 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E194718029DAC for ; Thu, 30 Sep 2021 14:31:20 +0000 (UTC) X-FDA: 78644477520.01.4EDC697 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by imf11.hostedemail.com (Postfix) with ESMTP id 906BDF000211 for ; Thu, 30 Sep 2021 14:31:20 +0000 (UTC) Received: by mail-qv1-f53.google.com with SMTP id 11so3665565qvd.11 for ; Thu, 30 Sep 2021 07:31:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=wvhp7Shwzj2V1AcHSUNo3T+jO//upg214s/8JF7+8X4=; b=WzTIvObi8BmW8NTQjOqb0FOv0H/1NPIY+oIISBMpm8fDw4WooAALyuQ8nS0FbKt/SD oG484TyexjpEF9PLSUf6FoVGHUmJa7wCuNbtCA+2gWYOBP7IMxuKsxo2SB2qce1734pU xsEoWg7TnfyCy6fwEslgvv5G4WPjSmJb/ebM1dBbO3PAOPhAMBgXqHNADEKlZZw80hfB UHXFu0nSxEI896qgpyWWYjO5i4k8yAw4Zdf5kcNiCy9l3kfnM4g1YbizWJfXBfT0X1pO i58tA/zr32BfcRs2J3R6EKHkuX/khIa1jSiNc1733FUETlk2PlBrtbTQuqXUQUOrPrxJ 0vIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wvhp7Shwzj2V1AcHSUNo3T+jO//upg214s/8JF7+8X4=; b=ApMUGWMZv5ufbwathDE1+f/m4ORT/jW9BtgL9RzJMoNzmEizd6utIUIWV5CjEaBiGT MaFTByWIwpObdWEO5S+6Jk4iFp8zdbJq52xxheu2A60JQ8Sdf8meeKklc+oZPh+4oFGc 6h2NzbfMLLEk7ms1BhgJYDkGn+o6V1CxEdgtNM9olXiWowVEbREcVD4w39k9nKDOcSE6 NGkLBKjxzL+4ekuxRN6EDPZP/LfRx203D7DdmhMgxW/kuLs8MAeVBobpDmlW+/KYKlI6 ih4SM1XkC0LAvlf9u8JsCipi8nhOSO8hZu1579oG1yLsTWe8QE/R4FdYFZc7yE6Mv3s5 /5Vg== X-Gm-Message-State: AOAM532rQHaV5Y7XFEuZWEkx/fQyQfl9+np1eLKxPyD694MSrOQiR1db 5WH9P1WxdsXw3BjC/C202TqvRg== X-Google-Smtp-Source: ABdhPJyRpe+0jXAcRshmB05WHMM4GvLP/u4h4SOYX1etnyUhK8wl9VzQ7qbqdCBBQWvQ1rXkOCyYsA== X-Received: by 2002:a05:6214:90:: with SMTP id n16mr4150225qvr.1.1633012279941; Thu, 30 Sep 2021 07:31:19 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:19 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 04/15] arm64: kexec: flush image and lists during kexec load time Date: Thu, 30 Sep 2021 14:31:02 +0000 Message-Id: <20210930143113.1502553-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 906BDF000211 X-Stat-Signature: bds5d4qxkyju94hb6rg4su6hfi6udgjf Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=WzTIvObi; dmarc=none; spf=pass (imf11.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.219.53 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-HE-Tag: 1633012280-31717 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, during kexec load we are copying relocation function and flushing it. However, we can also flush kexec relocation buffers and if new kernel image is already in place (i.e. crash kernel), we can also flush the new kernel image itself. Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/kernel/machine_kexec.c | 58 ++++++++++++++----------------- 1 file changed, 26 insertions(+), 32 deletions(-) diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 213d56c14f60..b6d5a02cba2e 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -60,29 +60,6 @@ void machine_kexec_cleanup(struct kimage *kimage) /* Empty routine needed to avoid build errors. */ } -int machine_kexec_post_load(struct kimage *kimage) -{ - void *reloc_code = page_to_virt(kimage->control_code_page); - - memcpy(reloc_code, arm64_relocate_new_kernel, - arm64_relocate_new_kernel_size); - kimage->arch.kern_reloc = __pa(reloc_code); - kexec_image_info(kimage); - - /* - * For execution with the MMU off, reloc_code needs to be cleaned to the - * PoC and invalidated from the I-cache. - */ - dcache_clean_inval_poc((unsigned long)reloc_code, - (unsigned long)reloc_code + - arm64_relocate_new_kernel_size); - icache_inval_pou((uintptr_t)reloc_code, - (uintptr_t)reloc_code + - arm64_relocate_new_kernel_size); - - return 0; -} - /** * machine_kexec_prepare - Prepare for a kexec reboot. * @@ -163,6 +140,32 @@ static void kexec_segment_flush(const struct kimage *kimage) } } +int machine_kexec_post_load(struct kimage *kimage) +{ + void *reloc_code = page_to_virt(kimage->control_code_page); + + /* If in place flush new kernel image, else flush lists and buffers */ + if (kimage->head & IND_DONE) + kexec_segment_flush(kimage); + else + kexec_list_flush(kimage); + + memcpy(reloc_code, arm64_relocate_new_kernel, + arm64_relocate_new_kernel_size); + kimage->arch.kern_reloc = __pa(reloc_code); + kexec_image_info(kimage); + + /* Flush the reloc_code in preparation for its execution. */ + dcache_clean_inval_poc((unsigned long)reloc_code, + (unsigned long)reloc_code + + arm64_relocate_new_kernel_size); + icache_inval_pou((uintptr_t)reloc_code, + (uintptr_t)reloc_code + + arm64_relocate_new_kernel_size); + + return 0; +} + /** * machine_kexec - Do the kexec reboot. * @@ -180,13 +183,6 @@ void machine_kexec(struct kimage *kimage) WARN(in_kexec_crash && (stuck_cpus || smp_crash_stop_failed()), "Some CPUs may be stale, kdump will be unreliable.\n"); - /* Flush the kimage list and its buffers. */ - kexec_list_flush(kimage); - - /* Flush the new image if already in place. */ - if ((kimage != kexec_crash_image) && (kimage->head & IND_DONE)) - kexec_segment_flush(kimage); - pr_info("Bye!\n"); local_daif_mask(); @@ -261,8 +257,6 @@ void arch_kexec_protect_crashkres(void) { int i; - kexec_segment_flush(kexec_crash_image); - for (i = 0; i < kexec_crash_image->nr_segments; i++) set_memory_valid( __phys_to_virt(kexec_crash_image->segment[i].mem), From patchwork Thu Sep 30 14:31:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A9FCC433EF for ; Thu, 30 Sep 2021 14:31:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D17FE61A0D for ; Thu, 30 Sep 2021 14:31:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D17FE61A0D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id A34F79400A5; Thu, 30 Sep 2021 10:31:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9BF7194003A; Thu, 30 Sep 2021 10:31:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8104D9400A5; Thu, 30 Sep 2021 10:31:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0250.hostedemail.com [216.40.44.250]) by kanga.kvack.org (Postfix) with ESMTP id 6D32894003A for ; Thu, 30 Sep 2021 10:31:22 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 180128249980 for ; Thu, 30 Sep 2021 14:31:22 +0000 (UTC) X-FDA: 78644477604.12.9D5EA5A Received: from mail-qv1-f47.google.com (mail-qv1-f47.google.com [209.85.219.47]) by imf04.hostedemail.com (Postfix) with ESMTP id B96245001046 for ; Thu, 30 Sep 2021 14:31:21 +0000 (UTC) Received: by mail-qv1-f47.google.com with SMTP id dk4so3693790qvb.2 for ; Thu, 30 Sep 2021 07:31:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=2byFwZTqL6H5T1HwZ3UMFR9jHqf1akMl4mBN/vDMX94=; b=CnuAcVa/z2KIcemUvmuCQG3R70S7y0YjOx0S55gdxctiytScgPrJDP6M1fLmRGPYXJ YAT3mIXZVlLiki2mw9xhu8NnySDq7DVByfEQjdj7oIecSzThsjjHYPVJQ4VGIOBh5cqz VlYyGJ6x1aJFFhC+csTa7NfnPJGU6R7oBZD8MmNrJ4z6rW5X+/OgtPXmPF9tCdlDw8iv 5gBfKad8IQFE/KHebxTVW7m95rJ16FTAVG6bAoPT9dRSVK92yIipf0bAMwhKETG99lhJ JBbcprcZPUyjLRS4jyYAX+2cwdviTgJKdrRmaXT6Z1jEDDgCDbBiPpKZPoxA4uQvLe6A kMFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2byFwZTqL6H5T1HwZ3UMFR9jHqf1akMl4mBN/vDMX94=; b=uic+dROYQG4ZB8mmdnKPOPAGD0/+hJV+Xjnc1u72Ux4141wBhA4sXZyJUU6hKvgI+W miFKQfbTpzmK0GVHlClwg6eUk7UbE9gb04RBvhgSL/iKI3al4j3ursiZiQFI2I9u3vBr dArU4EFKlivdCbX5qFjclgLFg/IGo/KfKnCOEg8uzsQgW2wcKvkEv/fYlyEfCUaOlVyn 91mkEWQZTQCTGmaFirdEDHZj4k6/vNpYa7vgU79zb8sANrOqf8HC6LByM7pYxdWWO57T Yh8cMJv0y3rk6gfFEP/25vGTuAoVXS9iPX9f4G6FXJaxbsxXR86KLlgt+pLyjlHx3eVL aMIg== X-Gm-Message-State: AOAM531Khyi2vueXjh/RbGbf7rHZoNuJopWqGMXqGTKpZyYpd43R3mIX j4akNEhCATtXdHDgMNMDyCLvsA== X-Google-Smtp-Source: ABdhPJx2GA26RZN03kRYmBDCi/gcboAxmrhaWBYq1chXopjjFm1J8tvOWaWK45xUGQBMAOa3uCnDFA== X-Received: by 2002:ad4:45f0:: with SMTP id q16mr331461qvu.4.1633012281074; Thu, 30 Sep 2021 07:31:21 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:20 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 05/15] arm64: kexec: skip relocation code for inplace kexec Date: Thu, 30 Sep 2021 14:31:03 +0000 Message-Id: <20210930143113.1502553-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B96245001046 X-Stat-Signature: 3mckgrj7ps1jy9tsysqiuuo1mrakiyre Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b="CnuAcVa/"; spf=pass (imf04.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.219.47 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none X-Rspamd-Server: rspam06 X-HE-Tag: 1633012281-290265 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In case of kdump or when segments are already in place the relocation is not needed, therefore the setup of relocation function and call to it can be skipped. Signed-off-by: Pasha Tatashin Suggested-by: James Morse Acked-by: Catalin Marinas --- arch/arm64/kernel/machine_kexec.c | 34 ++++++++++++++++++----------- arch/arm64/kernel/relocate_kernel.S | 3 --- 2 files changed, 21 insertions(+), 16 deletions(-) diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index b6d5a02cba2e..7f1cb5a2a463 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -144,16 +144,16 @@ int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); - /* If in place flush new kernel image, else flush lists and buffers */ - if (kimage->head & IND_DONE) + /* If in place, relocation is not used, only flush next kernel */ + if (kimage->head & IND_DONE) { kexec_segment_flush(kimage); - else - kexec_list_flush(kimage); + kexec_image_info(kimage); + return 0; + } memcpy(reloc_code, arm64_relocate_new_kernel, arm64_relocate_new_kernel_size); kimage->arch.kern_reloc = __pa(reloc_code); - kexec_image_info(kimage); /* Flush the reloc_code in preparation for its execution. */ dcache_clean_inval_poc((unsigned long)reloc_code, @@ -162,6 +162,8 @@ int machine_kexec_post_load(struct kimage *kimage) icache_inval_pou((uintptr_t)reloc_code, (uintptr_t)reloc_code + arm64_relocate_new_kernel_size); + kexec_list_flush(kimage); + kexec_image_info(kimage); return 0; } @@ -188,19 +190,25 @@ void machine_kexec(struct kimage *kimage) local_daif_mask(); /* - * cpu_soft_restart will shutdown the MMU, disable data caches, then - * transfer control to the kern_reloc which contains a copy of - * the arm64_relocate_new_kernel routine. arm64_relocate_new_kernel - * uses physical addressing to relocate the new image to its final - * position and transfers control to the image entry point when the - * relocation is complete. + * Both restart and cpu_soft_restart will shutdown the MMU, disable data + * caches. However, restart will start new kernel or purgatory directly, + * cpu_soft_restart will transfer control to arm64_relocate_new_kernel * In kexec case, kimage->start points to purgatory assuming that * kernel entry and dtb address are embedded in purgatory by * userspace (kexec-tools). * In kexec_file case, the kernel starts directly without purgatory. */ - cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start, - kimage->arch.dtb_mem); + if (kimage->head & IND_DONE) { + typeof(__cpu_soft_restart) *restart; + + cpu_install_idmap(); + restart = (void *)__pa_symbol(function_nocfi(__cpu_soft_restart)); + restart(is_hyp_nvhe(), kimage->start, kimage->arch.dtb_mem, + 0, 0); + } else { + cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, + kimage->start, kimage->arch.dtb_mem); + } BUG(); /* Should never get here. */ } diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index b78ea5de97a4..8058fabe0a76 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -32,8 +32,6 @@ SYM_CODE_START(arm64_relocate_new_kernel) mov x16, x0 /* x16 = kimage_head */ mov x14, xzr /* x14 = entry ptr */ mov x13, xzr /* x13 = copy dest */ - /* Check if the new image needs relocation. */ - tbnz x16, IND_DONE_BIT, .Ldone raw_dcache_line_size x15, x1 /* x15 = dcache line size */ .Lloop: and x12, x16, PAGE_MASK /* x12 = addr */ @@ -65,7 +63,6 @@ SYM_CODE_START(arm64_relocate_new_kernel) .Lnext: ldr x16, [x14], #8 /* entry = *ptr++ */ tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */ -.Ldone: /* wait for writes from copy_page to finish */ dsb nsh ic iallu From patchwork Thu Sep 30 14:31:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8B40C433F5 for ; Thu, 30 Sep 2021 14:31:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 66DF761A2F for ; Thu, 30 Sep 2021 14:31:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 66DF761A2F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 79D229400A6; Thu, 30 Sep 2021 10:31:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DDB194003A; Thu, 30 Sep 2021 10:31:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DD519400A6; Thu, 30 Sep 2021 10:31:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0059.hostedemail.com [216.40.44.59]) by kanga.kvack.org (Postfix) with ESMTP id 39C8494003A for ; Thu, 30 Sep 2021 10:31:23 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E43341830D199 for ; Thu, 30 Sep 2021 14:31:22 +0000 (UTC) X-FDA: 78644477604.31.2386443 Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) by imf01.hostedemail.com (Postfix) with ESMTP id 94DFC506B165 for ; Thu, 30 Sep 2021 14:31:22 +0000 (UTC) Received: by mail-qk1-f175.google.com with SMTP id f130so5961005qke.6 for ; Thu, 30 Sep 2021 07:31:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=UIkM47FXs5P7Z/Da3Mn+8DCMex1zjJveQjNKvVhGh4k=; b=HAsc0xpvY/J7ECsq2WRyYU/gr+1qYLJdXXgcW7LBuP6Ck7/Ze+a0P8y/kFcZ9s4sBG CwKWrt/wRlgYhSyE8RDbsVvBXDwWX3j3zpgu3zaWiQbEL57hwBFqnzhPBbq/ZXYEa9X6 OTV8mpwqGNCSOvXYIvWJjjiVqAb255gIiz0u656noA+ql4bIpfbuLR/FvM+oxIDWJYxC Ovp2KOoR3zDAdTJ8kp0W1Doa1VICEloK8cfOUhjbl9VsuR0lbeMoG8zNTtpiVfDxuEAD XQQHdoPSdene+TR0RbeUo3xCYIjqnjPoTuNAvD4reELA4S2ULq+Z1lbpIp9kkqH0jtVV hnXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UIkM47FXs5P7Z/Da3Mn+8DCMex1zjJveQjNKvVhGh4k=; b=4gEk6HbY5EnAtdIgFu25x2qPiR84y4hjzUyMWd7rmfafK0C60NSCt7Khf8IdgrGG0l BKybVf9WZ/GRDv5cUApkWudgNAUzNhXmYZys/F45FObC0bnFemOmomHeY10MgqT9ALcL Ixgq7U50el0SG4I6E5rx7QV7sBILgrLedv0Ys4DEXosfJn20h77ym2JsiZhyDI1bex/i 6txFUsZb8YWK3kcLxZTQyUY/jun83QW57jEZuvyXtqXOE7Hvbwwq/FBChITqmXFsTO9B AzzrBgUWXnevQbnSjcQTBV5Cbfg+o+uuVYu1RNSroHXWOdavCXggKizssaBGmKWOSTpQ QNGg== X-Gm-Message-State: AOAM532ItlkgurmqmNDIU6sfM1Qb84DqR9+fKpcqvoU25Wi3pQCz1m+G 6KOkolWHGvORzjydxwpjoobHpQ== X-Google-Smtp-Source: ABdhPJz+yokIGfxLu3ulCxnz5lUdNKG81GcyHkZE4+G8gOjEmo4IXXWq+pXgLlUabAlOLSxioSB5Qg== X-Received: by 2002:a37:993:: with SMTP id 141mr4982102qkj.488.1633012281915; Thu, 30 Sep 2021 07:31:21 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:21 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 06/15] arm64: kexec: Use dcache ops macros instead of open-coding Date: Thu, 30 Sep 2021 14:31:04 +0000 Message-Id: <20210930143113.1502553-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 94DFC506B165 X-Stat-Signature: ucn9w5ee7ayxt6rnjduqo8qoiqcnakd7 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=HAsc0xpv; spf=pass (imf01.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.175 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none X-Rspamd-Server: rspam06 X-HE-Tag: 1633012282-449261 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: kexec does dcache maintenance when it re-writes all memory. Our dcache_by_line_op macro depends on reading the sanitized DminLine from memory. Kexec may have overwritten this, so open-codes the sequence. dcache_by_line_op is a whole set of macros, it uses dcache_line_size which uses read_ctr for the sanitsed DminLine. Reading the DminLine is the first thing the dcache_by_line_op does. Rename dcache_by_line_op dcache_by_myline_op and take DminLine as an argument. Kexec can now use the slightly smaller macro. This makes up-coming changes to the dcache maintenance easier on the eye. Code generated by the existing callers is unchanged. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/include/asm/assembler.h | 30 ++++++++++++++++++++++------- arch/arm64/kernel/relocate_kernel.S | 13 +++---------- 2 files changed, 26 insertions(+), 17 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index bfa58409a4d4..d5281f75a58d 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -405,19 +405,19 @@ alternative_endif /* * Macro to perform a data cache maintenance for the interval - * [start, end) + * [start, end) with dcache line size explicitly provided. * * op: operation passed to dc instruction * domain: domain used in dsb instruciton * start: starting virtual address of the region * end: end virtual address of the region + * linesz: dcache line size * fixup: optional label to branch to on user fault - * Corrupts: start, end, tmp1, tmp2 + * Corrupts: start, end, tmp */ - .macro dcache_by_line_op op, domain, start, end, tmp1, tmp2, fixup - dcache_line_size \tmp1, \tmp2 - sub \tmp2, \tmp1, #1 - bic \start, \start, \tmp2 + .macro dcache_by_myline_op op, domain, start, end, linesz, tmp, fixup + sub \tmp, \linesz, #1 + bic \start, \start, \tmp .Ldcache_op\@: .ifc \op, cvau __dcache_op_workaround_clean_cache \op, \start @@ -436,7 +436,7 @@ alternative_endif .endif .endif .endif - add \start, \start, \tmp1 + add \start, \start, \linesz cmp \start, \end b.lo .Ldcache_op\@ dsb \domain @@ -444,6 +444,22 @@ alternative_endif _cond_extable .Ldcache_op\@, \fixup .endm +/* + * Macro to perform a data cache maintenance for the interval + * [start, end) + * + * op: operation passed to dc instruction + * domain: domain used in dsb instruciton + * start: starting virtual address of the region + * end: end virtual address of the region + * fixup: optional label to branch to on user fault + * Corrupts: start, end, tmp1, tmp2 + */ + .macro dcache_by_line_op op, domain, start, end, tmp1, tmp2, fixup + dcache_line_size \tmp1, \tmp2 + dcache_by_myline_op \op, \domain, \start, \end, \tmp1, \tmp2, \fixup + .endm + /* * Macro to perform an instruction cache maintenance for the interval * [start, end) diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 8058fabe0a76..8c43779e8cc6 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -41,16 +41,9 @@ SYM_CODE_START(arm64_relocate_new_kernel) tbz x16, IND_SOURCE_BIT, .Ltest_indirection /* Invalidate dest page to PoC. */ - mov x2, x13 - add x20, x2, #PAGE_SIZE - sub x1, x15, #1 - bic x2, x2, x1 -2: dc ivac, x2 - add x2, x2, x15 - cmp x2, x20 - b.lo 2b - dsb sy - + mov x2, x13 + add x1, x2, #PAGE_SIZE + dcache_by_myline_op ivac, sy, x2, x1, x15, x20 copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 b .Lnext .Ltest_indirection: From patchwork Thu Sep 30 14:31:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CF87C433EF for ; Thu, 30 Sep 2021 14:31:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E870C61A10 for ; Thu, 30 Sep 2021 14:31:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E870C61A10 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 4DDCD9400A7; Thu, 30 Sep 2021 10:31:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49FDA94003A; Thu, 30 Sep 2021 10:31:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 307709400A7; Thu, 30 Sep 2021 10:31:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0224.hostedemail.com [216.40.44.224]) by kanga.kvack.org (Postfix) with ESMTP id 1B21394003A for ; Thu, 30 Sep 2021 10:31:24 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C3EEE2C6B5 for ; Thu, 30 Sep 2021 14:31:23 +0000 (UTC) X-FDA: 78644477646.38.E964628 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) by imf21.hostedemail.com (Postfix) with ESMTP id 7D46CD0345A6 for ; Thu, 30 Sep 2021 14:31:23 +0000 (UTC) Received: by mail-qt1-f174.google.com with SMTP id a13so5814490qtw.10 for ; Thu, 30 Sep 2021 07:31:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=5tAiOQNbsPwYQy1X9AyljFW/jKh8PdcAlWcG2o1mwWI=; b=XCzx76XQyqYzRm1ibea0WoCNtcgtXq1gejadijr7/Tsm8GnPNst3oyZj3R9I9AnbJG lrBnaIGN9DLqClY6Iu3a0cPk49EhlCdK/rDuxQQYANtGNfP8HFek80IgFaJdn1NGVLDJ 27ChNmLpSzUczm8ERB32kFy54GWlk57lRkLWF4zD1P65qWvy/Iad8ko3u57FmlRhAxP9 hho1onk8yY5gm6Iw+NSNJ1q5UqXxrZlGyOQYSWjs498DAm/VFk2pIkmf5qWtWOKGMdgF ygnW+Ww/+cSezwh8z/XfxMhB2qr8HyapKNi3SW/qFQ+y1Vy8tWorf86vcsOyv9ttqucp +12Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5tAiOQNbsPwYQy1X9AyljFW/jKh8PdcAlWcG2o1mwWI=; b=jVamrOmIYluLDnBL6xWAGDpgLsg40X6vuWa/raDgjddFCl5Nsr6Xr4Xjy0nnMLIjr1 2orjERgZEP4+6C9SR0qXuMFcbNdTO6iOL/Avo0qePb+o3W225BlI4BkM/bI71FnPLQ1B kJB93fI/kPjTIV0v2OkEVJjR0sIx/LbJ+0z1p/HgrnxGls4bnDJJyzGnr/oZw7upehfr /NC54mDtciDHYD/gQiXj7EiaYh/k1+aJfsYHN10CtyGqsr9xztHTyjuHkqQA/FEMNo5o Lc7Y/khmW1hFt51QlfNlVAKGJWenEF+AFjzkCD3i/0klHZ+dvEsWknDj822nqP9oYmC0 LPrg== X-Gm-Message-State: AOAM532Vxo98U/LPA5LBUGjQnDshEtkucM0YVaQM/gd5Hn23TEV8JMa1 jOsYJ1eLJuWfcdW0gyWk+F2xUg== X-Google-Smtp-Source: ABdhPJy8KG5Co2knAgWKxrqUz+zTSFnivorQOkpsDEWq0y8x/CqeHR1bJLHAsbozgKtfArAnLhmX4Q== X-Received: by 2002:ac8:4e8f:: with SMTP id 15mr6748304qtp.189.1633012282856; Thu, 30 Sep 2021 07:31:22 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:22 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 07/15] arm64: kexec: pass kimage as the only argument to relocation function Date: Thu, 30 Sep 2021 14:31:05 +0000 Message-Id: <20210930143113.1502553-8-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=XCzx76XQ; spf=pass (imf21.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.174 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7D46CD0345A6 X-Stat-Signature: weofmsw3k6s31bnxkzbzqrs38wbsdfzb X-HE-Tag: 1633012283-886801 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, kexec relocation function (arm64_relocate_new_kernel) accepts the following arguments: head: start of array that contains relocation information. entry: entry point for new kernel or purgatory. dtb_mem: first and only argument to entry. The number of arguments cannot be easily expended, because this function is also called from HVC_SOFT_RESTART, which preserves only three arguments. And, also arm64_relocate_new_kernel is written in assembly but called without stack, thus no place to move extra arguments to free registers. Soon, we will need to pass more arguments: once we enable MMU we will need to pass information about page tables. Pass kimage to arm64_relocate_new_kernel, and teach it to get the required fields from kimage. Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/kernel/asm-offsets.c | 7 +++++++ arch/arm64/kernel/machine_kexec.c | 7 +++++-- arch/arm64/kernel/relocate_kernel.S | 10 ++++------ 3 files changed, 16 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 551427ae8cc5..1d3319c7518e 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -170,6 +171,12 @@ int main(void) DEFINE(PTRAUTH_KERNEL_KEY_APIA, offsetof(struct ptrauth_keys_kernel, apia)); #endif BLANK(); +#endif +#ifdef CONFIG_KEXEC_CORE + DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); + DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); + DEFINE(KIMAGE_START, offsetof(struct kimage, start)); + BLANK(); #endif return 0; } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 7f1cb5a2a463..e210b19592c6 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -84,6 +84,9 @@ static void kexec_list_flush(struct kimage *kimage) { kimage_entry_t *entry; + dcache_clean_inval_poc((unsigned long)kimage, + (unsigned long)kimage + sizeof(*kimage)); + for (entry = &kimage->head; ; entry++) { unsigned int flag; unsigned long addr; @@ -206,8 +209,8 @@ void machine_kexec(struct kimage *kimage) restart(is_hyp_nvhe(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else { - cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, - kimage->start, kimage->arch.dtb_mem); + cpu_soft_restart(kimage->arch.kern_reloc, virt_to_phys(kimage), + 0, 0); } BUG(); /* Should never get here. */ diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 8c43779e8cc6..63ea19868f63 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -27,9 +27,7 @@ */ SYM_CODE_START(arm64_relocate_new_kernel) /* Setup the list loop variables. */ - mov x18, x2 /* x18 = dtb address */ - mov x17, x1 /* x17 = kimage_start */ - mov x16, x0 /* x16 = kimage_head */ + ldr x16, [x0, #KIMAGE_HEAD] /* x16 = kimage_head */ mov x14, xzr /* x14 = entry ptr */ mov x13, xzr /* x13 = copy dest */ raw_dcache_line_size x15, x1 /* x15 = dcache line size */ @@ -63,12 +61,12 @@ SYM_CODE_START(arm64_relocate_new_kernel) isb /* Start new image. */ - mov x0, x18 + ldr x4, [x0, #KIMAGE_START] /* relocation start */ + ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ mov x1, xzr mov x2, xzr mov x3, xzr - br x17 - + br x4 SYM_CODE_END(arm64_relocate_new_kernel) .align 3 /* To keep the 64-bit values below naturally aligned. */ From patchwork Thu Sep 30 14:31:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1406FC433F5 for ; Thu, 30 Sep 2021 14:31:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9DD1F60551 for ; Thu, 30 Sep 2021 14:31:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9DD1F60551 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 5CA9C9400A8; Thu, 30 Sep 2021 10:31:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 52BAB94003A; Thu, 30 Sep 2021 10:31:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 355419400A8; Thu, 30 Sep 2021 10:31:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0181.hostedemail.com [216.40.44.181]) by kanga.kvack.org (Postfix) with ESMTP id 26E3F94003A for ; Thu, 30 Sep 2021 10:31:25 -0400 (EDT) Received: from smtpin33.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id D86B72B357 for ; Thu, 30 Sep 2021 14:31:24 +0000 (UTC) X-FDA: 78644477688.33.B085D84 Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) by imf14.hostedemail.com (Postfix) with ESMTP id 8533760022C3 for ; Thu, 30 Sep 2021 14:31:24 +0000 (UTC) Received: by mail-qv1-f41.google.com with SMTP id o15so3706131qvq.4 for ; Thu, 30 Sep 2021 07:31:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=WoiHQItxXcz2mVRE9xPdmbdljbzjyyQiR7Sfhvuof+4=; b=XkNv35QVeDXA+9ICJFiqcH3hry1Rj1XAj0QdnxsbRz6fwc9oUluTUbenVe9NBN3pnV CUtSC8fikhV2y56qoIbIQdAUA9zdrd5Nxa+TFhd+oYTNqu8c8kf0g+ku6eMcAWK/g19w GKkIGpvfDKObl1kWlsfqY6H6Si+vIw6XJTqX5Dep2sFYe+/Afg1UNtOAlbIuRexpz+QQ pTQo68B12adRqkcXim8b7BbrVZYUDQPxnDXJui5IcH58P0vuYghNKFrgnGhK4AXfsEyX 03B4BBSipxHXtkVuQI16JQ/KClI21eNiaZVTH04d5cDKAa2eEbm/SAqlRsZ3FbyKuoTh ElDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WoiHQItxXcz2mVRE9xPdmbdljbzjyyQiR7Sfhvuof+4=; b=TPXcUoY0vpeHv73HF6AbBIvehE4e3u3vcLpAxUGSgk5L8fKqpaLH1Yt6paa1M7dN4o WbdKgYqxCIxnJieY2Pp1ZWJ3zTNJkjb8mVpG7ynF20ClCX1PkNOHIZ6AmotkvtH+sms9 pjbpWI8piChWD2T5xteCc3Rz8JZ6xIdXKshmq9fAQEA3/5ksiO2CTcATnnUH4oWbkyWO fnY6Q8tbD6h0dH747Z40c+kwflVLbHg8g+dfUt1GchXuM58Upt3P29vDn/DrmNfA3ucb GRCO7VHdovKJh55BTvYLATDBcjwMqmr52mVeHb+zbz79SlzMTVlVtU/3voiXVvmhEK2l HRpA== X-Gm-Message-State: AOAM530XtjYUR5FjxOXp6wDv428CEQG7X+ellZno8ucGwZrG5dgmHYbP lZr0Y2xU50ysd43pnPW4M1NQiQ== X-Google-Smtp-Source: ABdhPJycCwt+koTjzhe5LqBKA+zcsOrf0LEcpfr3AGB09z+yDDgnPiU9k7YZduVSbOerYBZTV4y+2Q== X-Received: by 2002:ad4:4989:: with SMTP id t9mr4126913qvx.29.1633012283934; Thu, 30 Sep 2021 07:31:23 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:23 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 08/15] arm64: kexec: configure EL2 vectors for kexec Date: Thu, 30 Sep 2021 14:31:06 +0000 Message-Id: <20210930143113.1502553-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 8533760022C3 X-Stat-Signature: prxryqeamoyczp1bqfiphccbdehzzju3 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=XkNv35QV; dmarc=none; spf=pass (imf14.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.219.41 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-HE-Tag: 1633012284-153087 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we have a EL2 mode without VHE, the EL2 vectors are needed in order to switch to EL2 and jump to new world with hypervisor privileges. In preparation to MMU enabled relocation, configure our EL2 table now. Kexec uses #HVC_SOFT_RESTART to branch to the new world, so extend el1_sync vector that is provided by trans_pgd_copy_el2_vectors() to support this case. Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/kexec.h | 1 + arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/machine_kexec.c | 31 +++++++++++++++++++++++++++++++ arch/arm64/mm/trans_pgd-asm.S | 9 ++++++++- 5 files changed, 42 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 5c7ae4c3954b..552a057b40af 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1135,7 +1135,7 @@ config CRASH_DUMP config TRANS_TABLE def_bool y - depends on HIBERNATION + depends on HIBERNATION || KEXEC_CORE config XEN_DOM0 def_bool y diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 00dbcc71aeb2..753a1c398898 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -96,6 +96,7 @@ struct kimage_arch { void *dtb; phys_addr_t dtb_mem; phys_addr_t kern_reloc; + phys_addr_t el2_vectors; }; #ifdef CONFIG_KEXEC_FILE diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 1d3319c7518e..6a2b8b1a4872 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -174,6 +174,7 @@ int main(void) #endif #ifdef CONFIG_KEXEC_CORE DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); + DEFINE(KIMAGE_ARCH_EL2_VECTORS, offsetof(struct kimage, arch.el2_vectors)); DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); DEFINE(KIMAGE_START, offsetof(struct kimage, start)); BLANK(); diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index e210b19592c6..59a4b4172b68 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "cpu-reset.h" @@ -43,7 +44,9 @@ static void _kexec_image_info(const char *func, int line, pr_debug(" start: %lx\n", kimage->start); pr_debug(" head: %lx\n", kimage->head); pr_debug(" nr_segments: %lu\n", kimage->nr_segments); + pr_debug(" dtb_mem: %pa\n", &kimage->arch.dtb_mem); pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc); + pr_debug(" el2_vectors: %pa\n", &kimage->arch.el2_vectors); for (i = 0; i < kimage->nr_segments; i++) { pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n", @@ -143,9 +146,27 @@ static void kexec_segment_flush(const struct kimage *kimage) } } +/* Allocates pages for kexec page table */ +static void *kexec_page_alloc(void *arg) +{ + struct kimage *kimage = (struct kimage *)arg; + struct page *page = kimage_alloc_control_pages(kimage, 0); + + if (!page) + return NULL; + + memset(page_address(page), 0, PAGE_SIZE); + + return page_address(page); +} + int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); + struct trans_pgd_info info = { + .trans_alloc_page = kexec_page_alloc, + .trans_alloc_arg = kimage, + }; /* If in place, relocation is not used, only flush next kernel */ if (kimage->head & IND_DONE) { @@ -154,6 +175,14 @@ int machine_kexec_post_load(struct kimage *kimage) return 0; } + kimage->arch.el2_vectors = 0; + if (is_hyp_nvhe()) { + int rc = trans_pgd_copy_el2_vectors(&info, + &kimage->arch.el2_vectors); + if (rc) + return rc; + } + memcpy(reloc_code, arm64_relocate_new_kernel, arm64_relocate_new_kernel_size); kimage->arch.kern_reloc = __pa(reloc_code); @@ -209,6 +238,8 @@ void machine_kexec(struct kimage *kimage) restart(is_hyp_nvhe(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else { + if (is_hyp_nvhe()) + __hyp_set_vectors(kimage->arch.el2_vectors); cpu_soft_restart(kimage->arch.kern_reloc, virt_to_phys(kimage), 0, 0); } diff --git a/arch/arm64/mm/trans_pgd-asm.S b/arch/arm64/mm/trans_pgd-asm.S index 8c4bffe3089d..021c31573bcb 100644 --- a/arch/arm64/mm/trans_pgd-asm.S +++ b/arch/arm64/mm/trans_pgd-asm.S @@ -24,7 +24,14 @@ SYM_CODE_START_LOCAL(el1_sync) msr vbar_el2, x1 mov x0, xzr eret -1: /* Unexpected argument, set an error */ +1: cmp x0, #HVC_SOFT_RESTART /* Called from kexec */ + b.ne 2f + mov x0, x2 + mov x2, x4 + mov x4, x1 + mov x1, x3 + br x4 +2: /* Unexpected argument, set an error */ mov_q x0, HVC_STUB_ERR eret SYM_CODE_END(el1_sync) From patchwork Thu Sep 30 14:31:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C68C8C433EF for ; Thu, 30 Sep 2021 14:31:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7EAB560551 for ; Thu, 30 Sep 2021 14:31:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7EAB560551 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 63CF09400A9; Thu, 30 Sep 2021 10:31:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C46094003A; Thu, 30 Sep 2021 10:31:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 444169400A9; Thu, 30 Sep 2021 10:31:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0174.hostedemail.com [216.40.44.174]) by kanga.kvack.org (Postfix) with ESMTP id 2C82F94003A for ; Thu, 30 Sep 2021 10:31:26 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id DC2C38249980 for ; Thu, 30 Sep 2021 14:31:25 +0000 (UTC) X-FDA: 78644477730.16.27BB5D8 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) by imf19.hostedemail.com (Postfix) with ESMTP id 92448B000446 for ; Thu, 30 Sep 2021 14:31:25 +0000 (UTC) Received: by mail-qk1-f174.google.com with SMTP id 194so5935111qkj.11 for ; Thu, 30 Sep 2021 07:31:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=8cW/IVlTGR5Obb7nU8rBJjg2uf8tpcGiOc+9iw339z8=; b=LGZ+V3ieH2k3Ak86m2WgJk69Yb4/UpPLxocU8YOxOu6VywnSRkV5tG6QX1CZAFVkjH IdxIVertagbOxiCikHfXeI+gyRxyIK4n4WNp1gD2dM3hVmXHeWY/0MZVIVU7p16dt0EE kQyfCpW5wHlohwk4aa6PjA7uPCBnCZ3aqzRo52eBJjEIs/y+7yHsylM6P6A0lmWxLvdl W4ughsB3UmWmnmLbk1MTeuaS/nugc+CYzDd6T7qr/a1d7xhATy3542wZgxbkBlmDuy4P Vz0qwo0+V3u7WtKUnsS7vgeAv3YUSK4CE8SJAitCqTQ88TYgfnftSHQblwWyCxVgwAFj 5hNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8cW/IVlTGR5Obb7nU8rBJjg2uf8tpcGiOc+9iw339z8=; b=zrwRgk77IZdBMUJ801UI81FysYVQpF4m7kmTWyr3uMo7xZxYWOzSxCQwLfNzcYJYkA WmTHly2YEdou7QTOJqX+jGhZmfucK7qJ1zJ+tC4qhey/s9rVD8Da66kta3udTQsfHHKb 5UrKSIyfjiboEqsdDPMB6zoZeXsqxVL2cQLrbGDjjkw4maZ5xZrz40JowU6G5reqD0Lq Z/DWsPCJLZmbv0CtO1/0okEMHJ+7b8/08f00ehUeqWmivNqtTpq1EMvrkejkDtFZ5LNJ pREP/Tf3LxR34EirRU6VWA28n587vRdcGN/Ll1/3CgUBbfM2z+HAdTpuXf5BJ0gQtpwy ZGjg== X-Gm-Message-State: AOAM530JK9B9yTc29Ii4Apy59NLN4Fztm3IkSdErkv3g0bzvOa9PP39f ggY/mM9q+yXPr9gUdWGSCNU3Rw== X-Google-Smtp-Source: ABdhPJyVmacE1sVXxqp5kGQ3RLtcUFdTcFqDSjNppAD3P445XkdfyacytvCYjB45IVcqvbFv/tXoJQ== X-Received: by 2002:a37:9e8d:: with SMTP id h135mr5202679qke.189.1633012284930; Thu, 30 Sep 2021 07:31:24 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:24 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 09/15] arm64: kexec: relocate in EL1 mode Date: Thu, 30 Sep 2021 14:31:07 +0000 Message-Id: <20210930143113.1502553-10-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 92448B000446 X-Stat-Signature: r4sfxcjr77rmqwh55e66ip11tgtx1pir Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=LGZ+V3ie; dmarc=none; spf=pass (imf19.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.174 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-HE-Tag: 1633012285-473975 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since we are going to keep MMU enabled during relocation, we need to keep EL1 mode throughout the relocation. Keep EL1 enabled, and switch EL2 only before entering the new world. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/kernel/cpu-reset.h | 3 +-- arch/arm64/kernel/machine_kexec.c | 4 ++-- arch/arm64/kernel/relocate_kernel.S | 13 +++++++++++-- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h index 81b3d0fe7a63..296abbac7192 100644 --- a/arch/arm64/kernel/cpu-reset.h +++ b/arch/arm64/kernel/cpu-reset.h @@ -20,11 +20,10 @@ static inline void __noreturn __nocfi cpu_soft_restart(unsigned long entry, { typeof(__cpu_soft_restart) *restart; - unsigned long el2_switch = is_hyp_nvhe(); restart = (void *)__pa_symbol(function_nocfi(__cpu_soft_restart)); cpu_install_idmap(); - restart(el2_switch, entry, arg0, arg1, arg2); + restart(0, entry, arg0, arg1, arg2); unreachable(); } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 59a4b4172b68..cf5d6f22a041 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -240,8 +240,8 @@ void machine_kexec(struct kimage *kimage) } else { if (is_hyp_nvhe()) __hyp_set_vectors(kimage->arch.el2_vectors); - cpu_soft_restart(kimage->arch.kern_reloc, virt_to_phys(kimage), - 0, 0); + cpu_soft_restart(kimage->arch.kern_reloc, + virt_to_phys(kimage), 0, 0); } BUG(); /* Should never get here. */ diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 63ea19868f63..b4fb97312a80 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -13,6 +13,7 @@ #include #include #include +#include /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. @@ -61,12 +62,20 @@ SYM_CODE_START(arm64_relocate_new_kernel) isb /* Start new image. */ + ldr x1, [x0, #KIMAGE_ARCH_EL2_VECTORS] /* relocation start */ + cbz x1, .Lel1 + ldr x1, [x0, #KIMAGE_START] /* relocation start */ + ldr x2, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ + mov x3, xzr + mov x4, xzr + mov x0, #HVC_SOFT_RESTART + hvc #0 /* Jumps from el2 */ +.Lel1: ldr x4, [x0, #KIMAGE_START] /* relocation start */ ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ - mov x1, xzr mov x2, xzr mov x3, xzr - br x4 + br x4 /* Jumps from el1 */ SYM_CODE_END(arm64_relocate_new_kernel) .align 3 /* To keep the 64-bit values below naturally aligned. */ From patchwork Thu Sep 30 14:31:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CE13C433F5 for ; Thu, 30 Sep 2021 14:31:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2D07D6108E for ; Thu, 30 Sep 2021 14:31:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2D07D6108E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 554159400AC; Thu, 30 Sep 2021 10:31:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B35194003A; Thu, 30 Sep 2021 10:31:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32E099400AC; Thu, 30 Sep 2021 10:31:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0038.hostedemail.com [216.40.44.38]) by kanga.kvack.org (Postfix) with ESMTP id 1C7F594003A for ; Thu, 30 Sep 2021 10:31:27 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D1B48183DEA9D for ; Thu, 30 Sep 2021 14:31:26 +0000 (UTC) X-FDA: 78644477772.29.FB1137C Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) by imf05.hostedemail.com (Postfix) with ESMTP id 6E5EC506B171 for ; Thu, 30 Sep 2021 14:31:26 +0000 (UTC) Received: by mail-qk1-f172.google.com with SMTP id c7so5989014qka.2 for ; Thu, 30 Sep 2021 07:31:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=a4UWBG19ndOi78HFaDxtpOvDb/2uQd3EH24/B6Fqa/w=; b=Ds4z7LD5Z+Mi2pRwCBj8ySV7y8sqlAsz82jDxaarB1CA8ARQGfNTDZKfqF8Sw6E4Wh nq/ud6gBw+F5e/Y2OW/7cL/d/DDweMQR48tmkTJJY5qFyYVoBFrpCziwIyh1lASxIO5x ed4MDzfcQ1jkwRMQPZHI8rnx9767iPUYN830/oSF4GY1gF10sLA5M4FSVxSNqSVAh4Vy 2ItlwMPGrCVYIJfqXFG934ycmp8xkFGhONWCDFfKWRVf2ybRcTQpWx2px7piZpzRGj1A c7CbY+TLQanIe+9leu1vf1U2p22xlSETEb/ACrt8zfvpRqXRIt9cbG/H1bR2Ht39X65y 8Uow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=a4UWBG19ndOi78HFaDxtpOvDb/2uQd3EH24/B6Fqa/w=; b=h6FF4NYy+Yb5b08NsMS18gqURND3J96kPs6iIUo+Noam4RdN9b68hU9y33owBKg6bC jTh33+m9Ht6Q47uKkKDd3x7PANW8RgLtI7oH64HV9p8RdslSxtFUaykHGMxKWf+6yv+M cgQxh2EOMYryqP0FFJbhjkS4cuJMGUoaHwfSpkh6CyQ7vx8lGFsqjl5cnWgGn6kkU5eO 3ZSkV4XOpG8ziZYNu/iCcRpw2pbbLzsEsjuKupFe/aI6Ya2drLJ2Vp/yseek9HqrHfPM oceCjT90GurSfGP5YqOONMYNeezyySWqC1NBMnsuWITbdWAQ+Kjp2c88Q+3VAtgwEHsq XRXA== X-Gm-Message-State: AOAM533v0Ug/72/aS+Y2ZyQI4pqsIIiSiYUMXIip0SY0ayU345IXSh0S ZzV7ZU42i+n3G2Ubiz/pX8nwvQ== X-Google-Smtp-Source: ABdhPJyqI9O+dVJs+vntZTBGZ9pxzbqB0UkHVZ/IqqI1agRY7/nbveVWZblYlDFFkKLclptvtLx9aQ== X-Received: by 2002:a37:9144:: with SMTP id t65mr5175741qkd.170.1633012285793; Thu, 30 Sep 2021 07:31:25 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:25 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 10/15] arm64: kexec: use ld script for relocation function Date: Thu, 30 Sep 2021 14:31:08 +0000 Message-Id: <20210930143113.1502553-11-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6E5EC506B171 X-Stat-Signature: rwkrn8xu4d5srextx4iwe61m4ti41csg Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=Ds4z7LD5; dmarc=none; spf=pass (imf05.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.172 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-HE-Tag: 1633012286-160022 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, relocation code declares start and end variables which are used to compute its size. The better way to do this is to use ld script, and put relocation function in its own section. Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/machine_kexec.c | 16 ++++++---------- arch/arm64/kernel/relocate_kernel.S | 14 +------------- arch/arm64/kernel/vmlinux.lds.S | 19 +++++++++++++++++++ 4 files changed, 27 insertions(+), 23 deletions(-) diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index e4ad9db53af1..152cb35bf9df 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -21,5 +21,6 @@ extern char __exittext_begin[], __exittext_end[]; extern char __irqentry_text_start[], __irqentry_text_end[]; extern char __mmuoff_data_start[], __mmuoff_data_end[]; extern char __entry_tramp_text_start[], __entry_tramp_text_end[]; +extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[]; #endif /* __ASM_SECTIONS_H */ diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index cf5d6f22a041..320442d35811 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -21,14 +21,11 @@ #include #include #include +#include #include #include "cpu-reset.h" -/* Global variables for the arm64_relocate_new_kernel routine. */ -extern const unsigned char arm64_relocate_new_kernel[]; -extern const unsigned long arm64_relocate_new_kernel_size; - /** * kexec_image_info - For debugging output. */ @@ -163,6 +160,7 @@ static void *kexec_page_alloc(void *arg) int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); + long reloc_size; struct trans_pgd_info info = { .trans_alloc_page = kexec_page_alloc, .trans_alloc_arg = kimage, @@ -183,17 +181,15 @@ int machine_kexec_post_load(struct kimage *kimage) return rc; } - memcpy(reloc_code, arm64_relocate_new_kernel, - arm64_relocate_new_kernel_size); + reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; + memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code); /* Flush the reloc_code in preparation for its execution. */ dcache_clean_inval_poc((unsigned long)reloc_code, - (unsigned long)reloc_code + - arm64_relocate_new_kernel_size); + (unsigned long)reloc_code + reloc_size); icache_inval_pou((uintptr_t)reloc_code, - (uintptr_t)reloc_code + - arm64_relocate_new_kernel_size); + (uintptr_t)reloc_code + reloc_size); kexec_list_flush(kimage); kexec_image_info(kimage); diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index b4fb97312a80..2227741b96fa 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -15,6 +15,7 @@ #include #include +.section ".kexec_relocate.text", "ax" /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. * @@ -77,16 +78,3 @@ SYM_CODE_START(arm64_relocate_new_kernel) mov x3, xzr br x4 /* Jumps from el1 */ SYM_CODE_END(arm64_relocate_new_kernel) - -.align 3 /* To keep the 64-bit values below naturally aligned. */ - -.Lcopy_end: -.org KEXEC_CONTROL_PAGE_SIZE - -/* - * arm64_relocate_new_kernel_size - Number of bytes to copy to the - * control_code_page. - */ -.globl arm64_relocate_new_kernel_size -arm64_relocate_new_kernel_size: - .quad .Lcopy_end - arm64_relocate_new_kernel diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index f6b1a88245db..0760331af85c 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -63,6 +63,7 @@ #include #include #include +#include #include #include @@ -100,6 +101,16 @@ jiffies = jiffies_64; #define HIBERNATE_TEXT #endif +#ifdef CONFIG_KEXEC_CORE +#define KEXEC_TEXT \ + . = ALIGN(SZ_4K); \ + __relocate_new_kernel_start = .; \ + *(.kexec_relocate.text) \ + __relocate_new_kernel_end = .; +#else +#define KEXEC_TEXT +#endif + #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 #define TRAMP_TEXT \ . = ALIGN(PAGE_SIZE); \ @@ -160,6 +171,7 @@ SECTIONS HYPERVISOR_TEXT IDMAP_TEXT HIBERNATE_TEXT + KEXEC_TEXT TRAMP_TEXT *(.fixup) *(.gnu.warning) @@ -348,3 +360,10 @@ ASSERT(swapper_pg_dir - reserved_pg_dir == RESERVED_SWAPPER_OFFSET, ASSERT(swapper_pg_dir - tramp_pg_dir == TRAMP_SWAPPER_OFFSET, "TRAMP_SWAPPER_OFFSET is wrong!") #endif + +#ifdef CONFIG_KEXEC_CORE +/* kexec relocation code should fit into one KEXEC_CONTROL_PAGE_SIZE */ +ASSERT(__relocate_new_kernel_end - (__relocate_new_kernel_start & ~(SZ_4K - 1)) + <= SZ_4K, "kexec relocation code is too big or misaligned") +ASSERT(KEXEC_CONTROL_PAGE_SIZE >= SZ_4K, "KEXEC_CONTROL_PAGE_SIZE is broken") +#endif From patchwork Thu Sep 30 14:31:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CECC0C433FE for ; Thu, 30 Sep 2021 14:31:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 853636108E for ; Thu, 30 Sep 2021 14:31:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 853636108E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id C5A4C9400AD; Thu, 30 Sep 2021 10:31:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE15A94003A; Thu, 30 Sep 2021 10:31:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5B119400AD; Thu, 30 Sep 2021 10:31:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0096.hostedemail.com [216.40.44.96]) by kanga.kvack.org (Postfix) with ESMTP id 93A0B94003A for ; Thu, 30 Sep 2021 10:31:28 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 521278249980 for ; Thu, 30 Sep 2021 14:31:28 +0000 (UTC) X-FDA: 78644477856.22.0051D74 Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) by imf04.hostedemail.com (Postfix) with ESMTP id 077FA5000321 for ; Thu, 30 Sep 2021 14:31:27 +0000 (UTC) Received: by mail-qk1-f181.google.com with SMTP id 194so5935238qkj.11 for ; Thu, 30 Sep 2021 07:31:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=uM7nePTc3Pr5alLDIhJT26d9ASl4ItsEhUSF3SoRlOw=; b=NwQN2f451Un2tZ6PYU+nIoolI85dWRHZ/cijulDx9n/s9hwGzuXkW3IZ70abRtcZr1 UihJG5WIsBZze7Uhc9U6n8ZhFUVV6sjgMxUHqKkALpbBH1aLmHdxIqHLZotMPYuJMUG5 dtROBU6G+v4Hc99a9FIbqRwzucEMLrSQUt4fG/SlKhwvEU0CMmXsDVfXg4sG/MTZVYZw 3VfpxwF+KE06wtbGousa78FexB4Ux4sRoD/m79nmaRw9GPDPwN2Qah0Ucj9D5Wit2dNh pE1NViWkWEz1soNCMqDHhOse8+PSySe2+kn6mCe+F1oMlNpsMhFX0TDirDbZNOpVKxBo mhYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uM7nePTc3Pr5alLDIhJT26d9ASl4ItsEhUSF3SoRlOw=; b=p6kjAIwtdlNk2TcK3ijayDXS2FyuimkK1rWCOihRtNIvNTztOwUcdy3EN/C0NZE4F1 I8n/Pr0LOFIqLSpzwjtQBdb8VN51iN/yfYjTq22y1XKzA8irjk09ESWdgN0VN3iy3F4+ fMqsNXJN5kAiQ0+p1z0KCo1SQVIdNbGAwqbU69tgRjLb6qTud6QqCjUZvHjGfyFq7nZG qn4L8cpwOTJZbbp/d017LEfOAjUKsavZMlZbWtnU0Uvs5TEsilesNRSZfPz290lPBdXL 9SB/+c0DcX8TnDC79AfvZ5uMb9T57v6qOy69MKKakTINaIV21qF7dqGq56tgpbys1wp7 TNBQ== X-Gm-Message-State: AOAM530boXdlomE959+4VJ6ycIQYfBF7vdULnsBlRP3hdCWiEdu8Stav pver3ESJKMd2WCCHF6PwBxcorA== X-Google-Smtp-Source: ABdhPJyqaxYjUoVqqI6Z2WRqe7YV7XGmKu6XQqMsK8o/rIRI06qkcwlSoFfm5Mh/Y4OegOULvC+4kQ== X-Received: by 2002:ae9:eb58:: with SMTP id b85mr4879543qkg.323.1633012287361; Thu, 30 Sep 2021 07:31:27 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:26 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 11/15] arm64: kexec: install a copy of the linear-map Date: Thu, 30 Sep 2021 14:31:09 +0000 Message-Id: <20210930143113.1502553-12-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 077FA5000321 X-Stat-Signature: t5mc43pkt5siqsp1s1qhj6nw8surok7i Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=NwQN2f45; dmarc=none; spf=pass (imf04.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.181 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-HE-Tag: 1633012287-813733 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To perform the kexec relocation with the MMU enabled, we need a copy of the linear map. Create one, and install it from the relocation code. This has to be done from the assembly code as it will be idmapped with TTBR0. The kernel runs in TTRB1, so can't use the break-before-make sequence on the mapping it is executing from. The makes no difference yet as the relocation code runs with the MMU disabled. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/include/asm/assembler.h | 19 +++++++++++++++++++ arch/arm64/include/asm/kexec.h | 2 ++ arch/arm64/kernel/asm-offsets.c | 2 ++ arch/arm64/kernel/hibernate-asm.S | 20 -------------------- arch/arm64/kernel/machine_kexec.c | 16 ++++++++++++++-- arch/arm64/kernel/relocate_kernel.S | 3 +++ 6 files changed, 40 insertions(+), 22 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index d5281f75a58d..5da176dd37a9 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -483,6 +483,25 @@ alternative_endif _cond_extable .Licache_op\@, \fixup .endm +/* + * To prevent the possibility of old and new partial table walks being visible + * in the tlb, switch the ttbr to a zero page when we invalidate the old + * records. D4.7.1 'General TLB maintenance requirements' in ARM DDI 0487A.i + * Even switching to our copied tables will cause a changed output address at + * each stage of the walk. + */ + .macro break_before_make_ttbr_switch zero_page, page_table, tmp, tmp2 + phys_to_ttbr \tmp, \zero_page + msr ttbr1_el1, \tmp + isb + tlbi vmalle1 + dsb nsh + phys_to_ttbr \tmp, \page_table + offset_ttbr1 \tmp, \tmp2 + msr ttbr1_el1, \tmp + isb + .endm + /* * reset_pmuserenr_el0 - reset PMUSERENR_EL0 if PMUv3 present */ diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 753a1c398898..d678f0ceb7ee 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -97,6 +97,8 @@ struct kimage_arch { phys_addr_t dtb_mem; phys_addr_t kern_reloc; phys_addr_t el2_vectors; + phys_addr_t ttbr1; + phys_addr_t zero_page; }; #ifdef CONFIG_KEXEC_FILE diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 6a2b8b1a4872..1f565224dafd 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -175,6 +175,8 @@ int main(void) #ifdef CONFIG_KEXEC_CORE DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); DEFINE(KIMAGE_ARCH_EL2_VECTORS, offsetof(struct kimage, arch.el2_vectors)); + DEFINE(KIMAGE_ARCH_ZERO_PAGE, offsetof(struct kimage, arch.zero_page)); + DEFINE(KIMAGE_ARCH_TTBR1, offsetof(struct kimage, arch.ttbr1)); DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); DEFINE(KIMAGE_START, offsetof(struct kimage, start)); BLANK(); diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S index a30a2c3f905e..0e1d9c3c6a93 100644 --- a/arch/arm64/kernel/hibernate-asm.S +++ b/arch/arm64/kernel/hibernate-asm.S @@ -15,26 +15,6 @@ #include #include -/* - * To prevent the possibility of old and new partial table walks being visible - * in the tlb, switch the ttbr to a zero page when we invalidate the old - * records. D4.7.1 'General TLB maintenance requirements' in ARM DDI 0487A.i - * Even switching to our copied tables will cause a changed output address at - * each stage of the walk. - */ -.macro break_before_make_ttbr_switch zero_page, page_table, tmp, tmp2 - phys_to_ttbr \tmp, \zero_page - msr ttbr1_el1, \tmp - isb - tlbi vmalle1 - dsb nsh - phys_to_ttbr \tmp, \page_table - offset_ttbr1 \tmp, \tmp2 - msr ttbr1_el1, \tmp - isb -.endm - - /* * Resume from hibernate * diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 320442d35811..fbff545565f1 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -159,6 +159,8 @@ static void *kexec_page_alloc(void *arg) int machine_kexec_post_load(struct kimage *kimage) { + int rc; + pgd_t *trans_pgd; void *reloc_code = page_to_virt(kimage->control_code_page); long reloc_size; struct trans_pgd_info info = { @@ -175,12 +177,22 @@ int machine_kexec_post_load(struct kimage *kimage) kimage->arch.el2_vectors = 0; if (is_hyp_nvhe()) { - int rc = trans_pgd_copy_el2_vectors(&info, - &kimage->arch.el2_vectors); + rc = trans_pgd_copy_el2_vectors(&info, + &kimage->arch.el2_vectors); if (rc) return rc; } + /* Create a copy of the linear map */ + trans_pgd = kexec_page_alloc(kimage); + if (!trans_pgd) + return -ENOMEM; + rc = trans_pgd_create_copy(&info, &trans_pgd, PAGE_OFFSET, PAGE_END); + if (rc) + return rc; + kimage->arch.ttbr1 = __pa(trans_pgd); + kimage->arch.zero_page = __pa(empty_zero_page); + reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code); diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 2227741b96fa..2b80232246f7 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -29,10 +29,13 @@ */ SYM_CODE_START(arm64_relocate_new_kernel) /* Setup the list loop variables. */ + ldr x18, [x0, #KIMAGE_ARCH_ZERO_PAGE] /* x18 = zero page for BBM */ + ldr x17, [x0, #KIMAGE_ARCH_TTBR1] /* x17 = linear map copy */ ldr x16, [x0, #KIMAGE_HEAD] /* x16 = kimage_head */ mov x14, xzr /* x14 = entry ptr */ mov x13, xzr /* x13 = copy dest */ raw_dcache_line_size x15, x1 /* x15 = dcache line size */ + break_before_make_ttbr_switch x18, x17, x1, x2 /* set linear map */ .Lloop: and x12, x16, PAGE_MASK /* x12 = addr */ From patchwork Thu Sep 30 14:31:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E304C433F5 for ; Thu, 30 Sep 2021 14:31:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F3F8E60551 for ; Thu, 30 Sep 2021 14:31:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org F3F8E60551 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 0E82D9400AE; Thu, 30 Sep 2021 10:31:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06F1B94003A; Thu, 30 Sep 2021 10:31:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E54539400AE; Thu, 30 Sep 2021 10:31:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0119.hostedemail.com [216.40.44.119]) by kanga.kvack.org (Postfix) with ESMTP id CD15694003A for ; Thu, 30 Sep 2021 10:31:29 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 873A38249980 for ; Thu, 30 Sep 2021 14:31:29 +0000 (UTC) X-FDA: 78644477898.14.E952EFB Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) by imf20.hostedemail.com (Postfix) with ESMTP id 3191FD0000BC for ; Thu, 30 Sep 2021 14:31:29 +0000 (UTC) Received: by mail-qt1-f180.google.com with SMTP id c20so5846154qtb.2 for ; Thu, 30 Sep 2021 07:31:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=hiDX8HDwZgFFsas4OkEqmNALnkk9CtONfrYzDPONMsg=; b=gbFRTimS7Mkao9xkEPYbRHZaSmf78Aj8pSH6G1/KFohe9hMG6P20jSLuzhJDqeF/Bo tOa6kt9PbJe2vDBc1ovjbpdsqm1TPyx3FgReKagqq4bG9MhZV8T6vLvX0sd/uhkoutJL I9pB7aMGLq2Cua3b0/e+Xf79ceuAeJpM/o8h0KJUdEtVN2ujzbjuExv12XVoKsZqexrD qpXTF95mNaT6dsxe31Y9MmwQDZIVaYs6cCbmfPNU/o8Z9FI42qFo3GcidOPlNqTY8YbM CQmGE1huR7tHEKQ2FAp3ufKRAX2aocQNPrTBnUMWj8KoVckqjmEQiiGSAwjUz+a5g0lG ki9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hiDX8HDwZgFFsas4OkEqmNALnkk9CtONfrYzDPONMsg=; b=Yif0FpeEFf0bhPf5rKUhHPQoqiuQg33lATkXFq3X1+0C5xKKcprSGqryPvqK7rvH8W tZpUnxEhUpb0nkBm0EcwfBfc/TJ7OW2eSkdN660TRvYW2i+d8mIbNGloxvSmAi/hB1uO A1C1ke1FHKniYVb7+Ja8EmSSUGNk26+Kax8QA/Bz8bMTYcXqRTohtPnsP/swhKkiEGyQ Kptu2KaNQ+dxHpRXqcM/YSqYVvbD3BfO9zTP1Za+sGz8Qm97ycOEiU2R3issUMehjwuy PhVEsCxrVuIn6SuC1HtMP5b7vB9OvCARP1vU1/a2KYa3E3yFreYi9RPFSlni1TMcZzs8 9DMA== X-Gm-Message-State: AOAM530RDQk8p2eiWMYX7UUwRbzvbtI/VRNzoxzwduVGOLAzCfUGeO0D RyJw+CRzJ9I0tXDc5YYSbM1C4w== X-Google-Smtp-Source: ABdhPJyfgNR+pR4HEDq+pM1gsVASNpdjnBQkyGm2HuLMBJBmfbiE+cFDJ7JjKeGzrdgbRSvxiuKivA== X-Received: by 2002:ac8:4c8d:: with SMTP id j13mr6612858qtv.155.1633012288410; Thu, 30 Sep 2021 07:31:28 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:27 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 12/15] arm64: kexec: keep MMU enabled during kexec relocation Date: Thu, 30 Sep 2021 14:31:10 +0000 Message-Id: <20210930143113.1502553-13-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=gbFRTimS; spf=pass (imf20.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.180 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3191FD0000BC X-Stat-Signature: c6wk3bhy1jmisakexbxwgqj3pgo69qpm X-HE-Tag: 1633012289-22079 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now, that we have linear map page tables configured, keep MMU enabled to allow faster relocation of segments to final destination. Cavium ThunderX2: Kernel Image size: 38M Iniramfs size: 46M Total relocation size: 84M MMU-disabled: relocation 7.489539915s MMU-enabled: relocation 0.03946095s Broadcom Stingray: The performance data: for a moderate size kernel + initramfs: 25M the relocation was taking 0.382s, with enabled MMU it now takes 0.019s only or x20 improvement. The time is proportional to the size of relocation, therefore if initramfs is larger, 100M it could take over a second. Signed-off-by: Pasha Tatashin Tested-by: Pingfan Liu Acked-by: Catalin Marinas --- arch/arm64/include/asm/kexec.h | 3 +++ arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/machine_kexec.c | 16 +++++++++++---- arch/arm64/kernel/relocate_kernel.S | 31 +++++++++++++++++++---------- 4 files changed, 36 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index d678f0ceb7ee..dca6dedc3b25 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -97,8 +97,11 @@ struct kimage_arch { phys_addr_t dtb_mem; phys_addr_t kern_reloc; phys_addr_t el2_vectors; + phys_addr_t ttbr0; phys_addr_t ttbr1; phys_addr_t zero_page; + unsigned long phys_offset; + unsigned long t0sz; }; #ifdef CONFIG_KEXEC_FILE diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 1f565224dafd..2124357c2075 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -176,6 +176,7 @@ int main(void) DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); DEFINE(KIMAGE_ARCH_EL2_VECTORS, offsetof(struct kimage, arch.el2_vectors)); DEFINE(KIMAGE_ARCH_ZERO_PAGE, offsetof(struct kimage, arch.zero_page)); + DEFINE(KIMAGE_ARCH_PHYS_OFFSET, offsetof(struct kimage, arch.phys_offset)); DEFINE(KIMAGE_ARCH_TTBR1, offsetof(struct kimage, arch.ttbr1)); DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); DEFINE(KIMAGE_START, offsetof(struct kimage, start)); diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index fbff545565f1..1e9a2a45e016 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -196,6 +196,11 @@ int machine_kexec_post_load(struct kimage *kimage) reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code); + rc = trans_pgd_idmap_page(&info, &kimage->arch.ttbr0, + &kimage->arch.t0sz, reloc_code); + if (rc) + return rc; + kimage->arch.phys_offset = virt_to_phys(kimage) - (long)kimage; /* Flush the reloc_code in preparation for its execution. */ dcache_clean_inval_poc((unsigned long)reloc_code, @@ -230,9 +235,9 @@ void machine_kexec(struct kimage *kimage) local_daif_mask(); /* - * Both restart and cpu_soft_restart will shutdown the MMU, disable data + * Both restart and kernel_reloc will shutdown the MMU, disable data * caches. However, restart will start new kernel or purgatory directly, - * cpu_soft_restart will transfer control to arm64_relocate_new_kernel + * kernel_reloc contains the body of arm64_relocate_new_kernel * In kexec case, kimage->start points to purgatory assuming that * kernel entry and dtb address are embedded in purgatory by * userspace (kexec-tools). @@ -246,10 +251,13 @@ void machine_kexec(struct kimage *kimage) restart(is_hyp_nvhe(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else { + void (*kernel_reloc)(struct kimage *kimage); + if (is_hyp_nvhe()) __hyp_set_vectors(kimage->arch.el2_vectors); - cpu_soft_restart(kimage->arch.kern_reloc, - virt_to_phys(kimage), 0, 0); + cpu_install_ttbr0(kimage->arch.ttbr0, kimage->arch.t0sz); + kernel_reloc = (void *)kimage->arch.kern_reloc; + kernel_reloc(kimage); } BUG(); /* Should never get here. */ diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 2b80232246f7..f0a3df9e18a3 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -4,6 +4,8 @@ * * Copyright (C) Linaro. * Copyright (C) Huawei Futurewei Technologies. + * Copyright (C) 2021, Microsoft Corporation. + * Pasha Tatashin */ #include @@ -15,6 +17,13 @@ #include #include +.macro turn_off_mmu tmp1, tmp2 + mov_q \tmp1, INIT_SCTLR_EL1_MMU_OFF + pre_disable_mmu_workaround + msr sctlr_el1, \tmp1 + isb +.endm + .section ".kexec_relocate.text", "ax" /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. @@ -32,22 +41,21 @@ SYM_CODE_START(arm64_relocate_new_kernel) ldr x18, [x0, #KIMAGE_ARCH_ZERO_PAGE] /* x18 = zero page for BBM */ ldr x17, [x0, #KIMAGE_ARCH_TTBR1] /* x17 = linear map copy */ ldr x16, [x0, #KIMAGE_HEAD] /* x16 = kimage_head */ - mov x14, xzr /* x14 = entry ptr */ - mov x13, xzr /* x13 = copy dest */ + ldr x22, [x0, #KIMAGE_ARCH_PHYS_OFFSET] /* x22 phys_offset */ raw_dcache_line_size x15, x1 /* x15 = dcache line size */ break_before_make_ttbr_switch x18, x17, x1, x2 /* set linear map */ .Lloop: and x12, x16, PAGE_MASK /* x12 = addr */ - + sub x12, x12, x22 /* Convert x12 to virt */ /* Test the entry flags. */ .Ltest_source: tbz x16, IND_SOURCE_BIT, .Ltest_indirection /* Invalidate dest page to PoC. */ - mov x2, x13 - add x1, x2, #PAGE_SIZE - dcache_by_myline_op ivac, sy, x2, x1, x15, x20 + mov x19, x13 copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 + add x1, x19, #PAGE_SIZE + dcache_by_myline_op civac, sy, x19, x1, x15, x20 b .Lnext .Ltest_indirection: tbz x16, IND_INDIRECTION_BIT, .Ltest_destination @@ -64,19 +72,20 @@ SYM_CODE_START(arm64_relocate_new_kernel) ic iallu dsb nsh isb + ldr x4, [x0, #KIMAGE_START] /* relocation start */ + ldr x1, [x0, #KIMAGE_ARCH_EL2_VECTORS] /* relocation start */ + ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ + turn_off_mmu x12, x13 /* Start new image. */ - ldr x1, [x0, #KIMAGE_ARCH_EL2_VECTORS] /* relocation start */ cbz x1, .Lel1 - ldr x1, [x0, #KIMAGE_START] /* relocation start */ - ldr x2, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ + mov x1, x4 /* relocation start */ + mov x2, x0 /* dtb address */ mov x3, xzr mov x4, xzr mov x0, #HVC_SOFT_RESTART hvc #0 /* Jumps from el2 */ .Lel1: - ldr x4, [x0, #KIMAGE_START] /* relocation start */ - ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ mov x2, xzr mov x3, xzr br x4 /* Jumps from el1 */ From patchwork Thu Sep 30 14:31:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5461C433EF for ; Thu, 30 Sep 2021 14:31:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 957FC60FC2 for ; Thu, 30 Sep 2021 14:31:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 957FC60FC2 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id F40DE9400AF; Thu, 30 Sep 2021 10:31:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E9F5B94003A; Thu, 30 Sep 2021 10:31:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1B959400AF; Thu, 30 Sep 2021 10:31:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0205.hostedemail.com [216.40.44.205]) by kanga.kvack.org (Postfix) with ESMTP id B976994003A for ; Thu, 30 Sep 2021 10:31:30 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6D7222A00D for ; Thu, 30 Sep 2021 14:31:30 +0000 (UTC) X-FDA: 78644477940.25.8010F7E Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by imf26.hostedemail.com (Postfix) with ESMTP id 1C0BE200172B for ; Thu, 30 Sep 2021 14:31:29 +0000 (UTC) Received: by mail-qv1-f42.google.com with SMTP id a9so3727568qvf.0 for ; Thu, 30 Sep 2021 07:31:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=5Swv6lplDzjvDpp+CqlYUWQ4CihY4pr+PdPAAcX6X0k=; b=QIl+s6AgZb/E9V+qFkaJv28iDbmNKtriCkBKOedA+vTjWGPQUHwOWsSN82vUhYFDPa aKGX8pbht4XajcOxe9AsUPnPHyFeMsy53kNmxOp3NUiw7gCTOFbJHD2xFmkobfsYc6tc +a3sdvCpBBLPEhOdR01yAiMVZgRcqbM5QUcmBf0qkoXY2cJCdwNF4UEA2JfG0IDurSJL wCqpUukwDUB5zPuu1BM5g/ZOGHHARNjAM79AbZES71bLfS3MSsmn1x/xQsLYN7rTb6s9 X7JRzwlBTy1hpr+G/1SwuZkqioteecdFDqEFHGew47hr8gs54RBmUJ+WmmZ3E1UZGl1E MNFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5Swv6lplDzjvDpp+CqlYUWQ4CihY4pr+PdPAAcX6X0k=; b=GK1Z0uqq74aalPoMm0rbMjHEkz0xBOS/tDBKxwBbFquKCnGFrKd4G5f0jivP94Nfnt FHIhjNod0Ji9gRuLwMsZHmM3wVcc8qIdbhnUdFQ5cZgqxQlGkW9+n4rggC/swef01INK u+6OiwPDaUKlxmmajTIItndx5JZp1EBLK7O0dZaw5roK7jLQlfNrD0TZ/GeP/J2Vm+Lc MhLQX6KJMOrjbgg54g+0inQCCi4cORJTaeogTDnOFEwj/HymiIzdfApmTwEWt3nMh9bc uRk6dx3nKk3Eu73siD9QuSEJVnhC6CTjGSrdDIAE9mx+jARoPiobTlmYf48GuJw/vlet hseg== X-Gm-Message-State: AOAM530LFWBTTBl6TGJMIReaTaJ7Bta9dBLlGVoxK9LMNitXMKcjecIv O8NHgvVX84x7kUbGbb2QkHrp3Q== X-Google-Smtp-Source: ABdhPJxh6s/187JmlvOIcH5ROC+cTxyh2kXDCoA5lkqv5O44XaWCE6Q0LOi1e6FSLI9Y930/MBRD8A== X-Received: by 2002:a05:6214:1427:: with SMTP id o7mr5627954qvx.45.1633012289388; Thu, 30 Sep 2021 07:31:29 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:28 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 13/15] arm64: kexec: remove the pre-kexec PoC maintenance Date: Thu, 30 Sep 2021 14:31:11 +0000 Message-Id: <20210930143113.1502553-14-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=QIl+s6Ag; spf=pass (imf26.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.219.42 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1C0BE200172B X-Stat-Signature: kfecuqebubaid4y6s47wjfecue5qjqm4 X-HE-Tag: 1633012289-484337 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that kexec does its relocations with the MMU enabled, we no longer need to clean the relocation data to the PoC. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/kernel/machine_kexec.c | 43 ------------------------------- 1 file changed, 43 deletions(-) diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 1e9a2a45e016..559d47a3c59c 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -77,48 +77,6 @@ int machine_kexec_prepare(struct kimage *kimage) return 0; } -/** - * kexec_list_flush - Helper to flush the kimage list and source pages to PoC. - */ -static void kexec_list_flush(struct kimage *kimage) -{ - kimage_entry_t *entry; - - dcache_clean_inval_poc((unsigned long)kimage, - (unsigned long)kimage + sizeof(*kimage)); - - for (entry = &kimage->head; ; entry++) { - unsigned int flag; - unsigned long addr; - - /* flush the list entries. */ - dcache_clean_inval_poc((unsigned long)entry, - (unsigned long)entry + - sizeof(kimage_entry_t)); - - flag = *entry & IND_FLAGS; - if (flag == IND_DONE) - break; - - addr = (unsigned long)phys_to_virt(*entry & PAGE_MASK); - - switch (flag) { - case IND_INDIRECTION: - /* Set entry point just before the new list page. */ - entry = (kimage_entry_t *)addr - 1; - break; - case IND_SOURCE: - /* flush the source pages. */ - dcache_clean_inval_poc(addr, addr + PAGE_SIZE); - break; - case IND_DESTINATION: - break; - default: - BUG(); - } - } -} - /** * kexec_segment_flush - Helper to flush the kimage segments to PoC. */ @@ -207,7 +165,6 @@ int machine_kexec_post_load(struct kimage *kimage) (unsigned long)reloc_code + reloc_size); icache_inval_pou((uintptr_t)reloc_code, (uintptr_t)reloc_code + reloc_size); - kexec_list_flush(kimage); kexec_image_info(kimage); return 0; From patchwork Thu Sep 30 14:31:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A31BC433F5 for ; Thu, 30 Sep 2021 14:31:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2B2C86108E for ; Thu, 30 Sep 2021 14:31:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2B2C86108E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id E2DF09400B0; Thu, 30 Sep 2021 10:31:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D673594003A; Thu, 30 Sep 2021 10:31:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF6A99400B0; Thu, 30 Sep 2021 10:31:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0245.hostedemail.com [216.40.44.245]) by kanga.kvack.org (Postfix) with ESMTP id A099794003A for ; Thu, 30 Sep 2021 10:31:31 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 617222D007 for ; Thu, 30 Sep 2021 14:31:31 +0000 (UTC) X-FDA: 78644477982.11.B46A6B6 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) by imf27.hostedemail.com (Postfix) with ESMTP id 12BED7001968 for ; Thu, 30 Sep 2021 14:31:30 +0000 (UTC) Received: by mail-qk1-f171.google.com with SMTP id b65so5923132qkc.13 for ; Thu, 30 Sep 2021 07:31:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=+AOkYJOnQvYKR/38NvELzatiZDcWOHgVXKELSMlb5SM=; b=J5UQ7UZyfmWWA8PGgeNyC7jbgKspoR4nC8w8x68ivVY8HpNv+1af8uvYuEZzW7fTQT JW0pPONsRs0dngUFtIDqKg3T4bcpIwi6oVM+aCnbkzDE5Di4J7g/yTcWL7Gu2lQbh+Zm qg+s5ZIK4LCaICSgJO9NMMahZfisKm+U5Tzzbo2ep7S11va1mhIojJHfvKTJq4OocfLr XB80LgfKaKVLrEsgV+kcLXqnC35HNyCkz/OrUln0LzZBCP4N+1sG2ROW9TSujy+hUpJ5 GVJv9mLRsSKM2ypF2prF1I1gIh9AnkvoN8SFFYqEfooatv7xr4Pv+NuWlrcZ5U4ihEAw A5OA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+AOkYJOnQvYKR/38NvELzatiZDcWOHgVXKELSMlb5SM=; b=omMyOzO2a/+wG6blK0X1lpl1vwd0reblWHN3V9oJp1ezOQ3SmlaZF6Bkye/AeXrQyv 3E02A6jKBEJEVvCLcemEIjTGBFRY3o6PPmOy9MKrN7m1ONEOdFcf/NKgQXNiiMPKsEWB hgx7wQdIJXVwU8BGMhbA1FK7qgbciIJjUXin7Jt0J+npfPWrLgn+ZKALoutgYJnv5evA nnOvM6rKuRVT3irEWuQUqQGBvZZbxdCuYOmxKjQaSax9WqAqiQs/E62Wn4iIsB1dSTDe gN75FkPtVitXOIEfvxlsfuT8B5+1FYpXFZC60rU3dC/tiUo9W96jYi0fSP63/7Wwb6k0 r0/w== X-Gm-Message-State: AOAM531ucd/9fySTVMzdTDXvvLT3ACsPw8lQJHJR2zaTCygiEGWTazm3 hhh18uLfIcThO3MiVZ8kBShTDg== X-Google-Smtp-Source: ABdhPJwcDK47/5GmWQlsLpv+of/CmJ8uJnwVzFmIbGlzIotuh9aCabUvIq/XQKbO0rOygE1OJAfqqg== X-Received: by 2002:a37:8883:: with SMTP id k125mr4898086qkd.458.1633012290355; Thu, 30 Sep 2021 07:31:30 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:29 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 14/15] arm64: kexec: remove cpu-reset.h Date: Thu, 30 Sep 2021 14:31:12 +0000 Message-Id: <20210930143113.1502553-15-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 12BED7001968 X-Stat-Signature: e4q8pmaaho3cg1iho31914pesqz1dtmk Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=J5UQ7UZy; spf=pass (imf27.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.171 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none X-HE-Tag: 1633012290-861931 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This header contains only cpu_soft_restart() which is never used directly anymore. So, remove this header, and rename the helper to be cpu_soft_restart(). Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/include/asm/kexec.h | 6 ++++++ arch/arm64/kernel/cpu-reset.S | 7 +++---- arch/arm64/kernel/cpu-reset.h | 30 ------------------------------ arch/arm64/kernel/machine_kexec.c | 6 ++---- 4 files changed, 11 insertions(+), 38 deletions(-) delete mode 100644 arch/arm64/kernel/cpu-reset.h diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index dca6dedc3b25..9839bfc163d7 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -90,6 +90,12 @@ static inline void crash_prepare_suspend(void) {} static inline void crash_post_resume(void) {} #endif +#if defined(CONFIG_KEXEC_CORE) +void cpu_soft_restart(unsigned long el2_switch, unsigned long entry, + unsigned long arg0, unsigned long arg1, + unsigned long arg2); +#endif + #define ARCH_HAS_KIMAGE_ARCH struct kimage_arch { diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S index d47ff63a5b66..48a8af97faa9 100644 --- a/arch/arm64/kernel/cpu-reset.S +++ b/arch/arm64/kernel/cpu-reset.S @@ -16,8 +16,7 @@ .pushsection .idmap.text, "awx" /* - * __cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) - Helper for - * cpu_soft_restart. + * cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) * * @el2_switch: Flag to indicate a switch to EL2 is needed. * @entry: Location to jump to for soft reset. @@ -29,7 +28,7 @@ * branch to what would be the reset vector. It must be executed with the * flat identity mapping. */ -SYM_CODE_START(__cpu_soft_restart) +SYM_CODE_START(cpu_soft_restart) mov_q x12, INIT_SCTLR_EL1_MMU_OFF pre_disable_mmu_workaround /* @@ -48,6 +47,6 @@ SYM_CODE_START(__cpu_soft_restart) mov x1, x3 // arg1 mov x2, x4 // arg2 br x8 -SYM_CODE_END(__cpu_soft_restart) +SYM_CODE_END(cpu_soft_restart) .popsection diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h deleted file mode 100644 index 296abbac7192..000000000000 --- a/arch/arm64/kernel/cpu-reset.h +++ /dev/null @@ -1,30 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * CPU reset routines - * - * Copyright (C) 2015 Huawei Futurewei Technologies. - */ - -#ifndef _ARM64_CPU_RESET_H -#define _ARM64_CPU_RESET_H - -#include - -void __cpu_soft_restart(unsigned long el2_switch, unsigned long entry, - unsigned long arg0, unsigned long arg1, unsigned long arg2); - -static inline void __noreturn __nocfi cpu_soft_restart(unsigned long entry, - unsigned long arg0, - unsigned long arg1, - unsigned long arg2) -{ - typeof(__cpu_soft_restart) *restart; - - restart = (void *)__pa_symbol(function_nocfi(__cpu_soft_restart)); - - cpu_install_idmap(); - restart(0, entry, arg0, arg1, arg2); - unreachable(); -} - -#endif diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 559d47a3c59c..1038494135c8 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -24,8 +24,6 @@ #include #include -#include "cpu-reset.h" - /** * kexec_image_info - For debugging output. */ @@ -201,10 +199,10 @@ void machine_kexec(struct kimage *kimage) * In kexec_file case, the kernel starts directly without purgatory. */ if (kimage->head & IND_DONE) { - typeof(__cpu_soft_restart) *restart; + typeof(cpu_soft_restart) *restart; cpu_install_idmap(); - restart = (void *)__pa_symbol(function_nocfi(__cpu_soft_restart)); + restart = (void *)__pa_symbol(function_nocfi(cpu_soft_restart)); restart(is_hyp_nvhe(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else { From patchwork Thu Sep 30 14:31:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12528529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0AD00C433EF for ; Thu, 30 Sep 2021 14:31:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AF9CF60551 for ; Thu, 30 Sep 2021 14:31:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AF9CF60551 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 307B29400B1; Thu, 30 Sep 2021 10:31:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26CFF94003A; Thu, 30 Sep 2021 10:31:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06F5F9400B1; Thu, 30 Sep 2021 10:31:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0191.hostedemail.com [216.40.44.191]) by kanga.kvack.org (Postfix) with ESMTP id E7C0794003A for ; Thu, 30 Sep 2021 10:31:32 -0400 (EDT) Received: from smtpin35.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A43822D007 for ; Thu, 30 Sep 2021 14:31:32 +0000 (UTC) X-FDA: 78644478024.35.6C05317 Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) by imf27.hostedemail.com (Postfix) with ESMTP id 5EE3770000A0 for ; Thu, 30 Sep 2021 14:31:32 +0000 (UTC) Received: by mail-qt1-f171.google.com with SMTP id m26so5853898qtn.1 for ; Thu, 30 Sep 2021 07:31:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=XxlQwO+zvTPvOSLEI+q8JmpmwLvHiOqSq9H/D1w+4mk=; b=aOdtcLsSMWhdmH/KKMmZnRau19vAPajS02V43ANM6ifhO/tnr/ulMkrIJTQmX7vAlq YADiIG39r9Vb8SZLAASMRqAQkCynKyOg0JkQwWMSIFTPKBulzt9tXQmn3UQ4nHZ3Vec4 voVkB4YK0EgaS0YEkK4o633fgSi4CFFC1DAx3UaBZ+BZVZx8Bh8Rv4FppGd+Ii7Uv5Ii gccTAgM1+8aZq5FXtWSip+4iu1L56aFhwQS1FUhvvzUT1IpJKlNrC0qCJuQm/EWJhkSX yeK0+vcszokE9LKwJMyqxq71n5JxLS0hdUyuJ15uNVFeMWMaLUwrC8M2NJ6UH9kOWleh WTtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XxlQwO+zvTPvOSLEI+q8JmpmwLvHiOqSq9H/D1w+4mk=; b=kb4cZMIFZxxbwTieeSJC8cli7ndKpXt9fYUlMaWoBC+9KYLW5tfTSqCxJNa7MFqR3V CfECmYFAynKzFj+M/FfdgZlJIWmlZi628Y+WChGNE8ZXBA5MJGSk25Kqb/lX9PoNmCPR 48CEggufLGaxkPm2ShUb+o8IfTCqtUdi7Xgj3hijVDRQxHVC1GAAE3OyBfVc41GkOKP4 +ivi5fcynitPW+KRYY0/PMiJJOHKlWDc5OFCfQF1E5nmRVM4ipX6sxXfHJIxxYucAf8E W8CKsMWbNlMj7fM8QUE5WOj+a/lSEO6dh4NaMNl6ooM8xdbeG4jgPp9kBL9h84JZ37hT 6bwg== X-Gm-Message-State: AOAM530RkN60TS2ruL6Z0KIwbvsFAE/tILFtQkpdpy4LMQ0m1NV0B4ao OSdMqG2p0OtEhJm4f0mE7AFA5A== X-Google-Smtp-Source: ABdhPJzecZm5xCqX2NKvPHbAfDt/pCmj5N9m76pk7Eb3EnDGgwSGvIf0RbDVFg/Bu7rhjM9gly/kXw== X-Received: by 2002:ac8:138b:: with SMTP id h11mr6860629qtj.163.1633012291650; Thu, 30 Sep 2021 07:31:31 -0700 (PDT) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id l195sm1528731qke.98.2021.09.30.07.31.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 07:31:30 -0700 (PDT) From: Pasha Tatashin To: pasha.tatashin@soleen.com, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, kernelfans@gmail.com, akpm@linux-foundation.org Subject: [PATCH v18 15/15] arm64: trans_pgd: remove trans_pgd_map_page() Date: Thu, 30 Sep 2021 14:31:13 +0000 Message-Id: <20210930143113.1502553-16-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog In-Reply-To: <20210930143113.1502553-1-pasha.tatashin@soleen.com> References: <20210930143113.1502553-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 5EE3770000A0 X-Stat-Signature: sq1i1h8pxb6mf4sh9cq89y83sjs3yd96 Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=aOdtcLsS; dmarc=none; spf=pass (imf27.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.171 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-HE-Tag: 1633012292-21406 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The intend of trans_pgd_map_page() was to map contiguous range of VA memory to the memory that is getting relocated during kexec. However, since we are now using linear map instead of contiguous range this function is not needed Suggested-by: Pingfan Liu Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas --- arch/arm64/include/asm/trans_pgd.h | 5 +-- arch/arm64/mm/trans_pgd.c | 57 ------------------------------ 2 files changed, 1 insertion(+), 61 deletions(-) diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h index 7b04d32b102c..033d400a4ea4 100644 --- a/arch/arm64/include/asm/trans_pgd.h +++ b/arch/arm64/include/asm/trans_pgd.h @@ -15,7 +15,7 @@ /* * trans_alloc_page * - Allocator that should return exactly one zeroed page, if this - * allocator fails, trans_pgd_create_copy() and trans_pgd_map_page() + * allocator fails, trans_pgd_create_copy() and trans_pgd_idmap_page() * return -ENOMEM error. * * trans_alloc_arg @@ -30,9 +30,6 @@ struct trans_pgd_info { int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **trans_pgd, unsigned long start, unsigned long end); -int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, - void *page, unsigned long dst_addr, pgprot_t pgprot); - int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0, unsigned long *t0sz, void *page); diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 26bd8f2d95af..d7da8ca40d2e 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -217,63 +217,6 @@ int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **dst_pgdp, return rc; } -/* - * Add map entry to trans_pgd for a base-size page at PTE level. - * info: contains allocator and its argument - * trans_pgd: page table in which new map is added. - * page: page to be mapped. - * dst_addr: new VA address for the page - * pgprot: protection for the page. - * - * Returns 0 on success, and -ENOMEM on failure. - */ -int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, - void *page, unsigned long dst_addr, pgprot_t pgprot) -{ - pgd_t *pgdp; - p4d_t *p4dp; - pud_t *pudp; - pmd_t *pmdp; - pte_t *ptep; - - pgdp = pgd_offset_pgd(trans_pgd, dst_addr); - if (pgd_none(READ_ONCE(*pgdp))) { - p4dp = trans_alloc(info); - if (!pgdp) - return -ENOMEM; - pgd_populate(NULL, pgdp, p4dp); - } - - p4dp = p4d_offset(pgdp, dst_addr); - if (p4d_none(READ_ONCE(*p4dp))) { - pudp = trans_alloc(info); - if (!pudp) - return -ENOMEM; - p4d_populate(NULL, p4dp, pudp); - } - - pudp = pud_offset(p4dp, dst_addr); - if (pud_none(READ_ONCE(*pudp))) { - pmdp = trans_alloc(info); - if (!pmdp) - return -ENOMEM; - pud_populate(NULL, pudp, pmdp); - } - - pmdp = pmd_offset(pudp, dst_addr); - if (pmd_none(READ_ONCE(*pmdp))) { - ptep = trans_alloc(info); - if (!ptep) - return -ENOMEM; - pmd_populate_kernel(NULL, pmdp, ptep); - } - - ptep = pte_offset_kernel(pmdp, dst_addr); - set_pte(ptep, pfn_pte(virt_to_pfn(page), pgprot)); - - return 0; -} - /* * The page we want to idmap may be outside the range covered by VA_BITS that * can be built using the kernel's p?d_populate() helpers. As a one off, for a