From patchwork Wed Mar 3 00:22:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D509C433E6 for ; Wed, 3 Mar 2021 00:22:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AB13264E58 for ; Wed, 3 Mar 2021 00:22:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AB13264E58 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8161E8D00FC; Tue, 2 Mar 2021 19:22:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 791558D0101; Tue, 2 Mar 2021 19:22:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 628DF8D00FC; Tue, 2 Mar 2021 19:22:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0245.hostedemail.com [216.40.44.245]) by kanga.kvack.org (Postfix) with ESMTP id D09818D0101 for ; Tue, 2 Mar 2021 19:22:35 -0500 (EST) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 8C8C9824999B for ; Wed, 3 Mar 2021 00:22:35 +0000 (UTC) X-FDA: 77876661870.27.EA16A54 Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) by imf26.hostedemail.com (Postfix) with ESMTP id 6CD39407F8DB for ; Wed, 3 Mar 2021 00:22:34 +0000 (UTC) Received: by mail-qk1-f181.google.com with SMTP id n79so10742965qke.3 for ; Tue, 02 Mar 2021 16:22:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=2ZHo6bqqoFQHDt1XsnxWzUIHllyauSrjIJToDTBiZLY=; b=HFHYvchu6CpdqexxuDSsNYf6zc2J6VEKWIBzKSjPVt7Oj5tn2/oRKB1PtdVtAhgBcu 8QtnWiz/7P8/LqX5wFb1aqUQju3puj7mBA1eY2moIHPzNRpZr6tHqSrfP6M19tFtCndA Dw7uvs/8X/wLRsL+Y2oTYcwh4VUTm1gb0QoWI27HLxniuFGEj0KhR+xLWCeoEvbTnu+7 iDZO6hGQaznymI4Z7Kti3JjhODZC9KYEwl0lPssNr6T7Q8YrqF4+AmY67n+7+DzGelJs WYrCxrVf4ENesVrOPXLzb1mOiHh9nKU3H9yCHxRR06iUcT7IO02sj3oVqitQee/e+0NO HyTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2ZHo6bqqoFQHDt1XsnxWzUIHllyauSrjIJToDTBiZLY=; b=eQtQ2qzTC1tV4Xl/2YTOVRFqoLXN+q10DYWWjv2x1njREXMreM3G+aBQtvrmnKBUuQ UsfXA3K3+BTlEyMtBlxZbgG4zw+qqW4t9vdxIR69FM59lKH4OkZ0BIg94Vrw1y/O7vUL l7ekzqYr6hywzmWcAQjxHNKlacc2NIuI0zizRgd0A3pmHtSAdlArmLmcPNBgtFQMS235 /n0Y8pUNxQO3C/XKrvxWriyOnyzMl2X6Wt9PQqCA6NSEVYrxLVVX1h7QHfroKu2iYvgm 4xYrhxE0kFKI5ql+R3FFqTRH6LOI0az55X2J0f4RIrgdv6CghPIqt0qUJ1KEivttXzmp ty0w== X-Gm-Message-State: AOAM5320W3Waa4u8kn1Y5wn5eoR1p9N2Xu2wQgnxFytGKMIYNfzNFqOc F0jqBxcjUoyesAamJXzTxU+/Sw== X-Google-Smtp-Source: ABdhPJwsTtNwrwyfksF0JxhDL724PEtC95EO+MO9/7fXbkyN8TnG2jdsy9RcmwXwPeG1L2V0hRnmfQ== X-Received: by 2002:a37:4743:: with SMTP id u64mr16003296qka.350.1614730954363; Tue, 02 Mar 2021 16:22:34 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:34 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 01/17] arm64: hyp-stub: Check the size of the HYP stub's vectors Date: Tue, 2 Mar 2021 19:22:14 -0500 Message-Id: <20210303002230.1083176-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6CD39407F8DB X-Stat-Signature: 5akhjchfmw7kmxyzhngn7byrbhgsxjyw Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=mail-qk1-f181.google.com; client-ip=209.85.222.181 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730954-196848 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: James Morse Hibernate contains a set of temporary EL2 vectors used to 'park' EL2 somewhere safe while all the memory is thrown in the air. Making kexec do its relocations with the MMU on means they have to be done at EL1, so EL2 has to be parked. This means yet another set of vectors. All these things do is HVC_SET_VECTORS and HVC_SOFT_RESTART, both of which are implemented by the hyp-stub. Lets copy it instead of re-inventing it. To do this the hyp-stub's entrails need to be packed neatly inside its 2K vectors. Start by moving the final 2K alignment inside the end marker, and add a build check that we didn't overflow 2K. Signed-off-by: James Morse Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/hyp-stub.S | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S index 5eccbd62fec8..572b28646005 100644 --- a/arch/arm64/kernel/hyp-stub.S +++ b/arch/arm64/kernel/hyp-stub.S @@ -41,9 +41,13 @@ SYM_CODE_START(__hyp_stub_vectors) ventry el1_irq_invalid // IRQ 32-bit EL1 ventry el1_fiq_invalid // FIQ 32-bit EL1 ventry el1_error_invalid // Error 32-bit EL1 + .align 11 +SYM_INNER_LABEL(__hyp_stub_vectors_end, SYM_L_LOCAL) SYM_CODE_END(__hyp_stub_vectors) - .align 11 +# Check the __hyp_stub_vectors didn't overflow +.org . - (__hyp_stub_vectors_end - __hyp_stub_vectors) + SZ_2K + SYM_CODE_START_LOCAL(el1_sync) cmp x0, #HVC_SET_VECTORS From patchwork Wed Mar 3 00:22:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34C42C433DB for ; Wed, 3 Mar 2021 00:22:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BF68564FB4 for ; Wed, 3 Mar 2021 00:22:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF68564FB4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 20D388D0102; Tue, 2 Mar 2021 19:22:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 178548D0101; Tue, 2 Mar 2021 19:22:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D511C8D0102; Tue, 2 Mar 2021 19:22:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0253.hostedemail.com [216.40.44.253]) by kanga.kvack.org (Postfix) with ESMTP id 00C988D0101 for ; Tue, 2 Mar 2021 19:22:36 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id B24F118141ED9 for ; Wed, 3 Mar 2021 00:22:36 +0000 (UTC) X-FDA: 77876661912.19.DB081AE Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) by imf27.hostedemail.com (Postfix) with ESMTP id CAC6580192C7 for ; Wed, 3 Mar 2021 00:22:35 +0000 (UTC) Received: by mail-qt1-f174.google.com with SMTP id w1so16293923qto.2 for ; Tue, 02 Mar 2021 16:22:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Me+IJOmpVZMJE6OZITjjNJXxLybVS0PvpkZtNmZgu9s=; b=TexeuK1LzjZUxl+qx3qQULwT87vQYc1MD6Bj68BOVjP3cGLOT2rRwv1i74chPgGfYO UbwIjulEE5WjuUopzaIzxJPEuV7hmvshndmHJBpW/eW8BnrLt1Cr+b2u4gSWJdSJ3OGP JwY+w0YB4TdGP9WGlGzIaL5gqIKtQQYXEyCZUEkKWc4+LfQjREDeMeDJf799nFNdtcYa 3loS2FGH6MBJ5vfpVkn8MnukQ54Se5vlVbHC6yPt9D67uZmJrJJh8vJDHEUtq80ftZPG l026k6HCjmsCE6HHevk/8lCVPEc5jzpfn6+0cF6QpQLBswS06CStNq8tM/bnGM6z0lET sIMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Me+IJOmpVZMJE6OZITjjNJXxLybVS0PvpkZtNmZgu9s=; b=PeEF8wCFX7AdyvLk2lTqZPo0iTtT1Ove1WDqfbqqtsR7s5ORimGJOKlPGE+7bS0W4H 4ZwH4F9feMAUG+iHb6qvHcCnZWEvbyr6s3qGNGKvlHKtCZZjibcqPSQNR26sBwH4qY41 18ks6TD9hKLKDV7SF+ViWOu52ykaNCQhz/61PsnSgNyPiOANnz9LGIBPLMN90Qre3Mtg TEAk1vngPQyRQlxGkBxA6P1WIUH0hoF/FpUVr0hPiN/JuR3nYtKG4aKUOPKyU3CVdYHU 7MO1dd/O5WDcE1Tnx2Zb2B0eKwV5jLLo8BuihHc+WVL7kOvAHeCMUtA1ZLVGzg4SAvE0 wNYA== X-Gm-Message-State: AOAM533iaWvVADNZMl+ne8XqLvR3JfXfgAL92p8enveDnfWvbuU1QeqY iOWiZsfLsEzdyqWp1qbSdUZDRw== X-Google-Smtp-Source: ABdhPJwug5BfAFqjtQc6bxUz5+jervNuBn1c/vyjkzVFvBEuDqyurtMGznEMcQrgeoHuudD6UVUuOA== X-Received: by 2002:ac8:544:: with SMTP id c4mr20060173qth.245.1614730955693; Tue, 02 Mar 2021 16:22:35 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:35 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 02/17] arm64: hyp-stub: Move invalid vector entries into the vectors Date: Tue, 2 Mar 2021 19:22:15 -0500 Message-Id: <20210303002230.1083176-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: CAC6580192C7 X-Stat-Signature: gnmsuh9psuhe3ww49hsq6u1tedrq4qz3 Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf27; identity=mailfrom; envelope-from=""; helo=mail-qt1-f174.google.com; client-ip=209.85.160.174 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730955-662133 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: James Morse Most of the hyp-stub's vector entries are invalid. These are each a unique function that branches to itself. To move these into the vectors, merge the ventry and invalid_vector macros and give each one a unique name. This means we can copy the hyp-stub as it is self contained within its vectors. Signed-off-by: James Morse [Fixed merging issues] Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/hyp-stub.S | 56 +++++++++++++++--------------------- 1 file changed, 23 insertions(+), 33 deletions(-) diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S index 572b28646005..ff329c5c074d 100644 --- a/arch/arm64/kernel/hyp-stub.S +++ b/arch/arm64/kernel/hyp-stub.S @@ -16,31 +16,38 @@ #include #include +.macro invalid_vector label +SYM_CODE_START_LOCAL(\label) + .align 7 + b \label +SYM_CODE_END(\label) +.endm + .text .pushsection .hyp.text, "ax" .align 11 SYM_CODE_START(__hyp_stub_vectors) - ventry el2_sync_invalid // Synchronous EL2t - ventry el2_irq_invalid // IRQ EL2t - ventry el2_fiq_invalid // FIQ EL2t - ventry el2_error_invalid // Error EL2t + invalid_vector hyp_stub_el2t_sync_invalid // Synchronous EL2t + invalid_vector hyp_stub_el2t_irq_invalid // IRQ EL2t + invalid_vector hyp_stub_el2t_fiq_invalid // FIQ EL2t + invalid_vector hyp_stub_el2t_error_invalid // Error EL2t - ventry el2_sync_invalid // Synchronous EL2h - ventry el2_irq_invalid // IRQ EL2h - ventry el2_fiq_invalid // FIQ EL2h - ventry el2_error_invalid // Error EL2h + invalid_vector hyp_stub_el2h_sync_invalid // Synchronous EL2h + invalid_vector hyp_stub_el2h_irq_invalid // IRQ EL2h + invalid_vector hyp_stub_el2h_fiq_invalid // FIQ EL2h + invalid_vector hyp_stub_el2h_error_invalid // Error EL2h ventry el1_sync // Synchronous 64-bit EL1 - ventry el1_irq_invalid // IRQ 64-bit EL1 - ventry el1_fiq_invalid // FIQ 64-bit EL1 - ventry el1_error_invalid // Error 64-bit EL1 - - ventry el1_sync_invalid // Synchronous 32-bit EL1 - ventry el1_irq_invalid // IRQ 32-bit EL1 - ventry el1_fiq_invalid // FIQ 32-bit EL1 - ventry el1_error_invalid // Error 32-bit EL1 + invalid_vector hyp_stub_el1_irq_invalid // IRQ 64-bit EL1 + invalid_vector hyp_stub_el1_fiq_invalid // FIQ 64-bit EL1 + invalid_vector hyp_stub_el1_error_invalid // Error 64-bit EL1 + + invalid_vector hyp_stub_32b_el1_sync_invalid // Synchronous 32-bit EL1 + invalid_vector hyp_stub_32b_el1_irq_invalid // IRQ 32-bit EL1 + invalid_vector hyp_stub_32b_el1_fiq_invalid // FIQ 32-bit EL1 + invalid_vector hyp_stub_32b_el1_error_invalid // Error 32-bit EL1 .align 11 SYM_INNER_LABEL(__hyp_stub_vectors_end, SYM_L_LOCAL) SYM_CODE_END(__hyp_stub_vectors) @@ -173,23 +180,6 @@ SYM_CODE_END(enter_vhe) .popsection -.macro invalid_vector label -SYM_CODE_START_LOCAL(\label) - b \label -SYM_CODE_END(\label) -.endm - - invalid_vector el2_sync_invalid - invalid_vector el2_irq_invalid - invalid_vector el2_fiq_invalid - invalid_vector el2_error_invalid - invalid_vector el1_sync_invalid - invalid_vector el1_irq_invalid - invalid_vector el1_fiq_invalid - invalid_vector el1_error_invalid - - .popsection - /* * __hyp_set_vectors: Call this after boot to set the initial hypervisor * vectors as part of hypervisor installation. On an SMP system, this should From patchwork Wed Mar 3 00:22:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34684C433E6 for ; Wed, 3 Mar 2021 00:22:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BE26B64F25 for ; Wed, 3 Mar 2021 00:22:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BE26B64F25 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F04668D0103; Tue, 2 Mar 2021 19:22:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EDF028D0101; Tue, 2 Mar 2021 19:22:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D02828D0103; Tue, 2 Mar 2021 19:22:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0249.hostedemail.com [216.40.44.249]) by kanga.kvack.org (Postfix) with ESMTP id 6B8608D0101 for ; Tue, 2 Mar 2021 19:22:38 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1EBA2181AF5D3 for ; Wed, 3 Mar 2021 00:22:38 +0000 (UTC) X-FDA: 77876661996.25.617E00A Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf07.hostedemail.com (Postfix) with ESMTP id 5AC97A0009DD for ; Wed, 3 Mar 2021 00:22:37 +0000 (UTC) Received: by mail-qk1-f180.google.com with SMTP id l4so7558147qkl.0 for ; Tue, 02 Mar 2021 16:22:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ys4BurbzE8f3xOtcZckwDgg+eZ/2ibCIBaYK/PrLc5I=; b=dcWduAuTpEnsZ23w4q/nGHITQ9VeRVbopJtAQolCLmSGYvvi8NkPc5iT5pqOtLEI+X /sN64YP6ApTGMwT4gN0aCuIbX2Wf5v4YZFTXQwBOPHm5src59LNHJDHloF+tQBkzuW6H W24eGwHiIzWRAaystxYlL0kVLtHGNHklPZtuwO+YFXmkTv8YCN+qaNaRdJ574xHDHN7n Xl5EbudHJckK44CfwjCJvtC80arwVKn0HT4+PXXkI8IRd+qhia0D+KE2nv8hEhlvqYWx osSeISMrNU39E2zAoOMWs+dLxNpFl9AVxtMy9JdcplGBSucizAVfKfRAnp8EPZLiGRQv reLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ys4BurbzE8f3xOtcZckwDgg+eZ/2ibCIBaYK/PrLc5I=; b=GtISJozgNNpt7y0p4sdOLAGtnKxwvMdDOwS58q8hHb9/EQ5wQQ5xXbhQXOp9mLoDAm 9Dl1OudX3HrbhFWijUSyIPV/c+q60/BVreSUKhF7w5a/s67L69CoRabzkbdcSW+ZBHQb 2RYstk69hYPDTQ8NKQ0Q8tWcYQnmcluz+LCdN3ZIeX+gA1AbX7ADDpVa3P4VOYvxEFRt UaPMrnOYP0uHmMJawd05TqqCvtjjVSTw5FiSdhf7DcUvaMlYBnQtNG/0XY8FerDvFjlr 80azPn7oNiMnrpfw+SgmXR71BdNXGGZswh8OkFQU/w4JWsx78+MjZJX5ndF9wrXfptC8 MXTQ== X-Gm-Message-State: AOAM533SCFgU2ZnUcpfldbB9zY3kEKSQgaduuONXo5BLxVIZ/7o8k2fs 2OLOPSvkeC63Dqs7o9cU3QebgA== X-Google-Smtp-Source: ABdhPJy9cSXd6XyxnY9AZClyXZ4f6veKnYkkla2SRvqY4v7p7SiG4DwUFuL6Yj3oldPgCXfdo+UiPw== X-Received: by 2002:a05:620a:c0e:: with SMTP id l14mr13557520qki.91.1614730957001; Tue, 02 Mar 2021 16:22:37 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:36 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 03/17] arm64: hyp-stub: Move el1_sync into the vectors Date: Tue, 2 Mar 2021 19:22:16 -0500 Message-Id: <20210303002230.1083176-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: y8mrsqq1epa36smonipnwnihjipurtqm X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 5AC97A0009DD Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=mail-qk1-f180.google.com; client-ip=209.85.222.180 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730957-190287 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: James Morse The hyp-stub's el1_sync code doesn't do very much, this can easily fit in the vectors. With this, all of the hyp-stubs behaviour is contained in its vectors. This lets kexec and hibernate copy the hyp-stub when they need its behaviour, instead of re-implementing it. Signed-off-by: James Morse [Fixed merging issues] Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/hyp-stub.S | 59 ++++++++++++++++++------------------ 1 file changed, 29 insertions(+), 30 deletions(-) diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S index ff329c5c074d..d1a73d0f74e0 100644 --- a/arch/arm64/kernel/hyp-stub.S +++ b/arch/arm64/kernel/hyp-stub.S @@ -21,6 +21,34 @@ SYM_CODE_START_LOCAL(\label) .align 7 b \label SYM_CODE_END(\label) +.endm + +.macro hyp_stub_el1_sync +SYM_CODE_START_LOCAL(hyp_stub_el1_sync) + .align 7 + cmp x0, #HVC_SET_VECTORS + b.ne 2f + msr vbar_el2, x1 + b 9f + +2: cmp x0, #HVC_SOFT_RESTART + b.ne 3f + mov x0, x2 + mov x2, x4 + mov x4, x1 + mov x1, x3 + br x4 // no return + +3: cmp x0, #HVC_RESET_VECTORS + beq 9f // Nothing to reset! + + /* Someone called kvm_call_hyp() against the hyp-stub... */ + mov_q x0, HVC_STUB_ERR + eret + +9: mov x0, xzr + eret +SYM_CODE_END(hyp_stub_el1_sync) .endm .text @@ -39,7 +67,7 @@ SYM_CODE_START(__hyp_stub_vectors) invalid_vector hyp_stub_el2h_fiq_invalid // FIQ EL2h invalid_vector hyp_stub_el2h_error_invalid // Error EL2h - ventry el1_sync // Synchronous 64-bit EL1 + hyp_stub_el1_sync // Synchronous 64-bit EL1 invalid_vector hyp_stub_el1_irq_invalid // IRQ 64-bit EL1 invalid_vector hyp_stub_el1_fiq_invalid // FIQ 64-bit EL1 invalid_vector hyp_stub_el1_error_invalid // Error 64-bit EL1 @@ -55,35 +83,6 @@ SYM_CODE_END(__hyp_stub_vectors) # Check the __hyp_stub_vectors didn't overflow .org . - (__hyp_stub_vectors_end - __hyp_stub_vectors) + SZ_2K - -SYM_CODE_START_LOCAL(el1_sync) - cmp x0, #HVC_SET_VECTORS - b.ne 1f - msr vbar_el2, x1 - b 9f - -1: cmp x0, #HVC_VHE_RESTART - b.eq mutate_to_vhe - -2: cmp x0, #HVC_SOFT_RESTART - b.ne 3f - mov x0, x2 - mov x2, x4 - mov x4, x1 - mov x1, x3 - br x4 // no return - -3: cmp x0, #HVC_RESET_VECTORS - beq 9f // Nothing to reset! - - /* Someone called kvm_call_hyp() against the hyp-stub... */ - mov_q x0, HVC_STUB_ERR - eret - -9: mov x0, xzr - eret -SYM_CODE_END(el1_sync) - // nVHE? No way! Give me the real thing! SYM_CODE_START_LOCAL(mutate_to_vhe) // Sanity check: MMU *must* be off From patchwork Wed Mar 3 00:22:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B30AC433E0 for ; Wed, 3 Mar 2021 00:22:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9784064E58 for ; Wed, 3 Mar 2021 00:22:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9784064E58 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 99B8A8D0104; Tue, 2 Mar 2021 19:22:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 95A368D0101; Tue, 2 Mar 2021 19:22:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6361E8D0104; Tue, 2 Mar 2021 19:22:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0174.hostedemail.com [216.40.44.174]) by kanga.kvack.org (Postfix) with ESMTP id A9EC48D0101 for ; Tue, 2 Mar 2021 19:22:39 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6BD581EFD for ; Wed, 3 Mar 2021 00:22:39 +0000 (UTC) X-FDA: 77876662038.24.D817DE7 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by imf09.hostedemail.com (Postfix) with ESMTP id 649796000104 for ; Wed, 3 Mar 2021 00:22:38 +0000 (UTC) Received: by mail-qt1-f169.google.com with SMTP id o1so11849773qta.13 for ; Tue, 02 Mar 2021 16:22:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=t5r+nQTDiOBjAcARZmxYD2C/XCnn3pF+1dbE/Rdt76g=; b=SwKTL4eCvkIvgxU6lGAV//PLN7/clkFuLLU3DXUzK1tQP8M45hnpri8l5cR3f1+Ylp sMpfdEfEz+Et5fTiJRblDRcFcdDME1BtQOImc3LBxmY2ee/EnHXeyesUtIl0vD64exx+ VcRzv+DTWT/hDHKT9/XTJeTMiBflR4d/CWf1PPm9mvSFV7yUEEIVgbmppDsdR6CWOocz V0blw/3FCGwiuLgjQvSQ/B5NmZjurWUX6kzX83Lt5n0pSVAHFZPPAh+CUnOKE143JiH9 EaWCHQ+48dXFnyzY6WGGo0svwQnyOB1kX9/+fl90p2Ln1YIFMmaJnFRL2kdEe5kov6l2 jBiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=t5r+nQTDiOBjAcARZmxYD2C/XCnn3pF+1dbE/Rdt76g=; b=tTB55aexR8WhUqI038V5ChC2clAsGZNcTdFX0rbxC1wF+Sx+f/FsX/u//MLu4/30jz FmnBJTcCBf4D/NTvsthtPClkY6R8t89iLFENc0dCTBSfr92zWQOIABM46Ftdv+ArFU6s tYaJlPTvXU9RDkgrKhhz+bVmWjnqvRwMsZGu6+/1ksVwubsvsGqkaKDfEI263FEU8e05 XFCggKWupEWf7MGtJuxSECaoM3ovPa4yblJRM+mjc83MreDbHbWlyeLltuVVDwWUQ6aI XSP68//gns55KsIPm5e3yrdw0XwRgILb9w1xB2/oSSjqHw8QrKZM+ozIsg7DHzm+jyPV n7bg== X-Gm-Message-State: AOAM531bzmO21HgKlrBQZDaW+k+NYHbK/m6GWfczv3bCA05DVrFLIq6L bkCQyVsRQo5S7bIMvVCvpyxzIg== X-Google-Smtp-Source: ABdhPJzO9otEd6iJTRYqtDhi+XwWHgyDvgnqwPpNWgOkK7byt0RQkwHr9BNXQF6YdGG3LDA/I4eZOQ== X-Received: by 2002:ac8:d44:: with SMTP id r4mr19942609qti.7.1614730958326; Tue, 02 Mar 2021 16:22:38 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:37 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 04/17] arm64: kernel: add helper for booted at EL2 and not VHE Date: Tue, 2 Mar 2021 19:22:17 -0500 Message-Id: <20210303002230.1083176-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 649796000104 X-Stat-Signature: g5gyfrgqmi71s875z9hmc54qa3qxnwy6 Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=mail-qt1-f169.google.com; client-ip=209.85.160.169 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730958-820781 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace places that contain logic like this: is_hyp_mode_available() && !is_kernel_in_hyp_mode() With a dedicated boolean function is_hyp_callable(). This will be needed later in kexec in order to sooner switch back to EL2. Suggested-by: James Morse Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/virt.h | 5 +++++ arch/arm64/kernel/cpu-reset.h | 3 +-- arch/arm64/kernel/hibernate.c | 9 +++------ arch/arm64/kernel/sdei.c | 2 +- 4 files changed, 10 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h index 7379f35ae2c6..4216c8623538 100644 --- a/arch/arm64/include/asm/virt.h +++ b/arch/arm64/include/asm/virt.h @@ -128,6 +128,11 @@ static __always_inline bool is_protected_kvm_enabled(void) return cpus_have_final_cap(ARM64_KVM_PROTECTED_MODE); } +static inline bool is_hyp_callable(void) +{ + return is_hyp_mode_available() && !is_kernel_in_hyp_mode(); +} + #endif /* __ASSEMBLY__ */ #endif /* ! __ASM__VIRT_H */ diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h index ed50e9587ad8..1922e7a690f8 100644 --- a/arch/arm64/kernel/cpu-reset.h +++ b/arch/arm64/kernel/cpu-reset.h @@ -20,8 +20,7 @@ static inline void __noreturn cpu_soft_restart(unsigned long entry, { typeof(__cpu_soft_restart) *restart; - unsigned long el2_switch = !is_kernel_in_hyp_mode() && - is_hyp_mode_available(); + unsigned long el2_switch = is_hyp_callable(); restart = (void *)__pa_symbol(__cpu_soft_restart); cpu_install_idmap(); diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index b1cef371df2b..c764574a1acb 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -48,9 +48,6 @@ */ extern int in_suspend; -/* Do we need to reset el2? */ -#define el2_reset_needed() (is_hyp_mode_available() && !is_kernel_in_hyp_mode()) - /* temporary el2 vectors in the __hibernate_exit_text section. */ extern char hibernate_el2_vectors[]; @@ -125,7 +122,7 @@ int arch_hibernation_header_save(void *addr, unsigned int max_size) hdr->reenter_kernel = _cpu_resume; /* We can't use __hyp_get_vectors() because kvm may still be loaded */ - if (el2_reset_needed()) + if (is_hyp_callable()) hdr->__hyp_stub_vectors = __pa_symbol(__hyp_stub_vectors); else hdr->__hyp_stub_vectors = 0; @@ -387,7 +384,7 @@ int swsusp_arch_suspend(void) dcache_clean_range(__idmap_text_start, __idmap_text_end); /* Clean kvm setup code to PoC? */ - if (el2_reset_needed()) { + if (is_hyp_callable()) { dcache_clean_range(__hyp_idmap_text_start, __hyp_idmap_text_end); dcache_clean_range(__hyp_text_start, __hyp_text_end); } @@ -482,7 +479,7 @@ int swsusp_arch_resume(void) * * We can skip this step if we booted at EL1, or are running with VHE. */ - if (el2_reset_needed()) { + if (is_hyp_callable()) { phys_addr_t el2_vectors = (phys_addr_t)hibernate_exit; el2_vectors += hibernate_el2_vectors - __hibernate_exit_text_start; /* offset */ diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c index 2c7ca449dd51..af0ac2f920cf 100644 --- a/arch/arm64/kernel/sdei.c +++ b/arch/arm64/kernel/sdei.c @@ -200,7 +200,7 @@ unsigned long sdei_arch_get_entry_point(int conduit) * dropped to EL1 because we don't support VHE, then we can't support * SDEI. */ - if (is_hyp_mode_available() && !is_kernel_in_hyp_mode()) { + if (is_hyp_callable()) { pr_err("Not supported on this hardware/boot configuration\n"); goto out_err; } From patchwork Wed Mar 3 00:22:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 300FDC433DB for ; Wed, 3 Mar 2021 00:22:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BFC1464FB5 for ; Wed, 3 Mar 2021 00:22:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BFC1464FB5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B3D158D0105; Tue, 2 Mar 2021 19:22:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ABACE8D0101; Tue, 2 Mar 2021 19:22:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BA308D0105; Tue, 2 Mar 2021 19:22:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0055.hostedemail.com [216.40.44.55]) by kanga.kvack.org (Postfix) with ESMTP id F0D1D8D0101 for ; Tue, 2 Mar 2021 19:22:40 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B5B1583E2 for ; Wed, 3 Mar 2021 00:22:40 +0000 (UTC) X-FDA: 77876662080.17.FBBD830 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf30.hostedemail.com (Postfix) with ESMTP id 2A8CBE0011E6 for ; Wed, 3 Mar 2021 00:22:39 +0000 (UTC) Received: by mail-qk1-f173.google.com with SMTP id q85so22341911qke.8 for ; Tue, 02 Mar 2021 16:22:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=+EiR776LGRIVUqs8E6nTUaQXNk2yNVD65j0QM2uVzow=; b=iC7nX8qZIRxsD5OhB+90MycRKzn7baBcfRaKmKM42+Rfyoo/TzKiZgG5k29FGJj22A 3IRibCdly/NeZnW6O+0v5jPgsqfvXNS1Ri+Wt6eQSXzKy3TbGQQV4zSr6bMCrDJuT7CZ wbYECoGcv5wZ7z7t/QWfP/9Xvt9vqJltxAJ86LAPyLkWWe6VZP61mFU9hqtH0rfIll3j yqgpLmU5q7CK4YhI+VLTcek0xOakTRKi1EyHf/AWlhklroigd/U3mJP42p7NVtYd41ss E/16iSCm4wnaioO47C7o9vp4jWJnN/N+Wd8sno0i9qIHZvkE+SUfGDtxCGR/W6bqKUJ3 pAFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+EiR776LGRIVUqs8E6nTUaQXNk2yNVD65j0QM2uVzow=; b=UDnmibncNz9issKfktooV5FVjCiKTQKIo3A3uSV3NAB3YbBOLyZAPUiRaxrc8I+MFk mlt8LuM3gZjSqASTNyT7ADVyle0pQ0bpLYDW+avVEA1D5008HOFMH97ak5bJDlbG/X4g Qf9e66qZv+hFl81y27GfeCk2RYooa1mwG0RKnAgjEaSxl6GGg8xC8DZGt7efkbSGxM3h rBbAhV95e6qtODgRRAKX4QKRAZalroNnl1THHpRLtAMiRVPG7OqANw8I0mZr6HZ9FMPQ X4wZeBI/vPY8dD7CViB/Eellj5k8L1tEuK8y+msFGhvM44TIOtR1K60MxDNydIKtwgxD E9/w== X-Gm-Message-State: AOAM532tDcFXONi9yI9sTOJwz3koBby+QJw939feakzIKBTkxCu8K63E k/mH2NZ53Zuyfa3FNHvOznRz0FmDN9ndxw== X-Google-Smtp-Source: ABdhPJzAuGpf1um8csVUUBUG3gga/6awbhEJ2kTfUdApasxiQpu4oFmL5IVED6iXYZoNv2iCme9QRw== X-Received: by 2002:a37:6115:: with SMTP id v21mr14220445qkb.239.1614730959621; Tue, 02 Mar 2021 16:22:39 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:39 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 05/17] arm64: trans_pgd: hibernate: Add trans_pgd_copy_el2_vectors Date: Tue, 2 Mar 2021 19:22:18 -0500 Message-Id: <20210303002230.1083176-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: 4g14wp4ma1sr4a4jatmir4hp76kcg17s X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 2A8CBE0011E6 Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=mail-qk1-f173.google.com; client-ip=209.85.222.173 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730959-586106 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Users of trans_pgd may also need a copy of vector table because it is also may be overwritten if a linear map can be overwritten. Move setup of EL2 vectors from hibernate to trans_pgd, so it can be later shared with kexec as well. Suggested-by: James Morse Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/trans_pgd.h | 3 +++ arch/arm64/include/asm/virt.h | 3 +++ arch/arm64/kernel/hibernate.c | 28 ++++++++++------------------ arch/arm64/mm/trans_pgd.c | 20 ++++++++++++++++++++ 4 files changed, 36 insertions(+), 18 deletions(-) diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h index 5d08e5adf3d5..e0760e52d36d 100644 --- a/arch/arm64/include/asm/trans_pgd.h +++ b/arch/arm64/include/asm/trans_pgd.h @@ -36,4 +36,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0, unsigned long *t0sz, void *page); +int trans_pgd_copy_el2_vectors(struct trans_pgd_info *info, + phys_addr_t *el2_vectors); + #endif /* _ASM_TRANS_TABLE_H */ diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h index 4216c8623538..bfbb66018114 100644 --- a/arch/arm64/include/asm/virt.h +++ b/arch/arm64/include/asm/virt.h @@ -67,6 +67,9 @@ */ extern u32 __boot_cpu_mode[2]; +extern char __hyp_stub_vectors[]; +#define ARM64_VECTOR_TABLE_LEN SZ_2K + void __hyp_set_vectors(phys_addr_t phys_vector_base); void __hyp_reset_vectors(void); diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index c764574a1acb..0b8bad8bb6eb 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -48,12 +48,6 @@ */ extern int in_suspend; -/* temporary el2 vectors in the __hibernate_exit_text section. */ -extern char hibernate_el2_vectors[]; - -/* hyp-stub vectors, used to restore el2 during resume from hibernate. */ -extern char __hyp_stub_vectors[]; - /* * The logical cpu number we should resume on, initialised to a non-cpu * number. @@ -428,6 +422,7 @@ int swsusp_arch_resume(void) void *zero_page; size_t exit_size; pgd_t *tmp_pg_dir; + phys_addr_t el2_vectors; void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *, void *, phys_addr_t, phys_addr_t); struct trans_pgd_info trans_info = { @@ -455,6 +450,14 @@ int swsusp_arch_resume(void) return -ENOMEM; } + if (is_hyp_callable()) { + rc = trans_pgd_copy_el2_vectors(&trans_info, &el2_vectors); + if (rc) { + pr_err("Failed to setup el2 vectors\n"); + return rc; + } + } + exit_size = __hibernate_exit_text_end - __hibernate_exit_text_start; /* * Copy swsusp_arch_suspend_exit() to a safe page. This will generate @@ -467,25 +470,14 @@ int swsusp_arch_resume(void) return rc; } - /* - * The hibernate exit text contains a set of el2 vectors, that will - * be executed at el2 with the mmu off in order to reload hyp-stub. - */ - __flush_dcache_area(hibernate_exit, exit_size); - /* * KASLR will cause the el2 vectors to be in a different location in * the resumed kernel. Load hibernate's temporary copy into el2. * * We can skip this step if we booted at EL1, or are running with VHE. */ - if (is_hyp_callable()) { - phys_addr_t el2_vectors = (phys_addr_t)hibernate_exit; - el2_vectors += hibernate_el2_vectors - - __hibernate_exit_text_start; /* offset */ - + if (is_hyp_callable()) __hyp_set_vectors(el2_vectors); - } hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1, resume_hdr.reenter_kernel, restore_pblist, diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 527f0a39c3da..61549451ed3a 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -322,3 +322,23 @@ int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0, return 0; } + +/* + * Create a copy of the vector table so we can call HVC_SET_VECTORS or + * HVC_SOFT_RESTART from contexts where the table may be overwritten. + */ +int trans_pgd_copy_el2_vectors(struct trans_pgd_info *info, + phys_addr_t *el2_vectors) +{ + void *hyp_stub = trans_alloc(info); + + if (!hyp_stub) + return -ENOMEM; + *el2_vectors = virt_to_phys(hyp_stub); + memcpy(hyp_stub, &__hyp_stub_vectors, ARM64_VECTOR_TABLE_LEN); + __flush_icache_range((unsigned long)hyp_stub, + (unsigned long)hyp_stub + ARM64_VECTOR_TABLE_LEN); + __flush_dcache_area(hyp_stub, ARM64_VECTOR_TABLE_LEN); + + return 0; +} From patchwork Wed Mar 3 00:22:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32A98C433E6 for ; Wed, 3 Mar 2021 00:22:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B558964FA9 for ; Wed, 3 Mar 2021 00:22:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B558964FA9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D67A78D0107; Tue, 2 Mar 2021 19:22:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CFC338D0101; Tue, 2 Mar 2021 19:22:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B12B18D0107; Tue, 2 Mar 2021 19:22:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0026.hostedemail.com [216.40.44.26]) by kanga.kvack.org (Postfix) with ESMTP id 3CB588D0101 for ; Tue, 2 Mar 2021 19:22:42 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id E9661824999B for ; Wed, 3 Mar 2021 00:22:41 +0000 (UTC) X-FDA: 77876662122.17.9E2A1E4 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf16.hostedemail.com (Postfix) with ESMTP id 3CD3380192C0 for ; Wed, 3 Mar 2021 00:22:41 +0000 (UTC) Received: by mail-qk1-f180.google.com with SMTP id q85so22341972qke.8 for ; Tue, 02 Mar 2021 16:22:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=PfFJiiZEssC2qRvy3KJXm7j/Y4Op5RWRizR2E1HngfE=; b=jMDvuz/Zgmq1B1MpVMXtPYeQ4yIQVYunhOd0cuqlnPu73G25rI6A8bbQBy7YIsq+XL QLFaC6VqVzMJKOhhn6eYQS9l2TbV/YMXjSMgRvzpw+kAe19ABWobfO0KCffKshahDMdI bUlq1RwNGtmC7j7E3GIqW51fgJ+qLsCRs7WDT07CRyQw/7RmxffV2qQ5/UKUDv4FWBAd rEakxu/EEIwCQfy3eqO5Nq7V6xiG538snw78opWM2Sc0e/ol/nnofnFY7H9ogH44hG/Y QWBHfv7b8Hjc3HJ2evECDpilQuTYVfBG/XhchNb4WN/jfozGm70uSgAosPcJBkyAtyS7 tJWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PfFJiiZEssC2qRvy3KJXm7j/Y4Op5RWRizR2E1HngfE=; b=rj9efx5MLWDqZg6aT89cL2BbNmt/d6lGnpmwKXJFMYkULp75hF8TqB0FBnipJDo7XD JeqxsOvGMPW1qpHPa26QKV7xKl1pwjwN77cp7pzqorwLC5hVVcBWN466CpMhXUsabvZM ZYZndwm1psngVQzUjJXSmS0R0MJKHyodhSEVhtyjBsym+r4fnf4LAO9G5+78dlJoHFX/ 6ESIQmfAvLEjMC2CEvfyB64B/HoIVWPKQQhCTh2szZP5KJWkBro+AaLY3+i8fPi2NaDC FncKp0fi5x7O4ekUPHqX0ZRjgxUP5911Ubj24TkMXMssuXLqjBRs9zUms/+Put3TRncM /lqw== X-Gm-Message-State: AOAM532H6rtDukMFLyHc88tK+6TSiRL6pB/ah/6ig+6r5IJPl6TUrHeR cFgYxp+QJk8jMoPVz7Gt6K8n8Q== X-Google-Smtp-Source: ABdhPJz+uVEaSRAvPV9aeu5i6C2Vj6mlXfoQyiLGMJfrnW+9tKudvptZdZIYZEjfRvwkDzISw2eE+Q== X-Received: by 2002:a37:4e01:: with SMTP id c1mr10344514qkb.16.1614730960877; Tue, 02 Mar 2021 16:22:40 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:40 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 06/17] arm64: hibernate: abstract ttrb0 setup function Date: Tue, 2 Mar 2021 19:22:19 -0500 Message-Id: <20210303002230.1083176-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 3CD3380192C0 X-Stat-Signature: 65k6brbpir5ikm1gpxwbjd4wi7peftnz Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=mail-qk1-f180.google.com; client-ip=209.85.222.180 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730961-283461 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, only hibernate sets custom ttbr0 with safe idmaped function. Kexec, is also going to be using this functinality when relocation code is going to be idmapped. Move the setup seqeuence to a dedicated cpu_install_ttbr0() for custom ttbr0. Suggested-by: James Morse Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/mmu_context.h | 24 ++++++++++++++++++++++++ arch/arm64/kernel/hibernate.c | 21 +-------------------- 2 files changed, 25 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 70ce8c1d2b07..c6521c8c06ac 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -132,6 +132,30 @@ static inline void cpu_install_idmap(void) cpu_switch_mm(lm_alias(idmap_pg_dir), &init_mm); } +/* + * Load our new page tables. A strict BBM approach requires that we ensure that + * TLBs are free of any entries that may overlap with the global mappings we are + * about to install. + * + * For a real hibernate/resume/kexec cycle TTBR0 currently points to a zero + * page, but TLBs may contain stale ASID-tagged entries (e.g. for EFI runtime + * services), while for a userspace-driven test_resume cycle it points to + * userspace page tables (and we must point it at a zero page ourselves). + * + * We change T0SZ as part of installing the idmap. This is undone by + * cpu_uninstall_idmap() in __cpu_suspend_exit(). + */ +static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) +{ + cpu_set_reserved_ttbr0(); + local_flush_tlb_all(); + __cpu_set_tcr_t0sz(t0sz); + + /* avoid cpu_switch_mm() and its SW-PAN and CNP interactions */ + write_sysreg(ttbr0, ttbr0_el1); + isb(); +} + /* * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 0b8bad8bb6eb..ded5115bcb63 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -206,26 +206,7 @@ static int create_safe_exec_page(void *src_start, size_t length, if (rc) return rc; - /* - * Load our new page tables. A strict BBM approach requires that we - * ensure that TLBs are free of any entries that may overlap with the - * global mappings we are about to install. - * - * For a real hibernate/resume cycle TTBR0 currently points to a zero - * page, but TLBs may contain stale ASID-tagged entries (e.g. for EFI - * runtime services), while for a userspace-driven test_resume cycle it - * points to userspace page tables (and we must point it at a zero page - * ourselves). - * - * We change T0SZ as part of installing the idmap. This is undone by - * cpu_uninstall_idmap() in __cpu_suspend_exit(). - */ - cpu_set_reserved_ttbr0(); - local_flush_tlb_all(); - __cpu_set_tcr_t0sz(t0sz); - write_sysreg(trans_ttbr0, ttbr0_el1); - isb(); - + cpu_install_ttbr0(trans_ttbr0, t0sz); *phys_dst_addr = virt_to_phys(page); return 0; From patchwork Wed Mar 3 00:22:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E80EDC433E0 for ; Wed, 3 Mar 2021 00:22:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 812FC64FB5 for ; Wed, 3 Mar 2021 00:22:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 812FC64FB5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 306378D0106; Tue, 2 Mar 2021 19:22:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 24FFC8D0101; Tue, 2 Mar 2021 19:22:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00CFA8D0108; Tue, 2 Mar 2021 19:22:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0133.hostedemail.com [216.40.44.133]) by kanga.kvack.org (Postfix) with ESMTP id 6A5C28D0106 for ; Tue, 2 Mar 2021 19:22:43 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2DD0518163930 for ; Wed, 3 Mar 2021 00:22:43 +0000 (UTC) X-FDA: 77876662206.24.FDCFBC7 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf05.hostedemail.com (Postfix) with ESMTP id 3D845E0011E6 for ; Wed, 3 Mar 2021 00:22:42 +0000 (UTC) Received: by mail-qk1-f180.google.com with SMTP id l132so21138247qke.7 for ; Tue, 02 Mar 2021 16:22:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=FcIEg/sDUKkdURMCkTF1KW5fxOPpkw674yFxDZdydcg=; b=O7cPe49tVOcGseTiLHnzDyJZBZLGwQ8Mqe5Dn8mcQ7k5iBMuCobmbyqA2O4I+bELpB zHLp06elBMsXF5GXqnhfiFgVl8HzIeeAxXZanZh9XuGoGveIfu8Logb3klD20cg4Lajg camd8RfSe3wGmyeRjz3ajPwHXzU4RxSIsDAkaf3k/Ym8xBkcBk/VVx0mn/4+26u0Jk9y E9oPu8omML486M7rrPC5Fi8osaWQbS4+CuhExOmrALnpB9CgOvT1M9JrxHC17qiWLnOl lJtLtc7A4zTfzbV0WqzqanyzS2CyWzkHN3ltfjyEQs54FPOEDCa4pJ3trCc+b9GEUq4a XcFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FcIEg/sDUKkdURMCkTF1KW5fxOPpkw674yFxDZdydcg=; b=ZIWduSuPAhD49A1E8BqkVgv0o6yaT/99rG7SD8UMr5pTjvAE10e+7yqK2mYnslcQwn qQ3YsUq3I4pFeG2c9IE8cL+pIrfrhWCbo6l4udnUtx7LIA67EpyofpUsVfSjSUsGIs6b dMDQTasxlVUEw76z9uXx5ulgmzbI6Tz/3tpOxRxrqMBP3IqxTlMxZbITZQn2Tlzu8KrL mmFORusn1DkhnevtO7uQCOrofgcMZRI7PYgdNwr/m1Brgn/EM7B/cD2b9wOutNtQIpZy suyz6T/wuWu9ObN7BtbdYjqXeqyEY+jl1EeqXSUJ4e7VrtplaLYWLqNpZI8IzTAb5eGW QUxA== X-Gm-Message-State: AOAM530ryg9JkBY4u0mYbCJOIhBisUN7Ueet498xrPcRUyeeP7jRYa2a EsSMfhKY/7iVk40yG8Eiejaoqw== X-Google-Smtp-Source: ABdhPJwfuxp+t9/gww6zzmUbohUN1pBfabKCa8Gw5J69qpWws0xk/h6jJgeII7h77ZjFF8nEckCeZQ== X-Received: by 2002:a37:2795:: with SMTP id n143mr21781313qkn.292.1614730962187; Tue, 02 Mar 2021 16:22:42 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:41 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 07/17] arm64: kexec: flush image and lists during kexec load time Date: Tue, 2 Mar 2021 19:22:20 -0500 Message-Id: <20210303002230.1083176-8-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 3D845E0011E6 X-Stat-Signature: yj65hrwbjdc6i3xnt4hzwkk7sbcmijwx Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=mail-qk1-f180.google.com; client-ip=209.85.222.180 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730962-120561 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, during kexec load we are copying relocation function and flushing it. However, we can also flush kexec relocation buffers and if new kernel image is already in place (i.e. crash kernel), we can also flush the new kernel image itself. Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/machine_kexec.c | 49 +++++++++++++++---------------- 1 file changed, 23 insertions(+), 26 deletions(-) diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 90a335c74442..3a034bc25709 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -59,23 +59,6 @@ void machine_kexec_cleanup(struct kimage *kimage) /* Empty routine needed to avoid build errors. */ } -int machine_kexec_post_load(struct kimage *kimage) -{ - void *reloc_code = page_to_virt(kimage->control_code_page); - - memcpy(reloc_code, arm64_relocate_new_kernel, - arm64_relocate_new_kernel_size); - kimage->arch.kern_reloc = __pa(reloc_code); - kexec_image_info(kimage); - - /* Flush the reloc_code in preparation for its execution. */ - __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size); - flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code + - arm64_relocate_new_kernel_size); - - return 0; -} - /** * machine_kexec_prepare - Prepare for a kexec reboot. * @@ -152,6 +135,29 @@ static void kexec_segment_flush(const struct kimage *kimage) } } +int machine_kexec_post_load(struct kimage *kimage) +{ + void *reloc_code = page_to_virt(kimage->control_code_page); + + /* If in place flush new kernel image, else flush lists and buffers */ + if (kimage->head & IND_DONE) + kexec_segment_flush(kimage); + else + kexec_list_flush(kimage); + + memcpy(reloc_code, arm64_relocate_new_kernel, + arm64_relocate_new_kernel_size); + kimage->arch.kern_reloc = __pa(reloc_code); + kexec_image_info(kimage); + + /* Flush the reloc_code in preparation for its execution. */ + __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size); + flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code + + arm64_relocate_new_kernel_size); + + return 0; +} + /** * machine_kexec - Do the kexec reboot. * @@ -169,13 +175,6 @@ void machine_kexec(struct kimage *kimage) WARN(in_kexec_crash && (stuck_cpus || smp_crash_stop_failed()), "Some CPUs may be stale, kdump will be unreliable.\n"); - /* Flush the kimage list and its buffers. */ - kexec_list_flush(kimage); - - /* Flush the new image if already in place. */ - if ((kimage != kexec_crash_image) && (kimage->head & IND_DONE)) - kexec_segment_flush(kimage); - pr_info("Bye!\n"); local_daif_mask(); @@ -250,8 +249,6 @@ void arch_kexec_protect_crashkres(void) { int i; - kexec_segment_flush(kexec_crash_image); - for (i = 0; i < kexec_crash_image->nr_segments; i++) set_memory_valid( __phys_to_virt(kexec_crash_image->segment[i].mem), From patchwork Wed Mar 3 00:22:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DF75C433E0 for ; Wed, 3 Mar 2021 00:22:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 75D6964FB2 for ; Wed, 3 Mar 2021 00:22:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 75D6964FB2 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5E78E8D0108; Tue, 2 Mar 2021 19:22:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AFFC8D0101; Tue, 2 Mar 2021 19:22:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C0B28D0108; Tue, 2 Mar 2021 19:22:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0092.hostedemail.com [216.40.44.92]) by kanga.kvack.org (Postfix) with ESMTP id CF1F18D0101 for ; Tue, 2 Mar 2021 19:22:44 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 8F49A2C9D for ; Wed, 3 Mar 2021 00:22:44 +0000 (UTC) X-FDA: 77876662248.29.8CD8B45 Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by imf18.hostedemail.com (Postfix) with ESMTP id B30E02000397 for ; Wed, 3 Mar 2021 00:22:43 +0000 (UTC) Received: by mail-qk1-f176.google.com with SMTP id 130so7703986qkh.11 for ; Tue, 02 Mar 2021 16:22:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=hiQbfer3Cy+5MCvp23I2bat0V/tLiwOPOJLrW/6qX9o=; b=UueDkZWeVdt0ZVUeFCaAS95OcbctawuVt2jRXcfNleEQHioaIPsafjb3iN3wPEqMdt J9stLwiFQM/z9W3CKU1wRsTGAe+nD318y5F9IIMtfpYl9ooGJOj9fq3oWyPfDl8EafIQ QkbzJVXwjyMJ3s+WeBV5VvNz7u8LvjFGWo2A0yPGEvKqoLl2EWQ4TA6kdAzhcd0pCOxi bNZKhhZ2PTEsy6gzqf5i+WWcAhW6s6kBywZl2oPrlqd186gvUOmG2SJ5qEzykeb+bgo7 zNWjDcSlC0bR7+IcSuyKZ2OqdH/81WqWH4B4D8mKrZgAcxAnonHq8pCXd9qjhhgHBr97 fnAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hiQbfer3Cy+5MCvp23I2bat0V/tLiwOPOJLrW/6qX9o=; b=KX0xG4YLK6B34nIHB8dDLPS81DyhjEmskve4d+DrvTYvsASpOGStEDjTctlxCdyi/X ww6Ysq4w9ZbpV/vFV1ze3dOtppEGJa8OXRT2YCXnKBxlejJchnZEV6JDZqFEXC/+wcxn ddfY6k8v7tbbZ8MfDuNXQ/0MHKb2aXTCVH5Uc6/VLd2dhp/5hew7ww4bXPEmhim5WrdV jr9WQKJD4NZmf8HSnK4o52v9Oo05s7fmzLQ6LZFM8FxloIecB6W7p8sBy9AndCuE3PLB S+DbynRkPcOOTfOgKnv6bkod8RGt+V7/z4FrGnBr1MnrmiaEf0OxYJF4nNh2xh6uSzvn YOXQ== X-Gm-Message-State: AOAM530HC/HsmZtIBNqcpNYz7OW5PhlLkWNpnN+s0BOukWAPQipIoG7n WkX4S962gRjJQY26Y+QPYaCDIg== X-Google-Smtp-Source: ABdhPJx73Id09SRSbyQ7yttHUhQ1XFsUCayEKdjf5Um64FVS0lPYLKHoeNVlJXLwSpbmAURdNlizvQ== X-Received: by 2002:a37:2785:: with SMTP id n127mr12916932qkn.320.1614730963459; Tue, 02 Mar 2021 16:22:43 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:43 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 08/17] arm64: kexec: skip relocation code for inplace kexec Date: Tue, 2 Mar 2021 19:22:21 -0500 Message-Id: <20210303002230.1083176-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: B30E02000397 X-Stat-Signature: prptuthgjucdooxchhog8syym1uz69ib Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf18; identity=mailfrom; envelope-from=""; helo=mail-qk1-f176.google.com; client-ip=209.85.222.176 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730963-394311 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In case of kdump or when segments are already in place the relocation is not needed, therefore the setup of relocation function and call to it can be skipped. Signed-off-by: Pavel Tatashin Suggested-by: James Morse --- arch/arm64/kernel/machine_kexec.c | 34 ++++++++++++++++++----------- arch/arm64/kernel/relocate_kernel.S | 3 --- 2 files changed, 21 insertions(+), 16 deletions(-) diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 3a034bc25709..b150b65f0b84 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -139,21 +139,23 @@ int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); - /* If in place flush new kernel image, else flush lists and buffers */ - if (kimage->head & IND_DONE) + /* If in place, relocation is not used, only flush next kernel */ + if (kimage->head & IND_DONE) { kexec_segment_flush(kimage); - else - kexec_list_flush(kimage); + kexec_image_info(kimage); + return 0; + } memcpy(reloc_code, arm64_relocate_new_kernel, arm64_relocate_new_kernel_size); kimage->arch.kern_reloc = __pa(reloc_code); - kexec_image_info(kimage); /* Flush the reloc_code in preparation for its execution. */ __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size); flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code + arm64_relocate_new_kernel_size); + kexec_list_flush(kimage); + kexec_image_info(kimage); return 0; } @@ -180,19 +182,25 @@ void machine_kexec(struct kimage *kimage) local_daif_mask(); /* - * cpu_soft_restart will shutdown the MMU, disable data caches, then - * transfer control to the kern_reloc which contains a copy of - * the arm64_relocate_new_kernel routine. arm64_relocate_new_kernel - * uses physical addressing to relocate the new image to its final - * position and transfers control to the image entry point when the - * relocation is complete. + * Both restart and cpu_soft_restart will shutdown the MMU, disable data + * caches. However, restart will start new kernel or purgatory directly, + * cpu_soft_restart will transfer control to arm64_relocate_new_kernel * In kexec case, kimage->start points to purgatory assuming that * kernel entry and dtb address are embedded in purgatory by * userspace (kexec-tools). * In kexec_file case, the kernel starts directly without purgatory. */ - cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start, - kimage->arch.dtb_mem); + if (kimage->head & IND_DONE) { + typeof(__cpu_soft_restart) *restart; + + cpu_install_idmap(); + restart = (void *)__pa_symbol(__cpu_soft_restart); + restart(is_hyp_callable(), kimage->start, kimage->arch.dtb_mem, + 0, 0); + } else { + cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, + kimage->start, kimage->arch.dtb_mem); + } BUG(); /* Should never get here. */ } diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index b78ea5de97a4..8058fabe0a76 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -32,8 +32,6 @@ SYM_CODE_START(arm64_relocate_new_kernel) mov x16, x0 /* x16 = kimage_head */ mov x14, xzr /* x14 = entry ptr */ mov x13, xzr /* x13 = copy dest */ - /* Check if the new image needs relocation. */ - tbnz x16, IND_DONE_BIT, .Ldone raw_dcache_line_size x15, x1 /* x15 = dcache line size */ .Lloop: and x12, x16, PAGE_MASK /* x12 = addr */ @@ -65,7 +63,6 @@ SYM_CODE_START(arm64_relocate_new_kernel) .Lnext: ldr x16, [x14], #8 /* entry = *ptr++ */ tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */ -.Ldone: /* wait for writes from copy_page to finish */ dsb nsh ic iallu From patchwork Wed Mar 3 00:22:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD89BC433E6 for ; Wed, 3 Mar 2021 00:22:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 70E1964FAB for ; Wed, 3 Mar 2021 00:22:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 70E1964FAB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 937978D0109; Tue, 2 Mar 2021 19:22:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D8C48D0101; Tue, 2 Mar 2021 19:22:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F7F88D0109; Tue, 2 Mar 2021 19:22:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0241.hostedemail.com [216.40.44.241]) by kanga.kvack.org (Postfix) with ESMTP id 1E1218D0101 for ; Tue, 2 Mar 2021 19:22:46 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id CB3528249980 for ; Wed, 3 Mar 2021 00:22:45 +0000 (UTC) X-FDA: 77876662290.07.1DD1C99 Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by imf24.hostedemail.com (Postfix) with ESMTP id 634ECA0009CD for ; Wed, 3 Mar 2021 00:22:44 +0000 (UTC) Received: by mail-qk1-f176.google.com with SMTP id f124so4158797qkj.5 for ; Tue, 02 Mar 2021 16:22:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=d9prc1UZUFN5dwsf86cy1HbKMQhioOGgsdfiYkk/1yQ=; b=J384aoqFXPkOEjGgXRUpI8bPGle2xZ/d/8Mv1cPGwzPmmCqCRjHEm8PmZPVcr9NZpT Z/KgtFrb4DGavOcxweTEDqhVwUSVb1KHU+Z037e6LKv3ESLtvkWkFGCvdLtPmlCQZTxT RCBUz0Wn/BU+REqHmXxhFqO6OJhLWdMwlbEFdOvgxGxtN5Z2/RSnuvKm9x2xboFy4ck6 ZaSRuztXVrXoPZDDSlAgvE9ZvAUzTrAYXdlp1fhgAD/cLkj2PII4I8mvog8wI+/OCKQF SpzPPSl9SYrpcKF3EJVTlYtPkA5IMuj47TmLjStHmcLs5ptke7rHEW6NWdgao9sd7QOO Z+FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=d9prc1UZUFN5dwsf86cy1HbKMQhioOGgsdfiYkk/1yQ=; b=hXsRk0uJNOoFCH8yPf6K2rSF3FAUzvBPGQlGfAyWA0VaifszxdP6NTtdJcPDH/48Ed zt9+VnzYJZgpkbSIALqJJIKivE221Y7Iei42CHG9mTU4YWA3VYeRVFyRd/4Lwyljbc3M ewa31LUWdmwkaUhViHDquJ0FUwXipcKdZ0zSQroMO2aAEQdrj7drCS8l0zuAWjAE2rhJ SHvTz4Sq3ip/7FS/eBjRzapMW4n5S84cdXnYomqw7Qahp0b/mu8QvwAETZXd0OwhMsnz f08Q/NCFObJwk1eeE93qgKf6m6nwM1/UQtkyu7xLbfFN01qQfKOvcTIMg7BYiKsotFmu 2ruA== X-Gm-Message-State: AOAM532SQ3W+2ZdmPGTaE8H9u+/Gmy+vmq70G72OMZUBCYoa8uUz5Etn WzEIRRD/HTIrE5R0dtPvQX4Z/Q== X-Google-Smtp-Source: ABdhPJzrhYCjfrgecXKXC5DvGpsZHuFJheu6IGy3c5MWfmYc4aXycDNvYiyhxoSY+nCVc07vrEcfIw== X-Received: by 2002:a37:a48a:: with SMTP id n132mr13712448qke.359.1614730964775; Tue, 02 Mar 2021 16:22:44 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:44 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 09/17] arm64: kexec: Use dcache ops macros instead of open-coding Date: Tue, 2 Mar 2021 19:22:22 -0500 Message-Id: <20210303002230.1083176-10-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 634ECA0009CD X-Stat-Signature: hswhqn79sso8wqnxwyc6j14d87wgfb8d Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf24; identity=mailfrom; envelope-from=""; helo=mail-qk1-f176.google.com; client-ip=209.85.222.176 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730964-514679 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: James Morse kexec does dcache maintenance when it re-writes all memory. Our dcache_by_line_op macro depends on reading the sanitised DminLine from memory. Kexec may have overwritten this, so open-codes the sequence. dcache_by_line_op is a whole set of macros, it uses dcache_line_size which uses read_ctr for the sanitsed DminLine. Reading the DminLine is the first thing the dcache_by_line_op does. Rename dcache_by_line_op dcache_by_myline_op and take DminLine as an argument. Kexec can now use the slightly smaller macro. This makes up-coming changes to the dcache maintenance easier on the eye. Code generated by the existing callers is unchanged. Signed-off-by: James Morse [Fixed merging issues] Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/assembler.h | 12 ++++++++---- arch/arm64/kernel/relocate_kernel.S | 13 +++---------- 2 files changed, 11 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index ca31594d3d6c..29061b76aab6 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -371,10 +371,9 @@ alternative_else alternative_endif .endm - .macro dcache_by_line_op op, domain, kaddr, size, tmp1, tmp2 - dcache_line_size \tmp1, \tmp2 + .macro dcache_by_myline_op op, domain, kaddr, size, linesz, tmp2 add \size, \kaddr, \size - sub \tmp2, \tmp1, #1 + sub \tmp2, \linesz, #1 bic \kaddr, \kaddr, \tmp2 9998: .ifc \op, cvau @@ -394,12 +393,17 @@ alternative_endif .endif .endif .endif - add \kaddr, \kaddr, \tmp1 + add \kaddr, \kaddr, \linesz cmp \kaddr, \size b.lo 9998b dsb \domain .endm + .macro dcache_by_line_op op, domain, kaddr, size, tmp1, tmp2 + dcache_line_size \tmp1, \tmp2 + dcache_by_myline_op \op, \domain, \kaddr, \size, \tmp1, \tmp2 + .endm + /* * Macro to perform an instruction cache maintenance for the interval * [start, end) diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 8058fabe0a76..718037bef560 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -41,16 +41,9 @@ SYM_CODE_START(arm64_relocate_new_kernel) tbz x16, IND_SOURCE_BIT, .Ltest_indirection /* Invalidate dest page to PoC. */ - mov x2, x13 - add x20, x2, #PAGE_SIZE - sub x1, x15, #1 - bic x2, x2, x1 -2: dc ivac, x2 - add x2, x2, x15 - cmp x2, x20 - b.lo 2b - dsb sy - + mov x2, x13 + mov x1, #PAGE_SIZE + dcache_by_myline_op ivac, sy, x2, x1, x15, x20 copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 b .Lnext .Ltest_indirection: From patchwork Wed Mar 3 00:22:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EA14C433E0 for ; Wed, 3 Mar 2021 00:22:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 183ED64FA9 for ; Wed, 3 Mar 2021 00:22:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 183ED64FA9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 716668D010A; Tue, 2 Mar 2021 19:22:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CEBF8D0101; Tue, 2 Mar 2021 19:22:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42DAC8D010A; Tue, 2 Mar 2021 19:22:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0030.hostedemail.com [216.40.44.30]) by kanga.kvack.org (Postfix) with ESMTP id 549F98D0101 for ; Tue, 2 Mar 2021 19:22:47 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 14AE98249980 for ; Wed, 3 Mar 2021 00:22:47 +0000 (UTC) X-FDA: 77876662374.25.4ED9549 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) by imf06.hostedemail.com (Postfix) with ESMTP id D91ADC0007C3 for ; Wed, 3 Mar 2021 00:22:45 +0000 (UTC) Received: by mail-qk1-f171.google.com with SMTP id g185so3058616qkf.6 for ; Tue, 02 Mar 2021 16:22:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=tCKT8dQy3j0JXLfUAkLHptP8jUCHy5Mtlem3XeT2uzw=; b=DhKQLAF9gGWy+VQ4vEFv8vyOX67wEL5g+Ibv7M6YGg2GyOsWVpcLrO8l/KN0L5IZAc RxUn6cVCLaJEnn1GP49dKDyfXS2S5pAiSCfkfdBs3217gnmAIZ5KJF9kQwUjn08WVXoD eVb/xAgyG6HCeSJF560IMqYfvfngZgwPC0Gmto4sET2VI1otYwBlS0xkYWwsZi+5Y5wu CiGi9fY2dPpyuaQ0XGJefCQyLBo5uJyBd2Zt5atz0jeIcn/r/0MV3W88wPds72buWKfc OID9WDzMy8wulcxcHzZaME1ROXYkHzAQzPpJnO8T7pARrDaLZCrx6HDCoBT9CIwWJebN jArg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tCKT8dQy3j0JXLfUAkLHptP8jUCHy5Mtlem3XeT2uzw=; b=qHdghZhhDlxPoafH064IZgDnEGT9J/CX6rtO6aWWef4c1VZ3Z77bXRDJSXTSTUUI90 4cKjGUZEKPsw1vImhNrQv0mZSRLdOVZ648rmGCmwaZ2jtMGC54BoaLyidIpdRggCKg17 blNAUVtgJIbrDOEPn9v8eGZaBJ659TdR3/PKmcP0gsPo6xxFEyCaMWDflh7gpL5VYXbX P8vs2Cu4crtVGq8LaTDgeEfdlNLzOAXx1AAD5SmXf3mN9ICaHslbUdH4FonQWrGEeLo4 QgJz6bOEVjpxS5v9xWe5LQOxgZpT4pMPOsTNzKZCZp1oKQOtzQsFXBBcxYnA+6dR2kzu FSrQ== X-Gm-Message-State: AOAM53301ZPA0KUtKvcB4ltuKCDYDbkkjV72Pvqi0v0vHW1AI+0+nQPh Th2w/iQiD1A0FgGLLRNZiljtE6DMKHeeQw== X-Google-Smtp-Source: ABdhPJz8rw5a4BVvRDYMXLHg55FhSEneNbuZ0yxoLq7h3VikY6X4yYzXWKIMomKucbL+q9Rc23LaTA== X-Received: by 2002:a05:620a:13a5:: with SMTP id m5mr2243290qki.498.1614730966050; Tue, 02 Mar 2021 16:22:46 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:45 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 10/17] arm64: kexec: pass kimage as the only argument to relocation function Date: Tue, 2 Mar 2021 19:22:23 -0500 Message-Id: <20210303002230.1083176-11-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D91ADC0007C3 X-Stat-Signature: pkdh7kgspedqouu738wb534ypjrst14r Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=mail-qk1-f171.google.com; client-ip=209.85.222.171 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730965-788109 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, kexec relocation function (arm64_relocate_new_kernel) accepts the following arguments: head: start of array that contains relocation information. entry: entry point for new kernel or purgatory. dtb_mem: first and only argument to entry. The number of arguments cannot be easily expended, because this function is also called from HVC_SOFT_RESTART, which preserves only three arguments. And, also arm64_relocate_new_kernel is written in assembly but called without stack, thus no place to move extra arguments to free registers. Soon, we will need to pass more arguments: once we enable MMU we will need to pass information about page tables. Pass kimage to arm64_relocate_new_kernel, and teach it to get the required fields from kimage. Suggested-by: James Morse Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/asm-offsets.c | 7 +++++++ arch/arm64/kernel/machine_kexec.c | 6 ++++-- arch/arm64/kernel/relocate_kernel.S | 10 ++++------ 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index a36e2fc330d4..0c92e193f866 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -153,6 +154,12 @@ int main(void) DEFINE(PTRAUTH_USER_KEY_APGA, offsetof(struct ptrauth_keys_user, apga)); DEFINE(PTRAUTH_KERNEL_KEY_APIA, offsetof(struct ptrauth_keys_kernel, apia)); BLANK(); +#endif +#ifdef CONFIG_KEXEC_CORE + DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); + DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); + DEFINE(KIMAGE_START, offsetof(struct kimage, start)); + BLANK(); #endif return 0; } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index b150b65f0b84..2e734e4ae12e 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -83,6 +83,8 @@ static void kexec_list_flush(struct kimage *kimage) { kimage_entry_t *entry; + __flush_dcache_area(kimage, sizeof(*kimage)); + for (entry = &kimage->head; ; entry++) { unsigned int flag; void *addr; @@ -198,8 +200,8 @@ void machine_kexec(struct kimage *kimage) restart(is_hyp_callable(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else { - cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, - kimage->start, kimage->arch.dtb_mem); + cpu_soft_restart(kimage->arch.kern_reloc, virt_to_phys(kimage), + 0, 0); } BUG(); /* Should never get here. */ diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 718037bef560..36b4496524c3 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -27,9 +27,7 @@ */ SYM_CODE_START(arm64_relocate_new_kernel) /* Setup the list loop variables. */ - mov x18, x2 /* x18 = dtb address */ - mov x17, x1 /* x17 = kimage_start */ - mov x16, x0 /* x16 = kimage_head */ + ldr x16, [x0, #KIMAGE_HEAD] /* x16 = kimage_head */ mov x14, xzr /* x14 = entry ptr */ mov x13, xzr /* x13 = copy dest */ raw_dcache_line_size x15, x1 /* x15 = dcache line size */ @@ -63,12 +61,12 @@ SYM_CODE_START(arm64_relocate_new_kernel) isb /* Start new image. */ - mov x0, x18 + ldr x4, [x0, #KIMAGE_START] /* relocation start */ + ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ mov x1, xzr mov x2, xzr mov x3, xzr - br x17 - + br x4 SYM_CODE_END(arm64_relocate_new_kernel) .align 3 /* To keep the 64-bit values below naturally aligned. */ From patchwork Wed Mar 3 00:22:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A080C433E6 for ; Wed, 3 Mar 2021 00:22:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 13D6164E58 for ; Wed, 3 Mar 2021 00:22:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 13D6164E58 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3DB698D010B; Tue, 2 Mar 2021 19:22:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 258808D0101; Tue, 2 Mar 2021 19:22:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D12FE8D010B; Tue, 2 Mar 2021 19:22:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0022.hostedemail.com [216.40.44.22]) by kanga.kvack.org (Postfix) with ESMTP id A034F8D0101 for ; Tue, 2 Mar 2021 19:22:48 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5B43B612F for ; Wed, 3 Mar 2021 00:22:48 +0000 (UTC) X-FDA: 77876662416.28.F63D467 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by imf04.hostedemail.com (Postfix) with ESMTP id 6E1F8132 for ; Wed, 3 Mar 2021 00:22:47 +0000 (UTC) Received: by mail-qt1-f169.google.com with SMTP id d11so15555669qtx.9 for ; Tue, 02 Mar 2021 16:22:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=CkOHd4Ubs6QxUZPpaIRvKfllHz4QaZDE7Cb9NwMEurE=; b=kv+djcD5Dn8tl5qEcKKg9wUfgrKRtIeHMDiyEGXGQs7F/ZmuhWe//JnMzxmKu3F7AH 88SbhKfkE78CfHCAU44Ln2BV0iQYNwXNNODQZqJTwJvNp3lruyG1V2IxzYqcxJbNeIZ6 girfcwSI1nPDGrreCNMXLSEJQko02mF+3Yihp7JjLbeVQdjBz1rF1xU7FCCpI1MADl1I pOeZtuW+wh2ejaP/7dgGE0DRh69BVBAUaJ+0RDEenZ88YSFdaBOLm7hTX3jYfk39tjpV 9gPTcFqCKqbCgpdfet/cTN4tW9o3BcjhJkKgOcD4wkV5+2WV0S3t0adzPBj5dTCa3CUU saSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CkOHd4Ubs6QxUZPpaIRvKfllHz4QaZDE7Cb9NwMEurE=; b=eu6OTIXMQtJZRt6QVZPkV73Jx0qKdDvtP4n8lLqf4WsLBd4rkKvdHUxZyNqgp7pCsh 1IozZCGsyrahLCeI+LphgS3PtGD1y1aR2FlPRhycqllI/csGE0qziTMfbaMAt+ZlEySJ jgc/tobw8r1zxbKrnNwNhz/7bLmB49hw30v/8WdGkt9/EKF8DfASAytZeWu2R56TAOA3 vPR7pxtM2Vw/zGcJGU7J0itzS1F5ctZwr0RXupvkJTJ4yUkOY1pWNbEzdi9MrvRn5f9e 3yr8NZ8Ab+zU5m6XNxicmdZuPHnktOCUCnrO4+Y3xEJ59KZuwfVPAFvw37eJLLHMjAsc K8cw== X-Gm-Message-State: AOAM531FLJ4fuRMevubL5iulKfpauovfpy8dBED5yKISdDjf4pEsJEPg jn9fgFzHGVTjFiM7IsIyjD4gEA== X-Google-Smtp-Source: ABdhPJzh2Sttz/3luOleODmtp3yKBF23rhlq0H6VUfZnw3KEBQCZesiL+EeFQlA4puGBiExt3HvrBQ== X-Received: by 2002:ac8:5947:: with SMTP id 7mr8369297qtz.60.1614730967312; Tue, 02 Mar 2021 16:22:47 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:46 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 11/17] arm64: kexec: kexec may require EL2 vectors Date: Tue, 2 Mar 2021 19:22:24 -0500 Message-Id: <20210303002230.1083176-12-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 6E1F8132 X-Stat-Signature: 8f7wm4mkct7yz4mwhs5pawxahq8iodw6 Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf04; identity=mailfrom; envelope-from=""; helo=mail-qt1-f169.google.com; client-ip=209.85.160.169 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730967-816585 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we have a EL2 mode without VHE, the EL2 vectors are needed in order to switch to EL2 and jump to new world with hypervisor privileges. In preporation to MMU enabled relocation, configure our EL2 table now. Suggested-by: James Morse Signed-off-by: Pavel Tatashin --- arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/kexec.h | 1 + arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/machine_kexec.c | 31 +++++++++++++++++++++++++++++++ 4 files changed, 34 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1f212b47a48a..825fe88b7c08 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1141,7 +1141,7 @@ config CRASH_DUMP config TRANS_TABLE def_bool y - depends on HIBERNATION + depends on HIBERNATION || KEXEC_CORE config XEN_DOM0 def_bool y diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 9befcd87e9a8..305cf0840ed3 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -96,6 +96,7 @@ struct kimage_arch { void *dtb; phys_addr_t dtb_mem; phys_addr_t kern_reloc; + phys_addr_t el2_vectors; /* Core ELF header buffer */ void *elf_headers; unsigned long elf_headers_mem; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 0c92e193f866..2e3278df1fc3 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -157,6 +157,7 @@ int main(void) #endif #ifdef CONFIG_KEXEC_CORE DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); + DEFINE(KIMAGE_ARCH_EL2_VECTORS, offsetof(struct kimage, arch.el2_vectors)); DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); DEFINE(KIMAGE_START, offsetof(struct kimage, start)); BLANK(); diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 2e734e4ae12e..fb03b6676fb9 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -20,6 +20,7 @@ #include #include #include +#include #include "cpu-reset.h" @@ -42,7 +43,9 @@ static void _kexec_image_info(const char *func, int line, pr_debug(" start: %lx\n", kimage->start); pr_debug(" head: %lx\n", kimage->head); pr_debug(" nr_segments: %lu\n", kimage->nr_segments); + pr_debug(" dtb_mem: %pa\n", &kimage->arch.dtb_mem); pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc); + pr_debug(" el2_vectors: %pa\n", &kimage->arch.el2_vectors); for (i = 0; i < kimage->nr_segments; i++) { pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n", @@ -137,9 +140,27 @@ static void kexec_segment_flush(const struct kimage *kimage) } } +/* Allocates pages for kexec page table */ +static void *kexec_page_alloc(void *arg) +{ + struct kimage *kimage = (struct kimage *)arg; + struct page *page = kimage_alloc_control_pages(kimage, 0); + + if (!page) + return NULL; + + memset(page_address(page), 0, PAGE_SIZE); + + return page_address(page); +} + int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); + struct trans_pgd_info info = { + .trans_alloc_page = kexec_page_alloc, + .trans_alloc_arg = kimage, + }; /* If in place, relocation is not used, only flush next kernel */ if (kimage->head & IND_DONE) { @@ -148,6 +169,14 @@ int machine_kexec_post_load(struct kimage *kimage) return 0; } + kimage->arch.el2_vectors = 0; + if (is_hyp_callable()) { + int rc = trans_pgd_copy_el2_vectors(&info, + &kimage->arch.el2_vectors); + if (rc) + return rc; + } + memcpy(reloc_code, arm64_relocate_new_kernel, arm64_relocate_new_kernel_size); kimage->arch.kern_reloc = __pa(reloc_code); @@ -200,6 +229,8 @@ void machine_kexec(struct kimage *kimage) restart(is_hyp_callable(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else { + if (is_hyp_callable()) + __hyp_set_vectors(kimage->arch.el2_vectors); cpu_soft_restart(kimage->arch.kern_reloc, virt_to_phys(kimage), 0, 0); } From patchwork Wed Mar 3 00:22:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 511C3C433DB for ; Wed, 3 Mar 2021 00:22:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DB31D64FA9 for ; Wed, 3 Mar 2021 00:22:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DB31D64FA9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0FA048D010C; Tue, 2 Mar 2021 19:22:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0456D8D0101; Tue, 2 Mar 2021 19:22:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E6BC8D010C; Tue, 2 Mar 2021 19:22:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0156.hostedemail.com [216.40.44.156]) by kanga.kvack.org (Postfix) with ESMTP id F2FDE8D0101 for ; Tue, 2 Mar 2021 19:22:49 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id AF79B181AF5D7 for ; Wed, 3 Mar 2021 00:22:49 +0000 (UTC) X-FDA: 77876662458.19.FD4FF07 Received: from mail-qt1-f178.google.com (mail-qt1-f178.google.com [209.85.160.178]) by imf20.hostedemail.com (Postfix) with ESMTP id 29F85139 for ; Wed, 3 Mar 2021 00:22:48 +0000 (UTC) Received: by mail-qt1-f178.google.com with SMTP id j3so578214qtj.12 for ; Tue, 02 Mar 2021 16:22:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ATHDsIlQRij0i8p3z1/oZchje9DfRT1VSEQgV22ghC4=; b=FBdyqZaIcDyDYLlxElrZkuvpRkRku/nV1NAZj9UA7Kl0fuBORRQply+OelppUmA5/X YdOr7jgb6xCxZMVvxMAcCH+UB76E2UqPr8+Unv7yNzOfN0g/mRQJU850K0bipeneaRdC 0WfXYfl0XdrewOTEEVkNkr4Lc/IGmFcluOa3uHtSXXXbq7gKZ5vwgyBe/7c9xGU1wiFK Q38hSdvjT0G2vbcJ1e+1tLkWB8lGsOtC1XnvGT+Bg/yC/a1whgpNW0yKjwt/kUY1FC93 zcHgH6K4aNPWKbx2D1PifjppE8zXUC4vlX60NN8LCSNtkLWhJOCrvpiJkGwJyeoZ6GOV CuTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ATHDsIlQRij0i8p3z1/oZchje9DfRT1VSEQgV22ghC4=; b=G+ze1tObGU9GqBVJpi5+RJ4vbQ5PMHKslQBJDZFgz1SkNeaJ2moWjfB0wpt4JKeFbe nUZE4bSUmnLHJLD259/oUgs6PE6l8JMcct/Nrf1nanglEjD8vt7kvRGxoYHfKgFxJiwV 8L8BhCbutzsnhSoJqHU/7+QnsrQD1D2wPd0rPnYnC/ylLCOuNharvP/M9UUjjAPziBKm zyWBQambG40oIWxTjuIx7aRgTnrHRqExIGGb6X412ZjI3GZkWxFNp3WGOvg65zFpvSPF hAQpap7+BQ8eTDMVkN9ltiGU/ZugjLs1GBIrcupHe6I2no4ZTqF7HweujdSr/1fQgl/M lGRw== X-Gm-Message-State: AOAM531DpcnpibC8D4VHy5GR3clWjGV/0JiwlLN3tSsmJdPR0bCkCm9c xS9tXpBFxx+iOu3ccEk/hxOFBg== X-Google-Smtp-Source: ABdhPJyyN8JxnvqauMS4QIYOtU4flOLUY/VfPIcIZzPJfu3L3JbZXHRoVShtyq5YtQyvU4C/7Kg4+A== X-Received: by 2002:ac8:5953:: with SMTP id 19mr20290245qtz.263.1614730968594; Tue, 02 Mar 2021 16:22:48 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:48 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 12/17] arm64: kexec: relocate in EL1 mode Date: Tue, 2 Mar 2021 19:22:25 -0500 Message-Id: <20210303002230.1083176-13-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: ab51hg5fj36pypatpxiqqwbafcx5i8n4 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 29F85139 Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf20; identity=mailfrom; envelope-from=""; helo=mail-qt1-f178.google.com; client-ip=209.85.160.178 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730968-376769 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since we are going to keep MMU enabled during relocation, we need to keep EL1 mode throughout the relocation. Keep EL1 enabled, and switch EL2 only before enterying the new world. Suggested-by: James Morse Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/cpu-reset.h | 3 +-- arch/arm64/kernel/machine_kexec.c | 4 ++-- arch/arm64/kernel/relocate_kernel.S | 13 +++++++++++-- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h index 1922e7a690f8..f6d95512fec6 100644 --- a/arch/arm64/kernel/cpu-reset.h +++ b/arch/arm64/kernel/cpu-reset.h @@ -20,11 +20,10 @@ static inline void __noreturn cpu_soft_restart(unsigned long entry, { typeof(__cpu_soft_restart) *restart; - unsigned long el2_switch = is_hyp_callable(); restart = (void *)__pa_symbol(__cpu_soft_restart); cpu_install_idmap(); - restart(el2_switch, entry, arg0, arg1, arg2); + restart(0, entry, arg0, arg1, arg2); unreachable(); } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index fb03b6676fb9..d5940b7889f8 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -231,8 +231,8 @@ void machine_kexec(struct kimage *kimage) } else { if (is_hyp_callable()) __hyp_set_vectors(kimage->arch.el2_vectors); - cpu_soft_restart(kimage->arch.kern_reloc, virt_to_phys(kimage), - 0, 0); + cpu_soft_restart(kimage->arch.kern_reloc, + virt_to_phys(kimage), 0, 0); } BUG(); /* Should never get here. */ diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 36b4496524c3..df023b82544b 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -13,6 +13,7 @@ #include #include #include +#include /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. @@ -61,12 +62,20 @@ SYM_CODE_START(arm64_relocate_new_kernel) isb /* Start new image. */ + ldr x1, [x0, #KIMAGE_ARCH_EL2_VECTORS] /* relocation start */ + cbz x1, .Lel1 + ldr x1, [x0, #KIMAGE_START] /* relocation start */ + ldr x2, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ + mov x3, xzr + mov x4, xzr + mov x0, #HVC_SOFT_RESTART + hvc #0 /* Jumps from el2 */ +.Lel1: ldr x4, [x0, #KIMAGE_START] /* relocation start */ ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ - mov x1, xzr mov x2, xzr mov x3, xzr - br x4 + br x4 /* Jumps from el1 */ SYM_CODE_END(arm64_relocate_new_kernel) .align 3 /* To keep the 64-bit values below naturally aligned. */ From patchwork Wed Mar 3 00:22:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB739C433E0 for ; Wed, 3 Mar 2021 00:23:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 900E664F48 for ; Wed, 3 Mar 2021 00:23:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 900E664F48 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AB2DA8D0101; Tue, 2 Mar 2021 19:22:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DE978D010D; Tue, 2 Mar 2021 19:22:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 680ED8D0101; Tue, 2 Mar 2021 19:22:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0127.hostedemail.com [216.40.44.127]) by kanga.kvack.org (Postfix) with ESMTP id 2C6DB8D010D for ; Tue, 2 Mar 2021 19:22:51 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id E26048249980 for ; Wed, 3 Mar 2021 00:22:50 +0000 (UTC) X-FDA: 77876662500.05.B491051 Received: from mail-qv1-f51.google.com (mail-qv1-f51.google.com [209.85.219.51]) by imf02.hostedemail.com (Postfix) with ESMTP id AA9AA407F8FE for ; Wed, 3 Mar 2021 00:22:48 +0000 (UTC) Received: by mail-qv1-f51.google.com with SMTP id cw15so10120681qvb.11 for ; Tue, 02 Mar 2021 16:22:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=IBZj7dYh8lQRd+RkarkK6Bs3S98VDCxIDzsjaOHMyWU=; b=bKq3DOfy+ZZ2Br+ZXPwnr++JLsk2BOWy0QyrfsLY6aZS5OS4WSDGnNHfH8Icr0Z8I+ HOk/5OYLaspLekY1uItNjP9HFswk0cGO5JyOgS4Xbrfa9H7cSB4LdBlvUxuJhK4H+cib ru04FyGfJRo+sC3ysELsdQ2o9h1TMC4Jld9NJUUtDG6UjhaCyL1ezmZMZh5s6WU0PGc4 8bQe9pI4CW2cns5OQYOpxkOSPbubGDDyxQeJyQ9wZ3fHz7qOj0xd3HFKiaLqOqqwWAfP tbh9FZQ+wJC+teNKGy/f19ohcQicVlGPjTjHkfLWWPH8aSxnj73rrhZgLmKrV3EOd8Z5 f6Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IBZj7dYh8lQRd+RkarkK6Bs3S98VDCxIDzsjaOHMyWU=; b=gGLnDzzujEZI2UAzfOgottbc70hE4nnEgloTIGZn6Yq8DaZb4/aWeKoM6XtqAuI/63 ivfoPWw+MXisyDKBGNmf0C3p3bQzFRfNeAaRK1/xGz1brHhCt5y7hbdQYQwkw+qwV6eG YbRLSLxNqvG5crLOirlWag0jJEEycc5GkuA3JmMWV0Jh/pWyYVUERS3OpXb3PEp5e9qj QBqdL9KlPZ91Wo2WhIV/fzvNqsUNWL6knswL0MJlShYL8KwucfMMO6kXWU1Xp27nlklj ZAsGbIXsVVVEDydnNmAPc0bF4XbjbCZ75ojDojCMoP2SPjCOx9OaWm2gH9ibEB1me4iu w+9A== X-Gm-Message-State: AOAM530uJX0hzYDyxVpPsn0gZpilND6vIe79p/VSaXnnpA8SlDEa3CGw REHi/1btqzYxIIES30ZYMwXYXw== X-Google-Smtp-Source: ABdhPJxvNv+/84aSaILV/bO4Xb7Cban1AhQ63vws8517ykCszMsFD3aYB5zfOTZP/wFR41IfZGWghg== X-Received: by 2002:a0c:8f1a:: with SMTP id z26mr350314qvd.51.1614730969894; Tue, 02 Mar 2021 16:22:49 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:49 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 13/17] arm64: kexec: use ld script for relocation function Date: Tue, 2 Mar 2021 19:22:26 -0500 Message-Id: <20210303002230.1083176-14-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: AA9AA407F8FE X-Stat-Signature: o1jjzaezbrpf1ojmt7jc636b9ie76e58 Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=mail-qv1-f51.google.com; client-ip=209.85.219.51 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730968-517882 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, relocation code declares start and end variables which are used to compute its size. The better way to do this is to use ld script incited, and put relocation function in its own section. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/machine_kexec.c | 14 ++++++-------- arch/arm64/kernel/relocate_kernel.S | 15 ++------------- arch/arm64/kernel/vmlinux.lds.S | 19 +++++++++++++++++++ 4 files changed, 28 insertions(+), 21 deletions(-) diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index 2f36b16a5b5d..31e459af89f6 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -20,5 +20,6 @@ extern char __exittext_begin[], __exittext_end[]; extern char __irqentry_text_start[], __irqentry_text_end[]; extern char __mmuoff_data_start[], __mmuoff_data_end[]; extern char __entry_tramp_text_start[], __entry_tramp_text_end[]; +extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[]; #endif /* __ASM_SECTIONS_H */ diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index d5940b7889f8..f1451d807708 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -20,14 +20,11 @@ #include #include #include +#include #include #include "cpu-reset.h" -/* Global variables for the arm64_relocate_new_kernel routine. */ -extern const unsigned char arm64_relocate_new_kernel[]; -extern const unsigned long arm64_relocate_new_kernel_size; - /** * kexec_image_info - For debugging output. */ @@ -157,6 +154,7 @@ static void *kexec_page_alloc(void *arg) int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); + long reloc_size; struct trans_pgd_info info = { .trans_alloc_page = kexec_page_alloc, .trans_alloc_arg = kimage, @@ -177,14 +175,14 @@ int machine_kexec_post_load(struct kimage *kimage) return rc; } - memcpy(reloc_code, arm64_relocate_new_kernel, - arm64_relocate_new_kernel_size); + reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; + memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code); /* Flush the reloc_code in preparation for its execution. */ - __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size); + __flush_dcache_area(reloc_code, reloc_size); flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code + - arm64_relocate_new_kernel_size); + reloc_size); kexec_list_flush(kimage); kexec_image_info(kimage); diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index df023b82544b..7a600ba33ae1 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -15,6 +15,7 @@ #include #include +.pushsection ".kexec_relocate.text", "ax" /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. * @@ -77,16 +78,4 @@ SYM_CODE_START(arm64_relocate_new_kernel) mov x3, xzr br x4 /* Jumps from el1 */ SYM_CODE_END(arm64_relocate_new_kernel) - -.align 3 /* To keep the 64-bit values below naturally aligned. */ - -.Lcopy_end: -.org KEXEC_CONTROL_PAGE_SIZE - -/* - * arm64_relocate_new_kernel_size - Number of bytes to copy to the - * control_code_page. - */ -.globl arm64_relocate_new_kernel_size -arm64_relocate_new_kernel_size: - .quad .Lcopy_end - arm64_relocate_new_kernel +.popsection diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 7eea7888bb02..0d9d5e6af66f 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -92,6 +93,16 @@ jiffies = jiffies_64; #define HIBERNATE_TEXT #endif +#ifdef CONFIG_KEXEC_CORE +#define KEXEC_TEXT \ + . = ALIGN(SZ_4K); \ + __relocate_new_kernel_start = .; \ + *(.kexec_relocate.text) \ + __relocate_new_kernel_end = .; +#else +#define KEXEC_TEXT +#endif + #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 #define TRAMP_TEXT \ . = ALIGN(PAGE_SIZE); \ @@ -152,6 +163,7 @@ SECTIONS HYPERVISOR_TEXT IDMAP_TEXT HIBERNATE_TEXT + KEXEC_TEXT TRAMP_TEXT *(.fixup) *(.gnu.warning) @@ -336,3 +348,10 @@ ASSERT(swapper_pg_dir - reserved_pg_dir == RESERVED_SWAPPER_OFFSET, ASSERT(swapper_pg_dir - tramp_pg_dir == TRAMP_SWAPPER_OFFSET, "TRAMP_SWAPPER_OFFSET is wrong!") #endif + +#ifdef CONFIG_KEXEC_CORE +/* kexec relocation code should fit into one KEXEC_CONTROL_PAGE_SIZE */ +ASSERT(__relocate_new_kernel_end - (__relocate_new_kernel_start & ~(SZ_4K - 1)) + <= SZ_4K, "kexec relocation code is too big or misaligned") +ASSERT(KEXEC_CONTROL_PAGE_SIZE >= SZ_4K, "KEXEC_CONTROL_PAGE_SIZE is brokern") +#endif From patchwork Wed Mar 3 00:22:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BFD5C433DB for ; Wed, 3 Mar 2021 00:23:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 942F064E58 for ; Wed, 3 Mar 2021 00:23:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 942F064E58 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8FEC68D010E; Tue, 2 Mar 2021 19:22:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 822F78D010D; Tue, 2 Mar 2021 19:22:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50FBB8D010E; Tue, 2 Mar 2021 19:22:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id 950CE8D010D for ; Tue, 2 Mar 2021 19:22:52 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 51E0718144F6A for ; Wed, 3 Mar 2021 00:22:52 +0000 (UTC) X-FDA: 77876662584.09.C93291F Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf18.hostedemail.com (Postfix) with ESMTP id 705DA200038F for ; Wed, 3 Mar 2021 00:22:51 +0000 (UTC) Received: by mail-qt1-f175.google.com with SMTP id 18so13634403qty.3 for ; Tue, 02 Mar 2021 16:22:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Olm+fYXYbuVgp6bWsix/kMAWKOrqJlCkAHr9xCmjIpk=; b=K8kaTNPYcS3zquvbMQ0dtlLepeH8HKg/xCkwC1qj6nsKzbvTfQ63+Ks/RZEV3bUkmE y+QkmQaX6vcgEtn1YnwU7oJSM7haYdmSxw2yqC1Qf3YNDgEFTj1vUr6PvuQRH/KJFtHo JD59CklExk570Vv9Pr1/M82SYFlrf93ssmow3dcobqItrZ9QMnpounC4Sh2DTaIcy3oc CvvomkmEgsTKwkBVp9BC0SmnLqI5CuktCtdt20kzsgz1kO8YMSQGsgMHC9E+2iXhK6As rp6QpILmBre+MfVCQrotSwc9ZOttqOwTTzIP/JkW/2P0pHt12j8jhRTB6iZgS2quUD27 /56w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Olm+fYXYbuVgp6bWsix/kMAWKOrqJlCkAHr9xCmjIpk=; b=pA3Q0RBdiGLbTf1Jaq+DkDivhiaTMhkDYEahFQzw3lxIJZyDZU5MCUCe7b7aGTTVBn qMIxAVUPX7G/mRlVECbQrbFkKvHVCqQDp30hl9Ql5dJsGU3N9TpQkkfgjjrb2BbN+5Kv +B3wFuM2fEEPGNtkcylPqYqQCh52RgsZQlH0d8b8OVPrrDRAw+xx+abcrNLFuIIhL0ia mp8OSaqAM3n4xbPLEiE7psuFqj5uoFqx7iINhowRVRomTiGgChyDgL/o733tLBEDodM7 O/ttla0ueUJR85/+t3VodCBsTWOXlVFOPQdUhiAvCtnECouzGsLnOH5pMwfPX1pc3jlh zQng== X-Gm-Message-State: AOAM533VKQGXQjtBcii3hjXIoBQFvUStthNa8hJ/GmbPTLEcQufgPKSu Z3orx286RMEGFPmiIRR4VRQvUQ== X-Google-Smtp-Source: ABdhPJzanxLZpfCCAQZtVgjL5PevxfgLuq3u4oQDxWDQ5ASJZdnQkNQpbYmMikL7M6jXu/HxEyd+hA== X-Received: by 2002:ac8:75d4:: with SMTP id z20mr20660664qtq.61.1614730971180; Tue, 02 Mar 2021 16:22:51 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:50 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 14/17] arm64: kexec: install a copy of the linear-map Date: Tue, 2 Mar 2021 19:22:27 -0500 Message-Id: <20210303002230.1083176-15-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 705DA200038F X-Stat-Signature: jmbwue9y8oq5tb6d1f6st943bbruzgiy Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf18; identity=mailfrom; envelope-from=""; helo=mail-qt1-f175.google.com; client-ip=209.85.160.175 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730971-29455 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To perform the kexec relocations with the MMU enabled, we need a copy of the linear map. Create one, and install it from the relocation code. This has to be done from the assembly code as it will be idmapped with TTBR0. The kernel runs in TTRB1, so can't use the break-before-make sequence on the mapping it is executing from. The makes no difference yet as the relocation code runs with the MMU disabled. Co-developed-by: James Morse Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/assembler.h | 19 +++++++++++++++++++ arch/arm64/include/asm/kexec.h | 2 ++ arch/arm64/kernel/asm-offsets.c | 2 ++ arch/arm64/kernel/hibernate-asm.S | 20 -------------------- arch/arm64/kernel/machine_kexec.c | 16 ++++++++++++++-- arch/arm64/kernel/relocate_kernel.S | 3 +++ 6 files changed, 40 insertions(+), 22 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 29061b76aab6..3ce8131ad660 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -425,6 +425,25 @@ USER(\label, ic ivau, \tmp2) // invalidate I line PoU isb .endm +/* + * To prevent the possibility of old and new partial table walks being visible + * in the tlb, switch the ttbr to a zero page when we invalidate the old + * records. D4.7.1 'General TLB maintenance requirements' in ARM DDI 0487A.i + * Even switching to our copied tables will cause a changed output address at + * each stage of the walk. + */ + .macro break_before_make_ttbr_switch zero_page, page_table, tmp, tmp2 + phys_to_ttbr \tmp, \zero_page + msr ttbr1_el1, \tmp + isb + tlbi vmalle1 + dsb nsh + phys_to_ttbr \tmp, \page_table + offset_ttbr1 \tmp, \tmp2 + msr ttbr1_el1, \tmp + isb + .endm + /* * reset_pmuserenr_el0 - reset PMUSERENR_EL0 if PMUv3 present */ diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 305cf0840ed3..59ac166daf53 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -97,6 +97,8 @@ struct kimage_arch { phys_addr_t dtb_mem; phys_addr_t kern_reloc; phys_addr_t el2_vectors; + phys_addr_t ttbr1; + phys_addr_t zero_page; /* Core ELF header buffer */ void *elf_headers; unsigned long elf_headers_mem; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 2e3278df1fc3..609362b5aa76 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -158,6 +158,8 @@ int main(void) #ifdef CONFIG_KEXEC_CORE DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); DEFINE(KIMAGE_ARCH_EL2_VECTORS, offsetof(struct kimage, arch.el2_vectors)); + DEFINE(KIMAGE_ARCH_ZERO_PAGE, offsetof(struct kimage, arch.zero_page)); + DEFINE(KIMAGE_ARCH_TTBR1, offsetof(struct kimage, arch.ttbr1)); DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); DEFINE(KIMAGE_START, offsetof(struct kimage, start)); BLANK(); diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S index 8ccca660034e..a31e621ba867 100644 --- a/arch/arm64/kernel/hibernate-asm.S +++ b/arch/arm64/kernel/hibernate-asm.S @@ -15,26 +15,6 @@ #include #include -/* - * To prevent the possibility of old and new partial table walks being visible - * in the tlb, switch the ttbr to a zero page when we invalidate the old - * records. D4.7.1 'General TLB maintenance requirements' in ARM DDI 0487A.i - * Even switching to our copied tables will cause a changed output address at - * each stage of the walk. - */ -.macro break_before_make_ttbr_switch zero_page, page_table, tmp, tmp2 - phys_to_ttbr \tmp, \zero_page - msr ttbr1_el1, \tmp - isb - tlbi vmalle1 - dsb nsh - phys_to_ttbr \tmp, \page_table - offset_ttbr1 \tmp, \tmp2 - msr ttbr1_el1, \tmp - isb -.endm - - /* * Resume from hibernate * diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index f1451d807708..c875ef522e53 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -153,6 +153,8 @@ static void *kexec_page_alloc(void *arg) int machine_kexec_post_load(struct kimage *kimage) { + int rc; + pgd_t *trans_pgd; void *reloc_code = page_to_virt(kimage->control_code_page); long reloc_size; struct trans_pgd_info info = { @@ -169,12 +171,22 @@ int machine_kexec_post_load(struct kimage *kimage) kimage->arch.el2_vectors = 0; if (is_hyp_callable()) { - int rc = trans_pgd_copy_el2_vectors(&info, - &kimage->arch.el2_vectors); + rc = trans_pgd_copy_el2_vectors(&info, + &kimage->arch.el2_vectors); if (rc) return rc; } + /* Create a copy of the linear map */ + trans_pgd = kexec_page_alloc(kimage); + if (!trans_pgd) + return -ENOMEM; + rc = trans_pgd_create_copy(&info, &trans_pgd, PAGE_OFFSET, PAGE_END); + if (rc) + return rc; + kimage->arch.ttbr1 = __pa(trans_pgd); + kimage->arch.zero_page = __pa(empty_zero_page); + reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code); diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 7a600ba33ae1..e83b6380907d 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -29,10 +29,13 @@ */ SYM_CODE_START(arm64_relocate_new_kernel) /* Setup the list loop variables. */ + ldr x18, [x0, #KIMAGE_ARCH_ZERO_PAGE] /* x18 = zero page for BBM */ + ldr x17, [x0, #KIMAGE_ARCH_TTBR1] /* x17 = linear map copy */ ldr x16, [x0, #KIMAGE_HEAD] /* x16 = kimage_head */ mov x14, xzr /* x14 = entry ptr */ mov x13, xzr /* x13 = copy dest */ raw_dcache_line_size x15, x1 /* x15 = dcache line size */ + break_before_make_ttbr_switch x18, x17, x1, x2 /* set linear map */ .Lloop: and x12, x16, PAGE_MASK /* x12 = addr */ From patchwork Wed Mar 3 00:22:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5199C433E0 for ; Wed, 3 Mar 2021 00:23:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 85A8264E58 for ; Wed, 3 Mar 2021 00:23:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 85A8264E58 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 69EFE8D010F; Tue, 2 Mar 2021 19:22:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 651D88D010D; Tue, 2 Mar 2021 19:22:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 378228D010F; Tue, 2 Mar 2021 19:22:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0150.hostedemail.com [216.40.44.150]) by kanga.kvack.org (Postfix) with ESMTP id DF6208D010D for ; Tue, 2 Mar 2021 19:22:53 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A26346C2D for ; Wed, 3 Mar 2021 00:22:53 +0000 (UTC) X-FDA: 77876662626.24.39FAD00 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf25.hostedemail.com (Postfix) with ESMTP id 2BA946000101 for ; Wed, 3 Mar 2021 00:22:51 +0000 (UTC) Received: by mail-qk1-f179.google.com with SMTP id a9so7782328qkn.13 for ; Tue, 02 Mar 2021 16:22:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=UKjtjEDCRzQ4WDzfqfuqgp2BgU7fhP5QVy8I5s6gkYg=; b=fSLQuV82Jj2QUma3YmUBsnpeEsIRTLafxYwB28IFsK6PJ2gWAr7BlPSOvdbgCDrY0V 7HXoOxavXo2LYem8r3aWvfCujfpeTOoQ23KaKeI7QTwWr+XBRtc9A+OLPcoKhyscI06N uIGWARRpGBQZaGjaHiw/VVaw/7yLctzYfwfMwMbH/polx4wQEMsB/johD9Y3JLxmUZVz aZA24DlpuLA+BHJcNwtQl10b5bntN1CLS/hFgv6E0o7lLCbLeeoW1elkT8BymHot13xH 0zF/IAF12YxFrhHx+fREOYaZzOCleNBzrBSYZsJZm6LrwYV6MWmBlqfc8CNjPuVFotY8 NcIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UKjtjEDCRzQ4WDzfqfuqgp2BgU7fhP5QVy8I5s6gkYg=; b=Pw4caQfhCfkUdA+xnwkYc+JnRrOhHDDyX8mLrxylCwtIr4BdUaTRmVTr67++Z0Hv2E HrW72+TWq0Oa+QJORdaG/3vC2GLnQCksGfHznk1DqGUkOVBESk7ehGxYoLHSOLORJyFK 7XBfIi0CkBbJZuvaxY8QQOWoi+Xd1KOlVDsd1AzpJ4pUfAcQb1fhAxSSm6YwLg2QPZUp 5BoUOkqpidQVteSIKJWt6wYzY6Jdw8eOea1lV3gBQqSVrhtgejnnBm7XUbEZurya1rJp 8Ue/XF6L+Cztj9hwKCHQ0e2MUSRD997xCSxhNy22iy+QjtG9omxyCczGhEU9mB6Pknny Eq2g== X-Gm-Message-State: AOAM533jtC2Vl5SPpifkR3NPxMWkz/HkmhX91nTjZf3hm4CU7oZDcLKM +S6c+hbionpUqKVK/1EfU3s06/BRO9cTHg== X-Google-Smtp-Source: ABdhPJyKPWdMLDIUss61XZukaIXsHhimeP40sTJYFttLXHFVC8t+0q7qOi4TzBxxIkgsER4FQ5kv4g== X-Received: by 2002:a37:4743:: with SMTP id u64mr16004437qka.350.1614730972452; Tue, 02 Mar 2021 16:22:52 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:52 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 15/17] arm64: kexec: keep MMU enabled during kexec relocation Date: Tue, 2 Mar 2021 19:22:28 -0500 Message-Id: <20210303002230.1083176-16-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: d1ygt1zsy6i83nihxzy7gee38j46d5zx X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 2BA946000101 Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=mail-qk1-f179.google.com; client-ip=209.85.222.179 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730971-269202 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now, that we have linear map page tables configured, keep MMU enabled to allow faster relocation of segments to final destination. The performance data: for a moderate size kernel + initramfs: 25M the relocation was taking 0.382s, with enabled MMU it now takes 0.019s only or x20 improvement. The time is proportional to the size of relocation, therefore if initramfs is larger, 100M it could take over a second. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 3 +++ arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/machine_kexec.c | 16 ++++++++++---- arch/arm64/kernel/relocate_kernel.S | 33 +++++++++++++++++++---------- 4 files changed, 38 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 59ac166daf53..5fc87b51f8a9 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -97,8 +97,11 @@ struct kimage_arch { phys_addr_t dtb_mem; phys_addr_t kern_reloc; phys_addr_t el2_vectors; + phys_addr_t ttbr0; phys_addr_t ttbr1; phys_addr_t zero_page; + unsigned long phys_offset; + unsigned long t0sz; /* Core ELF header buffer */ void *elf_headers; unsigned long elf_headers_mem; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 609362b5aa76..ec7bb80aedc8 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -159,6 +159,7 @@ int main(void) DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); DEFINE(KIMAGE_ARCH_EL2_VECTORS, offsetof(struct kimage, arch.el2_vectors)); DEFINE(KIMAGE_ARCH_ZERO_PAGE, offsetof(struct kimage, arch.zero_page)); + DEFINE(KIMAGE_ARCH_PHYS_OFFSET, offsetof(struct kimage, arch.phys_offset)); DEFINE(KIMAGE_ARCH_TTBR1, offsetof(struct kimage, arch.ttbr1)); DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); DEFINE(KIMAGE_START, offsetof(struct kimage, start)); diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index c875ef522e53..d5c8aefc66f3 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -190,6 +190,11 @@ int machine_kexec_post_load(struct kimage *kimage) reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code); + rc = trans_pgd_idmap_page(&info, &kimage->arch.ttbr0, + &kimage->arch.t0sz, reloc_code); + if (rc) + return rc; + kimage->arch.phys_offset = virt_to_phys(kimage) - (long)kimage; /* Flush the reloc_code in preparation for its execution. */ __flush_dcache_area(reloc_code, reloc_size); @@ -223,9 +228,9 @@ void machine_kexec(struct kimage *kimage) local_daif_mask(); /* - * Both restart and cpu_soft_restart will shutdown the MMU, disable data + * Both restart and kernel_reloc will shutdown the MMU, disable data * caches. However, restart will start new kernel or purgatory directly, - * cpu_soft_restart will transfer control to arm64_relocate_new_kernel + * kernel_reloc contains the body of arm64_relocate_new_kernel * In kexec case, kimage->start points to purgatory assuming that * kernel entry and dtb address are embedded in purgatory by * userspace (kexec-tools). @@ -239,10 +244,13 @@ void machine_kexec(struct kimage *kimage) restart(is_hyp_callable(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else { + void (*kernel_reloc)(struct kimage *kimage); + if (is_hyp_callable()) __hyp_set_vectors(kimage->arch.el2_vectors); - cpu_soft_restart(kimage->arch.kern_reloc, - virt_to_phys(kimage), 0, 0); + cpu_install_ttbr0(kimage->arch.ttbr0, kimage->arch.t0sz); + kernel_reloc = (void *)kimage->arch.kern_reloc; + kernel_reloc(kimage); } BUG(); /* Should never get here. */ diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index e83b6380907d..8ac4b2d7f5e8 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -4,6 +4,8 @@ * * Copyright (C) Linaro. * Copyright (C) Huawei Futurewei Technologies. + * Copyright (C) 2020, Microsoft Corporation. + * Pavel Tatashin */ #include @@ -15,6 +17,15 @@ #include #include +.macro turn_off_mmu tmp1, tmp2 + mrs \tmp1, sctlr_el1 + mov_q \tmp2, SCTLR_ELx_FLAGS + bic \tmp1, \tmp1, \tmp2 + pre_disable_mmu_workaround + msr sctlr_el1, \tmp1 + isb +.endm + .pushsection ".kexec_relocate.text", "ax" /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. @@ -32,22 +43,21 @@ SYM_CODE_START(arm64_relocate_new_kernel) ldr x18, [x0, #KIMAGE_ARCH_ZERO_PAGE] /* x18 = zero page for BBM */ ldr x17, [x0, #KIMAGE_ARCH_TTBR1] /* x17 = linear map copy */ ldr x16, [x0, #KIMAGE_HEAD] /* x16 = kimage_head */ - mov x14, xzr /* x14 = entry ptr */ - mov x13, xzr /* x13 = copy dest */ + ldr x22, [x0, #KIMAGE_ARCH_PHYS_OFFSET] /* x22 phys_offset */ raw_dcache_line_size x15, x1 /* x15 = dcache line size */ break_before_make_ttbr_switch x18, x17, x1, x2 /* set linear map */ .Lloop: and x12, x16, PAGE_MASK /* x12 = addr */ - + sub x12, x12, x22 /* Convert x12 to virt */ /* Test the entry flags. */ .Ltest_source: tbz x16, IND_SOURCE_BIT, .Ltest_indirection /* Invalidate dest page to PoC. */ - mov x2, x13 - mov x1, #PAGE_SIZE - dcache_by_myline_op ivac, sy, x2, x1, x15, x20 + mov x19, x13 copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 + mov x1, #PAGE_SIZE + dcache_by_myline_op ivac, sy, x19, x1, x15, x20 b .Lnext .Ltest_indirection: tbz x16, IND_INDIRECTION_BIT, .Ltest_destination @@ -64,19 +74,20 @@ SYM_CODE_START(arm64_relocate_new_kernel) ic iallu dsb nsh isb + ldr x4, [x0, #KIMAGE_START] /* relocation start */ + ldr x1, [x0, #KIMAGE_ARCH_EL2_VECTORS] /* relocation start */ + ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ + turn_off_mmu x12, x13 /* Start new image. */ - ldr x1, [x0, #KIMAGE_ARCH_EL2_VECTORS] /* relocation start */ cbz x1, .Lel1 - ldr x1, [x0, #KIMAGE_START] /* relocation start */ - ldr x2, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ + mov x1, x4 /* relocation start */ + mov x2, x0 /* dtb address */ mov x3, xzr mov x4, xzr mov x0, #HVC_SOFT_RESTART hvc #0 /* Jumps from el2 */ .Lel1: - ldr x4, [x0, #KIMAGE_START] /* relocation start */ - ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ mov x2, xzr mov x3, xzr br x4 /* Jumps from el1 */ From patchwork Wed Mar 3 00:22:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16FD0C433DB for ; Wed, 3 Mar 2021 00:23:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AB5AB64E58 for ; Wed, 3 Mar 2021 00:23:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AB5AB64E58 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 789948D0110; Tue, 2 Mar 2021 19:22:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 715F28D010D; Tue, 2 Mar 2021 19:22:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A4118D0110; Tue, 2 Mar 2021 19:22:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0162.hostedemail.com [216.40.44.162]) by kanga.kvack.org (Postfix) with ESMTP id 0A1148D010D for ; Tue, 2 Mar 2021 19:22:55 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C88EA6D6E for ; Wed, 3 Mar 2021 00:22:54 +0000 (UTC) X-FDA: 77876662668.12.08B242E Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf06.hostedemail.com (Postfix) with ESMTP id 91ACDC0007C0 for ; Wed, 3 Mar 2021 00:22:53 +0000 (UTC) Received: by mail-qt1-f177.google.com with SMTP id 18so13634477qty.3 for ; Tue, 02 Mar 2021 16:22:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=mQmK3SsBkKt3YHt5HRugbQ4C32Fw5itH9aDNQf+nmJI=; b=cbwp13Q2ddpME4q7SnIYDmvYSLO9Ch0h6chGS0mJcuKYrVSiyW9RqugBaVOLREGq/3 RaXjQUi+1f2H7Dq7uhH7XbTZd4G8/Xc+rFwYUEEx/AdUSO30WgYG1LUfV2w1G6wCcjc2 /8GAhHNr0K4pH8SgFuD2VradryKueR66b6D+IX/xPN+SlF18R9pC4oMGtJaJMn5wE4eO 2KXKyC3D/rElN2+qfLl+X13RT9wKObgzzMT/qtKkvQmWMY9NClwSB/YVP+mMnf17tCpK BcdzbAKBOoD7Tz3sqa63lxj0/n8gza47BWFQQyLZKmLhuN8xvSXm97fDqDAG6jrXOUvD BH8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mQmK3SsBkKt3YHt5HRugbQ4C32Fw5itH9aDNQf+nmJI=; b=asa0FcqC1gIFlbwPEHHtooVQP0KPmMgshaNlnKNQUnXN4xMa2CUZXCPMbgu/VeGB5x xK6LBRgRv9JzF8koARYPCDpOWNMH0Df+3Ye9Uh8KkQedUA0tTnT4/gnG3WhrMSrxnEEi ZMoLGBS7zBBOu7J/x3TaueE+jkSO6bw6sB2/tE0nrxpeaFTSuF3ghn026nIMFxZymt9+ gdtS7xgy0l2bP7B/I2oszAJC2mMy9u4zNBeT8sJR0XHTq1qOjr+cI2jHI6eEoVH1Yq9s 8N8YXLSj0SDtOmT195za8TFw1zbwT6YrN78oOophfC7zXHDoc9uz4ug8Zh0W5CKj6LjM 0Tow== X-Gm-Message-State: AOAM531/qODQtNHmyKsu1WgCv2LIEvee9lBmLuV/8VlDgGGNlKJkiQgs mcJ6cs3AsOCIVYCT3/kZwrXrgg== X-Google-Smtp-Source: ABdhPJzDM5pLJqM5foQKA752jyI+AxEsX2Dhp+xecxEILxLFKkHZG6f5Ei6Ery303Y4YtrOKPjg5sg== X-Received: by 2002:ac8:72cf:: with SMTP id o15mr15747863qtp.118.1614730973753; Tue, 02 Mar 2021 16:22:53 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:53 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 16/17] arm64: kexec: remove the pre-kexec PoC maintenance Date: Tue, 2 Mar 2021 19:22:29 -0500 Message-Id: <20210303002230.1083176-17-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: oq7jzihesrghqq4rpra87ay63qbnzssg X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 91ACDC0007C0 Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=mail-qt1-f177.google.com; client-ip=209.85.160.177 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730973-764605 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that kexec does its relocations with the MMU enabled, we no longer need to clean the relocation data to the PoC. Co-developed-by: James Morse Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/machine_kexec.c | 40 ------------------------------- 1 file changed, 40 deletions(-) diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index d5c8aefc66f3..a1c9bee0cddd 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -76,45 +76,6 @@ int machine_kexec_prepare(struct kimage *kimage) return 0; } -/** - * kexec_list_flush - Helper to flush the kimage list and source pages to PoC. - */ -static void kexec_list_flush(struct kimage *kimage) -{ - kimage_entry_t *entry; - - __flush_dcache_area(kimage, sizeof(*kimage)); - - for (entry = &kimage->head; ; entry++) { - unsigned int flag; - void *addr; - - /* flush the list entries. */ - __flush_dcache_area(entry, sizeof(kimage_entry_t)); - - flag = *entry & IND_FLAGS; - if (flag == IND_DONE) - break; - - addr = phys_to_virt(*entry & PAGE_MASK); - - switch (flag) { - case IND_INDIRECTION: - /* Set entry point just before the new list page. */ - entry = (kimage_entry_t *)addr - 1; - break; - case IND_SOURCE: - /* flush the source pages. */ - __flush_dcache_area(addr, PAGE_SIZE); - break; - case IND_DESTINATION: - break; - default: - BUG(); - } - } -} - /** * kexec_segment_flush - Helper to flush the kimage segments to PoC. */ @@ -200,7 +161,6 @@ int machine_kexec_post_load(struct kimage *kimage) __flush_dcache_area(reloc_code, reloc_size); flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code + reloc_size); - kexec_list_flush(kimage); kexec_image_info(kimage); return 0; From patchwork Wed Mar 3 00:22:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12112443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00081C433DB for ; Wed, 3 Mar 2021 00:23:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 94F1264F25 for ; Wed, 3 Mar 2021 00:23:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 94F1264F25 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6872F8D0111; Tue, 2 Mar 2021 19:22:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C6178D010D; Tue, 2 Mar 2021 19:22:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3936D8D0111; Tue, 2 Mar 2021 19:22:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0250.hostedemail.com [216.40.44.250]) by kanga.kvack.org (Postfix) with ESMTP id 6DBF28D010D for ; Tue, 2 Mar 2021 19:22:56 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2E7F06C2D for ; Wed, 3 Mar 2021 00:22:56 +0000 (UTC) X-FDA: 77876662752.25.46E0F75 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf17.hostedemail.com (Postfix) with ESMTP id 8115240001DE for ; Wed, 3 Mar 2021 00:22:55 +0000 (UTC) Received: by mail-qt1-f176.google.com with SMTP id r24so16258372qtt.8 for ; Tue, 02 Mar 2021 16:22:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=xvI3SX7JlP3yoyw6A9a93CvHYXbBacBuW7cN/tDXEXM=; b=oPIjUKZfKiJwU7NlOxCWJ5OPY57CbXoshRc624GxpBnFgpup2P/nV58n0kK9/w2XTx brLP+1+u8lqb56VBhBgco7Y8J95WM0noHIV2xeYo4P7naJMOF8jyTmpN1bSO7ZzqBYkf wLnTE9IuEPsdQ9DhW53OsI8CGals1KWGCUuu9oUKLerRbT0Eefa0xtN6bNdQt8IR07uA 2nsXk3Z0vFKr1jZmw1N75kg8R6bKWnQATRIxJn4n7gJcwpld3SrUrVk/3xub3eHr43iP T9Uf5ZzJh8r4nx+VCew3/uiDHEi1lSTdW1/r3/oOd9UBCwABx/E9DzUpEJ1T7iG2PJER ybYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xvI3SX7JlP3yoyw6A9a93CvHYXbBacBuW7cN/tDXEXM=; b=mdq9wT6uoaV5Kq7avIxxbSujPZYbavapvtHB347uOyPhfkBE/2FWZu5Qod55aV9vsg /J54NRxX5cN6L0gS2RsnQh/VBkA8JDhw7MdQcrtcBJmql/kAZVsQ/MfIvg8WJhf8PshB lPIRUiW8Ius9wks+yo0GSHQaq3N4JpJr4zdKhyq4tbcqNXjuWPHgPyDtRgpcRKzw3yHg EHoHFIOJLNG/nDoEaNbooAfZrMRW0dyxwA6RM7UbKNkaM/cC4LKTXFpiCnLJS2aRRWL8 Nncx4h1Q2e8HUX63b6EWUar4lSeDzxwmkzSQWUaB5h3Brb3YVWln+h46pnRUNHQlWfSe x8mg== X-Gm-Message-State: AOAM532mC7o+x1dyLp5wRueXbPo4r5K3wvoemmSiGLqOLJxW4n9Ls461 Dtvs9OZnRqe/MhsyHxbaUt3WdA== X-Google-Smtp-Source: ABdhPJx5815Pblab/QccqNxqmgkFadcK+ZfVV2aeyidjFjH4QiHxtUtFGn8129DzTiPluIgZh9HpDw== X-Received: by 2002:ac8:64c:: with SMTP id e12mr20040140qth.103.1614730975050; Tue, 02 Mar 2021 16:22:55 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r3sm16690512qkm.129.2021.03.02.16.22.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 16:22:54 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v12 17/17] arm64: kexec: Remove cpu-reset.h Date: Tue, 2 Mar 2021 19:22:30 -0500 Message-Id: <20210303002230.1083176-18-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210303002230.1083176-1-pasha.tatashin@soleen.com> References: <20210303002230.1083176-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: awi6cfyw8wb7nhwukeg6mmsd17qzpcym X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 8115240001DE Received-SPF: none (soleen.com>: No applicable sender policy available) receiver=imf17; identity=mailfrom; envelope-from=""; helo=mail-qt1-f176.google.com; client-ip=209.85.160.176 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614730975-395043 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This header contains only cpu_soft_restart() which is never used directly anymore. So, remove this header, and rename the helper to be cpu_soft_restart(). Suggested-by: James Morse Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 6 ++++++ arch/arm64/kernel/cpu-reset.S | 7 +++---- arch/arm64/kernel/cpu-reset.h | 30 ------------------------------ arch/arm64/kernel/machine_kexec.c | 6 ++---- 4 files changed, 11 insertions(+), 38 deletions(-) delete mode 100644 arch/arm64/kernel/cpu-reset.h diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 5fc87b51f8a9..ee71ae3b93ed 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -90,6 +90,12 @@ static inline void crash_prepare_suspend(void) {} static inline void crash_post_resume(void) {} #endif +#if defined(CONFIG_KEXEC_CORE) +void cpu_soft_restart(unsigned long el2_switch, unsigned long entry, + unsigned long arg0, unsigned long arg1, + unsigned long arg2); +#endif + #define ARCH_HAS_KIMAGE_ARCH struct kimage_arch { diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S index 37721eb6f9a1..5d47d6c92634 100644 --- a/arch/arm64/kernel/cpu-reset.S +++ b/arch/arm64/kernel/cpu-reset.S @@ -16,8 +16,7 @@ .pushsection .idmap.text, "awx" /* - * __cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) - Helper for - * cpu_soft_restart. + * cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) * * @el2_switch: Flag to indicate a switch to EL2 is needed. * @entry: Location to jump to for soft reset. @@ -29,7 +28,7 @@ * branch to what would be the reset vector. It must be executed with the * flat identity mapping. */ -SYM_CODE_START(__cpu_soft_restart) +SYM_CODE_START(cpu_soft_restart) /* Clear sctlr_el1 flags. */ mrs x12, sctlr_el1 mov_q x13, SCTLR_ELx_FLAGS @@ -51,6 +50,6 @@ SYM_CODE_START(__cpu_soft_restart) mov x1, x3 // arg1 mov x2, x4 // arg2 br x8 -SYM_CODE_END(__cpu_soft_restart) +SYM_CODE_END(cpu_soft_restart) .popsection diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h deleted file mode 100644 index f6d95512fec6..000000000000 --- a/arch/arm64/kernel/cpu-reset.h +++ /dev/null @@ -1,30 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * CPU reset routines - * - * Copyright (C) 2015 Huawei Futurewei Technologies. - */ - -#ifndef _ARM64_CPU_RESET_H -#define _ARM64_CPU_RESET_H - -#include - -void __cpu_soft_restart(unsigned long el2_switch, unsigned long entry, - unsigned long arg0, unsigned long arg1, unsigned long arg2); - -static inline void __noreturn cpu_soft_restart(unsigned long entry, - unsigned long arg0, - unsigned long arg1, - unsigned long arg2) -{ - typeof(__cpu_soft_restart) *restart; - - restart = (void *)__pa_symbol(__cpu_soft_restart); - - cpu_install_idmap(); - restart(0, entry, arg0, arg1, arg2); - unreachable(); -} - -#endif diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index a1c9bee0cddd..ef7ba93f2bd6 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -23,8 +23,6 @@ #include #include -#include "cpu-reset.h" - /** * kexec_image_info - For debugging output. */ @@ -197,10 +195,10 @@ void machine_kexec(struct kimage *kimage) * In kexec_file case, the kernel starts directly without purgatory. */ if (kimage->head & IND_DONE) { - typeof(__cpu_soft_restart) *restart; + typeof(cpu_soft_restart) *restart; cpu_install_idmap(); - restart = (void *)__pa_symbol(__cpu_soft_restart); + restart = (void *)__pa_symbol(cpu_soft_restart); restart(is_hyp_callable(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else {