From patchwork Wed Dec 4 15:59:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11273261 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 06BE36C1 for ; Wed, 4 Dec 2019 16:08:06 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CE3EE2081B for ; Wed, 4 Dec 2019 16:08:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="qkW3dtdS"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="rDS4Lsw9"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=soleen.com header.i=@soleen.com header.b="gn5Q4wOC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CE3EE2081B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=eKj25lyRpmSMbUUcdQxHdlBgrbUcp4uw6u/L02KdFsI=; b=qkW3dtdSd1wh8j f91MIf/e/8rRE8PTSKf/dzLntLqZOD+0iz2hGrtOCyZs18HKjmbI2l02vzf8oocYw/eFmo+5AitDT 37j51c5JKfEqyQGUX2nRY3pPQeHd/pnHbUpLDfiTdTwbQysYjUOZca63vS8KkSMWICC2TJggoCsSi iAMvbB4ggLnSGeVWlUjaiGHssnvpFX1TJjhv+nTn27pMhK5o73o998FDNf8kpw9TjJ122O9MbB9ij WJP268p7yjKiT86TDaN2q4qkv1rKtMWgsyyRITP+6Q9UeLqsW/RnphnH63WatHyTS7/Q8Q/rLr2CD PpkgwGFbMWl9yB6Gp8KQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1icXC5-00008P-DH; Wed, 04 Dec 2019 16:08:01 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1icX4g-0000YG-86 for linux-arm-kernel@bombadil.infradead.org; Wed, 04 Dec 2019 16:00:22 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:To:From:Sender:Reply-To:Cc: Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=+hrNsEc0GRu30jUNwwgijvfvJA3Gw0vbGFinPeI+s7A=; b=rDS4Lsw9lOcfzoYVgT6V/CAZOc ModJDfsxer5uBzJB42CxdWLvDt/vkIsTNcqRy6EOmraC0O9tjBsOVU8PKg5fEFkoGV/DrebhIkrLA XN85TR8lKeFZaoJpiSMiZOu5lM9OcIZH2R8RC12+oOJAJUO9Qms4+cPFh9RER1PwM0gnlvOTRh+kW 4qi9pVgZUCN47N6KEkOAZ/fmnCR1c8t2gYLVgX8XZCOfDO0e+va0Tib7NdtUuiZdR0nRX4QTs4KwQ 0dFbgrbrCYWI4XJU3rrv6Wm51FUvAvr45SE/RiYzVskyGyUvj3AbRFMwweJ7GZSngGBVK7CqBfk9I 96493JAA==; Received: from mail-qk1-x742.google.com ([2607:f8b0:4864:20::742]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1icX4d-00028s-3p for linux-arm-kernel@lists.infradead.org; Wed, 04 Dec 2019 16:00:20 +0000 Received: by mail-qk1-x742.google.com with SMTP id i18so320872qkl.11 for ; Wed, 04 Dec 2019 08:00:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=+hrNsEc0GRu30jUNwwgijvfvJA3Gw0vbGFinPeI+s7A=; b=gn5Q4wOC/CWivBpaA0eyfip0aKM+qjW33B6QF5A/NHl5LG9ptLiIgrJx0VgtTY1YxR DtuNvXHvp/qSyUrI8MlkiGRI0p7IjuXYhZfma9Xp0MS9/kM9mBGgBx19r1zPiRO2Qsgh 1EubFBjJyJ79d/HDRqSHhulsugbf5yARPr1DF21T60p4zhlo5btmqa9oDvrM9dsQWayQ vZWr5mTUBYOIB5IZGQN8kRZeVY1OVboOKhkrvN9XAc97uj9L4ivZ+7eVjgQS1i18AhMG R0yI4e2vdElgCmvOc1VtDY52eRFf95njifE5hCv/JTuVpDtRQ0dvkeXoNLKIIhdIoqzr Hj6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+hrNsEc0GRu30jUNwwgijvfvJA3Gw0vbGFinPeI+s7A=; b=LpQiwT9usFIwbgGEovj6w7LN4oNjLA0bDse6vejtgsffNZQVg6S+zfhtsUUyZ1hEuD 6i8PIVXa2WqY6akAcZUbrc4vewVtqREKtvdBuw0LbcB6bXKSxDxOzxeq93NC26wlixeE o0nS2KFC9uMayMbO3im+6+mQyESMM0iLZRikbJRIsZYggVOhpr2tRQigPQsfSFrM4+x8 84PGjDxeDP5va35SZdeZr+Lto5p7QcRnryq8wUCTTgF7T4gW1JmAugDQBmZYrMo0/Pu9 j2RGFe+U/rAYDFlzyavfxicAvkvjbHQ3ln6aYKf4r8NXq4VYsWIPX7MMaFCnXnZvUotd eNOQ== X-Gm-Message-State: APjAAAUykMMURTFxj6+3EaraF6MKGYO4CQ7dc9WGqNJcW68Zq5SrtSl1 twULF2UdZKsNwlHuB1nIcjZmCQ== X-Google-Smtp-Source: APXvYqxGHmN/Bb5BgVqDSbpZGghOEE64QncuRmgF6H7L6Kzy+CxFQPsdvZXIYBmorhLxk5bGaaM22w== X-Received: by 2002:a37:a40d:: with SMTP id n13mr3811268qke.167.1575475216583; Wed, 04 Dec 2019 08:00:16 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id w21sm4177585qth.17.2019.12.04.08.00.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Dec 2019 08:00:15 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, bhsharma@redhat.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de Subject: [PATCH v8 24/25] arm64: kexec: enable MMU during kexec relocation Date: Wed, 4 Dec 2019 10:59:37 -0500 Message-Id: <20191204155938.2279686-25-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191204155938.2279686-1-pasha.tatashin@soleen.com> References: <20191204155938.2279686-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Spam-Note: CRM114 invocation failed X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.2 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:742 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Now, that we have transitional page tables configured, temporarily enable MMU to allow faster relocation of segments to final destination. The performance data: for a moderate size kernel + initramfs: 25M the relocation was taking 0.382s, with enabled MMU it now takes 0.019s only or x20 improvement. The time is proportional to the size of relocation, therefore if initramfs is larger, 100M it could take over a second. Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/relocate_kernel.S | 135 +++++++++++++++++----------- 1 file changed, 83 insertions(+), 52 deletions(-) diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 67efa42575a5..e623b02f72a3 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -4,6 +4,8 @@ * * Copyright (C) Linaro. * Copyright (C) Huawei Futurewei Technologies. + * Copyright (c) 2019, Microsoft Corporation. + * Pavel Tatashin */ #include @@ -15,6 +17,48 @@ #include GLOBAL(kexec_relocate_code_start) +/* Invalidae TLB */ +.macro tlb_invalidate + dsb sy + dsb ish + tlbi vmalle1 + dsb ish + isb +.endm + +/* Turn-off mmu at level specified by sctlr */ +.macro turn_off_mmu sctlr, tmp1, tmp2 + mrs \tmp1, \sctlr + ldr \tmp2, =SCTLR_ELx_FLAGS + bic \tmp1, \tmp1, \tmp2 + pre_disable_mmu_workaround + msr \sctlr, \tmp1 + isb +.endm + +/* Turn-on mmu at level specified by sctlr */ +.macro turn_on_mmu sctlr, tmp1, tmp2 + mrs \tmp1, \sctlr + ldr \tmp2, =SCTLR_ELx_FLAGS + orr \tmp1, \tmp1, \tmp2 + msr \sctlr, \tmp1 + ic iallu + dsb nsh + isb +.endm + +/* + * Set ttbr0 and ttbr1, called while MMU is disabled, so no need to temporarily + * set zero_page table. Invalidate TLB after new tables are set. + */ +.macro set_ttbr arg, tmp + ldr \tmp, [\arg, #KEXEC_KRELOC_TRANS_TTBR0] + msr ttbr0_el1, \tmp + ldr \tmp, [\arg, #KEXEC_KRELOC_TRANS_TTBR1] + offset_ttbr1 \tmp + msr ttbr1_el1, \tmp + isb +.endm /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. @@ -26,65 +70,52 @@ GLOBAL(kexec_relocate_code_start) * symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. The * machine_kexec() routine will copy arm64_relocate_new_kernel to the kexec * safe memory that has been set up to be preserved during the copy operation. + * + * This function temporarily enables MMU if kernel relocation is needed. + * Also, if we enter this function at EL2 on non-VHE kernel, we temporarily go + * to EL1 to enable MMU, and escalate back to EL2 at the end to do the jump to + * the new kernel. This is determined by presence of el2_vector. */ ENTRY(arm64_relocate_new_kernel) - /* Clear the sctlr_el2 flags. */ - mrs x2, CurrentEL - cmp x2, #CurrentEL_EL2 + mrs x1, CurrentEL + cmp x1, #CurrentEL_EL2 b.ne 1f - mrs x2, sctlr_el2 - ldr x1, =SCTLR_ELx_FLAGS - bic x2, x2, x1 - pre_disable_mmu_workaround - msr sctlr_el2, x2 - isb -1: /* Check if the new image needs relocation. */ - ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 = kimage_head */ - tbnz x16, IND_DONE_BIT, .Ldone - raw_dcache_line_size x15, x1 /* x15 = dcache line size */ -.Lloop: - and x12, x16, PAGE_MASK /* x12 = addr */ - /* Test the entry flags. */ -.Ltest_source: - tbz x16, IND_SOURCE_BIT, .Ltest_indirection - - /* Invalidate dest page to PoC. */ - mov x2, x13 - add x20, x2, #PAGE_SIZE - sub x1, x15, #1 - bic x2, x2, x1 -2: dc ivac, x2 - add x2, x2, x15 - cmp x2, x20 - b.lo 2b - dsb sy - - copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 - b .Lnext -.Ltest_indirection: - tbz x16, IND_INDIRECTION_BIT, .Ltest_destination - mov x14, x12 /* ptr = addr */ - b .Lnext -.Ltest_destination: - tbz x16, IND_DESTINATION_BIT, .Lnext - mov x13, x12 /* dest = addr */ -.Lnext: - ldr x16, [x14], #8 /* entry = *ptr++ */ - tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */ -.Ldone: - /* wait for writes from copy_page to finish */ - dsb nsh - ic iallu - dsb nsh - isb - - /* Start new image. */ - ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */ + turn_off_mmu sctlr_el2, x1, x2 /* Turn off MMU at EL2 */ +1: mov x20, xzr /* x20 will hold vector value */ + ldr x11, [x0, #KEXEC_KRELOC_COPY_LEN] + cbz x11, 5f /* Check if need to relocate */ + ldr x20, [x0, #KEXEC_KRELOC_EL2_VECTOR] + cbz x20, 2f /* need to reduce to EL1? */ + msr vbar_el2, x20 /* el2_vector present, means */ + adr x1, 2f /* we will do copy in el1 but */ + msr elr_el2, x1 /* do final jump from el2 */ + eret /* Reduce to EL1 */ +2: set_ttbr x0, x1 /* Set our page tables */ + tlb_invalidate + turn_on_mmu sctlr_el1, x1, x2 /* Turn MMU back on */ + ldr x1, [x0, #KEXEC_KRELOC_DST_ADDR]; + ldr x2, [x0, #KEXEC_KRELOC_SRC_ADDR]; + mov x12, x1 /* x12 dst backup */ +3: copy_page x1, x2, x3, x4, x5, x6, x7, x8, x9, x10 + sub x11, x11, #PAGE_SIZE + cbnz x11, 3b /* page copy loop */ + raw_dcache_line_size x2, x3 /* x2 = dcache line size */ + sub x3, x2, #1 /* x3 = dcache_size - 1 */ + bic x12, x12, x3 +4: dc cvau, x12 /* Flush D-cache */ + add x12, x12, x2 + cmp x12, x1 /* Compare to dst + len */ + b.ne 4b /* D-cache flush loop */ + turn_off_mmu sctlr_el1, x1, x2 /* Turn off MMU */ + tlb_invalidate /* Invalidate TLB */ +5: ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */ ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3] ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2] ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1] ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */ - br x4 + cbnz x20, 6f /* need to escalate to el2? */ + br x4 /* Jump to new world */ +6: hvc #0 /* enters kexec_el1_sync */ .ltorg END(arm64_relocate_new_kernel)