From patchwork Thu Dec 13 17:20:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10729239 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BD1BC16B1 for ; Thu, 13 Dec 2018 17:21:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9E1152C75D for ; Thu, 13 Dec 2018 17:21:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D0192C79B; Thu, 13 Dec 2018 17:21:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3BDBB2C772 for ; Thu, 13 Dec 2018 17:21:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yBNLzx16fY7KziV5DOq5v2c3wCWNBphg8FDuQ2hRHoQ=; b=gXrJTeb7KY5Lp5 V1doM0ScvvPb5L1v2Ay5LvGyzEUuzzUrYgYVUrAJPMqWyl82mCWkEd/VLYechZ2axXDrOFem2/Usb pkJnvRd9Gbkso3UZyou+nz30bWcBuDnapnZxTIVpvt9T1u9/bn+5JaQPYSs8ut6DmF6ASFWBjCpPH f1k58kwkHKgqqCAT/wthtnUksT/SmUJ2MLeZ2fAx76CD9ShS9w4q/NqqpLiZHXu0CFk48TILhMpyM eEK1sRru+L1mYvomgPgYZmCh403LX3LdolxPFkDWuhoTKHUTd9unsY3FOZl1C0gcr1NGxD1enY2Hr H/uMlE/TeC1Rw2ZhLKmg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gXUff-0007RB-Eg; Thu, 13 Dec 2018 17:21:11 +0000 Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gXUfN-0007BA-Ro for linux-arm-kernel@lists.infradead.org; Thu, 13 Dec 2018 17:20:55 +0000 Received: by mail-wr1-x444.google.com with SMTP id x10so2817914wrs.8 for ; Thu, 13 Dec 2018 09:20:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3kU5A9Z2aTiJS/REKXgB3s9e5vNfCn6o+UyWBf65Za0=; b=KVR5amhXEn2pScWPCPmR2LKrv82Z5stuocDVaxtFDjBZ4j4rLtWiuL1vq2w3Rp5aW4 fZYi4rxay+5/a2MiyxmJOYHndr45fGP6tAdGF1WEl906qUgyV3IPuTGMe7lxIxZsxlqJ xHlCli8ljdFUpT37uIgg7uQbpTASnuS3j/TfA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3kU5A9Z2aTiJS/REKXgB3s9e5vNfCn6o+UyWBf65Za0=; b=afABgHLxujzQlJuG0+a5T9V8XdFmI/nLm1QzIB9ACJ/kRZokRHLL1THR2byRV7ZPz7 eickGv6EfB6u3hP1Rk4+nrbgxOQb4IX8zFCFv0w0iXp8dMHGKtN0AR5jR9F1bi3lJ5HG JaUN6TkyM9iB2GmTMvug4EHWwCf98IHdPZcskIRmMgjKmVwX9yivPBRGHJ7zutGgM6Dv c5s4121Yqf5JLH6a1caM9X3xPIT7vkk5bIEcdhH7keJ5FoJFkGlhRvk6hiztUZnnmjs3 Y8sSFFdoAEdN4R/fNPoylcLgVsujSmjjXFPNHQeWj1tjaj0dviw8LuMUotdRolUkcIPe k98g== X-Gm-Message-State: AA+aEWYWZsGBfFD9HDvkCAdT5IAUE1ZU9RMcpNOEBEhAMCvX2ckDUYaX SYgP4owv/ABj/PutB/Oftx6MFJ/TgGEQgw== X-Google-Smtp-Source: AFSGD/W/6wXCdP0wLDN1fEjald4j+PoVOdiYZnBJr+t6Cm0033e42NSDO/3BP8D9zxpOiyDr2ZIeaQ== X-Received: by 2002:adf:e64d:: with SMTP id b13mr4570953wrn.276.1544721641942; Thu, 13 Dec 2018 09:20:41 -0800 (PST) Received: from localhost.localdomain (laubervilliers-657-1-83-120.w92-154.abo.wanadoo.fr. [92.154.90.120]) by smtp.gmail.com with ESMTPSA id q12sm2902753wrx.31.2018.12.13.09.20.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Dec 2018 09:20:41 -0800 (PST) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 1/4] arm64: kpti: enable KPTI only when KASLR is truly enabled. Date: Thu, 13 Dec 2018 18:20:32 +0100 Message-Id: <20181213172036.14504-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20181213172036.14504-1-ard.biesheuvel@linaro.org> References: <20181213172036.14504-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181213_092053_892514_497500D8 X-CRM114-Status: GOOD ( 15.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Robin Murphy , John Garry , Will Deacon , Suzuki K Poulose , Ard Biesheuvel Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Kernels built with CONFIG_RANDOMIZE_BASE=y may run with KASLR disabled when no RNG is provided by the firmware, or when it has been turned off explicitly by putting 'nokaslr' on the command line. In this case, there is no point in enabling KPTI on cores that have no need for it otherwise, so take kaslr_offset() into account here. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index aec5ecb85737..ef8118274ca9 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -937,7 +937,7 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, } /* Useful for KASLR robustness */ - if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE) && kaslr_offset() > 0) return true; /* Don't force KPTI for CPUs that are not vulnerable */ From patchwork Thu Dec 13 17:20:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10729241 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 748B291E for ; Thu, 13 Dec 2018 17:21:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 537882C6F5 for ; Thu, 13 Dec 2018 17:21:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 50AC12C7A3; Thu, 13 Dec 2018 17:21:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 0D55D2C788 for ; Thu, 13 Dec 2018 17:21:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=y13STmxZkJhf3f5iEFuUcbtni1m8wefZoNbR+3FGyLQ=; b=MgQVrpD8pcpLEv 6nNnr1JzvjWIGKk6SEoUHwF555TnSy0wIhcniYVXgS+3EyLQT1opfl3jBapo2XlF9w42kk7lW78T3 tKNSFCC6x5tEXUwtiJKCJcdzmPesjEKxpAASj9COPDrxL35FOsT6P4Ngdsb/t5v/Sg4gCLidAsfIK enWZT28VsBrbvJkMDQOCFiHYfR2GcSFfwMz7PrDb9aMWPVs+IUPeMfEr7P4I01kTRbJjYiy5aR+V/ 36QF6vi+ACho/L8EEn8b33gluHlbMUG4yKLJNk+s909I8sjD7srzgYOsNV1dUGF5TU0WOs1VCmlLs bAVl3hH6KGCBRWcRmDzA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gXUfs-0007e4-KZ; Thu, 13 Dec 2018 17:21:24 +0000 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gXUfP-0007BB-7s for linux-arm-kernel@lists.infradead.org; Thu, 13 Dec 2018 17:20:56 +0000 Received: by mail-wr1-x441.google.com with SMTP id l9so2799850wrt.13 for ; Thu, 13 Dec 2018 09:20:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YJvOwVwCfOfBo2c0GhHmvY07l9vz/ddNv4xdwbKx6M0=; b=juK9YiLqlqreBKNbwkw2HcjeBCwxen35tZ0BCLy2uvOwcvZ122l85lmeBF/Qcuxgtj OFtW6HArUVjSSj4l/yrE3NGs+QRWRI/hMBkId+MiPPy+c5LlCieYFTCr0ZRU1sBWxmcU lQwSRQiVVnQGchzPb32Fmvgh0cpkZR3ZjjrJA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YJvOwVwCfOfBo2c0GhHmvY07l9vz/ddNv4xdwbKx6M0=; b=mEzaI0aOaScdmO8nIsa/EDMp8G8pWVdD9YUDITcvOLSD2GZk43lSeW8df++Oc66vSW lWqropoMfSZRgYJHpPeLPqppiiM9i87Psm8mHyF2hjIHbTmeD2QNix0V0mYwE0p5ML9Y N8mmb7hhX1Bfg1yH2iRGJHfuaI3iDEMRjUF5vSqax5pp8qTEZfqCwtD9/3n05A3v2PA5 6zBAGuDbH2pSAdB1gp/3lovSNT//gliJxN/1EF2UVe6PbxuE+AXRpvx/L5gLTwH7cyV1 ELZ6HAGFbpSJZNmxAz0Jo/hSJ0/nvAyqOVxn6I61uJv1WI6U4NmzxrSORUkQV0xh/0xH KlCw== X-Gm-Message-State: AA+aEWZuEWobtVRln8z9/fh5A3bAfaTlIR3lOQPwT1BCqBBHDVX/TeTP LcX4IO4VE7kFyfticETasFBOD8OlaT4DTw== X-Google-Smtp-Source: AFSGD/XfYvj69DboSWkS6xBBxhVxKAGRWkb+0FgjmEUAVbBHzFt1uZdeXCztlVCj1hxL+EjHfMUlhQ== X-Received: by 2002:adf:8264:: with SMTP id 91mr21055689wrb.312.1544721643300; Thu, 13 Dec 2018 09:20:43 -0800 (PST) Received: from localhost.localdomain (laubervilliers-657-1-83-120.w92-154.abo.wanadoo.fr. [92.154.90.120]) by smtp.gmail.com with ESMTPSA id q12sm2902753wrx.31.2018.12.13.09.20.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Dec 2018 09:20:42 -0800 (PST) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 2/4] arm64: kpti: add helper to decide whether nG mappings should be used early Date: Thu, 13 Dec 2018 18:20:33 +0100 Message-Id: <20181213172036.14504-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20181213172036.14504-1-ard.biesheuvel@linaro.org> References: <20181213172036.14504-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181213_092055_279870_4A6B953A X-CRM114-Status: GOOD ( 18.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Robin Murphy , John Garry , Will Deacon , Suzuki K Poulose , Ard Biesheuvel Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP idmap_kpti_install_ng_mappings() traverses all kernel page tables with the caches off to replace global with non-global attributes, so that KPTI may be enabled safely. This is costly, and can be avoided in cases where we know we will be enabling KPTI regardless of whether any cores are present that are susceptible to Meltdown. So add a helper that tells us whether KPTI was force en/disabled, which we will help use decide whether to use nG mappings when creating the mappings of the kernel address space. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 7 ++++ arch/arm64/kernel/cpufeature.c | 37 ++++++++++++++------ 2 files changed, 34 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 7e2ec64aa414..91bcab94a725 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -558,6 +558,13 @@ static inline u32 id_aa64mmfr0_parange_to_phys_shift(int parange) default: return CONFIG_ARM64_PA_BITS; } } + +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +extern bool kpti_is_forced(bool *enabled); +#else +static bool kpti_is_forced(bool *enabled) { return false; } +#endif + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index ef8118274ca9..ecd8c65dd2d7 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -908,13 +908,11 @@ has_useable_cnp(const struct arm64_cpu_capabilities *entry, int scope) #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */ -static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, - int scope) +bool kpti_is_forced(bool *enabled) { - /* List of CPUs that are not vulnerable and don't need KPTI */ - static const struct midr_range kpti_safe_list[] = { - MIDR_ALL_VERSIONS(MIDR_CAVIUM_THUNDERX2), - MIDR_ALL_VERSIONS(MIDR_BRCM_VULCAN), + static const struct midr_range kpti_blacklist[] = { + MIDR_RANGE(MIDR_THUNDERX, 0, 0, 1, 1), + MIDR_RANGE(MIDR_THUNDERX_81XX, 0, 0, 0, 0), { /* sentinel */ } }; char const *str = "command line option"; @@ -924,8 +922,8 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, * ThunderX leads to apparent I-cache corruption of kernel text, which * ends as well as you might imagine. Don't even try. */ - if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_27456)) { - str = "ARM64_WORKAROUND_CAVIUM_27456"; + if (IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456) && + is_midr_in_range_list(read_cpuid_id(), kpti_blacklist)) { __kpti_forced = -1; } @@ -933,12 +931,31 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, if (__kpti_forced) { pr_info_once("kernel page table isolation forced %s by %s\n", __kpti_forced > 0 ? "ON" : "OFF", str); - return __kpti_forced > 0; + *enabled = __kpti_forced > 0; + return true; } /* Useful for KASLR robustness */ - if (IS_ENABLED(CONFIG_RANDOMIZE_BASE) && kaslr_offset() > 0) + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE) && kaslr_offset() > 0) { + *enabled = true; return true; + } + return false; +} + +static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, + int scope) +{ + /* List of CPUs that are not vulnerable and don't need KPTI */ + static const struct midr_range kpti_safe_list[] = { + MIDR_ALL_VERSIONS(MIDR_CAVIUM_THUNDERX2), + MIDR_ALL_VERSIONS(MIDR_BRCM_VULCAN), + { /* sentinel */ } + }; + bool enabled; + + if (kpti_is_forced(&enabled)) + return enabled; /* Don't force KPTI for CPUs that are not vulnerable */ if (is_midr_in_range_list(read_cpuid_id(), kpti_safe_list)) From patchwork Thu Dec 13 17:20:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10729243 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 961A891E for ; Thu, 13 Dec 2018 17:21:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 74E212C7AB for ; Thu, 13 Dec 2018 17:21:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 735282C7C0; Thu, 13 Dec 2018 17:21:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 11C5A2C7AB for ; Thu, 13 Dec 2018 17:21:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Nyvw5GeTbfP5ISiwcxpUPGUE7VNQ4ggWGa7CrmWNfXM=; b=R1sjdBFG6c1yJw ByzBjluuDr6FUsemASYZDrcaUaaivEfVntFs4D7iJo4636/5vtGjXfq8pUJn1ZwcJQ6560AGxyYgw AsC/gXbEq5f29cvgRCG3gf82J6UlmeRcYPFydVGvVTHRhL56rihU6ox+3ucM8lIx8lUORgnsjXwVR CHzHXMto2Hgy/N/BSAqTLZw7MhgRBz+snkJOrN2AAeInSwjYzwdtShOtVnjfrBurS2xSIYOCano5V VJMWyjS7HcQxD8hT1785goxiXfGmtftplUFZ5JnFjioKm6hA0E2bpXu4TEyM5kAbjYRTbtXPgs81Z YQLymjgV8UNgCfJz8rOg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gXUg5-0007so-Fx; Thu, 13 Dec 2018 17:21:37 +0000 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gXUfQ-0007BC-Bf for linux-arm-kernel@lists.infradead.org; Thu, 13 Dec 2018 17:21:00 +0000 Received: by mail-wr1-x443.google.com with SMTP id v13so2836476wrw.5 for ; Thu, 13 Dec 2018 09:20:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PfN4/9eI1NIZOQUGnM6jMBmFJSgj4lLHz8nP2g1zbqo=; b=EO8B54+mJRDGEX1JaupC3h2MbD0JMZNcOIQpnnKumX9w2nlzg0l61hHweqDWihCLsC qnmRGL9y79O5pPWlco7a9CmsMQy447e34vx8mUx13c9RJYWfJTwcgCK+VOJPHiYEBrh3 2n6gnNxo8opla3vaYcXt0vYLp1l/t9b0t/UPQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PfN4/9eI1NIZOQUGnM6jMBmFJSgj4lLHz8nP2g1zbqo=; b=fpfkQAyPSqMw/Tmr4S/58zoFMJ/ZlhelxKQIVlCWZ1oXkjE7b4I3gPF+mxyQxreNem k0JtpNalUgtda5rsozMUMtNGqKjF7Z9BRmvvhjuSmx48svB1e1gAfXnO+4ol+d4htEOa QkOrOYg9QGwZwiUc8cKTztwxf3EeNibfoQHEwhRyTcI7ocwgdRMwkuuKFnpNvSXsYCin H+6x8RUFJijCr0mOjE9/hFXFopH9UzptyAq5RYFcL7KkMLIvmb6GtNbryMwAwcwneoQB adQGsUPNKFTjLzR79KDtf3XIUDUDsrWX13wjYMZbgOFNooKb6iFGwlhYoYa/414kdySK HR+Q== X-Gm-Message-State: AA+aEWbWLl4cETMiZWxu4kiWbIPpt9ZtV7DtwsUhwSYzCZajckt9uH7h KEcHEFfhQ0ortlhoDiyGiO2Xs6fKkwBIDQ== X-Google-Smtp-Source: AFSGD/UFzD47UVpN70e+KLDKu6QirlzTQqexSB0qzWKD6SG3CB0a1C9WwYguvDIv6qKfNcLpo/oqmA== X-Received: by 2002:a05:6000:51:: with SMTP id k17mr20731523wrx.259.1544721644399; Thu, 13 Dec 2018 09:20:44 -0800 (PST) Received: from localhost.localdomain (laubervilliers-657-1-83-120.w92-154.abo.wanadoo.fr. [92.154.90.120]) by smtp.gmail.com with ESMTPSA id q12sm2902753wrx.31.2018.12.13.09.20.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Dec 2018 09:20:43 -0800 (PST) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 3/4] arm64: kpti: use nG mappings from the outset if kpti is force enabled Date: Thu, 13 Dec 2018 18:20:34 +0100 Message-Id: <20181213172036.14504-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20181213172036.14504-1-ard.biesheuvel@linaro.org> References: <20181213172036.14504-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181213_092056_628379_3A0EAD57 X-CRM114-Status: GOOD ( 15.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Robin Murphy , John Garry , Will Deacon , Suzuki K Poulose , Ard Biesheuvel Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Instead of relying on a slow asm routine executing from the idmap to change all global mappings into non-global ones, just apply non-global mappings from the outset if KPTI is going to be enabled regardless of CPU capabilities (i.e, when running with KASLR enabled) Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 3 ++- arch/arm64/mm/mmu.c | 9 +++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index ecd8c65dd2d7..11ef6aadeb0c 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -965,6 +965,8 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, return !has_cpuid_feature(entry, scope); } +bool kpti_applied = false; + static void kpti_install_ng_mappings(const struct arm64_cpu_capabilities *__unused) { @@ -972,7 +974,6 @@ kpti_install_ng_mappings(const struct arm64_cpu_capabilities *__unused) extern kpti_remap_fn idmap_kpti_install_ng_mappings; kpti_remap_fn *remap_fn; - static bool kpti_applied = false; int cpu = smp_processor_id(); if (kpti_applied) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d1d6601b385d..ab70834b45b8 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -648,6 +648,15 @@ static void __init map_kernel(pgd_t *pgdp) void __init paging_init(void) { pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); + bool kpti_enabled; + + /* create nG mappings if KPTI is enabled regardless of CPU features */ + if (kpti_is_forced(&kpti_enabled) && kpti_enabled) { + extern bool kpti_applied; + + cpus_set_cap(ARM64_UNMAP_KERNEL_AT_EL0); + kpti_applied = true; + } map_kernel(pgdp); map_mem(pgdp); From patchwork Thu Dec 13 17:20:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10729245 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0E5A16B1 for ; Thu, 13 Dec 2018 17:21:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 906412C79F for ; Thu, 13 Dec 2018 17:21:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 83AB22C7A8; Thu, 13 Dec 2018 17:21:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F3EA42C7AD for ; Thu, 13 Dec 2018 17:21:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CnHDSPf4xrkmV/B2FENtQHAdi3Dh6ytmqwGX2/wFAUg=; b=Jf/IYpJOABfM/N UoVFRe1Jho8zK4Y6clx88Dfxj0Xs/erHzcZawOHUIZOzLvCWNnuk4oXhCKcePMYAk2pjqs0grF3En yfSNo8ne02n/TugjZRyLzWpC2f2DvaXmALmflFRQH4zeomre7Dk+L6sNSPscRocD1AwH39igYn8wa tbQWTByBGBZ1hpqs5SRQA/zsXJKOsrXnuupTpkGjjxOChQjtxX29HIj5g+TFlq0Vls47CfksXBT1t qBHevE5zgQI1G7gDAJ5Ec7AGOliWVaAV9SuwgaBsDexxc7QecPSAjvnInKb91ogSgekFLlnqedi9O LSQEdm9Z12zBXNRBEHXg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gXUgH-00085g-PJ; Thu, 13 Dec 2018 17:21:49 +0000 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gXUfR-0007BF-Kb for linux-arm-kernel@lists.infradead.org; Thu, 13 Dec 2018 17:21:08 +0000 Received: by mail-wm1-x342.google.com with SMTP id q26so3173190wmf.5 for ; Thu, 13 Dec 2018 09:20:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Tbch3xLF+M5jA4HSG7+rG1VY+S90zpEVNMyhSNqIGss=; b=ZKe+m4Mg7rP/DOIidkYfXtaZCZ/e/KMui+yjDwqdcRlbY5+6qXCFRHmaZ8AXokaB8v XUse906riwQipHYWvW7IeBCG6iZEAN0n+9220qPZB6qeQx9651RZg3x2X/HZZdVCDpx5 5d2ppvjA42mfS3ccS4vZIQCDavN9OwtcswD+M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Tbch3xLF+M5jA4HSG7+rG1VY+S90zpEVNMyhSNqIGss=; b=Ao49WOrJMKjU2qa7VwVRka7hpMelxTZ4znSbHrqnRSE1O1iIoYW3PkdNf6ohCW9b8E C0ijfCHKH9ITIXR/JgQHzWM5YoQsLqVWmYjAK8HXV4+yRD3qmCztKHP2ffralYe/GLFu 2zqHNsh1Uaqe6sL/Pnkuoj/JYIbW3xe+9WOvc6Um0x401i64oG2nUrFFiri0dbttW++b 9QTpjHoc37L0slIAtPGMRpa+lCWqLx1iqbLOCf9QH9xmQs3H9dwLeSpICywwUhGRvsYH 50TLhdKeSjhdzD1Ktuv0Ar+okJ+mXoc0VjZWbIgL24/neAHC+KJYsuBk5uZrZzoKBP8q adOw== X-Gm-Message-State: AA+aEWbpRBAhWF3lIwmPWnyy35GrFSqfwiFObpa8HCJD4T7z7+IjaZKS p21+Mz8caHYxXLslOTyo4XeY8hRKnj9WYg== X-Google-Smtp-Source: AFSGD/VXj+yZZEpfabzrC0RWUskiRJl8wSPMg0qsU0VmQjWbodFsTZxZ1cIGylUX84pD+ogoLnxt9w== X-Received: by 2002:a1c:ac42:: with SMTP id v63mr172508wme.119.1544721645560; Thu, 13 Dec 2018 09:20:45 -0800 (PST) Received: from localhost.localdomain (laubervilliers-657-1-83-120.w92-154.abo.wanadoo.fr. [92.154.90.120]) by smtp.gmail.com with ESMTPSA id q12sm2902753wrx.31.2018.12.13.09.20.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Dec 2018 09:20:44 -0800 (PST) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 4/4] arm64: kpti: use non-global mappings unless KPTI is forced off Date: Thu, 13 Dec 2018 18:20:35 +0100 Message-Id: <20181213172036.14504-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20181213172036.14504-1-ard.biesheuvel@linaro.org> References: <20181213172036.14504-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181213_092058_008067_A5FA63A9 X-CRM114-Status: GOOD ( 22.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Robin Murphy , John Garry , Will Deacon , Suzuki K Poulose , Ard Biesheuvel Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP KPTI requires non-global mappings but the converse is not true: we can usually tolerate non-global mappings when KPTI is disabled (with the exception of some ThunderX cores), but the increased TLB footprint of kernel mappings may adversely affect performance in some cases. So let's invert the early mapping logic to always create non-global mappings unless KPTI was forced off, allowing us to get rid of the costly and fragile remapping code that changes kernel mappings from global to non-global at CPU feature detection time. In cases where the increased TLB footprint does in fact cause performance issues and Meltdown mitigations or KASLR are not required or desired, kpti=off may be passed on the kernel command line to switch back to global kernel mappings unconditionally. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 27 --- arch/arm64/mm/mmu.c | 15 +- arch/arm64/mm/proc.S | 189 -------------------- 3 files changed, 10 insertions(+), 221 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 11ef6aadeb0c..649937753587 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -965,32 +965,6 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, return !has_cpuid_feature(entry, scope); } -bool kpti_applied = false; - -static void -kpti_install_ng_mappings(const struct arm64_cpu_capabilities *__unused) -{ - typedef void (kpti_remap_fn)(int, int, phys_addr_t); - extern kpti_remap_fn idmap_kpti_install_ng_mappings; - kpti_remap_fn *remap_fn; - - int cpu = smp_processor_id(); - - if (kpti_applied) - return; - - remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings); - - cpu_install_idmap(); - remap_fn(cpu, num_online_cpus(), __pa_symbol(swapper_pg_dir)); - cpu_uninstall_idmap(); - - if (!cpu) - kpti_applied = true; - - return; -} - static int __init parse_kpti(char *str) { bool enabled; @@ -1260,7 +1234,6 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .field_pos = ID_AA64PFR0_CSV3_SHIFT, .min_field_value = 1, .matches = unmap_kernel_at_el0, - .cpu_enable = kpti_install_ng_mappings, }, #endif { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index ab70834b45b8..74e27f4ae6ea 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -650,12 +650,17 @@ void __init paging_init(void) pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); bool kpti_enabled; - /* create nG mappings if KPTI is enabled regardless of CPU features */ - if (kpti_is_forced(&kpti_enabled) && kpti_enabled) { - extern bool kpti_applied; - + /* create nG mappings unless KPTI is forced off */ + if (!kpti_is_forced(&kpti_enabled) || kpti_enabled) { + /* + * Set the capability so that PTE_MAYBE_NG will evaluate to + * nG enabled. This capability will be cleared again in case + * we decide not to enable KPTI after all at CPU feature + * detection time, in which case we will end up running with + * a mix of non-global and global kernel mappings but this + * shouldn't hurt in practice. + */ cpus_set_cap(ARM64_UNMAP_KERNEL_AT_EL0); - kpti_applied = true; } map_kernel(pgdp); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 2c75b0b903ae..b80d4220f7d0 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -209,195 +209,6 @@ ENTRY(idmap_cpu_replace_ttbr1) ENDPROC(idmap_cpu_replace_ttbr1) .popsection -#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 - .pushsection ".idmap.text", "awx" - - .macro __idmap_kpti_get_pgtable_ent, type - dc cvac, cur_\()\type\()p // Ensure any existing dirty - dmb sy // lines are written back before - ldr \type, [cur_\()\type\()p] // loading the entry - tbz \type, #0, skip_\()\type // Skip invalid and - tbnz \type, #11, skip_\()\type // non-global entries - .endm - - .macro __idmap_kpti_put_pgtable_ent_ng, type - orr \type, \type, #PTE_NG // Same bit for blocks and pages - str \type, [cur_\()\type\()p] // Update the entry and ensure - dmb sy // that it is visible to all - dc civac, cur_\()\type\()p // CPUs. - .endm - -/* - * void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper) - * - * Called exactly once from stop_machine context by each CPU found during boot. - */ -__idmap_kpti_flag: - .long 1 -ENTRY(idmap_kpti_install_ng_mappings) - cpu .req w0 - num_cpus .req w1 - swapper_pa .req x2 - swapper_ttb .req x3 - flag_ptr .req x4 - cur_pgdp .req x5 - end_pgdp .req x6 - pgd .req x7 - cur_pudp .req x8 - end_pudp .req x9 - pud .req x10 - cur_pmdp .req x11 - end_pmdp .req x12 - pmd .req x13 - cur_ptep .req x14 - end_ptep .req x15 - pte .req x16 - - mrs swapper_ttb, ttbr1_el1 - adr flag_ptr, __idmap_kpti_flag - - cbnz cpu, __idmap_kpti_secondary - - /* We're the boot CPU. Wait for the others to catch up */ - sevl -1: wfe - ldaxr w18, [flag_ptr] - eor w18, w18, num_cpus - cbnz w18, 1b - - /* We need to walk swapper, so turn off the MMU. */ - pre_disable_mmu_workaround - mrs x18, sctlr_el1 - bic x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 - isb - - /* Everybody is enjoying the idmap, so we can rewrite swapper. */ - /* PGD */ - mov cur_pgdp, swapper_pa - add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8) -do_pgd: __idmap_kpti_get_pgtable_ent pgd - tbnz pgd, #1, walk_puds -next_pgd: - __idmap_kpti_put_pgtable_ent_ng pgd -skip_pgd: - add cur_pgdp, cur_pgdp, #8 - cmp cur_pgdp, end_pgdp - b.ne do_pgd - - /* Publish the updated tables and nuke all the TLBs */ - dsb sy - tlbi vmalle1is - dsb ish - isb - - /* We're done: fire up the MMU again */ - mrs x18, sctlr_el1 - orr x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 - isb - - /* Set the flag to zero to indicate that we're all done */ - str wzr, [flag_ptr] - ret - - /* PUD */ -walk_puds: - .if CONFIG_PGTABLE_LEVELS > 3 - pte_to_phys cur_pudp, pgd - add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8) -do_pud: __idmap_kpti_get_pgtable_ent pud - tbnz pud, #1, walk_pmds -next_pud: - __idmap_kpti_put_pgtable_ent_ng pud -skip_pud: - add cur_pudp, cur_pudp, 8 - cmp cur_pudp, end_pudp - b.ne do_pud - b next_pgd - .else /* CONFIG_PGTABLE_LEVELS <= 3 */ - mov pud, pgd - b walk_pmds -next_pud: - b next_pgd - .endif - - /* PMD */ -walk_pmds: - .if CONFIG_PGTABLE_LEVELS > 2 - pte_to_phys cur_pmdp, pud - add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8) -do_pmd: __idmap_kpti_get_pgtable_ent pmd - tbnz pmd, #1, walk_ptes -next_pmd: - __idmap_kpti_put_pgtable_ent_ng pmd -skip_pmd: - add cur_pmdp, cur_pmdp, #8 - cmp cur_pmdp, end_pmdp - b.ne do_pmd - b next_pud - .else /* CONFIG_PGTABLE_LEVELS <= 2 */ - mov pmd, pud - b walk_ptes -next_pmd: - b next_pud - .endif - - /* PTE */ -walk_ptes: - pte_to_phys cur_ptep, pmd - add end_ptep, cur_ptep, #(PTRS_PER_PTE * 8) -do_pte: __idmap_kpti_get_pgtable_ent pte - __idmap_kpti_put_pgtable_ent_ng pte -skip_pte: - add cur_ptep, cur_ptep, #8 - cmp cur_ptep, end_ptep - b.ne do_pte - b next_pmd - - /* Secondary CPUs end up here */ -__idmap_kpti_secondary: - /* Uninstall swapper before surgery begins */ - __idmap_cpu_set_reserved_ttbr1 x18, x17 - - /* Increment the flag to let the boot CPU we're ready */ -1: ldxr w18, [flag_ptr] - add w18, w18, #1 - stxr w17, w18, [flag_ptr] - cbnz w17, 1b - - /* Wait for the boot CPU to finish messing around with swapper */ - sevl -1: wfe - ldxr w18, [flag_ptr] - cbnz w18, 1b - - /* All done, act like nothing happened */ - msr ttbr1_el1, swapper_ttb - isb - ret - - .unreq cpu - .unreq num_cpus - .unreq swapper_pa - .unreq swapper_ttb - .unreq flag_ptr - .unreq cur_pgdp - .unreq end_pgdp - .unreq pgd - .unreq cur_pudp - .unreq end_pudp - .unreq pud - .unreq cur_pmdp - .unreq end_pmdp - .unreq pmd - .unreq cur_ptep - .unreq end_ptep - .unreq pte -ENDPROC(idmap_kpti_install_ng_mappings) - .popsection -#endif - /* * __cpu_setup *