From patchwork Tue Nov 16 09:47:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12621909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38344C433F5 for ; Tue, 16 Nov 2021 10:03:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C8CE26152B for ; Tue, 16 Nov 2021 10:03:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C8CE26152B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 502316B0085; Tue, 16 Nov 2021 05:03:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B2B96B0088; Tue, 16 Nov 2021 05:03:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A0CE6B0089; Tue, 16 Nov 2021 05:03:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2A9EB6B0085 for ; Tue, 16 Nov 2021 05:03:51 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id DF758180D6FA6 for ; Tue, 16 Nov 2021 10:03:50 +0000 (UTC) X-FDA: 78814357020.22.894A976 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf27.hostedemail.com (Postfix) with ESMTP id 96B6770000B9 for ; Tue, 16 Nov 2021 10:03:49 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id E3B4E61B3E; Tue, 16 Nov 2021 09:47:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1637056075; bh=HuxCBSVOLmJNXWCjSAgdud3xi8eb8teRW9t905R2Svg=; h=From:To:Cc:Subject:Date:From; b=CMrX0thXaa+iuiRomOyKQWLsIwep4MqXGG77tyVdoffwAdC8oBNtWEOGBoZ2tFVwq d50hpHZjCCFGk/uGC5AVfPABoXLtPXxdqF6wENfLvw3Fdl58aVuhs+F0aXWJfBSmIE egY2sP3f5ffqOzNu1lLcra4+6fRMKaDAyQqGYFASPBvsARw6ilFw8xwexWnXD1JOfT mjMxzUWToe5SNuTpYBTQ+0PbQQQ8DKjsYjMMFERxYL7pCuoszWw7sxaQZO92bSovS+ mevfLA/kUZYAz8Veh1Zqlq7AC0n4FAEZKSKRR4+DTCmPiVowD74h+vsJtVsIZbQtTs xgqfM7gxJaGvA== From: Ard Biesheuvel To: linux-mm@kvack.org Cc: Ard Biesheuvel , Quanyang Wang , Linus Walleij , Thomas Gleixner , Russell King , Andrew Morton Subject: [PATCH resend] kmap_local: don't assume kmap PTEs are linear arrays in memory Date: Tue, 16 Nov 2021 10:47:37 +0100 Message-Id: <20211116094737.7391-1-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5670; h=from:subject; bh=HuxCBSVOLmJNXWCjSAgdud3xi8eb8teRW9t905R2Svg=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhk344T9lQKv38HqebUfRBdnB9eg7ku6M8bj+KZZ8K W/NNAHuJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYZN+OAAKCRDDTyI5ktmPJKvtC/ 45JfBAlWMBxOT6AAoLsDg0vzW81GVa8mtnw/+fg0wX8AGGykcLqjFrExCoy8iyhk0etP+pGSurqnjS r9vgDMjVXmZHWdSJ7Kcm4vj+GSXoiR/nqCZRoTSC8eOI8kjthgNbB48FUZHERaknNKA18WURI+OPHr TWLJO0FUPxyx+YowhWzn3gzZU8RCQr0rIha9GuiwTTq0iCdG4wMbs8gdQ9ez7/APkcy0o7c3OfQMY4 ftY/nFRaabha5lpGNPHWhgoKlavaYt9B35TiRO/K5JKfzKzkNxmsxiHvru/ktQ+553pH6XsQkV53GB PdJti2LJ1PTX3oKvo/+7FwZIthjOv1zoGrKL0KI9sxY9OHIuVe1/SsWNK4XSw/CDTFmnKzmGuJYTqr q3An3Rd0NgJyNQqsNNZ+piWThykMwHEdmSatbaJDAf8Tsl2V8bsJ4a2eupS0jDjebaW/LkWGwzpoGD WoJkQ6pDf11v11Aah9pd4ZFJUVfuiNwrEJN38SvnzWTAM= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=CMrX0thX; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf27.hostedemail.com: domain of ardb@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=ardb@kernel.org X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 96B6770000B9 X-Stat-Signature: 6obutwmrmwthstci761c3eo6t4yzt1rx X-HE-Tag: 1637057029-336925 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The kmap_local conversion broke the ARM architecture, because the new code assumes that all PTEs used for creating kmaps form a linear array in memory, and uses array indexing to look up the kmap PTE belonging to a certain kmap index. On ARM, this cannot work, not only because the PTE pages may be non-adjacent in memory, but also because ARM/!LPAE interleaves hardware entries and extended entries (carrying software-only bits) in a way that is not compatible with array indexing. Fortunately, this only seems to affect configurations with more than 8 CPUs, due to the way the per-CPU kmap slots are organized in memory. Work around this by permitting an architecture to set a Kconfig symbol that signifies that the kmap PTEs do not form a lineary array in memory, and so the only way to locate the appropriate one is to walk the page tables. Link: https://lore.kernel.org/linux-arm-kernel/20211026131249.3731275-1-ardb@kernel.org/ Reported-by: Quanyang Wang Reviewed-by: Linus Walleij Cc: Thomas Gleixner Cc: Russell King Cc: Andrew Morton Signed-off-by: Ard Biesheuvel Acked-by: Russell King (Oracle) Reviewed-by: Thomas Gleixner --- Resent to linux-mm on akpm's request arch/arm/Kconfig | 1 + mm/Kconfig | 3 ++ mm/highmem.c | 32 +++++++++++++------- 3 files changed, 25 insertions(+), 11 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 899830259bed..5d4ab8e8477f 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1470,6 +1470,7 @@ config HIGHMEM bool "High Memory Support" depends on MMU select KMAP_LOCAL + select KMAP_LOCAL_NON_LINEAR_PTE_ARRAY help The address space of ARM processors is only 4 Gigabytes large and it has to accommodate user address space, kernel address diff --git a/mm/Kconfig b/mm/Kconfig index 068ce591a13a..28edafc820ad 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -890,6 +890,9 @@ config MAPPING_DIRTY_HELPERS config KMAP_LOCAL bool +config KMAP_LOCAL_NON_LINEAR_PTE_ARRAY + bool + # struct io_mapping based helper. Selected by drivers that need them config IO_MAPPING bool diff --git a/mm/highmem.c b/mm/highmem.c index 88f65f155845..ca9fa8c92593 100644 --- a/mm/highmem.c +++ b/mm/highmem.c @@ -503,16 +503,22 @@ static inline int kmap_local_calc_idx(int idx) static pte_t *__kmap_pte; -static pte_t *kmap_get_pte(void) +static pte_t *kmap_get_pte(unsigned long vaddr, int idx) { + if (IS_ENABLED(CONFIG_KMAP_LOCAL_NON_LINEAR_PTE_ARRAY)) + /* + * Set by the arch if __kmap_pte[-idx] does not produce + * the correct entry. + */ + return virt_to_kpte(vaddr); if (!__kmap_pte) __kmap_pte = virt_to_kpte(__fix_to_virt(FIX_KMAP_BEGIN)); - return __kmap_pte; + return &__kmap_pte[-idx]; } void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot) { - pte_t pteval, *kmap_pte = kmap_get_pte(); + pte_t pteval, *kmap_pte; unsigned long vaddr; int idx; @@ -524,9 +530,10 @@ void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot) preempt_disable(); idx = arch_kmap_local_map_idx(kmap_local_idx_push(), pfn); vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); - BUG_ON(!pte_none(*(kmap_pte - idx))); + kmap_pte = kmap_get_pte(vaddr, idx); + BUG_ON(!pte_none(*kmap_pte)); pteval = pfn_pte(pfn, prot); - arch_kmap_local_set_pte(&init_mm, vaddr, kmap_pte - idx, pteval); + arch_kmap_local_set_pte(&init_mm, vaddr, kmap_pte, pteval); arch_kmap_local_post_map(vaddr, pteval); current->kmap_ctrl.pteval[kmap_local_idx()] = pteval; preempt_enable(); @@ -559,7 +566,7 @@ EXPORT_SYMBOL(__kmap_local_page_prot); void kunmap_local_indexed(void *vaddr) { unsigned long addr = (unsigned long) vaddr & PAGE_MASK; - pte_t *kmap_pte = kmap_get_pte(); + pte_t *kmap_pte; int idx; if (addr < __fix_to_virt(FIX_KMAP_END) || @@ -584,8 +591,9 @@ void kunmap_local_indexed(void *vaddr) idx = arch_kmap_local_unmap_idx(kmap_local_idx(), addr); WARN_ON_ONCE(addr != __fix_to_virt(FIX_KMAP_BEGIN + idx)); + kmap_pte = kmap_get_pte(addr, idx); arch_kmap_local_pre_unmap(addr); - pte_clear(&init_mm, addr, kmap_pte - idx); + pte_clear(&init_mm, addr, kmap_pte); arch_kmap_local_post_unmap(addr); current->kmap_ctrl.pteval[kmap_local_idx()] = __pte(0); kmap_local_idx_pop(); @@ -607,7 +615,7 @@ EXPORT_SYMBOL(kunmap_local_indexed); void __kmap_local_sched_out(void) { struct task_struct *tsk = current; - pte_t *kmap_pte = kmap_get_pte(); + pte_t *kmap_pte; int i; /* Clear kmaps */ @@ -634,8 +642,9 @@ void __kmap_local_sched_out(void) idx = arch_kmap_local_map_idx(i, pte_pfn(pteval)); addr = __fix_to_virt(FIX_KMAP_BEGIN + idx); + kmap_pte = kmap_get_pte(addr, idx); arch_kmap_local_pre_unmap(addr); - pte_clear(&init_mm, addr, kmap_pte - idx); + pte_clear(&init_mm, addr, kmap_pte); arch_kmap_local_post_unmap(addr); } } @@ -643,7 +652,7 @@ void __kmap_local_sched_out(void) void __kmap_local_sched_in(void) { struct task_struct *tsk = current; - pte_t *kmap_pte = kmap_get_pte(); + pte_t *kmap_pte; int i; /* Restore kmaps */ @@ -663,7 +672,8 @@ void __kmap_local_sched_in(void) /* See comment in __kmap_local_sched_out() */ idx = arch_kmap_local_map_idx(i, pte_pfn(pteval)); addr = __fix_to_virt(FIX_KMAP_BEGIN + idx); - set_pte_at(&init_mm, addr, kmap_pte - idx, pteval); + kmap_pte = kmap_get_pte(addr, idx); + set_pte_at(&init_mm, addr, kmap_pte, pteval); arch_kmap_local_post_map(addr, pteval); } }