From patchwork Thu Apr 4 13:36:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Huacai Chen X-Patchwork-Id: 13617864 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 277DDCD1284 for ; Thu, 4 Apr 2024 13:37:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3DD96B0087; Thu, 4 Apr 2024 09:37:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9EE376B0088; Thu, 4 Apr 2024 09:37:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DDF66B0089; Thu, 4 Apr 2024 09:37:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6C7876B0087 for ; Thu, 4 Apr 2024 09:37:22 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DF3ACA0195 for ; Thu, 4 Apr 2024 13:37:21 +0000 (UTC) X-FDA: 81971951082.07.2D98D23 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf07.hostedemail.com (Postfix) with ESMTP id A8A9240005 for ; Thu, 4 Apr 2024 13:37:19 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; spf=pass (imf07.hostedemail.com: domain of "SRS0=Mf8M=LJ=loongson.cn=chenhuacai@kernel.org" designates 145.40.73.55 as permitted sender) smtp.mailfrom="SRS0=Mf8M=LJ=loongson.cn=chenhuacai@kernel.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712237840; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3f99SFlkVL5OOnE6U6HiiQqsAZsGzkWrHeOu0tQxUQ0=; b=PQyWcHcXoo6ght3/rf/Q9HqSMYoi3vdPNQ3Et4gGoEij4RmJXNnpm3bFo90okHZHRz/Xi9 jZZ8RJjv/lr16e76N/mA7AvklsjCyLrIMwDITScSYX7pKOhx8AnldpKvYq/oDZt+0vad5g 44QWDXxwu3t3FuY9i9uUfsUSewYRUkA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712237840; a=rsa-sha256; cv=none; b=qYE++ZrmbE7CBfe5QFEtS7e3vaBvJRupYHapF365cLwgV36/fKrJ6Pq08HmN0BnKrekotC DwutOyyNK+D/8HV/0g3l9y58gxIXXJJhF5AitXGWGNQOaSrlvr9X1praPJd7ab4yIsiZUY KtVDr26no8qISEMHEoSCjT98D4fIC1o= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; spf=pass (imf07.hostedemail.com: domain of "SRS0=Mf8M=LJ=loongson.cn=chenhuacai@kernel.org" designates 145.40.73.55 as permitted sender) smtp.mailfrom="SRS0=Mf8M=LJ=loongson.cn=chenhuacai@kernel.org"; dmarc=none Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 93924CE2B37; Thu, 4 Apr 2024 13:37:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3756EC433C7; Thu, 4 Apr 2024 13:37:13 +0000 (UTC) From: Huacai Chen To: Andrew Morton , Huacai Chen Cc: loongarch@lists.linux.dev, linux-mm@kvack.org, Xuefeng Li , Guo Ren , Xuerui Wang , Jiaxun Yang , linux-kernel@vger.kernel.org, loongson-kernel@lists.loongnix.cn, Huacai Chen Subject: [PATCH 2/3] LoongArch: Make {virt, phys, page, pfn} translation work with KFENCE Date: Thu, 4 Apr 2024 21:36:35 +0800 Message-ID: <20240404133642.971583-3-chenhuacai@loongson.cn> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240404133642.971583-1-chenhuacai@loongson.cn> References: <20240404133642.971583-1-chenhuacai@loongson.cn> MIME-Version: 1.0 X-Rspamd-Queue-Id: A8A9240005 X-Rspam-User: X-Stat-Signature: dfdg97p4zyfbfwyyaremypj9pte8qsjf X-Rspamd-Server: rspam03 X-HE-Tag: 1712237839-391752 X-HE-Meta: U2FsdGVkX1/y7sHCSpPmWKRJaetb0M0J5sWyKHhmRFt9ZOQCCOIYewLKbrcq05zfA4Vj8+edxqPEbbrpGsbukxOHcRvHKV8IwVyHWaTJpKq9Txgx7LdDEoLyV09bs1LhO4YoHSy9m4uVsvqc2bF+ZLRTK7Cis+u824ySdv/D7ZSQF5VPHYPNQzd4Ew/5oQkRtZ640kiLabMCulgsFBSD1onukhi/FgICtdtdYRchvz5sq+Uljz15f1LWu+Mc1uywuk3U18ItQNnWMHXXCt4RnqVMDaCbqBHpdZvUxjjLExe/qHi8DXittcolPugrcAPz8W7HvhXc4NeWsdmy81l7NjM86frIos4feVGXmyBtQ0F3tMepFwUagiRaSfumtcbVoNAQe1CpuVx3i/sBgnUeKxtaCCTVCQ5ZkAasbDEgbBaSd8CYrQ4wwbWOYkOjEeo0IwKcch+cLbWhITJHSH8znOw82pqN3QBQ9tyzz4I153OOlH1TSZ4TELc8ZvuPIGDeghNp570FfLGHxNZz9IeZCC5YY6525VUq0y0O1+rwbAKmlL9CIEszbeJY27FL+bwljyBxeWaUdKvAfwaMlmyAHHEyGo0XOGuMa+YK/8OT5LHRLrREJsGr+Ek697mCHM4d3zfmcj5mdAMtWM5szKFm/whsc2TSyjIRd9oHuK6F14DaaSUSSyzVdypo/vG9El/myHRFvSNXAk1dJS9DtagA/yp1IANpgnxOf3sS5JHd2z7fZpk97Acobwr3RtHG+rm9zJUIYuTJc4yDca/oaEaIjFBCLKh7G/Znqd41HJ+fjlrQ95tEiQW3SegLQmNTdJwSA6fqVn9q974Na4K7EyuG2/eT/fSxhGBc0NIJznewPTUIM+eTj3g4E/vPGrYvPZuHjtYuPWdU8BXGeIgHdweq5n0momFKqPLB+BAHnYn2KxA0MtHTLDowWpGSa5gmrqtmkUDvGmkNvDQvHhmXmmx 6y9KoIB3 G6wjD0gvQjX2OtyMDZVcjk4j1La2xL6a8PF2bZ/OcP2GZlguqpTlRe1aQxDRcMGgsQVAfLtGT6FdPPmKCbjIJsneJXophfVYDGxzzoGthqCh3FTFwoMFfm8GOEvB7SK1W18jBoRyVluEjqCLUpqQ2NNwQhF7NLzb81xywuaYMlVD09l80XFbM0g2wkH6+zYdbdlMlBJEb9hdieGVlSVLYe3D9yBfmFff1lpywYMW+6g27o4aue5rHr5r6crn+9jkqYgAkP/iXzKUxybSC/KQsa1nYS7XooroNkvsLA1AHbHhzskE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: KFENCE changes virt_to_page() to be able to translate tlb mapped virtual addresses, but forget to change virt_to_phys()/phys_to_virt() and other translation functions as well. This patch fix it, otherwise some drivers (such as nvme and virtio-blk) cannot work with KFENCE. All {virt, phys, page, pfn} translation functions are updated: 1, virt_to_pfn()/pfn_to_virt(); 2, virt_to_page()/page_to_virt(); 3, virt_to_phys()/phys_to_virt(). DMW/TLB mapped addresses are distinguished by comparing the vaddress with vm_map_base in virt_to_xyz(), and we define WANT_PAGE_VIRTUAL in the KFENCE case for the reverse translations, xyz_to_virt(). Signed-off-by: Huacai Chen --- arch/loongarch/include/asm/io.h | 20 +++++++++++++++----- arch/loongarch/include/asm/kfence.h | 9 +++++++++ arch/loongarch/include/asm/page.h | 26 +++++++++++++++++++++++++- arch/loongarch/mm/pgtable.c | 4 ++-- 4 files changed, 51 insertions(+), 8 deletions(-) diff --git a/arch/loongarch/include/asm/io.h b/arch/loongarch/include/asm/io.h index 4a8adcca329b..c2f9979b2979 100644 --- a/arch/loongarch/include/asm/io.h +++ b/arch/loongarch/include/asm/io.h @@ -14,11 +14,6 @@ #include #include -/* - * Change "struct page" to physical address. - */ -#define page_to_phys(page) ((phys_addr_t)page_to_pfn(page) << PAGE_SHIFT) - extern void __init __iomem *early_ioremap(u64 phys_addr, unsigned long size); extern void __init early_iounmap(void __iomem *addr, unsigned long size); @@ -73,6 +68,21 @@ extern void __memcpy_fromio(void *to, const volatile void __iomem *from, size_t #define __io_aw() mmiowb() +#ifdef CONFIG_KFENCE +#define virt_to_phys(kaddr) \ +({ \ + (likely((unsigned long)kaddr < vm_map_base)) ? __pa((unsigned long)kaddr) : \ + page_to_phys(tlb_virt_to_page((unsigned long)kaddr)) + offset_in_page((unsigned long)kaddr);\ +}) + +#define phys_to_virt(paddr) \ +({ \ + extern char *__kfence_pool; \ + (unlikely(__kfence_pool == NULL)) ? __va((unsigned long)paddr) : \ + page_address(phys_to_page((unsigned long)paddr)) + offset_in_page((unsigned long)paddr);\ +}) +#endif + #include #define ARCH_HAS_VALID_PHYS_ADDR_RANGE diff --git a/arch/loongarch/include/asm/kfence.h b/arch/loongarch/include/asm/kfence.h index 6c82aea1c993..a6a5760da3a3 100644 --- a/arch/loongarch/include/asm/kfence.h +++ b/arch/loongarch/include/asm/kfence.h @@ -16,6 +16,7 @@ static inline bool arch_kfence_init_pool(void) { int err; + char *kaddr, *vaddr; char *kfence_pool = __kfence_pool; struct vm_struct *area; @@ -35,6 +36,14 @@ static inline bool arch_kfence_init_pool(void) return false; } + kaddr = kfence_pool; + vaddr = __kfence_pool; + while (kaddr < kfence_pool + KFENCE_POOL_SIZE) { + set_page_address(virt_to_page(kaddr), vaddr); + kaddr += PAGE_SIZE; + vaddr += PAGE_SIZE; + } + return true; } diff --git a/arch/loongarch/include/asm/page.h b/arch/loongarch/include/asm/page.h index 44027060c54a..e85df33f11c7 100644 --- a/arch/loongarch/include/asm/page.h +++ b/arch/loongarch/include/asm/page.h @@ -78,7 +78,26 @@ typedef struct { unsigned long pgprot; } pgprot_t; struct page *dmw_virt_to_page(unsigned long kaddr); struct page *tlb_virt_to_page(unsigned long kaddr); -#define virt_to_pfn(kaddr) PFN_DOWN(PHYSADDR(kaddr)) +#define pfn_to_phys(pfn) __pfn_to_phys(pfn) +#define phys_to_pfn(paddr) __phys_to_pfn(paddr) + +#define page_to_phys(page) pfn_to_phys(page_to_pfn(page)) +#define phys_to_page(paddr) pfn_to_page(phys_to_pfn(paddr)) + +#ifndef CONFIG_KFENCE + +#define page_to_virt(page) __va(page_to_phys(page)) +#define virt_to_page(kaddr) phys_to_page(__pa(kaddr)) + +#else + +#define WANT_PAGE_VIRTUAL + +#define page_to_virt(page) \ +({ \ + extern char *__kfence_pool; \ + (__kfence_pool == NULL) ? __va(page_to_phys(page)) : page_address(page); \ +}) #define virt_to_page(kaddr) \ ({ \ @@ -86,6 +105,11 @@ struct page *tlb_virt_to_page(unsigned long kaddr); dmw_virt_to_page((unsigned long)kaddr) : tlb_virt_to_page((unsigned long)kaddr);\ }) +#endif + +#define pfn_to_virt(pfn) page_to_virt(pfn_to_page(pfn)) +#define virt_to_pfn(kaddr) page_to_pfn(virt_to_page(kaddr)) + extern int __virt_addr_valid(volatile void *kaddr); #define virt_addr_valid(kaddr) __virt_addr_valid((volatile void *)(kaddr)) diff --git a/arch/loongarch/mm/pgtable.c b/arch/loongarch/mm/pgtable.c index 2aae72e63871..bda018150000 100644 --- a/arch/loongarch/mm/pgtable.c +++ b/arch/loongarch/mm/pgtable.c @@ -11,13 +11,13 @@ struct page *dmw_virt_to_page(unsigned long kaddr) { - return pfn_to_page(virt_to_pfn(kaddr)); + return phys_to_page(__pa(kaddr)); } EXPORT_SYMBOL(dmw_virt_to_page); struct page *tlb_virt_to_page(unsigned long kaddr) { - return pfn_to_page(pte_pfn(*virt_to_kpte(kaddr))); + return phys_to_page(pfn_to_phys(pte_pfn(*virt_to_kpte(kaddr)))); } EXPORT_SYMBOL(tlb_virt_to_page);