From patchwork Thu Nov 23 06:56:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13465843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C970FC61D97 for ; Thu, 23 Nov 2023 06:57:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=VYnV1DSVrL2cqE8XR8wlF3/QtMDANXFPNVIER76J1fI=; b=dSvO1CSoBxuwtE 44n5i7XZ8IgaPz035G20AKk0sHGKXmqXcpsS6T6FfB6sNwkkEqlzLJK9DrABPo/6eml1/J9A3Gvso 8AXs4vlVSOLjPoqTg3Yvyxf6f9+k8JoFiIb9IpGK7hBZB3g3QYKN9bNAZiRt5CFNXheu04lIiu0gs aaMySMJqX5HCE111kc/extYCIR+HJWTJtt0SNTukZLPlHlMKw0X5zKFL6eipBXFgdw3001JDu5doO 6bdNNCY+0vRuQD0YjOPzNiAd901FbthsMob3w8XmfiDzoC1sYiXW7n+uTjW2ONZWBVidtee+ZM/sq kOxCgXVkt6Z/eAjbiA0A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r63eZ-003xE3-1u; Thu, 23 Nov 2023 06:57:35 +0000 Received: from mail-oi1-x233.google.com ([2607:f8b0:4864:20::233]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r63eW-003xCV-2i for linux-riscv@lists.infradead.org; Thu, 23 Nov 2023 06:57:34 +0000 Received: by mail-oi1-x233.google.com with SMTP id 5614622812f47-3b84402923fso329219b6e.0 for ; Wed, 22 Nov 2023 22:57:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1700722651; x=1701327451; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9WCHkg8EwYgV8K7NhoHBdWRiWjVGGiMNhMP8Dvq/T3c=; b=MnLAIqbAb0iZvNvD7tmygn81i3W04X7qiKNvcbWD4WH5dSzaT/XqPA9RJ2aj91zEiB NKnSXYslBS7WFZq0ls7hDW1NcxDnKFGz5mTJZf+nN5YqKjLQFFnwJGdjpSQSS5hhTzyN onH9e9hrV5yo2zXKPuzmE5v4SPsWmxNf6KWPK6FJbR3/bqT9lL1AaVEEtsK9ru9e8YyD crxIDUFlJWZt3EvNlIjqPMkymBucpjtygBwBhjh2uDK8sAycT+i7fG/o/AxBHVnI1l91 1xyOtW+76mIQh/T1rYnPlASEPkLSN4NWzH7akTSdM+e/pDbLca4BHKG0bQT1XPcWIeh6 JfUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700722651; x=1701327451; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9WCHkg8EwYgV8K7NhoHBdWRiWjVGGiMNhMP8Dvq/T3c=; b=fmZSVOILLMLrEjvTfdXMSwiJtJZ697TXpLlNSlkKR5RH53GI7gl+Yo8vzK6tMYN/Y0 hWx6gYliJuHXnhsQOEJf/3cEGqZph5vKbpBn08GtgBnHzszwHtqunJuiTsaA2MOXswjd hQCQxurWncImplPtcui6CRGDAUC+ps/+jLbMSw9UZEWk6tqTKBgZJ7egt7FrB41jj6nL hVr6dpL1VJhG1nlC73NtpPJYl+GRwq66gRg4iJCgt7+T/DGK++xu6WFfzBrW/BFMdsuJ NeBgabliYCXOUIAuD8PVQIATfwTEG7t7gjRh2BS2ZweUAf1ZGgEV98B1QAZrD0gDyfzz AtIw== X-Gm-Message-State: AOJu0Yy5/LJ3LLtUDyzCXZXxhjW3Jn0L2+7kqDPVr511lvYLI1C27S/P xxEeJiIHky+Q9mhFj6JMSF2UtA== X-Google-Smtp-Source: AGHT+IF+JUV2qs3GxZcMJ8mDN8tfMhVpkCkyXhPxn9II1VI5nQr9SgqkEEhOwu5qk/0BmZa46AiHEg== X-Received: by 2002:a05:6808:2096:b0:3b2:e461:bcfc with SMTP id s22-20020a056808209600b003b2e461bcfcmr5671424oiw.20.1700722651004; Wed, 22 Nov 2023 22:57:31 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w37-20020a634765000000b005bd2b3a03eesm615437pgk.6.2023.11.22.22.57.26 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Nov 2023 22:57:30 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: dengliang.1214@bytedance.com, xieyongji@bytedance.com, lihangjing@bytedance.com, songmuchun@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH V1 01/11] mm: Fix misused APIs on huge pte Date: Thu, 23 Nov 2023 14:56:58 +0800 Message-Id: <20231123065708.91345-2-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231123065708.91345-1-luxu.kernel@bytedance.com> References: <20231123065708.91345-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_225732_885709_ED57477A X-CRM114-Status: GOOD ( 12.05 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org There exist some paths that try to get value of huge pte via normal pte API ptep_get instead of huge pte API huge_ptep_get. This commit corrects these misused APIs. Signed-off-by: Xu Lu --- arch/riscv/mm/hugetlbpage.c | 2 +- fs/proc/task_mmu.c | 2 +- include/asm-generic/hugetlb.h | 7 +++++++ mm/hugetlb.c | 2 +- mm/migrate.c | 5 ++++- mm/mprotect.c | 2 +- mm/rmap.c | 10 ++++++++-- mm/vmalloc.c | 3 ++- 8 files changed, 25 insertions(+), 8 deletions(-) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index b52f0210481f..d7cf8e2d3c5b 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -74,7 +74,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, out: if (pte) { - pte_t pteval = ptep_get_lockless(pte); + pte_t pteval = huge_ptep_get_lockless(pte); WARN_ON_ONCE(pte_present(pteval) && !pte_huge(pteval)); } diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index ef2eb12906da..0fe9d23aa062 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -726,7 +726,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, struct mem_size_stats *mss = walk->private; struct vm_area_struct *vma = walk->vma; struct page *page = NULL; - pte_t ptent = ptep_get(pte); + pte_t ptent = huge_ptep_get(pte); if (pte_present(ptent)) { page = vm_normal_page(vma, addr, ptent); diff --git a/include/asm-generic/hugetlb.h b/include/asm-generic/hugetlb.h index 6dcf4d576970..52c299db971a 100644 --- a/include/asm-generic/hugetlb.h +++ b/include/asm-generic/hugetlb.h @@ -150,6 +150,13 @@ static inline pte_t huge_ptep_get(pte_t *ptep) } #endif +#ifndef __HAVE_ARCH_HUGE_PTEP_GET_LOCKLESS +static inline pte_t huge_ptep_get_lockless(pte_t *ptep) +{ + return huge_ptep_get(ptep); +} +#endif + #ifndef __HAVE_ARCH_GIGANTIC_PAGE_RUNTIME_SUPPORTED static inline bool gigantic_page_runtime_supported(void) { diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1169ef2f2176..9f773eb95b3b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -7406,7 +7406,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, } if (pte) { - pte_t pteval = ptep_get_lockless(pte); + pte_t pteval = huge_ptep_get_lockless(pte); BUG_ON(pte_present(pteval) && !pte_huge(pteval)); } diff --git a/mm/migrate.c b/mm/migrate.c index 35a88334bb3c..d0daf58e486e 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -210,7 +210,10 @@ static bool remove_migration_pte(struct folio *folio, folio_get(folio); pte = mk_pte(new, READ_ONCE(vma->vm_page_prot)); - old_pte = ptep_get(pvmw.pte); + if (folio_test_hugetlb(folio)) + old_pte = huge_ptep_get(pvmw.pte); + else + old_pte = ptep_get(pvmw.pte); if (pte_swp_soft_dirty(old_pte)) pte = pte_mksoft_dirty(pte); diff --git a/mm/mprotect.c b/mm/mprotect.c index 81991102f785..b9129c03f451 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -555,7 +555,7 @@ static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask, unsigned long addr, unsigned long next, struct mm_walk *walk) { - return pfn_modify_allowed(pte_pfn(ptep_get(pte)), + return pfn_modify_allowed(pte_pfn(huge_ptep_get(pte)), *(pgprot_t *)(walk->private)) ? 0 : -EACCES; } diff --git a/mm/rmap.c b/mm/rmap.c index 7a27a2b41802..d93c6dabbdf4 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1577,7 +1577,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, break; } - pfn = pte_pfn(ptep_get(pvmw.pte)); + if (folio_test_hugetlb(folio)) + pfn = pte_pfn(huge_ptep_get(pvmw.pte)); + else + pfn = pte_pfn(ptep_get(pvmw.pte)); subpage = folio_page(folio, pfn - folio_pfn(folio)); address = pvmw.address; anon_exclusive = folio_test_anon(folio) && @@ -1931,7 +1934,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, /* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO(!pvmw.pte, folio); - pfn = pte_pfn(ptep_get(pvmw.pte)); + if (folio_test_hugetlb(folio)) + pfn = pte_pfn(huge_ptep_get(pvmw.pte)); + else + pfn = pte_pfn(ptep_get(pvmw.pte)); if (folio_is_zone_device(folio)) { /* diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d12a17fc0c17..1a451b82a7ac 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -103,7 +103,6 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, if (!pte) return -ENOMEM; do { - BUG_ON(!pte_none(ptep_get(pte))); #ifdef CONFIG_HUGETLB_PAGE size = arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift); @@ -111,11 +110,13 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pte_t entry = pfn_pte(pfn, prot); entry = arch_make_huge_pte(entry, ilog2(size), 0); + BUG_ON(!pte_none(huge_ptep_get(pte))); set_huge_pte_at(&init_mm, addr, pte, entry, size); pfn += PFN_DOWN(size); continue; } #endif + BUG_ON(!pte_none(ptep_get(pte))); set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); pfn++; } while (pte += PFN_DOWN(size), addr += size, addr != end); From patchwork Thu Nov 23 06:56:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13465844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6E4AAC61D85 for ; Thu, 23 Nov 2023 06:57:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=aUje3nUZ37HZW4ANa5XtRkxG4tR5OzZLztv+6bP+hzg=; b=iPONH0fKOFtGUY DiddjE5xAGjWpPfL14Vt8OQj/rwuHprcJDeGqyWzPYG8CiEjp5ufzTlK/8/VLEw+gLFnmMgEeluDR G0esi5SaXKb0aGhrKKEROfU7ERO0fLbwhjJDxU0AAa5NMdEwKiA8rrsruGYJNIOT6Syfkg9HeMeqa +NYP+s2crzgHHQ9cqhcvpRafNYBONueqNMZyXZuL+5lHUAtkG6J1YRQpUQC5bEN6pdeN7zVY90Bd+ LdvIRyysvJrV91Ujn2qRpfJ16UBBw6ZCEigyZ2Qmei5MO7bml+zwXx9FnBvTwdKClkWri8QK8b4Wk X6LcwAbW7nZkdaCWbzFg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r63ed-003xFw-22; Thu, 23 Nov 2023 06:57:39 +0000 Received: from mail-ot1-x32f.google.com ([2607:f8b0:4864:20::32f]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r63eb-003xEi-2P for linux-riscv@lists.infradead.org; Thu, 23 Nov 2023 06:57:39 +0000 Received: by mail-ot1-x32f.google.com with SMTP id 46e09a7af769-6cd1918afb2so354595a34.0 for ; Wed, 22 Nov 2023 22:57:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1700722656; x=1701327456; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x1SlIwZQZIPlzGLzwkEo+aJE2Wdd8/z5GIv+0pytAfk=; b=N8zum1ZX+DVCojnEsZnzOm+63fnAyJEqWCbYN1urRo1bQG1E2a8ikuBSNyx2OB3o2/ fzYhGwG60oBLvE1/4uELczuvYi7nDpEM5veXbRp6GV4XfSn1fY1RWGnm+WV1hjsST8sf a5K3xkNOYLl2ocX4lS5e0FUOGIhp7sWaBzkUfQv1RhKE0UaRdmYfLXC+VpRUF84HcZwS ObzP80RE6juDb4ZRjIHpXXWJW6EoZ+oI2iPwcmgh0K/amXCvYOTL/IgSAhDzerxGvZcr Hq4FM4KHDpPSjb0W9iGgvj3dMCdwzLyoGi+8SYz//4TGPF2Y+yI003oEnfWRWwDSK/nc iZqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700722656; x=1701327456; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x1SlIwZQZIPlzGLzwkEo+aJE2Wdd8/z5GIv+0pytAfk=; b=vBkkkszEPdaWW8LbSNSE2FXbGY6lVgNxlqodaRgtTi2ZNFXR80BSzLBeuCqXZmtFwr FAHEkMac2SkNFe8h8mlcWOx2liM4kl1RYJZZ85aTh2EweP30koy9izyjN7zxgJdK8W3n e7mUyZghfIb5zgnyvmTZwty7sEBTmA95WWU2bx0Cfvea2BJRvibRH+cY37ot2C7YfRz8 D6+V7iPN/qCztpMnI6f8Oo6TILzSZSSN273OnYTssRra1LG+NlQKxDhhVRmVjaGTLKsn m5P1C//LyKEETfMZFlrGXkg9taMY5kB3DrPvetHuRRHSOiVcd8Xb7K28do9wA9/L2948 JpGA== X-Gm-Message-State: AOJu0Ywi2kCK9jleQbuuIKDE+4Zx3GhsPW0LQXJ4+bdooSBdLjb6ZvF6 bSjvg9ycW9r4tKF14VpBqWlFxw== X-Google-Smtp-Source: AGHT+IEzA3h3BSIYMfAOV+OKk+4/Lh1uORa+ogLGTQQeOgfArkr2f4ATFwv21Gu8GGvcH6ELupclgg== X-Received: by 2002:a9d:6481:0:b0:6d7:f02f:dcc3 with SMTP id g1-20020a9d6481000000b006d7f02fdcc3mr3690469otl.28.1700722656378; Wed, 22 Nov 2023 22:57:36 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w37-20020a634765000000b005bd2b3a03eesm615437pgk.6.2023.11.22.22.57.31 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Nov 2023 22:57:35 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: dengliang.1214@bytedance.com, xieyongji@bytedance.com, lihangjing@bytedance.com, songmuchun@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH V1 02/11] riscv: Introduce concept of hardware base page Date: Thu, 23 Nov 2023 14:56:59 +0800 Message-Id: <20231123065708.91345-3-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231123065708.91345-1-luxu.kernel@bytedance.com> References: <20231123065708.91345-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_225737_804733_0B09B442 X-CRM114-Status: GOOD ( 10.73 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The key idea to implement larger base page based on MMU that only supports 4K page is to decouple the MMU page from the software page in view of kernel mm. In contrary to software page, we denote the MMU page as hardware page. To decouple these two kinds of pages, we should manage, allocate and map memory at a granularity of software page, which is exactly what existing mm code does. The page table operations, however, should configure page table entries at a granularity of hardware page, which is the responsibility of arch code. This commit introduces the concept of hardware base page for RISCV. Signed-off-by: Xu Lu --- arch/riscv/Kconfig | 8 ++++++++ arch/riscv/include/asm/page.h | 6 +++++- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 95a2a06acc6a..105cbb3ca797 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -221,6 +221,14 @@ config PAGE_OFFSET default 0x80000000 if !MMU default 0xff60000000000000 if 64BIT +config RISCV_HW_PAGE_SHIFT + int + default 12 + +config RISCV_PAGE_SHIFT + int + default 12 + config KASAN_SHADOW_OFFSET hex depends on KASAN_GENERIC diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index 57e887bfa34c..a8c59d80683c 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -12,7 +12,11 @@ #include #include -#define PAGE_SHIFT (12) +#define HW_PAGE_SHIFT CONFIG_RISCV_HW_PAGE_SHIFT +#define HW_PAGE_SIZE (_AC(1, UL) << HW_PAGE_SHIFT) +#define HW_PAGE_MASK (~(HW_PAGE_SIZE - 1)) + +#define PAGE_SHIFT CONFIG_RISCV_PAGE_SHIFT #define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT) #define PAGE_MASK (~(PAGE_SIZE - 1)) From patchwork Thu Nov 23 06:57:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13465845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AAE6AC61D85 for ; Thu, 23 Nov 2023 06:57:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jaUpQ/WJrSz2jsEiiTxhuRPPBeaz0YHOE/UWSC/LuPk=; b=WyswLNPlNupCdT gFyEYQaA1eljzbOwuGNpqIIdH8z4iX4kpy/ef+ufzoN5qPBdgBij/lM506Ma4xubhB2XlRsuJoHS8 f0Ph9h4ciTV5/zilgUbY1FWuRDE9bKLrX3lKFflRjTHfpJHF41t4jITXcJuXS+8hsX2Szeb1TGQn5 VTGa0fvjaEhRUnInDO0U3HNg6hKN1rUS4IFuhjo02ZsNFVH9YGT0Ap0Dsso+P+YltzXL2/czghxqh /KmyRDdiRHlVuCZiDF/dU2Lo9RUpPDFoPIUdPbWkqEC9EJWZtD/OmLrceFdbea0fYSZJDlAnIDtWZ fknrcVwOpg+f9X0Mj+Uw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r63el-003xIZ-1G; Thu, 23 Nov 2023 06:57:47 +0000 Received: from mail-pf1-x434.google.com ([2607:f8b0:4864:20::434]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r63eh-003xH5-2J for linux-riscv@lists.infradead.org; Thu, 23 Nov 2023 06:57:45 +0000 Received: by mail-pf1-x434.google.com with SMTP id d2e1a72fcca58-6cb749044a2so599632b3a.0 for ; Wed, 22 Nov 2023 22:57:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1700722662; x=1701327462; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1pQabAU2EIOA7IYsCP/RDHpXwqUYYVOXCKbRpkKm/sI=; b=F6bERXC4SfWoigX/XWC0O6jfkQpTbUU+xmX2xUmtqnKxYxPV70N1xqBlIdDSuUFvmb YeZNoYlx/Q4/NgIDr+wuQet6d9wwuff9xynLY+HRxEXhKkMnI0WzJNoYY8H4U6Jdh1Rb pmKHjDhfhDW+NVl61+SUFSSO+Rql7olf+9jev2fioATNM3ycjk5BXdarg6tT3PrpfDNn gFyw/jE44+nBhn8EM0suAVuE5lNZZam/IAbNRvEldX9nYdVBGPAWo6HlXBCtHP7Zzi2D Xp1C2u2bCo5J8ELUoPutKN6mH5JBVfVWa1CuJ0T7exL4N2ET6ouic94MDcMRgDkcjlJT Fc8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700722662; x=1701327462; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1pQabAU2EIOA7IYsCP/RDHpXwqUYYVOXCKbRpkKm/sI=; b=iJSTsvUytT3vPm8K04ymwp7+/HweKB12X0szDEL6BG7ILJs0Gs/t/zyqXFhy0i3TsK fb/6TKkpWQkpMl0Wv/rAWx0F/cgd+UPmt3rx3CkBKaLsi0/FEuBT5YB6xCVoFAalLNsy 8whJjzbZfMEScbjyJr0Dt9sPC2QLUkEPmOz1zd2CYOzG6IKLcC8TCnO8YR4FIpd86ZZ/ AkKqGGaRVAyNW/SZbw11sxXMsmqqgFeNdztCaldpcCgf8Z6CTyctjgOtuNSwwFVZ22qR OwR1J3R+WiiGoocN5U5/D9+YLGuUJ7r9fqFNHT/0LUG1i4bJlDm0xCQX+GjdZMdGb2G/ 2ZRg== X-Gm-Message-State: AOJu0YyjPAxGoONR9SfS7DgdH74MqaZDbSVN47oeH1ch52uyd7JOLWgt bgvf8gOg/7gfGmLwwcY7xEsbng== X-Google-Smtp-Source: AGHT+IEjrB3TX75kVy8EQWnz59Mr28Ta2QXhohdgO7PmW2QllvYs+v4WN85i+Msea7MqDkKVKQnXQg== X-Received: by 2002:a05:6a00:8d97:b0:68f:bb02:fdf with SMTP id im23-20020a056a008d9700b0068fbb020fdfmr5401752pfb.27.1700722661962; Wed, 22 Nov 2023 22:57:41 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w37-20020a634765000000b005bd2b3a03eesm615437pgk.6.2023.11.22.22.57.36 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Nov 2023 22:57:41 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: dengliang.1214@bytedance.com, xieyongji@bytedance.com, lihangjing@bytedance.com, songmuchun@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH V1 03/11] riscv: Adapt pte struct to gap between hw page and sw page Date: Thu, 23 Nov 2023 14:57:00 +0800 Message-Id: <20231123065708.91345-4-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231123065708.91345-1-luxu.kernel@bytedance.com> References: <20231123065708.91345-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_225743_756430_C25C0CD6 X-CRM114-Status: GOOD ( 20.23 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The pte_t struct maps a virtual page to a physical page, both of which are the concept of software page in view of kernel mm. In contrary, per page table entry refers to a single hardware page. When sw page is larger than hw page, the existing pte_t struct with only one page table entry can not represent a software page anymore. This commit extends pte_t struct to contain an array of page table entries. The pte_t struct now maps a software page to an exact number of hardware pages, the total size of which still matches sw page size. Signed-off-by: Xu Lu --- arch/riscv/include/asm/page.h | 7 ++- arch/riscv/include/asm/pgtable-64.h | 3 +- arch/riscv/include/asm/pgtable.h | 91 ++++++++++++++++++++++++++--- arch/riscv/kernel/efi.c | 2 +- arch/riscv/mm/hugetlbpage.c | 2 +- arch/riscv/mm/pageattr.c | 2 +- 6 files changed, 92 insertions(+), 15 deletions(-) diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index a8c59d80683c..cbaa7e027f9a 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -68,9 +68,11 @@ typedef struct { unsigned long pgd; } pgd_t; +#define PTES_PER_PAGE (1 << (PAGE_SHIFT - HW_PAGE_SHIFT)) + /* Page Table entry */ typedef struct { - unsigned long pte; + unsigned long ptes[PTES_PER_PAGE]; } pte_t; typedef struct { @@ -79,11 +81,10 @@ typedef struct { typedef struct page *pgtable_t; -#define pte_val(x) ((x).pte) +#define pte_val(x) ((x).ptes[0]) #define pgd_val(x) ((x).pgd) #define pgprot_val(x) ((x).pgprot) -#define __pte(x) ((pte_t) { (x) }) #define __pgd(x) ((pgd_t) { (x) }) #define __pgprot(x) ((pgprot_t) { (x) }) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 9a2c780a11e9..c08db54594a9 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -99,7 +99,8 @@ enum napot_cont_order { #define for_each_napot_order_rev(order) \ for (order = NAPOT_ORDER_MAX - 1; \ order >= NAPOT_CONT_ORDER_BASE; order--) -#define napot_cont_order(val) (__builtin_ctzl((val.pte >> _PAGE_PFN_SHIFT) << 1)) +#define napot_cont_order(val) \ + (__builtin_ctzl((pte_val(val) >> _PAGE_PFN_SHIFT) << 1)) #define napot_cont_shift(order) ((order) + PAGE_SHIFT) #define napot_cont_size(order) BIT(napot_cont_shift(order)) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 294044429e8e..342be2112fd2 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -212,6 +212,31 @@ extern pgd_t swapper_pg_dir[]; extern pgd_t trampoline_pg_dir[]; extern pgd_t early_pg_dir[]; +static __always_inline int __pte_present(unsigned long pteval) +{ + return (pteval & (_PAGE_PRESENT | _PAGE_PROT_NONE)); +} + +static __always_inline unsigned long __pte_napot(unsigned long pteval) +{ + return pteval & _PAGE_NAPOT; +} + +static inline pte_t __pte(unsigned long pteval) +{ + pte_t pte; + unsigned int i; + + for (i = 0; i < PTES_PER_PAGE; i++) { + pte.ptes[i] = pteval; + if (__pte_present(pteval) && !__pte_napot(pteval)) + pteval += 1 << _PAGE_PFN_SHIFT; + } + + return pte; +} +#define __pte __pte + #ifdef CONFIG_TRANSPARENT_HUGEPAGE static inline int pmd_present(pmd_t pmd) { @@ -300,7 +325,7 @@ static __always_inline bool has_svnapot(void) static inline unsigned long pte_napot(pte_t pte) { - return pte_val(pte) & _PAGE_NAPOT; + return __pte_napot(pte_val(pte)); } static inline pte_t pte_mknapot(pte_t pte, unsigned int order) @@ -350,7 +375,7 @@ static inline pte_t pfn_pte(unsigned long pfn, pgprot_t prot) static inline int pte_present(pte_t pte) { - return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)); + return __pte_present(pte_val(pte)); } static inline int pte_none(pte_t pte) @@ -439,6 +464,36 @@ static inline pte_t pte_mkhuge(pte_t pte) return pte; } +static inline pte_t ptep_get(pte_t *ptep) +{ + unsigned int i; + pte_t pte = *ptep; + + for (i = 0; i < PTES_PER_PAGE; i++) { + if (pte.ptes[i] & _PAGE_DIRTY) { + pte = pte_mkdirty(pte); + break; + } + } + for (i = 0; i < PTES_PER_PAGE; i++) { + if (pte.ptes[i] & _PAGE_ACCESSED) { + pte = pte_mkyoung(pte); + break; + } + } + + return pte; +} +#define ptep_get ptep_get + +static inline pte_t ptep_get_lockless(pte_t *ptep) +{ + unsigned long pteval = READ_ONCE(ptep->ptes[0]); + + return __pte(pteval); +} +#define ptep_get_lockless ptep_get_lockless + #ifdef CONFIG_NUMA_BALANCING /* * See the comment in include/asm-generic/pgtable.h @@ -526,6 +581,8 @@ static inline void __set_pte_at(pte_t *ptep, pte_t pteval) static inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr) { + unsigned int i; + page_table_check_ptes_set(mm, ptep, pteval, nr); for (;;) { @@ -533,7 +590,10 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, if (--nr == 0) break; ptep++; - pte_val(pteval) += 1 << _PAGE_PFN_SHIFT; + if (pte_present(pteval) && !pte_napot(pteval)) { + for (i = 0; i < PTES_PER_PAGE; i++) + pteval.ptes[i] += PTES_PER_PAGE << _PAGE_PFN_SHIFT; + } } } #define set_ptes set_ptes @@ -562,7 +622,11 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, pte_t *ptep) { - pte_t pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); + pte_t pte; + unsigned int i; + + for (i = 0; i < PTES_PER_PAGE; i++) + pte.ptes[i] = atomic_long_xchg((atomic_long_t *)(&ptep->ptes[i]), 0); page_table_check_pte_clear(mm, pte); @@ -574,16 +638,27 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - if (!pte_young(*ptep)) + int ret = 0; + unsigned int i; + + if (!pte_young(ptep_get(ptep))) return 0; - return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep)); + + for (i = 0; i < PTES_PER_PAGE; i++) + ret |= test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &ptep->ptes[i]); + + return ret; } #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) { - atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); + unsigned int i; + + for (i = 0; i < PTES_PER_PAGE; i++) + atomic_long_and(~(unsigned long)_PAGE_WRITE, + (atomic_long_t *)(&ptep->ptes[i])); } #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH @@ -829,7 +904,7 @@ extern pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, ((offset) << __SWP_OFFSET_SHIFT) }) #define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) }) -#define __swp_entry_to_pte(x) ((pte_t) { (x).val }) +#define __swp_entry_to_pte(x) __pte((x).val) static inline int pte_swp_exclusive(pte_t pte) { diff --git a/arch/riscv/kernel/efi.c b/arch/riscv/kernel/efi.c index aa6209a74c83..b64bf1624a05 100644 --- a/arch/riscv/kernel/efi.c +++ b/arch/riscv/kernel/efi.c @@ -60,7 +60,7 @@ int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md) static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) { efi_memory_desc_t *md = data; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = ptep_get(ptep); unsigned long val; if (md->attribute & EFI_MEMORY_RO) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index d7cf8e2d3c5b..67fd71c36853 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -293,7 +293,7 @@ void huge_pte_clear(struct mm_struct *mm, pte_t *ptep, unsigned long sz) { - pte_t pte = READ_ONCE(*ptep); + pte_t pte = ptep_get(ptep); int i, pte_num; if (!pte_napot(pte)) { diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c index fc5fc4f785c4..b8e30df2e7df 100644 --- a/arch/riscv/mm/pageattr.c +++ b/arch/riscv/mm/pageattr.c @@ -68,7 +68,7 @@ static int pageattr_pmd_entry(pmd_t *pmd, unsigned long addr, static int pageattr_pte_entry(pte_t *pte, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pte_t val = READ_ONCE(*pte); + pte_t val = ptep_get(pte); val = __pte(set_pageattr_masks(pte_val(val), walk)); set_pte(pte, val); From patchwork Thu Nov 23 06:57:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13465846 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 67623C5AD4C for ; Thu, 23 Nov 2023 06:57:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=XmPBk2Kh84SW7wzL34Oy4ZNf2grCfUw+0sSuU474vA8=; b=2W2XSD1jyX5zcb 54al64+AmkX4+QQLx0ZF8i4Cpk8FFiYXwSqph9mi/fmn35cVuQhzXI3z2Lgd4tyIf9xPB4NHMvduG tNqE+N/TnANXrhCLvGB9kKdBUQUd5H/IQttYPpvlE2m86xLJ76Fnaq3L4wNYh621Yiuivtc64hflF eXn69PWyjxn9jQFLl478+hu1cQYTYuq/WwsSGUofsC78osNLnkK7IRxDhrzLuwFjRJsE2bn8D3hud lWTWZJNACvsQXczeMTDtJi5OKQFOaXdPQWWYzhYdM7fr/VBhw6wfoAO/ID00fv0QCNPbNE4PBznLN qUBlGkBJnK5rIrlB+5Gw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r63es-003xLA-0i; Thu, 23 Nov 2023 06:57:54 +0000 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r63eo-003xJ5-0r for linux-riscv@lists.infradead.org; Thu, 23 Nov 2023 06:57:52 +0000 Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-6cbd24d9557so429766b3a.1 for ; Wed, 22 Nov 2023 22:57:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1700722667; x=1701327467; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FqqTxLyku7Pb+ctpWN2ROyQtTus2hT3Qjo5g/GoNKE8=; b=lFNDpdKLNs/WqKns5cv0TGag+7MCfIAsJ02SVbmQtKYZwqvAXb7ee3nk/boRcVtvYh IxXUA5r7QoKdMipVdPdsMKGp+clSg3D+Vzf6+GrzBcywdKnaBsXQISfmR2KX02AVvtDd pmppJb27qwn9QZTdXUjtWqT38y9j5rMr5rxTe5V6by3575PNL1VoyZmqgdOpkDuPMkDC vkLSijSGc2y9AoT4wpNDhVC1sn/R9zJ+z8nrRRyhIoYgke7q6O649urUDQJqR+Zot2I0 y1IwChIdewl9cG2AdEAct0yd49KBSw/lEy/MSbWA+mobG1fC3pAYT2mHsWBsUFe3Xcc5 9k1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700722667; x=1701327467; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FqqTxLyku7Pb+ctpWN2ROyQtTus2hT3Qjo5g/GoNKE8=; b=scGy26weIRIlVLFfblYV9fcT1TemkZ00X+BGbCO69C264WjhGwUAy4Ls9z4X9ZTSlG K2rm8Ny+dvRVA4gptkyxHmYPiL8t1iECaEHOqnX7fj85VAWxk8HwQBK7ReiF9ruflSZj LLgZ6hUa13nJJq+MrtwfZEzl/t32mQakOq0FzVo+Fk582lCjZR0d3+nBMEehbAMfSVCf QLD7xdpmZvDhLnhCh3713rdjb/MW4hqFoB4yjH+SvMXuJh+tU/FPnlltpMkNCpFKtPvl oySYg1IYny9NAMW0T//ptVyga1ly3xv0deaj28IyydYj4c3pgWOZLum95rksCZmXz+oa Hrmw== X-Gm-Message-State: AOJu0YxTIMtSZPYtAcm6g8HgVsBKm+wvI2jYikyXL3yHwClb8abDEfyo CtHDfwH0hAbc42xRxobuHlHgDA== X-Google-Smtp-Source: AGHT+IFPkQpg+ey3ertHh/TcNBgUDX7fnhdQHqcFtkkyI3tzskgpcz/a4p2oxDN3tA6VRXfdP6SYzg== X-Received: by 2002:a05:6a00:a11:b0:68f:c078:b0c9 with SMTP id p17-20020a056a000a1100b0068fc078b0c9mr3267435pfh.11.1700722667294; Wed, 22 Nov 2023 22:57:47 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w37-20020a634765000000b005bd2b3a03eesm615437pgk.6.2023.11.22.22.57.42 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Nov 2023 22:57:46 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: dengliang.1214@bytedance.com, xieyongji@bytedance.com, lihangjing@bytedance.com, songmuchun@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH V1 04/11] riscv: Adapt pte operations to gap between hw page and sw page Date: Thu, 23 Nov 2023 14:57:01 +0800 Message-Id: <20231123065708.91345-5-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231123065708.91345-1-luxu.kernel@bytedance.com> References: <20231123065708.91345-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_225750_312013_32A9C88D X-CRM114-Status: GOOD ( 16.62 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org MMU handles pages at a granularity of hardware page. That is, for 4K MMU, the pfn decoded from page table entry will be regarded as 4K page frame number, no matter how large the software page is. Thus, page table entries should always be encoded at the granularity of hardware page. This commit makes pte operations aware of the gap between hw page and sw page. All pte operations now configure page table entries via hardware page frame number. Signed-off-by: Xu Lu --- arch/riscv/include/asm/page.h | 3 +++ arch/riscv/include/asm/pgalloc.h | 21 ++++++++++----- arch/riscv/include/asm/pgtable-32.h | 2 +- arch/riscv/include/asm/pgtable-64.h | 40 ++++++++++++++++++----------- arch/riscv/include/asm/pgtable.h | 19 +++++++------- arch/riscv/mm/init.c | 18 ++++++------- 6 files changed, 62 insertions(+), 41 deletions(-) diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index cbaa7e027f9a..12f2e73ed55b 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -177,6 +177,9 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x); #define __pa(x) __virt_to_phys((unsigned long)(x)) #define __va(x) ((void *)__pa_to_va_nodebug((phys_addr_t)(x))) +#define pfn_to_hwpfn(pfn) (pfn << (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define hwpfn_to_pfn(hwpfn) (hwpfn >> (PAGE_SHIFT - HW_PAGE_SHIFT)) + #define phys_to_pfn(phys) (PFN_DOWN(phys)) #define pfn_to_phys(pfn) (PFN_PHYS(pfn)) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index d169a4f41a2e..eab75d5f7093 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -19,32 +19,36 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) { unsigned long pfn = virt_to_pfn(pte); + unsigned long hwpfn = pfn_to_hwpfn(pfn); - set_pmd(pmd, __pmd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + set_pmd(pmd, __pmd((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t pte) { unsigned long pfn = virt_to_pfn(page_address(pte)); + unsigned long hwpfn = pfn_to_hwpfn(pfn); - set_pmd(pmd, __pmd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + set_pmd(pmd, __pmd((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } #ifndef __PAGETABLE_PMD_FOLDED static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) { unsigned long pfn = virt_to_pfn(pmd); + unsigned long hwpfn = pfn_to_hwpfn(pfn); - set_pud(pud, __pud((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + set_pud(pud, __pud((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4d, pud_t *pud) { if (pgtable_l4_enabled) { unsigned long pfn = virt_to_pfn(pud); + unsigned long hwpfn = pfn_to_hwpfn(pfn); - set_p4d(p4d, __p4d((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + set_p4d(p4d, __p4d((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } } @@ -53,9 +57,10 @@ static inline void p4d_populate_safe(struct mm_struct *mm, p4d_t *p4d, { if (pgtable_l4_enabled) { unsigned long pfn = virt_to_pfn(pud); + unsigned long hwpfn = pfn_to_hwpfn(pfn); set_p4d_safe(p4d, - __p4d((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + __p4d((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } } @@ -63,8 +68,9 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4d) { if (pgtable_l5_enabled) { unsigned long pfn = virt_to_pfn(p4d); + unsigned long hwpfn = pfn_to_hwpfn(pfn); - set_pgd(pgd, __pgd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + set_pgd(pgd, __pgd((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } } @@ -73,9 +79,10 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, { if (pgtable_l5_enabled) { unsigned long pfn = virt_to_pfn(p4d); + unsigned long hwpfn = pfn_to_hwpfn(pfn); set_pgd_safe(pgd, - __pgd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + __pgd((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } } diff --git a/arch/riscv/include/asm/pgtable-32.h b/arch/riscv/include/asm/pgtable-32.h index 00f3369570a8..dec436e146ae 100644 --- a/arch/riscv/include/asm/pgtable-32.h +++ b/arch/riscv/include/asm/pgtable-32.h @@ -20,7 +20,7 @@ /* * rv32 PTE format: * | XLEN-1 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 - * PFN reserved for SW D A G U X W R V + * HW_PFN reserved for SW D A G U X W R V */ #define _PAGE_PFN_MASK GENMASK(31, 10) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index c08db54594a9..1926727698fc 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -50,7 +50,7 @@ typedef struct { #define p4d_val(x) ((x).p4d) #define __p4d(x) ((p4d_t) { (x) }) -#define PTRS_PER_P4D (PAGE_SIZE / sizeof(p4d_t)) +#define PTRS_PER_P4D (HW_PAGE_SIZE / sizeof(p4d_t)) /* Page Upper Directory entry */ typedef struct { @@ -59,7 +59,7 @@ typedef struct { #define pud_val(x) ((x).pud) #define __pud(x) ((pud_t) { (x) }) -#define PTRS_PER_PUD (PAGE_SIZE / sizeof(pud_t)) +#define PTRS_PER_PUD (HW_PAGE_SIZE / sizeof(pud_t)) /* Page Middle Directory entry */ typedef struct { @@ -69,12 +69,12 @@ typedef struct { #define pmd_val(x) ((x).pmd) #define __pmd(x) ((pmd_t) { (x) }) -#define PTRS_PER_PMD (PAGE_SIZE / sizeof(pmd_t)) +#define PTRS_PER_PMD (HW_PAGE_SIZE / sizeof(pmd_t)) /* * rv64 PTE format: * | 63 | 62 61 | 60 54 | 53 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 - * N MT RSV PFN reserved for SW D A G U X W R V + * N MT RSV HW_PFN reserved for SW D A G U X W R V */ #define _PAGE_PFN_MASK GENMASK(53, 10) @@ -94,13 +94,23 @@ enum napot_cont_order { NAPOT_ORDER_MAX, }; -#define for_each_napot_order(order) \ - for (order = NAPOT_CONT_ORDER_BASE; order < NAPOT_ORDER_MAX; order++) -#define for_each_napot_order_rev(order) \ - for (order = NAPOT_ORDER_MAX - 1; \ - order >= NAPOT_CONT_ORDER_BASE; order--) -#define napot_cont_order(val) \ - (__builtin_ctzl((pte_val(val) >> _PAGE_PFN_SHIFT) << 1)) +#define NAPOT_PAGE_ORDER_BASE \ + ((NAPOT_CONT_ORDER_BASE >= (PAGE_SHIFT - HW_PAGE_SHIFT)) ? \ + (NAPOT_CONT_ORDER_BASE - (PAGE_SHIFT - HW_PAGE_SHIFT)) : 1) +#define NAPOT_PAGE_ORDER_MAX \ + ((NAPOT_ORDER_MAX > (PAGE_SHIFT - HW_PAGE_SHIFT)) ? \ + (NAPOT_ORDER_MAX - (PAGE_SHIFT - HW_PAGE_SHIFT)) : \ + NAPOT_PAGE_ORDER_BASE) + +#define for_each_napot_order(order) \ + for (order = NAPOT_PAGE_ORDER_BASE; \ + order < NAPOT_PAGE_ORDER_MAX; order++) +#define for_each_napot_order_rev(order) \ + for (order = NAPOT_PAGE_ORDER_MAX - 1; \ + order >= NAPOT_PAGE_ORDER_BASE; order--) +#define napot_cont_order(val) \ + (__builtin_ctzl((pte_val(val) >> _PAGE_PFN_SHIFT) << 1) \ + - (PAGE_SHIFT - HW_PAGE_SHIFT)) #define napot_cont_shift(order) ((order) + PAGE_SHIFT) #define napot_cont_size(order) BIT(napot_cont_shift(order)) @@ -108,7 +118,7 @@ enum napot_cont_order { #define napot_pte_num(order) BIT(order) #ifdef CONFIG_RISCV_ISA_SVNAPOT -#define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_CONT_ORDER_BASE)) +#define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_PAGE_ORDER_BASE)) #else #define HUGE_MAX_HSTATE 2 #endif @@ -213,7 +223,7 @@ static inline void pud_clear(pud_t *pudp) static inline pud_t pfn_pud(unsigned long pfn, pgprot_t prot) { - return __pud((pfn << _PAGE_PFN_SHIFT) | pgprot_val(prot)); + return __pud((pfn_to_hwpfn(pfn) << _PAGE_PFN_SHIFT) | pgprot_val(prot)); } static inline unsigned long _pud_pfn(pud_t pud) @@ -257,7 +267,7 @@ static inline pmd_t pfn_pmd(unsigned long pfn, pgprot_t prot) ALT_THEAD_PMA(prot_val); - return __pmd((pfn << _PAGE_PFN_SHIFT) | prot_val); + return __pmd((pfn_to_hwpfn(pfn) << _PAGE_PFN_SHIFT) | prot_val); } static inline unsigned long _pmd_pfn(pmd_t pmd) @@ -316,7 +326,7 @@ static inline void p4d_clear(p4d_t *p4d) static inline p4d_t pfn_p4d(unsigned long pfn, pgprot_t prot) { - return __p4d((pfn << _PAGE_PFN_SHIFT) | pgprot_val(prot)); + return __p4d((pfn_to_hwpfn(pfn) << _PAGE_PFN_SHIFT) | pgprot_val(prot)); } static inline unsigned long _p4d_pfn(p4d_t p4d) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 342be2112fd2..d50c4588c1ed 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -26,9 +26,9 @@ #endif /* Number of entries in the page global directory */ -#define PTRS_PER_PGD (PAGE_SIZE / sizeof(pgd_t)) +#define PTRS_PER_PGD (HW_PAGE_SIZE / sizeof(pgd_t)) /* Number of entries in the page table */ -#define PTRS_PER_PTE (PAGE_SIZE / sizeof(pte_t)) +#define PTRS_PER_PTE (HW_PAGE_SIZE / sizeof(pte_t)) /* * Half of the kernel address space (1/4 of the entries of the page global @@ -118,7 +118,8 @@ #include #include -#define __page_val_to_pfn(_val) (((_val) & _PAGE_PFN_MASK) >> _PAGE_PFN_SHIFT) +#define __page_val_to_hwpfn(_val) (((_val) & _PAGE_PFN_MASK) >> _PAGE_PFN_SHIFT) +#define __page_val_to_pfn(_val) hwpfn_to_pfn(__page_val_to_hwpfn(_val)) #ifdef CONFIG_64BIT #include @@ -287,7 +288,7 @@ static inline pgd_t pfn_pgd(unsigned long pfn, pgprot_t prot) ALT_THEAD_PMA(prot_val); - return __pgd((pfn << _PAGE_PFN_SHIFT) | prot_val); + return __pgd((pfn_to_hwpfn(pfn) << _PAGE_PFN_SHIFT) | prot_val); } static inline unsigned long _pgd_pfn(pgd_t pgd) @@ -351,12 +352,12 @@ static inline unsigned long pte_napot(pte_t pte) /* Yields the page frame number (PFN) of a page table entry */ static inline unsigned long pte_pfn(pte_t pte) { - unsigned long res = __page_val_to_pfn(pte_val(pte)); + unsigned long res = __page_val_to_hwpfn(pte_val(pte)); if (has_svnapot() && pte_napot(pte)) res = res & (res - 1UL); - return res; + return hwpfn_to_pfn(res); } #define pte_page(x) pfn_to_page(pte_pfn(x)) @@ -368,7 +369,7 @@ static inline pte_t pfn_pte(unsigned long pfn, pgprot_t prot) ALT_THEAD_PMA(prot_val); - return __pte((pfn << _PAGE_PFN_SHIFT) | prot_val); + return __pte((pfn_to_hwpfn(pfn) << _PAGE_PFN_SHIFT) | prot_val); } #define mk_pte(page, prot) pfn_pte(page_to_pfn(page), prot) @@ -723,14 +724,14 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) return __pmd(pmd_val(pmd) & ~(_PAGE_PRESENT|_PAGE_PROT_NONE)); } -#define __pmd_to_phys(pmd) (__page_val_to_pfn(pmd_val(pmd)) << PAGE_SHIFT) +#define __pmd_to_phys(pmd) (__page_val_to_hwpfn(pmd_val(pmd)) << HW_PAGE_SHIFT) static inline unsigned long pmd_pfn(pmd_t pmd) { return ((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT); } -#define __pud_to_phys(pud) (__page_val_to_pfn(pud_val(pud)) << PAGE_SHIFT) +#define __pud_to_phys(pud) (__page_val_to_hwpfn(pud_val(pud)) << HW_PAGE_SHIFT) static inline unsigned long pud_pfn(pud_t pud) { diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 2e011cbddf3a..a768b2b3ff05 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -466,7 +466,7 @@ static void __init create_pmd_mapping(pmd_t *pmdp, pte_phys = pt_ops.alloc_pte(va); pmdp[pmd_idx] = pfn_pmd(PFN_DOWN(pte_phys), PAGE_TABLE); ptep = pt_ops.get_pte_virt(pte_phys); - memset(ptep, 0, PAGE_SIZE); + memset(ptep, 0, PTRS_PER_PTE * sizeof(pte_t)); } else { pte_phys = PFN_PHYS(_pmd_pfn(pmdp[pmd_idx])); ptep = pt_ops.get_pte_virt(pte_phys); @@ -569,7 +569,7 @@ static void __init create_pud_mapping(pud_t *pudp, next_phys = pt_ops.alloc_pmd(va); pudp[pud_index] = pfn_pud(PFN_DOWN(next_phys), PAGE_TABLE); nextp = pt_ops.get_pmd_virt(next_phys); - memset(nextp, 0, PAGE_SIZE); + memset(nextp, 0, PTRS_PER_PMD * sizeof(pmd_t)); } else { next_phys = PFN_PHYS(_pud_pfn(pudp[pud_index])); nextp = pt_ops.get_pmd_virt(next_phys); @@ -596,7 +596,7 @@ static void __init create_p4d_mapping(p4d_t *p4dp, next_phys = pt_ops.alloc_pud(va); p4dp[p4d_index] = pfn_p4d(PFN_DOWN(next_phys), PAGE_TABLE); nextp = pt_ops.get_pud_virt(next_phys); - memset(nextp, 0, PAGE_SIZE); + memset(nextp, 0, PTRS_PER_PUD * sizeof(pud_t)); } else { next_phys = PFN_PHYS(_p4d_pfn(p4dp[p4d_index])); nextp = pt_ops.get_pud_virt(next_phys); @@ -654,7 +654,7 @@ void __init create_pgd_mapping(pgd_t *pgdp, next_phys = alloc_pgd_next(va); pgdp[pgd_idx] = pfn_pgd(PFN_DOWN(next_phys), PAGE_TABLE); nextp = get_pgd_next_virt(next_phys); - memset(nextp, 0, PAGE_SIZE); + memset(nextp, 0, PTRS_PER_P4D * sizeof(p4d_t)); } else { next_phys = PFN_PHYS(_pgd_pfn(pgdp[pgd_idx])); nextp = get_pgd_next_virt(next_phys); @@ -815,16 +815,16 @@ static __init void set_satp_mode(uintptr_t dtb_pa) if (hw_satp != identity_satp) { if (pgtable_l5_enabled) { disable_pgtable_l5(); - memset(early_pg_dir, 0, PAGE_SIZE); + memset(early_pg_dir, 0, PTRS_PER_PGD * sizeof(pgd_t)); goto retry; } disable_pgtable_l4(); } - memset(early_pg_dir, 0, PAGE_SIZE); - memset(early_p4d, 0, PAGE_SIZE); - memset(early_pud, 0, PAGE_SIZE); - memset(early_pmd, 0, PAGE_SIZE); + memset(early_pg_dir, 0, PTRS_PER_PGD * sizeof(pgd_t)); + memset(early_p4d, 0, PTRS_PER_P4D * sizeof(p4d_t)); + memset(early_pud, 0, PTRS_PER_PUD * sizeof(pud_t)); + memset(early_pmd, 0, PTRS_PER_PMD * sizeof(pmd_t)); } #endif From patchwork Thu Nov 23 06:57:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13465847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EAD0CC61D85 for ; Thu, 23 Nov 2023 06:58:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kcSE2WOBeIhXGxfpYNP3UP1XF+EnkY0Fyp7tQyVcBfI=; b=HeHrtN/hg1oSUJ SXkQWE3n085JYXNAMGDiXZ09fNRU7Os9ZC4evynBTNCWeOmj/TGZYJLklp07/EcqK4/Q12qKiDsaD UK03C/297+UaB4K6E/j3xBopMS/I/xpuBfE1zCfWNPkr12aZ2jTsOLYw59wQp+K29hdhKP1tpJIKR t06aXH8c/H43l7/kEnbPb9QpyKHjXMRWzaRICpELnGXZrVqOpKe/nLc21CfNVNhX3eKZ6zfZqGALg qfi2JOwzXAl5exXAfMvYCn1arln4FYLU3lUoDMMARiopNgCVH7eNGHi8C2hL44jLwgBM1DG4Sowcq CrM+6xaenxVqANDhXELA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r63ew-003xMe-0H; Thu, 23 Nov 2023 06:57:58 +0000 Received: from mail-oi1-x22f.google.com ([2607:f8b0:4864:20::22f]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r63et-003xL0-04 for linux-riscv@lists.infradead.org; Thu, 23 Nov 2023 06:57:56 +0000 Received: by mail-oi1-x22f.google.com with SMTP id 5614622812f47-3b83fc26e4cso383935b6e.2 for ; Wed, 22 Nov 2023 22:57:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1700722673; x=1701327473; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EBzEAwR1u2ognGkokbPIcjTus0Wt1Liku1PeiEAYDPA=; b=Zz+zskETm3fePiSC2+4mE5StmUXIsmN3Cyj50LqoMgL0Hyn4v/dpEtl6Eq335wafgs wjuLjhPNU1gt+THNalTpUMaQgM1HXXOgw72vVceshS8v3z2fEArYknoQQjzaAQXDTgNq RRPGns7b3men3gvX7vZS437hE81+ZEHU64Tb9TJK8FEgT7IJIutjox9SvNYf+526AZiK Jpph6Cv0Oo8NaYKi3RCF2wiONv5yQwyhktuwzwKV3XP1612x27Mq3fwCO0Wl296xFR1w //svaB2I4dEj30me3qySC6iqV/a17v8NpmnbUV5SWe5LA0APXQVgW1Pwwv7NMP+8QDF8 7Wbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700722673; x=1701327473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EBzEAwR1u2ognGkokbPIcjTus0Wt1Liku1PeiEAYDPA=; b=DfXgEwJKyvY1s/qKUpzvExaTDcK1epiHqtAqDT2m0+9T8HRa8T3VcHclPR0kqa61tg GdIEip+hTLIfGXLnrD/ECOF2M/ynPCgI7YM2rqqEz/9V5V3Je2Fo5rJoccHoKb17/LXh j8g/bOxHSTLrSWuJ1yIrPZuglqfCvznMHfPO6Trwku/8wK9isbulDYHJanPZNluLAh5u nBpVkY6HWxc8YwKy+yQ75uyXHnjugVK7BATK1qY72vsrNiEwhuiQ1uKXyeWeyMZzXgeM u3LaQrv+2hP9t5JXCeN1o1Hl3bdtaTkofmwflG1hV8OWGZSSg7hyzqGQmiOkf4rvb9N/ tpPA== X-Gm-Message-State: AOJu0YzE9fhU68NXi41/XDOfT87v0jUiBoBUOXhYUPRalEs1xhKFo7rf GLg1mXji8muXIrBwC6VmVCMVzQ== X-Google-Smtp-Source: AGHT+IGHKuk0EJS7EQTCzidInz2QrgpsRuY4bxBytkNm9OOr3Q4F5OcNYuewzXuJd5Dzw7DfQuJfmg== X-Received: by 2002:a54:4885:0:b0:3af:b6d3:cda0 with SMTP id r5-20020a544885000000b003afb6d3cda0mr5487295oic.40.1700722673045; Wed, 22 Nov 2023 22:57:53 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w37-20020a634765000000b005bd2b3a03eesm615437pgk.6.2023.11.22.22.57.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Nov 2023 22:57:52 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: dengliang.1214@bytedance.com, xieyongji@bytedance.com, lihangjing@bytedance.com, songmuchun@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH V1 05/11] riscv: Decouple pmd operations and pte operations Date: Thu, 23 Nov 2023 14:57:02 +0800 Message-Id: <20231123065708.91345-6-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231123065708.91345-1-luxu.kernel@bytedance.com> References: <20231123065708.91345-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_225755_059983_FF5086C0 X-CRM114-Status: GOOD ( 13.05 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Existing pmd operations are usually implemented via pte operations. For example, the pmd_mkdirty function, which is used to mark a pmd_t struct as dirty, will transfer pmd_t struct to pte_t struct via pmd_pte first, mark the generated pte_t as dirty then, and finally transfer it back to pmd_t struct via pte_pmd function. Such implementation introduces unnecessary overhead of struct transferring. Also, Now that pte_t struct is a number of page table entries, which can be larger than pmd_t struct, functions like set_pmd_at implemented via set_pte_at will cause write amplifications. This commit decouples pmd operations and pte operations. Pmd operations are now implemented independently of pte operations. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable-64.h | 6 ++ arch/riscv/include/asm/pgtable.h | 124 +++++++++++++++++++++------- include/asm-generic/pgtable-nopmd.h | 1 + include/linux/pgtable.h | 6 ++ 4 files changed, 108 insertions(+), 29 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 1926727698fc..95e785f2160c 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -206,6 +206,12 @@ static inline int pud_leaf(pud_t pud) return pud_present(pud) && (pud_val(pud) & _PAGE_LEAF); } +#define pud_exec pud_exec +static inline int pud_exec(pud_t pud) +{ + return pud_val(pud) & _PAGE_EXEC; +} + static inline int pud_user(pud_t pud) { return pud_val(pud) & _PAGE_USER; diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index d50c4588c1ed..9f81fe046cb8 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -272,6 +272,18 @@ static inline int pmd_leaf(pmd_t pmd) return pmd_present(pmd) && (pmd_val(pmd) & _PAGE_LEAF); } +#define pmd_exec pmd_exec +static inline int pmd_exec(pmd_t pmd) +{ + return pmd_val(pmd) & _PAGE_EXEC; +} + +#define __HAVE_ARCH_PMD_SAME +static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b) +{ + return pmd_val(pmd_a) == pmd_val(pmd_b); +} + static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) { *pmdp = pmd; @@ -506,7 +518,7 @@ static inline int pte_protnone(pte_t pte) static inline int pmd_protnone(pmd_t pmd) { - return pte_protnone(pmd_pte(pmd)); + return (pmd_val(pmd) & (_PAGE_PRESENT | _PAGE_PROT_NONE)) == _PAGE_PROT_NONE; } #endif @@ -740,73 +752,95 @@ static inline unsigned long pud_pfn(pud_t pud) static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) { - return pte_pmd(pte_modify(pmd_pte(pmd), newprot)); + unsigned long newprot_val = pgprot_val(newprot); + + ALT_THEAD_PMA(newprot_val); + + return __pmd((pmd_val(pmd) & _PAGE_CHG_MASK) | newprot_val); } #define pmd_write pmd_write static inline int pmd_write(pmd_t pmd) { - return pte_write(pmd_pte(pmd)); + return pmd_val(pmd) & _PAGE_WRITE; } static inline int pmd_dirty(pmd_t pmd) { - return pte_dirty(pmd_pte(pmd)); + return pmd_val(pmd) & _PAGE_DIRTY; } #define pmd_young pmd_young static inline int pmd_young(pmd_t pmd) { - return pte_young(pmd_pte(pmd)); + return pmd_val(pmd) & _PAGE_ACCESSED; } static inline int pmd_user(pmd_t pmd) { - return pte_user(pmd_pte(pmd)); + return pmd_val(pmd) & _PAGE_USER; } static inline pmd_t pmd_mkold(pmd_t pmd) { - return pte_pmd(pte_mkold(pmd_pte(pmd))); + return __pmd(pmd_val(pmd) & ~(_PAGE_ACCESSED)); } static inline pmd_t pmd_mkyoung(pmd_t pmd) { - return pte_pmd(pte_mkyoung(pmd_pte(pmd))); + return __pmd(pmd_val(pmd) | _PAGE_ACCESSED); } static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) { - return pte_pmd(pte_mkwrite_novma(pmd_pte(pmd))); + return __pmd(pmd_val(pmd) | _PAGE_WRITE); } static inline pmd_t pmd_wrprotect(pmd_t pmd) { - return pte_pmd(pte_wrprotect(pmd_pte(pmd))); + return __pmd(pmd_val(pmd) & (~_PAGE_WRITE)); } static inline pmd_t pmd_mkclean(pmd_t pmd) { - return pte_pmd(pte_mkclean(pmd_pte(pmd))); + return __pmd(pmd_val(pmd) & (~_PAGE_DIRTY)); } static inline pmd_t pmd_mkdirty(pmd_t pmd) { - return pte_pmd(pte_mkdirty(pmd_pte(pmd))); + return __pmd(pmd_val(pmd) | _PAGE_DIRTY); +} + +#define pmd_accessible(mm, pmd) ((void)(pmd), 1) + +static inline void __set_pmd_at(pmd_t *pmdp, pmd_t pmd) +{ + if (pmd_present(pmd) && pmd_exec(pmd)) + flush_icache_pte(pmd_pte(pmd)); + + set_pmd(pmdp, pmd); } static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd) { page_table_check_pmd_set(mm, pmdp, pmd); - return __set_pte_at((pte_t *)pmdp, pmd_pte(pmd)); + return __set_pmd_at(pmdp, pmd); +} + +static inline void __set_pud_at(pud_t *pudp, pud_t pud) +{ + if (pud_present(pud) && pud_exec(pud)) + flush_icache_pte(pud_pte(pud)); + + set_pud(pudp, pud); } static inline void set_pud_at(struct mm_struct *mm, unsigned long addr, pud_t *pudp, pud_t pud) { page_table_check_pud_set(mm, pudp, pud); - return __set_pte_at((pte_t *)pudp, pud_pte(pud)); + return __set_pud_at(pudp, pud); } #ifdef CONFIG_PAGE_TABLE_CHECK @@ -826,25 +860,64 @@ static inline bool pud_user_accessible_page(pud_t pud) } #endif -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -static inline int pmd_trans_huge(pmd_t pmd) -{ - return pmd_leaf(pmd); -} - #define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS static inline int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty) { - return ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); + if (!pmd_same(*pmdp, entry)) + set_pmd_at(vma->vm_mm, address, pmdp, entry); + /* + * update_mmu_cache will unconditionally execute, handling both + * the case that the PMD changed and the spurious fault case. + */ + return true; +} + +#define __HAVE_ARCH_PMDP_GET_AND_CLEAR +static inline pmd_t pmdp_get_and_clear(struct mm_struct *mm, + unsigned long address, pmd_t *pmdp) +{ + pmd_t pmd = __pmd(atomic_long_xchg((atomic_long_t *)pmdp, 0)); + + page_table_check_pmd_clear(mm, pmd); + + return pmd; +} + +#define __HAVE_ARCH_PMDP_SET_WRPROTECT +static inline void pmdp_set_wrprotect(struct mm_struct *mm, + unsigned long address, pmd_t *pmdp) +{ + atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)pmdp); +} + +#define __HAVE_ARCH_PMDP_CLEAR_FLUSH +static inline pmd_t pmdp_clear_flush(struct vm_area_struct *vma, + unsigned long address, pmd_t *pmdp) +{ + struct mm_struct *mm = (vma)->vm_mm; + pmd_t pmd = pmdp_get_and_clear(mm, address, pmdp); + + if (pmd_accessible(mm, pmd)) + flush_tlb_page(vma, address); + + return pmd; } #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - return ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); + if (!pmd_young(*pmdp)) + return 0; + return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pmd_val(*pmdp)); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static inline int pmd_trans_huge(pmd_t pmd) +{ + return pmd_leaf(pmd); } #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR @@ -858,13 +931,6 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, return pmd; } -#define __HAVE_ARCH_PMDP_SET_WRPROTECT -static inline void pmdp_set_wrprotect(struct mm_struct *mm, - unsigned long address, pmd_t *pmdp) -{ - ptep_set_wrprotect(mm, address, (pte_t *)pmdp); -} - #define pmdp_establish pmdp_establish static inline pmd_t pmdp_establish(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t pmd) diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h index 8ffd64e7a24c..acef201b29f5 100644 --- a/include/asm-generic/pgtable-nopmd.h +++ b/include/asm-generic/pgtable-nopmd.h @@ -32,6 +32,7 @@ static inline int pud_bad(pud_t pud) { return 0; } static inline int pud_present(pud_t pud) { return 1; } static inline int pud_user(pud_t pud) { return 0; } static inline int pud_leaf(pud_t pud) { return 0; } +static inline int pud_exec(pud_t pud) { return 0; } static inline void pud_clear(pud_t *pud) { } #define pmd_ERROR(pmd) (pud_ERROR((pmd).pud)) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index af7639c3b0a3..b8d6e39fefc2 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1630,9 +1630,15 @@ typedef unsigned int pgtbl_mod_mask; #ifndef pud_leaf #define pud_leaf(x) 0 #endif +#ifndef pud_exec +#define pud_exec(x) 0 +#endif #ifndef pmd_leaf #define pmd_leaf(x) 0 #endif +#ifndef pmd_exec +#define pmd_exec(x) 0 +#endif #ifndef pgd_leaf_size #define pgd_leaf_size(x) (1ULL << PGDIR_SHIFT) From patchwork Thu Nov 23 06:57:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13465848 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C14A9C61D85 for ; Thu, 23 Nov 2023 06:58:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=atfAOuwG5T2uQN564PVTXBEzUW9CJInCf2sEQbCQN7o=; b=W8iaijJXYTI0mT pCdvxvlAzP0G2ZZOWtxu160SrQN9KZpuQZWbr2w/UklSGwF2+YZ8jX8RE4BPCz7F3dXQBktrVeHc0 fcIZUYnuKKNxmEPteh9pIO9LaboYlkHpePseSvQPM4EuQVOaSJYbLtjvWIW7HOZJdJdOFbf5bhsaF TzUZiob/Vme6C4UmhOKMTXfmOineTkvGn2JfRkdH9lVD6R5iJ9mm6HIfa6EeMDDWApQTxAviD6Is+ eO0WlG6Ca4ZXuqr0TqhMcwSxudrTpWfkodGPfwx1ly0+VXs3uZlJDW+n3h8fDV4Kuk1Y+AGUv5xCI lOLWEc0bUZfhDbWlMgGQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r63f0-003xPj-2e; Thu, 23 Nov 2023 06:58:02 +0000 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r63ex-003xNa-2f for linux-riscv@lists.infradead.org; Thu, 23 Nov 2023 06:58:01 +0000 Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-6cbd24d9557so429852b3a.1 for ; Wed, 22 Nov 2023 22:57:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1700722678; x=1701327478; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C94qAqj8zbtSDuRHlmOHD8Q2VGFM2PRqxiBp3mJrf0U=; b=Vu2g4hzrvkT2A8oF7szasjV6qRmiNUU05wCacTArVG5BpEIHsGDEK9Xbfeks2NgX72 Uf86dnNFnzQwIxORb2QgfS2aqq9J0/tAAPMKT9r0ZlJhbyz76MWmwgvRPgGM6GA1MEMz ctlr6E/dYWdDZFPYaKdG4SV9P/D/fb0pYQlXJC2TC2XhboJoBiK79WCgRiHc9gJAnzfO 8gBzN9Nsz9UX7ULiCbwEOsnCujYv3BJLcY7xliCHs8nYMDEOGB6mEZlD4auuHsTp3ZDB /o8TR9ONZ6TvHMdNg3McW4TwxemSoxL5lC6hyvq/odIclbTnXfM603Rq4aIDy5kWy4BX CUHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700722678; x=1701327478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C94qAqj8zbtSDuRHlmOHD8Q2VGFM2PRqxiBp3mJrf0U=; b=BiGqh+zYt1BBAH34X2M5/fGeQgXO6/6stsNFvInXhVlheIyZ0+nE+hpLRfApAWiOVe vKFuPqDP7VjHV7hvAIAra15OBIn41PzYtzCEgEi2V5vjCaO8eQxlxQlJhS1AjkoINbWM 9b4EtA0TP5xbD80v4852INwhQYC8b2iZ7TKaXklapy1BGiWKQSPvxwbcV2EqUtHL4cta iZYC9OHsbcHct/kLExVCh2w17vQJ64CG6FbQg0ij+K4zsSWDejKQBoK5AYcIsr0/n1te Y2UACy5g/jZ1FZ8HR+puoKPWtrt6kZJQPTE3umnwIurov7GNSMAF5iqfMx2oJUuvExbU 7s9Q== X-Gm-Message-State: AOJu0Yzlncp7N46llTvspTnrYFz838MhQpkZGRvIepWtiO+KT0QOsWGQ fzV74FMDGtbTbmB1d1z1KFjppA== X-Google-Smtp-Source: AGHT+IGnHCkqYYh5NefAhg3MFCZ+EXB4xUd+vxXu7NalDhM8+VhB4KPw88X/a6YywB+kIuA/EhnA0w== X-Received: by 2002:a05:6a00:2d94:b0:6cb:db40:4568 with SMTP id fb20-20020a056a002d9400b006cbdb404568mr2701949pfb.17.1700722678597; Wed, 22 Nov 2023 22:57:58 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w37-20020a634765000000b005bd2b3a03eesm615437pgk.6.2023.11.22.22.57.53 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Nov 2023 22:57:58 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: dengliang.1214@bytedance.com, xieyongji@bytedance.com, lihangjing@bytedance.com, songmuchun@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH V1 06/11] riscv: Distinguish pmd huge pte and napot huge pte Date: Thu, 23 Nov 2023 14:57:03 +0800 Message-Id: <20231123065708.91345-7-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231123065708.91345-1-luxu.kernel@bytedance.com> References: <20231123065708.91345-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_225759_868090_842383BA X-CRM114-Status: GOOD ( 13.68 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org There exist two kinds of huge pte on RISC-V: pmd and napot pte. For pmd kind huge pte, the huge page it represents is much larger than base page. Thus pmd kind huge pte can be represented via a single pmd entry and needs no special handling. For napot kind huge pte, it is actually several normal pte entries encoded in Svnapot format. Thus napot kind huge pte should be represented via pte_t struct and handled via pte operations. This commit distinguishes these two kinds of huge pte and handles them with different operations. Signed-off-by: Xu Lu --- arch/riscv/include/asm/hugetlb.h | 71 +++++++++++++++++++++++++++++++- arch/riscv/mm/hugetlbpage.c | 40 ++++++++++++------ 2 files changed, 97 insertions(+), 14 deletions(-) diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 4c5b0e929890..1cdd5a26e6d4 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -4,6 +4,7 @@ #include #include +#include static inline void arch_clear_hugepage_flags(struct page *page) { @@ -12,6 +13,7 @@ static inline void arch_clear_hugepage_flags(struct page *page) #define arch_clear_hugepage_flags arch_clear_hugepage_flags #ifdef CONFIG_RISCV_ISA_SVNAPOT + #define __HAVE_ARCH_HUGE_PTE_CLEAR void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz); @@ -41,10 +43,77 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, #define __HAVE_ARCH_HUGE_PTEP_GET pte_t huge_ptep_get(pte_t *ptep); +#define __HAVE_ARCH_HUGE_PTEP_GET_LOCKLESS +static inline pte_t huge_ptep_get_lockless(pte_t *ptep) +{ + unsigned long pteval = READ_ONCE(ptep->ptes[0]); + + return __pte(pteval); +} + pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#endif /*CONFIG_RISCV_ISA_SVNAPOT*/ +#else /* CONFIG_RISCV_ISA_SVNAPOT */ + +#define __HAVE_ARCH_HUGE_PTEP_GET +static inline pte_t huge_ptep_get(pte_t *ptep) +{ + pmd_t *pmdp = (pmd_t *)ptep; + + return pmd_pte(pmdp_get(pdmp)); +} + +#define __HAVE_ARCH_HUGE_PTEP_GET_LOCKLESS +static inline pte_t huge_ptep_get_lockless(pte_t *ptep) +{ + return huge_ptep_get(ptep); +} + +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT +static inline void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + set_pmd_at(mm, addr, (pmd_t *)ptep, pte_pmd(pte)); +} + +#define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGS +static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, int dirty) +{ + return pmdp_set_access_flags(vma, addr, (pmd_t *)ptep, pte_pmd(pte), dirty); +} + +#define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR +static inline pte_t huge_ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + return pmd_pte(pmdp_get_and_clear(mm, addr, (pmd_t *)ptep)); +} + +#define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT +static inline void huge_ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + pmdp_set_wrprotect(mm, addr, (pmd_t *)ptep); +} + +#define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH +static inline pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + return pmd_pte(pmdp_clear_flush(vma, addr, (pmd_t *)ptep)); +} + +#define __HAVE_ARCH_HUGE_PTE_CLEAR +static inline void huge_pte_clear(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned long sz) +{ + pmd_clear((pmd_t *)ptep); +} + +#endif /* CONFIG_RISCV_ISA_SVNAPOT */ #include diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 67fd71c36853..4a2ad8657502 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -7,8 +7,13 @@ pte_t huge_ptep_get(pte_t *ptep) { unsigned long pte_num; int i; - pte_t orig_pte = ptep_get(ptep); + pmd_t *pmdp = (pmd_t *)ptep; + pte_t orig_pte = pmd_pte(pmdp_get(pmdp)); + /* + * Non napot pte indicates a middle page table entry and + * should be treated as a pmd. + */ if (!pte_present(orig_pte) || !pte_napot(orig_pte)) return orig_pte; @@ -198,6 +203,8 @@ void set_huge_pte_at(struct mm_struct *mm, hugepage_shift = PAGE_SHIFT; pte_num = sz >> hugepage_shift; + if (pte_num == 1) + set_pmd_at(mm, addr, (pmd_t *)ptep, pte_pmd(pte)); for (i = 0; i < pte_num; i++, ptep++, addr += (1 << hugepage_shift)) set_pte_at(mm, addr, ptep, pte); } @@ -214,7 +221,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, int i, pte_num; if (!pte_napot(pte)) - return ptep_set_access_flags(vma, addr, ptep, pte, dirty); + return pmdp_set_access_flags(vma, addr, (pmd_t *)ptep, + pte_pmd(pte), dirty); order = napot_cont_order(pte); pte_num = napot_pte_num(order); @@ -237,11 +245,12 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - pte_t orig_pte = ptep_get(ptep); + pmd_t *pmdp = (pmd_t *)ptep; + pte_t orig_pte = pmd_pte(pmdp_get(pmdp)); int pte_num; if (!pte_napot(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); + return pmd_pte(pmdp_get_and_clear(mm, addr, pmdp)); pte_num = napot_pte_num(napot_cont_order(orig_pte)); @@ -252,13 +261,14 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - pte_t pte = ptep_get(ptep); + pmd_t *pmdp = (pmd_t *)ptep; + pte_t pte = pmd_pte(pmdp_get(pmdp)); unsigned long order; pte_t orig_pte; int i, pte_num; if (!pte_napot(pte)) { - ptep_set_wrprotect(mm, addr, ptep); + pmdp_set_wrprotect(mm, addr, pmdp); return; } @@ -277,11 +287,12 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { - pte_t pte = ptep_get(ptep); + pmd_t *pmdp = (pmd_t *)ptep; + pte_t pte = pmd_pte(pmdp_get(pmdp)); int pte_num; if (!pte_napot(pte)) - return ptep_clear_flush(vma, addr, ptep); + return pmd_pte(pmdp_clear_flush(vma, addr, pmdp)); pte_num = napot_pte_num(napot_cont_order(pte)); @@ -293,11 +304,12 @@ void huge_pte_clear(struct mm_struct *mm, pte_t *ptep, unsigned long sz) { - pte_t pte = ptep_get(ptep); + pmd_t *pmdp = (pmd_t *)ptep; + pte_t pte = pmd_pte(pmdp_get(pmdp)); int i, pte_num; if (!pte_napot(pte)) { - pte_clear(mm, addr, ptep); + pmd_clear(pmdp); return; } @@ -325,8 +337,10 @@ static __init int napot_hugetlbpages_init(void) if (has_svnapot()) { unsigned long order; - for_each_napot_order(order) - hugetlb_add_hstate(order); + for_each_napot_order(order) { + if (napot_cont_shift(order) > PAGE_SHIFT) + hugetlb_add_hstate(order); + } } return 0; } @@ -357,7 +371,7 @@ bool __init arch_hugetlb_valid_size(unsigned long size) return true; else if (IS_ENABLED(CONFIG_64BIT) && size == PUD_SIZE) return true; - else if (is_napot_size(size)) + else if (is_napot_size(size) && size > PAGE_SIZE) return true; else return false; From patchwork Thu Nov 23 06:57:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13465849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3E983C61D85 for ; Thu, 23 Nov 2023 06:58:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=EQnvrTGeGtLQMMsWJ6VkHXglD44uz1GQVc78mP8ehko=; b=qzB8SnJnSmIA19 Oatic/BqFzThTftvOCzGlN0em5ZMOpdGsNwZj0AcTRS+XuFZhSBbZ6OQ/cZZFoo0B4WhFGz3Duixd 6tsWszPzAdN/4rMRd0mnr+Hl7ZObnl3WfiXB/biE3g8Spf0cMbMld85j0a0mP0RUXpGlCcKMxdlxC tOwUzKVPtAfmc5043KBQqoHz+/kAQDscwpG6iHvkjFp9vheE9c35Psa5GSgx4U88TlrUJFwMZ561B Oilr3fbjrzVDROIBib3KAlk7rmlG7fWbDxpEr+kq8aCmQJrXkOnQkLrFBmOsI/zSfmeB9MldFhY8D dXBqhdWeoSlgAIClLclg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r63f6-003xSk-1j; Thu, 23 Nov 2023 06:58:08 +0000 Received: from mail-pf1-x432.google.com ([2607:f8b0:4864:20::432]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r63f3-003xQr-1O for linux-riscv@lists.infradead.org; Thu, 23 Nov 2023 06:58:07 +0000 Received: by mail-pf1-x432.google.com with SMTP id d2e1a72fcca58-6cbd24d9557so429905b3a.1 for ; Wed, 22 Nov 2023 22:58:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1700722684; x=1701327484; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WlPuhP739U6c+OrO0VAY/Be/8JPEzg6mBNajKv/ssXQ=; b=Mpu4HmlIXCLa3hyHHLEpFiTARq9HXAaRvvx6TOZZZPc62kBhYchIbt5b97Td50m9QK 03uoyDm/Bp7S858n2YT1qvVz7hNjY7XyZuE+9qlcOrzLRfQ5WWWHlWNg1XMz56Y0k5jj vsc2Xs/7FZQUmBfzFZcTBHNhry05h6zdnh6fMugzoKyI61sr3fSO74L0UIFXq5UdUUsd sq3YfhQHt9KDO0JEvjBkb4qNObKoQxrtihhWx4u0cGGVm67EQ4G+/e99QDS3I2HzmMOa BDBjz4gaKOjnUkSKtUt5TFKzPDnVDpQfiPkm7AWiwpcaNS+S5ZFDCOsohmayvTFQhoOG PvWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700722684; x=1701327484; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WlPuhP739U6c+OrO0VAY/Be/8JPEzg6mBNajKv/ssXQ=; b=pti+ChPnVYFI2LkQKmutcVRomyPhIrZfSbIPMPj2vjU4Br6rjZLwbhScrcflXa/LaX vwi+TIH6KkHKkkEGEUop8kUihRSG3Naw+iLETdiAN4BJ60c0384rmTh1Nn9ix9OSeUwx tSO3RRciU5T+eNcm4u/8LUW7GU+wi6zxf5U0aiKEGsU0loSQjmdOj+R4C88QeFWFGIhY anOS0gmnRmR8RjChcF2xaiU9x1hxiRsTJ4zpShXsV2sn7D6G4lOd5GR7BxQ6lgbM6jr5 Zd2xWGe8fUatlDOY0IcW2L6cesAlHPZjz/8hMu+M1z5wUygns7zxli+LHqKvzNQMiFIC cFCA== X-Gm-Message-State: AOJu0YyuaEo39xcpfqmg2qMU7F0kj0+vxqROINDHaq3nWNKtT/MjB3X8 i10oU7+w5kH8thw86+1lPR1ELg== X-Google-Smtp-Source: AGHT+IE8riuXo5L/cjCywKIfFVpKle7h5wIh28b47x4mdmNZCzuyeGv68fbHbrOiwUZ6PsJ9tMJIhw== X-Received: by 2002:a05:6a20:7484:b0:187:72e7:6d98 with SMTP id p4-20020a056a20748400b0018772e76d98mr2523467pzd.3.1700722683901; Wed, 22 Nov 2023 22:58:03 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w37-20020a634765000000b005bd2b3a03eesm615437pgk.6.2023.11.22.22.57.59 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Nov 2023 22:58:03 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: dengliang.1214@bytedance.com, xieyongji@bytedance.com, lihangjing@bytedance.com, songmuchun@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH V1 07/11] riscv: Adapt satp operations to gap between hw page and sw page Date: Thu, 23 Nov 2023 14:57:04 +0800 Message-Id: <20231123065708.91345-8-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231123065708.91345-1-luxu.kernel@bytedance.com> References: <20231123065708.91345-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_225805_474361_E3C4141F X-CRM114-Status: GOOD ( 15.42 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The control register CSR_SATP on RISC-V, which points to the root page table page, is used by MMU to translate va to pa when TLB miss happens. Thus it should be encoded at a granularity of hardware page, while existing code usually encodes it via software page frame number. This commit corrects encoding operations of CSR_SATP register. To get developers rid of the annoying encoding format of CSR_SATP and the conversion between sw pfn and hw pfn, we abstract the encoding operations of CSR_SATP into a specific function. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 7 +++++++ arch/riscv/kernel/head.S | 4 ++-- arch/riscv/kernel/hibernate.c | 3 ++- arch/riscv/mm/context.c | 7 +++---- arch/riscv/mm/fault.c | 1 + arch/riscv/mm/init.c | 7 +++++-- arch/riscv/mm/kasan_init.c | 7 +++++-- 7 files changed, 25 insertions(+), 11 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 9f81fe046cb8..56366f07985d 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -213,6 +213,13 @@ extern pgd_t swapper_pg_dir[]; extern pgd_t trampoline_pg_dir[]; extern pgd_t early_pg_dir[]; +static inline unsigned long make_satp(unsigned long pfn, + unsigned long asid, unsigned long satp_mode) +{ + return (pfn_to_hwpfn(pfn) | + ((asid & SATP_ASID_MASK) << SATP_ASID_SHIFT) | satp_mode); +} + static __always_inline int __pte_present(unsigned long pteval) { return (pteval & (_PAGE_PRESENT | _PAGE_PROT_NONE)); diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index b77397432403..dace2e4e6164 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -87,7 +87,7 @@ relocate_enable_mmu: csrw CSR_TVEC, a2 /* Compute satp for kernel page tables, but don't load it yet */ - srl a2, a0, PAGE_SHIFT + srl a2, a0, HW_PAGE_SHIFT la a1, satp_mode REG_L a1, 0(a1) or a2, a2, a1 @@ -100,7 +100,7 @@ relocate_enable_mmu: */ la a0, trampoline_pg_dir XIP_FIXUP_OFFSET a0 - srl a0, a0, PAGE_SHIFT + srl a0, a0, HW_PAGE_SHIFT or a0, a0, a1 sfence.vma csrw CSR_SATP, a0 diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c index 671b686c0158..155be6b1d32c 100644 --- a/arch/riscv/kernel/hibernate.c +++ b/arch/riscv/kernel/hibernate.c @@ -395,7 +395,8 @@ int swsusp_arch_resume(void) if (ret) return ret; - hibernate_restore_image(resume_hdr.saved_satp, (PFN_DOWN(__pa(resume_pg_dir)) | satp_mode), + hibernate_restore_image(resume_hdr.saved_satp, + make_satp(PFN_DOWN(__pa(resume_pg_dir)), 0, satp_mode), resume_hdr.restore_cpu_addr); return 0; diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 217fd4de6134..2ecf87433dfc 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -190,9 +190,8 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) raw_spin_unlock_irqrestore(&context_lock, flags); switch_mm_fast: - csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - ((cntx & asid_mask) << SATP_ASID_SHIFT) | - satp_mode); + csr_write(CSR_SATP, make_satp(virt_to_pfn(mm->pgd), (cntx & asid_mask), + satp_mode)); if (need_flush_tlb) local_flush_tlb_all(); @@ -201,7 +200,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ - csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); + csr_write(CSR_SATP, make_satp(virt_to_pfn(mm->pgd), 0, satp_mode)); local_flush_tlb_all(); } diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 90d4ba36d1d0..026ac007febf 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -133,6 +133,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a */ index = pgd_index(addr); pfn = csr_read(CSR_SATP) & SATP_PPN; + pfn = hwpfn_to_pfn(pfn); pgd = (pgd_t *)pfn_to_virt(pfn) + index; pgd_k = init_mm.pgd + index; diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index a768b2b3ff05..c33a90d0c51d 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -805,7 +805,7 @@ static __init void set_satp_mode(uintptr_t dtb_pa) (uintptr_t)early_p4d : (uintptr_t)early_pud, PGDIR_SIZE, PAGE_TABLE); - identity_satp = PFN_DOWN((uintptr_t)&early_pg_dir) | satp_mode; + identity_satp = make_satp(PFN_DOWN((uintptr_t)&early_pg_dir), 0, satp_mode); local_flush_tlb_all(); csr_write(CSR_SATP, identity_satp); @@ -1285,6 +1285,8 @@ static void __init create_linear_mapping_page_table(void) static void __init setup_vm_final(void) { + unsigned long satp; + /* Setup swapper PGD for fixmap */ #if !defined(CONFIG_64BIT) /* @@ -1318,7 +1320,8 @@ static void __init setup_vm_final(void) clear_fixmap(FIX_P4D); /* Move to swapper page table */ - csr_write(CSR_SATP, PFN_DOWN(__pa_symbol(swapper_pg_dir)) | satp_mode); + satp = make_satp(PFN_DOWN(__pa_symbol(swapper_pg_dir)), 0, satp_mode); + csr_write(CSR_SATP, satp); local_flush_tlb_all(); pt_ops_set_late(); diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index 5e39dcf23fdb..72269e9f1964 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -471,11 +471,13 @@ static void __init create_tmp_mapping(void) void __init kasan_init(void) { + unsigned long satp; phys_addr_t p_start, p_end; u64 i; create_tmp_mapping(); - csr_write(CSR_SATP, PFN_DOWN(__pa(tmp_pg_dir)) | satp_mode); + satp = make_satp(PFN_DOWN(__pa(tmp_pg_dir)), 0, satp_mode); + csr_write(CSR_SATP, satp); kasan_early_clear_pgd(pgd_offset_k(KASAN_SHADOW_START), KASAN_SHADOW_START, KASAN_SHADOW_END); @@ -520,6 +522,7 @@ void __init kasan_init(void) memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); init_task.kasan_depth = 0; - csr_write(CSR_SATP, PFN_DOWN(__pa(swapper_pg_dir)) | satp_mode); + satp = make_satp(PFN_DOWN(__pa(swapper_pg_dir)), 0, satp_mode); + csr_write(CSR_SATP, satp); local_flush_tlb_all(); } From patchwork Thu Nov 23 06:57:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13465850 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B1084C61D85 for ; Thu, 23 Nov 2023 06:58:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=LDhRUiItKkGpps0WUcqAlolJKnE9Mzb2/HDCkk09X+I=; b=MOqxwymwwfAjYf LPJjcz/m+rSRjxoeedZDyQPpoC4DmSNXzSDSfviN/HllbEnHP+KqZ8uVL8LyavpmhHKpBSy+DwNP/ rZA5VUhs/mzOXB40zXRn1/pXgqpI9wWQgEfruHMS0oNenyrUuzo9wmFMH7ewpko5fEUVbwIClZHJq HgrSuj+2RJC5W/+xG9HaGsrzWcRxrBKxmfKerBPZUdb6zCT/v+fLCXSHBDQIlHnxxo31MiAGmXBtG gxYyHLhPlPvFlkbSpB7MK9rmgiB92hLqZsTv7HPXMQh3rfU8W+84v7oIO0ascEizfXJ/4nUbptzo6 dgfaA88U24NaaWFm7Sgg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r63fB-003xVZ-1X; Thu, 23 Nov 2023 06:58:13 +0000 Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r63f8-003xTN-0q for linux-riscv@lists.infradead.org; Thu, 23 Nov 2023 06:58:11 +0000 Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-6c33ab26dddso511078b3a.0 for ; Wed, 22 Nov 2023 22:58:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1700722689; x=1701327489; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LItBiu3njgwOLf534DBFrKwLVF0r3rYUowPJh3TGf00=; b=REERRMXFt1KFxF6U5zz2z+MWy+D7gu8zSJqNaP5U5Hg+1M9e8KVELh2DwUbG74mK0P dmm+CIYyJbqxANvHhAo63B4rrqZ8K0T5gcYk1FC9MGZbC50fdAaOdPit6yWv9FDaP8+9 Ll6AG82jfpIppXoXd1v7OHuN5GwpbZcS4H8iGc19YBSfhMY1n1H8lQQggfLI0dVvpzp3 cNXfJfUpCWP+Fffg/8yLMAT+QoEqNV6qu5ilaAkpNnsV3C1SG/Dv/yKOlN0LtYg5JAE8 6lwZ4ikrK4vPfitru4RXGlxJXtjBiwEC2KAeqLfDjCABYlgj7n0l4RC9l981aeb5i6MK UWcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700722689; x=1701327489; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LItBiu3njgwOLf534DBFrKwLVF0r3rYUowPJh3TGf00=; b=H04/SA5OyLo3F1/s2cPvkQiikyEF9rDGf6153OJsj8u297888Trhot1TAAHfNSXJuq h/tqMBAUDdwEpnoOCZhEfllBnbJd7qiRkvo4QmXF0ODgSXfD0Nm9oG269NGCpmct4A6Y lGXsEBknpVyOs5aCpuV7iQAIAZw3WAYUpgwa8DIMdtfR72n/f5R34kSk+sswX1N3YRxZ bQoFhITSBp41BkMKmLC6qKSavAQ03X8oUzK5xC2g3B4e8fRWGIktVBhcicuJ8gCPtqiH Z6OqXIuvIJDtA3IARSI9AJ0UW8NB8TZrfQ+DO8pL54rXYEBJGA5DJEs5VF+oEvpRiVrS s1rQ== X-Gm-Message-State: AOJu0YwcGUF4Mz0RB/jcKM2T2kLRqCP2L+7yfITUxB0CGdwHlOXnxvK6 jMkCBP2D6JXbSUMl4gq6UXXqxg== X-Google-Smtp-Source: AGHT+IHDftjTLrWgyN3MNpT75XgaN32WdbFQKAWGC+MLKZT2N1z3VTeVkMs+OclMTvua0pcY3LDEGg== X-Received: by 2002:a05:6a00:35cf:b0:6cb:8c70:4790 with SMTP id dc15-20020a056a0035cf00b006cb8c704790mr5824334pfb.1.1700722689341; Wed, 22 Nov 2023 22:58:09 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w37-20020a634765000000b005bd2b3a03eesm615437pgk.6.2023.11.22.22.58.04 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Nov 2023 22:58:09 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: dengliang.1214@bytedance.com, xieyongji@bytedance.com, lihangjing@bytedance.com, songmuchun@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH V1 08/11] riscv: Apply Svnapot for base page mapping Date: Thu, 23 Nov 2023 14:57:05 +0800 Message-Id: <20231123065708.91345-9-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231123065708.91345-1-luxu.kernel@bytedance.com> References: <20231123065708.91345-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_225810_317343_DF6C3BAB X-CRM114-Status: GOOD ( 12.05 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The Svnapot extension on RISC-V is like contiguous PTE on ARM64. It allows ptes of a naturally aligned power-of 2 (NAPOT) memory range be encoded in the same format to save the TLB space. This commit applies Svnapot for each base page's mapping. This commit is the key to achieving larger base page's performance optimization. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 34 +++++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 56366f07985d..803dc5fb6314 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -230,6 +230,16 @@ static __always_inline unsigned long __pte_napot(unsigned long pteval) return pteval & _PAGE_NAPOT; } +static __always_inline unsigned long __pte_mknapot(unsigned long pteval, + unsigned int order) +{ + int pos = order - 1 + _PAGE_PFN_SHIFT; + unsigned long napot_bit = BIT(pos); + unsigned long napot_mask = ~GENMASK(pos, _PAGE_PFN_SHIFT); + + return (pteval & napot_mask) | napot_bit | _PAGE_NAPOT; +} + static inline pte_t __pte(unsigned long pteval) { pte_t pte; @@ -348,13 +358,11 @@ static inline unsigned long pte_napot(pte_t pte) return __pte_napot(pte_val(pte)); } -static inline pte_t pte_mknapot(pte_t pte, unsigned int order) +static inline pte_t pte_mknapot(pte_t pte, unsigned int page_order) { - int pos = order - 1 + _PAGE_PFN_SHIFT; - unsigned long napot_bit = BIT(pos); - unsigned long napot_mask = ~GENMASK(pos, _PAGE_PFN_SHIFT); + unsigned int hw_page_order = page_order + (PAGE_SHIFT - HW_PAGE_SHIFT); - return __pte((pte_val(pte) & napot_mask) | napot_bit | _PAGE_NAPOT); + return __pte(__pte_mknapot(pte_val(pte), hw_page_order)); } #else @@ -366,6 +374,11 @@ static inline unsigned long pte_napot(pte_t pte) return 0; } +static inline pte_t pte_mknapot(pte_t pte, unsigned int page_order) +{ + return pte; +} + #endif /* CONFIG_RISCV_ISA_SVNAPOT */ /* Yields the page frame number (PFN) of a page table entry */ @@ -585,6 +598,17 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b) */ static inline void set_pte(pte_t *ptep, pte_t pteval) { + unsigned long order; + + /* + * has_svnapot() always return false before riscv_isa is initialized. + */ + if (has_svnapot() && pte_present(pteval) && !pte_napot(pteval)) { + for_each_napot_order(order) { + if (napot_cont_shift(order) == PAGE_SHIFT) + pteval = pte_mknapot(pteval, order); + } + } *ptep = pteval; } From patchwork Thu Nov 23 06:57:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13465851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C595FC61D85 for ; Thu, 23 Nov 2023 06:58:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AQYKJB+kUzWxsOs1RMoOu0jqPyDAjdojCk2DG4DGDaU=; b=NMQa/Wd+vU49fd RXZRrtQprz6iol6HT7J3VgX72o7f3ZN39jcNnWcDky4V32yHeekFiZ0ZVrNv4ngtKoIhHkRvBSTxx gGWBkYd3fC+a7iJCd1mZ44CdkrOWVlIYo3VsXxvGIJPs74bW+NO4uC5kZTLPNThm4+9auAcL0swEW MSkrNHR+BRNEMZKJdKEy6GVHo6xW0CBp9tRtOlTer8u2q7R61obgKHASCB+E9wIaJkkS/TcRLdNVC CYzqkGY59dZOV7Yn29YaeLeZmMGNPIk6cmi2/EI6BnnGJFNP9aRlR1JdJ5WAD5dt0rpdWnryZb++h Nky+/8kLi3E5/JSBa3oA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r63fG-003xY5-0k; Thu, 23 Nov 2023 06:58:18 +0000 Received: from mail-oi1-x22c.google.com ([2607:f8b0:4864:20::22c]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r63fD-003xWR-3D for linux-riscv@lists.infradead.org; Thu, 23 Nov 2023 06:58:17 +0000 Received: by mail-oi1-x22c.google.com with SMTP id 5614622812f47-3b3f55e1bbbso397636b6e.2 for ; Wed, 22 Nov 2023 22:58:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1700722695; x=1701327495; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Xy98WgGNwj9rOwWzhsI6VzEfMnk+JFUolM0E3kCQh54=; b=Y49/9YszNULlgacYQss+yKarlq9NRaE0G56W5Inluh0hVlzmsqMN4tyqzQFpgdOm0c QLPUA7BkajpxHlb4+0+NwKU4SfwfLUIutnv0TE04NSCNrGb1FGKgHt2AKHOUa2M7XepR Dsl+LuGO2WpARfhzAFMpOvWJcv0PhRYL2L4qDu45wMahz8e9w9rCMMTq9aXVcqcVHKye imNFIXwR+w1uvQCjw/jTaeUEdZQyfhM7ctQpugV3psQDKeIb0kszBEKUGOyiWXIuD9tQ 7xGAmWkkT18slXYeEHrFN6d21b9M62nD8KzqcDtS0qZlt7I/yJ+FW6ZbVd8N8vRGakdP zS/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700722695; x=1701327495; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xy98WgGNwj9rOwWzhsI6VzEfMnk+JFUolM0E3kCQh54=; b=aJMMcvQtRbsrGW2SKoLFijzR7XIP8f4UHOgyLtcYd1syxIRH/s18SpSrtPwWQW8aJS aGY0MBrxpq67FkUrbsZbYNUHoiTI8TisVvU7mSlEiHMEYvTbDMqehHigQ/bmYHfcdsRA vYNszbWMQuEnk3wJsk0lmmx/6lkbotriV1KxCGiZ+mJPFu/iMx3otupfPgTxYfKIaLay KxDcmSkchatsl63hko399d/G9vLwDoiWP6KwWwKNTfPTQJNLBsKSgfjS+OUzeKhTKIRy QkzMyKf3g5Qgm22MGcwM8x1cKD8pcHTzH8DiuFR43zokxFFDyIUVQFbAp5T1qXF9s8oE MfsA== X-Gm-Message-State: AOJu0YynEvAqfi0+eRiLKjwLyKv/9/KR4nfoE3cQIPPRz0jh5zhr3dPY pI4sYecUrEu4kY1E0V3KL6Nh7Q== X-Google-Smtp-Source: AGHT+IEh9UjPGyaaQjuaIBLcYWiSd7ay7EOeP118dTtKlJ+CAM34oaGt+k986nYAGOfCpnNSdObjog== X-Received: by 2002:a05:6808:ab8:b0:3b2:f557:666e with SMTP id r24-20020a0568080ab800b003b2f557666emr5093225oij.19.1700722694829; Wed, 22 Nov 2023 22:58:14 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w37-20020a634765000000b005bd2b3a03eesm615437pgk.6.2023.11.22.22.58.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Nov 2023 22:58:14 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: dengliang.1214@bytedance.com, xieyongji@bytedance.com, lihangjing@bytedance.com, songmuchun@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH V1 09/11] riscv: Adjust fix_btmap slots number to match variable page size Date: Thu, 23 Nov 2023 14:57:06 +0800 Message-Id: <20231123065708.91345-10-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231123065708.91345-1-luxu.kernel@bytedance.com> References: <20231123065708.91345-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_225816_034970_74E8AD2D X-CRM114-Status: UNSURE ( 7.92 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The existing fixmap slot number will cause the fixmap size to exceed FIX_FDT_SIZE when base page becomes larger than 4K. This patch adjusts the slot number to make them always match. Signed-off-by: Xu Lu --- arch/riscv/include/asm/fixmap.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/fixmap.h b/arch/riscv/include/asm/fixmap.h index 0a55099bb734..17bf31334bd5 100644 --- a/arch/riscv/include/asm/fixmap.h +++ b/arch/riscv/include/asm/fixmap.h @@ -44,7 +44,8 @@ enum fixed_addresses { * before ioremap() is functional. */ #define NR_FIX_BTMAPS (SZ_256K / PAGE_SIZE) -#define FIX_BTMAPS_SLOTS 7 +#define FIX_BTMAPS_SIZE (FIXADDR_SIZE - ((FIX_BTMAP_END + 1) << PAGE_SHIFT)) +#define FIX_BTMAPS_SLOTS (FIX_BTMAPS_SIZE / SZ_256K) #define TOTAL_FIX_BTMAPS (NR_FIX_BTMAPS * FIX_BTMAPS_SLOTS) FIX_BTMAP_END = __end_of_permanent_fixed_addresses, From patchwork Thu Nov 23 06:57:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13465852 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4DF95C61D85 for ; Thu, 23 Nov 2023 06:58:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=XvXodfBtRb6e7Xk8wx0D0zYtzouhbUAI/Vb8Vyodx4Q=; b=0gp6i7AjHKPEkz uWQ7Q9DxPRM5PqRjGKl4Fa4NI34lqe3y5Q9JpFZjsqfWjDP0KvilT7BqJjNX87d+Gpu4ApsbX75KE ILYWDoWLbX77Q0DJzslx2ZZIKzSqOw6SucdPa4ZOyt4ApX6azrD/tFY4pSx3Jh7NCqPZHrL33Zl/B ykij2rnoh5CZj1rF1+jygO7FAeI3M4lrfNvG7oEUUmZ5igIGiu3vxLsxMRiRH5KImH2KpAqmA6ZQP CLZ3mwSjpCJ+nCg52zygOt9SqNm6olkNZRazV2qs9/YmBTYquoKpoGNpW9Oz8IqiSz6div2kbKh1h UCmihvy0BHJ9oFE8+tYw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r63fN-003xdb-1D; Thu, 23 Nov 2023 06:58:25 +0000 Received: from mail-ot1-x335.google.com ([2607:f8b0:4864:20::335]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r63fJ-003xZv-29 for linux-riscv@lists.infradead.org; Thu, 23 Nov 2023 06:58:23 +0000 Received: by mail-ot1-x335.google.com with SMTP id 46e09a7af769-6d67d32adc2so353360a34.2 for ; Wed, 22 Nov 2023 22:58:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1700722700; x=1701327500; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VzxcE+HP292pep10tUlqLGBeuvx8vBA9uQw9Wbs1mQo=; b=bU7JraRPxS9aIonh6RAmtPOYg7KdDrpnQEYBUi7vRMi4k51ZhlGtqAs8axy1BPvowC Y3gFPemvrrmZZ6+jRPrTAFQVyBaW3fK/uY2CZWPjaG6BhjzX1i8+l0b23t0Tbt82LC3R 3BH7cDoWC/AwuZWYuxCTRFIgp9ylD2mAaTmJPV71hjDvC6LDNjPZMTlDwUu5YpUKde6W TNwBDWLUoSwECYSuVvELvPSC9/qKB3rcScZoc8N9HthGgNU4LHWJ8IGPg3iYvsXdUUGw cSpvV0IDb/YqOWWxdk9O1iGu08p1xf4gVn50F6fRpd6FlNGpSwnVf4yu38bo94s5xTko LYhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700722700; x=1701327500; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VzxcE+HP292pep10tUlqLGBeuvx8vBA9uQw9Wbs1mQo=; b=C0Kym03XHdHa6a/ol2JTIz+pFdfuZLRvyvXxFFQ2DwynuzOPZWxHie7IkmOdEYr0hP SvNVNeZ2kcswh8TgBncG1EeaHUN7vBZZ9khMH1fhXDQt7d4I9CD+HAXVhctbo8hU85SX bJ8YJJ8VEiOWyPP8rCHjfXBF0vDpIdbg6MkuOiS4ZBBvM+d9mGt05P83JvKnMuhte5KN tosMHxpFjwqNcx+qR8UT6MuvB3NksHxi8LW5+VFvvkoLxNoIULu8/s4xu5tFOYoWNeZz wFYCJmE9AiOdHUa+wDVQy1v4oZLCXwy9+KZLhtskHx1ZIR7Xp4MHXhuf4eClUKulp0xn uEtQ== X-Gm-Message-State: AOJu0YwzYbHp/DKpM7DIvvI0MjPrMUMp2lxqu7+1SKVjCyXsV7iZMm6a wxgTdu3YCCFxjTwkpyYMV73C6g== X-Google-Smtp-Source: AGHT+IHfN56opjKVRtmf8nkhgYvS+GzTzNtB590B3ZTYABAvfpqWqq/B6z3vd8AMP9SjHn+EpQsRvw== X-Received: by 2002:a05:6830:2646:b0:6d7:f8c1:e473 with SMTP id f6-20020a056830264600b006d7f8c1e473mr1973461otu.19.1700722700289; Wed, 22 Nov 2023 22:58:20 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w37-20020a634765000000b005bd2b3a03eesm615437pgk.6.2023.11.22.22.58.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Nov 2023 22:58:19 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: dengliang.1214@bytedance.com, xieyongji@bytedance.com, lihangjing@bytedance.com, songmuchun@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH V1 10/11] riscv: kvm: Adapt kvm to gap between hw page and sw page Date: Thu, 23 Nov 2023 14:57:07 +0800 Message-Id: <20231123065708.91345-11-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231123065708.91345-1-luxu.kernel@bytedance.com> References: <20231123065708.91345-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_225821_718617_0785590E X-CRM114-Status: GOOD ( 20.10 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Existing mmu code in kvm handles middle level page table entry and the last level page table entry in the same way, which is insufficient when base page becomes larger. For example, for 64K base page, per pte_t contains 16 page table entries while per pmd_t still contains one and thus needs to be handled in different ways. This commit refines kvm mmu code to handle middle level page table entries and last level page table entries distinctively. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 7 ++ arch/riscv/kvm/mmu.c | 198 +++++++++++++++++++++---------- 2 files changed, 145 insertions(+), 60 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 803dc5fb6314..9bed1512b3d2 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -220,6 +220,13 @@ static inline unsigned long make_satp(unsigned long pfn, ((asid & SATP_ASID_MASK) << SATP_ASID_SHIFT) | satp_mode); } +static inline unsigned long make_hgatp(unsigned long pfn, + unsigned long vmid, unsigned long hgatp_mode) +{ + return ((pfn_to_hwpfn(pfn) & HGATP_PPN) | + ((vmid << HGATP_VMID_SHIFT) & HGATP_VMID) | hgatp_mode); +} + static __always_inline int __pte_present(unsigned long pteval) { return (pteval & (_PAGE_PRESENT | _PAGE_PROT_NONE)); diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 068c74593871..f26d3e94fe17 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -36,22 +36,36 @@ static unsigned long gstage_pgd_levels __ro_after_init = 2; gstage_pgd_xbits) #define gstage_gpa_size ((gpa_t)(1ULL << gstage_gpa_bits)) +#define gstage_pmd_leaf(__pmdp) \ + (pmd_val(pmdp_get(__pmdp)) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC)) + #define gstage_pte_leaf(__ptep) \ - (pte_val(*(__ptep)) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC)) + (pte_val(ptep_get(__ptep)) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC)) -static inline unsigned long gstage_pte_index(gpa_t addr, u32 level) +static inline unsigned long gstage_pmd_index(gpa_t addr, u32 level) { unsigned long mask; unsigned long shift = HGATP_PAGE_SHIFT + (gstage_index_bits * level); + BUG_ON(level == 0); if (level == (gstage_pgd_levels - 1)) - mask = (PTRS_PER_PTE * (1UL << gstage_pgd_xbits)) - 1; + mask = (PTRS_PER_PMD * (1UL << gstage_pgd_xbits)) - 1; else - mask = PTRS_PER_PTE - 1; + mask = PTRS_PER_PMD - 1; return (addr >> shift) & mask; } +static inline unsigned long gstage_pte_index(gpa_t addr, u32 level) +{ + return (addr >> PAGE_SHIFT) & (PTRS_PER_PTE - 1); +} + +static inline unsigned long gstage_pmd_page_vaddr(pmd_t pmd) +{ + return (unsigned long)pfn_to_virt(__page_val_to_pfn(pmd_val(pmd))); +} + static inline unsigned long gstage_pte_page_vaddr(pte_t pte) { return (unsigned long)pfn_to_virt(__page_val_to_pfn(pte_val(pte))); @@ -60,9 +74,13 @@ static inline unsigned long gstage_pte_page_vaddr(pte_t pte) static int gstage_page_size_to_level(unsigned long page_size, u32 *out_level) { u32 i; - unsigned long psz = 1UL << 12; + unsigned long psz = 1UL << HW_PAGE_SHIFT; - for (i = 0; i < gstage_pgd_levels; i++) { + if (page_size == PAGE_SIZE) { + *out_level = 0; + return 0; + } + for (i = 1; i < gstage_pgd_levels; i++) { if (page_size == (psz << (i * gstage_index_bits))) { *out_level = i; return 0; @@ -77,7 +95,11 @@ static int gstage_level_to_page_order(u32 level, unsigned long *out_pgorder) if (gstage_pgd_levels < level) return -EINVAL; - *out_pgorder = 12 + (level * gstage_index_bits); + if (level == 0) + *out_pgorder = PAGE_SHIFT; + else + *out_pgorder = HW_PAGE_SHIFT + (level * gstage_index_bits); + return 0; } @@ -95,30 +117,40 @@ static int gstage_level_to_page_size(u32 level, unsigned long *out_pgsize) } static bool gstage_get_leaf_entry(struct kvm *kvm, gpa_t addr, - pte_t **ptepp, u32 *ptep_level) + void **ptepp, u32 *ptep_level) { - pte_t *ptep; + pmd_t *pmdp = NULL; + pte_t *ptep = NULL; u32 current_level = gstage_pgd_levels - 1; *ptep_level = current_level; - ptep = (pte_t *)kvm->arch.pgd; - ptep = &ptep[gstage_pte_index(addr, current_level)]; - while (ptep && pte_val(*ptep)) { - if (gstage_pte_leaf(ptep)) { + pmdp = (pmd_t *)kvm->arch.pgd; + pmdp = &pmdp[gstage_pmd_index(addr, current_level)]; + while (current_level && pmdp && pmd_val(pmdp_get(pmdp))) { + if (gstage_pmd_leaf(pmdp)) { *ptep_level = current_level; - *ptepp = ptep; + *ptepp = (void *)pmdp; return true; } + current_level--; + *ptep_level = current_level; + pmdp = (pmd_t *)gstage_pmd_page_vaddr(pmdp_get(pmdp)); if (current_level) { - current_level--; - *ptep_level = current_level; - ptep = (pte_t *)gstage_pte_page_vaddr(*ptep); - ptep = &ptep[gstage_pte_index(addr, current_level)]; + pmdp = &pmdp[gstage_pmd_index(addr, current_level)]; } else { - ptep = NULL; + ptep = (pte_t *)pmdp; + ptep = &ptep[gstage_pte_index(addr, current_level)]; } } + if (ptep && pte_val(ptep_get(ptep))) { + if (gstage_pte_leaf(ptep)) { + *ptep_level = current_level; + *ptepp = (void *)ptep; + return true; + } + ptep = NULL; + } return false; } @@ -136,40 +168,53 @@ static void gstage_remote_tlb_flush(struct kvm *kvm, u32 level, gpa_t addr) static int gstage_set_pte(struct kvm *kvm, u32 level, struct kvm_mmu_memory_cache *pcache, - gpa_t addr, const pte_t *new_pte) + gpa_t addr, const void *new_pte) { u32 current_level = gstage_pgd_levels - 1; - pte_t *next_ptep = (pte_t *)kvm->arch.pgd; - pte_t *ptep = &next_ptep[gstage_pte_index(addr, current_level)]; + pmd_t *next_pmdp = (pmd_t *)kvm->arch.pgd; + pmd_t *pmdp = &next_pmdp[gstage_pmd_index(addr, current_level)]; + pte_t *next_ptep = NULL; + pte_t *ptep = NULL; if (current_level < level) return -EINVAL; while (current_level != level) { - if (gstage_pte_leaf(ptep)) + if (gstage_pmd_leaf(pmdp)) return -EEXIST; - if (!pte_val(*ptep)) { + if (!pmd_val(pmdp_get(pmdp))) { if (!pcache) return -ENOMEM; - next_ptep = kvm_mmu_memory_cache_alloc(pcache); - if (!next_ptep) + next_pmdp = kvm_mmu_memory_cache_alloc(pcache); + if (!next_pmdp) return -ENOMEM; - *ptep = pfn_pte(PFN_DOWN(__pa(next_ptep)), - __pgprot(_PAGE_TABLE)); + set_pmd(pmdp, pfn_pmd(PFN_DOWN(__pa(next_pmdp)), + __pgprot(_PAGE_TABLE))); } else { - if (gstage_pte_leaf(ptep)) + if (gstage_pmd_leaf(pmdp)) return -EEXIST; - next_ptep = (pte_t *)gstage_pte_page_vaddr(*ptep); + next_pmdp = (pmd_t *)gstage_pmd_page_vaddr(pmdp_get(pmdp)); } current_level--; - ptep = &next_ptep[gstage_pte_index(addr, current_level)]; + if (current_level) { + pmdp = &next_pmdp[gstage_pmd_index(addr, current_level)]; + } else { + next_ptep = (pte_t *)next_pmdp; + ptep = &next_ptep[gstage_pte_index(addr, current_level)]; + } } - *ptep = *new_pte; - if (gstage_pte_leaf(ptep)) - gstage_remote_tlb_flush(kvm, current_level, addr); + if (current_level) { + set_pmd(pmdp, pmdp_get((pmd_t *)new_pte)); + if (gstage_pmd_leaf(pmdp)) + gstage_remote_tlb_flush(kvm, current_level, addr); + } else { + set_pte(ptep, ptep_get((pte_t *)new_pte)); + if (gstage_pte_leaf(ptep)) + gstage_remote_tlb_flush(kvm, current_level, addr); + } return 0; } @@ -182,6 +227,7 @@ static int gstage_map_page(struct kvm *kvm, { int ret; u32 level = 0; + pmd_t new_pmd; pte_t new_pte; pgprot_t prot; @@ -213,10 +259,15 @@ static int gstage_map_page(struct kvm *kvm, else prot = PAGE_WRITE; } - new_pte = pfn_pte(PFN_DOWN(hpa), prot); - new_pte = pte_mkdirty(new_pte); - - return gstage_set_pte(kvm, level, pcache, gpa, &new_pte); + if (level) { + new_pmd = pfn_pmd(PFN_DOWN(hpa), prot); + new_pmd = pmd_mkdirty(new_pmd); + return gstage_set_pte(kvm, level, pcache, gpa, &new_pmd); + } else { + new_pte = pfn_pte(PFN_DOWN(hpa), prot); + new_pte = pte_mkdirty(new_pte); + return gstage_set_pte(kvm, level, pcache, gpa, &new_pte); + } } enum gstage_op { @@ -226,9 +277,12 @@ enum gstage_op { }; static void gstage_op_pte(struct kvm *kvm, gpa_t addr, - pte_t *ptep, u32 ptep_level, enum gstage_op op) + void *__ptep, u32 ptep_level, enum gstage_op op) { int i, ret; + pmd_t *pmdp = (pmd_t *)__ptep; + pte_t *ptep = (pte_t *)__ptep; + pmd_t *next_pmdp; pte_t *next_ptep; u32 next_ptep_level; unsigned long next_page_size, page_size; @@ -239,11 +293,13 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, BUG_ON(addr & (page_size - 1)); - if (!pte_val(*ptep)) + if (ptep_level && !pmd_val(pmdp_get(pmdp))) + return; + if (!ptep_level && !pte_val(ptep_get(ptep))) return; - if (ptep_level && !gstage_pte_leaf(ptep)) { - next_ptep = (pte_t *)gstage_pte_page_vaddr(*ptep); + if (ptep_level && !gstage_pmd_leaf(pmdp)) { + next_pmdp = (pmd_t *)gstage_pmd_page_vaddr(pmdp_get(pmdp)); next_ptep_level = ptep_level - 1; ret = gstage_level_to_page_size(next_ptep_level, &next_page_size); @@ -251,17 +307,33 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, return; if (op == GSTAGE_OP_CLEAR) - set_pte(ptep, __pte(0)); - for (i = 0; i < PTRS_PER_PTE; i++) - gstage_op_pte(kvm, addr + i * next_page_size, - &next_ptep[i], next_ptep_level, op); + set_pmd(pmdp, __pmd(0)); + if (next_ptep_level) { + for (i = 0; i < PTRS_PER_PMD; i++) + gstage_op_pte(kvm, addr + i * next_page_size, + &next_pmdp[i], next_ptep_level, op); + } else { + next_ptep = (pte_t *)next_pmdp; + for (i = 0; i < PTRS_PER_PTE; i++) + gstage_op_pte(kvm, addr + i * next_page_size, + &next_ptep[i], next_ptep_level, op); + } if (op == GSTAGE_OP_CLEAR) - put_page(virt_to_page(next_ptep)); + put_page(virt_to_page(next_pmdp)); } else { - if (op == GSTAGE_OP_CLEAR) - set_pte(ptep, __pte(0)); - else if (op == GSTAGE_OP_WP) - set_pte(ptep, __pte(pte_val(*ptep) & ~_PAGE_WRITE)); + if (ptep_level) { + if (op == GSTAGE_OP_CLEAR) + set_pmd(pmdp, __pmd(0)); + else if (op == GSTAGE_OP_WP) + set_pmd(pmdp, + __pmd(pmd_val(pmdp_get(pmdp)) & ~_PAGE_WRITE)); + } else { + if (op == GSTAGE_OP_CLEAR) + set_pte(ptep, __pte(0)); + else if (op == GSTAGE_OP_WP) + set_pte(ptep, + __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE)); + } gstage_remote_tlb_flush(kvm, ptep_level, addr); } } @@ -270,7 +342,7 @@ static void gstage_unmap_range(struct kvm *kvm, gpa_t start, gpa_t size, bool may_block) { int ret; - pte_t *ptep; + void *ptep; u32 ptep_level; bool found_leaf; unsigned long page_size; @@ -305,7 +377,7 @@ static void gstage_unmap_range(struct kvm *kvm, gpa_t start, static void gstage_wp_range(struct kvm *kvm, gpa_t start, gpa_t end) { int ret; - pte_t *ptep; + void *ptep; u32 ptep_level; bool found_leaf; gpa_t addr = start; @@ -572,7 +644,7 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - pte_t *ptep; + void *ptep; u32 ptep_level = 0; u64 size = (range->end - range->start) << PAGE_SHIFT; @@ -585,12 +657,15 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) &ptep, &ptep_level)) return false; - return ptep_test_and_clear_young(NULL, 0, ptep); + if (ptep_level) + return pmdp_test_and_clear_young(NULL, 0, (pmd_t *)ptep); + else + return ptep_test_and_clear_young(NULL, 0, (pte_t *)ptep); } bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - pte_t *ptep; + void *ptep; u32 ptep_level = 0; u64 size = (range->end - range->start) << PAGE_SHIFT; @@ -603,7 +678,10 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) &ptep, &ptep_level)) return false; - return pte_young(*ptep); + if (ptep_level) + return pmd_young(pmdp_get((pmd_t *)ptep)); + else + return pte_young(ptep_get((pte_t *)ptep)); } int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, @@ -746,11 +824,11 @@ void kvm_riscv_gstage_free_pgd(struct kvm *kvm) void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vcpu) { - unsigned long hgatp = gstage_mode; + unsigned long hgatp; struct kvm_arch *k = &vcpu->kvm->arch; - hgatp |= (READ_ONCE(k->vmid.vmid) << HGATP_VMID_SHIFT) & HGATP_VMID; - hgatp |= (k->pgd_phys >> PAGE_SHIFT) & HGATP_PPN; + hgatp = make_hgatp(PFN_DOWN(k->pgd_phys), READ_ONCE(k->vmid.vmid), + gstage_mode); csr_write(CSR_HGATP, hgatp); From patchwork Thu Nov 23 06:57:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13465853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B071BC5AD4C for ; Thu, 23 Nov 2023 06:58:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=si2yhcDwZhqho+QrldZzB9KOYj1EdLjGN97bEyLBj3M=; b=0EWdhKqie+hajh NfkV34pPt9eP22EjYmMVnX9X/DVGtcEgE8UZFS8p9zH4y6gmvwd1HqtEV6YeHUBsqXh14BbBO5g86 3jzLzqlt0gJnjMLFhMwqS5QvxAcRPRuqC5H+Aj/75YLWj3j8nn6AvDkCrEnIa6VZkao454kxo29Hy w1Ro/Dh3g0LGhji5HZJ5ebbd6ed/0SAcmQV61gBOMJF2TWG02+IG/9iOJfz2DjiFqIsacl/fIfB+s YRzY1Ex1OsjrQWuhhNbCYeNH4e2pAQJzReG8tQ9ekHcH3Z1APzOY0NkfIux8y+0w6YPbXcBk5evhA euFYxbooszkZjy8glPZw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r63fR-003xgq-0f; Thu, 23 Nov 2023 06:58:29 +0000 Received: from mail-ot1-x331.google.com ([2607:f8b0:4864:20::331]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r63fO-003xef-2w for linux-riscv@lists.infradead.org; Thu, 23 Nov 2023 06:58:28 +0000 Received: by mail-ot1-x331.google.com with SMTP id 46e09a7af769-6d7f2058213so349848a34.1 for ; Wed, 22 Nov 2023 22:58:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1700722706; x=1701327506; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cXVVh5tHQq6H+NSQA4JeCtH/rNgHUAw+NLKBVun1FJI=; b=W4FBE+5eZHoq4bc6sOnwuM9tDT40g7IpZNcrWUeLhT7XU/RDWtaXorW/3q/CVxJMFd IM0CufGpQrJsbFqXfdztFv4NiHkp528U0TX9ZmJU6M0mdGogoXcSOhLvfJDeD4MujRE9 ovW8mRdPGprSE26464hbIWdZDhSV1BBX1w3QbLFoXwG1O/10Y/C8OBiB89WSu+YouafM jTZxgUkXPQLvFrGtYr5cd7AcrTAXQ+YEqE8h0mfqOYdcUjd/l4TKCZPSYqvWpZUIJbil INI/GPMktSTY1MEZDdyR2KInr0BUhoPkcYoeDNFujFplbrMGrTPAxSBHKHznVvFU180n JNjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700722706; x=1701327506; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cXVVh5tHQq6H+NSQA4JeCtH/rNgHUAw+NLKBVun1FJI=; b=Nad+DL5b4iRjpvrn6d2fnRRIAHJRVq1Z/muQJbC4SFYoUrpuhHk2WkaQ5xCAQGZxxt g0F7E7AI69Wmuxs3d8qH1zE3lI5wPIMrb7RyxE/IpFY65jZCRMfIZAFHD3PFz9cWgeZg 8bD59RrIbfuAkdNx8zMN6n/aL22WWgI4x95zFLyWXkNIBl7Xiso2gXgTV4ctjlZaAW17 XPrps4Dv+xpKbfRSiRw8Rr1DCmmCoxjMCr8/1t/e3U33cLEq0dxlCd0gLYM09rfdUvC3 Uq+YsPuB/Loa0BNMYMA1g6LeT9HG+ct5HFO/AwuGedVADivxt8s3hvtot3hfSDPTB4wS BBKg== X-Gm-Message-State: AOJu0YzIv3VferS34LEXGoKTTgT+HpSojR6wvEeUeD2/5rdRCLiHgJws yArtPWd4ohOZj7ASPZutAHtm0g== X-Google-Smtp-Source: AGHT+IHjXbEiIjdrtzBQes9pWsmlDe44Anh7ZX4igq4dgVqzJwwbqJ+5ff/GxnQby0i2Dm9GfBQciw== X-Received: by 2002:a9d:620d:0:b0:6cd:8c3:5b40 with SMTP id g13-20020a9d620d000000b006cd08c35b40mr4649798otj.36.1700722705824; Wed, 22 Nov 2023 22:58:25 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w37-20020a634765000000b005bd2b3a03eesm615437pgk.6.2023.11.22.22.58.20 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Nov 2023 22:58:25 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: dengliang.1214@bytedance.com, xieyongji@bytedance.com, lihangjing@bytedance.com, songmuchun@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH V1 11/11] riscv: Introduce 64K page size Date: Thu, 23 Nov 2023 14:57:08 +0800 Message-Id: <20231123065708.91345-12-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231123065708.91345-1-luxu.kernel@bytedance.com> References: <20231123065708.91345-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_225826_951200_0F9666C4 X-CRM114-Status: UNSURE ( 7.66 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch introduces new config to control whether enabling the 64K base page feature on RISC-V. Signed-off-by: Xu Lu --- arch/Kconfig | 1 + arch/riscv/Kconfig | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index f4b210ab0612..66f64450d409 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1087,6 +1087,7 @@ config HAVE_ARCH_COMPAT_MMAP_BASES config PAGE_SIZE_LESS_THAN_64KB def_bool y + depends on !RISCV_64K_PAGES depends on !ARM64_64K_PAGES depends on !PAGE_SIZE_64KB depends on !PARISC_PAGE_SIZE_64KB diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 105cbb3ca797..d561f9f7f9b4 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -227,6 +227,7 @@ config RISCV_HW_PAGE_SHIFT config RISCV_PAGE_SHIFT int + default 16 if RISCV_64K_PAGES default 12 config KASAN_SHADOW_OFFSET @@ -692,6 +693,25 @@ config RISCV_BOOT_SPINWAIT If unsure what to do here, say N. +choice + prompt "Page size" + default RISCV_4K_PAGES + help + Page size (translation granule) configuration. + +config RISCV_4K_PAGES + bool "4KB" + help + This feature enables 4KB pages support. + +config RISCV_64K_PAGES + bool "64KB" + depends on ARCH_HAS_STRICT_KERNEL_RWX && 64BIT + help + This feature enables 64KB pages support (4KB by default) + +endchoice + config ARCH_SUPPORTS_KEXEC def_bool MMU