From patchwork Fri Mar 1 09:14:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13578200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EB32C5478C for ; Fri, 1 Mar 2024 09:21:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02FA86B008A; Fri, 1 Mar 2024 04:21:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F22246B009E; Fri, 1 Mar 2024 04:21:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DEAB66B00A0; Fri, 1 Mar 2024 04:21:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CF7D86B008A for ; Fri, 1 Mar 2024 04:21:21 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A69BFA11F8 for ; Fri, 1 Mar 2024 09:21:21 +0000 (UTC) X-FDA: 81847926762.01.D98F82B Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by imf16.hostedemail.com (Postfix) with ESMTP id E0BA8180009 for ; Fri, 1 Mar 2024 09:21:19 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b="UTe/JdCA"; dmarc=none; spf=pass (imf16.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.47 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709284880; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=roMginZSvi1o0fjo+EUJOo1pu0LVGawrxcxtyn24YPc=; b=vi9RSyr+Aj7U76207zsUDxWUGhJFeKtdCvll8NMlW6fP9Sh5KlFbwFi4Po6HHxDvWctq7L rdglhZo7XbnaMfsBc4ipacQKnoiU8dCfmf1Lk7mSma3PlAwR+8jdAZA1yjKdgmPKlw0b5r EoA5kYoTERPIqSsH6a7Uo8EcP39F8xs= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b="UTe/JdCA"; dmarc=none; spf=pass (imf16.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.47 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709284880; a=rsa-sha256; cv=none; b=pM2F2n1z08F/vNgvqo5nq3Rh/IQ4X381JYuqDRrPp/2Fg/3g+0F2IfVgNg8oETK4LDiTP+ iAGCuXUNFQzzrfmLcPvHdvCIfYjA3C015yA1QMVLvxB3cKmhVWpIJ7zEe7AATuUGNaSEM3 CVXrfEwWPzkpgQ6300K7M69Gv1sWtKY= Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-33e17342ea7so527403f8f.2 for ; Fri, 01 Mar 2024 01:21:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709284878; x=1709889678; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=roMginZSvi1o0fjo+EUJOo1pu0LVGawrxcxtyn24YPc=; b=UTe/JdCABqY/FsPQw1gnaRYs5DBYWmVurX490RR4nn+BDG4TGIcBGP7+BhJ1weQeTO wvC8A5xGSb7H2AYm1CRZtZKN/XpfF3A+0Yh6q0NDbfKUHKux9/9SPxFopOlFf+Zeyw+3 +uLmtsOJyMgaTG1BdshrfkwcJxrp4k44Ps78hQqfbpq6cgLjKvPQ50hUaZUV76dtTq3Y eEnxo3h0k+9rLf8zoGItmc6izRSpUWbQFLJZJ6oMSYLr48pF2lVv9bvZFAEFOEAwUUvq QDFbzLmPFYycd55+hLlxcjxiJY2+TVmeMziN6PddEw1vzIo+ruDlzitexJCSFicInOEo 4LVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709284878; x=1709889678; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=roMginZSvi1o0fjo+EUJOo1pu0LVGawrxcxtyn24YPc=; b=WQ6HnZPeGH5tBVDFfJmdobE6Vn8H8NLwRow8RXriaJVDlB5u51pFE2eaya3YJDZg5U juAmGXSfJOQsioybN0/DiJMtgaTjjcUkyCHOaXD6NMgIOMLoiNIvqQLr/bjur3s1DiZf OCXbxmbstkw/WRaMV14TD7rbUcI4fsTTVHfkESt7JLBVeqJERo0ZDhiMsXNfMMLsDyZ6 UVO4TOsA+97JLnZYFoAJAapgRZ9sDkEy/ZSUPAiRatImNPEvvtXS5NnI+YvK97CtGqe6 k/GBr3D+PCS0A6VHfYbxrT/Q47M+7L+RKFeCS0C9rp7sRnfEYBuPcHDSo8Wdif8vd/nx bbxg== X-Forwarded-Encrypted: i=1; AJvYcCUUPgUWP9DsiXmpx1ELKXjZQSOuDdqGX9UbvmAh/DkPPdYPqIpYzV5eq3KRjUN00nU1itBPNchjQCT9fvlsFcptaIk= X-Gm-Message-State: AOJu0Yx1e4PJob/7Pm/wDAmeaZ1UAAxEc8KUXmvzcfIv/UioQG4a4w4r UQxSgaa6N8yv95SSvviyMIEgel1XIoppCpRJUsblPPXtt7bsfQ+oqWPPng2OtPA= X-Google-Smtp-Source: AGHT+IHE36Cx0pzkf7/PwROG0sFnOnPILtKg/5tjC2XGRavDiE3m4OmzpuOVj7dxMVbSTb8/7tKiwA== X-Received: by 2002:adf:f24e:0:b0:33d:c5c5:9bc6 with SMTP id b14-20020adff24e000000b0033dc5c59bc6mr893274wrp.54.1709284878472; Fri, 01 Mar 2024 01:21:18 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id i6-20020adfb646000000b0033e033898c5sm4044660wre.20.2024.03.01.01.21.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 01:21:18 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 6/9] mm: Use common huge_ptep_get_and_clear() function for riscv/arm64 Date: Fri, 1 Mar 2024 10:14:52 +0100 Message-Id: <20240301091455.246686-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240301091455.246686-1-alexghiti@rivosinc.com> References: <20240301091455.246686-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E0BA8180009 X-Stat-Signature: du3hicqpq3kig8a3e4mj6fh7jrrhcwae X-Rspam-User: X-HE-Tag: 1709284879-937886 X-HE-Meta: U2FsdGVkX18Xj4PVNBI8AI8uHuNGdOA5v1lXaQLOVswolcxP2KcSzu4QuqwJfwqdcOGnYClrI8bUq7T890WPZDzVGT4bFhxtPgXSWi6rj5PuDsoQHQyMKlx5f2KFR4XQyauGKk77/61O0VjzfDHJcNS2FrgEyoTZFqKtVs4yrTBbK57+un1YD5f4hIKNFrWBP9ICwgqbI8OxSy9VevvpzCXo6uwXzG+4N0g7USCcVFcjmJns0hGBE81ULnvkzJWXfL6+yG+fn2T/4i8Dwc8wCVd7945VeBdxDCaz3pyhJmh8BZCBTNv50QtZK372mI9aPen+mG3Zdt/Iuwufh9lW1iBXBBBDS7U6M3r171HepDyrt0789RhfKGusnUkJuditegfzN0drTiR61CLc2YVJuVv3wJHnFQpG2Zu5XOi/9Fa1wWMyoqbACkufCbafovMrNPt1Xcmh7BkGU1qKy0EFFdgKwJQoVXi5Wke4brlpMt2rQpQ7RS6+BV3UdGR/g6iEOMmqObNhYTGjqk3LC96YTfncSueuWdWG5rt0w9GfDDPuLpBSzhx2X2rkG2PwiXp91HFw6ZtejqnEDkgsjxrmB35pr6XuAwRi4z1KjVDyNDruypEDouQme2tV+L8OIhBUMVv4dS7WkxGQ6hqw2v0JwdiLevjqjY7SpKc5HswfVQy7JPKgv2mkd4e7cuyLV5jvVRVZQbzvqz9GKKicERY7zQLhDZAvKtrQz93LDU8b4QSvcM+Dh4spCUI+yAA6evAqoOQq188rNlCOebPV0HuyxS2KRBYYSwGLXG9BvzqUlq0ApKiR3Jn7ARXyYeRoyV5lpIWyRVW74OMo6FbxX1MwlSTPz6nHqeGAo0y/f2VUCioITUqfUAx7QcFIVmOwLtKgN/F4R2xwsBM7w4TU6zE0wvhuLpFenlOBM/q+zXdgnT702eA+zjQcInO5OyC0mL+g/q4YQR1IYF5Ps36VdHt XvJSG8KM PXKVnJa8be0xdL7jmujis5dx5uph9FfkkEfUsqX3SFwf0wNpxYWqtYJYmY2zSPiWL/llZKHIL4L23CzvygoCx91P49AR+x/Pclj3rcSpVid3excxd2BERqC4t0vNV/WATiltBZSdXJtGf1IXVqwdA7/1wsBBvtG9EuzZzTqKTuepl12GL+JusLpDVXzSGfDvZtyzxAoZxYxhcMM3Xg1kxoEigbw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After some adjustments, both architectures have the same implementation so move it to the generic code. Note that get_clear_contig() function is duplicated in the generic and the arm64 code because it is still used by some arm64 functions that will, in the next commits, be moved to the generic code. Once all have been moved, the arm64 version will be removed. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 14 +++++++++- arch/arm64/mm/hugetlbpage.c | 19 ++----------- arch/riscv/include/asm/pgtable.h | 4 ++- arch/riscv/mm/hugetlbpage.c | 21 ++------------ mm/contpte.c | 48 ++++++++++++++++++++++++++++++-- 5 files changed, 66 insertions(+), 40 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 437e9638b2b9..b5caf7a9d03f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1131,11 +1131,23 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); -static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, +int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize); + +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { int contig_ptes = 0; + /* + * If the size is not passed, we need to go through the page table to + * find out the number of contiguous ptes. + */ + if (size == 0) + return find_num_contig(mm, addr, ptep, pgsize); + *pgsize = size; switch (size) { diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 09b55bac8c7c..a2d33cbc7da5 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -100,8 +100,8 @@ int pud_huge(pud_t pud) #endif } -static int find_num_contig(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, size_t *pgsize) +int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize) { pgd_t *pgdp = pgd_offset(mm, addr); p4d_t *p4dp; @@ -284,21 +284,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - int ncontig; - size_t pgsize; - pte_t orig_pte = ptep_get(ptep); - - if (!pte_cont(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - - return get_clear_contig(mm, addr, ptep, pgsize, ncontig); -} - /* * huge_ptep_set_access_flags will update access flags (dirty, accesssed) * and write permission. diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 03f8ced8b26a..1cb877f0d0ce 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -530,7 +530,9 @@ static inline void __set_pte_at(pte_t *ptep, pte_t pteval) } #ifdef CONFIG_RISCV_ISA_SVNAPOT -static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { unsigned long hugepage_shift; diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 01b1403dd4cb..4c441664db8a 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -187,7 +187,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); @@ -202,21 +202,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return true; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep) -{ - pte_t orig_pte = ptep_get(ptep); - int pte_num; - - if (!pte_napot(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); - - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); - - return get_clear_contig(mm, addr, ptep, pte_num); -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -231,7 +216,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, return; } - pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + pte_num = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); @@ -249,7 +234,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_clear_flush(vma, addr, ptep); - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); + pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, NULL); return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } diff --git a/mm/contpte.c b/mm/contpte.c index 5200c234404d..8d2ab391e0d8 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -20,6 +20,7 @@ * huge_ptep_get() * set_huge_pte_at() * huge_pte_clear() + * huge_ptep_get_and_clear() */ pte_t huge_ptep_get(pte_t *ptep) @@ -31,7 +32,7 @@ pte_t huge_ptep_get(pte_t *ptep) if (!pte_present(orig_pte) || !pte_cont(orig_pte)) return orig_pte; - ncontig = arch_contpte_get_num_contig(ptep, + ncontig = arch_contpte_get_num_contig(NULL, 0, ptep, page_size(pte_page(orig_pte)), &pgsize); @@ -85,7 +86,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, int i; int ncontig; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) @@ -109,8 +110,49 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, int i, ncontig; size_t pgsize; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) pte_clear(mm, addr, ptep); } + +static pte_t get_clear_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + pte_t orig_pte = ptep_get(ptep); + unsigned long i; + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { + pte_t pte = ptep_get_and_clear(mm, addr, ptep); + + /* + * If HW_AFDBM (arm64) or svadu (riscv) is enabled, then the HW + * could turn on the dirty or accessed bit for any page in the + * set, so check them all. + */ + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + return orig_pte; +} + +pte_t huge_ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + int ncontig; + size_t pgsize; + pte_t orig_pte = ptep_get(ptep); + + if (!pte_cont(orig_pte)) + return ptep_get_and_clear(mm, addr, ptep); + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + return get_clear_contig(mm, addr, ptep, pgsize, ncontig); +}