From patchwork Wed May 8 11:18:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13658564 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86595C19F4F for ; Wed, 8 May 2024 11:19:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25ED36B007B; Wed, 8 May 2024 07:19:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 20F496B0131; Wed, 8 May 2024 07:19:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B0236B0133; Wed, 8 May 2024 07:19:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E22C56B007B for ; Wed, 8 May 2024 07:19:41 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9BD7F140D51 for ; Wed, 8 May 2024 11:19:41 +0000 (UTC) X-FDA: 82094983362.26.E2435B5 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by imf19.hostedemail.com (Postfix) with ESMTP id AF5CD1A0017 for ; Wed, 8 May 2024 11:19:39 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=jupCytli; dmarc=none; spf=pass (imf19.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.48 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715167179; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eptZ1TC6/9ugjFjSR/N9yJ6dEI/3FvKeew1j70YJt7s=; b=U8IkbZ+H10NPByGtkY01Qo4jP9NATxp8vWH1h+cwXwHYAxgRGFpTxb5PxRi8+6i+nOGUN8 8qCq1OFuawytWk93Zwosj+T0XafVWcIXML6KhAZptfPLvyQL94v5CecOpnZwjwlnc+IWfv doJ6HZTY9wItmgDOGeEvxl+tgYZxov0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715167179; a=rsa-sha256; cv=none; b=0gDB5OqMtr/TuDU6CZ2foMml5awzqXr4+cNvZE07wdT0rk0aYIaV3akmw1UMh9Bw2Bpt5x fS2ws9WHjfw6mXEg9gYbzGd9sC9uznguKndPRuzPhdXmf+7P/4tVcLOht4YLVezyoa9hc0 mB+nUZyaSAdEYL/MYa8VCq1Sx03OeLk= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=jupCytli; dmarc=none; spf=pass (imf19.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.48 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-41bab13ca80so30715095e9.0 for ; Wed, 08 May 2024 04:19:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715167178; x=1715771978; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eptZ1TC6/9ugjFjSR/N9yJ6dEI/3FvKeew1j70YJt7s=; b=jupCytliTWsUhlYIMmUz99ZAwYmgEs/S+od3wG3ptBQI4Ag0PhYwWrIRii90IGUM0k pO1f5VbR69vWQwlaDIL8K+RaeDTSEq4nVv1+l781gdKm0Yc4SS4tPI+z+9/l+nZiNoX3 ud4gwNahpod3OspVXGf4RphFCF+dMU9ebz5gRpZRLpz+gNaLRsRuVdTrdU4/81DzItqU zUyo2saBRyR6ID4n8GIzifbbzajDJ/FoID4Ln4dkuaTkhWKTqt45TDMJllZcaL1646dM 37igOWSZiuKO+C+pwewFVQ+ypwe6BpDWgkou942rNtPzjgmqlTLW4oFKHGzZikqSFUT5 IrTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715167178; x=1715771978; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eptZ1TC6/9ugjFjSR/N9yJ6dEI/3FvKeew1j70YJt7s=; b=YmtZYzhxu1ci9Q/WZVrHhHJ7ljb3rrMSQI++1lVp0WrlRcK0fFyFPfe7GsjfD/fe6M dXFTTtvasf/RAaivQsz+rnABjt080HBANp9pp1X0sNimu49RXtzkyAOOu0utRZtCvlgE sfX47p2/F8LU2RIaRpv/jekD6fbNde4pq61XrurMPaY5HgB7Yc0b4lcNEvp+V+nIFKZu E+Eq4k4mn4dBtorgK+4T4mMD/zI8KFrqdG9yvEid1WlvaWJRWcAhXVkSzdtlw3hGXzr+ gx43BEc6kDmJecXLgq3Y7y9DATpGn1PZRx2w63AbUnFqo166p+qoz6M6Qrm1UMMs1RQs 4OkA== X-Forwarded-Encrypted: i=1; AJvYcCUkWwKa1L7nVN4+nHCwZSl2IpFJtLWjRETud/3XbR8FEw6cEfrOD2WZCWtODQQ/pgiau1q6wF3ijt7crqYousAcisU= X-Gm-Message-State: AOJu0Yza6Fo9Ji85Ww7D6uJVpslD105QOOYdU+P8TnUHxgxBoHXmKx+/ no7BAFKYN2uTUmUb1DUsPxmF8/MaCu87xKXFiUDjKWDqNiNhnL4pyb0Y3UJndDg= X-Google-Smtp-Source: AGHT+IHkzhqoYQlQehKFtJ3LJJ5+7RQutnDUAdNcSC3wCxgx+3uG7RvB8HMwIjEauHZgX9T5Lzrpyg== X-Received: by 2002:adf:f7cb:0:b0:34d:f6c0:cc69 with SMTP id ffacd0b85a97d-34fca2446c2mr1975865f8f.41.1715167177905; Wed, 08 May 2024 04:19:37 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id q27-20020adfab1b000000b0034e24be18a1sm15242609wrc.15.2024.05.08.04.19.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 04:19:37 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v2 1/9] riscv: Restore the pfn in a NAPOT pte when manipulated by core mm code Date: Wed, 8 May 2024 13:18:21 +0200 Message-Id: <20240508111829.16891-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508111829.16891-1-alexghiti@rivosinc.com> References: <20240508111829.16891-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AF5CD1A0017 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: xrgj9m7w6a46ho55tn5aof3eamduk8tf X-HE-Tag: 1715167179-156229 X-HE-Meta: U2FsdGVkX1/F0ywn/PK2W/vze0YGhvU0+mxdEBhNXe4PIQzPavv8Xom/uTCtggMo7bdYQYmmkhJyGPpjlRFSE+iAzUf1kYQzrIVj5fh7uWCJb2dcBpsfGynqzRfADi95C6PIX8D3g5EHrEFxdQwHk1GjOXLgIM3XV5s9K5Ch6Ihf1a2QzA/6kfDTkQc8gR1+9JvY0ezx25GcfwebxD73k83QoVcgNkG4zo4T7KlXq2UPhuQnKKA29oyeBOxDbXfcTYd/U2WbWAo2x5NnTscp5+vMVaqke/xnnxSXT/vdJh9gmNHzNQoYq6kYKxaNDMKSA2VyKsI2P2NX/BHmhUCbRAACLk4TAqLqFIl7IV6BcDogN2nw8ozXT8lfVAFtoqc0er/CBbyl6QyILKIK4HXGjta/CCoMBOKLj42rqdPnNgZQp8TWwudN9yR7SUA07CI2f+20fP96wAP+oBX8Uz5FB8m/iDFfsUDmjDh3J95sRHyzUT99EqsLBles+L0hnE3pZE1qvBhWC09yZx0hvq0VZSKWQbj8RRvf8Ie4P6988pU2S+9QOeArugGA3jC/JakKQg8IPwQVa+l7m2Bt5XGgEVr4UVVPDX5f20/vW0xC9+gWuyErV7rsgcenOFxRlvM7yymBF95xPCxBLkeutjRleOizTb1Xtx/fMv2m+DSua+60VGAiei51SBU4RA0FshjuAnj7Yq+UFaA+toeGD22u4RJulrcdVeaurV9p878TtCdCQD+PjbqWxT/1PKykvTKl0Ca5PHWCX+daXhs5UXU+Unu6HDZCkK7ocyhSaUJJTdZJVg1keLsM1TY3xHcP8MAMz9UwEUvJV0PZN05525kz0wTxQODjMOKj9hLzyLTf5MyHJ36UoMTqCkDevcwsaA7f61JSDnHW84Iog/NzrFQmUB8QwD3ZUfq1REV8QSlgA2bzCbe6Whw7jLO1WqFdU619AvZZUMtqQtUBOxJweHW +erKltoQ Hhge0+Gt8kTcsiuwoQAeaxr5s2zEff0ynK3Te9saW92hmaYetAKIx6GLS7KYi3neaFCLcasQ1HzYEmlZFDwGz5rCcaWPviwoInZVIdWxy5krp51zbFKGwrLbxpzPFYESRPYRMVaEPCGpgqHnSUIAVheNxH+VtkIzrTcqxr1iO/zT1r14CYmVbmgaIQfb9z3NlHpm8b8VVNKrqsUyPp696lI1vrLP2OyqrXfBIytIVUKcpTb6dT0eoLVlOXzVNFZDYeC7JHsR6goGU8SB7LGjSNHfZdflD1f0qGH33ahHZaFj7voOL8+7nslIRm/mmNxgKF+pVDE6ldnf+/w74x2MLTGZr7DKPhK3QgbpmhLgISd24W4PU4EW4nT2m63PTTEGy5FtqzVxNtnvJ0X408zwNedKF4wIz3tKYMCtAuRWSlSjy48O2qhPN/AIWxbgwVNsRtYhQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The core mm code expects to be able to extract the pfn from a pte. NAPOT mappings work differently since its ptes actually point to the first pfn of the mapping, the other bits being used to encode the size of the mapping. So modify ptep_get() so that it returns a pte value that contains the *real* pfn (which is then different from what the HW expects) and right before storing the ptes to the page table, reset the pfn LSBs to the size of the mapping. And make sure that all NAPOT mappings are set using set_ptes(). Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/pgtable-64.h | 11 +++ arch/riscv/include/asm/pgtable.h | 105 ++++++++++++++++++++++++++-- arch/riscv/mm/hugetlbpage.c | 38 +++++----- 3 files changed, 128 insertions(+), 26 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 221a5c1ee287..9fe076fc503e 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -106,6 +106,17 @@ enum napot_cont_order { #define napot_cont_mask(order) (~(napot_cont_size(order) - 1UL)) #define napot_pte_num(order) BIT(order) +static inline bool is_napot_order(unsigned int order) +{ + unsigned int napot_order; + + for_each_napot_order(napot_order) + if (order == napot_order) + return true; + + return false; +} + #ifdef CONFIG_RISCV_ISA_SVNAPOT #define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_CONT_ORDER_BASE)) #else diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 9f8ea0e33eb1..268c828f5152 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -297,6 +297,8 @@ static inline unsigned long pte_napot(pte_t pte) return pte_val(pte) & _PAGE_NAPOT; } +#define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) + static inline pte_t pte_mknapot(pte_t pte, unsigned int order) { int pos = order - 1 + _PAGE_PFN_SHIFT; @@ -306,6 +308,12 @@ static inline pte_t pte_mknapot(pte_t pte, unsigned int order) return __pte((pte_val(pte) & napot_mask) | napot_bit | _PAGE_NAPOT); } +/* pte at entry must *not* encode the mapping size in the pfn LSBs. */ +static inline pte_t pte_clear_napot(pte_t pte) +{ + return __pte(pte_val(pte) & ~_PAGE_NAPOT); +} + #else static __always_inline bool has_svnapot(void) { return false; } @@ -315,17 +323,14 @@ static inline unsigned long pte_napot(pte_t pte) return 0; } +#define pte_valid_napot(pte) false + #endif /* CONFIG_RISCV_ISA_SVNAPOT */ /* Yields the page frame number (PFN) of a page table entry */ static inline unsigned long pte_pfn(pte_t pte) { - unsigned long res = __page_val_to_pfn(pte_val(pte)); - - if (has_svnapot() && pte_napot(pte)) - res = res & (res - 1UL); - - return res; + return __page_val_to_pfn(pte_val(pte)); } #define pte_page(x) pfn_to_page(pte_pfn(x)) @@ -525,9 +530,91 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) #define PFN_PTE_SHIFT _PAGE_PFN_SHIFT +#ifdef CONFIG_RISCV_ISA_SVNAPOT +static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, + size_t *pgsize) +{ + pte_t __pte; + + /* We must read the raw value of the pte to get the size of the mapping */ + __pte = READ_ONCE(*ptep); + + if (pgsize) { + if (size >= PGDIR_SIZE) + *pgsize = PGDIR_SIZE; + else if (size >= P4D_SIZE) + *pgsize = P4D_SIZE; + else if (size >= PUD_SIZE) + *pgsize = PUD_SIZE; + else if (size >= PMD_SIZE) + *pgsize = PMD_SIZE; + else + *pgsize = PAGE_SIZE; + } + + /* Make sure __pte is not a swap entry */ + if (pte_valid_napot(__pte)) + return napot_pte_num(napot_cont_order(__pte)); + + return 1; +} +#endif + +static inline pte_t ptep_get(pte_t *ptep) +{ + pte_t pte = READ_ONCE(*ptep); + +#ifdef CONFIG_RISCV_ISA_SVNAPOT + /* + * The pte we load has the N bit set and the size of the mapping in + * the pfn LSBs: keep the N bit and replace the mapping size with + * the *real* pfn since the core mm code expects to find it there. + * The mapping size will be reset just before being written to the + * page table in set_ptes(). + */ + if (unlikely(pte_valid_napot(pte))) { + unsigned int order = napot_cont_order(pte); + int pos = order - 1 + _PAGE_PFN_SHIFT; + unsigned long napot_mask = ~GENMASK(pos, _PAGE_PFN_SHIFT); + pte_t *orig_ptep = PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * napot_pte_num(order)); + + pte = __pte((pte_val(pte) & napot_mask) + ((ptep - orig_ptep) << _PAGE_PFN_SHIFT)); + } +#endif + + return pte; +} +#define ptep_get ptep_get + static inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr) { +#ifdef CONFIG_RISCV_ISA_SVNAPOT + if (unlikely(pte_valid_napot(pteval))) { + unsigned int order = ilog2(nr); + + if (!is_napot_order(order)) { + /* + * Something's weird, we are given a NAPOT pte but the + * size of the mapping is not a known NAPOT mapping + * size, so clear the NAPOT bit and map this without + * NAPOT support: core mm only manipulates pte with the + * real pfn so we know the pte is valid without the N + * bit. + */ + pr_err("Incorrect NAPOT mapping, resetting.\n"); + pteval = pte_clear_napot(pteval); + } else { + /* + * NAPOT ptes that arrive here only have the N bit set + * and their pfn does not contain the mapping size, so + * set that here. + */ + pteval = pte_mknapot(pteval, order); + } + } +#endif + page_table_check_ptes_set(mm, ptep, pteval, nr); for (;;) { @@ -535,6 +622,12 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, if (--nr == 0) break; ptep++; + +#ifdef CONFIG_RISCV_ISA_SVNAPOT + if (unlikely(pte_valid_napot(pteval))) + continue; +#endif + pte_val(pteval) += 1 << _PAGE_PFN_SHIFT; } } diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 5ef2a6891158..fe8067ee71b4 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -256,8 +256,7 @@ void set_huge_pte_at(struct mm_struct *mm, clear_flush(mm, addr, ptep, pgsize, pte_num); - for (i = 0; i < pte_num; i++, ptep++, addr += pgsize) - set_pte_at(mm, addr, ptep, pte); + set_ptes(mm, addr, ptep, pte, pte_num); } int huge_ptep_set_access_flags(struct vm_area_struct *vma, @@ -267,16 +266,16 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, int dirty) { struct mm_struct *mm = vma->vm_mm; - unsigned long order; + size_t pgsize; pte_t orig_pte; - int i, pte_num; + int pte_num; if (!pte_napot(pte)) return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - order = napot_cont_order(pte); - pte_num = napot_pte_num(order); - ptep = huge_pte_offset(mm, addr, napot_cont_size(order)); + pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + ptep = huge_pte_offset(mm, addr, pte_num * pgsize); + orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); if (pte_dirty(orig_pte)) @@ -285,8 +284,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, if (pte_young(orig_pte)) pte = pte_mkyoung(pte); - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) - set_pte_at(mm, addr, ptep, pte); + set_ptes(mm, addr, ptep, pte, pte_num); return true; } @@ -301,7 +299,7 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, if (!pte_napot(orig_pte)) return ptep_get_and_clear(mm, addr, ptep); - pte_num = napot_pte_num(napot_cont_order(orig_pte)); + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); return get_clear_contig(mm, addr, ptep, pte_num); } @@ -311,24 +309,23 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t *ptep) { pte_t pte = ptep_get(ptep); - unsigned long order; + size_t pgsize; pte_t orig_pte; - int i, pte_num; + int pte_num; if (!pte_napot(pte)) { ptep_set_wrprotect(mm, addr, ptep); return; } - order = napot_cont_order(pte); - pte_num = napot_pte_num(order); - ptep = huge_pte_offset(mm, addr, napot_cont_size(order)); + pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + ptep = huge_pte_offset(mm, addr, pte_num * pgsize); + orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) - set_pte_at(mm, addr, ptep, orig_pte); + set_ptes(mm, addr, ptep, orig_pte, pte_num); } pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, @@ -341,7 +338,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_clear_flush(vma, addr, ptep); - pte_num = napot_pte_num(napot_cont_order(pte)); + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } @@ -351,6 +348,7 @@ void huge_pte_clear(struct mm_struct *mm, pte_t *ptep, unsigned long sz) { + size_t pgsize; pte_t pte = ptep_get(ptep); int i, pte_num; @@ -359,8 +357,8 @@ void huge_pte_clear(struct mm_struct *mm, return; } - pte_num = napot_pte_num(napot_cont_order(pte)); - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) + pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + for (i = 0; i < pte_num; i++, addr += pgsize, ptep++) pte_clear(mm, addr, ptep); } From patchwork Wed May 8 11:18:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13658565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3873FC25B5F for ; Wed, 8 May 2024 11:20:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C22E66B009D; Wed, 8 May 2024 07:20:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD3CD6B00A9; Wed, 8 May 2024 07:20:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A9BE66B00AC; Wed, 8 May 2024 07:20:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8AAA46B009D for ; Wed, 8 May 2024 07:20:42 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3FE2B1C09B0 for ; Wed, 8 May 2024 11:20:42 +0000 (UTC) X-FDA: 82094985924.23.2AA999A Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by imf07.hostedemail.com (Postfix) with ESMTP id 5A37140011 for ; Wed, 8 May 2024 11:20:40 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=jQpShfqu; dmarc=none; spf=pass (imf07.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.43 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715167240; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oM+WuwVvF178uIr+5Dc8IcJyYSbzDoUf3sQGwgQ7LEI=; b=n07FUnf409/GWA+pq6NmLHSVTIWIPxiw5a8UI1CsS0ErmGTX3bNZ8ouGLWiDJtjGFfu+RQ dChlnTosea8aLbzzQSagV+py2dLZB7qmA/FAGm4VScTjIE83JjIswx8Tw9MEALlYSe7Qz5 BgOW9M3gydzuRqzoiugqL6vS5aF72Nk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715167240; a=rsa-sha256; cv=none; b=Go7MQSV9v/6PBoCwLIXIqFyoDZgK9e3dVIROAGR4/5FwiG8RUEd8u+Lo7GHsSxDVi83nFv nZoMCzvdoLzCSc0OAgIKJgAUWhd2+q+tzmPK7YGUebyadHzc/bKGO/Ik61IM5WEqyVWVQx QV3mI507qdTy1ax/A7jy4c/vZ2cvcLo= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=jQpShfqu; dmarc=none; spf=pass (imf07.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.43 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-41b869326daso28586055e9.0 for ; Wed, 08 May 2024 04:20:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715167239; x=1715772039; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oM+WuwVvF178uIr+5Dc8IcJyYSbzDoUf3sQGwgQ7LEI=; b=jQpShfquj5GTn2UPO3dgwjixSssOdffM/zD4T48Z39tSUrEiPQuV69IZ7rYV5GCsNO 06dqAOD13qjjUw2eWQ5fzm4DHtTZi2ENDTKJVXi1qbwG/YC5usPaqZPaWpYBPCyBNFZs jKYOnNPJShgRegJEPS36RNUYctJoKGj59FCvicS5u15xHizKxGOoCIlhxfBHR0nJXnNq vz/s3STxMe0AMonrq9J+N1Lp6Zi8Zy3APJRs9IP+VsHNxO5aPl43DXFSA2HCrYPncPqt VfvQDgO2Y04RO+ic3KMpFHHb05zx2ByXfqZlRNFRqKfZdGaVHVX4vqJRuVe6Ejjfhk7p kXuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715167239; x=1715772039; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oM+WuwVvF178uIr+5Dc8IcJyYSbzDoUf3sQGwgQ7LEI=; b=cb7Q/QsbwOMC9eczX+5SJTgGABxfdQuQGWv8pqciZjo43HMx3n9NrIeDOhfOFM63Dw 0dEakL5KP+0XaaQZ9+0WxF1alb2VNLjGFV+VnP03PtEr6OF9w0oNqQv43QaShTvlmIbj 75FKOlKhNIrvXj4dWrttO+ZV2Nq86c2kuRQX81Pa/rvDgxizjAgbQytwEK1Appi54odV db/IiWBkQhf2GmwcKogcaREOmbiLLzVjwDPQKp+pvzzCry+HYPBNLln0cJVWysBUQC9K 0VA/dt7WWK2BO7P0P9IXNRuzdTnY9PTwFc38JClhbe+P5bIhYevd53oVLd05DFXfRTFG XONg== X-Forwarded-Encrypted: i=1; AJvYcCUIGisYqv/9Y2PEnJCGtafIItBfp0VMMrdqwPPIi4Hd7nvOanFmbjSxpTmBNxNtJCtB0NzrcSPCqZHn8Yxx0Ufvtbc= X-Gm-Message-State: AOJu0Yx9PH1NQdix4kUbmL/QFOUkj4iyvvdhvRPfR8k/nLkQMt8Sr10e ttD6aXLi6LOjXlT5GHxd2a+O1KswbZ3g9WFkS1mEPL2s1btJrB1dEb72dttLy3I= X-Google-Smtp-Source: AGHT+IEtMJJRHnAlx2d7EmAnwEp8LMaoiRtpdi+BWrYkSsfaWBlt5vRm4skQUYamx6SwwnjaoJRU9Q== X-Received: by 2002:a05:600c:1395:b0:41b:e201:eedb with SMTP id 5b1f17b1804b1-41f7269010amr20541325e9.41.1715167238957; Wed, 08 May 2024 04:20:38 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-41f87c254f1sm19678785e9.14.2024.05.08.04.20.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 04:20:38 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v2 2/9] riscv: Safely remove huge_pte_offset() when manipulating NAPOT ptes Date: Wed, 8 May 2024 13:18:22 +0200 Message-Id: <20240508111829.16891-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508111829.16891-1-alexghiti@rivosinc.com> References: <20240508111829.16891-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 5A37140011 X-Stat-Signature: iqan8mpqn75im3wtbt1sw3ye4m8ebsck X-Rspam-User: X-HE-Tag: 1715167240-465060 X-HE-Meta: U2FsdGVkX18VpWho7fE2mjsHD289S428d1woTX43yvl7EnXg88jQbSKKpMkgjHl1RUQRxlKT0yBmX14pUUsSZVOaZGSngl3NzQ9I5dyvm2EO18wtpnwBeZMg7AoU4E9bLEcSnDiuaKT+xV2QmziDP1Vdqjk74feOlG/Fu1rZwjvlAf53iT2seLqK11YQlhybIvV+R1Q45dvS2tyNqmSSRPW2GzGQDCn/oB1ksYXlta01iP+PPfIn0mkuDWH3dKHLUKXJyZbPsrp9ROCSCgxRlH+yz5LeMKoG5F0m+f19AEiffnZswXwzh4xsOfvsSjtlCmgSoZn3nZMrKeXaiVLuuaj+Qi1vNqKhNZwdqfg1hAnpPS6DJJxoNu4A/QummbLAJCqzMtChqJ0cbBLJ5GXHH1vKMEKyNd5nvu1Vlr2hH4hC+pOR85+DCs0sMEtXw6VHZLeEfDWrXFEBC11StedPx99m7MmdqSBq6x+jst7+axCbb14o1SaDeNaENW5aXp28N3VlvC2YVliSX0DhfaaLpymmxXknzjLAoR51qEiIBtePACdz4zh2ZKywBNf00HT3DEhWwxYC5TMI26H+GLbByURBwu3Jj+fhxW4oxTHuMf9+Q6f8Zu6/UjjsbHkbglY7Kc622cukqUgBfQSg23zOfsxstC722hVV/RUS7ZFWaedhfg06kUx405OC1BwaJ+lazsGSB7zt8Q9HTTDRg3hirpIZyr15j3EDYbn0JsfOMQhclhbZ+qJZiR9CwkuqVYsRfR4uGNm9/J3x+7dWLXWOsARwtkJDl9RMhTtzIy064Rq9Seg4zSQMN7MPTjxArenaaSiMiEAl9FtbyBZbSU6e0JGXz4GbTVXafbW0MbhOtjTgTFHWQR6lI2quM/OUHkjMxp1eJkypoNyaHgDalQIGH13UkMm6McTB1pOr82wNvO32+oK9FzFTSp/RA1nAuRNUHXrMEtDJy8CKhH8mmt7 r1gSQHZ3 LJ/CKuvuzmAB2sYGEBoNklGQUO+fzWWeq+CxkQd5mz1UD5st9vWJZguqprrqGpbIPv2rS3luzpCIw+NceLXVIUqoR4UizUoxzmpXTxWyn9TPGojASZQ4sS8bqg6E5jLxYuc94SbxkL3om4y2y7jZGrVuHck0qa9hu4DhAVEUVkG38MqvWhvsDfEFsLsMo3koJoIr/KpgjTkshJmvL7ksVVWyfbWVorvgAaG97/9JwyM2Ak0KTxusOyeg269BjkHuNPsAMqbWbTkK6f5JpTvzNidi3qihP8a5/0RsZ4AW8B585KiSjF23ZxudfnabWvNkz/3qPfJ7+b9/exQ43gOuhZifbGXw31PSiLMum1/PSVXGLyo2/THVgh8Zhe+GSGD98ynSZYa6ry8psp/rq/hzZ2ZkJknKGzJEyCr+wmWHuEm6iTIAFkgOYAvQxNDPuIU0IVJX9NaoM1i6jDwcypFA85gf13VDWwybC6PJHQFmO41WiE4FePyrLffXhSw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pte_t pointer is expected to point to the first entry of the NAPOT mapping so no need to use huge_pte_offset(), similarly to what is done in arm64. Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/hugetlbpage.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index fe8067ee71b4..f042f5c8bdb7 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -274,7 +274,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return ptep_set_access_flags(vma, addr, ptep, pte, dirty); pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); - ptep = huge_pte_offset(mm, addr, pte_num * pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); @@ -319,10 +318,8 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, } pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); - ptep = huge_pte_offset(mm, addr, pte_num * pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); - orig_pte = pte_wrprotect(orig_pte); set_ptes(mm, addr, ptep, orig_pte, pte_num); From patchwork Wed May 8 11:18:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13658578 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6DB5C04FFE for ; Wed, 8 May 2024 11:21:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6562D6B013A; Wed, 8 May 2024 07:21:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6066A6B013B; Wed, 8 May 2024 07:21:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A72B6B013C; Wed, 8 May 2024 07:21:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2E4EF6B013A for ; Wed, 8 May 2024 07:21:44 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D43041A0D75 for ; Wed, 8 May 2024 11:21:43 +0000 (UTC) X-FDA: 82094988486.14.27762E6 Received: from mail-lf1-f54.google.com (mail-lf1-f54.google.com [209.85.167.54]) by imf27.hostedemail.com (Postfix) with ESMTP id E42294001C for ; Wed, 8 May 2024 11:21:41 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=CAZMK13b; spf=pass (imf27.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.167.54 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715167302; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YO7yDIQr+QJFeAZFjf5GxhWsbGb5WjqA2EGZTufF1Fc=; b=yErFzmu9gOWF6d1hmXn3efJLj9FeF0ah02gCg5WL6236Bji6AUtOicpBbsI9IXSoRe4WnE PQnfe1mQvG8FYUsrF17iOV+MKJL2evRA3EfVtFQenpQ+rHFMkdRPfujvCKa0AQcp3VZIvg 825c8HktSqXuisugivAi/MLELBUTubI= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=CAZMK13b; spf=pass (imf27.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.167.54 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715167302; a=rsa-sha256; cv=none; b=FvsYprhqWetLfTo0RpIwt/ZFVC2s9dhhE9wbl5aSkAXNWDyf89mCf+PMJ/i1CKM2b4p6+k VtlLvcZfzsiAur5oLZvNY4fdGVHrYglm1IP9EffaVDLsQoMYiG4tWtktRJ12f3daGhTL0V hNyZT6W5aJno5hdh7VjQ/MCaLk2Crw0= Received: by mail-lf1-f54.google.com with SMTP id 2adb3069b0e04-51f45104ef0so4603802e87.3 for ; Wed, 08 May 2024 04:21:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715167300; x=1715772100; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YO7yDIQr+QJFeAZFjf5GxhWsbGb5WjqA2EGZTufF1Fc=; b=CAZMK13bC07jAhJcTtUSwXj7Q2x/u2jBc/BIRNcs/xNlTeE6qpWdg+X4eF9UA4s3/c TBJyhSX1HSEg7mNR493rlGtCncezR4hTsU92JrpKt34bnfQ0oBz8UxC17HR5T/gcSoGO OxjPCBMfd7jM3OIg9cBnVzOZGsXmsOp3F3+LpgOJ+Oqu+fC+ZA4bnv/8kARi3HwpSNnF ZC6yWETg1VS4JXpfFi+x8JWRZ/rEGL6xHKlcTek11gtd+slZCZxo+QfJHYzGZ7dNGCuY 2DrCjg+2K2QYpeFLroLuaA02B3ItQ9z1lj8dC5zisg89N4rQYA0k4t9qd6ASlpujWl8F MNJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715167300; x=1715772100; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YO7yDIQr+QJFeAZFjf5GxhWsbGb5WjqA2EGZTufF1Fc=; b=SEBWCpfWSZ7zOZXzoqwQCJWeOq2OK/g4d9xk/dNrnKG9S46XByy2zJ+s/Z1OfXhhct kK52NNCBhZ+xWfoDzQEBnUuH/wD5cq5tOA3WyCCeZWME6lX+AJ/2bVR1OCIoMavtFJ7D 0/B6ZEfIwZxbYhrFGS54R0baYqXAgcUNyGobxee/CLEUoNRo1av5cB5zzbFm4e4A8Lkp DeCXTCqzRb2VjP6K1xFQ/pcsP5/60srRj6kYisd6FNIezxoPWoZQguXWrwsdSGGl+Pze hUzzuwDW5zdAn1ie/9cPbZjwCuMJ7AgavG+CIp5l+oOppwLJkR6GKvIzIsGSAcZU5aSk hSKA== X-Forwarded-Encrypted: i=1; AJvYcCUvC7eFAtSPr2pomb0HD2l93lYF5u+eeCNNvpvyPM7bWt+Zo/nG1aZjEny1mNWTHaBWiuuamhQU5MwR1l/y80g0xdA= X-Gm-Message-State: AOJu0YxBMjAn1OFZ/Pxq1vX9I0/bf0a/rimmG/kGqSmpCjxWrC52iAXG mHimh0/arFbkcxitvr9LVXof18MDzTmlVc9xlZiFFMKFUyCj+CTk4G3yT2Savnc= X-Google-Smtp-Source: AGHT+IGlIF5qrXIZLx6k07SYUl4uhRcj6nfG/t1miugJV1fwoSAC2HF4MbIjFtO5Wp6+YeTrOF+Wmg== X-Received: by 2002:a05:6512:3ba6:b0:520:dc1c:3c5d with SMTP id 2adb3069b0e04-5217cc458a8mr1935422e87.42.1715167300050; Wed, 08 May 2024 04:21:40 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id h4-20020adff4c4000000b0034cee43238fsm15052972wrp.27.2024.05.08.04.21.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 04:21:39 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v2 3/9] mm: Use common huge_ptep_get() function for riscv/arm64 Date: Wed, 8 May 2024 13:18:23 +0200 Message-Id: <20240508111829.16891-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508111829.16891-1-alexghiti@rivosinc.com> References: <20240508111829.16891-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Stat-Signature: b1xg7kythfbm7wi79qaafcsjcj9ibn3g X-Rspam-User: X-Rspamd-Queue-Id: E42294001C X-Rspamd-Server: rspam05 X-HE-Tag: 1715167301-892774 X-HE-Meta: U2FsdGVkX1/dYud8IbStudpcJZXe2lE/I2TTNSyAW1pZOYGygfugUxUjj8Xr1FLnAl97Tyv5RtSVCVFYIie1ol5JceAbeFqCLq4UG7m7Dr/7+DIkzZEuzYPT5JpjQ2VI1/RqxFNmYuHvtoMCEYkNFWZI51y1HLHCguol/Ms9bAV6BxRSxLLi/DlJPF0kdcLdc8kubALWhTzoLcPsE+5Gv48NK8OwQ0E9UthumqHObV9N36vYxyd74sIW7ouFqkx2saUL48xzHuIs2EvYgcRP0/2jJeBr5C4nztW8TD3RYb9m16Hg4eWwCHKmo6zGoYrf26ZBceAqEF9mpCSsuP64d4eu7fr0VvI2QwECpx6ZWM2qTfXZcPLqKqUeoWtl+hdvYIlXne1JXwleDjtYAh4AQG/D++nHJhAyRe+G+VCJcPi8XnabBupvU82i/L7GgBSOiOFIQSJa+q0Xkcdfxtc8FCoSi75nsOM2xrIrnHjXpINNn06ErkJVhd1qzd00S89j2B9/EnjeqI7OyuqB1pbOsYNNYQDQycIHEa2LUbknQjT0KTkOBPnBGr0lrTvOVO9Jy6yQyjfjDeBEZIXEi5AciSAR5OPFMzReYbfirc62Mu/vvKu1KA1iuKu9kNIQJsYzqxgly+ylmF3QfvPSsYqV3/FCbAn2tAmGV03xRpXi6/nzek/TFQZjq+CgzgVeH4nU9B02XbQKXSIu1VI2qznyHPE5aL3H0UNS9ghdW0VYBLve6Kt07zspSaLCj0vYHGHjM6jpUkCCbURku0wTw9WD6qolCVwemAPRyLXxA51Wv9reC1dlvc5CTW4oImYYSH37gGsvRxn3tCd3Xv1PVsuOOr0IGnjEbio5oceuBJGXxDeAC82ngPXlwwLGG8gPa4ffBUrcMFaJ/MUshd2yf5xbdzjT8QDOKCNngEBfXtwgI9WVgcwdp7VwI5dyPedATbH5/lEdsx6pd/5UZ1EvvN1 ey5fpLOx 2+eEFA+ycuR6uULDYjl+P/iE7d8Vbce4nVVV1g+S7OJGJBxvI6e8VjUXMEABB8OGu3o2VWjeWhd4804EEZRnKaALn0NGZoQZbV0GLqtUFuAbeqD401QIHskQTpFsrqsShKl4kC/n0clJHv8/lpKSEBCnMXXI77PmSFuq3FQ8eHy8kAIllqsoECdFXopRbBR9FKGhsTq9QxNqr/T9NHRM4UkPsGmtg2I/V2QNJZUR1NorKADHe7NhCemiOfdVUcBNIpVxkSqax/5K2wb3VeGfJ5ulRKfGBSDsx1D4j9X1apIpJarBYxN4jqmMVznnLlSiQc8Ej39Q6y/gO2dR82dU95vvbc/lujKFLFUDoPPaIcylvauUoG5f0lvg/dQRdU57k5CENemYxLkIHeFCeF5xb9ZddemenFousGrua6ASaUMWTLVMqvBp6l1Ig4TSMRSHfua5w2DixqpdBgAuFr2224BkbkNw1RiWfhfyHZnH26G4Fimk05gkjKvwmTA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For that, we need to introduce: - a new config: ARCH_HAS_CONTPTE, - a new arch specific function which returns the number of contiguous PTE in a mapping and its base page size, - a pte_cont() helper, only introduced for riscv since we keep the arm64 naming (contpte) which is more explicit than the riscv's (napot). Signed-off-by: Alexandre Ghiti --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 30 +++++++++++++++++ arch/arm64/mm/hugetlbpage.c | 55 ++------------------------------ arch/riscv/Kconfig | 1 + arch/riscv/include/asm/hugetlb.h | 2 +- arch/riscv/include/asm/pgtable.h | 6 ++-- arch/riscv/mm/hugetlbpage.c | 24 -------------- mm/Kconfig | 3 ++ mm/Makefile | 1 + mm/contpte.c | 45 ++++++++++++++++++++++++++ 10 files changed, 88 insertions(+), 80 deletions(-) create mode 100644 mm/contpte.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7b11c98b3e84..ac2f6d906cc3 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -20,6 +20,7 @@ config ARM64 select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_CACHE_LINE_SIZE + select ARCH_HAS_CONTPTE select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index afdd56d26ad7..e30149a128f2 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1638,6 +1638,36 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #endif /* CONFIG_ARM64_CONTPTE */ +static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, + size_t *pgsize) +{ + int contig_ptes = 0; + + *pgsize = size; + + switch (size) { +#ifndef __PAGETABLE_PMD_FOLDED + case PUD_SIZE: + if (pud_sect_supported()) + contig_ptes = 1; + break; +#endif + case PMD_SIZE: + contig_ptes = 1; + break; + case CONT_PMD_SIZE: + *pgsize = PMD_SIZE; + contig_ptes = CONT_PMDS; + break; + case CONT_PTE_SIZE: + *pgsize = PAGE_SIZE; + contig_ptes = CONT_PTES; + break; + } + + return contig_ptes; +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 0f0e10bb0a95..9e9c80ec6e74 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -112,57 +112,6 @@ static int find_num_contig(struct mm_struct *mm, unsigned long addr, return CONT_PTES; } -static inline int num_contig_ptes(unsigned long size, size_t *pgsize) -{ - int contig_ptes = 0; - - *pgsize = size; - - switch (size) { -#ifndef __PAGETABLE_PMD_FOLDED - case PUD_SIZE: - if (pud_sect_supported()) - contig_ptes = 1; - break; -#endif - case PMD_SIZE: - contig_ptes = 1; - break; - case CONT_PMD_SIZE: - *pgsize = PMD_SIZE; - contig_ptes = CONT_PMDS; - break; - case CONT_PTE_SIZE: - *pgsize = PAGE_SIZE; - contig_ptes = CONT_PTES; - break; - } - - return contig_ptes; -} - -pte_t huge_ptep_get(pte_t *ptep) -{ - int ncontig, i; - size_t pgsize; - pte_t orig_pte = __ptep_get(ptep); - - if (!pte_present(orig_pte) || !pte_cont(orig_pte)) - return orig_pte; - - ncontig = num_contig_ptes(page_size(pte_page(orig_pte)), &pgsize); - for (i = 0; i < ncontig; i++, ptep++) { - pte_t pte = __ptep_get(ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - return orig_pte; -} - /* * Changing some bits of contiguous entries requires us to follow a * Break-Before-Make approach, breaking the whole contiguous set @@ -243,7 +192,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, unsigned long pfn, dpfn; pgprot_t hugeprot; - ncontig = num_contig_ptes(sz, &pgsize); + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) @@ -390,7 +339,7 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, int i, ncontig; size_t pgsize; - ncontig = num_contig_ptes(sz, &pgsize); + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) __pte_clear(mm, addr, ptep); diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 8a0f403432e8..38d93cf44456 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -19,6 +19,7 @@ config RISCV select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_BINFMT_FLAT + select ARCH_HAS_CONTPTE if RISCV_ISA_SVNAPOT select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEBUG_VM_PGTABLE diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 22deb7a2a6ec..f195f611722b 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -49,7 +49,7 @@ pte_t huge_ptep_get(pte_t *ptep); pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#endif /*CONFIG_RISCV_ISA_SVNAPOT*/ +#endif /* CONFIG_RISCV_ISA_SVNAPOT */ #include diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 268c828f5152..66061002ff36 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -296,6 +296,7 @@ static inline unsigned long pte_napot(pte_t pte) { return pte_val(pte) & _PAGE_NAPOT; } +#define pte_cont pte_napot #define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) @@ -560,7 +561,7 @@ static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, } #endif -static inline pte_t ptep_get(pte_t *ptep) +static inline pte_t __ptep_get(pte_t *ptep) { pte_t pte = READ_ONCE(*ptep); @@ -584,7 +585,6 @@ static inline pte_t ptep_get(pte_t *ptep) return pte; } -#define ptep_get ptep_get static inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr) @@ -686,6 +686,8 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return ptep_test_and_clear_young(vma, address, ptep); } +#define ptep_get __ptep_get + #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index f042f5c8bdb7..be129f4f1503 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -3,30 +3,6 @@ #include #ifdef CONFIG_RISCV_ISA_SVNAPOT -pte_t huge_ptep_get(pte_t *ptep) -{ - unsigned long pte_num; - int i; - pte_t orig_pte = ptep_get(ptep); - - if (!pte_present(orig_pte) || !pte_napot(orig_pte)) - return orig_pte; - - pte_num = napot_pte_num(napot_cont_order(orig_pte)); - - for (i = 0; i < pte_num; i++, ptep++) { - pte_t pte = ptep_get(ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - - return orig_pte; -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, diff --git a/mm/Kconfig b/mm/Kconfig index b1448aa81e15..c325003d6552 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -981,6 +981,9 @@ config ARCH_HAS_CPU_CACHE_ALIASING config ARCH_HAS_CACHE_LINE_SIZE bool +config ARCH_HAS_CONTPTE + bool + config ARCH_HAS_CURRENT_STACK_POINTER bool help diff --git a/mm/Makefile b/mm/Makefile index 4abb40b911ec..605ead58403b 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -89,6 +89,7 @@ obj-$(CONFIG_MIGRATION) += migrate.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o +obj-$(CONFIG_ARCH_HAS_CONTPTE) += contpte.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o ifdef CONFIG_SWAP diff --git a/mm/contpte.c b/mm/contpte.c new file mode 100644 index 000000000000..e8574051d0b9 --- /dev/null +++ b/mm/contpte.c @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 Rivos Inc. + */ + +#include +#include +#include + +/* + * Any arch that wants to use that needs to define: + * - __ptep_get() + * - pte_cont() + * - arch_contpte_get_num_contig() + */ + +/* + * This file implements the following contpte aware API: + * - huge_ptep_get() + */ + +pte_t huge_ptep_get(pte_t *ptep) +{ + int ncontig, i; + size_t pgsize; + pte_t orig_pte = __ptep_get(ptep); + + if (!pte_present(orig_pte) || !pte_cont(orig_pte)) + return orig_pte; + + ncontig = arch_contpte_get_num_contig(ptep, + page_size(pte_page(orig_pte)), + &pgsize); + + for (i = 0; i < ncontig; i++, ptep++) { + pte_t pte = __ptep_get(ptep); + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + return orig_pte; +} From patchwork Wed May 8 11:18:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13658579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA694C04FFE for ; Wed, 8 May 2024 11:22:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F2DE6B013B; Wed, 8 May 2024 07:22:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67ACE6B013D; Wed, 8 May 2024 07:22:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4CED16B013E; Wed, 8 May 2024 07:22:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 25F846B013B for ; Wed, 8 May 2024 07:22:45 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C5FB3140E88 for ; Wed, 8 May 2024 11:22:44 +0000 (UTC) X-FDA: 82094991048.22.F4BAC15 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by imf23.hostedemail.com (Postfix) with ESMTP id CC25D14001B for ; Wed, 8 May 2024 11:22:42 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=TjwEqfEK; dmarc=none; spf=pass (imf23.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.47 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715167362; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9tID4t35K2Xq8XfdKgfehJoKp8xw/RZTmJgN0BGYZu8=; b=DH2GqN21tSNiOU/6MAZskiZKDDjS7VlYIoOaw9fg2oFeRutLv8vUmGbX90OG4MmJdouy88 oODCl7Yk0wp7t3qK1SQjbOHumisWZSuqohTvnYc+z+awnW/btgXstKSwpYnyrqZEnnlwVf kqRN5TzpVa6cqc4+vScakAI3EEeBydM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715167362; a=rsa-sha256; cv=none; b=zwxlzhXx2xE4eMNcaH8yQeNUZ7IRsuiQr/hbimN3wBQS4wiBc7R6FEkvwX8rI2zGN6xx/B BWsWtDvO4IaIjDBCKsRYUTRVppO+jYujweXQi/FhUxfNth7FMpCfyLDLxy3uB/oyUt03pl /TPSBEwb23KorjvyO+NTuViijoz+YOo= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=TjwEqfEK; dmarc=none; spf=pass (imf23.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.47 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-34ddc9fe4a1so2946899f8f.0 for ; Wed, 08 May 2024 04:22:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715167361; x=1715772161; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9tID4t35K2Xq8XfdKgfehJoKp8xw/RZTmJgN0BGYZu8=; b=TjwEqfEK8/cL2wLB7PQErYSWb6Chb2+XfrUFqs+gyErRqmktXgPzp2tWy0tmlGvF+W opEAt0DC+uxqv5ygwSGUNfCMiDHdq/XujnKAivSwt+nqPJF7ajmDUonZ3qpFlaT8nuRp pTQqXMbD9oQ/vRBZGTI1ik+eC76K25plYOiezM47sHl6638ri5X+Y/4bP/bjCCdSkrqL /goCllKcUqJ/Pbd24zC79vUUwH961pfd3gSqUUaab41S8DyAmdiGzaOWa/htloU3xeut L1P979eSMEXEI5Jmo/QMYBW86PWKZAFg5i2cWuz/E2xK7U52btUkkTWzk5ZXAtNFbavs ct0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715167361; x=1715772161; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9tID4t35K2Xq8XfdKgfehJoKp8xw/RZTmJgN0BGYZu8=; b=XkT3QYa7xtrYeuOatEa1fyG5PlgFzexybI88MFlbhTMkGGWCkqoDXSIbWI7dMXGMWE jeukFarjbr8Wf9KeQYDeSq0EPrzPo4pDKw31fdQLV0MHEQvsCI+RxyFL5F7lgN3c79tU CbbxC1IyEotZTMeEpjm5omqzF7jJILjCc4KVb/pTZq/XiTsPWFzZUCCcUZlZaO2bjq6p 7QEYwgNfzp1vTmqmXQTOtFP8JOeqGjIH7Jp9u6Gs99O5MudFSPUOOc8uaepzuLdybCXl NVXFklUX0y0o1745n7PixJ8Ud4ex5KJg0TMo/8D9/SpjrSxLCspDFo9zxKC0Z1sDyXvt lu8Q== X-Forwarded-Encrypted: i=1; AJvYcCVJf70EAefaZbh65YeUp8eOnExVQ382rLbj1XETyDkcZGVdH0yFADsPOxSLpRGu5EQX8WK2bXZmdkDQ8O+N45Hfl4M= X-Gm-Message-State: AOJu0Yy5JIDchP1GoRPgudT1kfVdf6TK/2cjbz1pzeUGHlSjESk7Mg3e ZtFjYU7QvJ1AB/BzLs59ox9FpVQ9S1KvoU0z3ShTRhX3Uk90oOBb3EkZJQ1TFWk= X-Google-Smtp-Source: AGHT+IEhj4x90HF7qk4DaGHuIQAJSSwdJ9BqGTmqhMoZKfcOAwPZQ6Sx+t8ghfaCru80kzJxCAbXqg== X-Received: by 2002:a05:6000:1b85:b0:349:7e8e:b29a with SMTP id ffacd0b85a97d-34fca80e9a8mr1884287f8f.62.1715167361084; Wed, 08 May 2024 04:22:41 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id p12-20020adfe60c000000b0034ccd06a6a3sm15154549wrm.18.2024.05.08.04.22.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 04:22:40 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v2 4/9] mm: Use common set_huge_pte_at() function for riscv/arm64 Date: Wed, 8 May 2024 13:18:24 +0200 Message-Id: <20240508111829.16891-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508111829.16891-1-alexghiti@rivosinc.com> References: <20240508111829.16891-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CC25D14001B X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: es5ung5kjw9xg8b8gag7c36gmofsf3nz X-HE-Tag: 1715167362-87333 X-HE-Meta: U2FsdGVkX19a+Xl5WHglQM6V/Ra51Eyhk529lPms6yKhM5TwOJ+9IiL2B91IKGmj5YtjJj9KXQCHbrGEVH2fP6eGRIc4zO/2uL12BBhqhC6EGjI3mv5KB+Bs0pStbrSfCM4ODkGaMOuPzeiUTeDG7r+NuhNjlQxryedbWnsUPPnM0yepwACxZdxdN6W8VOi9sN4Pdd2qPnK7UmAeviS568qZyiHOLHpfsFsWikjjeXkMY6taFBjSZkNjIzfPsrTi7TVwFMs1ub9s11o22a8ig3wDlYtxCQrNOfhO151iZJHAVM81X0/WKk1fR5FP9LgxAO92N4WINaouBAN4D+TAuHSwazEOtVCcwnJuymZ1ltNZu5apTFkw0zrduE0vy6aVaHjIt4Yh3pfkxthSUenZbG9QLyCox45U6GPXwbCcPxaACF/W+eVJSR9pcpUrsS/dhLDU2br6X0J8fi0S8+hZdu0aRTDuIjlsNfDkmNG3z+Nuv4yPAf57EHAPI/sZeLUbx3PUTJ2o8+MH0ZPcfBIc/CxpNN7evxXNtP0NOzVBWWGRJL9NLo6tvtgJmxHmwxFgPaZDCtxzuYJuQ1loLYfvIfqDJWnrnGBRbb5mjY5HYVkdrTjy6TrdfxgbKep7sTNEWdcDQXIDPeLz2KNTIAlHe1UN1vqDHVY0MjszfdA282sMOuYxTnE1jllOGT7BzBAUNvzZVR9p1orFQC+KH/RNtVNiSYQ+zok/EVEGLIQBr9S7X/p/ERVOl3q/O1iYPhz+hTC5+sYKPZzMxcpUsy++HMQR/HqNTrYAvHwEC8UJtF91nNw+MmqZzU8pEEG9CuBBg3ee59Xe4zYMEAoY539N+O0VfVskniKZ9D1N9L1By6k/xe8kyttDDfp/E5IqHe9pRzAlGOGrEDr8IA6bM6shtzkdKbLuuDwf2MMGZbaJvwUeTNpLUCrExVYTVRbAvIhkHJ8E6HuwkKow4DGARkp 4CR+FNuN OvkDvJnTe9U2oJfb2AvCrFfKr+msvtNFXQgfvboLtq/i+0bG+LwrhlyIpnOWWXE+kaKajWJkr4aGjToXMc3hFfr7xIVrMN1oj2/qpsMXiQVfT99ZLddr/bFpenY3I7bYCXBpeZVJpTI6iDyOEfPaqFMNtqlN4B59vjHfjhFlubnUNm2zJ1PL7GFUPuZFWCsXKiBUWxC35s/1/gJyLmuFk8UJalLg82iSTvB7Lw4s5XeJNZjfKLZR/3lmJYOn0OyBMnGG8YA7UIxO33OXDt0UVYvSQDf1uD5jzyuWyw+zLz0fUccXT5lRX36a/7317pG/5dN12JgeaCR0MqNYokuTZDptstgBnmwqGIhIlb8LpDW61wMPk83mK3+Pwn23RK8bmAYXggd/WMX/yzYPBxF0jRqGRHzOC4069CLRspur3y2y9+HFjvgJg66Mt8/Wmlm/BkcAzwBJTpg15YJLkvPRNg+5EPYNh9tGBlKy+LVZKfm2+EHiERHT2Psdbsw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After some adjustments, both architectures have the same implementation so move it to generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 14 +++++--- arch/arm64/mm/hugetlbpage.c | 56 ----------------------------- arch/riscv/include/asm/pgtable.h | 39 +++++++++++++------- arch/riscv/mm/hugetlbpage.c | 62 -------------------------------- mm/contpte.c | 59 ++++++++++++++++++++++++++++++ 5 files changed, 95 insertions(+), 135 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index e30149a128f2..2e0415fd5083 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -372,9 +372,10 @@ static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) return pfn_pte(pte_pfn(pte) + nr, pte_pgprot(pte)); } -static inline void __set_ptes(struct mm_struct *mm, - unsigned long __always_unused addr, - pte_t *ptep, pte_t pte, unsigned int nr) +static inline void ___set_ptes(struct mm_struct *mm, + unsigned long __always_unused addr, + pte_t *ptep, pte_t pte, unsigned int nr, + size_t pgsize) { page_table_check_ptes_set(mm, ptep, pte, nr); __sync_cache_and_tags(pte, nr); @@ -385,10 +386,15 @@ static inline void __set_ptes(struct mm_struct *mm, if (--nr == 0) break; ptep++; - pte = pte_advance_pfn(pte, 1); + pte = pte_advance_pfn(pte, pgsize >> PAGE_SHIFT); } } +#define __set_ptes(mm, addr, ptep, pte, nr) \ + ___set_ptes(mm, addr, ptep, pte, nr, PAGE_SIZE) + +#define set_contptes ___set_ptes + /* * Huge pte definitions. */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 9e9c80ec6e74..b8353b0a273c 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -159,62 +159,6 @@ static pte_t get_clear_contig_flush(struct mm_struct *mm, return orig_pte; } -/* - * Changing some bits of contiguous entries requires us to follow a - * Break-Before-Make approach, breaking the whole contiguous set - * before we can change any entries. See ARM DDI 0487A.k_iss10775, - * "Misprogramming of the Contiguous bit", page D4-1762. - * - * This helper performs the break step for use cases where the - * original pte is not needed. - */ -static void clear_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long i, saddr = addr; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - __ptep_get_and_clear(mm, addr, ptep); - - flush_tlb_range(&vma, saddr, addr); -} - -void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned long sz) -{ - size_t pgsize; - int i; - int ncontig; - unsigned long pfn, dpfn; - pgprot_t hugeprot; - - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); - - if (!pte_present(pte)) { - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) - __set_ptes(mm, addr, ptep, pte, 1); - return; - } - - if (!pte_cont(pte)) { - __set_ptes(mm, addr, ptep, pte, 1); - return; - } - - pfn = pte_pfn(pte); - dpfn = pgsize >> PAGE_SHIFT; - hugeprot = pte_pgprot(pte); - - clear_flush(mm, addr, ptep, pgsize, ncontig); - - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz) { diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 66061002ff36..5d1d3a6c7c44 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -535,29 +535,39 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, size_t *pgsize) { + unsigned long hugepage_shift; pte_t __pte; /* We must read the raw value of the pte to get the size of the mapping */ __pte = READ_ONCE(*ptep); - if (pgsize) { - if (size >= PGDIR_SIZE) + if (size >= PGDIR_SIZE) { + if (pgsize) *pgsize = PGDIR_SIZE; - else if (size >= P4D_SIZE) + hugepage_shift = PGDIR_SHIFT; + } else if (size >= P4D_SIZE) { + if (pgsize) *pgsize = P4D_SIZE; - else if (size >= PUD_SIZE) + hugepage_shift = P4D_SHIFT; + } else if (size >= PUD_SIZE) { + if (pgsize) *pgsize = PUD_SIZE; - else if (size >= PMD_SIZE) + hugepage_shift = PUD_SHIFT; + } else if (size >= PMD_SIZE) { + if (pgsize) *pgsize = PMD_SIZE; - else + hugepage_shift = PMD_SHIFT; + } else { + if (pgsize) *pgsize = PAGE_SIZE; + hugepage_shift = PAGE_SHIFT; } /* Make sure __pte is not a swap entry */ if (pte_valid_napot(__pte)) return napot_pte_num(napot_cont_order(__pte)); - return 1; + return size >> hugepage_shift; } #endif @@ -586,8 +596,8 @@ static inline pte_t __ptep_get(pte_t *ptep) return pte; } -static inline void set_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pteval, unsigned int nr) +static inline void __set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pteval, unsigned int nr) { #ifdef CONFIG_RISCV_ISA_SVNAPOT if (unlikely(pte_valid_napot(pteval))) { @@ -631,7 +641,8 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_val(pteval) += 1 << _PAGE_PFN_SHIFT; } } -#define set_ptes set_ptes +#define set_contptes(mm, addr, ptep, pte, nr, pgsize) \ + __set_ptes(mm, addr, ptep, pte, nr) static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -646,9 +657,8 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addre extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { pte_t pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); @@ -687,6 +697,9 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, } #define ptep_get __ptep_get +#define set_ptes __set_ptes +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +#define ptep_get_and_clear __ptep_get_and_clear #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index be129f4f1503..d8f07aef758b 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -173,68 +173,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -static void clear_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long i, saddr = addr; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - ptep_get_and_clear(mm, addr, ptep); - - flush_tlb_range(&vma, saddr, addr); -} - -/* - * When dealing with NAPOT mappings, the privileged specification indicates that - * "if an update needs to be made, the OS generally should first mark all of the - * PTEs invalid, then issue SFENCE.VMA instruction(s) covering all 4 KiB regions - * within the range, [...] then update the PTE(s), as described in Section - * 4.2.1.". That's the equivalent of the Break-Before-Make approach used by - * arm64. - */ -void set_huge_pte_at(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - pte_t pte, - unsigned long sz) -{ - unsigned long hugepage_shift, pgsize; - int i, pte_num; - - if (sz >= PGDIR_SIZE) - hugepage_shift = PGDIR_SHIFT; - else if (sz >= P4D_SIZE) - hugepage_shift = P4D_SHIFT; - else if (sz >= PUD_SIZE) - hugepage_shift = PUD_SHIFT; - else if (sz >= PMD_SIZE) - hugepage_shift = PMD_SHIFT; - else - hugepage_shift = PAGE_SHIFT; - - pte_num = sz >> hugepage_shift; - pgsize = 1 << hugepage_shift; - - if (!pte_present(pte)) { - for (i = 0; i < pte_num; i++, ptep++, addr += pgsize) - set_ptes(mm, addr, ptep, pte, 1); - return; - } - - if (!pte_napot(pte)) { - set_ptes(mm, addr, ptep, pte, 1); - return; - } - - clear_flush(mm, addr, ptep, pgsize, pte_num); - - set_ptes(mm, addr, ptep, pte, pte_num); -} - int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, diff --git a/mm/contpte.c b/mm/contpte.c index e8574051d0b9..2320ee23478a 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -10,6 +10,8 @@ /* * Any arch that wants to use that needs to define: * - __ptep_get() + * - __set_ptes() + * - __ptep_get_and_clear() * - pte_cont() * - arch_contpte_get_num_contig() */ @@ -17,6 +19,7 @@ /* * This file implements the following contpte aware API: * - huge_ptep_get() + * - set_huge_pte_at() */ pte_t huge_ptep_get(pte_t *ptep) @@ -43,3 +46,59 @@ pte_t huge_ptep_get(pte_t *ptep) } return orig_pte; } + +/* + * ARM64: Changing some bits of contiguous entries requires us to follow a + * Break-Before-Make approach, breaking the whole contiguous set + * before we can change any entries. See ARM DDI 0487A.k_iss10775, + * "Misprogramming of the Contiguous bit", page D4-1762. + * + * RISCV: When dealing with NAPOT mappings, the privileged specification + * indicates that "if an update needs to be made, the OS generally should first + * mark all of the PTEs invalid, then issue SFENCE.VMA instruction(s) covering + * all 4 KiB regions within the range, [...] then update the PTE(s), as + * described in Section 4.2.1.". That's the equivalent of the Break-Before-Make + * approach used by arm64. + * + * This helper performs the break step for use cases where the + * original pte is not needed. + */ +static void clear_flush(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long i, saddr = addr; + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) + __ptep_get_and_clear(mm, addr, ptep); + + flush_tlb_range(&vma, saddr, addr); +} + +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned long sz) +{ + size_t pgsize; + int i; + int ncontig; + + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + + if (!pte_present(pte)) { + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) + __set_ptes(mm, addr, ptep, pte, 1); + return; + } + + if (!pte_cont(pte)) { + __set_ptes(mm, addr, ptep, pte, 1); + return; + } + + clear_flush(mm, addr, ptep, pgsize, ncontig); + + set_contptes(mm, addr, ptep, pte, ncontig, pgsize); +} From patchwork Wed May 8 11:18:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13658580 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 674A8C04FFE for ; Wed, 8 May 2024 11:23:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EBBEE6B0140; Wed, 8 May 2024 07:23:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E441C6B0141; Wed, 8 May 2024 07:23:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE5666B0142; Wed, 8 May 2024 07:23:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id ACB906B0140 for ; Wed, 8 May 2024 07:23:46 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1420440EA9 for ; Wed, 8 May 2024 11:23:46 +0000 (UTC) X-FDA: 82094993652.20.DE034A8 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by imf08.hostedemail.com (Postfix) with ESMTP id 26AC316000C for ; Wed, 8 May 2024 11:23:43 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=SnCZ+Y1s; spf=pass (imf08.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715167424; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hkOJ479CewvDdU5HWPEgLgm2sAjKEVUPHaCHVUUKnDI=; b=qPQsTIyCpFEffcmv7lBIP8ha6J0bOU6bMgKGsYnRZDRNp8JMFCv8Xc5fD7f4+c3mjBKQ3H M2bA0G0JCemMH5/QiJc3VL15RXZ6Gf2CDlV675vwRoRYJazhdCgQhSZH15FXk/CX4x58rc AOda0P8AXruSVQZcymFyPsMrEK1GAsA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=SnCZ+Y1s; spf=pass (imf08.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715167424; a=rsa-sha256; cv=none; b=WXNTyglUCtEmFR7ECzg10Mx/4gBPnHUcVD9fFetmEqEY9OHbDWbAvLKH926eM2/sZWIGF2 0Q/mHGANOM/RfybHJvJyzBXitz+Kx3NE26wa0sSmVgCv+PWj7FB6wHKwWoq7Z0sIZOuZV3 XlJyfwyR5nKxwvqmIY/U0YrhWIuoxb4= Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-41f9ce16ed8so2632005e9.0 for ; Wed, 08 May 2024 04:23:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715167423; x=1715772223; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hkOJ479CewvDdU5HWPEgLgm2sAjKEVUPHaCHVUUKnDI=; b=SnCZ+Y1sHY93pqQUVz6WpqFECUyzUn/YvejmFCqa3oQ8ssW6SXa2Ewk959K8xAdYOI 573grAYBftJypB3MwGeIE+a8yuSmdnQk9U+cBbotpQX2GzBCyGS/B/dxPqieRLVQiME8 cgb4U0qwM3r1Td2BWTuU2muQ55dq9Yt43jWOLitEqcli8mIaCJBXO3yz7DGkrdz/tson 6DTPXCujOwiWO+MEtZFdmaND3otTrA3hHGiNDOW5nTaIoA0TpVGzE8niMwRDS1QtCbhK IHKkg2/hA7rjdsIoiLN0Dkswq+/B4rqU1AIVdh9WoGYQObP+AUUoqK/NTTVXeX9kZcLs bZ5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715167423; x=1715772223; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hkOJ479CewvDdU5HWPEgLgm2sAjKEVUPHaCHVUUKnDI=; b=f/ewKT5PqiBmdMyZeIoltZeuClJVjtyZ5/9cFC9d4PudldCYHehRZCDhVkx8SzqwYK HRijTC2FPNP8LWfaHrUkxxoC4ryTaRR2j2SSfkrV8TwWC7ggOm12WTGMp5ln8ZZRb6vM JFqre7D8lm2f2cfhXuDQP0DQAfQYtH1ffD2IdOwotqHsrOCtbBZ7vF/2IlI7EGcEMBJU NXxBLRHWZgCfLsIBbHjMwmdBC93gcCb5+Iz3CQWJzxftIuK5R7L4zuYx98dpmzAhG6p3 VSNJKdof/2NIU3zV54FxOQDXBiji/DHO/5xYAXMBeCdfMz6zPYFO+2w9tMnvssgUwB9Q WTaQ== X-Forwarded-Encrypted: i=1; AJvYcCX8CCaBXu+yijWldqOoCIBIsGK4vIf3gj5wQrqbb+2Tc68vjUqDmIhXaPk+eicTblzmxEW+i0WnDVS9X6i/KtHi+Zs= X-Gm-Message-State: AOJu0YxhzGQp9E6GXLO1HHsbh3yL0iQSVcm4DNWJCjG37Yxzx8nE/Msr WXwIuQJ1rfNb/cWEvjzCHDk/IQFLdMeGnXWHPixU2jfXLITGqOE5WWt9f58Txfo= X-Google-Smtp-Source: AGHT+IFALljM8hm4oiWBT04Jw0JLiRkkyPDpkUx9yCVH0D1QlIeoTfNU3Xx59AzAW7ujf7brtTm4rA== X-Received: by 2002:a05:600c:3d93:b0:41a:adc3:f777 with SMTP id 5b1f17b1804b1-41f7130a17bmr26482125e9.16.1715167422759; Wed, 08 May 2024 04:23:42 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id o14-20020a05600c4fce00b0041bf685921dsm1992749wmq.0.2024.05.08.04.23.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 04:23:41 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v2 5/9] mm: Use common huge_pte_clear() function for riscv/arm64 Date: Wed, 8 May 2024 13:18:25 +0200 Message-Id: <20240508111829.16891-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508111829.16891-1-alexghiti@rivosinc.com> References: <20240508111829.16891-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Stat-Signature: aq3irhpw5xsf3tie6rm1jmniqb5dhqo4 X-Rspam-User: X-Rspamd-Queue-Id: 26AC316000C X-Rspamd-Server: rspam05 X-HE-Tag: 1715167423-352914 X-HE-Meta: U2FsdGVkX1/X88puKrSGxpFDaGaVqydh59TGycqlWvGM7GKex9pDhcFShxKhsIo13wT+2PXeLmULUnv6KTprqXaqBMwAXClQVK9ixWMWigeF9KjjxzrAGhAP2ACAFK8wyTdI/yWdO/LyLhEcsh+M+fKLsZm1JyoOr/SicHTK2SK1ITgRESHAiJGcvDwnPPEM6p92h4JCnbmrwJE5pLKDNMlBgrZHNF4lkdB9BIsHZQJljgZDTo28+Z2UBkZShBvwr12E/lGUSjIwOzHFMvpyuoRIkxpTYzr8lJwFmptd10mvHNcN+tCyLn50EPp1W4dn9FxOpKpeD2IMNUr7JZNYDTBqGEZr0SBVebMvMl2IYeI14sUxt4MEQyVKqGX1AU/v3xE11PBwUGsAwlAwFF4O+wC8hz0IwNKhKN2f3a8v5HYvoUco3capgQNGC972ZlyHFZT3Mdu9H++V7jInBcd9c9OoFtgTuikqZmpUZotdn1t1AoxSBGozv2UKhVun1Qm+NysKBIJkqSx3EIAeqJpSKJLuckFTnd8I9ok9wN9ScYj+k5h1B9XK3X2VVWysCVMA4Pge+4D/5ubmFXcxX/4g5JXrPF6xn7fkZt1GnLUUfj090TEPTxMu3PwJfsQd2U7R05PjrZ2awKYp3d2LDaxUg2liKU3aswrPIafiRsUKF71p76gyCu2Z3UIjuOAVF4wf+Iiga7JAqUHrb1GPerKfXoe44UTMN4JGrbqjUug/lgpljuafb8jo2u9BVVtXYyTz4CZh0vrU6BOkqnAkIRGONYWIhMnm0Zpg2Vr09zKBuXX6zYvlqJuEk3N+WhV6Pk/YmTyAqp2LYMUQeUJnbdWgYHPxptCn70SrjiMCqJWGibUlT1z+s52+2NPB3n8u99ShPB9P91laIH958x8V4WKuaJ1y04JH+fvZ/h1pACIEkhL1X9uFIgZO/UL+U1Az6UdT2Vek5fH98XYBRPRo9rK O6Wc4ERa kp0owqFyRYgKZ11/HaQJMWj1Za4i1F+HZ2UgY96dx/Rn76fDkHDOh/CPP+P0p7l11F49C/OeeicudS2ZGcai+RPw2z3iOnZNQ8jBs0GqHApmcNzXBucFyG5Rqo1ZcwrjZLhtlJhJjMSJm4mgP7WAWcwkZSrTjJJ1qijJfGsxtz173KJoxCo3a6J5WoYynS08CNPqW6iVBNHQNiveWKunWuciBGzcEeQPO2V2r7SECctrYUv44yesi/WcXkaaczYGk1QXHwG1gmGdSk/fsGysshxySlNsvR4GkgPX2QZ4kv5er3qwAFxNkACfRU3NVRh+6umdVfAo5GTxg3aOMpKD5j4foX/uEr/kY4eJFLqo6c/qkUr9mogSWgrkHzRhqqUU2PXk7zAhpfEdQIUMwxI+t5IDQEtCr+LPP7LgGqxX1K+hY2LF7++PL60wG0j5fSqGhwVfUqOPJnVOPJ34hG2TY5v8Wv/GpCZLO1zj3XqMQFjKEZbWedQBBiU+ekg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Both architectures have the same implementation so move it to generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/mm/hugetlbpage.c | 12 ------------ arch/riscv/include/asm/pgtable.h | 5 +++-- arch/riscv/mm/hugetlbpage.c | 19 ------------------- mm/contpte.c | 14 ++++++++++++++ 4 files changed, 17 insertions(+), 33 deletions(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index b8353b0a273c..cf44837369be 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -277,18 +277,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -void huge_pte_clear(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned long sz) -{ - int i, ncontig; - size_t pgsize; - - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - __pte_clear(mm, addr, ptep); -} - pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 5d1d3a6c7c44..0847a7fb8661 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -644,8 +644,8 @@ static inline void __set_ptes(struct mm_struct *mm, unsigned long addr, #define set_contptes(mm, addr, ptep, pte, nr, pgsize) \ __set_ptes(mm, addr, ptep, pte, nr) -static inline void pte_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) +static inline void __pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) { __set_pte_at(mm, ptep, __pte(0)); } @@ -700,6 +700,7 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, #define set_ptes __set_ptes #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear +#define pte_clear __pte_clear #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index d8f07aef758b..437b1df059eb 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -254,25 +254,6 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } -void huge_pte_clear(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long sz) -{ - size_t pgsize; - pte_t pte = ptep_get(ptep); - int i, pte_num; - - if (!pte_napot(pte)) { - pte_clear(mm, addr, ptep); - return; - } - - pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); - for (i = 0; i < pte_num; i++, addr += pgsize, ptep++) - pte_clear(mm, addr, ptep); -} - static bool is_napot_size(unsigned long size) { unsigned long order; diff --git a/mm/contpte.c b/mm/contpte.c index 2320ee23478a..22e0de197bd3 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -12,6 +12,7 @@ * - __ptep_get() * - __set_ptes() * - __ptep_get_and_clear() + * - __pte_clear() * - pte_cont() * - arch_contpte_get_num_contig() */ @@ -20,6 +21,7 @@ * This file implements the following contpte aware API: * - huge_ptep_get() * - set_huge_pte_at() + * - huge_pte_clear() */ pte_t huge_ptep_get(pte_t *ptep) @@ -102,3 +104,15 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, set_contptes(mm, addr, ptep, pte, ncontig, pgsize); } + +void huge_pte_clear(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned long sz) +{ + int i, ncontig; + size_t pgsize; + + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) + __pte_clear(mm, addr, ptep); +} From patchwork Wed May 8 11:18:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13658581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC68FC04FFE for ; Wed, 8 May 2024 11:24:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 483826B009B; Wed, 8 May 2024 07:24:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 40C886B0141; Wed, 8 May 2024 07:24:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 289896B0143; Wed, 8 May 2024 07:24:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 08C896B009B for ; Wed, 8 May 2024 07:24:48 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 89D80120AAB for ; Wed, 8 May 2024 11:24:47 +0000 (UTC) X-FDA: 82094996214.05.5ABB080 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by imf06.hostedemail.com (Postfix) with ESMTP id 91285180008 for ; Wed, 8 May 2024 11:24:45 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=gfaCVvmC; spf=pass (imf06.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.42 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715167485; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k7TbOelFCBPjD//aDEZ907vn9YCYdQxapVfhTuFqwGo=; b=JIf1ZPwLfUKBlJoXF5mVnchatmdv1FH15B/pK6WxQHh4+ea3DgWuuRuhG1ZIXi+WzuYZu+ MKO5op0Clvs/u4bOvKhO5FQueGSAGWcPcta2Zi0pQorgtvUR1jLmfA1GSMz6a2F+kkqSy8 nHyXtshOv3Y6BMzwRdPvjhKR99RAJYs= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=gfaCVvmC; spf=pass (imf06.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.42 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715167485; a=rsa-sha256; cv=none; b=eDYbeTQiEHNVfTA4hKlPvVylavJoSfbEn9Be4hgvJPvn8GQ4LeNy+tg8e+7RAyN17t2E+c QKGu7qpF7j1L6JVkGeqdg69/KjRulOhfTqveT8OQyiaSErHTcHSj6yMHPifLQ/uu9uRIys UHUjV7yMAgFzzycc1cf/KLB1ihMVI1U= Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-41dc9f98e8dso3941755e9.1 for ; Wed, 08 May 2024 04:24:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715167484; x=1715772284; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=k7TbOelFCBPjD//aDEZ907vn9YCYdQxapVfhTuFqwGo=; b=gfaCVvmCeI+Hxo1OnMsnknIgwvBY/AwEguKZHoeYYrIUiX0QnxbknLFSctJEtJkJ7b f7jMc9m+ORhZpmK8o3hY89HbqmNOotfGKLpcSkzATjf0Qtc0zh5XJg5FyCBl8Nn8p5zC Ptd6mj8UpRA6PmLHZ3aobfjS9XwZbb1k+8bXeGARiPcHlyezmYSyfYPOPPuOuwQ7PqKt xjyZ+X0SXriinZuFV2BghMIDt9BiceZRGqliDTmHPLn2D2SmwfVse4ch+hgm3Evo29E+ SJqEu0LI0vP2TbmlyZUoa/wtjYhPYTqINnr+OaxM8yLccv1NOcjo6Oe1GGQQ51H6nn2q mtbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715167484; x=1715772284; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k7TbOelFCBPjD//aDEZ907vn9YCYdQxapVfhTuFqwGo=; b=gb09FbPgKBTvaSYQt7Lso5TFZyqXhNd/REjv6Kj6vd+aY6s9LeZLwTmHBHvwuMwFUf UaoEyFiFYGKMPRZRL8iJ+zxgJKpqDJxBWt5/6R1AVOlpre9LN8IQ0zRHZ/RlPPoW8b3x zyto0cXaFNFd+z4zPg4RM+SJCfvhaRQsHOUEp0KVfVTJG7OUssqPEXP8soRG1EhC9+vn pRgHGTNi7gJOpQ4Q/JcQB+FKC9Mnk/A7HFFJ+1oTUquYCljek7lXWvHg0sMpam/L42DM M10tbfsWqD4WRrDlDLcB+LUTuYNCvW9PG9jCXHsWcVc419eFojOmQ5g7mAy0++PR01Ep hIiw== X-Forwarded-Encrypted: i=1; AJvYcCVcnZ5IHLdW1rYAw6UHtf5OcaFpBSI8I4v9GBBrnqR1vIjTlRqpWnMcemPMF9p9w/zeFhWq8cisn9VCniaixq3Blao= X-Gm-Message-State: AOJu0YyZIaC5srGI02HR/3DlzpXLRtitOWt/CJ+fv//ykKeYROIsL6Dq JBW/yy/YhEYYcpVY7PqcXb4I6SXYBmtEh5vaBX+a9v2f4eb8FQJJwfMypYgUnqI= X-Google-Smtp-Source: AGHT+IGIT5fRmSU5mTHAtfUSvBO8bSYE3S8hCevxI+ZKnPLL6EPNP04SFEUKamUPrL/Drlv8BwdKKg== X-Received: by 2002:a5d:6d52:0:b0:34d:b3a0:5614 with SMTP id ffacd0b85a97d-34fca4326b6mr2190198f8f.8.1715167483962; Wed, 08 May 2024 04:24:43 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id dn15-20020a0560000c0f00b0034d9012e74bsm15257565wrb.72.2024.05.08.04.24.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 04:24:43 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v2 6/9] mm: Use common huge_ptep_get_and_clear() function for riscv/arm64 Date: Wed, 8 May 2024 13:18:26 +0200 Message-Id: <20240508111829.16891-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508111829.16891-1-alexghiti@rivosinc.com> References: <20240508111829.16891-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 91285180008 X-Rspamd-Server: rspam06 X-Stat-Signature: bqspgwhf5ae5xndrdsc8e4k3siqz3uss X-HE-Tag: 1715167485-767694 X-HE-Meta: U2FsdGVkX19SuxEHRLmpopSJOmDuxVA7QFJvnXbAo4eD9oS2cjClFc64ojXFQ5s9018MqAoVKxBxRjTWPeJGWJaLDpYoj7UEMi+C0p/rsNV/35h+2Q3lm2Nw3noYX25QxOf8qvZ1/YEtyOepD97mf5edg6zsylhQGTBQdFxoaALanxv2WzczoOCiUNzQTi4SA9461o2TIEBKAOCUuVClRdU2KI3XPbqLDIGk9GEoWr0rGDe+lnEpep/36HLUKE+VwcXQ2H01MEvSELKCUKcY+zFJ1cf9Pcg2mk5ZtP9JsCTTYYohGDP0ztcm+Qxdd0z8hK3eUf2cC9f5WWFZ3/Co0FBqYQtieQxyBcYbvXnc3YdFopfPw0BF85QV0qeaQqQzmWtCW6n0bHUNdQ7e8eIfJWL2DXu/X1d4vN4OSrg7WkfCO7YWNVEO3IPhL1aM14g6WxfxDMTgnm32mQB2tp0xoJ6gI6YVlWr9yT+DJUhurXaGDSAroOVbBzdpNKy1HcR2LD+/OYGaenLZc/79Kcc1NjNMlw7uadXhGVJH1zIFPc3ALDBF+8rK5VJBEF6akhX3tfprERLV6pEWCT1I7Lisn4z4v4sK9FcRNDg0CvTpeHBSlC+P3eAWd+kUWLr0odeZmn54rCcuRfdrQPhynpS9jMjAJxGXWq9cCO4LeSiuvnrOFn+wtwBqpWVpjQ6w008lUNZalkGx1INE1RQ/wxU/qSDJMoXWgmc1SUK5KNcLxuueaZ03xEHWqM8uq5xuPS6+QuCQKblCnejoqIjiS7Pb1MFs+GGwPHAVada/LtzNVMuzaeZSyqtUT27l2IshX6W/BSXBrJckKJm45cHms4XGl1EkI+wDqK0FIwm+WY0a5UFNxLC93ILGmGpqhDhcSkzKR6sdgVZpjHDjXr0DtivQGErkopypz0ktP9CCo/WyTAC9M9s8yyq4n9gvmBzHdo7DbOT7ArSGlaIOg3RcGjU tZPxyQ8f l3jzg4yxj1atAPC6JKfgehysqMIMcZYrhj0iRirjpEABDGLEpgdHVb9IDsxEfQxqDftAviCf7DEAL8vEAAURjzILIQlxcKHrPbV67Y8qP0jAUKiumgwjhugSH+4HEBmAvXgEL+pD8wfZr5UHX5D4iPw024sITOH+IqOCMeURDRtecT+jsT6xQ6E3drToeZ2/2G416xnDQDH3X2DX7dXzox4EFZ2loUvCQSsKFimE19y3TX8LHl6dRG4eEchu4Ax/4B6YEzE8Q59HAwlaw1O8FTMedjJS8f0CUl4Pki4oApfV49fHZ4inB1j1uCii95Hpcs8E0HdoQeKdCGBXZn/OQJy3nqzFyjQICXEVP3MqDxgFMiPBxUbtyUhKGZU1YOGQlD+2nNBWK2EyUTDDje4gaxPhpqNuDR17SeDDep4OV++xit+7RC9JvR7sP+vQmrf+zJsOnBj+LjbcNNVDx2hybahANlOBl7lb17NrFQupReC0tXiwAW8RMidHPFQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After some adjustments, both architectures have the same implementation so move it to the generic code. Note that get_clear_contig() function is duplicated in the generic and the arm64 code because it is still used by some arm64 functions that will, in the next commits, be moved to the generic code. Once all have been moved, the arm64 version will be removed. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 14 +++++++++- arch/arm64/mm/hugetlbpage.c | 19 ++----------- arch/riscv/include/asm/pgtable.h | 4 ++- arch/riscv/mm/hugetlbpage.c | 21 ++------------ mm/contpte.c | 48 ++++++++++++++++++++++++++++++-- 5 files changed, 66 insertions(+), 40 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 2e0415fd5083..7c2938cb70b9 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1644,11 +1644,23 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #endif /* CONFIG_ARM64_CONTPTE */ -static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, +int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize); + +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { int contig_ptes = 0; + /* + * If the size is not passed, we need to go through the page table to + * find out the number of contiguous ptes. + */ + if (size == 0) + return find_num_contig(mm, addr, ptep, pgsize); + *pgsize = size; switch (size) { diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index cf44837369be..5ace4bf7ce35 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -93,8 +93,8 @@ int pud_huge(pud_t pud) #endif } -static int find_num_contig(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, size_t *pgsize) +int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize) { pgd_t *pgdp = pgd_offset(mm, addr); p4d_t *p4dp; @@ -277,21 +277,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - int ncontig; - size_t pgsize; - pte_t orig_pte = __ptep_get(ptep); - - if (!pte_cont(orig_pte)) - return __ptep_get_and_clear(mm, addr, ptep); - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - - return get_clear_contig(mm, addr, ptep, pgsize, ncontig); -} - /* * huge_ptep_set_access_flags will update access flags (dirty, accesssed) * and write permission. diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 0847a7fb8661..d976113a370d 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -532,7 +532,9 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) #define PFN_PTE_SHIFT _PAGE_PFN_SHIFT #ifdef CONFIG_RISCV_ISA_SVNAPOT -static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { unsigned long hugepage_shift; diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 437b1df059eb..a757e0b2f090 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -187,7 +187,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); @@ -202,21 +202,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return true; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep) -{ - pte_t orig_pte = ptep_get(ptep); - int pte_num; - - if (!pte_napot(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); - - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); - - return get_clear_contig(mm, addr, ptep, pte_num); -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -231,7 +216,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, return; } - pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + pte_num = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); @@ -249,7 +234,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_clear_flush(vma, addr, ptep); - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); + pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, NULL); return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } diff --git a/mm/contpte.c b/mm/contpte.c index 22e0de197bd3..68eb1634b922 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -22,6 +22,7 @@ * - huge_ptep_get() * - set_huge_pte_at() * - huge_pte_clear() + * - huge_ptep_get_and_clear() */ pte_t huge_ptep_get(pte_t *ptep) @@ -33,7 +34,7 @@ pte_t huge_ptep_get(pte_t *ptep) if (!pte_present(orig_pte) || !pte_cont(orig_pte)) return orig_pte; - ncontig = arch_contpte_get_num_contig(ptep, + ncontig = arch_contpte_get_num_contig(NULL, 0, ptep, page_size(pte_page(orig_pte)), &pgsize); @@ -87,7 +88,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, int i; int ncontig; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) @@ -111,8 +112,49 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, int i, ncontig; size_t pgsize; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) __pte_clear(mm, addr, ptep); } + +static pte_t get_clear_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + pte_t orig_pte = __ptep_get(ptep); + unsigned long i; + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { + pte_t pte = __ptep_get_and_clear(mm, addr, ptep); + + /* + * If HW_AFDBM (arm64) or svadu (riscv) is enabled, then the HW + * could turn on the dirty or accessed bit for any page in the + * set, so check them all. + */ + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + return orig_pte; +} + +pte_t huge_ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + int ncontig; + size_t pgsize; + pte_t orig_pte = __ptep_get(ptep); + + if (!pte_cont(orig_pte)) + return __ptep_get_and_clear(mm, addr, ptep); + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + return get_clear_contig(mm, addr, ptep, pgsize, ncontig); +}