From patchwork Mon Feb 17 14:07:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13977910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91941C021A9 for ; Mon, 17 Feb 2025 14:08:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0FAC628005D; Mon, 17 Feb 2025 09:08:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AAF7280059; Mon, 17 Feb 2025 09:08:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB3DB28005D; Mon, 17 Feb 2025 09:08:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CE5B3280059 for ; Mon, 17 Feb 2025 09:08:35 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7FAFB80496 for ; Mon, 17 Feb 2025 14:08:35 +0000 (UTC) X-FDA: 83129616990.24.73859E5 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id A545B1C001F for ; Mon, 17 Feb 2025 14:08:33 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739801313; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1cElz/iHfGJIJrYOZM+nGWb3qr/VxQv3tlR6HOGX+84=; b=bO4e1AwJBItfHja5KmlGyndQHSGN7TwaJVNt6vuV0Q5G68b96xD1KNGXgcqXddxmWOJrYO idPki5q6RDfsTZ1mjFpFaoUHc3P4QW2cxswxaSaQm6/zfMjdcEgqc1eCVqeORqg6EIatMa gbirHSIvTAK7zy8/OO1bP7qJp2lZPRw= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739801313; a=rsa-sha256; cv=none; b=UJ2ajb9+QB4L29cIoPXm0mZKCD+HfziEby1jBBbMQI+l4f/FYVc9K+ZSJOw8ufag8HBk+s B32NzKJOQ95cjO6E+LdaOgwkDSRu4adRDyYAxrWdjx3XQb5x9ZWW5mcQLemHy0NjMBdKRP Kp7bsEwC7Ci9aa+EtR9YBgvhAvdY7jw= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 45EC71A2D; Mon, 17 Feb 2025 06:08:52 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CC4143F6A8; Mon, 17 Feb 2025 06:08:30 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , David Hildenbrand , "Matthew Wilcox (Oracle)" , Mark Rutland , Anshuman Khandual , Alexandre Ghiti , Kevin Brodsky Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 05/14] arm64: hugetlb: Use set_ptes_anysz() and ptep_get_and_clear_anysz() Date: Mon, 17 Feb 2025 14:07:57 +0000 Message-ID: <20250217140809.1702789-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250217140809.1702789-1-ryan.roberts@arm.com> References: <20250217140809.1702789-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: A545B1C001F X-Stat-Signature: dx5cxack8km8sawka6kswp3jkzcokxj3 X-Rspamd-Server: rspam03 X-HE-Tag: 1739801313-229571 X-HE-Meta: U2FsdGVkX19YCxG7073V9nf7GhxzqIbYLCnCXzZw6ZAk0l0laiyYHsYXbxJWimHh0lVshc1ONMgLdtV/AemCRGiFIL/9cHNCKfDJQTHpm+OByLDFAw/dVU3AD31//9rIf9GUcoNqCOx3tVNVjiY1R7wKPOornQRMKOnUGiMKGHUalyUErvwX3IHQ/q3GUu5KpFAJBBXrD9kzHf0wx4YeJSBAtIj+1s4B8X5VwE/17q1pEpCA30aFOIrdhe5CUwPRZegHhbg5Hgka8ZW+OF+qSx6dJrPJOFT6K9lDD0Rx2l+b8ayG/5X+AHI2QUCJzD//JOI07xCERQ9WF78oT+FCErPHJ7KKyj/wu2Y/0vFprurj+8Ar4iCAuMgUvBiG1q0OlvvPwsUNqSGGpOdYbmO9dXxUmJct/Z9lvWxC9LLfKNPpsXkXOvRzU2WKfMHPX9Zt1l1l7GLee2xAycdOXBFZnlMsnpHrYbUQR9iANN/scuuqKzYZT8HGXtN3N9w2LQeUDektFO++x7rNxqgd9dcsgkJ8SOIghshPgwRZdUdYWT2p2RDtTOIXjWMFVbQa4ITmu8Xqr+gqNvPeLX/gamO8R5N/VvW0BKckw59kBMju9FzzBlBwro85X1IB47jp5ICBPDU8E4SlVAXlM1YbUlHP0oAetICOp2k+MrbrDZjcdYmAOobmOkVVPsoAG5SAErGERiZ8nk22WXXRXU/sDPoDSSrNQxbfgw1d/nGkDPaia6GNl7KDjT1cZ4SzJl7tLSiW1shSwiPPSD+I8MK4wlSpSQWYao+hXxyqL6wZBIDZV+i7UaB7V6eW6oujqZs3eQih6pAPuOuqsGxhYzxewxiS6gDr1qvplbM4AUCOJ95CwzuXs14BqIctQWRP+985TqIMRuCFUES6reR/LNKlo4avafgwQI8jn9XS6W02coDH0ZqmI3DsWeWnNmOrceMTZr98rc7dD96EgbrFcw7F7tg 4u0D7VFb 0DmrrnE8vNOuzZyAE+iMCot6KbBQgmtsv0m1mATYKO7umqW6588oPlCo2SdlX0DWfi1gW+lvQuiF/SVCc0vJ+uw82elaDhDZBq5tj00GABLeKqOZrS1vRKaUhaUPtfKklzSDxeTYM9bU3jwhC8Dj/PnF5P5dVXdPWXDn9TRLmbBubKEJUcRCw1poz/mZKqpO+ASLbmY1Azmwc8Dw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refactor the huge_pte helpers to use the new common set_ptes_anysz() and ptep_get_and_clear_anysz() APIs. This provides 2 benefits; First, when page_table_check=on, hugetlb is now properly/fully checked. Previously only the first page of a hugetlb folio was checked. Second, instead of having to call __set_ptes(nr=1) for each pte in a loop, the whole contiguous batch can now be set in one go, which enables some efficiencies and cleans up the code. One detail to note is that huge_ptep_clear_flush() was previously calling ptep_clear_flush() for a non-contiguous pte (i.e. a pud or pmd block mapping). This has a couple of disadvantages; first ptep_clear_flush() calls ptep_get_and_clear() which transparently handles contpte. Given we only call for non-contiguous ptes, it would be safe, but a waste of effort. It's preferable to go straight to the layer below. However, more problematic is that ptep_get_and_clear() is for PAGE_SIZE entries so it calls page_table_check_pte_clear() and would not clear the whole hugetlb folio. So let's stop special-casing the non-cont case and just rely on get_clear_contig_flush() to do the right thing for non-cont entries. Signed-off-by: Ryan Roberts --- arch/arm64/mm/hugetlbpage.c | 52 +++++++------------------------------ 1 file changed, 10 insertions(+), 42 deletions(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index b7434ed1b93b..8ac86cd180b3 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -166,12 +166,12 @@ static pte_t get_clear_contig(struct mm_struct *mm, pte_t pte, tmp_pte; bool present; - pte = __ptep_get_and_clear(mm, addr, ptep); + pte = ptep_get_and_clear_anysz(mm, ptep, pgsize); present = pte_present(pte); while (--ncontig) { ptep++; addr += pgsize; - tmp_pte = __ptep_get_and_clear(mm, addr, ptep); + tmp_pte = ptep_get_and_clear_anysz(mm, ptep, pgsize); if (present) { if (pte_dirty(tmp_pte)) pte = pte_mkdirty(pte); @@ -215,7 +215,7 @@ static void clear_flush(struct mm_struct *mm, unsigned long i, saddr = addr; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - __ptep_get_and_clear(mm, addr, ptep); + ptep_get_and_clear_anysz(mm, ptep, pgsize); __flush_hugetlb_tlb_range(&vma, saddr, addr, pgsize, true); } @@ -226,32 +226,20 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, size_t pgsize; int i; int ncontig; - unsigned long pfn, dpfn; - pgprot_t hugeprot; ncontig = num_contig_ptes(sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) - __set_ptes(mm, addr, ptep, pte, 1); + set_ptes_anysz(mm, ptep, pte, 1, pgsize); return; } - if (!pte_cont(pte)) { - __set_ptes(mm, addr, ptep, pte, 1); - return; - } - - pfn = pte_pfn(pte); - dpfn = pgsize >> PAGE_SHIFT; - hugeprot = pte_pgprot(pte); - /* Only need to "break" if transitioning valid -> valid. */ - if (pte_valid(__ptep_get(ptep))) + if (pte_cont(pte) && pte_valid(__ptep_get(ptep))) clear_flush(mm, addr, ptep, pgsize, ncontig); - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); + set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); } pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, @@ -441,11 +429,9 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty) { - int ncontig, i; + int ncontig; size_t pgsize = 0; - unsigned long pfn = pte_pfn(pte), dpfn; struct mm_struct *mm = vma->vm_mm; - pgprot_t hugeprot; pte_t orig_pte; VM_WARN_ON(!pte_present(pte)); @@ -454,7 +440,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); ncontig = num_contig_ptes(huge_page_size(hstate_vma(vma)), &pgsize); - dpfn = pgsize >> PAGE_SHIFT; if (!__cont_access_flags_changed(ptep, pte, ncontig)) return 0; @@ -469,19 +454,14 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, if (pte_young(orig_pte)) pte = pte_mkyoung(pte); - hugeprot = pte_pgprot(pte); - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); - + set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); return 1; } void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - unsigned long pfn, dpfn; - pgprot_t hugeprot; - int ncontig, i; + int ncontig; size_t pgsize; pte_t pte; @@ -494,16 +474,11 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, } ncontig = find_num_contig(mm, addr, ptep, &pgsize); - dpfn = pgsize >> PAGE_SHIFT; pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); pte = pte_wrprotect(pte); - hugeprot = pte_pgprot(pte); - pfn = pte_pfn(pte); - - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); + set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); } pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, @@ -512,13 +487,6 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; size_t pgsize; int ncontig; - pte_t pte; - - pte = __ptep_get(ptep); - VM_WARN_ON(!pte_present(pte)); - - if (!pte_cont(pte)) - return ptep_clear_flush(vma, addr, ptep); ncontig = num_contig_ptes(huge_page_size(hstate_vma(vma)), &pgsize); return get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig);