From patchwork Mon May 20 19:47:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13668624 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F5DCC25B77 for ; Mon, 20 May 2024 19:48:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F03036B008C; Mon, 20 May 2024 15:48:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB2726B0092; Mon, 20 May 2024 15:48:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D53856B0093; Mon, 20 May 2024 15:48:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B50036B008C for ; Mon, 20 May 2024 15:48:50 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5B95E120F5D for ; Mon, 20 May 2024 19:48:50 +0000 (UTC) X-FDA: 82139812020.29.20B5122 Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com [209.85.128.182]) by imf04.hostedemail.com (Postfix) with ESMTP id A18814000D for ; Mon, 20 May 2024 19:48:48 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AQTCCbBc; spf=pass (imf04.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.182 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716234528; a=rsa-sha256; cv=none; b=paIDUXep40F1kxroJMYODrJMyv9gK2Tt8xlD4IQqRfwSUwL5QfMUb6ttGaxqZKr/5x38O+ nRMddBHEXtZbNaTzWMHqUfvpZDk2P/4JqvvpJSbcxT3l4GwRi63sPFv+YIOD2Domfo/5Oo juIj9UUaMl23eVdKCB0bbNUHcyKvT38= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AQTCCbBc; spf=pass (imf04.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.182 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716234528; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=CxHZ56zm4ItHAREln0MAqCnqcbTO0AQUe9UM1z9zLfY=; b=YaEr+8W3kXs1qDcyMt3+yi3VvKjvGOKfgAqH01jqxG/F3PY+A5SxJAdQAybySQKiJn+Om/ OaVnc5Y7BbZ/BbUvZrnRQ9UqCGseuiGRhJ/jVe0sUOMQTGcmoDXQ7Bq56LmgUky7wkVB9j +kcK0aNAr2+BRGWquR8NP8GMAExl0MU= Received: by mail-yw1-f182.google.com with SMTP id 00721157ae682-61b2218ab6fso21559747b3.1 for ; Mon, 20 May 2024 12:48:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716234527; x=1716839327; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=CxHZ56zm4ItHAREln0MAqCnqcbTO0AQUe9UM1z9zLfY=; b=AQTCCbBcnhnC1SnxganfWwM7UZ0j0YhKWY/tr9+CEFqJJBydx1Bu90mdC/G7L8vP/3 VF4bot8ot2XLDVFtLeXJWr1UTpnw35AeQyClwX9WRF2yVqRpnRwIZ7L/b11sOj32o/Q8 GcZP5GhesaD+Vdxr2N3CB8M5dlhxaelsZKu4hikXgGFQ1FN4JzOcARrRcTMTeCjN3qse sOUAT+BZrR++FP5l7RkeORf9hTKiufqwkw4kGqjPa6gXcGrGq7VOq1ZNzXUfXxb4Pz+c BwmscFFagdBvBoKw933pxvJUpTdR7ZaXibJOQADNEmS0hJ3Mc06o7emqZRp5HDsTCQXe 5zIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716234527; x=1716839327; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=CxHZ56zm4ItHAREln0MAqCnqcbTO0AQUe9UM1z9zLfY=; b=LwmgSPfPr49G0Hbp9RlVNpbjGqBQu3iSdJ1p8oyUIt/vvhdJjJKohfwDZ3w1tTESsx S6sTW0LvuHJpMTSw6x7XMu82Tz4AMl3K34A2SlC4HM7N3d+Runb89xYsmZ3UXtbCbjBt NqZQrjh8jryZbAKFiacIGoG+DHL/v0l3y+S7Y/ehNRRsVv8MlvpPRqhvaoYRmYVfgd96 efjd9jEJmR1WFMS2ENpeHjsbvrtNDVSZ9mIBep4C+4BVa1f99pPZp4vXLFFkqJfNkAMu 1AGuKV2jN+ylZ96jvgy4Ul7VPdudJxo7hGjjbRrKZQoWwCCOp6ZF0c3zB5AV4u087wkP CFlw== X-Gm-Message-State: AOJu0YyZk36v9kmO+B3MH8iANXfHLiYqH+WBlqZxUr5YK6/PbGoftPil KQ9a/FHz1hst1FO3rMihR9jM1HyQa4+eh1FYwcJxpf7DbsSN1+LwcbF2lXYq X-Google-Smtp-Source: AGHT+IGRVnzRuJy3FzjwWKxx39LtXKiCJCYMf1/eg0CGPu5354+L2uqAHc6D7tg5dpC89Hd4MY3IAg== X-Received: by 2002:a81:e24d:0:b0:61a:c316:9953 with SMTP id 00721157ae682-622aff42b46mr301350797b3.11.1716234527493; Mon, 20 May 2024 12:48:47 -0700 (PDT) Received: from fedora.attlocal.net ([2600:1700:2f7d:1800::42]) by smtp.googlemail.com with ESMTPSA id 00721157ae682-6209e346760sm50695827b3.78.2024.05.20.12.48.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 May 2024 12:48:47 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: akpm@linux-foundation.org, muchun.song@linux.dev, linux-kernel@vger.kernel.org, "Vishal Moola (Oracle)" Subject: [PATCH] Convert __unmap_hugepage_range() to folios Date: Mon, 20 May 2024 12:47:49 -0700 Message-ID: <20240520194749.334896-1-vishal.moola@gmail.com> X-Mailer: git-send-email 2.45.0 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A18814000D X-Stat-Signature: 974urijiodcuty1uwryqyo37jc459579 X-HE-Tag: 1716234528-283240 X-HE-Meta: U2FsdGVkX18MgWNBJCBPYLMwYP3Hz0IazM6UfHuMO6jDZIwvd3IKx1rrEOAnv4ds6sEBdmyLCYom6nohJQmbXCGglHboKBu8SINIVwEqjNvU2uRxPTuPzIxaXzvjN1p2k10kqA3IsK1qpNMlTW8Avdo4/xc8DJPbm/YNenqEGSIT5Gqo9LjTCS9SJWLGmJQtSE/V77rkadzz5bDEWIoSrwGjpUdsnpYxhvET0Nrbhlom4Ml/wZR36Sp0Qg8qHfriF3rEedfa9vjGNJHrvUEWFZ5rHQw96llLRYwHOABZCFxFvAwgasqCZ15K///L72UdYLfRQ6kIFhSoO/ezZetTg7fMnuIO0KEMTDTAqVWjlQPkCKdIkTY3EiQDfQM5inRLqJO10XxMzBM1lz89vDtSEZLl/U0Z48UkT+rFqENwXJfQ5OEszpxQrnKLRpxsO9Rf0pLi64Kq7dvNFNqDX4CevxPiN3uivzqjMx0nAFV1DYtWKxEMzUZDC2mYMyaft8xda4moI2FIKXs33yTucSZvUc1emPr5FadAF1cqB3cqnma4LGw7/FBnlzChhYAsVPKZLKDKtdX8L/QJWnNqRGmGX22iSxywkKgsSp0+DK/xjnN34oNGlW2sryrA/zJcFOEA66XuSVCPL3wLVH51x6I6HPiZLuDx7PVXhTpQ5zp6e9/RW/MLcUXD4/iBqW9sGU/ua7zamclwyK+fQD2YP0FwBx8B6hq7ZVW943FDytmMjXZ2XGxuMBZHi1UnuNVDdjnmWSysIoCNAla4tstx3vnfB3uRaVo2/5Y3uhExqvoncxtlmv68rF6HV68hmkI1/dnUIlvhiontUSUNIPo6o7EgYvAlVWfBjAESwX9heJP6DtHUjgc/X8Ot5zDfb3N+0CrMo0Feifent7IaYcORUgP9dn2F5zUId7FKyxElxeOTLjGaD97BBDamCQip6r1zftwXTxdAxnBY/oxYeiPebwq 9ddabRmA RV9Y7Mq9By44IUniqcLPoTZx+nCX/xLAgG8NkmNgvsU2ATl0faJPykjReFG6vYlD3XJDrTXYzgNP074eYwe+VMyl9ffW7OU2bcp943oRqoo6i49l/Z/Aes4kUuRTvRBe1js4qJoQn+115rAJv9iaiuBZ+KHNW3RdfZTi8uVh4Rg57Lj4s4bFhh6Y9KzDyr3/0RqJl36VcYmsKuymonNwcsxxgV7U7AiL8X1nk1giO+oA9NO+JUvnEOQ9EDMNiAQ6rpbX5YmjPSnRbp0Sk7IhTBkqNzuEngONMrxH5AnH6iqnIOliuxNIxlmE7J6wGU4QoS5HCTAMkbKef0FG+rZX9DoA3j5KdqwJM+ZJvjD0T+45c/WOpuroYx5Xpvs9aZ8SYjVt5HcsF2m6FIHi7ebDtX09nig0H1eDYYSEM8Ejv0ZWmzx96DV3FJOaC+OXhJtsGzHRjqi4y9NK0ffqzcRlmOGSnzqFx9kKoLjTsodccJaHN5zRnaAXcgV1pIo8nP1z/DZj7C5vh0yAjkYlekT9clCln1bpkO+SdxuK/aeRMj4Fu0q58MwvpWvw0nV807AaxbGv6nvtIQrxcsec= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Replaces 4 calls to compound_head() with one. Also converts unmap_hugepage_range() and unmap_ref_private() to take in folios. Signed-off-by: Vishal Moola (Oracle) --- include/linux/hugetlb.h | 6 ++--- mm/hugetlb.c | 50 ++++++++++++++++++++--------------------- 2 files changed, 28 insertions(+), 28 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 68244bb3637a..6c7ce8679950 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -137,12 +137,12 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask); void unmap_hugepage_range(struct vm_area_struct *, - unsigned long, unsigned long, struct page *, + unsigned long, unsigned long, struct folio *, zap_flags_t); void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, - struct page *ref_page, zap_flags_t zap_flags); + struct folio *ref_folio, zap_flags_t zap_flags); void hugetlb_report_meminfo(struct seq_file *); int hugetlb_report_node_meminfo(char *buf, int len, int nid); void hugetlb_show_meminfo_node(int nid); @@ -458,7 +458,7 @@ static inline long hugetlb_change_protection( static inline void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page, + unsigned long end, struct folio *ref_folio, zap_flags_t zap_flags) { BUG(); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6be78e7d4f6e..02f40bfa3686 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5643,14 +5643,14 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, - struct page *ref_page, zap_flags_t zap_flags) + struct folio *ref_folio, zap_flags_t zap_flags) { struct mm_struct *mm = vma->vm_mm; unsigned long address; pte_t *ptep; pte_t pte; spinlock_t *ptl; - struct page *page; + struct folio *folio; struct hstate *h = hstate_vma(vma); unsigned long sz = huge_page_size(h); bool adjust_reservation = false; @@ -5663,7 +5663,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, /* * This is a hugetlb vma, all the pte entries should point - * to huge page. + * to huge folio. */ tlb_change_page_size(tlb, sz); tlb_start_vma(tlb, vma); @@ -5714,19 +5714,19 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, continue; } - page = pte_page(pte); + folio = page_folio(pte_page(pte)); /* - * If a reference page is supplied, it is because a specific - * page is being unmapped, not a range. Ensure the page we - * are about to unmap is the actual page of interest. + * If a reference folio is supplied, it is because a specific + * folio is being unmapped, not a range. Ensure the folio we + * are about to unmap is the actual folio of interest. */ - if (ref_page) { - if (page != ref_page) { + if (ref_folio) { + if (folio != ref_folio) { spin_unlock(ptl); continue; } /* - * Mark the VMA as having unmapped its page so that + * Mark the VMA as having unmapped its folio so that * future faults in this VMA will fail rather than * looking like data was lost */ @@ -5736,7 +5736,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pte = huge_ptep_get_and_clear(mm, address, ptep); tlb_remove_huge_tlb_entry(h, tlb, ptep, address); if (huge_pte_dirty(pte)) - set_page_dirty(page); + folio_mark_dirty(folio); /* Leave a uffd-wp pte marker if needed */ if (huge_pte_uffd_wp(pte) && !(zap_flags & ZAP_FLAG_DROP_MARKER)) @@ -5744,17 +5744,17 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, make_pte_marker(PTE_MARKER_UFFD_WP), sz); hugetlb_count_sub(pages_per_huge_page(h), mm); - hugetlb_remove_rmap(page_folio(page)); + hugetlb_remove_rmap(folio); /* - * Restore the reservation for anonymous page, otherwise the - * backing page could be stolen by someone. + * Restore the reservation for anonymous folio, otherwise the + * backing folio could be stolen by someone. * If there we are freeing a surplus, do not set the restore * reservation bit. */ if (!h->surplus_huge_pages && __vma_private_lock(vma) && - folio_test_anon(page_folio(page))) { - folio_set_hugetlb_restore_reserve(page_folio(page)); + folio_test_anon(folio)) { + folio_set_hugetlb_restore_reserve(folio); /* Reservation to be adjusted after the spin lock */ adjust_reservation = true; } @@ -5771,11 +5771,11 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, if (adjust_reservation && vma_needs_reservation(h, vma, address)) vma_add_reservation(h, vma, address); - tlb_remove_page_size(tlb, page, huge_page_size(h)); + tlb_remove_page_size(tlb, &folio->page, huge_page_size(h)); /* - * Bail out after unmapping reference page if supplied + * Bail out after unmapping reference folio if supplied */ - if (ref_page) + if (ref_folio) break; } tlb_end_vma(tlb, vma); @@ -5837,7 +5837,7 @@ void __hugetlb_zap_end(struct vm_area_struct *vma, } void unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page, + unsigned long end, struct folio *ref_folio, zap_flags_t zap_flags) { struct mmu_notifier_range range; @@ -5849,7 +5849,7 @@ void unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start, mmu_notifier_invalidate_range_start(&range); tlb_gather_mmu(&tlb, vma->vm_mm); - __unmap_hugepage_range(&tlb, vma, start, end, ref_page, zap_flags); + __unmap_hugepage_range(&tlb, vma, start, end, ref_folio, zap_flags); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); @@ -5862,7 +5862,7 @@ void unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start, * same region. */ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, - struct page *page, unsigned long address) + struct folio *folio, unsigned long address) { struct hstate *h = hstate_vma(vma); struct vm_area_struct *iter_vma; @@ -5898,7 +5898,7 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, continue; /* - * Unmap the page from other VMAs without their own reserves. + * Unmap the folio from other VMAs without their own reserves. * They get marked to be SIGKILLed if they fault in these * areas. This is because a future no-page fault on this VMA * could insert a zeroed page instead of the data existing @@ -5906,7 +5906,7 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, */ if (!is_vma_resv_set(iter_vma, HPAGE_RESV_OWNER)) unmap_hugepage_range(iter_vma, address, - address + huge_page_size(h), page, 0); + address + huge_page_size(h), folio, 0); } i_mmap_unlock_write(mapping); } @@ -6035,7 +6035,7 @@ static vm_fault_t hugetlb_wp(struct folio *pagecache_folio, hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); - unmap_ref_private(mm, vma, &old_folio->page, + unmap_ref_private(mm, vma, old_folio, vmf->address); mutex_lock(&hugetlb_fault_mutex_table[hash]);