From patchwork Tue Mar 26 20:28:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13605067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C775CD1283 for ; Tue, 26 Mar 2024 20:28:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 34B126B009C; Tue, 26 Mar 2024 16:28:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AAE16B0095; Tue, 26 Mar 2024 16:28:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 123296B009A; Tue, 26 Mar 2024 16:28:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EC02B6B0095 for ; Tue, 26 Mar 2024 16:28:39 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C7955160165 for ; Tue, 26 Mar 2024 20:28:39 +0000 (UTC) X-FDA: 81940328358.24.5BA9C23 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id B59BE10000B for ; Tue, 26 Mar 2024 20:28:37 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ST1EguAN; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711484918; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rtFLGbGqXpsjSue0ZqqRPXjXFuO8I+KTByqez1b7bEw=; b=ODPpPoPTz54DAyaPfhxxCxO6aLgsKhWqsVKfj3iEzWlexacbUoFSFW2HDDxPq1eDsdDgog oLNvhpd6a5JekqVPiXqRBee4JAiu5+09MTEfw3MtJdnkj2YbGo65pNxCF/+e1XxQMvEHqN FZS85XiYE3vMocQcRADqBKfWEGMXx2M= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ST1EguAN; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711484918; a=rsa-sha256; cv=none; b=lZDylh+o6WgqbEdmIXux/Mr+lzZDXtr2u8lRmkgRklpncxx4ZVwJt5wkV2qZ5OFNnPwgnS jOl23U6CD7ORQi2chVgWvi/8C2zDNN3LeD9Lud8em2aJ2T0OJEefJC68DhCc3PpyHbw8DU Y+r9VcPDpHerd0ks0CRpiEvz6W2eu3c= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rtFLGbGqXpsjSue0ZqqRPXjXFuO8I+KTByqez1b7bEw=; b=ST1EguANN2O3F1yfmR/V1z/Y6k CZdv5TxWFfImDgLBx+j2bO6GScpjQgXpRX/DbdZS2SByG7B/PLFS+2YdLZ2eznTf0Snf6VCnrS8RK yIIxrTN3GQd+KqFLfg8yY01elwA3jkHN3q5KPOIbQIwVzp3bjPHky2XLv9g6sI9Fkaxqt+MG+EWqk MdgpJGrfxRTu70wp4Wg0q3EZHebekaYYI9+bqXu5KZDxAsdPYv9vQkNPCNbUHI3+kfyGY7RfCggQe /GK8P7roppJTk3HGS6/4bT5ruhRPgzVA2FOw7E33xTPLTJQfH2B02vGBHEJoK8dlqZVeTPuVJjfAW b647SMPA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpDPQ-00000002CGL-1pEM; Tue, 26 Mar 2024 20:28:36 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH 2/8] mm: Add is_huge_zero_folio() Date: Tue, 26 Mar 2024 20:28:22 +0000 Message-ID: <20240326202833.523759-3-willy@infradead.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240326202833.523759-1-willy@infradead.org> References: <20240326202833.523759-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 1hgcmx6h74ejtca89ga5jmt468saidjo X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: B59BE10000B X-HE-Tag: 1711484917-653137 X-HE-Meta: U2FsdGVkX18IXcVytZz6WKRhbBzKL55yuTXnjtt/lWAGwV9CLzmrbCBXenubtSvuzy4X/bNNeHAnzxstKq9QFHTGvIkNW3mLpc7NkGcCCbVW7+7U0ME374pjAeGf/veKBkFc6o3rPcAuINHe+WNb9+s9XnzriDJs788+2Xj55vO1oCRyBexpSUtRwE3tSE401we1inhqor4FJfMRHsnc30mIwSBkAr3otBO8dUmBNtVW0rWps1zIU7E9zgKTjxCyy8EPrl7o75tM/HBFtd/Vwl9MO/fFbY18VTVxSt5+LguVg6oVnaawOHU2RZksX0NY5nJupY1Zc+m4iI3Ygmj5QFn/9Xm8fCOu7hWx0abJhR6h/gbHIgGDc9VjIIS14kjSOSOyGRhMwIQeNhhlldhRGe3i7rQDPAbOKF7IjN0V3nL5oJwICBEw0Ltc/hunzEyKYXuTyMK2OQoytzS1kJPkEXdAOTanPCle54G3LeBkxJrR/umqWED/K6Rv7uD11QUZTh3mVcpPSzcC+RTm+z1E9xYYAGWQkIhHpggy/6C5swS3s7xvcITwdAsUjYnG7vQ7HjRlP1I91RkysiuDhTUgyIEgunCGoZwOrCXTuzBtWu7sIouZCCFga9LWkOddAc+XcaVL3rzfASVKhLsbyTcoLnMlBJWb4WnBhh585EcdHRrg7xGUJF+thpmYspWHqyJOfmjedfIgBSN38Lhjxv2LQdbz11QB8KHLWtKeIbBGdzwCHj3qeYuUpJrrog8RNCcWWxEdq1k6qRFo2V3w5FfDnlNYOpOKC7AmlY8+xGDP6gCbLM9a7ZrMGy+CsJrJkVPBWGii6j5v2MgMRvjgurcOeBTmGseXZeD4TIZUQbBmaWOAxFFIReFXgTT4/hyoQ/1Uw0IXFBV6CnYsgdljXFO54/W5CBtVb52aWMFJid5YQW0gclRg/uWJ7NPpzGL4NU7Vsv2s0CsX8nPzPVrshN4 V5LVWine kx7CYtCYySiPbEL575I2y4iKWTEQtQHn9bT8Ubl6CY68He4YWnBbOvxr+Z7OOrAy7/5T2ENBrdIvwUlHO2svg4xGNQpvIZqQe1Ss72gygums0433xjoDj0EV1FaymB4SmX4F372Bq8EtJT1i2JdWcZX3Q0PXb7OZlFJUckj8SeYVVQ0xQ7jzXFxnaCNBaZEnue7Zq5oV38bgV3Xqh/t3rFc/RNhfM3YYc8DS3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is the folio equivalent of is_huge_zero_page(). It doesn't add any efficiency, but it does prevent the caller from passing a tail page and getting confused when the predicate returns false. Signed-off-by: Matthew Wilcox (Oracle) --- fs/proc/page.c | 2 +- include/linux/huge_mm.h | 10 ++++++++++ mm/huge_memory.c | 6 +++--- mm/mempolicy.c | 2 +- mm/swap.c | 2 +- mm/swap_state.c | 2 +- mm/userfaultfd.c | 2 +- 7 files changed, 18 insertions(+), 8 deletions(-) diff --git a/fs/proc/page.c b/fs/proc/page.c index d6953f95e3b4..5bc82828c6aa 100644 --- a/fs/proc/page.c +++ b/fs/proc/page.c @@ -155,7 +155,7 @@ u64 stable_page_flags(const struct page *page) else if (folio_test_large(folio)) { if ((k & PG_lru) || is_anon) u |= 1 << KPF_THP; - else if (is_huge_zero_page(&folio->page)) { + else if (is_huge_zero_folio(folio)) { u |= 1 << KPF_ZERO_PAGE; u |= 1 << KPF_THP; } diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 1540a1481daf..600c6008262b 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -356,6 +356,11 @@ static inline bool is_huge_zero_page(const struct page *page) return READ_ONCE(huge_zero_page) == page; } +static inline bool is_huge_zero_folio(const struct folio *folio) +{ + return READ_ONCE(huge_zero_page) == &folio->page; +} + static inline bool is_huge_zero_pmd(pmd_t pmd) { return pmd_present(pmd) && READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd); @@ -485,6 +490,11 @@ static inline bool is_huge_zero_page(const struct page *page) return false; } +static inline bool is_huge_zero_folio(const struct folio *folio) +{ + return false; +} + static inline bool is_huge_zero_pmd(pmd_t pmd) { return false; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4556144cff09..78f9132daa52 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -789,12 +789,12 @@ struct deferred_split *get_deferred_split_queue(struct folio *folio) } #endif -static inline bool is_transparent_hugepage(struct folio *folio) +static inline bool is_transparent_hugepage(const struct folio *folio) { if (!folio_test_large(folio)) return false; - return is_huge_zero_page(&folio->page) || + return is_huge_zero_folio(folio) || folio_test_large_rmappable(folio); } @@ -3085,7 +3085,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, } - is_hzp = is_huge_zero_page(&folio->page); + is_hzp = is_huge_zero_folio(folio); if (is_hzp) { pr_warn_ratelimited("Called split_huge_page for huge zero page\n"); return -EBUSY; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 3fae74b62c51..6e7069ecf713 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -510,7 +510,7 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk) return; } folio = pfn_folio(pmd_pfn(*pmd)); - if (is_huge_zero_page(&folio->page)) { + if (is_huge_zero_folio(folio)) { walk->action = ACTION_CONTINUE; return; } diff --git a/mm/swap.c b/mm/swap.c index 500a09a48dfd..f72364e92d5f 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -985,7 +985,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) struct folio *folio = folios->folios[i]; unsigned int nr_refs = refs ? refs[i] : 1; - if (is_huge_zero_page(&folio->page)) + if (is_huge_zero_folio(folio)) continue; if (folio_is_zone_device(folio)) { diff --git a/mm/swap_state.c b/mm/swap_state.c index bfc7e8c58a6d..2deac23633cd 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -301,7 +301,7 @@ void free_page_and_swap_cache(struct page *page) struct folio *folio = page_folio(page); free_swap_cache(folio); - if (!is_huge_zero_page(page)) + if (!is_huge_zero_folio(folio)) folio_put(folio); } diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 163511ab5896..4f1a392fe84f 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1699,7 +1699,7 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start, !pmd_none(dst_pmdval)) { struct folio *folio = pfn_folio(pmd_pfn(*src_pmd)); - if (!folio || (!is_huge_zero_page(&folio->page) && + if (!folio || (!is_huge_zero_folio(folio) && !PageAnonExclusive(&folio->page))) { spin_unlock(ptl); err = -EBUSY;