From patchwork Fri Nov 3 14:01:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13444576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48670C4332F for ; Fri, 3 Nov 2023 14:04:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5EDE28001D; Fri, 3 Nov 2023 10:04:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0DDE28000F; Fri, 3 Nov 2023 10:04:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD63D28001D; Fri, 3 Nov 2023 10:04:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AC26428000F for ; Fri, 3 Nov 2023 10:04:44 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7EEB1A079D for ; Fri, 3 Nov 2023 14:04:44 +0000 (UTC) X-FDA: 81416813688.24.5F055B9 Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by imf18.hostedemail.com (Postfix) with ESMTP id 2D0821C0035 for ; Fri, 3 Nov 2023 14:04:02 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699020243; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hFPL2wSMrPcu1hJjxD+4tynyVgkuYhp6u1rBQCBIGCw=; b=4biW4JfkrKs+PZXKrRXp5RTofrqwmWMiTQCUGz6PEb3PoagddN9j/YYXwDR3tjuho02S6v qvYKmraW/zR2OzPYb0mDz0qcxGYAEgqc3O2TPOtLNLPFTw5lc9HiQEFCnrLfy2hODBoZtg /Cyt1etECeaHRp0h6GAppoCMy+ZhyMc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699020243; a=rsa-sha256; cv=none; b=bt4lj+pJlzBvh57K+PLS7b9Hul+2iCLxKZ3ueR5RRZPCteT4UZj8YD6FiqZEPII6E+V3BS mQ0j1h+SMTqECTb3V/S0FVO8YgUXwpE658waJCgiY3/GGkJ59NXA5W/yT7ViojvVv8MFNx k5h378GMUUpHIW3+WyIT+CTDCSQrynU= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from dggpemm100001.china.huawei.com (unknown [172.30.72.54]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4SMMpQ1tzNz1P7pq; Fri, 3 Nov 2023 22:00:46 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Fri, 3 Nov 2023 22:03:49 +0800 From: Kefeng Wang To: Andrew Morton CC: , , Matthew Wilcox , David Hildenbrand , , Kefeng Wang Subject: [PATCH 16/18] mm: use a folio in zap_pte_range() Date: Fri, 3 Nov 2023 22:01:17 +0800 Message-ID: <20231103140119.2306578-17-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20231103140119.2306578-1-wangkefeng.wang@huawei.com> References: <20231103140119.2306578-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemm100001.china.huawei.com (7.185.36.93) X-CFilter-Loop: Reflected X-Stat-Signature: cerwuifnpdox56s36a4qdswtwt9cgcd9 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 2D0821C0035 X-Rspam-User: X-HE-Tag: 1699020242-911520 X-HE-Meta: U2FsdGVkX1/XHg+vbeNGKuvW8jiXVldtSfPaREOfinlJkVuVMIZmRLShzcAf1mhHjgg2zSt/CJSiw/002C0elwGY0WaohMAgJEqZTQc76csY3WDpRCUkMRempuBBQrY4mk2P2YurSsBjHgK053ahp4+6jSAsqZnmhUxV/bOzPrQvO7ncdA1tP4nS+X1CJueVlP1bf4xKvfZJa+zaYff02EoXi0MNKDNaKIdgsj0tEKhj1bGdCFkUhSdf93YbxBg0qlcidovU3lMzkCL9Gi52QTtzGVE3hClo8stZo1aoTYMMcWAoLm02Y8u2kCJdtusM/leGYUetlnKZPSJlq9OfOnnxzybd3RxNyBCsKAkbVcOIN3WUE30Tehl42Pur5hbkclovQg5mPIOV+VUaOLd/ojK/HQKblek9fR4s9PBr/j0y5/pPzkVaRKnAX/OUiCvQmswxwAcCIwiHVm69m4/z4LJH0mnS8XhulGuT3dbHaOhOzUtzNhVi2/9DQmDTJhTR1eq5ZgCqm7vc/IlG/sZISK6NAeMLQIr7Gywg4zmhPvllxycmvUL4hqqTRof5Q8aqKCEVv6RnMFA5yocd3mqVWs1mEqWVkHDV4HYHxtBD1iZl5t1IyEYLd9HusRlSsuf2rtadIYIuSd3tuKcrddEJpV6z5GOvT+XZZQngJA8kP43RkETDz7LnyXRPxonLaXo6g7husPM+0VsNYruRZkZB3ATcA5PDDSlxyrolfGNsAEMgBILxw3aw4eSjAlrdzBS4aSdoH1gy1YkiNCzEdFf/RJ3ad5iEIOmPvat+C/iWMDhSRSQO03wk6gskxcS/5pcF94g4QdYobNSJo3ZCJG+31v4T1QdVcGNIIyaHzf0sO0eHdcnGvCwCkUytsc0UoPu0hEqkk3uYAhDaVFBamZ9t0Srl2iL31vslJzdCFevKx3MQnoYuftGgLtcnFg51nGoJfTRwMOEYZbLscDob4YH wE6/YLQk j3vDXrFZ5VT3ap43LRvjCivdGSWpmGXNOxyUgkvEUOZSjVIg0C7c2gY1M7wC5snv1Q7CQbtbVPPs182oMw4TDGxJ6NfBISX7pcb86nVAYrXiaf5DsNuVyajx4Xs99AVdPeGSKw6suEClIka3KPlAGMRkzKJAX9VgnBh4+teBztRVX4WAHYrzKXJi4T834UQY8HGTm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make should_zap_page() to take a folio and use a folio in zap_pte_range(), which save several compound_head() calls. Signed-off-by: Kefeng Wang --- mm/memory.c | 43 ++++++++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 19 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index fcc04dce8e8a..9b4334de9bf0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1358,19 +1358,19 @@ static inline bool should_zap_cows(struct zap_details *details) return details->even_cows; } -/* Decides whether we should zap this page with the page pointer specified */ -static inline bool should_zap_page(struct zap_details *details, struct page *page) +/* Decides whether we should zap this folio with the folio pointer specified */ +static inline bool should_zap_page(struct zap_details *details, struct folio *folio) { - /* If we can make a decision without *page.. */ + /* If we can make a decision without *folio.. */ if (should_zap_cows(details)) return true; - /* E.g. the caller passes NULL for the case of a zero page */ - if (!page) + /* E.g. the caller passes NULL for the case of a zero folio */ + if (!folio) return true; - /* Otherwise we should only zap non-anon pages */ - return !PageAnon(page); + /* Otherwise we should only zap non-anon folios */ + return !folio_test_anon(folio); } static inline bool zap_drop_file_uffd_wp(struct zap_details *details) @@ -1423,6 +1423,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = ptep_get(pte); + struct folio *folio = NULL; struct page *page; if (pte_none(ptent)) @@ -1435,7 +1436,10 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, unsigned int delay_rmap; page = vm_normal_page(vma, addr, ptent); - if (unlikely(!should_zap_page(details, page))) + if (page) + folio = page_folio(page); + + if (unlikely(!should_zap_page(details, folio))) continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); @@ -1449,18 +1453,18 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, } delay_rmap = 0; - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { if (pte_dirty(ptent)) { - set_page_dirty(page); + folio_set_dirty(folio); if (tlb_delay_rmap(tlb)) { delay_rmap = 1; force_flush = 1; } } if (pte_young(ptent) && likely(vma_has_recency(vma))) - mark_page_accessed(page); + folio_mark_accessed(folio); } - rss[mm_counter(page)]--; + rss[mm_counter_folio(folio)]--; if (!delay_rmap) { page_remove_rmap(page, vma, false); if (unlikely(page_mapcount(page) < 0)) @@ -1477,9 +1481,10 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, entry = pte_to_swp_entry(ptent); if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { - page = pfn_swap_entry_to_page(entry); - if (unlikely(!should_zap_page(details, page))) + folio = pfn_swap_entry_to_folio(entry); + if (unlikely(!should_zap_page(details, folio))) continue; + /* * Both device private/exclusive mappings should only * work with anonymous page so far, so we don't need to @@ -1487,10 +1492,10 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, * see zap_install_uffd_wp_if_needed(). */ WARN_ON_ONCE(!vma_is_anonymous(vma)); - rss[mm_counter(page)]--; + rss[mm_counter_folio(folio)]--; if (is_device_private_entry(entry)) page_remove_rmap(page, vma, false); - put_page(page); + folio_put(folio); } else if (!non_swap_entry(entry)) { /* Genuine swap entry, hence a private anon page */ if (!should_zap_cows(details)) @@ -1499,10 +1504,10 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); } else if (is_migration_entry(entry)) { - page = pfn_swap_entry_to_page(entry); - if (!should_zap_page(details, page)) + folio = pfn_swap_entry_to_folio(entry); + if (!should_zap_page(details, folio)) continue; - rss[mm_counter(page)]--; + rss[mm_counter_folio(folio)]--; } else if (pte_marker_entry_uffd_wp(entry)) { /* * For anon: always drop the marker; for file: only