From patchwork Thu Oct 31 08:13:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13857705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F3CBE68958 for ; Thu, 31 Oct 2024 08:14:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C7FB96B0096; Thu, 31 Oct 2024 04:14:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C31136B0098; Thu, 31 Oct 2024 04:14:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB3446B0099; Thu, 31 Oct 2024 04:14:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8B7256B0096 for ; Thu, 31 Oct 2024 04:14:22 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 497918127A for ; Thu, 31 Oct 2024 08:14:22 +0000 (UTC) X-FDA: 82733184912.29.BD550F2 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf20.hostedemail.com (Postfix) with ESMTP id 39FFF1C001F for ; Thu, 31 Oct 2024 08:13:47 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hlwQEO5g; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730362299; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0uS9zjnEttUyVTWzhhVIHAdfZYrGflmgRk9QlOwbpbs=; b=ynF+3WBIHVxb50jtMSRumNq8DYOMuR/SCv90mY5Qi6lVf0uDAMXEIziB2Iu96gDSQkCV2S 3vaBy3YOZc/WI0rV2DFtB97VOezXWHPwZtfKaxQgiYAqhjRF1w+uROR0FdN4okDMUIjNYT JTXGnJFUuep28VlBS0izQC9Gjakk7OU= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hlwQEO5g; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730362299; a=rsa-sha256; cv=none; b=Nr9bJerQ5wJWFdbdCL/4uMgnV4rbBI5qdse1zL0KQydisBE0o13jvGNT8nEhKvIdA63USG HIybcEGOf8xmkNWLds+tCSWjzIHbPdBXW23mYWKvxNewvQH9RrKIaHLGlMwr6P4jeyXqKe +un6qi4meAmwu1mxJOIBeT8FWpVR8eg= Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-2e2eb9dde40so517888a91.0 for ; Thu, 31 Oct 2024 01:14:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1730362459; x=1730967259; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0uS9zjnEttUyVTWzhhVIHAdfZYrGflmgRk9QlOwbpbs=; b=hlwQEO5gY/G9XdiAhug443EbRd8aRGCjAiLUdceLENV8VpWO7x5Gn2XhkCmYFGY/iJ SS+uwr6cW4tHd3LEohIG4ksxvGtzSFi2gBL06zb4MKNgwBTnOAmuVmbJ0NF0JRqvdYyY GLrPPUwuGh1S7zmsPH0eiqhOR/ehzlZlCcCEDnVhavEWEgCEknyxk0mh9GbXtQTXNRvN orjRpqL/w6hR9fvEG5Ihyn6DMlSXdIS6CO6JYwYKBXNom3vYq72dBjB/A/BJGIkzcfaZ TPxuXHwLbygKARB3XQqh4QclzGaED+6NrCQOju4c4iwmNJGM3ZvUBOtr15D4gblUxHQ8 4vXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730362459; x=1730967259; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0uS9zjnEttUyVTWzhhVIHAdfZYrGflmgRk9QlOwbpbs=; b=WOIyG4zZeCNMFe1KVzz4g+45lDQFYM51h73tiAXeUNSpytzpOgHoCr/iT5+CbC4JM5 RHQ+wgjuis0TpixBGUcTNpue3X2LQoeWEr6NmoOZvDxFyvYCWyhnrhBlOHn7WqImDCet e3c2PXh10vdrUkspFno4WTi7SCdDh0R0mGDbEPi6LYgTiJE052qccX0lgezPY2Kf6M1q tx99U3IzHUOQrOuT8Fc/z+Fu9WBbPmQpKiIEgoTIURyLjxVW6PA0yj5OZP1+ltk5QTsv 1B5pxme4APeFzLcujL0qFydweO5kIaeI4lQ1iHwaHn4dqHzkjHOT8qRw4pZOHLKBpglk nTzw== X-Gm-Message-State: AOJu0Yxpu/ltqgqg2x/mP45T94UzgIzjF+uDxQz8xOwZV93MVb/vClms J+9HALpRo1LU0wrf9Dba4jnOCMTH3unTwW3zfEQhVpP1RAoNC/OAW3RHZyA6Heo= X-Google-Smtp-Source: AGHT+IHliHONlLjM71G0WcLE2jZg2du2F6KU7SIJsE29Lnbx/3IrRdww7bJdFluVjjQ6fAFe8KWB2A== X-Received: by 2002:a17:90b:53c3:b0:2e2:d859:1603 with SMTP id 98e67ed59e1d1-2e93c1d39b0mr2658162a91.25.1730362459134; Thu, 31 Oct 2024 01:14:19 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.149]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e9201b2586sm5163996a91.0.2024.10.31.01.14.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 31 Oct 2024 01:14:18 -0700 (PDT) From: Qi Zheng To: david@redhat.com, jannh@google.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, zokeefe@google.com, rientjes@google.com, peterx@redhat.com, catalin.marinas@arm.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, Qi Zheng Subject: [PATCH v2 2/7] mm: introduce zap_nonpresent_ptes() Date: Thu, 31 Oct 2024 16:13:18 +0800 Message-Id: <34ef26a763af7b2d8dee707b6f60ff85edda345f.1730360798.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 39FFF1C001F X-Stat-Signature: y1axpp3k4j9icc1n1g8gt16xkhyj9r7d X-Rspam-User: X-HE-Tag: 1730362427-37366 X-HE-Meta: U2FsdGVkX1+pBr9FnB+84iHHDE3cF0WEsoCJpA6kRwjKY/CPM70Q7F/y0zob+HtlCYW50VxOtCioNPA36EOzf5A8WIGYs1s9dw3A+nS6K4uJcSckVqF63EAGoXEP8id7lU8RWan3p994yilIhXZAkxsx08fnLsDFYzwT57Ran4/8mYcdbgsRZ5V4fqcoosTbdwkeamxEHc0MPogzBjDG4RfLXpe569w8yDcNGH1QkM1oweGhkhaHvhpTOzfT8vJ9ukM85QDtja5LqX9PH+rCqfdAZSzqcr9+PeO73Gz24m1Voq0Fk/gr4a52of2uzd2+Hh2JuOCBZus5d6tWV+mRsG182HyVRK+wVNtMlJupPlScLn605vZWwPE1NJO8adrJVzOlpGRTnllaXISumd3CbNMLtuz7NWkcmK/RH30G+920ryeRgsOgVp7QXg7ANZrd6SmGJa5b6FVnoTfS9mielxsO2MCDDJF7AR1IQXm883aHRQtj+zW6JVud/TpgMlGDZL1zLi8Uhwvrid6zKmPP0hboQ+T6xqyhvrEhRT8igBjTbQ28pm4x7D4dq7ugpp9nj/SomnPN+pRP1QuPZvauWqNUz9yCOlU5C05KdaSO72VopzzLCtuJB1jm7zy6lqnB9XwjouFexISwjcJXFNmeVj6NPO6jxXMUv0qDpNIdfA530DpoIFgkP5dGM91/YV1scu0Enlgj30+GGYOAkTOQ6ZRwY795qv4ibgYMhFv5UhkZ7YtAack2df/zsx3WKcIOWJhizzCSjdXrcxwdm0vOhYOlOejhVkNAc3jyJhZ/VoPmQHdlLqRbudnMPLTveeENPoWkg0G9EW9a6iT4b882yKvE/l/wnPQjyg+AC4u2MIFZtiRKG67Y0TCoLP3Poj600KqxnIyMK4ZzWXe3uKZQE6RS3SusEzwTgKq77+A2QM/esnmPNDKSPda2qZ5o8vEHHX4l012nQs1vJdV3x/N mqO+XRf1 ILwgdUdN3PN+HziTK69yJXsTNZIBZkI3H5vQqthcwj/sL8sDqFvN1+Ucb5gXwD+ok0Tk41/RbdM3gHg8Dpjx0PTb+tLooWEkYVj6LyN4mRyCouXtj+MN+zmslC5v0XEYeYzOrf/ujj3cymQ/6/gj4GW+KLm1O7Ozi9dwZHiUAnGeluN2sVTjbdsp2YqOVZz9qGkXE6Ls1Sgx/4Jl0zD5NPvgBY2GWxsL59J2SMVDmTophH/7JXMWQ1pW8FOJuA4qMHpeTZfGRlxe0SNmKHLRgDozP31LeGEyJGYweTDmR5BIvb1vfWuMgXVg+JOyIMpyBs52SMnRAN/dpWeNKk1u1dYEBmjHdzQIw1E6WE2kk9TvZ8Y8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similar to zap_present_ptes(), let's introduce zap_nonpresent_ptes() to handle non-present ptes, which can improve code readability. No functional change. Signed-off-by: Qi Zheng Reviewed-by: Jann Horn Acked-by: David Hildenbrand --- mm/memory.c | 136 ++++++++++++++++++++++++++++------------------------ 1 file changed, 73 insertions(+), 63 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 209885a4134f7..bd9ebe0f4471f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1587,6 +1587,76 @@ static inline int zap_present_ptes(struct mmu_gather *tlb, return 1; } +static inline int zap_nonpresent_ptes(struct mmu_gather *tlb, + struct vm_area_struct *vma, pte_t *pte, pte_t ptent, + unsigned int max_nr, unsigned long addr, + struct zap_details *details, int *rss) +{ + swp_entry_t entry; + int nr = 1; + + entry = pte_to_swp_entry(ptent); + if (is_device_private_entry(entry) || + is_device_exclusive_entry(entry)) { + struct page *page = pfn_swap_entry_to_page(entry); + struct folio *folio = page_folio(page); + + if (unlikely(!should_zap_folio(details, folio))) + return 1; + /* + * Both device private/exclusive mappings should only + * work with anonymous page so far, so we don't need to + * consider uffd-wp bit when zap. For more information, + * see zap_install_uffd_wp_if_needed(). + */ + WARN_ON_ONCE(!vma_is_anonymous(vma)); + rss[mm_counter(folio)]--; + if (is_device_private_entry(entry)) + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); + } else if (!non_swap_entry(entry)) { + /* Genuine swap entries, hence a private anon pages */ + if (!should_zap_cows(details)) + return 1; + + nr = swap_pte_batch(pte, max_nr, ptent); + rss[MM_SWAPENTS] -= nr; + free_swap_and_cache_nr(entry, nr); + } else if (is_migration_entry(entry)) { + struct folio *folio = pfn_swap_entry_folio(entry); + + if (!should_zap_folio(details, folio)) + return 1; + rss[mm_counter(folio)]--; + } else if (pte_marker_entry_uffd_wp(entry)) { + /* + * For anon: always drop the marker; for file: only + * drop the marker if explicitly requested. + */ + if (!vma_is_anonymous(vma) && !zap_drop_markers(details)) + return 1; + } else if (is_guard_swp_entry(entry)) { + /* + * Ordinary zapping should not remove guard PTE + * markers. Only do so if we should remove PTE markers + * in general. + */ + if (!zap_drop_markers(details)) + return 1; + } else if (is_hwpoison_entry(entry) || is_poisoned_swp_entry(entry)) { + if (!should_zap_cows(details)) + return 1; + } else { + /* We should have covered all the swap entry types */ + pr_alert("unrecognized swap entry 0x%lx\n", entry.val); + WARN_ON_ONCE(1); + } + clear_not_present_full_ptes(vma->vm_mm, addr, pte, nr, tlb->fullmm); + zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, ptent); + + return nr; +} + static unsigned long zap_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, @@ -1598,7 +1668,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, spinlock_t *ptl; pte_t *start_pte; pte_t *pte; - swp_entry_t entry; int nr; tlb_change_page_size(tlb, PAGE_SIZE); @@ -1611,8 +1680,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = ptep_get(pte); - struct folio *folio; - struct page *page; int max_nr; nr = 1; @@ -1622,8 +1689,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (need_resched()) break; + max_nr = (end - addr) / PAGE_SIZE; if (pte_present(ptent)) { - max_nr = (end - addr) / PAGE_SIZE; nr = zap_present_ptes(tlb, vma, pte, ptent, max_nr, addr, details, rss, &force_flush, &force_break); @@ -1631,67 +1698,10 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, addr += nr * PAGE_SIZE; break; } - continue; - } - - entry = pte_to_swp_entry(ptent); - if (is_device_private_entry(entry) || - is_device_exclusive_entry(entry)) { - page = pfn_swap_entry_to_page(entry); - folio = page_folio(page); - if (unlikely(!should_zap_folio(details, folio))) - continue; - /* - * Both device private/exclusive mappings should only - * work with anonymous page so far, so we don't need to - * consider uffd-wp bit when zap. For more information, - * see zap_install_uffd_wp_if_needed(). - */ - WARN_ON_ONCE(!vma_is_anonymous(vma)); - rss[mm_counter(folio)]--; - if (is_device_private_entry(entry)) - folio_remove_rmap_pte(folio, page, vma); - folio_put(folio); - } else if (!non_swap_entry(entry)) { - max_nr = (end - addr) / PAGE_SIZE; - nr = swap_pte_batch(pte, max_nr, ptent); - /* Genuine swap entries, hence a private anon pages */ - if (!should_zap_cows(details)) - continue; - rss[MM_SWAPENTS] -= nr; - free_swap_and_cache_nr(entry, nr); - } else if (is_migration_entry(entry)) { - folio = pfn_swap_entry_folio(entry); - if (!should_zap_folio(details, folio)) - continue; - rss[mm_counter(folio)]--; - } else if (pte_marker_entry_uffd_wp(entry)) { - /* - * For anon: always drop the marker; for file: only - * drop the marker if explicitly requested. - */ - if (!vma_is_anonymous(vma) && - !zap_drop_markers(details)) - continue; - } else if (is_guard_swp_entry(entry)) { - /* - * Ordinary zapping should not remove guard PTE - * markers. Only do so if we should remove PTE markers - * in general. - */ - if (!zap_drop_markers(details)) - continue; - } else if (is_hwpoison_entry(entry) || - is_poisoned_swp_entry(entry)) { - if (!should_zap_cows(details)) - continue; } else { - /* We should have covered all the swap entry types */ - pr_alert("unrecognized swap entry 0x%lx\n", entry.val); - WARN_ON_ONCE(1); + nr = zap_nonpresent_ptes(tlb, vma, pte, ptent, max_nr, + addr, details, rss); } - clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm); - zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, ptent); } while (pte += nr, addr += PAGE_SIZE * nr, addr != end); add_mm_rss_vec(mm, rss);