From patchwork Wed Apr 17 14:14:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13633429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 826E0C04FFF for ; Wed, 17 Apr 2024 14:15:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 05DBA6B0093; Wed, 17 Apr 2024 10:15:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0345F6B0095; Wed, 17 Apr 2024 10:15:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3E9E6B0098; Wed, 17 Apr 2024 10:15:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C61796B0093 for ; Wed, 17 Apr 2024 10:15:18 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 86FB2A0713 for ; Wed, 17 Apr 2024 14:15:18 +0000 (UTC) X-FDA: 82019221116.12.8F959D8 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf19.hostedemail.com (Postfix) with ESMTP id 887001A001E for ; Wed, 17 Apr 2024 14:15:16 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=K7tEYR0J; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713363316; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8zmDsYAtJOCrhdAwJbAGzI50/YgT/UfiWJqJuE4bljE=; b=KkLKku/A1+glUqSnq1CJsSkDNcmPlKf7P2pDWyPR4At2Rncw4emhJGS3VcoT9A3/dZ9vlj x/dggRItOLhYec0LTjdQymNuoA413aHT5Y5jn1qTjwY0g9/svRtW2t/OmLdA6ehWCEsAFQ n6DOdFb6t352o0XjPf+swXK7hMMY034= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=K7tEYR0J; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713363316; a=rsa-sha256; cv=none; b=r1PMaxAKnQZeCrmQps/iKUA2oYKWmmmQ4+lNTHbKY+fPPg3m1rumwcDZSwrRu8gqCpbM/X 0csGmGQ4DPEH2HdRw2iIeNZvdBnjg0/14lVEpaZOzV7fiOfBIh+5v7OOybdpA3l/LZUfQL HLzVEgTOk2kV+a4b0ROlelWEsh9IW/M= Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-6ead4093f85so4751455b3a.3 for ; Wed, 17 Apr 2024 07:15:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713363315; x=1713968115; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8zmDsYAtJOCrhdAwJbAGzI50/YgT/UfiWJqJuE4bljE=; b=K7tEYR0J9qq8YKH/D+M/dZ1cWnr4wAlBceIG1N/GqfBUtmQWfwKHbiHNHvNP00Vn7Q TcQ+AExfMrUbsqFjf0ZSNuYptI4/CabbOW8ZNjfnDmHn/cczV6edyj6CKOvasEoROjrJ XqHaGIX69cQhDIG21h13an2mCvsa6HBsiJAydQgaNzGxscwHKytVeKqJYUICbKInmR07 kN9PQG5Nqo/BTtq8jVDNWRWubgmN7os1DhfiXXqKkkJ+X3VRPxSYNNMw4uMTqwVFeP3X +yz3ZlNJBxvuJXg8HHketqsoOw4MmUp1E73IRIG1f/+cAxc58qkrkp1s96PcxFRo1siX HsYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713363315; x=1713968115; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8zmDsYAtJOCrhdAwJbAGzI50/YgT/UfiWJqJuE4bljE=; b=esnW2LgJj3Tjx8xHHwsfMkrNZagQ0wfsETSTHErZBsT+hq2wmZISUgpXbimVidgRYi zdsK5Qtr9albCsidvTmdGEDUWE/a0lSPc7eMP+GBkjy7HY3N/J597Y3jvnVrum2QwSr3 S7+mSaijLMr0cE94TevjvENzycoX4W28soUmSgVKi5WUAqfISqyPpIVIIYru+q+08UdD tQVKXz2sYf9r9wNAaI+kPWFTD09UTBDh31NiWDpMsRurSZVTO22BVAXr2lWkvkx4b07X h/aqb4Oe9njgoG/feCkMa+rPHmsOHOVzt+VZ7O9TIZo2eDFHxt0YRlHLE3cujY6LIs1T CNJQ== X-Forwarded-Encrypted: i=1; AJvYcCV8OViuZZzt2vr6kNNND6LlebuzZu9HHcWFNDRIohLRp+xzacjWIrIYv+wQh/ziEx0zbV9vbQWbXiDn++LFL65huIw= X-Gm-Message-State: AOJu0YwZcjWrxQwBNsZiUgwGEyWimGWnkVMFellBRV2br/AXvKH78+LZ 35KxS30r2jNiP/RPMgyDMhs4RMi2xnoBSwC82KL7J8pBwfZuZHWh X-Google-Smtp-Source: AGHT+IFHnRpIQz0volgPDyOzixsfBZR9W7TzOuIOwAy8+qIhGQ8COOnMGTqpfoWRzRuyD6dpMUBD1w== X-Received: by 2002:a05:6a20:3c9e:b0:1a8:2cd1:e437 with SMTP id b30-20020a056a203c9e00b001a82cd1e437mr17085789pzj.11.1713363315179; Wed, 17 Apr 2024 07:15:15 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id s14-20020a170902a50e00b001ddc83fda95sm11613562plq.186.2024.04.17.07.15.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Apr 2024 07:15:14 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v8 3/3] mm/madvise: optimize lazyfreeing with mTHP in madvise_free Date: Wed, 17 Apr 2024 22:14:36 +0800 Message-Id: <20240417141436.77963-4-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240417141436.77963-1-ioworker0@gmail.com> References: <20240417141436.77963-1-ioworker0@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 887001A001E X-Stat-Signature: wu8coc1c6ycg1sutzx5d69m3ij9hr76t X-HE-Tag: 1713363316-781917 X-HE-Meta: U2FsdGVkX188zT/PQc50A8LIfGZOJ+pmGOlufLf3QDPwTeD3pIWbH/tBX9fxPtVgR9ODARAByp8M9ivxiKVPPUDjHgms6eLUw9cDw18cnNpAHDc7TqMkrc8pmYOrYcoV11Q3527nXVGXha9nF4ozQnALHLfa/kvAQu/kgQnFRLVucxcFRljQ1G3YzgCkRAg2yD4FdrhTq77NRXu1UzOy8rzMa3NIprk9N1kxhBascBrnrHESsj9MoGupohYkD1DGh5SsXvRO72rgwArz4m4YebvzatX8kWUIhIT550NogrsyuzEXNh/j/T0329f479Z2PmhdlePyceMszxGTmB73zgKnGGU902ScaB/qXJ7ZUR7r+cxnI39tii9hIM6KywkBmIqE+K9COue89lpVrq01qILu0cZYYGvyZXIlsrgN+tfk1yiZ8xhTSCbp1UeaR6SrytY/jtY0q+4iLWYVfdk514LrhHqu2UBsu0Ys3o4nSCpLcnmM58A7kLg7m2V7Ib99jJ+uN/1bBzcpPT5IlDL/E9iJnPpdAH7VCN+ExBKOLHO029pp8WvRoJHiPZccNACRojH1DPCNB1lBJwpqCuX2Q1aNlQDA6dgNQYVkq88rWOxK1djb/k6UefLXrjvVOgxoipsfNX9XGuiqjA1J2ORD6LjOkWBDbXqQ/Kn96Q884IJayhvuSJ2mY65F88DIrcNwp87boMUs60uP1ZKcFUYHrOluU38d72bGZWiSUfon139UU+lVle+XVZT/pUZtikKNp7TtNU72rW8eBpFdRx7lRtawS4PNvfwIYbPqWjf3OnnhkQIIwSL2ivsxYkZTYklRC7sJprZcW0e6yYncbroJ3qvB3wHOuHwKQmM1ryv0xMAmVpPauzHNBdO5jH6R9Yw10QJg9SYemalFIb2iNF1oEicMPhgfNIN1GwxEiVIOan2vrYXn9c9vRVUnL0NhK6fwVj7376p2pmb831/V1uN 0B2Xp9qw UZslPFCZ6OOAzHswNI/VMMzncimgmo780L4ZOIlT4QLJZ5IkJdhfqofZ4aYcRWGvAdh5XUnUKqvnhXkhnA+31/lywBj4jXCzlVTaciFtGiQXWD7pD/RwWeoQuHVro3KX8DoIYnhzxHK/kxRPOG/6kfoFpENlcFfO4Pn9xnNGUogzizUlvsgIg0w9dbH5ApwkuXKvUR93WiudDw1SH70q8kU4vNmEuibmbVu47GgVwZNMeBdh+HBhmT9XvQdO6etSzKIevB8h+93hFXLAThfWuKOALBQEK17T5FTZGa4H+A375lXTywtaLUSzKVI1En/P04UQngyrnd2g5x9S4dvhJYiIWzgZ2KRq1Ni6Zr+M1yV3FjYzLlY0YVw1gQ1E5boK5C0aUSm1uzzCPN8PyzcBJ50h6xIdzmGKceqaDMV1M/ea77riivA6Ym+Hik3FsO3d3Pg0lTsrGdSbPRl0wWLYLVGqyqMQDTkyEt/2L+hbeq0d5+iOd26VRj1yAG0qybCRfvUmCRo5kUiQ6Mv9QgKJ8u8ghD/WBlt1n+MlNWNYn6bU9OauRUSkj2MF52qPgin8lOkOSwU6pzDo9pp4ePxzJTN8vFafZsbLosUesHwSe0AJ6u7YbgrFmm8lJNYQqZELmFnuC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch optimizes lazyfreeing with PTE-mapped mTHP[1] (Inspired by David Hildenbrand[2]). We aim to avoid unnecessary folio splitting if the large folio is fully mapped within the target range. If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities. On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better): Folio Size | Old | New | Change ------------------------------------------ 4KiB | 0.590251 | 0.590259 | 0% 16KiB | 2.990447 | 0.185655 | -94% 32KiB | 2.547831 | 0.104870 | -95% 64KiB | 2.457796 | 0.052812 | -97% 128KiB | 2.281034 | 0.032777 | -99% 256KiB | 2.230387 | 0.017496 | -99% 512KiB | 2.189106 | 0.010781 | -99% 1024KiB | 2.183949 | 0.007753 | -99% 2048KiB | 0.002799 | 0.002804 | 0% [1] https://lkml.kernel.org/r/20231207161211.2374093-5-ryan.roberts@arm.com [2] https://lore.kernel.org/linux-mm/20240214204435.167852-1-david@redhat.com Signed-off-by: Lance Yang Reviewed-by: Ryan Roberts --- mm/internal.h | 12 ++++- mm/madvise.c | 141 ++++++++++++++++++++++++++++---------------------- mm/memory.c | 4 +- 3 files changed, 91 insertions(+), 66 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index c6483f73ec13..daa59cef85d7 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -134,6 +134,8 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) * first one is writable. * @any_young: Optional pointer to indicate whether any entry except the * first one is young. + * @any_dirty: Optional pointer to indicate whether any entry except the + * first one is dirty. * * Detect a PTE batch: consecutive (present) PTEs that map consecutive * pages of the same large folio. @@ -149,18 +151,20 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) */ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags, - bool *any_writable, bool *any_young) + bool *any_writable, bool *any_young, bool *any_dirty) { unsigned long folio_end_pfn = folio_pfn(folio) + folio_nr_pages(folio); const pte_t *end_ptep = start_ptep + max_nr; pte_t expected_pte, *ptep; - bool writable, young; + bool writable, young, dirty; int nr; if (any_writable) *any_writable = false; if (any_young) *any_young = false; + if (any_dirty) + *any_dirty = false; VM_WARN_ON_FOLIO(!pte_present(pte), folio); VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio); @@ -176,6 +180,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, writable = !!pte_write(pte); if (any_young) young = !!pte_young(pte); + if (any_dirty) + dirty = !!pte_dirty(pte); pte = __pte_batch_clear_ignored(pte, flags); if (!pte_same(pte, expected_pte)) @@ -193,6 +199,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, *any_writable |= writable; if (any_young) *any_young |= young; + if (any_dirty) + *any_dirty |= dirty; nr = pte_batch_hint(ptep, pte); expected_pte = pte_advance_pfn(expected_pte, nr); diff --git a/mm/madvise.c b/mm/madvise.c index f5e3699e7b54..d6f1889d6308 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -321,6 +321,39 @@ static inline bool can_do_file_pageout(struct vm_area_struct *vma) file_permission(vma->vm_file, MAY_WRITE) == 0; } +static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end, + struct folio *folio, pte_t *ptep, + pte_t pte, bool *any_young, + bool *any_dirty) +{ + int max_nr = (end - addr) / PAGE_SIZE; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; + + return folio_pte_batch(folio, addr, ptep, pte, max_nr, fpb_flags, NULL, + any_young, any_dirty); +} + +static inline bool madvise_pte_split_folio(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, + struct folio *folio, pte_t **pte, + spinlock_t **ptl) +{ + int err; + + if (!folio_trylock(folio)) + return false; + + folio_get(folio); + pte_unmap_unlock(*pte, *ptl); + err = split_folio(folio); + folio_unlock(folio); + folio_put(folio); + + *pte = pte_offset_map_lock(mm, pmd, addr, ptl); + + return err == 0; +} + static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -456,41 +489,30 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, * next pte in the range. */ if (folio_test_large(folio)) { - const fpb_t fpb_flags = FPB_IGNORE_DIRTY | - FPB_IGNORE_SOFT_DIRTY; - int max_nr = (end - addr) / PAGE_SIZE; bool any_young; - nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, - fpb_flags, NULL, &any_young); - if (any_young) - ptent = pte_mkyoung(ptent); + nr = madvise_folio_pte_batch(addr, end, folio, pte, + ptent, &any_young, NULL); if (nr < folio_nr_pages(folio)) { - int err; - if (folio_likely_mapped_shared(folio)) continue; if (pageout_anon_only_filter && !folio_test_anon(folio)) continue; - if (!folio_trylock(folio)) - continue; - folio_get(folio); + arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(start_pte, ptl); - start_pte = NULL; - err = split_folio(folio); - folio_unlock(folio); - folio_put(folio); - start_pte = pte = - pte_offset_map_lock(mm, pmd, addr, &ptl); + if (madvise_pte_split_folio(mm, pmd, addr, + folio, &start_pte, &ptl)) + nr = 0; if (!start_pte) break; + pte = start_pte; arch_enter_lazy_mmu_mode(); - if (!err) - nr = 0; continue; } + + if (any_young) + ptent = pte_mkyoung(ptent); } /* @@ -688,44 +710,48 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, continue; /* - * If pmd isn't transhuge but the folio is large and - * is owned by only this process, split it and - * deactivate all pages. + * If we encounter a large folio, only split it if it is not + * fully mapped within the range we are operating on. Otherwise + * leave it as is so that it can be marked as lazyfree. If we + * fail to split a folio, leave it in place and advance to the + * next pte in the range. */ if (folio_test_large(folio)) { - int err; + bool any_young, any_dirty; - if (folio_likely_mapped_shared(folio)) - break; - if (!folio_trylock(folio)) - break; - folio_get(folio); - arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(start_pte, ptl); - start_pte = NULL; - err = split_folio(folio); - folio_unlock(folio); - folio_put(folio); - if (err) - break; - start_pte = pte = - pte_offset_map_lock(mm, pmd, addr, &ptl); - if (!start_pte) - break; - arch_enter_lazy_mmu_mode(); - pte--; - addr -= PAGE_SIZE; - continue; + nr = madvise_folio_pte_batch(addr, end, folio, pte, + ptent, &any_young, &any_dirty); + + if (nr < folio_nr_pages(folio)) { + if (folio_likely_mapped_shared(folio)) + continue; + + arch_leave_lazy_mmu_mode(); + if (madvise_pte_split_folio(mm, pmd, addr, + folio, &start_pte, &ptl)) + nr = 0; + if (!start_pte) + break; + pte = start_pte; + arch_enter_lazy_mmu_mode(); + continue; + } + + if (any_young) + ptent = pte_mkyoung(ptent); + if (any_dirty) + ptent = pte_mkdirty(ptent); } if (folio_test_swapcache(folio) || folio_test_dirty(folio)) { if (!folio_trylock(folio)) continue; /* - * If folio is shared with others, we mustn't clear - * the folio's dirty flag. + * If we have a large folio at this point, we know it is + * fully mapped so if its mapcount is the same as its + * number of pages, it must be exclusive. */ - if (folio_mapcount(folio) != 1) { + if (folio_mapcount(folio) != folio_nr_pages(folio)) { folio_unlock(folio); continue; } @@ -741,19 +767,10 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, } if (pte_young(ptent) || pte_dirty(ptent)) { - /* - * Some of architecture(ex, PPC) don't update TLB - * with set_pte_at and tlb_remove_tlb_entry so for - * the portability, remap the pte with old|clean - * after pte clearing. - */ - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - - ptent = pte_mkold(ptent); - ptent = pte_mkclean(ptent); - set_pte_at(mm, addr, pte, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + clear_young_dirty_ptes(vma, addr, pte, nr, + CYDP_CLEAR_YOUNG | + CYDP_CLEAR_DIRTY); + tlb_remove_tlb_entries(tlb, pte, nr, addr); } folio_mark_lazyfree(folio); } diff --git a/mm/memory.c b/mm/memory.c index 33d87b64d15d..9e07d1b9020c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -989,7 +989,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma flags |= FPB_IGNORE_SOFT_DIRTY; nr = folio_pte_batch(folio, addr, src_pte, pte, max_nr, flags, - &any_writable, NULL); + &any_writable, NULL, NULL); folio_ref_add(folio, nr); if (folio_test_anon(folio)) { if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page, @@ -1558,7 +1558,7 @@ static inline int zap_present_ptes(struct mmu_gather *tlb, */ if (unlikely(folio_test_large(folio) && max_nr != 1)) { nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, - NULL, NULL); + NULL, NULL, NULL); zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, nr, addr, details, rss, force_flush,