From patchwork Wed Jan 15 03:38:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13939799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2744EC02180 for ; Wed, 15 Jan 2025 03:39:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B28E0280006; Tue, 14 Jan 2025 22:39:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AD890280001; Tue, 14 Jan 2025 22:39:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95359280006; Tue, 14 Jan 2025 22:39:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 75D4E280001 for ; Tue, 14 Jan 2025 22:39:01 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 33A39160E56 for ; Wed, 15 Jan 2025 03:39:01 +0000 (UTC) X-FDA: 83008280082.14.72FCEA5 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf30.hostedemail.com (Postfix) with ESMTP id 3F77C8000A for ; Wed, 15 Jan 2025 03:38:59 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kwumTal9; spf=pass (imf30.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736912339; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9d8u+VrD2ONy4imzNddIFq+dSgSLZFsyXGu9GrVZVD8=; b=2ELA+DaYt/v+XQ7FG9IlJe71YM/nJzuaN020n7zkTK59qsnpKucvVrUZ2LC+0w6rQ07KvA jM0fQF+SftDGCqz4JrLd4mC1B6213ypET+8rT7j+h1tSEEk32lo/olHAZmxXGDK0HzUdBi k+Xxqx6fw4vhcBcEZgArY9L/VgMpaU0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736912339; a=rsa-sha256; cv=none; b=fzzzZvDezffHgtZrAolawr4uljIOaZXwSZd1SvRJDW3fiDUnPEQUC7vdWxRLtIelJOoeH6 3X4hfp657ip3KzjjIYOCAkUuksWDblPv1/g+LgaG4lW2DJElHMVWm1RcuiJn32tEfIoldO J5bxYb8uWpmAWJRaMHQtPfppvPXJsEg= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kwumTal9; spf=pass (imf30.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2ef8c012913so7821525a91.3 for ; Tue, 14 Jan 2025 19:38:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736912338; x=1737517138; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9d8u+VrD2ONy4imzNddIFq+dSgSLZFsyXGu9GrVZVD8=; b=kwumTal94D297ITpe6Gm/2FINrvBn05tr72iOb26fdQCf/9d6RSi9BFtcVZbj666qs kvVM8QJ5FTenENv/k2HO4eydCZWy6wsoZ2rBiuVTvVRtQEKQid+/Cyc4o3/MxpzcoOpg yweSFfrHN1qrgVtLahUaKO8WOXCI5g9t6v6UQIWfb+HV9i65nAycqR/tux1aSueItmm9 iZNQbESk6CMkwIWo06jDG0FxHk62pN5upv9OZ1TgHp9pxvUtWrKxp6KNnmBo9hjYIHlU S6MkG5/EZ5jjhhyysuL1onoKGJ159g25PUkx7JivkTFQMU/2T1syvjYYfng/JrW/SfeF Kufw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736912338; x=1737517138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9d8u+VrD2ONy4imzNddIFq+dSgSLZFsyXGu9GrVZVD8=; b=ErBq1OM6jDceyn9PT1AsdDROFsttrDmZkkdygEG9yQJ9bGlRU+NOF4y9V0ulRs+rgs iAEjxw9loVNtPtqOnfYxARZNJilOk0YnGOum3a8joAw0l1mQC/X4nTQ0RHSGW0GMPi5e pSc7/txp360gjju1imVyGLismAwK/Q2O/KxssUXWVuTl1z7/52zbQaUZA7hDfyYtvRYV PU7DQnD6rXNsJiZOaI7qHMIoP1SM0Jwmpap/LRhyad8cyMoKHPRyJ3/OXBSUGSzd3mpZ S9hUdZ1nYtpqX/KV8ocJKoG2H4urcdmKCgFaUDuEk7rA2IP9yYEZJsT+PS6pOeYo5LZy t0SQ== X-Forwarded-Encrypted: i=1; AJvYcCXdSeVwvKMIjiPGC8bcMlK5QRKDRFxDGb5zLzUJTha3XocvYbdbP5eo22gWVTbzgNCNiJ3Q4CQo8w==@kvack.org X-Gm-Message-State: AOJu0YzXpSjfYgL/tF2SRgOvK/xTNQgcnttmediS1zxDigzMyP2J/CnP 6re6LRUkpRviNkX8VJG4zNI+SUfATxoD6DbPb086Eij/q6/NTv1s X-Gm-Gg: ASbGncu9LJFF2ay2GKllTVtudQQUZ7opMxT/9NSUoNWf2z6YuCg7PkRIc2QDLl8yTco wLsrVtyz2i9y6PF2Ggu4R5qaT7O6k4Q1mGBETqRU4NPW7ErkQ+OXvo2foVFvsyPaBYxXa5rFVn1 4rNeoisS9tRyOfoBpD99FM2+hHNnHIIW1KvfA0VPINeWIl6jz0bocvS4SMbT5tJuZIRM6jT/LQC 3fGRlM1v2pfmAu7DqTRDPZEX3AcemBsji8mBIR0CbFiMJfIQ+SNle3j1KPyrd1YTt0VVyiT67LX w6Xmfwqv X-Google-Smtp-Source: AGHT+IEgu9NCoDqW1LR96/MyezmmdQXkL7jxU3Tb2bYXbRgZkZZNqA2mJFSkzEeWX1OvoHRYK4gZ3A== X-Received: by 2002:a17:90b:2f4e:b0:2ee:f550:3848 with SMTP id 98e67ed59e1d1-2f548e98ea9mr36383031a91.5.1736912337886; Tue, 14 Jan 2025 19:38:57 -0800 (PST) Received: from Barrys-MBP.hub ([2407:7000:af65:8200:e5d5:b870:ca9b:78f8]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f10dffbsm73368195ad.49.2025.01.14.19.38.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 14 Jan 2025 19:38:57 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: 21cnbao@gmail.com, baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, ioworker0@gmail.com, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, lorenzo.stoakes@oracle.com, ryan.roberts@arm.com, v-songbaohua@oppo.com, x86@kernel.org, ying.huang@intel.com, zhengtangquan@oppo.com Subject: [PATCH v3 4/4] mm: Avoid splitting pmd for lazyfree pmd-mapped THP in try_to_unmap Date: Wed, 15 Jan 2025 16:38:08 +1300 Message-Id: <20250115033808.40641-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250115033808.40641-1-21cnbao@gmail.com> References: <20250115033808.40641-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 3F77C8000A X-Stat-Signature: 8ptt7doxidzzfamak48tafx371ws7w6e X-Rspam-User: X-HE-Tag: 1736912339-244557 X-HE-Meta: U2FsdGVkX18lcAGJ1S44XQQBrY5JoTKXEYmydF+I7ySG5rUPi0hMJLJjimNT1eGCALYycqTmPdOyRb4exKuyx95nRIQpcVRY9zGZUXplYhEVDmf3qztySOZEWGP3ehx485xUpIxYUObfeA39uYWIZi1iDDiW5FCzqZ9PqxJa7iTIkrsq3fKzD9fhJKtMmDeL8mUXI5xHkm+T8Vd6+F7qaj8h4Xkc21kQt7ReYTB+JrIUQLUPY3727xLTHMhugs4RewZ0SXWuwMQTKMTpPAZF8CNUmohfDt/jVW1rEc+cld/aeZxAEkTJP2iBqDlaESz1bbc4DJ+RbdpSmbKSAqtI0+PLon2NqFqsWcDQIQzmeXppUPIaH4Hn/46OWrKGjskKC6YKU8JlNXNjV19CZ93PBhyKGAz72nGITTmQeBuf3Qd0Y/lFb4d5m7Le9yIacrhqK6NcNY1b6mjSv03vfv0OAqTl7SI9WdPRT1YgAMWvnza87jiqAegVs2i0FRYt96GpCEi5dxbrczxAiuoR4HhlsY+6xOqKO7Ro5/xAWuajh+VG5crWsnFRqHOA1eRlYEVdu2W0TmAEb9GROzOQqglKUsDgetMJ2JTDq+uppdglBVpSAfgEEWf92ZuGIofaBwhGOFjAyRu2SZkii3b9FT+g8mZl3dfxBHTyUqXBqbYpezaEuXU/TDBsWcCo2LG4nR11To0IDPoNRJFlgU9g5rJf3IIgIjy9fFo3xUJdNMj+TlLqo9aUP+2oQcdTZP0wMtQCT+yay+PyqPLwJEpeeVBdF/gMkk6ZfXKYuuEgLgrX7CjGWNmKrK7Cf+TJuYDZ63d8AOhd4CYuDLUI1eBORp68z/AmgXBJmYosl9PPFNtislaYIhPx89hd/DzaxgYVaI61oAF9etx0JJKKGxMhFMh1aw/cDtGN7co5FW7kSIZojzVAuR8j49ToPEPhCrvIo/Rc4SQYZWvJrpGOMu+r93i 426is9wx lV+pOmB8ujcEP5LTWbzW/+2thmm03H7ts5T/mbb1LPR7tB1pqVyzhGvMzxwvCYhdodyz7vzkQpwUe3LnbIzVduEQW37wEHypJzYffIcYwmAoLzLHv95VAgqzZ/uSqQZ+JHkp4Te1+23Y7XKLqeY35f9SBkAYW2X53RCZDv9TpGrkDFiyjcdXcMRsV5fDboDm2b5rCP6iMzjQCY6DJFhN2tN4fgASinUNtJk32Z6p+PekHcRjG4pQzWbg3N8/0G2DEdfLDKccn4gMJZ3Slp/Lcr282Oioe66pTpqLuKRkQQe3L2nD3at1yUKXvR7C60kjoXzP/S7k5fxjX0RYtSEeY6vJPMkPEXOE2x36LymH8h9xb07V0law2acu7CRnFTZifMfdLCiQpwuruBzjelhmxIGAR/JVg92iL8oVvs/Gu4+98LUeW0PwzE2biETlOyIkd8RvvmIWQhbRcDsk+6tiFBLygOcKFIbov9lQKzG7+1I3PlrNwB7B36rBRxy4mjNo7Nkk17/jsueKQ6rM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song The try_to_unmap_one() function currently handles PMD-mapped THPs inefficiently. It first splits the PMD into PTEs, copies the dirty state from the PMD to the PTEs, iterates over the PTEs to locate the dirty state, and then marks the THP as swap-backed. This process involves unnecessary PMD splitting and redundant iteration. Instead, this functionality can be efficiently managed in __discard_anon_folio_pmd_locked(), avoiding the extra steps and improving performance. The following microbenchmark redirties folios after invoking MADV_FREE, then measures the time taken to perform memory reclamation (actually set those folios swapbacked again) on the redirtied folios. #include #include #include #include #define SIZE 128*1024*1024 // 128 MB int main(int argc, char *argv[]) { while(1) { volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); memset((void *)p, 1, SIZE); madvise((void *)p, SIZE, MADV_FREE); /* redirty after MADV_FREE */ memset((void *)p, 1, SIZE); clock_t start_time = clock(); madvise((void *)p, SIZE, MADV_PAGEOUT); clock_t end_time = clock(); double elapsed_time = (double)(end_time - start_time) / CLOCKS_PER_SEC; printf("Time taken by reclamation: %f seconds\n", elapsed_time); munmap((void *)p, SIZE); } return 0; } Testing results are as below, w/o patch: ~ # ./a.out Time taken by reclamation: 0.007300 seconds Time taken by reclamation: 0.007226 seconds Time taken by reclamation: 0.007295 seconds Time taken by reclamation: 0.007731 seconds Time taken by reclamation: 0.007134 seconds Time taken by reclamation: 0.007285 seconds Time taken by reclamation: 0.007720 seconds Time taken by reclamation: 0.007128 seconds Time taken by reclamation: 0.007710 seconds Time taken by reclamation: 0.007712 seconds Time taken by reclamation: 0.007236 seconds Time taken by reclamation: 0.007690 seconds Time taken by reclamation: 0.007174 seconds Time taken by reclamation: 0.007670 seconds Time taken by reclamation: 0.007169 seconds Time taken by reclamation: 0.007305 seconds Time taken by reclamation: 0.007432 seconds Time taken by reclamation: 0.007158 seconds Time taken by reclamation: 0.007133 seconds … w/ patch ~ # ./a.out Time taken by reclamation: 0.002124 seconds Time taken by reclamation: 0.002116 seconds Time taken by reclamation: 0.002150 seconds Time taken by reclamation: 0.002261 seconds Time taken by reclamation: 0.002137 seconds Time taken by reclamation: 0.002173 seconds Time taken by reclamation: 0.002063 seconds Time taken by reclamation: 0.002088 seconds Time taken by reclamation: 0.002169 seconds Time taken by reclamation: 0.002124 seconds Time taken by reclamation: 0.002111 seconds Time taken by reclamation: 0.002224 seconds Time taken by reclamation: 0.002297 seconds Time taken by reclamation: 0.002260 seconds Time taken by reclamation: 0.002246 seconds Time taken by reclamation: 0.002272 seconds Time taken by reclamation: 0.002277 seconds Time taken by reclamation: 0.002462 seconds … This patch significantly speeds up try_to_unmap_one() by allowing it to skip redirtied THPs without splitting the PMD. Suggested-by: Baolin Wang Suggested-by: Lance Yang Signed-off-by: Barry Song Reviewed-by: Baolin Wang Reviewed-by: Lance Yang --- mm/huge_memory.c | 24 +++++++++++++++++------- mm/rmap.c | 13 ++++++++++--- 2 files changed, 27 insertions(+), 10 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3d3ebdc002d5..47cc8c3f8f80 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3070,8 +3070,12 @@ static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma, int ref_count, map_count; pmd_t orig_pmd = *pmdp; - if (folio_test_dirty(folio) || pmd_dirty(orig_pmd)) + if (pmd_dirty(orig_pmd)) + folio_set_dirty(folio); + if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) { + folio_set_swapbacked(folio); return false; + } orig_pmd = pmdp_huge_clear_flush(vma, addr, pmdp); @@ -3098,8 +3102,15 @@ static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma, * * The only folio refs must be one from isolation plus the rmap(s). */ - if (folio_test_dirty(folio) || pmd_dirty(orig_pmd) || - ref_count != map_count + 1) { + if (pmd_dirty(orig_pmd)) + folio_set_dirty(folio); + if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) { + folio_set_swapbacked(folio); + set_pmd_at(mm, addr, pmdp, orig_pmd); + return false; + } + + if (ref_count != map_count + 1) { set_pmd_at(mm, addr, pmdp, orig_pmd); return false; } @@ -3119,12 +3130,11 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, { VM_WARN_ON_FOLIO(!folio_test_pmd_mappable(folio), folio); VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(folio_test_swapbacked(folio), folio); VM_WARN_ON_ONCE(!IS_ALIGNED(addr, HPAGE_PMD_SIZE)); - if (folio_test_anon(folio) && !folio_test_swapbacked(folio)) - return __discard_anon_folio_pmd_locked(vma, addr, pmdp, folio); - - return false; + return __discard_anon_folio_pmd_locked(vma, addr, pmdp, folio); } static void remap_page(struct folio *folio, unsigned long nr, int flags) diff --git a/mm/rmap.c b/mm/rmap.c index be1978d2712d..a859c399ec7c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1724,9 +1724,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } if (!pvmw.pte) { - if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, - folio)) - goto walk_done; + if (folio_test_anon(folio) && !folio_test_swapbacked(folio)) { + if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, folio)) + goto walk_done; + /* + * unmap_huge_pmd_locked has either already marked + * the folio as swap-backed or decided to retain it + * due to GUP or speculative references. + */ + goto walk_abort; + } if (flags & TTU_SPLIT_HUGE_PMD) { /*