From patchwork Fri Feb 14 09:30:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13974668 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12A9CC02198 for ; Fri, 14 Feb 2025 09:31:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97EA86B0083; Fri, 14 Feb 2025 04:31:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 906906B0096; Fri, 14 Feb 2025 04:31:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 732516B0098; Fri, 14 Feb 2025 04:31:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5584C6B0083 for ; Fri, 14 Feb 2025 04:31:05 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 12B52141D6E for ; Fri, 14 Feb 2025 09:31:05 +0000 (UTC) X-FDA: 83118031290.26.B9AAEE1 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf20.hostedemail.com (Postfix) with ESMTP id 08A531C001D for ; Fri, 14 Feb 2025 09:31:02 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ps3D4MFW; spf=pass (imf20.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739525463; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OYdDqavtq9jUGmr9BTt10VNZtJWCJbWghGFQiFP0UEc=; b=eLg7lbJeRJ+4r8kCxHmOegdoUh8ITH+YmqLuNFqTPM9e/6gD22NJ4ILM/3ciM1KE9mP+vE 7R5++Dxsn8YnaYIQ+3R9F5O2OavtPwrE9OZlxmZ1hh10I3IstPSS+PIt5vC4xpWFVfvcCF MKY8GWONnfGD19VHiZt0syaYAyGMCvY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ps3D4MFW; spf=pass (imf20.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739525463; a=rsa-sha256; cv=none; b=twK/cbJWLkPboRIVEqut3QPO4ce7cRYodn3n4rxl/gKf1PXE8fbLtmFiTfWcBHvA3Zl9Sd RWAlV7wAXgHt9KCBf0I0B72TRdaldTtM0tHa+WXCTC/wkwKcYuMs50mK/nKAQD2VQZpauZ 3+zHBiDJFX7MaTU8dNgZiC6cBmHdQ8g= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-21f6a47d617so30942605ad.2 for ; Fri, 14 Feb 2025 01:31:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739525462; x=1740130262; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OYdDqavtq9jUGmr9BTt10VNZtJWCJbWghGFQiFP0UEc=; b=Ps3D4MFW1DV5u6ACBIr/mSC+9IpBYB4E2DfAB5cSRxFBoqUnrLREC9zqMdwaF8ed0V vR5fnCltypUufbgWvqT81R9Qjz+lyK8i6St0Vzb2o+t3mD9BMdLf34WqBqV0bu8rIEXL vvMrjlfK4cEM8hqFLGfMXY2TpUDgX3Mc16lecR3h3HprdRVesJ6u89feGMecyWI0dsAf UNF5rRAOa7qT/Y6eVT+cZnxYp+SJyvS4F7N+uvPIlPl0yNFqVSeZRt6JCHI3Hdl9ds1d gJii0yWD3lZ+J8cLcKvkPEayRyQLWA9y9ePq8v3oRDmt73Lx0fJExwiXYEGkiHBZ5DjZ tw8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739525462; x=1740130262; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OYdDqavtq9jUGmr9BTt10VNZtJWCJbWghGFQiFP0UEc=; b=HZMrSxXa8aHjEektcvsyJ1JM63ea2kK0UI7ID/OIdP2kfzENXhZpeZccyYaQd3qwjf jRasH05P48wcDKW0O9Pj6E5Qr2tQB8H2lJ3lo9fWYzSODuYf3dif+JPbBP3RZlzCnVmt ntYa0jErUchwzw8ZWcqbm6TbcLMd18z1gUEImIarlNAmEJU+vEMqvQyB58DS1rN0Ww/p 3mNIR9Mqeook54L2ht5y0uUJQco429uvULijrDuJth5cq5sTMmwAdz9As5qvPGcasdlm zf1UX64Dzv3e51p+Ld4xD7ltNn7qmQg21kTsHdxkr1Vr+Ao+inTdRQ+IGQy41kzPqvM/ g04w== X-Forwarded-Encrypted: i=1; AJvYcCWLEfsC9v1NiKlwOHBZjETLBlJTDDD77tG/LY9TuZT90BXVwZgovBLkJWr01phxQNYgpaUO8HYySQ==@kvack.org X-Gm-Message-State: AOJu0YxzuMpDKHFXcbmD5QQ/GcOOkxodqtBgGORROa6w2hvRA05Vvm/U smr6tOeB7GbzGhN19DyXjFYuKmOQtXPucR/TsMiNhI/a+jOrehUj X-Gm-Gg: ASbGnctkygd6xqNEJUr8RlFBg3uyqmdzqOvNWsQt2RpIPQXV+VuF+GA8Sk2yqrbT248 KVS7X+xhl2jUT8D1pcYLLE6hVm8xrPQFGzCCWOYhMIQLYPepS3axT0oea9+vZj+Am2wJSqeKMBw UBxL/+K46JboQdumce0Y+LDq1rTM/x3A+7tC9xckcLbyuCs3AuMBQscFXztp9acQYjIu3oJ1em/ rL4MJSFNOYJ7Hl8t5oE5RQNnn39ZoemAmQLX/YS9UiGHc1Ur2um6eRnBA2yYgBXPdKVu+gGogEO onKlHsq0Jpk+d5vCDm2bnPa2/xZf8Io= X-Google-Smtp-Source: AGHT+IHG1ACAyCVdNpdCj3QS2lybJkeXxN0YXvZYc6Wbp6cHggvocX10uqRNnZxi4u23qNKsTl+Yhw== X-Received: by 2002:a17:902:e5cc:b0:21f:3e2d:7d58 with SMTP id d9443c01a7336-220bbad65b4mr153804995ad.13.1739525461729; Fri, 14 Feb 2025 01:31:01 -0800 (PST) Received: from Barrys-MBP.hub ([118.92.30.135]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545c814sm25440515ad.148.2025.02.14.01.30.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 14 Feb 2025 01:31:01 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: 21cnbao@gmail.com, baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, ioworker0@gmail.com, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, lorenzo.stoakes@oracle.com, ryan.roberts@arm.com, v-songbaohua@oppo.com, x86@kernel.org, ying.huang@intel.com, zhengtangquan@oppo.com Subject: [PATCH v4 4/4] mm: Avoid splitting pmd for lazyfree pmd-mapped THP in try_to_unmap Date: Fri, 14 Feb 2025 22:30:15 +1300 Message-Id: <20250214093015.51024-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250214093015.51024-1-21cnbao@gmail.com> References: <20250214093015.51024-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 08A531C001D X-Stat-Signature: uf8ai7nswt77h9ku5qarm9j6cgss1mtw X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1739525462-770836 X-HE-Meta: U2FsdGVkX1964TpwYfnMXkYGFuyJoTPezK2Tru/TEWkeP6j+pOPioofrTECx7SL5Cudx+CEoYe8kuvEG0rHkr9cmfvw0Vr6CiNr/F9FRaNlMDq/pGy3USTcBDPacxOzPdgoiqA/e5Ka/nDtXR91a+cW9fiscwT/yZhUgGzuKoexLJ4kOQyz3xqgdPUahERaiUtpGaO0Or1/yKurXvhBz09NEK2obPf3IePEFiO/ikGj/3K8PzgjC91TuFC3OZ8/H7z4Q/+e/s0SlkJwanuzx344R+cvQH+dCcyBlL0O0JwP5zFOMI/bB+cFHKuuJOF1tPPAqwAz0ggmpxyscFOU0mN0Rlf1UhS7KcYqMZv2wvLAyb8TwCRIo+A/7pgWKboI+MEAFhiQC6M7IgRTq4PJvOgdpqqOZJPWX4PhBitekjf3RIh4t9+H2CEPs2HvUCkfvI1qHGak1QEh6D/IjbSNhD9DQ1yzsfL1cqhcCtP1WY/99uFmCJecuhYlNUf2X/iKmgC5KgUinRKDdAZAVQo6i/9pz4NHqdfDWtqXQacKRfCOiS1rgO3opimQJyE9flAbBntDlIYtKKQb55o3a3IQrEtBOrDZrv9owp5RimBWCFTagl4WkFFWtKzIk4tBj2yk5RjG6gxUD55XHj+9zlGZfXkay2Xsf0NhZ5u5ou129ym8yDex+5szR3P/0Thx0sbak4ZZFWjwtzNGwe2x3fY4VoQ5QFwdUA6vBkNi4pUl1ix2vTAkrg5efuloe77N3xKilIZRwWOMmdrL4avWOgkwfU6fAMaY0oWKJNXaPivv+13g+dklRFbSyERbLeOOOW6ijQKOv3hpT8gXIEG8IEsGvXPz8+9IQKJrj9gcTiTp7MaQHPgskZzR8TyM+jaUnngPyh0MSOu8s8RLxdQcT3N5QrxJELJQAJr31AYt304Gv5vq2k4POZzACRhBWgRNAtKqqkIdcWjjnkgOc2tL+AnK 3NUqi5/F jicQmQaVo1OGzqtRQ83bn6vCUU0ZIs3SObZqwGZ33Rtz+6wcU6rfdZu5Wge8B76HD4VpO2P6cfiaUK2t9KLWv8XcJMCYA5YCMb6azwKUEjKTGr5sxgmf+fDmLUpyn4WAxIqMvyewFz4bR8pJSYhfsd1pU8rUOOVFs6yt4ul0iIadEfgIjDnl179a9CXEQJmgUKGgCOmlqet+0oW+Rw8bbU/a6CicYDPzlU011shqLB6eO6V9s1C2i8oftVjGisvjx+F5f0ERsPRgEClMEk7JoIrBKfH2KqXNCgd357CEMO9kBiGBSr1wRh42auN7LhqD9Bgy1DaRjPneQaY002l05eCz257IWd2AJ6wMZQOqNPcEibUppFQ2aKK7UGK/xf9Llxv1rFGwC78KehfIysiAO3TDHL8ArsfO9Jg63GeuhMVdHQk2RiMImmGiVHsIQpbL/3XPFwMIDc6WYN9Wv2lrO4+6mUVbfh1WpAY2vp7WFnUaSxqSCMNSwVlrS3GGLIcIAoJ/i X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song The try_to_unmap_one() function currently handles PMD-mapped THPs inefficiently. It first splits the PMD into PTEs, copies the dirty state from the PMD to the PTEs, iterates over the PTEs to locate the dirty state, and then marks the THP as swap-backed. This process involves unnecessary PMD splitting and redundant iteration. Instead, this functionality can be efficiently managed in __discard_anon_folio_pmd_locked(), avoiding the extra steps and improving performance. The following microbenchmark redirties folios after invoking MADV_FREE, then measures the time taken to perform memory reclamation (actually set those folios swapbacked again) on the redirtied folios. #include #include #include #include #define SIZE 128*1024*1024 // 128 MB int main(int argc, char *argv[]) { while(1) { volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); memset((void *)p, 1, SIZE); madvise((void *)p, SIZE, MADV_FREE); /* redirty after MADV_FREE */ memset((void *)p, 1, SIZE); clock_t start_time = clock(); madvise((void *)p, SIZE, MADV_PAGEOUT); clock_t end_time = clock(); double elapsed_time = (double)(end_time - start_time) / CLOCKS_PER_SEC; printf("Time taken by reclamation: %f seconds\n", elapsed_time); munmap((void *)p, SIZE); } return 0; } Testing results are as below, w/o patch: ~ # ./a.out Time taken by reclamation: 0.007300 seconds Time taken by reclamation: 0.007226 seconds Time taken by reclamation: 0.007295 seconds Time taken by reclamation: 0.007731 seconds Time taken by reclamation: 0.007134 seconds Time taken by reclamation: 0.007285 seconds Time taken by reclamation: 0.007720 seconds Time taken by reclamation: 0.007128 seconds Time taken by reclamation: 0.007710 seconds Time taken by reclamation: 0.007712 seconds Time taken by reclamation: 0.007236 seconds Time taken by reclamation: 0.007690 seconds Time taken by reclamation: 0.007174 seconds Time taken by reclamation: 0.007670 seconds Time taken by reclamation: 0.007169 seconds Time taken by reclamation: 0.007305 seconds Time taken by reclamation: 0.007432 seconds Time taken by reclamation: 0.007158 seconds Time taken by reclamation: 0.007133 seconds … w/ patch ~ # ./a.out Time taken by reclamation: 0.002124 seconds Time taken by reclamation: 0.002116 seconds Time taken by reclamation: 0.002150 seconds Time taken by reclamation: 0.002261 seconds Time taken by reclamation: 0.002137 seconds Time taken by reclamation: 0.002173 seconds Time taken by reclamation: 0.002063 seconds Time taken by reclamation: 0.002088 seconds Time taken by reclamation: 0.002169 seconds Time taken by reclamation: 0.002124 seconds Time taken by reclamation: 0.002111 seconds Time taken by reclamation: 0.002224 seconds Time taken by reclamation: 0.002297 seconds Time taken by reclamation: 0.002260 seconds Time taken by reclamation: 0.002246 seconds Time taken by reclamation: 0.002272 seconds Time taken by reclamation: 0.002277 seconds Time taken by reclamation: 0.002462 seconds … This patch significantly speeds up try_to_unmap_one() by allowing it to skip redirtied THPs without splitting the PMD. Suggested-by: Baolin Wang Suggested-by: Lance Yang Signed-off-by: Barry Song Reviewed-by: Baolin Wang Reviewed-by: Lance Yang --- mm/huge_memory.c | 24 +++++++++++++++++------- mm/rmap.c | 13 ++++++++++--- 2 files changed, 27 insertions(+), 10 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2eda2a9ec8fc..ab80348f33dd 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3176,8 +3176,12 @@ static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma, int ref_count, map_count; pmd_t orig_pmd = *pmdp; - if (folio_test_dirty(folio) || pmd_dirty(orig_pmd)) + if (pmd_dirty(orig_pmd)) + folio_set_dirty(folio); + if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) { + folio_set_swapbacked(folio); return false; + } orig_pmd = pmdp_huge_clear_flush(vma, addr, pmdp); @@ -3204,8 +3208,15 @@ static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma, * * The only folio refs must be one from isolation plus the rmap(s). */ - if (folio_test_dirty(folio) || pmd_dirty(orig_pmd) || - ref_count != map_count + 1) { + if (pmd_dirty(orig_pmd)) + folio_set_dirty(folio); + if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) { + folio_set_swapbacked(folio); + set_pmd_at(mm, addr, pmdp, orig_pmd); + return false; + } + + if (ref_count != map_count + 1) { set_pmd_at(mm, addr, pmdp, orig_pmd); return false; } @@ -3225,12 +3236,11 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, { VM_WARN_ON_FOLIO(!folio_test_pmd_mappable(folio), folio); VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(folio_test_swapbacked(folio), folio); VM_WARN_ON_ONCE(!IS_ALIGNED(addr, HPAGE_PMD_SIZE)); - if (folio_test_anon(folio) && !folio_test_swapbacked(folio)) - return __discard_anon_folio_pmd_locked(vma, addr, pmdp, folio); - - return false; + return __discard_anon_folio_pmd_locked(vma, addr, pmdp, folio); } static void remap_page(struct folio *folio, unsigned long nr, int flags) diff --git a/mm/rmap.c b/mm/rmap.c index 8786704bd466..bcec8677f68d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1863,9 +1863,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } if (!pvmw.pte) { - if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, - folio)) - goto walk_done; + if (folio_test_anon(folio) && !folio_test_swapbacked(folio)) { + if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, folio)) + goto walk_done; + /* + * unmap_huge_pmd_locked has either already marked + * the folio as swap-backed or decided to retain it + * due to GUP or speculative references. + */ + goto walk_abort; + } if (flags & TTU_SPLIT_HUGE_PMD) { /*