From patchwork Mon Mar 10 17:23:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 14010448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1B90C282DE for ; Mon, 10 Mar 2025 17:23:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC70A280028; Mon, 10 Mar 2025 13:23:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B4D92280026; Mon, 10 Mar 2025 13:23:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 97B9C280028; Mon, 10 Mar 2025 13:23:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 754C9280026 for ; Mon, 10 Mar 2025 13:23:34 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 03A0DC12B9 for ; Mon, 10 Mar 2025 17:23:35 +0000 (UTC) X-FDA: 83206313190.27.4291E18 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf16.hostedemail.com (Postfix) with ESMTP id 55BD018000E for ; Mon, 10 Mar 2025 17:23:32 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=hSx76SP1; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf16.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741627412; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6bHFtBemQj4wFv7o9DpuwYSyeCUOKe2XoHkTMo5mXUs=; b=c9KRvPh2lxNEXjNpAzn5YVF/h6D3ksjl3Uv4E+mY/BYGJjHg3nD6hDPGrOpaLwIc+L8FHM SU7aeQEgqWaeIS5UhKikpqv41GY3YOjGqZCnXf6yEWs4UZQYo6FLRqwyG+TKs2j7BQmPxd 0zpcerdIhaTmKBXM+oUtloCCTSmX3nE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741627412; a=rsa-sha256; cv=none; b=YfWFwAMEAoZ0Cg7IxD8WYCX33+iY5044oNF+NEOp2UZdKMUwCt++8GpbVkhe06JBjyiFqs lvll/d2Xecr8Y4nrBNRUPSJmqTcAmA1SKNv2jMEfrSJFrbwX11KBBWxrI3KHlyRTWMDAWi C9s3MT+JL2JmnhAMU/MjdeGIrW38EZQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=hSx76SP1; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf16.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 3D9E85C14D9; Mon, 10 Mar 2025 17:21:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A5EDEC4CEED; Mon, 10 Mar 2025 17:23:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741627411; bh=RSFUpifN3HuVWoEj944CoZ0VVGpww5B6SbD8iILWaq8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hSx76SP1aMWn5lYQ40SpWdThFZAVcwsocl3ap/sjfMRB7710mfqiSRU6pLK6zibY+ /JVK/IYLx1GfYEXDCgblSfJbiMNgX96eIfr3qx8F+pOU+KqVLcvH3sqjmx4o8rUgS2 8xu2CsWKaUtKwmy6yCSvJLoHKNOkTkLlB25SLTWdsjZ4GzC0OaJhZtNXn798S5yRAE HHwcXczG6UftPqSdNetvITTqiLoPeMA3WDlTu4ynL3Wkd7HhCMVxVOvpYEGTmTkSf5 Dwo21kDvLQYj+Y2+Ka8/D3qsv0CE7h3+tuFfOS42mpzYM3XRqlPTHO4EyILJQjMfvr oP5hoQkQjgILA== From: SeongJae Park To: Andrew Morton Cc: SeongJae Park , "Liam R. Howlett" , David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Vlastimil Babka , kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 7/9] mm/madvise: let madvise_{dontneed,free}_single_vma() caller batches tlb flushes Date: Mon, 10 Mar 2025 10:23:16 -0700 Message-Id: <20250310172318.653630-8-sj@kernel.org> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250310172318.653630-1-sj@kernel.org> References: <20250310172318.653630-1-sj@kernel.org> MIME-Version: 1.0 X-Stat-Signature: 6h84irh3sy7nnt31akupms8hoecsp4m7 X-Rspamd-Server: rspam05 X-Rspam-User: X-Rspamd-Queue-Id: 55BD018000E X-HE-Tag: 1741627412-14690 X-HE-Meta: U2FsdGVkX19hXEBj/lPV7xiK3/UtgZU713t+HL5zfhy5mC8mBAYEMvJfQHG69i/G8nP/bVmZxw+KYVKie0OI1mlWh3jLyIjmvONzsAbraHdVytASTGqNZmSUghWNk37oSQXIvIs1vGaZf8AK+UGerntbvS3hhfF0QtZyEfDM2UZVyL87L2GMpgj191aHBfh5GFs66Vezg2oHbK07MEODz+KGzoawaUoE6hmvB3gUc9KtaSEaJ+VNiGR80oK4paEMdceUQzEKQuIM91TSKbNa41MEP8hGZsigVO4vrgCueIrmOtLW4MJa9/yZUGVX5mgOKsbcWixRiuSSQ9EJHYQvlfn0vZiTKsSfVgPlRfrQKTmFZallWN1IdrGV+PICmtc5S+DnBxG7DzjsS8RnYE02+HFLJOmytWDxSvYuMxwCjG2fQCvnAznI1VZ791fOkOqem93w9YdqcJd91//gBrtWsGrKjzJRMV/cOtbSeUYXixS//LbdOfR68WGV6EBQTSv2uc01mSVrE77YmjB4BTdCk+w/znnSJzoCsu3yG79WCI65FYuf6joHUpIUuvqjfw1qSqD2SSiIVlAfKgwVkd+3qZ7ZsFjpjRLCkE1Dd+XvtgzVc75q5Fro4aMf26IOoyCQ6UdY3vv33vYO4BcpdE1XtHl0Ve8LQMu8IVqIw7Om2LSxJb46JPr0FBgwUxbRR7KtEkPAk7lFCG1496zoc7fe+NtQ1vGz4tQ8ncxnTOBHvzXdBjxKn+fVAjiE+mFT55u648QTiFMb60AFKUJTsspkMRPJ8HxUjprU4YnJ2vP+UXYjrPY/k8vm+fzxJyd2s98dJblfCJNIzf8+QxZ6mXg4UK1y7JnnZBhJAdoxvYQu+ttRFbQZuFDKwSRzOZY/2IPeiQjcze6WR1xdbqjqIopWn/cdTeO04jQzDO+VCVIHIR95s4ZjrP1MLLy3NCKRcMJolNtQvletP/sZkHVv9sn HrU63VBR xCNn7FFvcP5e9m6TLTDD9fQO0gMV8KgoFPFj5xPD2NQryKq4gwccjlLAI9MJZ6Wu9nVj6XmF9nxHtxtUoNVpRM88tfiBh9x0DktlhsNO5RYmYGtRPgoXJon+1s56xzM3J/sWk9urTM656R2iS5oyZWHzBVeq9fbIDt+9dGTllse/rAKlgsvCxepa/fl+EIQ/gLKY6hxC8ZcZa0nx3lFLaRKG+6rJEGuGZFw6gdAx4u8PXy/ymJyj9JQlFCZ4UzmE5OGLZCFmVSiOfqAkhdDZjVnPK2GalZY9x0+FF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Update madvise_dontneed_single_vma() and madvise_free_single_vma() functions so that the caller can pass an mmu_gather object that should be initialized and will be finished outside, for batched tlb flushes. Also modify their internal code to support such usage by skipping the initialization and finishing of self-allocated mmu_gather object if it received a valid mmu_gather object. Signed-off-by: SeongJae Park --- mm/internal.h | 3 +++ mm/madvise.c | 37 +++++++++++++++++++++++++------------ mm/memory.c | 16 +++++++++++++--- 3 files changed, 41 insertions(+), 15 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 0caa64dc2cb7..ce7fb2383f65 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -438,6 +438,9 @@ void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, struct zap_details *details); +void unmap_vma_single(struct mmu_gather *tlb, struct vm_area_struct *vma, + unsigned long addr, unsigned long size, + struct zap_details *details); int folio_unmap_invalidate(struct address_space *mapping, struct folio *folio, gfp_t gfp); diff --git a/mm/madvise.c b/mm/madvise.c index ba2a78795207..d7ea71c6422c 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -794,12 +794,19 @@ static const struct mm_walk_ops madvise_free_walk_ops = { .walk_lock = PGWALK_RDLOCK, }; -static int madvise_free_single_vma(struct vm_area_struct *vma, - unsigned long start_addr, unsigned long end_addr) +static int madvise_free_single_vma( + struct mmu_gather *caller_tlb, struct vm_area_struct *vma, + unsigned long start_addr, unsigned long end_addr) { struct mm_struct *mm = vma->vm_mm; struct mmu_notifier_range range; - struct mmu_gather tlb; + struct mmu_gather self_tlb; + struct mmu_gather *tlb; + + if (caller_tlb) + tlb = caller_tlb; + else + tlb = &self_tlb; /* MADV_FREE works for only anon vma at the moment */ if (!vma_is_anonymous(vma)) @@ -815,16 +822,18 @@ static int madvise_free_single_vma(struct vm_area_struct *vma, range.start, range.end); lru_add_drain(); - tlb_gather_mmu(&tlb, mm); + if (!caller_tlb) + tlb_gather_mmu(tlb, mm); update_hiwater_rss(mm); mmu_notifier_invalidate_range_start(&range); - tlb_start_vma(&tlb, vma); + tlb_start_vma(tlb, vma); walk_page_range(vma->vm_mm, range.start, range.end, - &madvise_free_walk_ops, &tlb); - tlb_end_vma(&tlb, vma); + &madvise_free_walk_ops, tlb); + tlb_end_vma(tlb, vma); mmu_notifier_invalidate_range_end(&range); - tlb_finish_mmu(&tlb); + if (!caller_tlb) + tlb_finish_mmu(tlb); return 0; } @@ -848,7 +857,8 @@ static int madvise_free_single_vma(struct vm_area_struct *vma, * An interface that causes the system to free clean pages and flush * dirty pages is already available as msync(MS_INVALIDATE). */ -static long madvise_dontneed_single_vma(struct vm_area_struct *vma, +static long madvise_dontneed_single_vma(struct mmu_gather *tlb, + struct vm_area_struct *vma, unsigned long start, unsigned long end) { struct zap_details details = { @@ -856,7 +866,10 @@ static long madvise_dontneed_single_vma(struct vm_area_struct *vma, .even_cows = true, }; - zap_page_range_single(vma, start, end - start, &details); + if (!tlb) + zap_page_range_single(vma, start, end - start, &details); + else + unmap_vma_single(tlb, vma, start, end - start, &details); return 0; } @@ -951,9 +964,9 @@ static long madvise_dontneed_free(struct vm_area_struct *vma, } if (behavior == MADV_DONTNEED || behavior == MADV_DONTNEED_LOCKED) - return madvise_dontneed_single_vma(vma, start, end); + return madvise_dontneed_single_vma(NULL, vma, start, end); else if (behavior == MADV_FREE) - return madvise_free_single_vma(vma, start, end); + return madvise_free_single_vma(NULL, vma, start, end); else return -EINVAL; } diff --git a/mm/memory.c b/mm/memory.c index 88c478e2ed1a..3256b9713cbd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1995,9 +1995,19 @@ void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas, mmu_notifier_invalidate_range_end(&range); } -static void unmap_vma_single(struct mmu_gather *tlb, - struct vm_area_struct *vma, unsigned long address, - unsigned long size, struct zap_details *details) +/** + * unmap_vma_single - remove user pages in a given range + * @tlb: pointer to the caller's struct mmu_gather + * @vma: vm_area_struct holding the applicable pages + * @address: starting address of the pages + * @size: number of bytes to remove + * @details: details of shared cache invalidation + * + * @tlb shouldn't be NULL. The range must fit into one VMA. + */ +void unmap_vma_single(struct mmu_gather *tlb, struct vm_area_struct *vma, + unsigned long address, unsigned long size, + struct zap_details *details) { const unsigned long end = address + size; struct mmu_notifier_range range;