From patchwork Mon Mar 10 17:23:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 14010449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6D77C28B2E for ; Mon, 10 Mar 2025 17:23:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2375280029; Mon, 10 Mar 2025 13:23:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7CE1280026; Mon, 10 Mar 2025 13:23:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA8E4280029; Mon, 10 Mar 2025 13:23:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 85DC2280026 for ; Mon, 10 Mar 2025 13:23:35 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 140BD141343 for ; Mon, 10 Mar 2025 17:23:36 +0000 (UTC) X-FDA: 83206313232.23.26087B1 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf26.hostedemail.com (Postfix) with ESMTP id 67626140004 for ; Mon, 10 Mar 2025 17:23:33 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=krWQwApA; spf=pass (imf26.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741627413; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j/eJqt7TuvyP0SAnStLtA2j7PJ8ZDMjIk9dDwSEf/Lg=; b=ZnZpeErdVC7bGg2u3jHNLzSipXvP5kHhrHnzgfQBwoBqifX+9omEraXXB3HJ0QQ7p95f8U xVrWpOsIIJopAdFK1lnH2T0mE/mrWDYnaIV/0YIbxtBEP0O97At5G7S4ZJC5Mth8fQfMqx VJqKSK4ZJW6k3kLDGoWgz0+MSs0osCU= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=krWQwApA; spf=pass (imf26.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741627413; a=rsa-sha256; cv=none; b=NoY+iDIA06N+kD3H5xZXzoYP3HJyRLVpAVm8zg92eEtWU/WjrA8GzJCdOQ4EcucG4VKWTD ZTFh17TvKaQe0qCjUU16ZsNgyqF3mlUfnD+xYzRB51lCnpJyl+P3qmv6jhWCXykf4eTnTo e5G7TPrTfMi00QO3w+8AcI//PKVfYOI= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 5296D5C0608; Mon, 10 Mar 2025 17:21:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BC673C4CEEC; Mon, 10 Mar 2025 17:23:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741627412; bh=XwNyPWkuo3x8UEQgYUucQN44sSUMxDXOQT6wzHvLom0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=krWQwApAFooJw8RDR+VQklpBSPp/jb0qvzqLu5pCDpb7eW76XF5JyZwxHg4cy4FEU XvMGRSStteW7kyuxgIk/BsxYaqWeRdLQaa6PTij9bEWwE2+rClOI+uMaCmLkdK2M8K 8MzU+FiOthWHwlK/bdFrs0qShtkdkmaAE2UaZYZM8XMrmJtSnawjt/mLWf6uLnDD0d C+kVoxCRMP9beh4TlBf27mnPlUXccx8TLOwatzKscv0CoCZgrpLWWu7zmxnnpVRCyE Liha99fJhrQLYsMoDyQNiPeJZ4C//wNOTCnt8/6+cNVwvl2dr6Jp1ifIZUVfh+ZNIy g/6Q0LWdHYpWA== From: SeongJae Park To: Andrew Morton Cc: SeongJae Park , "Liam R. Howlett" , David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Vlastimil Babka , kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 8/9] mm/madvise: batch tlb flushes for [process_]madvise(MADV_{DONTNEED[_LOCKED],FREE}) Date: Mon, 10 Mar 2025 10:23:17 -0700 Message-Id: <20250310172318.653630-9-sj@kernel.org> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250310172318.653630-1-sj@kernel.org> References: <20250310172318.653630-1-sj@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 67626140004 X-Stat-Signature: u5zwcr5kcrwj3a8s7uq9krink6qzqjnh X-Rspam-User: X-HE-Tag: 1741627413-489294 X-HE-Meta: U2FsdGVkX1/vrbaEutRL6rTVo9WziShMBg15ADrC6JcQDLJgSWLTZHBKlZSy7D3KkH63NZv+oPPOg/ABBTdluVgEpaNOIQjEnu2XnGyoZmvxxKbbDktIRNgTZX1+28dXnyhV7O8ESTnELCnzNPn0gClx8E32eqOZbjEtpufBNRGmzdr70tzs84LKUNvDlNlqNP1Xtyp2HlS3z5Svav+sz3E4M6PhtlCS9GepUe+qfXGJEB1qEKCzwgbbsg7PvR2UkKJTJmjytUUEwR6/iYDqoQxcMyGLtYBl0dPjVHBAGx1U2Ggp4BSHvmz6FNXlbN5kxGgoYi2wP7bWxsK980oV2ozS8pYGPekmMpa7OYHvtwcmFyacZVeYhbzFIF6UhxW40TOWdSDB4ffk+sgmrg3TEM6NqM3usHzITnC2OdjXBd9g/JhAsvgnKotkWj79S5n5ZI4eC+KDIWk7T2rV+OLz9rLFDDrU5UrtwA1YaCMrB6c0HE6t+hCxNEuK3+thJX+3mAbFLnrsdqO9t327tDphsQAPqlJf/tBeIOO55QU3WY9EinTHDAejgUTNBWHTWalUNXqXti1eM2uloux2KNMDqjJ4QSd59A8g1lUmjEGkbTzppK0CVKVzP3zEfGx/iGYVerCXzl3SaUEqPB6VRuPOyzFv30s/JifDwUajzoLcivwwgDF2y9xKx5G4bEj53hEZXtFI1dYb4aoHL1YLgVBkNJG5G6XK4dQDz9E2HR0vnbCKXwcEzlJ0sX1DWG+CEYhnit9rwjW788Ne9vOD/yI1ngIWIOrlepkRMPCKS13clwskaEvwFI3j0j5zBa6sYPCpxsAgzQsrf2SB6ia3RlxgoWtWZYN8yajrLAJtiw2kgGMQvrmVS44cUUEhffnThmjxTArykYzLxUnbVUaDt+ubg3RTANjB4iNQ4TwgKey4hY2upCYsqk22aCK4nt3Udlp+A7rfXqSxL2EH5XzCVOK BJLRaJyE V9yiNP4KRKS46zQhzi6AgQeH81WoQ+sZRWYGfaiyeAGoWylruuDppm/MVWQNe/lOmR/6Z/gr2cQcwLC2gvCnnVudqi3R3veVyJD8OtdN+7WXbTWHCDzHuJM4za6L0w/ESaZbU08JD6GMmgtkg+FKEpzEIQr8Ced5Nw6lq3O6GeC37jgbIsM6HJJFLU+PuWoR4cEokuedT73amNyr4+B0iMKjhlgWWpOxUF8at/2nkQGuN+94aEFe/7snvXv8wlDAa0KDabt5SpPGCoX9S/YTigB/UpVqLZNKWuo/v X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: MADV_DONTNEED[_LOCKED] and MADV_FREE internal logics for [process_]madvise() can be invoked with batched tlb flushes. Update vector_madvise() and do_madvise(), which are called for the two system calls respectively, to use those in the efficient way. Initialize an mmu_gather object before starting the internal works, and flush the gathered tlb entries at once after all the internal works are done. Signed-off-by: SeongJae Park --- mm/madvise.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 47 insertions(+), 4 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index d7ea71c6422c..d5f4ce3041a4 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -905,6 +905,7 @@ static bool madvise_dontneed_free_valid_vma(struct vm_area_struct *vma, struct madvise_behavior { int behavior; + struct mmu_gather *tlb; }; static long madvise_dontneed_free(struct vm_area_struct *vma, @@ -964,9 +965,11 @@ static long madvise_dontneed_free(struct vm_area_struct *vma, } if (behavior == MADV_DONTNEED || behavior == MADV_DONTNEED_LOCKED) - return madvise_dontneed_single_vma(NULL, vma, start, end); + return madvise_dontneed_single_vma( + madv_behavior->tlb, vma, start, end); else if (behavior == MADV_FREE) - return madvise_free_single_vma(NULL, vma, start, end); + return madvise_free_single_vma( + madv_behavior->tlb, vma, start, end); else return -EINVAL; } @@ -1639,6 +1642,32 @@ static void madvise_unlock(struct mm_struct *mm, int behavior) mmap_read_unlock(mm); } +static bool madvise_batch_tlb_flush(int behavior) +{ + switch (behavior) { + case MADV_DONTNEED: + case MADV_DONTNEED_LOCKED: + return true; + default: + return false; + } +} + +static void madvise_init_tlb(struct madvise_behavior *madv_behavior, + struct mm_struct *mm) +{ + if (!madvise_batch_tlb_flush(madv_behavior->behavior)) + return; + tlb_gather_mmu(madv_behavior->tlb, mm); +} + +static void madvise_finish_tlb(struct madvise_behavior *madv_behavior) +{ + if (!madvise_batch_tlb_flush(madv_behavior->behavior)) + return; + tlb_finish_mmu(madv_behavior->tlb); +} + static bool is_valid_madvise(unsigned long start, size_t len_in, int behavior) { size_t len; @@ -1791,14 +1820,20 @@ static int madvise_do_behavior(struct mm_struct *mm, int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior) { int error; - struct madvise_behavior madv_behavior = {.behavior = behavior}; + struct mmu_gather tlb; + struct madvise_behavior madv_behavior = { + .behavior = behavior, + .tlb = &tlb, + }; if (madvise_should_skip(start, len_in, behavior, &error)) return error; error = madvise_lock(mm, behavior); if (error) return error; + madvise_init_tlb(&madv_behavior, mm); error = madvise_do_behavior(mm, start, len_in, &madv_behavior); + madvise_finish_tlb(&madv_behavior); madvise_unlock(mm, behavior); return error; @@ -1815,13 +1850,18 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, { ssize_t ret = 0; size_t total_len; - struct madvise_behavior madv_behavior = {.behavior = behavior}; + struct mmu_gather tlb; + struct madvise_behavior madv_behavior = { + .behavior = behavior, + .tlb = &tlb, + }; total_len = iov_iter_count(iter); ret = madvise_lock(mm, behavior); if (ret) return ret; + madvise_init_tlb(&madv_behavior, mm); while (iov_iter_count(iter)) { unsigned long start = (unsigned long)iter_iov_addr(iter); @@ -1850,14 +1890,17 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, } /* Drop and reacquire lock to unwind race. */ + madvise_finish_tlb(&madv_behavior); madvise_unlock(mm, behavior); madvise_lock(mm, behavior); + madvise_init_tlb(&madv_behavior, mm); continue; } if (ret < 0) break; iov_iter_advance(iter, iter_iov_len(iter)); } + madvise_finish_tlb(&madv_behavior); madvise_unlock(mm, behavior); ret = (total_len - iov_iter_count(iter)) ? : ret;