From patchwork Tue Apr 16 03:34:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13631163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC74FC4345F for ; Tue, 16 Apr 2024 03:35:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 791DD6B0087; Mon, 15 Apr 2024 23:35:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 740DC6B0088; Mon, 15 Apr 2024 23:35:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E21D6B008A; Mon, 15 Apr 2024 23:35:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3F8386B0087 for ; Mon, 15 Apr 2024 23:35:20 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D8EC014010F for ; Tue, 16 Apr 2024 03:35:19 +0000 (UTC) X-FDA: 82013979558.06.19EFC88 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf24.hostedemail.com (Postfix) with ESMTP id 0E46118001B for ; Tue, 16 Apr 2024 03:35:17 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ZlheVcKA; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713238518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Y26dypB16OA0SVhjpIF+OcTwWxmNUbTnikqr5y0M6Fc=; b=CqvbWZXH37eTwYYHwWnaW/Kp88z3R9NRb7H3KhgEL1d6BxZ18ABMIXP5AAKZCXANIafMYJ 7NEw97Q9KNGOay4ABVxYuZDMg0h3tRN9xjnRIbppNYa516DzYzHuDiuZbGnz17uwBLiWV5 1kkvsxLZOBKXYh5Nevb9/MSpeceIcWc= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ZlheVcKA; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713238518; a=rsa-sha256; cv=none; b=NocuCS5UWqL4DUWuSLViDkW5b8D3cH4e0FqwWb1MmtLga5ktG8ulRAvWE+MwXYzjahcWXt G7wdGlwuKQZa10HehXh2Jh+dP+aaJC/ommGyp6pZ8XqPC8iCb5NBc6K9nhhhOZECXR8K9e aqjVLWtx52bX8Rjt3kDqClj+srRUmx8= Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6ee12766586so2576289b3a.0 for ; Mon, 15 Apr 2024 20:35:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713238517; x=1713843317; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y26dypB16OA0SVhjpIF+OcTwWxmNUbTnikqr5y0M6Fc=; b=ZlheVcKAm1A+VjUpLVtGzhLaiH5mBrxGAOhnd2X0kfmYdUPtH4+TJ+Te69u+LIAZcI LoSzctAZdaV8pZj3Jpc5QHo54x6UZDQtyFyiLu6cUnpl8Kr3fmy+tAu9FIKoctDsnjr2 ttAA/O3s56YTp8YHymJY+2HmUl/JxIiQ11S6k5iekOiDEenKP6HXVhMPTZnyys+tY+2d yQlf+PwCFuIvF+LufC1dGJq74fxA5YpdzppXunvfeUeM62CHfw0fccK9z0PWFaDtvLUj T8nGL+teFuOP1Pfbrtsas4FSI81s4ugv+06uVaiOkeNlK+nmzt1lVWcBRw0Yp9bupOM0 mmbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713238517; x=1713843317; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Y26dypB16OA0SVhjpIF+OcTwWxmNUbTnikqr5y0M6Fc=; b=kBWp7yJ6A28G5EhpLYbXKJIHZ2pvj/jwK8gykdBiBrjBcGcvgC3qkuUBtKLdG3VFaV suMBbsTJtEzWgLEWFX0YMO9DuHzSze0m4v/1fQPTvCWTkeMzIIWmSWtHbI5dS2sDmd3f 6H0i8tQcEPOH97QT0UetJgWpe1Mg/j8bVUkBYkw7rJCXarOO5jssLxVZp7OuemZ8lUCC yxWGfOf9gR//vMz+TRhSkSm3uR678qKHUiLDO1G6fPGAFnJi4kZsyZNTzrIGI9URs3My bCgKyYWT2lkqshwV+kLDqNJn+3PUTCTlHHYOsri09SYhg172z7pQyOAhgQgtqg8MMHYB PQ0w== X-Forwarded-Encrypted: i=1; AJvYcCU/GUcF0P6Csn/iospYSEvp2v9TTvRkhimfdvnt2ooBRpAD9JciE/MROcef6BnaCFJE3MXDHDq80EvZ6iSOaMP5Ijo= X-Gm-Message-State: AOJu0Yxw8HtdEtmg1QZitiZ3E7HIvWdhYld6/eG4sA1IyLCrGie/jEOP N9o9i96kRd3T6/qMvNVzwqREVHjS+iCHS+IOJxNnCay4gRCoQwsn X-Google-Smtp-Source: AGHT+IFyV0ITDY9sKZfPnYg4et6b+fPHhV71ixJ7FoSU2qMU9HkaIvNCBw2dcpvFdL4KgCCgbcP34w== X-Received: by 2002:a05:6a00:2382:b0:6ed:ca65:68b with SMTP id f2-20020a056a00238200b006edca65068bmr2029849pfc.4.1713238516924; Mon, 15 Apr 2024 20:35:16 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id ka13-20020a056a00938d00b006e57247f4e5sm7906737pfb.8.2024.04.15.20.35.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Apr 2024 20:35:16 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v7 1/3] mm/madvise: introduce clear_young_dirty_ptes() batch helper Date: Tue, 16 Apr 2024 11:34:55 +0800 Message-Id: <20240416033457.32154-2-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240416033457.32154-1-ioworker0@gmail.com> References: <20240416033457.32154-1-ioworker0@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0E46118001B X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: jif16dzbgwqqmcsmmphz6qrswbtsdsxy X-HE-Tag: 1713238517-933246 X-HE-Meta: U2FsdGVkX1863UjDWQAvMnIykR3aCOe8MmRWvrDUKnAtf/+sAhiM5vKfX+EB0BfHDwRP89hm8WXTjoVQZHMLcc6K4h56Fx+x5mp4g09g3X/+xi2gzgBufP9Y/pmUoi2KklQ7xMsBQLbnN+MKxI6CHbkFr62C0bPnxWiGdQj9ihhKmqWeA6r1nTd64C0FcnmspCeLCtLTIEs+EsI5PDoQ9V1OTptns83qUero8CXKA9X3qQQr8tbiIsJFwhQLKdaHMJ0ri6tDsEA0D68feqo0+X4qZsmoui6RKzEg2lBQws1aecisM7MF5lpK3yuwZJsu5BSk0a9w8shoGa+GRenPH4Nwp7v79O3pSTbdxZCZgxFkIirCI2whwNYe7krpeFoKehDI7RoNDJ4WAfquciKCEMdD3yWmYK6JozzSE5wTYNQBfD9eSepH+iBPZcxpAgqzRrO+/ylcP04msUcSC6U03rUYV4dUD8y/NfUvvxFyRSn9FzRqyiTmV3E8iqb+/LPX3V3hTHFMhO0EEyMit3aj5EVzkXwsysG6M8lcpHE5NMU1rHauoPCoFovNEhptUWuo7KgunVPb2KkB0nKn4cmw7ejidaFW06WoohM0nRFaOjSGp3k1+Uvn4NF5QVsyG3kkV5PBkYxE8Sqvb4xZXU3I/xbNggQhP/k8nkq3baDpJ9Pu/nNDeviV+Nlc1CqY1/G/B7po1SlRbM93UID6gGfDh383qJ47EsWKtnT1pJ3BOxLUH6FekvWXpQ/XvPMb0UwgudcCSW0/5cZoQijV5RPHBBVAs2Wnu+z8vc5oeqrLtVy2xI9o734eyKc9YxcaVHpg/75LurDijZeRVzLGjXmZljqwR8pG9ikqb299qkQcM24OMUPsLAmvnz0zielkM5+vICOSKbq8dUYHKLDsjbSXbGxmmYgqZsMZLtcm0a2yK0NieNlfH2sc5X5zm2Q0arKYFXTv47nBs6WT4KHoZum 9zJvtY6j z31seb9s9trLSdFFvWzpyzGcw7VDW+BvjrLSqU56zjXVTCgx/T7TcJQcZu4yBNonsrkbdEcYoKugxYBUfnhhB+UHYMnLMLh7lLM9Efx2X7UESK57sng+h2dzRUBgED+xSTiAPrPjWwXewyOdOFuVeS3oBWNEJv+fXNFCeHbgKyGTEzCGcBBn/I4GOcDabjukK7WKh+xicvX+Z8EaJ+eFjqLuW00d/7mmI3ImpY+b/Nk7sOSfXMeelbLovFFCPhksV9yPibGCAkVRbGj86cOdgoppumTYVIgQJqqAV5FN8iDbNlb+8iDaVL4MxrpOck5VembhgLDsSCepDZsPlyJmee7zhi+sYeu5/OAvFbWP0IHDL872mFbAty9YtSq3Cs9urZwj+d1I360Curi2myGp/26yPhDTfFLXY7OaLSHXWWL110A4c3Y+NPwOyJohLowUzaOKc7UoF10GVAx7wb7nwDmEMTe2ifWDyaMZvjug9Glgvxp73VOEhEjrluurTiAwlPADFEb41lmzrErMjKvS/PZ8V+2uDRQPvOrLHnEcYLi0Ocj4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This commit introduces clear_young_dirty_ptes() to replace mkold_ptes(). By doing so, we can use the same function for both use cases (madvise_pageout and madvise_free), and it also provides the flexibility to only clear the dirty flag in the future if needed. Suggested-by: Ryan Roberts Signed-off-by: Lance Yang Reviewed-by: Ryan Roberts --- include/linux/mm_types.h | 9 +++++ include/linux/pgtable.h | 74 ++++++++++++++++++++++++---------------- mm/madvise.c | 3 +- 3 files changed, 55 insertions(+), 31 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c432add95913..28822cd65d2a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1367,6 +1367,15 @@ enum fault_flag { typedef unsigned int __bitwise zap_flags_t; +/* Flags for clear_young_dirty_ptes(). */ +typedef int __bitwise cydp_t; + +/* Clear the access bit */ +#define CYDP_CLEAR_YOUNG ((__force cydp_t)BIT(0)) + +/* Clear the dirty bit */ +#define CYDP_CLEAR_DIRTY ((__force cydp_t)BIT(1)) + /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each * other. Here is what they mean, and how to use them: diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e2f45e22a6d1..18019f037bae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -361,36 +361,6 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, } #endif -#ifndef mkold_ptes -/** - * mkold_ptes - Mark PTEs that map consecutive pages of the same folio as old. - * @vma: VMA the pages are mapped into. - * @addr: Address the first page is mapped at. - * @ptep: Page table pointer for the first entry. - * @nr: Number of entries to mark old. - * - * May be overridden by the architecture; otherwise, implemented as a simple - * loop over ptep_test_and_clear_young(). - * - * Note that PTE bits in the PTE range besides the PFN can differ. For example, - * some PTEs might be write-protected. - * - * Context: The caller holds the page table lock. The PTEs map consecutive - * pages that belong to the same folio. The PTEs are all in the same PMD. - */ -static inline void mkold_ptes(struct vm_area_struct *vma, unsigned long addr, - pte_t *ptep, unsigned int nr) -{ - for (;;) { - ptep_test_and_clear_young(vma, addr, ptep); - if (--nr == 0) - break; - ptep++; - addr += PAGE_SIZE; - } -} -#endif - #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, @@ -489,6 +459,50 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, } #endif +#ifndef clear_young_dirty_ptes +/** + * clear_young_dirty_ptes - Mark PTEs that map consecutive pages of the + * same folio as old/clean. + * @mm: Address space the pages are mapped into. + * @addr: Address the first page is mapped at. + * @ptep: Page table pointer for the first entry. + * @nr: Number of entries to mark old/clean. + * @flags: Flags to modify the PTE batch semantics. + * + * May be overridden by the architecture; otherwise, implemented by + * get_and_clear/modify/set for each pte in the range. + * + * Note that PTE bits in the PTE range besides the PFN can differ. For example, + * some PTEs might be write-protected. + * + * Context: The caller holds the page table lock. The PTEs map consecutive + * pages that belong to the same folio. The PTEs are all in the same PMD. + */ +static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + pte_t pte; + + for (;;) { + if (flags == CYDP_CLEAR_YOUNG) + ptep_test_and_clear_young(vma, addr, ptep); + else { + pte = ptep_get_and_clear(vma->vm_mm, addr, ptep); + if (flags & CYDP_CLEAR_YOUNG) + pte = pte_mkold(pte); + if (flags & CYDP_CLEAR_DIRTY) + pte = pte_mkclean(pte); + set_pte_at(vma->vm_mm, addr, ptep, pte); + } + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} +#endif + static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { diff --git a/mm/madvise.c b/mm/madvise.c index f59169888b8e..edb592adb749 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -507,7 +507,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, continue; if (!pageout && pte_young(ptent)) { - mkold_ptes(vma, addr, pte, nr); + clear_young_dirty_ptes(vma, addr, pte, nr, + CYDP_CLEAR_YOUNG); tlb_remove_tlb_entries(tlb, pte, nr, addr); } From patchwork Tue Apr 16 03:34:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13631164 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31846C4345F for ; Tue, 16 Apr 2024 03:35:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C00DC6B008A; Mon, 15 Apr 2024 23:35:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA1206B0092; Mon, 15 Apr 2024 23:35:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A763F6B0093; Mon, 15 Apr 2024 23:35:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 899F76B008A for ; Mon, 15 Apr 2024 23:35:30 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4DFBF1A08C2 for ; Tue, 16 Apr 2024 03:35:30 +0000 (UTC) X-FDA: 82013980020.02.541C6E8 Received: from mail-oa1-f46.google.com (mail-oa1-f46.google.com [209.85.160.46]) by imf30.hostedemail.com (Postfix) with ESMTP id 881E98000D for ; Tue, 16 Apr 2024 03:35:28 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PkPeRowk; spf=pass (imf30.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.160.46 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713238528; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xa5U5Lslaz+G0m1rVPRlrShOq/KcuLodfgFv0sEM+Zs=; b=fPBFULnKcJLjxGLD09rxZa96xsZPjC5Gm5lZZb6T1G7qq+ZyEoycjIp2t266VpQhOrLZOE /aCko181Px3jR2f/bz7/t/z6U3dLKxPkIG5HnUs5QvOaPE34lsmjjk86a3UN2mMb1x4GUb CtJD1/B5R9n6Uoi+bqocEpqsMtocaVw= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PkPeRowk; spf=pass (imf30.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.160.46 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713238528; a=rsa-sha256; cv=none; b=oqXT8SGrG5v+Hybyxzp96NAoPaybWRn2x665ubmY8Oes2ugadQ9Wgicb5Wjk5tOLJSUvgo 2Ur79L1wggbGEa7D263wQm334OpOnHeQULSTdCMvdC98N/moC6xeHMyo4cZD5WSiR3PubV nlUbTN+dqrGUt2rql1G+1ghdMa7z2Lc= Received: by mail-oa1-f46.google.com with SMTP id 586e51a60fabf-23489b56559so1084355fac.1 for ; Mon, 15 Apr 2024 20:35:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713238527; x=1713843327; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Xa5U5Lslaz+G0m1rVPRlrShOq/KcuLodfgFv0sEM+Zs=; b=PkPeRowkXN16gKhbbr2m3smTpECn8yPAJMBk1HQBWXtaacyBA5G1CVN9TvLD1a2DbT 9octQeiKiwUkpgNzugGX6xQAHxRxMHVhfcap5WpiwBnWxlOrm3XDO621MmH9jZSax6FA TaJEuJufjSqQ7G1zwyUxAIS2nXnAK9RY+f84ozvGsogEJDUn97+vxl5UIRPD4ccdiOql m99HbeUrgwAPp4uPMe915R7e5ImO7AyPvIhGTEJrgWFuiQ0H4x/wX7/5KNtCYB1iHvOX JIRm77g6iOj8XxF1iniOpUDZuyKG1txA473e1/0tgyqELYGlSSmGTJk1LUFEVaUIA/pt h2cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713238527; x=1713843327; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xa5U5Lslaz+G0m1rVPRlrShOq/KcuLodfgFv0sEM+Zs=; b=BAHjUiaBgdQ2pO74+y2qPRw3nTUjBlgDoi/HZ9w5Y30P1BTnkFiN7FI0oJbMdnhY3S jGVTjVE1v2xh6GsZ7pma/aQ0F3jMGP/SdXZosAPYAJ4BaZnpiCiSlWZfAAsjxh7ztZmc yrbGOsZ4cQvxSME3mmGR51Cdi48RA7I3jY1m3xBG6ys7yde7xp5n7sCyzQXhdpDNgjl+ NQB/0ioZXNdQravQklR4YOYWjmYEqeIS12L6KBQkHksb89fZuracipRLUyJ61/Js8CrJ 4NCFaP9HbkFcpbJ9oHbb1Mb8RlvWNNhM77M50uCph9ACm8mqKHf+L5wFeQnzTYS40Wb/ FQWA== X-Forwarded-Encrypted: i=1; AJvYcCUs3vSpu4kA2I3Xjjvmf+Qif9ZANcAB6T5GQl2uJmBIPTJxsCe/1YWO81VkJq1JersGh9UhxwXzBY0+PAastqxyIII= X-Gm-Message-State: AOJu0Yx/151f68WhwI831fdZ//8yaZk68n7T+gCFkctHggd93SzRxTmr 5Ibmxhw4H29sPZ4KPhG7CGxb++IkSBGBLbzueI8gLehBJBlaBOLE X-Google-Smtp-Source: AGHT+IHwk5g3eb1ccJ96pB5R5kMJw+NcK5BFd59uPH2p5YulEU/Mhp5PXvWdZqm/wREsRZ7pVF+wYA== X-Received: by 2002:a05:6870:63a9:b0:22e:d382:87dc with SMTP id t41-20020a05687063a900b0022ed38287dcmr14773843oap.22.1713238527433; Mon, 15 Apr 2024 20:35:27 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id ka13-20020a056a00938d00b006e57247f4e5sm7906737pfb.8.2024.04.15.20.35.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Apr 2024 20:35:27 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v7 2/3] mm/arm64: override clear_young_dirty_ptes() batch helper Date: Tue, 16 Apr 2024 11:34:56 +0800 Message-Id: <20240416033457.32154-3-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240416033457.32154-1-ioworker0@gmail.com> References: <20240416033457.32154-1-ioworker0@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 881E98000D X-Rspam-User: X-Stat-Signature: bnrbkzj1o9quttfo1w3ap7wx6qksgxfg X-Rspamd-Server: rspam01 X-HE-Tag: 1713238528-368521 X-HE-Meta: U2FsdGVkX1+mF6vQAnoyQZOZ502vn6nfwXlU3dzcFQWc1DD7pLPJml/BJm1N6m0OF9utzjllRSDjHfBlD8NlG3VR+ZzYvzsb14i/DGa4UTI9Qe8hofKZlFlCJp2KPXnb3aWALJE+s7N9GVENrKxPMO7weo2E4Zo+5ubaN8xkxzT7Rjni6GJggcRiHEOdEmOB5w/Obz633f5NCRvshw4drAGkNS7fMJ5+55RlEIexm8/8epO2lTAV53IGvwlGN3yiRns9+WKhWvrOIm4mMmuEepC9GFxYehbgML8V4y6XgKlIcKumL8hLoVeYjtqZXTdhPolNMGhyDiisVpi8SmZWM7g/b0T13y/5iI2M5VEcKYLHMUYA+0FzxpQz6PNjuSOQ7ZUxntGs7KSbcJL6mQCv9o5ioypYaj2WOfetacHFfiMWJwVdULt4L64+BE0uOQaBWSgY4iJlAf7xvjDynPIwp50tEFMdduhyelX1n3lJYzkIQd0KsNxUNSFemElV1/9DS5XGtYI38DM4Ks5AKR76LPlx81/VSOvUIE0HgF9Pr3j8fMMrxrQynS60i9CCq9CZdzl6HPMbxuLwNHDQIEOX7yvlgcqvMv/pMLNtmJQbZJAfWV1DArktcR2CuSe8ThwGU7MEarPc10+BZZhEukO+otcb6RhtRd8cE69RiAXhHolh6OkXBSKOKwtfOXR/STQzetvSLyd9w3r1i1tL5ILvFJTYloN4xZoaiKwMnq8+NxGunAZ3thsQ31cVZTgSGK8/6xNGH++kdRRGWpekHGf8V63C7Z1z8v7yy3EJEDL2TQNWq8mqH6DqR3n9HoijDC1AQVkiqMek7yx7Z5B1CTY+ANBZzcBlvToiYGEn68eklRF7XNZUTWSIbZtRKDuVn7/PODfwLI1jWy8iddjKPtcYLdAPtyZgta3kav4Lnt9inmA1l9DIlRDxZCXYV/iWRz3J8hdcmf+NOz+2BybPv9D 5Z3OUj0Y kOKn4NuADAnpShcsWAyk/bRwggSyCO8F7e9psdhvgpwvG9TQbH9eVQ6Wt+0OeQLG3lS2d75V4BsyJtA/zVXa9sW/mgnlpdNHGp7R/8vTEqs7QyTe1zoNtEikLLI6w3BRTtjDSZxJQzLUeYBIVB8roCvJiyd2vi/WFBixekuzXB22b5Nzc7EYcscdA0zYDexl5jwuwJsUqWyTlTLDmwh4giteqY3j4D1Mm7YO9nN6dFtuXOFXn8m2oCwF0trOfcCQLtOTM6spoMPxU2xdiLTUGWlG8rKTFTuoJt9Xesc8b60j5Xd8uLH0s/6Q1i9nCsOFVEc9LOV6vzitkYiuJsR26oqTRPkj1DKW14ze/peLYXfmpFtVBDKh+XAwGytDTHB47327ideqLp5BUxWZOCfm3XLiQHGCcHLSjDxqaEYGQd/rNcXz2sJ9yEzAho7J+Wwg/EBhFlF+hwzoJwHVzkoNCvBs0Xl2r6gV6Fthy3roJXmmVtjSxT1fz278THsm8rsumM0XiPD2pc5h9VwQPdR02vSLMyMfR5IuCk4bMCQpotfu9gI3BY4vHlW5Rodb3B+alfpzR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The per-pte get_and_clear/modify/set approach would result in unfolding/refolding for contpte mappings on arm64. So we need to override clear_young_dirty_ptes() for arm64 to avoid it. Suggested-by: David Hildenbrand Suggested-by: Barry Song <21cnbao@gmail.com> Suggested-by: Ryan Roberts Signed-off-by: Lance Yang Reviewed-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 55 ++++++++++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 29 +++++++++++++++++ 2 files changed, 84 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9fd8613b2db2..1303d30287dc 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1223,6 +1223,46 @@ static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, __ptep_set_wrprotect(mm, address, ptep); } +static inline void __clear_young_dirty_pte(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, cydp_t flags) +{ + pte_t old_pte; + + do { + old_pte = pte; + + if (flags & CYDP_CLEAR_YOUNG) + pte = pte_mkold(pte); + if (flags & CYDP_CLEAR_DIRTY) + pte = pte_mkclean(pte); + + pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep), + pte_val(old_pte), pte_val(pte)); + } while (pte_val(pte) != pte_val(old_pte)); +} + +static inline void __clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + pte_t pte; + + for (;;) { + pte = __ptep_get(ptep); + + if (flags == (CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY)) + __set_pte(ptep, pte_mkclean(pte_mkold(pte))); + else + __clear_young_dirty_pte(vma, addr, ptep, pte, flags); + + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PMDP_SET_WRPROTECT static inline void pmdp_set_wrprotect(struct mm_struct *mm, @@ -1379,6 +1419,9 @@ extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); +extern void contpte_clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags); static __always_inline void contpte_try_fold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) @@ -1603,6 +1646,17 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); } +#define clear_young_dirty_ptes clear_young_dirty_ptes +static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + if (likely(nr == 1 && !pte_cont(__ptep_get(ptep)))) + __clear_young_dirty_ptes(vma, addr, ptep, nr, flags); + else + contpte_clear_young_dirty_ptes(vma, addr, ptep, nr, flags); +} + #else /* CONFIG_ARM64_CONTPTE */ #define ptep_get __ptep_get @@ -1622,6 +1676,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #define wrprotect_ptes __wrprotect_ptes #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags +#define clear_young_dirty_ptes __clear_young_dirty_ptes #endif /* CONFIG_ARM64_CONTPTE */ diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 1b64b4c3f8bf..9f9486de0004 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -361,6 +361,35 @@ void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); +void contpte_clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + /* + * We can safely clear access/dirty without needing to unfold from + * the architectures perspective, even when contpte is set. If the + * range starts or ends midway through a contpte block, we can just + * expand to include the full contpte block. While this is not + * exactly what the core-mm asked for, it tracks access/dirty per + * folio, not per page. And since we only create a contpte block + * when it is covered by a single folio, we can get away with + * clearing access/dirty for the whole block. + */ + unsigned long start = addr; + unsigned long end = start + nr; + + if (pte_cont(__ptep_get(ptep + nr - 1))) + end = ALIGN(end, CONT_PTE_SIZE); + + if (pte_cont(__ptep_get(ptep))) { + start = ALIGN_DOWN(start, CONT_PTE_SIZE); + ptep = contpte_align_down(ptep); + } + + __clear_young_dirty_ptes(vma, start, ptep, end - start, flags); +} +EXPORT_SYMBOL_GPL(contpte_clear_young_dirty_ptes); + int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty) From patchwork Tue Apr 16 03:34:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13631165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B786C4345F for ; Tue, 16 Apr 2024 03:35:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B4666B0093; Mon, 15 Apr 2024 23:35:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 965B56B0095; Mon, 15 Apr 2024 23:35:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82D026B0096; Mon, 15 Apr 2024 23:35:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 643AA6B0093 for ; Mon, 15 Apr 2024 23:35:36 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0914A1608A0 for ; Tue, 16 Apr 2024 03:35:36 +0000 (UTC) X-FDA: 82013980272.23.DF0967B Received: from mail-oa1-f49.google.com (mail-oa1-f49.google.com [209.85.160.49]) by imf25.hostedemail.com (Postfix) with ESMTP id 32138A0010 for ; Tue, 16 Apr 2024 03:35:33 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YJ0c9S70; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.160.49 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713238534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m0HcbiaRgpJF51M0mSKqxhETC3OHCqm5NVbvpi65tTw=; b=SfiT1MQ3fVdYPVPodRRjVXt1gPffm9z7Jw95DisVMSTYrt35l+Tr7vHOj1yTA0CdwM1AML i+Rrd0loJxrNkWnWQXTn+ZolM/kYamnVvChWwxz9R9jcGCDXrPvo/q31/0qxquniL9jG07 WluDh81DvGiQUwdrUT3I7AH+hr+Quug= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YJ0c9S70; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.160.49 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713238534; a=rsa-sha256; cv=none; b=B5vOedCajM1JQ5PFIIvUzt3aKdSJhxoaWVx5XIlQny7t4EirRB4vD3PZeWcdWbowubDTbn sZofrF23EZ5je3fPoVAtTBeWjvz74+Du1ZW+cQQhKmTzqPTfEo1NRLoJ+0BSIIU/tvtNhq yhsJBYqqpsb2zHVHD8FttZKYw7KKKbE= Received: by mail-oa1-f49.google.com with SMTP id 586e51a60fabf-22f01274622so2626157fac.1 for ; Mon, 15 Apr 2024 20:35:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713238533; x=1713843333; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m0HcbiaRgpJF51M0mSKqxhETC3OHCqm5NVbvpi65tTw=; b=YJ0c9S70f31vHPlaTWjkq+1Fo9Nmic4WhtrihPsDbWGlbW+hD8Kpm+oOFS3Wq5Xt33 OlYdnh8U3eb4UV3lzVEBlIHKj4btJ8J2WK4r5mZrBgJyPUe78I05KMhFB/EAvPx1I84l 6KM52S5l8M7e3YGDfJBSksXh5a9lfMEZkl8hI75SSuWtKp3tmdi85vlRTeRQyLGxN6HE xa9RniE/bBKN1gCYAlhVFSFq25ty+MzZ3OxOa4VkPlZ/Wz3Qe4G558ovhQB2+wC93Y9j I2iM/6Z9p/924bzXjStsmJ6zHZVh5GwiZDl9m1v+Z7fiTKBiwPS3csfVA9SkUJFD+EA3 m9/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713238533; x=1713843333; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m0HcbiaRgpJF51M0mSKqxhETC3OHCqm5NVbvpi65tTw=; b=Yzou/g9Z5Ks/y5GNyXIZnIOcOQAY0ib1yi8Wp1Nqol9zXKN5NTn9xbo1D/dOZ0MyEW fhcxs9Y5IOWxa8/OelOmuyX31YQyBpC539JDJrFc8cwEdEtnD8qlHLyLRfYczNNjcr5P /81T+vvOeDhYJ2ShpRqgBzW83bvdBRRmcqtl+1vbIi3CEgfchRXBWwFRQod5Gi4CU6HV nWIbsQs/B+lokqKpGIGn/+Qt13zG+t5MjHneT5kCeUsf0mM2zNdB3RlvO3yXMNZvPWfj GEqSel0oO9Mu2OKM5aB4twBStvj8PibszZm4opfUYtfQWbW/lxFPvlh4NTgqCezQVPiQ Raow== X-Forwarded-Encrypted: i=1; AJvYcCUpIYEsvtNSVBOBaWwEarY69TcxXUwa/4hDiTY+SuXo955GIM7nhtZtZT4iwF37qI94xRAvLTnhdzVn5mVuHyqt1Ac= X-Gm-Message-State: AOJu0Yx2Vv3gKUUBN9C6CRpbWUO26LTwN/nx2v7zSZtW8/mTO2xVfnh1 HWPCb/LIGHJH5k8NbJDx8jfxFumkGERWhJvN4pJUVcYIOVWBtLlD X-Google-Smtp-Source: AGHT+IGo1uJQRZkUMtOh4wICpGfarDFnG/8P7ACfcqPgczPdZo+tB+ZgYtW7iG5epQEsPT0CmQiKhw== X-Received: by 2002:a05:6870:718f:b0:229:fa5a:2fec with SMTP id d15-20020a056870718f00b00229fa5a2fecmr14635844oah.54.1713238533202; Mon, 15 Apr 2024 20:35:33 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id ka13-20020a056a00938d00b006e57247f4e5sm7906737pfb.8.2024.04.15.20.35.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Apr 2024 20:35:32 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v7 3/3] mm/madvise: optimize lazyfreeing with mTHP in madvise_free Date: Tue, 16 Apr 2024 11:34:57 +0800 Message-Id: <20240416033457.32154-4-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240416033457.32154-1-ioworker0@gmail.com> References: <20240416033457.32154-1-ioworker0@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 32138A0010 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 9kfiakkgt65musgpy5qybebsywpz3jos X-HE-Tag: 1713238533-924894 X-HE-Meta: U2FsdGVkX18Im1oqPNWNjJmvw0oGb/AWooB9fKx58kic1OL8436ieDB9HGySVjsp41F03E12k4YjKkLAFwRyVqNxbGQpEfcZVpXcZj0oKQVjWly0uBfyDZd4JlI8gDjnZ5yl77soQQVidx4qJbavsnFQVYgz5IgBniChb9aTf8SKPpwiI9iZPpDm2BellA3JyNhUy3uOfuH/DoMA0UaEdQv6r2ss3mDiKuhAPI3Ytdx25TwWczptei4wyrRQLQu2qgivjw2dTmTk3jtn9QbdPOMxlcdXTIH6vu294hp9onZOajUz9+0SIhmDiReChATlYZ+oNm4mYFB9QjWsuOltF3PhIatASAZwy5JYB9tifR6uCuKTXBj1/bjfBBmsZRiw5y9BXMUDipke83kkWbLiWyFktngBxBBVY2AEV/C8ibA/l0WWW+pCxmJw60HZf/YpLsb+R2Cq1TN8Rmkyh+6PlNyM4pjkpoatq9Px5qS7ibyE8jvNTdNz12bEebwWq6pGWbQstKsa1BW2kHYXJC+QOc2wEr/00eMrltncJ8Tvsvk8shvwQ35mM+oE3HxA7KhIIZ9gVZuRq3XeRYMqYn3B+qbm1+KJzXDeVion6qDi2Z15B8XJHH0jObjGEGOzSfZHY3lHiQrQ17zaxSOSinu4pL2tkUkwsdK66BnVGfYmmjhI2RY51BCCoXTA/aVs+Yk6ULLmQpqVx/O6S9S32FmfHcR3ep/Tih25k2BYpyKVIE2k3AuUYS4cG0r04ThnpZZgDYdgz69h3gU9P+LRfU/vlD3zFp6R6uAKUeuj4zWUe+tf8OYE0FyFW69LMFqCRJ3k6i3vAXWmb37AaIE7E6KPYN2DLv75r5QOq/kxCQnXE7R5dckDNBP8wbpP5aXjNqtcmLwEFnQpgMg64OZX/4cjI3dXcrrON2f0YABX6f9O45myVUQxFmRKJv0NDNN7xH5kVg8HgyhmNow5FTocDH8 8IrPPcfX 6zOrsupzTA+Vnnt2syUKhuQnWwbDer2SAieKK8JEjbF4NP39oOlArAfaCXJb5JkL3S8UmAqoIxU7rBrVfd59qjDN6CPRH53d2MLfu4QwaA+R9mRaZsPkzryPQEn6H2H+P+WbF4HUjH5aPQuVO96SVzeC86ROP7KiEonbRkGDNv3NckS2gos2rbzkF+zwYjErjXSxUk7Y3wma6Bw+hW6Eg7KrdDpfTO5tIS0nsZ60g4YRY4wSvQk7WjKsdALiKepXpEN5NN7Ft7otE9uSAXzwoKmyYBK3P6DlKt4+Snn+eqlumKOK3Padsuucny1WpJ70C7fUYKOLglITsDsMTGCit4Byn0pN59uCqnPwGmpj9Bo8iVZ13HmGmk91+3PCJUki+3xIugbh+r1FlC07hEsHHGlkOGoM/xdLoLyG94WyeQnJEwQwrpKUOJ4IRIeDRH9giwUDghiRqqqAzBSlqHl2A8dRKYD8Moi+8k23qzyD0mlTMxcBHThUmULsTSXOKo1mQ7zmKD1Jd/6r2fjZ/T1C8zFzLVf8uVRj+zXnK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch optimizes lazyfreeing with PTE-mapped mTHP[1] (Inspired by David Hildenbrand[2]). We aim to avoid unnecessary folio splitting if the large folio is fully mapped within the target range. If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities. On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better): Folio Size | Old | New | Change ------------------------------------------ 4KiB | 0.590251 | 0.590259 | 0% 16KiB | 2.990447 | 0.185655 | -94% 32KiB | 2.547831 | 0.104870 | -95% 64KiB | 2.457796 | 0.052812 | -97% 128KiB | 2.281034 | 0.032777 | -99% 256KiB | 2.230387 | 0.017496 | -99% 512KiB | 2.189106 | 0.010781 | -99% 1024KiB | 2.183949 | 0.007753 | -99% 2048KiB | 0.002799 | 0.002804 | 0% [1] https://lkml.kernel.org/r/20231207161211.2374093-5-ryan.roberts@arm.com [2] https://lore.kernel.org/linux-mm/20240214204435.167852-1-david@redhat.com Signed-off-by: Lance Yang --- mm/internal.h | 12 ++++- mm/madvise.c | 144 ++++++++++++++++++++++++++++---------------------- mm/memory.c | 4 +- 3 files changed, 94 insertions(+), 66 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 2adc3f616b71..5d5e49b86fe3 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -134,6 +134,8 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) * first one is writable. * @any_young: Optional pointer to indicate whether any entry except the * first one is young. + * @any_dirty: Optional pointer to indicate whether any entry except the + * first one is dirty. * * Detect a PTE batch: consecutive (present) PTEs that map consecutive * pages of the same large folio. @@ -149,18 +151,20 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) */ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags, - bool *any_writable, bool *any_young) + bool *any_writable, bool *any_young, bool *any_dirty) { unsigned long folio_end_pfn = folio_pfn(folio) + folio_nr_pages(folio); const pte_t *end_ptep = start_ptep + max_nr; pte_t expected_pte, *ptep; - bool writable, young; + bool writable, young, dirty; int nr; if (any_writable) *any_writable = false; if (any_young) *any_young = false; + if (any_dirty) + *any_dirty = false; VM_WARN_ON_FOLIO(!pte_present(pte), folio); VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio); @@ -176,6 +180,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, writable = !!pte_write(pte); if (any_young) young = !!pte_young(pte); + if (any_dirty) + dirty = !!pte_dirty(pte); pte = __pte_batch_clear_ignored(pte, flags); if (!pte_same(pte, expected_pte)) @@ -193,6 +199,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, *any_writable |= writable; if (any_young) *any_young |= young; + if (any_dirty) + *any_dirty |= dirty; nr = pte_batch_hint(ptep, pte); expected_pte = pte_advance_pfn(expected_pte, nr); diff --git a/mm/madvise.c b/mm/madvise.c index edb592adb749..a6bfbbd881e9 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -321,6 +321,39 @@ static inline bool can_do_file_pageout(struct vm_area_struct *vma) file_permission(vma->vm_file, MAY_WRITE) == 0; } +static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end, + struct folio *folio, pte_t *ptep, + pte_t pte, bool *any_young, + bool *any_dirty) +{ + int max_nr = (end - addr) / PAGE_SIZE; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; + + return folio_pte_batch(folio, addr, ptep, pte, max_nr, fpb_flags, NULL, + any_young, any_dirty); +} + +static inline bool madvise_pte_split_folio(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, + struct folio *folio, pte_t **pte, + spinlock_t **ptl) +{ + int err; + + if (!folio_trylock(folio)) + return false; + + folio_get(folio); + pte_unmap_unlock(*pte, *ptl); + err = split_folio(folio); + folio_unlock(folio); + folio_put(folio); + + *pte = pte_offset_map_lock(mm, pmd, addr, ptl); + + return err == 0; +} + static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -456,41 +489,30 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, * next pte in the range. */ if (folio_test_large(folio)) { - const fpb_t fpb_flags = FPB_IGNORE_DIRTY | - FPB_IGNORE_SOFT_DIRTY; - int max_nr = (end - addr) / PAGE_SIZE; bool any_young; - nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, - fpb_flags, NULL, &any_young); - if (any_young) - ptent = pte_mkyoung(ptent); + nr = madvise_folio_pte_batch(addr, end, folio, pte, + ptent, &any_young, NULL); if (nr < folio_nr_pages(folio)) { - int err; - if (folio_likely_mapped_shared(folio)) continue; if (pageout_anon_only_filter && !folio_test_anon(folio)) continue; - if (!folio_trylock(folio)) - continue; - folio_get(folio); + arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(start_pte, ptl); - start_pte = NULL; - err = split_folio(folio); - folio_unlock(folio); - folio_put(folio); - start_pte = pte = - pte_offset_map_lock(mm, pmd, addr, &ptl); + if (madvise_pte_split_folio(mm, pmd, addr, + folio, &start_pte, &ptl)) + nr = 0; if (!start_pte) break; + pte = start_pte; arch_enter_lazy_mmu_mode(); - if (!err) - nr = 0; continue; } + + if (any_young) + ptent = pte_mkyoung(ptent); } /* @@ -688,44 +710,51 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, continue; /* - * If pmd isn't transhuge but the folio is large and - * is owned by only this process, split it and - * deactivate all pages. + * If we encounter a large folio, only split it if it is not + * fully mapped within the range we are operating on. Otherwise + * leave it as is so that it can be marked as lazyfree. If we + * fail to split a folio, leave it in place and advance to the + * next pte in the range. */ if (folio_test_large(folio)) { - int err; + bool any_young, any_dirty; - if (folio_likely_mapped_shared(folio)) - break; - if (!folio_trylock(folio)) - break; - folio_get(folio); - arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(start_pte, ptl); - start_pte = NULL; - err = split_folio(folio); - folio_unlock(folio); - folio_put(folio); - if (err) - break; - start_pte = pte = - pte_offset_map_lock(mm, pmd, addr, &ptl); - if (!start_pte) - break; - arch_enter_lazy_mmu_mode(); - pte--; - addr -= PAGE_SIZE; - continue; + nr = madvise_folio_pte_batch(addr, end, folio, pte, + ptent, &any_young, &any_dirty); + + if (nr < folio_nr_pages(folio)) { + if (folio_likely_mapped_shared(folio)) + continue; + + arch_leave_lazy_mmu_mode(); + if (madvise_pte_split_folio(mm, pmd, addr, + folio, &start_pte, &ptl)) + nr = 0; + if (!start_pte) + break; + pte = start_pte; + arch_enter_lazy_mmu_mode(); + continue; + } + + if (any_young) + ptent = pte_mkyoung(ptent); + if (any_dirty) + ptent = pte_mkdirty(ptent); } + if (folio_mapcount(folio) != folio_nr_pages(folio)) + continue; + if (folio_test_swapcache(folio) || folio_test_dirty(folio)) { if (!folio_trylock(folio)) continue; /* - * If folio is shared with others, we mustn't clear - * the folio's dirty flag. + * If we have a large folio at this point, we know it is + * fully mapped so if its mapcount is the same as its + * number of pages, it must be exclusive. */ - if (folio_mapcount(folio) != 1) { + if (folio_mapcount(folio) != folio_nr_pages(folio)) { folio_unlock(folio); continue; } @@ -741,19 +770,10 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, } if (pte_young(ptent) || pte_dirty(ptent)) { - /* - * Some of architecture(ex, PPC) don't update TLB - * with set_pte_at and tlb_remove_tlb_entry so for - * the portability, remap the pte with old|clean - * after pte clearing. - */ - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - - ptent = pte_mkold(ptent); - ptent = pte_mkclean(ptent); - set_pte_at(mm, addr, pte, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + clear_young_dirty_ptes(vma, addr, pte, nr, + CYDP_CLEAR_YOUNG | + CYDP_CLEAR_DIRTY); + tlb_remove_tlb_entries(tlb, pte, nr, addr); } folio_mark_lazyfree(folio); } diff --git a/mm/memory.c b/mm/memory.c index 33d87b64d15d..9e07d1b9020c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -989,7 +989,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma flags |= FPB_IGNORE_SOFT_DIRTY; nr = folio_pte_batch(folio, addr, src_pte, pte, max_nr, flags, - &any_writable, NULL); + &any_writable, NULL, NULL); folio_ref_add(folio, nr); if (folio_test_anon(folio)) { if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page, @@ -1558,7 +1558,7 @@ static inline int zap_present_ptes(struct mmu_gather *tlb, */ if (unlikely(folio_test_large(folio) && max_nr != 1)) { nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, - NULL, NULL); + NULL, NULL, NULL); zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, nr, addr, details, rss, force_flush,