From patchwork Thu Apr 18 10:57:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13634513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B937BC4345F for ; Thu, 18 Apr 2024 10:58:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4ED0C6B0085; Thu, 18 Apr 2024 06:58:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49D0E6B0089; Thu, 18 Apr 2024 06:58:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33DCE6B008C; Thu, 18 Apr 2024 06:58:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 131086B0085 for ; Thu, 18 Apr 2024 06:58:28 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0C966121359 for ; Thu, 18 Apr 2024 10:58:22 +0000 (UTC) X-FDA: 82022353644.20.3C99EA3 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf24.hostedemail.com (Postfix) with ESMTP id 1106218000A for ; Thu, 18 Apr 2024 10:58:19 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jaFtFRrU; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713437900; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lkdNcUi/cFtD91tOAv56m6RXgS21TIueDzxwluwn/Lw=; b=R0CVYbHyG2fwsqs5SDZYuq5i7qVMAEsE3hASqp3qZA2gy4BeUccYO4k6y2xqcz8xooPlFW GFExe8tSF2m9DsH0Z+z81MjftsOLTbchIhltWBdr0wXkkijBSUWwcASP9qxcOSQ00k2QPi KkewHSuk8SyuTRp4ZtnlgeEi09/dntY= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jaFtFRrU; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713437900; a=rsa-sha256; cv=none; b=yRrNuvhESPHAYwJ3yGftfnTC5AEdTliy66w4+hC9XkKhNmTg7E74mu5ZHJ/GGu3tLm64vg wD7/CveYKKpQgvut1Ryhbq19zlnQsvxDwUhpbIOwudTxBDXND2wW8ygVwLbitYDPr2ec9x pwXa6ibeQjrFSBiIdkhv9ujpep/SO0s= Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6ed691fb83eso688969b3a.1 for ; Thu, 18 Apr 2024 03:58:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713437899; x=1714042699; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lkdNcUi/cFtD91tOAv56m6RXgS21TIueDzxwluwn/Lw=; b=jaFtFRrU0KjbvX4Vx9yTxdV2ZBONAJq4oIOoPq0zv884gDGuFoJd4lmH6ZJatK0APG z9lfRNvsBPxkl1eiYufy3+W50nhADo4P+fsa69ApFjjLVTMlbJNadxqbC4GbNlNVV1pk hXVm8xudkp70G/7Yvk+DMV3VE3xwt+85/eaE5EbGKYOtXuDaByjGYj4XJm7hzok74sWz h/aQTTU2SzqeLdcHBimOE/fz1tZ1pRu0nZvU7EI9ulFSadlxrEKbP7CJh1xocqQAiKrg 5yHdIHtC54DQOHbV+wYYkb1y5j7pcRvO0e9MFQEEBOTXoCPw7xK/HntJ/Ilxm82jx4vi wjnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713437899; x=1714042699; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lkdNcUi/cFtD91tOAv56m6RXgS21TIueDzxwluwn/Lw=; b=lG4SGotC44w//Tiuitxa6U77SKv+JjEZemkKjx/J9gDmPBklDNF8ey2SYUS09boKMs 0zOTBCtoPCHb48fzsQeuudMJc64ZZu3zEW7XW1eJ7cN4YrUdaasvb86y7fz8iftoq4Io 7eyiUjnbyBhxcMldhSv9FhsnIa1MZlAkTKF7VzY0TJl2DzFSdi/NUvwDv7TYsQA2JqND SG6JbcqGekRxub41KJB8FRsrIczpF70Gu9bRWmXUC3mEdGzXH2lM35WemHtLxaRsxCvI /gFFCgH5YIfPZCBAp0RqT0MMEDbsgI9vXji2yoR6X8mNG2chJPwQIh1nlzWhsV8yUlTT F/wA== X-Forwarded-Encrypted: i=1; AJvYcCVsOA+R3TRhAXE980b/iHMvPUwsl2wYYtbBvVsy4eKC2YtWRG8QMknAzeN78fGYUBKRSEF4Kv/pzkXfY9v/ohqqpFg= X-Gm-Message-State: AOJu0Yz/eM4Cc5bo7vjwGvjDmfBPY0F1vgMNTQVGFQPx8OKNPBZvKSEM OKxIFtmOKM9iRS+Gi8K+0rqyHcoe2xm7FqrirSaT7Y+xWGp1LPSf X-Google-Smtp-Source: AGHT+IF4GTJLmtDlsAiq1UBHnJLKlM7tfb3a1b5r2w5ICLJMQHsjTAFqJxOVL/DbBjBw6Z9Mr8QknQ== X-Received: by 2002:a05:6a00:1749:b0:6ea:d740:62a4 with SMTP id j9-20020a056a00174900b006ead74062a4mr2701566pfc.25.1713437898934; Thu, 18 Apr 2024 03:58:18 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id gd26-20020a056a00831a00b006ea923678a6sm1200487pfb.137.2024.04.18.03.58.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 03:58:18 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v9 1/4] mm/madvise: introduce clear_young_dirty_ptes() batch helper Date: Thu, 18 Apr 2024 18:57:47 +0800 Message-Id: <20240418105750.98866-2-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240418105750.98866-1-ioworker0@gmail.com> References: <20240418105750.98866-1-ioworker0@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 1106218000A X-Stat-Signature: mx3htfpr4juf39z95mtcuan6n8nbudug X-HE-Tag: 1713437899-753715 X-HE-Meta: U2FsdGVkX19lV+AdJSsJjt0aPdAQq1B4MZR7TAwHaKD0yWJ7qOOR4Z/g8EmhHtg4KfgOo/4SbuWowFaFffiH0QSZGnbjvtM4OZyBw4WV4S4B3ZwKpsdhQ8EP83wb4rkHSHAOoDGtMixLZYtn2NKAxhiQtGYubfu0oDjdTHsOqWsmaNDGE1B0BmgyWANlzK6R+jfp25di0uACnxApHSd0J+wT0WyL24K9fcDZKKeVqoCNujrdGnbIXc9XpnurGGWBp2E8FKascFfLeXkCNBqJP7oL5H0xLUkmFKH0Vge5gfjDSFIJYOdy44As2BJ7SSBNDwaPsEwcWVShjegTuP+HEO84N9C/IDVw/ErLIFU/WB6J4g40mDiYCOKJ65HMHAMxIEJ8deJrh5X31sM+qTvHrKP2CwmwSv3O7jnI5F2AMloye0bw5og8fTSVVu2wQZ2YDwHV8lV28DwGekRcKgsFWkO/PsCgzVRbf5K5yF4FNjQ0HJRlXZMC0NkROpxoh7xTN8gdxftGZgXJ/XmBr61ct8gAdc7OKESqYHJX7Wz1C5Hq4pVIFOXVA3uEOTulsltpMQvrkH2XmluXjdIr+8xOsGg2igG0RZY/YB8ihBr+uFwk5CZ9KzP2hxD76O+ODs8Kv0yEKRVqyrv6SfF0MpArEaV80NBgjAfuUq3WYbF+5A+gfbckfTXOgKiAYXgfVwbGM/uVz2KjBvvQT0QlqPtnqtRgLYDA+IbnwG/F5dIRjvoUP7po8zK7WeBUMr18+/42Tjb9GNGXJSxQza2Ex318F7eurmKGeCey5Rpjrs3WDGZ05gDr9XLKeF8315FlveDZSYyS+FP3I1r+kQ0h9zJtDzwAjVXZSjSPgvxPc19yfRApDqfy3sQIbP6c+glGfy7GBonEF3zfwPzC04WowXnO+JW4/BK4mTWWFT1aOKJg4U+xJCKsHuam8ojR/z7wwCJK5tCV5mvfh2MtAHYrKM2 QRIrLQSU jO5jpRyojw5k/a/Re6QdUfw76/b+RZuc8sMpC0pr2WctXIJmpUppnQXThBfquChBIFCWosrzIKFecl9jyOfXmt1Ri+O95vmbh8Ncn3ZlJXJH8c0Ydljqd2JGdiOFsfjlWc2l+xNY0SI3nLgWD4E6ZM4FDcu92/Q3UrlxtJauawE5WgnwDmNiAPtRNYM6EJDFadJDdqeQp59BUO9U0ubV2ZVrYw/OcDWnLxiXW2QWb7O29ux3ZDkdTTVu5UYvx2LtZz7W0eUIEoDuFw2iawnzTsayihEZsOaZi+hR3z/TmsQ+r6MzKDPqX18NTYh9COsMsiORCu3DA63mke78P4lG0yMZKKsMdjjE+0306XGdJL/YhrMPquR5YY6iJMFLcxs+iRui/AbLtztqWVCBQ+zIte0FhjQIqbwnriteof/k30CTu+cQnPEP50SwRKTxHrjvQxf/0n2AedJhNefTZzXda+qWxTPu9waOWwyteWP1E8UAXNkGd5LR2PpVMXjcKR4czNYC8Nq8z7O/706hzz05IaCHmRzSlFcWtfnEsqImt3Dwwa+FpcvysoPGzvZknfSJvxMU3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This commit introduces clear_young_dirty_ptes() to replace mkold_ptes(). By doing so, we can use the same function for both use cases (madvise_pageout and madvise_free), and it also provides the flexibility to only clear the dirty flag in the future if needed. Suggested-by: Ryan Roberts Acked-by: David Hildenbrand Reviewed-by: Ryan Roberts Signed-off-by: Lance Yang --- include/linux/mm_types.h | 9 +++++ include/linux/pgtable.h | 74 ++++++++++++++++++++++++---------------- mm/madvise.c | 3 +- 3 files changed, 55 insertions(+), 31 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index db0adf5721cc..24323c7d0bd4 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1368,6 +1368,15 @@ enum fault_flag { typedef unsigned int __bitwise zap_flags_t; +/* Flags for clear_young_dirty_ptes(). */ +typedef int __bitwise cydp_t; + +/* Clear the access bit */ +#define CYDP_CLEAR_YOUNG ((__force cydp_t)BIT(0)) + +/* Clear the dirty bit */ +#define CYDP_CLEAR_DIRTY ((__force cydp_t)BIT(1)) + /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each * other. Here is what they mean, and how to use them: diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e2f45e22a6d1..18019f037bae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -361,36 +361,6 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, } #endif -#ifndef mkold_ptes -/** - * mkold_ptes - Mark PTEs that map consecutive pages of the same folio as old. - * @vma: VMA the pages are mapped into. - * @addr: Address the first page is mapped at. - * @ptep: Page table pointer for the first entry. - * @nr: Number of entries to mark old. - * - * May be overridden by the architecture; otherwise, implemented as a simple - * loop over ptep_test_and_clear_young(). - * - * Note that PTE bits in the PTE range besides the PFN can differ. For example, - * some PTEs might be write-protected. - * - * Context: The caller holds the page table lock. The PTEs map consecutive - * pages that belong to the same folio. The PTEs are all in the same PMD. - */ -static inline void mkold_ptes(struct vm_area_struct *vma, unsigned long addr, - pte_t *ptep, unsigned int nr) -{ - for (;;) { - ptep_test_and_clear_young(vma, addr, ptep); - if (--nr == 0) - break; - ptep++; - addr += PAGE_SIZE; - } -} -#endif - #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, @@ -489,6 +459,50 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, } #endif +#ifndef clear_young_dirty_ptes +/** + * clear_young_dirty_ptes - Mark PTEs that map consecutive pages of the + * same folio as old/clean. + * @mm: Address space the pages are mapped into. + * @addr: Address the first page is mapped at. + * @ptep: Page table pointer for the first entry. + * @nr: Number of entries to mark old/clean. + * @flags: Flags to modify the PTE batch semantics. + * + * May be overridden by the architecture; otherwise, implemented by + * get_and_clear/modify/set for each pte in the range. + * + * Note that PTE bits in the PTE range besides the PFN can differ. For example, + * some PTEs might be write-protected. + * + * Context: The caller holds the page table lock. The PTEs map consecutive + * pages that belong to the same folio. The PTEs are all in the same PMD. + */ +static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + pte_t pte; + + for (;;) { + if (flags == CYDP_CLEAR_YOUNG) + ptep_test_and_clear_young(vma, addr, ptep); + else { + pte = ptep_get_and_clear(vma->vm_mm, addr, ptep); + if (flags & CYDP_CLEAR_YOUNG) + pte = pte_mkold(pte); + if (flags & CYDP_CLEAR_DIRTY) + pte = pte_mkclean(pte); + set_pte_at(vma->vm_mm, addr, ptep, pte); + } + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} +#endif + static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { diff --git a/mm/madvise.c b/mm/madvise.c index 4b869b682fd5..f5e3699e7b54 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -507,7 +507,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, continue; if (!pageout && pte_young(ptent)) { - mkold_ptes(vma, addr, pte, nr); + clear_young_dirty_ptes(vma, addr, pte, nr, + CYDP_CLEAR_YOUNG); tlb_remove_tlb_entries(tlb, pte, nr, addr); } From patchwork Thu Apr 18 10:57:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13634514 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63F5AC04FFF for ; Thu, 18 Apr 2024 10:58:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDE806B008C; Thu, 18 Apr 2024 06:58:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C8EDF6B0093; Thu, 18 Apr 2024 06:58:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE0506B0092; Thu, 18 Apr 2024 06:58:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8E8806B0089 for ; Thu, 18 Apr 2024 06:58:29 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 608DF81241 for ; Thu, 18 Apr 2024 10:58:27 +0000 (UTC) X-FDA: 82022353854.10.00C73E8 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf07.hostedemail.com (Postfix) with ESMTP id B733940009 for ; Thu, 18 Apr 2024 10:58:25 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ajy0DBoa; spf=pass (imf07.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713437905; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XITq4/hgHmAHZ0ZHlNrd2qmUtOIaf33ePb9J0ZwzOSE=; b=FP30WpHD/2b6SZJDeZc6WOpex8hM5KlSZLtyAXHX+VrM4gRvRzIEgV8qiiM6gbAL0nPMSA TsPlEBNXtBqqqiO6zS1f0z0lHO0VS1G/77zOThWeZ5e77Vdfj4NxHH+vmIWJxPT4EYrImX wjgYbIwZJkBBpdiz92g3HWocT5egWMQ= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ajy0DBoa; spf=pass (imf07.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713437905; a=rsa-sha256; cv=none; b=wYELvfUbiMCeReY6BXSpQIdaxXIBGh065ucBCST36Ub/NLPQSrEwFWVLuBLdUEO9B7EJZn +3+wQVzWeYKnxao1D762R19rm27QwsmB4IV3cuWlNCk+YifzoRxWpBosoAf4xF7bIXn9zE WwZJpbcAGfhg37gdakcJg2+u5aZDsRg= Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-6ed2dc03df6so683555b3a.1 for ; Thu, 18 Apr 2024 03:58:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713437905; x=1714042705; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XITq4/hgHmAHZ0ZHlNrd2qmUtOIaf33ePb9J0ZwzOSE=; b=ajy0DBoaMX7uC8eBV/ssrbQ8sEjC+qf2tGJjhmJiDPYoCe6oSrTXJzJHx7Wsj89P4q Q1+4hYPjtiFKb5n/nxlVsMlVZJhAfboAFTAQB6f9Gd/rsEKiIBAtN4jpT1JKIvd0tDT7 yPhbwwlSGBseK3U9ZieO50g9cpgY0UeAM06xIJCClLVVPoBFdU9G6OAuhR/Qu8/smaGx A4ji8ZJ0e1Lg2IIGpOtWYMJrMNNAhIkpf9z1Qv+VoExpVtFe9ZhoHK7G+UX8HL134F78 iAZoCGK2XWvJIczDwX6pWPvvSaxJwetnbtPMqZ7CZKbMBMgYtra3W/0CytnkUBedgbZ0 5lsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713437905; x=1714042705; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XITq4/hgHmAHZ0ZHlNrd2qmUtOIaf33ePb9J0ZwzOSE=; b=AHreFMWr7Ij+tYIYfoOZ7uyRchRXp1mruZMO2XBDgfSogRvJKRCJZ6xGwTVpeSwzjM LAQHcIY9eBhxTQhA4B0sRl0F27VRA3Ft2aeL6/xMjltWKjbgCIesdzlcnAaFQFR1tgWW 28oaveQQqHkMPylLzLm4HMtPz/BL1SHHxQjSyr6yw3laq+Z0sbpIbqzcV/Sa+3oZ0/KP HtYC3mH2fdwHCC/eG1jP+8G4SDri4xV6QZObBFZXgrMMPg7op5Q6P3miJXlcXzHGqi4/ 9ZbeidncCXb6Vl3W72hFm6bpnDJiVkbR+YYWCpAbpObzpIA6Bq1LPaEPT3i252MYLmFP UWMg== X-Forwarded-Encrypted: i=1; AJvYcCVQrefRmsk7LbZgyDCBxTmcFwzuoXSGt4/1EkFxr9MnAbB1WwNKwEyHYgDNY36y2G771idgcl7VxnLQY3c0UIWFt90= X-Gm-Message-State: AOJu0Yw1lk50PYSPEP1ZvyDTmcxjTtECAfh743OzGMr5b/zDuHi8PMCn cZhlHOFa/pg1TIPoQj8HJ4AaN3Va/VAKzLBIDghyhjkEgGPmEeka X-Google-Smtp-Source: AGHT+IESWKGXs5SVUDQDCP4J36Ikudx/CjpP5NVMmnbx2ui5miMXKMf1AI2xrY+wo35NeL00v+rfzA== X-Received: by 2002:a05:6a00:1acf:b0:6ec:db05:36d2 with SMTP id f15-20020a056a001acf00b006ecdb0536d2mr3186097pfv.4.1713437904694; Thu, 18 Apr 2024 03:58:24 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id gd26-20020a056a00831a00b006ea923678a6sm1200487pfb.137.2024.04.18.03.58.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 03:58:24 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v9 2/4] mm/arm64: override clear_young_dirty_ptes() batch helper Date: Thu, 18 Apr 2024 18:57:48 +0800 Message-Id: <20240418105750.98866-3-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240418105750.98866-1-ioworker0@gmail.com> References: <20240418105750.98866-1-ioworker0@gmail.com> MIME-Version: 1.0 X-Stat-Signature: ctteswnttgrawre5i47d9bkzeio4qkhn X-Rspamd-Queue-Id: B733940009 X-Rspamd-Server: rspam10 X-Rspam-User: X-HE-Tag: 1713437905-569215 X-HE-Meta: U2FsdGVkX18rureDANi9up3oYxkRhXgEu9BUHtWbggoxnCGD3aV3lMQbSkJhx8jrc26YW7hNfN3sv//Z3YwVDB11IclXI+2lPKXxv4A6kqoCRtNh1CFgF1eZcvgar/9qD459Kw4WmEh0ECrfBmjgCINL54lP6vK1YY6xZgfLhzAFNcOQDzedKXLwP8doiXlKDKIs8mMstBbXXvgmgGG9YggLKHCnnpBrVdcA1KOw9dThooLghaBvDVoKXuekWRodoKspLlRDV5eahK6t3Xwgb2eA5AC4EBARbh9OdaTtEaX5so7Ey2skC82XT3PPeFnESW7At+w9ONhrXPC3WhWQhgoXTx51l98xC8lNcsKwSnoojqlSGo3u5MnRFwev01jzin0EiEAT93GSjzAeXeKWQ0WCySosLXL/YkH7UYYEUuaiWoNcWm5I2vseDqNIUDatPhrims+Otld991j/bBunjPbIr/EkfdcdUEnWQ6HS/iV2PxqxuPeE17O+rVJz3xNOdqIrAiXsQ0u90qm12xgiCFUN6FnoaAeUhssMpBoz2iVCxQgLCY+wESOg2xXu62LpnPCsvv1OAHi+RsqrxMC0UYA+3UBeTeARvkMARPGoQa3EWXU//zCMi3ilF/bQM7QLiq8hqWBy0xrZ0v5xCHlIVz6pB3eRdCtvCpUdIxfFUAB6u0eoBB4vmYqOepu8vltVOE12dtcfC+REZ1ZkRjk2KDatGsJTrQ6/qXdT0P14X3xEcugMwFWikmwudmzKGFXIThiE4wYV23BFzuL7YywCez/GK/IJRV6VFyPTmgKWm8LFrsJ1P9PPLCNKUJesunINPc/gqXbpXWNC8WyjgSn50L80CNdJCU2bvOgouICaR1zLU6nKsKI5taYC3W3v4fs7ROZg99sFG2SEM5TEWlcGM+knPnua4c9ai/npVYOK2xfyOvfveFvA2+OT21Cx+qraAUikmXam0MnVchE50DZ YfX6Qmmf uzrz9lUWiKWM1JCvyeS39kSQ0+Du2c0XbzYCyggfgI/35zuTSjrDWy3JZ/Ujw/nZzwX73oKVNeEVaM1QYUcyntzORMtxIp+c+MpeJmVDW+3cloM6lDtYh/5QRkjqbFjRGVwGk+S1BLbeoskaNyHlynYQIo5Wf8DgESk1P X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The per-pte get_and_clear/modify/set approach would result in unfolding/refolding for contpte mappings on arm64. So we need to override clear_young_dirty_ptes() for arm64 to avoid it. Suggested-by: Barry Song <21cnbao@gmail.com> Suggested-by: Ryan Roberts Reviewed-by: Ryan Roberts Signed-off-by: Lance Yang --- arch/arm64/include/asm/pgtable.h | 55 ++++++++++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 29 +++++++++++++++++ 2 files changed, 84 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9fd8613b2db2..1303d30287dc 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1223,6 +1223,46 @@ static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, __ptep_set_wrprotect(mm, address, ptep); } +static inline void __clear_young_dirty_pte(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, cydp_t flags) +{ + pte_t old_pte; + + do { + old_pte = pte; + + if (flags & CYDP_CLEAR_YOUNG) + pte = pte_mkold(pte); + if (flags & CYDP_CLEAR_DIRTY) + pte = pte_mkclean(pte); + + pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep), + pte_val(old_pte), pte_val(pte)); + } while (pte_val(pte) != pte_val(old_pte)); +} + +static inline void __clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + pte_t pte; + + for (;;) { + pte = __ptep_get(ptep); + + if (flags == (CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY)) + __set_pte(ptep, pte_mkclean(pte_mkold(pte))); + else + __clear_young_dirty_pte(vma, addr, ptep, pte, flags); + + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PMDP_SET_WRPROTECT static inline void pmdp_set_wrprotect(struct mm_struct *mm, @@ -1379,6 +1419,9 @@ extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); +extern void contpte_clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags); static __always_inline void contpte_try_fold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) @@ -1603,6 +1646,17 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); } +#define clear_young_dirty_ptes clear_young_dirty_ptes +static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + if (likely(nr == 1 && !pte_cont(__ptep_get(ptep)))) + __clear_young_dirty_ptes(vma, addr, ptep, nr, flags); + else + contpte_clear_young_dirty_ptes(vma, addr, ptep, nr, flags); +} + #else /* CONFIG_ARM64_CONTPTE */ #define ptep_get __ptep_get @@ -1622,6 +1676,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #define wrprotect_ptes __wrprotect_ptes #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags +#define clear_young_dirty_ptes __clear_young_dirty_ptes #endif /* CONFIG_ARM64_CONTPTE */ diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 1b64b4c3f8bf..9f9486de0004 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -361,6 +361,35 @@ void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); +void contpte_clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + /* + * We can safely clear access/dirty without needing to unfold from + * the architectures perspective, even when contpte is set. If the + * range starts or ends midway through a contpte block, we can just + * expand to include the full contpte block. While this is not + * exactly what the core-mm asked for, it tracks access/dirty per + * folio, not per page. And since we only create a contpte block + * when it is covered by a single folio, we can get away with + * clearing access/dirty for the whole block. + */ + unsigned long start = addr; + unsigned long end = start + nr; + + if (pte_cont(__ptep_get(ptep + nr - 1))) + end = ALIGN(end, CONT_PTE_SIZE); + + if (pte_cont(__ptep_get(ptep))) { + start = ALIGN_DOWN(start, CONT_PTE_SIZE); + ptep = contpte_align_down(ptep); + } + + __clear_young_dirty_ptes(vma, start, ptep, end - start, flags); +} +EXPORT_SYMBOL_GPL(contpte_clear_young_dirty_ptes); + int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty) From patchwork Thu Apr 18 10:57:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13634515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63F95C4345F for ; Thu, 18 Apr 2024 10:58:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E16506B0096; Thu, 18 Apr 2024 06:58:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DC7976B0098; Thu, 18 Apr 2024 06:58:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C402E6B0099; Thu, 18 Apr 2024 06:58:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A560F6B0096 for ; Thu, 18 Apr 2024 06:58:35 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 677B7A1318 for ; Thu, 18 Apr 2024 10:58:35 +0000 (UTC) X-FDA: 82022354190.23.62A06B2 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) by imf09.hostedemail.com (Postfix) with ESMTP id 938F6140015 for ; Thu, 18 Apr 2024 10:58:33 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DC8bXvnN; spf=pass (imf09.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.167.176 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713437913; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fCDKyy/HK4fLjIHRec29GbYvLewmbRt2hV7S2QYdA34=; b=HQ6iqXj82qiG8O+9LCEy3eLzwRLKwjNm9Nl1p/mVPdnzZgiucUuDWsWrJsUdq/0jfr2WeN AGahlMy2tX3G21fJYmFOIxDWTBjZF2T3cznhjsMqYfVXFOZJPYoFjUuzMPuJr5fmQcUrA+ 8WKKjgm2X7wNtnnb/VnI/GvugjTOtTI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713437913; a=rsa-sha256; cv=none; b=gFJGN8D6g69Z2k5bOxbzYiIVLbGDUmYNmHeo27NThsMBGzJMeBlm09sPjH4OX6bCX8k9VJ DUEi+uEqiqAEbAbQr48aFaiXIZmOip97nEWE31Ol2YM0xklGBFhwDgJA8zqyZEWbhGSErI FH6TcITO4F2eghF4sfp0Y43Am+PKi+Q= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DC8bXvnN; spf=pass (imf09.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.167.176 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3c709e5e4f9so504198b6e.3 for ; Thu, 18 Apr 2024 03:58:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713437912; x=1714042712; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fCDKyy/HK4fLjIHRec29GbYvLewmbRt2hV7S2QYdA34=; b=DC8bXvnNMRKBSMlXeummNCCgC9K/Cbd8WgwYfLm18xkXhMLCtp0Mp/xQdYCEXBDuRm 7TKYdbjsdrD310VqKEUQEvYI8+AhXY27WvirF+oU9zTXVUlz+jWLvkC2zWf7lSk82Ch2 D25paheXFsReCMxTh0srfskMzkvrsoSFqxt4RDxyh5RRdnvuKGlQEPhq86pXkJws/X2M iDYt/vUqED2Ii8ujsgwByacvIgws8D8ChzfiiG9djZ6K2MgcF8Ar4EyXkwfEZno9zjBL dzG32rHbdWXKp1qSMJJQkBkUVTgFzjjlOFel20cbDNKbIDqNMFa9sS1qd42Sjiy2BXO1 i23w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713437912; x=1714042712; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fCDKyy/HK4fLjIHRec29GbYvLewmbRt2hV7S2QYdA34=; b=d1qLLGD8Asut2dUaVT1UWrAYVojbMZ1GPHu6bNSrHmt16jzXsQCDFKlCf+pTx/3l/v EeQ8NwkI50Rs2E9PH5rC6k+KN8HYlwE/BKIvxxu92Y6fpJ0M+oUSGqRVfL/B8MXchziV T28VPJ+sxfoFUKGj0EkHES04AlTBLqTqvYcDf8khyAlaW6Aisx6PaV0ZMbRs5Ko3BsRJ M2JcJ6FE5wtdg12jXHRRBYTqrOtUX0L5slHFarXJk/fxYavu075lBPD2jo7MwyvJ2ZnH MoU5R+vwBPoaFxCq1MLGd7API3n0wPllxLCEubGDMgTcTbVtlNq7Cq5sjnRUGx5aZzuU b1IQ== X-Forwarded-Encrypted: i=1; AJvYcCUaCNq1nI+Y2CY0KbwrpSJBbFrqr41L10wnpTjSah6Ky0jegwjEpUnfFiaAylENScefOcCd4ZFOCRH23ug7Z3wOAI8= X-Gm-Message-State: AOJu0YyjN6Lsl2LnSJJRhLF3KwPyfywW+206sR8oHrYvNh8vfOqVKCYl wjxbKia8u/R2uJnIX0liy8Wdxs2kqEypOmNQNOijpGPAUibz9t09 X-Google-Smtp-Source: AGHT+IFiJWFYcIrSf2RPRltS/8C34Y5gwlKmlTV5x2wUCcZ1LqdcLz5TQFv4JEnEdVetJ6UiO6Fzxg== X-Received: by 2002:a05:6808:8f3:b0:3c7:3b1d:bb59 with SMTP id d19-20020a05680808f300b003c73b1dbb59mr1553476oic.2.1713437911086; Thu, 18 Apr 2024 03:58:31 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id gd26-20020a056a00831a00b006ea923678a6sm1200487pfb.137.2024.04.18.03.58.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 03:58:30 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v9 3/4] mm/memory: add any_dirty optional pointer to folio_pte_batch() Date: Thu, 18 Apr 2024 18:57:49 +0800 Message-Id: <20240418105750.98866-4-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240418105750.98866-1-ioworker0@gmail.com> References: <20240418105750.98866-1-ioworker0@gmail.com> MIME-Version: 1.0 X-Stat-Signature: du8ubufesgedbnmzh8ge5xo7mtzbe1br X-Rspamd-Queue-Id: 938F6140015 X-Rspamd-Server: rspam06 X-Rspam-User: X-HE-Tag: 1713437913-278683 X-HE-Meta: U2FsdGVkX19TJlyI9ytQ5tIppR+xgqPadkwAUaAzrm3MMiwSXaalxfTLc4GRZse8yIYu6EMR35BqnKOqcjKlmMRTVjIOaI5OL01rE2Asv47g05DJhfWfg1z+L9YA3UE0ddDEeBn7d8dz8KzaM5xcZCP5uqool35L8Zr+ctRTzM0jab33Y934nHlixdgXRUIK+3eMBv5TMNWIQ15jcFgUywOjlnzBxyCDlZhkI55Eo0C1u3dSnBg8NlDrauQH0iEbcJ5dDvLJC1W7J0xKJAlPE2Bn8ZrB2O4itVd6THy9xxy9/ba2J0tCDxhkSGmmKmjDUH34e8H1Dd3uqq+CxcXD+1Jxfak/33HPzJP8ZQ6wTzCyjR6U+XVHSXKInie7dmeK/hN6xLirejstdXz1PW6uie61k5dPDu+y0YBc7OOaSrZn+5j6n8o0l9LV+IFCKb5nAc5C6zMc0PwDajkdoiwE6Ab1kEZxvoBo73c8N3LPtqu3PVWdzAqvbVlTSSbL6GpnpD6jImNzi5xl7eK8eNydeISjwHa6quaBxx4/sLBm3f3qU08K/m3PpmK0cSe7UMVz+48Xj/WxuHQa56sv2y9HF1wfxQR5dQqmluD/Bti+UVO23D61N6YOW0dX4uPWog+hI2URnFXLOwPGNQ1udQU8av9tPFC43gLPQPajkZlsnAz7mdIpn+KGzqkHvODEU2TxRqp/qJYIkeGv95tcGxjme8hx9Vd4YL+iyC/AzuSzXKrloTtPMD2oHOhJlUtSiXWCy/4IF+aEFGFjIftNFKAN7Wl5nu16twHEcW7AFWgrPRI5BAIfXrWfIT2TMfW1AcawS0Mktf+nBa753vcaucEqq4KmMrpEyaaMDfsebc6+crWTHLRIjcuqPUMTNIoGdaH9g8IS8FPlR0Slc9ulQ84NRclsHwbUDhPWI7vs4rmV40b2g/XVrG+q8kkS8HaKfcpGGi3meKGQjd701lZzpUO uhgIN9LK oFPOQoGokcOdguPMk7PPgvi8+z1U2qRPy2Pf6V7See3hCO7tA+qb9lRBjz/zj7rMwS4OlvZXkBpF2Ufo8U5W2txd0ffebwHFOWjjOtLaOxUeCO6yE4YdP2JOscPaILd8YcxHxqmYWN7RlXGg9TLyG8n1SxJKd1uKDye9PMfclU/Iv4Gq8tvd3bOZnkH4o4M8SO73p6QTg8ZbIbpIjR1gxfKIMs7P1fkspU/+IW2MGyuck4ilPQgxo27EsNF6TvHhx6HR5POUCb7czhLEFt6jWn9cTibxSxZ9BejspgF4Wbd9PfnfNa59hSxj3S7lbz506iTgAvLLc9qee1nnOHyFw1avJCb7vdIgZrg5qoZJUhp+JVQ+kSlLAPztok/8DOgFHTVAUTscyKR3mONrhZvUHSl2MlfNw3FEpxDlT1SAfh1DQzA/Y4fEZ7qLa1JmjhYmzCDnT6QREHP0S25VT9j3A2QuTsMvMBnhpVuWMdig0XwYk/R/JSwBibu9SfcRB59EEMLLs8ugnWThdCuw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This commit adds the any_dirty pointer as an optional parameter to folio_pte_batch() function. By using both the any_young and any_dirty pointers, madvise_free can make smarter decisions about whether to clear the PTEs when marking large folios as lazyfree. Suggested-by: David Hildenbrand Signed-off-by: Lance Yang Acked-by: David Hildenbrand --- mm/internal.h | 12 ++++++++++-- mm/madvise.c | 19 ++++++++++++++----- mm/memory.c | 4 ++-- 3 files changed, 26 insertions(+), 9 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index c6483f73ec13..daa59cef85d7 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -134,6 +134,8 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) * first one is writable. * @any_young: Optional pointer to indicate whether any entry except the * first one is young. + * @any_dirty: Optional pointer to indicate whether any entry except the + * first one is dirty. * * Detect a PTE batch: consecutive (present) PTEs that map consecutive * pages of the same large folio. @@ -149,18 +151,20 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) */ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags, - bool *any_writable, bool *any_young) + bool *any_writable, bool *any_young, bool *any_dirty) { unsigned long folio_end_pfn = folio_pfn(folio) + folio_nr_pages(folio); const pte_t *end_ptep = start_ptep + max_nr; pte_t expected_pte, *ptep; - bool writable, young; + bool writable, young, dirty; int nr; if (any_writable) *any_writable = false; if (any_young) *any_young = false; + if (any_dirty) + *any_dirty = false; VM_WARN_ON_FOLIO(!pte_present(pte), folio); VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio); @@ -176,6 +180,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, writable = !!pte_write(pte); if (any_young) young = !!pte_young(pte); + if (any_dirty) + dirty = !!pte_dirty(pte); pte = __pte_batch_clear_ignored(pte, flags); if (!pte_same(pte, expected_pte)) @@ -193,6 +199,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, *any_writable |= writable; if (any_young) *any_young |= young; + if (any_dirty) + *any_dirty |= dirty; nr = pte_batch_hint(ptep, pte); expected_pte = pte_advance_pfn(expected_pte, nr); diff --git a/mm/madvise.c b/mm/madvise.c index f5e3699e7b54..4597a3568e7e 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -321,6 +321,18 @@ static inline bool can_do_file_pageout(struct vm_area_struct *vma) file_permission(vma->vm_file, MAY_WRITE) == 0; } +static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end, + struct folio *folio, pte_t *ptep, + pte_t pte, bool *any_young, + bool *any_dirty) +{ + const fpb_t fpb_flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; + int max_nr = (end - addr) / PAGE_SIZE; + + return folio_pte_batch(folio, addr, ptep, pte, max_nr, fpb_flags, NULL, + any_young, any_dirty); +} + static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -456,13 +468,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, * next pte in the range. */ if (folio_test_large(folio)) { - const fpb_t fpb_flags = FPB_IGNORE_DIRTY | - FPB_IGNORE_SOFT_DIRTY; - int max_nr = (end - addr) / PAGE_SIZE; bool any_young; - nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, - fpb_flags, NULL, &any_young); + nr = madvise_folio_pte_batch(addr, end, folio, pte, + ptent, &any_young, NULL); if (any_young) ptent = pte_mkyoung(ptent); diff --git a/mm/memory.c b/mm/memory.c index 33d87b64d15d..9e07d1b9020c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -989,7 +989,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma flags |= FPB_IGNORE_SOFT_DIRTY; nr = folio_pte_batch(folio, addr, src_pte, pte, max_nr, flags, - &any_writable, NULL); + &any_writable, NULL, NULL); folio_ref_add(folio, nr); if (folio_test_anon(folio)) { if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page, @@ -1558,7 +1558,7 @@ static inline int zap_present_ptes(struct mmu_gather *tlb, */ if (unlikely(folio_test_large(folio) && max_nr != 1)) { nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, - NULL, NULL); + NULL, NULL, NULL); zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, nr, addr, details, rss, force_flush, From patchwork Thu Apr 18 10:57:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13634516 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CE8DC4345F for ; Thu, 18 Apr 2024 10:58:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C68656B009A; Thu, 18 Apr 2024 06:58:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C173B6B009B; Thu, 18 Apr 2024 06:58:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB86A6B009C; Thu, 18 Apr 2024 06:58:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8FB516B009A for ; Thu, 18 Apr 2024 06:58:41 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 20411A232B for ; Thu, 18 Apr 2024 10:58:41 +0000 (UTC) X-FDA: 82022354442.26.556C157 Received: from mail-ot1-f46.google.com (mail-ot1-f46.google.com [209.85.210.46]) by imf30.hostedemail.com (Postfix) with ESMTP id 5B34F80004 for ; Thu, 18 Apr 2024 10:58:39 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=X8uZxOUq; spf=pass (imf30.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.46 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713437919; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O8NkjRR32gwNlYRbH1lR13f1NodvtqYs34036pu1WAk=; b=TA8iB4j9iNNOErcV/wNZUGEacE4NxA/zDnb6+AhRA5NWbn7h/0cuyF+wbbnlf/dwPBf/SW CNVUQ2j12sf3Jfq5YakdxH7ucoiclGBgzeiZo9DumBrY4KwJ8ajNEeCXVzIaTDJXDHgx4q 6ghf58hoIvi9BG2sdP4gbffEO5zxilQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713437919; a=rsa-sha256; cv=none; b=rviWZSInkIX87ESavy+HXT2f/iaaacJtSXr/6fW8K+A9GRvY2gAXghcdqLokrBLuBnGsZ9 SHKG/W/qc6SH16f9AnxeC+DO13Wr6YfCVZno7G+S+OjY7E1EFl2aJVxxb4KZztdf+kt1zY LpVWrYgqKnXMRBKeRGlK+vPHazSV2OE= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=X8uZxOUq; spf=pass (imf30.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.46 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ot1-f46.google.com with SMTP id 46e09a7af769-6eb77e56b20so361246a34.3 for ; Thu, 18 Apr 2024 03:58:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713437918; x=1714042718; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O8NkjRR32gwNlYRbH1lR13f1NodvtqYs34036pu1WAk=; b=X8uZxOUqPfs5yTV+ccXRtN1XW+t72jGmT6bo+qvt8ZwPWM7bppv398jrJWeZlfKB03 CiNHmF2/vYShplIusWRgG5WlPcnDBB5vf4ek0euu1HR+FWlAplkigLiIEpy8ldagqyAU NrYRS5YCkbQweh36Lk55Jj0xJ/CBPzfsXbja1Fw2EVBLBJsV7+9FV4OnOmPN0ZSLlL3o ae4n9DHoNYERtv5dTxpAksx1dloqgotzsk/4HmcwcGb1ClcOK2IKbDWqqpVbwzIOxSvE xicKBY5q2xzluAH08DQYphLGu7hPvXCqQb7STLjIlp4Wav2ChzKvg0RBdGha6zPJaeuQ ktjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713437918; x=1714042718; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O8NkjRR32gwNlYRbH1lR13f1NodvtqYs34036pu1WAk=; b=dmK3ekbtyFf/FrsPof0HktvWWsFf//AhR0DnmhOq+2oyBjLnTOisRuiPZZ86iEMYDi nfLOWLDpl5GVUW3odJTGSbtx5GRzIyczEvgjbPdd/iWx60ie+yI3c2RYvvoRvGpe44Ga c4e2I6Hip9C3pkMJu9eoX0rChSPhEgvLW25K3Db5FwlYr/giP+e7Xeo9HKLkwDlQEbgU uUnLgsc78dcoZn52zzS+gcApAsWmrXe8cmVEfbPUhzu7OHzEuqeZhbja1Uw7gQsX4KZZ pxB7nI/H7IECgNHCY+zHJLPZzfQvwXp2NC+qRkY9rXbdyEEZseslVM56Bw3FiVe28esJ AabQ== X-Forwarded-Encrypted: i=1; AJvYcCVlqAMNLH+pX7dYid2CCHlyHChlviypLA42JS7fjY0zeniKYVL4YOunGrgXbfWdWH108lyJPsr0VrbpXxQu6YQEVIU= X-Gm-Message-State: AOJu0YzhcGOyJTVdz4Hg04WevCWygFEPBVP55vKC063ymufsUul6aN3K Oacvs+Mu276T/Dhxo+BlUB9Hm063D4BBMvYCpx0udJsuK5+8WqXQ X-Google-Smtp-Source: AGHT+IE0Wend3e5EpVQTSC9tsdUxn2WDCsJjwIeBC6hJ6WUDqPXTIzMmKsKYHmthb5COY6jTF/NdRg== X-Received: by 2002:a05:6870:7096:b0:22e:dfbc:4d9d with SMTP id v22-20020a056870709600b0022edfbc4d9dmr2903143oae.1.1713437917381; Thu, 18 Apr 2024 03:58:37 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id gd26-20020a056a00831a00b006ea923678a6sm1200487pfb.137.2024.04.18.03.58.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 03:58:37 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v9 4/4] mm/madvise: optimize lazyfreeing with mTHP in madvise_free Date: Thu, 18 Apr 2024 18:57:50 +0800 Message-Id: <20240418105750.98866-5-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240418105750.98866-1-ioworker0@gmail.com> References: <20240418105750.98866-1-ioworker0@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 5B34F80004 X-Rspam-User: X-Stat-Signature: s9fthwgqkced4u67qbeqcw8g86kx96a6 X-HE-Tag: 1713437919-1896 X-HE-Meta: U2FsdGVkX19VhDP91XpUo8E598obIQ7x4hfU+jLNncLoESbJvpRq3oLKXKMzK87LBv3pIel4Hd14Y83ZSi1tdEzKMkzK0Oz4XQ/XQcFgrZ+jdfhcfyK1mX4qjG60aQrEtjoAgSx5YFUUrn/ZezZUx3iR9YyVXB+ifdUnh+NyEOgK/6wnpAvXbr/BjThmSlt2pWRP+9RkMFHa5pwnaNyPUDYuhfDXenvlC3XMlve/SIyyC3i1/nMI/2lWgsCl8XsE8uf/Ni37TzntckFMwc5mHJcrpmOCWZJmenXhU2Arp5ogeD0wXjA794niA4lacfo4GkmOSqWYFKUqHOUU+CfHyCHnj6tORSoQB/At7uIPztYX+qE2ylecf8e5RW1mBvBqbkaRmtUsyWsdNYK9ieAdvt6Kgl7Kzz6wSzmN1o7mml+SmAIddm3xYnKBfEMZF/IUEGI5g90qC4yRLiGLxkOj+YBt5S2V59mk0uUdMFEuvuwp4Vnudmn5QtXBvk8z28rKogW0EfDKf6PbNayGXDIiWGhnifUS9hT6b0AufvsxZRaVs7vc9hf5V3JF65OaNfDyWy1/+sUWDbanUIrvWyc5jz0rq0UjYhLM87pXjdGxi5Y5Gc1ykQkx6QhX86POSmCMF+2jXasGIyZS+l6OioXc8wFh7s6EtqhYKVKa31D0aWrqPiDkXTu4U9hb7MCMtFyxVYTi70o3t/2E5RpAfCsLH02YGUOVe6gYSLLKqPnAU1Jd9NV2sVQiWmqDhKxSsUO+rMlYH17aMdkH0eLHDz53m9Lv115BBbwv5iIIOvoxqMCy5x24SHOafeY8REnktuxIrLwcaENlV0iuwzjw4Y2WqIL3M0d64o3+MUSYlDQKjR9eiovnFXt+vPK0Kn/JdklzmkJVtO7l2+Lc6I3FyrrOlplJcSz02Syjh8FRuiWUmeDJZ5lBT9JKoATsoAoSEaV61KHu66jdY8Ov2/E8QA1 RZt62GJl yilZOkGKmhQVvP9OsG9fExHsEHHnPLuHGr7cvyNOHBVEvunhttawUacydo32ChkrxOqYHlaoudbX7/QdtEhhIP38PWijCIUNKHHCf35Ax1dWsvEEUQcyudrWTzrBgWuqG/bh2PGmzgbiCXNNtxnMQb/+4DKSw6WX0oqwazNqmeDnFI2rPtayJJT9UqmlUNQZ+A2ze9sOel7p/5h8ZbnjvaHQ7HnRKQu9IHr4FxyyTHTS1QtmH6oLcaCDtFj6Z/22fE/t7NjlQs5Gm2R/bRY275lbWbsELCdWXIPdvXPM6zbSyJhzcMAjTjDLT5RlcWvlkG6DE4IUOmXkdVK++Y3x8LM5Lg0Ok4XiHTc3s4+A03cwJxeG8U6urX1MQCB8qWPgYkYTA5pqhREC6qQBC7flN1PcfCCppVzlX5IVB8+HrN2Pn/T5wGQLcpUad4tSQx9Fc52+Htpq8gMvwp5EPieXPt9R99Y+1K2PMv6/KLb9B6RZlqzaCzZ2zqDkoFoeldmUoyANZRnI0T26sEK+c960C8LuxA/QMgBVMC2zS+K/f1apmhTgCZW9EoJa3KSFMiTZGLv0xfskWMvSrZbxKDP/uiOfBzIhzYQgjltHHhIWMCaUCohC++QQAZutaWD9BaPieLjUZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch optimizes lazyfreeing with PTE-mapped mTHP[1] (Inspired by David Hildenbrand[2]). We aim to avoid unnecessary folio splitting if the large folio is fully mapped within the target range. If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities. On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better): Folio Size | Old | New | Change ------------------------------------------ 4KiB | 0.590251 | 0.590259 | 0% 16KiB | 2.990447 | 0.185655 | -94% 32KiB | 2.547831 | 0.104870 | -95% 64KiB | 2.457796 | 0.052812 | -97% 128KiB | 2.281034 | 0.032777 | -99% 256KiB | 2.230387 | 0.017496 | -99% 512KiB | 2.189106 | 0.010781 | -99% 1024KiB | 2.183949 | 0.007753 | -99% 2048KiB | 0.002799 | 0.002804 | 0% [1] https://lkml.kernel.org/r/20231207161211.2374093-5-ryan.roberts@arm.com [2] https://lore.kernel.org/linux-mm/20240214204435.167852-1-david@redhat.com Reviewed-by: Ryan Roberts Signed-off-by: Lance Yang Acked-by: David Hildenbrand --- mm/madvise.c | 85 +++++++++++++++++++++++++++------------------------- 1 file changed, 44 insertions(+), 41 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 4597a3568e7e..375ab3234603 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -643,6 +643,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) { + const cydp_t cydp_flags = CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY; struct mmu_gather *tlb = walk->private; struct mm_struct *mm = tlb->mm; struct vm_area_struct *vma = walk->vma; @@ -697,44 +698,57 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, continue; /* - * If pmd isn't transhuge but the folio is large and - * is owned by only this process, split it and - * deactivate all pages. + * If we encounter a large folio, only split it if it is not + * fully mapped within the range we are operating on. Otherwise + * leave it as is so that it can be marked as lazyfree. If we + * fail to split a folio, leave it in place and advance to the + * next pte in the range. */ if (folio_test_large(folio)) { - int err; + bool any_young, any_dirty; - if (folio_likely_mapped_shared(folio)) - break; - if (!folio_trylock(folio)) - break; - folio_get(folio); - arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(start_pte, ptl); - start_pte = NULL; - err = split_folio(folio); - folio_unlock(folio); - folio_put(folio); - if (err) - break; - start_pte = pte = - pte_offset_map_lock(mm, pmd, addr, &ptl); - if (!start_pte) - break; - arch_enter_lazy_mmu_mode(); - pte--; - addr -= PAGE_SIZE; - continue; + nr = madvise_folio_pte_batch(addr, end, folio, pte, + ptent, &any_young, NULL); + + if (nr < folio_nr_pages(folio)) { + int err; + + if (folio_likely_mapped_shared(folio)) + continue; + if (!folio_trylock(folio)) + continue; + folio_get(folio); + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + start_pte = NULL; + err = split_folio(folio); + folio_unlock(folio); + folio_put(folio); + start_pte = pte = + pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!start_pte) + break; + arch_enter_lazy_mmu_mode(); + if (!err) + nr = 0; + continue; + } + + if (any_young) + ptent = pte_mkyoung(ptent); + if (any_dirty) + ptent = pte_mkdirty(ptent); } if (folio_test_swapcache(folio) || folio_test_dirty(folio)) { if (!folio_trylock(folio)) continue; /* - * If folio is shared with others, we mustn't clear - * the folio's dirty flag. + * If we have a large folio at this point, we know it is + * fully mapped so if its mapcount is the same as its + * number of pages, it must be exclusive. */ - if (folio_mapcount(folio) != 1) { + if (folio_mapcount(folio) != folio_nr_pages(folio)) { folio_unlock(folio); continue; } @@ -750,19 +764,8 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, } if (pte_young(ptent) || pte_dirty(ptent)) { - /* - * Some of architecture(ex, PPC) don't update TLB - * with set_pte_at and tlb_remove_tlb_entry so for - * the portability, remap the pte with old|clean - * after pte clearing. - */ - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - - ptent = pte_mkold(ptent); - ptent = pte_mkclean(ptent); - set_pte_at(mm, addr, pte, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + clear_young_dirty_ptes(vma, addr, pte, nr, cydp_flags); + tlb_remove_tlb_entries(tlb, pte, nr, addr); } folio_mark_lazyfree(folio); }