From patchwork Thu Feb 23 08:31:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13149974 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D82DDC61DA4 for ; Thu, 23 Feb 2023 08:30:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DEA2B6B0074; Thu, 23 Feb 2023 03:30:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CF81C6B007E; Thu, 23 Feb 2023 03:30:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8B256B0080; Thu, 23 Feb 2023 03:30:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 904866B0074 for ; Thu, 23 Feb 2023 03:30:35 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 672611C6B6E for ; Thu, 23 Feb 2023 08:30:35 +0000 (UTC) X-FDA: 80497885230.27.368AC8E Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by imf06.hostedemail.com (Postfix) with ESMTP id 7068C180014 for ; Thu, 23 Feb 2023 08:30:33 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=m3fRzbbO; spf=pass (imf06.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677141033; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LPz2ETUja8kHTfsT0rbBvT0gxlRSH1hwzWLcea4rwfo=; b=As2wEm7KMNcwWwXCxyicClxAVTgag/bJasy+RJFZjqE47Q4oeJwCOJ1IHLPEFlP0KuxVLD MQsxRN+uFAR2RBK1ipxpH3ZTD1WQrAwB8AzbDNLk5MebXe71ZCj8+XHJSJP+L/7u3V09Bc ASDXDKRttV+r3cFK8VhJTQ8f0NfsMlk= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=m3fRzbbO; spf=pass (imf06.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677141033; a=rsa-sha256; cv=none; b=pxVGWy+s+zmFkAfe0+Y0dxU9wSj3JSsp/PmpVBqONtTJwJk1OuAmY2MfwVcdFM7snHPjHC cBYnRw8yHqLRKZJoGyzxYnT0N/VfWEdLyoJe2RxY0kz7tbV7xqeRgImDahnxR47Ia1Ef+s /Ptz5ZWIQE0Z0Bi++lGTd9HchxSfGOc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677141033; x=1708677033; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Frr3tP9iMcF/ayTmwJtPkPezLAtUZ5ZeFYQb1fE7bdE=; b=m3fRzbbOkP5HTJSfQYex5Zm5JX2ZehASxSnv9PVlCZzCnPnnwoFlXuq4 DPhTlDbmTfIy7YizZHEBD1v7URsJceprzdvueOixTvaWSyKTFWojIDdmR Ji9lGs9QgBl0boYkeiqgV46jNrINEdZj3iA0hkIad7rGyOVGCVdepZxzT bEHmadb+PhkwcoGaUJKm0BI/HlZdb4rXLvkcZyKprf5HyCyGueIRF1TkN RdCLs5OjjBsgRiM6sGgNS5/4BdN19sGyQZIsSxO0WJzyDpaUoWTlvJLac LMxrFrq/yT8OeX1gyorpvNq7pMooO/nZesbzZvxdtghmGFtUZhy8MfAqE Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10629"; a="321298502" X-IronPort-AV: E=Sophos;i="5.97,320,1669104000"; d="scan'208";a="321298502" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Feb 2023 00:30:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10629"; a="674437911" X-IronPort-AV: E=Sophos;i="5.97,320,1669104000"; d="scan'208";a="674437911" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga007.fm.intel.com with ESMTP; 23 Feb 2023 00:30:31 -0800 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org Cc: fengwei.yin@intel.com Subject: [PATCH 4/5] rmap:addd folio_remove_rmap_range() Date: Thu, 23 Feb 2023 16:31:59 +0800 Message-Id: <20230223083200.3149015-5-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230223083200.3149015-1-fengwei.yin@intel.com> References: <20230223083200.3149015-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7068C180014 X-Stat-Signature: djd9qfhsbgb8o9e119mhdan6wst5r7wa X-Rspam-User: X-HE-Tag: 1677141033-845092 X-HE-Meta: U2FsdGVkX18Psft3E2GvKKBrbJHOVsMugby/Z93IcIVNAYqizrJyjTislH8swDUOkjia3t+YuTtoc6BJofE4TEz1STua4qxIvEKPdWF8E3stn6KDY9pGkzonM4xzFk8mLqwRXvVCkdmNKDnM/tnm0/PfmF0QbQZg7LDuUphkXDfDB1ejg21DSDALStjMgYpo5Y8LwvY6ySH2hQfrWPPyARnvSDmmCvhHijb9Mdk/9msULGN0G5BPMa/0pB88W7ZuCQ8TcesNxq9smsUDgq07YgW0TdzXOW7feR31T37j7Xd4irUPwTJxLyKENBKRIHCtdqgU0acUmf200ESxpov8zz1yfahH2Ld0FW5ywaGdxwaQ37suOIkNwMj5i40m4vYXHz8OC0xg/aYBgUR47v7+4AWcVFCefPGCcKqA0nEFZRiA3x5y60zpJ3uPNsTYYTnE24SZQl2/tNaMkyX4qU/Og0eJZGjHiI4yD0dBKSC5/Ta9j0d8iqX4Az2f502lgfK1kSLcYf17VQE127Yp7cceRysTs+Q1RSDaIkPkCLBsJG+W+ypKER9JFP6rwkxKFOu1LidnaiDg65jC/3RUluzB0eYr1ho6P0H4zDLShxhSAN8vAECXsGHYuUW82uOV0r4GNMYrLyhCTIGdHXCh9e5OuvbpU/uafpW/ZfHhy2GhMARJ4PQ3HN58BDVP4NTDj6m7b1j34XEjKSJnGiOBMY+sqGGMETuE+Y2VK93g5ip6QLFc66K7iNZku97sWSOdiIvA1SqIJ0rfxUiBoFJqmPsZUWiP6YNjPO6b0kbPmhmu5uTqmAZmY+k/ZUSnY/UsWMTwbVUIY2l/+oW13x2VTfgevxN09AfIgdrlwOrbxsqEy3LHO4GOfPzH8zYClcgFGeJnzg888FVT8WRly3TA9NnxVUE8V9Que7Cm7e8xjmIMZjEeVCSZbGowBy1O+tdwBtkQAn+zb71Siok2JavYlFu Zn/1HK9H UwlTZUA7uXQE+lh3GIe8QcTIUEAN+8feslphNni7HL+odk/x2gEL7xd5+vxO0c9I2v1lnoALwHKC/7M5lV8RXATpPc299IYT6ktptrMJz9ymm3oEtqyIhxbOmM8H7pHpgaOyjAbEknVyjjdV9FEapSdPKRtgYA/nxfZAznXQGkVDNfn5OoElEetwbrhKG7AR+lnV0DHuEPmIVSHgPhihjMdvCaSsBWAEeSLDrnmmcMBgnQopgbBLGNd0BjElzjHuJZOb0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: folio_remove_rmap_range() allows to take down the pte mapping to a specific range of folio. Comparing to page_remove_rmap(), it batched updates __lruvec_stat for large folio. Signed-off-by: Yin Fengwei --- include/linux/rmap.h | 4 +++ mm/rmap.c | 58 +++++++++++++++++++++++++++++++++----------- 2 files changed, 48 insertions(+), 14 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index a4570da03e58..d7a51b96f379 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -200,6 +200,10 @@ void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_remove_rmap_range(struct folio *, struct page *, + unsigned int nr_pages, struct vm_area_struct *, + bool compound); + void hugepage_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); diff --git a/mm/rmap.c b/mm/rmap.c index 097774c809a0..3680765b7ec8 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1357,23 +1357,25 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, } /** - * page_remove_rmap - take down pte mapping from a page - * @page: page to remove mapping from + * folio_remove_rmap_range - take down pte mapping from a range of pages + * @folio: folio to remove mapping from + * @page: The first page to take down pte mapping + * @nr_pages: The number of pages which will be take down pte mapping * @vma: the vm area from which the mapping is removed * @compound: uncharge the page as compound or small page * * The caller needs to hold the pte lock. */ -void page_remove_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) +void folio_remove_rmap_range(struct folio *folio, struct page *page, + unsigned int nr_pages, struct vm_area_struct *vma, + bool compound) { - struct folio *folio = page_folio(page); atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool last; + int nr = 0, nr_pmdmapped = 0, last; enum node_stat_item idx; - VM_BUG_ON_PAGE(compound && !PageHead(page), page); + VM_BUG_ON_FOLIO(compound && (nr_pages != folio_nr_pages(folio)), folio); + VM_BUG_ON_FOLIO(compound && (page != &folio->page), folio); /* Hugetlb pages are not counted in NR_*MAPPED */ if (unlikely(folio_test_hugetlb(folio))) { @@ -1384,12 +1386,16 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, /* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely(!compound)) { - last = atomic_add_negative(-1, &page->_mapcount); - nr = last; - if (last && folio_test_large(folio)) { - nr = atomic_dec_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } + do { + last = atomic_add_negative(-1, &page->_mapcount); + if (last && folio_test_large(folio)) { + last = atomic_dec_return_relaxed(mapped); + last = (last < COMPOUND_MAPPED); + } + + if (last) + nr++; + } while (page++, --nr_pages > 0); } else if (folio_test_pmd_mappable(folio)) { /* That test is redundant: it's for safety or to optimize out */ @@ -1443,6 +1449,30 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, munlock_vma_folio(folio, vma, compound); } +/** + * page_remove_rmap - take down pte mapping from a page + * @page: page to remove mapping from + * @vma: the vm area from which the mapping is removed + * @compound: uncharge the page as compound or small page + * + * The caller needs to hold the pte lock. + */ +void page_remove_rmap(struct page *page, struct vm_area_struct *vma, + bool compound) +{ + struct folio *folio = page_folio(page); + unsigned int nr_pages; + + VM_BUG_ON_FOLIO(compound && (page != &folio->page), folio); + + if (likely(!compound)) + nr_pages = 1; + else + nr_pages = folio_nr_pages(folio); + + folio_remove_rmap_range(folio, page, nr_pages, vma, compound); +} + static bool try_to_unmap_one_hugetlb(struct folio *folio, struct vm_area_struct *vma, struct mmu_notifier_range range, struct page_vma_mapped_walk pvmw, unsigned long address,