From patchwork Wed Jul 19 13:54:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13319021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 764BBEB64DA for ; Wed, 19 Jul 2023 13:55:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFAB028005F; Wed, 19 Jul 2023 09:55:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B844428004C; Wed, 19 Jul 2023 09:55:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A489D28005F; Wed, 19 Jul 2023 09:55:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 93E4928004C for ; Wed, 19 Jul 2023 09:55:11 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 55B82C029D for ; Wed, 19 Jul 2023 13:55:11 +0000 (UTC) X-FDA: 81028508022.15.A91CB64 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf03.hostedemail.com (Postfix) with ESMTP id 4D7722001B for ; Wed, 19 Jul 2023 13:55:08 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf03.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689774909; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2dqIz3C526jfk09wr8sX5z1Ztv6RWuu5WBWCEVsOmTA=; b=zAmF/HGimKE8Zlw/8HD+XWz0WJcDTYmAYjZ5HOk/9oPtoCZtSg1sPWRG7XzeLUhJljQEW6 NeR1pGriHhgNYRs8qgZHX4O3XqlX/4mj0I8RI3thKxHMMKZ0O/wM6uYioAj1E6CHjmMSfs 23XW7Cj6Uyl1sWfXX+8FgOq2LMuLPt0= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf03.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689774909; a=rsa-sha256; cv=none; b=kzEKfiTQtd3KXwPmiQjwBwbPRX8RvB+V/AD3G3jCz5sebe9hLH22uAZUGZsLWivctQMIb4 NSA9Ya17gif+058EXpvUx4+qQiqmCesFSQ7cUC/efgB0t2AQxzfXowrB/Y9NlN9dspd0/Q WtOAG/mufsvG2a89XRuGHiXmUHGHVcw= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4238512FC; Wed, 19 Jul 2023 06:55:51 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 672B43F6C4; Wed, 19 Jul 2023 06:55:06 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Yang Shi , "Huang, Ying" , Zi Yan Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 2/3] mm: Implement folio_remove_rmap_range() Date: Wed, 19 Jul 2023 14:54:49 +0100 Message-Id: <20230719135450.545227-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230719135450.545227-1-ryan.roberts@arm.com> References: <20230719135450.545227-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 8h7f1yxfgnkpk7i1xfh5hmsrpe81uz73 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4D7722001B X-HE-Tag: 1689774908-686510 X-HE-Meta: U2FsdGVkX1/tNk+/gfmVcdSj/lqgVHTC0S9cfhFNpt4V+tpv3vdMhwblguUjgQhvDEgJKJAcpAzRMw2fRHRq+AryjVVpbG4eLGPPpfTn7uPYunRjHF+aC7iW0KarkpomeUqkGr1lWl49341Q8ID+X0SPeslxj7Zy+vM2O/q56jwvufbuP1gagcYblQeL5MCnRvXIk1dqDnNIYBxvjkXHjoobpP3zWpt6g71pi+Rwt6NsFGcIebQgJj2Q5aCRHQ74qDJxFBk0URThxOuF59RwAG009zoJSpmN1fSBYyU6oOxiX68XLtmuAMMImVfaTV3sppLR/0tdBUpzBUq0wfoDwCUGiNVENdPg7/OCjXfyvGV661bgX9c9ML1aekZ8gwkkPSbc0FSjseWNi+xDpI7bcxqzflR8zSbtKYi0Gsc4Swp03xbRLb58NQyTygdpBb23CCXJP6qjf9OVTg2kZrTqqTfIbx7xJjLCAkPkgdW7m0UIeTR4QZd27Tnb7lSpB7hLTJ6pr1+n0R8Ig2VAe21aB9fgOp8QfUIZBZ3VlJVOwP7+8KyqUs113yThPoOYA7foMJpBvKKHKRht/oqM5UgW2zXU7ufgANrcfMyI1pSmt4qyDomvWxER26E98TU4KxGDB9gfv3jWdaWM/MUyzeVOUWrBnYqZ2sSgjF0T0E0IKTnJMrre++/Tz1lmSZz/Y/LQ1jIZ+8aePtSvnki+JXe900+wzYBaZfWetzR3kqr/ggB7cTtwZ1xlsRY7CTs8JmmxdVXyYBTiC6UnNN42lRvZu0hGse6ysXGGocT709AQ9nGEKw2vRk/8u2LIJMgN4cNkEjurAVHKKmAlVXDnu7BM+oo3fS4xLsnXdMFP30M9Bo7+AhjN43RwOhJHOxMGDH7ycOY+yMhBoHQZ7KZ9S+dEBsXVC9gIOuEun3angmoqy99vi9u3X86ftAPHhX/PPt0mLMnTVC8xHBSoDNWewYS e+NnoUpy NWpYmDwGti7N8TR7afWbZ5n3Ys0E2iI6O22iDzpQ3qjLeVZCqGmTBeIpcFiEIpIio+NQGziyga618O2m7+iWNpgV86m9KFRY1j6hstdyyv4VNxLPw+uaFq0ZEI16GmYDAkEutQ+xfHLG3rGv626IQYHtc/v1YVDwnzQtEV+Cupl/4pzDFmf5ci7ugYOWcvVBH22G3uDFltI4vE79NfHAysDfEEEE1X3MmDSfFGBAt+ENtvDiwpprEExOTTw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like page_remove_rmap() but batch-removes the rmap for a range of pages belonging to a folio. This can provide a small speedup due to less manipuation of the various counters. But more crucially, if removing the rmap for all pages of a folio in a batch, there is no need to (spuriously) add it to the deferred split list, which saves significant cost when there is contention for the split queue lock. All contained pages are accounted using the order-0 folio (or base page) scheme. Reviewed-by: Yin Fengwei Reviewed-by: Zi Yan Signed-off-by: Ryan Roberts --- include/linux/rmap.h | 2 ++ mm/rmap.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b87d01660412..f578975c12c0 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -200,6 +200,8 @@ void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_remove_rmap_range(struct folio *folio, struct page *page, + int nr, struct vm_area_struct *vma); void hugepage_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); diff --git a/mm/rmap.c b/mm/rmap.c index eb0bb00dae34..4022a3ab73a8 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1359,6 +1359,78 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, mlock_vma_folio(folio, vma, compound); } +/** + * folio_remove_rmap_range - Take down PTE mappings from a range of pages. + * @folio: Folio containing all pages in range. + * @page: First page in range to unmap. + * @nr: Number of pages to unmap. + * @vma: The VM area containing the range. + * + * All pages in the range must belong to the same VMA & folio. They + * must be mapped with PTEs, not a PMD. + * + * Context: Caller holds the pte lock. + */ +void folio_remove_rmap_range(struct folio *folio, struct page *page, + int nr, struct vm_area_struct *vma) +{ + atomic_t *mapped = &folio->_nr_pages_mapped; + int nr_unmapped = 0; + int nr_mapped; + bool last; + enum node_stat_item idx; + + if (unlikely(folio_test_hugetlb(folio))) { + VM_WARN_ON_FOLIO(1, folio); + return; + } + + VM_WARN_ON_ONCE(page < &folio->page || + page + nr > (&folio->page + folio_nr_pages(folio))); + + if (!folio_test_large(folio)) { + /* Is this the page's last map to be removed? */ + last = atomic_add_negative(-1, &page->_mapcount); + nr_unmapped = last; + } else { + for (; nr != 0; nr--, page++) { + /* Is this the page's last map to be removed? */ + last = atomic_add_negative(-1, &page->_mapcount); + if (last) + nr_unmapped++; + } + + /* Pages still mapped if folio mapped entirely */ + nr_mapped = atomic_sub_return_relaxed(nr_unmapped, mapped); + if (nr_mapped >= COMPOUND_MAPPED) + nr_unmapped = 0; + } + + if (nr_unmapped) { + idx = folio_test_anon(folio) ? NR_ANON_MAPPED : NR_FILE_MAPPED; + __lruvec_stat_mod_folio(folio, idx, -nr_unmapped); + + /* + * Queue anon large folio for deferred split if at least one + * page of the folio is unmapped and at least one page is still + * mapped. + */ + if (folio_test_large(folio) && + folio_test_anon(folio) && nr_mapped) + deferred_split_folio(folio); + } + + /* + * It would be tidy to reset folio_test_anon mapping when fully + * unmapped, but that might overwrite a racing page_add_anon_rmap + * which increments mapcount after us but sets mapping before us: + * so leave the reset to free_pages_prepare, and remember that + * it's only reliable while mapped. + */ + + munlock_vma_folio(folio, vma, false); +} + /** * page_remove_rmap - take down pte mapping from a page * @page: page to remove mapping from